[{"id":5374,"web_url":"https://patchwork.libcamera.org/comment/5374/","msgid":"<35bd63f1-45e4-8b5d-b914-84375b662ce9@ideasonboard.com>","date":"2020-06-24T09:34:39","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Show,\n\nThank you, I'm quite excited to see this update, and I know Niklas is\nkeen for this to get in too, as he uses a Rockchip target mostly so this\nwill improve the performances for him quite a lot.\n\nMost of my comments below are stylistic, (but theres a couple of other\ntopics too), so as suggested below, could you install the checkstyle.py\nhook as a git commit hook please?\n\nI personally prefer having it as a 'post-commit' hook, rather than a\npre-commit so that it's easier to 'ignore' suggestions that I don't care\nfor ;-)\n\n--\nKieran\n\n\nOn 24/06/2020 08:37, Show Liu wrote:\n> The patch is to render the NV family YUV formats by OpenGL shader.\n> V3: refine the fragment shader for better pixel color.\n> \n> Signed-off-by: Show Liu \n> ---\n> src/qcam/main.cpp | 2 +\n> src/qcam/main_window.cpp | 19 ++-\n> src/qcam/main_window.h | 3 +-\n> src/qcam/meson.build | 2 +\n> src/qcam/shader.h | 104 ++++++++++++\n> src/qcam/viewfinder.cpp | 18 +-\n> src/qcam/viewfinder.h | 23 ++-\n> src/qcam/viewfinderGL.cpp | 335 ++++++++++++++++++++++++++++++++++++++\n> src/qcam/viewfinderGL.h | 101 ++++++++++++\n> 9 files changed, 593 insertions(+), 14 deletions(-)\n> create mode 100644 src/qcam/shader.h\n> create mode 100644 src/qcam/viewfinderGL.cpp\n> create mode 100644 src/qcam/viewfinderGL.h\n> \n> diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> index b3468cb..a3ea471 100644\n> --- a/src/qcam/main.cpp\n> +++ b/src/qcam/main.cpp\n> @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc, char *argv[])\n> \t\t\t \"help\");\n> \tparser.addOption(OptStream, &streamKeyValue,\n> \t\t\t \"Set configuration of a camera stream\", \"stream\", true);\n> +\tparser.addOption(OptOpenGL, OptionNone, \"Render YUV formats frame via OpenGL shader\",\n> +\t\t\t \"opengl\");\n\n(Just a question to everyone) - Should we default to the OpenGL\nviewfinder if open-gl is available, and fall back to the QWidget version\notherwise? (We can always do such actions on top/later if we choose.)\n\n\n> \n> \tOptionsParser::Options options = parser.parse(argc, argv);\n> \tif (options.isSet(OptHelp))\n> diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> index 7bc1360..6edf370 100644\n> --- a/src/qcam/main_window.cpp\n> +++ b/src/qcam/main_window.cpp\n> @@ -28,6 +28,9 @@\n> #include \n> \n> #include \"dng_writer.h\"\n> +#include \"main_window.h\"\n> +#include \"viewfinder.h\"\n> +#include \"viewfinderGL.h\"\n> \n> using namespace libcamera;\n> \n> @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm, const OptionsParser::Options &options)\n> \tsetWindowTitle(title_);\n> \tconnect(&titleTimer_, SIGNAL(timeout()), this, SLOT(updateTitle()));\n> \n> -\tviewfinder_ = new ViewFinder(this);\n> -\tconnect(viewfinder_, &ViewFinder::renderComplete,\n> -\t\tthis, &MainWindow::queueRequest);\n> -\tsetCentralWidget(viewfinder_);\n> +\tif (options_.isSet(OptOpenGL)) {\n> +\t\tviewfinder_ = new ViewFinderGL(this);\n> +\t\tconnect(dynamic_cast(viewfinder_), &ViewFinderGL::renderComplete,\n> +\t\t\t\tthis, &MainWindow::queueRequest);\n\ncheckstyle.py highlights that the indentation of the second line should\nbe pulled back, and I agree:\n \t\tconnect(dynamic_cast(viewfinder_),\n&ViewFinderGL::renderComplete,\n-\t\t\t\tthis, &MainWindow::queueRequest);\n+\t\t\tthis, &MainWindow::queueRequest);\n\n\n\n> +\t\tsetCentralWidget(dynamic_cast(viewfinder_));\n\nDoes the setCentralWidget need to have the dynamic_cast? Or can it just\nbe a call to\n\n\tsetCentralWidget(viewfinder_);\n\nafter this conditional block?\n\n\nPerhaps if the base viewfinder class had a method to call which handled\nthe connect, that could remove casting requirements there - but it would\nprobably end up needing to deal with complex template things for the\nsignal/slot parsing ... so I suspect that part isn't worth it...\n\n\n> +\t} else {\n> +\t\tviewfinder_ = new ViewFinder(this);\n> +\t\tconnect(dynamic_cast(viewfinder_), &ViewFinder::renderComplete,\n> +\t\t\t\tthis, &MainWindow::queueRequest);\n\nSame indentation fault here.\n\nIf you add the checkstyle.py hook to your libcamera sources, you'll get\nthese notifications as you commit (you can then decide if checkstyle was\nright or not ;D):\n\n $ cp utils/hooks/post-commit .git/hooks/post-commit\n\n> +\t\tsetCentralWidget(dynamic_cast(viewfinder_));\n> +\t}\n> +\n> \tadjustSize();\n> \n> \t/* Hotplug/unplug support */\n> diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> index 4606fe4..a852ef4 100644\n> --- a/src/qcam/main_window.h\n> +++ b/src/qcam/main_window.h\n> @@ -38,6 +38,7 @@ enum {\n> \tOptCamera = 'c',\n> \tOptHelp = 'h',\n> \tOptStream = 's',\n> +\tOptOpenGL = 'o',\n> };\n> \n> class CaptureRequest\n> @@ -102,7 +103,7 @@ private:\n> \tQAction *startStopAction_;\n> \tQComboBox *cameraCombo_;\n> \tQAction *saveRaw_;\n> -\tViewFinder *viewfinder_;\n> +\tViewFinderHandler *viewfinder_;\n\nHandler?\n\nI guess I'll see below, but I think I'd expect the base class interface\nto be 'ViewFinder', and then make specific derived implementations of that?\n\n> \n> \tQIcon iconPlay_;\n> \tQIcon iconStop_;\n> diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> index 045db52..6a58947 100644\n> --- a/src/qcam/meson.build\n> +++ b/src/qcam/meson.build\n> @@ -7,11 +7,13 @@ qcam_sources = files([\n> 'main.cpp',\n> 'main_window.cpp',\n> 'viewfinder.cpp',\n> + 'viewfinderGL.cpp'\n> ])\n> \n> qcam_moc_headers = files([\n> 'main_window.h',\n> 'viewfinder.h',\n> + 'viewfinderGL.h'\n> ])\n> \n> qcam_resources = files([\n> diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> new file mode 100644\n> index 0000000..f54c264\n> --- /dev/null\n> +++ b/src/qcam/shader.h\n> @@ -0,0 +1,104 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Linaro\n> + *\n> + * shader.h - YUV format converter shader source code\n> + */\n> +#ifndef __SHADER_H__\n> +#define __SHADER_H__\n> +\n> +/* Vertex shader for NV formats */\n> +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n> + \"attribute vec2 textureIn;\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"gl_Position = vertexIn;\\n\"\n> + \"textureOut = textureIn;\\n\"\n> + \"}\\n\";\n\nI'd pull that indentation back so that they all match, including moving\nthe first line to a new line to match...\n\nchar NV_Vertex_shader[] =\n\t\"attribute vec4 vertexIn;\\n\"\n \"attribute vec2 textureIn;\\n\"\n \"varying vec2 textureOut;\\n\"\n \"void main(void)\\n\"\n \"{\\n\"\n \"gl_Position = vertexIn;\\n\"\n \"textureOut = textureIn;\\n\"\n \"}\\n\";\n\n\n\n> +\n> +/* Fragment shader for NV12, NV16 and NV24 */\n> +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n\nAnd ofcourse be consistent with the code block indentation style throughout.\n\ncheckstyle might complain whatever you do here, so I would take\ncheckstyle with a pinch of salt and just make sure the file is consitent\nand well laid out.\n\n\n\n> + \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> + \"}\\n\";\n> +\n> +/* Fragment shader for NV21, NV61 and NV42 */\n> +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> + \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> + \"}\\n\";\n> +\n> +/* Fragment shader for YUV420 */\n> +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> + \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"uniform sampler2D tex_v;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> + \"}\\n\";\n> +\n> +/* Fragment shader for YVU420 */\n> +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"uniform sampler2D tex_v;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> + \"}\\n\";\n> +#endif // __SHADER_H__\n\nI can't comment on the shaders without a lot of research into shaders...\nso I'm going to say \"Yay, if this black magic works, then it works for\nme\" ;-D\n\n\n> diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> index dcf8a15..d3a2422 100644\n> --- a/src/qcam/viewfinder.cpp\n> +++ b/src/qcam/viewfinder.cpp\n> @@ -33,22 +33,30 @@ static const QMap nativeFormats\n> \t{ libcamera::formats::BGR888, QImage::Format_RGB888 },\n> };\n> \n> -ViewFinder::ViewFinder(QWidget *parent)\n> -\t: QWidget(parent), buffer_(nullptr)\n> +ViewFinderHandler::ViewFinderHandler()\n> {\n> -\ticon_ = QIcon(\":camera-off.svg\");\n> }\n> \n> -ViewFinder::~ViewFinder()\n> +ViewFinderHandler::~ViewFinderHandler()\n> {\n> }\n> \n> -const QList &ViewFinder::nativeFormats() const\n> +const QList &ViewFinderHandler::nativeFormats() const\n> {\n> \tstatic const QList formats = ::nativeFormats.keys();\n> \treturn formats;\n> }\n> \n> +ViewFinder::ViewFinder(QWidget *parent)\n> +\t: QWidget(parent), buffer_(nullptr)\n> +{\n> +\ticon_ = QIcon(\":camera-off.svg\");\n> +}\n> +\n> +ViewFinder::~ViewFinder()\n> +{\n> +}\n> +\n> int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> \t\t\t const QSize &size)\n> {\n> diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> index 26a1320..741d694 100644\n> --- a/src/qcam/viewfinder.h\n> +++ b/src/qcam/viewfinder.h\n> @@ -13,6 +13,8 @@\n> #include \n> #include \n> #include \n> +#include \n> +#include \n\nShouldn't those includes be in\nsrc/qcam/viewfinderGL.cpp or src/qcam/viewfinderGL.h only?\n\nAlso - they should be alphabetical order.\n (Checkstyle will highlight that for you).\n\n> #include \n> #include \n> \n> @@ -28,7 +30,23 @@ struct MappedBuffer {\n> \tsize_t size;\n> };\n> \n> -class ViewFinder : public QWidget\n> +class ViewFinderHandler\n\nI think it's just naming, but I would have called this ViewFinder,...\n\n> +{\n> +public:\n> +\tViewFinderHandler();\n> +\tvirtual ~ViewFinderHandler();\n> +\n> +\tconst QList &nativeFormats() const;\n> +\n> +\tvirtual int setFormat(const libcamera::PixelFormat &format, const QSize &size) =0;\n> +\tvirtual void render(libcamera::FrameBuffer *buffer, MappedBuffer *map) =0;\n> +\tvirtual void stop() =0;\n> +\n> +\tvirtual QImage getCurrentImage() =0;\n\non each of those, a space after the = looks better:\n functionName() = 0;\n\n\n> +\n> +};\n> +\n> +class ViewFinder : public QWidget, public ViewFinderHandler\n\nAnd this, ViewFinderQT or ViewFinderQWidget? (probably the later..)\n\nTo keep consistent with coding style, I think the viewfinder.h would\nthen only contain the base interface, and create a new file+header for\nthe QWidget version.\n\nOr maybe that's over kill, as the QWidget version would be the 'default'\nViewFinder anyway ...\n\n\n> {\n> \tQ_OBJECT\n> \n> @@ -36,8 +54,6 @@ public:\n> \tViewFinder(QWidget *parent);\n> \t~ViewFinder();\n> \n> -\tconst QList &nativeFormats() const;\n> -\n> \tint setFormat(const libcamera::PixelFormat &format, const QSize &size);\n> \tvoid render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> \tvoid stop();\n> @@ -67,5 +83,4 @@ private:\n> \tQImage image_;\n> \tQMutex mutex_; /* Prevent concurrent access to image_ */\n> };\n> -\n> #endif /* __QCAM_VIEWFINDER__ */\n> diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> new file mode 100644\n> index 0000000..7b47beb\n> --- /dev/null\n> +++ b/src/qcam/viewfinderGL.cpp\n> @@ -0,0 +1,335 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Linaro\n> + *\n> + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> + */\n> +\n> +#include \"shader.h\"\n> +#include \"viewfinderGL.h\"\n> +\n> +#include \n\nQImage is provided in viewfinder.h ... And do you actually use it in here?\n\n\n> +\n> +#include \n> +\n> +#define ATTRIB_VERTEX 0\n> +#define ATTRIB_TEXTURE 1\n> +\n> +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> +\t: QOpenGLWidget(parent),\n> +\tglBuffer(QOpenGLBuffer::VertexBuffer),\n> +\tpFShader(nullptr),\n> +\tpVShader(nullptr),\n> +\ttextureU(QOpenGLTexture::Target2D),\n> +\ttextureV(QOpenGLTexture::Target2D),\n> +\ttextureY(QOpenGLTexture::Target2D),\n> +\tyuvDataPtr(nullptr)\n\nCheckstyle will tell you to align those vertically with the first\nQOpenGLWidget(parent), rather than the ':'.\n\n\n> +\n> +{\n> +}\n> +\n> +ViewFinderGL::~ViewFinderGL()\n> +{\n> +\tremoveShader();\n> +\n> +\tif(textureY.isCreated())\n\nSpace after if: \"if ()\"\n\n> +\t\ttextureY.destroy();\n> +\n> +\tif(textureU.isCreated())\n> +\t\ttextureU.destroy();\n> +\n> +\tif(textureV.isCreated())\n> +\t\ttextureV.destroy();\n\nFor each of those ;-)\n\nThere's plenty more too, but I'll let you find them with checkstyle.\n\n\n> +\n> +\tglBuffer.destroy();\n> +}\n> +\n> +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> +\t\t\t const QSize &size)\n> +{\n> +\tformat_ = format;\n> +\tsize_ = size;\n> +\n> +\tupdateGeometry();\n> +\treturn 0;\n> +}\n> +\n> +void ViewFinderGL::stop()\n> +{\n> +\tif (buffer_) {\n> +\t\trenderComplete(buffer_);\n> +\t\tbuffer_ = nullptr;\n> +\t}\n> +}\n> +\n> +QImage ViewFinderGL::getCurrentImage()\n> +{\n> +\tQMutexLocker locker(&mutex_);\n> +\n> +\treturn(grabFramebuffer());\n\nI think that could be :\n\n\treturn grabFrameBuffer();\n\n> +}\n> +\n> +void ViewFinderGL::render(libcamera::FrameBuffer *buffer, MappedBuffer *map)\n> +{\n> +\tif (buffer->planes().size() != 1) {\n> +\t\tqWarning() << \"Multi-planar buffers are not supported\";\n> +\t\treturn;\n> +\t}\n> +\n> +\tunsigned char *memory = static_cast(map->memory);\n> +\tif(memory) {\n> +\t\tyuvDataPtr = memory;\n> +\t\tupdate();\n> +\t\tbuffer_ = buffer;\n> +\t}\n> +\n> +\tif (buffer)\n> +\t\trenderComplete(buffer);\n> +}\n> +\n> +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> +{\n> +\tif(buffer) {\n> +\t\tyuvDataPtr = buffer;\n> +\t\tupdate();\n> +\t}\n> +}\n> +\n> +void ViewFinderGL::setFrameSize(int width, int height)\n\nCan width/height be negative? Maybe the should be unsigned int?\n\nWhat about using a QSize too rather than individually passing width/height?\n\nI can't see what calls setFrameSize(), is this an override from\n\n> +{\n> +\tif(width > 0 && height > 0) {\n> +\t\twidth_ = width;\n> +\t\theight_ = height;\n> +\t}\n\nIn fact, don't we already have a size_ in the class? (which could also\nbe in the base class too most likely...\n\n\n> +}\n> +\n> +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> +{\n> +\tchar *fsrc = nullptr;\n> +\n> +\tswitch(format) {\n> +\tcase libcamera::formats::NV12:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 2;\n> +\t\tfsrc = NV_2_planes_UV;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV21:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 2;\n> +\t\tfsrc = NV_2_planes_VU;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV16:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_UV;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV61:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_VU;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV24:\n> +\t\thorzSubSample_ = 1;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_UV;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV42:\n> +\t\thorzSubSample_ = 1;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_VU;\n> +\t\tbreak;\n> +\tcase libcamera::formats::YUV420:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 2;\n> +\t\tfsrc = NV_3_planes_UV;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tbreak;\n> +\t};\n> +\n> +\tif(fsrc == nullptr) {\n> +\t\tqDebug() << __func__ << \"[ERROR]:\" <<\" No suitable fragment shader can be select.\";\n\n'selected'\n\n\n> +\t}\n> +\treturn fsrc;\n> +}\n> +\n> +void ViewFinderGL::createFragmentShader()\n> +{\n> +\tbool bCompile;\n> +\n> +\tpFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> +\tchar *fsrc = selectFragmentShader(format_);\n> +\n> +\tbCompile = pFShader->compileSourceCode(fsrc);\n> +\tif(!bCompile)\n> +\t{\n> +\t\tqDebug() << __func__ << \":\" << pFShader->log();\n> +\t}\n> +\n> +\tshaderProgram.addShader(pFShader);\n> +\n> +\t// Link shader pipeline\n\nEven though it's C++, we usually use C style comments...\n\n\n> +\tif (!shaderProgram.link()) {\n> +\t\tqDebug() << __func__ << \": shader program link failed.\\n\" << shaderProgram.log();\n> +\t\tclose();\n> +\t}\n> +\n> +\t// Bind shader pipeline for use\n> +\tif (!shaderProgram.bind()) {\n> +\t\tqDebug() << __func__ << \": shader program binding failed.\\n\" << shaderProgram.log();\n> +\t\tclose();\n> +\t}\n> +\n> +\tshaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> +\tshaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> +\n> +\tshaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> +\tshaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> +\n> +\ttextureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> +\ttextureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> +\ttextureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> +\n> +\tif(!textureY.isCreated())\n> +\t\ttextureY.create();\n> +\n> +\tif(!textureU.isCreated())\n> +\t\ttextureU.create();\n> +\n> +\tif(!textureV.isCreated())\n> +\t\ttextureV.create();\n> +\n> +\tid_y = textureY.textureId();\n> +\tid_u = textureU.textureId();\n> +\tid_v = textureV.textureId();\n> +}\n> +\n> +void ViewFinderGL::configureTexture(unsigned int id)\n> +{\n> +\tglBindTexture(GL_TEXTURE_2D, id);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);\n> +}\n> +\n> +void ViewFinderGL::removeShader()\n> +{\n> +\tif (shaderProgram.isLinked()) {\n> +\t\tshaderProgram.release();\n> +\t\tshaderProgram.removeAllShaders();\n> +\t}\n> +\n> +\tif(pFShader)\n> +\t\tdelete pFShader;\n> +\n> +\tif(pVShader)\n> +\t\tdelete pVShader;\n> +}\n> +\n> +void ViewFinderGL::initializeGL()\n> +{\n> +\tbool bCompile;\n> +\n> +\tinitializeOpenGLFunctions();\n> +\tglEnable(GL_TEXTURE_2D);\n> +\tglDisable(GL_DEPTH_TEST);\n> +\n> +\tstatic const GLfloat vertices[] {\n> +\t\t-1.0f,-1.0f,\n> +\t\t-1.0f,+1.0f,\n> +\t\t+1.0f,+1.0f,\n> +\t\t+1.0f,-1.0f,\n> +\t\t0.0f,1.0f,\n> +\t\t0.0f,0.0f,\n> +\t\t1.0f,0.0f,\n> +\t\t1.0f,1.0f,\n\nWhen you get here, checkstyle will suggest putting this all in one\nvertical column.\n\nIgnore it... (checkstyle is just advice, and in this case your layout is\nbetter).\n\nThough I would add a space at least after the first ',' on each line to\nmake the columns clearer., Or go further and align the 'f's... ?\n\n> +\t};\n> +\n> +\tglBuffer.create();\n> +\tglBuffer.bind();\n> +\tglBuffer.allocate(vertices,sizeof(vertices));\n> +\n> +\t/* Create Vertex Shader */\n> +\tpVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> +\tchar *vsrc = NV_Vertex_shader;\n> +\n> +\tbCompile = pVShader->compileSourceCode(vsrc);\n> +\tif(!bCompile) {\n> +\t\tqDebug() << __func__<< \":\" << pVShader->log();\n> +\t}\n> +\n> +\tshaderProgram.addShader(pVShader);\n> +\n> +\tglClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> +}\n> +\n> +void ViewFinderGL::render()\n> +{\n> +\tswitch(format_) {\n> +\tcase libcamera::formats::NV12:\n> +\tcase libcamera::formats::NV21:\n> +\tcase libcamera::formats::NV16:\n> +\tcase libcamera::formats::NV61:\n> +\tcase libcamera::formats::NV24:\n> +\tcase libcamera::formats::NV42:\n> +\t\t/* activate texture 0 */\n> +\t\tglActiveTexture(GL_TEXTURE0);\n> +\t\tconfigureTexture(id_y);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> +\t\tglUniform1i(textureUniformY, 0);\n> +\n> +\t\t/* activate texture 1 */\n> +\t\tglActiveTexture(GL_TEXTURE1);\n> +\t\tconfigureTexture(id_u);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> +\t\tglUniform1i(textureUniformU, 1);\n> +\t\tbreak;\n> +\tcase libcamera::formats::YUV420:\n> +\t\t/* activate texture 0 */\n> +\t\tglActiveTexture(GL_TEXTURE0);\n> +\t\tconfigureTexture(id_y);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> +\t\tglUniform1i(textureUniformY, 0);\n> +\n> +\t\t/* activate texture 1 */\n> +\t\tglActiveTexture(GL_TEXTURE1);\n> +\t\tconfigureTexture(id_u);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> +\t\tglUniform1i(textureUniformU, 1);\n> +\n> +\t\t/* activate texture 2 */\n> +\t\tglActiveTexture(GL_TEXTURE2);\n> +\t\tconfigureTexture(id_v);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> +\t\tglUniform1i(textureUniformV, 2);\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tbreak;\n> +\t};\n> +}\n> +\n> +void ViewFinderGL::paintGL()\n> +{\n> +\tif(pFShader == nullptr)\n> +\t\tcreateFragmentShader();\n> +\n> +\tif(yuvDataPtr)\n> +\t{\n> +\t\tglClearColor(0.0, 0.0, 0.0, 1.0);\n> +\t\tglClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> +\n> +\t\trender();\n> +\t\tglDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> +\t}\n> +}\n> +\n> +void ViewFinderGL::resizeGL(int w, int h)\n> +{\n> +\tglViewport(0,0,w,h);\n> +}\n> +\n> +QSize ViewFinderGL::sizeHint() const\n> +{\n> +\treturn size_.isValid() ? size_ : QSize(640, 480);\n> +}\n\nI wonder if this sizeHint should go to the base class?\n\nDoes ViewFinderGL use it? I think it's there to provide a required\noverload for the QWidget ... if it's common it could go to the base, and\nif it's not used, it could get dropped.\n\n\n\n> diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> new file mode 100644\n> index 0000000..08662ca\n> --- /dev/null\n> +++ b/src/qcam/viewfinderGL.h\n> @@ -0,0 +1,101 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Linaro\n> + *\n> + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> + */\n> +#ifndef __VIEWFINDERGL_H__\n\nPerhaps to make this clearer use this: __VIEWFINDER_GL_H__ ?\nAn underscore between the ViewFinder and the GL would separate the base,\nand the derived names...\n\n\n> +#define __VIEWFINDERGL_H__\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \"viewfinder.h\"\n> +\n> +class ViewFinderGL : public QOpenGLWidget,\n> +\t\t\t\t\t public ViewFinderHandler,\n> +\t\t\t\t\t protected QOpenGLFunctions\n\nThat indentation looks off, I think they should be aligned vertically at\nleast.\n\n\n> +{\n> +\tQ_OBJECT\n> +\n> +public:\n> +\tViewFinderGL(QWidget *parent = 0);\n> +\t~ViewFinderGL();\n> +\n> +\tint setFormat(const libcamera::PixelFormat &format, const QSize &size);\n> +\tvoid render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> +\tvoid stop();\n> +\n> +\tQImage getCurrentImage();\n\nAha - that's why we need QImage, I think that's part of the save routine\nperhaps isn't it ...\n\n> +\n> +\tvoid setFrameSize(int width, int height);\n\nI can't see what calls setFrameSize... is it redundant? Or is it an\noverride used externally or such. If so I think it needs to be marked as\naccordingly as an override?\n\n> +\tvoid updateFrame(unsigned char *buffer);\n\nSame here... What calls updateFrame()?\n\n\n> +\n> +\tchar *selectFragmentShader(unsigned int format);\n> +\tvoid createFragmentShader();\n> +\tvoid render();\n> +\n> +Q_SIGNALS:\n> +\tvoid renderComplete(libcamera::FrameBuffer *buffer);\n> +\n> +protected:\n> +\tvoid initializeGL() Q_DECL_OVERRIDE;\n> +\tvoid paintGL() Q_DECL_OVERRIDE;\n> +\tvoid resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> +\tQSize sizeHint() const override;\n\nis sizeHint() used?\n\n\n> +\n> +private:\n> +\n> +\tvoid configureTexture(unsigned int id);\n> +\tvoid removeShader();\n> +\n> +\t/* Captured image size, format and buffer */\n> +\tlibcamera::FrameBuffer *buffer_;\n> +\tlibcamera::PixelFormat format_;\n> +\tQOpenGLBuffer glBuffer;\n\nglBuffer_;\n\n\n\n> +\tQSize size_;\n> +\n> +\tGLuint id_u;\n> +\tGLuint id_v;\n> +\tGLuint id_y;\n> +\n> +\tQMutex mutex_; /* Prevent concurrent access to image_ */\n> +\n\nAll of the member variables below should have a '_' suffix...\n\n> +\tQOpenGLShader *pFShader;\n> +\tQOpenGLShader *pVShader;\n> +\tQOpenGLShaderProgram shaderProgram;\n> +\n> +\tGLuint textureUniformU;\n> +\tGLuint textureUniformV;\n> +\tGLuint textureUniformY;\n> +\n> +\tQOpenGLTexture textureU;\n> +\tQOpenGLTexture textureV;\n> +\tQOpenGLTexture textureY;\n> +\n> +\tunsigned int width_;\n> +\tunsigned int height_;\n\nThere is already a QSize size_, do these duplicate that purpose?\n\n> +\n> +\tunsigned char* yuvDataPtr;\n> +\n> +\t/* NV parameters */\n> +\tunsigned int horzSubSample_ ;\n\nExtra space?\n\n> +\tunsigned int vertSubSample_;\n> +};\n> +#endif // __VIEWFINDERGL_H__\n>","headers":{"Return-Path":"","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EE8AE609A7\n\tfor ;\n\tWed, 24 Jun 2020 11:34:43 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 4E0CD2A8;\n\tWed, 24 Jun 2020 11:34:43 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"RXIIFa3Q\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1592991283;\n\tbh=Y9icy5OdAZDxfxPG2xYXwTc1tQ/w8s35PUu97XYIpZI=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=RXIIFa3Q8cRS++Yw88ztvHaOfD1fIAsoRdT/iDDRdX3AGRcJd7QszNSt5jkI7ztw9\n\tpEOBVrkuwRymJEpZgzIjurJ0BBUvRkDWkFGjMD/lUqb+vnmBG0InPt2R2+wTrqYb6h\n\tXMIeke/lVBGlsE/5+hPaqSOxfHQqSXvxScr6Csfo=","Reply-To":"kieran.bingham@ideasonboard.com","To":"Show Liu , libcamera-devel@lists.libcamera.org","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<35bd63f1-45e4-8b5d-b914-84375b662ce9@ideasonboard.com>","Date":"Wed, 24 Jun 2020 10:34:39 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.8.0","MIME-Version":"1.0","In-Reply-To":"<20200624073705.14737-2-show.liu@linaro.org>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-GB","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","X-List-Received-Date":"Wed, 24 Jun 2020 09:34:44 -0000"}},{"id":5413,"web_url":"https://patchwork.libcamera.org/comment/5413/","msgid":"","date":"2020-06-25T08:43:54","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":24,"url":"https://patchwork.libcamera.org/api/people/24/","name":"Show Liu","email":"show.liu@linaro.org"},"content":"Hi Kieran,\n\nThanks for your review.\nYour suggestions and opinions are really good for me.\n\nI will try to hook the checkstyle.py and check my patch first. I believe\nthat will fix most of problems below. Sorry for the inconvenience.\n\n- Show\n\n\n\n\nKieran Bingham 於 2020年6月24日 週三 下午5:34 寫道：\n\n> Hi Show,\n>\n> Thank you, I'm quite excited to see this update, and I know Niklas is\n> keen for this to get in too, as he uses a Rockchip target mostly so this\n> will improve the performances for him quite a lot.\n>\n> Most of my comments below are stylistic, (but theres a couple of other\n> topics too), so as suggested below, could you install the checkstyle.py\n> hook as a git commit hook please?\n>\n\nTh\n\n>\n> I personally prefer having it as a 'post-commit' hook, rather than a\n> pre-commit so that it's easier to 'ignore' suggestions that I don't care\n> for ;-)\n>\n> --\n> Kieran\n>\n>\n> On 24/06/2020 08:37, Show Liu wrote:\n> > The patch is to render the NV family YUV formats by OpenGL shader.\n> > V3: refine the fragment shader for better pixel color.\n> >\n> > Signed-off-by: Show Liu \n> > ---\n> > src/qcam/main.cpp | 2 +\n> > src/qcam/main_window.cpp | 19 ++-\n> > src/qcam/main_window.h | 3 +-\n> > src/qcam/meson.build | 2 +\n> > src/qcam/shader.h | 104 ++++++++++++\n> > src/qcam/viewfinder.cpp | 18 +-\n> > src/qcam/viewfinder.h | 23 ++-\n> > src/qcam/viewfinderGL.cpp | 335 ++++++++++++++++++++++++++++++++++++++\n> > src/qcam/viewfinderGL.h | 101 ++++++++++++\n> > 9 files changed, 593 insertions(+), 14 deletions(-)\n> > create mode 100644 src/qcam/shader.h\n> > create mode 100644 src/qcam/viewfinderGL.cpp\n> > create mode 100644 src/qcam/viewfinderGL.h\n> >\n> > diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> > index b3468cb..a3ea471 100644\n> > --- a/src/qcam/main.cpp\n> > +++ b/src/qcam/main.cpp\n> > @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc, char\n> *argv[])\n> > \"help\");\n> > parser.addOption(OptStream, &streamKeyValue,\n> > \"Set configuration of a camera stream\", \"stream\",\n> true);\n> > + parser.addOption(OptOpenGL, OptionNone, \"Render YUV formats frame\n> via OpenGL shader\",\n> > + \"opengl\");\n>\n> (Just a question to everyone) - Should we default to the OpenGL\n> viewfinder if open-gl is available, and fall back to the QWidget version\n> otherwise? (We can always do such actions on top/later if we choose.)\n>\n>\n> >\n> > OptionsParser::Options options = parser.parse(argc, argv);\n> > if (options.isSet(OptHelp))\n> > diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> > index 7bc1360..6edf370 100644\n> > --- a/src/qcam/main_window.cpp\n> > +++ b/src/qcam/main_window.cpp\n> > @@ -28,6 +28,9 @@\n> > #include \n> >\n> > #include \"dng_writer.h\"\n> > +#include \"main_window.h\"\n> > +#include \"viewfinder.h\"\n> > +#include \"viewfinderGL.h\"\n> >\n> > using namespace libcamera;\n> >\n> > @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm, const\n> OptionsParser::Options &options)\n> > setWindowTitle(title_);\n> > connect(&titleTimer_, SIGNAL(timeout()), this,\n> SLOT(updateTitle()));\n> >\n> > - viewfinder_ = new ViewFinder(this);\n> > - connect(viewfinder_, &ViewFinder::renderComplete,\n> > - this, &MainWindow::queueRequest);\n> > - setCentralWidget(viewfinder_);\n> > + if (options_.isSet(OptOpenGL)) {\n> > + viewfinder_ = new ViewFinderGL(this);\n> > + connect(dynamic_cast(viewfinder_),\n> &ViewFinderGL::renderComplete,\n> > + this, &MainWindow::queueRequest);\n>\n> checkstyle.py highlights that the indentation of the second line should\n> be pulled back, and I agree:\n> connect(dynamic_cast(viewfinder_),\n> &ViewFinderGL::renderComplete,\n> - this, &MainWindow::queueRequest);\n> + this, &MainWindow::queueRequest);\n>\n>\n>\n> > + setCentralWidget(dynamic_cast *>(viewfinder_));\n>\n> Does the setCentralWidget need to have the dynamic_cast? Or can it just\n> be a call to\n>\n> setCentralWidget(viewfinder_);\n\nI got build error if use above if no dynamic_cast.\n\n>\n> after this conditional block?\n>\n>\n> Perhaps if the base viewfinder class had a method to call which handled\n> the connect, that could remove casting requirements there - but it would\n> probably end up needing to deal with complex template things for the\n> signal/slot parsing ... so I suspect that part isn't worth it...\n>\n\nI can trying to refine this parts.\n\n>\n>\n> > + } else {\n> > + viewfinder_ = new ViewFinder(this);\n> > + connect(dynamic_cast(viewfinder_),\n> &ViewFinder::renderComplete,\n> > + this, &MainWindow::queueRequest);\n>\n> Same indentation fault here.\n>\n> If you add the checkstyle.py hook to your libcamera sources, you'll get\n> these notifications as you commit (you can then decide if checkstyle was\n> right or not ;D):\n>\n> $ cp utils/hooks/post-commit .git/hooks/post-commit\n>\n> > + setCentralWidget(dynamic_cast(viewfinder_));\n> > + }\n> > +\n> > adjustSize();\n> >\n> > /* Hotplug/unplug support */\n> > diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> > index 4606fe4..a852ef4 100644\n> > --- a/src/qcam/main_window.h\n> > +++ b/src/qcam/main_window.h\n> > @@ -38,6 +38,7 @@ enum {\n> > OptCamera = 'c',\n> > OptHelp = 'h',\n> > OptStream = 's',\n> > + OptOpenGL = 'o',\n> > };\n> >\n> > class CaptureRequest\n> > @@ -102,7 +103,7 @@ private:\n> > QAction *startStopAction_;\n> > QComboBox *cameraCombo_;\n> > QAction *saveRaw_;\n> > - ViewFinder *viewfinder_;\n> > + ViewFinderHandler *viewfinder_;\n>\n> Handler?\n>\n> I guess I'll see below, but I think I'd expect the base class interface\n> to be 'ViewFinder', and then make specific derived implementations of that?\n>\nOk I will think about how to refine this too.\n\n>\n> >\n> > QIcon iconPlay_;\n> > QIcon iconStop_;\n> > diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> > index 045db52..6a58947 100644\n> > --- a/src/qcam/meson.build\n> > +++ b/src/qcam/meson.build\n> > @@ -7,11 +7,13 @@ qcam_sources = files([\n> > 'main.cpp',\n> > 'main_window.cpp',\n> > 'viewfinder.cpp',\n> > + 'viewfinderGL.cpp'\n> > ])\n> >\n> > qcam_moc_headers = files([\n> > 'main_window.h',\n> > 'viewfinder.h',\n> > + 'viewfinderGL.h'\n> > ])\n> >\n> > qcam_resources = files([\n> > diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> > new file mode 100644\n> > index 0000000..f54c264\n> > --- /dev/null\n> > +++ b/src/qcam/shader.h\n> > @@ -0,0 +1,104 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * shader.h - YUV format converter shader source code\n> > + */\n> > +#ifndef __SHADER_H__\n> > +#define __SHADER_H__\n> > +\n> > +/* Vertex shader for NV formats */\n> > +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n> > + \"attribute vec2 textureIn;\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"gl_Position = vertexIn;\\n\"\n> > + \"textureOut = textureIn;\\n\"\n> > + \"}\\n\";\n>\n> I'd pull that indentation back so that they all match, including moving\n> the first line to a new line to match...\n>\n> char NV_Vertex_shader[] =\n> \"attribute vec4 vertexIn;\\n\"\n> \"attribute vec2 textureIn;\\n\"\n> \"varying vec2 textureOut;\\n\"\n> \"void main(void)\\n\"\n> \"{\\n\"\n> \"gl_Position = vertexIn;\\n\"\n> \"textureOut = textureIn;\\n\"\n> \"}\\n\";\n>\n>\n>\n> > +\n> > +/* Fragment shader for NV12, NV16 and NV24 */\n> > +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n>\n> And ofcourse be consistent with the code block indentation style\n> throughout.\n>\n> checkstyle might complain whatever you do here, so I would take\n> checkstyle with a pinch of salt and just make sure the file is consitent\n> and well laid out.\n>\n>\n>\n> > + \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > + \"}\\n\";\n> > +\n> > +/* Fragment shader for NV21, NV61 and NV42 */\n> > +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> > + \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > + \"}\\n\";\n> > +\n> > +/* Fragment shader for YUV420 */\n> > +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > + \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"uniform sampler2D tex_v;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > + \"}\\n\";\n> > +\n> > +/* Fragment shader for YVU420 */\n> > +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"uniform sampler2D tex_v;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > + \"}\\n\";\n> > +#endif // __SHADER_H__\n>\n> I can't comment on the shaders without a lot of research into shaders...\n> so I'm going to say \"Yay, if this black magic works, then it works for\n> me\" ;-D\n>\n>\n> > diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> > index dcf8a15..d3a2422 100644\n> > --- a/src/qcam/viewfinder.cpp\n> > +++ b/src/qcam/viewfinder.cpp\n> > @@ -33,22 +33,30 @@ static const QMap QImage::Format> nativeFormats\n> > { libcamera::formats::BGR888, QImage::Format_RGB888 },\n> > };\n> >\n> > -ViewFinder::ViewFinder(QWidget *parent)\n> > - : QWidget(parent), buffer_(nullptr)\n> > +ViewFinderHandler::ViewFinderHandler()\n> > {\n> > - icon_ = QIcon(\":camera-off.svg\");\n> > }\n> >\n> > -ViewFinder::~ViewFinder()\n> > +ViewFinderHandler::~ViewFinderHandler()\n> > {\n> > }\n> >\n> > -const QList &ViewFinder::nativeFormats() const\n> > +const QList &ViewFinderHandler::nativeFormats()\n> const\n> > {\n> > static const QList formats =\n> ::nativeFormats.keys();\n> > return formats;\n> > }\n> >\n> > +ViewFinder::ViewFinder(QWidget *parent)\n> > + : QWidget(parent), buffer_(nullptr)\n> > +{\n> > + icon_ = QIcon(\":camera-off.svg\");\n> > +}\n> > +\n> > +ViewFinder::~ViewFinder()\n> > +{\n> > +}\n> > +\n> > int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> > const QSize &size)\n> > {\n> > diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> > index 26a1320..741d694 100644\n> > --- a/src/qcam/viewfinder.h\n> > +++ b/src/qcam/viewfinder.h\n> > @@ -13,6 +13,8 @@\n> > #include \n> > #include \n> > #include \n> > +#include \n> > +#include \n>\n> Shouldn't those includes be in\n> src/qcam/viewfinderGL.cpp or src/qcam/viewfinderGL.h only?\n>\n> Also - they should be alphabetical order.\n> (Checkstyle will highlight that for you).\n>\n> > #include \n> > #include \n> >\n> > @@ -28,7 +30,23 @@ struct MappedBuffer {\n> > size_t size;\n> > };\n> >\n> > -class ViewFinder : public QWidget\n> > +class ViewFinderHandler\n>\n> I think it's just naming, but I would have called this ViewFinder,...\n>\n> > +{\n> > +public:\n> > + ViewFinderHandler();\n> > + virtual ~ViewFinderHandler();\n> > +\n> > + const QList &nativeFormats() const;\n> > +\n> > + virtual int setFormat(const libcamera::PixelFormat &format, const\n> QSize &size) =0;\n> > + virtual void render(libcamera::FrameBuffer *buffer, MappedBuffer\n> *map) =0;\n> > + virtual void stop() =0;\n> > +\n> > + virtual QImage getCurrentImage() =0;\n>\n> on each of those, a space after the = looks better:\n> functionName() = 0;\n>\n>\n> > +\n> > +};\n> > +\n> > +class ViewFinder : public QWidget, public ViewFinderHandler\n>\n> And this, ViewFinderQT or ViewFinderQWidget? (probably the later..)\n>\n> To keep consistent with coding style, I think the viewfinder.h would\n> then only contain the base interface, and create a new file+header for\n> the QWidget version.\n>\n> Or maybe that's over kill, as the QWidget version would be the 'default'\n> ViewFinder anyway ...\n>\n>\n> > {\n> > Q_OBJECT\n> >\n> > @@ -36,8 +54,6 @@ public:\n> > ViewFinder(QWidget *parent);\n> > ~ViewFinder();\n> >\n> > - const QList &nativeFormats() const;\n> > -\n> > int setFormat(const libcamera::PixelFormat &format, const QSize\n> &size);\n> > void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > void stop();\n> > @@ -67,5 +83,4 @@ private:\n> > QImage image_;\n> > QMutex mutex_; /* Prevent concurrent access to image_ */\n> > };\n> > -\n> > #endif /* __QCAM_VIEWFINDER__ */\n> > diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> > new file mode 100644\n> > index 0000000..7b47beb\n> > --- /dev/null\n> > +++ b/src/qcam/viewfinderGL.cpp\n> > @@ -0,0 +1,335 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> > + */\n> > +\n> > +#include \"shader.h\"\n> > +#include \"viewfinderGL.h\"\n> > +\n> > +#include \n>\n> QImage is provided in viewfinder.h ... And do you actually use it in here?\n>\n>\n> > +\n> > +#include \n> > +\n> > +#define ATTRIB_VERTEX 0\n> > +#define ATTRIB_TEXTURE 1\n> > +\n> > +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> > + : QOpenGLWidget(parent),\n> > + glBuffer(QOpenGLBuffer::VertexBuffer),\n> > + pFShader(nullptr),\n> > + pVShader(nullptr),\n> > + textureU(QOpenGLTexture::Target2D),\n> > + textureV(QOpenGLTexture::Target2D),\n> > + textureY(QOpenGLTexture::Target2D),\n> > + yuvDataPtr(nullptr)\n>\n> Checkstyle will tell you to align those vertically with the first\n> QOpenGLWidget(parent), rather than the ':'.\n>\n>\n> > +\n> > +{\n> > +}\n> > +\n> > +ViewFinderGL::~ViewFinderGL()\n> > +{\n> > + removeShader();\n> > +\n> > + if(textureY.isCreated())\n>\n> Space after if: \"if ()\"\n>\n> > + textureY.destroy();\n> > +\n> > + if(textureU.isCreated())\n> > + textureU.destroy();\n> > +\n> > + if(textureV.isCreated())\n> > + textureV.destroy();\n>\n> For each of those ;-)\n>\n> There's plenty more too, but I'll let you find them with checkstyle.\n>\n>\n> > +\n> > + glBuffer.destroy();\n> > +}\n> > +\n> > +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> > + const QSize &size)\n> > +{\n> > + format_ = format;\n> > + size_ = size;\n> > +\n> > + updateGeometry();\n> > + return 0;\n> > +}\n> > +\n> > +void ViewFinderGL::stop()\n> > +{\n> > + if (buffer_) {\n> > + renderComplete(buffer_);\n> > + buffer_ = nullptr;\n> > + }\n> > +}\n> > +\n> > +QImage ViewFinderGL::getCurrentImage()\n> > +{\n> > + QMutexLocker locker(&mutex_);\n> > +\n> > + return(grabFramebuffer());\n>\n> I think that could be :\n>\n> return grabFrameBuffer();\n>\n> > +}\n> > +\n> > +void ViewFinderGL::render(libcamera::FrameBuffer *buffer, MappedBuffer\n> *map)\n> > +{\n> > + if (buffer->planes().size() != 1) {\n> > + qWarning() << \"Multi-planar buffers are not supported\";\n> > + return;\n> > + }\n> > +\n> > + unsigned char *memory = static_cast(map->memory);\n> > + if(memory) {\n> > + yuvDataPtr = memory;\n> > + update();\n> > + buffer_ = buffer;\n> > + }\n> > +\n> > + if (buffer)\n> > + renderComplete(buffer);\n> > +}\n> > +\n> > +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> > +{\n> > + if(buffer) {\n> > + yuvDataPtr = buffer;\n> > + update();\n> > + }\n> > +}\n> > +\n> > +void ViewFinderGL::setFrameSize(int width, int height)\n>\n> Can width/height be negative? Maybe the should be unsigned int?\n>\n> What about using a QSize too rather than individually passing width/height?\n>\n> I can't see what calls setFrameSize(), is this an override from\n>\n> > +{\n> > + if(width > 0 && height > 0) {\n> > + width_ = width;\n> > + height_ = height;\n> > + }\n>\n> In fact, don't we already have a size_ in the class? (which could also\n> be in the base class too most likely...\n>\nSure, I will use QSize instead.\n\n>\n>\n> > +}\n> > +\n> > +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> > +{\n> > + char *fsrc = nullptr;\n> > +\n> > + switch(format) {\n> > + case libcamera::formats::NV12:\n> > + horzSubSample_ = 2;\n> > + vertSubSample_ = 2;\n> > + fsrc = NV_2_planes_UV;\n> > + break;\n> > + case libcamera::formats::NV21:\n> > + horzSubSample_ = 2;\n> > + vertSubSample_ = 2;\n> > + fsrc = NV_2_planes_VU;\n> > + break;\n> > + case libcamera::formats::NV16:\n> > + horzSubSample_ = 2;\n> > + vertSubSample_ = 1;\n> > + fsrc = NV_2_planes_UV;\n> > + break;\n> > + case libcamera::formats::NV61:\n> > + horzSubSample_ = 2;\n> > + vertSubSample_ = 1;\n> > + fsrc = NV_2_planes_VU;\n> > + break;\n> > + case libcamera::formats::NV24:\n> > + horzSubSample_ = 1;\n> > + vertSubSample_ = 1;\n> > + fsrc = NV_2_planes_UV;\n> > + break;\n> > + case libcamera::formats::NV42:\n> > + horzSubSample_ = 1;\n> > + vertSubSample_ = 1;\n> > + fsrc = NV_2_planes_VU;\n> > + break;\n> > + case libcamera::formats::YUV420:\n> > + horzSubSample_ = 2;\n> > + vertSubSample_ = 2;\n> > + fsrc = NV_3_planes_UV;\n> > + break;\n> > + default:\n> > + break;\n> > + };\n> > +\n> > + if(fsrc == nullptr) {\n> > + qDebug() << __func__ << \"[ERROR]:\" <<\" No suitable\n> fragment shader can be select.\";\n>\n> 'selected'\n>\nOk. Will fix it.\n\n>\n>\n> > + }\n> > + return fsrc;\n> > +}\n> > +\n> > +void ViewFinderGL::createFragmentShader()\n> > +{\n> > + bool bCompile;\n> > +\n> > + pFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> > + char *fsrc = selectFragmentShader(format_);\n> > +\n> > + bCompile = pFShader->compileSourceCode(fsrc);\n> > + if(!bCompile)\n> > + {\n> > + qDebug() << __func__ << \":\" << pFShader->log();\n> > + }\n> > +\n> > + shaderProgram.addShader(pFShader);\n> > +\n> > + // Link shader pipeline\n>\n> Even though it's C++, we usually use C style comments...\n>\n>\n> > + if (!shaderProgram.link()) {\n> > + qDebug() << __func__ << \": shader program link failed.\\n\"\n> << shaderProgram.log();\n> > + close();\n> > + }\n> > +\n> > + // Bind shader pipeline for use\n> > + if (!shaderProgram.bind()) {\n> > + qDebug() << __func__ << \": shader program binding\n> failed.\\n\" << shaderProgram.log();\n> > + close();\n> > + }\n> > +\n> > + shaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> > + shaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> > +\n> > +\n> shaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> > +\n> shaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> > +\n> > + textureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> > + textureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> > + textureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> > +\n> > + if(!textureY.isCreated())\n> > + textureY.create();\n> > +\n> > + if(!textureU.isCreated())\n> > + textureU.create();\n> > +\n> > + if(!textureV.isCreated())\n> > + textureV.create();\n> > +\n> > + id_y = textureY.textureId();\n> > + id_u = textureU.textureId();\n> > + id_v = textureV.textureId();\n> > +}\n> > +\n> > +void ViewFinderGL::configureTexture(unsigned int id)\n> > +{\n> > + glBindTexture(GL_TEXTURE_2D, id);\n> > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);\n> > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);\n> > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S,\n> GL_CLAMP_TO_EDGE);\n> > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T,\n> GL_CLAMP_TO_EDGE);\n> > +}\n> > +\n> > +void ViewFinderGL::removeShader()\n> > +{\n> > + if (shaderProgram.isLinked()) {\n> > + shaderProgram.release();\n> > + shaderProgram.removeAllShaders();\n> > + }\n> > +\n> > + if(pFShader)\n> > + delete pFShader;\n> > +\n> > + if(pVShader)\n> > + delete pVShader;\n> > +}\n> > +\n> > +void ViewFinderGL::initializeGL()\n> > +{\n> > + bool bCompile;\n> > +\n> > + initializeOpenGLFunctions();\n> > + glEnable(GL_TEXTURE_2D);\n> > + glDisable(GL_DEPTH_TEST);\n> > +\n> > + static const GLfloat vertices[] {\n> > + -1.0f,-1.0f,\n> > + -1.0f,+1.0f,\n> > + +1.0f,+1.0f,\n> > + +1.0f,-1.0f,\n> > + 0.0f,1.0f,\n> > + 0.0f,0.0f,\n> > + 1.0f,0.0f,\n> > + 1.0f,1.0f,\n>\n> When you get here, checkstyle will suggest putting this all in one\n> vertical column.\n>\n> Ignore it... (checkstyle is just advice, and in this case your layout is\n> better).\n>\n> Though I would add a space at least after the first ',' on each line to\n> make the columns clearer., Or go further and align the 'f's... ?\n>\n> > + };\n> > +\n> > + glBuffer.create();\n> > + glBuffer.bind();\n> > + glBuffer.allocate(vertices,sizeof(vertices));\n> > +\n> > + /* Create Vertex Shader */\n> > + pVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> > + char *vsrc = NV_Vertex_shader;\n> > +\n> > + bCompile = pVShader->compileSourceCode(vsrc);\n> > + if(!bCompile) {\n> > + qDebug() << __func__<< \":\" << pVShader->log();\n> > + }\n> > +\n> > + shaderProgram.addShader(pVShader);\n> > +\n> > + glClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> > +}\n> > +\n> > +void ViewFinderGL::render()\n> > +{\n> > + switch(format_) {\n> > + case libcamera::formats::NV12:\n> > + case libcamera::formats::NV21:\n> > + case libcamera::formats::NV16:\n> > + case libcamera::formats::NV61:\n> > + case libcamera::formats::NV24:\n> > + case libcamera::formats::NV42:\n> > + /* activate texture 0 */\n> > + glActiveTexture(GL_TEXTURE0);\n> > + configureTexture(id_y);\n> > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(),\n> size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > + glUniform1i(textureUniformY, 0);\n> > +\n> > + /* activate texture 1 */\n> > + glActiveTexture(GL_TEXTURE1);\n> > + configureTexture(id_u);\n> > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > + glUniform1i(textureUniformU, 1);\n> > + break;\n> > + case libcamera::formats::YUV420:\n> > + /* activate texture 0 */\n> > + glActiveTexture(GL_TEXTURE0);\n> > + configureTexture(id_y);\n> > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(),\n> size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > + glUniform1i(textureUniformY, 0);\n> > +\n> > + /* activate texture 1 */\n> > + glActiveTexture(GL_TEXTURE1);\n> > + configureTexture(id_u);\n> > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > + glUniform1i(textureUniformU, 1);\n> > +\n> > + /* activate texture 2 */\n> > + glActiveTexture(GL_TEXTURE2);\n> > + configureTexture(id_v);\n> > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> > + glUniform1i(textureUniformV, 2);\n> > + break;\n> > + default:\n> > + break;\n> > + };\n> > +}\n> > +\n> > +void ViewFinderGL::paintGL()\n> > +{\n> > + if(pFShader == nullptr)\n> > + createFragmentShader();\n> > +\n> > + if(yuvDataPtr)\n> > + {\n> > + glClearColor(0.0, 0.0, 0.0, 1.0);\n> > + glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> > +\n> > + render();\n> > + glDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> > + }\n> > +}\n> > +\n> > +void ViewFinderGL::resizeGL(int w, int h)\n> > +{\n> > + glViewport(0,0,w,h);\n> > +}\n> > +\n> > +QSize ViewFinderGL::sizeHint() const\n> > +{\n> > + return size_.isValid() ? size_ : QSize(640, 480);\n> > +}\n>\n> I wonder if this sizeHint should go to the base class?\n>\n> Does ViewFinderGL use it? I think it's there to provide a required\n> overload for the QWidget ... if it's common it could go to the base, and\n> if it's not used, it could get dropped.\n>\nGot it.\n\n>\n>\n>\n> > diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> > new file mode 100644\n> > index 0000000..08662ca\n> > --- /dev/null\n> > +++ b/src/qcam/viewfinderGL.h\n> > @@ -0,0 +1,101 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> > + */\n> > +#ifndef __VIEWFINDERGL_H__\n>\n> Perhaps to make this clearer use this: __VIEWFINDER_GL_H__ ?\n> An underscore between the ViewFinder and the GL would separate the base,\n> and the derived names...\n> Ok, I will fix this.\n>\n> > +#define __VIEWFINDERGL_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \"viewfinder.h\"\n> > +\n> > +class ViewFinderGL : public QOpenGLWidget,\n> > + public ViewFinderHandler,\n> > + protected QOpenGLFunctions\n>\n> That indentation looks off, I think they should be aligned vertically at\n> least.\n>\nSure.\n\n>\n>\n> > +{\n> > + Q_OBJECT\n> > +\n> > +public:\n> > + ViewFinderGL(QWidget *parent = 0);\n> > + ~ViewFinderGL();\n> > +\n> > + int setFormat(const libcamera::PixelFormat &format, const QSize\n> &size);\n> > + void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > + void stop();\n> > +\n> > + QImage getCurrentImage();\n>\n> Aha - that's why we need QImage, I think that's part of the save routine\n> perhaps isn't it ...\n>\n> > +\n> > + void setFrameSize(int width, int height);\n>\n> I can't see what calls setFrameSize... is it redundant? Or is it an\n> override used externally or such. If so I think it needs to be marked as\n> accordingly as an override?\n>\n> > + void updateFrame(unsigned char *buffer);\n>\n> Same here... What calls updateFrame()?\n>\nActually, only NV12, I can verify with Libcamera on rock pi 4, I use an\nexternal application to verify other shaders and above functions are for\nthat.\nI will remove that in next version.\n\n>\n>\n> > +\n> > + char *selectFragmentShader(unsigned int format);\n> > + void createFragmentShader();\n> > + void render();\n> > +\n> > +Q_SIGNALS:\n> > + void renderComplete(libcamera::FrameBuffer *buffer);\n> > +\n> > +protected:\n> > + void initializeGL() Q_DECL_OVERRIDE;\n> > + void paintGL() Q_DECL_OVERRIDE;\n> > + void resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> > + QSize sizeHint() const override;\n>\n> is sizeHint() used?\n>\n>\n> > +\n> > +private:\n> > +\n> > + void configureTexture(unsigned int id);\n> > + void removeShader();\n> > +\n> > + /* Captured image size, format and buffer */\n> > + libcamera::FrameBuffer *buffer_;\n> > + libcamera::PixelFormat format_;\n> > + QOpenGLBuffer glBuffer;\n>\n> glBuffer_;\n>\nWill fix it.\n\n>\n>\n>\n> > + QSize size_;\n> > +\n> > + GLuint id_u;\n> > + GLuint id_v;\n> > + GLuint id_y;\n> > +\n> > + QMutex mutex_; /* Prevent concurrent access to image_ */\n> > +\n>\n> All of the member variables below should have a '_' suffix...\n>\n> > + QOpenGLShader *pFShader;\n> > + QOpenGLShader *pVShader;\n> > + QOpenGLShaderProgram shaderProgram;\n> > +\n> > + GLuint textureUniformU;\n> > + GLuint textureUniformV;\n> > + GLuint textureUniformY;\n> > +\n> > + QOpenGLTexture textureU;\n> > + QOpenGLTexture textureV;\n> > + QOpenGLTexture textureY;\n> > +\n> > + unsigned int width_;\n> > + unsigned int height_;\n>\n> There is already a QSize size_, do these duplicate that purpose?\n>\nYeah, I will use size_ instead.\n\n>\n> > +\n> > + unsigned char* yuvDataPtr;\n> > +\n> > + /* NV parameters */\n> > + unsigned int horzSubSample_ ;\n>\n> Extra space?\n>\nOk will fix it.\n\n>\n> > + unsigned int vertSubSample_;\n> > +};\n> > +#endif // __VIEWFINDERGL_H__\n> >\n>\n>\n>\n> --\n> Regards\n> --\n> Kieran\n>\n\nThank you for your comments again.\nI will post next version asap.\n\n--\nRegards\n--\nShow\n\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3947EC0101\n\tfor ;\n\tThu, 25 Jun 2020 08:44:10 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B4E26609C4;\n\tThu, 25 Jun 2020 10:44:09 +0200 (CEST)","from mail-ej1-x632.google.com (mail-ej1-x632.google.com\n\t[IPv6:2a00:1450:4864:20::632])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0509F609A3\n\tfor ;\n\tThu, 25 Jun 2020 10:44:07 +0200 (CEST)","by mail-ej1-x632.google.com with SMTP id y10so5151426eje.1\n\tfor ;\n\tThu, 25 Jun 2020 01:44:06 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"cFqwEID+\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=8yvE45IqntndWs3J6TEC9ZS+qmOE9KXR0LW9WtGp6Qs=;\n\tb=cFqwEID+LXObyecdjIyIqgdPcV3TF0kHEUn8BfpXsAxdIVXPBdV0Y51EhrpHRBtJj+\n\tmX70dYi0Io5psuKCDxEKaNdOmDEmH0psIrnFUAd29t/DFMPCCMnsXfca0xg73sdRiNZ/\n\tz3i8AmcclDfwjIE6vZghrtJxbbjBYmuUWpasK+rakn05YwcrCtt/Y+fwghwNmyv2rYZg\n\tSah1+dFWuygHzWdTdsJ1cmcXCmCMTIjnzVIYsmpcoeC68Fk5O/2CeifsZzp1CIPHbo8Q\n\tgB4NQpdrHERS/zzVU0Cj+0q4YZ86vvZ7ySDIGWj8jKeyixqT+ApyE/4thhvFhQG1wGx+\n\tNxSg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=8yvE45IqntndWs3J6TEC9ZS+qmOE9KXR0LW9WtGp6Qs=;\n\tb=rcGMqe9jD2rNE9oGUkWTBwUPWhqJW9jHf+JjnfhEwMM9UA5u2Bnlcw+AkOEGuGZ/Ct\n\t21iMfh9Lidt0Sr9XACcr5SxTX1nWH87gjpk3O5ucH8RoPgmaErkJFTdYLsPLESwVp7Sx\n\t5YohiCEIhGTGqrkjb9i5EI8Zx8WIukRYxB92c8B2Z1ScwY9g/PuV627TKQxLXfKWvvwq\n\tsMz4xK7xOR+N3dpC7wX08tFQRieQOLrZvZUXd0ccR3UiiAxEEpajT8dzytqEYEbaF61x\n\ti4Dz8GZivHGPdk6d4wC/zx31tWeu5i3Y20yaqVF68BwzA3BA/4nfTN5Y9vI0lL2n8BHj\n\tWeNw==","X-Gm-Message-State":"AOAM5311KIPUL/wiV7PnIhoMr0eTBEGAIj6o23fmr8p+AztEgln6EZJu\n\t2v/W6B52fGXVXUMgLF1v6GeylgQr577jDPqUC1RyLw==","X-Google-Smtp-Source":"ABdhPJwcaQtd0KO/3Wj/4nHLrYAxQt7ZGJRR/shtGVE7NMCF+Mx6H6jwNTejve0AJLj+7XYNhpdqfMHKy6ldsCOhDec=","X-Received":"by 2002:a17:906:68ca:: with SMTP id\n\ty10mr27933188ejr.322.1593074646249; \n\tThu, 25 Jun 2020 01:44:06 -0700 (PDT)","MIME-Version":"1.0","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>\n\t<35bd63f1-45e4-8b5d-b914-84375b662ce9@ideasonboard.com>","In-Reply-To":"<35bd63f1-45e4-8b5d-b914-84375b662ce9@ideasonboard.com>","From":"Show Liu ","Date":"Thu, 25 Jun 2020 16:43:54 +0800","Message-ID":"","To":"kieran.bingham@ideasonboard.com","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============6380582454777619078==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":10858,"web_url":"https://patchwork.libcamera.org/comment/10858/","msgid":"<3a34e009-5877-4420-3da4-6f3153394f5f@ideasonboard.com>","date":"2020-06-25T11:36:43","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Show,\n\nOn 25/06/2020 09:43, Show Liu wrote:\n> Hi Kieran,\n> \n> Thanks for your review.\n> Your suggestions and opinions are really good for me.\n> \n> I will try to hook the checkstyle.py and check my patch first. I believe\n> that will fix most of problems below. Sorry for the inconvenience.\n\nNo problem, or inconvenience, that's why we (well Laurent) made\ncheckstyle.py :-D\n\nLooking forward to your update.\n\nKieran\n\n\n> \n> - Show\n> \n> \n> \n> \n> Kieran Bingham > 於 2020年6月24日 週三 下午5:34\n> 寫道：\n> \n> Hi Show,\n> \n> Thank you, I'm quite excited to see this update, and I know Niklas is\n> keen for this to get in too, as he uses a Rockchip target mostly so this\n> will improve the performances for him quite a lot.\n> \n> Most of my comments below are stylistic, (but theres a couple of other\n> topics too), so as suggested below, could you install the checkstyle.py\n> hook as a git commit hook please?\n> \n> \n> Th\n> \n> \n> I personally prefer having it as a 'post-commit' hook, rather than a\n> pre-commit so that it's easier to 'ignore' suggestions that I don't care\n> for ;-)\n> \n> --\n> Kieran\n> \n> \n> On 24/06/2020 08:37, Show Liu wrote:\n> > The patch is to render the NV family YUV formats by OpenGL shader.\n> > V3: refine the fragment shader for better pixel color.\n> >\n> > Signed-off-by: Show Liu >\n> > ---\n> >  src/qcam/main.cpp         |   2 +\n> >  src/qcam/main_window.cpp  |  19 ++-\n> >  src/qcam/main_window.h    |   3 +-\n> >  src/qcam/meson.build      |   2 +\n> >  src/qcam/shader.h         | 104 ++++++++++++\n> >  src/qcam/viewfinder.cpp   |  18 +-\n> >  src/qcam/viewfinder.h     |  23 ++-\n> >  src/qcam/viewfinderGL.cpp | 335\n> ++++++++++++++++++++++++++++++++++++++\n> >  src/qcam/viewfinderGL.h   | 101 ++++++++++++\n> >  9 files changed, 593 insertions(+), 14 deletions(-)\n> >  create mode 100644 src/qcam/shader.h\n> >  create mode 100644 src/qcam/viewfinderGL.cpp\n> >  create mode 100644 src/qcam/viewfinderGL.h\n> >\n> > diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> > index b3468cb..a3ea471 100644\n> > --- a/src/qcam/main.cpp\n> > +++ b/src/qcam/main.cpp\n> > @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc,\n> char *argv[])\n> >                        \"help\");\n> >       parser.addOption(OptStream, &streamKeyValue,\n> >                        \"Set configuration of a camera stream\",\n> \"stream\", true);\n> > +     parser.addOption(OptOpenGL, OptionNone, \"Render YUV formats\n> frame via OpenGL shader\",\n> > +                      \"opengl\");\n> \n> (Just a question to everyone) - Should we default to the OpenGL\n> viewfinder if open-gl is available, and fall back to the QWidget version\n> otherwise? (We can always do such actions on top/later if we choose.)\n> \n> \n> > \n> >       OptionsParser::Options options = parser.parse(argc, argv);\n> >       if (options.isSet(OptHelp))\n> > diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> > index 7bc1360..6edf370 100644\n> > --- a/src/qcam/main_window.cpp\n> > +++ b/src/qcam/main_window.cpp\n> > @@ -28,6 +28,9 @@\n> >  #include \n> > \n> >  #include \"dng_writer.h\"\n> > +#include \"main_window.h\"\n> > +#include \"viewfinder.h\"\n> > +#include \"viewfinderGL.h\"\n> > \n> >  using namespace libcamera;\n> > \n> > @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm,\n> const OptionsParser::Options &options)\n> >       setWindowTitle(title_);\n> >       connect(&titleTimer_, SIGNAL(timeout()), this,\n> SLOT(updateTitle()));\n> > \n> > -     viewfinder_ = new ViewFinder(this);\n> > -     connect(viewfinder_, &ViewFinder::renderComplete,\n> > -             this, &MainWindow::queueRequest);\n> > -     setCentralWidget(viewfinder_);\n> > +     if (options_.isSet(OptOpenGL)) {\n> > +             viewfinder_ = new ViewFinderGL(this);\n> > +             connect(dynamic_cast(viewfinder_),\n> &ViewFinderGL::renderComplete,\n> > +                             this, &MainWindow::queueRequest);\n> \n> checkstyle.py highlights that the indentation of the second line should\n> be pulled back, and I agree:\n>                 connect(dynamic_cast(viewfinder_),\n> &ViewFinderGL::renderComplete,\n> -                               this, &MainWindow::queueRequest);\n> +                       this, &MainWindow::queueRequest);\n> \n> \n> \n> > +             setCentralWidget(dynamic_cast *>(viewfinder_));\n> \n> Does the setCentralWidget need to have the dynamic_cast? Or can it just\n> be a call to\n> \n>         setCentralWidget(viewfinder_);\n> \n> I got build error if use above if no dynamic_cast.\n> \n> \n> after this conditional block?\n> \n> \n> Perhaps if the base viewfinder class had a method to call which handled\n> the connect, that could remove casting requirements there - but it would\n> probably end up needing to deal with complex template things for the\n> signal/slot parsing ... so I suspect that part isn't worth it...\n> \n> \n> I can trying to refine this parts.\n> \n> \n> \n> > +     } else {\n> > +             viewfinder_ = new ViewFinder(this);\n> > +             connect(dynamic_cast(viewfinder_),\n> &ViewFinder::renderComplete,\n> > +                             this, &MainWindow::queueRequest);\n> \n> Same indentation fault here.\n> \n> If you add the checkstyle.py hook to your libcamera sources, you'll get\n> these notifications as you commit (you can then decide if checkstyle was\n> right or not ;D):\n> \n>  $ cp utils/hooks/post-commit .git/hooks/post-commit\n> \n> > +             setCentralWidget(dynamic_cast *>(viewfinder_));\n> > +     }\n> > +\n> >       adjustSize();\n> > \n> >       /* Hotplug/unplug support */\n> > diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> > index 4606fe4..a852ef4 100644\n> > --- a/src/qcam/main_window.h\n> > +++ b/src/qcam/main_window.h\n> > @@ -38,6 +38,7 @@ enum {\n> >       OptCamera = 'c',\n> >       OptHelp = 'h',\n> >       OptStream = 's',\n> > +     OptOpenGL = 'o',\n> >  };\n> > \n> >  class CaptureRequest\n> > @@ -102,7 +103,7 @@ private:\n> >       QAction *startStopAction_;\n> >       QComboBox *cameraCombo_;\n> >       QAction *saveRaw_;\n> > -     ViewFinder *viewfinder_;\n> > +     ViewFinderHandler *viewfinder_;\n> \n> Handler?\n> \n> I guess I'll see below, but I think I'd expect the base class interface\n> to be 'ViewFinder', and then make specific derived implementations\n> of that?\n> \n> Ok I will think about how to refine this too. \n> \n> \n> > \n> >       QIcon iconPlay_;\n> >       QIcon iconStop_;\n> > diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> > index 045db52..6a58947 100644\n> > --- a/src/qcam/meson.build\n> > +++ b/src/qcam/meson.build\n> > @@ -7,11 +7,13 @@ qcam_sources = files([\n> >      'main.cpp',\n> >      'main_window.cpp',\n> >      'viewfinder.cpp',\n> > +    'viewfinderGL.cpp'\n> >  ])\n> > \n> >  qcam_moc_headers = files([\n> >      'main_window.h',\n> >      'viewfinder.h',\n> > +    'viewfinderGL.h'\n> >  ])\n> > \n> >  qcam_resources = files([\n> > diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> > new file mode 100644\n> > index 0000000..f54c264\n> > --- /dev/null\n> > +++ b/src/qcam/shader.h\n> > @@ -0,0 +1,104 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * shader.h - YUV format converter shader source code\n> > + */\n> > +#ifndef __SHADER_H__\n> > +#define __SHADER_H__\n> > +\n> > +/* Vertex shader for NV formats */\n> > +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n> > +                        \"attribute vec2 textureIn;\\n\"\n> > +                        \"varying vec2 textureOut;\\n\"\n> > +                        \"void main(void)\\n\"\n> > +                        \"{\\n\"\n> > +                        \"gl_Position = vertexIn;\\n\"\n> > +                        \"textureOut = textureIn;\\n\"\n> > +                        \"}\\n\";\n> \n> I'd pull that indentation back so that they all match, including moving\n> the first line to a new line to match...\n> \n> char NV_Vertex_shader[] =\n>         \"attribute vec4 vertexIn;\\n\"\n>         \"attribute vec2 textureIn;\\n\"\n>         \"varying vec2 textureOut;\\n\"\n>         \"void main(void)\\n\"\n>         \"{\\n\"\n>         \"gl_Position = vertexIn;\\n\"\n>         \"textureOut = textureIn;\\n\"\n>         \"}\\n\";\n> \n> \n> \n> > +\n> > +/* Fragment shader for NV12, NV16 and NV24 */\n> > +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n> \n> And ofcourse be consistent with the code block indentation style\n> throughout.\n> \n> checkstyle might complain whatever you do here, so I would take\n> checkstyle with a pinch of salt and just make sure the file is consitent\n> and well laid out.\n> \n> \n> \n> > +                \"precision highp float;\\n\"\n> > +                \"#endif\\n\"\n> > +                \"varying vec2 textureOut;\\n\"\n> > +                \"uniform sampler2D tex_y;\\n\"\n> > +                \"uniform sampler2D tex_u;\\n\"\n> > +                \"void main(void)\\n\"\n> > +                \"{\\n\"\n> > +                \"vec3 yuv;\\n\"\n> > +                \"vec3 rgb;\\n\"\n> > +                \"mat3 convert_mat = mat3(vec3(1.1640625, \n> 1.1640625, 1.1640625),\\n\"\n> > +                \"                        vec3(0.0,   -0.390625,\n> 2.015625),\\n\"\n> > +                \"                        vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > +                \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > +                \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > +                \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > +                \"rgb = convert_mat * yuv;\\n\"\n> > +                \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > +                \"}\\n\";\n> > +\n> > +/* Fragment shader for NV21, NV61 and NV42 */\n> > +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> > +                \"precision highp float;\\n\"\n> > +                \"#endif\\n\"\n> > +                \"varying vec2 textureOut;\\n\"\n> > +                \"uniform sampler2D tex_y;\\n\"\n> > +                \"uniform sampler2D tex_u;\\n\"\n> > +                \"void main(void)\\n\"\n> > +                \"{\\n\"\n> > +                \"vec3 yuv;\\n\"\n> > +                \"vec3 rgb;\\n\"\n> > +                \"mat3 convert_mat = mat3(vec3(1.1640625, \n> 1.1640625, 1.1640625),\\n\"\n> > +                \"                        vec3(0.0,   -0.390625,\n> 2.015625),\\n\"\n> > +                \"                        vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > +                \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > +                \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > +                \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > +                \"rgb = convert_mat * yuv;\\n\"\n> > +                \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > +                \"}\\n\";\n> > +\n> > +/* Fragment shader for YUV420 */\n> > +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > +                \"precision highp float;\\n\"\n> > +                \"#endif\\n\"\n> > +                \"varying vec2 textureOut;\\n\"\n> > +                \"uniform sampler2D tex_y;\\n\"\n> > +                \"uniform sampler2D tex_u;\\n\"\n> > +                \"uniform sampler2D tex_v;\\n\"\n> > +                \"void main(void)\\n\"\n> > +                \"{\\n\"\n> > +                \"vec3 yuv;\\n\"\n> > +                \"vec3 rgb;\\n\"\n> > +                \"mat3 convert_mat = mat3(vec3(1.1640625, \n> 1.1640625, 1.1640625),\\n\"\n> > +                \"                        vec3(0.0,   -0.390625,\n> 2.015625),\\n\"\n> > +                \"                        vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > +                \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > +                \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > +                \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > +                \"rgb = convert_mat * yuv;\\n\"\n> > +                \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > +                \"}\\n\";\n> > +\n> > +/* Fragment shader for YVU420 */\n> > +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> > +                \"#endif\\n\"\n> > +                \"varying vec2 textureOut;\\n\"\n> > +                \"uniform sampler2D tex_y;\\n\"\n> > +                \"uniform sampler2D tex_u;\\n\"\n> > +                \"uniform sampler2D tex_v;\\n\"\n> > +                \"void main(void)\\n\"\n> > +                \"{\\n\"\n> > +                \"vec3 yuv;\\n\"\n> > +                \"vec3 rgb;\\n\"\n> > +                \"mat3 convert_mat = mat3(vec3(1.1640625, \n> 1.1640625, 1.1640625),\\n\"\n> > +                \"                        vec3(0.0,   -0.390625,\n> 2.015625),\\n\"\n> > +                \"                        vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > +                \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > +                \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > +                \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > +                \"rgb = convert_mat * yuv;\\n\"\n> > +                \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > +                \"}\\n\";\n> > +#endif // __SHADER_H__\n> \n> I can't comment on the shaders without a lot of research into shaders...\n> so I'm going to say \"Yay, if this black magic works, then it works for\n> me\"  ;-D\n> \n> \n> > diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> > index dcf8a15..d3a2422 100644\n> > --- a/src/qcam/viewfinder.cpp\n> > +++ b/src/qcam/viewfinder.cpp\n> > @@ -33,22 +33,30 @@ static const QMap QImage::Format> nativeFormats\n> >       { libcamera::formats::BGR888, QImage::Format_RGB888 },\n> >  };\n> > \n> > -ViewFinder::ViewFinder(QWidget *parent)\n> > -     : QWidget(parent), buffer_(nullptr)\n> > +ViewFinderHandler::ViewFinderHandler()\n> >  {\n> > -     icon_ = QIcon(\":camera-off.svg\");\n> >  }\n> > \n> > -ViewFinder::~ViewFinder()\n> > +ViewFinderHandler::~ViewFinderHandler()\n> >  {\n> >  }\n> > \n> > -const QList &ViewFinder::nativeFormats()\n> const\n> > +const QList\n> &ViewFinderHandler::nativeFormats() const\n> >  {\n> >       static const QList formats =\n> ::nativeFormats.keys();\n> >       return formats;\n> >  }\n> > \n> > +ViewFinder::ViewFinder(QWidget *parent)\n> > +     : QWidget(parent), buffer_(nullptr)\n> > +{\n> > +     icon_ = QIcon(\":camera-off.svg\");\n> > +}\n> > +\n> > +ViewFinder::~ViewFinder()\n> > +{\n> > +}\n> > +\n> >  int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> >                         const QSize &size)\n> >  {\n> > diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> > index 26a1320..741d694 100644\n> > --- a/src/qcam/viewfinder.h\n> > +++ b/src/qcam/viewfinder.h\n> > @@ -13,6 +13,8 @@\n> >  #include \n> >  #include \n> >  #include \n> > +#include \n> > +#include \n> \n> Shouldn't those includes be in\n> src/qcam/viewfinderGL.cpp or src/qcam/viewfinderGL.h only?\n> \n> Also - they should be alphabetical order.\n>  (Checkstyle will highlight that for you).\n> \n> >  #include \n> >  #include \n> > \n> > @@ -28,7 +30,23 @@ struct MappedBuffer {\n> >       size_t size;\n> >  };\n> > \n> > -class ViewFinder : public QWidget\n> > +class ViewFinderHandler\n> \n> I think it's just naming, but I would have called this ViewFinder,...\n> \n> > +{\n> > +public:\n> > +     ViewFinderHandler();\n> > +     virtual ~ViewFinderHandler();\n> > +\n> > +     const QList &nativeFormats() const;\n> > +\n> > +     virtual int setFormat(const libcamera::PixelFormat &format,\n> const QSize &size) =0;\n> > +     virtual void render(libcamera::FrameBuffer *buffer,\n> MappedBuffer *map) =0;\n> > +     virtual void stop() =0;\n> > +\n> > +     virtual QImage getCurrentImage() =0;\n> \n> on each of those, a space after the = looks better:\n>   functionName() = 0;\n> \n> \n> > +\n> > +};\n> > +\n> > +class ViewFinder : public QWidget, public ViewFinderHandler\n> \n> And this, ViewFinderQT or ViewFinderQWidget? (probably the later..)\n> \n> To keep consistent with coding style, I think the viewfinder.h would\n> then only contain the base interface, and create a new file+header for\n> the QWidget version.\n> \n> Or maybe that's over kill, as the QWidget version would be the 'default'\n> ViewFinder anyway ...\n> \n> \n> >  {\n> >       Q_OBJECT\n> > \n> > @@ -36,8 +54,6 @@ public:\n> >       ViewFinder(QWidget *parent);\n> >       ~ViewFinder();\n> > \n> > -     const QList &nativeFormats() const;\n> > -\n> >       int setFormat(const libcamera::PixelFormat &format, const\n> QSize &size);\n> >       void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> >       void stop();\n> > @@ -67,5 +83,4 @@ private:\n> >       QImage image_;\n> >       QMutex mutex_; /* Prevent concurrent access to image_ */\n> >  };\n> > -\n> >  #endif /* __QCAM_VIEWFINDER__ */\n> > diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> > new file mode 100644\n> > index 0000000..7b47beb\n> > --- /dev/null\n> > +++ b/src/qcam/viewfinderGL.cpp\n> > @@ -0,0 +1,335 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> > + */\n> > +\n> > +#include \"shader.h\"\n> > +#include \"viewfinderGL.h\"\n> > +\n> > +#include \n> \n> QImage is provided in viewfinder.h ... And do you actually use it in\n> here?\n> \n> \n> > +\n> > +#include \n> > +\n> > +#define ATTRIB_VERTEX 0\n> > +#define ATTRIB_TEXTURE 1\n> > +\n> > +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> > +     : QOpenGLWidget(parent),\n> > +     glBuffer(QOpenGLBuffer::VertexBuffer),\n> > +     pFShader(nullptr),\n> > +     pVShader(nullptr),\n> > +     textureU(QOpenGLTexture::Target2D),\n> > +     textureV(QOpenGLTexture::Target2D),\n> > +     textureY(QOpenGLTexture::Target2D),\n> > +     yuvDataPtr(nullptr)\n> \n> Checkstyle  will tell you to align those vertically with the first\n> QOpenGLWidget(parent), rather than the ':'.\n> \n> \n> > +\n> > +{\n> > +}\n> > +\n> > +ViewFinderGL::~ViewFinderGL()\n> > +{\n> > +     removeShader();\n> > +\n> > +     if(textureY.isCreated())\n> \n> Space after if: \"if ()\"\n> \n> > +             textureY.destroy();\n> > +\n> > +     if(textureU.isCreated())\n> > +             textureU.destroy();\n> > +\n> > +     if(textureV.isCreated())\n> > +             textureV.destroy();\n> \n> For each of those ;-)\n> \n> There's plenty more too, but I'll let you find them with checkstyle.\n> \n> \n> > +\n> > +     glBuffer.destroy();\n> > +}\n> > +\n> > +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> > +                         const QSize &size)\n> > +{\n> > +     format_ = format;\n> > +     size_ = size;\n> > +\n> > +     updateGeometry();\n> > +     return 0;\n> > +}\n> > +\n> > +void ViewFinderGL::stop()\n> > +{\n> > +     if (buffer_) {\n> > +             renderComplete(buffer_);\n> > +             buffer_ = nullptr;\n> > +     }\n> > +}\n> > +\n> > +QImage ViewFinderGL::getCurrentImage()\n> > +{\n> > +     QMutexLocker locker(&mutex_);\n> > +\n> > +     return(grabFramebuffer());\n> \n> I think that could be :\n> \n>         return grabFrameBuffer();\n> \n> > +}\n> > +\n> > +void ViewFinderGL::render(libcamera::FrameBuffer *buffer,\n> MappedBuffer *map)\n> > +{\n> > +     if (buffer->planes().size() != 1) {\n> > +             qWarning() << \"Multi-planar buffers are not supported\";\n> > +             return;\n> > +     }\n> > +\n> > +     unsigned char *memory = static_cast *>(map->memory);\n> > +     if(memory) {\n> > +             yuvDataPtr = memory;\n> > +             update();\n> > +             buffer_ = buffer;\n> > +     }\n> > +\n> > +     if (buffer)\n> > +             renderComplete(buffer);\n> > +}\n> > +\n> > +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> > +{\n> > +     if(buffer) {\n> > +             yuvDataPtr = buffer;\n> > +             update();\n> > +     }\n> > +}\n> > +\n> > +void ViewFinderGL::setFrameSize(int width, int height)\n> \n> Can width/height be negative? Maybe the should be unsigned int?\n> \n> What about using a QSize too rather than individually passing\n> width/height?\n> \n> I can't see what calls setFrameSize(), is this an override from\n> \n> > +{\n> > +     if(width > 0 && height > 0) {\n> > +             width_ = width;\n> > +             height_ = height;\n> > +     }\n> \n> In fact, don't we already have a size_ in the class? (which could also\n> be in the base class too most likely...\n> \n> Sure, I will use QSize instead.\n> \n> \n> \n> > +}\n> > +\n> > +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> > +{\n> > +     char *fsrc = nullptr;\n> > +\n> > +     switch(format) {\n> > +     case libcamera::formats::NV12:\n> > +             horzSubSample_ = 2;\n> > +             vertSubSample_ = 2;\n> > +             fsrc = NV_2_planes_UV;\n> > +             break;\n> > +     case libcamera::formats::NV21:\n> > +             horzSubSample_ = 2;\n> > +             vertSubSample_ = 2;\n> > +             fsrc = NV_2_planes_VU;\n> > +             break;\n> > +     case libcamera::formats::NV16:\n> > +             horzSubSample_ = 2;\n> > +             vertSubSample_ = 1;\n> > +             fsrc = NV_2_planes_UV;\n> > +             break;\n> > +     case libcamera::formats::NV61:\n> > +             horzSubSample_ = 2;\n> > +             vertSubSample_ = 1;\n> > +             fsrc = NV_2_planes_VU;\n> > +             break;\n> > +     case libcamera::formats::NV24:\n> > +             horzSubSample_ = 1;\n> > +             vertSubSample_ = 1;\n> > +             fsrc = NV_2_planes_UV;\n> > +             break;\n> > +     case libcamera::formats::NV42:\n> > +             horzSubSample_ = 1;\n> > +             vertSubSample_ = 1;\n> > +             fsrc = NV_2_planes_VU;\n> > +             break;\n> > +     case libcamera::formats::YUV420:\n> > +             horzSubSample_ = 2;\n> > +             vertSubSample_ = 2;\n> > +             fsrc = NV_3_planes_UV;\n> > +             break;\n> > +     default:\n> > +             break;\n> > +     };\n> > +\n> > +     if(fsrc == nullptr) {\n> > +             qDebug() << __func__ << \"[ERROR]:\" <<\" No suitable\n> fragment shader can be select.\";\n> \n> 'selected'\n> \n> Ok. Will fix it.\n> \n> \n> \n> > +     }\n> > +     return fsrc;\n> > +}\n> > +\n> > +void ViewFinderGL::createFragmentShader()\n> > +{\n> > +     bool bCompile;\n> > +\n> > +     pFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> > +     char *fsrc = selectFragmentShader(format_);\n> > +\n> > +     bCompile = pFShader->compileSourceCode(fsrc);\n> > +     if(!bCompile)\n> > +     {\n> > +             qDebug() << __func__ << \":\" << pFShader->log();\n> > +     }\n> > +\n> > +     shaderProgram.addShader(pFShader);\n> > +\n> > +     // Link shader pipeline\n> \n> Even though it's C++, we usually use C style comments...\n> \n> \n> > +     if (!shaderProgram.link()) {\n> > +             qDebug() << __func__ << \": shader program link\n> failed.\\n\" << shaderProgram.log();\n> > +             close();\n> > +     }\n> > +\n> > +     // Bind shader pipeline for use\n> > +     if (!shaderProgram.bind()) {\n> > +             qDebug() << __func__ << \": shader program binding\n> failed.\\n\" << shaderProgram.log();\n> > +             close();\n> > +     }\n> > +\n> > +     shaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> > +     shaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> > +\n> > +   \n>  shaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> > +   \n>  shaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> > +\n> > +     textureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> > +     textureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> > +     textureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> > +\n> > +     if(!textureY.isCreated())\n> > +             textureY.create();\n> > +\n> > +     if(!textureU.isCreated())\n> > +             textureU.create();\n> > +\n> > +     if(!textureV.isCreated())\n> > +             textureV.create();\n> > +\n> > +     id_y = textureY.textureId();\n> > +     id_u = textureU.textureId();\n> > +     id_v = textureV.textureId();\n> > +}\n> > +\n> > +void ViewFinderGL::configureTexture(unsigned int id)\n> > +{\n> > +     glBindTexture(GL_TEXTURE_2D, id);\n> > +     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER,\n> GL_LINEAR);\n> > +     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER,\n> GL_LINEAR);\n> > +     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S,\n> GL_CLAMP_TO_EDGE);\n> > +     glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T,\n> GL_CLAMP_TO_EDGE);\n> > +}\n> > +\n> > +void ViewFinderGL::removeShader()\n> > +{\n> > +     if (shaderProgram.isLinked()) {\n> > +             shaderProgram.release();\n> > +             shaderProgram.removeAllShaders();\n> > +     }\n> > +\n> > +     if(pFShader)\n> > +             delete pFShader;\n> > +\n> > +     if(pVShader)\n> > +             delete pVShader;\n> > +}\n> > +\n> > +void ViewFinderGL::initializeGL()\n> > +{\n> > +     bool bCompile;\n> > +\n> > +     initializeOpenGLFunctions();\n> > +     glEnable(GL_TEXTURE_2D);\n> > +     glDisable(GL_DEPTH_TEST);\n> > +\n> > +     static const GLfloat vertices[] {\n> > +             -1.0f,-1.0f,\n> > +             -1.0f,+1.0f,\n> > +             +1.0f,+1.0f,\n> > +             +1.0f,-1.0f,\n> > +             0.0f,1.0f,\n> > +             0.0f,0.0f,\n> > +             1.0f,0.0f,\n> > +             1.0f,1.0f,\n> \n> When you get here, checkstyle will suggest putting this all in one\n> vertical column.\n> \n> Ignore it... (checkstyle is just advice, and in this case your layout is\n> better).\n> \n> Though I would add a space at least after the first ',' on each line to\n> make the columns clearer., Or go further and align the 'f's... ?\n> \n> > +     };\n> > +\n> > +     glBuffer.create();\n> > +     glBuffer.bind();\n> > +     glBuffer.allocate(vertices,sizeof(vertices));\n> > +\n> > +     /* Create Vertex Shader */\n> > +     pVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> > +     char *vsrc =  NV_Vertex_shader;\n> > +\n> > +��    bCompile = pVShader->compileSourceCode(vsrc);\n> > +     if(!bCompile) {\n> > +             qDebug() << __func__<< \":\" << pVShader->log();\n> > +     }\n> > +\n> > +     shaderProgram.addShader(pVShader);\n> > +\n> > +     glClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> > +}\n> > +\n> > +void ViewFinderGL::render()\n> > +{\n> > +     switch(format_) {\n> > +     case libcamera::formats::NV12:\n> > +     case libcamera::formats::NV21:\n> > +     case libcamera::formats::NV16:\n> > +     case libcamera::formats::NV61:\n> > +     case libcamera::formats::NV24:\n> > +     case libcamera::formats::NV42:\n> > +             /* activate texture 0 */\n> > +             glActiveTexture(GL_TEXTURE0);\n> > +             configureTexture(id_y);\n> > +             glTexImage2D(GL_TEXTURE_2D, 0, GL_RED,\n> size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > +             glUniform1i(textureUniformY, 0);\n> > +\n> > +             /* activate texture 1 */\n> > +             glActiveTexture(GL_TEXTURE1);\n> > +             configureTexture(id_u);\n> > +             glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0,\n> GL_RG, GL_UNSIGNED_BYTE,\n> (char*)yuvDataPtr+size_.width()*size_.height());\n> > +             glUniform1i(textureUniformU, 1);\n> > +             break;\n> > +     case libcamera::formats::YUV420:\n> > +             /* activate texture 0 */\n> > +             glActiveTexture(GL_TEXTURE0);\n> > +             configureTexture(id_y);\n> > +             glTexImage2D(GL_TEXTURE_2D, 0, GL_RED,\n> size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > +             glUniform1i(textureUniformY, 0);\n> > +\n> > +             /* activate texture 1 */\n> > +             glActiveTexture(GL_TEXTURE1);\n> > +             configureTexture(id_u);\n> > +             glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0,\n> GL_RG, GL_UNSIGNED_BYTE,\n> (char*)yuvDataPtr+size_.width()*size_.height());\n> > +             glUniform1i(textureUniformU, 1);\n> > +\n> > +             /* activate texture 2 */\n> > +             glActiveTexture(GL_TEXTURE2);\n> > +             configureTexture(id_v);\n> > +             glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0,\n> GL_RG, GL_UNSIGNED_BYTE,\n> (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> > +             glUniform1i(textureUniformV, 2);\n> > +             break;\n> > +     default:\n> > +             break;\n> > +     };\n> > +}\n> > +\n> > +void ViewFinderGL::paintGL()\n> > +{\n> > +     if(pFShader == nullptr)\n> > +             createFragmentShader();\n> > +\n> > +     if(yuvDataPtr)\n> > +     {\n> > +             glClearColor(0.0, 0.0, 0.0, 1.0);\n> > +             glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> > +\n> > +             render();\n> > +             glDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> > +     }\n> > +}\n> > +\n> > +void ViewFinderGL::resizeGL(int w, int h)\n> > +{\n> > +     glViewport(0,0,w,h);\n> > +}\n> > +\n> > +QSize ViewFinderGL::sizeHint() const\n> > +{\n> > +     return size_.isValid() ? size_ : QSize(640, 480);\n> > +}\n> \n> I wonder if this sizeHint should go to the base class?\n> \n> Does ViewFinderGL use it? I think it's there to provide a required\n> overload for the QWidget ... if it's common it could go to the base, and\n> if it's not used, it could get dropped.\n> \n> Got  it.\n> \n> \n> \n> \n> > diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> > new file mode 100644\n> > index 0000000..08662ca\n> > --- /dev/null\n> > +++ b/src/qcam/viewfinderGL.h\n> > @@ -0,0 +1,101 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> > + */\n> > +#ifndef __VIEWFINDERGL_H__\n> \n> Perhaps to make this clearer use this: __VIEWFINDER_GL_H__ ?\n> An underscore between the ViewFinder and the GL would separate the base,\n> and the derived names...\n> Ok, I will fix this.\n> \n> > +#define __VIEWFINDERGL_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \"viewfinder.h\"\n> > +\n> > +class ViewFinderGL : public QOpenGLWidget,\n> > +                                      public ViewFinderHandler,\n> > +                                      protected QOpenGLFunctions\n> \n> That indentation looks off, I think they should be aligned vertically at\n> least.\n> \n> Sure.\n> \n> \n> \n> > +{\n> > +     Q_OBJECT\n> > +\n> > +public:\n> > +     ViewFinderGL(QWidget *parent = 0);\n> > +     ~ViewFinderGL();\n> > +\n> > +     int setFormat(const libcamera::PixelFormat &format, const\n> QSize &size);\n> > +     void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > +     void stop();\n> > +\n> > +     QImage getCurrentImage();\n> \n> Aha - that's why we need QImage, I think that's part of the save routine\n> perhaps isn't it ...\n> \n> > +\n> > +     void setFrameSize(int width, int height);\n> \n> I can't see what calls setFrameSize... is it redundant? Or is it an\n> override used externally or such. If so I think it needs to be marked as\n> accordingly as an override?\n> \n> > +     void updateFrame(unsigned char  *buffer);\n> \n> Same here... What calls updateFrame()?\n> \n> Actually, only NV12, I can verify with Libcamera on rock pi 4, I use an\n> external application to verify other shaders and above functions are for\n> that.\n> I will remove that in next version.\n> \n> \n> \n> > +\n> > +     char *selectFragmentShader(unsigned int format);\n> > +     void createFragmentShader();\n> > +     void render();\n> > +\n> > +Q_SIGNALS:\n> > +     void renderComplete(libcamera::FrameBuffer *buffer);\n> > +\n> > +protected:\n> > +     void initializeGL() Q_DECL_OVERRIDE;\n> > +     void paintGL() Q_DECL_OVERRIDE;\n> > +     void resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> > +     QSize sizeHint() const override;\n> \n> is sizeHint() used?\n> \n> \n> > +\n> > +private:\n> > +\n> > +     void configureTexture(unsigned int id);\n> > +     void removeShader();\n> > +\n> > +     /* Captured image size, format and buffer */\n> > +     libcamera::FrameBuffer *buffer_;\n> > +     libcamera::PixelFormat format_;\n> > +     QOpenGLBuffer glBuffer;\n> \n> glBuffer_;\n> \n> Will fix it.\n> \n> \n> \n> \n> > +     QSize size_;\n> > +\n> > +     GLuint id_u;\n> > +     GLuint id_v;\n> > +     GLuint id_y;\n> > +\n> > +     QMutex mutex_; /* Prevent concurrent access to image_ */\n> > +\n> \n> All of the member variables below should have a '_' suffix...\n> \n> > +     QOpenGLShader *pFShader;\n> > +     QOpenGLShader *pVShader;\n> > +     QOpenGLShaderProgram shaderProgram;\n> > +\n> > +     GLuint textureUniformU;\n> > +     GLuint textureUniformV;\n> > +     GLuint textureUniformY;\n> > +\n> > +     QOpenGLTexture textureU;\n> > +     QOpenGLTexture textureV;\n> > +     QOpenGLTexture textureY;\n> > +\n> > +     unsigned int width_;\n> > +     unsigned int height_;\n> \n> There is already a QSize size_, do these duplicate that purpose?\n> \n> Yeah, I will use size_ instead.\n> \n> \n> > +\n> > +     unsigned char* yuvDataPtr;\n> > +\n> > +     /* NV parameters */\n> > +     unsigned int horzSubSample_ ;\n> \n> Extra space?\n> \n> Ok will fix it.\n> \n> \n> > +     unsigned int vertSubSample_;\n> > +};\n> > +#endif // __VIEWFINDERGL_H__\n> >\n> \n> \n> \n> -- \n> Regards\n> --\n> Kieran\n> \n> \n> Thank you for your comments again.\n> I will post next version asap.\n> \n> --\n> Regards\n> --\n> Show\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 69981C0109\n\tfor ;\n\tThu, 25 Jun 2020 11:36:49 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E17C4609C2;\n\tThu, 25 Jun 2020 13:36:48 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D790B609A5\n\tfor ;\n\tThu, 25 Jun 2020 13:36:46 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 10026535;\n\tThu, 25 Jun 2020 13:36:45 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"eOjaYSdT\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1593085006;\n\tbh=21gA9BrQIIW9yT4bB41WigjYWmeswX3brXVM6KXs1es=;\n\th=Reply-To:Subject:To:Cc:References:From:Date:In-Reply-To:From;\n\tb=eOjaYSdTcVTx3ifScmpD8w/w50Q4Va72dnC2odK0njVOz8N+aQg5jiQni14w5ZfVp\n\t5uGZDW1XNo8dh0l4c23KGaiphry+SuY4K5seIzhqZOvKrypg6pT0gVtUyHyoazlEmH\n\trQsWxA2FGH8f/yc4NpQJ3tkaobEtsYakNSo7OIhI=","To":"Show Liu ","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>\n\t<35bd63f1-45e4-8b5d-b914-84375b662ce9@ideasonboard.com>\n\t","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<3a34e009-5877-4420-3da4-6f3153394f5f@ideasonboard.com>","Date":"Thu, 25 Jun 2020 12:36:43 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.8.0","MIME-Version":"1.0","In-Reply-To":"","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11026,"web_url":"https://patchwork.libcamera.org/comment/11026/","msgid":"<20200630150343.GA2259976@oden.dyn.berto.se>","date":"2020-06-30T15:03:43","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Show,\n\nThanks for your work.\n\nI really like this version! The structure is almost there and much \nbetter then previous versions. As Kieran points out there are style \nerrors that checkstyle.py will help you point out so I will ignore them \nin this review.\n\nOn 2020-06-24 15:37:05 +0800, Show Liu wrote:\n> The patch is to render the NV family YUV formats by OpenGL shader.\n> V3: refine the fragment shader for better pixel color.\n> \n> Signed-off-by: Show Liu \n> ---\n> src/qcam/main.cpp | 2 +\n> src/qcam/main_window.cpp | 19 ++-\n> src/qcam/main_window.h | 3 +-\n> src/qcam/meson.build | 2 +\n> src/qcam/shader.h | 104 ++++++++++++\n> src/qcam/viewfinder.cpp | 18 +-\n> src/qcam/viewfinder.h | 23 ++-\n> src/qcam/viewfinderGL.cpp | 335 ++++++++++++++++++++++++++++++++++++++\n> src/qcam/viewfinderGL.h | 101 ++++++++++++\n> 9 files changed, 593 insertions(+), 14 deletions(-)\n> create mode 100644 src/qcam/shader.h\n> create mode 100644 src/qcam/viewfinderGL.cpp\n> create mode 100644 src/qcam/viewfinderGL.h\n> \n> diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> index b3468cb..a3ea471 100644\n> --- a/src/qcam/main.cpp\n> +++ b/src/qcam/main.cpp\n> @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc, char *argv[])\n> \t\t\t \"help\");\n> \tparser.addOption(OptStream, &streamKeyValue,\n> \t\t\t \"Set configuration of a camera stream\", \"stream\", true);\n> +\tparser.addOption(OptOpenGL, OptionNone, \"Render YUV formats frame via OpenGL shader\",\n> +\t\t\t \"opengl\");\n> \n> \tOptionsParser::Options options = parser.parse(argc, argv);\n> \tif (options.isSet(OptHelp))\n> diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> index 7bc1360..6edf370 100644\n> --- a/src/qcam/main_window.cpp\n> +++ b/src/qcam/main_window.cpp\n> @@ -28,6 +28,9 @@\n> #include \n> \n> #include \"dng_writer.h\"\n> +#include \"main_window.h\"\n> +#include \"viewfinder.h\"\n> +#include \"viewfinderGL.h\"\n> \n> using namespace libcamera;\n> \n> @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm, const OptionsParser::Options &options)\n> \tsetWindowTitle(title_);\n> \tconnect(&titleTimer_, SIGNAL(timeout()), this, SLOT(updateTitle()));\n> \n> -\tviewfinder_ = new ViewFinder(this);\n> -\tconnect(viewfinder_, &ViewFinder::renderComplete,\n> -\t\tthis, &MainWindow::queueRequest);\n> -\tsetCentralWidget(viewfinder_);\n> +\tif (options_.isSet(OptOpenGL)) {\n> +\t\tviewfinder_ = new ViewFinderGL(this);\n> +\t\tconnect(dynamic_cast(viewfinder_), &ViewFinderGL::renderComplete,\n> +\t\t\t\tthis, &MainWindow::queueRequest);\n> +\t\tsetCentralWidget(dynamic_cast(viewfinder_));\n> +\t} else {\n> +\t\tviewfinder_ = new ViewFinder(this);\n> +\t\tconnect(dynamic_cast(viewfinder_), &ViewFinder::renderComplete,\n> +\t\t\t\tthis, &MainWindow::queueRequest);\n> +\t\tsetCentralWidget(dynamic_cast(viewfinder_));\n> +\t}\n\nI understand that one can't inherit from QObject twice, but this looks \nodd. Is there someway the base class could inherit from QObject or \npossibly QWidget and the derived classes hide their specilization \nsomehow? I'm no Qt export so maybe I'm asking for something impossible \nor really stupid.\n\n> +\n> \tadjustSize();\n> \n> \t/* Hotplug/unplug support */\n> diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> index 4606fe4..a852ef4 100644\n> --- a/src/qcam/main_window.h\n> +++ b/src/qcam/main_window.h\n> @@ -38,6 +38,7 @@ enum {\n> \tOptCamera = 'c',\n> \tOptHelp = 'h',\n> \tOptStream = 's',\n> +\tOptOpenGL = 'o',\n> };\n> \n> class CaptureRequest\n> @@ -102,7 +103,7 @@ private:\n> \tQAction *startStopAction_;\n> \tQComboBox *cameraCombo_;\n> \tQAction *saveRaw_;\n> -\tViewFinder *viewfinder_;\n> +\tViewFinderHandler *viewfinder_;\n> \n> \tQIcon iconPlay_;\n> \tQIcon iconStop_;\n> diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> index 045db52..6a58947 100644\n> --- a/src/qcam/meson.build\n> +++ b/src/qcam/meson.build\n> @@ -7,11 +7,13 @@ qcam_sources = files([\n> 'main.cpp',\n> 'main_window.cpp',\n> 'viewfinder.cpp',\n> + 'viewfinderGL.cpp'\n> ])\n> \n> qcam_moc_headers = files([\n> 'main_window.h',\n> 'viewfinder.h',\n> + 'viewfinderGL.h'\n> ])\n> \n> qcam_resources = files([\n> diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> new file mode 100644\n> index 0000000..f54c264\n> --- /dev/null\n> +++ b/src/qcam/shader.h\n> @@ -0,0 +1,104 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Linaro\n> + *\n> + * shader.h - YUV format converter shader source code\n> + */\n> +#ifndef __SHADER_H__\n> +#define __SHADER_H__\n\nI think the content of this file should be moved inside viewfinderGL.cpp\n\n> +\n> +/* Vertex shader for NV formats */\n> +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n\ncould this (and bellow) be made static const ?\n\n> + \"attribute vec2 textureIn;\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"gl_Position = vertexIn;\\n\"\n> + \"textureOut = textureIn;\\n\"\n> + \"}\\n\";\n> +\n> +/* Fragment shader for NV12, NV16 and NV24 */\n> +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n> + \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> + \"}\\n\";\n> +\n> +/* Fragment shader for NV21, NV61 and NV42 */\n> +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> + \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> + \"}\\n\";\n> +\n> +/* Fragment shader for YUV420 */\n> +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> + \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"uniform sampler2D tex_v;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> + \"}\\n\";\n> +\n> +/* Fragment shader for YVU420 */\n> +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> + \"#endif\\n\"\n> + \"varying vec2 textureOut;\\n\"\n> + \"uniform sampler2D tex_y;\\n\"\n> + \"uniform sampler2D tex_u;\\n\"\n> + \"uniform sampler2D tex_v;\\n\"\n> + \"void main(void)\\n\"\n> + \"{\\n\"\n> + \"vec3 yuv;\\n\"\n> + \"vec3 rgb;\\n\"\n> + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> + \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> + \"rgb = convert_mat * yuv;\\n\"\n> + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> + \"}\\n\";\n> +#endif // __SHADER_H__\n> diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> index dcf8a15..d3a2422 100644\n> --- a/src/qcam/viewfinder.cpp\n> +++ b/src/qcam/viewfinder.cpp\n> @@ -33,22 +33,30 @@ static const QMap nativeFormats\n> \t{ libcamera::formats::BGR888, QImage::Format_RGB888 },\n> };\n> \n> -ViewFinder::ViewFinder(QWidget *parent)\n> -\t: QWidget(parent), buffer_(nullptr)\n> +ViewFinderHandler::ViewFinderHandler()\n> {\n> -\ticon_ = QIcon(\":camera-off.svg\");\n> }\n> \n> -ViewFinder::~ViewFinder()\n> +ViewFinderHandler::~ViewFinderHandler()\n> {\n> }\n> \n> -const QList &ViewFinder::nativeFormats() const\n> +const QList &ViewFinderHandler::nativeFormats() const\n> {\n> \tstatic const QList formats = ::nativeFormats.keys();\n> \treturn formats;\n> }\n> \n> +ViewFinder::ViewFinder(QWidget *parent)\n> +\t: QWidget(parent), buffer_(nullptr)\n> +{\n> +\ticon_ = QIcon(\":camera-off.svg\");\n> +}\n\nI agree with Kieran here I think the base class should be named \nViewFinder and the two derived classes ViewFinderQt and ViewFinderGL or \nsomething similar.\n\nI also think you should move the base class to its own .h file (and .cpp \nfile if needed).\n\n> +\n> +ViewFinder::~ViewFinder()\n> +{\n> +}\n> +\n> int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> \t\t\t const QSize &size)\n> {\n> diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> index 26a1320..741d694 100644\n> --- a/src/qcam/viewfinder.h\n> +++ b/src/qcam/viewfinder.h\n> @@ -13,6 +13,8 @@\n> #include \n> #include \n> #include \n> +#include \n> +#include \n> #include \n> #include \n> \n> @@ -28,7 +30,23 @@ struct MappedBuffer {\n> \tsize_t size;\n> };\n> \n> -class ViewFinder : public QWidget\n> +class ViewFinderHandler\n> +{\n> +public:\n> +\tViewFinderHandler();\n> +\tvirtual ~ViewFinderHandler();\n> +\n> +\tconst QList &nativeFormats() const;\n> +\n> +\tvirtual int setFormat(const libcamera::PixelFormat &format, const QSize &size) =0;\n> +\tvirtual void render(libcamera::FrameBuffer *buffer, MappedBuffer *map) =0;\n> +\tvirtual void stop() =0;\n> +\n> +\tvirtual QImage getCurrentImage() =0;\n> +\n> +};\n> +\n> +class ViewFinder : public QWidget, public ViewFinderHandler\n> {\n> \tQ_OBJECT\n> \n> @@ -36,8 +54,6 @@ public:\n> \tViewFinder(QWidget *parent);\n> \t~ViewFinder();\n> \n> -\tconst QList &nativeFormats() const;\n> -\n> \tint setFormat(const libcamera::PixelFormat &format, const QSize &size);\n> \tvoid render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> \tvoid stop();\n> @@ -67,5 +83,4 @@ private:\n> \tQImage image_;\n> \tQMutex mutex_; /* Prevent concurrent access to image_ */\n> };\n> -\n> #endif /* __QCAM_VIEWFINDER__ */\n> diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> new file mode 100644\n> index 0000000..7b47beb\n> --- /dev/null\n> +++ b/src/qcam/viewfinderGL.cpp\n> @@ -0,0 +1,335 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Linaro\n> + *\n> + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> + */\n> +\n> +#include \"shader.h\"\n> +#include \"viewfinderGL.h\"\n> +\n> +#include \n> +\n> +#include \n> +\n> +#define ATTRIB_VERTEX 0\n> +#define ATTRIB_TEXTURE 1\n> +\n> +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> +\t: QOpenGLWidget(parent),\n> +\tglBuffer(QOpenGLBuffer::VertexBuffer),\n> +\tpFShader(nullptr),\n> +\tpVShader(nullptr),\n> +\ttextureU(QOpenGLTexture::Target2D),\n> +\ttextureV(QOpenGLTexture::Target2D),\n> +\ttextureY(QOpenGLTexture::Target2D),\n> +\tyuvDataPtr(nullptr)\n> +\n> +{\n> +}\n> +\n> +ViewFinderGL::~ViewFinderGL()\n> +{\n> +\tremoveShader();\n> +\n> +\tif(textureY.isCreated())\n> +\t\ttextureY.destroy();\n> +\n> +\tif(textureU.isCreated())\n> +\t\ttextureU.destroy();\n> +\n> +\tif(textureV.isCreated())\n> +\t\ttextureV.destroy();\n> +\n> +\tglBuffer.destroy();\n> +}\n> +\n> +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> +\t\t\t const QSize &size)\n> +{\n> +\tformat_ = format;\n> +\tsize_ = size;\n> +\n> +\tupdateGeometry();\n> +\treturn 0;\n> +}\n> +\n> +void ViewFinderGL::stop()\n> +{\n> +\tif (buffer_) {\n> +\t\trenderComplete(buffer_);\n> +\t\tbuffer_ = nullptr;\n> +\t}\n> +}\n> +\n> +QImage ViewFinderGL::getCurrentImage()\n> +{\n> +\tQMutexLocker locker(&mutex_);\n> +\n> +\treturn(grabFramebuffer());\n> +}\n> +\n> +void ViewFinderGL::render(libcamera::FrameBuffer *buffer, MappedBuffer *map)\n> +{\n> +\tif (buffer->planes().size() != 1) {\n> +\t\tqWarning() << \"Multi-planar buffers are not supported\";\n> +\t\treturn;\n> +\t}\n> +\n> +\tunsigned char *memory = static_cast(map->memory);\n> +\tif(memory) {\n> +\t\tyuvDataPtr = memory;\n> +\t\tupdate();\n> +\t\tbuffer_ = buffer;\n> +\t}\n> +\n> +\tif (buffer)\n> +\t\trenderComplete(buffer);\n> +}\n> +\n> +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> +{\n> +\tif(buffer) {\n> +\t\tyuvDataPtr = buffer;\n> +\t\tupdate();\n> +\t}\n> +}\n> +\n> +void ViewFinderGL::setFrameSize(int width, int height)\n> +{\n> +\tif(width > 0 && height > 0) {\n> +\t\twidth_ = width;\n> +\t\theight_ = height;\n> +\t}\n> +}\n> +\n> +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> +{\n> +\tchar *fsrc = nullptr;\n> +\n> +\tswitch(format) {\n> +\tcase libcamera::formats::NV12:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 2;\n> +\t\tfsrc = NV_2_planes_UV;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV21:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 2;\n> +\t\tfsrc = NV_2_planes_VU;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV16:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_UV;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV61:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_VU;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV24:\n> +\t\thorzSubSample_ = 1;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_UV;\n> +\t\tbreak;\n> +\tcase libcamera::formats::NV42:\n> +\t\thorzSubSample_ = 1;\n> +\t\tvertSubSample_ = 1;\n> +\t\tfsrc = NV_2_planes_VU;\n> +\t\tbreak;\n> +\tcase libcamera::formats::YUV420:\n> +\t\thorzSubSample_ = 2;\n> +\t\tvertSubSample_ = 2;\n> +\t\tfsrc = NV_3_planes_UV;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tbreak;\n> +\t};\n> +\n> +\tif(fsrc == nullptr) {\n> +\t\tqDebug() << __func__ << \"[ERROR]:\" <<\" No suitable fragment shader can be select.\";\n> +\t}\n> +\treturn fsrc;\n> +}\n> +\n> +void ViewFinderGL::createFragmentShader()\n> +{\n> +\tbool bCompile;\n> +\n> +\tpFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> +\tchar *fsrc = selectFragmentShader(format_);\n> +\n> +\tbCompile = pFShader->compileSourceCode(fsrc);\n> +\tif(!bCompile)\n> +\t{\n> +\t\tqDebug() << __func__ << \":\" << pFShader->log();\n> +\t}\n> +\n> +\tshaderProgram.addShader(pFShader);\n> +\n> +\t// Link shader pipeline\n> +\tif (!shaderProgram.link()) {\n> +\t\tqDebug() << __func__ << \": shader program link failed.\\n\" << shaderProgram.log();\n> +\t\tclose();\n> +\t}\n> +\n> +\t// Bind shader pipeline for use\n> +\tif (!shaderProgram.bind()) {\n> +\t\tqDebug() << __func__ << \": shader program binding failed.\\n\" << shaderProgram.log();\n> +\t\tclose();\n> +\t}\n> +\n> +\tshaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> +\tshaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> +\n> +\tshaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> +\tshaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> +\n> +\ttextureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> +\ttextureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> +\ttextureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> +\n> +\tif(!textureY.isCreated())\n> +\t\ttextureY.create();\n> +\n> +\tif(!textureU.isCreated())\n> +\t\ttextureU.create();\n> +\n> +\tif(!textureV.isCreated())\n> +\t\ttextureV.create();\n> +\n> +\tid_y = textureY.textureId();\n> +\tid_u = textureU.textureId();\n> +\tid_v = textureV.textureId();\n> +}\n> +\n> +void ViewFinderGL::configureTexture(unsigned int id)\n> +{\n> +\tglBindTexture(GL_TEXTURE_2D, id);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);\n> +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);\n> +}\n> +\n> +void ViewFinderGL::removeShader()\n> +{\n> +\tif (shaderProgram.isLinked()) {\n> +\t\tshaderProgram.release();\n> +\t\tshaderProgram.removeAllShaders();\n> +\t}\n> +\n> +\tif(pFShader)\n> +\t\tdelete pFShader;\n> +\n> +\tif(pVShader)\n> +\t\tdelete pVShader;\n> +}\n> +\n> +void ViewFinderGL::initializeGL()\n> +{\n> +\tbool bCompile;\n> +\n> +\tinitializeOpenGLFunctions();\n> +\tglEnable(GL_TEXTURE_2D);\n> +\tglDisable(GL_DEPTH_TEST);\n> +\n> +\tstatic const GLfloat vertices[] {\n> +\t\t-1.0f,-1.0f,\n> +\t\t-1.0f,+1.0f,\n> +\t\t+1.0f,+1.0f,\n> +\t\t+1.0f,-1.0f,\n> +\t\t0.0f,1.0f,\n> +\t\t0.0f,0.0f,\n> +\t\t1.0f,0.0f,\n> +\t\t1.0f,1.0f,\n> +\t};\n> +\n> +\tglBuffer.create();\n> +\tglBuffer.bind();\n> +\tglBuffer.allocate(vertices,sizeof(vertices));\n> +\n> +\t/* Create Vertex Shader */\n> +\tpVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> +\tchar *vsrc = NV_Vertex_shader;\n> +\n> +\tbCompile = pVShader->compileSourceCode(vsrc);\n> +\tif(!bCompile) {\n> +\t\tqDebug() << __func__<< \":\" << pVShader->log();\n> +\t}\n> +\n> +\tshaderProgram.addShader(pVShader);\n> +\n> +\tglClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> +}\n> +\n> +void ViewFinderGL::render()\n> +{\n> +\tswitch(format_) {\n> +\tcase libcamera::formats::NV12:\n> +\tcase libcamera::formats::NV21:\n> +\tcase libcamera::formats::NV16:\n> +\tcase libcamera::formats::NV61:\n> +\tcase libcamera::formats::NV24:\n> +\tcase libcamera::formats::NV42:\n> +\t\t/* activate texture 0 */\n> +\t\tglActiveTexture(GL_TEXTURE0);\n> +\t\tconfigureTexture(id_y);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> +\t\tglUniform1i(textureUniformY, 0);\n> +\n> +\t\t/* activate texture 1 */\n> +\t\tglActiveTexture(GL_TEXTURE1);\n> +\t\tconfigureTexture(id_u);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> +\t\tglUniform1i(textureUniformU, 1);\n> +\t\tbreak;\n> +\tcase libcamera::formats::YUV420:\n> +\t\t/* activate texture 0 */\n> +\t\tglActiveTexture(GL_TEXTURE0);\n> +\t\tconfigureTexture(id_y);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> +\t\tglUniform1i(textureUniformY, 0);\n> +\n> +\t\t/* activate texture 1 */\n> +\t\tglActiveTexture(GL_TEXTURE1);\n> +\t\tconfigureTexture(id_u);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> +\t\tglUniform1i(textureUniformU, 1);\n> +\n> +\t\t/* activate texture 2 */\n> +\t\tglActiveTexture(GL_TEXTURE2);\n> +\t\tconfigureTexture(id_v);\n> +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> +\t\tglUniform1i(textureUniformV, 2);\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tbreak;\n> +\t};\n> +}\n> +\n> +void ViewFinderGL::paintGL()\n> +{\n> +\tif(pFShader == nullptr)\n> +\t\tcreateFragmentShader();\n> +\n> +\tif(yuvDataPtr)\n> +\t{\n> +\t\tglClearColor(0.0, 0.0, 0.0, 1.0);\n> +\t\tglClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> +\n> +\t\trender();\n> +\t\tglDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> +\t}\n> +}\n> +\n> +void ViewFinderGL::resizeGL(int w, int h)\n> +{\n> +\tglViewport(0,0,w,h);\n> +}\n> +\n> +QSize ViewFinderGL::sizeHint() const\n> +{\n> +\treturn size_.isValid() ? size_ : QSize(640, 480);\n> +}\n> diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> new file mode 100644\n> index 0000000..08662ca\n> --- /dev/null\n> +++ b/src/qcam/viewfinderGL.h\n> @@ -0,0 +1,101 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Linaro\n> + *\n> + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> + */\n> +#ifndef __VIEWFINDERGL_H__\n> +#define __VIEWFINDERGL_H__\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \"viewfinder.h\"\n> +\n> +class ViewFinderGL : public QOpenGLWidget,\n> +\t\t\t\t\t public ViewFinderHandler,\n> +\t\t\t\t\t protected QOpenGLFunctions\n> +{\n> +\tQ_OBJECT\n> +\n> +public:\n> +\tViewFinderGL(QWidget *parent = 0);\n> +\t~ViewFinderGL();\n> +\n> +\tint setFormat(const libcamera::PixelFormat &format, const QSize &size);\n> +\tvoid render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> +\tvoid stop();\n> +\n> +\tQImage getCurrentImage();\n> +\n> +\tvoid setFrameSize(int width, int height);\n> +\tvoid updateFrame(unsigned char *buffer);\n> +\n> +\tchar *selectFragmentShader(unsigned int format);\n> +\tvoid createFragmentShader();\n> +\tvoid render();\n> +\n> +Q_SIGNALS:\n> +\tvoid renderComplete(libcamera::FrameBuffer *buffer);\n> +\n> +protected:\n> +\tvoid initializeGL() Q_DECL_OVERRIDE;\n> +\tvoid paintGL() Q_DECL_OVERRIDE;\n> +\tvoid resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> +\tQSize sizeHint() const override;\n> +\n> +private:\n> +\n> +\tvoid configureTexture(unsigned int id);\n> +\tvoid removeShader();\n> +\n> +\t/* Captured image size, format and buffer */\n> +\tlibcamera::FrameBuffer *buffer_;\n> +\tlibcamera::PixelFormat format_;\n> +\tQOpenGLBuffer glBuffer;\n> +\tQSize size_;\n> +\n> +\tGLuint id_u;\n> +\tGLuint id_v;\n> +\tGLuint id_y;\n> +\n> +\tQMutex mutex_; /* Prevent concurrent access to image_ */\n> +\n> +\tQOpenGLShader *pFShader;\n> +\tQOpenGLShader *pVShader;\n> +\tQOpenGLShaderProgram shaderProgram;\n> +\n> +\tGLuint textureUniformU;\n> +\tGLuint textureUniformV;\n> +\tGLuint textureUniformY;\n> +\n> +\tQOpenGLTexture textureU;\n> +\tQOpenGLTexture textureV;\n> +\tQOpenGLTexture textureY;\n> +\n> +\tunsigned int width_;\n> +\tunsigned int height_;\n> +\n> +\tunsigned char* yuvDataPtr;\n> +\n> +\t/* NV parameters */\n> +\tunsigned int horzSubSample_ ;\n> +\tunsigned int vertSubSample_;\n> +};\n> +#endif // __VIEWFINDERGL_H__\n> -- \n> 2.20.1\n> \n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 16138BFFE2\n\tfor ;\n\tTue, 30 Jun 2020 15:03:48 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D98AA60C5B;\n\tTue, 30 Jun 2020 17:03:47 +0200 (CEST)","from mail-lj1-x235.google.com (mail-lj1-x235.google.com\n\t[IPv6:2a00:1450:4864:20::235])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 14750609C5\n\tfor ;\n\tTue, 30 Jun 2020 17:03:47 +0200 (CEST)","by mail-lj1-x235.google.com with SMTP id d17so8287544ljl.3\n\tfor ;\n\tTue, 30 Jun 2020 08:03:47 -0700 (PDT)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tf24sm802645ljk.125.2020.06.30.08.03.43\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tTue, 30 Jun 2020 08:03:43 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"QTNIG55v\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=a8ROWEiqubE0v/xz0K2dTtgY7KSkAmvA9Y0JXXZEPr0=;\n\tb=QTNIG55vLfxF2YXJg4zu48nisG3paApCDyYVA3ZYtpHFeDCt+kHqI7B77nykZwuK84\n\tASfspUn4g3ybNh3trEiU5QkrbvfKQzWpDPNlSaIPJa0ECW+sIgmrVl4+BmNFwRyfoMg6\n\thCkjxnOQIDvr2R3cKZoXb9rECdthAnQOVYsc5ojS/vakh+SFwis9sBrIFFEfIn9NLw1z\n\t1fKznPngU3tCVkmli1MoXT/OcIuj2d4SdoPCB4mCMzJ1KYxqn6+TGZnxgo5+ojSqAacA\n\tPgU4BSXZX0r18zpjqLHpODwR8Dcztn753eDT0zIH9MNQuRQ22BOVQKwYMgrrE3spcRxJ\n\tIcMA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=a8ROWEiqubE0v/xz0K2dTtgY7KSkAmvA9Y0JXXZEPr0=;\n\tb=S+MyW5k7rSzmYhaZ3GcGT1dujvQnoE6X+so3z8tRCt+MI+FLHqqRhyH6KqS3VkcYDS\n\td8Y+GksiArA/VY6IIowLLaBk6b0JPmjb7t/CbeymLGQZJed6ONF7nt56Kf0oLVKz9rQb\n\t7MRNrosYdm42bup+3TsuVGUaeTfKptVnLiKUMdIrlJ+OVi9iUxodddO2pQJ5kX0DFWLC\n\tuGm/7OgQLoha/WBI26/HSQo2rj0TVSMZIk0Q7gH1F63T3fa0EaJlai1evsGnVwlXyOGg\n\tQN0kMQhmpVuU+SilaDh3ivNLBFeRjYSVfa/Ydj3Q/lHFoigKBOM5SatUxdLBFSJZvSQ5\n\tWXjg==","X-Gm-Message-State":"AOAM531CdIm10NFGIkW6Iv3CH9QJNTTw05hpDKvyDZwRZzWOP2YTKaPW\n\t+9bF3FVE3Y/Mpsnbzy4OUAntwA==","X-Google-Smtp-Source":"ABdhPJxBx8IC0kTspGQ05E+mMtnamshnI2tMLSxQji4NixUrBZ9Nu3ia4cfEu7RWj2YsMWFUUO1cEQ==","X-Received":"by 2002:a2e:a484:: with SMTP id\n\th4mr11000485lji.468.1593529424448; \n\tTue, 30 Jun 2020 08:03:44 -0700 (PDT)","Date":"Tue, 30 Jun 2020 17:03:43 +0200","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Show Liu ","Message-ID":"<20200630150343.GA2259976@oden.dyn.berto.se>","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20200624073705.14737-2-show.liu@linaro.org>","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11030,"web_url":"https://patchwork.libcamera.org/comment/11030/","msgid":"<20200630162624.GM5850@pendragon.ideasonboard.com>","date":"2020-06-30T16:26:24","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hello,\n\nOn Tue, Jun 30, 2020 at 05:03:43PM +0200, Niklas Söderlund wrote:\n> Hi Show,\n> \n> Thanks for your work.\n\nLikewise :-)\n\n> I really like this version! The structure is almost there and much \n> better then previous versions. As Kieran points out there are style \n> errors that checkstyle.py will help you point out so I will ignore them \n> in this review.\n> \n> On 2020-06-24 15:37:05 +0800, Show Liu wrote:\n> > The patch is to render the NV family YUV formats by OpenGL shader.\n> > V3: refine the fragment shader for better pixel color.\n> > \n> > Signed-off-by: Show Liu \n> > ---\n> > src/qcam/main.cpp | 2 +\n> > src/qcam/main_window.cpp | 19 ++-\n> > src/qcam/main_window.h | 3 +-\n> > src/qcam/meson.build | 2 +\n> > src/qcam/shader.h | 104 ++++++++++++\n> > src/qcam/viewfinder.cpp | 18 +-\n> > src/qcam/viewfinder.h | 23 ++-\n> > src/qcam/viewfinderGL.cpp | 335 ++++++++++++++++++++++++++++++++++++++\n> > src/qcam/viewfinderGL.h | 101 ++++++++++++\n> > 9 files changed, 593 insertions(+), 14 deletions(-)\n> > create mode 100644 src/qcam/shader.h\n> > create mode 100644 src/qcam/viewfinderGL.cpp\n> > create mode 100644 src/qcam/viewfinderGL.h\n> > \n> > diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> > index b3468cb..a3ea471 100644\n> > --- a/src/qcam/main.cpp\n> > +++ b/src/qcam/main.cpp\n> > @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc, char *argv[])\n> > \t\t\t \"help\");\n> > \tparser.addOption(OptStream, &streamKeyValue,\n> > \t\t\t \"Set configuration of a camera stream\", \"stream\", true);\n> > +\tparser.addOption(OptOpenGL, OptionNone, \"Render YUV formats frame via OpenGL shader\",\n> > +\t\t\t \"opengl\");\n\nShould we default to OpenGL when possible, and add an option to force a\nparticular backend ? Maybe -r/--render={gles,qt}\n\n> > \n> > \tOptionsParser::Options options = parser.parse(argc, argv);\n> > \tif (options.isSet(OptHelp))\n> > diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> > index 7bc1360..6edf370 100644\n> > --- a/src/qcam/main_window.cpp\n> > +++ b/src/qcam/main_window.cpp\n> > @@ -28,6 +28,9 @@\n> > #include \n> > \n> > #include \"dng_writer.h\"\n> > +#include \"main_window.h\"\n> > +#include \"viewfinder.h\"\n> > +#include \"viewfinderGL.h\"\n\nAccording to the name scheme we use, I think this should be\nviewfinder_gl.h.\n\n> > \n> > using namespace libcamera;\n> > \n> > @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm, const OptionsParser::Options &options)\n> > \tsetWindowTitle(title_);\n> > \tconnect(&titleTimer_, SIGNAL(timeout()), this, SLOT(updateTitle()));\n> > \n> > -\tviewfinder_ = new ViewFinder(this);\n> > -\tconnect(viewfinder_, &ViewFinder::renderComplete,\n> > -\t\tthis, &MainWindow::queueRequest);\n> > -\tsetCentralWidget(viewfinder_);\n> > +\tif (options_.isSet(OptOpenGL)) {\n> > +\t\tviewfinder_ = new ViewFinderGL(this);\n> > +\t\tconnect(dynamic_cast(viewfinder_), &ViewFinderGL::renderComplete,\n> > +\t\t\t\tthis, &MainWindow::queueRequest);\n> > +\t\tsetCentralWidget(dynamic_cast(viewfinder_));\n> > +\t} else {\n> > +\t\tviewfinder_ = new ViewFinder(this);\n> > +\t\tconnect(dynamic_cast(viewfinder_), &ViewFinder::renderComplete,\n> > +\t\t\t\tthis, &MainWindow::queueRequest);\n> > +\t\tsetCentralWidget(dynamic_cast(viewfinder_));\n> > +\t}\n> \n> I understand that one can't inherit from QObject twice, but this looks \n> odd. Is there someway the base class could inherit from QObject or \n> possibly QWidget and the derived classes hide their specilization \n> somehow? I'm no Qt export so maybe I'm asking for something impossible \n> or really stupid.\n\nIf we assume all subclasses of Viewfinder will be QWidget, we could do\n\n\tif (options_.isSet(OptOpenGL))\n\t\tviewfinder_ = new ViewFinderGL(this);\n\telse\n\t\tviewfinder_ = new ViewFinder(this);\n\n\tQWidget *vf>idget = dynamic_cast(viewfinder_);\n\tconnect(vfWidget, &ViewFinderHandler::renderComplete,\n\t\tthis, &MainWindow::queueRequest);\n\tsetCentralWidget(vfWidget);\n\nWith ViewFinderHandler::renderComplete() being a signal in the base\nclass.\n\n> > +\n> > \tadjustSize();\n> > \n> > \t/* Hotplug/unplug support */\n> > diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> > index 4606fe4..a852ef4 100644\n> > --- a/src/qcam/main_window.h\n> > +++ b/src/qcam/main_window.h\n> > @@ -38,6 +38,7 @@ enum {\n> > \tOptCamera = 'c',\n> > \tOptHelp = 'h',\n> > \tOptStream = 's',\n> > +\tOptOpenGL = 'o',\n> > };\n> > \n> > class CaptureRequest\n> > @@ -102,7 +103,7 @@ private:\n> > \tQAction *startStopAction_;\n> > \tQComboBox *cameraCombo_;\n> > \tQAction *saveRaw_;\n> > -\tViewFinder *viewfinder_;\n> > +\tViewFinderHandler *viewfinder_;\n\nI'd split this patch in two, with one patch that renames the existing\nViewFinder to ViewFinderQt and creates a ViewFinder base class (which\nyou call ViewFinderHandler here), and a second patch that adds\nViewFinderGL.\n\n> > \n> > \tQIcon iconPlay_;\n> > \tQIcon iconStop_;\n> > diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> > index 045db52..6a58947 100644\n> > --- a/src/qcam/meson.build\n> > +++ b/src/qcam/meson.build\n> > @@ -7,11 +7,13 @@ qcam_sources = files([\n> > 'main.cpp',\n> > 'main_window.cpp',\n> > 'viewfinder.cpp',\n> > + 'viewfinderGL.cpp'\n> > ])\n> > \n> > qcam_moc_headers = files([\n> > 'main_window.h',\n> > 'viewfinder.h',\n> > + 'viewfinderGL.h'\n> > ])\n> > \n> > qcam_resources = files([\n> > diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> > new file mode 100644\n> > index 0000000..f54c264\n> > --- /dev/null\n> > +++ b/src/qcam/shader.h\n> > @@ -0,0 +1,104 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * shader.h - YUV format converter shader source code\n> > + */\n> > +#ifndef __SHADER_H__\n> > +#define __SHADER_H__\n> \n> I think the content of this file should be moved inside viewfinderGL.cpp\n\nOr maybe to viewfinder_gl_shader.cpp if we want to keep it separate.\nHeader files should only contain the declarations in any case, not the\nactual variable contents.\n\nIdeally we should store the shaders in files of their own, not in a C\narray, and have them pulled in as Qt resources\n(https://doc.qt.io/qt-5/resources.html#using-resources-in-the-application).\nThey can be read through a QFile. It's something we can do on top of\nthis patch.\n\n> > +\n> > +/* Vertex shader for NV formats */\n> > +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n> \n> could this (and bellow) be made static const ?\n> \n> > + \"attribute vec2 textureIn;\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"gl_Position = vertexIn;\\n\"\n> > + \"textureOut = textureIn;\\n\"\n> > + \"}\\n\";\n> > +\n> > +/* Fragment shader for NV12, NV16 and NV24 */\n> > +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > + \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > + \"}\\n\";\n> > +\n> > +/* Fragment shader for NV21, NV61 and NV42 */\n> > +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> > + \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > + \"}\\n\";\n> > +\n> > +/* Fragment shader for YUV420 */\n> > +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > + \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"uniform sampler2D tex_v;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > + \"}\\n\";\n> > +\n> > +/* Fragment shader for YVU420 */\n> > +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> > + \"#endif\\n\"\n> > + \"varying vec2 textureOut;\\n\"\n> > + \"uniform sampler2D tex_y;\\n\"\n> > + \"uniform sampler2D tex_u;\\n\"\n> > + \"uniform sampler2D tex_v;\\n\"\n> > + \"void main(void)\\n\"\n> > + \"{\\n\"\n> > + \"vec3 yuv;\\n\"\n> > + \"vec3 rgb;\\n\"\n> > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > + \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > + \"rgb = convert_mat * yuv;\\n\"\n> > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > + \"}\\n\";\n> > +#endif // __SHADER_H__\n> > diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> > index dcf8a15..d3a2422 100644\n> > --- a/src/qcam/viewfinder.cpp\n> > +++ b/src/qcam/viewfinder.cpp\n> > @@ -33,22 +33,30 @@ static const QMap nativeFormats\n> > \t{ libcamera::formats::BGR888, QImage::Format_RGB888 },\n> > };\n> > \n> > -ViewFinder::ViewFinder(QWidget *parent)\n> > -\t: QWidget(parent), buffer_(nullptr)\n> > +ViewFinderHandler::ViewFinderHandler()\n> > {\n> > -\ticon_ = QIcon(\":camera-off.svg\");\n> > }\n\nYou don't need an empty constructor in the base class, you can just drop\nit.\n\n> > \n> > -ViewFinder::~ViewFinder()\n> > +ViewFinderHandler::~ViewFinderHandler()\n> > {\n> > }\n> > \n> > -const QList &ViewFinder::nativeFormats() const\n> > +const QList &ViewFinderHandler::nativeFormats() const\n> > {\n> > \tstatic const QList formats = ::nativeFormats.keys();\n> > \treturn formats;\n> > }\n\nI expect native formats to be different for ViewFinderQt and\nViewFinderGL, so I'd make this a pure virtual function with\nimplementations in the derived classes. ViewFinderGL should report all\nthe formats for which you have conversion shaders.\n\n> > \n> > +ViewFinder::ViewFinder(QWidget *parent)\n> > +\t: QWidget(parent), buffer_(nullptr)\n> > +{\n> > +\ticon_ = QIcon(\":camera-off.svg\");\n> > +}\n> \n> I agree with Kieran here I think the base class should be named \n> ViewFinder and the two derived classes ViewFinderQt and ViewFinderGL or \n> something similar.\n\nSeems we all agree then :-)\n\n> I also think you should move the base class to its own .h file (and .cpp \n> file if needed).\n\nAgreed.\n\n> > +\n> > +ViewFinder::~ViewFinder()\n> > +{\n> > +}\n> > +\n> > int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> > \t\t\t const QSize &size)\n> > {\n> > diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> > index 26a1320..741d694 100644\n> > --- a/src/qcam/viewfinder.h\n> > +++ b/src/qcam/viewfinder.h\n> > @@ -13,6 +13,8 @@\n> > #include \n> > #include \n> > #include \n> > +#include \n> > +#include \n> > #include \n> > #include \n> > \n> > @@ -28,7 +30,23 @@ struct MappedBuffer {\n> > \tsize_t size;\n> > };\n> > \n> > -class ViewFinder : public QWidget\n> > +class ViewFinderHandler\n> > +{\n> > +public:\n> > +\tViewFinderHandler();\n> > +\tvirtual ~ViewFinderHandler();\n> > +\n> > +\tconst QList &nativeFormats() const;\n> > +\n> > +\tvirtual int setFormat(const libcamera::PixelFormat &format, const QSize &size) =0;\n> > +\tvirtual void render(libcamera::FrameBuffer *buffer, MappedBuffer *map) =0;\n> > +\tvirtual void stop() =0;\n> > +\n> > +\tvirtual QImage getCurrentImage() =0;\n> > +\n> > +};\n> > +\n> > +class ViewFinder : public QWidget, public ViewFinderHandler\n> > {\n> > \tQ_OBJECT\n> > \n> > @@ -36,8 +54,6 @@ public:\n> > \tViewFinder(QWidget *parent);\n> > \t~ViewFinder();\n> > \n> > -\tconst QList &nativeFormats() const;\n> > -\n> > \tint setFormat(const libcamera::PixelFormat &format, const QSize &size);\n> > \tvoid render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > \tvoid stop();\n> > @@ -67,5 +83,4 @@ private:\n> > \tQImage image_;\n> > \tQMutex mutex_; /* Prevent concurrent access to image_ */\n> > };\n> > -\n\nThis blank line should be kept.\n\n> > #endif /* __QCAM_VIEWFINDER__ */\n> > diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> > new file mode 100644\n> > index 0000000..7b47beb\n> > --- /dev/null\n> > +++ b/src/qcam/viewfinderGL.cpp\n> > @@ -0,0 +1,335 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> > + */\n> > +\n> > +#include \"shader.h\"\n> > +#include \"viewfinderGL.h\"\n> > +\n> > +#include \n> > +\n> > +#include \n> > +\n> > +#define ATTRIB_VERTEX 0\n> > +#define ATTRIB_TEXTURE 1\n> > +\n> > +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> > +\t: QOpenGLWidget(parent),\n> > +\tglBuffer(QOpenGLBuffer::VertexBuffer),\n> > +\tpFShader(nullptr),\n> > +\tpVShader(nullptr),\n> > +\ttextureU(QOpenGLTexture::Target2D),\n> > +\ttextureV(QOpenGLTexture::Target2D),\n> > +\ttextureY(QOpenGLTexture::Target2D),\n> > +\tyuvDataPtr(nullptr)\n> > +\n> > +{\n> > +}\n> > +\n> > +ViewFinderGL::~ViewFinderGL()\n> > +{\n> > +\tremoveShader();\n> > +\n> > +\tif(textureY.isCreated())\n> > +\t\ttextureY.destroy();\n> > +\n> > +\tif(textureU.isCreated())\n> > +\t\ttextureU.destroy();\n> > +\n> > +\tif(textureV.isCreated())\n> > +\t\ttextureV.destroy();\n> > +\n> > +\tglBuffer.destroy();\n> > +}\n> > +\n> > +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> > +\t\t\t const QSize &size)\n> > +{\n> > +\tformat_ = format;\n> > +\tsize_ = size;\n> > +\n> > +\tupdateGeometry();\n> > +\treturn 0;\n> > +}\n> > +\n> > +void ViewFinderGL::stop()\n> > +{\n> > +\tif (buffer_) {\n> > +\t\trenderComplete(buffer_);\n> > +\t\tbuffer_ = nullptr;\n> > +\t}\n> > +}\n> > +\n> > +QImage ViewFinderGL::getCurrentImage()\n> > +{\n> > +\tQMutexLocker locker(&mutex_);\n> > +\n> > +\treturn(grabFramebuffer());\n> > +}\n> > +\n> > +void ViewFinderGL::render(libcamera::FrameBuffer *buffer, MappedBuffer *map)\n> > +{\n> > +\tif (buffer->planes().size() != 1) {\n> > +\t\tqWarning() << \"Multi-planar buffers are not supported\";\n> > +\t\treturn;\n> > +\t}\n> > +\n> > +\tunsigned char *memory = static_cast(map->memory);\n> > +\tif(memory) {\n\nCan memory be null here ?\n\n> > +\t\tyuvDataPtr = memory;\n> > +\t\tupdate();\n> > +\t\tbuffer_ = buffer;\n> > +\t}\n> > +\n> > +\tif (buffer)\n> > +\t\trenderComplete(buffer);\n\nHere's you're supposed to signal completion of the previous buffer, not\nthe current buffer. That's why there's a std::swap() in the existing\nViewFinder::render().\n\n> > +}\n> > +\n> > +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> > +{\n> > +\tif(buffer) {\n> > +\t\tyuvDataPtr = buffer;\n> > +\t\tupdate();\n> > +\t}\n> > +}\n> > +\n> > +void ViewFinderGL::setFrameSize(int width, int height)\n> > +{\n> > +\tif(width > 0 && height > 0) {\n> > +\t\twidth_ = width;\n> > +\t\theight_ = height;\n> > +\t}\n> > +}\n> > +\n> > +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> > +{\n> > +\tchar *fsrc = nullptr;\n> > +\n> > +\tswitch(format) {\n> > +\tcase libcamera::formats::NV12:\n> > +\t\thorzSubSample_ = 2;\n> > +\t\tvertSubSample_ = 2;\n> > +\t\tfsrc = NV_2_planes_UV;\n> > +\t\tbreak;\n> > +\tcase libcamera::formats::NV21:\n> > +\t\thorzSubSample_ = 2;\n> > +\t\tvertSubSample_ = 2;\n> > +\t\tfsrc = NV_2_planes_VU;\n> > +\t\tbreak;\n> > +\tcase libcamera::formats::NV16:\n> > +\t\thorzSubSample_ = 2;\n> > +\t\tvertSubSample_ = 1;\n> > +\t\tfsrc = NV_2_planes_UV;\n> > +\t\tbreak;\n> > +\tcase libcamera::formats::NV61:\n> > +\t\thorzSubSample_ = 2;\n> > +\t\tvertSubSample_ = 1;\n> > +\t\tfsrc = NV_2_planes_VU;\n> > +\t\tbreak;\n> > +\tcase libcamera::formats::NV24:\n> > +\t\thorzSubSample_ = 1;\n> > +\t\tvertSubSample_ = 1;\n> > +\t\tfsrc = NV_2_planes_UV;\n> > +\t\tbreak;\n> > +\tcase libcamera::formats::NV42:\n> > +\t\thorzSubSample_ = 1;\n> > +\t\tvertSubSample_ = 1;\n> > +\t\tfsrc = NV_2_planes_VU;\n> > +\t\tbreak;\n> > +\tcase libcamera::formats::YUV420:\n> > +\t\thorzSubSample_ = 2;\n> > +\t\tvertSubSample_ = 2;\n> > +\t\tfsrc = NV_3_planes_UV;\n> > +\t\tbreak;\n> > +\tdefault:\n> > +\t\tbreak;\n> > +\t};\n> > +\n> > +\tif(fsrc == nullptr) {\n> > +\t\tqDebug() << __func__ << \"[ERROR]:\" <<\" No suitable fragment shader can be select.\";\n> > +\t}\n> > +\treturn fsrc;\n> > +}\n> > +\n> > +void ViewFinderGL::createFragmentShader()\n> > +{\n> > +\tbool bCompile;\n> > +\n> > +\tpFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> > +\tchar *fsrc = selectFragmentShader(format_);\n> > +\n> > +\tbCompile = pFShader->compileSourceCode(fsrc);\n> > +\tif(!bCompile)\n> > +\t{\n> > +\t\tqDebug() << __func__ << \":\" << pFShader->log();\n> > +\t}\n> > +\n> > +\tshaderProgram.addShader(pFShader);\n> > +\n> > +\t// Link shader pipeline\n> > +\tif (!shaderProgram.link()) {\n> > +\t\tqDebug() << __func__ << \": shader program link failed.\\n\" << shaderProgram.log();\n> > +\t\tclose();\n> > +\t}\n> > +\n> > +\t// Bind shader pipeline for use\n> > +\tif (!shaderProgram.bind()) {\n> > +\t\tqDebug() << __func__ << \": shader program binding failed.\\n\" << shaderProgram.log();\n> > +\t\tclose();\n> > +\t}\n> > +\n> > +\tshaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> > +\tshaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> > +\n> > +\tshaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> > +\tshaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> > +\n> > +\ttextureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> > +\ttextureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> > +\ttextureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> > +\n> > +\tif(!textureY.isCreated())\n> > +\t\ttextureY.create();\n> > +\n> > +\tif(!textureU.isCreated())\n> > +\t\ttextureU.create();\n> > +\n> > +\tif(!textureV.isCreated())\n> > +\t\ttextureV.create();\n> > +\n> > +\tid_y = textureY.textureId();\n> > +\tid_u = textureU.textureId();\n> > +\tid_v = textureV.textureId();\n> > +}\n> > +\n> > +void ViewFinderGL::configureTexture(unsigned int id)\n> > +{\n> > +\tglBindTexture(GL_TEXTURE_2D, id);\n> > +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);\n> > +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);\n> > +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);\n> > +\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);\n> > +}\n> > +\n> > +void ViewFinderGL::removeShader()\n> > +{\n> > +\tif (shaderProgram.isLinked()) {\n> > +\t\tshaderProgram.release();\n> > +\t\tshaderProgram.removeAllShaders();\n> > +\t}\n> > +\n> > +\tif(pFShader)\n> > +\t\tdelete pFShader;\n> > +\n> > +\tif(pVShader)\n> > +\t\tdelete pVShader;\n\nIf we stop and restart the stream with a different format, the previous\nshader will be used, as removeShader() isn't called. I don't think\nthat's right. I believe you need to call removeShader() at the beginning\nof setFormat().\n\n> > +}\n> > +\n> > +void ViewFinderGL::initializeGL()\n> > +{\n> > +\tbool bCompile;\n> > +\n> > +\tinitializeOpenGLFunctions();\n> > +\tglEnable(GL_TEXTURE_2D);\n> > +\tglDisable(GL_DEPTH_TEST);\n> > +\n> > +\tstatic const GLfloat vertices[] {\n> > +\t\t-1.0f,-1.0f,\n> > +\t\t-1.0f,+1.0f,\n> > +\t\t+1.0f,+1.0f,\n> > +\t\t+1.0f,-1.0f,\n> > +\t\t0.0f,1.0f,\n> > +\t\t0.0f,0.0f,\n> > +\t\t1.0f,0.0f,\n> > +\t\t1.0f,1.0f,\n> > +\t};\n> > +\n> > +\tglBuffer.create();\n> > +\tglBuffer.bind();\n> > +\tglBuffer.allocate(vertices,sizeof(vertices));\n> > +\n> > +\t/* Create Vertex Shader */\n> > +\tpVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> > +\tchar *vsrc = NV_Vertex_shader;\n> > +\n> > +\tbCompile = pVShader->compileSourceCode(vsrc);\n> > +\tif(!bCompile) {\n> > +\t\tqDebug() << __func__<< \":\" << pVShader->log();\n> > +\t}\n> > +\n> > +\tshaderProgram.addShader(pVShader);\n> > +\n> > +\tglClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> > +}\n> > +\n> > +void ViewFinderGL::render()\n> > +{\n> > +\tswitch(format_) {\n> > +\tcase libcamera::formats::NV12:\n> > +\tcase libcamera::formats::NV21:\n> > +\tcase libcamera::formats::NV16:\n> > +\tcase libcamera::formats::NV61:\n> > +\tcase libcamera::formats::NV24:\n> > +\tcase libcamera::formats::NV42:\n> > +\t\t/* activate texture 0 */\n> > +\t\tglActiveTexture(GL_TEXTURE0);\n> > +\t\tconfigureTexture(id_y);\n> > +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > +\t\tglUniform1i(textureUniformY, 0);\n> > +\n> > +\t\t/* activate texture 1 */\n> > +\t\tglActiveTexture(GL_TEXTURE1);\n> > +\t\tconfigureTexture(id_u);\n> > +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > +\t\tglUniform1i(textureUniformU, 1);\n> > +\t\tbreak;\n> > +\tcase libcamera::formats::YUV420:\n> > +\t\t/* activate texture 0 */\n> > +\t\tglActiveTexture(GL_TEXTURE0);\n> > +\t\tconfigureTexture(id_y);\n> > +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > +\t\tglUniform1i(textureUniformY, 0);\n> > +\n> > +\t\t/* activate texture 1 */\n> > +\t\tglActiveTexture(GL_TEXTURE1);\n> > +\t\tconfigureTexture(id_u);\n> > +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > +\t\tglUniform1i(textureUniformU, 1);\n> > +\n> > +\t\t/* activate texture 2 */\n> > +\t\tglActiveTexture(GL_TEXTURE2);\n> > +\t\tconfigureTexture(id_v);\n> > +\t\tglTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> > +\t\tglUniform1i(textureUniformV, 2);\n> > +\t\tbreak;\n> > +\tdefault:\n> > +\t\tbreak;\n> > +\t};\n> > +}\n> > +\n> > +void ViewFinderGL::paintGL()\n> > +{\n> > +\tif(pFShader == nullptr)\n\nWe tend to write\n\n\tif (!pFShader)\n\n> > +\t\tcreateFragmentShader();\n\nI wonder if we could do this in setFormat().\n\n> > +\n> > +\tif(yuvDataPtr)\n> > +\t{\n> > +\t\tglClearColor(0.0, 0.0, 0.0, 1.0);\n> > +\t\tglClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> > +\n> > +\t\trender();\n> > +\t\tglDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> > +\t}\n> > +}\n> > +\n> > +void ViewFinderGL::resizeGL(int w, int h)\n> > +{\n> > +\tglViewport(0,0,w,h);\n> > +}\n> > +\n> > +QSize ViewFinderGL::sizeHint() const\n> > +{\n> > +\treturn size_.isValid() ? size_ : QSize(640, 480);\n> > +}\n> > diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> > new file mode 100644\n> > index 0000000..08662ca\n> > --- /dev/null\n> > +++ b/src/qcam/viewfinderGL.h\n> > @@ -0,0 +1,101 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Linaro\n> > + *\n> > + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> > + */\n> > +#ifndef __VIEWFINDERGL_H__\n> > +#define __VIEWFINDERGL_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n\nDo you need all these headers in the .h file ? They seem to belong to\nthe .cpp file.\n\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n\nBlank line missing here.\n\n> > +#include \"viewfinder.h\"\n> > +\n> > +class ViewFinderGL : public QOpenGLWidget,\n> > +\t\t\t\t\t public ViewFinderHandler,\n> > +\t\t\t\t\t protected QOpenGLFunctions\n> > +{\n> > +\tQ_OBJECT\n> > +\n> > +public:\n> > +\tViewFinderGL(QWidget *parent = 0);\n> > +\t~ViewFinderGL();\n> > +\n> > +\tint setFormat(const libcamera::PixelFormat &format, const QSize &size);\n\nYou should add \"override\" to qualify all overridden functions.\n\n\tint setFormat(const libcamera::PixelFormat &format,\n\t\t const QSize &size) override;\n\n> > +\tvoid render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > +\tvoid stop();\n> > +\n> > +\tQImage getCurrentImage();\n> > +\n> > +\tvoid setFrameSize(int width, int height);\n> > +\tvoid updateFrame(unsigned char *buffer);\n\nThose two functions seem unused.\n\n> > +\n> > +\tchar *selectFragmentShader(unsigned int format);\n> > +\tvoid createFragmentShader();\n\nAnd these two functions can be private.\n\n> > +\tvoid render();\n> > +\n> > +Q_SIGNALS:\n> > +\tvoid renderComplete(libcamera::FrameBuffer *buffer);\n> > +\n> > +protected:\n> > +\tvoid initializeGL() Q_DECL_OVERRIDE;\n\nYou can use \"override\" directly, we know the compiler supports it.\n\n> > +\tvoid paintGL() Q_DECL_OVERRIDE;\n> > +\tvoid resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> > +\tQSize sizeHint() const override;\n> > +\n> > +private:\n> > +\n> > +\tvoid configureTexture(unsigned int id);\n> > +\tvoid removeShader();\n> > +\n> > +\t/* Captured image size, format and buffer */\n> > +\tlibcamera::FrameBuffer *buffer_;\n> > +\tlibcamera::PixelFormat format_;\n> > +\tQOpenGLBuffer glBuffer;\n> > +\tQSize size_;\n> > +\n> > +\tGLuint id_u;\n> > +\tGLuint id_v;\n> > +\tGLuint id_y;\n> > +\n> > +\tQMutex mutex_; /* Prevent concurrent access to image_ */\n> > +\n> > +\tQOpenGLShader *pFShader;\n> > +\tQOpenGLShader *pVShader;\n> > +\tQOpenGLShaderProgram shaderProgram;\n> > +\n> > +\tGLuint textureUniformU;\n> > +\tGLuint textureUniformV;\n> > +\tGLuint textureUniformY;\n> > +\n> > +\tQOpenGLTexture textureU;\n> > +\tQOpenGLTexture textureV;\n> > +\tQOpenGLTexture textureY;\n> > +\n> > +\tunsigned int width_;\n> > +\tunsigned int height_;\n> > +\n> > +\tunsigned char* yuvDataPtr;\n> > +\n> > +\t/* NV parameters */\n> > +\tunsigned int horzSubSample_ ;\n> > +\tunsigned int vertSubSample_;\n> > +};\n> > +#endif // __VIEWFINDERGL_H__","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 823C2BF415\n\tfor ;\n\tTue, 30 Jun 2020 16:26:30 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id DF96860C58;\n\tTue, 30 Jun 2020 18:26:29 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9ABE1603B4\n\tfor ;\n\tTue, 30 Jun 2020 18:26:28 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi\n\t[81.175.216.236])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id E22DC29F;\n\tTue, 30 Jun 2020 18:26:27 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"rDGOhyB2\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1593534388;\n\tbh=GdvNkQdk5HKbbTXO0Xqu0/IZgrgA/xh3n6nCZDIdNKI=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=rDGOhyB2XSXuDen1m66KrMK+z3PTUywjaAml/RFk3TOzZmKYQIDWLGQ+bdfMSJVvJ\n\tV1rrXiT74XHKNup5sUX3nOlhskpJSQcSX4D7y4xN5qHMvl1tZ2Zybnn39rAae3BBAt\n\twRSAEZzymKF33D3OBFVu9t/ncIJtgep+rS6CGsn4=","Date":"Tue, 30 Jun 2020 19:26:24 +0300","From":"Laurent Pinchart ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Message-ID":"<20200630162624.GM5850@pendragon.ideasonboard.com>","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>\n\t<20200630150343.GA2259976@oden.dyn.berto.se>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20200630150343.GA2259976@oden.dyn.berto.se>","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11114,"web_url":"https://patchwork.libcamera.org/comment/11114/","msgid":"","date":"2020-07-03T07:46:32","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":24,"url":"https://patchwork.libcamera.org/api/people/24/","name":"Show Liu","email":"show.liu@linaro.org"},"content":"Hi Niklas, Laurent,\n\nThanks for the review.\n\n\n\nOn Wed, Jul 1, 2020 at 12:26 AM Laurent Pinchart <\nlaurent.pinchart@ideasonboard.com> wrote:\n\n> Hello,\n>\n> On Tue, Jun 30, 2020 at 05:03:43PM +0200, Niklas Söderlund wrote:\n> > Hi Show,\n> >\n> > Thanks for your work.\n>\n> Likewise :-)\n>\n> > I really like this version! The structure is almost there and much\n> > better then previous versions. As Kieran points out there are style\n> > errors that checkstyle.py will help you point out so I will ignore them\n> > in this review.\n>\nThank you.:-)\n\n> >\n> > On 2020-06-24 15:37:05 +0800, Show Liu wrote:\n> > > The patch is to render the NV family YUV formats by OpenGL shader.\n> > > V3: refine the fragment shader for better pixel color.\n> > >\n> > > Signed-off-by: Show Liu \n> > > ---\n> > > src/qcam/main.cpp | 2 +\n> > > src/qcam/main_window.cpp | 19 ++-\n> > > src/qcam/main_window.h | 3 +-\n> > > src/qcam/meson.build | 2 +\n> > > src/qcam/shader.h | 104 ++++++++++++\n> > > src/qcam/viewfinder.cpp | 18 +-\n> > > src/qcam/viewfinder.h | 23 ++-\n> > > src/qcam/viewfinderGL.cpp | 335 ++++++++++++++++++++++++++++++++++++++\n> > > src/qcam/viewfinderGL.h | 101 ++++++++++++\n> > > 9 files changed, 593 insertions(+), 14 deletions(-)\n> > > create mode 100644 src/qcam/shader.h\n> > > create mode 100644 src/qcam/viewfinderGL.cpp\n> > > create mode 100644 src/qcam/viewfinderGL.h\n> > >\n> > > diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> > > index b3468cb..a3ea471 100644\n> > > --- a/src/qcam/main.cpp\n> > > +++ b/src/qcam/main.cpp\n> > > @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc, char\n> *argv[])\n> > > \"help\");\n> > > parser.addOption(OptStream, &streamKeyValue,\n> > > \"Set configuration of a camera stream\", \"stream\",\n> true);\n> > > + parser.addOption(OptOpenGL, OptionNone, \"Render YUV formats frame\n> via OpenGL shader\",\n> > > + \"opengl\");\n>\n> Should we default to OpenGL when possible, and add an option to force a\n> particular backend ? Maybe -r/--render={gles,qt}\n>\nI'd like to mention why I set OpenGL rendering as an option.\nFirst, this depends on the \"GPU enabled\" platforms,\nand it always needs some additional steps to make sure the GPU is ready.\nSecond, as I said this patch is only for NV family YUV formats at present,\nand I believe it just covers very small parts of camera stream formats.\nThat's why I am still trying to make it support more formats as I can.:-)\n\n\n>\n> > >\n> > > OptionsParser::Options options = parser.parse(argc, argv);\n> > > if (options.isSet(OptHelp))\n> > > diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> > > index 7bc1360..6edf370 100644\n> > > --- a/src/qcam/main_window.cpp\n> > > +++ b/src/qcam/main_window.cpp\n> > > @@ -28,6 +28,9 @@\n> > > #include \n> > >\n> > > #include \"dng_writer.h\"\n> > > +#include \"main_window.h\"\n> > > +#include \"viewfinder.h\"\n> > > +#include \"viewfinderGL.h\"\n>\n> According to the name scheme we use, I think this should be\n> viewfinder_gl.h.\n\nOK. will fix it.\n\n>\n\n\n> > >\n> > > using namespace libcamera;\n> > >\n> > > @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm, const\n> OptionsParser::Options &options)\n> > > setWindowTitle(title_);\n> > > connect(&titleTimer_, SIGNAL(timeout()), this,\n> SLOT(updateTitle()));\n> > >\n> > > - viewfinder_ = new ViewFinder(this);\n> > > - connect(viewfinder_, &ViewFinder::renderComplete,\n> > > - this, &MainWindow::queueRequest);\n> > > - setCentralWidget(viewfinder_);\n> > > + if (options_.isSet(OptOpenGL)) {\n> > > + viewfinder_ = new ViewFinderGL(this);\n> > > + connect(dynamic_cast(viewfinder_),\n> &ViewFinderGL::renderComplete,\n> > > + this, &MainWindow::queueRequest);\n> > > + setCentralWidget(dynamic_cast *>(viewfinder_));\n> > > + } else {\n> > > + viewfinder_ = new ViewFinder(this);\n> > > + connect(dynamic_cast(viewfinder_),\n> &ViewFinder::renderComplete,\n> > > + this, &MainWindow::queueRequest);\n> > > + setCentralWidget(dynamic_cast(viewfinder_));\n> > > + }\n> >\n> > I understand that one can't inherit from QObject twice, but this looks\n> > odd. Is there someway the base class could inherit from QObject or\n> > possibly QWidget and the derived classes hide their specilization\n> > somehow? I'm no Qt export so maybe I'm asking for something impossible\n> > or really stupid.\n>\nNo, I really appreciate your opinions, it'll help me to make this better. :)\n\n>\n> If we assume all subclasses of Viewfinder will be QWidget, we could do\n>\n> if (options_.isSet(OptOpenGL))\n> viewfinder_ = new ViewFinderGL(this);\n> else\n> viewfinder_ = new ViewFinder(this);\n>\n> QWidget *vf>idget = dynamic_cast(viewfinder_);\n> connect(vfWidget, &ViewFinderHandler::renderComplete,\n> this, &MainWindow::queueRequest);\n> setCentralWidget(vfWidget);\n>\n> With ViewFinderHandler::renderComplete() being a signal in the base\n> class.\n>\nI will try this way in the next version.\n\n>\n> > > +\n> > > adjustSize();\n> > >\n> > > /* Hotplug/unplug support */\n> > > diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> > > index 4606fe4..a852ef4 100644\n> > > --- a/src/qcam/main_window.h\n> > > +++ b/src/qcam/main_window.h\n> > > @@ -38,6 +38,7 @@ enum {\n> > > OptCamera = 'c',\n> > > OptHelp = 'h',\n> > > OptStream = 's',\n> > > + OptOpenGL = 'o',\n> > > };\n> > >\n> > > class CaptureRequest\n> > > @@ -102,7 +103,7 @@ private:\n> > > QAction *startStopAction_;\n> > > QComboBox *cameraCombo_;\n> > > QAction *saveRaw_;\n> > > - ViewFinder *viewfinder_;\n> > > + ViewFinderHandler *viewfinder_;\n>\n> I'd split this patch in two, with one patch that renames the existing\n> ViewFinder to ViewFinderQt and creates a ViewFinder base class (which\n> you call ViewFinderHandler here), and a second patch that adds\n> ViewFinderGL.\n\nOK. will do.\n\n>\n\n\n> > >\n> > > QIcon iconPlay_;\n> > > QIcon iconStop_;\n> > > diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> > > index 045db52..6a58947 100644\n> > > --- a/src/qcam/meson.build\n> > > +++ b/src/qcam/meson.build\n> > > @@ -7,11 +7,13 @@ qcam_sources = files([\n> > > 'main.cpp',\n> > > 'main_window.cpp',\n> > > 'viewfinder.cpp',\n> > > + 'viewfinderGL.cpp'\n> > > ])\n> > >\n> > > qcam_moc_headers = files([\n> > > 'main_window.h',\n> > > 'viewfinder.h',\n> > > + 'viewfinderGL.h'\n> > > ])\n> > >\n> > > qcam_resources = files([\n> > > diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> > > new file mode 100644\n> > > index 0000000..f54c264\n> > > --- /dev/null\n> > > +++ b/src/qcam/shader.h\n> > > @@ -0,0 +1,104 @@\n> > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Linaro\n> > > + *\n> > > + * shader.h - YUV format converter shader source code\n> > > + */\n> > > +#ifndef __SHADER_H__\n> > > +#define __SHADER_H__\n> >\n> > I think the content of this file should be moved inside viewfinderGL.cpp\n>\n> Or maybe to viewfinder_gl_shader.cpp if we want to keep it separate.\n> Header files should only contain the declarations in any case, not the\n> actual variable contents.\n>\n> Ideally we should store the shaders in files of their own, not in a C\n> array, and have them pulled in as Qt resources\n> (https://doc.qt.io/qt-5/resources.html#using-resources-in-the-application\n> ).\n> They can be read through a QFile. It's something we can do on top of\n> this patch.\n>\nActually, QOpenGLShader/QOpenGLShaderProgram are provide some methods\nto build/compile the shader source code from file directly but if I load it\nfrom file which means\nadditional shader installation steps are required and also there are some\npotential risks\nif the shader is not in the right position. That's why I push them into one\nheader file.\nForgive my lazy ...:-P\nAnyway refine this part in the next version.\n\n\n> > > +\n> > > +/* Vertex shader for NV formats */\n> > > +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n> >\n> > could this (and bellow) be made static const ?\n> >\n> > > + \"attribute vec2 textureIn;\\n\"\n> > > + \"varying vec2 textureOut;\\n\"\n> > > + \"void main(void)\\n\"\n> > > + \"{\\n\"\n> > > + \"gl_Position = vertexIn;\\n\"\n> > > + \"textureOut = textureIn;\\n\"\n> > > + \"}\\n\";\n> > > +\n> > > +/* Fragment shader for NV12, NV16 and NV24 */\n> > > +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > > + \"precision highp float;\\n\"\n> > > + \"#endif\\n\"\n> > > + \"varying vec2 textureOut;\\n\"\n> > > + \"uniform sampler2D tex_y;\\n\"\n> > > + \"uniform sampler2D tex_u;\\n\"\n> > > + \"void main(void)\\n\"\n> > > + \"{\\n\"\n> > > + \"vec3 yuv;\\n\"\n> > > + \"vec3 rgb;\\n\"\n> > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > + \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > > + \"rgb = convert_mat * yuv;\\n\"\n> > > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > > + \"}\\n\";\n> > > +\n> > > +/* Fragment shader for NV21, NV61 and NV42 */\n> > > +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> > > + \"precision highp float;\\n\"\n> > > + \"#endif\\n\"\n> > > + \"varying vec2 textureOut;\\n\"\n> > > + \"uniform sampler2D tex_y;\\n\"\n> > > + \"uniform sampler2D tex_u;\\n\"\n> > > + \"void main(void)\\n\"\n> > > + \"{\\n\"\n> > > + \"vec3 yuv;\\n\"\n> > > + \"vec3 rgb;\\n\"\n> > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > + \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > + \"rgb = convert_mat * yuv;\\n\"\n> > > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > > + \"}\\n\";\n> > > +\n> > > +/* Fragment shader for YUV420 */\n> > > +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > > + \"precision highp float;\\n\"\n> > > + \"#endif\\n\"\n> > > + \"varying vec2 textureOut;\\n\"\n> > > + \"uniform sampler2D tex_y;\\n\"\n> > > + \"uniform sampler2D tex_u;\\n\"\n> > > + \"uniform sampler2D tex_v;\\n\"\n> > > + \"void main(void)\\n\"\n> > > + \"{\\n\"\n> > > + \"vec3 yuv;\\n\"\n> > > + \"vec3 rgb;\\n\"\n> > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > + \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > > + \"rgb = convert_mat * yuv;\\n\"\n> > > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > > + \"}\\n\";\n> > > +\n> > > +/* Fragment shader for YVU420 */\n> > > +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> > > + \"#endif\\n\"\n> > > + \"varying vec2 textureOut;\\n\"\n> > > + \"uniform sampler2D tex_y;\\n\"\n> > > + \"uniform sampler2D tex_u;\\n\"\n> > > + \"uniform sampler2D tex_v;\\n\"\n> > > + \"void main(void)\\n\"\n> > > + \"{\\n\"\n> > > + \"vec3 yuv;\\n\"\n> > > + \"vec3 rgb;\\n\"\n> > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625,\n> 1.1640625),\\n\"\n> > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > + \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > + \"rgb = convert_mat * yuv;\\n\"\n> > > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > > + \"}\\n\";\n> > > +#endif // __SHADER_H__\n> > > diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> > > index dcf8a15..d3a2422 100644\n> > > --- a/src/qcam/viewfinder.cpp\n> > > +++ b/src/qcam/viewfinder.cpp\n> > > @@ -33,22 +33,30 @@ static const QMap QImage::Format> nativeFormats\n> > > { libcamera::formats::BGR888, QImage::Format_RGB888 },\n> > > };\n> > >\n> > > -ViewFinder::ViewFinder(QWidget *parent)\n> > > - : QWidget(parent), buffer_(nullptr)\n> > > +ViewFinderHandler::ViewFinderHandler()\n> > > {\n> > > - icon_ = QIcon(\":camera-off.svg\");\n> > > }\n>\n> You don't need an empty constructor in the base class, you can just drop\n> it.\n>\n> > >\n> > > -ViewFinder::~ViewFinder()\n> > > +ViewFinderHandler::~ViewFinderHandler()\n> > > {\n> > > }\n> > >\n> > > -const QList &ViewFinder::nativeFormats() const\n> > > +const QList\n> &ViewFinderHandler::nativeFormats() const\n> > > {\n> > > static const QList formats =\n> ::nativeFormats.keys();\n> > > return formats;\n> > > }\n>\n> I expect native formats to be different for ViewFinderQt and\n> ViewFinderGL, so I'd make this a pure virtual function with\n> implementations in the derived classes. ViewFinderGL should report all\n> the formats for which you have conversion shaders.\n>\n ok, I will rewrite this.\n\n>\n> > >\n> > > +ViewFinder::ViewFinder(QWidget *parent)\n> > > + : QWidget(parent), buffer_(nullptr)\n> > > +{\n> > > + icon_ = QIcon(\":camera-off.svg\");\n> > > +}\n> >\n> > I agree with Kieran here I think the base class should be named\n> > ViewFinder and the two derived classes ViewFinderQt and ViewFinderGL or\n> > something similar.\n>\n> Seems we all agree then :-)\n>\n> > I also think you should move the base class to its own .h file (and .cpp\n> > file if needed).\n>\n> Agreed.\n>\nI will modify it accordingly.\n\n>\n> > > +\n> > > +ViewFinder::~ViewFinder()\n> > > +{\n> > > +}\n> > > +\n> > > int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> > > const QSize &size)\n> > > {\n> > > diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> > > index 26a1320..741d694 100644\n> > > --- a/src/qcam/viewfinder.h\n> > > +++ b/src/qcam/viewfinder.h\n> > > @@ -13,6 +13,8 @@\n> > > #include \n> > > #include \n> > > #include \n> > > +#include \n> > > +#include \n> > > #include \n> > > #include \n> > >\n> > > @@ -28,7 +30,23 @@ struct MappedBuffer {\n> > > size_t size;\n> > > };\n> > >\n> > > -class ViewFinder : public QWidget\n> > > +class ViewFinderHandler\n> > > +{\n> > > +public:\n> > > + ViewFinderHandler();\n> > > + virtual ~ViewFinderHandler();\n> > > +\n> > > + const QList &nativeFormats() const;\n> > > +\n> > > + virtual int setFormat(const libcamera::PixelFormat &format, const\n> QSize &size) =0;\n> > > + virtual void render(libcamera::FrameBuffer *buffer, MappedBuffer\n> *map) =0;\n> > > + virtual void stop() =0;\n> > > +\n> > > + virtual QImage getCurrentImage() =0;\n> > > +\n> > > +};\n> > > +\n> > > +class ViewFinder : public QWidget, public ViewFinderHandler\n> > > {\n> > > Q_OBJECT\n> > >\n> > > @@ -36,8 +54,6 @@ public:\n> > > ViewFinder(QWidget *parent);\n> > > ~ViewFinder();\n> > >\n> > > - const QList &nativeFormats() const;\n> > > -\n> > > int setFormat(const libcamera::PixelFormat &format, const QSize\n> &size);\n> > > void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > > void stop();\n> > > @@ -67,5 +83,4 @@ private:\n> > > QImage image_;\n> > > QMutex mutex_; /* Prevent concurrent access to image_ */\n> > > };\n> > > -\n>\n> This blank line should be kept.\n>\nsure, will fix it.\n\n>\n> > > #endif /* __QCAM_VIEWFINDER__ */\n> > > diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> > > new file mode 100644\n> > > index 0000000..7b47beb\n> > > --- /dev/null\n> > > +++ b/src/qcam/viewfinderGL.cpp\n> > > @@ -0,0 +1,335 @@\n> > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Linaro\n> > > + *\n> > > + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> > > + */\n> > > +\n> > > +#include \"shader.h\"\n> > > +#include \"viewfinderGL.h\"\n> > > +\n> > > +#include \n> > > +\n> > > +#include \n> > > +\n> > > +#define ATTRIB_VERTEX 0\n> > > +#define ATTRIB_TEXTURE 1\n> > > +\n> > > +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> > > + : QOpenGLWidget(parent),\n> > > + glBuffer(QOpenGLBuffer::VertexBuffer),\n> > > + pFShader(nullptr),\n> > > + pVShader(nullptr),\n> > > + textureU(QOpenGLTexture::Target2D),\n> > > + textureV(QOpenGLTexture::Target2D),\n> > > + textureY(QOpenGLTexture::Target2D),\n> > > + yuvDataPtr(nullptr)\n> > > +\n> > > +{\n> > > +}\n> > > +\n> > > +ViewFinderGL::~ViewFinderGL()\n> > > +{\n> > > + removeShader();\n> > > +\n> > > + if(textureY.isCreated())\n> > > + textureY.destroy();\n> > > +\n> > > + if(textureU.isCreated())\n> > > + textureU.destroy();\n> > > +\n> > > + if(textureV.isCreated())\n> > > + textureV.destroy();\n> > > +\n> > > + glBuffer.destroy();\n> > > +}\n> > > +\n> > > +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> > > + const QSize &size)\n> > > +{\n> > > + format_ = format;\n> > > + size_ = size;\n> > > +\n> > > + updateGeometry();\n> > > + return 0;\n> > > +}\n> > > +\n> > > +void ViewFinderGL::stop()\n> > > +{\n> > > + if (buffer_) {\n> > > + renderComplete(buffer_);\n> > > + buffer_ = nullptr;\n> > > + }\n> > > +}\n> > > +\n> > > +QImage ViewFinderGL::getCurrentImage()\n> > > +{\n> > > + QMutexLocker locker(&mutex_);\n> > > +\n> > > + return(grabFramebuffer());\n> > > +}\n> > > +\n> > > +void ViewFinderGL::render(libcamera::FrameBuffer *buffer,\n> MappedBuffer *map)\n> > > +{\n> > > + if (buffer->planes().size() != 1) {\n> > > + qWarning() << \"Multi-planar buffers are not supported\";\n> > > + return;\n> > > + }\n> > > +\n> > > + unsigned char *memory = static_cast(map->memory);\n> > > + if(memory) {\n>\n> Can memory be null here ?\n>\nI will rewrite this.\n\n>\n> > > + yuvDataPtr = memory;\n> > > + update();\n> > > + buffer_ = buffer;\n> > > + }\n> > > +\n> > > + if (buffer)\n> > > + renderComplete(buffer);\n>\n> Here's you're supposed to signal completion of the previous buffer, not\n> the current buffer. That's why there's a std::swap() in the existing\n> ViewFinder::render().\n\nIs That means I should\nemit renderComplete(buffer_); after update();\nthen\nbuffer_ = buffer;\n...\nI don't quite understand what you mean, would you please explain it in\ndetail?\nForgive my stupid..:-P\n\n\n>\n\n\n> > > +}\n> > > +\n> > > +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> > > +{\n> > > + if(buffer) {\n> > > + yuvDataPtr = buffer;\n> > > + update();\n> > > + }\n> > > +}\n> > > +\n> > > +void ViewFinderGL::setFrameSize(int width, int height)\n> > > +{\n> > > + if(width > 0 && height > 0) {\n> > > + width_ = width;\n> > > + height_ = height;\n> > > + }\n> > > +}\n> > > +\n> > > +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> > > +{\n> > > + char *fsrc = nullptr;\n> > > +\n> > > + switch(format) {\n> > > + case libcamera::formats::NV12:\n> > > + horzSubSample_ = 2;\n> > > + vertSubSample_ = 2;\n> > > + fsrc = NV_2_planes_UV;\n> > > + break;\n> > > + case libcamera::formats::NV21:\n> > > + horzSubSample_ = 2;\n> > > + vertSubSample_ = 2;\n> > > + fsrc = NV_2_planes_VU;\n> > > + break;\n> > > + case libcamera::formats::NV16:\n> > > + horzSubSample_ = 2;\n> > > + vertSubSample_ = 1;\n> > > + fsrc = NV_2_planes_UV;\n> > > + break;\n> > > + case libcamera::formats::NV61:\n> > > + horzSubSample_ = 2;\n> > > + vertSubSample_ = 1;\n> > > + fsrc = NV_2_planes_VU;\n> > > + break;\n> > > + case libcamera::formats::NV24:\n> > > + horzSubSample_ = 1;\n> > > + vertSubSample_ = 1;\n> > > + fsrc = NV_2_planes_UV;\n> > > + break;\n> > > + case libcamera::formats::NV42:\n> > > + horzSubSample_ = 1;\n> > > + vertSubSample_ = 1;\n> > > + fsrc = NV_2_planes_VU;\n> > > + break;\n> > > + case libcamera::formats::YUV420:\n> > > + horzSubSample_ = 2;\n> > > + vertSubSample_ = 2;\n> > > + fsrc = NV_3_planes_UV;\n> > > + break;\n> > > + default:\n> > > + break;\n> > > + };\n> > > +\n> > > + if(fsrc == nullptr) {\n> > > + qDebug() << __func__ << \"[ERROR]:\" <<\" No suitable\n> fragment shader can be select.\";\n> > > + }\n> > > + return fsrc;\n> > > +}\n> > > +\n> > > +void ViewFinderGL::createFragmentShader()\n> > > +{\n> > > + bool bCompile;\n> > > +\n> > > + pFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> > > + char *fsrc = selectFragmentShader(format_);\n> > > +\n> > > + bCompile = pFShader->compileSourceCode(fsrc);\n> > > + if(!bCompile)\n> > > + {\n> > > + qDebug() << __func__ << \":\" << pFShader->log();\n> > > + }\n> > > +\n> > > + shaderProgram.addShader(pFShader);\n> > > +\n> > > + // Link shader pipeline\n> > > + if (!shaderProgram.link()) {\n> > > + qDebug() << __func__ << \": shader program link failed.\\n\"\n> << shaderProgram.log();\n> > > + close();\n> > > + }\n> > > +\n> > > + // Bind shader pipeline for use\n> > > + if (!shaderProgram.bind()) {\n> > > + qDebug() << __func__ << \": shader program binding\n> failed.\\n\" << shaderProgram.log();\n> > > + close();\n> > > + }\n> > > +\n> > > + shaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> > > + shaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> > > +\n> > > +\n> shaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> > > +\n> shaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> > > +\n> > > + textureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> > > + textureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> > > + textureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> > > +\n> > > + if(!textureY.isCreated())\n> > > + textureY.create();\n> > > +\n> > > + if(!textureU.isCreated())\n> > > + textureU.create();\n> > > +\n> > > + if(!textureV.isCreated())\n> > > + textureV.create();\n> > > +\n> > > + id_y = textureY.textureId();\n> > > + id_u = textureU.textureId();\n> > > + id_v = textureV.textureId();\n> > > +}\n> > > +\n> > > +void ViewFinderGL::configureTexture(unsigned int id)\n> > > +{\n> > > + glBindTexture(GL_TEXTURE_2D, id);\n> > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);\n> > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);\n> > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S,\n> GL_CLAMP_TO_EDGE);\n> > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T,\n> GL_CLAMP_TO_EDGE);\n> > > +}\n> > > +\n> > > +void ViewFinderGL::removeShader()\n> > > +{\n> > > + if (shaderProgram.isLinked()) {\n> > > + shaderProgram.release();\n> > > + shaderProgram.removeAllShaders();\n> > > + }\n> > > +\n> > > + if(pFShader)\n> > > + delete pFShader;\n> > > +\n> > > + if(pVShader)\n> > > + delete pVShader;\n>\n> If we stop and restart the stream with a different format, the previous\n> shader will be used, as removeShader() isn't called. I don't think\n> that's right. I believe you need to call removeShader() at the beginning\n> of setFormat().\n>\nYes. you're right. I will refine the setFormat() function.\n\n>\n> > > +}\n> > > +\n> > > +void ViewFinderGL::initializeGL()\n> > > +{\n> > > + bool bCompile;\n> > > +\n> > > + initializeOpenGLFunctions();\n> > > + glEnable(GL_TEXTURE_2D);\n> > > + glDisable(GL_DEPTH_TEST);\n> > > +\n> > > + static const GLfloat vertices[] {\n> > > + -1.0f,-1.0f,\n> > > + -1.0f,+1.0f,\n> > > + +1.0f,+1.0f,\n> > > + +1.0f,-1.0f,\n> > > + 0.0f,1.0f,\n> > > + 0.0f,0.0f,\n> > > + 1.0f,0.0f,\n> > > + 1.0f,1.0f,\n> > > + };\n> > > +\n> > > + glBuffer.create();\n> > > + glBuffer.bind();\n> > > + glBuffer.allocate(vertices,sizeof(vertices));\n> > > +\n> > > + /* Create Vertex Shader */\n> > > + pVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> > > + char *vsrc = NV_Vertex_shader;\n> > > +\n> > > + bCompile = pVShader->compileSourceCode(vsrc);\n> > > + if(!bCompile) {\n> > > + qDebug() << __func__<< \":\" << pVShader->log();\n> > > + }\n> > > +\n> > > + shaderProgram.addShader(pVShader);\n> > > +\n> > > + glClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> > > +}\n> > > +\n> > > +void ViewFinderGL::render()\n> > > +{\n> > > + switch(format_) {\n> > > + case libcamera::formats::NV12:\n> > > + case libcamera::formats::NV21:\n> > > + case libcamera::formats::NV16:\n> > > + case libcamera::formats::NV61:\n> > > + case libcamera::formats::NV24:\n> > > + case libcamera::formats::NV42:\n> > > + /* activate texture 0 */\n> > > + glActiveTexture(GL_TEXTURE0);\n> > > + configureTexture(id_y);\n> > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(),\n> size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > > + glUniform1i(textureUniformY, 0);\n> > > +\n> > > + /* activate texture 1 */\n> > > + glActiveTexture(GL_TEXTURE1);\n> > > + configureTexture(id_u);\n> > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > > + glUniform1i(textureUniformU, 1);\n> > > + break;\n> > > + case libcamera::formats::YUV420:\n> > > + /* activate texture 0 */\n> > > + glActiveTexture(GL_TEXTURE0);\n> > > + configureTexture(id_y);\n> > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(),\n> size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > > + glUniform1i(textureUniformY, 0);\n> > > +\n> > > + /* activate texture 1 */\n> > > + glActiveTexture(GL_TEXTURE1);\n> > > + configureTexture(id_u);\n> > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > > + glUniform1i(textureUniformU, 1);\n> > > +\n> > > + /* activate texture 2 */\n> > > + glActiveTexture(GL_TEXTURE2);\n> > > + configureTexture(id_v);\n> > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> > > + glUniform1i(textureUniformV, 2);\n> > > + break;\n> > > + default:\n> > > + break;\n> > > + };\n> > > +}\n> > > +\n> > > +void ViewFinderGL::paintGL()\n> > > +{\n> > > + if(pFShader == nullptr)\n>\n> We tend to write\n\n\n> if (!pFShader)\n>\nok will fix it.\n\n>\n> > > + createFragmentShader();\n>\n> I wonder if we could do this in setFormat().\n>\nThat should be ok, I will try to move this when setting Format.\n\n>\n> > > +\n> > > + if(yuvDataPtr)\n> > > + {\n> > > + glClearColor(0.0, 0.0, 0.0, 1.0);\n> > > + glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> > > +\n> > > + render();\n> > > + glDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> > > + }\n> > > +}\n> > > +\n> > > +void ViewFinderGL::resizeGL(int w, int h)\n> > > +{\n> > > + glViewport(0,0,w,h);\n> > > +}\n> > > +\n> > > +QSize ViewFinderGL::sizeHint() const\n> > > +{\n> > > + return size_.isValid() ? size_ : QSize(640, 480);\n> > > +}\n> > > diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> > > new file mode 100644\n> > > index 0000000..08662ca\n> > > --- /dev/null\n> > > +++ b/src/qcam/viewfinderGL.h\n> > > @@ -0,0 +1,101 @@\n> > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Linaro\n> > > + *\n> > > + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> > > + */\n> > > +#ifndef __VIEWFINDERGL_H__\n> > > +#define __VIEWFINDERGL_H__\n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n>\n> Do you need all these headers in the .h file ? They seem to belong to\n> the .cpp file.\n>\nOK I will check or just remove it.\n\n>\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +#include \n>\n> Blank line missing here.\n>\nsure, will do .\n\n>\n> > > +#include \"viewfinder.h\"\n> > > +\n> > > +class ViewFinderGL : public QOpenGLWidget,\n> > > + public ViewFinderHandler,\n> > > + protected QOpenGLFunctions\n> > > +{\n> > > + Q_OBJECT\n> > > +\n> > > +public:\n> > > + ViewFinderGL(QWidget *parent = 0);\n> > > + ~ViewFinderGL();\n> > > +\n> > > + int setFormat(const libcamera::PixelFormat &format, const QSize\n> &size);\n>\n> You should add \"override\" to qualify all overridden functions.\n>\n> int setFormat(const libcamera::PixelFormat &format,\n> const QSize &size) override;\n>\nok will do.\n\n>\n> > > + void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > > + void stop();\n> > > +\n> > > + QImage getCurrentImage();\n> > > +\n> > > + void setFrameSize(int width, int height);\n> > > + void updateFrame(unsigned char *buffer);\n>\n> Those two functions seem unused.\n\nsure, I will remove it.\n\n>\n\n\n> > > +\n> > > + char *selectFragmentShader(unsigned int format);\n> > > + void createFragmentShader();\n>\n> And these two functions can be private.\n>\nwill move to private.\n\n>\n> > > + void render();\n> > > +\n> > > +Q_SIGNALS:\n> > > + void renderComplete(libcamera::FrameBuffer *buffer);\n> > > +\n> > > +protected:\n> > > + void initializeGL() Q_DECL_OVERRIDE;\n>\n> You can use \"override\" directly, we know the compiler supports it.\n>\nok, I will fix it up.\n\nBest Regards,\nShow Liu\n\n>\n> > > + void paintGL() Q_DECL_OVERRIDE;\n> > > + void resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> > > + QSize sizeHint() const override;\n> > > +\n> > > +private:\n> > > +\n> > > + void configureTexture(unsigned int id);\n> > > + void removeShader();\n> > > +\n> > > + /* Captured image size, format and buffer */\n> > > + libcamera::FrameBuffer *buffer_;\n> > > + libcamera::PixelFormat format_;\n> > > + QOpenGLBuffer glBuffer;\n> > > + QSize size_;\n> > > +\n> > > + GLuint id_u;\n> > > + GLuint id_v;\n> > > + GLuint id_y;\n> > > +\n> > > + QMutex mutex_; /* Prevent concurrent access to image_ */\n> > > +\n> > > + QOpenGLShader *pFShader;\n> > > + QOpenGLShader *pVShader;\n> > > + QOpenGLShaderProgram shaderProgram;\n> > > +\n> > > + GLuint textureUniformU;\n> > > + GLuint textureUniformV;\n> > > + GLuint textureUniformY;\n> > > +\n> > > + QOpenGLTexture textureU;\n> > > + QOpenGLTexture textureV;\n> > > + QOpenGLTexture textureY;\n> > > +\n> > > + unsigned int width_;\n> > > + unsigned int height_;\n> > > +\n> > > + unsigned char* yuvDataPtr;\n> > > +\n> > > + /* NV parameters */\n> > > + unsigned int horzSubSample_ ;\n> > > + unsigned int vertSubSample_;\n> > > +};\n> > > +#endif // __VIEWFINDERGL_H__\n>\n> --\n> Regards,\n>\n> Laurent Pinchart\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7ACC7BE905\n\tfor ;\n\tFri, 3 Jul 2020 07:46:48 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id F004C60C56;\n\tFri, 3 Jul 2020 09:46:47 +0200 (CEST)","from mail-pg1-x529.google.com (mail-pg1-x529.google.com\n\t[IPv6:2607:f8b0:4864:20::529])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3118F603AD\n\tfor ;\n\tFri, 3 Jul 2020 09:46:46 +0200 (CEST)","by mail-pg1-x529.google.com with SMTP id w2so13931508pgg.10\n\tfor ;\n\tFri, 03 Jul 2020 00:46:46 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"G2/q4243\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=PB+vjZlIPiROPvUlcfTbZi8LG8VZssbytztwfWWUcRA=;\n\tb=G2/q4243Z6jkTOis7c0Rg5RUFs2VrMRHLh9ytZIDk7h4bMfE3pDVqfpD063tnIgMy8\n\t3CcMhDHyJ34O2LQTE2YsoVDlf9LvDZdAAwGlz4356ND4440WPzwZRKxH7F3q5FPnnXTU\n\taeMtME/vN7lnl4rmz02k4o1rkSlKQtdHwc+2IajDn8dfRwnbE44e/DSfm17rrOnkUjpv\n\tjEFICKZU8GAKzEb79VtCUYohMyVF7/c/8ytH3u+hZf1Oc2pienDKZ5DU7zilYl4pNI0L\n\tE/2rh0buOKomSjDNrMAkCIScvg3fIYyk0LOEQqqsCfj9H4VvuG5diX7JSXTkjry62K4w\n\tdyhg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=PB+vjZlIPiROPvUlcfTbZi8LG8VZssbytztwfWWUcRA=;\n\tb=cL1FvDfz9lA7ehXtAQEq6r5wgXB0aoJYXB5BrvtuUKMj4y61EnkEJ3KnrNkrd7Nhkd\n\tefbNeuT/EKUnkA0BONRaP/gfEwiZNA5Ce0JcJUOTBrCdkDpPmG7o+y06lMqORas1Jrci\n\tbDddbylcdl3sfQX9UHQm/dm4Lx6IvvBZa+CtEd15WyD+yCgtTpBLCnI5X5RMIoNXzYUh\n\tPqMxSAcWMdIX1YJXuRaE09+1Odk87uskyBqHIVADVVZkstjMNNe8xLi4Wd/8gxIg2BeG\n\tRkzPfjp5Ds0iMSKH4WuxJ4XR+OBv7V7Un3CUMmNWXYtjiz3WKafwexBvpS3mP2B4qqo8\n\tkEMA==","X-Gm-Message-State":"AOAM531VhHHRmHPEAi0VMOxfqY+2gYLIN06K6vFgB5hwCvTwuYIW27KJ\n\t4KeEup+qn6X5R+ClEtcmOT7M2v43K8NtwM25nYyLmDx6z04=","X-Google-Smtp-Source":"ABdhPJzqIpixZ8S3N8oJTuYvVTAEyUxUoW5OQYUDnFWEg9ARq1jP8fd4zKZetw6OPk3eeAvtzHMCvOaLbHIU9qAC5yU=","X-Received":"by 2002:a05:6a00:1507:: with SMTP id\n\tq7mr30391663pfu.131.1593762403964; \n\tFri, 03 Jul 2020 00:46:43 -0700 (PDT)","MIME-Version":"1.0","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>\n\t<20200630150343.GA2259976@oden.dyn.berto.se>\n\t<20200630162624.GM5850@pendragon.ideasonboard.com>","In-Reply-To":"<20200630162624.GM5850@pendragon.ideasonboard.com>","From":"Show Liu ","Date":"Fri, 3 Jul 2020 15:46:32 +0800","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============6510288707954390809==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11136,"web_url":"https://patchwork.libcamera.org/comment/11136/","msgid":"<20200703101405.GA5963@pendragon.ideasonboard.com>","date":"2020-07-03T10:14:05","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Show,\n\nOn Fri, Jul 03, 2020 at 03:46:32PM +0800, Show Liu wrote:\n> On Wed, Jul 1, 2020 at 12:26 AM Laurent Pinchart wrote:\n> > On Tue, Jun 30, 2020 at 05:03:43PM +0200, Niklas Söderlund wrote:\n> > > Hi Show,\n> > >\n> > > Thanks for your work.\n> >\n> > Likewise :-)\n> >\n> > > I really like this version! The structure is almost there and much\n> > > better then previous versions. As Kieran points out there are style\n> > > errors that checkstyle.py will help you point out so I will ignore them\n> > > in this review.\n>\n> Thank you.:-)\n> \n> > > On 2020-06-24 15:37:05 +0800, Show Liu wrote:\n> > > > The patch is to render the NV family YUV formats by OpenGL shader.\n> > > > V3: refine the fragment shader for better pixel color.\n> > > >\n> > > > Signed-off-by: Show Liu \n> > > > ---\n> > > > src/qcam/main.cpp | 2 +\n> > > > src/qcam/main_window.cpp | 19 ++-\n> > > > src/qcam/main_window.h | 3 +-\n> > > > src/qcam/meson.build | 2 +\n> > > > src/qcam/shader.h | 104 ++++++++++++\n> > > > src/qcam/viewfinder.cpp | 18 +-\n> > > > src/qcam/viewfinder.h | 23 ++-\n> > > > src/qcam/viewfinderGL.cpp | 335 ++++++++++++++++++++++++++++++++++++++\n> > > > src/qcam/viewfinderGL.h | 101 ++++++++++++\n> > > > 9 files changed, 593 insertions(+), 14 deletions(-)\n> > > > create mode 100644 src/qcam/shader.h\n> > > > create mode 100644 src/qcam/viewfinderGL.cpp\n> > > > create mode 100644 src/qcam/viewfinderGL.h\n> > > >\n> > > > diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> > > > index b3468cb..a3ea471 100644\n> > > > --- a/src/qcam/main.cpp\n> > > > +++ b/src/qcam/main.cpp\n> > > > @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc, char *argv[])\n> > > > \"help\");\n> > > > parser.addOption(OptStream, &streamKeyValue,\n> > > > \"Set configuration of a camera stream\", \"stream\", true);\n> > > > + parser.addOption(OptOpenGL, OptionNone, \"Render YUV formats frame via OpenGL shader\",\n> > > > + \"opengl\");\n> >\n> > Should we default to OpenGL when possible, and add an option to force a\n> > particular backend ? Maybe -r/--render={gles,qt}\n>\n> I'd like to mention why I set OpenGL rendering as an option.\n> First, this depends on the \"GPU enabled\" platforms,\n> and it always needs some additional steps to make sure the GPU is ready.\n> Second, as I said this patch is only for NV family YUV formats at present,\n> and I believe it just covers very small parts of camera stream formats.\n> That's why I am still trying to make it support more formats as I can.:-)\n\nThat's a good point, defaulting to the Qt renderer for now may be best.\nI'd still prefer a -r/--render option if possible though, to make it\neasier to change the default and to add more renderers later if needed.\n\n> > > >\n> > > > OptionsParser::Options options = parser.parse(argc, argv);\n> > > > if (options.isSet(OptHelp))\n> > > > diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> > > > index 7bc1360..6edf370 100644\n> > > > --- a/src/qcam/main_window.cpp\n> > > > +++ b/src/qcam/main_window.cpp\n> > > > @@ -28,6 +28,9 @@\n> > > > #include \n> > > >\n> > > > #include \"dng_writer.h\"\n> > > > +#include \"main_window.h\"\n> > > > +#include \"viewfinder.h\"\n> > > > +#include \"viewfinderGL.h\"\n> >\n> > According to the name scheme we use, I think this should be\n> > viewfinder_gl.h.\n> \n> OK. will fix it.\n> \n> > > > using namespace libcamera;\n> > > >\n> > > > @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm, const OptionsParser::Options &options)\n> > > > setWindowTitle(title_);\n> > > > connect(&titleTimer_, SIGNAL(timeout()), this, SLOT(updateTitle()));\n> > > >\n> > > > - viewfinder_ = new ViewFinder(this);\n> > > > - connect(viewfinder_, &ViewFinder::renderComplete,\n> > > > - this, &MainWindow::queueRequest);\n> > > > - setCentralWidget(viewfinder_);\n> > > > + if (options_.isSet(OptOpenGL)) {\n> > > > + viewfinder_ = new ViewFinderGL(this);\n> > > > + connect(dynamic_cast(viewfinder_), &ViewFinderGL::renderComplete,\n> > > > + this, &MainWindow::queueRequest);\n> > > > + setCentralWidget(dynamic_cast(viewfinder_));\n> > > > + } else {\n> > > > + viewfinder_ = new ViewFinder(this);\n> > > > + connect(dynamic_cast(viewfinder_), &ViewFinder::renderComplete,\n> > > > + this, &MainWindow::queueRequest);\n> > > > + setCentralWidget(dynamic_cast(viewfinder_));\n> > > > + }\n> > >\n> > > I understand that one can't inherit from QObject twice, but this looks\n> > > odd. Is there someway the base class could inherit from QObject or\n> > > possibly QWidget and the derived classes hide their specilization\n> > > somehow? I'm no Qt export so maybe I'm asking for something impossible\n> > > or really stupid.\n>\n> No, I really appreciate your opinions, it'll help me to make this better. :)\n> \n> > If we assume all subclasses of Viewfinder will be QWidget, we could do\n> >\n> > if (options_.isSet(OptOpenGL))\n> > viewfinder_ = new ViewFinderGL(this);\n> > else\n> > viewfinder_ = new ViewFinder(this);\n> >\n> > QWidget *vf>idget = dynamic_cast(viewfinder_);\n> > connect(vfWidget, &ViewFinderHandler::renderComplete,\n> > this, &MainWindow::queueRequest);\n> > setCentralWidget(vfWidget);\n> >\n> > With ViewFinderHandler::renderComplete() being a signal in the base\n> > class.\n>\n> I will try this way in the next version.\n> \n> > > > +\n> > > > adjustSize();\n> > > >\n> > > > /* Hotplug/unplug support */\n> > > > diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> > > > index 4606fe4..a852ef4 100644\n> > > > --- a/src/qcam/main_window.h\n> > > > +++ b/src/qcam/main_window.h\n> > > > @@ -38,6 +38,7 @@ enum {\n> > > > OptCamera = 'c',\n> > > > OptHelp = 'h',\n> > > > OptStream = 's',\n> > > > + OptOpenGL = 'o',\n> > > > };\n> > > >\n> > > > class CaptureRequest\n> > > > @@ -102,7 +103,7 @@ private:\n> > > > QAction *startStopAction_;\n> > > > QComboBox *cameraCombo_;\n> > > > QAction *saveRaw_;\n> > > > - ViewFinder *viewfinder_;\n> > > > + ViewFinderHandler *viewfinder_;\n> >\n> > I'd split this patch in two, with one patch that renames the existing\n> > ViewFinder to ViewFinderQt and creates a ViewFinder base class (which\n> > you call ViewFinderHandler here), and a second patch that adds\n> > ViewFinderGL.\n> \n> OK. will do.\n> \n> > > > QIcon iconPlay_;\n> > > > QIcon iconStop_;\n> > > > diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> > > > index 045db52..6a58947 100644\n> > > > --- a/src/qcam/meson.build\n> > > > +++ b/src/qcam/meson.build\n> > > > @@ -7,11 +7,13 @@ qcam_sources = files([\n> > > > 'main.cpp',\n> > > > 'main_window.cpp',\n> > > > 'viewfinder.cpp',\n> > > > + 'viewfinderGL.cpp'\n> > > > ])\n> > > >\n> > > > qcam_moc_headers = files([\n> > > > 'main_window.h',\n> > > > 'viewfinder.h',\n> > > > + 'viewfinderGL.h'\n> > > > ])\n> > > >\n> > > > qcam_resources = files([\n> > > > diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> > > > new file mode 100644\n> > > > index 0000000..f54c264\n> > > > --- /dev/null\n> > > > +++ b/src/qcam/shader.h\n> > > > @@ -0,0 +1,104 @@\n> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > +/*\n> > > > + * Copyright (C) 2020, Linaro\n> > > > + *\n> > > > + * shader.h - YUV format converter shader source code\n> > > > + */\n> > > > +#ifndef __SHADER_H__\n> > > > +#define __SHADER_H__\n> > >\n> > > I think the content of this file should be moved inside viewfinderGL.cpp\n> >\n> > Or maybe to viewfinder_gl_shader.cpp if we want to keep it separate.\n> > Header files should only contain the declarations in any case, not the\n> > actual variable contents.\n> >\n> > Ideally we should store the shaders in files of their own, not in a C\n> > array, and have them pulled in as Qt resources\n> > (https://doc.qt.io/qt-5/resources.html#using-resources-in-the-application\n> > ).\n> > They can be read through a QFile. It's something we can do on top of\n> > this patch.\n>\n> Actually, QOpenGLShader/QOpenGLShaderProgram are provide some methods\n> to build/compile the shader source code from file directly but if I load it\n> from file which means\n> additional shader installation steps are required and also there are some\n> potential risks\n> if the shader is not in the right position. That's why I push them into one\n> header file.\n> Forgive my lazy ...:-P\n> Anyway refine this part in the next version.\n\nI agree with you that storing shaders in files that need to be installed\nisn't very nice, for the reasons you described. However, if you store\nthem in files in the source tree that are then compiled as Qt resources,\nthe data will be in the same binary, and always accessible. I think that\nwould be a good middle-ground, making the shader sources nicer to read,\nmodify and review, and still easy to use in the code through QFile. If\nQFile::map() works on resources, that will be very simple, otherwise\nQFile::readAll() will give you a QByteArray with the data.\n\n> > > > +\n> > > > +/* Vertex shader for NV formats */\n> > > > +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n> > >\n> > > could this (and bellow) be made static const ?\n> > >\n> > > > + \"attribute vec2 textureIn;\\n\"\n> > > > + \"varying vec2 textureOut;\\n\"\n> > > > + \"void main(void)\\n\"\n> > > > + \"{\\n\"\n> > > > + \"gl_Position = vertexIn;\\n\"\n> > > > + \"textureOut = textureIn;\\n\"\n> > > > + \"}\\n\";\n> > > > +\n> > > > +/* Fragment shader for NV12, NV16 and NV24 */\n> > > > +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > > > + \"precision highp float;\\n\"\n> > > > + \"#endif\\n\"\n> > > > + \"varying vec2 textureOut;\\n\"\n> > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > + \"void main(void)\\n\"\n> > > > + \"{\\n\"\n> > > > + \"vec3 yuv;\\n\"\n> > > > + \"vec3 rgb;\\n\"\n> > > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > > > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > > > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > + \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > > > + \"}\\n\";\n> > > > +\n> > > > +/* Fragment shader for NV21, NV61 and NV42 */\n> > > > +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> > > > + \"precision highp float;\\n\"\n> > > > + \"#endif\\n\"\n> > > > + \"varying vec2 textureOut;\\n\"\n> > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > + \"void main(void)\\n\"\n> > > > + \"{\\n\"\n> > > > + \"vec3 yuv;\\n\"\n> > > > + \"vec3 rgb;\\n\"\n> > > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > > > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > > > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > > + \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > > > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > > > + \"}\\n\";\n> > > > +\n> > > > +/* Fragment shader for YUV420 */\n> > > > +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > > > + \"precision highp float;\\n\"\n> > > > + \"#endif\\n\"\n> > > > + \"varying vec2 textureOut;\\n\"\n> > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > + \"uniform sampler2D tex_v;\\n\"\n> > > > + \"void main(void)\\n\"\n> > > > + \"{\\n\"\n> > > > + \"vec3 yuv;\\n\"\n> > > > + \"vec3 rgb;\\n\"\n> > > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > > > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > > > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > + \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > > > + \"}\\n\";\n> > > > +\n> > > > +/* Fragment shader for YVU420 */\n> > > > +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> > > > + \"#endif\\n\"\n> > > > + \"varying vec2 textureOut;\\n\"\n> > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > + \"uniform sampler2D tex_v;\\n\"\n> > > > + \"void main(void)\\n\"\n> > > > + \"{\\n\"\n> > > > + \"vec3 yuv;\\n\"\n> > > > + \"vec3 rgb;\\n\"\n> > > > + \"mat3 convert_mat = mat3(vec3(1.1640625, 1.1640625, 1.1640625),\\n\"\n> > > > + \" vec3(0.0, -0.390625, 2.015625),\\n\"\n> > > > + \" vec3(1.5975625, -0.8125, 0.0));\\n\"\n> > > > + \"yuv.x = texture2D(tex_y, textureOut).r - 0.0625;\\n\"\n> > > > + \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > > > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > > > + \"}\\n\";\n> > > > +#endif // __SHADER_H__\n> > > > diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> > > > index dcf8a15..d3a2422 100644\n> > > > --- a/src/qcam/viewfinder.cpp\n> > > > +++ b/src/qcam/viewfinder.cpp\n> > > > @@ -33,22 +33,30 @@ static const QMap nativeFormats\n> > > > { libcamera::formats::BGR888, QImage::Format_RGB888 },\n> > > > };\n> > > >\n> > > > -ViewFinder::ViewFinder(QWidget *parent)\n> > > > - : QWidget(parent), buffer_(nullptr)\n> > > > +ViewFinderHandler::ViewFinderHandler()\n> > > > {\n> > > > - icon_ = QIcon(\":camera-off.svg\");\n> > > > }\n> >\n> > You don't need an empty constructor in the base class, you can just drop\n> > it.\n> >\n> > > >\n> > > > -ViewFinder::~ViewFinder()\n> > > > +ViewFinderHandler::~ViewFinderHandler()\n> > > > {\n> > > > }\n> > > >\n> > > > -const QList &ViewFinder::nativeFormats() const\n> > > > +const QList &ViewFinderHandler::nativeFormats() const\n> > > > {\n> > > > static const QList formats = ::nativeFormats.keys();\n> > > > return formats;\n> > > > }\n> >\n> > I expect native formats to be different for ViewFinderQt and\n> > ViewFinderGL, so I'd make this a pure virtual function with\n> > implementations in the derived classes. ViewFinderGL should report all\n> > the formats for which you have conversion shaders.\n>\n> ok, I will rewrite this.\n> \n> > > >\n> > > > +ViewFinder::ViewFinder(QWidget *parent)\n> > > > + : QWidget(parent), buffer_(nullptr)\n> > > > +{\n> > > > + icon_ = QIcon(\":camera-off.svg\");\n> > > > +}\n> > >\n> > > I agree with Kieran here I think the base class should be named\n> > > ViewFinder and the two derived classes ViewFinderQt and ViewFinderGL or\n> > > something similar.\n> >\n> > Seems we all agree then :-)\n> >\n> > > I also think you should move the base class to its own .h file (and .cpp\n> > > file if needed).\n> >\n> > Agreed.\n>\n> I will modify it accordingly.\n> \n> > > > +\n> > > > +ViewFinder::~ViewFinder()\n> > > > +{\n> > > > +}\n> > > > +\n> > > > int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> > > > const QSize &size)\n> > > > {\n> > > > diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> > > > index 26a1320..741d694 100644\n> > > > --- a/src/qcam/viewfinder.h\n> > > > +++ b/src/qcam/viewfinder.h\n> > > > @@ -13,6 +13,8 @@\n> > > > #include \n> > > > #include \n> > > > #include \n> > > > +#include \n> > > > +#include \n> > > > #include \n> > > > #include \n> > > >\n> > > > @@ -28,7 +30,23 @@ struct MappedBuffer {\n> > > > size_t size;\n> > > > };\n> > > >\n> > > > -class ViewFinder : public QWidget\n> > > > +class ViewFinderHandler\n> > > > +{\n> > > > +public:\n> > > > + ViewFinderHandler();\n> > > > + virtual ~ViewFinderHandler();\n> > > > +\n> > > > + const QList &nativeFormats() const;\n> > > > +\n> > > > + virtual int setFormat(const libcamera::PixelFormat &format, const QSize &size) =0;\n> > > > + virtual void render(libcamera::FrameBuffer *buffer, MappedBuffer *map) =0;\n> > > > + virtual void stop() =0;\n> > > > +\n> > > > + virtual QImage getCurrentImage() =0;\n> > > > +\n> > > > +};\n> > > > +\n> > > > +class ViewFinder : public QWidget, public ViewFinderHandler\n> > > > {\n> > > > Q_OBJECT\n> > > >\n> > > > @@ -36,8 +54,6 @@ public:\n> > > > ViewFinder(QWidget *parent);\n> > > > ~ViewFinder();\n> > > >\n> > > > - const QList &nativeFormats() const;\n> > > > -\n> > > > int setFormat(const libcamera::PixelFormat &format, const QSize &size);\n> > > > void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > > > void stop();\n> > > > @@ -67,5 +83,4 @@ private:\n> > > > QImage image_;\n> > > > QMutex mutex_; /* Prevent concurrent access to image_ */\n> > > > };\n> > > > -\n> >\n> > This blank line should be kept.\n>\n> sure, will fix it.\n> \n> > > > #endif /* __QCAM_VIEWFINDER__ */\n> > > > diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> > > > new file mode 100644\n> > > > index 0000000..7b47beb\n> > > > --- /dev/null\n> > > > +++ b/src/qcam/viewfinderGL.cpp\n> > > > @@ -0,0 +1,335 @@\n> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > +/*\n> > > > + * Copyright (C) 2020, Linaro\n> > > > + *\n> > > > + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> > > > + */\n> > > > +\n> > > > +#include \"shader.h\"\n> > > > +#include \"viewfinderGL.h\"\n> > > > +\n> > > > +#include \n> > > > +\n> > > > +#include \n> > > > +\n> > > > +#define ATTRIB_VERTEX 0\n> > > > +#define ATTRIB_TEXTURE 1\n> > > > +\n> > > > +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> > > > + : QOpenGLWidget(parent),\n> > > > + glBuffer(QOpenGLBuffer::VertexBuffer),\n> > > > + pFShader(nullptr),\n> > > > + pVShader(nullptr),\n> > > > + textureU(QOpenGLTexture::Target2D),\n> > > > + textureV(QOpenGLTexture::Target2D),\n> > > > + textureY(QOpenGLTexture::Target2D),\n> > > > + yuvDataPtr(nullptr)\n> > > > +\n> > > > +{\n> > > > +}\n> > > > +\n> > > > +ViewFinderGL::~ViewFinderGL()\n> > > > +{\n> > > > + removeShader();\n> > > > +\n> > > > + if(textureY.isCreated())\n> > > > + textureY.destroy();\n> > > > +\n> > > > + if(textureU.isCreated())\n> > > > + textureU.destroy();\n> > > > +\n> > > > + if(textureV.isCreated())\n> > > > + textureV.destroy();\n> > > > +\n> > > > + glBuffer.destroy();\n> > > > +}\n> > > > +\n> > > > +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> > > > + const QSize &size)\n> > > > +{\n> > > > + format_ = format;\n> > > > + size_ = size;\n> > > > +\n> > > > + updateGeometry();\n> > > > + return 0;\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::stop()\n> > > > +{\n> > > > + if (buffer_) {\n> > > > + renderComplete(buffer_);\n> > > > + buffer_ = nullptr;\n> > > > + }\n> > > > +}\n> > > > +\n> > > > +QImage ViewFinderGL::getCurrentImage()\n> > > > +{\n> > > > + QMutexLocker locker(&mutex_);\n> > > > +\n> > > > + return(grabFramebuffer());\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::render(libcamera::FrameBuffer *buffer, MappedBuffer *map)\n> > > > +{\n> > > > + if (buffer->planes().size() != 1) {\n> > > > + qWarning() << \"Multi-planar buffers are not supported\";\n> > > > + return;\n> > > > + }\n> > > > +\n> > > > + unsigned char *memory = static_cast(map->memory);\n> > > > + if(memory) {\n> >\n> > Can memory be null here ?\n>\n> I will rewrite this.\n> \n> > > > + yuvDataPtr = memory;\n> > > > + update();\n> > > > + buffer_ = buffer;\n> > > > + }\n> > > > +\n> > > > + if (buffer)\n> > > > + renderComplete(buffer);\n> >\n> > Here's you're supposed to signal completion of the previous buffer, not\n> > the current buffer. That's why there's a std::swap() in the existing\n> > ViewFinder::render().\n> \n> Is That means I should\n> emit renderComplete(buffer_); after update();\n> then\n> buffer_ = buffer;\n> ...\n> I don't quite understand what you mean, would you please explain it in\n> detail?\n> Forgive my stupid..:-P\n\nA stupid person couldn't have sent this patch in the first place :-)\n\nThe ViewFinder component performs rendering asynchronously. It receives\na libcamera buffer in render(), and is tasked to display it on the\nscreen. The render() function returns before the rendering is complete.\nWe thus need to tell the caller when the rendering is complete, to let\nit reuse the buffer once ViewFinder doesn't need it anymore. That's what\nrenderComplete() is for.\n\nWith the code above, you signal that the buffer has been rendered right\nafter calling update(). That's not true, rendering is still in progress\nat that point. You can only give the buffer back to the caller with\nrenderComplete() once a new buffer has been received to be rendered.\nThat's why the current ViewFinder implementation has\n\n\tstd::swap(buffer_, buffer);\n\nAfter this call, buffer contains the old buffer, and we then call\n\n\tif (buffer)\n\t\trenderComplete(buffer);\n\nto give the old buffer back. We need to check here if buffer is null, as\nwhen the first buffer is rendered, there's no old buffer to be given\nback.\n\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> > > > +{\n> > > > + if(buffer) {\n> > > > + yuvDataPtr = buffer;\n> > > > + update();\n> > > > + }\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::setFrameSize(int width, int height)\n> > > > +{\n> > > > + if(width > 0 && height > 0) {\n> > > > + width_ = width;\n> > > > + height_ = height;\n> > > > + }\n> > > > +}\n> > > > +\n> > > > +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> > > > +{\n> > > > + char *fsrc = nullptr;\n> > > > +\n> > > > + switch(format) {\n> > > > + case libcamera::formats::NV12:\n> > > > + horzSubSample_ = 2;\n> > > > + vertSubSample_ = 2;\n> > > > + fsrc = NV_2_planes_UV;\n> > > > + break;\n> > > > + case libcamera::formats::NV21:\n> > > > + horzSubSample_ = 2;\n> > > > + vertSubSample_ = 2;\n> > > > + fsrc = NV_2_planes_VU;\n> > > > + break;\n> > > > + case libcamera::formats::NV16:\n> > > > + horzSubSample_ = 2;\n> > > > + vertSubSample_ = 1;\n> > > > + fsrc = NV_2_planes_UV;\n> > > > + break;\n> > > > + case libcamera::formats::NV61:\n> > > > + horzSubSample_ = 2;\n> > > > + vertSubSample_ = 1;\n> > > > + fsrc = NV_2_planes_VU;\n> > > > + break;\n> > > > + case libcamera::formats::NV24:\n> > > > + horzSubSample_ = 1;\n> > > > + vertSubSample_ = 1;\n> > > > + fsrc = NV_2_planes_UV;\n> > > > + break;\n> > > > + case libcamera::formats::NV42:\n> > > > + horzSubSample_ = 1;\n> > > > + vertSubSample_ = 1;\n> > > > + fsrc = NV_2_planes_VU;\n> > > > + break;\n> > > > + case libcamera::formats::YUV420:\n> > > > + horzSubSample_ = 2;\n> > > > + vertSubSample_ = 2;\n> > > > + fsrc = NV_3_planes_UV;\n> > > > + break;\n> > > > + default:\n> > > > + break;\n> > > > + };\n> > > > +\n> > > > + if(fsrc == nullptr) {\n> > > > + qDebug() << __func__ << \"[ERROR]:\" <<\" No suitable fragment shader can be select.\";\n> > > > + }\n> > > > + return fsrc;\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::createFragmentShader()\n> > > > +{\n> > > > + bool bCompile;\n> > > > +\n> > > > + pFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> > > > + char *fsrc = selectFragmentShader(format_);\n> > > > +\n> > > > + bCompile = pFShader->compileSourceCode(fsrc);\n> > > > + if(!bCompile)\n> > > > + {\n> > > > + qDebug() << __func__ << \":\" << pFShader->log();\n> > > > + }\n> > > > +\n> > > > + shaderProgram.addShader(pFShader);\n> > > > +\n> > > > + // Link shader pipeline\n> > > > + if (!shaderProgram.link()) {\n> > > > + qDebug() << __func__ << \": shader program link failed.\\n\" << shaderProgram.log();\n> > > > + close();\n> > > > + }\n> > > > +\n> > > > + // Bind shader pipeline for use\n> > > > + if (!shaderProgram.bind()) {\n> > > > + qDebug() << __func__ << \": shader program binding failed.\\n\" << shaderProgram.log();\n> > > > + close();\n> > > > + }\n> > > > +\n> > > > + shaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> > > > + shaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> > > > +\n> > > > + shaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> > > > + shaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> > > > +\n> > > > + textureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> > > > + textureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> > > > + textureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> > > > +\n> > > > + if(!textureY.isCreated())\n> > > > + textureY.create();\n> > > > +\n> > > > + if(!textureU.isCreated())\n> > > > + textureU.create();\n> > > > +\n> > > > + if(!textureV.isCreated())\n> > > > + textureV.create();\n> > > > +\n> > > > + id_y = textureY.textureId();\n> > > > + id_u = textureU.textureId();\n> > > > + id_v = textureV.textureId();\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::configureTexture(unsigned int id)\n> > > > +{\n> > > > + glBindTexture(GL_TEXTURE_2D, id);\n> > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);\n> > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);\n> > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);\n> > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::removeShader()\n> > > > +{\n> > > > + if (shaderProgram.isLinked()) {\n> > > > + shaderProgram.release();\n> > > > + shaderProgram.removeAllShaders();\n> > > > + }\n> > > > +\n> > > > + if(pFShader)\n> > > > + delete pFShader;\n> > > > +\n> > > > + if(pVShader)\n> > > > + delete pVShader;\n> >\n> > If we stop and restart the stream with a different format, the previous\n> > shader will be used, as removeShader() isn't called. I don't think\n> > that's right. I believe you need to call removeShader() at the beginning\n> > of setFormat().\n>\n> Yes. you're right. I will refine the setFormat() function.\n> \n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::initializeGL()\n> > > > +{\n> > > > + bool bCompile;\n> > > > +\n> > > > + initializeOpenGLFunctions();\n> > > > + glEnable(GL_TEXTURE_2D);\n> > > > + glDisable(GL_DEPTH_TEST);\n> > > > +\n> > > > + static const GLfloat vertices[] {\n> > > > + -1.0f,-1.0f,\n> > > > + -1.0f,+1.0f,\n> > > > + +1.0f,+1.0f,\n> > > > + +1.0f,-1.0f,\n> > > > + 0.0f,1.0f,\n> > > > + 0.0f,0.0f,\n> > > > + 1.0f,0.0f,\n> > > > + 1.0f,1.0f,\n> > > > + };\n> > > > +\n> > > > + glBuffer.create();\n> > > > + glBuffer.bind();\n> > > > + glBuffer.allocate(vertices,sizeof(vertices));\n> > > > +\n> > > > + /* Create Vertex Shader */\n> > > > + pVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> > > > + char *vsrc = NV_Vertex_shader;\n> > > > +\n> > > > + bCompile = pVShader->compileSourceCode(vsrc);\n> > > > + if(!bCompile) {\n> > > > + qDebug() << __func__<< \":\" << pVShader->log();\n> > > > + }\n> > > > +\n> > > > + shaderProgram.addShader(pVShader);\n> > > > +\n> > > > + glClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::render()\n> > > > +{\n> > > > + switch(format_) {\n> > > > + case libcamera::formats::NV12:\n> > > > + case libcamera::formats::NV21:\n> > > > + case libcamera::formats::NV16:\n> > > > + case libcamera::formats::NV61:\n> > > > + case libcamera::formats::NV24:\n> > > > + case libcamera::formats::NV42:\n> > > > + /* activate texture 0 */\n> > > > + glActiveTexture(GL_TEXTURE0);\n> > > > + configureTexture(id_y);\n> > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > > > + glUniform1i(textureUniformY, 0);\n> > > > +\n> > > > + /* activate texture 1 */\n> > > > + glActiveTexture(GL_TEXTURE1);\n> > > > + configureTexture(id_u);\n> > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > > > + glUniform1i(textureUniformU, 1);\n> > > > + break;\n> > > > + case libcamera::formats::YUV420:\n> > > > + /* activate texture 0 */\n> > > > + glActiveTexture(GL_TEXTURE0);\n> > > > + configureTexture(id_y);\n> > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(), size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > > > + glUniform1i(textureUniformY, 0);\n> > > > +\n> > > > + /* activate texture 1 */\n> > > > + glActiveTexture(GL_TEXTURE1);\n> > > > + configureTexture(id_u);\n> > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > > > + glUniform1i(textureUniformU, 1);\n> > > > +\n> > > > + /* activate texture 2 */\n> > > > + glActiveTexture(GL_TEXTURE2);\n> > > > + configureTexture(id_v);\n> > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG, size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG, GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> > > > + glUniform1i(textureUniformV, 2);\n> > > > + break;\n> > > > + default:\n> > > > + break;\n> > > > + };\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::paintGL()\n> > > > +{\n> > > > + if(pFShader == nullptr)\n> >\n> > We tend to write\n> >\n> > if (!pFShader)\n>\n> ok will fix it.\n> \n> > > > + createFragmentShader();\n> >\n> > I wonder if we could do this in setFormat().\n>\n> That should be ok, I will try to move this when setting Format.\n> \n> > > > +\n> > > > + if(yuvDataPtr)\n> > > > + {\n> > > > + glClearColor(0.0, 0.0, 0.0, 1.0);\n> > > > + glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> > > > +\n> > > > + render();\n> > > > + glDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> > > > + }\n> > > > +}\n> > > > +\n> > > > +void ViewFinderGL::resizeGL(int w, int h)\n> > > > +{\n> > > > + glViewport(0,0,w,h);\n> > > > +}\n> > > > +\n> > > > +QSize ViewFinderGL::sizeHint() const\n> > > > +{\n> > > > + return size_.isValid() ? size_ : QSize(640, 480);\n> > > > +}\n> > > > diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> > > > new file mode 100644\n> > > > index 0000000..08662ca\n> > > > --- /dev/null\n> > > > +++ b/src/qcam/viewfinderGL.h\n> > > > @@ -0,0 +1,101 @@\n> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > +/*\n> > > > + * Copyright (C) 2020, Linaro\n> > > > + *\n> > > > + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> > > > + */\n> > > > +#ifndef __VIEWFINDERGL_H__\n> > > > +#define __VIEWFINDERGL_H__\n> > > > +\n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +\n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +\n> >\n> > Do you need all these headers in the .h file ? They seem to belong to\n> > the .cpp file.\n>\n> OK I will check or just remove it.\n> \n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +\n> > > > +#include \n> > > > +#include \n> >\n> > Blank line missing here.\n>\n> sure, will do .\n> \n> > > > +#include \"viewfinder.h\"\n> > > > +\n> > > > +class ViewFinderGL : public QOpenGLWidget,\n> > > > + public ViewFinderHandler,\n> > > > + protected QOpenGLFunctions\n> > > > +{\n> > > > + Q_OBJECT\n> > > > +\n> > > > +public:\n> > > > + ViewFinderGL(QWidget *parent = 0);\n> > > > + ~ViewFinderGL();\n> > > > +\n> > > > + int setFormat(const libcamera::PixelFormat &format, const QSize &size);\n> >\n> > You should add \"override\" to qualify all overridden functions.\n> >\n> > int setFormat(const libcamera::PixelFormat &format,\n> > const QSize &size) override;\n>\n> ok will do.\n> \n> > > > + void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > > > + void stop();\n> > > > +\n> > > > + QImage getCurrentImage();\n> > > > +\n> > > > + void setFrameSize(int width, int height);\n> > > > + void updateFrame(unsigned char *buffer);\n> >\n> > Those two functions seem unused.\n> \n> sure, I will remove it.\n> \n> > > > +\n> > > > + char *selectFragmentShader(unsigned int format);\n> > > > + void createFragmentShader();\n> >\n> > And these two functions can be private.\n>\n> will move to private.\n> \n> > > > + void render();\n> > > > +\n> > > > +Q_SIGNALS:\n> > > > + void renderComplete(libcamera::FrameBuffer *buffer);\n> > > > +\n> > > > +protected:\n> > > > + void initializeGL() Q_DECL_OVERRIDE;\n> >\n> > You can use \"override\" directly, we know the compiler supports it.\n>\n> ok, I will fix it up.\n> \n> > > > + void paintGL() Q_DECL_OVERRIDE;\n> > > > + void resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> > > > + QSize sizeHint() const override;\n> > > > +\n> > > > +private:\n> > > > +\n> > > > + void configureTexture(unsigned int id);\n> > > > + void removeShader();\n> > > > +\n> > > > + /* Captured image size, format and buffer */\n> > > > + libcamera::FrameBuffer *buffer_;\n> > > > + libcamera::PixelFormat format_;\n> > > > + QOpenGLBuffer glBuffer;\n> > > > + QSize size_;\n> > > > +\n> > > > + GLuint id_u;\n> > > > + GLuint id_v;\n> > > > + GLuint id_y;\n> > > > +\n> > > > + QMutex mutex_; /* Prevent concurrent access to image_ */\n> > > > +\n> > > > + QOpenGLShader *pFShader;\n> > > > + QOpenGLShader *pVShader;\n> > > > + QOpenGLShaderProgram shaderProgram;\n> > > > +\n> > > > + GLuint textureUniformU;\n> > > > + GLuint textureUniformV;\n> > > > + GLuint textureUniformY;\n> > > > +\n> > > > + QOpenGLTexture textureU;\n> > > > + QOpenGLTexture textureV;\n> > > > + QOpenGLTexture textureY;\n> > > > +\n> > > > + unsigned int width_;\n> > > > + unsigned int height_;\n> > > > +\n> > > > + unsigned char* yuvDataPtr;\n> > > > +\n> > > > + /* NV parameters */\n> > > > + unsigned int horzSubSample_ ;\n> > > > + unsigned int vertSubSample_;\n> > > > +};\n> > > > +#endif // __VIEWFINDERGL_H__","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 96883BE905\n\tfor ;\n\tFri, 3 Jul 2020 10:14:12 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id CF8B060C59;\n\tFri, 3 Jul 2020 12:14:11 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5181E603AE\n\tfor ;\n\tFri, 3 Jul 2020 12:14:10 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi\n\t[81.175.216.236])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 92A7651B;\n\tFri, 3 Jul 2020 12:14:09 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"vFqxu8of\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1593771249;\n\tbh=h8X5ZxvuAl6kOc8seSvNwblLkK5tyC/S+wE2UhtBUlo=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=vFqxu8ofcUOEUrij1cElyWOeKU3kqLpk7p7dNr/udPdB44X5y2aWhEmueBgOc9hFq\n\tAs3MToe6CXh0YJXuY3Bw43iBBbCWhf4so5P9/07TS+2yn9N0mSK5dLfXylOsM+LQ1d\n\tAygosOPG3f1TEI5+yVqeM/f+ikQQyG0QhrP5MSlU=","Date":"Fri, 3 Jul 2020 13:14:05 +0300","From":"Laurent Pinchart ","To":"Show Liu ","Message-ID":"<20200703101405.GA5963@pendragon.ideasonboard.com>","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>\n\t<20200630150343.GA2259976@oden.dyn.berto.se>\n\t<20200630162624.GM5850@pendragon.ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11190,"web_url":"https://patchwork.libcamera.org/comment/11190/","msgid":"","date":"2020-07-06T08:06:04","subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","submitter":{"id":24,"url":"https://patchwork.libcamera.org/api/people/24/","name":"Show Liu","email":"show.liu@linaro.org"},"content":"Hi Laurent,\n\n\n\nOn Fri, Jul 3, 2020 at 6:14 PM Laurent Pinchart <\nlaurent.pinchart@ideasonboard.com> wrote:\n\n> Hi Show,\n>\n> On Fri, Jul 03, 2020 at 03:46:32PM +0800, Show Liu wrote:\n> > On Wed, Jul 1, 2020 at 12:26 AM Laurent Pinchart wrote:\n> > > On Tue, Jun 30, 2020 at 05:03:43PM +0200, Niklas Söderlund wrote:\n> > > > Hi Show,\n> > > >\n> > > > Thanks for your work.\n> > >\n> > > Likewise :-)\n> > >\n> > > > I really like this version! The structure is almost there and much\n> > > > better then previous versions. As Kieran points out there are style\n> > > > errors that checkstyle.py will help you point out so I will ignore\n> them\n> > > > in this review.\n> >\n> > Thank you.:-)\n> >\n> > > > On 2020-06-24 15:37:05 +0800, Show Liu wrote:\n> > > > > The patch is to render the NV family YUV formats by OpenGL shader.\n> > > > > V3: refine the fragment shader for better pixel color.\n> > > > >\n> > > > > Signed-off-by: Show Liu \n> > > > > ---\n> > > > > src/qcam/main.cpp | 2 +\n> > > > > src/qcam/main_window.cpp | 19 ++-\n> > > > > src/qcam/main_window.h | 3 +-\n> > > > > src/qcam/meson.build | 2 +\n> > > > > src/qcam/shader.h | 104 ++++++++++++\n> > > > > src/qcam/viewfinder.cpp | 18 +-\n> > > > > src/qcam/viewfinder.h | 23 ++-\n> > > > > src/qcam/viewfinderGL.cpp | 335\n> ++++++++++++++++++++++++++++++++++++++\n> > > > > src/qcam/viewfinderGL.h | 101 ++++++++++++\n> > > > > 9 files changed, 593 insertions(+), 14 deletions(-)\n> > > > > create mode 100644 src/qcam/shader.h\n> > > > > create mode 100644 src/qcam/viewfinderGL.cpp\n> > > > > create mode 100644 src/qcam/viewfinderGL.h\n> > > > >\n> > > > > diff --git a/src/qcam/main.cpp b/src/qcam/main.cpp\n> > > > > index b3468cb..a3ea471 100644\n> > > > > --- a/src/qcam/main.cpp\n> > > > > +++ b/src/qcam/main.cpp\n> > > > > @@ -35,6 +35,8 @@ OptionsParser::Options parseOptions(int argc,\n> char *argv[])\n> > > > > \"help\");\n> > > > > parser.addOption(OptStream, &streamKeyValue,\n> > > > > \"Set configuration of a camera stream\",\n> \"stream\", true);\n> > > > > + parser.addOption(OptOpenGL, OptionNone, \"Render YUV formats\n> frame via OpenGL shader\",\n> > > > > + \"opengl\");\n> > >\n> > > Should we default to OpenGL when possible, and add an option to force a\n> > > particular backend ? Maybe -r/--render={gles,qt}\n> >\n> > I'd like to mention why I set OpenGL rendering as an option.\n> > First, this depends on the \"GPU enabled\" platforms,\n> > and it always needs some additional steps to make sure the GPU is ready.\n> > Second, as I said this patch is only for NV family YUV formats at\n> present,\n> > and I believe it just covers very small parts of camera stream formats.\n> > That's why I am still trying to make it support more formats as I can.:-)\n>\n> That's a good point, defaulting to the Qt renderer for now may be best.\n> I'd still prefer a -r/--render option if possible though, to make it\n> easier to change the default and to add more renderers later if needed.\n>\nOK. Understood. I will try this in the next version.\n\n>\n> > > > >\n> > > > > OptionsParser::Options options = parser.parse(argc, argv);\n> > > > > if (options.isSet(OptHelp))\n> > > > > diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp\n> > > > > index 7bc1360..6edf370 100644\n> > > > > --- a/src/qcam/main_window.cpp\n> > > > > +++ b/src/qcam/main_window.cpp\n> > > > > @@ -28,6 +28,9 @@\n> > > > > #include \n> > > > >\n> > > > > #include \"dng_writer.h\"\n> > > > > +#include \"main_window.h\"\n> > > > > +#include \"viewfinder.h\"\n> > > > > +#include \"viewfinderGL.h\"\n> > >\n> > > According to the name scheme we use, I think this should be\n> > > viewfinder_gl.h.\n> >\n> > OK. will fix it.\n> >\n> > > > > using namespace libcamera;\n> > > > >\n> > > > > @@ -105,10 +108,18 @@ MainWindow::MainWindow(CameraManager *cm,\n> const OptionsParser::Options &options)\n> > > > > setWindowTitle(title_);\n> > > > > connect(&titleTimer_, SIGNAL(timeout()), this,\n> SLOT(updateTitle()));\n> > > > >\n> > > > > - viewfinder_ = new ViewFinder(this);\n> > > > > - connect(viewfinder_, &ViewFinder::renderComplete,\n> > > > > - this, &MainWindow::queueRequest);\n> > > > > - setCentralWidget(viewfinder_);\n> > > > > + if (options_.isSet(OptOpenGL)) {\n> > > > > + viewfinder_ = new ViewFinderGL(this);\n> > > > > + connect(dynamic_cast(viewfinder_),\n> &ViewFinderGL::renderComplete,\n> > > > > + this, &MainWindow::queueRequest);\n> > > > > + setCentralWidget(dynamic_cast *>(viewfinder_));\n> > > > > + } else {\n> > > > > + viewfinder_ = new ViewFinder(this);\n> > > > > + connect(dynamic_cast(viewfinder_),\n> &ViewFinder::renderComplete,\n> > > > > + this, &MainWindow::queueRequest);\n> > > > > + setCentralWidget(dynamic_cast *>(viewfinder_));\n> > > > > + }\n> > > >\n> > > > I understand that one can't inherit from QObject twice, but this\n> looks\n> > > > odd. Is there someway the base class could inherit from QObject or\n> > > > possibly QWidget and the derived classes hide their specilization\n> > > > somehow? I'm no Qt export so maybe I'm asking for something\n> impossible\n> > > > or really stupid.\n> >\n> > No, I really appreciate your opinions, it'll help me to make this\n> better. :)\n> >\n> > > If we assume all subclasses of Viewfinder will be QWidget, we could do\n> > >\n> > > if (options_.isSet(OptOpenGL))\n> > > viewfinder_ = new ViewFinderGL(this);\n> > > else\n> > > viewfinder_ = new ViewFinder(this);\n> > >\n> > > QWidget *vf>idget = dynamic_cast(viewfinder_);\n> > > connect(vfWidget, &ViewFinderHandler::renderComplete,\n> > > this, &MainWindow::queueRequest);\n> > > setCentralWidget(vfWidget);\n> > >\n> > > With ViewFinderHandler::renderComplete() being a signal in the base\n> > > class.\n> >\n> > I will try this way in the next version.\n> >\n> > > > > +\n> > > > > adjustSize();\n> > > > >\n> > > > > /* Hotplug/unplug support */\n> > > > > diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h\n> > > > > index 4606fe4..a852ef4 100644\n> > > > > --- a/src/qcam/main_window.h\n> > > > > +++ b/src/qcam/main_window.h\n> > > > > @@ -38,6 +38,7 @@ enum {\n> > > > > OptCamera = 'c',\n> > > > > OptHelp = 'h',\n> > > > > OptStream = 's',\n> > > > > + OptOpenGL = 'o',\n> > > > > };\n> > > > >\n> > > > > class CaptureRequest\n> > > > > @@ -102,7 +103,7 @@ private:\n> > > > > QAction *startStopAction_;\n> > > > > QComboBox *cameraCombo_;\n> > > > > QAction *saveRaw_;\n> > > > > - ViewFinder *viewfinder_;\n> > > > > + ViewFinderHandler *viewfinder_;\n> > >\n> > > I'd split this patch in two, with one patch that renames the existing\n> > > ViewFinder to ViewFinderQt and creates a ViewFinder base class (which\n> > > you call ViewFinderHandler here), and a second patch that adds\n> > > ViewFinderGL.\n> >\n> > OK. will do.\n> >\n> > > > > QIcon iconPlay_;\n> > > > > QIcon iconStop_;\n> > > > > diff --git a/src/qcam/meson.build b/src/qcam/meson.build\n> > > > > index 045db52..6a58947 100644\n> > > > > --- a/src/qcam/meson.build\n> > > > > +++ b/src/qcam/meson.build\n> > > > > @@ -7,11 +7,13 @@ qcam_sources = files([\n> > > > > 'main.cpp',\n> > > > > 'main_window.cpp',\n> > > > > 'viewfinder.cpp',\n> > > > > + 'viewfinderGL.cpp'\n> > > > > ])\n> > > > >\n> > > > > qcam_moc_headers = files([\n> > > > > 'main_window.h',\n> > > > > 'viewfinder.h',\n> > > > > + 'viewfinderGL.h'\n> > > > > ])\n> > > > >\n> > > > > qcam_resources = files([\n> > > > > diff --git a/src/qcam/shader.h b/src/qcam/shader.h\n> > > > > new file mode 100644\n> > > > > index 0000000..f54c264\n> > > > > --- /dev/null\n> > > > > +++ b/src/qcam/shader.h\n> > > > > @@ -0,0 +1,104 @@\n> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > > +/*\n> > > > > + * Copyright (C) 2020, Linaro\n> > > > > + *\n> > > > > + * shader.h - YUV format converter shader source code\n> > > > > + */\n> > > > > +#ifndef __SHADER_H__\n> > > > > +#define __SHADER_H__\n> > > >\n> > > > I think the content of this file should be moved inside\n> viewfinderGL.cpp\n> > >\n> > > Or maybe to viewfinder_gl_shader.cpp if we want to keep it separate.\n> > > Header files should only contain the declarations in any case, not the\n> > > actual variable contents.\n> > >\n> > > Ideally we should store the shaders in files of their own, not in a C\n> > > array, and have them pulled in as Qt resources\n> > > (\n> https://doc.qt.io/qt-5/resources.html#using-resources-in-the-application\n> > > ).\n> > > They can be read through a QFile. It's something we can do on top of\n> > > this patch.\n> >\n> > Actually, QOpenGLShader/QOpenGLShaderProgram are provide some methods\n> > to build/compile the shader source code from file directly but if I load\n> it\n> > from file which means\n> > additional shader installation steps are required and also there are some\n> > potential risks\n> > if the shader is not in the right position. That's why I push them into\n> one\n> > header file.\n> > Forgive my lazy ...:-P\n> > Anyway refine this part in the next version.\n>\n> I agree with you that storing shaders in files that need to be installed\n> isn't very nice, for the reasons you described. However, if you store\n> them in files in the source tree that are then compiled as Qt resources,\n> the data will be in the same binary, and always accessible. I think that\n> would be a good middle-ground, making the shader sources nicer to read,\n> modify and review, and still easy to use in the code through QFile. If\n> QFile::map() works on resources, that will be very simple, otherwise\n> QFile::readAll() will give you a QByteArray with the data.\n>\nOK. sounds good. I will try this in the next version too.\n\n>\n> > > > > +\n> > > > > +/* Vertex shader for NV formats */\n> > > > > +char NV_Vertex_shader[] = \"attribute vec4 vertexIn;\\n\"\n> > > >\n> > > > could this (and bellow) be made static const ?\n> > > >\n> > > > > + \"attribute vec2 textureIn;\\n\"\n> > > > > + \"varying vec2 textureOut;\\n\"\n> > > > > + \"void main(void)\\n\"\n> > > > > + \"{\\n\"\n> > > > > + \"gl_Position = vertexIn;\\n\"\n> > > > > + \"textureOut = textureIn;\\n\"\n> > > > > + \"}\\n\";\n> > > > > +\n> > > > > +/* Fragment shader for NV12, NV16 and NV24 */\n> > > > > +char NV_2_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > > > > + \"precision highp float;\\n\"\n> > > > > + \"#endif\\n\"\n> > > > > + \"varying vec2 textureOut;\\n\"\n> > > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > > + \"void main(void)\\n\"\n> > > > > + \"{\\n\"\n> > > > > + \"vec3 yuv;\\n\"\n> > > > > + \"vec3 rgb;\\n\"\n> > > > > + \"mat3 convert_mat = mat3(vec3(1.1640625,\n> 1.1640625, 1.1640625),\\n\"\n> > > > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > > > + \"yuv.x = texture2D(tex_y, textureOut).r -\n> 0.0625;\\n\"\n> > > > > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > > + \"yuv.z = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > > > > + \"}\\n\";\n> > > > > +\n> > > > > +/* Fragment shader for NV21, NV61 and NV42 */\n> > > > > +char NV_2_planes_VU[] = \"#ifdef GL_ES\\n\"\n> > > > > + \"precision highp float;\\n\"\n> > > > > + \"#endif\\n\"\n> > > > > + \"varying vec2 textureOut;\\n\"\n> > > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > > + \"void main(void)\\n\"\n> > > > > + \"{\\n\"\n> > > > > + \"vec3 yuv;\\n\"\n> > > > > + \"vec3 rgb;\\n\"\n> > > > > + \"mat3 convert_mat = mat3(vec3(1.1640625,\n> 1.1640625, 1.1640625),\\n\"\n> > > > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > > > + \"yuv.x = texture2D(tex_y, textureOut).r -\n> 0.0625;\\n\"\n> > > > > + \"yuv.y = texture2D(tex_u, textureOut).g - 0.5;\\n\"\n> > > > > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > > + \"gl_FragColor = vec4(rgb, 1.0);\\n\"\n> > > > > + \"}\\n\";\n> > > > > +\n> > > > > +/* Fragment shader for YUV420 */\n> > > > > +char NV_3_planes_UV[] = \"#ifdef GL_ES\\n\"\n> > > > > + \"precision highp float;\\n\"\n> > > > > + \"#endif\\n\"\n> > > > > + \"varying vec2 textureOut;\\n\"\n> > > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > > + \"uniform sampler2D tex_v;\\n\"\n> > > > > + \"void main(void)\\n\"\n> > > > > + \"{\\n\"\n> > > > > + \"vec3 yuv;\\n\"\n> > > > > + \"vec3 rgb;\\n\"\n> > > > > + \"mat3 convert_mat = mat3(vec3(1.1640625,\n> 1.1640625, 1.1640625),\\n\"\n> > > > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > > > + \"yuv.x = texture2D(tex_y, textureOut).r -\n> 0.0625;\\n\"\n> > > > > + \"yuv.y = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > > + \"yuv.z = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > > > > + \"}\\n\";\n> > > > > +\n> > > > > +/* Fragment shader for YVU420 */\n> > > > > +char NV_3_planes_VU[] = \"precision highp float;\\n\"\n> > > > > + \"#endif\\n\"\n> > > > > + \"varying vec2 textureOut;\\n\"\n> > > > > + \"uniform sampler2D tex_y;\\n\"\n> > > > > + \"uniform sampler2D tex_u;\\n\"\n> > > > > + \"uniform sampler2D tex_v;\\n\"\n> > > > > + \"void main(void)\\n\"\n> > > > > + \"{\\n\"\n> > > > > + \"vec3 yuv;\\n\"\n> > > > > + \"vec3 rgb;\\n\"\n> > > > > + \"mat3 convert_mat = mat3(vec3(1.1640625,\n> 1.1640625, 1.1640625),\\n\"\n> > > > > + \" vec3(0.0, -0.390625,\n> 2.015625),\\n\"\n> > > > > + \" vec3(1.5975625, -0.8125,\n> 0.0));\\n\"\n> > > > > + \"yuv.x = texture2D(tex_y, textureOut).r -\n> 0.0625;\\n\"\n> > > > > + \"yuv.y = texture2D(tex_v, textureOut).g - 0.5;\\n\"\n> > > > > + \"yuv.z = texture2D(tex_u, textureOut).r - 0.5;\\n\"\n> > > > > + \"rgb = convert_mat * yuv;\\n\"\n> > > > > + \"gl_FragColor = vec4(rgb, 1);\\n\"\n> > > > > + \"}\\n\";\n> > > > > +#endif // __SHADER_H__\n> > > > > diff --git a/src/qcam/viewfinder.cpp b/src/qcam/viewfinder.cpp\n> > > > > index dcf8a15..d3a2422 100644\n> > > > > --- a/src/qcam/viewfinder.cpp\n> > > > > +++ b/src/qcam/viewfinder.cpp\n> > > > > @@ -33,22 +33,30 @@ static const QMap QImage::Format> nativeFormats\n> > > > > { libcamera::formats::BGR888, QImage::Format_RGB888 },\n> > > > > };\n> > > > >\n> > > > > -ViewFinder::ViewFinder(QWidget *parent)\n> > > > > - : QWidget(parent), buffer_(nullptr)\n> > > > > +ViewFinderHandler::ViewFinderHandler()\n> > > > > {\n> > > > > - icon_ = QIcon(\":camera-off.svg\");\n> > > > > }\n> > >\n> > > You don't need an empty constructor in the base class, you can just\n> drop\n> > > it.\n> > >\n> > > > >\n> > > > > -ViewFinder::~ViewFinder()\n> > > > > +ViewFinderHandler::~ViewFinderHandler()\n> > > > > {\n> > > > > }\n> > > > >\n> > > > > -const QList &ViewFinder::nativeFormats()\n> const\n> > > > > +const QList\n> &ViewFinderHandler::nativeFormats() const\n> > > > > {\n> > > > > static const QList formats =\n> ::nativeFormats.keys();\n> > > > > return formats;\n> > > > > }\n> > >\n> > > I expect native formats to be different for ViewFinderQt and\n> > > ViewFinderGL, so I'd make this a pure virtual function with\n> > > implementations in the derived classes. ViewFinderGL should report all\n> > > the formats for which you have conversion shaders.\n> >\n> > ok, I will rewrite this.\n> >\n> > > > >\n> > > > > +ViewFinder::ViewFinder(QWidget *parent)\n> > > > > + : QWidget(parent), buffer_(nullptr)\n> > > > > +{\n> > > > > + icon_ = QIcon(\":camera-off.svg\");\n> > > > > +}\n> > > >\n> > > > I agree with Kieran here I think the base class should be named\n> > > > ViewFinder and the two derived classes ViewFinderQt and ViewFinderGL\n> or\n> > > > something similar.\n> > >\n> > > Seems we all agree then :-)\n> > >\n> > > > I also think you should move the base class to its own .h file (and\n> .cpp\n> > > > file if needed).\n> > >\n> > > Agreed.\n> >\n> > I will modify it accordingly.\n> >\n> > > > > +\n> > > > > +ViewFinder::~ViewFinder()\n> > > > > +{\n> > > > > +}\n> > > > > +\n> > > > > int ViewFinder::setFormat(const libcamera::PixelFormat &format,\n> > > > > const QSize &size)\n> > > > > {\n> > > > > diff --git a/src/qcam/viewfinder.h b/src/qcam/viewfinder.h\n> > > > > index 26a1320..741d694 100644\n> > > > > --- a/src/qcam/viewfinder.h\n> > > > > +++ b/src/qcam/viewfinder.h\n> > > > > @@ -13,6 +13,8 @@\n> > > > > #include \n> > > > > #include \n> > > > > #include \n> > > > > +#include \n> > > > > +#include \n> > > > > #include \n> > > > > #include \n> > > > >\n> > > > > @@ -28,7 +30,23 @@ struct MappedBuffer {\n> > > > > size_t size;\n> > > > > };\n> > > > >\n> > > > > -class ViewFinder : public QWidget\n> > > > > +class ViewFinderHandler\n> > > > > +{\n> > > > > +public:\n> > > > > + ViewFinderHandler();\n> > > > > + virtual ~ViewFinderHandler();\n> > > > > +\n> > > > > + const QList &nativeFormats() const;\n> > > > > +\n> > > > > + virtual int setFormat(const libcamera::PixelFormat &format,\n> const QSize &size) =0;\n> > > > > + virtual void render(libcamera::FrameBuffer *buffer,\n> MappedBuffer *map) =0;\n> > > > > + virtual void stop() =0;\n> > > > > +\n> > > > > + virtual QImage getCurrentImage() =0;\n> > > > > +\n> > > > > +};\n> > > > > +\n> > > > > +class ViewFinder : public QWidget, public ViewFinderHandler\n> > > > > {\n> > > > > Q_OBJECT\n> > > > >\n> > > > > @@ -36,8 +54,6 @@ public:\n> > > > > ViewFinder(QWidget *parent);\n> > > > > ~ViewFinder();\n> > > > >\n> > > > > - const QList &nativeFormats() const;\n> > > > > -\n> > > > > int setFormat(const libcamera::PixelFormat &format, const\n> QSize &size);\n> > > > > void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > > > > void stop();\n> > > > > @@ -67,5 +83,4 @@ private:\n> > > > > QImage image_;\n> > > > > QMutex mutex_; /* Prevent concurrent access to image_ */\n> > > > > };\n> > > > > -\n> > >\n> > > This blank line should be kept.\n> >\n> > sure, will fix it.\n> >\n> > > > > #endif /* __QCAM_VIEWFINDER__ */\n> > > > > diff --git a/src/qcam/viewfinderGL.cpp b/src/qcam/viewfinderGL.cpp\n> > > > > new file mode 100644\n> > > > > index 0000000..7b47beb\n> > > > > --- /dev/null\n> > > > > +++ b/src/qcam/viewfinderGL.cpp\n> > > > > @@ -0,0 +1,335 @@\n> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > > +/*\n> > > > > + * Copyright (C) 2020, Linaro\n> > > > > + *\n> > > > > + * viewfinderGL.cpp - Render YUV format frame by OpenGL shader\n> > > > > + */\n> > > > > +\n> > > > > +#include \"shader.h\"\n> > > > > +#include \"viewfinderGL.h\"\n> > > > > +\n> > > > > +#include \n> > > > > +\n> > > > > +#include \n> > > > > +\n> > > > > +#define ATTRIB_VERTEX 0\n> > > > > +#define ATTRIB_TEXTURE 1\n> > > > > +\n> > > > > +ViewFinderGL::ViewFinderGL(QWidget *parent)\n> > > > > + : QOpenGLWidget(parent),\n> > > > > + glBuffer(QOpenGLBuffer::VertexBuffer),\n> > > > > + pFShader(nullptr),\n> > > > > + pVShader(nullptr),\n> > > > > + textureU(QOpenGLTexture::Target2D),\n> > > > > + textureV(QOpenGLTexture::Target2D),\n> > > > > + textureY(QOpenGLTexture::Target2D),\n> > > > > + yuvDataPtr(nullptr)\n> > > > > +\n> > > > > +{\n> > > > > +}\n> > > > > +\n> > > > > +ViewFinderGL::~ViewFinderGL()\n> > > > > +{\n> > > > > + removeShader();\n> > > > > +\n> > > > > + if(textureY.isCreated())\n> > > > > + textureY.destroy();\n> > > > > +\n> > > > > + if(textureU.isCreated())\n> > > > > + textureU.destroy();\n> > > > > +\n> > > > > + if(textureV.isCreated())\n> > > > > + textureV.destroy();\n> > > > > +\n> > > > > + glBuffer.destroy();\n> > > > > +}\n> > > > > +\n> > > > > +int ViewFinderGL::setFormat(const libcamera::PixelFormat &format,\n> > > > > + const QSize &size)\n> > > > > +{\n> > > > > + format_ = format;\n> > > > > + size_ = size;\n> > > > > +\n> > > > > + updateGeometry();\n> > > > > + return 0;\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::stop()\n> > > > > +{\n> > > > > + if (buffer_) {\n> > > > > + renderComplete(buffer_);\n> > > > > + buffer_ = nullptr;\n> > > > > + }\n> > > > > +}\n> > > > > +\n> > > > > +QImage ViewFinderGL::getCurrentImage()\n> > > > > +{\n> > > > > + QMutexLocker locker(&mutex_);\n> > > > > +\n> > > > > + return(grabFramebuffer());\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::render(libcamera::FrameBuffer *buffer,\n> MappedBuffer *map)\n> > > > > +{\n> > > > > + if (buffer->planes().size() != 1) {\n> > > > > + qWarning() << \"Multi-planar buffers are not supported\";\n> > > > > + return;\n> > > > > + }\n> > > > > +\n> > > > > + unsigned char *memory = static_cast *>(map->memory);\n> > > > > + if(memory) {\n> > >\n> > > Can memory be null here ?\n> >\n> > I will rewrite this.\n> >\n> > > > > + yuvDataPtr = memory;\n> > > > > + update();\n> > > > > + buffer_ = buffer;\n> > > > > + }\n> > > > > +\n> > > > > + if (buffer)\n> > > > > + renderComplete(buffer);\n> > >\n> > > Here's you're supposed to signal completion of the previous buffer, not\n> > > the current buffer. That's why there's a std::swap() in the existing\n> > > ViewFinder::render().\n> >\n> > Is That means I should\n> > emit renderComplete(buffer_); after update();\n> > then\n> > buffer_ = buffer;\n> > ...\n> > I don't quite understand what you mean, would you please explain it in\n> > detail?\n> > Forgive my stupid..:-P\n>\n> A stupid person couldn't have sent this patch in the first place :-)\n>\n> The ViewFinder component performs rendering asynchronously. It receives\n> a libcamera buffer in render(), and is tasked to display it on the\n> screen. The render() function returns before the rendering is complete.\n> We thus need to tell the caller when the rendering is complete, to let\n> it reuse the buffer once ViewFinder doesn't need it anymore. That's what\n> renderComplete() is for.\n>\n> With the code above, you signal that the buffer has been rendered right\n> after calling update(). That's not true, rendering is still in progress\n> at that point. You can only give the buffer back to the caller with\n> renderComplete() once a new buffer has been received to be rendered.\n> That's why the current ViewFinder implementation has\n>\n> std::swap(buffer_, buffer);\n>\n> After this call, buffer contains the old buffer, and we then call\n>\n> if (buffer)\n> renderComplete(buffer);\n>\n> to give the old buffer back. We need to check here if buffer is null, as\n> when the first buffer is rendered, there's no old buffer to be given\n> back.\n>\nThanks for the detailed explanation. I totally misunderstood this portion\nbefore.\n\n\nBR,\nShow Liu\n\n>\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::updateFrame(unsigned char *buffer)\n> > > > > +{\n> > > > > + if(buffer) {\n> > > > > + yuvDataPtr = buffer;\n> > > > > + update();\n> > > > > + }\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::setFrameSize(int width, int height)\n> > > > > +{\n> > > > > + if(width > 0 && height > 0) {\n> > > > > + width_ = width;\n> > > > > + height_ = height;\n> > > > > + }\n> > > > > +}\n> > > > > +\n> > > > > +char *ViewFinderGL::selectFragmentShader(unsigned int format)\n> > > > > +{\n> > > > > + char *fsrc = nullptr;\n> > > > > +\n> > > > > + switch(format) {\n> > > > > + case libcamera::formats::NV12:\n> > > > > + horzSubSample_ = 2;\n> > > > > + vertSubSample_ = 2;\n> > > > > + fsrc = NV_2_planes_UV;\n> > > > > + break;\n> > > > > + case libcamera::formats::NV21:\n> > > > > + horzSubSample_ = 2;\n> > > > > + vertSubSample_ = 2;\n> > > > > + fsrc = NV_2_planes_VU;\n> > > > > + break;\n> > > > > + case libcamera::formats::NV16:\n> > > > > + horzSubSample_ = 2;\n> > > > > + vertSubSample_ = 1;\n> > > > > + fsrc = NV_2_planes_UV;\n> > > > > + break;\n> > > > > + case libcamera::formats::NV61:\n> > > > > + horzSubSample_ = 2;\n> > > > > + vertSubSample_ = 1;\n> > > > > + fsrc = NV_2_planes_VU;\n> > > > > + break;\n> > > > > + case libcamera::formats::NV24:\n> > > > > + horzSubSample_ = 1;\n> > > > > + vertSubSample_ = 1;\n> > > > > + fsrc = NV_2_planes_UV;\n> > > > > + break;\n> > > > > + case libcamera::formats::NV42:\n> > > > > + horzSubSample_ = 1;\n> > > > > + vertSubSample_ = 1;\n> > > > > + fsrc = NV_2_planes_VU;\n> > > > > + break;\n> > > > > + case libcamera::formats::YUV420:\n> > > > > + horzSubSample_ = 2;\n> > > > > + vertSubSample_ = 2;\n> > > > > + fsrc = NV_3_planes_UV;\n> > > > > + break;\n> > > > > + default:\n> > > > > + break;\n> > > > > + };\n> > > > > +\n> > > > > + if(fsrc == nullptr) {\n> > > > > + qDebug() << __func__ << \"[ERROR]:\" <<\" No suitable\n> fragment shader can be select.\";\n> > > > > + }\n> > > > > + return fsrc;\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::createFragmentShader()\n> > > > > +{\n> > > > > + bool bCompile;\n> > > > > +\n> > > > > + pFShader = new QOpenGLShader(QOpenGLShader::Fragment, this);\n> > > > > + char *fsrc = selectFragmentShader(format_);\n> > > > > +\n> > > > > + bCompile = pFShader->compileSourceCode(fsrc);\n> > > > > + if(!bCompile)\n> > > > > + {\n> > > > > + qDebug() << __func__ << \":\" << pFShader->log();\n> > > > > + }\n> > > > > +\n> > > > > + shaderProgram.addShader(pFShader);\n> > > > > +\n> > > > > + // Link shader pipeline\n> > > > > + if (!shaderProgram.link()) {\n> > > > > + qDebug() << __func__ << \": shader program link\n> failed.\\n\" << shaderProgram.log();\n> > > > > + close();\n> > > > > + }\n> > > > > +\n> > > > > + // Bind shader pipeline for use\n> > > > > + if (!shaderProgram.bind()) {\n> > > > > + qDebug() << __func__ << \": shader program binding\n> failed.\\n\" << shaderProgram.log();\n> > > > > + close();\n> > > > > + }\n> > > > > +\n> > > > > + shaderProgram.enableAttributeArray(ATTRIB_VERTEX);\n> > > > > + shaderProgram.enableAttributeArray(ATTRIB_TEXTURE);\n> > > > > +\n> > > > > +\n> shaderProgram.setAttributeBuffer(ATTRIB_VERTEX,GL_FLOAT,0,2,2*sizeof(GLfloat));\n> > > > > +\n> shaderProgram.setAttributeBuffer(ATTRIB_TEXTURE,GL_FLOAT,8*sizeof(GLfloat),2,2*sizeof(GLfloat));\n> > > > > +\n> > > > > + textureUniformY = shaderProgram.uniformLocation(\"tex_y\");\n> > > > > + textureUniformU = shaderProgram.uniformLocation(\"tex_u\");\n> > > > > + textureUniformV = shaderProgram.uniformLocation(\"tex_v\");\n> > > > > +\n> > > > > + if(!textureY.isCreated())\n> > > > > + textureY.create();\n> > > > > +\n> > > > > + if(!textureU.isCreated())\n> > > > > + textureU.create();\n> > > > > +\n> > > > > + if(!textureV.isCreated())\n> > > > > + textureV.create();\n> > > > > +\n> > > > > + id_y = textureY.textureId();\n> > > > > + id_u = textureU.textureId();\n> > > > > + id_v = textureV.textureId();\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::configureTexture(unsigned int id)\n> > > > > +{\n> > > > > + glBindTexture(GL_TEXTURE_2D, id);\n> > > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER,\n> GL_LINEAR);\n> > > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER,\n> GL_LINEAR);\n> > > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S,\n> GL_CLAMP_TO_EDGE);\n> > > > > + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T,\n> GL_CLAMP_TO_EDGE);\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::removeShader()\n> > > > > +{\n> > > > > + if (shaderProgram.isLinked()) {\n> > > > > + shaderProgram.release();\n> > > > > + shaderProgram.removeAllShaders();\n> > > > > + }\n> > > > > +\n> > > > > + if(pFShader)\n> > > > > + delete pFShader;\n> > > > > +\n> > > > > + if(pVShader)\n> > > > > + delete pVShader;\n> > >\n> > > If we stop and restart the stream with a different format, the previous\n> > > shader will be used, as removeShader() isn't called. I don't think\n> > > that's right. I believe you need to call removeShader() at the\n> beginning\n> > > of setFormat().\n> >\n> > Yes. you're right. I will refine the setFormat() function.\n> >\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::initializeGL()\n> > > > > +{\n> > > > > + bool bCompile;\n> > > > > +\n> > > > > + initializeOpenGLFunctions();\n> > > > > + glEnable(GL_TEXTURE_2D);\n> > > > > + glDisable(GL_DEPTH_TEST);\n> > > > > +\n> > > > > + static const GLfloat vertices[] {\n> > > > > + -1.0f,-1.0f,\n> > > > > + -1.0f,+1.0f,\n> > > > > + +1.0f,+1.0f,\n> > > > > + +1.0f,-1.0f,\n> > > > > + 0.0f,1.0f,\n> > > > > + 0.0f,0.0f,\n> > > > > + 1.0f,0.0f,\n> > > > > + 1.0f,1.0f,\n> > > > > + };\n> > > > > +\n> > > > > + glBuffer.create();\n> > > > > + glBuffer.bind();\n> > > > > + glBuffer.allocate(vertices,sizeof(vertices));\n> > > > > +\n> > > > > + /* Create Vertex Shader */\n> > > > > + pVShader = new QOpenGLShader(QOpenGLShader::Vertex, this);\n> > > > > + char *vsrc = NV_Vertex_shader;\n> > > > > +\n> > > > > + bCompile = pVShader->compileSourceCode(vsrc);\n> > > > > + if(!bCompile) {\n> > > > > + qDebug() << __func__<< \":\" << pVShader->log();\n> > > > > + }\n> > > > > +\n> > > > > + shaderProgram.addShader(pVShader);\n> > > > > +\n> > > > > + glClearColor(1.0f, 1.0f, 1.0f, 0.0f);\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::render()\n> > > > > +{\n> > > > > + switch(format_) {\n> > > > > + case libcamera::formats::NV12:\n> > > > > + case libcamera::formats::NV21:\n> > > > > + case libcamera::formats::NV16:\n> > > > > + case libcamera::formats::NV61:\n> > > > > + case libcamera::formats::NV24:\n> > > > > + case libcamera::formats::NV42:\n> > > > > + /* activate texture 0 */\n> > > > > + glActiveTexture(GL_TEXTURE0);\n> > > > > + configureTexture(id_y);\n> > > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(),\n> size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > > > > + glUniform1i(textureUniformY, 0);\n> > > > > +\n> > > > > + /* activate texture 1 */\n> > > > > + glActiveTexture(GL_TEXTURE1);\n> > > > > + configureTexture(id_u);\n> > > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > > > > + glUniform1i(textureUniformU, 1);\n> > > > > + break;\n> > > > > + case libcamera::formats::YUV420:\n> > > > > + /* activate texture 0 */\n> > > > > + glActiveTexture(GL_TEXTURE0);\n> > > > > + configureTexture(id_y);\n> > > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RED, size_.width(),\n> size_.height(), 0, GL_RED, GL_UNSIGNED_BYTE, yuvDataPtr);\n> > > > > + glUniform1i(textureUniformY, 0);\n> > > > > +\n> > > > > + /* activate texture 1 */\n> > > > > + glActiveTexture(GL_TEXTURE1);\n> > > > > + configureTexture(id_u);\n> > > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height());\n> > > > > + glUniform1i(textureUniformU, 1);\n> > > > > +\n> > > > > + /* activate texture 2 */\n> > > > > + glActiveTexture(GL_TEXTURE2);\n> > > > > + configureTexture(id_v);\n> > > > > + glTexImage2D(GL_TEXTURE_2D, 0, GL_RG,\n> size_.width()/horzSubSample_, size_.height()/vertSubSample_, 0, GL_RG,\n> GL_UNSIGNED_BYTE, (char*)yuvDataPtr+size_.width()*size_.height()*5/4);\n> > > > > + glUniform1i(textureUniformV, 2);\n> > > > > + break;\n> > > > > + default:\n> > > > > + break;\n> > > > > + };\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::paintGL()\n> > > > > +{\n> > > > > + if(pFShader == nullptr)\n> > >\n> > > We tend to write\n> > >\n> > > if (!pFShader)\n> >\n> > ok will fix it.\n> >\n> > > > > + createFragmentShader();\n> > >\n> > > I wonder if we could do this in setFormat().\n> >\n> > That should be ok, I will try to move this when setting Format.\n> >\n> > > > > +\n> > > > > + if(yuvDataPtr)\n> > > > > + {\n> > > > > + glClearColor(0.0, 0.0, 0.0, 1.0);\n> > > > > + glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );\n> > > > > +\n> > > > > + render();\n> > > > > + glDrawArrays(GL_TRIANGLE_FAN, 0, 4);\n> > > > > + }\n> > > > > +}\n> > > > > +\n> > > > > +void ViewFinderGL::resizeGL(int w, int h)\n> > > > > +{\n> > > > > + glViewport(0,0,w,h);\n> > > > > +}\n> > > > > +\n> > > > > +QSize ViewFinderGL::sizeHint() const\n> > > > > +{\n> > > > > + return size_.isValid() ? size_ : QSize(640, 480);\n> > > > > +}\n> > > > > diff --git a/src/qcam/viewfinderGL.h b/src/qcam/viewfinderGL.h\n> > > > > new file mode 100644\n> > > > > index 0000000..08662ca\n> > > > > --- /dev/null\n> > > > > +++ b/src/qcam/viewfinderGL.h\n> > > > > @@ -0,0 +1,101 @@\n> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > > +/*\n> > > > > + * Copyright (C) 2020, Linaro\n> > > > > + *\n> > > > > + * viewfinderGL.h - Render YUV format frame by OpenGL shader\n> > > > > + */\n> > > > > +#ifndef __VIEWFINDERGL_H__\n> > > > > +#define __VIEWFINDERGL_H__\n> > > > > +\n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +\n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +\n> > >\n> > > Do you need all these headers in the .h file ? They seem to belong to\n> > > the .cpp file.\n> >\n> > OK I will check or just remove it.\n> >\n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +\n> > > > > +#include \n> > > > > +#include \n> > >\n> > > Blank line missing here.\n> >\n> > sure, will do .\n> >\n> > > > > +#include \"viewfinder.h\"\n> > > > > +\n> > > > > +class ViewFinderGL : public QOpenGLWidget,\n> > > > > + public ViewFinderHandler,\n> > > > > + protected QOpenGLFunctions\n> > > > > +{\n> > > > > + Q_OBJECT\n> > > > > +\n> > > > > +public:\n> > > > > + ViewFinderGL(QWidget *parent = 0);\n> > > > > + ~ViewFinderGL();\n> > > > > +\n> > > > > + int setFormat(const libcamera::PixelFormat &format, const\n> QSize &size);\n> > >\n> > > You should add \"override\" to qualify all overridden functions.\n> > >\n> > > int setFormat(const libcamera::PixelFormat &format,\n> > > const QSize &size) override;\n> >\n> > ok will do.\n> >\n> > > > > + void render(libcamera::FrameBuffer *buffer, MappedBuffer *map);\n> > > > > + void stop();\n> > > > > +\n> > > > > + QImage getCurrentImage();\n> > > > > +\n> > > > > + void setFrameSize(int width, int height);\n> > > > > + void updateFrame(unsigned char *buffer);\n> > >\n> > > Those two functions seem unused.\n> >\n> > sure, I will remove it.\n> >\n> > > > > +\n> > > > > + char *selectFragmentShader(unsigned int format);\n> > > > > + void createFragmentShader();\n> > >\n> > > And these two functions can be private.\n> >\n> > will move to private.\n> >\n> > > > > + void render();\n> > > > > +\n> > > > > +Q_SIGNALS:\n> > > > > + void renderComplete(libcamera::FrameBuffer *buffer);\n> > > > > +\n> > > > > +protected:\n> > > > > + void initializeGL() Q_DECL_OVERRIDE;\n> > >\n> > > You can use \"override\" directly, we know the compiler supports it.\n> >\n> > ok, I will fix it up.\n> >\n> > > > > + void paintGL() Q_DECL_OVERRIDE;\n> > > > > + void resizeGL(int w, int h) Q_DECL_OVERRIDE;\n> > > > > + QSize sizeHint() const override;\n> > > > > +\n> > > > > +private:\n> > > > > +\n> > > > > + void configureTexture(unsigned int id);\n> > > > > + void removeShader();\n> > > > > +\n> > > > > + /* Captured image size, format and buffer */\n> > > > > + libcamera::FrameBuffer *buffer_;\n> > > > > + libcamera::PixelFormat format_;\n> > > > > + QOpenGLBuffer glBuffer;\n> > > > > + QSize size_;\n> > > > > +\n> > > > > + GLuint id_u;\n> > > > > + GLuint id_v;\n> > > > > + GLuint id_y;\n> > > > > +\n> > > > > + QMutex mutex_; /* Prevent concurrent access to image_ */\n> > > > > +\n> > > > > + QOpenGLShader *pFShader;\n> > > > > + QOpenGLShader *pVShader;\n> > > > > + QOpenGLShaderProgram shaderProgram;\n> > > > > +\n> > > > > + GLuint textureUniformU;\n> > > > > + GLuint textureUniformV;\n> > > > > + GLuint textureUniformY;\n> > > > > +\n> > > > > + QOpenGLTexture textureU;\n> > > > > + QOpenGLTexture textureV;\n> > > > > + QOpenGLTexture textureY;\n> > > > > +\n> > > > > + unsigned int width_;\n> > > > > + unsigned int height_;\n> > > > > +\n> > > > > + unsigned char* yuvDataPtr;\n> > > > > +\n> > > > > + /* NV parameters */\n> > > > > + unsigned int horzSubSample_ ;\n> > > > > + unsigned int vertSubSample_;\n> > > > > +};\n> > > > > +#endif // __VIEWFINDERGL_H__\n>\n> --\n> Regards,\n>\n> Laurent Pinchart\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 57ED5BD792\n\tfor ;\n\tMon, 6 Jul 2020 08:06:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id DB71D60E09;\n\tMon, 6 Jul 2020 10:06:19 +0200 (CEST)","from mail-pg1-x531.google.com (mail-pg1-x531.google.com\n\t[IPv6:2607:f8b0:4864:20::531])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EE2DF603B1\n\tfor ;\n\tMon, 6 Jul 2020 10:06:17 +0200 (CEST)","by mail-pg1-x531.google.com with SMTP id k27so1939478pgm.2\n\tfor ;\n\tMon, 06 Jul 2020 01:06:17 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"q8abbQMk\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=4xYMpr1yLrdJa/RFBEYCqH1p5OOjHMqB3t8nchY0dIs=;\n\tb=q8abbQMkCX0MoONUJMk723gLIqodTH5HXqea+QPM5fcKpZjfiBRp4O9ligvAy2F+0v\n\tMq751RM4cCwddIusfhLXH/NUv55O1Xyz06vPpaimeA9BKc6PXv8cOihLJcIicKzvkJ/I\n\tUaGUU5OWhkNO4h0fEjzSQZ++IJYRrsQyCdY6ZRcnVlvEZm1YVRbT/xkOlEkS00qrWbjE\n\tBQ90RP3RONWJdqvXXVcNaemN7mCQPvZbr+jwYSBTWwjHgv0A3mVIUKKS4L3guR7Ww6RJ\n\taQUoWnoFHnxSXvKX0nZqCEHemyqewweSISRomBMtYvXiO73uttk6L0l0NYURhWmt6zE6\n\tF1HQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=4xYMpr1yLrdJa/RFBEYCqH1p5OOjHMqB3t8nchY0dIs=;\n\tb=BMhKssYVzsnTPZp6xTYa2pUO32GeZFHV0MAhTSVxCbbDL61hPSsbzVq9Xa/VVLJE05\n\t+RfLDrWUEfPaOeikznubF5oYJz9VF6gqM9FP8fjyGoixvAA31aSy2lWzAvtw24mWKINM\n\trV4sFznzSSLvUh+Y9+/fpUzwbnrUB2J++SO/8NG5Ou6ZmOFmRprtlI1nhKv9AWPddwNJ\n\tZ7FiUsIFw9W8ZeKjgH4NG6EAbjWwyQp6HONFVchck+jt7T1q/ToLAv1xouXz9ZwMShic\n\trkisjdLEFSQSB8obDZ3tlYRePyoukAQU0/gjai8IXU0OL71e0TdJIpagFzCcDt0oNWY0\n\tcyGA==","X-Gm-Message-State":"AOAM5301yyBfqCdkwU3rn0j6by5k6sHPqE7VZXV5b+7HAdXknCdEarZt\n\t5mJXwgYPYulZghIxiNwrhy8y9jeDYBW42T0lDRC35oY3","X-Google-Smtp-Source":"ABdhPJyMJUYMSPo4grwLp6Nc8KPkXlzU/vmg0EAe5mTQDKa9zG+kUxHcVZXVMrPWOVNiEiXYyp4b3GClEDj/7St4VJs=","X-Received":"by 2002:a05:6a00:1507:: with SMTP id\n\tq7mr41289401pfu.131.1594022775943; \n\tMon, 06 Jul 2020 01:06:15 -0700 (PDT)","MIME-Version":"1.0","References":"<20200624073705.14737-1-show.liu@linaro.org>\n\t<20200624073705.14737-2-show.liu@linaro.org>\n\t<20200630150343.GA2259976@oden.dyn.berto.se>\n\t<20200630162624.GM5850@pendragon.ideasonboard.com>\n\t\n\t<20200703101405.GA5963@pendragon.ideasonboard.com>","In-Reply-To":"<20200703101405.GA5963@pendragon.ideasonboard.com>","From":"Show Liu ","Date":"Mon, 6 Jul 2020 16:06:04 +0800","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [PATCH v3 1/1] qcam: Render YUV formats frame\n\tby OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============2876354417246489212==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]