[{"id":12102,"web_url":"https://patchwork.libcamera.org/comment/12102/","msgid":"<20200824001110.GR6002@pendragon.ideasonboard.com>","date":"2020-08-24T00:11:10","subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Show,\n\nOn Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:\n> Hi,\n> \n> This is the version 4 patchset for qcam to accelerate the YUV format conversion\n> by OpenGL shader. Most of the modifications are according to the comments\n> and suggestions from V3. \n> And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface\n> instead of the QOpenGLWidget, the main reason that use dynamic_cast\n> to get the viewfinder_ is really weird.\n> In this version, OpenGL rendering enabled by --render=gles,\n> and if any error occurs during OpenGL configuration, it will return to use QT rendering.\n> But compared with the previous version, this version performance dropped a lot\n> due to I using the toImage() function to get the image when rendering complete,\n> it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800).\n> I am still trying to improve this performance issue. Any suggestions are welcome.\n\nIsn't it expected though if you render to an off-screen surface and then\nconvert it to a QImage ? What was the issue with rendering directly on\nthe screen with QOpenGLWidget ? Just the dynamic cast ?\n\n> Show Liu (3):\n> qcam: add OpenGL shader code as QT resource\n> qcam: add OpenGL renderer\n> qcam: use the OpenGL renderer as NV family YUV format converter\n> \n> src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++\n> src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++\n> src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++\n> src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++\n> src/qcam/assets/shader/NV_vertex_shader.glsl | 16 +\n> src/qcam/assets/shader/shaders.qrc | 10 +\n> src/qcam/main.cpp | 3 +\n> src/qcam/main_window.cpp | 2 +\n> src/qcam/main_window.h | 1 +\n> src/qcam/meson.build | 3 +\n> src/qcam/renderer.cpp | 346 +++++++++++++++++++\n> src/qcam/renderer.h | 81 +++++\n> src/qcam/viewfinder.cpp | 46 ++-\n> src/qcam/viewfinder.h | 10 +\n> 14 files changed, 640 insertions(+), 8 deletions(-)\n> create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl\n> create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl\n> create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl\n> create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl\n> create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl\n> create mode 100644 src/qcam/assets/shader/shaders.qrc\n> create mode 100644 src/qcam/renderer.cpp\n> create mode 100644 src/qcam/renderer.h","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1BCFEBE173\n\tfor ;\n\tMon, 24 Aug 2020 00:11:32 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 91B64627B5;\n\tMon, 24 Aug 2020 02:11:31 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 77F5C60381\n\tfor ;\n\tMon, 24 Aug 2020 02:11:29 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DBBD7279;\n\tMon, 24 Aug 2020 02:11:28 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"pKsdVAvZ\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1598227889;\n\tbh=mBryFp8Iktv6634j/tdyb/p7tbx6ZPMgGAL0/vvGD1M=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=pKsdVAvZuYJt8PvYfAeUwi6ACvJPmCGfRG8urDy1wBsLrEhbss8XD/GV9iMPAeM6M\n\tbVyuj9WGeTaB2eLEYydCK31+URuowQ64M2jEyPO04ru1vmPtJ/NTghczRm12t4jQ7O\n\t/Mb5b9WmRQO2obpeFofOTMDXi2g2yGEc0KKHA2SQ=","Date":"Mon, 24 Aug 2020 03:11:10 +0300","From":"Laurent Pinchart ","To":"Show Liu ","Message-ID":"<20200824001110.GR6002@pendragon.ideasonboard.com>","References":"<20200821161602.5093-1-show.liu@linaro.org>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20200821161602.5093-1-show.liu@linaro.org>","Subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":12144,"web_url":"https://patchwork.libcamera.org/comment/12144/","msgid":"","date":"2020-08-25T09:21:38","subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","submitter":{"id":24,"url":"https://patchwork.libcamera.org/api/people/24/","name":"Show Liu","email":"show.liu@linaro.org"},"content":"Hi Laurent,\n\nThanks for your review.\n\n\n\nOn Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart <\nlaurent.pinchart@ideasonboard.com> wrote:\n\n> Hi Show,\n>\n> On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:\n> > Hi,\n> >\n> > This is the version 4 patchset for qcam to accelerate the YUV format\n> conversion\n> > by OpenGL shader. Most of the modifications are according to the comments\n> > and suggestions from V3.\n> > And the big change is using the QOpenGLFramebufferObject with\n> QOffscreenSurface\n> > instead of the QOpenGLWidget, the main reason that use dynamic_cast\n> > to get the viewfinder_ is really weird.\n> > In this version, OpenGL rendering enabled by --render=gles,\n> > and if any error occurs during OpenGL configuration, it will return to\n> use QT rendering.\n> > But compared with the previous version, this version performance dropped\n> a lot\n> > due to I using the toImage() function to get the image when rendering\n> complete,\n> > it's really an inefficient operation. the framerate down to 16.x ~ 18.x\n> fps(1280 x 800).\n> > I am still trying to improve this performance issue. Any suggestions are\n> welcome.\n>\n> Isn't it expected though if you render to an off-screen surface and then\n> convert it to a QImage ? What was the issue with rendering directly on\n> the screen with QOpenGLWidget ? Just the dynamic cast ?\n>\n\nSome other reasons are like:\n1. The viewfinder and viewfinderGL are very similar, too many parts are\nduplicate.\n2. My idea is that if viewfinder is able to handle both Qt and OpenGL\nrendering would be perfect.\n The original processing flow when viewfinder got a new frame from\ncamera is like below\n Frame -> rendering -> call update() -> paintEvent( painting)\n But in QOpenGLWidget way, the processing flow become\n Frame -> call update() -> rendering and painting in paintGL()\n That's why I try to render into off-screen to fit the original\nviewfinder process flow.\n\nBUT honestly it seems not a good idea to render to an external off screen\nsurface\nthen convert to QImage due to the performance issue. I am trying to find\nsome way\nto paint the QOpenGLFramebufferObject directly.\n\nOR just letting the viewfinder inherit from QOpenGLWidget\nand QOpenGLFunctions\nand make it able to handle Qt and OpenGL rendering both.\nI am still trying to figure out a good solution for that.\n\n\n\nBest Regards,\nShow Liu\n\n> > Show Liu (3):\n> > qcam: add OpenGL shader code as QT resource\n> > qcam: add OpenGL renderer\n> > qcam: use the OpenGL renderer as NV family YUV format converter\n> >\n> > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++\n> > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++\n> > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++\n> > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++\n> > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 +\n> > src/qcam/assets/shader/shaders.qrc | 10 +\n> > src/qcam/main.cpp | 3 +\n> > src/qcam/main_window.cpp | 2 +\n> > src/qcam/main_window.h | 1 +\n> > src/qcam/meson.build | 3 +\n> > src/qcam/renderer.cpp | 346 +++++++++++++++++++\n> > src/qcam/renderer.h | 81 +++++\n> > src/qcam/viewfinder.cpp | 46 ++-\n> > src/qcam/viewfinder.h | 10 +\n> > 14 files changed, 640 insertions(+), 8 deletions(-)\n> > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl\n> > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl\n> > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl\n> > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl\n> > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl\n> > create mode 100644 src/qcam/assets/shader/shaders.qrc\n> > create mode 100644 src/qcam/renderer.cpp\n> > create mode 100644 src/qcam/renderer.h\n>\n> --\n> Regards,\n>\n> Laurent Pinchart\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 81D91BE173\n\tfor ;\n\tTue, 25 Aug 2020 09:21:53 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0F92D62849;\n\tTue, 25 Aug 2020 11:21:53 +0200 (CEST)","from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com\n\t[IPv6:2607:f8b0:4864:20::1043])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4587E6037B\n\tfor ;\n\tTue, 25 Aug 2020 11:21:51 +0200 (CEST)","by mail-pj1-x1043.google.com with SMTP id i13so930933pjv.0\n\tfor ;\n\tTue, 25 Aug 2020 02:21:51 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"yDVByR7L\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=/kTEGGhnplpPYEFy/Q93cuFsn4rMPAw9/ashPQ3N5bw=;\n\tb=yDVByR7L0mYImWIAjw3yDqUBGwbt7g52HJtgdT64CQ+R96+qvaiCTVJjOc+ZXQNxLD\n\tbVDX3oic6QJgbW0DboPd2bg3Ht5/ml+F+zT/OLaPWctQIUtrlUjlljTxhiRerGkAtVx/\n\tfwAZJGNwE+Bx9Zcx4h8KaY2sGKtEFHjvyhrBwX4t0senyMMaiBz0vThl8xd3siKA/Zqq\n\tVLjc7LpJWIA2a0QKZLnCFTXht+27OFGflZ67KfzaJNb+aEAe1vVMiJOfT0bapxcynzmY\n\t/K/N0YCrSPs46LOJ6LvnquRnKC247krd1clF8xSqKQ72HMpirN4j3PgCoV0lD8RyL76K\n\t7eRQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=/kTEGGhnplpPYEFy/Q93cuFsn4rMPAw9/ashPQ3N5bw=;\n\tb=HCXHBd6VBLkJk+K2pccZGlqTbLHeBc+PalCYltSiX897jsgbTOqZ0PRFpvRv8ijjZC\n\tQ7mofVgMz0KTGu7g/VNCOTfrwlQ3tp37AfochVRfDR755e3IeevSG6GWPcIO3eKnIY1H\n\tnC35X/syIcuemDyYm7hJ9XJPlkx5E73WLkAXs9f4CxVveYecnjBPVmWxFZKjR/Ip0rA8\n\trT4Kz8EEvQLLvRK/oN7mwVm5zeHTUhROBpkgJ8FkZdNkkFrsta2qJh1SyFewF+/ha+GU\n\tpxK+uZq0SnKk4iUzEEwIhNr7+JTMoXw9vBSIOaa7kWLR/kaEr+DB3AuLbbm1LukY2H4S\n\t04/Q==","X-Gm-Message-State":"AOAM530WqGo8QG+HMuGi+P6rZuD4bq3I/YzOQZl2GnVBZmo7xRqIRfK3\n\t9p8es71DMwdyVPt/kFzL9JBckNFdJLE1mvLgFcJBvzIfGrYGXw==","X-Google-Smtp-Source":"ABdhPJwP8Z6vyvvJNcX7B8VE7KXshCU+LoUw3i5m95m2lAs/FfUhSLtBTl8/ngtBbMbJyPj7EvWibKsP6DRIMnY9SVg=","X-Received":"by 2002:a17:90b:a54:: with SMTP id\n\tgw20mr798814pjb.183.1598347309251; \n\tTue, 25 Aug 2020 02:21:49 -0700 (PDT)","MIME-Version":"1.0","References":"<20200821161602.5093-1-show.liu@linaro.org>\n\t<20200824001110.GR6002@pendragon.ideasonboard.com>","In-Reply-To":"<20200824001110.GR6002@pendragon.ideasonboard.com>","From":"Show Liu ","Date":"Tue, 25 Aug 2020 17:21:38 +0800","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============3679520991015166800==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":12146,"web_url":"https://patchwork.libcamera.org/comment/12146/","msgid":"<20200825190617.GI6767@pendragon.ideasonboard.com>","date":"2020-08-25T19:06:17","subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Show,\n\nOn Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote:\n> Hi Laurent,\n> \n> Thanks for your review.\n> \n> On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote:\n> > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:\n> > > Hi,\n> > >\n> > > This is the version 4 patchset for qcam to accelerate the YUV format conversion\n> > > by OpenGL shader. Most of the modifications are according to the comments\n> > > and suggestions from V3.\n> > > And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface\n> > > instead of the QOpenGLWidget, the main reason that use dynamic_cast\n> > > to get the viewfinder_ is really weird.\n> > > In this version, OpenGL rendering enabled by --render=gles,\n> > > and if any error occurs during OpenGL configuration, it will return to use QT rendering.\n> > > But compared with the previous version, this version performance dropped a lot\n> > > due to I using the toImage() function to get the image when rendering complete,\n> > > it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800).\n> > > I am still trying to improve this performance issue. Any suggestions are welcome.\n> >\n> > Isn't it expected though if you render to an off-screen surface and then\n> > convert it to a QImage ? What was the issue with rendering directly on\n> > the screen with QOpenGLWidget ? Just the dynamic cast ?\n> \n> Some other reasons are like:\n> 1. The viewfinder and viewfinderGL are very similar, too many parts are\n> duplicate.\n> 2. My idea is that if viewfinder is able to handle both Qt and OpenGL\n> rendering would be perfect.\n> The original processing flow when viewfinder got a new frame from\n> camera is like below\n> Frame -> rendering -> call update() -> paintEvent( painting)\n> But in QOpenGLWidget way, the processing flow become\n> Frame -> call update() -> rendering and painting in paintGL()\n> That's why I try to render into off-screen to fit the original\n> viewfinder process flow.\n> \n> BUT honestly it seems not a good idea to render to an external off screen surface\n> then convert to QImage due to the performance issue. I am trying to find some way\n> to paint the QOpenGLFramebufferObject directly.\n> \n> OR just letting the viewfinder inherit from QOpenGLWidgetand QOpenGLFunctions\n> and make it able to handle Qt and OpenGL rendering both.\n> I am still trying to figure out a good solution for that.\n\nHow about starting with a ViewfinderGL implementation that duplicates\ncode from the existing Viewfinder class, to get the feature merged, and\nthen reworking both classes on top to reduce code duplication ? I can\nhelp with the latter.\n\n> > > Show Liu (3):\n> > > qcam: add OpenGL shader code as QT resource\n> > > qcam: add OpenGL renderer\n> > > qcam: use the OpenGL renderer as NV family YUV format converter\n> > >\n> > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++\n> > > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++\n> > > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++\n> > > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++\n> > > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 +\n> > > src/qcam/assets/shader/shaders.qrc | 10 +\n> > > src/qcam/main.cpp | 3 +\n> > > src/qcam/main_window.cpp | 2 +\n> > > src/qcam/main_window.h | 1 +\n> > > src/qcam/meson.build | 3 +\n> > > src/qcam/renderer.cpp | 346 +++++++++++++++++++\n> > > src/qcam/renderer.h | 81 +++++\n> > > src/qcam/viewfinder.cpp | 46 ++-\n> > > src/qcam/viewfinder.h | 10 +\n> > > 14 files changed, 640 insertions(+), 8 deletions(-)\n> > > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl\n> > > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl\n> > > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl\n> > > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl\n> > > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl\n> > > create mode 100644 src/qcam/assets/shader/shaders.qrc\n> > > create mode 100644 src/qcam/renderer.cpp\n> > > create mode 100644 src/qcam/renderer.h","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C1067BD87E\n\tfor ;\n\tTue, 25 Aug 2020 19:06:38 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2882A62882;\n\tTue, 25 Aug 2020 21:06:38 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3212E6037C\n\tfor ;\n\tTue, 25 Aug 2020 21:06:37 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id AD2DF9A5;\n\tTue, 25 Aug 2020 21:06:36 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"H1G8Jd0f\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1598382396;\n\tbh=cSULHVWD/et/dULMhhpDvrlNf/Gwq9KglVDnk3IN1Bo=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=H1G8Jd0fAJ/Ct+/21kGv/gh41Rib1p93qpB7LeSI9jCCANYj3lIbEXE/OSu7UXDY9\n\tmf3J6Uf+kUB1cp/3KsM9T8k90IuA2Ge76Zl1cG/bDX+9Bkq4D14jLDMcxwibL27+ss\n\tB4oK2wVCKlMSch26O/S7xO/K4tZZBo4TkVGcgsdo=","Date":"Tue, 25 Aug 2020 22:06:17 +0300","From":"Laurent Pinchart ","To":"Show Liu ","Message-ID":"<20200825190617.GI6767@pendragon.ideasonboard.com>","References":"<20200821161602.5093-1-show.liu@linaro.org>\n\t<20200824001110.GR6002@pendragon.ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":12154,"web_url":"https://patchwork.libcamera.org/comment/12154/","msgid":"","date":"2020-08-26T06:25:43","subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","submitter":{"id":24,"url":"https://patchwork.libcamera.org/api/people/24/","name":"Show Liu","email":"show.liu@linaro.org"},"content":"Hi Laurent,\n\n\nOn Wed, Aug 26, 2020 at 3:06 AM Laurent Pinchart <\nlaurent.pinchart@ideasonboard.com> wrote:\n\n> Hi Show,\n>\n> On Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote:\n> > Hi Laurent,\n> >\n> > Thanks for your review.\n> >\n> > On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote:\n> > > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:\n> > > > Hi,\n> > > >\n> > > > This is the version 4 patchset for qcam to accelerate the YUV format\n> conversion\n> > > > by OpenGL shader. Most of the modifications are according to the\n> comments\n> > > > and suggestions from V3.\n> > > > And the big change is using the QOpenGLFramebufferObject with\n> QOffscreenSurface\n> > > > instead of the QOpenGLWidget, the main reason that use dynamic_cast\n> > > > to get the viewfinder_ is really weird.\n> > > > In this version, OpenGL rendering enabled by --render=gles,\n> > > > and if any error occurs during OpenGL configuration, it will return\n> to use QT rendering.\n> > > > But compared with the previous version, this version performance\n> dropped a lot\n> > > > due to I using the toImage() function to get the image when\n> rendering complete,\n> > > > it's really an inefficient operation. the framerate down to 16.x ~\n> 18.x fps(1280 x 800).\n> > > > I am still trying to improve this performance issue. Any suggestions\n> are welcome.\n> > >\n> > > Isn't it expected though if you render to an off-screen surface and\n> then\n> > > convert it to a QImage ? What was the issue with rendering directly on\n> > > the screen with QOpenGLWidget ? Just the dynamic cast ?\n> >\n> > Some other reasons are like:\n> > 1. The viewfinder and viewfinderGL are very similar, too many parts are\n> > duplicate.\n> > 2. My idea is that if viewfinder is able to handle both Qt and OpenGL\n> > rendering would be perfect.\n> > The original processing flow when viewfinder got a new frame from\n> > camera is like below\n> > Frame -> rendering -> call update() -> paintEvent( painting)\n> > But in QOpenGLWidget way, the processing flow become\n> > Frame -> call update() -> rendering and painting in paintGL()\n> > That's why I try to render into off-screen to fit the original\n> > viewfinder process flow.\n> >\n> > BUT honestly it seems not a good idea to render to an external off\n> screen surface\n> > then convert to QImage due to the performance issue. I am trying to find\n> some way\n> > to paint the QOpenGLFramebufferObject directly.\n> >\n> > OR just letting the viewfinder inherit from QOpenGLWidgetand\n> QOpenGLFunctions\n> > and make it able to handle Qt and OpenGL rendering both.\n> > I am still trying to figure out a good solution for that.\n>\n> How about starting with a ViewfinderGL implementation that duplicates\n> code from the existing Viewfinder class, to get the feature merged, and\n> then reworking both classes on top to reduce code duplication ? I can\n> help with the latter.\n>\n\nSounds Great.\nI will have the next version based on viewfinderGL ASAP.\n\nBest Regards,\nShow Liu\n\n>\n> > > > Show Liu (3):\n> > > > qcam: add OpenGL shader code as QT resource\n> > > > qcam: add OpenGL renderer\n> > > > qcam: use the OpenGL renderer as NV family YUV format converter\n> > > >\n> > > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++\n> > > > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++\n> > > > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++\n> > > > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++\n> > > > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 +\n> > > > src/qcam/assets/shader/shaders.qrc | 10 +\n> > > > src/qcam/main.cpp | 3 +\n> > > > src/qcam/main_window.cpp | 2 +\n> > > > src/qcam/main_window.h | 1 +\n> > > > src/qcam/meson.build | 3 +\n> > > > src/qcam/renderer.cpp | 346\n> +++++++++++++++++++\n> > > > src/qcam/renderer.h | 81 +++++\n> > > > src/qcam/viewfinder.cpp | 46 ++-\n> > > > src/qcam/viewfinder.h | 10 +\n> > > > 14 files changed, 640 insertions(+), 8 deletions(-)\n> > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl\n> > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl\n> > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl\n> > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl\n> > > > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl\n> > > > create mode 100644 src/qcam/assets/shader/shaders.qrc\n> > > > create mode 100644 src/qcam/renderer.cpp\n> > > > create mode 100644 src/qcam/renderer.h\n>\n> --\n> Regards,\n>\n> Laurent Pinchart\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2586CBE173\n\tfor ;\n\tWed, 26 Aug 2020 06:25:58 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 96E57628E6;\n\tWed, 26 Aug 2020 08:25:57 +0200 (CEST)","from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com\n\t[IPv6:2607:f8b0:4864:20::1043])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DAD0B61EA0\n\tfor ;\n\tWed, 26 Aug 2020 08:25:56 +0200 (CEST)","by mail-pj1-x1043.google.com with SMTP id q1so405327pjd.1\n\tfor ;\n\tTue, 25 Aug 2020 23:25:56 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"izlmvAzx\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=CqB7Fefq8DrCe0zDwJPNFd2slvAU41Yj4cB7G+EN2Ts=;\n\tb=izlmvAzxs+sbLCmJABODerx47kCg+vqMhCOnZPCtznWKZ220eM0s4fG4oN5k394nf6\n\tZPetnBwiNqmYzFfLJIAvi0MJaR4Ta59mDQjtATb9itrMr0GsCb4NNXwDEKUsEotUfrvv\n\tpVSvKmOxSiy/KdUWYzJEEuuwtwkMS0iHWWmb6GKGJwCtMYBD3XOqYpPJXiLm0MovhfrV\n\tKImQom8C9AERHe80qrwJbpSdbZRxx4hMXBY5W3vDOYvkuyf2DDAhJtrSqueqBK8Q8VXL\n\tVUdFbnd++7T37Q4ehItCnROK01QeHaoQkjj6mYtdXAq+KDNNusOcEy9CJRNivd77GyIJ\n\t5yIw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=CqB7Fefq8DrCe0zDwJPNFd2slvAU41Yj4cB7G+EN2Ts=;\n\tb=CchdQKQJ0Aj3WSdgFkk4lByhiPJvTckIJc7rjYpzrLnhniVbTQ7yDVoN6jA+gHmSbQ\n\tbFSWSTuZL7YSgsW196mZIUipQE6tr5ObM9wGAwaY4HEZ8g1vBZ/tLIVh+lreElKbyG2J\n\tbJBLscJgVGielIHlktTdd8bWPcM3w/RQVbyFvMvgPEge14M/JTBLz1tJDwqR9GL/+Pyu\n\tMk5X2VwjqHx6H8V+twXOIkCnzDvv7zteD4T8a1EcrcO9JEqtZYx30DUaNb+QLjPgsxkz\n\th9nwWDFO1LpBrdvsIKVTJ1X6dejzPhqzVOdS/0gvUD7UA1QQ/FsbCML+rjGrVHncVg/6\n\tM5HQ==","X-Gm-Message-State":"AOAM530+20q1bEy9L3fipGyhM8KHAsPBKuNLJxv553wVve+D7BeXWxTJ\n\tslx1xztKu+sY/KEjYcWT+p2xtlb6xJV6a95EoIb1Tw==","X-Google-Smtp-Source":"ABdhPJzahgR0CXs5cDsVRu9aygdHJtCNPhwF4o0cCQTCzTx3AEmSssf+JFDNO5odsa6OgO9B7rUqfdhbmJPKRhctfyc=","X-Received":"by 2002:a17:90a:520d:: with SMTP id\n\tv13mr4842668pjh.18.1598423154733; \n\tTue, 25 Aug 2020 23:25:54 -0700 (PDT)","MIME-Version":"1.0","References":"<20200821161602.5093-1-show.liu@linaro.org>\n\t<20200824001110.GR6002@pendragon.ideasonboard.com>\n\t\n\t<20200825190617.GI6767@pendragon.ideasonboard.com>","In-Reply-To":"<20200825190617.GI6767@pendragon.ideasonboard.com>","From":"Show Liu ","Date":"Wed, 26 Aug 2020 14:25:43 +0800","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format\n\tconversion by using OpenGL shader","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============1374812802213734876==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]