| Message ID | 20200821161602.5093-1-show.liu@linaro.org |
|---|---|
| Headers | show |
| Series |
|
| Related | show |
Hi Show, On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote: > Hi, > > This is the version 4 patchset for qcam to accelerate the YUV format conversion > by OpenGL shader. Most of the modifications are according to the comments > and suggestions from V3. > And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface > instead of the QOpenGLWidget, the main reason that use dynamic_cast > to get the viewfinder_ is really weird. > In this version, OpenGL rendering enabled by --render=gles, > and if any error occurs during OpenGL configuration, it will return to use QT rendering. > But compared with the previous version, this version performance dropped a lot > due to I using the toImage() function to get the image when rendering complete, > it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800). > I am still trying to improve this performance issue. Any suggestions are welcome. Isn't it expected though if you render to an off-screen surface and then convert it to a QImage ? What was the issue with rendering directly on the screen with QOpenGLWidget ? Just the dynamic cast ? > Show Liu (3): > qcam: add OpenGL shader code as QT resource > qcam: add OpenGL renderer > qcam: use the OpenGL renderer as NV family YUV format converter > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++ > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++ > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++ > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++ > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 + > src/qcam/assets/shader/shaders.qrc | 10 + > src/qcam/main.cpp | 3 + > src/qcam/main_window.cpp | 2 + > src/qcam/main_window.h | 1 + > src/qcam/meson.build | 3 + > src/qcam/renderer.cpp | 346 +++++++++++++++++++ > src/qcam/renderer.h | 81 +++++ > src/qcam/viewfinder.cpp | 46 ++- > src/qcam/viewfinder.h | 10 + > 14 files changed, 640 insertions(+), 8 deletions(-) > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl > create mode 100644 src/qcam/assets/shader/shaders.qrc > create mode 100644 src/qcam/renderer.cpp > create mode 100644 src/qcam/renderer.h
Hi Laurent, Thanks for your review. On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart < laurent.pinchart@ideasonboard.com> wrote: > Hi Show, > > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote: > > Hi, > > > > This is the version 4 patchset for qcam to accelerate the YUV format > conversion > > by OpenGL shader. Most of the modifications are according to the comments > > and suggestions from V3. > > And the big change is using the QOpenGLFramebufferObject with > QOffscreenSurface > > instead of the QOpenGLWidget, the main reason that use dynamic_cast > > to get the viewfinder_ is really weird. > > In this version, OpenGL rendering enabled by --render=gles, > > and if any error occurs during OpenGL configuration, it will return to > use QT rendering. > > But compared with the previous version, this version performance dropped > a lot > > due to I using the toImage() function to get the image when rendering > complete, > > it's really an inefficient operation. the framerate down to 16.x ~ 18.x > fps(1280 x 800). > > I am still trying to improve this performance issue. Any suggestions are > welcome. > > Isn't it expected though if you render to an off-screen surface and then > convert it to a QImage ? What was the issue with rendering directly on > the screen with QOpenGLWidget ? Just the dynamic cast ? > Some other reasons are like: 1. The viewfinder and viewfinderGL are very similar, too many parts are duplicate. 2. My idea is that if viewfinder is able to handle both Qt and OpenGL rendering would be perfect. The original processing flow when viewfinder got a new frame from camera is like below Frame -> rendering -> call update() -> paintEvent( painting) But in QOpenGLWidget way, the processing flow become Frame -> call update() -> rendering and painting in paintGL() That's why I try to render into off-screen to fit the original viewfinder process flow. BUT honestly it seems not a good idea to render to an external off screen surface then convert to QImage due to the performance issue. I am trying to find some way to paint the QOpenGLFramebufferObject directly. OR just letting the viewfinder inherit from QOpenGLWidget and QOpenGLFunctions and make it able to handle Qt and OpenGL rendering both. I am still trying to figure out a good solution for that. Best Regards, Show Liu > > Show Liu (3): > > qcam: add OpenGL shader code as QT resource > > qcam: add OpenGL renderer > > qcam: use the OpenGL renderer as NV family YUV format converter > > > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++ > > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++ > > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++ > > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++ > > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 + > > src/qcam/assets/shader/shaders.qrc | 10 + > > src/qcam/main.cpp | 3 + > > src/qcam/main_window.cpp | 2 + > > src/qcam/main_window.h | 1 + > > src/qcam/meson.build | 3 + > > src/qcam/renderer.cpp | 346 +++++++++++++++++++ > > src/qcam/renderer.h | 81 +++++ > > src/qcam/viewfinder.cpp | 46 ++- > > src/qcam/viewfinder.h | 10 + > > 14 files changed, 640 insertions(+), 8 deletions(-) > > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl > > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl > > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl > > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl > > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl > > create mode 100644 src/qcam/assets/shader/shaders.qrc > > create mode 100644 src/qcam/renderer.cpp > > create mode 100644 src/qcam/renderer.h > > -- > Regards, > > Laurent Pinchart >
Hi Show, On Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote: > Hi Laurent, > > Thanks for your review. > > On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote: > > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote: > > > Hi, > > > > > > This is the version 4 patchset for qcam to accelerate the YUV format conversion > > > by OpenGL shader. Most of the modifications are according to the comments > > > and suggestions from V3. > > > And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface > > > instead of the QOpenGLWidget, the main reason that use dynamic_cast > > > to get the viewfinder_ is really weird. > > > In this version, OpenGL rendering enabled by --render=gles, > > > and if any error occurs during OpenGL configuration, it will return to use QT rendering. > > > But compared with the previous version, this version performance dropped a lot > > > due to I using the toImage() function to get the image when rendering complete, > > > it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800). > > > I am still trying to improve this performance issue. Any suggestions are welcome. > > > > Isn't it expected though if you render to an off-screen surface and then > > convert it to a QImage ? What was the issue with rendering directly on > > the screen with QOpenGLWidget ? Just the dynamic cast ? > > Some other reasons are like: > 1. The viewfinder and viewfinderGL are very similar, too many parts are > duplicate. > 2. My idea is that if viewfinder is able to handle both Qt and OpenGL > rendering would be perfect. > The original processing flow when viewfinder got a new frame from > camera is like below > Frame -> rendering -> call update() -> paintEvent( painting) > But in QOpenGLWidget way, the processing flow become > Frame -> call update() -> rendering and painting in paintGL() > That's why I try to render into off-screen to fit the original > viewfinder process flow. > > BUT honestly it seems not a good idea to render to an external off screen surface > then convert to QImage due to the performance issue. I am trying to find some way > to paint the QOpenGLFramebufferObject directly. > > OR just letting the viewfinder inherit from QOpenGLWidgetand QOpenGLFunctions > and make it able to handle Qt and OpenGL rendering both. > I am still trying to figure out a good solution for that. How about starting with a ViewfinderGL implementation that duplicates code from the existing Viewfinder class, to get the feature merged, and then reworking both classes on top to reduce code duplication ? I can help with the latter. > > > Show Liu (3): > > > qcam: add OpenGL shader code as QT resource > > > qcam: add OpenGL renderer > > > qcam: use the OpenGL renderer as NV family YUV format converter > > > > > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++ > > > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++ > > > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++ > > > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++ > > > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 + > > > src/qcam/assets/shader/shaders.qrc | 10 + > > > src/qcam/main.cpp | 3 + > > > src/qcam/main_window.cpp | 2 + > > > src/qcam/main_window.h | 1 + > > > src/qcam/meson.build | 3 + > > > src/qcam/renderer.cpp | 346 +++++++++++++++++++ > > > src/qcam/renderer.h | 81 +++++ > > > src/qcam/viewfinder.cpp | 46 ++- > > > src/qcam/viewfinder.h | 10 + > > > 14 files changed, 640 insertions(+), 8 deletions(-) > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl > > > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl > > > create mode 100644 src/qcam/assets/shader/shaders.qrc > > > create mode 100644 src/qcam/renderer.cpp > > > create mode 100644 src/qcam/renderer.h
Hi Laurent, On Wed, Aug 26, 2020 at 3:06 AM Laurent Pinchart < laurent.pinchart@ideasonboard.com> wrote: > Hi Show, > > On Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote: > > Hi Laurent, > > > > Thanks for your review. > > > > On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote: > > > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote: > > > > Hi, > > > > > > > > This is the version 4 patchset for qcam to accelerate the YUV format > conversion > > > > by OpenGL shader. Most of the modifications are according to the > comments > > > > and suggestions from V3. > > > > And the big change is using the QOpenGLFramebufferObject with > QOffscreenSurface > > > > instead of the QOpenGLWidget, the main reason that use dynamic_cast > > > > to get the viewfinder_ is really weird. > > > > In this version, OpenGL rendering enabled by --render=gles, > > > > and if any error occurs during OpenGL configuration, it will return > to use QT rendering. > > > > But compared with the previous version, this version performance > dropped a lot > > > > due to I using the toImage() function to get the image when > rendering complete, > > > > it's really an inefficient operation. the framerate down to 16.x ~ > 18.x fps(1280 x 800). > > > > I am still trying to improve this performance issue. Any suggestions > are welcome. > > > > > > Isn't it expected though if you render to an off-screen surface and > then > > > convert it to a QImage ? What was the issue with rendering directly on > > > the screen with QOpenGLWidget ? Just the dynamic cast ? > > > > Some other reasons are like: > > 1. The viewfinder and viewfinderGL are very similar, too many parts are > > duplicate. > > 2. My idea is that if viewfinder is able to handle both Qt and OpenGL > > rendering would be perfect. > > The original processing flow when viewfinder got a new frame from > > camera is like below > > Frame -> rendering -> call update() -> paintEvent( painting) > > But in QOpenGLWidget way, the processing flow become > > Frame -> call update() -> rendering and painting in paintGL() > > That's why I try to render into off-screen to fit the original > > viewfinder process flow. > > > > BUT honestly it seems not a good idea to render to an external off > screen surface > > then convert to QImage due to the performance issue. I am trying to find > some way > > to paint the QOpenGLFramebufferObject directly. > > > > OR just letting the viewfinder inherit from QOpenGLWidgetand > QOpenGLFunctions > > and make it able to handle Qt and OpenGL rendering both. > > I am still trying to figure out a good solution for that. > > How about starting with a ViewfinderGL implementation that duplicates > code from the existing Viewfinder class, to get the feature merged, and > then reworking both classes on top to reduce code duplication ? I can > help with the latter. > Sounds Great. I will have the next version based on viewfinderGL ASAP. Best Regards, Show Liu > > > > > Show Liu (3): > > > > qcam: add OpenGL shader code as QT resource > > > > qcam: add OpenGL renderer > > > > qcam: use the OpenGL renderer as NV family YUV format converter > > > > > > > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 ++ > > > > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 ++ > > > > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++ > > > > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++ > > > > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 + > > > > src/qcam/assets/shader/shaders.qrc | 10 + > > > > src/qcam/main.cpp | 3 + > > > > src/qcam/main_window.cpp | 2 + > > > > src/qcam/main_window.h | 1 + > > > > src/qcam/meson.build | 3 + > > > > src/qcam/renderer.cpp | 346 > +++++++++++++++++++ > > > > src/qcam/renderer.h | 81 +++++ > > > > src/qcam/viewfinder.cpp | 46 ++- > > > > src/qcam/viewfinder.h | 10 + > > > > 14 files changed, 640 insertions(+), 8 deletions(-) > > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl > > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl > > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl > > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl > > > > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl > > > > create mode 100644 src/qcam/assets/shader/shaders.qrc > > > > create mode 100644 src/qcam/renderer.cpp > > > > create mode 100644 src/qcam/renderer.h > > -- > Regards, > > Laurent Pinchart >