[libcamera-devel,v4,0/3] qcam: NV family YUV format conversion by using OpenGL shader
mbox series

Message ID 20200821161602.5093-1-show.liu@linaro.org
Headers show
Series
  • qcam: NV family YUV format conversion by using OpenGL shader
Related show

Message

Show Liu Aug. 21, 2020, 4:15 p.m. UTC
Hi,

This is the version 4 patchset for qcam to accelerate the YUV format conversion
by OpenGL shader. Most of the modifications are according to the comments
and suggestions from V3. 
And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface
instead of the QOpenGLWidget, the main reason that use dynamic_cast
to get the viewfinder_ is really weird.
In this version, OpenGL rendering enabled by --render=gles,
and if any error occurs during OpenGL configuration, it will return to use QT rendering.
But compared with the previous version, this version performance dropped a lot
due to I using the toImage() function to get the image when rendering complete,
it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800).
I am still trying to improve this performance issue. Any suggestions are welcome.



BR,
Show Liu

Show Liu (3):
  qcam: add OpenGL shader code as QT resource
  qcam: add OpenGL renderer
  qcam: use the OpenGL renderer as NV family YUV format converter

 src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
 src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
 src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
 src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
 src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
 src/qcam/assets/shader/shaders.qrc           |  10 +
 src/qcam/main.cpp                            |   3 +
 src/qcam/main_window.cpp                     |   2 +
 src/qcam/main_window.h                       |   1 +
 src/qcam/meson.build                         |   3 +
 src/qcam/renderer.cpp                        | 346 +++++++++++++++++++
 src/qcam/renderer.h                          |  81 +++++
 src/qcam/viewfinder.cpp                      |  46 ++-
 src/qcam/viewfinder.h                        |  10 +
 14 files changed, 640 insertions(+), 8 deletions(-)
 create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
 create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
 create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
 create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
 create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
 create mode 100644 src/qcam/assets/shader/shaders.qrc
 create mode 100644 src/qcam/renderer.cpp
 create mode 100644 src/qcam/renderer.h

Comments

Laurent Pinchart Aug. 24, 2020, 12:11 a.m. UTC | #1
Hi Show,

On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:
> Hi,
> 
> This is the version 4 patchset for qcam to accelerate the YUV format conversion
> by OpenGL shader. Most of the modifications are according to the comments
> and suggestions from V3. 
> And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface
> instead of the QOpenGLWidget, the main reason that use dynamic_cast
> to get the viewfinder_ is really weird.
> In this version, OpenGL rendering enabled by --render=gles,
> and if any error occurs during OpenGL configuration, it will return to use QT rendering.
> But compared with the previous version, this version performance dropped a lot
> due to I using the toImage() function to get the image when rendering complete,
> it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800).
> I am still trying to improve this performance issue. Any suggestions are welcome.

Isn't it expected though if you render to an off-screen surface and then
convert it to a QImage ? What was the issue with rendering directly on
the screen with QOpenGLWidget ? Just the dynamic cast ?

> Show Liu (3):
>   qcam: add OpenGL shader code as QT resource
>   qcam: add OpenGL renderer
>   qcam: use the OpenGL renderer as NV family YUV format converter
> 
>  src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
>  src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
>  src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
>  src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
>  src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
>  src/qcam/assets/shader/shaders.qrc           |  10 +
>  src/qcam/main.cpp                            |   3 +
>  src/qcam/main_window.cpp                     |   2 +
>  src/qcam/main_window.h                       |   1 +
>  src/qcam/meson.build                         |   3 +
>  src/qcam/renderer.cpp                        | 346 +++++++++++++++++++
>  src/qcam/renderer.h                          |  81 +++++
>  src/qcam/viewfinder.cpp                      |  46 ++-
>  src/qcam/viewfinder.h                        |  10 +
>  14 files changed, 640 insertions(+), 8 deletions(-)
>  create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
>  create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
>  create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
>  create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
>  create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
>  create mode 100644 src/qcam/assets/shader/shaders.qrc
>  create mode 100644 src/qcam/renderer.cpp
>  create mode 100644 src/qcam/renderer.h
Show Liu Aug. 25, 2020, 9:21 a.m. UTC | #2
Hi Laurent,

Thanks for your review.



On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart <
laurent.pinchart@ideasonboard.com> wrote:

> Hi Show,
>
> On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:
> > Hi,
> >
> > This is the version 4 patchset for qcam to accelerate the YUV format
> conversion
> > by OpenGL shader. Most of the modifications are according to the comments
> > and suggestions from V3.
> > And the big change is using the QOpenGLFramebufferObject with
> QOffscreenSurface
> > instead of the QOpenGLWidget, the main reason that use dynamic_cast
> > to get the viewfinder_ is really weird.
> > In this version, OpenGL rendering enabled by --render=gles,
> > and if any error occurs during OpenGL configuration, it will return to
> use QT rendering.
> > But compared with the previous version, this version performance dropped
> a lot
> > due to I using the toImage() function to get the image when rendering
> complete,
> > it's really an inefficient operation. the framerate down to 16.x ~ 18.x
> fps(1280 x 800).
> > I am still trying to improve this performance issue. Any suggestions are
> welcome.
>
> Isn't it expected though if you render to an off-screen surface and then
> convert it to a QImage ? What was the issue with rendering directly on
> the screen with QOpenGLWidget ? Just the dynamic cast ?
>

Some other reasons are like:
1. The viewfinder and viewfinderGL are very similar, too many parts are
duplicate.
2. My idea is that if viewfinder is able to handle both Qt and OpenGL
rendering would be perfect.
    The original processing flow when viewfinder got a new frame from
camera is like below
          Frame -> rendering -> call update() -> paintEvent( painting)
    But in QOpenGLWidget way, the processing flow become
          Frame -> call update() -> rendering and painting in paintGL()
    That's why I try to render into off-screen to fit the original
viewfinder process flow.

BUT honestly it seems not a good idea to render to an external off screen
surface
then convert to QImage due to the performance issue. I am trying to find
some way
to paint the QOpenGLFramebufferObject directly.

OR just letting the viewfinder inherit from QOpenGLWidget
and QOpenGLFunctions
and make it able to handle Qt and OpenGL rendering both.
I am still trying to figure out a good solution for that.



Best Regards,
Show Liu

> > Show Liu (3):
> >   qcam: add OpenGL shader code as QT resource
> >   qcam: add OpenGL renderer
> >   qcam: use the OpenGL renderer as NV family YUV format converter
> >
> >  src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
> >  src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
> >  src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
> >  src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
> >  src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
> >  src/qcam/assets/shader/shaders.qrc           |  10 +
> >  src/qcam/main.cpp                            |   3 +
> >  src/qcam/main_window.cpp                     |   2 +
> >  src/qcam/main_window.h                       |   1 +
> >  src/qcam/meson.build                         |   3 +
> >  src/qcam/renderer.cpp                        | 346 +++++++++++++++++++
> >  src/qcam/renderer.h                          |  81 +++++
> >  src/qcam/viewfinder.cpp                      |  46 ++-
> >  src/qcam/viewfinder.h                        |  10 +
> >  14 files changed, 640 insertions(+), 8 deletions(-)
> >  create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> >  create mode 100644 src/qcam/assets/shader/shaders.qrc
> >  create mode 100644 src/qcam/renderer.cpp
> >  create mode 100644 src/qcam/renderer.h
>
> --
> Regards,
>
> Laurent Pinchart
>
Laurent Pinchart Aug. 25, 2020, 7:06 p.m. UTC | #3
Hi Show,

On Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote:
> Hi Laurent,
> 
> Thanks for your review.
> 
> On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote:
> > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:
> > > Hi,
> > >
> > > This is the version 4 patchset for qcam to accelerate the YUV format conversion
> > > by OpenGL shader. Most of the modifications are according to the comments
> > > and suggestions from V3.
> > > And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface
> > > instead of the QOpenGLWidget, the main reason that use dynamic_cast
> > > to get the viewfinder_ is really weird.
> > > In this version, OpenGL rendering enabled by --render=gles,
> > > and if any error occurs during OpenGL configuration, it will return to use QT rendering.
> > > But compared with the previous version, this version performance dropped a lot
> > > due to I using the toImage() function to get the image when rendering complete,
> > > it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800).
> > > I am still trying to improve this performance issue. Any suggestions are welcome.
> >
> > Isn't it expected though if you render to an off-screen surface and then
> > convert it to a QImage ? What was the issue with rendering directly on
> > the screen with QOpenGLWidget ? Just the dynamic cast ?
> 
> Some other reasons are like:
> 1. The viewfinder and viewfinderGL are very similar, too many parts are
> duplicate.
> 2. My idea is that if viewfinder is able to handle both Qt and OpenGL
> rendering would be perfect.
>     The original processing flow when viewfinder got a new frame from
> camera is like below
>           Frame -> rendering -> call update() -> paintEvent( painting)
>     But in QOpenGLWidget way, the processing flow become
>           Frame -> call update() -> rendering and painting in paintGL()
>     That's why I try to render into off-screen to fit the original
> viewfinder process flow.
> 
> BUT honestly it seems not a good idea to render to an external off screen surface
> then convert to QImage due to the performance issue. I am trying to find some way
> to paint the QOpenGLFramebufferObject directly.
> 
> OR just letting the viewfinder inherit from QOpenGLWidgetand QOpenGLFunctions
> and make it able to handle Qt and OpenGL rendering both.
> I am still trying to figure out a good solution for that.

How about starting with a ViewfinderGL implementation that duplicates
code from the existing Viewfinder class, to get the feature merged, and
then reworking both classes on top to reduce code duplication ? I can
help with the latter.

> > > Show Liu (3):
> > >   qcam: add OpenGL shader code as QT resource
> > >   qcam: add OpenGL renderer
> > >   qcam: use the OpenGL renderer as NV family YUV format converter
> > >
> > >  src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
> > >  src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
> > >  src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
> > >  src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
> > >  src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
> > >  src/qcam/assets/shader/shaders.qrc           |  10 +
> > >  src/qcam/main.cpp                            |   3 +
> > >  src/qcam/main_window.cpp                     |   2 +
> > >  src/qcam/main_window.h                       |   1 +
> > >  src/qcam/meson.build                         |   3 +
> > >  src/qcam/renderer.cpp                        | 346 +++++++++++++++++++
> > >  src/qcam/renderer.h                          |  81 +++++
> > >  src/qcam/viewfinder.cpp                      |  46 ++-
> > >  src/qcam/viewfinder.h                        |  10 +
> > >  14 files changed, 640 insertions(+), 8 deletions(-)
> > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> > >  create mode 100644 src/qcam/assets/shader/shaders.qrc
> > >  create mode 100644 src/qcam/renderer.cpp
> > >  create mode 100644 src/qcam/renderer.h
Show Liu Aug. 26, 2020, 6:25 a.m. UTC | #4
Hi Laurent,


On Wed, Aug 26, 2020 at 3:06 AM Laurent Pinchart <
laurent.pinchart@ideasonboard.com> wrote:

> Hi Show,
>
> On Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote:
> > Hi Laurent,
> >
> > Thanks for your review.
> >
> > On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote:
> > > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:
> > > > Hi,
> > > >
> > > > This is the version 4 patchset for qcam to accelerate the YUV format
> conversion
> > > > by OpenGL shader. Most of the modifications are according to the
> comments
> > > > and suggestions from V3.
> > > > And the big change is using the QOpenGLFramebufferObject with
> QOffscreenSurface
> > > > instead of the QOpenGLWidget, the main reason that use dynamic_cast
> > > > to get the viewfinder_ is really weird.
> > > > In this version, OpenGL rendering enabled by --render=gles,
> > > > and if any error occurs during OpenGL configuration, it will return
> to use QT rendering.
> > > > But compared with the previous version, this version performance
> dropped a lot
> > > > due to I using the toImage() function to get the image when
> rendering complete,
> > > > it's really an inefficient operation. the framerate down to 16.x ~
> 18.x fps(1280 x 800).
> > > > I am still trying to improve this performance issue. Any suggestions
> are welcome.
> > >
> > > Isn't it expected though if you render to an off-screen surface and
> then
> > > convert it to a QImage ? What was the issue with rendering directly on
> > > the screen with QOpenGLWidget ? Just the dynamic cast ?
> >
> > Some other reasons are like:
> > 1. The viewfinder and viewfinderGL are very similar, too many parts are
> > duplicate.
> > 2. My idea is that if viewfinder is able to handle both Qt and OpenGL
> > rendering would be perfect.
> >     The original processing flow when viewfinder got a new frame from
> > camera is like below
> >           Frame -> rendering -> call update() -> paintEvent( painting)
> >     But in QOpenGLWidget way, the processing flow become
> >           Frame -> call update() -> rendering and painting in paintGL()
> >     That's why I try to render into off-screen to fit the original
> > viewfinder process flow.
> >
> > BUT honestly it seems not a good idea to render to an external off
> screen surface
> > then convert to QImage due to the performance issue. I am trying to find
> some way
> > to paint the QOpenGLFramebufferObject directly.
> >
> > OR just letting the viewfinder inherit from QOpenGLWidgetand
> QOpenGLFunctions
> > and make it able to handle Qt and OpenGL rendering both.
> > I am still trying to figure out a good solution for that.
>
> How about starting with a ViewfinderGL implementation that duplicates
> code from the existing Viewfinder class, to get the feature merged, and
> then reworking both classes on top to reduce code duplication ? I can
> help with the latter.
>

Sounds Great.
I will have the next version based on viewfinderGL ASAP.

Best Regards,
Show Liu

>
> > > > Show Liu (3):
> > > >   qcam: add OpenGL shader code as QT resource
> > > >   qcam: add OpenGL renderer
> > > >   qcam: use the OpenGL renderer as NV family YUV format converter
> > > >
> > > >  src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
> > > >  src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
> > > >  src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
> > > >  src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
> > > >  src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
> > > >  src/qcam/assets/shader/shaders.qrc           |  10 +
> > > >  src/qcam/main.cpp                            |   3 +
> > > >  src/qcam/main_window.cpp                     |   2 +
> > > >  src/qcam/main_window.h                       |   1 +
> > > >  src/qcam/meson.build                         |   3 +
> > > >  src/qcam/renderer.cpp                        | 346
> +++++++++++++++++++
> > > >  src/qcam/renderer.h                          |  81 +++++
> > > >  src/qcam/viewfinder.cpp                      |  46 ++-
> > > >  src/qcam/viewfinder.h                        |  10 +
> > > >  14 files changed, 640 insertions(+), 8 deletions(-)
> > > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> > > >  create mode 100644 src/qcam/assets/shader/shaders.qrc
> > > >  create mode 100644 src/qcam/renderer.cpp
> > > >  create mode 100644 src/qcam/renderer.h
>
> --
> Regards,
>
> Laurent Pinchart
>