[v13,0/7] Add VirtualPipelineHandler
mbox series

Message ID 20240926163141.1593026-1-chenghaoyang@google.com
Headers show
Series
  • Add VirtualPipelineHandler
Related show

Message

Cheng-Hao Yang Sept. 26, 2024, 4:29 p.m. UTC
Harvey Yang New message
Tue, Sep 10, 12:48 PM
to libcamera-devel, me

Hi all,

This series adds virtual pipeline handler, which doesn't depend on any
hardware, like camera sensor or ISP. Currently the configuration
supports test patterns and images.

It passed the gitlab pipeline:
https://gitlab.freedesktop.org/chenghaoyang/libcamera/-/pipelines/1278799

I've also checked that it works on qcam.

I failed to pass multi_stream_test when trying to enable multiple
streams though. Please give me some hints what it tests and what I
missed:
```
[295:43:43.910237144] [1441841]  INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/usr/local/google/home/chenghaoyang/Workspace/libca
mera/build/src/ipa' to the IPA search path
[295:43:43.914030564] [1441841]  INFO Camera camera_manager.cpp:325 libcamera v0.3.1+79-8ca4f033-dirty (2024-08-29T19:18:51UTC)
[295:43:43.915493754] [1441844] ERROR DmaBufAllocator dma_buf_allocator.cpp:120 Could not open any dma-buf provider
[295:43:43.919118835] [1441841]  INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/usr/local/google/home/chenghaoyang/Workspace/libca
mera/build/src/ipa' to the IPA search path
[295:43:43.922245825] [1441841]  INFO Camera camera_manager.cpp:325 libcamera v0.3.1+79-8ca4f033-dirty (2024-08-29T19:18:51UTC)
[295:43:43.922820175] [1441846] ERROR DmaBufAllocator dma_buf_allocator.cpp:120 Could not open any dma-buf provider
Unable to set the pipeline to the playing state.
```

Gitlab pipeline failure:
https://gitlab.freedesktop.org/chenghaoyang/libcamera/-/pipelines/1260412

Updates in v13:
- Returned an errno in `FrameGenerete::generateFrame()`

Updates in v12:
- Added ImageFrameGenerator first, then add Parser.
- Removed ScaleMode argument, as only Fill is supported now.
- Fixed bug of Stream's configuration usage in configure().

Updates in v11:
- Allowed a single value in the config file's frame_rates field.

Updates in v10:
Apply fixes according to Jacopo's and Barnabás' comments.
- Split test_pattern and path fields in the yaml format.
- Let FrameGenerators control frameCount_.
- Fixed match() returning values.

Updates in v9: Allocate contiguous memory for planes in the same
FrameBuffer.

BR,
Harvey

Harvey Yang (6):
  libcamera: add DmaBufAllocator::exportBuffers()
  libcamera: Remove PipelineHandler Fatal check of non-empty
    MediaDevices
  libcamera: virtual: Add VirtualPipelineHandler
  libcamera: virtual: Add ImageFrameGenerator
  libcamera: virtual: Read config and register cameras based on the
    config
  libcamera: software_isp: Refactor SoftwareIsp to use
    DmaBufAllocator::exportBuffers

Konami Shu (1):
  libcamera: pipeline: Add test pattern for VirtualPipelineHandler

 .../libcamera/internal/dma_buf_allocator.h    |  13 +
 meson.build                                   |   1 +
 meson_options.txt                             |   3 +-
 src/android/meson.build                       |  19 -
 src/libcamera/dma_buf_allocator.cpp           |  57 +++
 src/libcamera/pipeline/virtual/README.md      |  48 +++
 .../pipeline/virtual/data/virtual.yaml        |  36 ++
 .../pipeline/virtual/frame_generator.h        |  29 ++
 .../virtual/image_frame_generator.cpp         | 175 ++++++++
 .../pipeline/virtual/image_frame_generator.h  |  51 +++
 src/libcamera/pipeline/virtual/meson.build    |  13 +
 src/libcamera/pipeline/virtual/parser.cpp     | 243 +++++++++++
 src/libcamera/pipeline/virtual/parser.h       |  39 ++
 .../virtual/test_pattern_generator.cpp        | 137 +++++++
 .../pipeline/virtual/test_pattern_generator.h |  53 +++
 src/libcamera/pipeline/virtual/virtual.cpp    | 382 ++++++++++++++++++
 src/libcamera/pipeline/virtual/virtual.h      |  61 +++
 src/libcamera/pipeline_handler.cpp            |  11 +-
 src/libcamera/software_isp/software_isp.cpp   |  20 +-
 src/meson.build                               |  19 +
 20 files changed, 1368 insertions(+), 42 deletions(-)
 create mode 100644 src/libcamera/pipeline/virtual/README.md
 create mode 100644 src/libcamera/pipeline/virtual/data/virtual.yaml
 create mode 100644 src/libcamera/pipeline/virtual/frame_generator.h
 create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.cpp
 create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.h
 create mode 100644 src/libcamera/pipeline/virtual/meson.build
 create mode 100644 src/libcamera/pipeline/virtual/parser.cpp
 create mode 100644 src/libcamera/pipeline/virtual/parser.h
 create mode 100644 src/libcamera/pipeline/virtual/test_pattern_generator.cpp
 create mode 100644 src/libcamera/pipeline/virtual/test_pattern_generator.h
 create mode 100644 src/libcamera/pipeline/virtual/virtual.cpp
 create mode 100644 src/libcamera/pipeline/virtual/virtual.h

Comments

Kieran Bingham Sept. 27, 2024, 4:32 p.m. UTC | #1
Quoting Harvey Yang (2024-09-26 17:29:49)
> 
> Harvey Yang New message
> Tue, Sep 10, 12:48 PM
> to libcamera-devel, me
> 
> Hi all,
> 
> This series adds virtual pipeline handler, which doesn't depend on any
> hardware, like camera sensor or ISP. Currently the configuration
> supports test patterns and images.
> 
> It passed the gitlab pipeline:
> https://gitlab.freedesktop.org/chenghaoyang/libcamera/-/pipelines/1278799
> 
> I've also checked that it works on qcam.
> 
> I failed to pass multi_stream_test when trying to enable multiple
> streams though. Please give me some hints what it tests and what I
> missed:
> ```
> [295:43:43.910237144] [1441841]  INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/usr/local/google/home/chenghaoyang/Workspace/libca
> mera/build/src/ipa' to the IPA search path
> [295:43:43.914030564] [1441841]  INFO Camera camera_manager.cpp:325 libcamera v0.3.1+79-8ca4f033-dirty (2024-08-29T19:18:51UTC)
> [295:43:43.915493754] [1441844] ERROR DmaBufAllocator dma_buf_allocator.cpp:120 Could not open any dma-buf provider
> [295:43:43.919118835] [1441841]  INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/usr/local/google/home/chenghaoyang/Workspace/libca
> mera/build/src/ipa' to the IPA search path
> [295:43:43.922245825] [1441841]  INFO Camera camera_manager.cpp:325 libcamera v0.3.1+79-8ca4f033-dirty (2024-08-29T19:18:51UTC)
> [295:43:43.922820175] [1441846] ERROR DmaBufAllocator dma_buf_allocator.cpp:120 Could not open any dma-buf provider
> Unable to set the pipeline to the playing state.

I presume this is some failure in the configuration of the virtual test
environment that isn't exposing access to the suitable udmabuf handler?

And I wonder if this series is now making the unit tests use the virtual
camera instead of vimc ?

If so - we probably want to have the tests more consistently defined
maybe - to either prefer specific cameras - or run on all cameras
available?

But I think fixing this issue probably needs investigation on the
libcamera-ci build (https://gitlab.freedesktop.org/camera/libcamera-ci)

--
Kieran


> ```
> 
> Gitlab pipeline failure:
> https://gitlab.freedesktop.org/chenghaoyang/libcamera/-/pipelines/1260412
> 
> Updates in v13:
> - Returned an errno in `FrameGenerete::generateFrame()`
> 
> Updates in v12:
> - Added ImageFrameGenerator first, then add Parser.
> - Removed ScaleMode argument, as only Fill is supported now.
> - Fixed bug of Stream's configuration usage in configure().
> 
> Updates in v11:
> - Allowed a single value in the config file's frame_rates field.
> 
> Updates in v10:
> Apply fixes according to Jacopo's and Barnabás' comments.
> - Split test_pattern and path fields in the yaml format.
> - Let FrameGenerators control frameCount_.
> - Fixed match() returning values.
> 
> Updates in v9: Allocate contiguous memory for planes in the same
> FrameBuffer.
> 
> BR,
> Harvey
> 
> Harvey Yang (6):
>   libcamera: add DmaBufAllocator::exportBuffers()
>   libcamera: Remove PipelineHandler Fatal check of non-empty
>     MediaDevices
>   libcamera: virtual: Add VirtualPipelineHandler
>   libcamera: virtual: Add ImageFrameGenerator
>   libcamera: virtual: Read config and register cameras based on the
>     config
>   libcamera: software_isp: Refactor SoftwareIsp to use
>     DmaBufAllocator::exportBuffers
> 
> Konami Shu (1):
>   libcamera: pipeline: Add test pattern for VirtualPipelineHandler
> 
>  .../libcamera/internal/dma_buf_allocator.h    |  13 +
>  meson.build                                   |   1 +
>  meson_options.txt                             |   3 +-
>  src/android/meson.build                       |  19 -
>  src/libcamera/dma_buf_allocator.cpp           |  57 +++
>  src/libcamera/pipeline/virtual/README.md      |  48 +++
>  .../pipeline/virtual/data/virtual.yaml        |  36 ++
>  .../pipeline/virtual/frame_generator.h        |  29 ++
>  .../virtual/image_frame_generator.cpp         | 175 ++++++++
>  .../pipeline/virtual/image_frame_generator.h  |  51 +++
>  src/libcamera/pipeline/virtual/meson.build    |  13 +
>  src/libcamera/pipeline/virtual/parser.cpp     | 243 +++++++++++
>  src/libcamera/pipeline/virtual/parser.h       |  39 ++
>  .../virtual/test_pattern_generator.cpp        | 137 +++++++
>  .../pipeline/virtual/test_pattern_generator.h |  53 +++
>  src/libcamera/pipeline/virtual/virtual.cpp    | 382 ++++++++++++++++++
>  src/libcamera/pipeline/virtual/virtual.h      |  61 +++
>  src/libcamera/pipeline_handler.cpp            |  11 +-
>  src/libcamera/software_isp/software_isp.cpp   |  20 +-
>  src/meson.build                               |  19 +
>  20 files changed, 1368 insertions(+), 42 deletions(-)
>  create mode 100644 src/libcamera/pipeline/virtual/README.md
>  create mode 100644 src/libcamera/pipeline/virtual/data/virtual.yaml
>  create mode 100644 src/libcamera/pipeline/virtual/frame_generator.h
>  create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.cpp
>  create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.h
>  create mode 100644 src/libcamera/pipeline/virtual/meson.build
>  create mode 100644 src/libcamera/pipeline/virtual/parser.cpp
>  create mode 100644 src/libcamera/pipeline/virtual/parser.h
>  create mode 100644 src/libcamera/pipeline/virtual/test_pattern_generator.cpp
>  create mode 100644 src/libcamera/pipeline/virtual/test_pattern_generator.h
>  create mode 100644 src/libcamera/pipeline/virtual/virtual.cpp
>  create mode 100644 src/libcamera/pipeline/virtual/virtual.h
> 
> -- 
> 2.46.1.824.gd892dcdcdd-goog
>
Cheng-Hao Yang Sept. 27, 2024, 5:55 p.m. UTC | #2
Hi Kieran,

On Sat, Sep 28, 2024 at 12:32 AM Kieran Bingham
<kieran.bingham@ideasonboard.com> wrote:
>
> Quoting Harvey Yang (2024-09-26 17:29:49)
> >
> > Harvey Yang New message
> > Tue, Sep 10, 12:48 PM
> > to libcamera-devel, me
> >
> > Hi all,
> >
> > This series adds virtual pipeline handler, which doesn't depend on any
> > hardware, like camera sensor or ISP. Currently the configuration
> > supports test patterns and images.
> >
> > It passed the gitlab pipeline:
> > https://gitlab.freedesktop.org/chenghaoyang/libcamera/-/pipelines/1278799
> >
> > I've also checked that it works on qcam.
> >
> > I failed to pass multi_stream_test when trying to enable multiple
> > streams though. Please give me some hints what it tests and what I
> > missed:
> > ```
> > [295:43:43.910237144] [1441841]  INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/usr/local/google/home/chenghaoyang/Workspace/libca
> > mera/build/src/ipa' to the IPA search path
> > [295:43:43.914030564] [1441841]  INFO Camera camera_manager.cpp:325 libcamera v0.3.1+79-8ca4f033-dirty (2024-08-29T19:18:51UTC)
> > [295:43:43.915493754] [1441844] ERROR DmaBufAllocator dma_buf_allocator.cpp:120 Could not open any dma-buf provider
> > [295:43:43.919118835] [1441841]  INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/usr/local/google/home/chenghaoyang/Workspace/libca
> > mera/build/src/ipa' to the IPA search path
> > [295:43:43.922245825] [1441841]  INFO Camera camera_manager.cpp:325 libcamera v0.3.1+79-8ca4f033-dirty (2024-08-29T19:18:51UTC)
> > [295:43:43.922820175] [1441846] ERROR DmaBufAllocator dma_buf_allocator.cpp:120 Could not open any dma-buf provider
> > Unable to set the pipeline to the playing state.
>
> I presume this is some failure in the configuration of the virtual test
> environment that isn't exposing access to the suitable udmabuf handler?

Actually, I don't think so. I added logs in exportFrameBuffer(), and the
test doesn't produce that log, which indicates the test fails before using
DmaBufAllocator. Also, the test still fails with memfd workaround:
https://gitlab.freedesktop.org/chenghaoyang/libcamera/-/pipelines/1279655

>
> And I wonder if this series is now making the unit tests use the virtual
> camera instead of vimc ?

In my own trials, it seems that multi_stream_test constructs both
virtual and vimc pipeline handlers. I'm not exactly sure what it tests
though.

>
> If so - we probably want to have the tests more consistently defined
> maybe - to either prefer specific cameras - or run on all cameras
> available?
>
> But I think fixing this issue probably needs investigation on the
> libcamera-ci build (https://gitlab.freedesktop.org/camera/libcamera-ci)
>
> --
> Kieran
>
>
> > ```
> >
> > Gitlab pipeline failure:
> > https://gitlab.freedesktop.org/chenghaoyang/libcamera/-/pipelines/1260412
> >
> > Updates in v13:
> > - Returned an errno in `FrameGenerete::generateFrame()`
> >
> > Updates in v12:
> > - Added ImageFrameGenerator first, then add Parser.
> > - Removed ScaleMode argument, as only Fill is supported now.
> > - Fixed bug of Stream's configuration usage in configure().
> >
> > Updates in v11:
> > - Allowed a single value in the config file's frame_rates field.
> >
> > Updates in v10:
> > Apply fixes according to Jacopo's and Barnabás' comments.
> > - Split test_pattern and path fields in the yaml format.
> > - Let FrameGenerators control frameCount_.
> > - Fixed match() returning values.
> >
> > Updates in v9: Allocate contiguous memory for planes in the same
> > FrameBuffer.
> >
> > BR,
> > Harvey
> >
> > Harvey Yang (6):
> >   libcamera: add DmaBufAllocator::exportBuffers()
> >   libcamera: Remove PipelineHandler Fatal check of non-empty
> >     MediaDevices
> >   libcamera: virtual: Add VirtualPipelineHandler
> >   libcamera: virtual: Add ImageFrameGenerator
> >   libcamera: virtual: Read config and register cameras based on the
> >     config
> >   libcamera: software_isp: Refactor SoftwareIsp to use
> >     DmaBufAllocator::exportBuffers
> >
> > Konami Shu (1):
> >   libcamera: pipeline: Add test pattern for VirtualPipelineHandler
> >
> >  .../libcamera/internal/dma_buf_allocator.h    |  13 +
> >  meson.build                                   |   1 +
> >  meson_options.txt                             |   3 +-
> >  src/android/meson.build                       |  19 -
> >  src/libcamera/dma_buf_allocator.cpp           |  57 +++
> >  src/libcamera/pipeline/virtual/README.md      |  48 +++
> >  .../pipeline/virtual/data/virtual.yaml        |  36 ++
> >  .../pipeline/virtual/frame_generator.h        |  29 ++
> >  .../virtual/image_frame_generator.cpp         | 175 ++++++++
> >  .../pipeline/virtual/image_frame_generator.h  |  51 +++
> >  src/libcamera/pipeline/virtual/meson.build    |  13 +
> >  src/libcamera/pipeline/virtual/parser.cpp     | 243 +++++++++++
> >  src/libcamera/pipeline/virtual/parser.h       |  39 ++
> >  .../virtual/test_pattern_generator.cpp        | 137 +++++++
> >  .../pipeline/virtual/test_pattern_generator.h |  53 +++
> >  src/libcamera/pipeline/virtual/virtual.cpp    | 382 ++++++++++++++++++
> >  src/libcamera/pipeline/virtual/virtual.h      |  61 +++
> >  src/libcamera/pipeline_handler.cpp            |  11 +-
> >  src/libcamera/software_isp/software_isp.cpp   |  20 +-
> >  src/meson.build                               |  19 +
> >  20 files changed, 1368 insertions(+), 42 deletions(-)
> >  create mode 100644 src/libcamera/pipeline/virtual/README.md
> >  create mode 100644 src/libcamera/pipeline/virtual/data/virtual.yaml
> >  create mode 100644 src/libcamera/pipeline/virtual/frame_generator.h
> >  create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.cpp
> >  create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.h
> >  create mode 100644 src/libcamera/pipeline/virtual/meson.build
> >  create mode 100644 src/libcamera/pipeline/virtual/parser.cpp
> >  create mode 100644 src/libcamera/pipeline/virtual/parser.h
> >  create mode 100644 src/libcamera/pipeline/virtual/test_pattern_generator.cpp
> >  create mode 100644 src/libcamera/pipeline/virtual/test_pattern_generator.h
> >  create mode 100644 src/libcamera/pipeline/virtual/virtual.cpp
> >  create mode 100644 src/libcamera/pipeline/virtual/virtual.h
> >
> > --
> > 2.46.1.824.gd892dcdcdd-goog
> >