[libcamera-devel,v5,00/23] IPA isolation implementation

10563 mbox series

This patchset implements IPA isolation, and is fully plumbed and
runnable on all pipeline handlers. This also includes code
generators for the IPA proxies, and headers and serializers for custom
data structures and functions. Unlike the RFC, this patch series is
structured like a normal patch series, in order that should be merged.
The only exception is patches 10/23-19/23, which must be squashed to
avoid bisection breakage, and are only kept separate to ease review.

To see samples of generated code (without running this), see the RFC
"IPA isolation example, with IPC, fully plumbed".

To restate the problem, we have two goals:
- to be able to run IPAs isolated in a separate process
- the isolation must be transparent to both the pipeline handler and
  the IPA
During design of the IPC mechanism, we realized that we could support
both custom fuctions and custom data structures, which would be a lot
nicer than the tedious manual de/serialization that we had before with
IPAOperationData.

The architecture of the whole thing is as follows:

pipeline handler -> IPAProxyRPi --thread--> IPARPi
                                \
                                 \-> IPCPipe --IPC--> IPAProxyRPiWorker -> IPARPi

Where the pipeline author defines the interface between the pipeline
handler and the IPA, as well as any data structures. Based on this,
headers, de/serializers, and the proxy (including the worker) will be
generated.

Patches 01/23 to 07/23 add the meat of this IPC and custom IPA interface
mechanisms. Patches 10/23 to 19/23 are modifications in the rest of
libcamera to make the whole mechanism work, and must be squashed
together to avoid bisection breakage. Patches 20/23, 21/23, and 23/23
add tests for the serializer, IPC components, and serializer generation.
Newly in v4, patches 22/23 adds an IPA guide, and 23/23 adds a test to test
generated serializers and headers.

Patch 16/23 may be of interest, as it shows how one would write the
interface and data definition. In v2 I have customized the raspberrypi
mojom file to take advantage of the freedom in definition the IPA
interface and data structures (as opposed to v1, where it was
practically a direct translation of IPAOperationData). In v3 I have
added namespace support to the data definition, and this is reflected in
its usage.

In v2 I have added some rules to the mojom file, such as start(),
stop(), and init() are required, along with the main and callback
interfaces. As of v5, event interfaces are also required to have at
least one event. These are documented in core.mojom in patch 15/23 and
enforced in the code generator in patch 01/23. core.mojom is a new
addition in v2, and is where common mojom "structs" (which represent
libcamera objects) and IPA interface definition documentaion are.

Patch 17/23 is also noteworthy, as it shows how one would use the
interface that has been defined. Since the headers and proxies are
autogenerated, there isn't a patch where the generated code can be seen,
however.

Patches 18/23 and 19/23 also define mojom files and use them, but are
for vimc and rkisp1, respectively. These will also give an idea on how
the mojom definition looks and how it will be used.

Changes in v5:
- rename IPAIPC to IPCPipe
- IPCPipe passes IPCMessages, instead of byte vector + fd vector
- rename the generated files so the naming is consistent
- make IPADataSerializer use const references and const iterators
- removal of IPA wrapper test moved earlier
- squash all patches related to replacing the C IPA interface with the
  customizable C++ IPA interface
- squash all patches related to making the IPAProxy one-per-pipeline

Changes in v4:
- add IPA guide
- add test for header and serializer generation
- fix style in generated code and IPADataSerializer
- add segfault protections in read/append helpers in de/serializer
- rename generated header and serializer
- rename IPA callback interface to IPA event interface

Changes in v3:
- add support for namespaces in the mojom file
  - note that they individually must not collide with existing
    namespaces (enforced by the C++ compiler)
- move IPAIPCUnixSocket helper functions away from global and into the
  class
- add SPDX and copyright to python scripts
- add parser.py to wrap the mojo parser
- make IPADataSerializer definition a bit cleaner with reimplemented POD
  manipulation functions
- expand mojom documentation in core.mojom

Changes in v2:
- upgrade documentation
- fix documentation compiling
- plumb the new IPC system through the other pipelines
- fix the issue where editing jinja templates wouldn't trigger
  recompile
- enforce IPA interface rules in code generator
- fix previously-untriggered code generation bugs
- add de/serializers for all primitive types, string, and const
  ControlList
- customized the RPi IPA interface
- add licenses
- add tests



Paul Elder (23):
  utils: ipc: add templates for code generation for IPC mechanism
  utils: ipc: add generator script
  utils: ipc: add parser script
  Documentation: skip generating documentation for generated code
  libcamera: Add IPADataSerializer
  libcamera: Add IPAIPC
  libcamera: Add IPAIPC implementation based on unix socket
  meson: ipa, proxy: Generate headers and proxy with mojo
  tests: Remove IPA wrappers test
  ipa: raspberrypi: meson: Add dependency on generated headers
  libcamera: IPAInterface: Replace C API with the new C++-only API
  libcamera: IPAProxy: Remove stop() override
  libcamera: IPAProxy, IPAManager: Switch to one-proxy-per-pipeline
    scheme
  libcamera: PipelineHandler: Remove IPA from base class
  ipa: Add core.mojom
  ipa: raspberrypi: Add mojom data definition file
  libcamera: pipeline, ipa: raspberrypi: Use new data definition
  libcamera: pipeline, ipa: vimc: Support the new IPC mechanism
  libcamera: pipeline, ipa: rkisp1: Support the new IPC mechanism
  tests: Add IPADataSerializer test
  tests: Add test for IPCPipeUnixSocket
  Documentation: Add IPA writers guide
  tests: Test IPA serializer generation

 Documentation/Doxyfile.in                     |   7 +-
 Documentation/guides/ipa.rst                  | 479 +++++++++
 Documentation/index.rst                       |   1 +
 Documentation/meson.build                     |   1 +
 .../libcamera/internal/ipa_context_wrapper.h  |  52 -
 .../libcamera/internal/ipa_data_serializer.h  | 932 ++++++++++++++++++
 include/libcamera/internal/ipa_manager.h      |  31 +-
 include/libcamera/internal/ipa_module.h       |   4 +-
 include/libcamera/internal/ipa_proxy.h        |  31 -
 include/libcamera/internal/ipc_pipe.h         |  70 ++
 .../libcamera/internal/ipc_pipe_unixsocket.h  |  50 +
 include/libcamera/internal/meson.build        |   1 -
 include/libcamera/internal/pipeline_handler.h |   1 -
 include/libcamera/ipa/core.mojom              |  48 +
 include/libcamera/ipa/ipa_interface.h         | 126 +--
 include/libcamera/ipa/meson.build             |  94 ++
 include/libcamera/ipa/raspberrypi.h           |  59 +-
 include/libcamera/ipa/raspberrypi.mojom       | 119 +++
 include/libcamera/ipa/rkisp1.h                |  18 +-
 include/libcamera/ipa/rkisp1.mojom            |  42 +
 include/libcamera/ipa/vimc.h                  |   8 +
 include/libcamera/ipa/vimc.mojom              |  13 +
 src/ipa/libipa/ipa_interface_wrapper.cpp      | 285 ------
 src/ipa/libipa/ipa_interface_wrapper.h        |  61 --
 src/ipa/libipa/meson.build                    |   2 -
 src/ipa/raspberrypi/meson.build               |   2 +-
 src/ipa/raspberrypi/raspberrypi.cpp           | 160 ++-
 src/ipa/rkisp1/meson.build                    |   2 +-
 src/ipa/rkisp1/rkisp1.cpp                     |  55 +-
 src/ipa/vimc/meson.build                      |   2 +-
 src/ipa/vimc/vimc.cpp                         |  18 +-
 src/libcamera/ipa_context_wrapper.cpp         | 297 ------
 src/libcamera/ipa_data_serializer.cpp         | 178 ++++
 src/libcamera/ipa_interface.cpp               | 534 +---------
 src/libcamera/ipa_manager.cpp                 |  47 +-
 src/libcamera/ipa_module.cpp                  |  18 +-
 src/libcamera/ipa_proxy.cpp                   | 101 --
 src/libcamera/ipc_pipe.cpp                    | 211 ++++
 src/libcamera/ipc_pipe_unixsocket.cpp         | 147 +++
 src/libcamera/meson.build                     |   5 +-
 .../pipeline/raspberrypi/raspberrypi.cpp      | 193 ++--
 src/libcamera/pipeline/rkisp1/rkisp1.cpp      |  44 +-
 src/libcamera/pipeline/vimc/vimc.cpp          |   8 +-
 src/libcamera/pipeline_handler.cpp            |   8 -
 src/libcamera/proxy/ipa_proxy_linux.cpp       | 103 --
 src/libcamera/proxy/ipa_proxy_thread.cpp      | 172 ----
 src/libcamera/proxy/meson.build               |  21 +-
 .../proxy/worker/ipa_proxy_linux_worker.cpp   |  90 --
 src/libcamera/proxy/worker/meson.build        |  23 +-
 test/ipa/ipa_interface_test.cpp               |   8 +-
 test/ipa/ipa_wrappers_test.cpp                | 452 ---------
 test/ipa/meson.build                          |   3 +-
 test/ipc/meson.build                          |   3 +-
 test/ipc/unixsocket_ipc.cpp                   | 237 +++++
 .../generated_serializer_test.cpp             | 128 +++
 .../generated_serializer/meson.build          |  49 +
 .../generated_serializer/vimc.mojom           |  24 +
 .../ipa_data_serializer_test.cpp              | 465 +++++++++
 test/serialization/meson.build                |   5 +-
 utils/ipc/generate.py                         |  29 +
 .../libcamera_templates/meson.build           |  11 +
 .../module_ipa_interface.h.tmpl               | 113 +++
 .../module_ipa_proxy.cpp.tmpl                 | 223 +++++
 .../module_ipa_proxy.h.tmpl                   | 121 +++
 .../module_ipa_proxy_worker.cpp.tmpl          | 224 +++++
 .../module_ipa_serializer.h.tmpl              |  47 +
 .../libcamera_templates/proxy_functions.tmpl  | 204 ++++
 .../libcamera_templates/serializer.tmpl       | 289 ++++++
 utils/ipc/generators/meson.build              |   3 +
 .../generators/mojom_libcamera_generator.py   | 487 +++++++++
 utils/ipc/meson.build                         |  13 +
 utils/ipc/parser.py                           |  20 +
 72 files changed, 5477 insertions(+), 2655 deletions(-)
 create mode 100644 Documentation/guides/ipa.rst
 delete mode 100644 include/libcamera/internal/ipa_context_wrapper.h
 create mode 100644 include/libcamera/internal/ipa_data_serializer.h
 create mode 100644 include/libcamera/internal/ipc_pipe.h
 create mode 100644 include/libcamera/internal/ipc_pipe_unixsocket.h
 create mode 100644 include/libcamera/ipa/core.mojom
 create mode 100644 include/libcamera/ipa/raspberrypi.mojom
 create mode 100644 include/libcamera/ipa/rkisp1.mojom
 create mode 100644 include/libcamera/ipa/vimc.mojom
 delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.cpp
 delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.h
 delete mode 100644 src/libcamera/ipa_context_wrapper.cpp
 create mode 100644 src/libcamera/ipa_data_serializer.cpp
 create mode 100644 src/libcamera/ipc_pipe.cpp
 create mode 100644 src/libcamera/ipc_pipe_unixsocket.cpp
 delete mode 100644 src/libcamera/proxy/ipa_proxy_linux.cpp
 delete mode 100644 src/libcamera/proxy/ipa_proxy_thread.cpp
 delete mode 100644 src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp
 delete mode 100644 test/ipa/ipa_wrappers_test.cpp
 create mode 100644 test/ipc/unixsocket_ipc.cpp
 create mode 100644 test/serialization/generated_serializer/generated_serializer_test.cpp
 create mode 100644 test/serialization/generated_serializer/meson.build
 create mode 100644 test/serialization/generated_serializer/vimc.mojom
 create mode 100644 test/serialization/ipa_data_serializer_test.cpp
 create mode 100755 utils/ipc/generate.py
 create mode 100644 utils/ipc/generators/libcamera_templates/meson.build
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl
 create mode 100644 utils/ipc/generators/meson.build
 create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py
 create mode 100755 utils/ipc/parser.py