{"id":10563,"url":"https://patchwork.libcamera.org/api/1.1/covers/10563/?format=json","web_url":"https://patchwork.libcamera.org/cover/10563/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20201205103106.242080-1-paul.elder@ideasonboard.com>","date":"2020-12-05T10:30:43","name":"[libcamera-devel,v5,00/23] IPA isolation implementation","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/1.1/people/17/?format=json","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"mbox":"https://patchwork.libcamera.org/cover/10563/mbox/","series":[{"id":1506,"url":"https://patchwork.libcamera.org/api/1.1/series/1506/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1506","date":"2020-12-05T10:30:43","name":"IPA isolation implementation","version":5,"mbox":"https://patchwork.libcamera.org/series/1506/mbox/"}],"comments":"https://patchwork.libcamera.org/api/covers/10563/comments/","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 793FBBDB20\n\tfor <parsemail@patchwork.libcamera.org>;\n\tSat,  5 Dec 2020 10:31:17 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 037CF635F5;\n\tSat,  5 Dec 2020 11:31:17 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CDA80635F0\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tSat,  5 Dec 2020 11:31:15 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2400:4051:61:600:2c71:1b79:d06d:5032])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 4A1C12A4;\n\tSat,  5 Dec 2020 11:31:14 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"IE01N11Y\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1607164275;\n\tbh=ciVUrqcojLqz2Bg7w14XC5OnQnSALhs0lWFv1h5vjkg=;\n\th=From:To:Cc:Subject:Date:From;\n\tb=IE01N11Yy4yGgYT+oyI+3yNiNh1K4HhYrEb6bk+5kgdC43t57HoKRoGWgoX0EQXCy\n\tcWwHJXnwrfRkOskiYBJIZ7y8VNhT7nfGA7zHh3zg05VOEdCY3uH8KNIoYrxeyb4Kb/\n\tGPS6BOdnqLYUnpgoiUrLoENfGZxevxiog+s5Dv6Q=","From":"Paul Elder <paul.elder@ideasonboard.com>","To":"libcamera-devel@lists.libcamera.org","Date":"Sat,  5 Dec 2020 19:30:43 +0900","Message-Id":"<20201205103106.242080-1-paul.elder@ideasonboard.com>","X-Mailer":"git-send-email 2.27.0","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v5 00/23] IPA isolation implementation","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"},"content":"This patchset implements IPA isolation, and is fully plumbed and\nrunnable on all pipeline handlers. This also includes code\ngenerators for the IPA proxies, and headers and serializers for custom\ndata structures and functions. Unlike the RFC, this patch series is\nstructured like a normal patch series, in order that should be merged.\nThe only exception is patches 10/23-19/23, which must be squashed to\navoid bisection breakage, and are only kept separate to ease review.\n\nTo see samples of generated code (without running this), see the RFC\n\"IPA isolation example, with IPC, fully plumbed\".\n\nTo restate the problem, we have two goals:\n- to be able to run IPAs isolated in a separate process\n- the isolation must be transparent to both the pipeline handler and\n  the IPA\nDuring design of the IPC mechanism, we realized that we could support\nboth custom fuctions and custom data structures, which would be a lot\nnicer than the tedious manual de/serialization that we had before with\nIPAOperationData.\n\nThe architecture of the whole thing is as follows:\n\npipeline handler -> IPAProxyRPi --thread--> IPARPi\n                                \\\n                                 \\-> IPCPipe --IPC--> IPAProxyRPiWorker -> IPARPi\n\nWhere the pipeline author defines the interface between the pipeline\nhandler and the IPA, as well as any data structures. Based on this,\nheaders, de/serializers, and the proxy (including the worker) will be\ngenerated.\n\nPatches 01/23 to 07/23 add the meat of this IPC and custom IPA interface\nmechanisms. Patches 10/23 to 19/23 are modifications in the rest of\nlibcamera to make the whole mechanism work, and must be squashed\ntogether to avoid bisection breakage. Patches 20/23, 21/23, and 23/23\nadd tests for the serializer, IPC components, and serializer generation.\nNewly in v4, patches 22/23 adds an IPA guide, and 23/23 adds a test to test\ngenerated serializers and headers.\n\nPatch 16/23 may be of interest, as it shows how one would write the\ninterface and data definition. In v2 I have customized the raspberrypi\nmojom file to take advantage of the freedom in definition the IPA\ninterface and data structures (as opposed to v1, where it was\npractically a direct translation of IPAOperationData). In v3 I have\nadded namespace support to the data definition, and this is reflected in\nits usage.\n\nIn v2 I have added some rules to the mojom file, such as start(),\nstop(), and init() are required, along with the main and callback\ninterfaces. As of v5, event interfaces are also required to have at\nleast one event. These are documented in core.mojom in patch 15/23 and\nenforced in the code generator in patch 01/23. core.mojom is a new\naddition in v2, and is where common mojom \"structs\" (which represent\nlibcamera objects) and IPA interface definition documentaion are.\n\nPatch 17/23 is also noteworthy, as it shows how one would use the\ninterface that has been defined. Since the headers and proxies are\nautogenerated, there isn't a patch where the generated code can be seen,\nhowever.\n\nPatches 18/23 and 19/23 also define mojom files and use them, but are\nfor vimc and rkisp1, respectively. These will also give an idea on how\nthe mojom definition looks and how it will be used.\n\nChanges in v5:\n- rename IPAIPC to IPCPipe\n- IPCPipe passes IPCMessages, instead of byte vector + fd vector\n- rename the generated files so the naming is consistent\n- make IPADataSerializer use const references and const iterators\n- removal of IPA wrapper test moved earlier\n- squash all patches related to replacing the C IPA interface with the\n  customizable C++ IPA interface\n- squash all patches related to making the IPAProxy one-per-pipeline\n\nChanges in v4:\n- add IPA guide\n- add test for header and serializer generation\n- fix style in generated code and IPADataSerializer\n- add segfault protections in read/append helpers in de/serializer\n- rename generated header and serializer\n- rename IPA callback interface to IPA event interface\n\nChanges in v3:\n- add support for namespaces in the mojom file\n  - note that they individually must not collide with existing\n    namespaces (enforced by the C++ compiler)\n- move IPAIPCUnixSocket helper functions away from global and into the\n  class\n- add SPDX and copyright to python scripts\n- add parser.py to wrap the mojo parser\n- make IPADataSerializer definition a bit cleaner with reimplemented POD\n  manipulation functions\n- expand mojom documentation in core.mojom\n\nChanges in v2:\n- upgrade documentation\n- fix documentation compiling\n- plumb the new IPC system through the other pipelines\n- fix the issue where editing jinja templates wouldn't trigger\n  recompile\n- enforce IPA interface rules in code generator\n- fix previously-untriggered code generation bugs\n- add de/serializers for all primitive types, string, and const\n  ControlList\n- customized the RPi IPA interface\n- add licenses\n- add tests\n\n\n\nPaul Elder (23):\n  utils: ipc: add templates for code generation for IPC mechanism\n  utils: ipc: add generator script\n  utils: ipc: add parser script\n  Documentation: skip generating documentation for generated code\n  libcamera: Add IPADataSerializer\n  libcamera: Add IPAIPC\n  libcamera: Add IPAIPC implementation based on unix socket\n  meson: ipa, proxy: Generate headers and proxy with mojo\n  tests: Remove IPA wrappers test\n  ipa: raspberrypi: meson: Add dependency on generated headers\n  libcamera: IPAInterface: Replace C API with the new C++-only API\n  libcamera: IPAProxy: Remove stop() override\n  libcamera: IPAProxy, IPAManager: Switch to one-proxy-per-pipeline\n    scheme\n  libcamera: PipelineHandler: Remove IPA from base class\n  ipa: Add core.mojom\n  ipa: raspberrypi: Add mojom data definition file\n  libcamera: pipeline, ipa: raspberrypi: Use new data definition\n  libcamera: pipeline, ipa: vimc: Support the new IPC mechanism\n  libcamera: pipeline, ipa: rkisp1: Support the new IPC mechanism\n  tests: Add IPADataSerializer test\n  tests: Add test for IPCPipeUnixSocket\n  Documentation: Add IPA writers guide\n  tests: Test IPA serializer generation\n\n Documentation/Doxyfile.in                     |   7 +-\n Documentation/guides/ipa.rst                  | 479 +++++++++\n Documentation/index.rst                       |   1 +\n Documentation/meson.build                     |   1 +\n .../libcamera/internal/ipa_context_wrapper.h  |  52 -\n .../libcamera/internal/ipa_data_serializer.h  | 932 ++++++++++++++++++\n include/libcamera/internal/ipa_manager.h      |  31 +-\n include/libcamera/internal/ipa_module.h       |   4 +-\n include/libcamera/internal/ipa_proxy.h        |  31 -\n include/libcamera/internal/ipc_pipe.h         |  70 ++\n .../libcamera/internal/ipc_pipe_unixsocket.h  |  50 +\n include/libcamera/internal/meson.build        |   1 -\n include/libcamera/internal/pipeline_handler.h |   1 -\n include/libcamera/ipa/core.mojom              |  48 +\n include/libcamera/ipa/ipa_interface.h         | 126 +--\n include/libcamera/ipa/meson.build             |  94 ++\n include/libcamera/ipa/raspberrypi.h           |  59 +-\n include/libcamera/ipa/raspberrypi.mojom       | 119 +++\n include/libcamera/ipa/rkisp1.h                |  18 +-\n include/libcamera/ipa/rkisp1.mojom            |  42 +\n include/libcamera/ipa/vimc.h                  |   8 +\n include/libcamera/ipa/vimc.mojom              |  13 +\n src/ipa/libipa/ipa_interface_wrapper.cpp      | 285 ------\n src/ipa/libipa/ipa_interface_wrapper.h        |  61 --\n src/ipa/libipa/meson.build                    |   2 -\n src/ipa/raspberrypi/meson.build               |   2 +-\n src/ipa/raspberrypi/raspberrypi.cpp           | 160 ++-\n src/ipa/rkisp1/meson.build                    |   2 +-\n src/ipa/rkisp1/rkisp1.cpp                     |  55 +-\n src/ipa/vimc/meson.build                      |   2 +-\n src/ipa/vimc/vimc.cpp                         |  18 +-\n src/libcamera/ipa_context_wrapper.cpp         | 297 ------\n src/libcamera/ipa_data_serializer.cpp         | 178 ++++\n src/libcamera/ipa_interface.cpp               | 534 +---------\n src/libcamera/ipa_manager.cpp                 |  47 +-\n src/libcamera/ipa_module.cpp                  |  18 +-\n src/libcamera/ipa_proxy.cpp                   | 101 --\n src/libcamera/ipc_pipe.cpp                    | 211 ++++\n src/libcamera/ipc_pipe_unixsocket.cpp         | 147 +++\n src/libcamera/meson.build                     |   5 +-\n .../pipeline/raspberrypi/raspberrypi.cpp      | 193 ++--\n src/libcamera/pipeline/rkisp1/rkisp1.cpp      |  44 +-\n src/libcamera/pipeline/vimc/vimc.cpp          |   8 +-\n src/libcamera/pipeline_handler.cpp            |   8 -\n src/libcamera/proxy/ipa_proxy_linux.cpp       | 103 --\n src/libcamera/proxy/ipa_proxy_thread.cpp      | 172 ----\n src/libcamera/proxy/meson.build               |  21 +-\n .../proxy/worker/ipa_proxy_linux_worker.cpp   |  90 --\n src/libcamera/proxy/worker/meson.build        |  23 +-\n test/ipa/ipa_interface_test.cpp               |   8 +-\n test/ipa/ipa_wrappers_test.cpp                | 452 ---------\n test/ipa/meson.build                          |   3 +-\n test/ipc/meson.build                          |   3 +-\n test/ipc/unixsocket_ipc.cpp                   | 237 +++++\n .../generated_serializer_test.cpp             | 128 +++\n .../generated_serializer/meson.build          |  49 +\n .../generated_serializer/vimc.mojom           |  24 +\n .../ipa_data_serializer_test.cpp              | 465 +++++++++\n test/serialization/meson.build                |   5 +-\n utils/ipc/generate.py                         |  29 +\n .../libcamera_templates/meson.build           |  11 +\n .../module_ipa_interface.h.tmpl               | 113 +++\n .../module_ipa_proxy.cpp.tmpl                 | 223 +++++\n .../module_ipa_proxy.h.tmpl                   | 121 +++\n .../module_ipa_proxy_worker.cpp.tmpl          | 224 +++++\n .../module_ipa_serializer.h.tmpl              |  47 +\n .../libcamera_templates/proxy_functions.tmpl  | 204 ++++\n .../libcamera_templates/serializer.tmpl       | 289 ++++++\n utils/ipc/generators/meson.build              |   3 +\n .../generators/mojom_libcamera_generator.py   | 487 +++++++++\n utils/ipc/meson.build                         |  13 +\n utils/ipc/parser.py                           |  20 +\n 72 files changed, 5477 insertions(+), 2655 deletions(-)\n create mode 100644 Documentation/guides/ipa.rst\n delete mode 100644 include/libcamera/internal/ipa_context_wrapper.h\n create mode 100644 include/libcamera/internal/ipa_data_serializer.h\n create mode 100644 include/libcamera/internal/ipc_pipe.h\n create mode 100644 include/libcamera/internal/ipc_pipe_unixsocket.h\n create mode 100644 include/libcamera/ipa/core.mojom\n create mode 100644 include/libcamera/ipa/raspberrypi.mojom\n create mode 100644 include/libcamera/ipa/rkisp1.mojom\n create mode 100644 include/libcamera/ipa/vimc.mojom\n delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.cpp\n delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.h\n delete mode 100644 src/libcamera/ipa_context_wrapper.cpp\n create mode 100644 src/libcamera/ipa_data_serializer.cpp\n create mode 100644 src/libcamera/ipc_pipe.cpp\n create mode 100644 src/libcamera/ipc_pipe_unixsocket.cpp\n delete mode 100644 src/libcamera/proxy/ipa_proxy_linux.cpp\n delete mode 100644 src/libcamera/proxy/ipa_proxy_thread.cpp\n delete mode 100644 src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp\n delete mode 100644 test/ipa/ipa_wrappers_test.cpp\n create mode 100644 test/ipc/unixsocket_ipc.cpp\n create mode 100644 test/serialization/generated_serializer/generated_serializer_test.cpp\n create mode 100644 test/serialization/generated_serializer/meson.build\n create mode 100644 test/serialization/generated_serializer/vimc.mojom\n create mode 100644 test/serialization/ipa_data_serializer_test.cpp\n create mode 100755 utils/ipc/generate.py\n create mode 100644 utils/ipc/generators/libcamera_templates/meson.build\n create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n create mode 100644 utils/ipc/generators/meson.build\n create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n create mode 100755 utils/ipc/parser.py"}