[libcamera-devel,v11] Documentation: Add IPA writers guide

12131 diff mbox series

Add a guide about writing IPAs.

Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

---
Changes in v11:
- expand introduction and concepts that should have been pre-explained

Changes in v10:
- fix typos and indentation and remove todos

Changes in v9:
- update documentation to include customizable init() and direct return
  of int32

No change in v8

Changes in v8:
- fix bullet points
- update wording about struct field names
- fix typos

Changes in v7:
- fix TODO syntax
- update the generated struct fiels
  - no more postfix underscore

Changes in v6:
- namespacing is now required
- start() is now customizable
- {pipeline_name} is no longer required
- fix code block indentations

Changes in v5:
- fix doxygen compile errors
- update example struct names from raspberry pi changes
- add todo for restricting pre-start() to sync and post-start() to async

Changes in v4.1:
- Add section on namespacing, custom data structures, compiling, and
  usage of the data structures and interface
- Add examples to the main ipa interface and event ipa interface
  sections

New in v4
---
 Documentation/guides/ipa.rst | 511 +++++++++++++++++++++++++++++++++++
 Documentation/index.rst      |   1 +
 Documentation/meson.build    |   1 +
 3 files changed, 513 insertions(+)
 create mode 100644 Documentation/guides/ipa.rst

Hey Paul,

On 28.04.2021 18:33, Paul Elder wrote:
>Add a guide about writing IPAs.

Thank you this was a nice read and very detailed as well.

>
>Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
>Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Sebastian Fricke <sebastian.fricke@posteo.net>

>
>---
>Changes in v11:
>- expand introduction and concepts that should have been pre-explained
>
>Changes in v10:
>- fix typos and indentation and remove todos
>
>Changes in v9:
>- update documentation to include customizable init() and direct return
>  of int32
>
>No change in v8
>
>Changes in v8:
>- fix bullet points
>- update wording about struct field names
>- fix typos
>
>Changes in v7:
>- fix TODO syntax
>- update the generated struct fiels
>  - no more postfix underscore
>
>Changes in v6:
>- namespacing is now required
>- start() is now customizable
>- {pipeline_name} is no longer required
>- fix code block indentations
>
>Changes in v5:
>- fix doxygen compile errors
>- update example struct names from raspberry pi changes
>- add todo for restricting pre-start() to sync and post-start() to async
>
>Changes in v4.1:
>- Add section on namespacing, custom data structures, compiling, and
>  usage of the data structures and interface
>- Add examples to the main ipa interface and event ipa interface
>  sections
>
>New in v4
>---
> Documentation/guides/ipa.rst | 511 +++++++++++++++++++++++++++++++++++
> Documentation/index.rst      |   1 +
> Documentation/meson.build    |   1 +
> 3 files changed, 513 insertions(+)
> create mode 100644 Documentation/guides/ipa.rst
>
>diff --git a/Documentation/guides/ipa.rst b/Documentation/guides/ipa.rst
>new file mode 100644
>index 00000000..e53486d1
>--- /dev/null
>+++ b/Documentation/guides/ipa.rst
>@@ -0,0 +1,511 @@
>+.. SPDX-License-Identifier: CC-BY-SA-4.0
>+
>+IPA Writer's Guide
>+==================
>+
>+IPA modules are Image Processing Algorithm modules. They provide functionality
>+that the pipeline handler can use for image processing.
>+
>+This guide covers the definition of the IPA interface, and how to plumb the
>+connection between the pipeline handler and the IPA.
>+
>+The IPA interface and protocol
>+------------------------------
>+
>+The IPA interface defines the interface between the pipeline handler and the
>+IPA. Specifically, it defines the functions that the IPA exposes that the
>+pipeline handler can call, and the signals that the pipeline handler can
>+connect to, in order to receive data from the IPA asynchronously. In addition,
>+it contains any custom data structures that the pipeline handler and IPA may
>+pass to each other.
>+
>+The IPA protocol refers to the agreement between the pipeline handler and the
>+IPA regarding the expected response(s) from the IPA for given calls to the IPA.
>+This protocol doesn't need to be declared anywhere in code, but it shall be
>+documented, as there may be multiple IPA implementations for one pipeline
>+handler.
>+
>+As part of the design of libcamera, IPAs may be isolated in a separate process,
>+or run in the same process but a different thread from libcamera. The pipeline
>+handler and IPA shall not have to change their operation based on whether the
>+IPA is isolated or not, but the possibility of isolation needs to be kept in
>+mind. Therefore all data that is passed between them must be serializable, so
>+they must be defined separately in the `mojo Interface Definition Language`_
>+(IDL), and a code generator will generate headers and serializers corresponding
>+to the definitions. Every interface is defined in a mojom file and includes:
>+
>+- the functions that the pipeline handler can call from the IPA
>+- signals in the pipeline handler that the IPA can emit
>+- any data structures that are to be passed between the pipeline handler and the IPA
>+
>+All IPA modules of a given pipeline handler use the same IPA interface. The IPA
>+interface definition is thus written by the pipeline handler author, based on
>+how they design the interactions between the pipeline handler and the IPA.
>+
>+The entire IPA interface, including the functions, signals, and any custom
>+structs shall be defined in a file named {pipeline_name}.mojom under
>+include/libcamera/ipa/.
>+
>+.. _mojo Interface Definition Language: https://chromium.googlesource.com/chromium/src.git/+/master/mojo/public/tools/bindings/README.md
>+
>+Namespacing
>+-----------
>+
>+To avoid name collisions between data types defined by different IPA interfaces
>+and data types defined by libcamera, each IPA interface must be defined in its
>+own namespace.
>+
>+The namespace is specific with mojo's module directive. It must be the first
>+non-comment line in the mojo data definition file. For example, the Raspberry
>+Pi IPA interface uses:
>+
>+.. code-block:: none
>+
>+        module ipa.rpi;
>+
>+This will become the ipa::rpi namespace in C++ code.
>+
>+Data containers
>+---------------
>+
>+Since the data passed between the pipeline handler and the IPA must support
>+serialization, any custom data containers must be defined with the mojo IDL.
>+
>+The following list of libcamera objects are supported in the interface
>+definition, and may be used as function parameter types or struct field types:
>+
>+- libcamera.CameraSensorInfo
>+- libcamera.ControlInfoMap
>+- libcamera.ControlList
>+- libcamera.FileDescriptor
>+- libcamera.IPABuffer
>+- libcamera.IPASettings
>+- libcamera.IPAStream
>+- libcamera.Point
>+- libcamera.Rectangle
>+- libcamera.Size
>+- libcamera.SizeRange
>+
>+To use them, core.mojom must be included in the mojo data definition file:
>+
>+.. code-block:: none
>+
>+        import "include/libcamera/ipa/core.mojom";
>+
>+Other custom structs may be defined and used as well. There is no requirement
>+that they must be defined before usage. enums and structs are supported.
>+
>+The following is an example of a definition of an enum, for the purpose of
>+being used as flags:
>+
>+.. code-block:: none
>+
>+        enum ConfigParameters {
>+                ConfigLsTable = 0x01,
>+                ConfigStaggeredWrite = 0x02,
>+                ConfigSensor = 0x04,
>+                ConfigDropFrames = 0x08,
>+        };
>+
>+The following is an example of a definition of a struct:
>+
>+.. code-block:: none
>+
>+        struct ConfigInput {
>+                uint32 op;
>+                uint32 transform;
>+                libcamera.FileDescriptor lsTableHandle;
>+                int32 lsTableHandleStatic = -1;
>+                map<uint32, libcamera.IPAStream> streamConfig;
>+                array<libcamera.IPABuffer> buffers;
>+        };
>+
>+This example has some special things about it. First of all, it uses the
>+FileDescriptor data type. This type must be used to ensure that the file
>+descriptor that it contains is translated properly across the IPC boundary
>+(when the IPA is in an isolated process).
>+
>+This does mean that if the file descriptor should be sent without being
>+translated (for example, for the IPA to tell the pipeline handler which
>+fd *that the pipeline handler holds* to act on), then it must be in a
>+regular int32 type.
>+
>+This example also illustrates that struct fields may have default values, as
>+is assigned to lsTableHandleStatic. This is the value that the field will
>+take when the struct is constructed with the default constructor.
>+
>+Arrays and maps are supported as well. They are translated to C++ vectors and
>+maps, respectively. The members of the arrays and maps are embedded, and cannot
>+be const.
>+
>+Note that nullable fields, static-length arrays, handles, and unions, which
>+are supported by mojo, are not supported by our code generator.
>+
>+The Main IPA interface
>+----------------------
>+
>+The IPA interface is split in two parts, the Main IPA interface, which
>+describes the functions that the pipeline handler can call from the IPA,
>+and the Event IPA interface, which describes the signals received by the
>+pipeline handler that the IPA can emit. Both must be defined. This section
>+focuses on the Main IPA interface.
>+
>+The main interface must be named as IPA{pipeline_name}Interface.
>+
>+The functions that the pipeline handler can call from the IPA may be
>+synchronous or asynchronous. Synchronous functions do not return until the IPA
>+returns from the function, while asynchronous functions return immediately
>+without waiting for the IPA to return.
>+
>+At a minimum, the following three functions must be present (and implemented):
>+
>+- init();
>+- start();
>+- stop();
>+
>+All three of these functions are synchronous. The parameters for start() and
>+init() may be customized.
>+
>+A configure() method is recommended. Any ControlInfoMap instances that will be
>+used by the IPA must be sent to the IPA from the pipeline handler, at configure
>+time, for example.
>+
>+All input parameters will become const references, except for arithmetic types,
>+which will be passed by value. Output parameters will become pointers, unless
>+the first output parameter is an int32, or there is only one primitive output
>+parameter, in which case it will become a regular return value.
>+
>+const is not allowed inside of arrays and maps. mojo arrays will become C++
>+std::vector<>.
>+
>+By default, all methods defined in the main interface are synchronous. This
>+means that in the case of IPC (i.e. isolated IPA), the function call will not
>+return until the return value or output parameters are ready. To specify an
>+asynchronous function, the [async] attribute can be used. Asynchronous
>+methods must not have any return value or output parameters, since in the
>+case of IPC the call needs to return immediately.
>+
>+It is also possible that the IPA will not be run in isolation. In this case,
>+the IPA thread will not exist until start() is called. This means that in the
>+case of no isolation, asynchronous calls cannot be made before start(). Since
>+the IPA interface must be the same regardless of isolation, the same
>+restriction applies to the case of isolation, and any function that will be
>+called before start() must be synchronous.
>+
>+In addition, any call made after start() and before stop() must be
>+asynchronous. The motivation for this is to avoid damaging real-time
>+performance of the pipeline handler. If the pipeline handler wants some data
>+from the IPA, the IPA should return the data asynchronously via an event
>+(see "The Event IPA interface").
>+
>+The following is an example of a main interface definition:
>+
>+.. code-block:: none
>+
>+        interface IPARPiInterface {
>+                init(libcamera.IPASettings settings, string sensorName)
>+                        => (int32 ret, bool metadataSupport);
>+                start() => (int32 ret);
>+                stop();
>+
>+                configure(libcamera.CameraSensorInfo sensorInfo,
>+                          map<uint32, libcamera.IPAStream> streamConfig,
>+                          map<uint32, libcamera.ControlInfoMap> entityControls,
>+                          ConfigInput ipaConfig)
>+                        => (int32 ret, ConfigOutput results);
>+
>+                mapBuffers(array<IPABuffer> buffers);
>+                unmapBuffers(array<uint32> ids);
>+
>+                [async] signalStatReady(uint32 bufferId);
>+                [async] signalQueueRequest(libcamera.ControlList controls);
>+                [async] signalIspPrepare(ISPConfig data);
>+        };
>+
>+
>+The first three functions are the required functions. Functions do not need to
>+have return values, like stop(), mapBuffers(), and unmapBuffers(). In the case
>+of asynchronous functions, as explained before, they *must not* have return
>+values.
>+
>+The Event IPA interface
>+-----------------------
>+
>+The event IPA interface describes the signals received by the pipeline handler
>+that the IPA can emit. It must be defined. If there are no event functions,
>+then it may be empty. These emissions are meant to notify the pipeline handler
>+of some event, such as request data is ready, and *must not* be used to drive
>+the camera pipeline from the IPA.
>+
>+The event interface must be named as IPA{pipeline_name}EventInterface.
>+
>+Methods defined in the event interface are implicitly asynchronous.
>+Thus they cannot return any value. Specifying the [async] tag is not
>+necessary.
>+
>+Methods defined in the event interface will become signals in the IPA
>+interface. The IPA can emit signals, while the pipeline handler can connect
>+slots to them.
>+
>+The following is an example of an event interface definition:
>+
>+.. code-block:: none
>+
>+        interface IPARPiEventInterface {
>+                statsMetadataComplete(uint32 bufferId,
>+                                      libcamera.ControlList controls);
>+                runIsp(uint32 bufferId);
>+                embeddedComplete(uint32 bufferId);
>+                setIsp(libcamera.ControlList controls);
>+                setStaggered(libcamera.ControlList controls);
>+        };
>+
>+Compiling the IPA interface
>+---------------------------
>+
>+After the IPA interface is defined in include/libcamera/ipa/{pipeline_name}.mojom,
>+an entry for it must be added in meson so that it can be compiled. The filename
>+must be added to the ipa_mojom_files object in include/libcamera/ipa/meson.build.
>+
>+For example, adding the raspberrypi.mojom file to meson:
>+
>+.. code-block:: none
>+
>+        ipa_mojom_files = [
>+            'raspberrypi.mojom',
>+        ]
>+
>+This will cause the mojo data definition file to be compiled. Specifically, it
>+generates five files:
>+
>+- a header describing the custom data structures, and the complete IPA
>+  interface (at {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_interface.h)
>+
>+- a serializer implementing de/serialization for the custom data structures (at
>+  {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_serializer.h)
>+
>+- a proxy header describing a specialized IPA proxy (at
>+  {$build_dir}/include/libcamera/ipa/{pipeline}_ipa_proxy.h)
>+
>+- a proxy source implementing the IPA proxy (at
>+  {$build_dir}/src/libcamera/proxy/{pipeline}_ipa_proxy.cpp)
>+
>+- a proxy worker source implementing the other end of the IPA proxy (at
>+  {$build_dir}/src/libcamera/proxy/worker/{pipeline}_ipa_proxy_worker.cpp)
>+
>+The IPA proxy serves as the layer between the pipeline handler and the IPA, and
>+handles threading vs isolation transparently. The pipeline handler and the IPA
>+only require the interface header and the proxy header. The serializer is only
>+used internally by the proxy.
>+
>+Using the custom data structures
>+--------------------------------
>+
>+To use the custom data structures that are defined in the mojo data definition
>+file, the following header must be included:
>+
>+.. code-block:: C++
>+
>+   #include <libcamera/ipa/{pipeline_name}_ipa_interface.h>
>+
>+The POD types of the structs simply become their C++ counterparts, eg. uint32
>+in mojo will become uint32_t in C++. mojo map becomes C++ std::map, and mojo
>+array becomes C++ std::vector. All members of maps and vectors are embedded,
>+and are not pointers. The members cannot be const.
>+
>+The names of all the fields of structs can be used in C++ in exactly the same
>+way as they are defined in the data definition file. For example, the following
>+struct as defined in the mojo file:
>+
>+.. code-block:: none
>+
>+   struct SensorConfig {
>+        uint32 gainDelay = 1;
>+        uint32 exposureDelay;
>+        uint32 sensorMetadata;
>+   };
>+
>+Will become this in C++:
>+
>+.. code-block:: C++
>+
>+   struct SensorConfig {
>+        uint32_t gainDelay;
>+        uint32_t exposureDelay;
>+        uint32_t sensorMetadata;
>+   };
>+
>+The generated structs will also have two constructors, a constructor that
>+fills all fields with the default values, and a second constructor that takes
>+a value for every field. The default value constructor will fill in the fields
>+with the specified default value if it exists. In the above example, `gainDelay_`
>+will be initialized to 1. If no default value is specified, then it will be
>+filled in as zero (or -1 for a FileDescriptor type).
>+
>+All fields and constructors/destructors in these generated structs are public.
>+
>+Using the IPA interface (pipeline handler)
>+------------------------------------------
>+
>+The following headers are necessary to use an IPA in the pipeline handler
>+(with raspberrypi as an example):
>+
>+.. code-block:: C++
>+
>+   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
>+   #include <libcamera/ipa/raspberrypi_ipa_proxy.h>
>+
>+The first header includes definitions of the custom data structures, and
>+the definition of the complete IPA interface (including both the Main and
>+the Event IPA interfaces). The name of the header file comes from the name
>+of the mojom file, which in this case was raspberrypi.mojom.
>+
>+The second header includes the definition of the specialized IPA proxy. It
>+exposes the complete IPA interface. We will see how to use it in this section.
>+
>+In the pipeline handler, we first need to construct a specialized IPA proxy.
>+From the point of view of the pipeline hander, this is the object that is the
>+IPA.
>+
>+To do so, we invoke the IPAManager:
>+
>+.. code-block:: C++
>+
>+        std::unique_ptr<ipa::rpi::IPAProxyRPi> ipa_ =
>+                IPAManager::createIPA<ipa::rpi::IPAProxyRPi>(pipe_, 1, 1);
>+
>+The ipa::rpi namespace comes from the namespace that we defined in the mojo
>+data definition file, in the "Namespacing" section. The name of the proxy,
>+IPAProxyRPi, comes from the name given to the main IPA interface,
>+IPARPiInterface, in the "The Main IPA interface" section.
>+
>+The return value of IPAManager::createIPA shall be error-checked, to confirm
>+that the returned pointer is not a nullptr.
>+
>+After this, before initializing the IPA, slots should be connected to all of
>+the IPA's signals, as defined in the Event IPA interface:
>+
>+.. code-block:: C++
>+
>+	ipa_->statsMetadataComplete.connect(this, &RPiCameraData::statsMetadataComplete);
>+	ipa_->runIsp.connect(this, &RPiCameraData::runIsp);
>+	ipa_->embeddedComplete.connect(this, &RPiCameraData::embeddedComplete);
>+	ipa_->setIsp.connect(this, &RPiCameraData::setIsp);
>+	ipa_->setStaggered.connect(this, &RPiCameraData::setStaggered);
>+
>+The slot functions have a function signature based on the function definition
>+in the Event IPA interface. All plain old data (POD) types are as-is (with
>+their C++ versions, eg. uint32 -> uint32_t), and all structs are const references.
>+
>+For example, for the following entry in the Event IPA interface:
>+
>+.. code-block:: none
>+
>+   statsMetadataComplete(uint32 bufferId, ControlList controls);
>+
>+A function with the following function signature shall be connected to the
>+signal:
>+
>+.. code-block:: C++
>+
>+   void statsMetadataComplete(uint32_t bufferId, const ControlList &controls);
>+
>+After connecting the slots to the signals, the IPA should be initialized
>+(using the main interface definition example from earlier):
>+
>+.. code-block:: C++
>+
>+   IPASettings settings{};
>+   bool metadataSupport;
>+   int ret = ipa_->init(settings, "sensor name", &metadataSupport);
>+
>+At this point, any IPA functions that were defined in the Main IPA interface
>+can be called as if they were regular member functions, for example (based on
>+the main interface definition example from earlier):
>+
>+.. code-block:: C++
>+
>+   ipa_->start();
>+   int ret = ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result);
>+   ipa_->signalStatReady(RPi::BufferMask::STATS | static_cast<unsigned int>(index));
>+
>+Remember that any functions designated as asynchronous *must not* be called
>+before start().
>+
>+Notice that for both init() and configure(), the first output parameter is a
>+direct return, since it is an int32, while the other output parameter is a
>+pointer-based output parameter.
>+
>+Using the IPA interface (IPA Module)
>+-----------------------------
>+
>+The following header is necessary to implement an IPA Module (with raspberrypi
>+as an example):
>+
>+.. code-block:: C++
>+
>+   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
>+
>+This header includes definitions of the custom data structures, and
>+the definition of the complete IPA interface (including both the Main and
>+the Event IPA interfaces). The name of the header file comes from the name
>+of the mojom file, which in this case was raspberrypi.mojom.
>+
>+The IPA module must implement the IPA interface class that is defined in the
>+header. In the case of our example, that is ipa::rpi::IPARPiInterface. The
>+ipa::rpi namespace comes from the namespace that we defined in the mojo data
>+definition file, in the "Namespacing" section. The name of the interface is the
>+same as the name given to the Main IPA interface.
>+
>+The function signature rules are the same as for the slots in the pipeline
>+handler side; PODs are passed by value, and structs are passed by const
>+reference. For the Main IPA interface, output values are also allowed (only
>+for synchronous calls), so there may be output parameters as well. If the
>+first output parameter is a POD it will be returned by value, otherwise
>+it will be returned by an output parameter pointer. The second and any other
>+output parameters will also be returned by output parameter pointers.
>+
>+For example, for the following function specification in the Main IPA interface
>+definition:
>+
>+.. code-block:: none
>+
>+   configure(libcamera.CameraSensorInfo sensorInfo,
>+             uint32 exampleNumber,
>+             map<uint32, libcamera.IPAStream> streamConfig,
>+             map<uint32, libcamera.ControlInfoMap> entityControls,
>+             ConfigInput ipaConfig)
>+   => (int32 ret, ConfigOutput results);
>+
>+We will need to implement a function with the following function signature:
>+
>+.. code-block:: C++
>+
>+        int configure(const CameraSensorInfo &sensorInfo,
>+                      uint32_t exampleNumber,
>+                      const std::map<unsigned int, IPAStream> &streamConfig,
>+                      const std::map<unsigned int, ControlInfoMap> &entityControls,
>+                      const ipa::rpi::ConfigInput &data,
>+                      ipa::rpi::ConfigOutput *response);
>+
>+The return value is int, because the first output parameter is int32.  The rest
>+of the output parameters (in this case, only response) become output parameter
>+pointers. The non-POD input parameters become const references, and the POD
>+input parameter is passed by value.
>+
>+At any time after start() and before stop() (though usually only in response to
>+an IPA call), the IPA may send data to the pipeline handler by emitting
>+signals. These signals are defined in the C++ IPA interface class (which is in
>+the generated and included header).
>+
>+For example, for the following function defined in the Event IPA interface:
>+
>+.. code-block:: none
>+
>+   statsMetadataComplete(uint32 bufferId, libcamera.ControlList controls);
>+
>+We can emit a signal like so:
>+
>+.. code-block:: C++
>+
>+   statsMetadataComplete.emit(bufferId & RPi::BufferMask::ID, libcameraMetadata_);
>diff --git a/Documentation/index.rst b/Documentation/index.rst
>index 7e3fe2f6..1f4fc485 100644
>--- a/Documentation/index.rst
>+++ b/Documentation/index.rst
>@@ -17,6 +17,7 @@
>    Developer Guide <guides/introduction>
>    Application Writer's Guide <guides/application-developer>
>    Pipeline Handler Writer's Guide <guides/pipeline-handler>
>+   IPA Writer's guide <guides/ipa>
>    Tracing guide <guides/tracing>
>    Environment variables <environment_variables>
>    Sensor driver requirements <sensor_driver_requirements>
>diff --git a/Documentation/meson.build b/Documentation/meson.build
>index 9950465d..8cf68a07 100644
>--- a/Documentation/meson.build
>+++ b/Documentation/meson.build
>@@ -54,6 +54,7 @@ if sphinx.found()
>         'environment_variables.rst',
>         'guides/application-developer.rst',
>         'guides/introduction.rst',
>+        'guides/ipa.rst',
>         'guides/pipeline-handler.rst',
>         'guides/tracing.rst',
>         'index.rst',
>-- 
>2.27.0
>
>_______________________________________________
>libcamera-devel mailing list
>libcamera-devel@lists.libcamera.org
>https://lists.libcamera.org/listinfo/libcamera-devel