[libcamera-devel,v4.1,36/37] Documentation: Add IPA writers guide

10409 diff mbox series

diff --git a/Documentation/guides/ipa.rst b/Documentation/guides/ipa.rst
new file mode 100644
index 00000000..575f3ca1
--- /dev/null
+++ b/Documentation/guides/ipa.rst
@@ -0,0 +1,456 @@ 
+.. SPDX-License-Identifier: CC-BY-SA-4.0
+
+IPA Writers Guide
+=================
+
+IPA are Image Processing Algorithm modules. They provide functionality that
+the pipeline handler can use for image processing.
+
+This guide so far only covers definition the IPA interface, and how to plumb
+the connection between the pipeline handler and the IPA.
+
+The IPA interface and protocol
+------------------------------
+
+The IPA interface defines the interface between the pipeline handler and the
+IPA. Specifically, it defines the functions that the IPA exposes that the
+pipeline handler can call, and the Signals that the pipeline handler can
+connect to to receive data from the IPA asyncrhonously. In addition, it
+contains any custom data structures that the pipeline handler and IPA may
+pass to each other.
+
+The IPA protocol refers to the agreement between the pipeline handler and the
+IPA regarding the expected response(s) from the IPA for given calls to the IPA.
+This protocol doesn't need to be declared anywhere in code, but it would be
+useful to document it, as there may be many different IPAs for one pipeline
+handler.
+
+The IPA interface must be defined in a mojom file. The interface includes:
+- the functions that the pipeline handler can call from the IPA
+- Signals in the pipeline handler that the IPA can emit
+- any data structures that are to be passed between the pipeline handler and
+  the IPA
+All IPA of a given pipeline handler use the same IPA interface. The IPA
+interface definition is thus likely to be written by the pipeline handler
+author, based on how they imagine the pipeline handler will interact with
+the IPA.
+
+The entire IPA interface, including the functions, Signals, and any custom
+structs shall be defined in in a file named {pipeline_name}.mojom under
+include/libcamera/ipa/ using the mojo interface definition language (IDL). This
+will be covered in detail in the following sections.
+
+Namespacing
+-----------
+
+It is recommended to use namespacing, to avoid potential collisions with
+libcamera types. Use mojo's module directive for this.
+
+It must be the first meaningful line in the mojo data definition file, for
+example (defining a raspberry pi IPA):
+
+.. code-block:: C++
+        module ipa.rpi;
+
+This will become the ipa::rpi namespace in C++ code.
+
+Data containers
+---------------
+
+Since the data passed between the pipeline handler and the IPA must support
+serialization, any custom data containers must be defined with the mojo IDL.
+
+The following list of libcamera objects are supported in the interface
+definition, and may be used as function parameter types or struct field types:
+
+- CameraSensorInfo
+- ControlInfoMap
+- ControlList
+- FileDescriptor
+- IPABuffer
+- IPASettings
+- IPAStream
+
+To use them, core.mojom must be included in the mojo data definition file:
+
+.. code-block:: C++
+        import "include/libcamera/ipa/core.mojom";
+
+Other custom structs may be defined and used as well. There is no requirement
+that they must be defined before usage. enums and structs are supported.
+
+The following is an example of a definition of an enum, for the purpose of
+being used as flags:
+
+.. code-block:: C++
+        enum ConfigParameters {
+        	RPI_IPA_CONFIG_LS_TABLE = 0x01,
+        	RPI_IPA_CONFIG_STAGGERED_WRITE = 0x02,
+        	RPI_IPA_CONFIG_SENSOR = 0x04,
+        	RPI_IPA_CONFIG_DROP_FRAMES = 0x08,
+        };
+
+The following is an example of a definition of a struct:
+
+.. code-block:: C++
+        struct ConfigInput {
+        	uint32 op;
+        	uint32 transform;
+        	FileDescriptor lsTableHandle;
+        	int32 lsTableHandleStatic = -1;
+                map<uint32, IPAStream> streamConfig;
+                array<IPABuffer> buffers;
+        };
+
+This example has some special things about it. First of all, it uses the
+FileDescriptor data type. This type must be used to ensure that the file
+descriptor that it contains is translated property across the IPC boundary
+(when the IPA is in an isolated process).
+
+This does mean that if the file descriptor should be sent without being
+translated (for example, for the IPA to tell the pipeline handler which
+fd *that the pipeline handler holds* to act on), then it must be in a
+regular int32 type.
+
+This example also illustrates that struct fields may have default values, as
+is assigned to lsTableHandleStatic. This is the value that the field will
+take when the struct is constructed with the default constructor.
+
+Arrays and maps are supported as well. They are translated to C++ vectors and
+maps, respectively. The members of the arrays and maps are embedded, and cannot
+be const.
+
+Note that nullable fields, static-length arrays, handles, and unions, which
+are supported by mojo, are not supported by our code generator.
+
+TODO what about versioning, and numbered fields?
+
+The Main IPA interface
+----------------------
+
+The IPA interface is split in two parts, the Main IPA interface, which
+describes the functions that the pipeline handler can call from the IPA,
+and the Event IPA interface, which describes the Signals in the pipeline
+handler that the IPA can emit. Both must be defined. This section focuses
+on the Main IPA interface.
+
+The main interface must be named as IPA{pipeline_name}Interface.
+
+At minimum, the following three functions must be present (and implemented):
+- init(IPASettings settings) => (int32 ret);
+- start() => (int32 ret);
+- stop();
+
+All three of these functions are synchronous.
+
+All input parameters will become const references, except for arithmetic types,
+which will be passed by value. Output parameters will become pointers, unless
+there is only one primitive output parameter, in which case it will become a
+a regular return value.
+
+const is not allowed inside of arrays and maps. mojo arrays will become C++
+std::vector<>.
+
+By default, all methods defined in the main interface are synchronous. This
+means that in the case of IPC (ie. isolated IPA), the function call will not
+return until the return value or output parameters are ready. To specify an
+asynchronous function, the [async] attribute can be used. Asynchronous
+methods must not have any return value or output parameters, since in the
+case of IPC the call needs to return immediately.
+
+It is also possible that the IPA will not be run in isolation. In this case,
+the IPA thread will not exist until start() is called. This means that in the
+case of no isolation, asynchronous calls cannot be made before start(). Since
+the IPA interface must be the same regardless of isolation, the same
+restriction applies to the case of isolation, and any function that will be
+called before start() must be synchronous.
+
+In addition, any call made after start() and before stop() must be
+asynchronous. The motivation for this is to avoid damaging real time
+performance of the pipeline handler. If the pipeline handler wants some data
+from the IPA, the IPA should return the data asynchronously via an event
+(see "The Event IPA interface").
+
+In addition the following must be defined in {pipeline_name}.h in the
+libcamera namespace:
+
+.. code-block:: C++
+   static ControlInfoMap {pipeline_name}::controls;
+
+It may be empty. This will be the default ControlInfoMap used when
+serializing and deserializing any ControlLists if it is not specified
+in the ControlList.
+
+The following is an example of a main interface definition:
+
+.. code-block:: C++
+        interface IPARPiInterface {
+        	init(IPASettings settings) => (int32 ret);
+        	start() => (int32 ret);
+        	stop();
+        
+        	configure(CameraSensorInfo sensorInfo,
+        		  map<uint32, IPAStream> streamConfig,
+        		  map<uint32, ControlInfoMap> entityControls,
+        		  ConfigInput ipaConfig)
+        		=> (ConfigOutput results);
+        
+        	mapBuffers(array<IPABuffer> buffers);
+        	unmapBuffers(array<uint32> ids);
+        
+        	[async] signalStatReady(uint32 bufferId);
+        	[async] signalQueueRequest(ControlList controls);
+        	[async] signalIspPrepare(IspPreparePayload data);
+        };
+
+
+The first three functions are the required functions. Functions do not need to
+have return values, like stop(), mapBuffers(), and unmapBuffers(). In the case
+of asynchronous functions, as explained before, they *must not* have return
+values.
+
+The Event IPA interface
+-----------------------
+
+The event IPA interface describes the Signals in the pipeline handler that the
+IPA can emit. It must be defined. If there are no event functions, then it may
+be empty. These emissions are meant to notify the pipeline handler of some
+event, such as reqeust data is ready, and *must not* be used to drive the
+camera pipeline from the IPA.
+
+The event interface must be named as IPA{pipeline_name}EventInterface.
+
+Methods defined in the event interface are implictly asynchronous.
+They thus cannot return any value. Specifying the [async] tag is not
+necessary.
+
+Methods defined in the event interface will become Signals in the IPA
+interface. The IPA can emit signals, while the pipeline handler can connect
+slots to them.
+
+The following is an example of an event interface definition:
+
+.. code-block:: C++
+        interface IPARPiEventInterface {
+        	statsMetadataComplete(uint32 bufferId, ControlList controls);
+        	runIsp(uint32 bufferId);
+        	embeddedComplete(uint32 bufferId);
+        	setIsp(ControlList controls);
+        	setStaggered(ControlList controls);
+        };
+
+Compiling the IPA interface
+---------------------------
+
+After the IPA interface is defined in include/libcamera/ipa/{pipeline_name}.mojom,
+and entry for it must be added in meson so that it can be compiled. The filename
+must be added to the ipa_mojom_files object in include/libcamera/ipa/meson.build.
+
+For example, adding the raspberrypi.mojom file to meson:
+
+.. code-block:: meson
+        ipa_mojom_files = [
+            'raspberrypi.mojom',
+        ]
+
+This will cause the mojo data definition file to be compiled. Specifically, it
+generated five files:
+- a header describing the custom data structures, and the complete IPA interface
+- a serializer implementing de/serialization for the custom data structures
+- a proxy header describing a specialized IPA proxy
+- a proxy source implementing the IPA proxy
+- a proxy worker source implementing the other end of the IPA proxy
+
+The pipeline handler and the IPA only require the header and the proxy header.
+The serializer is only used internally by the proxy.
+
+Using the custom data structures
+--------------------------------
+
+To use the custom data structures that are defined in the mojo data definition
+file, the follow header must be included:
+
+.. code-block:: C++
+   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
+
+The POD types of the structs simply become their C++ counterparts, eg. uint32
+in mojo will become uint32_t in C++. mojo map becomes C++ std::map, and mojo
+array becomes C++ std::vector. All members of maps and vectors are embedded,
+and are not pointers. The members cannot be const.
+
+All fields of structs are the name as specified in the data definition file,
+with an underscore at the end. For example, the following struct as defined
+in the mojo file:
+
+.. code-block:: C++
+   struct StaggeredWritePayload {
+        uint32 gainDelay = 1;
+        uint32 exposureDelay;
+        uint32 sensorMetadata;
+   };
+
+Will become this in C++:
+
+.. code-block:: C++
+   struct StaggeredWritePayload {
+        uint32_t gainDelay_;
+        uint32_t exposureDelay_;
+        uint32_t sensorMetadata_;
+   };
+
+The generated structs will also have two constructors, a constructor that
+fills all fields with the default values, and a second constructor that takes
+a value for every field. The default value constructor will fill in the fields
+with the specified default value, if it exists. In the above example, gainDelay_
+will be initialized to 1. If no default value is specified, then it will be
+filled in as zero (or -1 for a FileDescriptor type).
+
+All fields and constructors/deconstructors in these generated structs are public.
+
+Using the IPA interface (pipeline handler)
+-----------------------------------------
+
+The following headers are necessary to use an IPA in the pipeline handler
+(with raspberrypi as an example):
+
+.. code-block:: C++
+   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
+   #include <libcamera/ipa/ipa_proxy_raspberrypi.h>
+
+The first header includes definitions of the custom data structures, and
+the definition of the complete IPA interface (including both the Main and
+the Event IPA interfaces). The name of the header file is comes from the name
+of the mojom file, which in this case was raspberrypi.mojom.
+
+The second header inclues the definition of the specialized IPA proxy. It
+exposes the complete IPA interface. We will see how to use it in this section.
+
+In the pipeline handler, we first need to construct a specialized IPA proxy.
+From the point of view of the pipeline hander, this is the object that is the
+IPA.
+
+To do so, we invoke the IPAManager:
+
+.. code-block:: C++
+        std::unique_ptr<ipa::rpi::IPAProxyRPi> ipa_ =
+                IPAManager::createIPA<ipa::rpi::IPAProxyRPi>(pipe_, 1, 1);
+
+The ipa::rpi namespace comes from the namespace that we defined in the mojo
+data definition file, in the "Namespacing" section. The name of the proxy,
+IPAProxyRPi, comes from the name given to the main IPA interface,
+IPARPiInterface, in the "The Main IPA interface" section.
+
+The return value of IPAManager::createIPA shall be error-checked, to confirm
+that the returned pointer is not a nullptr.
+
+After this, before initializing the IPA, slots should be connected to all of
+the IPA's Signals, as defined in the Event IPA interface:
+
+.. code-block:: C++
+	ipa_->statsMetadataComplete.connect(this, &RPiCameraData::statsMetadataComplete);
+	ipa_->runIsp.connect(this, &RPiCameraData::runIsp);
+	ipa_->embeddedComplete.connect(this, &RPiCameraData::embeddedComplete);
+	ipa_->setIsp.connect(this, &RPiCameraData::setIsp);
+	ipa_->setStaggered.connect(this, &RPiCameraData::setStaggered);
+
+The slot functions have a function signature based on the function definition
+in the Event IPA interface. All plain old data (POD) types are as-is (with
+their C++ versions, eg. uint32 -> uint32_t), and all structs are const references.
+
+For example, for the following entry in the Event IPA interface:
+
+.. code-block:: C++
+   statsMetadataComplete(uint32 bufferId, ControlList controls);
+
+A function with the following function signature shall be connected to the
+signal:
+
+.. code-block:: C++
+   statsMetadataComplete(uint32_t bufferId, const ControlList &controls);
+
+After connecting the slots to the signals, the IPA should be initialized
+(fill in settings accordingly):
+
+.. code-block:: C++
+   IPASettings settings{};
+   ipa_->init(settings);
+
+At this point, any IPA functions that were defined in the Main IPA interface
+can be called as if they were regular member functions, for example:
+
+.. code-block:: C++
+   ipa_->start();
+   ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result);
+   ipa_->signalStatReady(RPi::BufferMask::STATS | static_cast<unsigned int>(index));
+
+Remember that any functions designated as asynchronous *must not* be called
+before start().
+
+TODO anything special about start() and stop() ?
+
+Using the IPA interface (IPA)
+-----------------------------
+
+The following header is necessary to implement an IPA (with raspberrypi as
+an example):
+
+.. code-block:: C++
+   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
+
+This header includes definitions of the custom data structures, and
+the definition of the complete IPA interface (including both the Main and
+the Event IPA interfaces). The name of the header file is comes from the name
+of the mojom file, which in this case was raspberrypi.mojom.
+
+The IPA must implement the IPA interface class that is defined in the header.
+In the case of our example, that is ipa::rpi::IPARPiInterface. The ipa::rpi
+namespace comes from the namespace that we defined in the mojo data definition
+file, in the "Namespacing" section. The name of the interface is the same as
+the name given to the Main IPA interface.
+
+The function signature rules are the same as for the slots in the pipeline
+handler side; PODs are passed by value, and structs are passed by const
+reference. For the Main IPA interface, output values are also allowed (only
+for synchronous calls), so there may be output parameters as well. If the
+output parameter is a single POD it will be returned by value, otherwise
+(multiple PODs or struct(s)) it will be returned by output parameter pointers.
+
+For example, for the following function specification in the Main IPA interface
+definition:
+
+.. code-block:: C++
+   configure(CameraSensorInfo sensorInfo,
+             uint32 exampleNumber,
+             map<uint32, IPAStream> streamConfig,
+             map<uint32, ControlInfoMap> entityControls,
+             ConfigInput ipaConfig)
+   => (ConfigOutput results);
+
+We will need to implement a function with the following function signature:
+
+.. code-block:: C++
+
+	void configure(const CameraSensorInfo &sensorInfo,
+                       uint32_t exampleNumber,
+		       const std::map<unsigned int, IPAStream> &streamConfig,
+		       const std::map<unsigned int, ControlInfoMap> &entityControls,
+		       const ipa::rpi::ConfigInput &data,
+		       ipa::rpi::ConfigOutput *response);
+
+The return value is void, because the output parameter is not a single POD.
+Instead, it becomes an output parameter pointer. The non-POD input parameters
+become const references, and the POD input parameter is passed by value.
+
+At any time (though usually only in response to an IPA call), the IPA may send
+data to the pipeline handler by emitting signals. These signals are defined
+in the C++ IPA interface class (which is in the generated and included header).
+
+For example, for the following function defined in the Event IPA interface:
+
+.. code-block:: C++
+   statsMetadataComplete(uint32 bufferId, ControlList controls);
+
+We can emit a signal like so:
+
+.. code-block:: C++
+   statsMetadataComplete.emit(bufferId & RPi::BufferMask::ID, libcameraMetadata_);
diff --git a/Documentation/index.rst b/Documentation/index.rst
index ff697d4f..8bc8922e 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -16,4 +16,5 @@ 
    Developer Guide <guides/introduction>
    Application Writer's Guide <guides/application-developer>
    Pipeline Handler Writer's Guide <guides/pipeline-handler>
+   IPA Writer's guide <guides/ipa>
    Tracing guide <guides/tracing>
diff --git a/Documentation/meson.build b/Documentation/meson.build
index 59107151..17f3b9d7 100644
--- a/Documentation/meson.build
+++ b/Documentation/meson.build
@@ -52,6 +52,7 @@  if sphinx.found()
         'docs.rst',
         'index.rst',
         'guides/introduction.rst',
+        'guides/ipa.rst',
         'guides/application-developer.rst',
         'guides/pipeline-handler.rst',
         'guides/tracing.rst',