{"id":11232,"url":"https://patchwork.libcamera.org/api/patches/11232/?format=json","web_url":"https://patchwork.libcamera.org/patch/11232/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210211072121.35229-4-paul.elder@ideasonboard.com>","date":"2021-02-11T07:21:20","name":"[libcamera-devel,v7,3/4] Documentation: Add IPA writers guide","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"aed757557801af98aedf7c740e35815aac054d2a","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/?format=json","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/11232/mbox/","series":[{"id":1677,"url":"https://patchwork.libcamera.org/api/series/1677/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1677","date":"2021-02-11T07:21:17","name":"IPA isolation: Part 3: Tests and documentation","version":7,"mbox":"https://patchwork.libcamera.org/series/1677/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/11232/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/11232/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3136ABD160\n\tfor ;\n\tThu, 11 Feb 2021 07:21:39 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 00EEA61692;\n\tThu, 11 Feb 2021 08:21:39 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id ABD5261662\n\tfor ;\n\tThu, 11 Feb 2021 08:21:36 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2400:4051:61:600:2c71:1b79:d06d:5032])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DE39BBB8;\n\tThu, 11 Feb 2021 08:21:34 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"rQEn6qin\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1613028096;\n\tbh=jjRyI9lZ4HXg+4teZmluz245yx6LajE+RxqUVvk95ew=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=rQEn6qinEwWThUuJC0oVlQRfhOQhouojNQM+2Hcqa1uPgs73bNOvJi/DwYlY/r1DR\n\t47qQ53nS52oAMb0uQcGOHFrg8qdCg2p0Ke2EVXjC2mDVO37FzSNkPclGP33H5BMlq0\n\tt8len3ED/axl+RFkxN0iY+47jhK8EcPidjUBBMzw=","From":"Paul Elder ","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 11 Feb 2021 16:21:20 +0900","Message-Id":"<20210211072121.35229-4-paul.elder@ideasonboard.com>","X-Mailer":"git-send-email 2.27.0","In-Reply-To":"<20210211072121.35229-1-paul.elder@ideasonboard.com>","References":"<20210211072121.35229-1-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v7 3/4] Documentation: Add IPA writers\n\tguide","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Add a guide about writing IPAs.\n\nSigned-off-by: Paul Elder \n\n---\nChanges in v7:\n- fix TODO syntax\n- update the generated struct fiels\n - no more postfix underscore\n\nChanges in v6:\n- namespacing is now required\n- start() is now customizable\n- {pipeline_name} is no longer required\n- fix code block indentations\n\nChanges in v5:\n- fix doxygen compile errors\n- update example struct names from raspberry pi changes\n- add todo for restricting pre-start() to sync and post-start() to async\n\nChanges in v4.1:\n- Add section on namespacing, custom data structures, compiling, and\n usage of the data structures and interface\n- Add examples to the main ipa interface and event ipa interface\n sections\n\nNew in v4\n---\n Documentation/guides/ipa.rst | 474 +++++++++++++++++++++++++++++++++++\n Documentation/index.rst | 1 +\n Documentation/meson.build | 1 +\n 3 files changed, 476 insertions(+)\n create mode 100644 Documentation/guides/ipa.rst","diff":"diff --git a/Documentation/guides/ipa.rst b/Documentation/guides/ipa.rst\nnew file mode 100644\nindex 00000000..a1050a03\n--- /dev/null\n+++ b/Documentation/guides/ipa.rst\n@@ -0,0 +1,474 @@\n+.. SPDX-License-Identifier: CC-BY-SA-4.0\n+\n+IPA Writers Guide\n+=================\n+\n+IPA are Image Processing Algorithm modules. They provide functionality that\n+the pipeline handler can use for image processing.\n+\n+This guide so far only covers definition the IPA interface, and how to plumb\n+the connection between the pipeline handler and the IPA.\n+\n+The IPA interface and protocol\n+------------------------------\n+\n+The IPA interface defines the interface between the pipeline handler and the\n+IPA. Specifically, it defines the functions that the IPA exposes that the\n+pipeline handler can call, and the Signals that the pipeline handler can\n+connect to to receive data from the IPA asyncrhonously. In addition, it\n+contains any custom data structures that the pipeline handler and IPA may\n+pass to each other.\n+\n+The IPA protocol refers to the agreement between the pipeline handler and the\n+IPA regarding the expected response(s) from the IPA for given calls to the IPA.\n+This protocol doesn't need to be declared anywhere in code, but it would be\n+useful to document it, as there may be many different IPAs for one pipeline\n+handler.\n+\n+The IPA interface must be defined in a mojom file. The interface includes:\n+- the functions that the pipeline handler can call from the IPA\n+- Signals in the pipeline handler that the IPA can emit\n+- any data structures that are to be passed between the pipeline handler and the IPA\n+All IPA of a given pipeline handler use the same IPA interface. The IPA\n+interface definition is thus likely to be written by the pipeline handler\n+author, based on how they imagine the pipeline handler will interact with\n+the IPA.\n+\n+The entire IPA interface, including the functions, Signals, and any custom\n+structs shall be defined in in a file named {pipeline_name}.mojom under\n+include/libcamera/ipa/ using the mojo interface definition language (IDL). This\n+will be covered in detail in the following sections.\n+\n+Namespacing\n+-----------\n+\n+Namespacing is required, to avoid potential collisions with libcamera types.\n+Use mojo's module directive for this.\n+\n+It must be the first meaningful line in the mojo data definition file, for\n+example (defining a raspberry pi IPA):\n+\n+.. code-block:: none\n+\n+ module ipa.rpi;\n+\n+This will become the ipa::rpi namespace in C++ code.\n+\n+Data containers\n+---------------\n+\n+Since the data passed between the pipeline handler and the IPA must support\n+serialization, any custom data containers must be defined with the mojo IDL.\n+\n+The following list of libcamera objects are supported in the interface\n+definition, and may be used as function parameter types or struct field types:\n+\n+- CameraSensorInfo\n+- ControlInfoMap\n+- ControlList\n+- FileDescriptor\n+- IPABuffer\n+- IPASettings\n+- IPAStream\n+\n+To use them, core.mojom must be included in the mojo data definition file:\n+\n+.. code-block:: none\n+\n+ import \"include/libcamera/ipa/core.mojom\";\n+\n+Other custom structs may be defined and used as well. There is no requirement\n+that they must be defined before usage. enums and structs are supported.\n+\n+The following is an example of a definition of an enum, for the purpose of\n+being used as flags:\n+\n+.. code-block:: none\n+\n+ enum ConfigParameters {\n+ ConfigLsTable = 0x01,\n+ ConfigStaggeredWrite = 0x02,\n+ ConfigSensor = 0x04,\n+ ConfigDropFrames = 0x08,\n+ };\n+\n+The following is an example of a definition of a struct:\n+\n+.. code-block:: none\n+\n+ struct ConfigInput {\n+ uint32 op;\n+ uint32 transform;\n+ FileDescriptor lsTableHandle;\n+ int32 lsTableHandleStatic = -1;\n+ map streamConfig;\n+ array buffers;\n+ };\n+\n+This example has some special things about it. First of all, it uses the\n+FileDescriptor data type. This type must be used to ensure that the file\n+descriptor that it contains is translated property across the IPC boundary\n+(when the IPA is in an isolated process).\n+\n+This does mean that if the file descriptor should be sent without being\n+translated (for example, for the IPA to tell the pipeline handler which\n+fd *that the pipeline handler holds* to act on), then it must be in a\n+regular int32 type.\n+\n+This example also illustrates that struct fields may have default values, as\n+is assigned to lsTableHandleStatic. This is the value that the field will\n+take when the struct is constructed with the default constructor.\n+\n+Arrays and maps are supported as well. They are translated to C++ vectors and\n+maps, respectively. The members of the arrays and maps are embedded, and cannot\n+be const.\n+\n+Note that nullable fields, static-length arrays, handles, and unions, which\n+are supported by mojo, are not supported by our code generator.\n+\n+TODO: what about versioning, and numbered fields?\n+\n+The Main IPA interface\n+----------------------\n+\n+The IPA interface is split in two parts, the Main IPA interface, which\n+describes the functions that the pipeline handler can call from the IPA,\n+and the Event IPA interface, which describes the Signals in the pipeline\n+handler that the IPA can emit. Both must be defined. This section focuses\n+on the Main IPA interface.\n+\n+The main interface must be named as IPA{pipeline_name}Interface.\n+\n+At minimum, the following three functions must be present (and implemented):\n+- init(IPASettings settings) => (int32 ret);\n+- start();\n+- stop();\n+\n+All three of these functions are synchronous.\n+\n+TODO: Restrict pre-start to synchronous, and post-start to asynchronous\n+\n+The parameters for start() may be customized.\n+\n+A configure() method is recommended. Any ContolInfoMap instances that will be\n+used by the IPA must be sent to the IPA from the pipeline handler, at configure\n+time, for example.\n+\n+All input parameters will become const references, except for arithmetic types,\n+which will be passed by value. Output parameters will become pointers, unless\n+there is only one primitive output parameter, in which case it will become a\n+a regular return value.\n+\n+const is not allowed inside of arrays and maps. mojo arrays will become C++\n+std::vector<>.\n+\n+By default, all methods defined in the main interface are synchronous. This\n+means that in the case of IPC (ie. isolated IPA), the function call will not\n+return until the return value or output parameters are ready. To specify an\n+asynchronous function, the [async] attribute can be used. Asynchronous\n+methods must not have any return value or output parameters, since in the\n+case of IPC the call needs to return immediately.\n+\n+It is also possible that the IPA will not be run in isolation. In this case,\n+the IPA thread will not exist until start() is called. This means that in the\n+case of no isolation, asynchronous calls cannot be made before start(). Since\n+the IPA interface must be the same regardless of isolation, the same\n+restriction applies to the case of isolation, and any function that will be\n+called before start() must be synchronous.\n+\n+In addition, any call made after start() and before stop() must be\n+asynchronous. The motivation for this is to avoid damaging real time\n+performance of the pipeline handler. If the pipeline handler wants some data\n+from the IPA, the IPA should return the data asynchronously via an event\n+(see \"The Event IPA interface\").\n+\n+The following is an example of a main interface definition:\n+\n+.. code-block:: none\n+\n+ interface IPARPiInterface {\n+ init(IPASettings settings) => (int32 ret);\n+ start() => (int32 ret);\n+ stop();\n+\n+ configure(CameraSensorInfo sensorInfo,\n+ map streamConfig,\n+ map entityControls,\n+ ConfigInput ipaConfig)\n+ => (ConfigOutput results);\n+\n+ mapBuffers(array buffers);\n+ unmapBuffers(array ids);\n+\n+ [async] signalStatReady(uint32 bufferId);\n+ [async] signalQueueRequest(ControlList controls);\n+ [async] signalIspPrepare(ISPConfig data);\n+ };\n+\n+\n+The first three functions are the required functions. Functions do not need to\n+have return values, like stop(), mapBuffers(), and unmapBuffers(). In the case\n+of asynchronous functions, as explained before, they *must not* have return\n+values.\n+\n+The Event IPA interface\n+-----------------------\n+\n+The event IPA interface describes the Signals in the pipeline handler that the\n+IPA can emit. It must be defined. If there are no event functions, then it may\n+be empty. These emissions are meant to notify the pipeline handler of some\n+event, such as reqeust data is ready, and *must not* be used to drive the\n+camera pipeline from the IPA.\n+\n+The event interface must be named as IPA{pipeline_name}EventInterface.\n+\n+Methods defined in the event interface are implictly asynchronous.\n+They thus cannot return any value. Specifying the [async] tag is not\n+necessary.\n+\n+Methods defined in the event interface will become Signals in the IPA\n+interface. The IPA can emit signals, while the pipeline handler can connect\n+slots to them.\n+\n+The following is an example of an event interface definition:\n+\n+.. code-block:: none\n+\n+ interface IPARPiEventInterface {\n+ statsMetadataComplete(uint32 bufferId, ControlList controls);\n+ runIsp(uint32 bufferId);\n+ embeddedComplete(uint32 bufferId);\n+ setIsp(ControlList controls);\n+ setStaggered(ControlList controls);\n+ };\n+\n+Compiling the IPA interface\n+---------------------------\n+\n+After the IPA interface is defined in include/libcamera/ipa/{pipeline_name}.mojom,\n+and entry for it must be added in meson so that it can be compiled. The filename\n+must be added to the ipa_mojom_files object in include/libcamera/ipa/meson.build.\n+\n+For example, adding the raspberrypi.mojom file to meson:\n+\n+.. code-block:: none\n+\n+ ipa_mojom_files = [\n+ 'raspberrypi.mojom',\n+ ]\n+\n+This will cause the mojo data definition file to be compiled. Specifically, it\n+generated five files:\n+- a header describing the custom data structures, and the complete IPA interface\n+- a serializer implementing de/serialization for the custom data structures\n+- a proxy header describing a specialized IPA proxy\n+- a proxy source implementing the IPA proxy\n+- a proxy worker source implementing the other end of the IPA proxy\n+\n+The pipeline handler and the IPA only require the header and the proxy header.\n+The serializer is only used internally by the proxy.\n+\n+Using the custom data structures\n+--------------------------------\n+\n+To use the custom data structures that are defined in the mojo data definition\n+file, the follow header must be included:\n+\n+.. code-block:: C++\n+\n+ #include \n+\n+The POD types of the structs simply become their C++ counterparts, eg. uint32\n+in mojo will become uint32_t in C++. mojo map becomes C++ std::map, and mojo\n+array becomes C++ std::vector. All members of maps and vectors are embedded,\n+and are not pointers. The members cannot be const.\n+\n+All fields of structs are the name as specified in the data definition file,\n+with an underscore at the end. For example, the following struct as defined\n+in the mojo file:\n+\n+.. code-block:: none\n+\n+ struct SensorConfig {\n+ uint32 gainDelay = 1;\n+ uint32 exposureDelay;\n+ uint32 sensorMetadata;\n+ };\n+\n+Will become this in C++:\n+\n+.. code-block:: C++\n+\n+ struct SensorConfig {\n+ uint32_t gainDelay;\n+ uint32_t exposureDelay;\n+ uint32_t sensorMetadata;\n+ };\n+\n+The generated structs will also have two constructors, a constructor that\n+fills all fields with the default values, and a second constructor that takes\n+a value for every field. The default value constructor will fill in the fields\n+with the specified default value, if it exists. In the above example, `gainDelay_`\n+will be initialized to 1. If no default value is specified, then it will be\n+filled in as zero (or -1 for a FileDescriptor type).\n+\n+All fields and constructors/deconstructors in these generated structs are public.\n+\n+Using the IPA interface (pipeline handler)\n+------------------------------------------\n+\n+The following headers are necessary to use an IPA in the pipeline handler\n+(with raspberrypi as an example):\n+\n+.. code-block:: C++\n+\n+ #include \n+ #include \n+\n+The first header includes definitions of the custom data structures, and\n+the definition of the complete IPA interface (including both the Main and\n+the Event IPA interfaces). The name of the header file is comes from the name\n+of the mojom file, which in this case was raspberrypi.mojom.\n+\n+The second header inclues the definition of the specialized IPA proxy. It\n+exposes the complete IPA interface. We will see how to use it in this section.\n+\n+In the pipeline handler, we first need to construct a specialized IPA proxy.\n+From the point of view of the pipeline hander, this is the object that is the\n+IPA.\n+\n+To do so, we invoke the IPAManager:\n+\n+.. code-block:: C++\n+\n+ std::unique_ptr ipa_ =\n+ IPAManager::createIPA(pipe_, 1, 1);\n+\n+The ipa::rpi namespace comes from the namespace that we defined in the mojo\n+data definition file, in the \"Namespacing\" section. The name of the proxy,\n+IPAProxyRPi, comes from the name given to the main IPA interface,\n+IPARPiInterface, in the \"The Main IPA interface\" section.\n+\n+The return value of IPAManager::createIPA shall be error-checked, to confirm\n+that the returned pointer is not a nullptr.\n+\n+After this, before initializing the IPA, slots should be connected to all of\n+the IPA's Signals, as defined in the Event IPA interface:\n+\n+.. code-block:: C++\n+\n+\tipa_->statsMetadataComplete.connect(this, &RPiCameraData::statsMetadataComplete);\n+\tipa_->runIsp.connect(this, &RPiCameraData::runIsp);\n+\tipa_->embeddedComplete.connect(this, &RPiCameraData::embeddedComplete);\n+\tipa_->setIsp.connect(this, &RPiCameraData::setIsp);\n+\tipa_->setStaggered.connect(this, &RPiCameraData::setStaggered);\n+\n+The slot functions have a function signature based on the function definition\n+in the Event IPA interface. All plain old data (POD) types are as-is (with\n+their C++ versions, eg. uint32 -> uint32_t), and all structs are const references.\n+\n+For example, for the following entry in the Event IPA interface:\n+\n+.. code-block:: none\n+\n+ statsMetadataComplete(uint32 bufferId, ControlList controls);\n+\n+A function with the following function signature shall be connected to the\n+signal:\n+\n+.. code-block:: C++\n+\n+ statsMetadataComplete(uint32_t bufferId, const ControlList &controls);\n+\n+After connecting the slots to the signals, the IPA should be initialized\n+(fill in settings accordingly):\n+\n+.. code-block:: C++\n+\n+ IPASettings settings{};\n+ ipa_->init(settings);\n+\n+At this point, any IPA functions that were defined in the Main IPA interface\n+can be called as if they were regular member functions, for example:\n+\n+.. code-block:: C++\n+\n+ ipa_->start();\n+ ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result);\n+ ipa_->signalStatReady(RPi::BufferMask::STATS | static_cast(index));\n+\n+Remember that any functions designated as asynchronous *must not* be called\n+before start().\n+\n+TODO: anything special about start() and stop() ?\n+\n+Using the IPA interface (IPA)\n+-----------------------------\n+\n+The following header is necessary to implement an IPA (with raspberrypi as\n+an example):\n+\n+.. code-block:: C++\n+\n+ #include \n+\n+This header includes definitions of the custom data structures, and\n+the definition of the complete IPA interface (including both the Main and\n+the Event IPA interfaces). The name of the header file is comes from the name\n+of the mojom file, which in this case was raspberrypi.mojom.\n+\n+The IPA must implement the IPA interface class that is defined in the header.\n+In the case of our example, that is ipa::rpi::IPARPiInterface. The ipa::rpi\n+namespace comes from the namespace that we defined in the mojo data definition\n+file, in the \"Namespacing\" section. The name of the interface is the same as\n+the name given to the Main IPA interface.\n+\n+The function signature rules are the same as for the slots in the pipeline\n+handler side; PODs are passed by value, and structs are passed by const\n+reference. For the Main IPA interface, output values are also allowed (only\n+for synchronous calls), so there may be output parameters as well. If the\n+output parameter is a single POD it will be returned by value, otherwise\n+(multiple PODs or struct(s)) it will be returned by output parameter pointers.\n+\n+For example, for the following function specification in the Main IPA interface\n+definition:\n+\n+.. code-block:: none\n+\n+ configure(CameraSensorInfo sensorInfo,\n+ uint32 exampleNumber,\n+ map streamConfig,\n+ map entityControls,\n+ ConfigInput ipaConfig)\n+ => (ConfigOutput results);\n+\n+We will need to implement a function with the following function signature:\n+\n+.. code-block:: C++\n+\n+\tvoid configure(const CameraSensorInfo &sensorInfo,\n+ uint32_t exampleNumber,\n+\t\t const std::map &streamConfig,\n+\t\t const std::map &entityControls,\n+\t\t const ipa::rpi::ConfigInput &data,\n+\t\t ipa::rpi::ConfigOutput *response);\n+\n+The return value is void, because the output parameter is not a single POD.\n+Instead, it becomes an output parameter pointer. The non-POD input parameters\n+become const references, and the POD input parameter is passed by value.\n+\n+At any time (though usually only in response to an IPA call), the IPA may send\n+data to the pipeline handler by emitting signals. These signals are defined\n+in the C++ IPA interface class (which is in the generated and included header).\n+\n+For example, for the following function defined in the Event IPA interface:\n+\n+.. code-block:: none\n+\n+ statsMetadataComplete(uint32 bufferId, ControlList controls);\n+\n+We can emit a signal like so:\n+\n+.. code-block:: C++\n+\n+ statsMetadataComplete.emit(bufferId & RPi::BufferMask::ID, libcameraMetadata_);\ndiff --git a/Documentation/index.rst b/Documentation/index.rst\nindex 285ca7c3..b40a4586 100644\n--- a/Documentation/index.rst\n+++ b/Documentation/index.rst\n@@ -16,6 +16,7 @@\n Developer Guide \n Application Writer's Guide \n Pipeline Handler Writer's Guide \n+ IPA Writer's guide \n Tracing guide \n Environment variables \n Sensor driver requirements \ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex 9950465d..8cf68a07 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -54,6 +54,7 @@ if sphinx.found()\n 'environment_variables.rst',\n 'guides/application-developer.rst',\n 'guides/introduction.rst',\n+ 'guides/ipa.rst',\n 'guides/pipeline-handler.rst',\n 'guides/tracing.rst',\n 'index.rst',\n","prefixes":["libcamera-devel","v7","3/4"]}