[libcamera-devel,v4,2/3] Documentation: Guides: Pipeline Handler Writers Guide

9339 diff mbox series

From: Chris Chinchilla <chris@gregariousmammal.com>

Introduce a pipeline-handler writers guide to provide a walk through of
the steps and concepts required to implement a new Pipeline Handler.

Signed-off-by: Chris Chinchilla <chris@gregariousmammal.com>
[Reflow/Rework, update to mainline API]
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
[Further reworks and review]
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 Documentation/guides/pipeline-handler.rst | 1473 +++++++++++++++++++++
 Documentation/index.rst                   |    1 +
 Documentation/meson.build                 |    1 +
 3 files changed, 1475 insertions(+)
 create mode 100644 Documentation/guides/pipeline-handler.rst

Hi Kieran,

Thank you for the patch.

On Thu, Aug 20, 2020 at 02:47:50PM +0100, Kieran Bingham wrote:
> From: Chris Chinchilla <chris@gregariousmammal.com>
> 
> Introduce a pipeline-handler writers guide to provide a walk through of
> the steps and concepts required to implement a new Pipeline Handler.
> 
> Signed-off-by: Chris Chinchilla <chris@gregariousmammal.com>
> [Reflow/Rework, update to mainline API]
> Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> [Further reworks and review]
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  Documentation/guides/pipeline-handler.rst | 1473 +++++++++++++++++++++
>  Documentation/index.rst                   |    1 +
>  Documentation/meson.build                 |    1 +
>  3 files changed, 1475 insertions(+)
>  create mode 100644 Documentation/guides/pipeline-handler.rst
> 
> diff --git a/Documentation/guides/pipeline-handler.rst b/Documentation/guides/pipeline-handler.rst
> new file mode 100644
> index 000000000000..eb795dcc35b6
> --- /dev/null
> +++ b/Documentation/guides/pipeline-handler.rst
> @@ -0,0 +1,1473 @@
> +.. SPDX-License-Identifier: CC-BY-SA-4.0
> +
> +Pipeline Handler Writers Guide
> +==============================
> +
> +Pipeline handlers are the abstraction layer for device-specific hardware
> +configuration. They access and control hardware through the V4L2 and Media
> +Controller kernel interfaces, and implement an internal API to control the ISP
> +and capture components of a pipeline directly.
> +
> +Prerequisite knowledge: system architecture
> +-------------------------------------------
> +
> +A pipeline handler configures and manages the image acquisition and
> +transformation pipeline realized by specialized system peripherals combined with
> +an image source connected to the system through a data and control bus. The
> +presence, number and characteristics of them vary depending on the system design
> +and the product integration of the target platform.
> +
> +System components can be classified in three macro-categories:
> +
> +.. TODO: Insert references to the open CSI-2 (and other) specification.
> +
> +- Input ports: Interfaces to external devices, usually image sensors,
> +  which transfer data from the physical bus to locations accessible by other
> +  system peripherals. An input port needs to be configured according to the
> +  input image format and size and could optionally apply basic transformations
> +  on the received images, most typically cropping/scaling and some formats
> +  conversion. The industry standard for the system typically targeted by
> +  libcamera is to have receivers compliant with the MIPI CSI-2 specifications,
> +  implemented on a compatible physical layer such as MIPI D-PHY or MIPI C-PHY.
> +  Other design are possible but less common, such as LVDS or the legacy BT.601
> +  and BT.656 parallel protocols.
> +
> +- Image Signal Processor (ISP): A specialized media processor which applies
> +  digital transformations on image streams. ISPs can be integrated as part of
> +  the SoC as a memory interfaced system peripheral or packaged as stand-alone
> +  chips connected to the application processor through a bus. Most hardware used
> +  by libcamera makes use of in-system ISP designs but pipelines can equally
> +  support external ISP chips or be instrumented to use other system resources
> +  such as a GPU or an FPGA IP block. ISPs expose a software programming
> +  interface that allows the configuration of multiple processing blocks which
> +  form an "Image Transformation Pipeline". An ISP usually produces 'processed'
> +  image streams along with the metadata describing the processing steps which
> +  have been applied to generate the output frames.
> +
> +- Camera Sensor: Digital components that integrate an image sensor with control
> +  electronics and usually a lens. It interfaces to the SoC image receiver ports
> +  and is programmed to produce images in a format and size suitable for the
> +  current system configuration. Complex camera modules can integrate on-board
> +  ISP or DSP chips and process images before delivering them to the system. Most
> +  systems with a dedicated ISP processor will usually integrate camera sensors
> +  which produce images in Raw Bayer format and defer processing to it.
> +
> +It is the responsibility of the pipeline handler to interface with these (and
> +possibly other) components of the system and implement the following
> +functionalities:
> +
> +- Detect and register camera devices available in the system with an associated
> +  set of image streams.
> +
> +- Configure the image acquisition and processing pipeline by assigning the
> +  system resources (memory, shared components, etc.) to satisfy the
> +  configuration requested by the application.
> +
> +- Start and the stop the image acquisition and processing sessions.
> +
> +- Apply configuration settings requested by applications and computed by image
> +  processing algorithms integrated in libcamera to the hardware devices.
> +
> +- Notify applications of the availability of new images and deliver them to the
> +  correct locations.
> +
> +Prerequisite knowledge: libcamera architecture
> +----------------------------------------------
> +
> +A pipeline handler makes use of the following libcamera classes to realize the
> +functionalities descibed above. Below is a brief overview of each of those:
> +
> +.. TODO: (All) Convert to sphinx refs
> +.. TODO: (MediaDevice) Reference to the Media Device API (possibly with versioning requirements)
> +.. TODO: (IPAInterface) refer to the IPA guide
> +
> +-  `MediaDevice <http://libcamera.org/api-html/classlibcamera_1_1MediaDevice.html>`_:
> +   Instances of this class are associated with a kernel media controller
> +   device and its connected objects.
> +
> +-  `DeviceEnumerator <http://libcamera.org/api-html/classlibcamera_1_1DeviceEnumerator.html>`_:
> +   Enumerates all media devices attached to the system and the media entities
> +   registered with it, by creating instances of the ``MediaDevice`` class and
> +   storing them.
> +
> +-  `DeviceMatch <http://libcamera.org/api-html/classlibcamera_1_1DeviceMatch.html>`_:
> +   Describes a media device search pattern using entity names, or other
> +   properties.
> +
> +-  `V4L2VideoDevice <http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html>`_:
> +   Models an instance of a V4L2 video device constructed with the path to a V4L2
> +   video device node.
> +
> +-  `V4L2SubDevice <http://libcamera.org/api-html/classlibcamera_1_1V4L2Subdevice.html>`_:
> +   Provides an API to the sub-devices that model the hardware components of a
> +   V4L2 device.
> +
> +-  `CameraSensor <http://libcamera.org/api-html/classlibcamera_1_1CameraSensor.html>`_:
> +   Abstracts camera sensor handling by hiding the details of the V4L2 subdevice
> +   kernel API and caching sensor information.
> +
> +-  `CameraData <http://libcamera.org/api-html/classlibcamera_1_1CameraData.html>`_:
> +   Represents device-specific data a pipeline handler associates to each Camera
> +   instance.
> +
> +-  `StreamConfiguration <http://libcamera.org/api-html/structlibcamera_1_1StreamConfiguration.html>`_:
> +   Models the current configuration of an image stream produced by the camera by
> +   reporting its format and sizes.
> +
> +-  `CameraConfiguration <http://libcamera.org/api-html/classlibcamera_1_1CameraConfiguration.html>`_:
> +   Represents the current configuration of a camera, which includes a list of
> +   stream configurations for each active stream in a capture session. When
> +   validated, it is applied to the camera.
> +
> +-  `IPAInterface <http://libcamera.org/api-html/classlibcamera_1_1IPAInterface.html>`_:
> +   The interface to the Image Processing Algorithm (IPA) module which performs
> +   the computation of the image processing pipeline tuning parameters.
> +
> +-  `ControlList <http://libcamera.org/api-html/classlibcamera_1_1ControlList.html>`_:
> +   A list of control items, indexed by Control<> instances or by numerical index
> +   which contains values used by application and IPA to change parameters of
> +   image streams, used to return to applications and share with IPA the metadata
> +   associated with the captured images, and to advertise the immutable camera
> +   characteristics enumerated at system initialization time.
> +
> +Creating a PipelineHandler
> +--------------------------
> +
> +This guide walks through the steps to create a simple pipeline handler
> +called “Vivid” that supports the `V4L2 Virtual Video Test Driver`_ (vivid).
> +
> +To use the vivid test driver, you first need to check that the vivid kernel
> +module is loaded, for example with the ``modprobe vivid`` command.
> +
> +.. _V4L2 Virtual Video Test Driver: https://www.kernel.org/doc/html/latest/admin-guide/media/vivid.html
> +
> +Create the skeleton file structure
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +To add a new pipeline handler, create a directory to hold the pipeline code in
> +the *src/libcamera/pipeline/* directory that matches the name of the pipeline
> +(in this case *vivid*). Inside the new directory add a *meson.build* file that
> +integrates with the libcamera build system, and a *vivid.cpp* file that matches
> +the name of the pipeline.
> +
> +In the *meson.build* file, add the *vivid.cpp* file as a build source for
> +libcamera by adding it to the global meson ``libcamera_sources`` variable:
> +
> +.. code-block:: none
> +
> +   # SPDX-License-Identifier: CC0-1.0
> +
> +   libcamera_sources += files([
> +       'vivid.cpp',
> +   ])
> +
> +Users of libcamera can selectively enable pipelines while building libcamera
> +using the ``pipelines`` option.
> +
> +For example, to enable only the IPU3, UVC, and VIVID pipelines, specify them as
> +a comma separated list with ``-Dpipelines`` when generating a build directory:
> +
> +.. code-block:: shell
> +
> +    meson build -Dpipelines=ipu3,uvcvideo,vivid
> +
> +Read the `Meson build configuration`_ documentation for more information on
> +configuring a build directory.
> +
> +.. _Meson build configuration: https://mesonbuild.com/Configuring-a-build-directory.html
> +
> +To add the new pipeline handler to this list of options, add its directory name
> +to the libcamera build options in the top level _meson_options.txt_.
> +
> +.. code-block:: none
> +
> +   option('pipelines',
> +           type : 'array',
> +           choices : ['ipu3', 'raspberrypi', 'rkisp1', 'simple', 'uvcvideo', 'vimc', 'vivid'],
> +           description : 'Select which pipeline handlers to include')
> +
> +
> +In *vivid.cpp* add the pipeline handler to the ``libcamera`` namespace, defining
> +a `PipelineHandler`_ derived class named PipelineHandlerVivid, and add stub
> +methods for the overridden class members.
> +
> +.. _PipelineHandler: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
> +
> +.. code-block:: cpp
> +
> +   namespace libcamera {
> +
> +   class PipelineHandlerVivid : public PipelineHandler
> +   {
> +   public:
> +          PipelineHandlerVivid(CameraManager *manager);
> +
> +          CameraConfiguration *generateConfiguration(Camera *camera,
> +          const StreamRoles &roles) override;
> +          int configure(Camera *camera, CameraConfiguration *config) override;
> +
> +          int exportFrameBuffers(Camera *camera, Stream *stream,
> +          std::vector<std::unique_ptr<FrameBuffer>> *buffers) override;
> +
> +          int start(Camera *camera) override;
> +          void stop(Camera *camera) override;
> +
> +          int queueRequestDevice(Camera *camera, Request *request) override;
> +
> +          bool match(DeviceEnumerator *enumerator) override;
> +   };
> +
> +   PipelineHandlerVivid::PipelineHandlerVivid(CameraManager *manager)
> +          : PipelineHandler(manager)
> +   {
> +   }
> +
> +   CameraConfiguration *PipelineHandlerVivid::generateConfiguration(Camera *camera,
> +                                                                    const StreamRoles &roles)
> +   {
> +          return nullptr;
> +   }
> +
> +   int PipelineHandlerVivid::configure(Camera *camera, CameraConfiguration *config)
> +   {
> +          return -1;
> +   }
> +
> +   int PipelineHandlerVivid::exportFrameBuffers(Camera *camera, Stream *stream,
> +                                                std::vector<std::unique_ptr<FrameBuffer>> *buffers)
> +   {
> +          return -1;
> +   }
> +
> +   int PipelineHandlerVivid::start(Camera *camera)
> +   {
> +          return -1;
> +   }
> +
> +   void PipelineHandlerVivid::stop(Camera *camera)
> +   {
> +   }
> +
> +   int PipelineHandlerVivid::queueRequestDevice(Camera *camera, Request *request)
> +   {
> +          return -1;
> +   }
> +
> +   bool PipelineHandlerVivid::match(DeviceEnumerator *enumerator)
> +   {
> +          return false;
> +   }
> +
> +   REGISTER_PIPELINE_HANDLER(PipelineHandlerVivid);
> +
> +   } /* namespace libcamera */
> +
> +Note that you must register the ``PipelineHandler`` subclass with the pipeline
> +handler factory using the `REGISTER_PIPELINE_HANDLER`_ macro which
> +registers it and creates a global symbol to reference the class and make it
> +available to try and match devices.
> +
> +.. _REGISTER_PIPELINE_HANDLER: http://libcamera.org/api-html/pipeline__handler_8h.html
> +
> +For debugging and testing a pipeline handler during development, you can define
> +a log message category for the pipeline handler. The ``LOG_DEFINE_CATEGORY``
> +macro and ``LIBCAMERA_LOG_LEVELS`` environment variable help you use the inbuilt
> +libcamera `logging infrastructure`_ that allow for the inspection of internal
> +operations in a user-configurable way.
> +
> +.. _logging infrastructure: http://libcamera.org/api-html/log_8h.html
> +
> +Add the following before the ``PipelineHandlerVivid`` class declaration:
> +
> +.. code-block:: cpp
> +
> +   LOG_DEFINE_CATEGORY(VIVID)
> +
> +At this point you need the following includes for logging and pipeline handler
> +features:
> +
> +.. code-block:: cpp
> +
> +   #include "libcamera/internal/log.h"
> +   #include "libcamera/internal/pipeline_handler.h"
> +
> +Run the following commands:
> +
> +.. code-block:: shell
> +
> +   meson build
> +   ninja -C build
> +
> +To build the libcamera code base, and confirm that the build system found the
> +new pipeline handler by running:
> +
> +.. code-block:: shell
> +
> +   LIBCAMERA_LOG_LEVELS=Pipeline:0 ./build/src/cam/cam -l
> +
> +And you should see output like the below:
> +
> +.. code-block:: shell
> +
> +    DEBUG Pipeline pipeline_handler.cpp:680 Registered pipeline handler "PipelineHandlerVivid"
> +
> +Matching devices
> +~~~~~~~~~~~~~~~~
> +
> +Each pipeline handler registered in libcamera gets tested against the current
> +system configuration, by matching a ``DeviceMatch`` with the system
> +``DeviceEnumerator``. A successful match makes sure all the requested components
> +have been registered in the system and allows the pipeline handler to be
> +initialized.
> +
> +The main entry point of a pipeline handler is the `match()`_ class member
> +function. When the ``CameraManager`` is started (using the `start()`_ method),
> +all the registered pipeline handlers are iterated and their ``match`` function
> +called with an enumerator of all devices it found on a system.
> +
> +The match method should identify if there are suitable devices available in the
> +``DeviceEnumerator`` which the pipeline supports, returning ``true`` if it
> +matches a device, and ``false`` if it does not. To do this, construct a
> +`DeviceMatch`_ class with the name of the ``MediaController`` device to match.
> +You can specify the search further by adding specific media entities to the
> +search using the ``.add()`` method on the DeviceMatch.
> +
> +.. _match(): https://www.libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#a7cd5b652a2414b543ec20ba9dabf61b6
> +.. _start(): http://libcamera.org/api-html/classlibcamera_1_1CameraManager.html#a49e322880a2a26013bb0076788b298c5
> +.. _DeviceMatch: http://libcamera.org/api-html/classlibcamera_1_1DeviceMatch.html
> +
> +This example uses search patterns that match vivid, but when developing a new
> +pipeline handler, you should change this value to suit your device identifier.
> +
> +Replace the contents of the ``PipelineHandlerVivid::match`` method with the
> +following:
> +
> +.. code-block:: cpp
> +
> +   DeviceMatch dm("vivid");
> +   dm.add("vivid-000-vid-cap");
> +   return false; // Prevent infinite loops for now
> +
> +With the device matching criteria defined, attempt to acquire exclusive access
> +to the matching media controller device with the `acquireMediaDevice`_ method.
> +If the method attempts to acquire a device it has already matched, it returns
> +``false``.
> +
> +.. _acquireMediaDevice: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#a77e424fe704e7b26094164b9189e0f84
> +
> +Add the following below ``dm.add("vivid-000-vid-cap");``:
> +
> +.. code-block:: cpp
> +
> +   MediaDevice *media = acquireMediaDevice(enumerator, dm);
> +   if (!media)
> +           return false;
> +
> +The pipeline handler now needs an additional include. Add the following to the
> +existing include block for device enumeration functionality:
> +
> +.. code-block:: cpp
> +
> +   #include "libcamera/internal/device_enumerator.h"
> +
> +At this stage, you should test that the pipeline handler can successfully match
> +the devices, but have not yet added any code to create a Camera which libcamera
> +reports to applications.
> +
> +As a temporary validation step, add a debug print with
> +
> +.. code-block:: cpp
> +
> +   LOG(VIVID, Debug) << "Vivid Device Identified";
> +
> +before the final closing return statement in the ``PipelineHandlerVivid::match``
> +method for when when the pipeline handler successfully matches the
> +``MediaDevice`` and ``MediaEntity`` names.
> +
> +Test that the pipeline handler matches and finds a device by rebuilding, and
> +running
> +
> +.. code-block:: shell
> +
> +   ninja -C build
> +   LIBCAMERA_LOG_LEVELS=Pipeline,VIVID:0 ./build/src/cam/cam -l
> +
> +And you should see output like the below:
> +
> +.. code-block:: shell
> +
> +    DEBUG VIVID vivid.cpp:74 Vivid Device Identified
> +
> +Creating camera devices
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +If the pipeline handler successfully matches with the system it is running on,
> +it can proceed to initialization, by creating all the required instances of the
> +``V4L2VideoDevice``, ``V4L2Subdevice`` and ``CameraSensor`` hardware abstraction
> +classes. If the Pipeline handler supports an ISP, it can then also Initialise
> +the IPA module before proceeding to the creation of the Camera devices.
> +
> +An image ``Stream`` represents a sequence of images and data of known size and
> +format, stored in application-accessible memory locations. Typical examples of
> +streams are the ISP processed outputs and the raw images captured at the
> +receivers port output.
> +
> +The Pipeline Handler is responsible for defining the set of Streams associated
> +with the Camera.
> +
> +Each Camera has instance-specific data represented using the `CameraData`_
> +class, which can be extended for the specific needs of the pipeline handler.
> +
> +.. _CameraData: http://libcamera.org/api-html/classlibcamera_1_1CameraData.html
> +
> +
> +To support the Camera we will later register, we need to create a CameraData
> +class that we can implement for our specific Pipeline Handler.
> +
> +Define a new ``VividCameraData()`` class derived from ``CameraData`` by adding
> +the following code before the PipelineHandlerVivid class definition where it
> +will be used:
> +
> +.. code-block:: cpp
> +
> +   class VividCameraData : public CameraData
> +   {
> +   public:
> +          VividCameraData(PipelineHandler *pipe, MediaDevice *media)
> +                : CameraData(pipe), media_(media), video_(nullptr)
> +          {
> +          }
> +
> +          ~VividCameraData()
> +          {
> +                delete video_;
> +          }
> +
> +          int init();
> +          void bufferReady(FrameBuffer *buffer);
> +
> +          MediaDevice *media_;
> +          V4L2VideoDevice *video_;
> +          Stream stream_;
> +   };
> +
> +This example pipeline handler handles a single video device and supports a
> +single stream, represented by the ``VividCameraData`` class members. More
> +complex pipeline handlers might register cameras composed of several video
> +devices and sub-devices, or multiple streams per camera that represent the
> +several components of the image capture pipeline. You should represent all these
> +components in the ``CameraData`` derived class when developing a custom
> +PipelineHandler.
> +
> +In our example VividCameraData we implement an ``init()`` function to prepare
> +the object from our PipelineHandler, however the CameraData class does not
> +specify the interface for initialisation and PipelineHandlers can manage this
> +based on their own needs. Derived CameraData classes are used only by their
> +respective pipeline handlers.
> +
> +The CameraData class stores the context required for each camera instance and
> +is usually responsible for opening all Devices used in the capture pipeline.
> +
> +We can now implement the ``init`` method for our example Pipeline Handler to
> +create a new V4L2 video device from the media entity, which we can specify using
> +the `MediaDevice::getEntityByName`_ method from the MediaDevice. As our example
> +is based upon the simplistic Vivid test device, we only need to open a single
> +capture device named 'vivid-000-vid-cap' by the device.
> +
> +.. _MediaDevice::getEntityByName: http://libcamera.org/api-html/classlibcamera_1_1MediaDevice.html#ad5d9279329ef4987ceece2694b33e230
> +
> +.. code-block:: cpp
> +
> +   int VividCameraData::init()
> +   {
> +          video_ = new V4L2VideoDevice(media_->getEntityByName("vivid-000-vid-cap"));
> +          if (video_->open())
> +                return -ENODEV;
> +
> +          return 0;
> +   }
> +
> +The CameraData should be created and initialised before we move on to register a
> +new Camera device so we need to construct and initialise our
> +VividCameraData after we have identified our device within
> +PipelineHandlerVivid::match(). The VividCameraData is wrapped by a
> +std::unique_ptr to help manage the lifetime of our CameraData instance.
> +
> +If the camera data initialization fails, return ``false`` to indicate the
> +failure to the ``match()`` method and prevent retrying of the pipeline handler.
> +
> +.. code-block:: cpp
> +
> +   std::unique_ptr<VividCameraData> data = std::make_unique<VividCameraData>(this, media);
> +
> +   if (data->init())
> +           return false;
> +
> +
> +Once the camera data has been initialized, the Camera device instances and the
> +associated streams have to be registered. Create a set of streams for the
> +camera, which for this device is only one. You create a camera using the static
> +`Camera::create`_ method, passing the pipeline handler, the id of the camera,
> +and the streams available. Then register the camera and its data with the
> +pipeline handler and camera manager using `registerCamera`_.
> +
> +Finally with a successful construction, we return 'true' indicating that the
> +PipelineHandler successfully matched and constructed a device.
> +
> +.. _Camera::create: http://libcamera.org/api-html/classlibcamera_1_1Camera.html#a453740e0d2a2f495048ae307a85a2574
> +.. _registerCamera: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#adf02a7f1bbd87aca73c0e8d8e0e6c98b
> +
> +.. code-block:: cpp
> +
> +   std::set<Stream *> streams{ &data->stream_ };
> +   std::shared_ptr<Camera> camera = Camera::create(this, data->video_->deviceName(), streams);
> +   registerCamera(std::move(camera), std::move(data));
> +
> +   return true;
> +
> +
> +Our match function should now look like the following:
> +
> +.. code-block:: cpp
> +
> +   bool PipelineHandlerVivid::match(DeviceEnumerator *enumerator)
> +   {
> +   	DeviceMatch dm("vivid");
> +   	dm.add("vivid-000-vid-cap");
> +
> +   	MediaDevice *media = acquireMediaDevice(enumerator, dm);
> +   	if (!media)
> +   		return false;
> +
> +   	std::unique_ptr<VividCameraData> data = std::make_unique<VividCameraData>(this, media);
> +
> +   	/* Locate and open the capture video node. */
> +   	if (data->init())
> +   		return false;
> +
> +   	/* Create and register the camera. */
> +   	std::set<Stream *> streams{ &data->stream_ };
> +   	std::shared_ptr<Camera> camera = Camera::create(this, data->video_->deviceName(), streams);
> +   	registerCamera(std::move(camera), std::move(data));
> +
> +   	return true;
> +   }
> +
> +We will need to use our custom CameraData class frequently throughout the
> +pipeline handler, so we add a private convenience helper to our Pipeline handler
> +to obtain and cast the custom CameraData instance from a Camera instance.
> +
> +.. code-block:: cpp
> +
> +   private:
> +       VividCameraData *cameraData(const Camera *camera)
> +       {
> +               return static_cast<VividCameraData *>(
> +                        PipelineHandler::cameraData(camera));
> +       }
> +
> +At this point, you need to add the following new includes to provide the Camera
> +interface, and device interaction interfaces.
> +
> +.. code-block:: cpp
> +
> +   #include <libcamera/camera.h>
> +   #include "libcamera/internal/media_device.h"
> +   #include "libcamera/internal/v4l2_videodevice.h"
> +
> +Registering controls and properties
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The libcamera `controls framework`_ allows an application to configure the
> +streams capture parameters on a per-frame basis and is also used to advertise
> +immutable properties of the ``Camera`` device.
> +
> +The libcamera controls and properties are defined in YAML form which is
> +processed to automatically generate documentation and interfaces. Controls are
> +defined by the src/libcamera/`control_ids.yaml`_ file and camera properties
> +are defined by src/libcamera/`properties_ids.yaml`_.
> +
> +.. _controls framework: http://libcamera.org/api-html/controls_8h.html
> +.. _control_ids.yaml: http://libcamera.org/api-html/control__ids_8h.html
> +.. _properties_ids.yaml: http://libcamera.org/api-html/property__ids_8h.html
> +
> +Pipeline handlers can optionally register the list of controls an application
> +can set as well as a list of immutable camera properties. Being both
> +Camera-specific values, they are represented in the ``CameraData`` base class,
> +which provides two members for this purpose: the `CameraData::controlInfo_`_ and
> +the `CameraData::properties_`_ fields.
> +
> +.. _CameraData::controlInfo_: http://libcamera.org/api-html/classlibcamera_1_1CameraData.html#ab9fecd05c655df6084a2233872144a52
> +.. _CameraData::properties_: http://libcamera.org/api-html/classlibcamera_1_1CameraData.html#a84002c29f45bd35566c172bb65e7ec0b
> +
> +The ``controlInfo_`` field represents a map of ``ControlId`` instances
> +associated with the limits of valid values supported for the control. More
> +information can be found in the `ControlInfoMap`_ class documentation.
> +
> +.. _ControlInfoMap: http://libcamera.org/api-html/classlibcamera_1_1ControlInfoMap.html
> +
> +Pipeline handlers register controls to expose the tunable device and IPA
> +parameters to applications. Our example pipeline handler only exposes trivial
> +controls of the video device, by registering a ``ControlId`` instance with
> +associated values for each supported V4L2 control but demonstrates the mapping
> +of V4L2 Controls to libcamera ControlIDs.
> +
> +Complete the initialization of the ``VividCameraData`` class by adding the
> +following code to the ``VividCameraData::init()`` method to initialise the
> +controls. For more complex control configurations, this could of course be
> +broken out to a separate function, but for now we just initialise the small set
> +inline in our CameraData init:
> +
> +.. code-block:: cpp
> +
> +   /* Initialise the supported controls. */
> +   const ControlInfoMap &controls = video_->controls();
> +   ControlInfoMap::Map ctrls;
> +
> +   for (const auto &ctrl : controls) {
> +           const ControlId *id;
> +           ControlInfo info;
> +
> +           switch (ctrl.first->id()) {
> +           case V4L2_CID_BRIGHTNESS:
> +                   id = &controls::Brightness;
> +                   info = ControlInfo{ { -1.0f }, { 1.0f }, { 0.0f } };
> +                   break;
> +           case V4L2_CID_CONTRAST:
> +                   id = &controls::Contrast;
> +                   info = ControlInfo{ { 0.0f }, { 2.0f }, { 1.0f } };
> +                   break;
> +           case V4L2_CID_SATURATION:
> +                   id = &controls::Saturation;
> +                   info = ControlInfo{ { 0.0f }, { 2.0f }, { 1.0f } };
> +                   break;
> +           default:
> +                   continue;
> +           }
> +
> +           ctrls.emplace(id, info);
> +   }
> +
> +   controlInfo_ = std::move(ctrls);
> +
> +The ``properties_`` field is  a list of ``ControlId`` instances
> +associated with immutable values, which represent static characteristics that can
> +be used by applications to identify camera devices in the system. Properties can be
> +registered by inspecting the values of V4L2 controls from the video devices and
> +camera sensor (for example to retrieve the position and orientation of a camera)
> +or to express other immutable characteristics. The example pipeline handler does
> +not register any property, but examples are available in the libcamera code
> +base.
> +
> +.. TODO: Add a property example to the pipeline handler. At least the model.
> +
> +At this point you need to add the following includes to the top of the file for
> +handling controls:
> +
> +.. code-block:: cpp
> +
> +   #include <libcamera/controls.h>
> +   #include <libcamera/control_ids.h>
> +
> +Generating a default configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Once ``Camera`` devices and the associated ``Streams`` have been registered, an
> +application can proceed to acquire and configure the camera to prepare it for a
> +frame capture session.
> +
> +Applications specify the requested configuration by assigning a
> +``StreamConfiguration`` instance to each stream they want to enable which
> +expresses the desired image size and pixel format. The stream configurations are
> +grouped in a ``CameraConfiguration`` which is inspected by the pipeline handler
> +and validated to adjust it to a supported configuration. This may involve
> +adjusting the formats or image sizes or alignments for example to match the
> +capabilities of the device.
> +
> +Applications may choose to repeat validation stages, adjusting paramters until a
> +set of validated StreamConfigurations are returned that is acceptable for the
> +applications needs. When the pipeline handler receives a valid camera
> +configuration it can use the image stream configurations to apply settings to
> +the hardware devices.
> +
> +This configuration and validation process is managed with another Pipeline
> +specific class derived from a common base implementation and interface.
> +
> +To support validation in our example pipeline handler, Create a new class called
> +``VividCameraConfiguration`` derived from the base `CameraConfiguration`_ class
> +which we can implement and use within our ``PipelineHandlerVivid`` class.
> +
> +.. _CameraConfiguration: http://libcamera.org/api-html/classlibcamera_1_1CameraConfiguration.html
> +
> +The derived ``CameraConfiguration`` class must override the base class
> +``validate()`` function, where the stream configuration inspection and
> +adjustment happens.
> +
> +.. code-block:: cpp
> +
> +    class VividCameraConfiguration : public CameraConfiguration
> +    {
> +    public:
> +           VividCameraConfiguration();
> +
> +           Status validate() override;
> +    };
> +
> +    VividCameraConfiguration::VividCameraConfiguration()
> +           : CameraConfiguration()
> +    {
> +    }
> +
> +Applications generate a ``CameraConfiguration`` instance by calling the
> +`Camera::generateConfiguration()`_ function, which calls into the pipeline
> +implementation of the overridden `PipelineHandler::generateConfiguration()`_
> +method.
> +
> +.. _Camera::generateConfiguration(): http://libcamera.org/api-html/classlibcamera_1_1Camera.html#a25c80eb7fc9b1cf32692ce0c7f09991d
> +.. _PipelineHandler::generateConfiguration(): http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#a7932e87735695500ce1f8c7ae449b65b
> +
> +Configurations are generated by receiving a list of ``StreamRoles`` instances,
> +which libcamera uses as predefined ways an application intends to use a camera
> +(You can read the full list in the `StreamRole API`_ documentation). These are
> +optional hints on how an application intends to use a stream, and a pipeline
> +handler should return an ideal configuration for each role that is requested.
> +
> +.. _StreamRole API: http://libcamera.org/api-html/stream_8h.html#file_a295d1f5e7828d95c0b0aabc0a8baac03
> +
> +In the pipeline handler ``generateConfiguration`` implementation, remove the
> +``return nullptr;``, create a new instance of the ``CameraConfiguration``
> +derived class, and assign it to a base class pointer.
> +
> +.. code-block:: cpp
> +
> +   VividCameraData *data = cameraData(camera);
> +   CameraConfiguration *config = new VividCameraConfiguration();
> +
> +A ``CameraConfiguration`` is specific to each pipeline, so you can only create
> +it from the pipeline handler code path. Applications can also generate an empty
> +configuration and add desired stream configurations manually. Pipelines must
> +allow for this by returning an empty configuration if no roles are requested.
> +
> +To support this in our PipelineHandlerVivid, next add the following check in
> +``generateConfiguration`` after the Cameraconfiguration has been constructed:
> +
> +.. code-block:: cpp
> +
> +   if (roles.empty())
> +           return config;
> +
> +A production pipeline handler should generate the ``StreamConfiguration`` for
> +all the appropriate stream roles a camera device supports. For this simpler
> +example (with only one stream), the pipeline handler always returns the same
> +configuration, inferred from the underlying V4L2VideoDevice.
> +
> +How it does this is shown below, but examination of the more full-featured
> +pipelines for IPU3, RKISP1 and RaspberryPi are recommend to explore more
> +complex examples.
> +
> +To generate a ``StreamConfiguration``, you need a list of pixel formats and
> +frame sizes which supported outputs of the stream. You can fetch a map of the
> +``V4LPixelFormat`` and ``SizeRange`` supported by the underlying output device,
> +but the pipeline handler needs to convert this to a ``libcamera::PixelFormat``
> +type to pass to applications. We do this here using ``std::transform`` to
> +convert the formats and populate a new ``PixelFormat`` map as shown below.
> +
> +Continue adding the following code example to our ``generateConfiguration``
> +implementation.
> +
> +.. code-block:: cpp
> +
> +   std::map<V4L2PixelFormat, std::vector<SizeRange>> v4l2Formats =
> +   data->video_->formats();
> +   std::map<PixelFormat, std::vector<SizeRange>> deviceFormats;
> +   std::transform(v4l2Formats.begin(), v4l2Formats.end(),
> +          std::inserter(deviceFormats, deviceFormats.begin()),
> +          [&](const decltype(v4l2Formats)::value_type &format) {
> +              return decltype(deviceFormats)::value_type{
> +                  format.first.toPixelFormat(),
> +                  format.second
> +              };
> +          });
> +
> +The `StreamFormats`_ class holds information about the pixel formats and frame
> +sizes that a stream can support. The class groups size information by the pixel
> +format, which can produce it.
> +
> +.. _StreamFormats: http://libcamera.org/api-html/classlibcamera_1_1StreamFormats.html
> +
> +The code below uses the ``StreamFormats`` class to represent all of the
> +supported pixel formats, associated with a list of frame sizes. It then
> +generates a supported StreamConfiguration to model the information an
> +application can use to configure a single stream.
> +
> +Continue adding the following code to support this:
> +
> +.. code-block:: cpp
> +
> +   StreamFormats formats(deviceFormats);
> +   StreamConfiguration cfg(formats);
> +
> +As well as a list of supported StreamFormats, the StreamConfiguration is also
> +expected to provide an initialsed default configuration. This may be arbitrary,
> +but depending on use case you may which to select an output that matches the
> +Sensor output, or prefer a pixelformat which might provide higher performance on
> +the hardware. The bufferCount represents the number of buffers required to
> +support functional continuous processing on this stream.
> +
> +.. code-block:: cpp
> +
> +   cfg.pixelFormat = formats::BGR888;
> +   cfg.size = { 1280, 720 };
> +   cfg.bufferCount = 4;
> +
> +Finally add each ``StreamConfiguration`` generated to the
> +``CameraConfiguration``, and ensure that it has been validated before returning
> +it to the application. With only a single supported stream, this code adds only
> +a single StreamConfiguration however a StreamConfiguration should be added for
> +each supported role in a device that can handle more streams.
> +
> +Add the following code to complete the implementation of
> +``generateConfiguration``:
> +
> +.. code-block:: cpp
> +
> +   config->addConfiguration(cfg);
> +
> +   config->validate();
> +
> +   return config;
> +
> +To validate a camera configuration, a pipeline handler must implement the
> +`CameraConfiguration::validate()`_ method in it's derived class to inspect all
> +the stream configuration associated to it, make any adjustments required to make
> +the configuration valid, and return the validation status.
> +
> +If changes are made, it marks the configuration as ``Adjusted``, however if the
> +requested configuration is not supported and cannot be adjusted it shall be
> +refused and marked as ``Invalid``.
> +
> +.. _CameraConfiguration::validate(): http://libcamera.org/api-html/classlibcamera_1_1CameraConfiguration.html#a29f8f263384c6149775b6011c7397093
> +
> +The validation phase makes sure all the platform-specific constraints are
> +respected by the requested configuration. The most trivial examples being making
> +sure the requested image formats are supported and the image alignment
> +restrictions adhered to. The pipeline handler specific implementation of
> +``validate()`` shall inspect all the configuration parameters received and never
> +assume they are correct, as applications are free to change the requested stream
> +parameters after the configuration has been generated.
> +
> +Again, this example pipeline handler is simpler, look at the more complex
> +implementations for a realistic example.
> +
> +Add the following function implementation to your file:
> +
> +.. code-block:: cpp
> +
> +   CameraConfiguration::Status VividCameraConfiguration::validate()
> +   {
> +           Status status = Valid;
> +
> +           if (config_.empty())
> +                  return Invalid;
> +
> +           if (config_.size() > 1) {
> +                  config_.resize(1);
> +                  status = Adjusted;
> +           }
> +
> +           StreamConfiguration &cfg = config_[0];
> +
> +           const std::vector<libcamera::PixelFormat> formats = cfg.formats().pixelformats();
> +           if (std::find(formats.begin(), formats.end(), cfg.pixelFormat) == formats.end()) {
> +                  cfg.pixelFormat = cfg.formats().pixelformats()[0];
> +                  LOG(VIVID, Debug) << "Adjusting format to " << cfg.pixelFormat.toString();
> +                  status = Adjusted;
> +           }
> +
> +           cfg.bufferCount = 4;
> +
> +           return status;
> +   }
> +
> +Now that we are handling the ``PixelFormat`` type, we also need to add
> +``#include <libcamera/formats.h>`` to the include section before we rebuild the
> +codebase, and test:
> +
> +.. code-block:: shell
> +
> +   ninja -C build
> +   LIBCAMERA_LOG_LEVELS=Pipeline,VIVID:0 ./build/src/cam/cam -c vivid -I
> +
> +You should see the following output showing the capabilites of our new pipeline
> +handler, and showing that our configurations have been generated:
> +
> +.. code-block:: shell
> +
> +    Using camera vivid
> +    0: 1280x720-BGR888
> +    * Pixelformat: NV21 (320x180)-(3840x2160)/(+0,+0)
> +    - 320x180
> +    - 640x360
> +    - 640x480
> +    - 1280x720
> +    - 1920x1080
> +    - 3840x2160
> +    * Pixelformat: NV12 (320x180)-(3840x2160)/(+0,+0)
> +    - 320x180
> +    - 640x360
> +    - 640x480
> +    - 1280x720
> +    - 1920x1080
> +    - 3840x2160
> +    * Pixelformat: BGRA8888 (320x180)-(3840x2160)/(+0,+0)
> +    - 320x180
> +    - 640x360
> +    - 640x480
> +    - 1280x720
> +    - 1920x1080
> +    - 3840x2160
> +    * Pixelformat: RGBA8888 (320x180)-(3840x2160)/(+0,+0)
> +    - 320x180
> +    - 640x360
> +    - 640x480
> +    - 1280x720
> +    - 1920x1080
> +    - 3840x2160
> +
> +Configuring a device
> +~~~~~~~~~~~~~~~~~~~~
> +
> +With the configuration generated, and optionally modified and re-validated, a
> +pipeline handler needs a method that allows an application to apply a
> +configuration to the hardware devices.
> +
> +The `PipelineHandler::configure()`_ method receives a valid
> +`CameraConfiguration`_ and applies the settings to hardware devices, using its
> +parameters to prepare a device for a streaming session with the desired
> +properties.
> +
> +.. _PipelineHandler::configure(): http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#a930f2a9cdfb51dfb4b9ca3824e84fc29
> +.. _CameraConfiguration: http://libcamera.org/api-html/classlibcamera_1_1CameraConfiguration.html
> +
> +Replace the contents of the stubbed ``PipelineHandlerVivid::configure`` method
> +with the following to obtain the camera data and stream configuration. This
> +pipeline handler supports only a single stream, so it directly obtains the first
> +``StreamConfiguration`` from the camera configuration. A pipeline handler with
> +multiple streams should inspect each StreamConfiguration and configure the
> +system accordingly.
> +
> +.. code-block:: cpp
> +
> +   VividCameraData *data = cameraData(camera);
> +   StreamConfiguration &cfg = config->at(0);
> +   int ret;
> +
> +The Vivid capture device is a V4L2 video device, so we use a `V4L2DeviceFormat`_
> +with the fourcc and size attributes to apply directly to the capture device
> +node. The fourcc attribute is a `V4L2PixelFormat`_ and differs from the
> +``libcamera::PixelFormat``. Converting the format requires knowledge of the
> +plane configuration for multiplanar formats, so you must explicitly convert it
> +using the helper ``V4L2VideoDevice::toV4L2PixelFormat()`` provided by the
> +V4L2VideoDevice instance of which the format will be applied on.
> +
> +.. _V4L2DeviceFormat: http://libcamera.org/api-html/classlibcamera_1_1V4L2DeviceFormat.html
> +.. _V4L2PixelFormat: http://libcamera.org/api-html/classlibcamera_1_1V4L2PixelFormat.html
> +
> +Add the following code beneath the code from above:
> +
> +.. code-block:: cpp
> +
> +   V4L2DeviceFormat format = {};
> +   format.fourcc = data->video_->toV4L2PixelFormat(cfg.pixelFormat);
> +   format.size = cfg.size;
> +
> +Set the video device format defined above using the
> +`V4L2VideoDevice::setFormat()`_ method. You should check if the kernel
> +driver has adjusted the format, as this shows the pipeline handler has failed to
> +handle the validation stages correctly, and the configure operation shall also
> +fail.
> +
> +.. _V4L2VideoDevice::setFormat(): http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#ad67b47dd9327ce5df43350b80c083cca
> +
> +Continue the implementation with the following code:
> +
> +.. code-block:: cpp
> +
> +   ret = data->video_->setFormat(&format);
> +   if (ret)
> +          return ret;
> +
> +   if (format.size != cfg.size ||
> +          format.fourcc != data->video_->toV4L2PixelFormat(cfg.pixelFormat))
> +          return -EINVAL;
> +
> +Finally, store and set stream-specific data reflecting the state of the stream.
> +Associate the configuration with the stream by using the
> +`StreamConfiguration::setStream`_ method, and set the values of individual
> +stream configuration members as required.
> +
> +.. _StreamConfiguration::setStream: http://libcamera.org/api-html/structlibcamera_1_1StreamConfiguration.html#a74a0eb44dad1b00112c7c0443ae54a12
> +
> +.. NOTE: the cfg.setStream() call here associates the stream to the
> +   StreamConfiguration however that should quite likely be done as part of
> +   the validation process. TBD
> +
> +Complete the configure implementation with the following code:
> +
> +.. code-block:: cpp
> +
> +   cfg.setStream(&data->stream_);
> +   cfg.stride = format.planes[0].bpl;
> +
> +   return 0;
> +
> +.. TODO: stride SHALL be assigned in validate
> +
> +Initializing device controls
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Pipeline handlers can optionally initialize the video devices and camera sensor
> +controls at system configuration time, to make sure to make sure they are
> +defaulted to sane values. Handling of device controls is again performed using
> +the libcamera `controls framework`_.
> +
> +.. _Controls Framework: http://libcamera.org/api-html/controls_8h.html
> +
> +This section is particularly specific to Vivid as it sets the initial values of
> +controls to match `Vivid Controls`_ defined by the kernel driver. You won’t need
> +any of the code below for your pipeline handler, but it’s included as an example
> +of how to implement functionality your pipeline handler might need.
> +
> +.. _Vivid Controls: https://www.kernel.org/doc/html/latest/admin-guide/media/vivid.html#controls
> +
> +We need to add some definitions at the top of the file for convenience. These
> +come directly from the kernel sources:
> +
> +.. code-block:: cpp
> +
> +   #define VIVID_CID_VIVID_BASE            (0x00f00000 | 0xf000)
> +   #define VIVID_CID_VIVID_CLASS           (0x00f00000 | 1)
> +   #define VIVID_CID_TEST_PATTERN          (VIVID_CID_VIVID_BASE  + 0)
> +   #define VIVID_CID_OSD_TEXT_MODE         (VIVID_CID_VIVID_BASE  + 1)
> +   #define VIVID_CID_HOR_MOVEMENT          (VIVID_CID_VIVID_BASE  + 2)
> +
> +We can now use the V4L2 control IDs to prepare a list of controls with the
> +`ControlList`_ class, and set them using the `ControlList::set()`_ method.
> +
> +.. _ControlList: http://libcamera.org/api-html/classlibcamera_1_1ControlList.html
> +.. _ControlList::set(): http://libcamera.org/api-html/classlibcamera_1_1ControlList.html#a74a1a29abff5243e6e37ace8e24eb4ba
> +
> +In our pipeline ``configure`` method, add the following code after the format
> +has been set and checked to initialise the ControlList and apply it to the
> +device:
> +
> +.. code-block:: cpp
> +
> +   ControlList controls(data->video_->controls());
> +   controls.set(VIVID_CID_TEST_PATTERN, 0);
> +   controls.set(VIVID_CID_OSD_TEXT_MODE, 0);
> +
> +   controls.set(V4L2_CID_BRIGHTNESS, 128);
> +   controls.set(V4L2_CID_CONTRAST, 128);
> +   controls.set(V4L2_CID_SATURATION, 128);
> +
> +   controls.set(VIVID_CID_HOR_MOVEMENT, 5);
> +
> +   ret = data->video_->setControls(&controls);
> +   if (ret) {
> +          LOG(VIVID, Error) << "Failed to set controls: " << ret;
> +          return ret < 0 ? ret : -EINVAL;
> +   }
> +
> +These controls configure VIVID to use a default test pattern, and enable all
> +on-screen display text, while configuring sensible brightness, contrast and
> +saturation values. Use the ``controls.set`` method to set individual controls.
> +
> +Buffer handling and stream control
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Once the system has been configured with the requested parameters, it is now
> +possible for applications to start capturing frames from the ``Camera`` device.
> +
> +Libcamera implements a per-frame request capture model, realized by queueing

s/Libcamera/libcamera/ (there are two other occurences)

> +``Request`` instances to a ``Camera`` object. Before applications can start
> +submitting capture requests the capture pipeline needs to be prepared to deliver
> +frames as soon as they are requested. Memory should be initialized and made
> +available to the devices which have to be started and ready to produce
> +images. At the end of a capture session the ``Camera`` device needs to be
> +stopped, to gracefully clean up any allocated memory and stop the hardware
> +devices. Pipeline handlers implement two methods for these purposes, the
> +``start()`` and ``stop()`` methods.
> +
> +The memory initialization phase that happens at ``start()`` time serves to
> +configure video devices to be able to use memory buffers exported as dma-buf
> +file descriptors. From the pipeline handlers perspective the video devices that
> +provide application facing streams always act as memory importers which use,
> +in V4L2 terminology, buffers of V4L2_MEMORY_DMABUF memory type.
> +
> +Libcamera also provides an API to allocate and export memory to applications
> +realized through the `exportFrameBuffers`_ function and the
> +`FrameBufferAllocator`_ class which will be presented later.
> +
> +.. _exportFrameBuffers: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#a6312a69da7129c2ed41f9d9f790adf7c
> +.. _FrameBufferAllocator: http://libcamera.org/api-html/classlibcamera_1_1FrameBufferAllocator.html
> +
> +Please refer to the V4L2VideoDevice API documentation, specifically the
> +`allocateBuffers`_, `importBuffers`_ and `exportBuffers`_ functions for a
> +detailed description of the video device memory management.
> +
> +.. _allocateBuffers: http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#a3a1a77e5e6c220ea7878e89485864a1c
> +.. _importBuffers: http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#a154f5283d16ebd5e15d63e212745cb64
> +.. _exportBuffers: http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#ae9c0b0a68f350725b63b73a6da5a2ecd
> +
> +Video memory buffers are represented in libcamera by the `FrameBuffer`_ class.
> +A ``FrameBuffer`` instance has to be associated to each ``Stream`` which is part
> +of a capture ``Request``. Pipeline handlers should prepare the capture devices
> +by importing the dma-buf file descriptors it needs to operate on. This operation
> +is performed by using the ``V4L2VideoDevice`` API, which provides an
> +``importBuffers()`` function that prepares the video device accordingly.
> +
> +.. _FrameBuffer: http://libcamera.org/api-html/classlibcamera_1_1FrameBuffer.html
> +
> +Implement the pipeline handler ``start()`` function by replacing the stub
> +version with the following code:
> +
> +.. code-block:: c++
> +
> +   VividCameraData *data = cameraData(camera);
> +   unsigned int count = data->stream_.configuration().bufferCount;
> +
> +   int ret = data->video_->importBuffers(count);
> +   if (ret < 0)
> +         return ret;
> +
> +   return 0;
> +
> +During the startup phase pipeline handlers allocate any internal buffer pool
> +required to transfer data between different components of the image capture
> +pipeline, for example, between the CSI-2 receiver and the ISP input. The example
> +pipeline does not require any internal pool, but examples are available in more
> +complex pipeline handlers in the libcamera code base.
> +
> +Applications might want to use memory allocated in the video devices instead of
> +allocating it from other parts of the system. libcamera provides an abstraction
> +to assist with this task in the `FrameBufferAllocator`_ class. The
> +``FrameBufferAllocator`` reserves memory for a ``Stream`` in the video device
> +and exports it as dma-buf file descriptors. From this point on, the allocated
> +``FrameBuffer`` are associated to ``Stream`` instances in a ``Request`` and then
> +imported by the pipeline hander in exactly the same fashion as if they were
> +allocated elsewhere.
> +
> +.. _FrameBufferAllocator: http://libcamera.org/api-html/classlibcamera_1_1FrameBufferAllocator.html
> +
> +Pipeline handlers support the ``FrameBufferAllocator`` operations by
> +implementing the `exportFrameBuffers`_ function, which will allocate memory in
> +the video device associated with a stream and export it.
> +
> +.. _exportFrameBuffers: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#a6312a69da7129c2ed41f9d9f790adf7c
> +
> +Implement the ``exportFrameBuffers`` stub method with the following code to
> +handle this:
> +
> +.. code-block:: cpp
> +
> +   unsigned int count = stream->configuration().bufferCount;
> +   VividCameraData *data = cameraData(camera);
> +
> +   return data->video_->exportBuffers(count, buffers);
> +
> +Once memory has been properly setup, the video devices can be started, to
> +prepare for capture operations. Complete the ``start`` method implementation
> +with the following code:
> +
> +.. code-block:: cpp
> +
> +   ret = data->video_->streamOn();
> +   if (ret < 0) {
> +          data->video_->releaseBuffers();
> +          return ret;
> +   }
> +
> +   return 0;
> +
> +The method starts the video device associated with the stream with the
> +`streamOn`_ method. If the call fails, the error value is propagated to the
> +caller and the `releaseBuffers`_ method releases any buffers to leave the device
> +in a consistent state. If your pipeline handler uses any image processing
> +algorithms, or other devices you should also stop them.
> +
> +.. _streamOn: http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#a588a5dc9d6f4c54c61136ac43ff9a8cc
> +.. _releaseBuffers: http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#a191619c152f764e03bc461611f3fcd35
> +
> +Of course we also need to handle the corresponding actions to stop streaming on
> +a device, Add the following to the ``stop`` method, to stop the stream with the
> +`streamOff`_ method and release all buffers.
> +
> +.. _streamOff: http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#a61998710615bdf7aa25a046c8565ed66
> +
> +.. code-block:: cpp
> +
> +   VividCameraData *data = cameraData(camera);
> +   data->video_->streamOff();
> +   data->video_->releaseBuffers();
> +
> +Queuing requests between applications and hardware
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +libcamera implements a streaming model based on capture requests queued by an
> +application to the ``Camera`` device. Each request contains at least one
> +``Stream`` instance with an associated ``FrameBuffer`` object.
> +
> +When an application sends a capture request, the pipeline handler identifies
> +which video devices have to be provided with buffers to generate a frame from
> +the enabled streams.
> +
> +This example pipeline handler identifies the buffer using the `findBuffer`_
> +helper from the only supported stream and queues it to the capture device
> +directly with the `queueBuffer`_ method provided by the V4L2VideoDevice.
> +
> +.. _findBuffer: http://libcamera.org/api-html/classlibcamera_1_1Request.html#ac66050aeb9b92c64218945158559c4d4
> +.. _queueBuffer: http://libcamera.org/api-html/classlibcamera_1_1V4L2VideoDevice.html#a594cd594686a8c1cf9ae8dba0b2a8a75
> +
> +Replace the stubbed contents of ``queueRequestDevice`` with the following:
> +
> +.. code-block:: cpp
> +
> +   VividCameraData *data = cameraData(camera);
> +   FrameBuffer *buffer = request->findBuffer(&data->stream_);
> +   if (!buffer) {
> +          LOG(VIVID, Error)
> +                  << "Attempt to queue request with invalid stream";
> +
> +          return -ENOENT;
> +    }
> +
> +   int ret = data->video_->queueBuffer(buffer);
> +   if (ret < 0)
> +          return ret;
> +
> +   return 0;
> +
> +Processing controls
> +~~~~~~~~~~~~~~~~~~~
> +
> +Capture requests not only contain streams and memory buffers, but can
> +optionally contain a list of controls the application has set to modify the
> +streaming parameters.
> +
> +Applications can set controls registered by the pipeline handler in the
> +initialization phase, as explained in the `Registering controls and properties`_
> +section.
> +
> +Implement a ``processControls`` method above the ``queueRequestDevice`` method
> +to loop through the control list received with each request, and inspect the
> +control values. Controls may need to be converted between the libcamera control
> +range definitions and their corresponding values on the device before being set.
> +
> +.. code-block:: cpp
> +
> +   int PipelineHandlerVivid::processControls(VividCameraData *data, Request *request)
> +   {
> +          ControlList controls(data->video_->controls());
> +
> +          for (auto it : request->controls()) {
> +                 unsigned int id = it.first;
> +                 unsigned int offset;
> +                 uint32_t cid;
> +
> +                 if (id == controls::Brightness) {
> +                        cid = V4L2_CID_BRIGHTNESS;
> +                        offset = 128;
> +                 } else if (id == controls::Contrast) {
> +                        cid = V4L2_CID_CONTRAST;
> +                        offset = 0;
> +                 } else if (id == controls::Saturation) {
> +                        cid = V4L2_CID_SATURATION;
> +                        offset = 0;
> +                 } else {
> +                        continue;
> +                 }
> +
> +                 int32_t value = lroundf(it.second.get<float>() * 128 + offset);
> +                 controls.set(cid, utils::clamp(value, 0, 255));
> +          }
> +
> +          for (const auto &ctrl : controls)
> +                 LOG(VIVID, Debug)
> +                        << "Setting control " << utils::hex(ctrl.first)
> +                        << " to " << ctrl.second.toString();
> +
> +          int ret = data->video_->setControls(&controls);
> +          if (ret) {
> +                 LOG(VIVID, Error) << "Failed to set controls: " << ret;
> +                 return ret < 0 ? ret : -EINVAL;
> +          }
> +
> +          return ret;
> +   }
> +
> +Declare the function prototype for the ``processControls`` method within the
> +private ``PipelineHandlerVivid`` class members, as it is only used internally as
> +a helper when processing Requests.
> +
> +.. code-block:: cpp
> +
> +   private:
> +        int processControls(VividCameraData *data, Request *request);
> +
> +A pipeline handler is responsible for applying controls provided in a Request to
> +the relevant hardware devices. This could be directly on the capture device, or
> +where appropriate by setting controls on V4L2Subdevices directly. Each pipeline
> +handler is responsible for understanding the correct procedure for applying
> +controls to the device they support.
> +
> +This example pipeline handler applies controls during the `queueRequestDevice`_
> +method for each request, and applies them to the capture device through the
> +capture node.
> +
> +.. _queueRequestDevice: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html#a106914cca210640c9da9ee1f0419e83c
> +
> +In the ``queueRequestDevice`` method, replace the following:
> +
> +.. code-block:: cpp
> +
> +   int ret = data->video_->queueBuffer(buffer);
> +   if (ret < 0)
> +        return ret;
> +
> +With the following code:
> +
> +.. code-block:: cpp
> +
> +   int ret = processControls(data, request);
> +   if (ret < 0)
> +        return ret;
> +
> +   ret = data->video_->queueBuffer(buffer);
> +   if (ret < 0)
> +        return ret;
> +
> +We also need to add the following include directive to support the control
> +value translation operations:
> +
> +.. code-block:: cpp
> +
> +   #include <math.h>
> +
> +Frame completion and event handling
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Libcamera implements a signals and slots mechanism (similar to `Qt Signals and
> +Slots`_) to connect event sources with callbacks to handle them.
> +
> +As a general summary, a ``Slot`` can be connected to a ``Signal``, which when
> +emitted triggers the execution of the connected slots.  A detailed description
> +of the libcamera implementation is available in the `libcamera Signal and Slot`_
> +classes documentation.
> +
> +.. _Qt Signals and Slots: https://doc.qt.io/qt-5/signalsandslots.html
> +.. _libcamera Signal and Slot: http://libcamera.org/api-html/classlibcamera_1_1Signal.html#details
> +
> +In order to notify applications about the availability of new frames and data,
> +the ``Camera`` device exposes two ``Signals`` which applications can connect to
> +be notified of frame completion events. The ``bufferComplete`` signal serves to
> +report to applications the completion event of a single ``Stream`` part of a
> +``Request``, while the ``requestComplete`` signal notifies the completion of all
> +the ``Streams`` and data submitted as part of a request. This mechanism allows
> +implementation of partial request completion, which allows an application to
> +inspect completed buffers associated with the single streams without waiting for
> +all of them to be ready.
> +
> +The ``bufferComplete`` and ``requestComplete`` signals are emitted by the
> +``Camera`` device upon notifications received from the pipeline handler, which
> +tracks the buffers and request completion status.
> +
> +The single buffer completion notification is implemented by pipeline handlers by
> +`connecting`_ the ``bufferReady`` signal of the capture devices they have queued
> +buffers to, to a member function slot that handles processing of the completed
> +frames. When a buffer is ready, the pipeline handler must propagate the
> +completion of that buffer to the Camera by using the PipelineHandler base class
> +``completeBuffer`` function. When all of the buffers referenced by a ``Request``
> +have been completed, the pipeline handler must again notify the ``Camera`` using
> +the PipelineHandler base class ``completeRequest`` function. The PipelineHandler
> +class implementation makes sure the request completion notifications are
> +delivered to applications in the same order as they have been submitted.
> +
> +.. _connecting: http://libcamera.org/api-html/classlibcamera_1_1Signal.html#aa04db72d5b3091ffbb4920565aeed382
> +
> +Returning to the ``int VividCameraData::init()`` method, add the following above
> +the closing ``return 0;`` to connects the pipeline handler ``bufferReady``
> +method to the V4L2 device buffer signal.
> +
> +.. code-block:: cpp
> +
> +   video_->bufferReady.connect(this, &VividCameraData::bufferReady);
> +
> +Create the matching ``VividCameraData::bufferReady`` method after your
> +VividCameradata::init() impelementation.
> +
> +The ``bufferReady`` method obtains the request from the buffer using the
> +``request`` method, and notifies the ``Camera`` that the buffer and
> +request are completed. In this simpler pipeline handler, there is only one
> +stream, so it completes the request immediately. You can find a more complex
> +example of event handling with supporting multiple streams in the libcamera
> +code-base.
> +
> +.. TODO: Add link
> +
> +.. code-block:: cpp
> +
> +   void VividCameraData::bufferReady(FrameBuffer *buffer)
> +   {
> +          Request *request = buffer->request();
> +
> +          pipe_->completeBuffer(camera_, request, buffer);
> +          pipe_->completeRequest(camera_, request);
> +   }
> +
> +Testing a pipeline handler
> +~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Once you've built the pipeline handler, we ca rebuild the code base, and test
> +capture through the pipeline through both of the cam and qcam utilities.
> +
> +.. code-block:: shell
> +
> +   ninja -C build
> +   ./build/src/cam/cam -c vivid -C 5
> +
> +To test that the pipeline handler can detect a device, and capture input.
> +
> +Running the command above outputs (a lot of) information about pixel formats,
> +and then starts capturing frame data, and should provide an output such as the
> +following:
> +
> +.. code-block:: none
> +
> +   user@dev:/home/libcamera$ ./build/src/cam/cam -c vivid -C5
> +   [42:34:08.573066847] [186470]  INFO IPAManager ipa_manager.cpp:136 libcamera is not installed. Adding '/home/libcamera/build/src/ipa' to the IPA search path
> +   [42:34:08.575908115] [186470]  INFO Camera camera_manager.cpp:287 libcamera v0.0.11+876-7b27d262
> +   [42:34:08.610334268] [186471]  INFO IPAProxy ipa_proxy.cpp:122 libcamera is not installed. Loading IPA configuration from '/home/libcamera/src/ipa/vimc/data'
> +   Using camera vivid
> +   [42:34:08.618462130] [186470]  WARN V4L2 v4l2_pixelformat.cpp:176 Unsupported V4L2 pixel format Y10
> +   ... <remaining Unsupported V4L2 pixel format warnings can be ignored>
> +   [42:34:08.619901297] [186470]  INFO Camera camera.cpp:793 configuring streams: (0) 1280x720-BGR888
> +   Capture 5 frames
> +   fps: 0.00 stream0 seq: 000000 bytesused: 2764800
> +   fps: 4.98 stream0 seq: 000001 bytesused: 2764800
> +   fps: 5.00 stream0 seq: 000002 bytesused: 2764800
> +   fps: 5.03 stream0 seq: 000003 bytesused: 2764800
> +   fps: 5.03 stream0 seq: 000004 bytesused: 2764800
> +
> +This demonstrates that the pipeline handler is successfully capturing frames,
> +but it is helpful to see the visual output and validate the images are being
> +processed correctly. The libcamera project also implements a Qt based
> +application which will render the frames in a window for visual inspection:
> +
> +.. code-block:: shell
> +
> +   ./build/src/qcam/qcam -c vivid
> +
> +.. TODO: Running qcam with the vivid pipeline handler appears to have a bug and
> +         no visual frames are seen. However disabling zero-copy on qcam renders
> +         them successfully.
> \ No newline at end of file
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index cfcfd388699b..fb391d2b6ebf 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -14,3 +14,4 @@
>     Contribute <contributing>
>  
>     Developer Guide <guides/introduction>
> +   Pipeline Handler Writers Guide <guides/pipeline-handler>

Writers or Writer's ?

Other improvements can go on top.

Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> diff --git a/Documentation/meson.build b/Documentation/meson.build
> index dd7ae700af11..9f6c67071da9 100644
> --- a/Documentation/meson.build
> +++ b/Documentation/meson.build
> @@ -53,6 +53,7 @@ if sphinx.found()
>          'docs.rst',
>          'index.rst',
>          'guides/introduction.rst',
> +        'guides/pipeline-handler.rst',
>      ]
>  
>      release = 'release=v' + libcamera_git_version