[{"id":30823,"web_url":"https://patchwork.libcamera.org/comment/30823/","msgid":"<20240815001824.GH22567@pendragon.ideasonboard.com>","date":"2024-08-15T00:18:24","subject":"Re: [PATCH 3/7] Documentation: Synchronise libcamera architecture\n\tdetails","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Dan,\n\nThank you for the patch.\n\nOn Fri, Aug 09, 2024 at 03:53:00PM +0100, Daniel Scally wrote:\n> There are two near-duplicate instances of the libcamera architecture\n> detail in the Documentation, in docs.rst and guides/introduction.rst.\n> The latter is more up-to-date, so remove it from the introduction\n> file (which will soon be deprecated) and update the section in docs.\n> \n> Signed-off-by: Daniel Scally \n> ---\n> Documentation/docs.rst | 221 ++++++++++++++------------\n> Documentation/guides/introduction.rst | 137 +---------------\n> 2 files changed, 121 insertions(+), 237 deletions(-)\n\nI like the diffstat :-)\n\n> diff --git a/Documentation/docs.rst b/Documentation/docs.rst\n> index 5871961c..10f07a9e 100644\n> --- a/Documentation/docs.rst\n> +++ b/Documentation/docs.rst\n> @@ -243,116 +243,135 @@ The camera stack comprises four software layers. From bottom to top:\n> libcamera Architecture\n> ======================\n> \n> +While offering a unified API towards upper layers, and presenting itself as a\n> +single library, libcamera isn't monolithic. It exposes multiple components\n> +through its public API and is built around a set of separate helpers internally.\n> +Hardware abstractions are handled through the use of device-specific components\n> +where required and dynamically loadable plugins are used to separate image\n> +processing algorithms from the core libcamera codebase.\n> +\n> ::\n> \n> - ---------------------------< libcamera Public API >---------------------------\n> - ^ ^\n> - | |\n> - v v\n> - +-------------+ +-------------------------------------------------+\n> - | Camera | | Camera Device |\n> - | Devices | | +---------------------------------------------+ |\n> - | Manager | | | Device-Agnostic | |\n> - +-------------+ | | | |\n> - ^ | | +------------------------+ |\n> - | | | | ~~~~~~~~~~~~~~~~~~~~~ |\n> - | | | | { +---------------+ } |\n> - | | | | } | ////Image//// | { |\n> - | | | | <-> | /Processing// | } |\n> - | | | | } | /Algorithms// | { |\n> - | | | | { +---------------+ } |\n> - | | | | ~~~~~~~~~~~~~~~~~~~~~ |\n> - | | | | ======================== |\n> - | | | | +---------------+ |\n> - | | | | | //Pipeline/// | |\n> - | | | | <-> | ///Handler/// | |\n> - | | | | | ///////////// | |\n> - | | +--------------------+ +---------------+ |\n> - | | Device-Specific |\n> - | +-------------------------------------------------+\n> - | ^ ^\n> - | | |\n> - v v v\n> - +--------------------------------------------------------------------+\n> - | Helpers and Support Classes |\n> - | +-------------+ +-------------+ +-------------+ +-------------+ |\n> - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |\n> - | | Support | | Allocator | | IPC | | Manager | |\n> - | +-------------+ +-------------+ +-------------+ +-------------+ |\n> - | +-------------+ +-------------+ |\n> - | | Pipeline | | ... | |\n> - | | Runner | | | |\n> - | +-------------+ +-------------+ |\n> - +--------------------------------------------------------------------+\n> -\n> - /// Device-Specific Components\n> - ~~~ Sandboxing\n> -\n> -While offering a unified API towards upper layers, and presenting\n> -itself as a single library, libcamera isn't monolithic. It exposes\n> -multiple components through its public API, is built around a set of\n> -separate helpers internally, uses device-specific components and can\n> -load dynamic plugins.\n> -\n> -Camera Devices Manager\n> - The Camera Devices Manager provides a view of available cameras\n> - in the system. It performs cold enumeration and runtime camera\n> - management, and supports a hotplug notification mechanism in its\n> - public API.\n> -\n> - To avoid the cost associated with cold enumeration of all devices\n> - at application start, and to arbitrate concurrent access to camera\n> - devices, the Camera Devices Manager could later be split to a\n> - separate service, possibly with integration in platform-specific\n> - device management.\n> + --------------------------< libcamera Public API >---------------------------\n> + ^ ^\n> + | |\n> + v v\n> + +-------------+ +---------------------------------------------------+\n> + | Camera | | Camera Device |\n> + | Manager | | +-----------------------------------------------+ |\n> + +-------------+ | | Device-Agnostic | |\n> + ^ | | | |\n> + | | | +--------------------------+ |\n> + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |\n> + | | | | { +-----------------+ } |\n> + | | | | } | //// Image //// | { |\n> + | | | | <-> | / Processing // | } |\n> + | | | | } | / Algorithms // | { |\n> + | | | | { +-----------------+ } |\n> + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |\n> + | | | | ========================== |\n> + | | | | +-----------------+ |\n> + | | | | | // Pipeline /// | |\n> + | | | | <-> | /// Handler /// | |\n> + | | | | | /////////////// | |\n> + | | +--------------------+ +-----------------+ |\n> + | | Device-Specific |\n> + | +---------------------------------------------------+\n> + | ^ ^\n> + | | |\n> + v v v\n> + +--------------------------------------------------------------------+\n> + | Helpers and Support Classes |\n> + | +-------------+ +-------------+ +-------------+ +-------------+ |\n> + | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |\n> + | | Support | | Allocator | | IPC | | Manager | |\n> + | +-------------+ +-------------+ +-------------+ +-------------+ |\n> + | +-------------+ +-------------+ |\n> + | | Pipeline | | ... | |\n> + | | Runner | | | |\n> + | +-------------+ +-------------+ |\n> + +--------------------------------------------------------------------+\n> +\n> + /// Device-Specific Components\n> + ~~~ Sandboxing\n> +\n> +Camera Manager\n> + The Camera Manager enumerates cameras and instantiates Pipeline Handlers to\n> + manage each Camera that libcamera supports. The Camera Manager supports\n> + hotplug detection and notification events when supported by the underlying\n> + kernel devices.\n> +\n> + There is only ever one instance of the Camera Manager running per application.\n> + Each application's instance of the Camera Manager ensures that only a single\n> + application can take control of a camera device at once.\n> +\n> + Read the `Camera Manager API`_ documentation for more details.\n> +\n> +.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html\n\nIt would be nice to avoid linking to an absolute URL, but that won't be\neasy to solve. As it's not a new issue, it's fine for now.\n\n> \n> Camera Device\n> - The Camera Device represents a camera device to upper layers. It\n> - exposes full control of the device through the public API, and is\n> - thus the highest level object exposed by libcamera.\n> + The Camera class represents a single item of camera hardware that is capable\n> + of producing one or more image streams, and provides the API to interact with\n> + the underlying device.\n> +\n> + If a system has multiple instances of the same hardware attached, each has its\n> + own instance of the camera class.\n> +\n> + The API exposes full control of the device to upper layers of libcamera through\n> + the public API, making it the highest level object libcamera exposes, and the\n> + object that all other API operations interact with from configuration to\n> + capture.\n> \n> - Camera Device instances are created by the Camera Devices\n> - Manager. An optional function to create new instances could be exposed\n> - through the public API to speed up initialization when the upper\n> - layer knows how to directly address camera devices present in the\n> - system.\n> + Read the `Camera API`_ documentation for more details.\n> +\n> +.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html\n> \n> Pipeline Handler\n> - The Pipeline Handler manages complex pipelines exposed by the kernel drivers\n> - through the Media Controller and V4L2 APIs. It abstracts pipeline handling to\n> - hide device-specific details to the rest of the library, and implements both\n> - pipeline configuration based on stream configuration, and pipeline runtime\n> - execution and scheduling when needed by the device.\n> -\n> - This component is device-specific and is part of the libcamera code base. As\n> - such it is covered by the same free software license as the rest of libcamera\n> - and needs to be contributed upstream by device vendors. The Pipeline Handler\n> - lives in the same process as the rest of the library, and has access to all\n> - helpers and kernel camera-related devices.\n> + The Pipeline Handler manages the complex pipelines exposed by the kernel\n> + drivers through the Media Controller and V4L2 APIs. It abstracts pipeline\n> + handling to hide device-specific details from the rest of the library, and\n> + implements both pipeline configuration based on stream configuration, and\n> + pipeline runtime execution and scheduling when needed by the device.\n> +\n> + The Pipeline Handler lives in the same process as the rest of the library, and\n> + has access to all helpers and kernel camera-related devices.\n> +\n> + Hardware abstraction is handled by device specific Pipeline Handlers which are\n> + derived from the Pipeline Handler base class allowing commonality to be shared\n> + among the implementations.\n> +\n> + Derived pipeline handlers create Camera device instances based on the devices\n> + they detect and support on the running system, and are responsible for\n> + managing the interactions with a camera device.\n> +\n> + More details can be found in the `PipelineHandler API`_ documentation, and the\n> + :doc:`Pipeline Handler Writers Guide `.\n> +\n> +.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html\n\nWith the documentation for the public and internal APIs being split,\nthis will be a dead link (once the copy on libcamera.org gets updated).\nIt should be fixed in a separate patch.\n\nReviewed-by: Laurent Pinchart \n\n> \n> Image Processing Algorithms\n> - Together with the hardware image processing and hardware statistics\n> - collection, the Image Processing Algorithms implement 3A (Auto-Exposure,\n> - Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU\n> - and interact with the kernel camera devices to control hardware image\n> - processing based on the parameters supplied by upper layers, closing the\n> - control loop of the ISP.\n> -\n> - This component is device-specific and is loaded as an external plugin. It can\n> - be part of the libcamera code base, in which case it is covered by the same\n> - license, or provided externally as an open-source or closed-source component.\n> -\n> - The component is sandboxed and can only interact with libcamera through\n> - internal APIs specifically marked as such. In particular it will have no\n> - direct access to kernel camera devices, and all its accesses to image and\n> - metadata will be mediated by dmabuf instances explicitly passed to the\n> - component. The component must be prepared to run in a process separate from\n> - the main libcamera process, and to have a very restricted view of the system,\n> - including no access to networking APIs and limited access to file systems.\n> -\n> - The sandboxing mechanism isn't defined by libcamera. One example\n> - implementation will be provided as part of the project, and platforms vendors\n> - will be able to provide their own sandboxing mechanism as a plugin.\n> + An image processing algorithm (IPA) component is a loadable plugin that\n> + implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other\n> + algorithms.\n> +\n> + The algorithms run on the CPU and interact with the camera devices through the\n> + Pipeline Handler to control hardware image processing based on the parameters\n> + supplied by upper layers, maintaining state and closing the control loop\n> + of the ISP.\n> +\n> + The component is sandboxed and can only interact with libcamera through the\n> + API provided by the Pipeline Handler and an IPA has no direct access to kernel\n> + camera devices.\n> +\n> + Open source IPA modules built with libcamera can be run in the same process\n> + space as libcamera, however external IPA modules are run in a separate process\n> + from the main libcamera process. IPA modules have a restricted view of the\n> + system, including no access to networking APIs and limited access to file\n> + systems.\n> +\n> + IPA modules are only required for platforms and devices with an ISP controlled\n> + by the host CPU. Camera sensors which have an integrated ISP are not\n> + controlled through the IPA module.\n> \n> libcamera should provide a basic implementation of Image Processing\n> Algorithms, to serve as a reference for the internal API. Device vendors are\n> diff --git a/Documentation/guides/introduction.rst b/Documentation/guides/introduction.rst\n> index 8368bd4a..1898d5fe 100644\n> --- a/Documentation/guides/introduction.rst\n> +++ b/Documentation/guides/introduction.rst\n> @@ -27,8 +27,7 @@ desirable results from the camera.\n> \n> \n> In this developers guide, we will explore the `Camera Stack`_ and how it is\n> -can be visualised at a high level, and explore the internal `Architecture`_ of\n> -the libcamera library with its components. The current `Platform Support`_ is\n> +can be visualised at a high level. The current `Platform Support`_ is\n> detailed, as well as an overview of the `Licensing`_ requirements of the\n> project.\n> \n> @@ -148,140 +147,6 @@ Native libcamera API\n> \n> .. _GStreamer element: https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html\n> \n> -Architecture\n> -------------\n> -\n> -While offering a unified API towards upper layers, and presenting itself as a\n> -single library, libcamera isn't monolithic. It exposes multiple components\n> -through its public API and is built around a set of separate helpers internally.\n> -Hardware abstractions are handled through the use of device-specific components\n> -where required and dynamically loadable plugins are used to separate image\n> -processing algorithms from the core libcamera codebase.\n> -\n> -::\n> -\n> - --------------------------< libcamera Public API >---------------------------\n> - ^ ^\n> - | |\n> - v v\n> - +-------------+ +---------------------------------------------------+\n> - | Camera | | Camera Device |\n> - | Manager | | +-----------------------------------------------+ |\n> - +-------------+ | | Device-Agnostic | |\n> - ^ | | | |\n> - | | | +--------------------------+ |\n> - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |\n> - | | | | { +-----------------+ } |\n> - | | | | } | //// Image //// | { |\n> - | | | | <-> | / Processing // | } |\n> - | | | | } | / Algorithms // | { |\n> - | | | | { +-----------------+ } |\n> - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |\n> - | | | | ========================== |\n> - | | | | +-----------------+ |\n> - | | | | | // Pipeline /// | |\n> - | | | | <-> | /// Handler /// | |\n> - | | | | | /////////////// | |\n> - | | +--------------------+ +-----------------+ |\n> - | | Device-Specific |\n> - | +---------------------------------------------------+\n> - | ^ ^\n> - | | |\n> - v v v\n> - +--------------------------------------------------------------------+\n> - | Helpers and Support Classes |\n> - | +-------------+ +-------------+ +-------------+ +-------------+ |\n> - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |\n> - | | Support | | Allocator | | IPC | | Manager | |\n> - | +-------------+ +-------------+ +-------------+ +-------------+ |\n> - | +-------------+ +-------------+ |\n> - | | Pipeline | | ... | |\n> - | | Runner | | | |\n> - | +-------------+ +-------------+ |\n> - +--------------------------------------------------------------------+\n> -\n> - /// Device-Specific Components\n> - ~~~ Sandboxing\n> -\n> -\n> -Camera Manager\n> - The Camera Manager enumerates cameras and instantiates Pipeline Handlers to\n> - manage each Camera that libcamera supports. The Camera Manager supports\n> - hotplug detection and notification events when supported by the underlying\n> - kernel devices.\n> -\n> - There is only ever one instance of the Camera Manager running per application.\n> - Each application's instance of the Camera Manager ensures that only a single\n> - application can take control of a camera device at once.\n> -\n> - Read the `Camera Manager API`_ documentation for more details.\n> -\n> -.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html\n> -\n> -Camera Device\n> - The Camera class represents a single item of camera hardware that is capable\n> - of producing one or more image streams, and provides the API to interact with\n> - the underlying device.\n> -\n> - If a system has multiple instances of the same hardware attached, each has its\n> - own instance of the camera class.\n> -\n> - The API exposes full control of the device to upper layers of libcamera through\n> - the public API, making it the highest level object libcamera exposes, and the\n> - object that all other API operations interact with from configuration to\n> - capture.\n> -\n> - Read the `Camera API`_ documentation for more details.\n> -\n> -.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html\n> -\n> -Pipeline Handler\n> - The Pipeline Handler manages the complex pipelines exposed by the kernel\n> - drivers through the Media Controller and V4L2 APIs. It abstracts pipeline\n> - handling to hide device-specific details from the rest of the library, and\n> - implements both pipeline configuration based on stream configuration, and\n> - pipeline runtime execution and scheduling when needed by the device.\n> -\n> - The Pipeline Handler lives in the same process as the rest of the library, and\n> - has access to all helpers and kernel camera-related devices.\n> -\n> - Hardware abstraction is handled by device specific Pipeline Handlers which are\n> - derived from the Pipeline Handler base class allowing commonality to be shared\n> - among the implementations.\n> -\n> - Derived pipeline handlers create Camera device instances based on the devices\n> - they detect and support on the running system, and are responsible for\n> - managing the interactions with a camera device.\n> -\n> - More details can be found in the `PipelineHandler API`_ documentation, and the\n> - `Pipeline Handler Writers Guide`_.\n> -\n> -.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html\n> -\n> -Image Processing Algorithms\n> - An image processing algorithm (IPA) component is a loadable plugin that\n> - implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other\n> - algorithms.\n> -\n> - The algorithms run on the CPU and interact with the camera devices through the\n> - Pipeline Handler to control hardware image processing based on the parameters\n> - supplied by upper layers, maintaining state and closing the control loop\n> - of the ISP.\n> -\n> - The component is sandboxed and can only interact with libcamera through the\n> - API provided by the Pipeline Handler and an IPA has no direct access to kernel\n> - camera devices.\n> -\n> - Open source IPA modules built with libcamera can be run in the same process\n> - space as libcamera, however external IPA modules are run in a separate process\n> - from the main libcamera process. IPA modules have a restricted view of the\n> - system, including no access to networking APIs and limited access to file\n> - systems.\n> -\n> - IPA modules are only required for platforms and devices with an ISP controlled\n> - by the host CPU. Camera sensors which have an integrated ISP are not\n> - controlled through the IPA module.\n> -\n> Platform Support\n> ----------------\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5BA5DBDB13\n\tfor ;\n\tThu, 15 Aug 2024 00:18:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2EF0B633BD;\n\tThu, 15 Aug 2024 02:18:51 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3919963382\n\tfor ;\n\tThu, 15 Aug 2024 02:18:50 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id B4746AD8;\n\tThu, 15 Aug 2024 02:17:51 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Y+ILbYK2\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1723681072;\n\tbh=TA/frlcMJhlGwSaTfCW4N8+bfwvidVj4ldg7QnvBzJo=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=Y+ILbYK2H9OJCGT91LRGS68pOp/MBMVLOa1cFwgST37PUhyMYL8eaNLvVec1jHZw2\n\tCw5KIhP8DScTg0kdPm65Lyrrbv8GLfaM49JQe+iiYnvGy0TjwsTh2irnos2+42KytJ\n\tlhlt5msTfhio0QePG2FKv4FEewRPAjIgwgCp712M=","Date":"Thu, 15 Aug 2024 03:18:24 +0300","From":"Laurent Pinchart ","To":"Daniel Scally ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH 3/7] Documentation: Synchronise libcamera architecture\n\tdetails","Message-ID":"<20240815001824.GH22567@pendragon.ideasonboard.com>","References":"<20240809145304.537551-1-dan.scally@ideasonboard.com>\n\t<20240809145304.537551-4-dan.scally@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20240809145304.537551-4-dan.scally@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]