@@ -243,116 +243,135 @@ The camera stack comprises four software layers. From bottom to top:
libcamera Architecture
======================
+While offering a unified API towards upper layers, and presenting itself as a
+single library, libcamera isn't monolithic. It exposes multiple components
+through its public API and is built around a set of separate helpers internally.
+Hardware abstractions are handled through the use of device-specific components
+where required and dynamically loadable plugins are used to separate image
+processing algorithms from the core libcamera codebase.
+
::
- ---------------------------< libcamera Public API >---------------------------
- ^ ^
- | |
- v v
- +-------------+ +-------------------------------------------------+
- | Camera | | Camera Device |
- | Devices | | +---------------------------------------------+ |
- | Manager | | | Device-Agnostic | |
- +-------------+ | | | |
- ^ | | +------------------------+ |
- | | | | ~~~~~~~~~~~~~~~~~~~~~ |
- | | | | { +---------------+ } |
- | | | | } | ////Image//// | { |
- | | | | <-> | /Processing// | } |
- | | | | } | /Algorithms// | { |
- | | | | { +---------------+ } |
- | | | | ~~~~~~~~~~~~~~~~~~~~~ |
- | | | | ======================== |
- | | | | +---------------+ |
- | | | | | //Pipeline/// | |
- | | | | <-> | ///Handler/// | |
- | | | | | ///////////// | |
- | | +--------------------+ +---------------+ |
- | | Device-Specific |
- | +-------------------------------------------------+
- | ^ ^
- | | |
- v v v
- +--------------------------------------------------------------------+
- | Helpers and Support Classes |
- | +-------------+ +-------------+ +-------------+ +-------------+ |
- | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
- | | Support | | Allocator | | IPC | | Manager | |
- | +-------------+ +-------------+ +-------------+ +-------------+ |
- | +-------------+ +-------------+ |
- | | Pipeline | | ... | |
- | | Runner | | | |
- | +-------------+ +-------------+ |
- +--------------------------------------------------------------------+
-
- /// Device-Specific Components
- ~~~ Sandboxing
-
-While offering a unified API towards upper layers, and presenting
-itself as a single library, libcamera isn't monolithic. It exposes
-multiple components through its public API, is built around a set of
-separate helpers internally, uses device-specific components and can
-load dynamic plugins.
-
-Camera Devices Manager
- The Camera Devices Manager provides a view of available cameras
- in the system. It performs cold enumeration and runtime camera
- management, and supports a hotplug notification mechanism in its
- public API.
-
- To avoid the cost associated with cold enumeration of all devices
- at application start, and to arbitrate concurrent access to camera
- devices, the Camera Devices Manager could later be split to a
- separate service, possibly with integration in platform-specific
- device management.
+ --------------------------< libcamera Public API >---------------------------
+ ^ ^
+ | |
+ v v
+ +-------------+ +---------------------------------------------------+
+ | Camera | | Camera Device |
+ | Manager | | +-----------------------------------------------+ |
+ +-------------+ | | Device-Agnostic | |
+ ^ | | | |
+ | | | +--------------------------+ |
+ | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
+ | | | | { +-----------------+ } |
+ | | | | } | //// Image //// | { |
+ | | | | <-> | / Processing // | } |
+ | | | | } | / Algorithms // | { |
+ | | | | { +-----------------+ } |
+ | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
+ | | | | ========================== |
+ | | | | +-----------------+ |
+ | | | | | // Pipeline /// | |
+ | | | | <-> | /// Handler /// | |
+ | | | | | /////////////// | |
+ | | +--------------------+ +-----------------+ |
+ | | Device-Specific |
+ | +---------------------------------------------------+
+ | ^ ^
+ | | |
+ v v v
+ +--------------------------------------------------------------------+
+ | Helpers and Support Classes |
+ | +-------------+ +-------------+ +-------------+ +-------------+ |
+ | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
+ | | Support | | Allocator | | IPC | | Manager | |
+ | +-------------+ +-------------+ +-------------+ +-------------+ |
+ | +-------------+ +-------------+ |
+ | | Pipeline | | ... | |
+ | | Runner | | | |
+ | +-------------+ +-------------+ |
+ +--------------------------------------------------------------------+
+
+ /// Device-Specific Components
+ ~~~ Sandboxing
+
+Camera Manager
+ The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
+ manage each Camera that libcamera supports. The Camera Manager supports
+ hotplug detection and notification events when supported by the underlying
+ kernel devices.
+
+ There is only ever one instance of the Camera Manager running per application.
+ Each application's instance of the Camera Manager ensures that only a single
+ application can take control of a camera device at once.
+
+ Read the `Camera Manager API`_ documentation for more details.
+
+.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
Camera Device
- The Camera Device represents a camera device to upper layers. It
- exposes full control of the device through the public API, and is
- thus the highest level object exposed by libcamera.
+ The Camera class represents a single item of camera hardware that is capable
+ of producing one or more image streams, and provides the API to interact with
+ the underlying device.
+
+ If a system has multiple instances of the same hardware attached, each has its
+ own instance of the camera class.
+
+ The API exposes full control of the device to upper layers of libcamera through
+ the public API, making it the highest level object libcamera exposes, and the
+ object that all other API operations interact with from configuration to
+ capture.
- Camera Device instances are created by the Camera Devices
- Manager. An optional function to create new instances could be exposed
- through the public API to speed up initialization when the upper
- layer knows how to directly address camera devices present in the
- system.
+ Read the `Camera API`_ documentation for more details.
+
+.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
Pipeline Handler
- The Pipeline Handler manages complex pipelines exposed by the kernel drivers
- through the Media Controller and V4L2 APIs. It abstracts pipeline handling to
- hide device-specific details to the rest of the library, and implements both
- pipeline configuration based on stream configuration, and pipeline runtime
- execution and scheduling when needed by the device.
-
- This component is device-specific and is part of the libcamera code base. As
- such it is covered by the same free software license as the rest of libcamera
- and needs to be contributed upstream by device vendors. The Pipeline Handler
- lives in the same process as the rest of the library, and has access to all
- helpers and kernel camera-related devices.
+ The Pipeline Handler manages the complex pipelines exposed by the kernel
+ drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
+ handling to hide device-specific details from the rest of the library, and
+ implements both pipeline configuration based on stream configuration, and
+ pipeline runtime execution and scheduling when needed by the device.
+
+ The Pipeline Handler lives in the same process as the rest of the library, and
+ has access to all helpers and kernel camera-related devices.
+
+ Hardware abstraction is handled by device specific Pipeline Handlers which are
+ derived from the Pipeline Handler base class allowing commonality to be shared
+ among the implementations.
+
+ Derived pipeline handlers create Camera device instances based on the devices
+ they detect and support on the running system, and are responsible for
+ managing the interactions with a camera device.
+
+ More details can be found in the `PipelineHandler API`_ documentation, and the
+ :doc:`Pipeline Handler Writers Guide <guides/pipeline-handler>`.
+
+.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
Image Processing Algorithms
- Together with the hardware image processing and hardware statistics
- collection, the Image Processing Algorithms implement 3A (Auto-Exposure,
- Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU
- and interact with the kernel camera devices to control hardware image
- processing based on the parameters supplied by upper layers, closing the
- control loop of the ISP.
-
- This component is device-specific and is loaded as an external plugin. It can
- be part of the libcamera code base, in which case it is covered by the same
- license, or provided externally as an open-source or closed-source component.
-
- The component is sandboxed and can only interact with libcamera through
- internal APIs specifically marked as such. In particular it will have no
- direct access to kernel camera devices, and all its accesses to image and
- metadata will be mediated by dmabuf instances explicitly passed to the
- component. The component must be prepared to run in a process separate from
- the main libcamera process, and to have a very restricted view of the system,
- including no access to networking APIs and limited access to file systems.
-
- The sandboxing mechanism isn't defined by libcamera. One example
- implementation will be provided as part of the project, and platforms vendors
- will be able to provide their own sandboxing mechanism as a plugin.
+ An image processing algorithm (IPA) component is a loadable plugin that
+ implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other
+ algorithms.
+
+ The algorithms run on the CPU and interact with the camera devices through the
+ Pipeline Handler to control hardware image processing based on the parameters
+ supplied by upper layers, maintaining state and closing the control loop
+ of the ISP.
+
+ The component is sandboxed and can only interact with libcamera through the
+ API provided by the Pipeline Handler and an IPA has no direct access to kernel
+ camera devices.
+
+ Open source IPA modules built with libcamera can be run in the same process
+ space as libcamera, however external IPA modules are run in a separate process
+ from the main libcamera process. IPA modules have a restricted view of the
+ system, including no access to networking APIs and limited access to file
+ systems.
+
+ IPA modules are only required for platforms and devices with an ISP controlled
+ by the host CPU. Camera sensors which have an integrated ISP are not
+ controlled through the IPA module.
libcamera should provide a basic implementation of Image Processing
Algorithms, to serve as a reference for the internal API. Device vendors are
@@ -27,8 +27,7 @@ desirable results from the camera.
In this developers guide, we will explore the `Camera Stack`_ and how it is
-can be visualised at a high level, and explore the internal `Architecture`_ of
-the libcamera library with its components. The current `Platform Support`_ is
+can be visualised at a high level. The current `Platform Support`_ is
detailed, as well as an overview of the `Licensing`_ requirements of the
project.
@@ -148,140 +147,6 @@ Native libcamera API
.. _GStreamer element: https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html
-Architecture
-------------
-
-While offering a unified API towards upper layers, and presenting itself as a
-single library, libcamera isn't monolithic. It exposes multiple components
-through its public API and is built around a set of separate helpers internally.
-Hardware abstractions are handled through the use of device-specific components
-where required and dynamically loadable plugins are used to separate image
-processing algorithms from the core libcamera codebase.
-
-::
-
- --------------------------< libcamera Public API >---------------------------
- ^ ^
- | |
- v v
- +-------------+ +---------------------------------------------------+
- | Camera | | Camera Device |
- | Manager | | +-----------------------------------------------+ |
- +-------------+ | | Device-Agnostic | |
- ^ | | | |
- | | | +--------------------------+ |
- | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
- | | | | { +-----------------+ } |
- | | | | } | //// Image //// | { |
- | | | | <-> | / Processing // | } |
- | | | | } | / Algorithms // | { |
- | | | | { +-----------------+ } |
- | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
- | | | | ========================== |
- | | | | +-----------------+ |
- | | | | | // Pipeline /// | |
- | | | | <-> | /// Handler /// | |
- | | | | | /////////////// | |
- | | +--------------------+ +-----------------+ |
- | | Device-Specific |
- | +---------------------------------------------------+
- | ^ ^
- | | |
- v v v
- +--------------------------------------------------------------------+
- | Helpers and Support Classes |
- | +-------------+ +-------------+ +-------------+ +-------------+ |
- | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
- | | Support | | Allocator | | IPC | | Manager | |
- | +-------------+ +-------------+ +-------------+ +-------------+ |
- | +-------------+ +-------------+ |
- | | Pipeline | | ... | |
- | | Runner | | | |
- | +-------------+ +-------------+ |
- +--------------------------------------------------------------------+
-
- /// Device-Specific Components
- ~~~ Sandboxing
-
-
-Camera Manager
- The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
- manage each Camera that libcamera supports. The Camera Manager supports
- hotplug detection and notification events when supported by the underlying
- kernel devices.
-
- There is only ever one instance of the Camera Manager running per application.
- Each application's instance of the Camera Manager ensures that only a single
- application can take control of a camera device at once.
-
- Read the `Camera Manager API`_ documentation for more details.
-
-.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
-
-Camera Device
- The Camera class represents a single item of camera hardware that is capable
- of producing one or more image streams, and provides the API to interact with
- the underlying device.
-
- If a system has multiple instances of the same hardware attached, each has its
- own instance of the camera class.
-
- The API exposes full control of the device to upper layers of libcamera through
- the public API, making it the highest level object libcamera exposes, and the
- object that all other API operations interact with from configuration to
- capture.
-
- Read the `Camera API`_ documentation for more details.
-
-.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
-
-Pipeline Handler
- The Pipeline Handler manages the complex pipelines exposed by the kernel
- drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
- handling to hide device-specific details from the rest of the library, and
- implements both pipeline configuration based on stream configuration, and
- pipeline runtime execution and scheduling when needed by the device.
-
- The Pipeline Handler lives in the same process as the rest of the library, and
- has access to all helpers and kernel camera-related devices.
-
- Hardware abstraction is handled by device specific Pipeline Handlers which are
- derived from the Pipeline Handler base class allowing commonality to be shared
- among the implementations.
-
- Derived pipeline handlers create Camera device instances based on the devices
- they detect and support on the running system, and are responsible for
- managing the interactions with a camera device.
-
- More details can be found in the `PipelineHandler API`_ documentation, and the
- `Pipeline Handler Writers Guide`_.
-
-.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
-
-Image Processing Algorithms
- An image processing algorithm (IPA) component is a loadable plugin that
- implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other
- algorithms.
-
- The algorithms run on the CPU and interact with the camera devices through the
- Pipeline Handler to control hardware image processing based on the parameters
- supplied by upper layers, maintaining state and closing the control loop
- of the ISP.
-
- The component is sandboxed and can only interact with libcamera through the
- API provided by the Pipeline Handler and an IPA has no direct access to kernel
- camera devices.
-
- Open source IPA modules built with libcamera can be run in the same process
- space as libcamera, however external IPA modules are run in a separate process
- from the main libcamera process. IPA modules have a restricted view of the
- system, including no access to networking APIs and limited access to file
- systems.
-
- IPA modules are only required for platforms and devices with an ISP controlled
- by the host CPU. Camera sensors which have an integrated ISP are not
- controlled through the IPA module.
-
Platform Support
----------------