| Message ID | 20240820130740.568243-6-dan.scally@ideasonboard.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series |
|
| Related | show |
Hi Dan, Thank you for the patch. On Tue, Aug 20, 2024 at 02:07:36PM +0100, Daniel Scally wrote: > The libcamera Architecture section of the introduction is largely a > duplicate of the section broken out from docs.rst. Remove it from the > introduction.rst file and consolidate anything that wasn't duplicated > into libcamera_architecture.rst and feature_requirements.rst. Take the > opportunity to also expand the list of Platform Support which is now a > bit out of date. > > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> > --- > Changes since v3: > > - Merged the contents rather than replaced one with the other, following > Laurent's suggestion. Also merged one section into the page detailing > feature requirements > > Changes since v2: > > - Reworked to change libcamera architecture section instead of camera > stack. DDropped R-b > - Updated the list of supported platforms on the new libcamera > architecture page > > Changes since v1: > > - None > > Documentation/feature_requirements.rst | 13 +- > Documentation/guides/introduction.rst | 156 +-------------- > Documentation/libcamera_architecture.rst | 244 +++++++++++++---------- > 3 files changed, 148 insertions(+), 265 deletions(-) > > diff --git a/Documentation/feature_requirements.rst b/Documentation/feature_requirements.rst > index cae7e9ab..e6b74a62 100644 > --- a/Documentation/feature_requirements.rst > +++ b/Documentation/feature_requirements.rst > @@ -90,10 +90,15 @@ on top of the minimum required set for the profile they expose. > 3A and Image Enhancement Algorithms > ----------------------------------- > > -The camera devices shall implement auto exposure, auto gain and auto white > -balance. Camera devices that include a focus lens shall implement auto > -focus. Additional image enhancement algorithms, such as noise reduction or > -video stabilization, may be implemented. > +The library shall provide a basic implementation of Image Processing Algorithms > +to serve as a reference for the internal API. This shall including auto exposure > +and gain and auto white balance. Camera devices that include a focus lens shall > +implement auto focus. Additional image enhancement algorithms, such as noise > +reduction or video stabilization, may be implemented. Device vendors are > +expected to provide a fully-fledged implementation compatible with their > +Pipeline Handler. One goal of the libcamera project is to create an environment > +in which the community will be able to compete with the closed-source vendor > +biaries and develop a high quality open source implementation. > > All algorithms may be implemented in hardware or firmware outside of the > library, or in software in the library. They shall all be controllable by > diff --git a/Documentation/guides/introduction.rst b/Documentation/guides/introduction.rst > index d1e67654..12d1b7d4 100644 > --- a/Documentation/guides/introduction.rst > +++ b/Documentation/guides/introduction.rst > @@ -26,10 +26,8 @@ desirable results from the camera. > .. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html > > > -In this developers guide, we will explore the internal `Architecture`_ of > -the libcamera library with its components. The current `Platform Support`_ is > -detailed, as well as an overview of the `Licensing`_ requirements of the > -project. > +In this developers guide the `Licensing`_ requirements of the project are > +detailed. > > This introduction is followed by a walkthrough tutorial to newcomers wishing to > support a new platform with the `Pipeline Handler Writers Guide`_ and for those > @@ -41,156 +39,6 @@ provides a tutorial of the key APIs exposed by libcamera. > > .. TODO: Correctly link to the other articles of the guide > > -Architecture > ------------- > - > -While offering a unified API towards upper layers, and presenting itself as a > -single library, libcamera isn't monolithic. It exposes multiple components > -through its public API and is built around a set of separate helpers internally. > -Hardware abstractions are handled through the use of device-specific components > -where required and dynamically loadable plugins are used to separate image > -processing algorithms from the core libcamera codebase. > - > -:: > - > - --------------------------< libcamera Public API >--------------------------- > - ^ ^ > - | | > - v v > - +-------------+ +---------------------------------------------------+ > - | Camera | | Camera Device | > - | Manager | | +-----------------------------------------------+ | > - +-------------+ | | Device-Agnostic | | > - ^ | | | | > - | | | +--------------------------+ | > - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | > - | | | | { +-----------------+ } | > - | | | | } | //// Image //// | { | > - | | | | <-> | / Processing // | } | > - | | | | } | / Algorithms // | { | > - | | | | { +-----------------+ } | > - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | > - | | | | ========================== | > - | | | | +-----------------+ | > - | | | | | // Pipeline /// | | > - | | | | <-> | /// Handler /// | | > - | | | | | /////////////// | | > - | | +--------------------+ +-----------------+ | > - | | Device-Specific | > - | +---------------------------------------------------+ > - | ^ ^ > - | | | > - v v v > - +--------------------------------------------------------------------+ > - | Helpers and Support Classes | > - | +-------------+ +-------------+ +-------------+ +-------------+ | > - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | > - | | Support | | Allocator | | IPC | | Manager | | > - | +-------------+ +-------------+ +-------------+ +-------------+ | > - | +-------------+ +-------------+ | > - | | Pipeline | | ... | | > - | | Runner | | | | > - | +-------------+ +-------------+ | > - +--------------------------------------------------------------------+ > - > - /// Device-Specific Components > - ~~~ Sandboxing > - > - > -Camera Manager > - The Camera Manager enumerates cameras and instantiates Pipeline Handlers to > - manage each Camera that libcamera supports. The Camera Manager supports > - hotplug detection and notification events when supported by the underlying > - kernel devices. > - > - There is only ever one instance of the Camera Manager running per application. > - Each application's instance of the Camera Manager ensures that only a single > - application can take control of a camera device at once. > - > - Read the `Camera Manager API`_ documentation for more details. > - > -.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html > - > -Camera Device > - The Camera class represents a single item of camera hardware that is capable > - of producing one or more image streams, and provides the API to interact with > - the underlying device. > - > - If a system has multiple instances of the same hardware attached, each has its > - own instance of the camera class. > - > - The API exposes full control of the device to upper layers of libcamera through > - the public API, making it the highest level object libcamera exposes, and the > - object that all other API operations interact with from configuration to > - capture. > - > - Read the `Camera API`_ documentation for more details. > - > -.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html > - > -Pipeline Handler > - The Pipeline Handler manages the complex pipelines exposed by the kernel > - drivers through the Media Controller and V4L2 APIs. It abstracts pipeline > - handling to hide device-specific details from the rest of the library, and > - implements both pipeline configuration based on stream configuration, and > - pipeline runtime execution and scheduling when needed by the device. > - > - The Pipeline Handler lives in the same process as the rest of the library, and > - has access to all helpers and kernel camera-related devices. > - > - Hardware abstraction is handled by device specific Pipeline Handlers which are > - derived from the Pipeline Handler base class allowing commonality to be shared > - among the implementations. > - > - Derived pipeline handlers create Camera device instances based on the devices > - they detect and support on the running system, and are responsible for > - managing the interactions with a camera device. > - > - More details can be found in the `PipelineHandler API`_ documentation, and the > - `Pipeline Handler Writers Guide`_. > - > -.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html > - > -Image Processing Algorithms > - An image processing algorithm (IPA) component is a loadable plugin that > - implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other > - algorithms. > - > - The algorithms run on the CPU and interact with the camera devices through the > - Pipeline Handler to control hardware image processing based on the parameters > - supplied by upper layers, maintaining state and closing the control loop > - of the ISP. > - > - The component is sandboxed and can only interact with libcamera through the > - API provided by the Pipeline Handler and an IPA has no direct access to kernel > - camera devices. > - > - Open source IPA modules built with libcamera can be run in the same process > - space as libcamera, however external IPA modules are run in a separate process > - from the main libcamera process. IPA modules have a restricted view of the > - system, including no access to networking APIs and limited access to file > - systems. > - > - IPA modules are only required for platforms and devices with an ISP controlled > - by the host CPU. Camera sensors which have an integrated ISP are not > - controlled through the IPA module. > - > -Platform Support > ----------------- > - > -The library currently supports the following hardware platforms specifically > -with dedicated pipeline handlers: > - > - - Intel IPU3 (ipu3) > - - Rockchip RK3399 (rkisp1) > - - RaspberryPi 3 and 4 (rpi/vc4) > - > -Furthermore, generic platform support is provided for the following: > - > - - USB video device class cameras (uvcvideo) > - - iMX7, Allwinner Sun6i (simple) > - - Virtual media controller driver for test use cases (vimc) > - > Licensing > --------- > > diff --git a/Documentation/libcamera_architecture.rst b/Documentation/libcamera_architecture.rst > index 1258db23..abbb0d17 100644 > --- a/Documentation/libcamera_architecture.rst > +++ b/Documentation/libcamera_architecture.rst > @@ -5,124 +5,136 @@ > libcamera Architecture > ====================== > > +While offering a unified API towards upper layers, and presenting itself as a > +single library, libcamera isn't monolithic. It exposes multiple components > +through its public API and is built around a set of separate helpers internally. > +Hardware abstractions are handled through the use of device-specific components > +where required and dynamically loadable plugins are used to separate image > +processing algorithms from the core libcamera codebase. > + > :: > > - ---------------------------< libcamera Public API >--------------------------- > - ^ ^ > - | | > - v v > - +-------------+ +-------------------------------------------------+ > - | Camera | | Camera Device | > - | Devices | | +---------------------------------------------+ | > - | Manager | | | Device-Agnostic | | > - +-------------+ | | | | > - ^ | | +------------------------+ | > - | | | | ~~~~~~~~~~~~~~~~~~~~~ | > - | | | | { +---------------+ } | > - | | | | } | ////Image//// | { | > - | | | | <-> | /Processing// | } | > - | | | | } | /Algorithms// | { | > - | | | | { +---------------+ } | > - | | | | ~~~~~~~~~~~~~~~~~~~~~ | > - | | | | ======================== | > - | | | | +---------------+ | > - | | | | | //Pipeline/// | | > - | | | | <-> | ///Handler/// | | > - | | | | | ///////////// | | > - | | +--------------------+ +---------------+ | > - | | Device-Specific | > - | +-------------------------------------------------+ > - | ^ ^ > - | | | > - v v v > - +--------------------------------------------------------------------+ > - | Helpers and Support Classes | > - | +-------------+ +-------------+ +-------------+ +-------------+ | > - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | > - | | Support | | Allocator | | IPC | | Manager | | > - | +-------------+ +-------------+ +-------------+ +-------------+ | > - | +-------------+ +-------------+ | > - | | Pipeline | | ... | | > - | | Runner | | | | > - | +-------------+ +-------------+ | > - +--------------------------------------------------------------------+ > - > - /// Device-Specific Components > - ~~~ Sandboxing > - > -While offering a unified API towards upper layers, and presenting > -itself as a single library, libcamera isn't monolithic. It exposes > -multiple components through its public API, is built around a set of > -separate helpers internally, uses device-specific components and can > -load dynamic plugins. > - > -Camera Devices Manager > - The Camera Devices Manager provides a view of available cameras > - in the system. It performs cold enumeration and runtime camera > - management, and supports a hotplug notification mechanism in its > - public API. > - > - To avoid the cost associated with cold enumeration of all devices > - at application start, and to arbitrate concurrent access to camera > - devices, the Camera Devices Manager could later be split to a > - separate service, possibly with integration in platform-specific > - device management. > + --------------------------< libcamera Public API >--------------------------- > + ^ ^ > + | | > + v v > + +-------------+ +---------------------------------------------------+ > + | Camera | | Camera Device | > + | Manager | | +-----------------------------------------------+ | > + +-------------+ | | Device-Agnostic | | > + ^ | | | | > + | | | +--------------------------+ | > + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | > + | | | | { +-----------------+ } | > + | | | | } | //// Image //// | { | > + | | | | <-> | / Processing // | } | > + | | | | } | / Algorithms // | { | > + | | | | { +-----------------+ } | > + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | > + | | | | ========================== | > + | | | | +-----------------+ | > + | | | | | // Pipeline /// | | > + | | | | <-> | /// Handler /// | | > + | | | | | /////////////// | | > + | | +--------------------+ +-----------------+ | > + | | Device-Specific | > + | +---------------------------------------------------+ > + | ^ ^ > + | | | > + v v v > + +--------------------------------------------------------------------+ > + | Helpers and Support Classes | > + | +-------------+ +-------------+ +-------------+ +-------------+ | > + | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | > + | | Support | | Allocator | | IPC | | Manager | | > + | +-------------+ +-------------+ +-------------+ +-------------+ | > + | +-------------+ +-------------+ | > + | | Pipeline | | ... | | > + | | Runner | | | | > + | +-------------+ +-------------+ | > + +--------------------------------------------------------------------+ > + > + /// Device-Specific Components > + ~~~ Sandboxing > + > + > +Camera Manager > + The Camera Manager enumerates cameras and instantiates Pipeline Handlers to > + manage each Camera that libcamera supports. The Camera Manager supports > + hotplug detection and notification events when supported by the underlying > + kernel devices. > + > + There is only ever one instance of the Camera Manager running per application. > + Each application's instance of the Camera Manager ensures that only a single > + application can take control of a camera device at once. > + > + Read the `Camera Manager API`_ documentation for more details. > + > +.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html > > Camera Device > - The Camera Device represents a camera device to upper layers. It > - exposes full control of the device through the public API, and is > - thus the highest level object exposed by libcamera. > + The Camera class represents a single item of camera hardware that is capable > + of producing one or more image streams, and provides the API to interact with > + the underlying device. > + > + If a system has multiple instances of the same hardware attached, each has its > + own instance of the camera class. > + > + The API exposes full control of the device to upper layers of libcamera through > + the public API, making it the highest level object libcamera exposes, and the > + object that all other API operations interact with from configuration to > + capture. > + > + Read the `Camera API`_ documentation for more details. > > - Camera Device instances are created by the Camera Devices > - Manager. An optional function to create new instances could be exposed > - through the public API to speed up initialization when the upper > - layer knows how to directly address camera devices present in the > - system. > +.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html > > Pipeline Handler > - The Pipeline Handler manages complex pipelines exposed by the kernel drivers > - through the Media Controller and V4L2 APIs. It abstracts pipeline handling to > - hide device-specific details to the rest of the library, and implements both > - pipeline configuration based on stream configuration, and pipeline runtime > - execution and scheduling when needed by the device. > - > - This component is device-specific and is part of the libcamera code base. As > - such it is covered by the same free software license as the rest of libcamera > - and needs to be contributed upstream by device vendors. The Pipeline Handler > - lives in the same process as the rest of the library, and has access to all > - helpers and kernel camera-related devices. > + The Pipeline Handler manages the complex pipelines exposed by the kernel > + drivers through the Media Controller and V4L2 APIs. It abstracts pipeline > + handling to hide device-specific details from the rest of the library, and > + implements both pipeline configuration based on stream configuration, and > + pipeline runtime execution and scheduling when needed by the device. > + > + The Pipeline Handler lives in the same process as the rest of the library, and > + has access to all helpers and kernel camera-related devices. > + > + Hardware abstraction is handled by device specific Pipeline Handlers which are > + derived from the Pipeline Handler base class allowing commonality to be shared > + among the implementations. > + > + Derived pipeline handlers create Camera device instances based on the devices > + they detect and support on the running system, and are responsible for > + managing the interactions with a camera device. > + > + More details can be found in the `PipelineHandler API`_ documentation, and the > + :doc:`Pipeline Handler Writers Guide <guides/pipeline-handler>`. > + > +.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html > > Image Processing Algorithms > Together with the hardware image processing and hardware statistics > - collection, the Image Processing Algorithms implement 3A (Auto-Exposure, > + collection, the Image Processing Algorithms (IPA) implement 3A (Auto-Exposure, > Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU > - and interact with the kernel camera devices to control hardware image > - processing based on the parameters supplied by upper layers, closing the > - control loop of the ISP. > - > - This component is device-specific and is loaded as an external plugin. It can > - be part of the libcamera code base, in which case it is covered by the same > - license, or provided externally as an open-source or closed-source component. > - > - The component is sandboxed and can only interact with libcamera through > - internal APIs specifically marked as such. In particular it will have no > - direct access to kernel camera devices, and all its accesses to image and > - metadata will be mediated by dmabuf instances explicitly passed to the > - component. The component must be prepared to run in a process separate from > - the main libcamera process, and to have a very restricted view of the system, > - including no access to networking APIs and limited access to file systems. > - > - The sandboxing mechanism isn't defined by libcamera. One example > - implementation will be provided as part of the project, and platforms vendors > - will be able to provide their own sandboxing mechanism as a plugin. > - > - libcamera should provide a basic implementation of Image Processing > - Algorithms, to serve as a reference for the internal API. Device vendors are > - expected to provide a full-fledged implementation compatible with their > - Pipeline Handler. One goal of the libcamera project is to create an > - environment in which the community will be able to compete with the > - closed-source vendor binaries and develop a high quality open source > - implementation. > + and control hardware image processing based on the parameters supplied by > + upper layers, closing the control loop of the ISP. > + > + IPAs are loaded as external plugins named IPA Modules. IPA Modules can be part > + of the libcamera code base or provided externally by camera vendors as > + open-source or closed-source components. > + > + Open source IPA Modules built with libcamera are run in the same process space > + as libcamera. External IPA Modules are run in a separate sandboxed process. In > + either case, they can only interact with libcamera through the API provided by > + the Pipeline Handler. They have a restricted view of the system, with no direct > + access to kernel camera devices, no access to networking APIs, and limited > + access to file systems. All their accesses to image and metadata are mediated > + by dmabuf instances explicitly passed by the Pipeline Handler to the IPA > + Module. > + > + IPA Modules are only required for platforms and devices with an ISP controlled > + by the host CPU. Camera sensors which have an integrated ISP are not > + controlled through the IPA Module. > > Helpers and Support Classes > While Pipeline Handlers are device-specific, implementations are expected to > @@ -136,3 +148,21 @@ Helpers and Support Classes > self-contained support classes, even if such code is present only once in the > code base, in order to keep the source code clean and easy to read. This > should be the case for instance for plugin management. > + > +Platform Support > +---------------- > + > +The library currently supports the following hardware platforms specifically > +with dedicated pipeline handlers: > + > + - Arm Mali-C55 > + - Intel IPU3 (ipu3) > + - NXP i.MX8MP (imx8-isi and rkisp1) > + - RaspberryPi 3, 4 and zero (rpi/vc4) > + - Rockchip RK3399 (rkisp1) > + > +Furthermore, generic platform support is provided for the following: > + > + - USB video device class cameras (uvcvideo) > + - iMX7, IPU6, Allwinner Sun6i (simple) > + - Virtual media controller driver for test use cases (vimc)
diff --git a/Documentation/feature_requirements.rst b/Documentation/feature_requirements.rst index cae7e9ab..e6b74a62 100644 --- a/Documentation/feature_requirements.rst +++ b/Documentation/feature_requirements.rst @@ -90,10 +90,15 @@ on top of the minimum required set for the profile they expose. 3A and Image Enhancement Algorithms ----------------------------------- -The camera devices shall implement auto exposure, auto gain and auto white -balance. Camera devices that include a focus lens shall implement auto -focus. Additional image enhancement algorithms, such as noise reduction or -video stabilization, may be implemented. +The library shall provide a basic implementation of Image Processing Algorithms +to serve as a reference for the internal API. This shall including auto exposure +and gain and auto white balance. Camera devices that include a focus lens shall +implement auto focus. Additional image enhancement algorithms, such as noise +reduction or video stabilization, may be implemented. Device vendors are +expected to provide a fully-fledged implementation compatible with their +Pipeline Handler. One goal of the libcamera project is to create an environment +in which the community will be able to compete with the closed-source vendor +biaries and develop a high quality open source implementation. All algorithms may be implemented in hardware or firmware outside of the library, or in software in the library. They shall all be controllable by diff --git a/Documentation/guides/introduction.rst b/Documentation/guides/introduction.rst index d1e67654..12d1b7d4 100644 --- a/Documentation/guides/introduction.rst +++ b/Documentation/guides/introduction.rst @@ -26,10 +26,8 @@ desirable results from the camera. .. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html -In this developers guide, we will explore the internal `Architecture`_ of -the libcamera library with its components. The current `Platform Support`_ is -detailed, as well as an overview of the `Licensing`_ requirements of the -project. +In this developers guide the `Licensing`_ requirements of the project are +detailed. This introduction is followed by a walkthrough tutorial to newcomers wishing to support a new platform with the `Pipeline Handler Writers Guide`_ and for those @@ -41,156 +39,6 @@ provides a tutorial of the key APIs exposed by libcamera. .. TODO: Correctly link to the other articles of the guide -Architecture ------------- - -While offering a unified API towards upper layers, and presenting itself as a -single library, libcamera isn't monolithic. It exposes multiple components -through its public API and is built around a set of separate helpers internally. -Hardware abstractions are handled through the use of device-specific components -where required and dynamically loadable plugins are used to separate image -processing algorithms from the core libcamera codebase. - -:: - - --------------------------< libcamera Public API >--------------------------- - ^ ^ - | | - v v - +-------------+ +---------------------------------------------------+ - | Camera | | Camera Device | - | Manager | | +-----------------------------------------------+ | - +-------------+ | | Device-Agnostic | | - ^ | | | | - | | | +--------------------------+ | - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | - | | | | { +-----------------+ } | - | | | | } | //// Image //// | { | - | | | | <-> | / Processing // | } | - | | | | } | / Algorithms // | { | - | | | | { +-----------------+ } | - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | - | | | | ========================== | - | | | | +-----------------+ | - | | | | | // Pipeline /// | | - | | | | <-> | /// Handler /// | | - | | | | | /////////////// | | - | | +--------------------+ +-----------------+ | - | | Device-Specific | - | +---------------------------------------------------+ - | ^ ^ - | | | - v v v - +--------------------------------------------------------------------+ - | Helpers and Support Classes | - | +-------------+ +-------------+ +-------------+ +-------------+ | - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | - | | Support | | Allocator | | IPC | | Manager | | - | +-------------+ +-------------+ +-------------+ +-------------+ | - | +-------------+ +-------------+ | - | | Pipeline | | ... | | - | | Runner | | | | - | +-------------+ +-------------+ | - +--------------------------------------------------------------------+ - - /// Device-Specific Components - ~~~ Sandboxing - - -Camera Manager - The Camera Manager enumerates cameras and instantiates Pipeline Handlers to - manage each Camera that libcamera supports. The Camera Manager supports - hotplug detection and notification events when supported by the underlying - kernel devices. - - There is only ever one instance of the Camera Manager running per application. - Each application's instance of the Camera Manager ensures that only a single - application can take control of a camera device at once. - - Read the `Camera Manager API`_ documentation for more details. - -.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html - -Camera Device - The Camera class represents a single item of camera hardware that is capable - of producing one or more image streams, and provides the API to interact with - the underlying device. - - If a system has multiple instances of the same hardware attached, each has its - own instance of the camera class. - - The API exposes full control of the device to upper layers of libcamera through - the public API, making it the highest level object libcamera exposes, and the - object that all other API operations interact with from configuration to - capture. - - Read the `Camera API`_ documentation for more details. - -.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html - -Pipeline Handler - The Pipeline Handler manages the complex pipelines exposed by the kernel - drivers through the Media Controller and V4L2 APIs. It abstracts pipeline - handling to hide device-specific details from the rest of the library, and - implements both pipeline configuration based on stream configuration, and - pipeline runtime execution and scheduling when needed by the device. - - The Pipeline Handler lives in the same process as the rest of the library, and - has access to all helpers and kernel camera-related devices. - - Hardware abstraction is handled by device specific Pipeline Handlers which are - derived from the Pipeline Handler base class allowing commonality to be shared - among the implementations. - - Derived pipeline handlers create Camera device instances based on the devices - they detect and support on the running system, and are responsible for - managing the interactions with a camera device. - - More details can be found in the `PipelineHandler API`_ documentation, and the - `Pipeline Handler Writers Guide`_. - -.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html - -Image Processing Algorithms - An image processing algorithm (IPA) component is a loadable plugin that - implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other - algorithms. - - The algorithms run on the CPU and interact with the camera devices through the - Pipeline Handler to control hardware image processing based on the parameters - supplied by upper layers, maintaining state and closing the control loop - of the ISP. - - The component is sandboxed and can only interact with libcamera through the - API provided by the Pipeline Handler and an IPA has no direct access to kernel - camera devices. - - Open source IPA modules built with libcamera can be run in the same process - space as libcamera, however external IPA modules are run in a separate process - from the main libcamera process. IPA modules have a restricted view of the - system, including no access to networking APIs and limited access to file - systems. - - IPA modules are only required for platforms and devices with an ISP controlled - by the host CPU. Camera sensors which have an integrated ISP are not - controlled through the IPA module. - -Platform Support ----------------- - -The library currently supports the following hardware platforms specifically -with dedicated pipeline handlers: - - - Intel IPU3 (ipu3) - - Rockchip RK3399 (rkisp1) - - RaspberryPi 3 and 4 (rpi/vc4) - -Furthermore, generic platform support is provided for the following: - - - USB video device class cameras (uvcvideo) - - iMX7, Allwinner Sun6i (simple) - - Virtual media controller driver for test use cases (vimc) - Licensing --------- diff --git a/Documentation/libcamera_architecture.rst b/Documentation/libcamera_architecture.rst index 1258db23..abbb0d17 100644 --- a/Documentation/libcamera_architecture.rst +++ b/Documentation/libcamera_architecture.rst @@ -5,124 +5,136 @@ libcamera Architecture ====================== +While offering a unified API towards upper layers, and presenting itself as a +single library, libcamera isn't monolithic. It exposes multiple components +through its public API and is built around a set of separate helpers internally. +Hardware abstractions are handled through the use of device-specific components +where required and dynamically loadable plugins are used to separate image +processing algorithms from the core libcamera codebase. + :: - ---------------------------< libcamera Public API >--------------------------- - ^ ^ - | | - v v - +-------------+ +-------------------------------------------------+ - | Camera | | Camera Device | - | Devices | | +---------------------------------------------+ | - | Manager | | | Device-Agnostic | | - +-------------+ | | | | - ^ | | +------------------------+ | - | | | | ~~~~~~~~~~~~~~~~~~~~~ | - | | | | { +---------------+ } | - | | | | } | ////Image//// | { | - | | | | <-> | /Processing// | } | - | | | | } | /Algorithms// | { | - | | | | { +---------------+ } | - | | | | ~~~~~~~~~~~~~~~~~~~~~ | - | | | | ======================== | - | | | | +---------------+ | - | | | | | //Pipeline/// | | - | | | | <-> | ///Handler/// | | - | | | | | ///////////// | | - | | +--------------------+ +---------------+ | - | | Device-Specific | - | +-------------------------------------------------+ - | ^ ^ - | | | - v v v - +--------------------------------------------------------------------+ - | Helpers and Support Classes | - | +-------------+ +-------------+ +-------------+ +-------------+ | - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | - | | Support | | Allocator | | IPC | | Manager | | - | +-------------+ +-------------+ +-------------+ +-------------+ | - | +-------------+ +-------------+ | - | | Pipeline | | ... | | - | | Runner | | | | - | +-------------+ +-------------+ | - +--------------------------------------------------------------------+ - - /// Device-Specific Components - ~~~ Sandboxing - -While offering a unified API towards upper layers, and presenting -itself as a single library, libcamera isn't monolithic. It exposes -multiple components through its public API, is built around a set of -separate helpers internally, uses device-specific components and can -load dynamic plugins. - -Camera Devices Manager - The Camera Devices Manager provides a view of available cameras - in the system. It performs cold enumeration and runtime camera - management, and supports a hotplug notification mechanism in its - public API. - - To avoid the cost associated with cold enumeration of all devices - at application start, and to arbitrate concurrent access to camera - devices, the Camera Devices Manager could later be split to a - separate service, possibly with integration in platform-specific - device management. + --------------------------< libcamera Public API >--------------------------- + ^ ^ + | | + v v + +-------------+ +---------------------------------------------------+ + | Camera | | Camera Device | + | Manager | | +-----------------------------------------------+ | + +-------------+ | | Device-Agnostic | | + ^ | | | | + | | | +--------------------------+ | + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | + | | | | { +-----------------+ } | + | | | | } | //// Image //// | { | + | | | | <-> | / Processing // | } | + | | | | } | / Algorithms // | { | + | | | | { +-----------------+ } | + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ | + | | | | ========================== | + | | | | +-----------------+ | + | | | | | // Pipeline /// | | + | | | | <-> | /// Handler /// | | + | | | | | /////////////// | | + | | +--------------------+ +-----------------+ | + | | Device-Specific | + | +---------------------------------------------------+ + | ^ ^ + | | | + v v v + +--------------------------------------------------------------------+ + | Helpers and Support Classes | + | +-------------+ +-------------+ +-------------+ +-------------+ | + | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | + | | Support | | Allocator | | IPC | | Manager | | + | +-------------+ +-------------+ +-------------+ +-------------+ | + | +-------------+ +-------------+ | + | | Pipeline | | ... | | + | | Runner | | | | + | +-------------+ +-------------+ | + +--------------------------------------------------------------------+ + + /// Device-Specific Components + ~~~ Sandboxing + + +Camera Manager + The Camera Manager enumerates cameras and instantiates Pipeline Handlers to + manage each Camera that libcamera supports. The Camera Manager supports + hotplug detection and notification events when supported by the underlying + kernel devices. + + There is only ever one instance of the Camera Manager running per application. + Each application's instance of the Camera Manager ensures that only a single + application can take control of a camera device at once. + + Read the `Camera Manager API`_ documentation for more details. + +.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html Camera Device - The Camera Device represents a camera device to upper layers. It - exposes full control of the device through the public API, and is - thus the highest level object exposed by libcamera. + The Camera class represents a single item of camera hardware that is capable + of producing one or more image streams, and provides the API to interact with + the underlying device. + + If a system has multiple instances of the same hardware attached, each has its + own instance of the camera class. + + The API exposes full control of the device to upper layers of libcamera through + the public API, making it the highest level object libcamera exposes, and the + object that all other API operations interact with from configuration to + capture. + + Read the `Camera API`_ documentation for more details. - Camera Device instances are created by the Camera Devices - Manager. An optional function to create new instances could be exposed - through the public API to speed up initialization when the upper - layer knows how to directly address camera devices present in the - system. +.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html Pipeline Handler - The Pipeline Handler manages complex pipelines exposed by the kernel drivers - through the Media Controller and V4L2 APIs. It abstracts pipeline handling to - hide device-specific details to the rest of the library, and implements both - pipeline configuration based on stream configuration, and pipeline runtime - execution and scheduling when needed by the device. - - This component is device-specific and is part of the libcamera code base. As - such it is covered by the same free software license as the rest of libcamera - and needs to be contributed upstream by device vendors. The Pipeline Handler - lives in the same process as the rest of the library, and has access to all - helpers and kernel camera-related devices. + The Pipeline Handler manages the complex pipelines exposed by the kernel + drivers through the Media Controller and V4L2 APIs. It abstracts pipeline + handling to hide device-specific details from the rest of the library, and + implements both pipeline configuration based on stream configuration, and + pipeline runtime execution and scheduling when needed by the device. + + The Pipeline Handler lives in the same process as the rest of the library, and + has access to all helpers and kernel camera-related devices. + + Hardware abstraction is handled by device specific Pipeline Handlers which are + derived from the Pipeline Handler base class allowing commonality to be shared + among the implementations. + + Derived pipeline handlers create Camera device instances based on the devices + they detect and support on the running system, and are responsible for + managing the interactions with a camera device. + + More details can be found in the `PipelineHandler API`_ documentation, and the + :doc:`Pipeline Handler Writers Guide <guides/pipeline-handler>`. + +.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html Image Processing Algorithms Together with the hardware image processing and hardware statistics - collection, the Image Processing Algorithms implement 3A (Auto-Exposure, + collection, the Image Processing Algorithms (IPA) implement 3A (Auto-Exposure, Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU - and interact with the kernel camera devices to control hardware image - processing based on the parameters supplied by upper layers, closing the - control loop of the ISP. - - This component is device-specific and is loaded as an external plugin. It can - be part of the libcamera code base, in which case it is covered by the same - license, or provided externally as an open-source or closed-source component. - - The component is sandboxed and can only interact with libcamera through - internal APIs specifically marked as such. In particular it will have no - direct access to kernel camera devices, and all its accesses to image and - metadata will be mediated by dmabuf instances explicitly passed to the - component. The component must be prepared to run in a process separate from - the main libcamera process, and to have a very restricted view of the system, - including no access to networking APIs and limited access to file systems. - - The sandboxing mechanism isn't defined by libcamera. One example - implementation will be provided as part of the project, and platforms vendors - will be able to provide their own sandboxing mechanism as a plugin. - - libcamera should provide a basic implementation of Image Processing - Algorithms, to serve as a reference for the internal API. Device vendors are - expected to provide a full-fledged implementation compatible with their - Pipeline Handler. One goal of the libcamera project is to create an - environment in which the community will be able to compete with the - closed-source vendor binaries and develop a high quality open source - implementation. + and control hardware image processing based on the parameters supplied by + upper layers, closing the control loop of the ISP. + + IPAs are loaded as external plugins named IPA Modules. IPA Modules can be part + of the libcamera code base or provided externally by camera vendors as + open-source or closed-source components. + + Open source IPA Modules built with libcamera are run in the same process space + as libcamera. External IPA Modules are run in a separate sandboxed process. In + either case, they can only interact with libcamera through the API provided by + the Pipeline Handler. They have a restricted view of the system, with no direct + access to kernel camera devices, no access to networking APIs, and limited + access to file systems. All their accesses to image and metadata are mediated + by dmabuf instances explicitly passed by the Pipeline Handler to the IPA + Module. + + IPA Modules are only required for platforms and devices with an ISP controlled + by the host CPU. Camera sensors which have an integrated ISP are not + controlled through the IPA Module. Helpers and Support Classes While Pipeline Handlers are device-specific, implementations are expected to @@ -136,3 +148,21 @@ Helpers and Support Classes self-contained support classes, even if such code is present only once in the code base, in order to keep the source code clean and easy to read. This should be the case for instance for plugin management. + +Platform Support +---------------- + +The library currently supports the following hardware platforms specifically +with dedicated pipeline handlers: + + - Arm Mali-C55 + - Intel IPU3 (ipu3) + - NXP i.MX8MP (imx8-isi and rkisp1) + - RaspberryPi 3, 4 and zero (rpi/vc4) + - Rockchip RK3399 (rkisp1) + +Furthermore, generic platform support is provided for the following: + + - USB video device class cameras (uvcvideo) + - iMX7, IPU6, Allwinner Sun6i (simple) + - Virtual media controller driver for test use cases (vimc)
The libcamera Architecture section of the introduction is largely a duplicate of the section broken out from docs.rst. Remove it from the introduction.rst file and consolidate anything that wasn't duplicated into libcamera_architecture.rst and feature_requirements.rst. Take the opportunity to also expand the list of Platform Support which is now a bit out of date. Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> --- Changes since v3: - Merged the contents rather than replaced one with the other, following Laurent's suggestion. Also merged one section into the page detailing feature requirements Changes since v2: - Reworked to change libcamera architecture section instead of camera stack. DDropped R-b - Updated the list of supported platforms on the new libcamera architecture page Changes since v1: - None Documentation/feature_requirements.rst | 13 +- Documentation/guides/introduction.rst | 156 +-------------- Documentation/libcamera_architecture.rst | 244 +++++++++++++---------- 3 files changed, 148 insertions(+), 265 deletions(-)