From patchwork Fri Jun 26 11:17:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Chris Chinchilla X-Patchwork-Id: 8441 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7B212C2E66 for ; Fri, 26 Jun 2020 11:17:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EC83D609C8; Fri, 26 Jun 2020 13:17:46 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gregariousmammal.com header.i=@gregariousmammal.com header.b="Ub3ihk5p"; dkim-atps=neutral Received: from mail-ej1-x644.google.com (mail-ej1-x644.google.com [IPv6:2a00:1450:4864:20::644]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A7FA5603BB for ; Fri, 26 Jun 2020 13:17:45 +0200 (CEST) Received: by mail-ej1-x644.google.com with SMTP id mb16so8998945ejb.4 for ; Fri, 26 Jun 2020 04:17:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gregariousmammal.com; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=irNRd0B2mrVcBqwbLL8Cgb6opUpKBL4iro9phtcZ+fE=; b=Ub3ihk5pjcozYyC1It3wI+cyzMl6mz1VDLykYntuQV2Z3sOBlknpxb3OYVRKXuk48w TFXB2w+nmN2KKazlc4J5z1PYhkL7tQWOPlQsNNs1TQti+PUFmF+90i5PGMmm85SEvBGt Gq1XwlQXptm43UbpeTL/hDP6pD0Sm97UAACqyLnSMsjkwBjq3GPqD5S8tFt1VIYVyXjJ rfrFSDXh0KqIuEgCg3KhEgx/tIJwcLu2jf/JiBtz4XvL058VeSszTIX+sCNalBjV9bTj 3vIHxths09uJv/x3h/LYZznBEfRVGnnVde2NtkImZB+sO60V1DuBSDi0okm8d2DW/rJH wFEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=irNRd0B2mrVcBqwbLL8Cgb6opUpKBL4iro9phtcZ+fE=; b=aHCYR7BW8O9z97F/kifV3Qgf5bHfLwhGEp3a1HP/m3UZttREBd/PQwCwW0GoPKMGeG VbWSFKd5SXXQArZSG4cdgdL/GxSnXMiE0sFlgJhmQ7olfKmBicX7HMwar9NCqFI2bWKr F0nzLifw0Kepa0r8UCP2XX4n6pcFGpji082JWLszNV9WIZ9Pw/MF+o9ynkFKg7jdCziX 3ayH6VcJ8Csl7K3NibQWlshnRDa2OIMR+/P3IO8wIR3E0eZL7bdeFULqnpWa8/BEICeM 4B2i0ZHCzD1uWH49fvhLZIPBIymmJjOL9Hbproe0YSalOr4DFEs8VSbIwnmkbXnKcDh6 P75g== X-Gm-Message-State: AOAM530xWwnnpZXW0n4hZXDqZUPlCMsERNPc4Vv2BPb3S4PP9u+PS/tN Isfx3GZ4ANUT/Th353buePIhtolKy+Im3A== X-Google-Smtp-Source: ABdhPJx1UCC1AO9k1FBfjPY+ovM3cnQTtRtQkFe44Ucgq9wnTcXvTL9Ac/fnoFxnvE9Bc2IUXResxA== X-Received: by 2002:a17:906:2f0d:: with SMTP id v13mr2021161eji.220.1593170264605; Fri, 26 Jun 2020 04:17:44 -0700 (PDT) Received: from localhost.localdomain ([185.199.104.22]) by smtp.gmail.com with ESMTPSA id a1sm12779638ejk.125.2020.06.26.04.17.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Jun 2020 04:17:44 -0700 (PDT) From: chris@gregariousmammal.com To: libcamera-devel@lists.libcamera.org Date: Fri, 26 Jun 2020 13:17:32 +0200 Message-Id: <20200626111732.62852-1-chris@gregariousmammal.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH] Create libcamera overview and gloassry X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Chinchilla Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Chris Chinchilla Creates a libcamera overview guide and glossary to give new developers an idea of why they should use libcamera, its architecture, and features it offers. Signed-off-by: Chris Chinchilla --- Documentation/introduction.rst | 195 +++++++++++++++++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 Documentation/introduction.rst diff --git a/Documentation/introduction.rst b/Documentation/introduction.rst new file mode 100644 index 0000000..0539e60 --- /dev/null +++ b/Documentation/introduction.rst @@ -0,0 +1,195 @@ +An introduction to libcamera +============================ + +The Video for Linux 2 (`V4L2 `_) API provides kernel drivers for devices that provide and manipulate +images and video. However, Linux was missing a convenient way for application developers to take +advantage of these kernel drivers in userspace. Vendors of devices that provide image input sources +referred to as “image signal processors” (ISPs) sometimes contribute open-source V4L2 drivers. +However, ISPs vary so much, it’s a hard task for developers to write portable ISP-based +applications. + +The libcamera library aims to fill this gap by providing a complete userspace camera +stack for Linux-based systems that supports a wide variety of ISPs, including systems with multiple ISPs attached. + +The library currently supports: + +- Hardware + + - Intel IPU3 + - Rockchip RK3399 + - RaspberryPi 3 and 4 + - USB video device class (UVC) cameras + - Virtual media controller (vimc) driver + +- Software + + - Other Linux-kernel-based operating systems (such as Android and ChromeOS) + - Image enhancement algorithms + - A V4L2 compatibility layer + +The library provides a public API for managing ISPs, frame capture, video streams, frame and +request metadata, events and callbacks, image processing, and more. + +Where libcamera sits in the camera stack +---------------------------------------- + +The libcamera library sits in userspace, just on top of the kernel drivers that directly interact +with hardware and the V4L2 family of APIs (Media Controller, V4L2 Video Device, and V4L2 sub-device APIs). + +When using libcamera in a camera stack, it is the core component, taking control of all camera +devices, and exposing a native C++ API to upper layers of the framework. Other language bindings are coming soon. + +Compatibility Layers +~~~~~~~~~~~~~~~~~~~~ + +In a layer above the core framework are compatibility libraries to help existing applications and their developers. + +V4L2 Compatibility Layer +^^^^^^^^^^^^^^^^^^^^^^^^ + +To emulate high-level V4L2 camera devices and capture all access to camera devices, libcamera uses +a shared library. The shared library is transparent to libcamera-based applications and injected +into a process address space using dynamic linking with ``LD_PRELOAD``. + +The compatibility layer exposes camera device features on a best-effort basis and aims for the +level of features traditionally available from a UVC camera designed for video conferencing. + +Android Camera HAL +^^^^^^^^^^^^^^^^^^ + +The libcamera library offers Android camera support through a generic `hardware abstraction layer (HAL) `_ implementation. +The HAL focuses on supporting features required by Android that are missing from libcamera, such as JPEG encoding. + +The Android camera HAL implementation initially targets the ``LIMITED`` +`hardware level `_, +with support for the ``FULL`` level then gradually implemented. + +gstreamer element +^^^^^^^^^^^^^^^^^ + +The library provides `a gstreamer element `_ for routing libcamera data to gstreamer for processing. + +libcamera Architecture +---------------------- + +The libcamera library exposes one unified API, but behind that is built from a +set of specific components for each supported device, components for different +functionalities, and helpers for those components. + +Camera Devices Manager +~~~~~~~~~~~~~~~~~~~~~~ + +Provides access and manages all cameras on a system. It enumerates cameras at +build and runtime and supports a hotplug notification mechanism. + +There is only ever one instance of the Camera Devices Manager running per application. + +`Read the Camera Manager API documentation for more details `_ + +Camera Device +~~~~~~~~~~~~~ + +A camera device is a device capable of producing one or more image streams from +a single image source and can include multiple identical camera devices. + +The libcamera library represents a camera device to the upper layers of the library. +It exposes full control of the device through the public API, making it the highest +level object libcamera exposes, and the object that all other API operations interact +with from configuration to capture. + +`Read the Camera API documentation for more details `_ + +Frame controls +^^^^^^^^^^^^^^ + +Depending on the capabilities of a camera device and the hardware it is connected +to, libcamera supports controlling capture parameters for each stream on a per-frame basis. + +These controls include auto exposure, gain, brightness, contrast, lux, white balance, color temperature, and saturation. + +`Read the Control API documentation for more details `_ + +Pipeline Handler +~~~~~~~~~~~~~~~~ + +A pipeline handler is a private API that manages the pipelines exposed by the kernel +drivers through the Media Controller and V4L2 APIs. + +Each supported camera device has its own pipeline handler component, which hides +and abstracts device-specific details, providing what the rest of the library +needs to interact with a device. + +A pipeline handler manages most aspects of interacting with a camera device including: + +- acquiring use of the camera device +- frame controls +- pipelines and stream configuration +- the data the camera produces, and the buffers it needs to hold the data +- creating requests applications need to access camera data + +A pipeline handler is part of the libcamera codebase, and hardware vendors generally need to create them. + +`Read the PipelineHandler API documentation for more details `_ + +Image Processing Algorithms +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An image processing algorithm (IPA) component is a private API that implements +3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) algorithms. + +Each supported camera device has its own IPA component that runs on the CPU and +interacts with the kernel camera devices to control hardware image processing +based on the parameters supplied by upper layers, and helps maintain state, +closing the control loop of the ISP. + +An IPA component can be part of the libcamera code base, in which case the same +license covers it, or provided externally as an open-source or closed-source component. + +.. TODO: Better in the IPA guide? + +The component is sandboxed and can only interact with libcamera through specifically +marked APIs. A component has no direct access to kernel camera devices, and dmabuf +instances explicitly passed to the component control its access to image data and +metadata. The component should run in a process separate from the main libcamera +process, and has a restricted view of the system, including no access to networking APIs +and limited access to file systems. + +The sandboxing mechanism isn’t defined by libcamera. Platform vendors need to provide sandboxing +mechanisms as a plugin. + +To support this security and sandboxing model, libcamera provides an IPC mechanism +for an IPA and Pipeline Handler to communicate, but developers need to create the +API themselves. Platform vendors can also use any other IPC mechanism that supports +passing file descriptors. + +3A and Image Enhancement Algorithms +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Camera devices should implement auto exposure, auto gain, and auto white balance, +and those that include a focus lens should also implement autofocus. + +Vendors can implement these algorithms in hardware or firmware outside of libcamera, +or in an IPA component. + + +.. TODO: Add link to guide when completed + +Helpers and Support Classes +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. TODO: Feel like this is implied in other places of the doc + +To help developers create device-specific pipeline handlers and image processing +algorithms, libcamera provides helpers and support classes that sit on top of the +Media Controller and V4L2 APIs. + +Glossary +-------- + +- **ISP**: Image Signal Processor. +- **SoC**: System on a Chip. +- **Media controller API**: `The Linux Kernel API `_ that handles audio and video input and output. +- **V4L2**: `Video for Linux API version 2 `_. Device drivers and API for video capture on Linux. +- **UVC camera**: `USB Video Class `_ that describes devices capable of streaming video. +- **3A**: Common algorithms used by camera devices for auto-exposure, auto-white balance and auto-focus). +- **IPC**: `Inter-process communications based protocol `_.