From patchwork Fri Jul 10 10:30:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Chinchilla X-Patchwork-Id: 8740 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 381FCBD790 for ; Fri, 10 Jul 2020 10:30:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0330A6118A; Fri, 10 Jul 2020 12:30:24 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gregariousmammal.com header.i=@gregariousmammal.com header.b="qqHDC2g9"; dkim-atps=neutral Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3211E61186 for ; Fri, 10 Jul 2020 12:30:22 +0200 (CEST) Received: by mail-ed1-x536.google.com with SMTP id d16so4232430edz.12 for ; Fri, 10 Jul 2020 03:30:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gregariousmammal.com; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MhEx4ZV+ukCmmJIPE6LpT6c4d9ghZUWOH33mWjtI7nQ=; b=qqHDC2g9lVgN6buCTxZKhFd6MmGfba7kjRZcMyBNjqBoplKrB6WLdnBWSRntV6I9In eIKURWbiyEesMzMOWoG0oFWYKmm8w9P2NEukc2b6ez0f3hvh7L7sdmyYwTDafg2hukoW GLZDtIlMqNVJRC59lE0s2/PVoMUNhog2kXXfw0FKLHrjhDzY2uCtRpAHxM/MsihtRa5D EWybhwbXmx2OALpGDMaLenyiHDzxPiRa6+z+hkUeTv5OSJWWcctkLzIIaLQnp9aAU0ev odgO6fTj3s7tIGR8CDh8N9bOWlfLrEXx/y/9VKsS+UaVtF6e1A1D7qzuVfA82s3acAkc 2TAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=MhEx4ZV+ukCmmJIPE6LpT6c4d9ghZUWOH33mWjtI7nQ=; b=thoj5gm5G1rioy8Dsf3IW5srtLbRxycAUxnSZDiiF0WOpTISGMZPI9bnU5vcKOLohm afwvl95/PkG+wdIR2nAV8vkRa/D2yJ+q9jJAbbn5gX3l7CR08xqUWxBxUMlKtBObE/JV JlNN7RuNU1LA+pcoieJPk63zBzxavmi2pyRb+J1PLULJ9ahcOtliA+Jw0UaC9CNqZgq0 lhJq7Qab9reMwoOzqlRpEXWGqdpFyk4VKZLkYgEYa0k/i4J44gwUOVYRFD1M+n47rzc9 MC4XAy8n1QWUjQm7XRggKXBGbAHOmagTtLRZwjn4EPFQ5OnxVLjhTS9Gb2U4DrT+50Tv 15ag== X-Gm-Message-State: AOAM532hMY5D+T4SeaB76cXunSVfC2WEqn792cAFIBFSK+OETSrB6jfV OkZ6RUKV2laS1bmUX18a7zArKf3WP3uwvQqE X-Google-Smtp-Source: ABdhPJxCA166YNbbwbYJsgAmT9oxSPJMN/P/v38gOa4Dzu506QVEPR7YYZCkfKbjpa+nBP3X4+q0ew== X-Received: by 2002:a50:f413:: with SMTP id r19mr80233139edm.17.1594377021245; Fri, 10 Jul 2020 03:30:21 -0700 (PDT) Received: from localhost.localdomain (business-90-187-212-1.pool2.vodafone-ip.de. [90.187.212.1]) by smtp.gmail.com with ESMTPSA id o17sm3463592ejb.105.2020.07.10.03.30.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 10 Jul 2020 03:30:20 -0700 (PDT) From: chris@gregariousmammal.com To: libcamera-devel@lists.libcamera.org Date: Fri, 10 Jul 2020 12:30:16 +0200 Message-Id: <20200710103016.85455-1-chris@gregariousmammal.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3] Create libcamera overview document and glossary X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Chinchilla Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Chris Chinchilla Creates a libcamera architecture overview guide and glossary. Signed-off-by: Chris Chinchilla --- Documentation/introduction.rst | 198 +++++++++++++++++++++++++++++++++ 1 file changed, 198 insertions(+) create mode 100644 Documentation/introduction.rst diff --git a/Documentation/introduction.rst b/Documentation/introduction.rst new file mode 100644 index 0000000..1e87218 --- /dev/null +++ b/Documentation/introduction.rst @@ -0,0 +1,198 @@ +An introduction to libcamera +============================ + +The Video for Linux 2 (`V4L2 `_) API provides kernel drivers for devices that provide and manipulate +images and video. However, Linux was missing a convenient way for application developers to take +advantage of these kernel drivers in userspace. Vendors of devices that provide image input sources +referred to as "image signal processors" (ISPs) sometimes contribute open-source V4L2 drivers. +However, ISPs vary so much, and it's a hard task for developers to write portable ISP-based +applications. + +The libcamera library aims to fill this gap by providing a complete userspace camera +stack for Linux-based systems that supports a wide variety of ISPs, including systems with multiple ISPs attached. + +The library currently supports: + +- Hardware + + - Intel IPU3 + - Rockchip RK3399 + - RaspberryPi 3 and 4 + - USB video device class (UVC) cameras + - Virtual media controller (vimc) driver + +- Software + + - Image processing algorithms + - A V4L2 compatibility layer for existing V4L2 based applications + - A gstreamer element for gstreamer pipeline based applications. + - ChromeOS support through an Android HAL3 adaptation layer (does not have V4L2 or gstreamer support) + +The library provides a public API for managing ISPs, frame capture, video streams, frame and +request metadata, events and callbacks, image processing, and more. + +Where libcamera sits in the camera stack +---------------------------------------- + +The libcamera library sits in userspace, just on top of the kernel drivers that directly interact +with hardware and the V4L2 family of APIs (Media Controller, V4L2 Video Device, and V4L2 sub-device APIs). + +When using libcamera in a camera stack, it is the core component, taking control of all camera +devices, and exposing a native C++ API to upper layers of the framework. Other language bindings are in development. + +Compatibility Layers +~~~~~~~~~~~~~~~~~~~~ + +In a layer above the core framework are compatibility libraries to help existing applications and their developers. + +V4L2 Compatibility Layer +^^^^^^^^^^^^^^^^^^^^^^^^ + +To emulate high-level V4L2 camera devices and capture all access to camera devices, libcamera uses +a shared library. The shared library is transparent to libcamera-based applications and injected +into a process address space using dynamic linking with "LD_PRELOAD ". + +The compatibility layer exposes camera device features on a best-effort basis and aims for the +level of features traditionally available from a UVC camera designed for video conferencing. + +Android and ChromeOS Camera HAL +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +ChromeOS uses the Android `hardware abstraction layer (HAL) `_ to provide camera support, and libcamera uses that to offer support +for ChromeOS-based devices. + +The HAL focuses on supporting features required by Android that are missing from libcamera, such as +JPEG encoding. + +The HAL implementation initially targets the "LIMITED " +`hardware level `_, +with support for the "FULL "level then gradually implemented. + +gstreamer element +^^^^^^^^^^^^^^^^^ + +The library provides `a gstreamer element `_ that routes libcamera data for +further processing in a gstreamer pipeline. + +libcamera Architecture +---------------------- + +The libcamera library exposes one unified API, but behind that is built from +re-usable components that provide hardware specific support and configuration +with a device agnostic API. + +Camera Manager +~~~~~~~~~~~~~~ + +It enumerates cameras at runtime and instantiates a `Pipeline Handler`_ to manage each Camera +device that libcamera supports. The Camera Manager supports hotplug detection +and notification events when supported by the underlying kernel devices. + +There is only ever one instance of the Camera Manager running per application. Each application's +instance of the Camera Manager ensures that only a single application can take control of a camera +device at once. + +`Read the Camera Manager API documentation for more details `_ + +Camera Device +~~~~~~~~~~~~~ + +The camera class represents a single item of camera hardware that is capable of producing one or more image streams, and provides the API to interact with the underlying device. + +If a system has multiple instances of the same hardware attached, each has it's own instance of the camera class. + +The API exposes full control of the device to upper layers of libcamera through the public API, +making it the highest level object libcamera exposes, and the object that all other API operations +interact with from configuration to capture. + +`Read the Camera API documentation for more details `_ + +Frame controls +^^^^^^^^^^^^^^ + +Depending on the capabilities of a camera device and the hardware it is connected +to, libcamera supports controlling capture parameters for each stream on a per-frame basis. + +These controls include auto exposure, gain, brightness, contrast, lux, white balance, color +temperature, and saturation. + +`Read the Control API documentation for more details `_ + +Pipeline Handler +~~~~~~~~~~~~~~~~ + +Pipeline handlers are the abstraction layer for platform-specific hardware configuration. They +access and control hardware through the V4L2 and Media Controller kernel interfaces, and implement +an internal API to control the ISP and Capture components of a pipeline directly. + +Pipeline handlers' create' Camera device instances based on the devices they detect and support on +the running system. + +A pipeline handler manages most aspects of interacting with a camera device including: + +- frame controls +- pipelines and stream configuration +- the data the camera produces, and the buffers it needs to hold the data +- granting access to camera data + +Pipeline handlers form part of the libcamera codebase, and developers need to implement them for +complex hardware with an ISP that requires device-specific configurations. + +`Read the PipelineHandler API documentation for more details `_ + +Image Processing Algorithms +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An image processing algorithm (IPA) component is a private API that implements +3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) algorithms. + +Each supported camera device has its own IPA component that runs on the CPU and +interacts with the kernel camera devices to control hardware image processing +based on the parameters supplied by upper layers, and helps maintain state, +closing the control loop of the ISP. + +An IPA component can be part of the libcamera code base, in which case the same +license covers it, or provided externally as an open-source or closed-source component. + +The component is sandboxed and can only interact with libcamera through specifically +marked APIs. A component has no direct access to kernel camera devices, and dmabuf +instances explicitly passed to the component control its access to image data and +metadata. The component should run in a process separate from the main libcamera +process, and has a restricted view of the system, including no access to networking APIs +and limited access to file systems. + +While libcamera requires sandboxing, the implementation is platform-specific, and handled by +platform-specific plugins. + +To support this security and sandboxing model, libcamera provides an IPC mechanism +for an IPA and Pipeline Handler to communicate, but developers need to create the +API themselves. Platform vendors can also use any other IPC mechanism that supports +passing file descriptors. + +3A and Image Enhancement Algorithms +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Camera devices should implement auto exposure, auto gain, and auto white balance, +and those that include a focus lens should also implement autofocus. + +Device vendors implement the control methods required to control these algorithms in hardware or +firmware outside of libcamera, or in an IPA component. + +.. TODO: Add link to guide when completed + +Helpers and Support Classes +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To help developers create device-specific pipeline handlers and image processing +algorithms, libcamera provides helpers and support classes that sit on top of the +Media Controller and V4L2 APIs. + +Glossary +-------- + +- **ISP**: Image Signal Processor. +- **Media controller API**: `The Linux Kernel API `_ that handles audio and video input and output. +- **V4L2**: `Video for Linux API version 2 `_. Device drivers and API for video capture on Linux. +- **UVC camera**: `USB Video Class `_ that describes devices capable of streaming video over the UVC protocol. +- **3A**: Common algorithms used by camera devices for auto-exposure, auto-white balance and auto-focus). +- **IPC**: `Inter-process communications based protocol `_.