{"id":9338,"url":"https://patchwork.libcamera.org/api/patches/9338/?format=json","web_url":"https://patchwork.libcamera.org/patch/9338/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20200820134751.278033-2-kieran.bingham@ideasonboard.com>","date":"2020-08-20T13:47:49","name":"[libcamera-devel,v4,1/3] Documentation: Guides: Developers Guide to Libcamera","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"57f65a763b8cf8289d38dfa2e1a18fa276c7da5f","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/?format=json","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/9338/mbox/","series":[{"id":1225,"url":"https://patchwork.libcamera.org/api/series/1225/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1225","date":"2020-08-20T13:47:48","name":"Developer Guides","version":4,"mbox":"https://patchwork.libcamera.org/series/1225/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/9338/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/9338/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C04C4BE173\n\tfor ;\n\tThu, 20 Aug 2020 13:47:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 791A961FE7;\n\tThu, 20 Aug 2020 15:47:59 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4F49760381\n\tfor ;\n\tThu, 20 Aug 2020 15:47:57 +0200 (CEST)","from Q.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net\n\t[86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id B17C9329;\n\tThu, 20 Aug 2020 15:47:56 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"AXm5BLm+\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1597931277;\n\tbh=chTvLguGK5lwpgNpV5CZqy8olyZPydGled1PQlfSzPY=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=AXm5BLm++Tqf1bOtFxO2UqFqc09hF5ThYwHdK1pdTWak9W58bGzY5Y51eECHvZbbb\n\tPA9rv7srNG93INg831NN3vhyRFdBlzJEfg3yRQ/llUXi21kqivoU4Y+OCdfk9BQj8R\n\t2QnkY5nmq44lB145hLE/ibBgBvQ2tI7ZE+JFb/bM=","From":"Kieran Bingham ","To":"libcamera devel ,\n\tChris Ward , Jacopo Mondi ","Date":"Thu, 20 Aug 2020 14:47:49 +0100","Message-Id":"<20200820134751.278033-2-kieran.bingham@ideasonboard.com>","X-Mailer":"git-send-email 2.25.1","In-Reply-To":"<20200820134751.278033-1-kieran.bingham@ideasonboard.com>","References":"<20200820134751.278033-1-kieran.bingham@ideasonboard.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v4 1/3] Documentation: Guides: Developers\n\tGuide to Libcamera","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"From: Chris Chinchilla \n\nCreate an introduction and overview for new developers to libcamera.\n\nProvide an overview of the Camera Stack, and Architecture of libcamera\nand introduce the main concepts of libcamera.\n\nSigned-off-by: Chris Chinchilla \n[Kieran: Rework/Reflow, add diagrams, licensing]\nSigned-off-by: Kieran Bingham \n\n---\n Documentation/guides/introduction.rst | 319 ++++++++++++++++++++++++++\n Documentation/index.rst | 2 +\n Documentation/meson.build | 1 +\n 3 files changed, 322 insertions(+)\n create mode 100644 Documentation/guides/introduction.rst","diff":"diff --git a/Documentation/guides/introduction.rst b/Documentation/guides/introduction.rst\nnew file mode 100644\nindex 000000000000..f34d2cf7cbdc\n--- /dev/null\n+++ b/Documentation/guides/introduction.rst\n@@ -0,0 +1,319 @@\n+.. SPDX-License-Identifier: CC-BY-SA-4.0\n+\n+Developers guide to libcamera\n+=============================\n+\n+The Linux kernel handles multimedia devices through the 'Linux media' subsystem\n+and provides a set of APIs (application programming interfaces) known\n+collectively as V4L2 (`Video for Linux 2`_) and the `Media Controller`_ API\n+which provide an interface to interact and control media devices.\n+\n+Included in this subsystem are drivers for camera sensors, CSI2 (Camera\n+Serial Interface) recievers, and ISPs (Image Signal Processors)\n+\n+The usage of these drivers to provide a functioning camera stack is a\n+responsibility that lies in userspace which is commonly implemented separately\n+by vendors without a common architecture or API for application developers.\n+\n+libcamera provides a complete camera stack for Linux based systems to abstract\n+functionality desired by camera application developers and process the\n+configuration of hardware and image control algorithms requried to obtain\n+desireable results from the camera.\n+\n+.. _Video for Linux 2: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/v4l2.html\n+.. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html\n+\n+\n+In this developers guide, we will explore the `Camera Stack`_ and how it is\n+can be visualised at a high level, and explore the internal `Architecture`_ of\n+the libcamera library with its components. The current `Platform Support`_ is\n+detailed, as well as an overview of the `Licensing`_ requirements of the\n+project.\n+\n+This introduction is followed by a walkthrough tutorial to newcomers wishing to\n+support a new platform with the `Pipeline Handler Writers Guide`_ and for those\n+looking to make use of the libcamera native API an `Application Writers Guide`_\n+provides a tutorial of the key APIs exposed by libcamera.\n+\n+.. _Pipeline Handler Writers Guide: pipeline-handler.html\n+.. _Application Writers Guide: application-developer.html\n+\n+.. TODO: Correctly link to the other articles of the guide\n+\n+Camera Stack\n+------------\n+\n+The libcamera library is implemented in userspace, and makes use of underlying\n+kernel drivers that directly interact with hardware.\n+\n+Applications can make use of libcamera through the native `libcamera API`_'s or\n+through an adaptation layer integrating libcamera into a larger framework.\n+\n+.. _libcamera API: https://www.libcamera.org/api-html/index.html\n+\n+::\n+\n+ Application Layer\n+ / +--------------+ +--------------+ +--------------+ +--------------+\n+ | | Native | | Framework | | Native | | Android |\n+ | | V4L2 | | Application | | libcamera | | Camera |\n+ | | Application | | (gstreamer) | | Application | | Framework |\n+ \\ +--------------+ +--------------+ +--------------+ +--------------+\n+\n+ ^ ^ ^ ^\n+ | | | |\n+ | | | |\n+ v v | v\n+ Adaptation Layer |\n+ / +--------------+ +--------------+ | +--------------+\n+ | | V4L2 | | gstreamer | | | Android |\n+ | | Compatability| | element | | | Camera |\n+ | | (preload) | |(libcamerasrc)| | | HAL |\n+ \\ +--------------+ +--------------+ | +--------------+\n+ |\n+ ^ ^ | ^\n+ | | | |\n+ | | | |\n+ v v v v\n+ libcamera Framework\n+ / +--------------------------------------------------------------------+\n+ | | |\n+ | | libcamera |\n+ | | |\n+ \\ +--------------------------------------------------------------------+\n+\n+ ^ ^ ^\n+ Userspace | | |\n+ --------------------- | ---------------- | ---------------- | ---------------\n+ Kernel | | |\n+ v v v\n+\n+ +-----------+ +-----------+ +-----------+\n+ | Media | <--> | Video | <--> | V4L2 |\n+ | Device | | Device | | Subdev |\n+ +-----------+ +-----------+ +-----------+\n+\n+The camera stack comprises of four software layers. From bottom to top:\n+\n+* The kernel drivers control the camera hardware and expose a low-level\n+ interface to userspace through the Linux kernel V4L2 family of APIs\n+ (Media Controller API, V4L2 Video Device API and V4L2 Subdev API).\n+\n+* The libcamera framework is the core part of the stack. It handles all control\n+ of the camera devices in its core component, libcamera, and exposes a native\n+ C++ API to upper layers.\n+\n+* The libcamera adaptation layer is an umbrella term designating the components\n+ that interface to libcamera in other frameworks. Notable examples are the V4L2\n+ compatibility layer, the gstreamer libcamera element, and the Android camera\n+ HAL implementation based on libcamera which are provided as a part of the\n+ libcamera project.\n+\n+* The applications and upper level frameworks are based on the libcamera\n+ framework or libcamera adaptation, and are outside of the scope of the\n+ libcamera project, however example native applications (cam, qcam) are\n+ provided for testing.\n+\n+\n+V4L2 Compatibility Layer\n+ V4L2 compatibility is achieved through a shared library that traps all\n+ accesses to camera devices and routes them to libcamera to emulate high-level\n+ V4L2 camera devices. It is injected in a process address space through\n+ `LD_PRELOAD` and is completely transparent for applications.\n+\n+ The compatibility layer exposes camera device features on a best-effort basis,\n+ and aims for the level of features traditionally available from a UVC camera\n+ designed for video conferencing.\n+\n+Android Camera HAL\n+ Camera support for Android is achieved through a generic Android camera HAL\n+ implementation on top of libcamera. The HAL implements features required by\n+ Android and out of scope from libcamera, such as JPEG encoding support.\n+\n+ This component is used to provide support for ChromeOS platforms\n+\n+GStreamer element (gstlibcamerasrc)\n+ A `GStreamer element`_ is provided to allow capture from libcamera supported\n+ devices through GStreamer pipelines, and connect to other elements for further\n+ processing.\n+\n+ Development of this element is ongoing and is limited to a single stream.\n+\n+Native libcamera API\n+ Applications can make use of the libcamera API directly using the C++\n+ API. An example application and walkthrough using the libcamera API can be\n+ followed in the `Application Writers Guide`_\n+\n+.. _GStreamer element: https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html\n+\n+Architecture\n+------------\n+\n+While offering a unified API towards upper layers, and presenting itself as a\n+single library, libcamera isn’t monolithic. It exposes multiple components\n+through its public API and is built around a set of separate helpers internally.\n+Hardware abstractractions are handled through the use of device-specific\n+components where required and dynamically loadable plugins are used to separate\n+image processing algorithms from the core libcamera codebase.\n+\n+::\n+\n+ --------------------------< libcamera Public API >---------------------------\n+ ^ ^\n+ | |\n+ v v\n+ +-------------+ +---------------------------------------------------+\n+ | Camera | | Camera Device |\n+ | Manager | | +-----------------------------------------------+ |\n+ +-------------+ | | Device-Agnostic | |\n+ ^ | | | |\n+ | | | +--------------------------+ |\n+ | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |\n+ | | | | { +-----------------+ } |\n+ | | | | } | //// Image //// | { |\n+ | | | | <-> | / Processing // | } |\n+ | | | | } | / Algorithms // | { |\n+ | | | | { +-----------------+ } |\n+ | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |\n+ | | | | ========================== |\n+ | | | | +-----------------+ |\n+ | | | | | // Pipeline /// | |\n+ | | | | <-> | /// Handler /// | |\n+ | | | | | /////////////// | |\n+ | | +--------------------+ +-----------------+ |\n+ | | Device-Specific |\n+ | +---------------------------------------------------+\n+ | ^ ^\n+ | | |\n+ v v v\n+ +--------------------------------------------------------------------+\n+ | Helpers and Support Classes |\n+ | +-------------+ +-------------+ +-------------+ +-------------+ |\n+ | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |\n+ | | Support | | Allocator | | IPC | | Manager | |\n+ | +-------------+ +-------------+ +-------------+ +-------------+ |\n+ | +-------------+ +-------------+ |\n+ | | Pipeline | | ... | |\n+ | | Runner | | | |\n+ | +-------------+ +-------------+ |\n+ +--------------------------------------------------------------------+\n+\n+ /// Device-Specific Components\n+ ~~~ Sandboxing\n+\n+\n+Camera Manager\n+ The Camera Manager enumerates cameras and instantiates Pipeline Handlers to\n+ manage each Camera that libcamera supports. The Camera Manager supports\n+ hotplug detection and notification events when supported by the underlying\n+ kernel devices.\n+\n+ There is only ever one instance of the Camera Manager running per application.\n+ Each application's instance of the Camera Manager ensures that only a single\n+ application can take control of a camera device at once.\n+\n+ Read the `Camera Manager API`_ documentation for more details.\n+\n+.. _Camera Manager API: http://libcamera.org/api-html/classlibcamera_1_1CameraManager.html\n+\n+Camera Device\n+ The Camera class represents a single item of camera hardware that is capable\n+ of producing one or more image streams, and provides the API to interact with\n+ the underlying device.\n+\n+ If a system has multiple instances of the same hardware attached, each has it's\n+ own instance of the camera class.\n+\n+ The API exposes full control of the device to upper layers of libcamera through\n+ the public API, making it the highest level object libcamera exposes, and the\n+ object that all other API operations interact with from configuration to\n+ capture.\n+\n+ Read the `Camera API`_ documentation for more details.\n+\n+.. _Camera API: http://libcamera.org/api-html/classlibcamera_1_1Camera.html\n+\n+Pipeline Handler\n+ The Pipeline Handler manages the complex pipelines exposed by the kernel\n+ drivers through the Media Controller and V4L2 APIs. It abstracts pipeline\n+ handling to hide device-specific details from the rest of the library, and\n+ implements both pipeline configuration based on stream configuration, and\n+ pipeline runtime execution and scheduling when needed by the device.\n+\n+ The Pipeline Handler lives in the same process as the rest of the library, and\n+ has access to all helpers and kernel camera-related devices.\n+\n+ Hardware abstraction is handled by device specific Pipeline Handlers which are\n+ derived from the Pipeline Handler base class allowing commonality to be shared\n+ among the implementations.\n+\n+ Derived pipeline handlers create Camera device instances based on the devices\n+ they detect and support on the running system, and are responsible for\n+ managing the interactions with a camera device.\n+\n+ More details can be found in the `PipelineHandler API`_ documentation, and the\n+ `Pipeline Handler Writers Guide`_.\n+\n+.. _PipelineHandler API: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html\n+\n+Image Processing Algorithms\n+ An image processing algorithm (IPA) component is a loadable plugin that\n+ implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other\n+ algorithms.\n+\n+ The algorithms run on the CPU and interact with the camera devices through the\n+ Pipeline Handler to control hardware image processing based on the parameters\n+ supplied by upper layers, maintaining state and closing the control loop\n+ of the ISP.\n+\n+ The component is sandboxed and can only interact with libcamera through the\n+ API provided by the Pipeline Handler and an IPA has no direct access to kernel\n+ camera devices.\n+\n+ Open source IPA modules built with libcamera can be run in the same process\n+ space as libcamera, however external IPA modules are run in a separate process\n+ from the main libcamera process. IPA modules have a restricted view of the\n+ system, including no access to networking APIs and limited access to file\n+ systems.\n+\n+ IPA modules are only required for platforms and devices with an ISP controlled\n+ by the host CPU. Camera sensors which have an integrated ISP are not\n+ controlled through the IPA module.\n+\n+Platform Support\n+----------------\n+\n+The library currently supports the following hardware platforms specifically\n+with dedicated pipeline handlers:\n+\n+ - Intel IPU3 (ipu3)\n+ - Rockchip RK3399 (rkisp1)\n+ - RaspberryPi 3 and 4 (raspberrypi)\n+\n+Furthermore, generic platform support is provided for the following:\n+\n+ - USB video device class cameras (uvcvideo)\n+ - iMX7, Allwinner Sun6i (simple)\n+ - Virtual media controller driver for test use cases (vimc)\n+\n+Licensing\n+---------\n+\n+The libcamera core, is covered by the `LGPL-2.1-or-later`_ license. Pipeline\n+Handlers are a part of the libcamera code base and need to be contributed\n+upstream by device vendors. IPA modules included in libcamera are covered by a\n+free software license, however third-parties may develop IPA modules outside of\n+libcamera and distribute them under a closed-source license, provided they do\n+not include source code from the libcamera project.\n+\n+The libcamera project itself contains multiple libraries, applications and\n+utilities. Licenses are expressed through SPDX tags in text-based files that\n+support comments, and through the .reuse/dep5 file otherwise. A copy of all\n+licenses are stored in the LICENSES directory, and a full summary of the\n+licensing used throughout the project can be found in the COPYING.rst document.\n+\n+Applications which link dynamically against libcamera and use only the public\n+API are an independent work of the authors and have no license restrictions\n+imposed upon them from libcamera.\n+\n+.. _LGPL-2.1-or-later: https://spdx.org/licenses/LGPL-2.1-or-later.html\n\\ No newline at end of file\ndiff --git a/Documentation/index.rst b/Documentation/index.rst\nindex 4e746bb17c4a..cfcfd388699b 100644\n--- a/Documentation/index.rst\n+++ b/Documentation/index.rst\n@@ -12,3 +12,5 @@\n Home \n Docs \n Contribute \n+\n+ Developer Guide \ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex 6d9a397cf1a3..dd7ae700af11 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -52,6 +52,7 @@ if sphinx.found()\n 'contributing.rst',\n 'docs.rst',\n 'index.rst',\n+ 'guides/introduction.rst',\n ]\n \n release = 'release=v' + libcamera_git_version\n","prefixes":["libcamera-devel","v4","1/3"]}