From patchwork Thu Dec 13 14:23:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 43 Return-Path: Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 88602600CC for ; Thu, 13 Dec 2018 15:22:43 +0100 (CET) Received: from avalon.bb.dnainternet.fi (dfj612ybrt5fhg77mgycy-3.rev.dnainternet.fi [IPv6:2001:14ba:21f5:5b00:2e86:4862:ef6a:2804]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id EB0FF549 for ; Thu, 13 Dec 2018 15:22:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1544710963; bh=8lCRNJ3bHk+d0Fr1t5oIx4H+epUSQcEkfWH6F5oB+Q0=; h=From:To:Subject:Date:From; b=bSnnWA3SErLa4LGSzC+H+LrNAWmBiDAhxMwcL0TDRewcj5RNxqGqCYMR9YlRGEK+8 lnD5tjUacgcPi0tD+z4ABhw18zZVV7uLP3OdE89LmKUmL/MAz8nVpCynIZ4eFYRZve mdyP7NXaHEkhry4OnMRPeVbEBd6D6J7UKAu8muG4= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Thu, 13 Dec 2018 16:23:25 +0200 Message-Id: <20181213142327.4455-1-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.19.2 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/3] Documentation: Remove _static and _templates directory X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Dec 2018 14:22:43 -0000 sphinx only requires those directories to be present because they're referenced in the configuration file. Remove the references and the directories. Signed-off-by: Laurent Pinchart Acked-by: Jacopo Mondi --- Documentation/_static/.keep_empty | 0 Documentation/_templates/.keep_empty | 0 Documentation/conf.py | 4 ++-- 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 Documentation/_static/.keep_empty delete mode 100644 Documentation/_templates/.keep_empty diff --git a/Documentation/_static/.keep_empty b/Documentation/_static/.keep_empty deleted file mode 100644 index e69de29bb2d1..000000000000 diff --git a/Documentation/_templates/.keep_empty b/Documentation/_templates/.keep_empty deleted file mode 100644 index e69de29bb2d1..000000000000 diff --git a/Documentation/conf.py b/Documentation/conf.py index 83d233e51e91..770358198f03 100644 --- a/Documentation/conf.py +++ b/Documentation/conf.py @@ -42,7 +42,7 @@ extensions = [ ] # Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] +templates_path = [] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: @@ -85,7 +85,7 @@ html_theme = 'alabaster' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] +html_static_path = [] # Custom sidebar templates, must be a dictionary that maps document names # to template names. From patchwork Thu Dec 13 14:23:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 44 Return-Path: Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B622860B1B for ; Thu, 13 Dec 2018 15:22:43 +0100 (CET) Received: from avalon.bb.dnainternet.fi (dfj612ybrt5fhg77mgycy-3.rev.dnainternet.fi [IPv6:2001:14ba:21f5:5b00:2e86:4862:ef6a:2804]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 437E5568 for ; Thu, 13 Dec 2018 15:22:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1544710963; bh=LzeSPerLSjg7an7bhNgKCmJPkuvN5j229Q7YhTvOHxg=; h=From:To:Subject:Date:In-Reply-To:References:From; b=iXKbrx9fyZG4SNm/u5kLL5EZHp7LexWRpcJb95tySNzlIRKwjbNyiVh4qRKB+zd3+ vTVljMXSYVxj8v9PGzEVpN2+NsQk4EBK+im9SmyJDaTb5jk3o+hvJjz8T2+XEKSWKT p366nF3dOwQNAul8Jtsrfy6sr91NpVRWWfjQ4IJM= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Thu, 13 Dec 2018 16:23:26 +0200 Message-Id: <20181213142327.4455-2-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181213142327.4455-1-laurent.pinchart@ideasonboard.com> References: <20181213142327.4455-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/3] Documentation: contributing: Add links to libcamera git and linuxtv.org X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Dec 2018 14:22:43 -0000 Signed-off-by: Laurent Pinchart --- Documentation/contributing.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Documentation/contributing.rst b/Documentation/contributing.rst index b0b67ebcc727..0eedadc94906 100644 --- a/Documentation/contributing.rst +++ b/Documentation/contributing.rst @@ -27,10 +27,13 @@ Source Code ----------- libcamera is in early stages of development, and no releases are available yet. -The source code is available from the project's git tree, hosted by LinuxTV. +The source code is available from the project's `git tree`_, hosted by `LinuxTV`_. $ git clone git://linuxtv.org/libcamera.git +.. _git tree: https://git.linuxtv.org/libcamera.git/ +.. _LinuxTV: https://linuxtv.org/ + Documentation ------------- From patchwork Thu Dec 13 14:23:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 45 Return-Path: Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0B07360B1B for ; Thu, 13 Dec 2018 15:22:44 +0100 (CET) Received: from avalon.bb.dnainternet.fi (dfj612ybrt5fhg77mgycy-3.rev.dnainternet.fi [IPv6:2001:14ba:21f5:5b00:2e86:4862:ef6a:2804]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9385658E for ; Thu, 13 Dec 2018 15:22:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1544710963; bh=QxvAKM6MylMaz5aOwZTmfazckLxq10hlPDbKQE65WyE=; h=From:To:Subject:Date:In-Reply-To:References:From; b=Fwl6EJgyjhgQFLMFiHxw9C5VKNk1PpmPfBOeym0tHvxFPmi+iyZ2dq55qctd+QAei 9LjW2rVFWHD4PpbvRuVqDQqwvPX53KVhVxJbusvVTnR1o86pImQ2OimC5uWXX36VGw UzF09auRkN89mKEz0xUisoymTyodc4oqjVQ5S6GY= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Thu, 13 Dec 2018 16:23:27 +0200 Message-Id: <20181213142327.4455-3-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181213142327.4455-1-laurent.pinchart@ideasonboard.com> References: <20181213142327.4455-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/3] Documentation: Add architecture documentation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Dec 2018 14:22:44 -0000 The documentation is copied mostly verbatim from the website, with small modifications to the ascii art diagrams to make them compile. Signed-off-by: Laurent Pinchart --- Documentation/docs.rst | 383 ++++++++++++++++++++++++++++++++++++++ Documentation/index.rst | 1 + Documentation/meson.build | 1 + 3 files changed, 385 insertions(+) create mode 100644 Documentation/docs.rst diff --git a/Documentation/docs.rst b/Documentation/docs.rst new file mode 100644 index 000000000000..4bbcd9aa35bf --- /dev/null +++ b/Documentation/docs.rst @@ -0,0 +1,383 @@ +************* +Documentation +************* + +Feature Requirements +==================== + +Device enumeration +------------------ + +The library shall support enumerating all camera devices available in the +system, including both fixed cameras and hotpluggable cameras. It shall +support cameras plugged and unplugged after the initialization of the +library, and shall offer a mechanism to notify applications of camera plug +and unplug. + +The following types of cameras shall be supported: + +* Internal cameras designed for point-and-shoot still image and video + capture usage, either controlled directly by the CPU, or exposed through + an internal USB bus as a UVC device. + +* External UVC cameras designed for video conferencing usage. + +Other types of camera, including analog cameras, depth cameras, thermal +cameras, external digital picture or movie cameras, are out of scope for +this project. + +A hardware device that includes independent camera sensors, such as front +and back sensors in a phone, shall be considered as multiple camera devices +for the purpose of this library. + +Independent Camera Devices +-------------------------- + +When multiple cameras are present in the system and are able to operate +independently from each other, the library shall expose them as multiple +camera devices and support parallel operation without any additional usage +restriction apart from the limitations inherent to the hardware (such as +memory bandwidth, CPU usage or number of CSI-2 receivers for instance). + +Independent processes shall be able to use independent cameras devices +without interfering with each other. A single camera device shall be +usable by a single process at a time. + +Multiple streams support +------------------------ + +The library shall support multiple video streams running in parallel +for each camera device, within the limits imposed by the system. + +Per frame controls +------------------ + +The library shall support controlling capture parameters for each stream +on a per-frame basis, on a best effort basis based on the capabilities of the +hardware and underlying software stack (including kernel drivers and +firmware). It shall apply capture parameters to the frame they target, and +report the value of the parameters that have effectively been used for each +captured frame. + +When a camera device supports multiple streams, the library shall allow both +control of each stream independently, and control of multiple streams +together. Streams that are controlled together shall be synchronized. No +synchronization is required for streams controlled independently. + +Capability Enumeration +---------------------- + +The library shall expose capabilities of each camera device in a way that +allows applications to discover those capabilities dynamically. Applications +shall be allowed to cache capabilities for as long as they are using the +library. If capabilities can change at runtime, the library shall offer a +mechanism to notify applications of such changes. Applications shall not +cache capabilities in long term storage between runs. + +Capabilities shall be discovered dynamically at runtime from the device when +possible, and may come, in part or in full, from platform configuration +data. + +Device Profiles +--------------- + +The library may define different camera device profiles, each with a minimum +set of required capabilities. Applications may use those profiles to quickly +determine the level of features exposed by a device without parsing the full +list of capabilities. Camera devices may implement additional capabilities +on top of the minimum required set for the profile they expose. + +3A and Image Enhancement Algorithms +----------------------------------- + +The camera devices shall implement auto exposure, auto gain and auto white +balance. Camera devices that include a focus lens shall implement auto +focus. Additional image enhancement algorithms, such as noise reduction or +video stabilization, may be implemented. + +All algorithms may be implemented in hardware or firmware outside of the +library, or in software in the library. They shall all be controllable by +applications. + +The library shall be architectured to isolate the 3A and image enhancement +algorithms in a component with a documented API, respectively called the 3A +component and the 3A API. The 3A API shall be stable, and shall allow both +open-source and closed-source implementations of the 3A component. + +The library may include statically-linked open-source 3A components, and +shall support dynamically-linked open-source and closed-source 3A +components. + +Closed-source 3A Component Sandboxing +------------------------------------- + +For security purposes, it may be desired to run closed-source 3A components +in a separate process. The 3A API would in such a case be transported over +IPC. The 3A API shall make it possible to use any IPC mechanism that +supports passing file descriptors. + +The library may implement an IPC mechanism, and shall support third-party +platform-specific IPC mechanisms through the implementation of a +platform-specific 3A API wrapper. No modification to the library shall be +needed to use such third-party IPC mechanisms. + +The 3A component shall not directly access any device node on the system. +Such accesses shall instead be performed through the 3A API. The library +shall validate all accesses and restrict them to what is absolutely required +by 3A components. + +V4L2 Compatibility Layer +------------------------ + +The project shall support traditional V4L2 application through an additional +libcamera wrapper library. The wrapper library shall trap all accesses to +camera devices through LD_PRELOAD, and route them through libcamera to +emulate a high-level V4L2 camera device. It shall expose camera device +features on a best-effort basis, and aim for the level of features +traditionally available from a UVC camera designed for video conferencing. + +Android Camera HAL v3 Compatibility +----------------------------------- + +The library API shall expose all the features required to implement an +Android Camera HAL v3 on top of libcamera. Some features of the HAL may be +omitted as long as they can be implemented separately in the HAL, such as +JPEG encoding, or YUV reprocessing. + + +Camera Stack +============ + +:: + + | a c / +-------------+ +-------------+ +-------------+ +-------------+ + | p a | | Native | | Framework | | Native | | Android | + | p t | | V4L2 | | Application | | libcamera | | Camera | + | l i | | Application | | (gstreamer) | | Application | | Framework | + | i o \ +-------------+ +-------------+ +-------------+ +-------------+ + | n ^ ^ ^ ^ + | | | | | + | l a | | | | + | i d v v | v + | b a / +-------------+ +-------------+ | +-------------+ + | c p | | V4L2 | | Camera | | | Android | + | a t | | Compat. | | Framework | | | Camera | + | m a | | | | (gstreamer) | | | HAL | + | e t \ +-------------+ +-------------+ | +-------------+ + | r i ^ ^ | ^ + | a o | | | | + | n | | | | + | / | ,................................................ + | | | ! : Language : ! + | l f | | ! : Bindings : ! + | i r | | ! : (optional) : ! + | b a | | \...............................................' + | c m | | | | | + | a e | | | | | + | m w | v v v v + | e o | +----------------------------------------------------------------+ + | r r | | | + | a k | | libcamera | + | | | | + | \ +----------------------------------------------------------------+ + | ^ ^ ^ + | Userspace | | | + | ------------------------ | ---------------- | ---------------- | --------------- + | Kernel | | | + | v v v + | +-----------+ +-----------+ +-----------+ + | | Media | <--> | Video | <--> | V4L2 | + | | Device | | Device | | Subdev | + | +-----------+ +-----------+ +-----------+ + +The camera stack comprises four software layers. From bottom to top: + +* The kernel drivers control the camera hardware and expose a + low-level interface to userspace through the Linux kernel V4L2 + family of APIs (Media Controller API, V4L2 Video Device API and + V4L2 Subdev API). + +* The libcamera framework is the core part of the stack. It + handles all control of the camera devices in its core component, + libcamera, and exposes a native C++ API to upper layers. Optional + language bindings allow interfacing to libcamera from other + programming languages. + + Those components live in the same source code repository and + all together constitute the libcamera framework. + +* The libcamera adaptation is an umbrella term designating the + components that interface to libcamera in other frameworks. + Notable examples are a V4L2 compatibility layer, a gstreamer + libcamera element, and an Android camera HAL implementation based + on libcamera. + + Those components can live in the libcamera project source code + in separate repositories, or move to their respective project's + repository (for instance the gstreamer libcamera element).

+ +* The applications and upper level frameworks are based on the + libcamera framework or libcamera adaptation, and are outside of + the scope of the libcamera project. + + +libcamera Architecture +====================== + +:: + + | ---------------------------< libcamera Public API >--------------------------- + | ^ ^ + | | | + | v v + | +-------------+ +-------------------------------------------------+ + | | Camera | | Camera Device | + | | Devices | | +---------------------------------------------+ | + | | Manager | | | Device-Agnostic | | + | +-------------+ | | | | + | ^ | | +------------------------+ | + | | | | | ~~~~~~~~~~~~~~~~~~~~~ | + | | | | | { +---------------+ } | + | | | | | } | ////Image//// | { | + | | | | | <-> | /Processing// | } | + | | | | | } | /Algorithms// | { | + | | | | | { +---------------+ } | + | | | | | ~~~~~~~~~~~~~~~~~~~~~ | + | | | | | ======================== | + | | | | | +---------------+ | + | | | | | | //Pipeline/// | | + | | | | | <-> | ///Handler/// | | + | | | | | | ///////////// | | + | | | +--------------------+ +---------------+ | + | | | Device-Specific | + | | +-------------------------------------------------+ + | | ^ ^ + | | | | + | v v v + | +--------------------------------------------------------------------+ + | | Helpers and Support Classes | + | | +-------------+ +-------------+ +-------------+ +-------------+ | + | | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | | + | | | Support | | Allocator | | IPC | | Manager | | + | | +-------------+ +-------------+ +-------------+ +-------------+ | + | | +-------------+ +-------------+ | + | | | Pipeline | | ... | | + | | | Runner | | | | + | | +-------------+ +-------------+ | + | +--------------------------------------------------------------------+ + | + | /// Device-Specific Components + | ~~~ Sandboxing + +While offering a unified API towards upper layers, and presenting +itself as a single library, libcamera isn't monolithic. It exposes +multiple components through its public API, is built around a set of +separate helpers internally, uses device-specific components and can +load dynamic plugins. + +Camera Devices Manager + The Camera Devices Manager provides a view of available cameras + in the system. It performs cold enumeration and runtime camera + management, and supports a hotplug notification mechanism in its + public API. + + To avoid the cost associated with cold enumeration of all devices + at application start, and to arbitrate concurrent access to camera + devices, the Camera Devices Manager could later be split to a + separate service, possibly with integration in platform-specific + device management. + +Camera Device + The Camera Device represents a camera device to upper layers. It + exposes full control of the device through the public API, and is + thus the highest level object exposed by libcamera. + + Camera Device instances are created by the Camera Devices + Manager. An optional method to create new instances could be exposed + through the public API to speed up initialization when the upper + layer knows how to directly address camera devices present in the + system. + +Pipeline Handler + The Pipeline Handler manages complex pipelines exposed by the kernel drivers + through the Media Controller and V4L2 APIs. It abstracts pipeline handling to + hide device-specific details to the rest of the library, and implements both + pipeline configuration based on stream configuration, and pipeline runtime + execution and scheduling when needed by the device. + + This component is device-specific and is part of the libcamera code base. As + such it is covered by the same free software license as the rest of libcamera + and needs to be contributed upstream by device vendors. The Pipeline Handler + lives in the same process as the rest of the library, and has access to all + helpers and kernel camera-related devices.

+ +Image Processing Algorithms + Together with the hardware image processing and hardware statistics + collection, the Image Processing Algorithms implement 3A (Auto-Exposure, + Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU + and interact with the kernel camera devices to control hardware image + processing based on the parameters supplied by upper layers, closing the + control loop of the ISP. + + This component is device-specific and is loaded as an external plugin. It can + be part of the libcamera code base, in which case it is covered by the same + license, or provided externally as an open-source or closed-source component. + + The component is sandboxed and can only interact with libcamera through + internal APIs specifically marked as such. In particular it will have no + direct access to kernel camera devices, and all its accesses to image and + metadata will be mediated by dmabuf instances explicitly passed to the + component. The component must be prepared to run in a process separate from + the main libcamera process, and to have a very restricted view of the system, + including no access to networking APIs and limited access to file systems. + + The sandboxing mechanism isn't defined by libcamera. One example + implementation will be provided as part of the project, and platforms vendors + will be able to provide their own sandboxing mechanism as a plugin. + + libcamera should provide a basic implementation of Image Processing + Algorithms, to serve as a reference for the internal API. Device vendors are + expected to provide a full-fledged implementation compatible with their + Pipeline Handler. One goal of the libcamera project is to create an + environment in which the community will be able to compete with the + closed-source vendor binaries and develop a high quality open source + implementation. + +Helpers and Support Classes + While Pipeline Handlers are device-specific, implementations are expected to + share code due to usage of identical APIs towards the kernel camera drivers + and the Image Processing Algorithms. This includes without limitation handling + of the MC and V4L2 APIs, buffer management through dmabuf, and pipeline + discovery, configuration and scheduling. Such code will be factored out to + helpers when applicable. + + Other parts of libcamera will also benefit from factoring code out to + self-contained support classes, even if such code is present only once in the + code base, in order to keep the source code clean and easy to read. This + should be the case for instance for plugin management. + + +V4L2 Compatibility Layer +------------------------ + +V4L2 compatibility is achieved through a shared library that traps all +accesses to camera devices and routes them to libcamera to emulate high-level +V4L2 camera devices. It is injected in a process address space through +LD_PRELOAD and is completely transparent for applications. + +The compatibility layer exposes camera device features on a best-effort basis, +and aims for the level of features traditionally available from a UVC camera +designed for video conferencing. + + +Android Camera HAL +------------------ + +Camera support for Android is achieved through a generic Android +camera HAL implementation on top of libcamera. The HAL will implement internally +features required by Android and missing from libcamera, such as JPEG encoding +support. + +The Android camera HAL implementation will initially target the +LIMITED harware level, with support for the FULL level then being gradually +implemented. diff --git a/Documentation/index.rst b/Documentation/index.rst index ba6c6c6ea56d..44176f230c0f 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -21,6 +21,7 @@ systems, including traditional Linux distributions, ChromeOS and Android. :caption: Contents: coding-style + docs contributing diff --git a/Documentation/meson.build b/Documentation/meson.build index 8264b5e6cdb1..dd43b2cbd401 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -44,6 +44,7 @@ if sphinx.found() 'coding-style.rst', 'conf.py', 'contributing.rst', + 'docs.rst', 'index.rst', ]