[{"id":10891,"web_url":"https://patchwork.libcamera.org/comment/10891/","msgid":"<65d52038-15a4-60c8-8d0b-93c43c576378@uajain.com>","date":"2020-06-26T15:38:46","subject":"Re: [libcamera-devel] [PATCH] Create libcamera overview and gloassry","submitter":{"id":1,"url":"https://patchwork.libcamera.org/api/people/1/","name":"Umang Jain","email":"email@uajain.com"},"content":"Hi Chris,\n\nThank you the work.\n\nI read through the entire overview and have commented my thoughts.\nHowever, I didn't feel I am the right person to review \"Image Processing \nAlgorithms\"\n(and sections below) so maybe a more experienced person can review those\nparts which might be actually helpful.\n\nOn 6/26/20 4:47 PM, chris@gregariousmammal.com wrote:\n> From: Chris Chinchilla \nYou can drop this as you already have 'Signed-off' Tag below.\n>\n> Creates a libcamera overview guide and glossary to give new developers an idea of why they should use libcamera, its architecture, and features it offers.\nCan you please honor 72-character line limit as per git's commit message\nconvention here? Then it would be perfect :)\n>\n> Signed-off-by: Chris Chinchilla \nPerfect. :)\n> ---\n> Documentation/introduction.rst | 195 +++++++++++++++++++++++++++++++++\n> 1 file changed, 195 insertions(+)\n> create mode 100644 Documentation/introduction.rst\n>\n> diff --git a/Documentation/introduction.rst b/Documentation/introduction.rst\n> new file mode 100644\n> index 0000000..0539e60\n> --- /dev/null\n> +++ b/Documentation/introduction.rst\n> @@ -0,0 +1,195 @@\n> +An introduction to libcamera\n> +============================\n> +\n> +The Video for Linux 2 (`V4L2 `_) API provides kernel drivers for devices that provide and manipulate\n> +images and video. However, Linux was missing a convenient way for application developers to take\n> +advantage of these kernel drivers in userspace. Vendors of devices that provide image input sources\ns/devices/camera devices/\n> \n> +referred to as “image signal processors” (ISPs) sometimes contribute open-source V4L2 drivers.\n> +However, ISPs vary so much, it’s a hard task for developers to write portable ISP-based\n> +applications.\n> +\n> +The libcamera library aims to fill this gap by providing a complete userspace camera\n> +stack for Linux-based systems that supports a wide variety of ISPs, including systems with multiple ISPs attached.\nhmm, I didn't understand the phrase \"with multiple ISPs attached\".\nDoes a vendor generally offers multiple ISPs with the camera? Does it \nhappen quite often?\nI honestly don't know, so probably a newbie reader(like me :P) won't \nknow too. Being a slight\nverbose about the multiple ISPs thing, might help.\n> +\n> +The library currently supports:\n> +\n> +- Hardware\n> +\n> + - Intel IPU3\n> + - Rockchip RK3399\n> + - RaspberryPi 3 and 4\n> + - USB video device class (UVC) cameras\n> + - Virtual media controller (vimc) driver\n> +\n> +- Software\n> +\n> + - Other Linux-kernel-based operating systems (such as Android and ChromeOS)\n> + - Image enhancement algorithms\n> + - A V4L2 compatibility layer\n> +\nI personally find that libcamera framework being able to integrate with \nproprietary IPAs,\na strong focus of public attention. Can we somehow flex that capability \nand say something\naround that a bit more 'explicitly' here ? :) No pressure though!\n> +The library provides a public API for managing ISPs, frame capture, video streams, frame and\n> +request metadata, events and callbacks, image processing, and more.\n> +\n> +Where libcamera sits in the camera stack\n> +----------------------------------------\n> +\n> +The libcamera library sits in userspace, just on top of the kernel drivers that directly interact\n> +with hardware and the V4L2 family of APIs (Media Controller, V4L2 Video Device, and V4L2 sub-device APIs).\n> +\n> +When using libcamera in a camera stack, it is the core component, taking control of all camera\n> +devices, and exposing a native C++ API to upper layers of the framework. Other language bindings are coming soon.\n> +\n> +Compatibility Layers\n> +~~~~~~~~~~~~~~~~~~~~\n> +\n> +In a layer above the core framework are compatibility libraries to help existing applications and their developers.\n> +\nDid you mean s/above/below/?\n> +V4L2 Compatibility Layer\n> +^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +To emulate high-level V4L2 camera devices and capture all access to camera devices, libcamera uses\n> +a shared library. The shared library is transparent to libcamera-based applications and injected\n> +into a process address space using dynamic linking with ``LD_PRELOAD``.\n> +\n> +The compatibility layer exposes camera device features on a best-effort basis and aims for the\n> +level of features traditionally available from a UVC camera designed for video conferencing.\n> +\n> +Android Camera HAL\n> +^^^^^^^^^^^^^^^^^^\n> +\n> +The libcamera library offers Android camera support through a generic `hardware abstraction layer (HAL) `_ implementation.\n> +The HAL focuses on supporting features required by Android that are missing from libcamera, such as JPEG encoding.\n> +\n> +The Android camera HAL implementation initially targets the ``LIMITED``\n> +`hardware level `_,\n> +with support for the ``FULL`` level then gradually implemented.\n> +\n> +gstreamer element\n> +^^^^^^^^^^^^^^^^^\n> +\n> +The library provides `a gstreamer element `_ for routing libcamera data to gstreamer for processing.\n> +\n> +libcamera Architecture\n> +----------------------\n> +\n> +The libcamera library exposes one unified API, but behind that is built from a\nI would slight repharse: s/behind that/on the backend side, it is built.../\n> +set of specific components for each supported device, components for different\n> +functionalities, and helpers for those components.\n> +\n> +Camera Devices Manager\n> +~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +Provides access and manages all cameras on a system. It enumerates cameras at\n> +build and runtime and supports a hotplug notification mechanism.\n> +\n> +There is only ever one instance of the Camera Devices Manager running per application.\n> +\n> +`Read the Camera Manager API documentation for more details `_\n> +\n> +Camera Device\n> +~~~~~~~~~~~~~\n> +\n> +A camera device is a device capable of producing one or more image streams from\n> +a single image source and can include multiple identical camera devices.\nThe 'multiple identical camera devices' bit confuses me here. I didn't \nhave time right\nnow to dig into this but I had a brief conversation on IRC. Attaching \nhere for your reference:\n\n From camera.cpp, can somebody clear me one question for me\n > * A camera device contains a single image source, and \nseparate camera device\n >  * instances relate to different image sources.\n The first 'camera device'  in the sentence acutally means a \n\"media device\" (Like a phone/laptop)\n and the second 'camera device' is a camera (i.e. instance of \nCamera class)\n Am I making any sense?\n no, both mean a Camera instance\n this means that there's a 1:1 mapping between camera sensors \nand Camera instances\n (for now at least)\n hmm, I see :S\n-*- uajain digs more\n I am acutally trying to de-reference Chris' statement on \n\"libcamera overview and glossary patch\"\n +A camera device is a device capable of producing one or more \nimage streams from\n +a single image source and can include multiple identical \ncamera devices.\n The 'can include multiple identical camera devices' part is \ntripping me up\n maybe he meant that a system can include multiple identical \ncamera devices, as in multiple UVC webcams for instance ?\n I'm not sure\n Yeah, I need to ask him when he's online.\n\n> +\n> +The libcamera library represents a camera device to the upper layers of the library.\n> +It exposes full control of the device through the public API, making it the highest\n> +level object libcamera exposes, and the object that all other API operations interact\n> +with from configuration to capture.\n> +\n> +`Read the Camera API documentation for more details `_\n> +\n> +Frame controls\n> +^^^^^^^^^^^^^^\n> +\n> +Depending on the capabilities of a camera device and the hardware it is connected\n> +to, libcamera supports controlling capture parameters for each stream on a per-frame basis.\n> +\n> +These controls include auto exposure, gain, brightness, contrast, lux, white balance, color temperature, and saturation.\n> +\n> +`Read the Control API documentation for more details `_\n> +\n> +Pipeline Handler\n> +~~~~~~~~~~~~~~~~\n> +\n> +A pipeline handler is a private API that manages the pipelines exposed by the kernel\n> +drivers through the Media Controller and V4L2 APIs.\n> +\n> +Each supported camera device has its own pipeline handler component, which hides\n> +and abstracts device-specific details, providing what the rest of the library\n> +needs to interact with a device.\n> +\n> +A pipeline handler manages most aspects of interacting with a camera device including:\n> +\n> +- acquiring use of the camera device\n> +- frame controls\n> +- pipelines and stream configuration\n> +- the data the camera produces, and the buffers it needs to hold the data\n> +- creating requests applications need to access camera data\nI would also include that, pipeline handler is responsible for \nregistering the camera with CameraManager\nso that it can be made available to the application.\n> +\n> +A pipeline handler is part of the libcamera codebase, and hardware vendors generally need to create them.\n> +\n> +`Read the PipelineHandler API documentation for more details `_\n> +\n> +Image Processing Algorithms\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +An image processing algorithm (IPA) component is a private API that implements\n> +3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) algorithms.\n> +\n> +Each supported camera device has its own IPA component that runs on the CPU and\n> +interacts with the kernel camera devices to control hardware image processing\n> +based on the parameters supplied by upper layers, and helps maintain state,\n> +closing the control loop of the ISP.\n> +\n> +An IPA component can be part of the libcamera code base, in which case the same\n> +license covers it, or provided externally as an open-source or closed-source component.\n> +\n> +.. TODO: Better in the IPA guide?\n> +\n> +The component is sandboxed and can only interact with libcamera through specifically\n> +marked APIs. A component has no direct access to kernel camera devices, and dmabuf\n> +instances explicitly passed to the component control its access to image data and\n> +metadata. The component should run in a process separate from the main libcamera\n> +process, and has a restricted view of the system, including no access to networking APIs\n> +and limited access to file systems.\n> +\n> +The sandboxing mechanism isn’t defined by libcamera. Platform vendors need to provide sandboxing\n> +mechanisms as a plugin.\n> +\n> +To support this security and sandboxing model, libcamera provides an IPC mechanism\n> +for an IPA and Pipeline Handler to communicate, but developers need to create the\n> +API themselves. Platform vendors can also use any other IPC mechanism that supports\n> +passing file descriptors.\n> +\n> +3A and Image Enhancement Algorithms\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +Camera devices should implement auto exposure, auto gain, and auto white balance,\n> +and those that include a focus lens should also implement autofocus.\n> +\n> +Vendors can implement these algorithms in hardware or firmware outside of libcamera,\n> +or in an IPA component.\n> +\n> +\n> +.. TODO: Add link to guide when completed\n> +\n> +Helpers and Support Classes\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +.. TODO: Feel like this is implied in other places of the doc\n> +\n> +To help developers create device-specific pipeline handlers and image processing\n> +algorithms, libcamera provides helpers and support classes that sit on top of the\n> +Media Controller and V4L2 APIs.\n> +\n> +Glossary\n> +--------\n> +\n> +- **ISP**: Image Signal Processor.\n> +- **SoC**: System on a Chip.\n> +- **Media controller API**: `The Linux Kernel API `_ that handles audio and video input and output.\n> +- **V4L2**: `Video for Linux API version 2 `_. Device drivers and API for video capture on Linux.\n> +- **UVC camera**: `USB Video Class `_ that describes devices capable of streaming video.\n> +- **3A**: Common algorithms used by camera devices for auto-exposure, auto-white balance and auto-focus).\n> +- **IPC**: `Inter-process communications based protocol `_.\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4952FC2E66\n\tfor ;\n\tFri, 26 Jun 2020 15:38:51 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AACF4609C7;\n\tFri, 26 Jun 2020 17:38:50 +0200 (CEST)","from o1.f.az.sendgrid.net (o1.f.az.sendgrid.net [208.117.55.132])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 44CAC603BB\n\tfor ;\n\tFri, 26 Jun 2020 17:38:48 +0200 (CEST)","by filterdrecv-p3mdw1-6dd9dd6d45-qzpsc with SMTP id\n\tfilterdrecv-p3mdw1-6dd9dd6d45-qzpsc-20-5EF61686-32\n\t2020-06-26 15:38:46.651270859 +0000 UTC m=+682579.644523707","from mail.uajain.com (unknown)\n\tby ismtpd0005p1maa1.sendgrid.net (SG) with ESMTP\n\tid agC7NQzZR3-Enc9V0Ot9Qw Fri, 26 Jun 2020 15:38:46.040 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=uajain.com header.i=@uajain.com\n\theader.b=\"jqx7Crb2\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=uajain.com;\n\th=subject:references:from:mime-version:in-reply-to:to:content-type:\n\tcontent-transfer-encoding;\n\ts=s1; bh=bytqa4e5ubHZkRFrLAQzk2Fz8cRnbODj1BQoePxD1oY=;\n\tb=jqx7Crb2py+11F9G8d2zDD7HXFh8WOn3RAKbdPwFqPQC4SE6tEFUIV6rTSJWndW0BO1j\n\tetUUds1svIXIUfk18Sy6jDd+KIiR1cVDxuR6XTbui+Re18VeA4MdZVOpyJ3+/9vv1OHZ1o\n\tRzeXmRrsTNLuX95cpjEAHelDYIy9hFsJY=","References":"<20200626111732.62852-1-chris@gregariousmammal.com>","From":"Umang Jain ","Message-ID":"<65d52038-15a4-60c8-8d0b-93c43c576378@uajain.com>","Date":"Fri, 26 Jun 2020 15:38:46 +0000 (UTC)","Mime-Version":"1.0","In-Reply-To":"<20200626111732.62852-1-chris@gregariousmammal.com>","X-SG-EID":"1Q40EQ7YGir8a9gjSIAdTjhngY657NMk9ckeo4dbHZDiOpywc/L3L9rFqlwE4KPc1X6pmd7do7wYP56agy+aLsbBrgmfzwGlE0/FvEunmysSiknVzCH3zxXIFcnTMDJwS+vc2FsEeuH9S2K0OLS9JPSt0v3RiKq8slN91j6CwFjUdM1lsBY6CtGGCWP63owUqLsYpwnttujg+nIkK03+FLedQOYjUSxrd0AYlzeZEgFNEjlcKarKUjO0if1zPQfsOfRkF0Gef996G/cDB1P1aw==","To":"chris@gregariousmammal.com, libcamera-devel@lists.libcamera.org","Content-Language":"en-US","Subject":"Re: [libcamera-devel] [PATCH] Create libcamera overview and gloassry","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Transfer-Encoding":"base64","Content-Type":"text/plain; charset=\"utf-8\"; Format=\"flowed\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11015,"web_url":"https://patchwork.libcamera.org/comment/11015/","msgid":"<465e4482-22e8-ed71-e5fd-a334579d9a20@ideasonboard.com>","date":"2020-06-30T12:42:57","subject":"Re: [libcamera-devel] [PATCH] Create libcamera overview and gloassry","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Chris,\n\nOn 26/06/2020 12:17, chris@gregariousmammal.com wrote:\n> From: Chris Chinchilla \n> \n> Creates a libcamera overview guide and glossary to give new developers an idea of why they should use libcamera, its architecture, and features it offers.\n> \n> Signed-off-by: Chris Chinchilla \n> ---\n> Documentation/introduction.rst | 195 +++++++++++++++++++++++++++++++++\n> 1 file changed, 195 insertions(+)\n> create mode 100644 Documentation/introduction.rst\n> \n> diff --git a/Documentation/introduction.rst b/Documentation/introduction.rst\n> new file mode 100644\n> index 0000000..0539e60\n> --- /dev/null\n> +++ b/Documentation/introduction.rst\n> @@ -0,0 +1,195 @@\n> +An introduction to libcamera\n> +============================\n> +\n> +The Video for Linux 2 (`V4L2 `_) API provides kernel drivers for devices that provide and manipulate\n> +images and video. However, Linux was missing a convenient way for application developers to take \n> +advantage of these kernel drivers in userspace. Vendors of devices that provide image input sources \n> +referred to as “image signal processors” (ISPs) sometimes contribute open-source V4L2 drivers. \n> +However, ISPs vary so much, it’s a hard task for developers to write portable ISP-based\n> +applications.\n> +\n> +The libcamera library aims to fill this gap by providing a complete userspace camera \n> +stack for Linux-based systems that supports a wide variety of ISPs, including systems with multiple ISPs attached.\n\nI'm not sure that 'systems with multiple ISPs attached' makes sense here.\n\n> +\n> +The library currently supports:\n> +\n> +- Hardware\n> +\n> + - Intel IPU3\n> + - Rockchip RK3399\n> + - RaspberryPi 3 and 4\n> + - USB video device class (UVC) cameras\n> + - Virtual media controller (vimc) driver\n> +\n> +- Software\n> +\n> + - Other Linux-kernel-based operating systems (such as Android and ChromeOS)\n\nI'd describe this as :\n - ChromeOS support through an Android HAL3 adaptation layer\n\n> + - A V4L2 compatibility layer\n\t - A V4L2 compatibility layer for existing V4L2 based applications\n\n - A gstreamer element for gstreamer pipeline based applications.\n\n\n\n> + - Image enhancement algorithms\n\nI'm not sure this point counts here yet, so I'd drop it.\nUnless it's meant to mean 'Image processing algorithms' ?\n\nMaybe that's the same thing...\n\n> +\n> +The library provides a public API for managing ISPs, frame capture, video streams, frame and \n> +request metadata, events and callbacks, image processing, and more.\n> +\n> +Where libcamera sits in the camera stack\n> +----------------------------------------\n> +\n> +The libcamera library sits in userspace, just on top of the kernel drivers that directly interact \n> +with hardware and the V4L2 family of APIs (Media Controller, V4L2 Video Device, and V4L2 sub-device APIs).\n> +\n> +When using libcamera in a camera stack, it is the core component, taking control of all camera \n> +devices, and exposing a native C++ API to upper layers of the framework. Other language bindings are coming soon.\n\nI'd maybe refrain from saying 'coming soon', as that could imply 'next\nweek' which wouldn't be accurate.\n\nHow about \" are expected potential developments.\"\n\n> +\n> +Compatibility Layers\n> +~~~~~~~~~~~~~~~~~~~~\n> +\n> +In a layer above the core framework are compatibility libraries to help existing applications and their developers.\n> +\n> +V4L2 Compatibility Layer\n> +^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +To emulate high-level V4L2 camera devices and capture all access to camera devices, libcamera uses \n> +a shared library. The shared library is transparent to libcamera-based applications and injected \n> +into a process address space using dynamic linking with ``LD_PRELOAD``.\n> +\n> +The compatibility layer exposes camera device features on a best-effort basis and aims for the \n> +level of features traditionally available from a UVC camera designed for video conferencing.\n> +\n> +Android Camera HAL\n> +^^^^^^^^^^^^^^^^^^\n> +\n> +The libcamera library offers Android camera support through a generic `hardware abstraction layer (HAL) `_ implementation. \n> +The HAL focuses on supporting features required by Android that are missing from libcamera, such as JPEG encoding.\n> +\n> +The Android camera HAL implementation initially targets the ``LIMITED``\n> +`hardware level `_,\n> +with support for the ``FULL`` level then gradually implemented.\n> +\n> +gstreamer element\n> +^^^^^^^^^^^^^^^^^\n> +\n> +The library provides `a gstreamer element `_ for routing libcamera data to gstreamer for processing.\n\n'for further processing in a gstreamer pipeline.' ?\n\n> +\n> +libcamera Architecture\n> +----------------------\n> +\n> +The libcamera library exposes one unified API, but behind that is built from a\n\n'it is built'\n\n> +set of specific components for each supported device, components for different \n> +functionalities, and helpers for those components.\n\nThis seems quite vague, so I think we need to expand here a bit, though\nI guess this is just an intro into this section ?\n\n\nHow about:\n\n'is built from a collection of re-usable components to support platform\nspecific configuration of hardware {with/to give/to provide?} a device\nagnostic API.'\n\n> +\n> +Camera Devices Manager\n\nCamera Manager\n\n> +~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +Provides access and manages all cameras on a system. It enumerates cameras at \n\n'Provides access control' ?\n\n> +build and runtime and supports a hotplug notification mechanism.\n\n\nIt can't enumerate cameras at build time. Only runtime.\n\n\"It enumerates cameras at runtime, and instantiates Pipeline Handlers to\nmanage each Camera device. The CameraManager supports hotplug detection\nand notification events when supported by the underlying kernel devices.\"\n\n\n> +\n> +There is only ever one instance of the Camera Devices Manager running per application.\n\nI'd probably call it the \"Camera Device Manager\"\nAnd lets expand that it prevents concurrent access to cameras even\nacross different applictions:\n\n\"The Camera Device Manager ensures that only a single application can\ntake control of a camera at any one time.\"\n\nEach application's instance of the CameraManager\n\n\n> +\n> +`Read the Camera Manager API documentation for more details `_\n> +\n> +Camera Device\n> +~~~~~~~~~~~~~\n> +\n> +A camera device is a device capable of producing one or more image streams from \n\nI wouldn't call a \"camera device\" a device like that... It's a class to\nrepresent the underlying hardware as a single object, providing the API\ninterface to interact with.\n\nI see that the term 'Camera device' comes from the opening brief in the\ndetailed description at\nhttp://libcamera.org/api-html/classlibcamera_1_1Camera.html#details, so\nI suspect that should be improved/expanded upon there too.\n\n\n\n> +a single image source and can include multiple identical camera devices.\n\nI'm not sure that's quite right (\"and can include multiple identical\ncamera devices...\"). A single instance of a Camera class represents a\nsingle underlying camera.\n\nIt can get complicated in the future where a logical camera might be\ncomposed of multiple sensors merged together (to produce exotic HDR or\nsuch I think) - but we don't need to worry about that for now. Currently\n- a Camera instance is a single camera, which can be multiple streams of\nthe same source capture image.\n\n> +\n> +The libcamera library represents a camera device to the upper layers of the library. \n> +It exposes full control of the device through the public API, making it the highest \n> +level object libcamera exposes, and the object that all other API operations interact \n> +with from configuration to capture.\n> +\n> +`Read the Camera API documentation for more details `_\n> +\n> +Frame controls\n> +^^^^^^^^^^^^^^\n> +\n> +Depending on the capabilities of a camera device and the hardware it is connected \n> +to, libcamera supports controlling capture parameters for each stream on a per-frame basis.\n> +\n> +These controls include auto exposure, gain, brightness, contrast, lux, white balance, color temperature, and saturation.\n> +\n> +`Read the Control API documentation for more details `_\n> +\n> +Pipeline Handler\n> +~~~~~~~~~~~~~~~~\n> +\n> +A pipeline handler is a private API that manages the pipelines exposed by the kernel \n\nHrm, it's more like - A pipeline handler implements the internal API\nrequired to manage ...\n\n> +drivers through the Media Controller and V4L2 APIs.\n\nHrm: I'd say something more like:\n\n\n\"A pipeline handler implements the internal API to manage, configure,\nand control the pipelines exposed by the kernel drivers through the\nMedia Controller and V4L2 kernel interfaces.\"\n\nOr\n\n\"Pipeline handlers are the underlying abstraction layer for platform\nspecific configuration of hardware. They access and control hardware\nthrough the V4L2 and Media Controller kernel interfaces, to implement an\ninternal API to directly control the ISP and Capture components of a\npipeline.\"\n\n\n\n> +\n> +Each supported camera device has its own pipeline handler component, which hides \n> +and abstracts device-specific details, providing what the rest of the library \n\nNot quite, Pipeline handlers 'create' Camera device instances, based on\nthe devices they detect and support in the running system.\n\n\n> +needs to interact with a device.\n\n\n\n> +\n> +A pipeline handler manages most aspects of interacting with a camera device including:\n> +\n> +- acquiring use of the camera device\n\nHrm ... Is that done by pipeline handlers, or the CameraManager ?\n\n> +- frame controls\n> +- pipelines and stream configuration\n> +- the data the camera produces, and the buffers it needs to hold the data\n> +- creating requests applications need to access camera data\n\nRequests are 'given' to a Pipeline handler to describe what the pipeline\nhandler must do ...\n\n> +\n> +A pipeline handler is part of the libcamera codebase, and hardware vendors generally need to create them.\n\nI'm not fond of that sentence, because while indeed a hardware vendor\nmight be best suited to create one for a target, they're probably not\nalways going to be the ones who do it.\n\nmaybe:\n\n\"Pipeline handlers form part of the internal libcamera codebase, and are\nneeded to be implemented for complex platforms with an ISP requiring\ndevice specific configurations.\"\n\n> +\n> +`Read the PipelineHandler API documentation for more details `_\n> +\n> +Image Processing Algorithms\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +An image processing algorithm (IPA) component is a private API that implements \n> +3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) algorithms.\n> +\n> +Each supported camera device has its own IPA component that runs on the CPU and \n> +interacts with the kernel camera devices to control hardware image processing \n> +based on the parameters supplied by upper layers, and helps maintain state, \n> +closing the control loop of the ISP.\n> +\n> +An IPA component can be part of the libcamera code base, in which case the same \n> +license covers it, or provided externally as an open-source or closed-source component.\n> +\n> +.. TODO: Better in the IPA guide?\n\nI would say there's benefit to having some description here.\n\nThe IPA guide will be about what is needed to write and design an IPA\nmodule.\n\n> +\n> +The component is sandboxed and can only interact with libcamera through specifically \n> +marked APIs. A component has no direct access to kernel camera devices, and dmabuf \n> +instances explicitly passed to the component control its access to image data and \n> +metadata. The component should run in a process separate from the main libcamera \n> +process, and has a restricted view of the system, including no access to networking APIs\n> +and limited access to file systems.\n> +\n> +The sandboxing mechanism isn’t defined by libcamera. Platform vendors need to provide sandboxing \n> +mechanisms as a plugin.\n\nHrm ... We only really support two platforms currently: Linux and ChromeOS.\n\nI would say we do define the requirements, but the implementation is\nplatform specific and handled via platform specific plugins?\n\n\n> +\n> +To support this security and sandboxing model, libcamera provides an IPC mechanism \n> +for an IPA and Pipeline Handler to communicate, but developers need to create the \n> +API themselves. Platform vendors can also use any other IPC mechanism that supports \n> +passing file descriptors.\n> +\n> +3A and Image Enhancement Algorithms\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +Camera devices should implement auto exposure, auto gain, and auto white balance, \n> +and those that include a focus lens should also implement autofocus.\n> +\n> +Vendors can implement these algorithms in hardware or firmware outside of libcamera, \n> +or in an IPA component.\n\nI think it's more like \"Vendors implement the control algorithms\nrequired to control\n\n> +\n> +\n> +.. TODO: Add link to guide when completed\n> +\n> +Helpers and Support Classes\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +.. TODO: Feel like this is implied in other places of the doc\n> +\n> +To help developers create device-specific pipeline handlers and image processing \n> +algorithms, libcamera provides helpers and support classes that sit on top of the \n> +Media Controller and V4L2 APIs.\n\nYes, either this section should be dropped, or it should actually\nintroduce and talk about the details of the MC/V4L2 classes, and the\nDeviceEnumerator APIs.\n\n\n> +\n> +Glossary\n> +--------\n> +\n> +- **ISP**: Image Signal Processor.\n> +- **SoC**: System on a Chip.\n\nSoC is never mentioned in this document so far, so I guess it doesn't\nneed to be in the glossary.\n\n\n> +- **Media controller API**: `The Linux Kernel API `_ that handles audio and video input and output.\n> +- **V4L2**: `Video for Linux API version 2 `_. Device drivers and API for video capture on Linux.\n> +- **UVC camera**: `USB Video Class `_ that describes devices capable of streaming video.\n\n\"of streaming video over the UVC protocol.\"\n\n> +- **3A**: Common algorithms used by camera devices for auto-exposure, auto-white balance and auto-focus).\n> +- **IPC**: `Inter-process communications based protocol `_.\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 490F9BFFE2\n\tfor ;\n\tTue, 30 Jun 2020 12:43:03 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AC23960C58;\n\tTue, 30 Jun 2020 14:43:02 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8762D603B4\n\tfor ;\n\tTue, 30 Jun 2020 14:43:00 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DA45E29F;\n\tTue, 30 Jun 2020 14:42:59 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Cnu05N/r\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1593520980;\n\tbh=3ZJnhMMN/XWGSDL7kveRMGs6D54TxO+zTdIA9GraBGw=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=Cnu05N/rmUUkgd+0DA+m41aRNQq3G/rupUsV+XRmr0WCsvqfiRU6DTMmn30OfQowZ\n\tgbKI9Evrat78kp+xtnyOILUSitV75PIxmALUWjFJI5m5fwafza7+CcXsxLyeMAi23i\n\t5XTUXmXUmOJG+N9oCf0F3hRgDGX9xNggqmdbHwMM=","To":"chris@gregariousmammal.com, libcamera-devel@lists.libcamera.org","References":"<20200626111732.62852-1-chris@gregariousmammal.com>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<465e4482-22e8-ed71-e5fd-a334579d9a20@ideasonboard.com>","Date":"Tue, 30 Jun 2020 13:42:57 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.8.0","MIME-Version":"1.0","In-Reply-To":"<20200626111732.62852-1-chris@gregariousmammal.com>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [PATCH] Create libcamera overview and gloassry","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]