From patchwork Thu Sep 21 16:55:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 19078 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 58DF7BE080 for ; Thu, 21 Sep 2023 20:51:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9FAA962944; Thu, 21 Sep 2023 22:51:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1695329514; bh=wQAOVVolZr+lzqz9rw0S0k1komJoFVjV9h6sK8wDq9M=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Y+RjyVzZ6rKVG21ph9D4VpmUJejrbaHPN5XP6/SAdn4t23kwZUSotseQm8zvBrAnD H1Ri48LZm+5tNtSz2p57VDHoq8iqJY0/ttGHliXxpRsVImmTDCy+ssBkhxha/W3xDb lxdhMcgYU1PDT6JegKGtVDZcckIjUYr2QfLw49XeyK9LkEAOZqknoWe+SG028jOJt6 mSMy9IDN+CJSqE/qY86UMIwKhKaCXzVtzMFSWt5BWDNH/RnKEGaLeMqFQ79clLV3Ku mJbV0KnhmkUh9jea5ziscMHyHIy+ETqtHanOeq77ko4Xpke7aRWlBie05C1s9PhoID 3tMNInA7LFkVQ== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 424AF628D8 for ; Thu, 21 Sep 2023 18:56:02 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="GDCEV/fP"; dkim-atps=neutral Received: from uno.lan (93-46-82-201.ip106.fastwebnet.it [93.46.82.201]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 476941257; Thu, 21 Sep 2023 18:54:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1695315264; bh=wQAOVVolZr+lzqz9rw0S0k1komJoFVjV9h6sK8wDq9M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GDCEV/fPrAh+ENtQbEUqp1svIumUB1cewZqCJt791i7OUOdhiaP+JX5CUfnfNXCRS r8etNCNGPScAWzc2dv/DXCZPxwWi0vtNNMkMqLOYXl/Y1T2sht2H5NGjbWOCyhvyqz pJ4o6HXeWyy4HFTT7O/i1bqKajKGpkqqUEvY1v1o= To: libcamera-devel@lists.libcamera.org Date: Thu, 21 Sep 2023 18:55:38 +0200 Message-ID: <20230921165550.50956-2-jacopo.mondi@ideasonboard.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20230921165550.50956-1-jacopo.mondi@ideasonboard.com> References: <20230921165550.50956-1-jacopo.mondi@ideasonboard.com> MIME-Version: 1.0 X-Mailman-Approved-At: Thu, 21 Sep 2023 22:51:53 +0200 Subject: [libcamera-devel] [PATCH v5 01/13] documentation: Introduce Camera Sensor Model X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Cc: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Introduce a documentation page about the 'camera sensor model' implemented by libcamera. The camera sensor model serves to provide to applications a reference description of the processing steps that take place in a camera sensor in order to precisely control the sensor configuration through the forthcoming SensorConfiguration class. Signed-off-by: Jacopo Mondi Reviewed-by: Naushir Patuck Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- .reuse/dep5 | 6 + Documentation/binning.svg | 5053 +++++++++++++++++++++++++ Documentation/camera-sensor-model.rst | 174 + Documentation/index.rst | 1 + Documentation/meson.build | 1 + Documentation/sensor_model.svg | 4870 ++++++++++++++++++++++++ Documentation/skipping.svg | 1720 +++++++++ 7 files changed, 11825 insertions(+) create mode 100644 Documentation/binning.svg create mode 100644 Documentation/camera-sensor-model.rst create mode 100644 Documentation/sensor_model.svg create mode 100644 Documentation/skipping.svg diff --git a/.reuse/dep5 b/.reuse/dep5 index d283a0c9e856..45e7eca3d714 100644 --- a/.reuse/dep5 +++ b/.reuse/dep5 @@ -23,3 +23,9 @@ Files: utils/ipc/mojo Copyright: Copyright 2013-2020 The Chromium Authors. All rights reserved. License: BSD-3-Clause Source: https://chromium.googlesource.com/chromium/src.git/ + +Files: Documentation/binning.svg + Documentation/camera-sensor-model.rst + Documentation/sensor_model.svg +Copyright: Copyright 2023 Ideas On Board Oy +License: CC-BY-SA-4.0 diff --git a/Documentation/binning.svg b/Documentation/binning.svg new file mode 100644 index 000000000000..c6a3b6394acd --- /dev/null +++ b/Documentation/binning.svg @@ -0,0 +1,5053 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/camera-sensor-model.rst b/Documentation/camera-sensor-model.rst new file mode 100644 index 000000000000..58bf70e1dc5a --- /dev/null +++ b/Documentation/camera-sensor-model.rst @@ -0,0 +1,174 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +.. _camera-sensor-model: + +.. todo: Move to Doxygen-generated documentation + +The libcamera camera sensor model +================================= + +libcamera defines an abstract camera sensor model in order to provide +a description of each of the processing steps that result in image data being +sent on the media bus and that form the image stream delivered to applications. + +Applications should use the abstract camera sensor model defined here to +precisely control the operations of the camera sensor. + +The libcamera camera sensor model targets image sensors producing frames in +RAW format, delivered through a MIPI CSI-2 compliant bus implementation. + +The abstract sensor model maps libcamera components to the characteristics and +operations of an image sensor, and serves as a reference to model the libcamera +CameraSensor class and SensorConfiguration classes and operations. + +In order to control the configuration of the camera sensor through the +SensorConfiguration class, applications should understand this model and map it +to the combination of image sensor and kernel driver in use. + +The camera sensor model defined here is based on the *MIPI CCS specification*, +particularly on *Section 8.2 - Image readout* of *Chapter 8 - Video Timings*. + + +Glossary +-------- + +.. glossary:: + + Pixel array + The full grid of pixels, active and inactive ones + + Pixel array active area + The portion(s) of the pixel array that contains valid and readable pixels; + corresponds to the libcamera properties::PixelArrayActiveAreas + + Analog crop rectangle + The portion of the *pixel array active area* which is read out and passed + to further processing stages + + Subsampling + Pixel processing techniques that reduce the image size by binning or by + skipping adjacent pixels + + Digital crop + Crop of the sub-sampled image data before scaling + + Frame output + The frame (image) as output on the media bus by the camera sensor + +Camera sensor model +------------------- + +The abstract sensor model is described in the following diagram. + +.. figure:: sensor_model.svg + + +1. The sensor reads pixels from the *pixel array*. The pixels being read out are + selected by the *analog crop rectangle*. + +2. The pixels can be subsampled to reduce the image size without affecting the + field of view. Two subsampling techniques can be used: + + - Binning: combines adjacent pixels of the same colour by averaging or + summing their values, in the analog domain and/or the digital domain. + + .. figure:: binning.svg + + + - Skipping: skips the read out of a number of adjacent pixels. + + .. figure:: skipping.svg + + +3. The output of the optional sub-sampling stage is then cropped after the + conversion of the analogue pixel values in the digital domain. + +4. The resulting output frame is sent on the media bus by the sensor. + +Camera Sensor configuration parameters +-------------------------------------- + +The libcamera camera sensor model defines parameters that allow users to +control: + +1. The image format bit depth + +2. The size and position of the *Analog crop rectangle* + +3. The subsampling factors used to downscale the pixel array readout data to a + smaller frame size without reducing the image *field of view*. Two + configuration parameters are made available to control the downscaling + factor: + + - binning + A vertical and horizontal binning factor can be specified, the image + will be downscaled in its vertical and horizontal sizes by the specified + factor. + + .. code-block:: + :caption: Definition: The horizontal and vertical binning factors + + horizontal_binning = xBin; + vertical_binning = yBin; + + + - skipping + Skipping reduces the image resolution by skipping the read-out of a number + of adjacent pixels. The skipping factor is specified by the 'increment' + number (number of pixels to 'skip') in the vertical and horizontal + directions and for even and odd rows and columns. + + .. code-block:: + :caption: Definition: The horizontal and vertical skipping factors + + horizontal_skipping = (xOddInc + xEvenInc) / 2 + vertical_skipping = (yOddInc + yEvenInc) / 2 + + Different sensors perform the binning and skipping stages in different + orders. For the sake of computing the final output image size the order of + execution is not relevant. The overall down-scaling factor is obtained by + combining the binning and skipping factors. + + .. code-block:: + :caption: Definition: The total scaling factor (binning + sub-sampling) + + total_horizontal_downscale = horizontal_binning + horizontal_skipping + total_vertical_downscale = vertical_binning + vertical_skipping + + +4. The output size is used to specify any additional cropping on the sub-sampled + frame. + +5. The total line length and frame height (*visibile* pixels + *blankings*) as + sent on the MIPI CSI-2 bus. + +6. The pixel transmission rate on the MIPI CSI-2 bus. + +The above parameters are combined to obtain the following high-level +configurations: + +- **frame output size** + + Obtained by applying a crop to the physical pixel array size in the analog + domain, followed by optional binning and sub-sampling (in any order), + followed by an optional crop step in the output digital domain. + +- **frame rate** + + The combination of the *total frame size*, the image format *bit depth* and + the *pixel rate* of the data sent on the MIPI CSI-2 bus allows to compute the + image stream frame rate. The equation is the well known: + + .. code-block:: + + frame_duration = total_frame_size / pixel_rate + frame_rate = 1 / frame_duration + + + where the *pixel_rate* parameter is the result of the sensor's configuration + of the MIPI CSI-2 bus *(the following formula applies to MIPI CSI-2 when + used on MIPI D-PHY physical protocol layer only)* + + .. code-block:: + + pixel_rate = CSI-2_link_freq * 2 * nr_of_lanes / bits_per_sample diff --git a/Documentation/index.rst b/Documentation/index.rst index 43d8b017d3b4..63fac72d11ed 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -23,3 +23,4 @@ Sensor driver requirements Lens driver requirements Python Bindings + Camera Sensor Model diff --git a/Documentation/meson.build b/Documentation/meson.build index b2a5bf15e6ea..7c1502592baa 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -63,6 +63,7 @@ endif if sphinx.found() docs_sources = [ + 'camera-sensor-model.rst', 'coding-style.rst', 'conf.py', 'contributing.rst', diff --git a/Documentation/sensor_model.svg b/Documentation/sensor_model.svg new file mode 100644 index 000000000000..02dc55a8018d --- /dev/null +++ b/Documentation/sensor_model.svg @@ -0,0 +1,4870 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Pixel array + Analog crop + Subsampling + Digital crop + Media Bus + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1 + + 2 + + 4 + + 3 + + diff --git a/Documentation/skipping.svg b/Documentation/skipping.svg new file mode 100644 index 000000000000..7bef37cfcc64 --- /dev/null +++ b/Documentation/skipping.svg @@ -0,0 +1,1720 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + x_even_inc = 1 + y_even_inc = 1 + 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 + x_odd_inc = 3 + y_odd_inc = 3 + 0 + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + + + +