From patchwork Sat Sep 16 12:19:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 19051 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D0496BE080 for ; Sat, 16 Sep 2023 13:01:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0D91162918; Sat, 16 Sep 2023 15:01:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1694869288; bh=5+z2tMp896rjc5PopxTdc83skhSb2ghYuCeTUPiouVI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=iHaIs7Ht1SwQfqhoaKIF7qlsCg8deuPSLf2CxBWn31KLC3rgHQNI5rzP9bjeVh766 IZ/4WWSAyjz4ZgiGTFps4FsX60oQeRLw/7VRgLCNeorNQCLIwYIA/dq+La0J+xAFuh zT3bS/CKIvRnvFu2ho5lPvOxlFpZKzJtWKm6kB2DDB19WhMxohZeBAo+mdJuKC1Hyf d3ZgtZqV/p2vTuU85cn2C0zE/JuBcTzoZZQBoVsT0wnHqY+VbMm8IJU4Gb1AI6LpQR 3xWIKPBN+LAp8WjbchR2D400AWq48+9SkhhApX0xcKFZIiC2cbVLsEeVQNVtRQV+eQ YFr9cBs1NnHOw== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C26C5628E9 for ; Sat, 16 Sep 2023 14:19:37 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pbp3L4HW"; dkim-atps=neutral Received: from uno.LocalDomain (93-61-96-190.ip145.fastwebnet.it [93.61.96.190]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1841C1536; Sat, 16 Sep 2023 14:18:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1694866683; bh=5+z2tMp896rjc5PopxTdc83skhSb2ghYuCeTUPiouVI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pbp3L4HWLOpFM8c4WRRzXiq+kohjhyE7UvpxHlcLUjma14ORIHcVx/X+G83Jh/FKm NeUP9kBGhZ+kW9squVGvaoObbYYaVxdAaQf4M7ZssPuVKkfFvTlsC/V8+uPXY56ieu 0l6smKfHud5WjKcKCM5zSptRTLOK/FgRrb9dhW7k= To: libcamera-devel@lists.libcamera.org Date: Sat, 16 Sep 2023 14:19:19 +0200 Message-ID: <20230916121930.29490-2-jacopo.mondi@ideasonboard.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20230916121930.29490-1-jacopo.mondi@ideasonboard.com> References: <20230916121930.29490-1-jacopo.mondi@ideasonboard.com> MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 16 Sep 2023 15:01:26 +0200 Subject: [libcamera-devel] [PATCH v4 01/12] documentation: Introduce Camera Sensor Model X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Cc: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Introduce a documentation page about the 'camera sensor model' implemented by libcamera. The camera sensor model serves to provide to applications a reference description of the processing steps that take place in a camera sensor in order to precisely control the sensor configuration through the forthcoming SensorConfiguration class. Signed-off-by: Jacopo Mondi Reviewed-by: Naushir Patuck Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- Documentation/binning.svg | 5053 +++++++++++++++++++++++++ Documentation/camera-sensor-model.rst | 170 + Documentation/index.rst | 1 + Documentation/meson.build | 1 + Documentation/sensor_model.svg | 4866 ++++++++++++++++++++++++ Documentation/skipping.svg | 1720 +++++++++ 6 files changed, 11811 insertions(+) create mode 100644 Documentation/binning.svg create mode 100644 Documentation/camera-sensor-model.rst create mode 100644 Documentation/sensor_model.svg create mode 100644 Documentation/skipping.svg diff --git a/Documentation/binning.svg b/Documentation/binning.svg new file mode 100644 index 000000000000..c6a3b6394acd --- /dev/null +++ b/Documentation/binning.svg @@ -0,0 +1,5053 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/camera-sensor-model.rst b/Documentation/camera-sensor-model.rst new file mode 100644 index 000000000000..7ff0bdc50564 --- /dev/null +++ b/Documentation/camera-sensor-model.rst @@ -0,0 +1,170 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +.. _camera-sensor-model: + +The libcamera camera sensor model +================================= + +libcamera defines an abstract camera sensor model in order to provide +a description of each of the processing steps that result in image data being +sent on the media bus and that form the image stream delivered to applications. + +Applications should use the abstracted camera sensor model defined here to +precisely control the operations of the camera sensor. + +The libcamera camera sensor model targets image sensors producing frames in +RAW format, delivered through a MIPI CSI-2 compliant bus implementation. + +The abstract sensor model maps libcamera components to the characteristics and +operations of an image sensor, and serves as a reference to model the libcamera +CameraSensor class and SensorConfiguration classes and operations. + +In order to control the configuration of the camera sensor through the +SensorConfiguration class, applications should understand this model and map it +to the combination of image sensor and kernel driver in use. + +The camera sensor model defined here is based on the *MIPI CCS specification*, +particularly on *Section 8.2 - Image readout* of *Chapter 8 - Video Timings*. + +Glossary +--------- + +- *Pixel array*: The full grid of pixels, active and inactive ones + +- *Pixel array active area*: The portion(s) of the pixel array that + contains valid and readable pixels; corresponds to the libcamera + properties::PixelArrayActiveAreas + +- *Analog crop rectangle*: The portion of the *pixel array active area* which + is read out and passed to further processing stages + +- *Subsampling*: Pixel processing techniques that reduce the image size by + binning or by skipping adjacent pixels + +- *Digital crop*: Crop of the sub-sampled image data before scaling + +- *Frame output*: The frame (image) as output on the media bus by the + camera sensor + +Camera sensor model +------------------- + +The abstract sensor model is described in the following diagram + +.. figure:: sensor_model.svg + + +1. The sensor reads pixels from the *pixel array*. The pixels being read out are + selected by the *analog crop rectangle*. + +2. The pixels can be subsampled to reduce the image size without affecting the + field of view. Two subsampling techniques can be used: + + - Binning: combines adjacent pixels of the same colour by averaging or + summing their values, in the analog domain and/or the digital domain. + + .. figure:: binning.svg + + + - Skipping: skips the read out of a number of adjacent pixels. + + .. figure:: skipping.svg + + +3. The output of the optional sub-sampling stage is then cropped after the +conversion of the analogue pixel values in the digital domain. + +4. The resulting output frame is sent on the media bus by the sensor. + +Camera Sensor configuration parameters +-------------------------------------- + +The libcamera camera sensor model defines parameters that allow users to +control: + +1. The image format bit depth + +2. The size and position of the *Analog crop rectangle* + +3. The subsampling factors used to downscale the pixel array readout data to a + smaller frame size without reducing the image *field of view*. Two + configuration parameters are made available to control the downscaling factor + + - binning + - a vertical and horizontal binning factor can be specified, the image + will be downscaled in its vertical and horizontal sizes by the specified + factor + + .. code-block:: + :caption: Definition: The horizontal and vertical binning factors + + horizontal_binning = xBin; + vertical_binning = yBin; + + + - skipping + - reduces the image resolution by skipping the read-out of a + number of adjacent pixels + - the skipping factor is specified by the 'increment' number (number of + pixels to 'skip') in the vertical and horizontal directions and for + even and odd rows and columns + + .. code-block:: + :caption: Definition: The horizontal and vertical skipping factors + + horizontal_skipping = (xOddInc + xEvenInc) / 2 + vertical_skipping = (yOddInc + yEvenInc) / 2 + + + - different sensors perform the binning and skipping stages in different + orders + - for the sake of computing the final output image size the order of + execution is not relevant. + + - the overall down-scaling factor is obtained by combining the binning and + skipping factors + + .. code-block:: + :caption: Definition: The total scaling factor (binning + sub-sampling) + + total_horizontal_downscale = horizontal_binning + horizontal_skipping + total_vertical_downscale = vertical_binning + vertical_skipping + + +4. The output data frame size + - the output size is used to specify any additional cropping on the + sub-sampled frame + +5. The total line length and frame height (*visibile* pixels + *blankings*) as + sent on the MIPI CSI-2 bus + +6. The pixel transmission rate on the MIPI CSI-2 bus + +The above parameters are combined to obtain the following high-level +configurations: + +- **frame output size** + + Obtained by applying a crop to the physical pixel array size in the analog + domain, followed by optional binning and sub-sampling (in any order), + followed by an optional crop step in the output digital domain. + +- **frame rate** + + The combination of the *total frame size*, the image format *bit depth* and + the *pixel rate* of the data sent on the MIPI CSI-2 bus allows to compute the + image stream frame rate. The equation is the well known: + + .. code-block:: + + frame_duration = total_frame_size / pixel_rate + frame_rate = 1 / frame_duration + + + where the *pixel_rate* parameter is the result of the sensor's configuration + of the MIPI CSI-2 bus *(the following formula applies to MIPI CSI-2 when + used on MIPI D-PHY physical protocol layer only)* + + .. code-block:: + + pixel_rate = CSI-2_link_freq * 2 * nr_of_lanes / bits_per_sample diff --git a/Documentation/index.rst b/Documentation/index.rst index 43d8b017d3b4..63fac72d11ed 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -23,3 +23,4 @@ Sensor driver requirements Lens driver requirements Python Bindings + Camera Sensor Model diff --git a/Documentation/meson.build b/Documentation/meson.build index b2a5bf15e6ea..7c1502592baa 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -63,6 +63,7 @@ endif if sphinx.found() docs_sources = [ + 'camera-sensor-model.rst', 'coding-style.rst', 'conf.py', 'contributing.rst', diff --git a/Documentation/sensor_model.svg b/Documentation/sensor_model.svg new file mode 100644 index 000000000000..1f76d41cf6a5 --- /dev/null +++ b/Documentation/sensor_model.svg @@ -0,0 +1,4866 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Pixel array + Analog crop + Subsampling + Digital crop + Media Bus + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1 + + 2 + + 4 + + 3 + + diff --git a/Documentation/skipping.svg b/Documentation/skipping.svg new file mode 100644 index 000000000000..7bef37cfcc64 --- /dev/null +++ b/Documentation/skipping.svg @@ -0,0 +1,1720 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + x_even_inc = 1 + y_even_inc = 1 + 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 + x_odd_inc = 3 + y_odd_inc = 3 + 0 + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + + + +