[v4,2/4] libcamera: Clarify meaning of PixelArraySize and other rectangles
diff mbox series

Message ID 20260918091835.2332386-3-stefan.klug@ideasonboard.com
State New
Headers show
Series
  • Clarify rectangle behavior
Related show

Commit Message

Stefan Klug Sept. 18, 2026, 9:10 a.m. UTC
Current documented behavior for PixelArraySize is to report the readable
array size of the sensor data. If the readable array is offset relative
to the physical sensor array, all related rectangles (ScalerCrop,
PixelArrayActiveAreas) shall be reported relative to PixelArraySize.

However no known pipeline and IPA implements it that way. All
pipelines/IPAs report PixelArrayActiveAreas relative to the physical
size resulting in an inconsistent behaviour where the right border of
the active area is outside of PixelArraySize.

Adjust the documentation to fit the actual implementations. Add another
property named PixelArrayReadableArea to convey the information that
PixelArraySize was intended for.

While at it, rename PixelArraySize to PixelArrayArea with the intention
to get rid of the awkward wording "... is relative to the PixelArraySize
rectangle" and to keep the door open to have a PixelArrayArea with a
negative top left corner (although that is very unlikely).

This has the added benefit that all rectangles are relative to the
physical sensor array, which helps in aligning tuning data like lens
shading correction. Using the active area to align tuning data had the
downside that the offset of the active area relative to the physical
pixel array was nowhere reported by libcamera and is a property that can
change in different versions of a sensor driver. Relying on physical
coordinates is expected to provide long-term stable absolute
coordinates.

To explain that a bit further, imagine a sensor driver was upstreamed
with a single mode with a centered FullHD crop. Now the next version of
the driver adds a centered 4:3 mode with the same width but taller as
the old mode. The active area of the sensor now changed which would
render polynomial lsc configuration from the tuning files useless as it
gets evaluated relative to the active areai. Having lsc data relative to
the physical pixel array solves this.

Another benefit is that it makes it easier to mix data coming from
libcamera with data coming from other sources like raw v4l2 tools, as
the coordinate systems match.

Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>

---

Changes in v4:
- Renamed PixelArraySize to PixelArrayArea
- Set readableArea in CameraSensorLegacy::initVimcDefaultProperties()

Changes in v3:
- Collected tag

Changes in v2:
- Fixed typos reported by Paul
- Collected tag
- Improved commit message
- Updated docs based on Jacpos review
- Added PixelArrayReadableArea to CameraSensorRaw
- Improved PixelArraySize in CameraSensorRaw but left a todo
---
 src/android/camera_capabilities.cpp           |   6 +-
 src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |   2 +-
 src/libcamera/property_ids_core.yaml          | 109 ++++++++++--------
 src/libcamera/sensor/camera_sensor_legacy.cpp |  34 ++++--
 src/libcamera/sensor/camera_sensor_raw.cpp    |  18 ++-
 5 files changed, 102 insertions(+), 67 deletions(-)

Patch
diff mbox series

diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp
index 795abc2d037c..00976f969d16 100644
--- a/src/android/camera_capabilities.cpp
+++ b/src/android/camera_capabilities.cpp
@@ -1066,9 +1066,9 @@  int CameraCapabilities::initializeStaticMetadata()
 	/* Sensor static metadata. */
 	std::array<int32_t, 2> pixelArraySize;
 	{
-		const Size &size = properties.get(properties::PixelArraySize).value_or(utils::defopt);
-		pixelArraySize[0] = size.width;
-		pixelArraySize[1] = size.height;
+		const Rectangle &area = properties.get(properties::PixelArrayArea).value_or(utils::defopt);
+		pixelArraySize[0] = area.width;
+		pixelArraySize[1] = area.height;
 		staticMetadata_->addEntry(ANDROID_SENSOR_INFO_PIXEL_ARRAY_SIZE,
 					  pixelArraySize);
 	}
diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
index 4b09bd6e2f7c..ecd6baace05f 100644
--- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
+++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
@@ -597,7 +597,7 @@  int UVCCameraData::init(std::shared_ptr<MediaDevice> media)
 
 	properties_.set(properties::Location, location);
 
-	properties_.set(properties::PixelArraySize, resolution);
+	properties_.set(properties::PixelArrayArea, Rectangle(resolution));
 	properties_.set(properties::PixelArrayActiveAreas, { Rectangle(resolution) });
 
 	/* Initialise the supported controls. */
diff --git a/src/libcamera/property_ids_core.yaml b/src/libcamera/property_ids_core.yaml
index 96c7b55d8383..06348c62db2d 100644
--- a/src/libcamera/property_ids_core.yaml
+++ b/src/libcamera/property_ids_core.yaml
@@ -424,16 +424,30 @@  controls:
         The property can be used to calculate the physical size of the sensor's
         pixel array area and for calibration purposes.
 
-  - PixelArraySize:
-      type: Size
+  - PixelArrayArea:
+      type: Rectangle
       description: |
-        The camera sensor pixel array readable area vertical and horizontal
-        sizes, in pixels.
+        The camera sensor pixel array area, in pixels.
 
-        The PixelArraySize property defines the size in pixel units of the
-        readable part of full pixel array matrix, including optical black
+        The PixelArrayArea property defines the size in pixel units of the full
+        pixel array matrix including readable and non-readable pixels. The top
+        left corner of this rectangle is always at (0,0). For the largest
+        readable area \see PixelArrayReadableArea.
+
+        All other properties that describe portions of the pixel array such as
+        PixelArrayActiveAreas and PixelArrayReadableArea are defined relative to
+        this rectangle.
+
+  - PixelArrayReadableArea:
+      type: Rectangle
+      description: |
+        The camera sensor pixel array readable area.
+
+        The PixelArrayReadableArea property defines the rectangle in pixel units
+        of the readable part of full pixel array matrix, including optical black
         pixels used for calibration, pixels which are not considered valid for
-        capture and active pixels containing valid image data.
+        capture and active pixels containing valid image data. It is defined
+        relative to the PixelArrayArea rectangle.
 
         The property describes the maximum size of the raw data captured by the
         camera, which might not correspond to the physical size of the sensor
@@ -442,23 +456,25 @@  controls:
 
         For example, let's consider a pixel array matrix assembled as follows
 
-             +--------------------------------------------------+
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
+                          PixelArrayArea.width
+             /--------------------------------------------------/
+             +--------------------------------------------------+ /
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | PixelArrayArea.height
              ...          ...           ...      ...          ...
 
              ...          ...           ...      ...          ...
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             +--------------------------------------------------+
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             +--------------------------------------------------+ /
 
         starting with two lines of non-readable pixels (x), followed by N lines
         of readable data (D) surrounded by two columns of non-readable pixels on
@@ -467,35 +483,32 @@  controls:
         sizes of the largest possible buffer of raw data that can be presented
         to applications.
 
-                             PixelArraySize.width
-               /----------------------------------------------/
-               +----------------------------------------------+ /
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | PixelArraySize.height
-               ...        ...           ...      ...        ...
-               ...        ...           ...      ...        ...
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               +----------------------------------------------+ /
+                          PixelArrayReadableArea.width
+                /--------------------------------------------/
+                PixelArrayReadableArea.x
+             +--o-----------------------------------------------+
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             oxxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| / PixelArrayReadableArea.y
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | PixelArrayReadableArea.height
+             ...          ...           ...      ...          ...
 
-        This defines a rectangle whose top-left corner is placed in position (0,
-        0) and whose vertical and horizontal sizes are defined by this property.
-        All other rectangles that describe portions of the pixel array, such as
-        the optical black pixels rectangles and active pixel areas, are defined
-        relatively to this rectangle.
+             ...          ...           ...      ...          ...
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| /
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             +--------------------------------------------------+
 
         All the coordinates are expressed relative to the default sensor readout
         direction, without any transformation (such as horizontal and vertical
         flipping) applied. When mapping them to the raw pixel buffer,
         applications shall take any configured transformation into account.
 
-        \todo Rename this property to Size once we will have property
-              categories (i.e. Properties::PixelArray::Size)
-
   - PixelArrayActiveAreas:
       type: Rectangle
       size: [n]
@@ -507,7 +520,7 @@  controls:
         This property describes an arbitrary number of overlapping rectangles,
         with each rectangle representing the maximum image size that the camera
         sensor can produce for a particular aspect ratio. They are defined
-        relatively to the PixelArraySize rectangle.
+        relative to the PixelArrayArea rectangle.
 
         When multiple rectangles are reported, they shall be ordered from the
         tallest to the shortest.
@@ -519,13 +532,13 @@  controls:
         and/or applying pixel sub-sampling techniques such as pixel skipping or
         binning.
 
-                   PixelArraySize.width
+                   PixelArrayArea.width
                     /----------------/
                       x1          x2
             (0,0)-> +-o------------o-+  /
                  y1 o +------------+ |  |
                     | |////////////| |  |
-                    | |////////////| |  | PixelArraySize.height
+                    | |////////////| |  | PixelArrayArea.height
                     | |////////////| |  |
                  y2 o +------------+ |  |
                     +----------------+  /
@@ -539,14 +552,14 @@  controls:
         resolutions will report several overlapping rectangles, one for each
         natively supported resolution.
 
-                     PixelArraySize.width
+                     PixelArrayArea.width
                     /------------------/
                       x1  x2    x3  x4
             (0,0)-> +o---o------o---o+  /
                  y1 o    +------+    |  |
                     |    |//////|    |  |
                  y2 o+---+------+---+|  |
-                    ||///|//////|///||  | PixelArraySize.height
+                    ||///|//////|///||  | PixelArrayArea.height
                  y3 o+---+------+---+|  |
                     |    |//////|    |  |
                  y4 o    +------+    |  |
diff --git a/src/libcamera/sensor/camera_sensor_legacy.cpp b/src/libcamera/sensor/camera_sensor_legacy.cpp
index 83e2c2593215..f8d05487202c 100644
--- a/src/libcamera/sensor/camera_sensor_legacy.cpp
+++ b/src/libcamera/sensor/camera_sensor_legacy.cpp
@@ -131,7 +131,8 @@  private:
 	std::vector<controls::draft::TestPatternModeEnum> testPatternModes_;
 	controls::draft::TestPatternModeEnum testPatternMode_;
 
-	Size pixelArraySize_;
+	Rectangle pixelArrayArea_;
+	Rectangle readableArea_;
 	Rectangle activeArea_;
 	const BayerFormat *bayerFormat_;
 	bool supportHFlips_;
@@ -396,33 +397,40 @@  int CameraSensorLegacy::validateSensorDriver()
 	 * \todo Make support for selection targets mandatory as soon as all
 	 * test platforms have been updated.
 	 */
-	Rectangle rect;
-	int ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_BOUNDS, &rect);
+	int ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_NATIVE_SIZE, &pixelArrayArea_);
 	if (ret) {
 		/*
 		 * Default the pixel array size to the largest size supported
 		 * by the sensor. The sizes_ vector is sorted in ascending
 		 * order, the largest size is thus the last element.
 		 */
-		pixelArraySize_ = sizes_.back();
+		pixelArrayArea_ = Rectangle(sizes_.back());
 
 		LOG(CameraSensor, Warning)
-			<< "The PixelArraySize property has been defaulted to "
-			<< pixelArraySize_;
+			<< "The PixelArrayArea property has been defaulted to "
+			<< pixelArrayArea_;
+		err = -EINVAL;
+	}
+
+	ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_BOUNDS, &readableArea_);
+	if (ret) {
+		readableArea_ = pixelArrayArea_;
+		LOG(CameraSensor, Warning)
+			<< "The PixelArrayReadableArea property has been defaulted to "
+			<< readableArea_;
 		err = -EINVAL;
-	} else {
-		pixelArraySize_ = rect.size();
 	}
 
 	ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_DEFAULT, &activeArea_);
 	if (ret) {
-		activeArea_ = Rectangle(pixelArraySize_);
+		activeArea_ = readableArea_;
 		LOG(CameraSensor, Warning)
 			<< "The PixelArrayActiveAreas property has been defaulted to "
 			<< activeArea_;
 		err = -EINVAL;
 	}
 
+	Rectangle rect;
 	ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP, &rect);
 	if (ret) {
 		LOG(CameraSensor, Warning)
@@ -476,8 +484,9 @@  int CameraSensorLegacy::validateSensorDriver()
 void CameraSensorLegacy::initVimcDefaultProperties()
 {
 	/* Use the largest supported size. */
-	pixelArraySize_ = sizes_.back();
-	activeArea_ = Rectangle(pixelArraySize_);
+	pixelArrayArea_ = Rectangle(sizes_.back());
+	readableArea_ = pixelArrayArea_;
+	activeArea_ = pixelArrayArea_;
 }
 
 void CameraSensorLegacy::initStaticProperties()
@@ -627,7 +636,8 @@  int CameraSensorLegacy::initProperties()
 		mountingOrientation_ = Orientation::Rotate0;
 	}
 
-	properties_.set(properties::PixelArraySize, pixelArraySize_);
+	properties_.set(properties::PixelArrayArea, pixelArrayArea_);
+	properties_.set(properties::PixelArrayReadableArea, readableArea_);
 	properties_.set(properties::PixelArrayActiveAreas, { activeArea_ });
 
 	/* Color filter array pattern, register only for RAW sensors. */
diff --git a/src/libcamera/sensor/camera_sensor_raw.cpp b/src/libcamera/sensor/camera_sensor_raw.cpp
index e50111f1d5c7..117da63ef364 100644
--- a/src/libcamera/sensor/camera_sensor_raw.cpp
+++ b/src/libcamera/sensor/camera_sensor_raw.cpp
@@ -141,7 +141,8 @@  private:
 	std::vector<controls::draft::TestPatternModeEnum> testPatternModes_;
 	controls::draft::TestPatternModeEnum testPatternMode_;
 
-	Size pixelArraySize_;
+	Rectangle pixelArrayArea_;
+	Rectangle readableArea_;
 	Rectangle activeArea_;
 	BayerFormat::Order cfaPattern_;
 	bool supportHFlips_;
@@ -431,7 +432,17 @@  std::optional<int> CameraSensorRaw::init()
 		return { ret };
 	}
 
-	pixelArraySize_ = rect.size();
+	/*
+	 * \todo Implement querying the physical sensor size based on pad 1,
+	 * stream 0. See
+	 * https://lore.kernel.org/linux-media/20260409201501.975242-23-sakari.ailus@linux.intel.com/
+	 */
+	LOG(CameraSensor, Warning) << "PixelArrayArea might be incorrect";
+	pixelArrayArea_ = Rectangle(0, 0,
+				    rect.x + rect.width,
+				    rect.y + rect.height);
+
+	readableArea_ = rect;
 
 	ret = subdev_->getSelection(streams_.image.sink, V4L2_SEL_TGT_CROP_DEFAULT,
 				    &activeArea_);
@@ -634,7 +645,8 @@  int CameraSensorRaw::initProperties()
 		mountingOrientation_ = Orientation::Rotate0;
 	}
 
-	properties_.set(properties::PixelArraySize, pixelArraySize_);
+	properties_.set(properties::PixelArrayArea, pixelArrayArea_);
+	properties_.set(properties::PixelArrayReadableArea, readableArea_);
 	properties_.set(properties::PixelArrayActiveAreas, { activeArea_ });
 
 	/* Color filter array pattern. */