[v1,1/2] libcamera: Clarify meaning of PixelArraySize and other rectangles
diff mbox series

Message ID 20260616120828.375547-2-stefan.klug@ideasonboard.com
State New
Headers show
Series
  • Clarify rectangle behavior
Related show

Commit Message

Stefan Klug June 16, 2026, 12:08 p.m. UTC 
Current documented behavior for PixelArraySize is to report the readable
array size of the sensor data. If the readable array is offset relative
to the physical sensor array, all related rectangles (ScalerCrop,
PixelArrayActiveAreas) shall be reported relative to PixelArraySize.

However no known pipeline and IPA implements it that way. All
pipelines/IPAs treat PixelArraySize as the physical sensor size and
handle the rectangles relative to that size.

Adjust the documentation to fit the actual implementations. Add another
property named PixelArrayReadableArea to convey the information that
PixelArraySize was intended for.

This has the added benefit that all rectangles are relative to the
physical sensor array, which is necessary to align tuning data like lens
shading correction.

Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>

---

ToDo:
- Implement the expected behavior in CameraSensorRaw
---
 include/libcamera/ipa/core.mojom              |   2 +-
 src/libcamera/property_ids_core.yaml          | 116 ++++++++++--------
 src/libcamera/sensor/camera_sensor_legacy.cpp |  26 ++--
 src/libcamera/sensor/camera_sensor_raw.cpp    |  16 +--
 4 files changed, 89 insertions(+), 71 deletions(-)

Patch
diff mbox series 

diff --git a/include/libcamera/ipa/core.mojom b/include/libcamera/ipa/core.mojom
index bce797245829..5da780f5d256 100644
--- a/include/libcamera/ipa/core.mojom
+++ b/include/libcamera/ipa/core.mojom
@@ -154,7 +154,7 @@  module libcamera;
 
 /**
  * \var IPACameraSensorInfo::activeAreaSize
- * \brief The size of the pixel array active area of the sensor
+ * \brief The size of the physical pixel array of the sensor in pixels
  */
 
 /**
diff --git a/src/libcamera/property_ids_core.yaml b/src/libcamera/property_ids_core.yaml
index d5b3d309c055..96c85f9233fd 100644
--- a/src/libcamera/property_ids_core.yaml
+++ b/src/libcamera/property_ids_core.yaml
@@ -427,11 +427,29 @@  controls:
   - PixelArraySize:
       type: Size
       description: |
-        The camera sensor pixel array readable area vertical and horizontal
-        sizes, in pixels.
+        The camera sensor pixel array vertical and horizontal sizes, in pixels.
 
-        The PixelArraySize property defines the size in pixel units of the
-        readable part of full pixel array matrix, including optical black
+        The PixelArraySize property defines the size in pixel units of the full
+        pixel array matrix. It is often not be possible to capture the whole
+        physical pixel array. For the largest readable area \see
+        PixelArrayReadableArea.
+
+        This defines a rectangle whose top-left corner is placed in position (0,
+        0) and whose vertical and horizontal sizes are defined by this property.
+        All other rectangles that describe portions of the pixel array, such as
+        the optical black pixels rectangles and active pixel areas, are defined
+        relatively to this rectangle.
+
+        \todo Rename this property to Size once we will have property
+              categories (i.e. Properties::PixelArray::Size)
+
+  - PixelArrayReadableArea:
+      type: Rectangle
+      description: |
+        The camera sensor pixel array readable area.
+
+        The PixelArrayReadableArea property defines the rectangle in pixel units
+        of the readable part of full pixel array matrix, including optical black
         pixels used for calibration, pixels which are not considered valid for
         capture and active pixels containing valid image data.
 
@@ -442,23 +460,25 @@  controls:
 
         For example, let's consider a pixel array matrix assembled as follows
 
-             +--------------------------------------------------+
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
+                          PixelArraySize.width
+             /--------------------------------------------------/
+             +--------------------------------------------------+ /
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | PixelArraySize.height
              ...          ...           ...      ...          ...
 
              ...          ...           ...      ...          ...
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx|
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
-             +--------------------------------------------------+
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+             +--------------------------------------------------+ /
 
         starting with two lines of non-readable pixels (x), followed by N lines
         of readable data (D) surrounded by two columns of non-readable pixels on
@@ -467,35 +487,32 @@  controls:
         sizes of the largest possible buffer of raw data that can be presented
         to applications.
 
-                             PixelArraySize.width
-               /----------------------------------------------/
-               +----------------------------------------------+ /
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | PixelArraySize.height
-               ...        ...           ...      ...        ...
-               ...        ...           ...      ...        ...
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| |
-               +----------------------------------------------+ /
+                          PixelArrayReadableArea.width
+                /--------------------------------------------/
+                PixelArrayReadableArea.x
+             +--o-----------------------------------------------+
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             oxxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| / PixelArrayReadableArea.y
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | PixelArrayReadableArea.height
+             ...          ...           ...      ...          ...
 
-        This defines a rectangle whose top-left corner is placed in position (0,
-        0) and whose vertical and horizontal sizes are defined by this property.
-        All other rectangles that describe portions of the pixel array, such as
-        the optical black pixels rectangles and active pixel areas, are defined
-        relatively to this rectangle.
+             ...          ...           ...      ...          ...
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| |
+             |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| /
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
+             +--------------------------------------------------+
 
         All the coordinates are expressed relative to the default sensor readout
         direction, without any transformation (such as horizontal and vertical
         flipping) applied. When mapping them to the raw pixel buffer,
         applications shall take any configured transformation into account.
 
-        \todo Rename this property to Size once we will have property
-              categories (i.e. Properties::PixelArray::Size)
-
   - PixelArrayOpticalBlackRectangles:
       type: Rectangle
       size: [n]
@@ -519,7 +536,7 @@  controls:
         they could be positioned too close to the chip margins or to the optical
         black shielding placed on top of optical black pixels.
 
-                             PixelArraySize.width
+                             PixelArrayReadableArea.width
                /----------------------------------------------/
                   x1                                       x2
                +--o---------------------------------------o---+ /
@@ -532,7 +549,7 @@  controls:
                |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII| |
                |IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII| |
             y3 |IIOOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOOII| |
-               |IIOOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOOII| | PixelArraySize.height
+               |IIOOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOOII| | PixelArrayReadableArea.height
                |IIOOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOOII| |
                ...          ...           ...     ...       ...
                ...          ...           ...     ...       ...
@@ -560,7 +577,7 @@  controls:
         skips the invalid lines and columns, producing the following data
         buffer, when captured to memory
 
-                             PixelArraySize.width
+                             PixelArrayReadableArea.width
                /----------------------------------------------/
                                                            x1
                +--------------------------------------------o-+ /
@@ -570,7 +587,7 @@  controls:
                |OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO| |
             y1 oOOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOO| |
                |OOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOO| |
-               |OOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOO| | PixelArraySize.height
+               |OOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOO| | PixelArrayReadableArea.height
                ...       ...          ...       ...         ... |
                ...       ...          ...       ...         ... |
                |OOPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPOO| |
@@ -578,14 +595,17 @@  controls:
                +----------------------------------------------+ /
 
         then the invalid lines and columns should not be reported as part of the
-        PixelArraySize property in first place.
+        PixelArrayReadableArea property in first place. As the coordinates of
+        PixelArrayReadableArea are relative to the physical pixel array, care
+        must be taken to report it in a way that the valid pixel data maps to
+        the correct physical position.
 
         In this case, the position of the black pixel rectangles will be
 
             PixelArrayOpticalBlackRectangles = {
-               { 0, 0, y1 + 1, PixelArraySize[0] },
-               { 0, y1, 2, PixelArraySize[1] - y1 + 1 },
-               { x1, y1, 2, PixelArraySize[1] - y1 + 1 },
+               { 0, 0, PixelArrayReadableArea.width, y1 + 1 },
+               { 0, y1, 2, PixelArrayReadableArea.height - y1 + 1 },
+               { x1, y1, 2, PixelArrayReadableArea.height - y1 + 1 },
             };
 
         \todo Rename this property to Size once we will have property
diff --git a/src/libcamera/sensor/camera_sensor_legacy.cpp b/src/libcamera/sensor/camera_sensor_legacy.cpp
index 6a683821f219..482e387d3b24 100644
--- a/src/libcamera/sensor/camera_sensor_legacy.cpp
+++ b/src/libcamera/sensor/camera_sensor_legacy.cpp
@@ -38,6 +38,7 @@ 
 #include "libcamera/internal/media_device.h"
 #include "libcamera/internal/sysfs.h"
 #include "libcamera/internal/v4l2_subdevice.h"
+#include "linux/v4l2-common.h"
 
 namespace libcamera {
 
@@ -131,6 +132,7 @@  private:
 	controls::draft::TestPatternModeEnum testPatternMode_;
 
 	Size pixelArraySize_;
+	Rectangle readableArea_;
 	Rectangle activeArea_;
 	const BayerFormat *bayerFormat_;
 	bool supportFlips_;
@@ -390,7 +392,7 @@  int CameraSensorLegacy::validateSensorDriver()
 	 * test platforms have been updated.
 	 */
 	Rectangle rect;
-	int ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_BOUNDS, &rect);
+	int ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_NATIVE_SIZE, &rect);
 	if (ret) {
 		/*
 		 * Default the pixel array size to the largest size supported
@@ -407,9 +409,18 @@  int CameraSensorLegacy::validateSensorDriver()
 		pixelArraySize_ = rect.size();
 	}
 
+	ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_BOUNDS, &readableArea_);
+	if (ret) {
+		readableArea_ = Rectangle(pixelArraySize_);
+		LOG(CameraSensor, Warning)
+			<< "The PixelArrayReadableArea property has been defaulted to "
+			<< readableArea_;
+		err = -EINVAL;
+	}
+
 	ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_DEFAULT, &activeArea_);
 	if (ret) {
-		activeArea_ = Rectangle(pixelArraySize_);
+		activeArea_ = readableArea_;
 		LOG(CameraSensor, Warning)
 			<< "The PixelArrayActiveAreas property has been defaulted to "
 			<< activeArea_;
@@ -621,6 +632,7 @@  int CameraSensorLegacy::initProperties()
 	}
 
 	properties_.set(properties::PixelArraySize, pixelArraySize_);
+	properties_.set(properties::PixelArrayReadableArea, readableArea_);
 	properties_.set(properties::PixelArrayActiveAreas, { activeArea_ });
 
 	/* Color filter array pattern, register only for RAW sensors. */
@@ -882,16 +894,6 @@  int CameraSensorLegacy::sensorInfo(IPACameraSensorInfo *info) const
 			<< "The analogue crop rectangle has been defaulted to the active area size";
 	}
 
-	/*
-	 * IPACameraSensorInfo::analogCrop::x and IPACameraSensorInfo::analogCrop::y
-	 * are defined relatively to the active pixel area, while V4L2's
-	 * TGT_CROP target is defined in respect to the full pixel array.
-	 *
-	 * Compensate it by subtracting the active area offset.
-	 */
-	info->analogCrop.x -= activeArea_.x;
-	info->analogCrop.y -= activeArea_.y;
-
 	/* The bit depth and image size depend on the currently applied format. */
 	V4L2SubdeviceFormat format{};
 	ret = subdev_->getFormat(pad_, &format);
diff --git a/src/libcamera/sensor/camera_sensor_raw.cpp b/src/libcamera/sensor/camera_sensor_raw.cpp
index 10eba0331fe8..410edeb68983 100644
--- a/src/libcamera/sensor/camera_sensor_raw.cpp
+++ b/src/libcamera/sensor/camera_sensor_raw.cpp
@@ -416,6 +416,12 @@  std::optional<int> CameraSensorRaw::init()
 	auto last = std::unique(sizes_.begin(), sizes_.end());
 	sizes_.erase(last, sizes_.end());
 
+	/*
+	 * \todo Implement querying the physical sensor size based on pad 1,
+	 * stream 0. See
+	 * https://lore.kernel.org/linux-media/20260409201501.975242-23-sakari.ailus@linux.intel.com/
+	 */
+
 	/*
 	 * 3. Query selection rectangles. Retrieve properties, and verify that
 	 * all the expected selection rectangles are supported.
@@ -999,16 +1005,6 @@  int CameraSensorRaw::sensorInfo(IPACameraSensorInfo *info) const
 	if (ret)
 		return ret;
 
-	/*
-	 * IPACameraSensorInfo::analogCrop::x and IPACameraSensorInfo::analogCrop::y
-	 * are defined relatively to the active pixel area, while V4L2's
-	 * TGT_CROP target is defined in respect to the full pixel array.
-	 *
-	 * Compensate it by subtracting the active area offset.
-	 */
-	info->analogCrop.x -= activeArea_.x;
-	info->analogCrop.y -= activeArea_.y;
-
 	/* The bit depth and image size depend on the currently applied format. */
 	V4L2SubdeviceFormat format{};
 	ret = subdev_->getFormat(streams_.image.source, &format);