From patchwork Fri Sep 18 09:10:49 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28335 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8DE5EBE175 for ; Fri, 18 Sep 2026 09:19:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F0D616871F; Fri, 18 Sep 2026 11:19:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="W4eMk15R"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5439E6849B for ; Fri, 18 Sep 2026 11:19:00 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:6c1f:355d:1c19:aba6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 83DE94DC; Fri, 18 Sep 2026 11:17:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789723037; bh=6G1FHSgmDiGK6yTjn55rMwVIgyPxvQcr2e55jUtdClY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W4eMk15RethUl7p6D/9RI9Etqz+kAHWQwsJhAzRjWuBZ+KNUPMMv2K9hK68duVZVn zdd6S0lWJz+wE3vLj68eiE2xWiEX5oSBk5/dXQkAzByzE4uREx8YzO0KoXClnwBqlB BjVhYe1m8MWJNvRoixwX1QKfLldGlaF0n91HhjkM= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder , Jacopo Mondi Subject: [PATCH v4 2/4] libcamera: Clarify meaning of PixelArraySize and other rectangles Date: Fri, 18 Sep 2026 11:10:49 +0200 Message-ID: <20260918091835.2332386-3-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260918091835.2332386-1-stefan.klug@ideasonboard.com> References: <20260918091835.2332386-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Current documented behavior for PixelArraySize is to report the readable array size of the sensor data. If the readable array is offset relative to the physical sensor array, all related rectangles (ScalerCrop, PixelArrayActiveAreas) shall be reported relative to PixelArraySize. However no known pipeline and IPA implements it that way. All pipelines/IPAs report PixelArrayActiveAreas relative to the physical size resulting in an inconsistent behaviour where the right border of the active area is outside of PixelArraySize. Adjust the documentation to fit the actual implementations. Add another property named PixelArrayReadableArea to convey the information that PixelArraySize was intended for. While at it, rename PixelArraySize to PixelArrayArea with the intention to get rid of the awkward wording "... is relative to the PixelArraySize rectangle" and to keep the door open to have a PixelArrayArea with a negative top left corner (although that is very unlikely). This has the added benefit that all rectangles are relative to the physical sensor array, which helps in aligning tuning data like lens shading correction. Using the active area to align tuning data had the downside that the offset of the active area relative to the physical pixel array was nowhere reported by libcamera and is a property that can change in different versions of a sensor driver. Relying on physical coordinates is expected to provide long-term stable absolute coordinates. To explain that a bit further, imagine a sensor driver was upstreamed with a single mode with a centered FullHD crop. Now the next version of the driver adds a centered 4:3 mode with the same width but taller as the old mode. The active area of the sensor now changed which would render polynomial lsc configuration from the tuning files useless as it gets evaluated relative to the active areai. Having lsc data relative to the physical pixel array solves this. Another benefit is that it makes it easier to mix data coming from libcamera with data coming from other sources like raw v4l2 tools, as the coordinate systems match. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Jacopo Mondi --- Changes in v4: - Renamed PixelArraySize to PixelArrayArea - Set readableArea in CameraSensorLegacy::initVimcDefaultProperties() Changes in v3: - Collected tag Changes in v2: - Fixed typos reported by Paul - Collected tag - Improved commit message - Updated docs based on Jacpos review - Added PixelArrayReadableArea to CameraSensorRaw - Improved PixelArraySize in CameraSensorRaw but left a todo --- src/android/camera_capabilities.cpp | 6 +- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 2 +- src/libcamera/property_ids_core.yaml | 109 ++++++++++-------- src/libcamera/sensor/camera_sensor_legacy.cpp | 34 ++++-- src/libcamera/sensor/camera_sensor_raw.cpp | 18 ++- 5 files changed, 102 insertions(+), 67 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 795abc2d037c..00976f969d16 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1066,9 +1066,9 @@ int CameraCapabilities::initializeStaticMetadata() /* Sensor static metadata. */ std::array pixelArraySize; { - const Size &size = properties.get(properties::PixelArraySize).value_or(utils::defopt); - pixelArraySize[0] = size.width; - pixelArraySize[1] = size.height; + const Rectangle &area = properties.get(properties::PixelArrayArea).value_or(utils::defopt); + pixelArraySize[0] = area.width; + pixelArraySize[1] = area.height; staticMetadata_->addEntry(ANDROID_SENSOR_INFO_PIXEL_ARRAY_SIZE, pixelArraySize); } diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 4b09bd6e2f7c..ecd6baace05f 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -597,7 +597,7 @@ int UVCCameraData::init(std::shared_ptr media) properties_.set(properties::Location, location); - properties_.set(properties::PixelArraySize, resolution); + properties_.set(properties::PixelArrayArea, Rectangle(resolution)); properties_.set(properties::PixelArrayActiveAreas, { Rectangle(resolution) }); /* Initialise the supported controls. */ diff --git a/src/libcamera/property_ids_core.yaml b/src/libcamera/property_ids_core.yaml index 96c7b55d8383..06348c62db2d 100644 --- a/src/libcamera/property_ids_core.yaml +++ b/src/libcamera/property_ids_core.yaml @@ -424,16 +424,30 @@ controls: The property can be used to calculate the physical size of the sensor's pixel array area and for calibration purposes. - - PixelArraySize: - type: Size + - PixelArrayArea: + type: Rectangle description: | - The camera sensor pixel array readable area vertical and horizontal - sizes, in pixels. + The camera sensor pixel array area, in pixels. - The PixelArraySize property defines the size in pixel units of the - readable part of full pixel array matrix, including optical black + The PixelArrayArea property defines the size in pixel units of the full + pixel array matrix including readable and non-readable pixels. The top + left corner of this rectangle is always at (0,0). For the largest + readable area \see PixelArrayReadableArea. + + All other properties that describe portions of the pixel array such as + PixelArrayActiveAreas and PixelArrayReadableArea are defined relative to + this rectangle. + + - PixelArrayReadableArea: + type: Rectangle + description: | + The camera sensor pixel array readable area. + + The PixelArrayReadableArea property defines the rectangle in pixel units + of the readable part of full pixel array matrix, including optical black pixels used for calibration, pixels which are not considered valid for - capture and active pixels containing valid image data. + capture and active pixels containing valid image data. It is defined + relative to the PixelArrayArea rectangle. The property describes the maximum size of the raw data captured by the camera, which might not correspond to the physical size of the sensor @@ -442,23 +456,25 @@ controls: For example, let's consider a pixel array matrix assembled as follows - +--------------------------------------------------+ - |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| + PixelArrayArea.width + /--------------------------------------------------/ + +--------------------------------------------------+ / + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | PixelArrayArea.height ... ... ... ... ... ... ... ... ... ... - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| - |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| - |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| - +--------------------------------------------------+ + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | + +--------------------------------------------------+ / starting with two lines of non-readable pixels (x), followed by N lines of readable data (D) surrounded by two columns of non-readable pixels on @@ -467,35 +483,32 @@ controls: sizes of the largest possible buffer of raw data that can be presented to applications. - PixelArraySize.width - /----------------------------------------------/ - +----------------------------------------------+ / - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | PixelArraySize.height - ... ... ... ... ... - ... ... ... ... ... - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | - |DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD| | - +----------------------------------------------+ / + PixelArrayReadableArea.width + /--------------------------------------------/ + PixelArrayReadableArea.x + +--o-----------------------------------------------+ + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| + oxxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| / PixelArrayReadableArea.y + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | PixelArrayReadableArea.height + ... ... ... ... ... - This defines a rectangle whose top-left corner is placed in position (0, - 0) and whose vertical and horizontal sizes are defined by this property. - All other rectangles that describe portions of the pixel array, such as - the optical black pixels rectangles and active pixel areas, are defined - relatively to this rectangle. + ... ... ... ... ... + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| | + |xxDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDxx| / + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| + +--------------------------------------------------+ All the coordinates are expressed relative to the default sensor readout direction, without any transformation (such as horizontal and vertical flipping) applied. When mapping them to the raw pixel buffer, applications shall take any configured transformation into account. - \todo Rename this property to Size once we will have property - categories (i.e. Properties::PixelArray::Size) - - PixelArrayActiveAreas: type: Rectangle size: [n] @@ -507,7 +520,7 @@ controls: This property describes an arbitrary number of overlapping rectangles, with each rectangle representing the maximum image size that the camera sensor can produce for a particular aspect ratio. They are defined - relatively to the PixelArraySize rectangle. + relative to the PixelArrayArea rectangle. When multiple rectangles are reported, they shall be ordered from the tallest to the shortest. @@ -519,13 +532,13 @@ controls: and/or applying pixel sub-sampling techniques such as pixel skipping or binning. - PixelArraySize.width + PixelArrayArea.width /----------------/ x1 x2 (0,0)-> +-o------------o-+ / y1 o +------------+ | | | |////////////| | | - | |////////////| | | PixelArraySize.height + | |////////////| | | PixelArrayArea.height | |////////////| | | y2 o +------------+ | | +----------------+ / @@ -539,14 +552,14 @@ controls: resolutions will report several overlapping rectangles, one for each natively supported resolution. - PixelArraySize.width + PixelArrayArea.width /------------------/ x1 x2 x3 x4 (0,0)-> +o---o------o---o+ / y1 o +------+ | | | |//////| | | y2 o+---+------+---+| | - ||///|//////|///|| | PixelArraySize.height + ||///|//////|///|| | PixelArrayArea.height y3 o+---+------+---+| | | |//////| | | y4 o +------+ | | diff --git a/src/libcamera/sensor/camera_sensor_legacy.cpp b/src/libcamera/sensor/camera_sensor_legacy.cpp index 83e2c2593215..f8d05487202c 100644 --- a/src/libcamera/sensor/camera_sensor_legacy.cpp +++ b/src/libcamera/sensor/camera_sensor_legacy.cpp @@ -131,7 +131,8 @@ private: std::vector testPatternModes_; controls::draft::TestPatternModeEnum testPatternMode_; - Size pixelArraySize_; + Rectangle pixelArrayArea_; + Rectangle readableArea_; Rectangle activeArea_; const BayerFormat *bayerFormat_; bool supportHFlips_; @@ -396,33 +397,40 @@ int CameraSensorLegacy::validateSensorDriver() * \todo Make support for selection targets mandatory as soon as all * test platforms have been updated. */ - Rectangle rect; - int ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_BOUNDS, &rect); + int ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_NATIVE_SIZE, &pixelArrayArea_); if (ret) { /* * Default the pixel array size to the largest size supported * by the sensor. The sizes_ vector is sorted in ascending * order, the largest size is thus the last element. */ - pixelArraySize_ = sizes_.back(); + pixelArrayArea_ = Rectangle(sizes_.back()); LOG(CameraSensor, Warning) - << "The PixelArraySize property has been defaulted to " - << pixelArraySize_; + << "The PixelArrayArea property has been defaulted to " + << pixelArrayArea_; + err = -EINVAL; + } + + ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_BOUNDS, &readableArea_); + if (ret) { + readableArea_ = pixelArrayArea_; + LOG(CameraSensor, Warning) + << "The PixelArrayReadableArea property has been defaulted to " + << readableArea_; err = -EINVAL; - } else { - pixelArraySize_ = rect.size(); } ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP_DEFAULT, &activeArea_); if (ret) { - activeArea_ = Rectangle(pixelArraySize_); + activeArea_ = readableArea_; LOG(CameraSensor, Warning) << "The PixelArrayActiveAreas property has been defaulted to " << activeArea_; err = -EINVAL; } + Rectangle rect; ret = subdev_->getSelection(pad_, V4L2_SEL_TGT_CROP, &rect); if (ret) { LOG(CameraSensor, Warning) @@ -476,8 +484,9 @@ int CameraSensorLegacy::validateSensorDriver() void CameraSensorLegacy::initVimcDefaultProperties() { /* Use the largest supported size. */ - pixelArraySize_ = sizes_.back(); - activeArea_ = Rectangle(pixelArraySize_); + pixelArrayArea_ = Rectangle(sizes_.back()); + readableArea_ = pixelArrayArea_; + activeArea_ = pixelArrayArea_; } void CameraSensorLegacy::initStaticProperties() @@ -627,7 +636,8 @@ int CameraSensorLegacy::initProperties() mountingOrientation_ = Orientation::Rotate0; } - properties_.set(properties::PixelArraySize, pixelArraySize_); + properties_.set(properties::PixelArrayArea, pixelArrayArea_); + properties_.set(properties::PixelArrayReadableArea, readableArea_); properties_.set(properties::PixelArrayActiveAreas, { activeArea_ }); /* Color filter array pattern, register only for RAW sensors. */ diff --git a/src/libcamera/sensor/camera_sensor_raw.cpp b/src/libcamera/sensor/camera_sensor_raw.cpp index e50111f1d5c7..117da63ef364 100644 --- a/src/libcamera/sensor/camera_sensor_raw.cpp +++ b/src/libcamera/sensor/camera_sensor_raw.cpp @@ -141,7 +141,8 @@ private: std::vector testPatternModes_; controls::draft::TestPatternModeEnum testPatternMode_; - Size pixelArraySize_; + Rectangle pixelArrayArea_; + Rectangle readableArea_; Rectangle activeArea_; BayerFormat::Order cfaPattern_; bool supportHFlips_; @@ -431,7 +432,17 @@ std::optional CameraSensorRaw::init() return { ret }; } - pixelArraySize_ = rect.size(); + /* + * \todo Implement querying the physical sensor size based on pad 1, + * stream 0. See + * https://lore.kernel.org/linux-media/20260409201501.975242-23-sakari.ailus@linux.intel.com/ + */ + LOG(CameraSensor, Warning) << "PixelArrayArea might be incorrect"; + pixelArrayArea_ = Rectangle(0, 0, + rect.x + rect.width, + rect.y + rect.height); + + readableArea_ = rect; ret = subdev_->getSelection(streams_.image.sink, V4L2_SEL_TGT_CROP_DEFAULT, &activeArea_); @@ -634,7 +645,8 @@ int CameraSensorRaw::initProperties() mountingOrientation_ = Orientation::Rotate0; } - properties_.set(properties::PixelArraySize, pixelArraySize_); + properties_.set(properties::PixelArrayArea, pixelArrayArea_); + properties_.set(properties::PixelArrayReadableArea, readableArea_); properties_.set(properties::PixelArrayActiveAreas, { activeArea_ }); /* Color filter array pattern. */