[{"id":25903,"web_url":"https://patchwork.libcamera.org/comment/25903/","msgid":"","date":"2022-11-24T15:52:17","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Jacopo\n\nThank you for this patch.\n\nOn Thu, 24 Nov 2022 at 12:12, Jacopo Mondi wrote:\n>\n> The two pipeline handlers that currently support Transform do so by\n> operating H/V flips on the image sensor, instead of rotating on the ISP.\n>\n> As the image sensor performs the actual rotation, centralize the code\n> that validates a requested Transform against the camera sensor rotation\n> and capabilities in the CameraSensor class.\n>\n> The implementation in the IPU3 pipeline handler was copied from the\n> RaspberryPi implementation, and is now centralized in CameraSensor to\n> make it easier for other platforms that do not rotate on the ISP to\n> implement support for Transform using the CameraSensor class.\n>\n> Signed-off-by: Jacopo Mondi \n> ---\n> include/libcamera/internal/camera_sensor.h | 4 ++\n> src/libcamera/camera_sensor.cpp | 59 +++++++++++++++++++\n> src/libcamera/pipeline/ipu3/ipu3.cpp | 45 ++------------\n> .../pipeline/raspberrypi/raspberrypi.cpp | 59 +++----------------\n> 4 files changed, 76 insertions(+), 91 deletions(-)\n>\n> diff --git a/include/libcamera/internal/camera_sensor.h b/include/libcamera/internal/camera_sensor.h\n> index 878f3c28a3c9..bea52badaff7 100644\n> --- a/include/libcamera/internal/camera_sensor.h\n> +++ b/include/libcamera/internal/camera_sensor.h\n> @@ -29,6 +29,8 @@ class BayerFormat;\n> class CameraLens;\n> class MediaEntity;\n>\n> +enum class Transform;\n> +\n> struct CameraSensorProperties;\n>\n> class CameraSensor : protected Loggable\n> @@ -68,6 +70,8 @@ public:\n>\n> CameraLens *focusLens() { return focusLens_.get(); }\n>\n> + Transform validateTransform(Transform *transform) const;\n> +\n> protected:\n> std::string logPrefix() const override;\n>\n> diff --git a/src/libcamera/camera_sensor.cpp b/src/libcamera/camera_sensor.cpp\n> index 3afcbc482095..a4ad0ba9a099 100644\n> --- a/src/libcamera/camera_sensor.cpp\n> +++ b/src/libcamera/camera_sensor.cpp\n> @@ -982,6 +982,65 @@ void CameraSensor::updateControlInfo()\n> * connected to the sensor\n> */\n>\n> +/**\n> + * \\brief Validate a transform request against the sensor capabilities\n> + * \\param[inout] transform The requested transformation, updated to match\n> + * the sensor capabilities\n> + *\n> + * The requested \\a transform is adjusted against the sensor rotation and its\n> + * capabilities.\n> + *\n> + * In example, if the requested \\a transform is Transform::Identity and the\n\ns/In example/For example/ reads better.\n\n> + * sensor rotation is 180 degrees, the resulting transform returned by the\n> + * function is Transform::Rot180 to automatically correct the image, but only if\n> + * the sensor can actually apply horizontal and vertical flips.\n> + *\n> + * \\return A Transform instance that represents how \\a transform is applied to\n> + * the camera sensor\n> + */\n> +Transform CameraSensor::validateTransform(Transform *transform) const\n\nTeeny nit-pick, but I wondered if another signature might be more\nconvenient for callers. For example\n\nStatus CameraSensor::validateTransform(Transform &transform, Transform\n&combined)\n\nIt would return \"Adjusted\" if the \"transform out\" is different from\nthe \"transform in\". But I'm not terribly bothered!!\n\n> +{\n> + /* Adjust the requested transform to the sensor rotation. */\n> + int32_t rotation = properties().get(properties::Rotation).value_or(0);\n> + bool success;\n> +\n> + Transform rotationTransform = transformFromRotation(rotation, &success);\n> + if (!success)\n> + LOG(CameraSensor, Warning) << \"Invalid rotation of \" << rotation\n> + << \" degrees - ignoring\";\n> +\n> + Transform combined = *transform * rotationTransform;\n\nI know I must have written this, but now I'm wondering if\n\"rotationTransform\" here should be \"-rotationTransform\". I guess it\nnever matters with our cameras which are always 0 or 180 degrees\nrotated, but maybe in other cases...?\n\n> +\n> + /*\n> + * The camera sensor cannot do Transpose. Adjust any combined result\n> + * that includes a transpose by flipping the transpose bit to notify\n> + * applications they either get the transform they requested, or have\n> + * to do a simple transpose themselves (they don't have to worry about\n> + * the other possible cases).\n> + */\n> + if (!!(combined & Transform::Transpose)) {\n> + /*\n> + * Flipping the transpose bit in \"transform\" flips it in the\n> + * combined result too (as it's the last thing that happens),\n> + * which is of course clearing it.\n> + */\n> + *transform ^= Transform::Transpose;\n> + combined &= ~Transform::Transpose;\n> + }\n> +\n> + /*\n> + * If the sensor can do no transforms, then combined must be changed to\n> + * the identity and the sensor rotation must be cleared from the\n> + * requested \"transform\".\n\nDon't really follow this comment. Maybe\n\n * If the sensor cannot do transforms, then combined must be changed to\n * the identity and the only transform the user can have is the sensor\n * rotation itself.\n\n(Which rather suggests the code following has got the transform the\nwrong way round...)\n\n> + */\n> + if (!supportFlips_ && !!combined) {\n> + *transform = -rotationTransform;\n\nThis might really be \"rotationTransform\" (without the inverse).\nDepending on which we round we view the sensor rotation (from in front\nof or behind the camera).\n\nAlso, as we remarked elsewhere, we should possibly consider the\nexistence of H and V flips separately.\n\nBut apart from my existential doubts about inverse rotations, it all\nseems reasonable to me, though I sense we'll be coming back to this\nagain anyway:\n\nReviewed-by: David Plowman \n\nThanks!\nDavid\n\n\n> + combined = Transform::Identity;\n> + }\n> +\n> + return combined;\n> +}\n> +\n> std::string CameraSensor::logPrefix() const\n> {\n> return \"'\" + entity_->name() + \"'\";\n> diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp\n> index e4d79ea44aed..a424ac914859 100644\n> --- a/src/libcamera/pipeline/ipu3/ipu3.cpp\n> +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp\n> @@ -184,48 +184,15 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate()\n> if (config_.empty())\n> return Invalid;\n>\n> - Transform combined = transform * data_->rotationTransform_;\n> -\n> - /*\n> - * We combine the platform and user transform, but must \"adjust away\"\n> - * any combined result that includes a transposition, as we can't do\n> - * those. In this case, flipping only the transpose bit is helpful to\n> - * applications - they either get the transform they requested, or have\n> - * to do a simple transpose themselves (they don't have to worry about\n> - * the other possible cases).\n> - */\n> - if (!!(combined & Transform::Transpose)) {\n> - /*\n> - * Flipping the transpose bit in \"transform\" flips it in the\n> - * combined result too (as it's the last thing that happens),\n> - * which is of course clearing it.\n> - */\n> - transform ^= Transform::Transpose;\n> - combined &= ~Transform::Transpose;\n> - status = Adjusted;\n> - }\n> -\n> /*\n> - * We also check if the sensor doesn't do h/vflips at all, in which\n> - * case we clear them, and the application will have to do everything.\n> + * Validate the requested transform against the sensor capabilities and\n> + * rotation and store the final combined transform that configure() will\n> + * need to apply to the sensor to save us working it out again.\n> */\n> - if (!data_->supportsFlips_ && !!combined) {\n> - /*\n> - * If the sensor can do no transforms, then combined must be\n> - * changed to the identity. The only user transform that gives\n> - * rise to this is the inverse of the rotation. (Recall that\n> - * combined = transform * rotationTransform.)\n> - */\n> - transform = -data_->rotationTransform_;\n> - combined = Transform::Identity;\n> + Transform requestedTransform = transform;\n> + combinedTransform_ = data_->cio2_.sensor()->validateTransform(&transform);\n> + if (transform != requestedTransform)\n> status = Adjusted;\n> - }\n> -\n> - /*\n> - * Store the final combined transform that configure() will need to\n> - * apply to the sensor to save us working it out again.\n> - */\n> - combinedTransform_ = combined;\n>\n> /* Cap the number of entries to the available streams. */\n> if (config_.size() > kMaxStreams) {\n> diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> index 087c71b65700..e83984985bce 100644\n> --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> @@ -366,59 +366,14 @@ CameraConfiguration::Status RPiCameraConfiguration::validate()\n> status = validateColorSpaces(ColorSpaceFlag::StreamsShareColorSpace);\n>\n> /*\n> - * What if the platform has a non-90 degree rotation? We can't even\n> - * \"adjust\" the configuration and carry on. Alternatively, raising an\n> - * error means the platform can never run. Let's just print a warning\n> - * and continue regardless; the rotation is effectively set to zero.\n> + * Validate the requested transform against the sensor capabilities and\n> + * rotation and store the final combined transform that configure() will\n> + * need to apply to the sensor to save us working it out again.\n> */\n> - int32_t rotation = data_->sensor_->properties().get(properties::Rotation).value_or(0);\n> - bool success;\n> - Transform rotationTransform = transformFromRotation(rotation, &success);\n> - if (!success)\n> - LOG(RPI, Warning) << \"Invalid rotation of \" << rotation\n> - << \" degrees - ignoring\";\n> - Transform combined = transform * rotationTransform;\n> -\n> - /*\n> - * We combine the platform and user transform, but must \"adjust away\"\n> - * any combined result that includes a transform, as we can't do those.\n> - * In this case, flipping only the transpose bit is helpful to\n> - * applications - they either get the transform they requested, or have\n> - * to do a simple transpose themselves (they don't have to worry about\n> - * the other possible cases).\n> - */\n> - if (!!(combined & Transform::Transpose)) {\n> - /*\n> - * Flipping the transpose bit in \"transform\" flips it in the\n> - * combined result too (as it's the last thing that happens),\n> - * which is of course clearing it.\n> - */\n> - transform ^= Transform::Transpose;\n> - combined &= ~Transform::Transpose;\n> - status = Adjusted;\n> - }\n> -\n> - /*\n> - * We also check if the sensor doesn't do h/vflips at all, in which\n> - * case we clear them, and the application will have to do everything.\n> - */\n> - if (!data_->supportsFlips_ && !!combined) {\n> - /*\n> - * If the sensor can do no transforms, then combined must be\n> - * changed to the identity. The only user transform that gives\n> - * rise to this the inverse of the rotation. (Recall that\n> - * combined = transform * rotationTransform.)\n> - */\n> - transform = -rotationTransform;\n> - combined = Transform::Identity;\n> + Transform requestedTransform = transform;\n> + combinedTransform_ = data_->sensor_->validateTransform(&transform);\n> + if (transform != requestedTransform)\n> status = Adjusted;\n> - }\n> -\n> - /*\n> - * Store the final combined transform that configure() will need to\n> - * apply to the sensor to save us working it out again.\n> - */\n> - combinedTransform_ = combined;\n>\n> unsigned int rawCount = 0, outCount = 0, count = 0, maxIndex = 0;\n> std::pair outSize[2];\n> @@ -453,7 +408,7 @@ CameraConfiguration::Status RPiCameraConfiguration::validate()\n> if (data_->flipsAlterBayerOrder_) {\n> BayerFormat bayer = BayerFormat::fromV4L2PixelFormat(fourcc);\n> bayer.order = data_->nativeBayerOrder_;\n> - bayer = bayer.transform(combined);\n> + bayer = bayer.transform(combinedTransform_);\n> fourcc = bayer.toV4L2PixelFormat();\n> }\n>\n> --\n> 2.38.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 085DDBDE6B\n\tfor ;\n\tThu, 24 Nov 2022 15:52:33 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3D6126331F;\n\tThu, 24 Nov 2022 16:52:32 +0100 (CET)","from mail-pg1-x531.google.com (mail-pg1-x531.google.com\n\t[IPv6:2607:f8b0:4864:20::531])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CBF366330D\n\tfor ;\n\tThu, 24 Nov 2022 16:52:30 +0100 (CET)","by mail-pg1-x531.google.com with SMTP id n17so1843583pgh.9\n\tfor ;\n\tThu, 24 Nov 2022 07:52:30 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1669305152;\n\tbh=7StPd/juq2cVzlMPRaIj6j5viJx2JLndUVMLY1XXDTk=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=M8D19jTXXhVuRVLIZN0IZvrdabMTD1CqBk9SFnXGUzfE5draj5cemHgD1gqhUMfLy\n\tlcyqp+BEnzKA3+pU8y5eMTZOJI8xGtfJhwd8a4I+K4xlX1/kVyLJPKdIGj6zzzXQXe\n\tDJXhb0bOF7fTiRiLVtyO1J3PPRhl7zu9pWJLBnhCvN/Rob9GstKrcpgcMuKl1FCMRt\n\tBVYZ58T0XE6tzHeh8eGTGsDpO2Lz5hSCT+JqVd5SlxzlZtJNPlGoWU7O4VsZL54elA\n\tMPUkOXjvzEqqiUen4Vl3/LyIZ4ITdwdkO+DpO41BoFvC1pmvx2jrUZaijA7MQmqgYB\n\t0JY0KG2pgphUg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=5ADjTqQbuC0V29rg2kPmKMzhnd681hXjrx4Cn52D9lE=;\n\tb=h/GabZS7Pt8Bkhv0rs4ILPWygG+H/hY7XgJHCsuJMRBfuo/J4+TpLCdALf22kMB1Gy\n\twze0kvifmgOfeqn4NHHtkpbYe42d5ANUFnU14SCzewfiFhEirxNdrB9W2f6vs2uWVQ14\n\tTtkaZsiJoo/Gp7aFVCyFV5qqr9/iOAXdTaM+vniH/DO1dPTF5O/HYwiUyGQxkQZG71eT\n\tr9xcLBFX+0Tfb2HZ793VhbhkjhYi7CdEx48ch5UVGFJ2out07kV15EvrpbhJqQ8tOFlx\n\txB9ZwOHY9MrNCw6JGqpimsclFNQmPErkhUUeOU9PzH6gWVgXcs1QnFvZxc/0oVkvdfVP\n\tkhcQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"h/GabZS7\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=5ADjTqQbuC0V29rg2kPmKMzhnd681hXjrx4Cn52D9lE=;\n\tb=q1/1v2GMn+gxD1nvOVrVFCNcnwGF8TzxSX1Eld8aUhR5UX6vaEWu6iVwD0zh/aiRsT\n\ttdzbVAQdmUSElA94TCtkriCqdTxx74IqNy1XPgKQHOZ/azc4W+ftf4xkD0VHSLFHLVh9\n\tl5JuaVemwYTD5VQfFlhAUIG2FIfXtq/doNuqRio2u6IPGaxDVeJlN/qXv+yq1tLeJSeP\n\t9rf81Gx2JjK3ZVBiC5ExH8AeATOG/p8y3Urg/vt9S3p6T4es96ujFQBo2VNQ6Umyr6eq\n\tJHjOTeve1ZKA20eJJwGZ6iXOEiTeuZP2XMmwmRqkRuY2rckvtTgXGTOM8f42vZeo7ZvU\n\twuaw==","X-Gm-Message-State":"ANoB5pnA1UiUrXZodO9wcQFjIWt0ymEvsxiYUdqQ/7sbUlnPlSpehUXF\n\t/ip/Y2w9RO2nzITytmFzqwn/dw76prklex/Y/s6HtA==","X-Google-Smtp-Source":"AA0mqf5AFtiJowSGliiju2/i4c9G/s3fdnXYY0XbJqtdLr8CwlvOJNzx4NTsxaE83xq2i0itxGJJr66wtcepaemdBwI=","X-Received":"by 2002:a05:6a00:4210:b0:56c:b8c2:ee89 with SMTP id\n\tcd16-20020a056a00421000b0056cb8c2ee89mr14600185pfb.21.1669305148718;\n\tThu, 24 Nov 2022 07:52:28 -0800 (PST)","MIME-Version":"1.0","References":"<20221124121220.47000-1-jacopo@jmondi.org>\n\t<20221124121220.47000-6-jacopo@jmondi.org>","In-Reply-To":"<20221124121220.47000-6-jacopo@jmondi.org>","Date":"Thu, 24 Nov 2022 15:52:17 +0000","Message-ID":"","To":"Jacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26019,"web_url":"https://patchwork.libcamera.org/comment/26019/","msgid":"<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>","date":"2022-12-07T12:32:04","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":140,"url":"https://patchwork.libcamera.org/api/people/140/","name":"Robert Mader","email":"robert.mader@collabora.com"},"content":"Hi!\n\nI'd love if we could clarify the following part to ensure the Pipewire \nimplementation is correct:\n\n> > +\n> > + /*\n> > + * The camera sensor cannot do Transpose. Adjust any combined result\n> > + * that includes a transpose by flipping the transpose bit to notify\n> > + * applications they either get the transform they requested, or have\n> > + * to do a simple transpose themselves (they don't have to worry about\n> > + * the other possible cases).\n> > + */\n> > + if (!!(combined & Transform::Transpose)) {\n> > + /*\n> > + * Flipping the transpose bit in \"transform\" flips it in the\n> > + * combined result too (as it's the last thing that happens),\n> > + * which is of course clearing it.\n> > + */\n> > + *transform ^= Transform::Transpose;\n> > + combined &= ~Transform::Transpose;\n> > + }\n> > +\n> > + /*\n> > + * If the sensor can do no transforms, then combined must be changed to\n> > + * the identity and the sensor rotation must be cleared from the\n> > + * requested \"transform\".\n>\n> Don't really follow this comment. Maybe\n>\n> * If the sensor cannot do transforms, then combined must be changed to\n> * the identity and the only transform the user can have is the sensor\n> * rotation itself.\n>\n> (Which rather suggests the code following has got the transform the\n> wrong way round...)\n>\n> > + */\n> > + if (!supportFlips_ && !!combined) {\n> > + *transform = -rotationTransform;\n>\n> This might really be \"rotationTransform\" (without the inverse).\n> Depending on which we round we view the sensor rotation (from in front\n> of or behind the camera).\n>\n> Also, as we remarked elsewhere, we should possibly consider the\n> existence of H and V flips separately.\n>\n> But apart from my existential doubts about inverse rotations, it all\n> seems reasonable to me, though I sense we'll be coming back to this\n> again anyway:\n\nAs example I'd like to take the back-camera of the Pinephone Pro \n(imx258). Pipewire will ask for an identity transform and the kernel \nwill most likely report `V4L2_CID_CAMERA_SENSOR_ROTATION == 270` (once \ndriver patches and device tree updates land), so the initial values \nwe'll get are  `properties::Rotation == 270` -> `combined == \nTransform::Rot270`.\n\nRight now neither transpose nor flips are supported, so the steps in the \ncurrent revision are:\n\n 1. `combined == rot270` / `transform == identity`\n 2. `combined == hflip` / `transform == transpose`\n 3. `combined == `identity` / `transform == rot90`\n\nThe value from `transform` will then be used for \n`CameraConfiguration::transform`, signalling to the client that it will \nhave to compensate accordingly. Following the comment to not use the \ninverse it would be `rot270` instead.\n\n From an API-point of view, I'd say `rot90` is the more intuitive value \nfor clients, because they can then just follow the documentation in \ncamera_sensor.cpp and apply a transformation accordingly. Would you \nagree that this should be the desired outcome? I.e. should we stick to \nthe proposed value?\n\nBest regards,\n\nRobert","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A9B7ABDE6B\n\tfor ;\n\tWed, 7 Dec 2022 12:32:10 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 083C16333F;\n\tWed, 7 Dec 2022 13:32:10 +0100 (CET)","from madras.collabora.co.uk (madras.collabora.co.uk\n\t[IPv6:2a00:1098:0:82:1000:25:2eeb:e5ab])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D41ED63335\n\tfor ;\n\tWed, 7 Dec 2022 13:32:08 +0100 (CET)","from [IPV6:2001:4091:a245:8030:aa38:3a69:ce40:58d2] (unknown\n\t[IPv6:2001:4091:a245:8030:aa38:3a69:ce40:58d2])\n\t(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (4096 bits)\n\tserver-digest SHA256)\n\t(No client certificate requested) (Authenticated sender: rmader)\n\tby madras.collabora.co.uk (Postfix) with ESMTPSA id 380DD660238B\n\tfor ;\n\tWed, 7 Dec 2022 12:32:08 +0000 (GMT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670416330;\n\tbh=HSkKaCZH4ARk4sySZHwaOpKTI8tnAK0pS79kUNzE7H4=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=NjxKwfl2U2eWXWE95Q0esF6SyVgoyvmDW4JllUb2PcMxVavo1IWYPZOqpC9rdTDkt\n\tOsx7XL7SyyjVNv1HyoFx8QcZ3RWtb2WBy4DVDR0dCHbFWaqqHI3nhAeXzD218l+h+B\n\tRe06NM1gDaxz7LihpOEYqyaLiPd+O05kiWcRgJPl2r8ySiGGBV09KA8gbgjp9Ijef9\n\tzS5PfG8GoW9fabEkWTJsjN+7Rs/HX0gOkgOIRlhnvsc7/MKuq9Ql0I7hyIPrc4jX4Z\n\t88PtMNj00L0LuX7K3sMJdjkKWhAbr6OQoOdc3ZKLT2WTOCkJ7I6Mu2mwEPO+egFFIn\n\thFFwdolN7VdYw==","v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com;\n\ts=mail; t=1670416328;\n\tbh=HSkKaCZH4ARk4sySZHwaOpKTI8tnAK0pS79kUNzE7H4=;\n\th=Date:To:References:Subject:From:In-Reply-To:From;\n\tb=Gn91AtlgXlLHZh4eUZ8xMcKjTNbzNRouwZgGRKZU4gcdGrzPLbte6TWsosY4XLJGy\n\tA7aAf9xaAJtz3Np6S5b1XR2ckg9Gz1CmZa4UAVFaPHl0xN4nTMU+s7bXPM2xsUg3gf\n\tf0msAOS8Qp7FuqL2MBNgFTXNMTZtBZQ+3XLG+we5oa2+fF8Rs5wM+IEy5MPx8G9hZZ\n\tJi4/3zGWXdlBAOzB/i6pymA2dEug842cBAMPazdSdddeWkVhvzC8kqlw+bY58rVBZo\n\tTJbn06R8AhjOdJvWkrlcsa0XVI7xwAR+QxjXRRP5AQdrv/v1jmosrxlaE33TMOV9Vj\n\tQskynzkzAbWyQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=collabora.com\n\theader.i=@collabora.com\n\theader.b=\"Gn91Atlg\"; dkim-atps=neutral","Content-Type":"multipart/alternative;\n\tboundary=\"------------W9Q7WJQqV0mX2k0Ic4X1erpQ\"","Message-ID":"<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>","Date":"Wed, 7 Dec 2022 13:32:04 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101\n\tThunderbird/102.5.0","To":"libcamera-devel@lists.libcamera.org","References":"","Content-Language":"en-US","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Robert Mader via libcamera-devel ","Reply-To":"Robert Mader ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26022,"web_url":"https://patchwork.libcamera.org/comment/26022/","msgid":"","date":"2022-12-07T14:28:07","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Robert\n\nThanks for prodding us all on this question!\n\nOn Wed, 7 Dec 2022 at 12:32, Robert Mader via libcamera-devel\n wrote:\n>\n> Hi!\n>\n> I'd love if we could clarify the following part to ensure the Pipewire implementation is correct:\n>\n> > +\n> > + /*\n> > + * The camera sensor cannot do Transpose. Adjust any combined result\n> > + * that includes a transpose by flipping the transpose bit to notify\n> > + * applications they either get the transform they requested, or have\n> > + * to do a simple transpose themselves (they don't have to worry about\n> > + * the other possible cases).\n> > + */\n> > + if (!!(combined & Transform::Transpose)) {\n> > + /*\n> > + * Flipping the transpose bit in \"transform\" flips it in the\n> > + * combined result too (as it's the last thing that happens),\n> > + * which is of course clearing it.\n> > + */\n> > + *transform ^= Transform::Transpose;\n> > + combined &= ~Transform::Transpose;\n> > + }\n> > +\n> > + /*\n> > + * If the sensor can do no transforms, then combined must be changed to\n> > + * the identity and the sensor rotation must be cleared from the\n> > + * requested \"transform\".\n>\n> Don't really follow this comment. Maybe\n>\n> * If the sensor cannot do transforms, then combined must be changed to\n> * the identity and the only transform the user can have is the sensor\n> * rotation itself.\n>\n> (Which rather suggests the code following has got the transform the\n> wrong way round...)\n>\n> > + */\n> > + if (!supportFlips_ && !!combined) {\n> > + *transform = -rotationTransform;\n>\n> This might really be \"rotationTransform\" (without the inverse).\n> Depending on which we round we view the sensor rotation (from in front\n> of or behind the camera).\n>\n> Also, as we remarked elsewhere, we should possibly consider the\n> existence of H and V flips separately.\n>\n> But apart from my existential doubts about inverse rotations, it all\n> seems reasonable to me, though I sense we'll be coming back to this\n> again anyway:\n>\n> As example I'd like to take the back-camera of the Pinephone Pro (imx258). Pipewire will ask for an identity transform and the kernel will most likely report `V4L2_CID_CAMERA_SENSOR_ROTATION == 270` (once driver patches and device tree updates land), so the initial values we'll get are `properties::Rotation == 270` -> `combined == Transform::Rot270`.\n>\n> Right now neither transpose nor flips are supported, so the steps in the current revision are:\n>\n> `combined == rot270` / `transform == identity`\n> `combined == hflip` / `transform == transpose`\n> `combined == `identity` / `transform == rot90`\n>\n> The value from `transform` will then be used for `CameraConfiguration::transform`, signalling to the client that it will have to compensate accordingly. Following the comment to not use the inverse it would be `rot270` instead.\n>\n> From an API-point of view, I'd say `rot90` is the more intuitive value for clients, because they can then just follow the documentation in camera_sensor.cpp and apply a\n\n(What document is that? It wasn't immediately clear to me...)\n\n> transformation accordingly. Would you agree that this should be the desired outcome? I.e. should we stick to the proposed value?\n\nI looked back over this again, and found Jacopo's description of the\nV4L2_CID_CAMERA_SENSOR_ROTATION control\n(https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\nis what I stumbled into)\n\nFrom that, and the nice example with the stick person, I deduce that\nif your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\ndegrees, then the image you capture (if nothing else happens to it)\nwill look like it has had a 90 degree clockwise rotation performed.\nEveryone agree with that?\n\nSo in turn, if you have a camera/ISP that can apply no transforms at\nall, then the only value for the user transform that will not come\nback as adjusted is \"rot90\", because it looks like it's had a 90\ndegree clockwise rotation performed (and libcamera transforms count\nrotations as being clockwise). Does that also make sense?\n\nSo by a slightly arbitrary mixture of conventions, it looks to me as\nthough the use of the inverse that I queried is indeed wrong, because\nwe \"need camera 90 degree rotation\" => \"user transform rot90\" in this\ncase.\n\nDoes that sound plausible... what do folks think?\n\nThanks everyone!\nDavid\n\n>\n> Best regards,\n>\n> Robert","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A6626BE08B\n\tfor ;\n\tWed, 7 Dec 2022 14:28:23 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id F02D763336;\n\tWed, 7 Dec 2022 15:28:22 +0100 (CET)","from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com\n\t[IPv6:2607:f8b0:4864:20::102d])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CEEE663335\n\tfor ;\n\tWed, 7 Dec 2022 15:28:20 +0100 (CET)","by mail-pj1-x102d.google.com with SMTP id q15so16715874pja.0\n\tfor ;\n\tWed, 07 Dec 2022 06:28:20 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670423303;\n\tbh=yPHTXWaGbSncgIdXn3dlyIsIIYZYI5P1rpg/8hSaPv0=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=lDnnyP4aPlMQxRWJbwSGPM+S7Zqc9M8W+CYjZdiVC5wlCQbSjypaRAYp3ZWMmwmYl\n\t006bNqRmjtoaLFiQIvIAzED98w9z2hGiUe7xGZKQL/IDrgAeg8beejVoGXmcUHAKoH\n\temiUvH348YXt/jXrPOqbSS0L7POofCRelwviRO7yu7kdOAVNsOm/RgPOA0j2U/7opq\n\t0/+pvlvoLYbCou5r8VUBNg8C0FQr5uJsGDYAG1FQ2it4pw6jij7CJf7LCuOIpkLe3R\n\tUQUN3e8GO5V25dZxBW5JKxHd0YGNV0VEIEqemoKn/aTmEUwtcNfAvkHayNWJdMjkRj\n\t+YhqAbkCOdWMQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=WLV/4/MzPYLVR68PzYMUzQFddfL/SUX36SL+5eyK0wg=;\n\tb=Mjyon3INeVdLEHsxUPtxU3Ptu5wT/bkKDcuJq6xWlbfLrQ3irYgM2PAyT+1xVwyiMi\n\t5SX/wHoAqHD3UPav8U+lTBbh4/GFJ4gD9yk7Xskhr0nRaeWVWxI/hUY53uR03fGaqefn\n\tWoaWZUIVILMgDJAwQTRbg8f6ZLuxu/fZ4szxiovJSLJdX5Yi5m17TMbGhL0WLmWagQka\n\t7b4cfZnRPgCijLmCsuKEjnz3B+0fCWu7zu0EQwb6a6k5qTXCvuyhOi6UnSAKUlVjba2v\n\tPsbS0xzE67TB4rlj1PG+QZv3+mmoBGQMO4yVqsA+JYzFz9E72ghE4QhB9jtFAt0IfT+j\n\tBldg=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"Mjyon3IN\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=WLV/4/MzPYLVR68PzYMUzQFddfL/SUX36SL+5eyK0wg=;\n\tb=hWxF3I21GYln6nDSmKD4RzxkgxSdm7m1/HBk+jiHHvYEQs4b14CSIO9rwIkd1OZ3YA\n\tLQ7d69GmrNAov0jK6hPpQp41RZ40bYsQnjoh8nnW7urLdK/vyp14CfBGEsLyALE+oJUG\n\tO8ZcxYBt3GmPCu3h/2EH+VEnejeW++f6IxHu3PIbL5tcvSkdlgxRMeS4xffyvxES7eoe\n\t3mRK2ODIMvFxJ07OfBzy9rpc5IPQ+h2wEm4tZEkp8g0/isS91sjZPCJbui9Wbqvi5oHi\n\td44dNnYEbVki012MzJQy0D7je0iWnOM0t/qrQVIoKiVCd1FcB5zP0PtqZomVO6XGVxpt\n\tmFqQ==","X-Gm-Message-State":"ANoB5pl3giG1WDKbn34xVFHdP8zk9Kb1F/FjOqOzK3Vz0xvmD7JBPsHI\n\t6C0LyWnz0K4oB8snKv3N9ev2GvowHtvbmNmqhZ+foQ==","X-Google-Smtp-Source":"AA0mqf4soQA3XW9JuYkj2IOlf/+O18+YFrtU72OnPsFLthubQkCKofxibGDbeTQq4QJQqlCrCjam73D4gntfY9rlP2w=","X-Received":"by 2002:a17:902:7d93:b0:186:9cf4:e53b with SMTP id\n\ta19-20020a1709027d9300b001869cf4e53bmr78700238plm.50.1670423298685;\n\tWed, 07 Dec 2022 06:28:18 -0800 (PST)","MIME-Version":"1.0","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>","In-Reply-To":"<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>","Date":"Wed, 7 Dec 2022 14:28:07 +0000","Message-ID":"","To":"Robert Mader ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26023,"web_url":"https://patchwork.libcamera.org/comment/26023/","msgid":"<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>","date":"2022-12-07T15:48:40","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":142,"url":"https://patchwork.libcamera.org/api/people/142/","name":"Robert Mader","email":"robert.mader@posteo.de"},"content":"On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> Hi Robert\n>\n> Thanks for prodding us all on this question!\n\nHi, thanks for the quick reply! And a pleasure :P\n\n> I looked back over this again, and found Jacopo's description of the\n> V4L2_CID_CAMERA_SENSOR_ROTATION control\n> (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> is what I stumbled into)\n\nThe current kernel docu can be found at\n\nhttps://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n\nthe libcamera one at\n\nhttps://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n\n> From that, and the nice example with the stick person, I deduce that\n> if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> degrees, then the image you capture (if nothing else happens to it)\n> will look like it has had a 90 degree clockwise rotation performed.\n> Everyone agree with that?\nYep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would \ntranslate to Transform::Rot90.\n> So in turn, if you have a camera/ISP that can apply no transforms at\n> all, then the only value for the user transform that will not come\n> back as adjusted is \"rot90\", because it looks like it's had a 90\n> degree clockwise rotation performed (and libcamera transforms count\n> rotations as being clockwise). Does that also make sense?\n>\n> So by a slightly arbitrary mixture of conventions, it looks to me as\n> though the use of the inverse that I queried is indeed wrong, because\n> we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> case.\n>\n> Does that sound plausible... what do folks think?\n\nAh, I think the question is whether the adjusted value means: \"this is \nwhat you'll get\" vs \"this is what you have to do yourself\".\n\nAbove I argued in favor of the later, but I guess the first one is what \nthe API is supposed to mean. That would imply that the client has \ncompute the difference between requested and adjusted value itself.\n\nIn my case that would mean:\n\nrequested CameraConfiguration::transform: Transform::Identity, \nV4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted \nCameraConfiguration::transform: Transform::Rot270\n\nTo compute what the client has to do, it would need to do: \nTransform::Identity - Transform::Rot270 = Transform::Rot90. And then \napply that, a 90 degr. rotation clockwise.\n\nDoes that sound reasonable? In that case the patch here should indeed be \n`*transform = rotationTransform;`, without the inverse.\n\n> Thanks everyone!\n> David\n\nThanks!\n\nRobert","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D92BDBDE6B\n\tfor ;\n\tWed, 7 Dec 2022 15:48:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3D79563336;\n\tWed, 7 Dec 2022 16:48:43 +0100 (CET)","from mout02.posteo.de (mout02.posteo.de [185.67.36.66])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 239CF63335\n\tfor ;\n\tWed, 7 Dec 2022 16:48:42 +0100 (CET)","from submission (posteo.de [185.67.36.169]) \n\tby mout02.posteo.de (Postfix) with ESMTPS id A2D43240106\n\tfor ;\n\tWed, 7 Dec 2022 16:48:41 +0100 (CET)","from customer (localhost [127.0.0.1])\n\tby submission (posteo.de) with ESMTPSA id 4NS1sj1cQxz6tlh\n\tfor ;\n\tWed, 7 Dec 2022 16:48:41 +0100 (CET)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670428123;\n\tbh=bWvvVRg9ca6efjh7XuJJqDcS45N5uryB05Ju2N6Cp50=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=a+cYBosAHGbdHLlziDiYnYMDOaXRd0LOCJ2XNzcGcsPqOAhY6VGp0wdckwtLIjiho\n\tBZ66+J2xMYBoYlY/iL/GXiYkVcl7Yv9R9FI5IKw6CZDBJivws1dAFrACJSgmYkWPtZ\n\txqZIwWTdZvoqZTXh3UmpQApvxLvBOvV8BMoR81h1OT5bJdAFl2al1Nj0MlLZoKg8PI\n\tsQhT80ht6oTcwZKeCcyslvkroUQ4X66ORV5MxIIWEg6xYea9ZwYAL6Pdy4RChlvDIr\n\t24oplqXXNSeBYc0Qe5gz5iwfxeFsNuFPvS72cEvWEWdKQ/8qRKpslA1pin9MrdEVat\n\t39piSHXew3uYQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017;\n\tt=1670428121; bh=bWvvVRg9ca6efjh7XuJJqDcS45N5uryB05Ju2N6Cp50=;\n\th=Date:Subject:To:From:From;\n\tb=Vz0RcaeYkYHecuyDgVwjMkgiHyVpu/T+d4B8TAJw+mtvJeSXfArHGB1WVm9vkV381\n\t1SgDMi66oKJ436hVz5I9CfswaqpXRzVwCWtccLVBX122uKK2x+lZZitPrEEjIWEq3S\n\t56UKGyQAQXQZlqRsPX83erXry/yREUXV7AmgHb+QxkAN88RIenWrLO95TrMGT0Gz+n\n\tIm+H/2RuB+oeBSeEAOKXnQDJFG865a4K3gSqkGb/v1eCEX42CcP4cpfin9QW9VSRPD\n\tNCQmiR1nLxX9GUJtwY1A94WddA72jrrb4tOJEegha/9OqL2K9UljQ0BXh7i4kBUru0\n\tFzpgqGBEZTw3Q=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=posteo.de header.i=@posteo.de\n\theader.b=\"Vz0RcaeY\"; dkim-atps=neutral","Message-ID":"<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>","Date":"Wed, 7 Dec 2022 15:48:40 +0000","MIME-Version":"1.0","To":"libcamera-devel@lists.libcamera.org","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t","Content-Language":"en-US","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Robert Mader via libcamera-devel ","Reply-To":"Robert Mader ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26033,"web_url":"https://patchwork.libcamera.org/comment/26033/","msgid":"<20221209111239.q6xgvaqhy4zdm4nl@uno.localdomain>","date":"2022-12-09T11:12:39","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi,\n\nOn Wed, Dec 07, 2022 at 01:32:04PM +0100, Robert Mader via libcamera-devel wrote:\n> Hi!\n>\n> I'd love if we could clarify the following part to ensure the Pipewire\n> implementation is correct:\n>\n> > > +\n> > > + /*\n> > > + * The camera sensor cannot do Transpose. Adjust any combined result\n> > > + * that includes a transpose by flipping the transpose bit to notify\n> > > + * applications they either get the transform they requested, or have\n> > > + * to do a simple transpose themselves (they don't have to worry about\n> > > + * the other possible cases).\n> > > + */\n> > > + if (!!(combined & Transform::Transpose)) {\n> > > + /*\n> > > + * Flipping the transpose bit in \"transform\" flips it in the\n> > > + * combined result too (as it's the last thing that happens),\n> > > + * which is of course clearing it.\n> > > + */\n> > > + *transform ^= Transform::Transpose;\n> > > + combined &= ~Transform::Transpose;\n> > > + }\n> > > +\n> > > + /*\n> > > + * If the sensor can do no transforms, then combined must be changed to\n> > > + * the identity and the sensor rotation must be cleared from the\n> > > + * requested \"transform\".\n> >\n> > Don't really follow this comment. Maybe\n> >\n> > * If the sensor cannot do transforms, then combined must be changed to\n> > * the identity and the only transform the user can have is the sensor\n> > * rotation itself.\n> >\n> > (Which rather suggests the code following has got the transform the\n> > wrong way round...)\n> >\n\nI should have replied earlier to David on the patch series for this\none.\n\nWhat I actually meant is \"If the sensor cannot do any flip\" and the\ncode has been copied from\nhttps://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp#n402\n\nWill reply to rest of the discussion below in a separate email.\n\n> > > + */\n> > > + if (!supportFlips_ && !!combined) {\n> > > + *transform = -rotationTransform;\n> >\n> > This might really be \"rotationTransform\" (without the inverse).\n> > Depending on which we round we view the sensor rotation (from in front\n> > of or behind the camera).\n> >\n> > Also, as we remarked elsewhere, we should possibly consider the\n> > existence of H and V flips separately.\n> >\n> > But apart from my existential doubts about inverse rotations, it all\n> > seems reasonable to me, though I sense we'll be coming back to this\n> > again anyway:\n>\n> As example I'd like to take the back-camera of the Pinephone Pro (imx258).\n> Pipewire will ask for an identity transform and the kernel will most likely\n> report `V4L2_CID_CAMERA_SENSOR_ROTATION == 270` (once driver patches and\n> device tree updates land), so the initial values we'll get are \n> `properties::Rotation == 270` -> `combined == Transform::Rot270`.\n>\n> Right now neither transpose nor flips are supported, so the steps in the\n> current revision are:\n>\n> 1. `combined == rot270` / `transform == identity`\n> 2. `combined == hflip` / `transform == transpose`\n> 3. `combined == `identity` / `transform == rot90`\n>\n> The value from `transform` will then be used for\n> `CameraConfiguration::transform`, signalling to the client that it will have\n> to compensate accordingly. Following the comment to not use the inverse it\n> would be `rot270` instead.\n>\n> From an API-point of view, I'd say `rot90` is the more intuitive value for\n> clients, because they can then just follow the documentation in\n> camera_sensor.cpp and apply a transformation accordingly. Would you agree\n> that this should be the desired outcome? I.e. should we stick to the\n> proposed value?\n>\n> Best regards,\n>\n> Robert","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 73F93BE08B\n\tfor ;\n\tFri, 9 Dec 2022 11:12:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C6EF46334B;\n\tFri, 9 Dec 2022 12:12:41 +0100 (CET)","from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net\n\t[217.70.183.195])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CCDF361F1B\n\tfor ;\n\tFri, 9 Dec 2022 12:12:40 +0100 (CET)","(Authenticated sender: jacopo@jmondi.org)\n\tby mail.gandi.net (Postfix) with ESMTPSA id 4CAFC60007;\n\tFri, 9 Dec 2022 11:12:39 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670584361;\n\tbh=kyhYPOKl4IT84Nz/5t4+Ap1cUqZp96z8okQCl4uI90U=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=Dew3cgfn1rVR67hy7VzErPF/7Ldbmr8S8g3C3RX95jReWmcmff2qpsgkdS5YozvUx\n\tU14/L7yelvyEMwRBVvB0Z4Qh4GT97WcvEa7aCYvO273a6OPVbTBBddZNxTTbzOikMi\n\t9CSI6m6MMYkSwToO8w6DM/QLMlphWEkx5WqWhIG7wd2AVacFApeZfQmO5zHU6nPYpK\n\tLEj8jEnWZxfWFLXBSWd8IfZ4hRtxV7810+zqcY7Ft20p+yUWa1yVZxvTE/GnkdS2xt\n\tyIwlvoLiAOdE9VF+ENIDXq8w0GFxIfHlA/N6ParVykQxdAvojlOjZiLvMMvjtbNtPi\n\tbriYhODFl/J+A==","Date":"Fri, 9 Dec 2022 12:12:39 +0100","To":"Robert Mader ","Message-ID":"<20221209111239.q6xgvaqhy4zdm4nl@uno.localdomain>","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jacopo Mondi via libcamera-devel ","Reply-To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26034,"web_url":"https://patchwork.libcamera.org/comment/26034/","msgid":"<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>","date":"2022-12-09T12:47:46","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi again,\n\nOn Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n> On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > Hi Robert\n> >\n> > Thanks for prodding us all on this question!\n>\n> Hi, thanks for the quick reply! And a pleasure :P\n>\n> > I looked back over this again, and found Jacopo's description of the\n> > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > is what I stumbled into)\n>\n> The current kernel docu can be found at\n>\n> https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n>\n> the libcamera one at\n>\n> https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n>\n> > From that, and the nice example with the stick person, I deduce that\n> > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> > degrees, then the image you capture (if nothing else happens to it)\n> > will look like it has had a 90 degree clockwise rotation performed.\n> > Everyone agree with that?\n> Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> translate to Transform::Rot90.\n\nLet's here put some definitions in place\n\nV4L2_CID_CAMERA_SENSOR_ROTATION\nThis read-only control describes the rotation correction in degrees in\nthe counter-clockwise direction to be applied to the captured images\nonce captured to memory to compensate for the camera sensor mounting\nrotation.\n\n \\var Transform::Rot270\n Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n\n Let's start by saying we did a sub-optimal (to be nice) job in\n having Transform and V4L2_CID_ROTATION operate in two different\n directions. Before the usage of Transform in libcamera gets widely\n adopted sould we align the two ?\n\n Hence if I'm not mistaken, if your camera has Rotation=90 and an\n application supplies Transform::Identity in\n CameraConfiguration::transform what you should get back is Rot270\n\n Correct ?\n\n> > So in turn, if you have a camera/ISP that can apply no transforms at\n> > all, then the only value for the user transform that will not come\n> > back as adjusted is \"rot90\", because it looks like it's had a 90\n> > degree clockwise rotation performed (and libcamera transforms count\n> > rotations as being clockwise). Does that also make sense?\n\nIt might be the case today, but I wonder if that's desirable.\n\nIf an application supplies CameraConfiguration::transform = Identity\nit means it doesn't want any rotation applied by the library. It will\nget back 270 to indicate that that's how the image is rotated and will\nget an Adjusted configuration.\n\nIf instead an application supplies CameraConfiguration::transform = Rot270\nand your camera -cannot- do any rotation, it will still receive back\nRot270 and the state is Valid.\n\nIf it asks for CameraConfiguration::transform = Rot270 and the camera\n-can- do that, it will clear CameraConfiguration::transform to\nIdentity and return Adjusted ?\n\nIs this what happens (I'm looking at the above code with David's\nsuggestion to use the inverse of roationTransform).\n\n\n> >\n> > So by a slightly arbitrary mixture of conventions, it looks to me as\n> > though the use of the inverse that I queried is indeed wrong, because\n> > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> > case.\n> >\n> > Does that sound plausible... what do folks think?\n>\n> Ah, I think the question is whether the adjusted value means: \"this is what\n> you'll get\" vs \"this is what you have to do yourself\".\n>\n> Above I argued in favor of the later, but I guess the first one is what the\n> API is supposed to mean. That would imply that the client has compute the\n> difference between requested and adjusted value itself.\n\nI always presumed the latter was meant to happen..\n\n>\n> In my case that would mean:\n>\n> requested CameraConfiguration::transform: Transform::Identity,\n> V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> CameraConfiguration::transform: Transform::Rot270\n>\n\nI guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n\n> To compute what the client has to do, it would need to do:\n> Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> that, a 90 degr. rotation clockwise.\n>\n> Does that sound reasonable? In that case the patch here should indeed be\n> `*transform = rotationTransform;`, without the inverse.\n>\n\n /0\\\n\nbefore we plumb this into and make assumptions that will be then\nset into stone in our adaption layers, can we take a step back and\ndefine what kind of API we would like to have ?\n\nLet's start by the definition of CameraConfiguration::transform.\n\nIt's a member of CameraConfiguration, and as by definition it is\nfilled-in by the application and eventually adjusted by the Camera to\na value that's acceptable/doable for the current configuration.\n\n1) Application to Camera = (counter?)clockwise degrees the camera\nsystem should rotate the image before presenting it to the user\n\n2) Camera to application = the rotation the camera can actually do\n(ie. only sensor flips == no Transpose)\n\nThe current documentation doesn't explain how it gets adjusted\n\n/**\n * \\var CameraConfiguration::transform\n * \\brief User-specified transform to be applied to the image\n *\n * The transform is a user-specified 2D plane transform that will be applied\n * to the camera images by the processing pipeline before being handed to\n * the application. This is subsequent to any transform that is already\n * required to fix up any platform-defined rotation.\n *\n * The usual 2D plane transforms are allowed here (horizontal/vertical\n * flips, multiple of 90-degree rotations etc.), but the validate() function\n * may adjust this field at its discretion if the selection is not supported.\n */\n\nThe last part in particular, \"at its discrection\" it's really too\ngeneric.\n\nLet's start from the beginning: what application should use transform\nfor ? I presume they should:\n\n1) Inspect propertis::Rotation\n2) Correct the image for the properties::Rotation amount (actually,\nthe inverse because of the clockwise/counter-clockwise thing)\n3) Add any additional rotation they would like on top of this\n\nHow should transform be adjusted ?\n\nIdentity:\nIf application->identity the camera does not have to apply any\ntransform. Should the transform be adjutes like it happens today to\ncompensate the Rotation (at least that's what I presume happens) ? I\npresume no, applications can infer rotation from properties and if they\nchose Identity either other layers above will do the transform or\nthey're fine with images as they are.\n\nA supported/non-supported transform:\nAn application asks for a VFLIP, the camera can do it, transform is\nnot changed. If the camera cannot flip, the transform is set back to\nIdentity and the state is adjusted.\n\nA partially non-supported transform:\nAn application asks for Rot270 (Transpose | HFLIP) and the camera can\nonly do HFLIP. transform sould be then set to HFLIP and applications\nknow that the remaining transformation should be handled elsewhere.\n\nAre we missing anything with such behaviour ? I presume it's rather\nsimilar to what happens today if not for the fact that the validate()\nimplementation adjusts transform to rotation in case it cannot flip\n\n Transform combined = transform * data_->rotationTransform_;\n if (!data_->supportsFlips_ && !!combined)\n transform = -data_->rotationTransform_;\n\nNot really sure I got why.\n\nAll of this, but an even more fundamental question: is it too late/is\nit worh it to adjust the Transform definition to adopt the same\ndirection as the kernel defined rotation ?\n\nSorry for the long email :/\n\n> > Thanks everyone!\n> > David\n>\n> Thanks!\n>\n> Robert\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 60F58BDE6B\n\tfor ;\n\tFri, 9 Dec 2022 12:47:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AF5826334B;\n\tFri, 9 Dec 2022 13:47:49 +0100 (CET)","from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net\n\t[217.70.183.199])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EE55F61F1B\n\tfor ;\n\tFri, 9 Dec 2022 13:47:47 +0100 (CET)","(Authenticated sender: jacopo@jmondi.org)\n\tby mail.gandi.net (Postfix) with ESMTPSA id 6AF7FFF815;\n\tFri, 9 Dec 2022 12:47:47 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670590069;\n\tbh=skmTO6Zl4AzU0sN19GjcknDdPjlP2z2Vl8aZIhJ9IZ8=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=TXCfHW8Ni2S75dbMc6T37ClAt4kbNRdXL8W+x1LErRmosraWUTCCqScuXMK0TXKUn\n\tUBdmv+hPspGAbkqIabCTnLBYeNt4yDku03W3BMXr2JV3C7YTxXZ37HsDIBYETWKJaD\n\tTyrTn/E9qfRL6BAGAWr6ARLatubwur+GWIvrWEgNDV8oCWcFBzkE9qdDHOm4vArxEx\n\t8kYCBeoRShM+9/hsrw+EAW3l7UHsqpIdWRP56yWp3Mt6KaSmISJtzdneTwDDVFTSQ1\n\tzxgngYm/E8M5pnIv+qYwiSSvtXUy2sS9Ltg46/rN5WnmXxLT7PXuKsFmACz9Zbsnez\n\txhbNItqLfok4g==","Date":"Fri, 9 Dec 2022 13:47:46 +0100","To":"Robert Mader ","Message-ID":"<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jacopo Mondi via libcamera-devel ","Reply-To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26046,"web_url":"https://patchwork.libcamera.org/comment/26046/","msgid":"","date":"2022-12-12T11:46:31","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Jacopo\n\nThanks for wading into the sea on this question! (Very appropriate now\nthat the stick person example has been turned into a shark...!)\n\nOn Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n wrote:\n>\n> Hi again,\n>\n> On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n> > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > > Hi Robert\n> > >\n> > > Thanks for prodding us all on this question!\n> >\n> > Hi, thanks for the quick reply! And a pleasure :P\n> >\n> > > I looked back over this again, and found Jacopo's description of the\n> > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > > is what I stumbled into)\n> >\n> > The current kernel docu can be found at\n> >\n> > https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n\nThanks for the updated reference. The stick person may have turned\ninto a shark, but apart from that I think the sense of the\ndocumentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n90 degrees, then your image in memory looks like it has had a 90\ndegree *clockwise* rotation applied.\n\n> >\n> > the libcamera one at\n> >\n> > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> >\n> > > From that, and the nice example with the stick person, I deduce that\n> > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> > > degrees, then the image you capture (if nothing else happens to it)\n> > > will look like it has had a 90 degree clockwise rotation performed.\n> > > Everyone agree with that?\n> > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> > translate to Transform::Rot90.\n>\n> Let's here put some definitions in place\n>\n> V4L2_CID_CAMERA_SENSOR_ROTATION\n> This read-only control describes the rotation correction in degrees in\n> the counter-clockwise direction to be applied to the captured images\n> once captured to memory to compensate for the camera sensor mounting\n> rotation.\n>\n> \\var Transform::Rot270\n> Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n>\n> Let's start by saying we did a sub-optimal (to be nice) job in\n> having Transform and V4L2_CID_ROTATION operate in two different\n> directions. Before the usage of Transform in libcamera gets widely\n> adopted sould we align the two ?\n\nI don't mind at all... we just have to decide what we want!\n\n>\n> Hence if I'm not mistaken, if your camera has Rotation=90 and an\n> application supplies Transform::Identity in\n> CameraConfiguration::transform what you should get back is Rot270\n>\n> Correct ?\n\nThe way things are currently, if the camera says Rotation=90, then\nyour image in memory looks like it is rotated 90 degrees clockwise, so\nthe Transform (if your pipeline and sensor can't apply any\nflips/rotations at all) will come back as Rot90.\n\nThat is, the only transform you can specify that won't be adjusted, is\na 90 degree clockwise rotation, so transform=Rot90 as things stand\ncurrently.\n\nIf we choose to change the sense of the libcamera.Transform rotation,\nthen we'd get transform=Rot270 instead.\n\n (If your pipeline handler can do flips and transposes, it may be able\nto do what you ask and so the transform would come back unchanged and\nwould be \"Valid\".)\n\n>\n> > > So in turn, if you have a camera/ISP that can apply no transforms at\n> > > all, then the only value for the user transform that will not come\n> > > back as adjusted is \"rot90\", because it looks like it's had a 90\n> > > degree clockwise rotation performed (and libcamera transforms count\n> > > rotations as being clockwise). Does that also make sense?\n>\n> It might be the case today, but I wonder if that's desirable.\n>\n> If an application supplies CameraConfiguration::transform = Identity\n> it means it doesn't want any rotation applied by the library. It will\n> get back 270 to indicate that that's how the image is rotated and will\n> get an Adjusted configuration.\n\nI'm not quite sure I get that, perhaps it depends how we interpret\n\"applied by the library\". I guess I agree if we mean \"applied by the\nlibrary together with any rotation of the sensor\"? Maybe a real\nexample is helpful:\n\nSo for a Raspberry Pi, for example, our sensors are mostly mounted\nupside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\nsay the transform they want is transform=Identity i.e. images come\nback \"the right way up\". Internally it means that we do apply an extra\nh+v flip to undo the 180 degree rotation of the sensor.\n\n>\n> If instead an application supplies CameraConfiguration::transform = Rot270\n> and your camera -cannot- do any rotation, it will still receive back\n> Rot270 and the state is Valid.\n\nI think that's true if we change the sense of the libcamera.Transform\nrotation. Currently, as I said earlier, I think we get Rot90?\n\n>\n> If it asks for CameraConfiguration::transform = Rot270 and the camera\n> -can- do that, it will clear CameraConfiguration::transform to\n> Identity and return Adjusted ?\n\nI'm not sure there... The CameraConfiguration::transform is the final\ntransform you want. If that can be done, it's left alone and you get\nback Valid. It's not telling you what else you would need to do\nafterwards.\n\n(To find out what you need to do afterwards you'd work out\n\"inverse(what-you-will-get) * what-you-wanted\", or something like\nthat.)\n\n>\n> Is this what happens (I'm looking at the above code with David's\n> suggestion to use the inverse of roationTransform).\n>\n>\n> > >\n> > > So by a slightly arbitrary mixture of conventions, it looks to me as\n> > > though the use of the inverse that I queried is indeed wrong, because\n> > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> > > case.\n> > >\n> > > Does that sound plausible... what do folks think?\n> >\n> > Ah, I think the question is whether the adjusted value means: \"this is what\n> > you'll get\" vs \"this is what you have to do yourself\".\n> >\n> > Above I argued in favor of the later, but I guess the first one is what the\n> > API is supposed to mean. That would imply that the client has compute the\n> > difference between requested and adjusted value itself.\n>\n> I always presumed the latter was meant to happen..\n\nYes, I think this is the crux of the matter. I believe it's \"what you\nwill get\". It's not \"what you need to do after\". (If it was \"what you\nneed to do after\", it would change every time you try to validate the\nconfiguration!)\n\n>\n> >\n> > In my case that would mean:\n> >\n> > requested CameraConfiguration::transform: Transform::Identity,\n> > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > CameraConfiguration::transform: Transform::Rot270\n> >\n>\n> I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n\nYes, looks right to me!\n\n>\n> > To compute what the client has to do, it would need to do:\n> > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> > that, a 90 degr. rotation clockwise.\n\nYes, I guess I wrote it above, it's \"inverse(what-you-got) *\nwhat-you-wanted\", in this case inverse(rot270) * identity which is\nrot90 (in the current clockwise sense).\n\n> >\n> > Does that sound reasonable? In that case the patch here should indeed be\n> > `*transform = rotationTransform;`, without the inverse.\n> >\n\nYes, agree!\n\n>\n> /0\\\n>\n> before we plumb this into and make assumptions that will be then\n> set into stone in our adaption layers, can we take a step back and\n> define what kind of API we would like to have ?\n>\n> Let's start by the definition of CameraConfiguration::transform.\n>\n> It's a member of CameraConfiguration, and as by definition it is\n> filled-in by the application and eventually adjusted by the Camera to\n> a value that's acceptable/doable for the current configuration.\n>\n> 1) Application to Camera = (counter?)clockwise degrees the camera\n> system should rotate the image before presenting it to the user\n\nThis may depend on what exactly we mean by \"the camera system\". I'd\nexplain it as\n\"the final resulting transform that the application wants to be\napplied to all the output images\". After all, this is actually the\nonly thing the application knows!\n\n>\n> 2) Camera to application = the rotation the camera can actually do\n> (ie. only sensor flips == no Transpose)\n\nAgree, mostly, though I would say \"transform\" rather than \"rotation\"\nbecause flips aren't rotations, you could in theory have a PH that\ncould do transposes (even though we probably don't currently).\n\n>\n> The current documentation doesn't explain how it gets adjusted\n\nThis is true, though I also think it's platform dependent. If you had\na PH that can't do transposes but can do flips, you have a choice how\nyou handle (for example) rot90. You could say \"oh I can't do that, so\nin this case I won't do anything at all\" and leave the application to\nsort everything out. The Pi tries to say \"well, I'll do everything\nexcept for the transpose, so an application only has to worry about\nthat one case\". But there is a choice...\n\n>\n> /**\n> * \\var CameraConfiguration::transform\n> * \\brief User-specified transform to be applied to the image\n> *\n> * The transform is a user-specified 2D plane transform that will be applied\n> * to the camera images by the processing pipeline before being handed to\n> * the application. This is subsequent to any transform that is already\n> * required to fix up any platform-defined rotation.\n> *\n> * The usual 2D plane transforms are allowed here (horizontal/vertical\n> * flips, multiple of 90-degree rotations etc.), but the validate() function\n> * may adjust this field at its discretion if the selection is not supported.\n> */\n>\n> The last part in particular, \"at its discrection\" it's really too\n> generic.\n\nI guess it's hinting at the fact that it's up to the PH, and we don't\nmandate the exact way in which it can fail to do what you asked for!\nBut it would be fine to have a discussion on that.\n\n>\n> Let's start from the beginning: what application should use transform\n> for ? I presume they should:\n>\n> 1) Inspect propertis::Rotation\n> 2) Correct the image for the properties::Rotation amount (actually,\n> the inverse because of the clockwise/counter-clockwise thing)\n> 3) Add any additional rotation they would like on top of this\n\nSo I think the point of the transform is that you *don't* have to\nworry about the properties::Rotation. It takes care of that for you.\nYou say \"I want upside down images\" and you get them, whatever the\nsensor's rotation.\n\n>\n> How should transform be adjusted ?\n>\n> Identity:\n> If application->identity the camera does not have to apply any\n> transform. Should the transform be adjutes like it happens today to\n> compensate the Rotation (at least that's what I presume happens) ? I\n> presume no, applications can infer rotation from properties and if they\n> chose Identity either other layers above will do the transform or\n> they're fine with images as they are.\n\nIf the application says it wants the \"identity\" transform, then it\nshould get output images with no transformation applied. But this does\nmean it will undo the sensor rotation (this being the normal case on\nthe Pi).\n\n>\n> A supported/non-supported transform:\n> An application asks for a VFLIP, the camera can do it, transform is\n> not changed. If the camera cannot flip, the transform is set back to\n> Identity and the state is adjusted.\n\nYes, mostly, except that the sensor may be mounted with a rotation,\nbut if it isn't, then I agree with this.\n\n>\n> A partially non-supported transform:\n> An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> only do HFLIP. transform sould be then set to HFLIP and applications\n> know that the remaining transformation should be handled elsewhere.\n\nAs I suggested above, I'm not sure whether we want to mandate this. I\nagree it seems useful behaviour, which is why the Pi does it, but I'm\nopen minded about requiring it or not.\n\n>\n> Are we missing anything with such behaviour ? I presume it's rather\n> similar to what happens today if not for the fact that the validate()\n> implementation adjusts transform to rotation in case it cannot flip\n>\n> Transform combined = transform * data_->rotationTransform_;\n> if (!data_->supportsFlips_ && !!combined)\n> transform = -data_->rotationTransform_;\n>\n> Not really sure I got why.\n>\n> All of this, but an even more fundamental question: is it too late/is\n> it worh it to adjust the Transform definition to adopt the same\n> direction as the kernel defined rotation ?\n>\n> Sorry for the long email :/\n\nNo problem! I like nothing more than discussing transforms, except\npossibly colour spaces!\n\nMore seriously, here are my key points:\n\n1. The CameraConfiguration::transform field says what the application\nwants the final output images to have.\n2. It takes account of the sensor rotation. The most obvious example\nis the Pi: if the sensor is mounted upside down, and the applications\nask for images \"the right way up\" (i.e. transform = identity), then\nthe PH will apply h and v flips to compensate for the sensor mounting.\n3. If the PH can do what the application requests, it leaves the\ntransform alone and your configuration is Valid.\n4. If the PH can't do what the application wants, it will set the\nvalue to something that it can do, and the configuration will be\nAdjusted.\n\nIf folks wanted a short conference call to iron any of this out, I'd\nbe very happy to join in!\n\nThanks\nDavid\n\n>\n> > > Thanks everyone!\n> > > David\n> >\n> > Thanks!\n> >\n> > Robert\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 634F7C31E9\n\tfor ;\n\tMon, 12 Dec 2022 11:46:46 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B244E63362;\n\tMon, 12 Dec 2022 12:46:45 +0100 (CET)","from mail-pg1-x533.google.com (mail-pg1-x533.google.com\n\t[IPv6:2607:f8b0:4864:20::533])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6088E6333F\n\tfor ;\n\tMon, 12 Dec 2022 12:46:44 +0100 (CET)","by mail-pg1-x533.google.com with SMTP id f9so7987858pgf.7\n\tfor ;\n\tMon, 12 Dec 2022 03:46:44 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670845605;\n\tbh=EWtFpJx38o/mrwcNbLrnQbj9wDWw5Tx2GkThwvEq9kM=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=HY6E/r524QPGFJ5i4YUmwVd2ZLccA7bB5FkhY3BtOOs9y8NibYVXgJ8ZYHFT78QAx\n\tbm5P+NXKB8jxOnv1fkmGJl9ZXKAnwxebg5dGoUY9aV6RTPb13Xa7k5G4onmrin2yov\n\tX4X/GX725TfhQuDQvlCwQWselu9YDegHxAABD5viBUNiUpLtnhBg0OW/BBZb4C0L+9\n\ts2vcmjkpLyy6JsjgIYYqsRulioz5o+Ptmm/LvzXaKhqhWIaRoSq7BPJCkADVi3g7tG\n\tvJ4Oy8EQnfkyFJ3AtrtZafu2YFRrdIzRsgsH3DyJsjVwuWPbREphvzhMBlmhcoMauK\n\tI2LuRHvws4ENA==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=bUOUAjKUIps29Gb2gGoTQm6z+H1V9XY84LreKOywMew=;\n\tb=s4uIP5+Q0mDw+3+kb8PDQ242P2bR89oYhp0b7lctJITA0u/GVcgyZe1iz+rwjGQspd\n\tmH75NTTOOtCU7iSzcyffgnZha4SD4bBnezC5piXoMroUczwOjTlU9QVzOwlTweStStob\n\tvM4QJDscdg+doB6XZI5QPN8N6nzkQk6SByfu3dKh3bhZc2JgH+sEkwQpBAYoBIr3DkaC\n\t12xjvscaT6Qb32hUBnoFC/o73c1d3p4nWmtoQyIxhcOCOX8QBTT1nOADLUNAeJrQqeoR\n\tGQN/C7QKOGFG9iqzfTHPX3/Qy+O5eEzngbOG86TvUVaThYEsXm+nWO/Wr9kfiSN7yMsQ\n\tFuNQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"s4uIP5+Q\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=bUOUAjKUIps29Gb2gGoTQm6z+H1V9XY84LreKOywMew=;\n\tb=vJd02MtJBUpmcXiRbXmHIuBcWBVqRNSvdukFeG5gW9BgL+svGlrlIN7Y2p6PDMLNAY\n\trqJ5/ZOOddep7qwqPZ1CYXOhaSPQFCy8z7Ll57Vw4u6QaFBlsCQRHXSMdG2FHIr+7CGK\n\tcMdO5Drx7a7LyrfNGSfLwdDjrb8LZdfogK0ckV0f35c2RVdyucNXo60cwvk1T87DMU4N\n\tD1EVQwtJpxKSWckTtTWfRcsn/YR0hQVAOum0aj7+WbYcFGiHXZcLqIXd0ayo9yqtfyr8\n\tJlLIcsRO2rcZYXjybgR1jiKIAR43+qMPDl0oSzTrxXyW6TjafFlgqormLht6oBxIae93\n\tK4Kw==","X-Gm-Message-State":"ANoB5pkAIgLmBJRF4xTXF3jw0pNfJkNoViufeJW6LJI1INLri6AOtCvC\n\t05HiGUOohwljkB0R6bcHWd+XihOmb2F3fnL3apAbqA==","X-Google-Smtp-Source":"AA0mqf73/B6N74ab1PgDYGLNyafsk9q/kAUU028N/jn/jEqkJDSYXPPLbmIci7JZpmh+k1MJCSWOyitJjg8NrF0dvZQ=","X-Received":"by 2002:a63:5b48:0:b0:478:b930:a35c with SMTP id\n\tl8-20020a635b48000000b00478b930a35cmr17054926pgm.494.1670845602680;\n\tMon, 12 Dec 2022 03:46:42 -0800 (PST)","MIME-Version":"1.0","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>","In-Reply-To":"<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>","Date":"Mon, 12 Dec 2022 11:46:31 +0000","Message-ID":"","To":"Jacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26047,"web_url":"https://patchwork.libcamera.org/comment/26047/","msgid":"","date":"2022-12-12T12:33:07","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":27,"url":"https://patchwork.libcamera.org/api/people/27/","name":"Dave Stevenson","email":"dave.stevenson@raspberrypi.com"},"content":"Hi David & Jacopo\n\nOn Mon, 12 Dec 2022 at 11:46, David Plowman via libcamera-devel\n wrote:\n>\n> Hi Jacopo\n>\n> Thanks for wading into the sea on this question! (Very appropriate now\n> that the stick person example has been turned into a shark...!)\n>\n> On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n> wrote:\n> >\n> > Hi again,\n> >\n> > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n> > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > > > Hi Robert\n> > > >\n> > > > Thanks for prodding us all on this question!\n> > >\n> > > Hi, thanks for the quick reply! And a pleasure :P\n> > >\n> > > > I looked back over this again, and found Jacopo's description of the\n> > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > > > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > > > is what I stumbled into)\n> > >\n> > > The current kernel docu can be found at\n> > >\n> > > https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n>\n> Thanks for the updated reference. The stick person may have turned\n> into a shark, but apart from that I think the sense of the\n> documentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n> 90 degrees, then your image in memory looks like it has had a 90\n> degree *clockwise* rotation applied.\n>\n> > >\n> > > the libcamera one at\n> > >\n> > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> > >\n> > > > From that, and the nice example with the stick person, I deduce that\n> > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> > > > degrees, then the image you capture (if nothing else happens to it)\n> > > > will look like it has had a 90 degree clockwise rotation performed.\n> > > > Everyone agree with that?\n> > > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> > > translate to Transform::Rot90.\n> >\n> > Let's here put some definitions in place\n> >\n> > V4L2_CID_CAMERA_SENSOR_ROTATION\n> > This read-only control describes the rotation correction in degrees in\n> > the counter-clockwise direction to be applied to the captured images\n> > once captured to memory to compensate for the camera sensor mounting\n> > rotation.\n> >\n> > \\var Transform::Rot270\n> > Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n> >\n> > Let's start by saying we did a sub-optimal (to be nice) job in\n> > having Transform and V4L2_CID_ROTATION operate in two different\n> > directions. Before the usage of Transform in libcamera gets widely\n> > adopted sould we align the two ?\n>\n> I don't mind at all... we just have to decide what we want!\n>\n> >\n> > Hence if I'm not mistaken, if your camera has Rotation=90 and an\n> > application supplies Transform::Identity in\n> > CameraConfiguration::transform what you should get back is Rot270\n> >\n> > Correct ?\n>\n> The way things are currently, if the camera says Rotation=90, then\n> your image in memory looks like it is rotated 90 degrees clockwise, so\n> the Transform (if your pipeline and sensor can't apply any\n> flips/rotations at all) will come back as Rot90.\n>\n> That is, the only transform you can specify that won't be adjusted, is\n> a 90 degree clockwise rotation, so transform=Rot90 as things stand\n> currently.\n>\n> If we choose to change the sense of the libcamera.Transform rotation,\n> then we'd get transform=Rot270 instead.\n>\n> (If your pipeline handler can do flips and transposes, it may be able\n> to do what you ask and so the transform would come back unchanged and\n> would be \"Valid\".)\n>\n> >\n> > > > So in turn, if you have a camera/ISP that can apply no transforms at\n> > > > all, then the only value for the user transform that will not come\n> > > > back as adjusted is \"rot90\", because it looks like it's had a 90\n> > > > degree clockwise rotation performed (and libcamera transforms count\n> > > > rotations as being clockwise). Does that also make sense?\n> >\n> > It might be the case today, but I wonder if that's desirable.\n> >\n> > If an application supplies CameraConfiguration::transform = Identity\n> > it means it doesn't want any rotation applied by the library. It will\n> > get back 270 to indicate that that's how the image is rotated and will\n> > get an Adjusted configuration.\n>\n> I'm not quite sure I get that, perhaps it depends how we interpret\n> \"applied by the library\". I guess I agree if we mean \"applied by the\n> library together with any rotation of the sensor\"? Maybe a real\n> example is helpful:\n>\n> So for a Raspberry Pi, for example, our sensors are mostly mounted\n> upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\n> say the transform they want is transform=Identity i.e. images come\n> back \"the right way up\". Internally it means that we do apply an extra\n> h+v flip to undo the 180 degree rotation of the sensor.\n>\n> >\n> > If instead an application supplies CameraConfiguration::transform = Rot270\n> > and your camera -cannot- do any rotation, it will still receive back\n> > Rot270 and the state is Valid.\n>\n> I think that's true if we change the sense of the libcamera.Transform\n> rotation. Currently, as I said earlier, I think we get Rot90?\n>\n> >\n> > If it asks for CameraConfiguration::transform = Rot270 and the camera\n> > -can- do that, it will clear CameraConfiguration::transform to\n> > Identity and return Adjusted ?\n>\n> I'm not sure there... The CameraConfiguration::transform is the final\n> transform you want. If that can be done, it's left alone and you get\n> back Valid. It's not telling you what else you would need to do\n> afterwards.\n>\n> (To find out what you need to do afterwards you'd work out\n> \"inverse(what-you-will-get) * what-you-wanted\", or something like\n> that.)\n>\n> >\n> > Is this what happens (I'm looking at the above code with David's\n> > suggestion to use the inverse of roationTransform).\n> >\n> >\n> > > >\n> > > > So by a slightly arbitrary mixture of conventions, it looks to me as\n> > > > though the use of the inverse that I queried is indeed wrong, because\n> > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> > > > case.\n> > > >\n> > > > Does that sound plausible... what do folks think?\n> > >\n> > > Ah, I think the question is whether the adjusted value means: \"this is what\n> > > you'll get\" vs \"this is what you have to do yourself\".\n> > >\n> > > Above I argued in favor of the later, but I guess the first one is what the\n> > > API is supposed to mean. That would imply that the client has compute the\n> > > difference between requested and adjusted value itself.\n> >\n> > I always presumed the latter was meant to happen..\n>\n> Yes, I think this is the crux of the matter. I believe it's \"what you\n> will get\". It's not \"what you need to do after\". (If it was \"what you\n> need to do after\", it would change every time you try to validate the\n> configuration!)\n>\n> >\n> > >\n> > > In my case that would mean:\n> > >\n> > > requested CameraConfiguration::transform: Transform::Identity,\n> > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > > CameraConfiguration::transform: Transform::Rot270\n> > >\n> >\n> > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n>\n> Yes, looks right to me!\n>\n> >\n> > > To compute what the client has to do, it would need to do:\n> > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> > > that, a 90 degr. rotation clockwise.\n>\n> Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> what-you-wanted\", in this case inverse(rot270) * identity which is\n> rot90 (in the current clockwise sense).\n>\n> > >\n> > > Does that sound reasonable? In that case the patch here should indeed be\n> > > `*transform = rotationTransform;`, without the inverse.\n> > >\n>\n> Yes, agree!\n>\n> >\n> > /0\\\n> >\n> > before we plumb this into and make assumptions that will be then\n> > set into stone in our adaption layers, can we take a step back and\n> > define what kind of API we would like to have ?\n> >\n> > Let's start by the definition of CameraConfiguration::transform.\n> >\n> > It's a member of CameraConfiguration, and as by definition it is\n> > filled-in by the application and eventually adjusted by the Camera to\n> > a value that's acceptable/doable for the current configuration.\n> >\n> > 1) Application to Camera = (counter?)clockwise degrees the camera\n> > system should rotate the image before presenting it to the user\n>\n> This may depend on what exactly we mean by \"the camera system\". I'd\n> explain it as\n> \"the final resulting transform that the application wants to be\n> applied to all the output images\". After all, this is actually the\n> only thing the application knows!\n>\n> >\n> > 2) Camera to application = the rotation the camera can actually do\n> > (ie. only sensor flips == no Transpose)\n>\n> Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n> because flips aren't rotations, you could in theory have a PH that\n> could do transposes (even though we probably don't currently).\n>\n> >\n> > The current documentation doesn't explain how it gets adjusted\n>\n> This is true, though I also think it's platform dependent. If you had\n> a PH that can't do transposes but can do flips, you have a choice how\n> you handle (for example) rot90. You could say \"oh I can't do that, so\n> in this case I won't do anything at all\" and leave the application to\n> sort everything out. The Pi tries to say \"well, I'll do everything\n> except for the transpose, so an application only has to worry about\n> that one case\". But there is a choice...\n>\n> >\n> > /**\n> > * \\var CameraConfiguration::transform\n> > * \\brief User-specified transform to be applied to the image\n> > *\n> > * The transform is a user-specified 2D plane transform that will be applied\n> > * to the camera images by the processing pipeline before being handed to\n> > * the application. This is subsequent to any transform that is already\n> > * required to fix up any platform-defined rotation.\n> > *\n> > * The usual 2D plane transforms are allowed here (horizontal/vertical\n> > * flips, multiple of 90-degree rotations etc.), but the validate() function\n> > * may adjust this field at its discretion if the selection is not supported.\n> > */\n> >\n> > The last part in particular, \"at its discrection\" it's really too\n> > generic.\n>\n> I guess it's hinting at the fact that it's up to the PH, and we don't\n> mandate the exact way in which it can fail to do what you asked for!\n> But it would be fine to have a discussion on that.\n>\n> >\n> > Let's start from the beginning: what application should use transform\n> > for ? I presume they should:\n> >\n> > 1) Inspect propertis::Rotation\n> > 2) Correct the image for the properties::Rotation amount (actually,\n> > the inverse because of the clockwise/counter-clockwise thing)\n> > 3) Add any additional rotation they would like on top of this\n>\n> So I think the point of the transform is that you *don't* have to\n> worry about the properties::Rotation. It takes care of that for you.\n> You say \"I want upside down images\" and you get them, whatever the\n> sensor's rotation.\n>\n> >\n> > How should transform be adjusted ?\n> >\n> > Identity:\n> > If application->identity the camera does not have to apply any\n> > transform. Should the transform be adjutes like it happens today to\n> > compensate the Rotation (at least that's what I presume happens) ? I\n> > presume no, applications can infer rotation from properties and if they\n> > chose Identity either other layers above will do the transform or\n> > they're fine with images as they are.\n>\n> If the application says it wants the \"identity\" transform, then it\n> should get output images with no transformation applied. But this does\n> mean it will undo the sensor rotation (this being the normal case on\n> the Pi).\n>\n> >\n> > A supported/non-supported transform:\n> > An application asks for a VFLIP, the camera can do it, transform is\n> > not changed. If the camera cannot flip, the transform is set back to\n> > Identity and the state is adjusted.\n>\n> Yes, mostly, except that the sensor may be mounted with a rotation,\n> but if it isn't, then I agree with this.\n>\n> >\n> > A partially non-supported transform:\n> > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> > only do HFLIP. transform sould be then set to HFLIP and applications\n> > know that the remaining transformation should be handled elsewhere.\n>\n> As I suggested above, I'm not sure whether we want to mandate this. I\n> agree it seems useful behaviour, which is why the Pi does it, but I'm\n> open minded about requiring it or not.\n>\n> >\n> > Are we missing anything with such behaviour ? I presume it's rather\n> > similar to what happens today if not for the fact that the validate()\n> > implementation adjusts transform to rotation in case it cannot flip\n> >\n> > Transform combined = transform * data_->rotationTransform_;\n> > if (!data_->supportsFlips_ && !!combined)\n> > transform = -data_->rotationTransform_;\n> >\n> > Not really sure I got why.\n> >\n> > All of this, but an even more fundamental question: is it too late/is\n> > it worh it to adjust the Transform definition to adopt the same\n> > direction as the kernel defined rotation ?\n> >\n> > Sorry for the long email :/\n>\n> No problem! I like nothing more than discussing transforms, except\n> possibly colour spaces!\n>\n> More seriously, here are my key points:\n>\n> 1. The CameraConfiguration::transform field says what the application\n> wants the final output images to have.\n> 2. It takes account of the sensor rotation. The most obvious example\n> is the Pi: if the sensor is mounted upside down, and the applications\n> ask for images \"the right way up\" (i.e. transform = identity), then\n> the PH will apply h and v flips to compensate for the sensor mounting.\n> 3. If the PH can do what the application requests, it leaves the\n> transform alone and your configuration is Valid.\n> 4. If the PH can't do what the application wants, it will set the\n> value to something that it can do, and the configuration will be\n> Adjusted.\n>\n> If folks wanted a short conference call to iron any of this out, I'd\n> be very happy to join in!\n>\n> Thanks\n> David\n\nI'll admit to having only skim read a chunk of this, but it seems\nworth throwing in how Android CameraHAL1 dealt with images (largely\nfrom memory of about 10 years ago!).\n\nALL images would have the native orientation of the sensor.\n\nPreview was told about the appropriate rotation to apply via\nsetDisplayOrientation().\nJPEG and MP4 stored the rotation in the EXIF and Composition Matrix\nrespectively.\n(The bit that many apps forgot about was that it didn't rotate the\npixels in a raw buffer, and they made assumptions. I remember fixing\nup the HAL to get around a limitation in the Skype app, and as soon as\nI'd done so they released a fixed app).\n\nThis obviously puts a requirement on the preview and playback\nsubsystems, but those are all pretty much mandatory these days.\nAs long as all the bits of the system know how to orient the images,\nthen it largely becomes a non-question. As long as libcamera has a\ndefined way to get all that metadata, then anything can be achieved.\n\n Dave","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 584E5BDE6B\n\tfor ;\n\tMon, 12 Dec 2022 12:33:26 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C307E63362;\n\tMon, 12 Dec 2022 13:33:25 +0100 (CET)","from mail-ua1-x92e.google.com (mail-ua1-x92e.google.com\n\t[IPv6:2607:f8b0:4864:20::92e])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 804166333F\n\tfor ;\n\tMon, 12 Dec 2022 13:33:24 +0100 (CET)","by mail-ua1-x92e.google.com with SMTP id v21so3134212uam.1\n\tfor ;\n\tMon, 12 Dec 2022 04:33:24 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670848405;\n\tbh=tYOMfhcvj7jVektgYMpIPnQWEGc4RXSTJL434W/LwYI=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=Md7VgWaGlDJ6djNzijjH36vuz329QTgTvq0aMSmVpZaBlhrXS5wuwPKQdo5Rg5AC1\n\tgifbCOo6d+cJQCSwJ0o+cYZ/xsqh2BuSnFxAsUKFjZpTNTiEssD8Y4Gn+EdKjddnfx\n\tt6EmctqVbUH8pb3RGypkhcOBTGLo5qxbrib/R2BO9tsOI0bnsf0nh8qA2cecO/qVLx\n\tgoiogcsbFP4evSEJU/xQ/Y6mcggskFtgXfQ2LqLU0n7FvNJ7M73I8Rq+sxuDwOns81\n\tVAZQpCBnETgQHpwcJ6m4vTs0dB8hGpSG8o3SqAY3supjRe5cTepXKHuQIb4DxwUF9l\n\tSCUHMcY8O1cug==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=PTBrbVPnuOQMsakGCOJH02mb4CWBTqxiAaZILyl7qNA=;\n\tb=Qc1IBuFOuagt7EgBTzq/rL+AcbL2InMF6P8Dlvsz0YpSuTRNqkiL6vHa9uwg53iiFx\n\tfHF3hGypYgM+wMQ5+iu6UPJDwP4J8mJzbYo8eyXhDgPU+z7urxVQMoXsxkUgqf794uAu\n\tpgcku1skhXijM0xqzpUpUbA3P+7qr4hT+O/qGOvNC11NSOXDNQudU1GA/wBe3n9ZbWXr\n\tskyL9s8ZldGEGdnsgEv8AwpE82D7+JUA/AVtafjNYh/MP3oH878sz3GWZd+WlrE23qJV\n\t+LP/5fxKnYg7LeOD+aPbVka9wUnP3mkpgznDPerseDtzkbPrzmegMdCHqqKzjsretDCf\n\tWr7w=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"Qc1IBuFO\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=PTBrbVPnuOQMsakGCOJH02mb4CWBTqxiAaZILyl7qNA=;\n\tb=g3N3kg9DYbxT0s6jv0O2MyEbHwrMMsGjV1FbnnhTWKTOJ50GAldwZwh1X3daiHHAJW\n\tIOiOkwHj3dVlBefhbwpAIR0DfZ0Clb1UaEclZz8xpTWpb2DKWzDB8P5D+wUm2L9Imx7e\n\t/vNbwbVxI5F1JiSJLMS81NI7nOZPFL289nXexdUUYlZy8XWiChvGRsvcxWMQRI8uSXu9\n\tGHGudO+f2HmqzQFUevhGrM8WqaaKlQWCOgHXZuGJL/g3u68MNVdb76LwMsP7VyAsYqzq\n\tTFRZDJP9Pte4VGf5YwRbgLEGaoExsaVCuDJEWe7Gp1bLA966f5nrZL+5DVmMiEPaGF2x\n\tNMHA==","X-Gm-Message-State":"ANoB5pmz6XyjpB1KUY2DFHfnXx1kS+YCdSqR/i/X/AJ59wtbDVcy6lH1\n\t/VFpn23fgaMcWHURlKsWDpg3pYG08zRm4M1gemTJ9g==","X-Google-Smtp-Source":"AA0mqf6ce64a8EUx/ee0l29FNNYWnByzgCpM2LgjkzacsR4Yb2VJ9NEjAhx2xpMkmmS0KDfKTzkImQAkFj7QnPLaRUQ=","X-Received":"by 2002:ab0:630e:0:b0:39f:682f:57df with SMTP id\n\ta14-20020ab0630e000000b0039f682f57dfmr50558443uap.83.1670848403163;\n\tMon, 12 Dec 2022 04:33:23 -0800 (PST)","MIME-Version":"1.0","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t","In-Reply-To":"","Date":"Mon, 12 Dec 2022 12:33:07 +0000","Message-ID":"","To":"David Plowman ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Dave Stevenson via libcamera-devel\n\t","Reply-To":"Dave Stevenson ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26053,"web_url":"https://patchwork.libcamera.org/comment/26053/","msgid":"<20221213112251.jnpgyzitusutkqm6@uno.localdomain>","date":"2022-12-13T11:22:51","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi David\n\nOn Mon, Dec 12, 2022 at 11:46:31AM +0000, David Plowman wrote:\n> Hi Jacopo\n>\n> Thanks for wading into the sea on this question! (Very appropriate now\n> that the stick person example has been turned into a shark...!)\n>\n> On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n> wrote:\n> >\n> > Hi again,\n> >\n> > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n> > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > > > Hi Robert\n> > > >\n> > > > Thanks for prodding us all on this question!\n> > >\n> > > Hi, thanks for the quick reply! And a pleasure :P\n> > >\n> > > > I looked back over this again, and found Jacopo's description of the\n> > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > > > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > > > is what I stumbled into)\n> > >\n> > > The current kernel docu can be found at\n> > >\n> > > https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n>\n> Thanks for the updated reference. The stick person may have turned\n> into a shark, but apart from that I think the sense of the\n> documentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n> 90 degrees, then your image in memory looks like it has had a 90\n> degree *clockwise* rotation applied.\n>\n\nYes!\n\n> > >\n> > > the libcamera one at\n> > >\n> > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> > >\n> > > > From that, and the nice example with the stick person, I deduce that\n> > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> > > > degrees, then the image you capture (if nothing else happens to it)\n> > > > will look like it has had a 90 degree clockwise rotation performed.\n> > > > Everyone agree with that?\n> > > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> > > translate to Transform::Rot90.\n> >\n> > Let's here put some definitions in place\n> >\n> > V4L2_CID_CAMERA_SENSOR_ROTATION\n> > This read-only control describes the rotation correction in degrees in\n> > the counter-clockwise direction to be applied to the captured images\n> > once captured to memory to compensate for the camera sensor mounting\n> > rotation.\n> >\n> > \\var Transform::Rot270\n> > Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n> >\n> > Let's start by saying we did a sub-optimal (to be nice) job in\n> > having Transform and V4L2_CID_ROTATION operate in two different\n> > directions. Before the usage of Transform in libcamera gets widely\n> > adopted sould we align the two ?\n>\n> I don't mind at all... we just have to decide what we want!\n>\n> >\n> > Hence if I'm not mistaken, if your camera has Rotation=90 and an\n> > application supplies Transform::Identity in\n> > CameraConfiguration::transform what you should get back is Rot270\n> >\n> > Correct ?\n>\n> The way things are currently, if the camera says Rotation=90, then\n> your image in memory looks like it is rotated 90 degrees clockwise, so\n> the Transform (if your pipeline and sensor can't apply any\n> flips/rotations at all) will come back as Rot90.\n>\n> That is, the only transform you can specify that won't be adjusted, is\n> a 90 degree clockwise rotation, so transform=Rot90 as things stand\n> currently.\n>\n> If we choose to change the sense of the libcamera.Transform rotation,\n> then we'd get transform=Rot270 instead.\n>\n> (If your pipeline handler can do flips and transposes, it may be able\n> to do what you ask and so the transform would come back unchanged and\n> would be \"Valid\".)\n>\n> >\n> > > > So in turn, if you have a camera/ISP that can apply no transforms at\n> > > > all, then the only value for the user transform that will not come\n> > > > back as adjusted is \"rot90\", because it looks like it's had a 90\n> > > > degree clockwise rotation performed (and libcamera transforms count\n> > > > rotations as being clockwise). Does that also make sense?\n> >\n> > It might be the case today, but I wonder if that's desirable.\n> >\n> > If an application supplies CameraConfiguration::transform = Identity\n> > it means it doesn't want any rotation applied by the library. It will\n> > get back 270 to indicate that that's how the image is rotated and will\n> > get an Adjusted configuration.\n>\n> I'm not quite sure I get that, perhaps it depends how we interpret\n> \"applied by the library\". I guess I agree if we mean \"applied by the\n> library together with any rotation of the sensor\"? Maybe a real\n\n\"Applied by the library\" == happens inside libcamera, ISP or sensor\nflips..\n\n> example is helpful:\n>\n> So for a Raspberry Pi, for example, our sensors are mostly mounted\n> upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\n> say the transform they want is transform=Identity i.e. images come\n> back \"the right way up\". Internally it means that we do apply an extra\n> h+v flip to undo the 180 degree rotation of the sensor.\n>\n\nHere you go, this is where we disconnect.\n\nAs I've interpreted the API so far, and the way we designed\nproperties::Rotation was to signal to the application what is the\nstatus of the image as it comes from the sensor. Application could then\ncorrect that by setting CameraConfiguration::transform, but in most\ncases the image will be corrected at rendering time by the application\nitself.\n\nIWO I was not expecting auto-correction and Transform::Identity to me\nmeans \"give me the images as they come from the sensor\"\n\n> >\n> > If instead an application supplies CameraConfiguration::transform = Rot270\n> > and your camera -cannot- do any rotation, it will still receive back\n> > Rot270 and the state is Valid.\n>\n> I think that's true if we change the sense of the libcamera.Transform\n> rotation. Currently, as I said earlier, I think we get Rot90?\n>\n> >\n> > If it asks for CameraConfiguration::transform = Rot270 and the camera\n> > -can- do that, it will clear CameraConfiguration::transform to\n> > Identity and return Adjusted ?\n>\n> I'm not sure there... The CameraConfiguration::transform is the final\n> transform you want. If that can be done, it's left alone and you get\n> back Valid. It's not telling you what else you would need to do\n> afterwards.\n>\n> (To find out what you need to do afterwards you'd work out\n> \"inverse(what-you-will-get) * what-you-wanted\", or something like\n> that.)\n>\n\nWhat I would like is to make as easier as possible for applications\nto know what they have to do to have the image rotated in the way they\nexpect. There are several \"transform\" steps on the image, the first\none happens in libcamera, all the other ones are left to the\nappplication.\n\n*) The image \"native\" transform is reported by Rotation (this assumes\nthat any implicit rotation buried in the sensor driver, as per our\nlast discussion with Dave is not taken into account)\n\n properties::Rotation = Rot270 = HFlip | Transpose\n\n*) Application sets CameraConfiguration::transform and call validate() to\ncheck what can be done by libcamera and what should be done by next\nlayers\n\n CameraConfigurat::Transform = Rot270\n\n*) The camera validate the transform and as it can only flip it sets\ntransform to HFlip, and the state is Adjusted\n\n validate() -> The camera can only flip\n -> CameraConfiguration = HFlip\n\n*) Application gets back what the camera can do and instrument the\nrest of the rendering pipeline to perform the additional transforms\nthe camera cannot do. To get what you have to do I presume you have to\n\n additionalTransform = what_you_want & !what_you_get\n\nSo yes, I was interpreting the usage of CameraConfiguration::transform\nas a negotiation of what the library can do and what applications have\nto do by themselves, which if I got you is different than what's\ncurrecntly implemented ?\n\n> >\n> > Is this what happens (I'm looking at the above code with David's\n> > suggestion to use the inverse of roationTransform).\n> >\n> >\n> > > >\n> > > > So by a slightly arbitrary mixture of conventions, it looks to me as\n> > > > though the use of the inverse that I queried is indeed wrong, because\n> > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> > > > case.\n> > > >\n> > > > Does that sound plausible... what do folks think?\n> > >\n> > > Ah, I think the question is whether the adjusted value means: \"this is what\n> > > you'll get\" vs \"this is what you have to do yourself\".\n> > >\n> > > Above I argued in favor of the later, but I guess the first one is what the\n> > > API is supposed to mean. That would imply that the client has compute the\n> > > difference between requested and adjusted value itself.\n> >\n> > I always presumed the latter was meant to happen..\n>\n> Yes, I think this is the crux of the matter. I believe it's \"what you\n> will get\". It's not \"what you need to do after\". (If it was \"what you\n> need to do after\", it would change every time you try to validate the\n> configuration!)\n>\n\nOn the \"it would change every time you try to validate the\nconfiguration\": only if the camera cannot satisfy the transform you\nhave requested. If you ask for a flip and the camera can do it,\ntransform stays un-touched and the state is Valid ?\n\n> >\n> > >\n> > > In my case that would mean:\n> > >\n> > > requested CameraConfiguration::transform: Transform::Identity,\n> > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > > CameraConfiguration::transform: Transform::Rot270\n> > >\n> >\n> > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n>\n> Yes, looks right to me!\n>\n> >\n> > > To compute what the client has to do, it would need to do:\n> > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> > > that, a 90 degr. rotation clockwise.\n>\n> Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> what-you-wanted\", in this case inverse(rot270) * identity which is\n> rot90 (in the current clockwise sense).\n>\n> > >\n> > > Does that sound reasonable? In that case the patch here should indeed be\n> > > `*transform = rotationTransform;`, without the inverse.\n> > >\n>\n> Yes, agree!\n>\n> >\n> > /0\\\n> >\n> > before we plumb this into and make assumptions that will be then\n> > set into stone in our adaption layers, can we take a step back and\n> > define what kind of API we would like to have ?\n> >\n> > Let's start by the definition of CameraConfiguration::transform.\n> >\n> > It's a member of CameraConfiguration, and as by definition it is\n> > filled-in by the application and eventually adjusted by the Camera to\n> > a value that's acceptable/doable for the current configuration.\n> >\n> > 1) Application to Camera = (counter?)clockwise degrees the camera\n> > system should rotate the image before presenting it to the user\n>\n> This may depend on what exactly we mean by \"the camera system\". I'd\n> explain it as\n> \"the final resulting transform that the application wants to be\n> applied to all the output images\". After all, this is actually the\n> only thing the application knows!\n>\n\nagreed\n\n> >\n> > 2) Camera to application = the rotation the camera can actually do\n> > (ie. only sensor flips == no Transpose)\n>\n> Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n\nyes, more correct\n\n> because flips aren't rotations, you could in theory have a PH that\n> could do transposes (even though we probably don't currently).\n>\n\nnone currently afaict\n\n> >\n> > The current documentation doesn't explain how it gets adjusted\n>\n> This is true, though I also think it's platform dependent. If you had\n> a PH that can't do transposes but can do flips, you have a choice how\n> you handle (for example) rot90. You could say \"oh I can't do that, so\n> in this case I won't do anything at all\" and leave the application to\n> sort everything out. The Pi tries to say \"well, I'll do everything\n> except for the transpose, so an application only has to worry about\n> that one case\". But there is a choice...\n>\n\nI agree it's platform-specific, but the way transform is negotiated with\napplication should be standardized\n\n> >\n> > /**\n> > * \\var CameraConfiguration::transform\n> > * \\brief User-specified transform to be applied to the image\n> > *\n> > * The transform is a user-specified 2D plane transform that will be applied\n> > * to the camera images by the processing pipeline before being handed to\n> > * the application. This is subsequent to any transform that is already\n> > * required to fix up any platform-defined rotation.\n> > *\n> > * The usual 2D plane transforms are allowed here (horizontal/vertical\n> > * flips, multiple of 90-degree rotations etc.), but the validate() function\n> > * may adjust this field at its discretion if the selection is not supported.\n> > */\n> >\n> > The last part in particular, \"at its discrection\" it's really too\n> > generic.\n>\n> I guess it's hinting at the fact that it's up to the PH, and we don't\n> mandate the exact way in which it can fail to do what you asked for!\n> But it would be fine to have a discussion on that.\n>\n\nThat's the point that most concerns me :)\n\n> >\n> > Let's start from the beginning: what application should use transform\n> > for ? I presume they should:\n> >\n> > 1) Inspect propertis::Rotation\n> > 2) Correct the image for the properties::Rotation amount (actually,\n> > the inverse because of the clockwise/counter-clockwise thing)\n> > 3) Add any additional rotation they would like on top of this\n>\n> So I think the point of the transform is that you *don't* have to\n> worry about the properties::Rotation. It takes care of that for you.\n> You say \"I want upside down images\" and you get them, whatever the\n> sensor's rotation.\n\nThat's acceptable as well if we prefer that, but it has to be\nstandardized among pipelines imho\n\n>\n> >\n> > How should transform be adjusted ?\n> >\n> > Identity:\n> > If application->identity the camera does not have to apply any\n> > transform. Should the transform be adjutes like it happens today to\n> > compensate the Rotation (at least that's what I presume happens) ? I\n> > presume no, applications can infer rotation from properties and if they\n> > chose Identity either other layers above will do the transform or\n> > they're fine with images as they are.\n>\n> If the application says it wants the \"identity\" transform, then it\n> should get output images with no transformation applied. But this does\n> mean it will undo the sensor rotation (this being the normal case on\n> the Pi).\n>\n\nright, it depends if we decide to account for the sensor rotation\ninside libcamera or not\n\n> >\n> > A supported/non-supported transform:\n> > An application asks for a VFLIP, the camera can do it, transform is\n> > not changed. If the camera cannot flip, the transform is set back to\n> > Identity and the state is adjusted.\n>\n> Yes, mostly, except that the sensor may be mounted with a rotation,\n> but if it isn't, then I agree with this.\n>\n> >\n> > A partially non-supported transform:\n> > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> > only do HFLIP. transform sould be then set to HFLIP and applications\n> > know that the remaining transformation should be handled elsewhere.\n>\n> As I suggested above, I'm not sure whether we want to mandate this. I\n> agree it seems useful behaviour, which is why the Pi does it, but I'm\n> open minded about requiring it or not.\n>\n> >\n> > Are we missing anything with such behaviour ? I presume it's rather\n> > similar to what happens today if not for the fact that the validate()\n> > implementation adjusts transform to rotation in case it cannot flip\n> >\n> > Transform combined = transform * data_->rotationTransform_;\n> > if (!data_->supportsFlips_ && !!combined)\n> > transform = -data_->rotationTransform_;\n> >\n> > Not really sure I got why.\n> >\n> > All of this, but an even more fundamental question: is it too late/is\n> > it worh it to adjust the Transform definition to adopt the same\n> > direction as the kernel defined rotation ?\n> >\n> > Sorry for the long email :/\n>\n> No problem! I like nothing more than discussing transforms, except\n> possibly colour spaces!\n>\n\n:)\n\n> More seriously, here are my key points:\n\nI'll try to summarize what I see differently\n\n>\n> 1. The CameraConfiguration::transform field says what the application\n> wants the final output images to have.\n\n1. The CameraConfiguration::transform field says what transform the\nlibrary has to apply to images as they arrive from the sensor in its\ndefault configuration\n\n> 2. It takes account of the sensor rotation. The most obvious example\n> is the Pi: if the sensor is mounted upside down, and the applications\n> ask for images \"the right way up\" (i.e. transform = identity), then\n> the PH will apply h and v flips to compensate for the sensor mounting.\n\n2. It doesn't automatically account for sensor rotation. Application\nmight decide to do that along the rendering pipeline, in example\n\n> 3. If the PH can do what the application requests, it leaves the\n> transform alone and your configuration is Valid.\n\nAgreed\n\n> 4. If the PH can't do what the application wants, it will set the\n> value to something that it can do, and the configuration will be\n> Adjusted.\n\nAgreed\n\n>\n> If folks wanted a short conference call to iron any of this out, I'd\n> be very happy to join in!\n\nI've asked others to have a look here, let's see if we need a call,\nI'm all for it !\n\nThanks\n j\n\n>\n> Thanks\n> David\n>\n> >\n> > > > Thanks everyone!\n> > > > David\n> > >\n> > > Thanks!\n> > >\n> > > Robert\n> > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id CEB75C31E9\n\tfor ;\n\tTue, 13 Dec 2022 11:22:55 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2EF7363362;\n\tTue, 13 Dec 2022 12:22:55 +0100 (CET)","from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net\n\t[IPv6:2001:4b98:dc4:8::225])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id F401A63340\n\tfor ;\n\tTue, 13 Dec 2022 12:22:53 +0100 (CET)","(Authenticated sender: jacopo@jmondi.org)\n\tby mail.gandi.net (Postfix) with ESMTPSA id 3F4981C0007;\n\tTue, 13 Dec 2022 11:22:52 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670930575;\n\tbh=8L8AJ+JKAYbDa4maqux1xjf3T/idYdB+fcZnzB3eY4A=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=quN7p7tGL+gUb5QJM39mkHTf1ddKQj1yvpwtb12V3OsRGy8NKebPYtxbMdRMnjwcl\n\tEi0kB1zXK4/g+16Vqk+3Do5FlrdNKcmUoEhKg7ZU8rJoNqGbne0LY9ml3ipYNQQpHs\n\t7hoVDNClesjfzPpuqNl4p4hC0JoMUd2O4zPNG/bI80OBxP9KROXidXd6L7cUXQlm0c\n\tBiXliqgkJEx9cN8kzd6niIcNlsqK5Viz7IgHQj4iR6G3r91AKcVXx/PP63DMxPeyeC\n\tf9JC824ODZQCD5JdXxB+dKdw+ir84HaWWfDgi2pkCSb8D4dmUIMXxFPEttu0LywIxX\n\tAxCVBxhUXXjhA==","Date":"Tue, 13 Dec 2022 12:22:51 +0100","To":"David Plowman ","Message-ID":"<20221213112251.jnpgyzitusutkqm6@uno.localdomain>","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jacopo Mondi via libcamera-devel ","Reply-To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26054,"web_url":"https://patchwork.libcamera.org/comment/26054/","msgid":"","date":"2022-12-13T14:30:24","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Jacopo\n\nThanks for the reply. I won't write a great deal this time round, I\nagree that possibly a call would be a good idea.\n\nOn Tue, 13 Dec 2022 at 11:22, Jacopo Mondi wrote:\n>\n> Hi David\n>\n> On Mon, Dec 12, 2022 at 11:46:31AM +0000, David Plowman wrote:\n> > Hi Jacopo\n> >\n> > Thanks for wading into the sea on this question! (Very appropriate now\n> > that the stick person example has been turned into a shark...!)\n> >\n> > On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n> > wrote:\n> > >\n> > > Hi again,\n> > >\n> > > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n> > > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > > > > Hi Robert\n> > > > >\n> > > > > Thanks for prodding us all on this question!\n> > > >\n> > > > Hi, thanks for the quick reply! And a pleasure :P\n> > > >\n> > > > > I looked back over this again, and found Jacopo's description of the\n> > > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > > > > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > > > > is what I stumbled into)\n> > > >\n> > > > The current kernel docu can be found at\n> > > >\n> > > > https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n> >\n> > Thanks for the updated reference. The stick person may have turned\n> > into a shark, but apart from that I think the sense of the\n> > documentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n> > 90 degrees, then your image in memory looks like it has had a 90\n> > degree *clockwise* rotation applied.\n> >\n>\n> Yes!\n>\n> > > >\n> > > > the libcamera one at\n> > > >\n> > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> > > >\n> > > > > From that, and the nice example with the stick person, I deduce that\n> > > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> > > > > degrees, then the image you capture (if nothing else happens to it)\n> > > > > will look like it has had a 90 degree clockwise rotation performed.\n> > > > > Everyone agree with that?\n> > > > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> > > > translate to Transform::Rot90.\n> > >\n> > > Let's here put some definitions in place\n> > >\n> > > V4L2_CID_CAMERA_SENSOR_ROTATION\n> > > This read-only control describes the rotation correction in degrees in\n> > > the counter-clockwise direction to be applied to the captured images\n> > > once captured to memory to compensate for the camera sensor mounting\n> > > rotation.\n> > >\n> > > \\var Transform::Rot270\n> > > Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n> > >\n> > > Let's start by saying we did a sub-optimal (to be nice) job in\n> > > having Transform and V4L2_CID_ROTATION operate in two different\n> > > directions. Before the usage of Transform in libcamera gets widely\n> > > adopted sould we align the two ?\n> >\n> > I don't mind at all... we just have to decide what we want!\n> >\n> > >\n> > > Hence if I'm not mistaken, if your camera has Rotation=90 and an\n> > > application supplies Transform::Identity in\n> > > CameraConfiguration::transform what you should get back is Rot270\n> > >\n> > > Correct ?\n> >\n> > The way things are currently, if the camera says Rotation=90, then\n> > your image in memory looks like it is rotated 90 degrees clockwise, so\n> > the Transform (if your pipeline and sensor can't apply any\n> > flips/rotations at all) will come back as Rot90.\n> >\n> > That is, the only transform you can specify that won't be adjusted, is\n> > a 90 degree clockwise rotation, so transform=Rot90 as things stand\n> > currently.\n> >\n> > If we choose to change the sense of the libcamera.Transform rotation,\n> > then we'd get transform=Rot270 instead.\n> >\n> > (If your pipeline handler can do flips and transposes, it may be able\n> > to do what you ask and so the transform would come back unchanged and\n> > would be \"Valid\".)\n> >\n> > >\n> > > > > So in turn, if you have a camera/ISP that can apply no transforms at\n> > > > > all, then the only value for the user transform that will not come\n> > > > > back as adjusted is \"rot90\", because it looks like it's had a 90\n> > > > > degree clockwise rotation performed (and libcamera transforms count\n> > > > > rotations as being clockwise). Does that also make sense?\n> > >\n> > > It might be the case today, but I wonder if that's desirable.\n> > >\n> > > If an application supplies CameraConfiguration::transform = Identity\n> > > it means it doesn't want any rotation applied by the library. It will\n> > > get back 270 to indicate that that's how the image is rotated and will\n> > > get an Adjusted configuration.\n> >\n> > I'm not quite sure I get that, perhaps it depends how we interpret\n> > \"applied by the library\". I guess I agree if we mean \"applied by the\n> > library together with any rotation of the sensor\"? Maybe a real\n>\n> \"Applied by the library\" == happens inside libcamera, ISP or sensor\n> flips..\n>\n> > example is helpful:\n> >\n> > So for a Raspberry Pi, for example, our sensors are mostly mounted\n> > upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\n> > say the transform they want is transform=Identity i.e. images come\n> > back \"the right way up\". Internally it means that we do apply an extra\n> > h+v flip to undo the 180 degree rotation of the sensor.\n> >\n>\n> Here you go, this is where we disconnect.\n>\n> As I've interpreted the API so far, and the way we designed\n> properties::Rotation was to signal to the application what is the\n> status of the image as it comes from the sensor. Application could then\n> correct that by setting CameraConfiguration::transform, but in most\n> cases the image will be corrected at rendering time by the application\n> itself.\n>\n> IWO I was not expecting auto-correction and Transform::Identity to me\n> means \"give me the images as they come from the sensor\"\n>\n> > >\n> > > If instead an application supplies CameraConfiguration::transform = Rot270\n> > > and your camera -cannot- do any rotation, it will still receive back\n> > > Rot270 and the state is Valid.\n> >\n> > I think that's true if we change the sense of the libcamera.Transform\n> > rotation. Currently, as I said earlier, I think we get Rot90?\n> >\n> > >\n> > > If it asks for CameraConfiguration::transform = Rot270 and the camera\n> > > -can- do that, it will clear CameraConfiguration::transform to\n> > > Identity and return Adjusted ?\n> >\n> > I'm not sure there... The CameraConfiguration::transform is the final\n> > transform you want. If that can be done, it's left alone and you get\n> > back Valid. It's not telling you what else you would need to do\n> > afterwards.\n> >\n> > (To find out what you need to do afterwards you'd work out\n> > \"inverse(what-you-will-get) * what-you-wanted\", or something like\n> > that.)\n> >\n>\n> What I would like is to make as easier as possible for applications\n> to know what they have to do to have the image rotated in the way they\n> expect. There are several \"transform\" steps on the image, the first\n> one happens in libcamera, all the other ones are left to the\n> appplication.\n\nI think this is exactly what I want, namely \"to make it as easy as\npossible for applications\".\n\nOn Raspberry Pi - and this has been the case for a couple of years\nnow! - if a user wants images the \"normal\" way up then they simply\npass Transform::Identity. If they want upside down images they pass\nTransform::Rot180. It's as easy as that, there is simply nothing else\nto do.\n\nRequiring all applications to look at the sensor rotation and then\ncombine various things to figure out what they have to ask for (and\npossibly get it wrong) seems, well, actually quite unhelpful.\n\nBut this is indeed what we need to talk about. Let's organise a call!\n\nThanks\nDavid\n\n>\n> *) The image \"native\" transform is reported by Rotation (this assumes\n> that any implicit rotation buried in the sensor driver, as per our\n> last discussion with Dave is not taken into account)\n>\n> properties::Rotation = Rot270 = HFlip | Transpose\n>\n> *) Application sets CameraConfiguration::transform and call validate() to\n> check what can be done by libcamera and what should be done by next\n> layers\n>\n> CameraConfigurat::Transform = Rot270\n>\n> *) The camera validate the transform and as it can only flip it sets\n> transform to HFlip, and the state is Adjusted\n>\n> validate() -> The camera can only flip\n> -> CameraConfiguration = HFlip\n>\n> *) Application gets back what the camera can do and instrument the\n> rest of the rendering pipeline to perform the additional transforms\n> the camera cannot do. To get what you have to do I presume you have to\n>\n> additionalTransform = what_you_want & !what_you_get\n>\n> So yes, I was interpreting the usage of CameraConfiguration::transform\n> as a negotiation of what the library can do and what applications have\n> to do by themselves, which if I got you is different than what's\n> currecntly implemented ?\n>\n> > >\n> > > Is this what happens (I'm looking at the above code with David's\n> > > suggestion to use the inverse of roationTransform).\n> > >\n> > >\n> > > > >\n> > > > > So by a slightly arbitrary mixture of conventions, it looks to me as\n> > > > > though the use of the inverse that I queried is indeed wrong, because\n> > > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> > > > > case.\n> > > > >\n> > > > > Does that sound plausible... what do folks think?\n> > > >\n> > > > Ah, I think the question is whether the adjusted value means: \"this is what\n> > > > you'll get\" vs \"this is what you have to do yourself\".\n> > > >\n> > > > Above I argued in favor of the later, but I guess the first one is what the\n> > > > API is supposed to mean. That would imply that the client has compute the\n> > > > difference between requested and adjusted value itself.\n> > >\n> > > I always presumed the latter was meant to happen..\n> >\n> > Yes, I think this is the crux of the matter. I believe it's \"what you\n> > will get\". It's not \"what you need to do after\". (If it was \"what you\n> > need to do after\", it would change every time you try to validate the\n> > configuration!)\n> >\n>\n> On the \"it would change every time you try to validate the\n> configuration\": only if the camera cannot satisfy the transform you\n> have requested. If you ask for a flip and the camera can do it,\n> transform stays un-touched and the state is Valid ?\n>\n> > >\n> > > >\n> > > > In my case that would mean:\n> > > >\n> > > > requested CameraConfiguration::transform: Transform::Identity,\n> > > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > > > CameraConfiguration::transform: Transform::Rot270\n> > > >\n> > >\n> > > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n> >\n> > Yes, looks right to me!\n> >\n> > >\n> > > > To compute what the client has to do, it would need to do:\n> > > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> > > > that, a 90 degr. rotation clockwise.\n> >\n> > Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> > what-you-wanted\", in this case inverse(rot270) * identity which is\n> > rot90 (in the current clockwise sense).\n> >\n> > > >\n> > > > Does that sound reasonable? In that case the patch here should indeed be\n> > > > `*transform = rotationTransform;`, without the inverse.\n> > > >\n> >\n> > Yes, agree!\n> >\n> > >\n> > > /0\\\n> > >\n> > > before we plumb this into and make assumptions that will be then\n> > > set into stone in our adaption layers, can we take a step back and\n> > > define what kind of API we would like to have ?\n> > >\n> > > Let's start by the definition of CameraConfiguration::transform.\n> > >\n> > > It's a member of CameraConfiguration, and as by definition it is\n> > > filled-in by the application and eventually adjusted by the Camera to\n> > > a value that's acceptable/doable for the current configuration.\n> > >\n> > > 1) Application to Camera = (counter?)clockwise degrees the camera\n> > > system should rotate the image before presenting it to the user\n> >\n> > This may depend on what exactly we mean by \"the camera system\". I'd\n> > explain it as\n> > \"the final resulting transform that the application wants to be\n> > applied to all the output images\". After all, this is actually the\n> > only thing the application knows!\n> >\n>\n> agreed\n>\n> > >\n> > > 2) Camera to application = the rotation the camera can actually do\n> > > (ie. only sensor flips == no Transpose)\n> >\n> > Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n>\n> yes, more correct\n>\n> > because flips aren't rotations, you could in theory have a PH that\n> > could do transposes (even though we probably don't currently).\n> >\n>\n> none currently afaict\n>\n> > >\n> > > The current documentation doesn't explain how it gets adjusted\n> >\n> > This is true, though I also think it's platform dependent. If you had\n> > a PH that can't do transposes but can do flips, you have a choice how\n> > you handle (for example) rot90. You could say \"oh I can't do that, so\n> > in this case I won't do anything at all\" and leave the application to\n> > sort everything out. The Pi tries to say \"well, I'll do everything\n> > except for the transpose, so an application only has to worry about\n> > that one case\". But there is a choice...\n> >\n>\n> I agree it's platform-specific, but the way transform is negotiated with\n> application should be standardized\n>\n> > >\n> > > /**\n> > > * \\var CameraConfiguration::transform\n> > > * \\brief User-specified transform to be applied to the image\n> > > *\n> > > * The transform is a user-specified 2D plane transform that will be applied\n> > > * to the camera images by the processing pipeline before being handed to\n> > > * the application. This is subsequent to any transform that is already\n> > > * required to fix up any platform-defined rotation.\n> > > *\n> > > * The usual 2D plane transforms are allowed here (horizontal/vertical\n> > > * flips, multiple of 90-degree rotations etc.), but the validate() function\n> > > * may adjust this field at its discretion if the selection is not supported.\n> > > */\n> > >\n> > > The last part in particular, \"at its discrection\" it's really too\n> > > generic.\n> >\n> > I guess it's hinting at the fact that it's up to the PH, and we don't\n> > mandate the exact way in which it can fail to do what you asked for!\n> > But it would be fine to have a discussion on that.\n> >\n>\n> That's the point that most concerns me :)\n>\n> > >\n> > > Let's start from the beginning: what application should use transform\n> > > for ? I presume they should:\n> > >\n> > > 1) Inspect propertis::Rotation\n> > > 2) Correct the image for the properties::Rotation amount (actually,\n> > > the inverse because of the clockwise/counter-clockwise thing)\n> > > 3) Add any additional rotation they would like on top of this\n> >\n> > So I think the point of the transform is that you *don't* have to\n> > worry about the properties::Rotation. It takes care of that for you.\n> > You say \"I want upside down images\" and you get them, whatever the\n> > sensor's rotation.\n>\n> That's acceptable as well if we prefer that, but it has to be\n> standardized among pipelines imho\n>\n> >\n> > >\n> > > How should transform be adjusted ?\n> > >\n> > > Identity:\n> > > If application->identity the camera does not have to apply any\n> > > transform. Should the transform be adjutes like it happens today to\n> > > compensate the Rotation (at least that's what I presume happens) ? I\n> > > presume no, applications can infer rotation from properties and if they\n> > > chose Identity either other layers above will do the transform or\n> > > they're fine with images as they are.\n> >\n> > If the application says it wants the \"identity\" transform, then it\n> > should get output images with no transformation applied. But this does\n> > mean it will undo the sensor rotation (this being the normal case on\n> > the Pi).\n> >\n>\n> right, it depends if we decide to account for the sensor rotation\n> inside libcamera or not\n>\n> > >\n> > > A supported/non-supported transform:\n> > > An application asks for a VFLIP, the camera can do it, transform is\n> > > not changed. If the camera cannot flip, the transform is set back to\n> > > Identity and the state is adjusted.\n> >\n> > Yes, mostly, except that the sensor may be mounted with a rotation,\n> > but if it isn't, then I agree with this.\n> >\n> > >\n> > > A partially non-supported transform:\n> > > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> > > only do HFLIP. transform sould be then set to HFLIP and applications\n> > > know that the remaining transformation should be handled elsewhere.\n> >\n> > As I suggested above, I'm not sure whether we want to mandate this. I\n> > agree it seems useful behaviour, which is why the Pi does it, but I'm\n> > open minded about requiring it or not.\n> >\n> > >\n> > > Are we missing anything with such behaviour ? I presume it's rather\n> > > similar to what happens today if not for the fact that the validate()\n> > > implementation adjusts transform to rotation in case it cannot flip\n> > >\n> > > Transform combined = transform * data_->rotationTransform_;\n> > > if (!data_->supportsFlips_ && !!combined)\n> > > transform = -data_->rotationTransform_;\n> > >\n> > > Not really sure I got why.\n> > >\n> > > All of this, but an even more fundamental question: is it too late/is\n> > > it worh it to adjust the Transform definition to adopt the same\n> > > direction as the kernel defined rotation ?\n> > >\n> > > Sorry for the long email :/\n> >\n> > No problem! I like nothing more than discussing transforms, except\n> > possibly colour spaces!\n> >\n>\n> :)\n>\n> > More seriously, here are my key points:\n>\n> I'll try to summarize what I see differently\n>\n> >\n> > 1. The CameraConfiguration::transform field says what the application\n> > wants the final output images to have.\n>\n> 1. The CameraConfiguration::transform field says what transform the\n> library has to apply to images as they arrive from the sensor in its\n> default configuration\n>\n> > 2. It takes account of the sensor rotation. The most obvious example\n> > is the Pi: if the sensor is mounted upside down, and the applications\n> > ask for images \"the right way up\" (i.e. transform = identity), then\n> > the PH will apply h and v flips to compensate for the sensor mounting.\n>\n> 2. It doesn't automatically account for sensor rotation. Application\n> might decide to do that along the rendering pipeline, in example\n>\n> > 3. If the PH can do what the application requests, it leaves the\n> > transform alone and your configuration is Valid.\n>\n> Agreed\n>\n> > 4. If the PH can't do what the application wants, it will set the\n> > value to something that it can do, and the configuration will be\n> > Adjusted.\n>\n> Agreed\n>\n> >\n> > If folks wanted a short conference call to iron any of this out, I'd\n> > be very happy to join in!\n>\n> I've asked others to have a look here, let's see if we need a call,\n> I'm all for it !\n>\n> Thanks\n> j\n>\n> >\n> > Thanks\n> > David\n> >\n> > >\n> > > > > Thanks everyone!\n> > > > > David\n> > > >\n> > > > Thanks!\n> > > >\n> > > > Robert\n> > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 6222BC31E9\n\tfor ;\n\tTue, 13 Dec 2022 14:30:40 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B646B63354;\n\tTue, 13 Dec 2022 15:30:39 +0100 (CET)","from mail-pg1-x535.google.com (mail-pg1-x535.google.com\n\t[IPv6:2607:f8b0:4864:20::535])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B0FBF63340\n\tfor ;\n\tTue, 13 Dec 2022 15:30:37 +0100 (CET)","by mail-pg1-x535.google.com with SMTP id f3so10543338pgc.2\n\tfor ;\n\tTue, 13 Dec 2022 06:30:37 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670941839;\n\tbh=lOsIMmGt6xfcksRyFTBLr18fjVcOjxpT/vZpnS1AMS4=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=oLLUuFtlXxDyqZ7xmaSYFDX6zQrZHD4AxEmII8/niEgvsHLyukUn+SVr2jYOLgYUY\n\tz/M/cz7hUaVKhLoaVvHCP4EwnDtT4y7VK6sNRAOyKZToa130f+1RCWnMrpB6UI8XVo\n\t4qnK6WKDiWu9o32mmjfti3lxaXmfgdhz2k3zy1vEeMKdh1HUhhGu0VnJ9Mp4cYpp8h\n\tHsA78HpjHDZJKrTQ33YfJYBtjBTiYNc+wKtiEnf80Z+d7CqFblgb8jsAdlVfP6jSrq\n\tNNnbZAuwSvoh1qXK4VQkZ+CISoVZ7yMdxeQ0zlZTD3IV8F2MUVH8HCjIxIO2ZHYiek\n\t6cI9DhXZ0zAGg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=udDkyUbbSqPytWpzfngCeLz+JCSfw1a1cjWAj/Ngrcc=;\n\tb=Ta1H9w4JReKcDRWkl+RIv0aodf4VGgp3naI+V5QEs0TUtHP0L92WAQWpcJ9MCFERvv\n\tPNqSQSy1jbwTU13dV1bMQu+u6bR+q+QrNX6QthCqqerODhZf/tKm2bCqkg7WhKT9t6Ad\n\tkuq6+B2CKPx51ieRfTHFvEocue0+0LY5xk0Fgk5TkFzXXiSKqYeDbkHuQXukAKHIzxW+\n\tGKdGMGlwExQQXrjbAFp1ytTpC7FGVseQCifKQ0sP0Tc49dfT8DYZ7kxqEbnPON9m9FxD\n\tgtw6rg6BcwUyGfyenyDTKhnwRcKNlb4lQ+2GMOs9DDuvqnmLx/UCkXnjCa3ILrxgRNVX\n\tCVtw=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"Ta1H9w4J\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=udDkyUbbSqPytWpzfngCeLz+JCSfw1a1cjWAj/Ngrcc=;\n\tb=yrKpOFJwjWLR2ebcQc1OKu45qjD95I3xqInsoCRHGPX/foGpMK10Qd2CAnp3tRxImV\n\tciIfeD2YiEacvy74ERtPUU9oxDTFPu+iSttjCEz5BCoi2+r6vSz0SoNFcLd5ZT10pjYu\n\t6TWievxQqYfUpMgdv7yW06HL3v1G2vxXlCuBCMtaTJCWpXYHLJMJaVEtQYrJJ7SXXMlB\n\t6++IWqWqeM5CIPn5+xcn7+WAow2RUf9TWGBoucAauvMza7JHRvweIbQdtZwrul5fHymh\n\tX05+fwFPA2yrrQoo7GdBlA6FmJKtg/AciW3zWk4nT1Odbiwb2eVIJCp5LThYGEImCNpN\n\tAqWw==","X-Gm-Message-State":"ANoB5pn2oSG2M+lfc8HbrQaTL1ZZQsPbrnAcLvHchDSE/PVkNuABeJKv\n\tTDSLO+5jMZPTHnHpkUrSGcc5NFvS/0M1GML9O6BKXQ==","X-Google-Smtp-Source":"AA0mqf5UuHNqp84cQYxfuAoPgCu1UGVy/It4K+ep+eKXNDyhI0g2DA0NWCIwGc0sQL+Pp6wu/nHBwHnBbIvrFwNC7cA=","X-Received":"by 2002:a63:5b48:0:b0:478:b930:a35c with SMTP id\n\tl8-20020a635b48000000b00478b930a35cmr17322611pgm.494.1670941835764;\n\tTue, 13 Dec 2022 06:30:35 -0800 (PST)","MIME-Version":"1.0","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t\n\t<20221213112251.jnpgyzitusutkqm6@uno.localdomain>","In-Reply-To":"<20221213112251.jnpgyzitusutkqm6@uno.localdomain>","Date":"Tue, 13 Dec 2022 14:30:24 +0000","Message-ID":"","To":"Jacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26055,"web_url":"https://patchwork.libcamera.org/comment/26055/","msgid":"<20221213144329.3pqib2cf4nkpfxqw@uno.localdomain>","date":"2022-12-13T14:43:29","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi David,\n\nOn Tue, Dec 13, 2022 at 02:30:24PM +0000, David Plowman wrote:\n> Hi Jacopo\n>\n> Thanks for the reply. I won't write a great deal this time round, I\n> agree that possibly a call would be a good idea.\n>\n\nSure thanks\n\n[snip]\n\n\n> >\n> > What I would like is to make as easier as possible for applications\n> > to know what they have to do to have the image rotated in the way they\n> > expect. There are several \"transform\" steps on the image, the first\n> > one happens in libcamera, all the other ones are left to the\n> > appplication.\n>\n> I think this is exactly what I want, namely \"to make it as easy as\n> possible for applications\".\n>\n> On Raspberry Pi - and this has been the case for a couple of years\n> now! - if a user wants images the \"normal\" way up then they simply\n> pass Transform::Identity. If they want upside down images they pass\n> Transform::Rot180. It's as easy as that, there is simply nothing else\n> to do.\n>\n> Requiring all applications to look at the sensor rotation and then\n> combine various things to figure out what they have to ask for (and\n> possibly get it wrong) seems, well, actually quite unhelpful.\n\nRobert, you're working on the pipewire integration. What's your\nopinion on this ?\n\n>\n> But this is indeed what we need to talk about. Let's organise a call!\n>\n> Thanks\n> David\n>\n> >\n> > *) The image \"native\" transform is reported by Rotation (this assumes\n> > that any implicit rotation buried in the sensor driver, as per our\n> > last discussion with Dave is not taken into account)\n> >\n> > properties::Rotation = Rot270 = HFlip | Transpose\n> >\n> > *) Application sets CameraConfiguration::transform and call validate() to\n> > check what can be done by libcamera and what should be done by next\n> > layers\n> >\n> > CameraConfigurat::Transform = Rot270\n> >\n> > *) The camera validate the transform and as it can only flip it sets\n> > transform to HFlip, and the state is Adjusted\n> >\n> > validate() -> The camera can only flip\n> > -> CameraConfiguration = HFlip\n> >\n> > *) Application gets back what the camera can do and instrument the\n> > rest of the rendering pipeline to perform the additional transforms\n> > the camera cannot do. To get what you have to do I presume you have to\n> >\n> > additionalTransform = what_you_want & !what_you_get\n> >\n> > So yes, I was interpreting the usage of CameraConfiguration::transform\n> > as a negotiation of what the library can do and what applications have\n> > to do by themselves, which if I got you is different than what's\n> > currecntly implemented ?\n> >\n> > > >\n> > > > Is this what happens (I'm looking at the above code with David's\n> > > > suggestion to use the inverse of roationTransform).\n> > > >\n> > > >\n> > > > > >\n> > > > > > So by a slightly arbitrary mixture of conventions, it looks to me as\n> > > > > > though the use of the inverse that I queried is indeed wrong, because\n> > > > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> > > > > > case.\n> > > > > >\n> > > > > > Does that sound plausible... what do folks think?\n> > > > >\n> > > > > Ah, I think the question is whether the adjusted value means: \"this is what\n> > > > > you'll get\" vs \"this is what you have to do yourself\".\n> > > > >\n> > > > > Above I argued in favor of the later, but I guess the first one is what the\n> > > > > API is supposed to mean. That would imply that the client has compute the\n> > > > > difference between requested and adjusted value itself.\n> > > >\n> > > > I always presumed the latter was meant to happen..\n> > >\n> > > Yes, I think this is the crux of the matter. I believe it's \"what you\n> > > will get\". It's not \"what you need to do after\". (If it was \"what you\n> > > need to do after\", it would change every time you try to validate the\n> > > configuration!)\n> > >\n> >\n> > On the \"it would change every time you try to validate the\n> > configuration\": only if the camera cannot satisfy the transform you\n> > have requested. If you ask for a flip and the camera can do it,\n> > transform stays un-touched and the state is Valid ?\n> >\n> > > >\n> > > > >\n> > > > > In my case that would mean:\n> > > > >\n> > > > > requested CameraConfiguration::transform: Transform::Identity,\n> > > > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > > > > CameraConfiguration::transform: Transform::Rot270\n> > > > >\n> > > >\n> > > > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n> > >\n> > > Yes, looks right to me!\n> > >\n> > > >\n> > > > > To compute what the client has to do, it would need to do:\n> > > > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> > > > > that, a 90 degr. rotation clockwise.\n> > >\n> > > Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> > > what-you-wanted\", in this case inverse(rot270) * identity which is\n> > > rot90 (in the current clockwise sense).\n> > >\n> > > > >\n> > > > > Does that sound reasonable? In that case the patch here should indeed be\n> > > > > `*transform = rotationTransform;`, without the inverse.\n> > > > >\n> > >\n> > > Yes, agree!\n> > >\n> > > >\n> > > > /0\\\n> > > >\n> > > > before we plumb this into and make assumptions that will be then\n> > > > set into stone in our adaption layers, can we take a step back and\n> > > > define what kind of API we would like to have ?\n> > > >\n> > > > Let's start by the definition of CameraConfiguration::transform.\n> > > >\n> > > > It's a member of CameraConfiguration, and as by definition it is\n> > > > filled-in by the application and eventually adjusted by the Camera to\n> > > > a value that's acceptable/doable for the current configuration.\n> > > >\n> > > > 1) Application to Camera = (counter?)clockwise degrees the camera\n> > > > system should rotate the image before presenting it to the user\n> > >\n> > > This may depend on what exactly we mean by \"the camera system\". I'd\n> > > explain it as\n> > > \"the final resulting transform that the application wants to be\n> > > applied to all the output images\". After all, this is actually the\n> > > only thing the application knows!\n> > >\n> >\n> > agreed\n> >\n> > > >\n> > > > 2) Camera to application = the rotation the camera can actually do\n> > > > (ie. only sensor flips == no Transpose)\n> > >\n> > > Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n> >\n> > yes, more correct\n> >\n> > > because flips aren't rotations, you could in theory have a PH that\n> > > could do transposes (even though we probably don't currently).\n> > >\n> >\n> > none currently afaict\n> >\n> > > >\n> > > > The current documentation doesn't explain how it gets adjusted\n> > >\n> > > This is true, though I also think it's platform dependent. If you had\n> > > a PH that can't do transposes but can do flips, you have a choice how\n> > > you handle (for example) rot90. You could say \"oh I can't do that, so\n> > > in this case I won't do anything at all\" and leave the application to\n> > > sort everything out. The Pi tries to say \"well, I'll do everything\n> > > except for the transpose, so an application only has to worry about\n> > > that one case\". But there is a choice...\n> > >\n> >\n> > I agree it's platform-specific, but the way transform is negotiated with\n> > application should be standardized\n> >\n> > > >\n> > > > /**\n> > > > * \\var CameraConfiguration::transform\n> > > > * \\brief User-specified transform to be applied to the image\n> > > > *\n> > > > * The transform is a user-specified 2D plane transform that will be applied\n> > > > * to the camera images by the processing pipeline before being handed to\n> > > > * the application. This is subsequent to any transform that is already\n> > > > * required to fix up any platform-defined rotation.\n> > > > *\n> > > > * The usual 2D plane transforms are allowed here (horizontal/vertical\n> > > > * flips, multiple of 90-degree rotations etc.), but the validate() function\n> > > > * may adjust this field at its discretion if the selection is not supported.\n> > > > */\n> > > >\n> > > > The last part in particular, \"at its discrection\" it's really too\n> > > > generic.\n> > >\n> > > I guess it's hinting at the fact that it's up to the PH, and we don't\n> > > mandate the exact way in which it can fail to do what you asked for!\n> > > But it would be fine to have a discussion on that.\n> > >\n> >\n> > That's the point that most concerns me :)\n> >\n> > > >\n> > > > Let's start from the beginning: what application should use transform\n> > > > for ? I presume they should:\n> > > >\n> > > > 1) Inspect propertis::Rotation\n> > > > 2) Correct the image for the properties::Rotation amount (actually,\n> > > > the inverse because of the clockwise/counter-clockwise thing)\n> > > > 3) Add any additional rotation they would like on top of this\n> > >\n> > > So I think the point of the transform is that you *don't* have to\n> > > worry about the properties::Rotation. It takes care of that for you.\n> > > You say \"I want upside down images\" and you get them, whatever the\n> > > sensor's rotation.\n> >\n> > That's acceptable as well if we prefer that, but it has to be\n> > standardized among pipelines imho\n> >\n> > >\n> > > >\n> > > > How should transform be adjusted ?\n> > > >\n> > > > Identity:\n> > > > If application->identity the camera does not have to apply any\n> > > > transform. Should the transform be adjutes like it happens today to\n> > > > compensate the Rotation (at least that's what I presume happens) ? I\n> > > > presume no, applications can infer rotation from properties and if they\n> > > > chose Identity either other layers above will do the transform or\n> > > > they're fine with images as they are.\n> > >\n> > > If the application says it wants the \"identity\" transform, then it\n> > > should get output images with no transformation applied. But this does\n> > > mean it will undo the sensor rotation (this being the normal case on\n> > > the Pi).\n> > >\n> >\n> > right, it depends if we decide to account for the sensor rotation\n> > inside libcamera or not\n> >\n> > > >\n> > > > A supported/non-supported transform:\n> > > > An application asks for a VFLIP, the camera can do it, transform is\n> > > > not changed. If the camera cannot flip, the transform is set back to\n> > > > Identity and the state is adjusted.\n> > >\n> > > Yes, mostly, except that the sensor may be mounted with a rotation,\n> > > but if it isn't, then I agree with this.\n> > >\n> > > >\n> > > > A partially non-supported transform:\n> > > > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> > > > only do HFLIP. transform sould be then set to HFLIP and applications\n> > > > know that the remaining transformation should be handled elsewhere.\n> > >\n> > > As I suggested above, I'm not sure whether we want to mandate this. I\n> > > agree it seems useful behaviour, which is why the Pi does it, but I'm\n> > > open minded about requiring it or not.\n> > >\n> > > >\n> > > > Are we missing anything with such behaviour ? I presume it's rather\n> > > > similar to what happens today if not for the fact that the validate()\n> > > > implementation adjusts transform to rotation in case it cannot flip\n> > > >\n> > > > Transform combined = transform * data_->rotationTransform_;\n> > > > if (!data_->supportsFlips_ && !!combined)\n> > > > transform = -data_->rotationTransform_;\n> > > >\n> > > > Not really sure I got why.\n> > > >\n> > > > All of this, but an even more fundamental question: is it too late/is\n> > > > it worh it to adjust the Transform definition to adopt the same\n> > > > direction as the kernel defined rotation ?\n> > > >\n> > > > Sorry for the long email :/\n> > >\n> > > No problem! I like nothing more than discussing transforms, except\n> > > possibly colour spaces!\n> > >\n> >\n> > :)\n> >\n> > > More seriously, here are my key points:\n> >\n> > I'll try to summarize what I see differently\n> >\n> > >\n> > > 1. The CameraConfiguration::transform field says what the application\n> > > wants the final output images to have.\n> >\n> > 1. The CameraConfiguration::transform field says what transform the\n> > library has to apply to images as they arrive from the sensor in its\n> > default configuration\n> >\n> > > 2. It takes account of the sensor rotation. The most obvious example\n> > > is the Pi: if the sensor is mounted upside down, and the applications\n> > > ask for images \"the right way up\" (i.e. transform = identity), then\n> > > the PH will apply h and v flips to compensate for the sensor mounting.\n> >\n> > 2. It doesn't automatically account for sensor rotation. Application\n> > might decide to do that along the rendering pipeline, in example\n> >\n> > > 3. If the PH can do what the application requests, it leaves the\n> > > transform alone and your configuration is Valid.\n> >\n> > Agreed\n> >\n> > > 4. If the PH can't do what the application wants, it will set the\n> > > value to something that it can do, and the configuration will be\n> > > Adjusted.\n> >\n> > Agreed\n> >\n> > >\n> > > If folks wanted a short conference call to iron any of this out, I'd\n> > > be very happy to join in!\n> >\n> > I've asked others to have a look here, let's see if we need a call,\n> > I'm all for it !\n> >\n> > Thanks\n> > j\n> >\n> > >\n> > > Thanks\n> > > David\n> > >\n> > > >\n> > > > > > Thanks everyone!\n> > > > > > David\n> > > > >\n> > > > > Thanks!\n> > > > >\n> > > > > Robert\n> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BABF5BDE6B\n\tfor ;\n\tTue, 13 Dec 2022 14:43:33 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 25F4663362;\n\tTue, 13 Dec 2022 15:43:33 +0100 (CET)","from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net\n\t[217.70.183.201])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 754DB63340\n\tfor ;\n\tTue, 13 Dec 2022 15:43:31 +0100 (CET)","(Authenticated sender: jacopo@jmondi.org)\n\tby mail.gandi.net (Postfix) with ESMTPSA id A71FD1BF204;\n\tTue, 13 Dec 2022 14:43:30 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670942613;\n\tbh=iDJuJv53AY5ZZRiO/Tlgt5uD7qCom51rSxu7lTo5PFQ=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=ZWdBCSlb4vdd5OUuzaOh/hcbFOqY3a/WC9VM/PUofOgT3ddPiLJ6haXz1ctQwhNhM\n\tHGJggcYSxPf5jFP7xc/y+5/BXEkISnPCHdLc5B8bLfLLWIBo5h8NL68136f/RXaQl4\n\tYalt6RQ5zqLQ0PqgptpwWIOXVFyPlF+hFR5WdYWuVIo5tp4jdTgeIAX23Owllw2NNF\n\tlT5EphqVuDXECfD0ZANwbtsKI4MFkePrXhvyR0Utq+Syz3IaFnviu+i57kzu54yfO5\n\tLZylOuL3pll8vDFplGw8Z2Cz8jYFyJ1Rs5tNZvL5um4x38pDhCj7TDA0465lkaEXN6\n\tggB4lTZA7uwzQ==","Date":"Tue, 13 Dec 2022 15:43:29 +0100","To":"David Plowman ","Message-ID":"<20221213144329.3pqib2cf4nkpfxqw@uno.localdomain>","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t\n\t<20221213112251.jnpgyzitusutkqm6@uno.localdomain>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jacopo Mondi via libcamera-devel ","Reply-To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26056,"web_url":"https://patchwork.libcamera.org/comment/26056/","msgid":"","date":"2022-12-13T14:59:54","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi all,\n\n\nOn Tue, 13 Dec 2022 at 14:30, David Plowman via libcamera-devel <\nlibcamera-devel@lists.libcamera.org> wrote:\n\n> Hi Jacopo\n>\n> Thanks for the reply. I won't write a great deal this time round, I\n> agree that possibly a call would be a good idea.\n>\n> On Tue, 13 Dec 2022 at 11:22, Jacopo Mondi wrote:\n> >\n> > Hi David\n> >\n> > On Mon, Dec 12, 2022 at 11:46:31AM +0000, David Plowman wrote:\n> > > Hi Jacopo\n> > >\n> > > Thanks for wading into the sea on this question! (Very appropriate now\n> > > that the stick person example has been turned into a shark...!)\n> > >\n> > > On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n> > > wrote:\n> > > >\n> > > > Hi again,\n> > > >\n> > > > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via\n> libcamera-devel wrote:\n> > > > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > > > > > Hi Robert\n> > > > > >\n> > > > > > Thanks for prodding us all on this question!\n> > > > >\n> > > > > Hi, thanks for the quick reply! And a pleasure :P\n> > > > >\n> > > > > > I looked back over this again, and found Jacopo's description of\n> the\n> > > > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > > > > > (\n> https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > > > > > is what I stumbled into)\n> > > > >\n> > > > > The current kernel docu can be found at\n> > > > >\n> > > > >\n> https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n> > >\n> > > Thanks for the updated reference. The stick person may have turned\n> > > into a shark, but apart from that I think the sense of the\n> > > documentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n> > > 90 degrees, then your image in memory looks like it has had a 90\n> > > degree *clockwise* rotation applied.\n> > >\n> >\n> > Yes!\n> >\n> > > > >\n> > > > > the libcamera one at\n> > > > >\n> > > > >\n> https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> > > > >\n> > > > > > From that, and the nice example with the stick person, I deduce\n> that\n> > > > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> > > > > > degrees, then the image you capture (if nothing else happens to\n> it)\n> > > > > > will look like it has had a 90 degree clockwise rotation\n> performed.\n> > > > > > Everyone agree with that?\n> > > > > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90\n> would\n> > > > > translate to Transform::Rot90.\n> > > >\n> > > > Let's here put some definitions in place\n> > > >\n> > > > V4L2_CID_CAMERA_SENSOR_ROTATION\n> > > > This read-only control describes the rotation correction in degrees\n> in\n> > > > the counter-clockwise direction to be applied to the captured images\n> > > > once captured to memory to compensate for the camera sensor mounting\n> > > > rotation.\n> > > >\n> > > > \\var Transform::Rot270\n> > > > Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n> > > >\n> > > > Let's start by saying we did a sub-optimal (to be nice) job in\n> > > > having Transform and V4L2_CID_ROTATION operate in two different\n> > > > directions. Before the usage of Transform in libcamera gets widely\n> > > > adopted sould we align the two ?\n> > >\n> > > I don't mind at all... we just have to decide what we want!\n> > >\n> > > >\n> > > > Hence if I'm not mistaken, if your camera has Rotation=90 and an\n> > > > application supplies Transform::Identity in\n> > > > CameraConfiguration::transform what you should get back is Rot270\n> > > >\n> > > > Correct ?\n> > >\n> > > The way things are currently, if the camera says Rotation=90, then\n> > > your image in memory looks like it is rotated 90 degrees clockwise, so\n> > > the Transform (if your pipeline and sensor can't apply any\n> > > flips/rotations at all) will come back as Rot90.\n> > >\n> > > That is, the only transform you can specify that won't be adjusted, is\n> > > a 90 degree clockwise rotation, so transform=Rot90 as things stand\n> > > currently.\n> > >\n> > > If we choose to change the sense of the libcamera.Transform rotation,\n> > > then we'd get transform=Rot270 instead.\n> > >\n> > > (If your pipeline handler can do flips and transposes, it may be able\n> > > to do what you ask and so the transform would come back unchanged and\n> > > would be \"Valid\".)\n> > >\n> > > >\n> > > > > > So in turn, if you have a camera/ISP that can apply no\n> transforms at\n> > > > > > all, then the only value for the user transform that will not\n> come\n> > > > > > back as adjusted is \"rot90\", because it looks like it's had a 90\n> > > > > > degree clockwise rotation performed (and libcamera transforms\n> count\n> > > > > > rotations as being clockwise). Does that also make sense?\n> > > >\n> > > > It might be the case today, but I wonder if that's desirable.\n> > > >\n> > > > If an application supplies CameraConfiguration::transform = Identity\n> > > > it means it doesn't want any rotation applied by the library. It will\n> > > > get back 270 to indicate that that's how the image is rotated and\n> will\n> > > > get an Adjusted configuration.\n> > >\n> > > I'm not quite sure I get that, perhaps it depends how we interpret\n> > > \"applied by the library\". I guess I agree if we mean \"applied by the\n> > > library together with any rotation of the sensor\"? Maybe a real\n> >\n> > \"Applied by the library\" == happens inside libcamera, ISP or sensor\n> > flips..\n> >\n> > > example is helpful:\n> > >\n> > > So for a Raspberry Pi, for example, our sensors are mostly mounted\n> > > upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\n> > > say the transform they want is transform=Identity i.e. images come\n> > > back \"the right way up\". Internally it means that we do apply an extra\n> > > h+v flip to undo the 180 degree rotation of the sensor.\n> > >\n> >\n> > Here you go, this is where we disconnect.\n> >\n> > As I've interpreted the API so far, and the way we designed\n> > properties::Rotation was to signal to the application what is the\n> > status of the image as it comes from the sensor. Application could then\n> > correct that by setting CameraConfiguration::transform, but in most\n> > cases the image will be corrected at rendering time by the application\n> > itself.\n> >\n> > IWO I was not expecting auto-correction and Transform::Identity to me\n> > means \"give me the images as they come from the sensor\"\n> >\n> > > >\n> > > > If instead an application supplies CameraConfiguration::transform =\n> Rot270\n> > > > and your camera -cannot- do any rotation, it will still receive back\n> > > > Rot270 and the state is Valid.\n> > >\n> > > I think that's true if we change the sense of the libcamera.Transform\n> > > rotation. Currently, as I said earlier, I think we get Rot90?\n> > >\n> > > >\n> > > > If it asks for CameraConfiguration::transform = Rot270 and the camera\n> > > > -can- do that, it will clear CameraConfiguration::transform to\n> > > > Identity and return Adjusted ?\n> > >\n> > > I'm not sure there... The CameraConfiguration::transform is the final\n> > > transform you want. If that can be done, it's left alone and you get\n> > > back Valid. It's not telling you what else you would need to do\n> > > afterwards.\n> > >\n> > > (To find out what you need to do afterwards you'd work out\n> > > \"inverse(what-you-will-get) * what-you-wanted\", or something like\n> > > that.)\n> > >\n> >\n> > What I would like is to make as easier as possible for applications\n> > to know what they have to do to have the image rotated in the way they\n> > expect. There are several \"transform\" steps on the image, the first\n> > one happens in libcamera, all the other ones are left to the\n> > appplication.\n>\n> I think this is exactly what I want, namely \"to make it as easy as\n> possible for applications\".\n>\n> On Raspberry Pi - and this has been the case for a couple of years\n> now! - if a user wants images the \"normal\" way up then they simply\n> pass Transform::Identity. If they want upside down images they pass\n> Transform::Rot180. It's as easy as that, there is simply nothing else\n> to do.\n>\n> Requiring all applications to look at the sensor rotation and then\n> combine various things to figure out what they have to ask for (and\n> possibly get it wrong) seems, well, actually quite unhelpful.\n>\n\nFWIW, I do agree that applications should not have to trouble themselves\nwith munging sensor rotation andrequested rotation to achieve their\ndesired outcome. This definitely wants to be hidden in the libcamera core\nor pipeline handler layers.\n\nNaush\n\n\n\n>\n> But this is indeed what we need to talk about. Let's organise a call!\n>\n> Thanks\n> David\n>\n> >\n> > *) The image \"native\" transform is reported by Rotation (this assumes\n> > that any implicit rotation buried in the sensor driver, as per our\n> > last discussion with Dave is not taken into account)\n> >\n> > properties::Rotation = Rot270 = HFlip | Transpose\n> >\n> > *) Application sets CameraConfiguration::transform and call validate() to\n> > check what can be done by libcamera and what should be done by next\n> > layers\n> >\n> > CameraConfigurat::Transform = Rot270\n> >\n> > *) The camera validate the transform and as it can only flip it sets\n> > transform to HFlip, and the state is Adjusted\n> >\n> > validate() -> The camera can only flip\n> > -> CameraConfiguration = HFlip\n> >\n> > *) Application gets back what the camera can do and instrument the\n> > rest of the rendering pipeline to perform the additional transforms\n> > the camera cannot do. To get what you have to do I presume you have to\n> >\n> > additionalTransform = what_you_want & !what_you_get\n> >\n> > So yes, I was interpreting the usage of CameraConfiguration::transform\n> > as a negotiation of what the library can do and what applications have\n> > to do by themselves, which if I got you is different than what's\n> > currecntly implemented ?\n> >\n> > > >\n> > > > Is this what happens (I'm looking at the above code with David's\n> > > > suggestion to use the inverse of roationTransform).\n> > > >\n> > > >\n> > > > > >\n> > > > > > So by a slightly arbitrary mixture of conventions, it looks to\n> me as\n> > > > > > though the use of the inverse that I queried is indeed wrong,\n> because\n> > > > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in\n> this\n> > > > > > case.\n> > > > > >\n> > > > > > Does that sound plausible... what do folks think?\n> > > > >\n> > > > > Ah, I think the question is whether the adjusted value means:\n> \"this is what\n> > > > > you'll get\" vs \"this is what you have to do yourself\".\n> > > > >\n> > > > > Above I argued in favor of the later, but I guess the first one is\n> what the\n> > > > > API is supposed to mean. That would imply that the client has\n> compute the\n> > > > > difference between requested and adjusted value itself.\n> > > >\n> > > > I always presumed the latter was meant to happen..\n> > >\n> > > Yes, I think this is the crux of the matter. I believe it's \"what you\n> > > will get\". It's not \"what you need to do after\". (If it was \"what you\n> > > need to do after\", it would change every time you try to validate the\n> > > configuration!)\n> > >\n> >\n> > On the \"it would change every time you try to validate the\n> > configuration\": only if the camera cannot satisfy the transform you\n> > have requested. If you ask for a flip and the camera can do it,\n> > transform stays un-touched and the state is Valid ?\n> >\n> > > >\n> > > > >\n> > > > > In my case that would mean:\n> > > > >\n> > > > > requested CameraConfiguration::transform: Transform::Identity,\n> > > > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > > > > CameraConfiguration::transform: Transform::Rot270\n> > > > >\n> > > >\n> > > > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n> > >\n> > > Yes, looks right to me!\n> > >\n> > > >\n> > > > > To compute what the client has to do, it would need to do:\n> > > > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And\n> then apply\n> > > > > that, a 90 degr. rotation clockwise.\n> > >\n> > > Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> > > what-you-wanted\", in this case inverse(rot270) * identity which is\n> > > rot90 (in the current clockwise sense).\n> > >\n> > > > >\n> > > > > Does that sound reasonable? In that case the patch here should\n> indeed be\n> > > > > `*transform = rotationTransform;`, without the inverse.\n> > > > >\n> > >\n> > > Yes, agree!\n> > >\n> > > >\n> > > > /0\\\n> > > >\n> > > > before we plumb this into and make assumptions that will be then\n> > > > set into stone in our adaption layers, can we take a step back and\n> > > > define what kind of API we would like to have ?\n> > > >\n> > > > Let's start by the definition of CameraConfiguration::transform.\n> > > >\n> > > > It's a member of CameraConfiguration, and as by definition it is\n> > > > filled-in by the application and eventually adjusted by the Camera to\n> > > > a value that's acceptable/doable for the current configuration.\n> > > >\n> > > > 1) Application to Camera = (counter?)clockwise degrees the camera\n> > > > system should rotate the image before presenting it to the user\n> > >\n> > > This may depend on what exactly we mean by \"the camera system\". I'd\n> > > explain it as\n> > > \"the final resulting transform that the application wants to be\n> > > applied to all the output images\". After all, this is actually the\n> > > only thing the application knows!\n> > >\n> >\n> > agreed\n> >\n> > > >\n> > > > 2) Camera to application = the rotation the camera can actually do\n> > > > (ie. only sensor flips == no Transpose)\n> > >\n> > > Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n> >\n> > yes, more correct\n> >\n> > > because flips aren't rotations, you could in theory have a PH that\n> > > could do transposes (even though we probably don't currently).\n> > >\n> >\n> > none currently afaict\n> >\n> > > >\n> > > > The current documentation doesn't explain how it gets adjusted\n> > >\n> > > This is true, though I also think it's platform dependent. If you had\n> > > a PH that can't do transposes but can do flips, you have a choice how\n> > > you handle (for example) rot90. You could say \"oh I can't do that, so\n> > > in this case I won't do anything at all\" and leave the application to\n> > > sort everything out. The Pi tries to say \"well, I'll do everything\n> > > except for the transpose, so an application only has to worry about\n> > > that one case\". But there is a choice...\n> > >\n> >\n> > I agree it's platform-specific, but the way transform is negotiated with\n> > application should be standardized\n> >\n> > > >\n> > > > /**\n> > > > * \\var CameraConfiguration::transform\n> > > > * \\brief User-specified transform to be applied to the image\n> > > > *\n> > > > * The transform is a user-specified 2D plane transform that will be\n> applied\n> > > > * to the camera images by the processing pipeline before being\n> handed to\n> > > > * the application. This is subsequent to any transform that is\n> already\n> > > > * required to fix up any platform-defined rotation.\n> > > > *\n> > > > * The usual 2D plane transforms are allowed here\n> (horizontal/vertical\n> > > > * flips, multiple of 90-degree rotations etc.), but the validate()\n> function\n> > > > * may adjust this field at its discretion if the selection is not\n> supported.\n> > > > */\n> > > >\n> > > > The last part in particular, \"at its discrection\" it's really too\n> > > > generic.\n> > >\n> > > I guess it's hinting at the fact that it's up to the PH, and we don't\n> > > mandate the exact way in which it can fail to do what you asked for!\n> > > But it would be fine to have a discussion on that.\n> > >\n> >\n> > That's the point that most concerns me :)\n> >\n> > > >\n> > > > Let's start from the beginning: what application should use transform\n> > > > for ? I presume they should:\n> > > >\n> > > > 1) Inspect propertis::Rotation\n> > > > 2) Correct the image for the properties::Rotation amount (actually,\n> > > > the inverse because of the clockwise/counter-clockwise thing)\n> > > > 3) Add any additional rotation they would like on top of this\n> > >\n> > > So I think the point of the transform is that you *don't* have to\n> > > worry about the properties::Rotation. It takes care of that for you.\n> > > You say \"I want upside down images\" and you get them, whatever the\n> > > sensor's rotation.\n> >\n> > That's acceptable as well if we prefer that, but it has to be\n> > standardized among pipelines imho\n> >\n> > >\n> > > >\n> > > > How should transform be adjusted ?\n> > > >\n> > > > Identity:\n> > > > If application->identity the camera does not have to apply any\n> > > > transform. Should the transform be adjutes like it happens today to\n> > > > compensate the Rotation (at least that's what I presume happens) ? I\n> > > > presume no, applications can infer rotation from properties and if\n> they\n> > > > chose Identity either other layers above will do the transform or\n> > > > they're fine with images as they are.\n> > >\n> > > If the application says it wants the \"identity\" transform, then it\n> > > should get output images with no transformation applied. But this does\n> > > mean it will undo the sensor rotation (this being the normal case on\n> > > the Pi).\n> > >\n> >\n> > right, it depends if we decide to account for the sensor rotation\n> > inside libcamera or not\n> >\n> > > >\n> > > > A supported/non-supported transform:\n> > > > An application asks for a VFLIP, the camera can do it, transform is\n> > > > not changed. If the camera cannot flip, the transform is set back to\n> > > > Identity and the state is adjusted.\n> > >\n> > > Yes, mostly, except that the sensor may be mounted with a rotation,\n> > > but if it isn't, then I agree with this.\n> > >\n> > > >\n> > > > A partially non-supported transform:\n> > > > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> > > > only do HFLIP. transform sould be then set to HFLIP and applications\n> > > > know that the remaining transformation should be handled elsewhere.\n> > >\n> > > As I suggested above, I'm not sure whether we want to mandate this. I\n> > > agree it seems useful behaviour, which is why the Pi does it, but I'm\n> > > open minded about requiring it or not.\n> > >\n> > > >\n> > > > Are we missing anything with such behaviour ? I presume it's rather\n> > > > similar to what happens today if not for the fact that the validate()\n> > > > implementation adjusts transform to rotation in case it cannot flip\n> > > >\n> > > > Transform combined = transform * data_->rotationTransform_;\n> > > > if (!data_->supportsFlips_ && !!combined)\n> > > > transform = -data_->rotationTransform_;\n> > > >\n> > > > Not really sure I got why.\n> > > >\n> > > > All of this, but an even more fundamental question: is it too late/is\n> > > > it worh it to adjust the Transform definition to adopt the same\n> > > > direction as the kernel defined rotation ?\n> > > >\n> > > > Sorry for the long email :/\n> > >\n> > > No problem! I like nothing more than discussing transforms, except\n> > > possibly colour spaces!\n> > >\n> >\n> > :)\n> >\n> > > More seriously, here are my key points:\n> >\n> > I'll try to summarize what I see differently\n> >\n> > >\n> > > 1. The CameraConfiguration::transform field says what the application\n> > > wants the final output images to have.\n> >\n> > 1. The CameraConfiguration::transform field says what transform the\n> > library has to apply to images as they arrive from the sensor in its\n> > default configuration\n> >\n> > > 2. It takes account of the sensor rotation. The most obvious example\n> > > is the Pi: if the sensor is mounted upside down, and the applications\n> > > ask for images \"the right way up\" (i.e. transform = identity), then\n> > > the PH will apply h and v flips to compensate for the sensor mounting.\n> >\n> > 2. It doesn't automatically account for sensor rotation. Application\n> > might decide to do that along the rendering pipeline, in example\n> >\n> > > 3. If the PH can do what the application requests, it leaves the\n> > > transform alone and your configuration is Valid.\n> >\n> > Agreed\n> >\n> > > 4. If the PH can't do what the application wants, it will set the\n> > > value to something that it can do, and the configuration will be\n> > > Adjusted.\n> >\n> > Agreed\n> >\n> > >\n> > > If folks wanted a short conference call to iron any of this out, I'd\n> > > be very happy to join in!\n> >\n> > I've asked others to have a look here, let's see if we need a call,\n> > I'm all for it !\n> >\n> > Thanks\n> > j\n> >\n> > >\n> > > Thanks\n> > > David\n> > >\n> > > >\n> > > > > > Thanks everyone!\n> > > > > > David\n> > > > >\n> > > > > Thanks!\n> > > > >\n> > > > > Robert\n> > > > >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D27DEC31E9\n\tfor ;\n\tTue, 13 Dec 2022 15:00:14 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 26A6763362;\n\tTue, 13 Dec 2022 16:00:14 +0100 (CET)","from mail-vk1-xa33.google.com (mail-vk1-xa33.google.com\n\t[IPv6:2607:f8b0:4864:20::a33])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 34D0B63340\n\tfor ;\n\tTue, 13 Dec 2022 16:00:12 +0100 (CET)","by mail-vk1-xa33.google.com with SMTP id f24so1664110vkl.9\n\tfor ;\n\tTue, 13 Dec 2022 07:00:12 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670943614;\n\tbh=a+oS0FEMqDyvKhX2EA+Bwqi6N3kkcO/OORMoZOLRbV0=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=oKvIm1jxKwGM6rhhAlUbjQQ19/ALaxTSxB2LZPKd/Nfp+zofjEa4h8XOoyYgmmOP9\n\tBDlGyTTaL0n6YrHtmZw49JJhLW1qWrbvvXumV4ChAw0FYhGMepnaZYyslGQbCI0AIK\n\t0fXY73MfqekP5h2bhAbUQlJK/3LuNRe4HKrtaQDhVN9XmMyrRjpl1CfnMmjLs0j0MK\n\tkXmOeS/OIMnhnFJZxtGv2WgTrfJvVNIzLOQkzY8YYToe+9yNAzrm6iEN/Vbu2MEfrz\n\tSuFMcP0dvzlzx3P4QgV7wdIvxLmhy6dg02CTSVBtMDfdJwzqvJSjRHAvhyYQ35zJ6i\n\tjO9jDggHG8Usg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=Qe6MnyVjiHmykAZxLEF3FH6b+i6ililwXSZZQqJy0eY=;\n\tb=keVq+eZZzRENBe+JdaScUMl1nTCGCMXXfrK0jZx7Qn3h4PRAlFJGJV7w2MUbIkcw1u\n\tdep4Bpv1ETgtgcohloBvgzHMmELWuOaulMovaaEgZu/lmF09NZ+dtEejuru9YFWN+og5\n\t2Um4KENZYF5Ee30/W4/NpkO3gU5dj9Fu+eWtyhasAQkhyqe7mf7TMN2NhgiFI8BQbvOh\n\t3y6kq+ZR8+4RkIIjmrSRkl49l2cuX9O3aIYYTEMF+QE94PvduPaKRzzHMH8Ms11oKSH5\n\tBJffkRngPNAXK8I1vnLdOafhbZn/hTq++IvnkeZJYrHL3koZXY2rks/gHFqtShi1jVW4\n\t/Vlw=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"keVq+eZZ\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=Qe6MnyVjiHmykAZxLEF3FH6b+i6ililwXSZZQqJy0eY=;\n\tb=7/9j0UfN01sCQBXShzq2xgK8rG+DkxOLx2LKoe/0qQSqJ2tr6xBbxaHyeuxfFk8X7o\n\t0wz5BSN0auYvdX4+M21vUGm9fxPrXXL1QBmDJqWe6Mkf3Xn6gkXZBUP1xCp0Z3O4pvTJ\n\tknVwlQVnv/uU1DIQA/g4EiP2Q1DpiCrQzr4e1HLfkVhvrAmG8jU4X12LZWBx4/b4Se23\n\tW1n/bh0WkluXz6xoJjWSDnsbrfkLFUFDuQU1LR52BZVpT6JRt7L5F/71v39unlgCor6A\n\tKJKUTQc25jXEaUTy+tCHcdJnWqfa1Rxb2Nw18+ydCCy6Xq4ZZ/LqWMgF8U5728LcBZx4\n\tGdpw==","X-Gm-Message-State":"ANoB5pmW1aEr5mtirbeOZmc40AKqTOTa1aWNFKJybyCUVfEuylGW8QZy\n\tvg8Ip2sTLR+/sRixhJ4/9SlmBqVJ9cAWHheeYi0I8gv75w9UxA==","X-Google-Smtp-Source":"AA0mqf7mDXxVZltAA5xMaDDM7j8SYqVSd9tHkN70zf+aJug7KuahpuWVdTu4eisPteSChZficXb/DBXjNbzxn4KhDdI=","X-Received":"by 2002:a05:6122:d8c:b0:3bd:fe9b:c10b with SMTP id\n\tbc12-20020a0561220d8c00b003bdfe9bc10bmr2995381vkb.19.1670943610767;\n\tTue, 13 Dec 2022 07:00:10 -0800 (PST)","MIME-Version":"1.0","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t\n\t<20221213112251.jnpgyzitusutkqm6@uno.localdomain>\n\t","In-Reply-To":"","Date":"Tue, 13 Dec 2022 14:59:54 +0000","Message-ID":"","To":"David Plowman ","Content-Type":"multipart/alternative; boundary=\"0000000000005c2acf05efb6e0df\"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26079,"web_url":"https://patchwork.libcamera.org/comment/26079/","msgid":"<1b0cd37c-75a1-03b7-7df7-cecf0deb0cc7@posteo.de>","date":"2022-12-14T15:55:46","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":142,"url":"https://patchwork.libcamera.org/api/people/142/","name":"Robert Mader","email":"robert.mader@posteo.de"},"content":"Hi,\n\njust wanted to drop that\n\n 1. I'm personally not very opinionated about the API design\n 2. the new Pipewire implementation matches the current Raspberry Pi code\n 3. I think/agree this patch series is the ideal place to\n change/stabilize this API\n 4. I'll happily adopt the Pipewire implementation as soon as a\n consensus is found / the implementation lands :)\n\nBest regards and thanks to everyone joining the discussion!\n\nRobert\n\nOn 13.12.22 15:59, Naushir Patuck via libcamera-devel wrote:\n> Hi all,\n>\n>\n> On Tue, 13 Dec 2022 at 14:30, David Plowman via libcamera-devel \n> wrote:\n>\n> Hi Jacopo\n>\n> Thanks for the reply. I won't write a great deal this time round, I\n> agree that possibly a call would be a good idea.\n>\n> On Tue, 13 Dec 2022 at 11:22, Jacopo Mondi wrote:\n> >\n> > Hi David\n> >\n> > On Mon, Dec 12, 2022 at 11:46:31AM +0000, David Plowman wrote:\n> > > Hi Jacopo\n> > >\n> > > Thanks for wading into the sea on this question! (Very\n> appropriate now\n> > > that the stick person example has been turned into a shark...!)\n> > >\n> > > On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n> > > wrote:\n> > > >\n> > > > Hi again,\n> > > >\n> > > > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via\n> libcamera-devel wrote:\n> > > > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > > > > > Hi Robert\n> > > > > >\n> > > > > > Thanks for prodding us all on this question!\n> > > > >\n> > > > > Hi, thanks for the quick reply! And a pleasure :P\n> > > > >\n> > > > > > I looked back over this again, and found Jacopo's\n> description of the\n> > > > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > > > > >\n> (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > > > > > is what I stumbled into)\n> > > > >\n> > > > > The current kernel docu can be found at\n> > > > >\n> > > > >\n> https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n> > >\n> > > Thanks for the updated reference. The stick person may have turned\n> > > into a shark, but apart from that I think the sense of the\n> > > documentation is unchanged. So if\n> V4L2_CID_CAMERA_SENSOR_ROTATION says\n> > > 90 degrees, then your image in memory looks like it has had a 90\n> > > degree *clockwise* rotation applied.\n> > >\n> >\n> > Yes!\n> >\n> > > > >\n> > > > > the libcamera one at\n> > > > >\n> > > > >\n> https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> > > > >\n> > > > > >  From that, and the nice example with the stick person,\n> I deduce that\n> > > > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION\n> property says 90\n> > > > > > degrees, then the image you capture (if nothing else\n> happens to it)\n> > > > > > will look like it has had a 90 degree clockwise rotation\n> performed.\n> > > > > > Everyone agree with that?\n> > > > > Yep, agreed. That would mean\n> V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> > > > > translate to Transform::Rot90.\n> > > >\n> > > > Let's here put some definitions in place\n> > > >\n> > > > V4L2_CID_CAMERA_SENSOR_ROTATION\n> > > > This read-only control describes the rotation correction in\n> degrees in\n> > > > the counter-clockwise direction to be applied to the\n> captured images\n> > > > once captured to memory to compensate for the camera sensor\n> mounting\n> > > > rotation.\n> > > >\n> > > >  \\var Transform::Rot270\n> > > >  Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n> > > >\n> > > >  Let's start by saying we did a sub-optimal (to be nice) job in\n> > > >  having Transform and V4L2_CID_ROTATION operate in two different\n> > > >  directions. Before the usage of Transform in libcamera gets\n> widely\n> > > >  adopted sould we align the two ?\n> > >\n> > > I don't mind at all... we just have to decide what we want!\n> > >\n> > > >\n> > > >  Hence if I'm not mistaken, if your camera has Rotation=90\n> and an\n> > > >  application supplies Transform::Identity in\n> > > >  CameraConfiguration::transform what you should get back is\n> Rot270\n> > > >\n> > > >  Correct ?\n> > >\n> > > The way things are currently, if the camera says Rotation=90, then\n> > > your image in memory looks like it is rotated 90 degrees\n> clockwise, so\n> > > the Transform (if your pipeline and sensor can't apply any\n> > > flips/rotations at all) will come back as Rot90.\n> > >\n> > > That is, the only transform you can specify that won't be\n> adjusted, is\n> > > a 90 degree clockwise rotation, so transform=Rot90 as things stand\n> > > currently.\n> > >\n> > > If we choose to change the sense of the libcamera.Transform\n> rotation,\n> > > then we'd get transform=Rot270 instead.\n> > >\n> > >  (If your pipeline handler can do flips and transposes, it may\n> be able\n> > > to do what you ask and so the transform would come back\n> unchanged and\n> > > would be \"Valid\".)\n> > >\n> > > >\n> > > > > > So in turn, if you have a camera/ISP that can apply no\n> transforms at\n> > > > > > all, then the only value for the user transform that\n> will not come\n> > > > > > back as adjusted is \"rot90\", because it looks like it's\n> had a 90\n> > > > > > degree clockwise rotation performed (and libcamera\n> transforms count\n> > > > > > rotations as being clockwise). Does that also make sense?\n> > > >\n> > > > It might be the case today, but I wonder if that's desirable.\n> > > >\n> > > > If an application supplies CameraConfiguration::transform =\n> Identity\n> > > > it means it doesn't want any rotation applied by the\n> library. It will\n> > > > get back 270 to indicate that that's how the image is\n> rotated and will\n> > > > get an Adjusted configuration.\n> > >\n> > > I'm not quite sure I get that, perhaps it depends how we interpret\n> > > \"applied by the library\". I guess I agree if we mean \"applied\n> by the\n> > > library together with any rotation of the sensor\"? Maybe a real\n> >\n> > \"Applied by the library\" == happens inside libcamera, ISP or sensor\n> > flips..\n> >\n> > > example is helpful:\n> > >\n> > > So for a Raspberry Pi, for example, our sensors are mostly mounted\n> > > upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our\n> users\n> > > say the transform they want is transform=Identity i.e. images come\n> > > back \"the right way up\". Internally it means that we do apply\n> an extra\n> > > h+v flip to undo the 180 degree rotation of the sensor.\n> > >\n> >\n> > Here you go, this is where we disconnect.\n> >\n> > As I've interpreted the API so far, and the way we designed\n> > properties::Rotation was to signal to the application what is the\n> > status of the image as it comes from the sensor. Application\n> could then\n> > correct that by setting CameraConfiguration::transform, but in most\n> > cases the image will be corrected at rendering time by the\n> application\n> > itself.\n> >\n> > IWO I was not expecting auto-correction and Transform::Identity\n> to me\n> > means \"give me the images as they come from the sensor\"\n> >\n> > > >\n> > > > If instead an application supplies\n> CameraConfiguration::transform = Rot270\n> > > > and your camera -cannot- do any rotation, it will still\n> receive back\n> > > > Rot270 and the state is Valid.\n> > >\n> > > I think that's true if we change the sense of the\n> libcamera.Transform\n> > > rotation. Currently, as I said earlier, I think we get Rot90?\n> > >\n> > > >\n> > > > If it asks for CameraConfiguration::transform = Rot270 and\n> the camera\n> > > > -can- do that, it will clear CameraConfiguration::transform to\n> > > > Identity and return Adjusted ?\n> > >\n> > > I'm not sure there... The CameraConfiguration::transform is\n> the final\n> > > transform you want. If that can be done, it's left alone and\n> you get\n> > > back Valid. It's not telling you what else you would need to do\n> > > afterwards.\n> > >\n> > > (To find out what you need to do afterwards you'd work out\n> > > \"inverse(what-you-will-get) * what-you-wanted\", or something like\n> > > that.)\n> > >\n> >\n> > What I would like is to make as easier as possible for applications\n> > to know what they have to do to have the image rotated in the\n> way they\n> > expect. There are several \"transform\" steps on the image, the first\n> > one happens in libcamera, all the other ones are left to the\n> > appplication.\n>\n> I think this is exactly what I want, namely \"to make it as easy as\n> possible for applications\".\n>\n> On Raspberry Pi - and this has been the case for a couple of years\n> now! - if a user wants images the \"normal\" way up then they simply\n> pass Transform::Identity. If they want upside down images they pass\n> Transform::Rot180. It's as easy as that, there is simply nothing else\n> to do.\n>\n> Requiring all applications to look at the sensor rotation and then\n> combine various things to figure out what they have to ask for (and\n> possibly get it wrong) seems, well, actually quite unhelpful.\n>\n>\n> FWIW, I do agree that applications should not have to trouble themselves\n> with munging sensor rotation andrequested rotation to achieve their\n> desired outcome.  This definitely wants to be hidden in the libcamera core\n> or pipeline handler layers.\n>\n> Naush\n>\n>\n> But this is indeed what we need to talk about. Let's organise a call!\n>\n> Thanks\n> David\n>\n> >\n> > *) The image \"native\" transform is reported by Rotation (this\n> assumes\n> > that any implicit rotation buried in the sensor driver, as per our\n> > last discussion with Dave is not taken into account)\n> >\n> >    properties::Rotation = Rot270 = HFlip | Transpose\n> >\n> > *) Application sets CameraConfiguration::transform and call\n> validate() to\n> > check what can be done by libcamera and what should be done by next\n> > layers\n> >\n> >    CameraConfigurat::Transform = Rot270\n> >\n> > *) The camera validate the transform and as it can only flip it sets\n> > transform to HFlip, and the state is Adjusted\n> >\n> >    validate() -> The camera can only flip\n> >               -> CameraConfiguration = HFlip\n> >\n> > *) Application gets back what the camera can do and instrument the\n> > rest of the rendering pipeline to perform the additional transforms\n> > the camera cannot do. To get what you have to do I presume you\n> have to\n> >\n> >         additionalTransform = what_you_want & !what_you_get\n> >\n> > So yes, I was interpreting the usage of\n> CameraConfiguration::transform\n> > as a negotiation of what the library can do and what\n> applications have\n> > to do by themselves, which if I got you is different than what's\n> > currecntly implemented ?\n> >\n> > > >\n> > > > Is this what happens (I'm looking at the above code with David's\n> > > > suggestion to use the inverse of roationTransform).\n> > > >\n> > > >\n> > > > > >\n> > > > > > So by a slightly arbitrary mixture of conventions, it\n> looks to me as\n> > > > > > though the use of the inverse that I queried is indeed\n> wrong, because\n> > > > > > we \"need camera 90 degree rotation\" => \"user transform\n> rot90\" in this\n> > > > > > case.\n> > > > > >\n> > > > > > Does that sound plausible... what do folks think?\n> > > > >\n> > > > > Ah, I think the question is whether the adjusted value\n> means: \"this is what\n> > > > > you'll get\" vs \"this is what you have to do yourself\".\n> > > > >\n> > > > > Above I argued in favor of the later, but I guess the\n> first one is what the\n> > > > > API is supposed to mean. That would imply that the client\n> has compute the\n> > > > > difference between requested and adjusted value itself.\n> > > >\n> > > > I always presumed the latter was meant to happen..\n> > >\n> > > Yes, I think this is the crux of the matter. I believe it's\n> \"what you\n> > > will get\". It's not \"what you need to do after\". (If it was\n> \"what you\n> > > need to do after\", it would change every time you try to\n> validate the\n> > > configuration!)\n> > >\n> >\n> > On the \"it would change every time you try to validate the\n> > configuration\": only if the camera cannot satisfy the transform you\n> > have requested. If you ask for a flip and the camera can do it,\n> > transform stays un-touched and the state is Valid ?\n> >\n> > > >\n> > > > >\n> > > > > In my case that would mean:\n> > > > >\n> > > > > requested CameraConfiguration::transform: Transform::Identity,\n> > > > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > > > > CameraConfiguration::transform: Transform::Rot270\n> > > > >\n> > > >\n> > > > I guess it's Rot90 because of the\n> clockwise/counter-clockwise issue ?\n> > >\n> > > Yes, looks right to me!\n> > >\n> > > >\n> > > > > To compute what the client has to do, it would need to do:\n> > > > > Transform::Identity - Transform::Rot270 =\n> Transform::Rot90. And then apply\n> > > > > that, a 90 degr. rotation clockwise.\n> > >\n> > > Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> > > what-you-wanted\", in this case inverse(rot270) * identity which is\n> > > rot90 (in the current clockwise sense).\n> > >\n> > > > >\n> > > > > Does that sound reasonable? In that case the patch here\n> should indeed be\n> > > > > `*transform = rotationTransform;`, without the inverse.\n> > > > >\n> > >\n> > > Yes, agree!\n> > >\n> > > >\n> > > >         /0\\\n> > > >\n> > > > before we plumb this into and make assumptions that will be then\n> > > > set into stone in our adaption layers, can we take a step\n> back and\n> > > > define what kind of API we would like to have ?\n> > > >\n> > > > Let's start by the definition of CameraConfiguration::transform.\n> > > >\n> > > > It's a member of CameraConfiguration, and as by definition it is\n> > > > filled-in by the application and eventually adjusted by the\n> Camera to\n> > > > a value that's acceptable/doable for the current configuration.\n> > > >\n> > > > 1) Application to Camera  = (counter?)clockwise degrees the\n> camera\n> > > > system should rotate the image before presenting it to the user\n> > >\n> > > This may depend on what exactly we mean by \"the camera\n> system\". I'd\n> > > explain it as\n> > > \"the final resulting transform that the application wants to be\n> > > applied to all the output images\". After all, this is actually the\n> > > only thing the application knows!\n> > >\n> >\n> > agreed\n> >\n> > > >\n> > > > 2) Camera to application = the rotation the camera can\n> actually do\n> > > > (ie. only sensor flips == no Transpose)\n> > >\n> > > Agree, mostly, though I would say \"transform\" rather than\n> \"rotation\"\n> >\n> > yes, more correct\n> >\n> > > because flips aren't rotations, you could in theory have a PH that\n> > > could do transposes (even though we probably don't currently).\n> > >\n> >\n> > none currently afaict\n> >\n> > > >\n> > > > The current documentation doesn't explain how it gets adjusted\n> > >\n> > > This is true, though I also think it's platform dependent. If\n> you had\n> > > a PH that can't do transposes but can do flips, you have a\n> choice how\n> > > you handle (for example) rot90. You could say \"oh I can't do\n> that, so\n> > > in this case I won't do anything at all\" and leave the\n> application to\n> > > sort everything out. The Pi tries to say \"well, I'll do everything\n> > > except for the transpose, so an application only has to worry\n> about\n> > > that one case\". But there is a choice...\n> > >\n> >\n> > I agree it's platform-specific, but the way transform is\n> negotiated with\n> > application should be standardized\n> >\n> > > >\n> > > > /**\n> > > >  * \\var CameraConfiguration::transform\n> > > >  * \\brief User-specified transform to be applied to the image\n> > > >  *\n> > > >  * The transform is a user-specified 2D plane transform that\n> will be applied\n> > > >  * to the camera images by the processing pipeline before\n> being handed to\n> > > >  * the application. This is subsequent to any transform that\n> is already\n> > > >  * required to fix up any platform-defined rotation.\n> > > >  *\n> > > >  * The usual 2D plane transforms are allowed here\n> (horizontal/vertical\n> > > >  * flips, multiple of 90-degree rotations etc.), but the\n> validate() function\n> > > >  * may adjust this field at its discretion if the selection\n> is not supported.\n> > > >  */\n> > > >\n> > > > The last part in particular, \"at its discrection\" it's\n> really too\n> > > > generic.\n> > >\n> > > I guess it's hinting at the fact that it's up to the PH, and\n> we don't\n> > > mandate the exact way in which it can fail to do what you\n> asked for!\n> > > But it would be fine to have a discussion on that.\n> > >\n> >\n> > That's the point that most concerns me :)\n> >\n> > > >\n> > > > Let's start from the beginning: what application should use\n> transform\n> > > > for ? I presume they should:\n> > > >\n> > > > 1) Inspect propertis::Rotation\n> > > > 2) Correct the image for the properties::Rotation amount\n> (actually,\n> > > > the inverse because of the clockwise/counter-clockwise thing)\n> > > > 3) Add any additional rotation they would like on top of this\n> > >\n> > > So I think the point of the transform is that you *don't* have to\n> > > worry about the properties::Rotation. It takes care of that\n> for you.\n> > > You say \"I want upside down images\" and you get them, whatever the\n> > > sensor's rotation.\n> >\n> > That's acceptable as well if we prefer that, but it has to be\n> > standardized among pipelines imho\n> >\n> > >\n> > > >\n> > > > How should transform be adjusted ?\n> > > >\n> > > > Identity:\n> > > > If application->identity the camera does not have to apply any\n> > > > transform. Should the transform be adjutes like it happens\n> today to\n> > > > compensate the Rotation (at least that's what I presume\n> happens) ? I\n> > > > presume no, applications can infer rotation from properties\n> and if they\n> > > > chose Identity either other layers above will do the\n> transform or\n> > > > they're fine with images as they are.\n> > >\n> > > If the application says it wants the \"identity\" transform, then it\n> > > should get output images with no transformation applied. But\n> this does\n> > > mean it will undo the sensor rotation (this being the normal\n> case on\n> > > the Pi).\n> > >\n> >\n> > right, it depends if we decide to account for the sensor rotation\n> > inside libcamera or not\n> >\n> > > >\n> > > > A supported/non-supported transform:\n> > > > An application asks for a VFLIP, the camera can do it,\n> transform is\n> > > > not changed. If the camera cannot flip, the transform is set\n> back to\n> > > > Identity and the state is adjusted.\n> > >\n> > > Yes, mostly, except that the sensor may be mounted with a\n> rotation,\n> > > but if it isn't, then I agree with this.\n> > >\n> > > >\n> > > > A partially non-supported transform:\n> > > > An application asks for Rot270 (Transpose | HFLIP) and the\n> camera can\n> > > > only do HFLIP. transform sould be then set to HFLIP and\n> applications\n> > > > know that the remaining transformation should be handled\n> elsewhere.\n> > >\n> > > As I suggested above, I'm not sure whether we want to mandate\n> this. I\n> > > agree it seems useful behaviour, which is why the Pi does it,\n> but I'm\n> > > open minded about requiring it or not.\n> > >\n> > > >\n> > > > Are we missing anything with such behaviour ? I presume it's\n> rather\n> > > > similar to what happens today if not for the fact that the\n> validate()\n> > > > implementation adjusts transform to rotation in case it\n> cannot flip\n> > > >\n> > > >         Transform combined = transform *\n> data_->rotationTransform_;\n> > > >         if (!data_->supportsFlips_ && !!combined)\n> > > >                 transform = -data_->rotationTransform_;\n> > > >\n> > > > Not really sure I got why.\n> > > >\n> > > > All of this, but an even more fundamental question: is it\n> too late/is\n> > > > it worh it to adjust the Transform definition to adopt the same\n> > > > direction as the kernel defined rotation ?\n> > > >\n> > > > Sorry for the long email :/\n> > >\n> > > No problem! I like nothing more than discussing transforms, except\n> > > possibly colour spaces!\n> > >\n> >\n> > :)\n> >\n> > > More seriously, here are my key points:\n> >\n> > I'll try to summarize what I see differently\n> >\n> > >\n> > > 1. The CameraConfiguration::transform field says what the\n> application\n> > > wants the final output images to have.\n> >\n> > 1. The CameraConfiguration::transform field says what transform the\n> > library has to apply to images as they arrive from the sensor in its\n> > default configuration\n> >\n> > > 2. It takes account of the sensor rotation. The most obvious\n> example\n> > > is the Pi: if the sensor is mounted upside down, and the\n> applications\n> > > ask for images \"the right way up\" (i.e. transform = identity),\n> then\n> > > the PH will apply h and v flips to compensate for the sensor\n> mounting.\n> >\n> > 2. It doesn't automatically account for sensor rotation. Application\n> > might decide to do that along the rendering pipeline, in example\n> >\n> > > 3. If the PH can do what the application requests, it leaves the\n> > > transform alone and your configuration is Valid.\n> >\n> > Agreed\n> >\n> > > 4. If the PH can't do what the application wants, it will set the\n> > > value to something that it can do, and the configuration will be\n> > > Adjusted.\n> >\n> > Agreed\n> >\n> > >\n> > > If folks wanted a short conference call to iron any of this\n> out, I'd\n> > > be very happy to join in!\n> >\n> > I've asked others to have a look here, let's see if we need a call,\n> > I'm all for it !\n> >\n> > Thanks\n> >   j\n> >\n> > >\n> > > Thanks\n> > > David\n> > >\n> > > >\n> > > > > > Thanks everyone!\n> > > > > > David\n> > > > >\n> > > > > Thanks!\n> > > > >\n> > > > > Robert\n> > > > >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9F44EC31E9\n\tfor ;\n\tWed, 14 Dec 2022 15:55:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 152B963363;\n\tWed, 14 Dec 2022 16:55:50 +0100 (CET)","from mout01.posteo.de (mout01.posteo.de [185.67.36.65])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AA19461F23\n\tfor ;\n\tWed, 14 Dec 2022 16:55:48 +0100 (CET)","from submission (posteo.de [185.67.36.169]) \n\tby mout01.posteo.de (Postfix) with ESMTPS id 2832824002B\n\tfor ;\n\tWed, 14 Dec 2022 16:55:48 +0100 (CET)","from customer (localhost [127.0.0.1])\n\tby submission (posteo.de) with ESMTPSA id 4NXKhg02bbz6tpd\n\tfor ;\n\tWed, 14 Dec 2022 16:55:46 +0100 (CET)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1671033350;\n\tbh=dnO9CkVEwJstwrKbPCizy1RuFhkivLSKjjgg2IQGsOc=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=xDRjcM1p9+YP2fBwSpKn+d2b0uoQF4CmIr4qNO0S0lnVnw5r9d4+ldN+84UpNTZVC\n\tc3GnN+tapb5YRfruo/hN/1FV9gmZBJbL5hVBfB7aIoqpKLAznNeR+EMEd01bGG2miy\n\tNFBEXpDIwGgMTsksGcWcDuE/7qp9d0PHwlQn9E5LMVnBG6BC1yvUq4uLbsCH5OdYHj\n\toBaRAwdQiQdZnX9FI7Qc0hmat+5CLXFa5xofvn/QbOIKnJ99PXPtGC1kTRTdHaaIgj\n\tgKmuo/nY7C1VkIq+CxFWf/RnuNWQDHCn16iEjvsjyaVhmGFAdWxOOA44jrS352uNZ5\n\t1dGaK8WLeX8JA==","v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017;\n\tt=1671033348; bh=dnO9CkVEwJstwrKbPCizy1RuFhkivLSKjjgg2IQGsOc=;\n\th=Date:Subject:To:From:From;\n\tb=al/WiFrG16Jz5RZfLixLKdKw8khkqcbS84Fntj2w79aUfXw+tjBKLFDvkmRAd2cSu\n\tav1Ek0uaxgjBwX9aSlR5CAncud3CuWJ4kchntsrp4IEN5e6HAEohxrNESWWDvH4sgv\n\tSNBLKKlWn2OYVPGTNKX+ukoTSGvyPhbLwAmfEoGzOeeP6F9yWPbMSTKpD9LM5NJuUC\n\tw/Y8JJyiMMdcmr0bIcfMi63WigzYLJTb8ADSE7pEm/J+vGgTNQa1SGfHrd//NUZ5kz\n\t8uNVyIqy9QNYU+ydPcpbE0hVa7rlIfOZH8d1ObG+ORdsZbhdwXrhVeJ+GBbjKehajf\n\ttsivx5mq92NYw=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=posteo.de header.i=@posteo.de\n\theader.b=\"al/WiFrG\"; dkim-atps=neutral","Content-Type":"multipart/alternative;\n\tboundary=\"------------dItWjmOiq587KyZ563YnUrhQ\"","Message-ID":"<1b0cd37c-75a1-03b7-7df7-cecf0deb0cc7@posteo.de>","Date":"Wed, 14 Dec 2022 15:55:46 +0000","MIME-Version":"1.0","To":"libcamera-devel@lists.libcamera.org","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t\n\t<20221213112251.jnpgyzitusutkqm6@uno.localdomain>\n\t\n\t","Content-Language":"en-US","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Robert Mader via libcamera-devel ","Reply-To":"Robert Mader ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26216,"web_url":"https://patchwork.libcamera.org/comment/26216/","msgid":"","date":"2023-01-12T11:18:51","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi everyone\n\nI wanted to give this discussion a little nudge because I don't think\nit was ever quite concluded, perhaps Christmas got in the way!\n\nTo summarise a little bit, at least from my point of view, my\nimpression was that the current approach is correct, and the transform\nfield is for the user to request the transform that they want, and may\nget adjusted to be the transform that they will get. Details of any\nrotation in the way the sensor is mounted is hidden from the user -\nthey can just say \"I want the image the right way up\" and it\n\"magically\" happens. (This is exactly how the Raspberry Pi pipeline\nhandler works today.)\n\nThere was a small detail about whether one of the transforms in the\ncode wanted reversing, that's mostly a matter of conventions about\nusing clockwise or anticlockwise rotations (or the direction of the\nrotation axis).\n\nDoes anyone have anything else to add here?\n\nThanks!\nDavid\n\nOn Wed, 14 Dec 2022 at 15:55, Robert Mader via libcamera-devel\n wrote:\n>\n> Hi,\n>\n> just wanted to drop that\n>\n> I'm personally not very opinionated about the API design\n> the new Pipewire implementation matches the current Raspberry Pi code\n> I think/agree this patch series is the ideal place to change/stabilize this API\n> I'll happily adopt the Pipewire implementation as soon as a consensus is found / the implementation lands :)\n>\n> Best regards and thanks to everyone joining the discussion!\n>\n> Robert\n>\n> On 13.12.22 15:59, Naushir Patuck via libcamera-devel wrote:\n>\n> Hi all,\n>\n>\n> On Tue, 13 Dec 2022 at 14:30, David Plowman via libcamera-devel wrote:\n>>\n>> Hi Jacopo\n>>\n>> Thanks for the reply. I won't write a great deal this time round, I\n>> agree that possibly a call would be a good idea.\n>>\n>> On Tue, 13 Dec 2022 at 11:22, Jacopo Mondi wrote:\n>> >\n>> > Hi David\n>> >\n>> > On Mon, Dec 12, 2022 at 11:46:31AM +0000, David Plowman wrote:\n>> > > Hi Jacopo\n>> > >\n>> > > Thanks for wading into the sea on this question! (Very appropriate now\n>> > > that the stick person example has been turned into a shark...!)\n>> > >\n>> > > On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n>> > > wrote:\n>> > > >\n>> > > > Hi again,\n>> > > >\n>> > > > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n>> > > > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n>> > > > > > Hi Robert\n>> > > > > >\n>> > > > > > Thanks for prodding us all on this question!\n>> > > > >\n>> > > > > Hi, thanks for the quick reply! And a pleasure :P\n>> > > > >\n>> > > > > > I looked back over this again, and found Jacopo's description of the\n>> > > > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n>> > > > > > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n>> > > > > > is what I stumbled into)\n>> > > > >\n>> > > > > The current kernel docu can be found at\n>> > > > >\n>> > > > > https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n>> > >\n>> > > Thanks for the updated reference. The stick person may have turned\n>> > > into a shark, but apart from that I think the sense of the\n>> > > documentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n>> > > 90 degrees, then your image in memory looks like it has had a 90\n>> > > degree *clockwise* rotation applied.\n>> > >\n>> >\n>> > Yes!\n>> >\n>> > > > >\n>> > > > > the libcamera one at\n>> > > > >\n>> > > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n>> > > > >\n>> > > > > > From that, and the nice example with the stick person, I deduce that\n>> > > > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n>> > > > > > degrees, then the image you capture (if nothing else happens to it)\n>> > > > > > will look like it has had a 90 degree clockwise rotation performed.\n>> > > > > > Everyone agree with that?\n>> > > > > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n>> > > > > translate to Transform::Rot90.\n>> > > >\n>> > > > Let's here put some definitions in place\n>> > > >\n>> > > > V4L2_CID_CAMERA_SENSOR_ROTATION\n>> > > > This read-only control describes the rotation correction in degrees in\n>> > > > the counter-clockwise direction to be applied to the captured images\n>> > > > once captured to memory to compensate for the camera sensor mounting\n>> > > > rotation.\n>> > > >\n>> > > > \\var Transform::Rot270\n>> > > > Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n>> > > >\n>> > > > Let's start by saying we did a sub-optimal (to be nice) job in\n>> > > > having Transform and V4L2_CID_ROTATION operate in two different\n>> > > > directions. Before the usage of Transform in libcamera gets widely\n>> > > > adopted sould we align the two ?\n>> > >\n>> > > I don't mind at all... we just have to decide what we want!\n>> > >\n>> > > >\n>> > > > Hence if I'm not mistaken, if your camera has Rotation=90 and an\n>> > > > application supplies Transform::Identity in\n>> > > > CameraConfiguration::transform what you should get back is Rot270\n>> > > >\n>> > > > Correct ?\n>> > >\n>> > > The way things are currently, if the camera says Rotation=90, then\n>> > > your image in memory looks like it is rotated 90 degrees clockwise, so\n>> > > the Transform (if your pipeline and sensor can't apply any\n>> > > flips/rotations at all) will come back as Rot90.\n>> > >\n>> > > That is, the only transform you can specify that won't be adjusted, is\n>> > > a 90 degree clockwise rotation, so transform=Rot90 as things stand\n>> > > currently.\n>> > >\n>> > > If we choose to change the sense of the libcamera.Transform rotation,\n>> > > then we'd get transform=Rot270 instead.\n>> > >\n>> > > (If your pipeline handler can do flips and transposes, it may be able\n>> > > to do what you ask and so the transform would come back unchanged and\n>> > > would be \"Valid\".)\n>> > >\n>> > > >\n>> > > > > > So in turn, if you have a camera/ISP that can apply no transforms at\n>> > > > > > all, then the only value for the user transform that will not come\n>> > > > > > back as adjusted is \"rot90\", because it looks like it's had a 90\n>> > > > > > degree clockwise rotation performed (and libcamera transforms count\n>> > > > > > rotations as being clockwise). Does that also make sense?\n>> > > >\n>> > > > It might be the case today, but I wonder if that's desirable.\n>> > > >\n>> > > > If an application supplies CameraConfiguration::transform = Identity\n>> > > > it means it doesn't want any rotation applied by the library. It will\n>> > > > get back 270 to indicate that that's how the image is rotated and will\n>> > > > get an Adjusted configuration.\n>> > >\n>> > > I'm not quite sure I get that, perhaps it depends how we interpret\n>> > > \"applied by the library\". I guess I agree if we mean \"applied by the\n>> > > library together with any rotation of the sensor\"? Maybe a real\n>> >\n>> > \"Applied by the library\" == happens inside libcamera, ISP or sensor\n>> > flips..\n>> >\n>> > > example is helpful:\n>> > >\n>> > > So for a Raspberry Pi, for example, our sensors are mostly mounted\n>> > > upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\n>> > > say the transform they want is transform=Identity i.e. images come\n>> > > back \"the right way up\". Internally it means that we do apply an extra\n>> > > h+v flip to undo the 180 degree rotation of the sensor.\n>> > >\n>> >\n>> > Here you go, this is where we disconnect.\n>> >\n>> > As I've interpreted the API so far, and the way we designed\n>> > properties::Rotation was to signal to the application what is the\n>> > status of the image as it comes from the sensor. Application could then\n>> > correct that by setting CameraConfiguration::transform, but in most\n>> > cases the image will be corrected at rendering time by the application\n>> > itself.\n>> >\n>> > IWO I was not expecting auto-correction and Transform::Identity to me\n>> > means \"give me the images as they come from the sensor\"\n>> >\n>> > > >\n>> > > > If instead an application supplies CameraConfiguration::transform = Rot270\n>> > > > and your camera -cannot- do any rotation, it will still receive back\n>> > > > Rot270 and the state is Valid.\n>> > >\n>> > > I think that's true if we change the sense of the libcamera.Transform\n>> > > rotation. Currently, as I said earlier, I think we get Rot90?\n>> > >\n>> > > >\n>> > > > If it asks for CameraConfiguration::transform = Rot270 and the camera\n>> > > > -can- do that, it will clear CameraConfiguration::transform to\n>> > > > Identity and return Adjusted ?\n>> > >\n>> > > I'm not sure there... The CameraConfiguration::transform is the final\n>> > > transform you want. If that can be done, it's left alone and you get\n>> > > back Valid. It's not telling you what else you would need to do\n>> > > afterwards.\n>> > >\n>> > > (To find out what you need to do afterwards you'd work out\n>> > > \"inverse(what-you-will-get) * what-you-wanted\", or something like\n>> > > that.)\n>> > >\n>> >\n>> > What I would like is to make as easier as possible for applications\n>> > to know what they have to do to have the image rotated in the way they\n>> > expect. There are several \"transform\" steps on the image, the first\n>> > one happens in libcamera, all the other ones are left to the\n>> > appplication.\n>>\n>> I think this is exactly what I want, namely \"to make it as easy as\n>> possible for applications\".\n>>\n>> On Raspberry Pi - and this has been the case for a couple of years\n>> now! - if a user wants images the \"normal\" way up then they simply\n>> pass Transform::Identity. If they want upside down images they pass\n>> Transform::Rot180. It's as easy as that, there is simply nothing else\n>> to do.\n>>\n>> Requiring all applications to look at the sensor rotation and then\n>> combine various things to figure out what they have to ask for (and\n>> possibly get it wrong) seems, well, actually quite unhelpful.\n>\n>\n> FWIW, I do agree that applications should not have to trouble themselves\n> with munging sensor rotation andrequested rotation to achieve their\n> desired outcome. This definitely wants to be hidden in the libcamera core\n> or pipeline handler layers.\n>\n> Naush\n>\n>\n>>\n>>\n>> But this is indeed what we need to talk about. Let's organise a call!\n>>\n>> Thanks\n>> David\n>>\n>> >\n>> > *) The image \"native\" transform is reported by Rotation (this assumes\n>> > that any implicit rotation buried in the sensor driver, as per our\n>> > last discussion with Dave is not taken into account)\n>> >\n>> > properties::Rotation = Rot270 = HFlip | Transpose\n>> >\n>> > *) Application sets CameraConfiguration::transform and call validate() to\n>> > check what can be done by libcamera and what should be done by next\n>> > layers\n>> >\n>> > CameraConfigurat::Transform = Rot270\n>> >\n>> > *) The camera validate the transform and as it can only flip it sets\n>> > transform to HFlip, and the state is Adjusted\n>> >\n>> > validate() -> The camera can only flip\n>> > -> CameraConfiguration = HFlip\n>> >\n>> > *) Application gets back what the camera can do and instrument the\n>> > rest of the rendering pipeline to perform the additional transforms\n>> > the camera cannot do. To get what you have to do I presume you have to\n>> >\n>> > additionalTransform = what_you_want & !what_you_get\n>> >\n>> > So yes, I was interpreting the usage of CameraConfiguration::transform\n>> > as a negotiation of what the library can do and what applications have\n>> > to do by themselves, which if I got you is different than what's\n>> > currecntly implemented ?\n>> >\n>> > > >\n>> > > > Is this what happens (I'm looking at the above code with David's\n>> > > > suggestion to use the inverse of roationTransform).\n>> > > >\n>> > > >\n>> > > > > >\n>> > > > > > So by a slightly arbitrary mixture of conventions, it looks to me as\n>> > > > > > though the use of the inverse that I queried is indeed wrong, because\n>> > > > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n>> > > > > > case.\n>> > > > > >\n>> > > > > > Does that sound plausible... what do folks think?\n>> > > > >\n>> > > > > Ah, I think the question is whether the adjusted value means: \"this is what\n>> > > > > you'll get\" vs \"this is what you have to do yourself\".\n>> > > > >\n>> > > > > Above I argued in favor of the later, but I guess the first one is what the\n>> > > > > API is supposed to mean. That would imply that the client has compute the\n>> > > > > difference between requested and adjusted value itself.\n>> > > >\n>> > > > I always presumed the latter was meant to happen..\n>> > >\n>> > > Yes, I think this is the crux of the matter. I believe it's \"what you\n>> > > will get\". It's not \"what you need to do after\". (If it was \"what you\n>> > > need to do after\", it would change every time you try to validate the\n>> > > configuration!)\n>> > >\n>> >\n>> > On the \"it would change every time you try to validate the\n>> > configuration\": only if the camera cannot satisfy the transform you\n>> > have requested. If you ask for a flip and the camera can do it,\n>> > transform stays un-touched and the state is Valid ?\n>> >\n>> > > >\n>> > > > >\n>> > > > > In my case that would mean:\n>> > > > >\n>> > > > > requested CameraConfiguration::transform: Transform::Identity,\n>> > > > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n>> > > > > CameraConfiguration::transform: Transform::Rot270\n>> > > > >\n>> > > >\n>> > > > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n>> > >\n>> > > Yes, looks right to me!\n>> > >\n>> > > >\n>> > > > > To compute what the client has to do, it would need to do:\n>> > > > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n>> > > > > that, a 90 degr. rotation clockwise.\n>> > >\n>> > > Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n>> > > what-you-wanted\", in this case inverse(rot270) * identity which is\n>> > > rot90 (in the current clockwise sense).\n>> > >\n>> > > > >\n>> > > > > Does that sound reasonable? In that case the patch here should indeed be\n>> > > > > `*transform = rotationTransform;`, without the inverse.\n>> > > > >\n>> > >\n>> > > Yes, agree!\n>> > >\n>> > > >\n>> > > > /0\\\n>> > > >\n>> > > > before we plumb this into and make assumptions that will be then\n>> > > > set into stone in our adaption layers, can we take a step back and\n>> > > > define what kind of API we would like to have ?\n>> > > >\n>> > > > Let's start by the definition of CameraConfiguration::transform.\n>> > > >\n>> > > > It's a member of CameraConfiguration, and as by definition it is\n>> > > > filled-in by the application and eventually adjusted by the Camera to\n>> > > > a value that's acceptable/doable for the current configuration.\n>> > > >\n>> > > > 1) Application to Camera = (counter?)clockwise degrees the camera\n>> > > > system should rotate the image before presenting it to the user\n>> > >\n>> > > This may depend on what exactly we mean by \"the camera system\". I'd\n>> > > explain it as\n>> > > \"the final resulting transform that the application wants to be\n>> > > applied to all the output images\". After all, this is actually the\n>> > > only thing the application knows!\n>> > >\n>> >\n>> > agreed\n>> >\n>> > > >\n>> > > > 2) Camera to application = the rotation the camera can actually do\n>> > > > (ie. only sensor flips == no Transpose)\n>> > >\n>> > > Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n>> >\n>> > yes, more correct\n>> >\n>> > > because flips aren't rotations, you could in theory have a PH that\n>> > > could do transposes (even though we probably don't currently).\n>> > >\n>> >\n>> > none currently afaict\n>> >\n>> > > >\n>> > > > The current documentation doesn't explain how it gets adjusted\n>> > >\n>> > > This is true, though I also think it's platform dependent. If you had\n>> > > a PH that can't do transposes but can do flips, you have a choice how\n>> > > you handle (for example) rot90. You could say \"oh I can't do that, so\n>> > > in this case I won't do anything at all\" and leave the application to\n>> > > sort everything out. The Pi tries to say \"well, I'll do everything\n>> > > except for the transpose, so an application only has to worry about\n>> > > that one case\". But there is a choice...\n>> > >\n>> >\n>> > I agree it's platform-specific, but the way transform is negotiated with\n>> > application should be standardized\n>> >\n>> > > >\n>> > > > /**\n>> > > > * \\var CameraConfiguration::transform\n>> > > > * \\brief User-specified transform to be applied to the image\n>> > > > *\n>> > > > * The transform is a user-specified 2D plane transform that will be applied\n>> > > > * to the camera images by the processing pipeline before being handed to\n>> > > > * the application. This is subsequent to any transform that is already\n>> > > > * required to fix up any platform-defined rotation.\n>> > > > *\n>> > > > * The usual 2D plane transforms are allowed here (horizontal/vertical\n>> > > > * flips, multiple of 90-degree rotations etc.), but the validate() function\n>> > > > * may adjust this field at its discretion if the selection is not supported.\n>> > > > */\n>> > > >\n>> > > > The last part in particular, \"at its discrection\" it's really too\n>> > > > generic.\n>> > >\n>> > > I guess it's hinting at the fact that it's up to the PH, and we don't\n>> > > mandate the exact way in which it can fail to do what you asked for!\n>> > > But it would be fine to have a discussion on that.\n>> > >\n>> >\n>> > That's the point that most concerns me :)\n>> >\n>> > > >\n>> > > > Let's start from the beginning: what application should use transform\n>> > > > for ? I presume they should:\n>> > > >\n>> > > > 1) Inspect propertis::Rotation\n>> > > > 2) Correct the image for the properties::Rotation amount (actually,\n>> > > > the inverse because of the clockwise/counter-clockwise thing)\n>> > > > 3) Add any additional rotation they would like on top of this\n>> > >\n>> > > So I think the point of the transform is that you *don't* have to\n>> > > worry about the properties::Rotation. It takes care of that for you.\n>> > > You say \"I want upside down images\" and you get them, whatever the\n>> > > sensor's rotation.\n>> >\n>> > That's acceptable as well if we prefer that, but it has to be\n>> > standardized among pipelines imho\n>> >\n>> > >\n>> > > >\n>> > > > How should transform be adjusted ?\n>> > > >\n>> > > > Identity:\n>> > > > If application->identity the camera does not have to apply any\n>> > > > transform. Should the transform be adjutes like it happens today to\n>> > > > compensate the Rotation (at least that's what I presume happens) ? I\n>> > > > presume no, applications can infer rotation from properties and if they\n>> > > > chose Identity either other layers above will do the transform or\n>> > > > they're fine with images as they are.\n>> > >\n>> > > If the application says it wants the \"identity\" transform, then it\n>> > > should get output images with no transformation applied. But this does\n>> > > mean it will undo the sensor rotation (this being the normal case on\n>> > > the Pi).\n>> > >\n>> >\n>> > right, it depends if we decide to account for the sensor rotation\n>> > inside libcamera or not\n>> >\n>> > > >\n>> > > > A supported/non-supported transform:\n>> > > > An application asks for a VFLIP, the camera can do it, transform is\n>> > > > not changed. If the camera cannot flip, the transform is set back to\n>> > > > Identity and the state is adjusted.\n>> > >\n>> > > Yes, mostly, except that the sensor may be mounted with a rotation,\n>> > > but if it isn't, then I agree with this.\n>> > >\n>> > > >\n>> > > > A partially non-supported transform:\n>> > > > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n>> > > > only do HFLIP. transform sould be then set to HFLIP and applications\n>> > > > know that the remaining transformation should be handled elsewhere.\n>> > >\n>> > > As I suggested above, I'm not sure whether we want to mandate this. I\n>> > > agree it seems useful behaviour, which is why the Pi does it, but I'm\n>> > > open minded about requiring it or not.\n>> > >\n>> > > >\n>> > > > Are we missing anything with such behaviour ? I presume it's rather\n>> > > > similar to what happens today if not for the fact that the validate()\n>> > > > implementation adjusts transform to rotation in case it cannot flip\n>> > > >\n>> > > > Transform combined = transform * data_->rotationTransform_;\n>> > > > if (!data_->supportsFlips_ && !!combined)\n>> > > > transform = -data_->rotationTransform_;\n>> > > >\n>> > > > Not really sure I got why.\n>> > > >\n>> > > > All of this, but an even more fundamental question: is it too late/is\n>> > > > it worh it to adjust the Transform definition to adopt the same\n>> > > > direction as the kernel defined rotation ?\n>> > > >\n>> > > > Sorry for the long email :/\n>> > >\n>> > > No problem! I like nothing more than discussing transforms, except\n>> > > possibly colour spaces!\n>> > >\n>> >\n>> > :)\n>> >\n>> > > More seriously, here are my key points:\n>> >\n>> > I'll try to summarize what I see differently\n>> >\n>> > >\n>> > > 1. The CameraConfiguration::transform field says what the application\n>> > > wants the final output images to have.\n>> >\n>> > 1. The CameraConfiguration::transform field says what transform the\n>> > library has to apply to images as they arrive from the sensor in its\n>> > default configuration\n>> >\n>> > > 2. It takes account of the sensor rotation. The most obvious example\n>> > > is the Pi: if the sensor is mounted upside down, and the applications\n>> > > ask for images \"the right way up\" (i.e. transform = identity), then\n>> > > the PH will apply h and v flips to compensate for the sensor mounting.\n>> >\n>> > 2. It doesn't automatically account for sensor rotation. Application\n>> > might decide to do that along the rendering pipeline, in example\n>> >\n>> > > 3. If the PH can do what the application requests, it leaves the\n>> > > transform alone and your configuration is Valid.\n>> >\n>> > Agreed\n>> >\n>> > > 4. If the PH can't do what the application wants, it will set the\n>> > > value to something that it can do, and the configuration will be\n>> > > Adjusted.\n>> >\n>> > Agreed\n>> >\n>> > >\n>> > > If folks wanted a short conference call to iron any of this out, I'd\n>> > > be very happy to join in!\n>> >\n>> > I've asked others to have a look here, let's see if we need a call,\n>> > I'm all for it !\n>> >\n>> > Thanks\n>> > j\n>> >\n>> > >\n>> > > Thanks\n>> > > David\n>> > >\n>> > > >\n>> > > > > > Thanks everyone!\n>> > > > > > David\n>> > > > >\n>> > > > > Thanks!\n>> > > > >\n>> > > > > Robert\n>> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 561BEC3292\n\tfor ;\n\tThu, 12 Jan 2023 11:19:07 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id CAE28625CF;\n\tThu, 12 Jan 2023 12:19:06 +0100 (CET)","from mail-oa1-x2e.google.com (mail-oa1-x2e.google.com\n\t[IPv6:2001:4860:4864:20::2e])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 99DF161F05\n\tfor ;\n\tThu, 12 Jan 2023 12:19:04 +0100 (CET)","by mail-oa1-x2e.google.com with SMTP id\n\t586e51a60fabf-15ebfdf69adso1111706fac.0\n\tfor ;\n\tThu, 12 Jan 2023 03:19:04 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1673522346;\n\tbh=EOvY0peMYR0u9Ye1q9OxmEBVIN27uJTrthIWTYHQd3w=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=KHevoMsk0lRHiKztbRgwgCnPsJqXvs3BjygZrksP5/x3HO2vLJAG7OP3de1jCtdKz\n\tZO5MZDP1Aop4Y/Iq5KLlUtZYMYJMAjAvXZK2qqyWTSn6mEU5XXM7YgQq6Md2VwAtk8\n\toOvTA3DlhUwccT2vSUBjodTcVnmD1vhGnXIS7qhmMOJ9hRfiK2G/JchrkbZ/wAZHHr\n\tPxlzsu8RmJsaTUKgrbsf1lsZBh5LhuRuKpJhCoyZEHRZKvCVeR5f9rcWShpxslO1AZ\n\t215QdFI8eo22eYP7Fn/OQAUvjrWNwtL4kvPMhFdIN2prXpAGD8TXEZChg5KGjb6dUK\n\tK0Rot0BrxDKyg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=AyI5LqY6Q4C3BH1a4iTRThR6zqsKFaFfbdCrOnV6rjY=;\n\tb=XoEZ0vXdieYtwYY1a1hSs/3rulYVBimRfjlkGVNM+I/SN9g/iHMOWoHqSjHJkEB0SJ\n\th4s5TwsD0CZ6xVWSfuyLpuV3+ZdqanNBQyx8XRjvorYXgWnhgJSz+eqauNuUmw3BykH9\n\tPhI2zbqU8AbFwgbr+zxdNKD2Lbz6Xm0JqMf3MhxZ3Jd1SdBlHgQf4LpIMGXpmvhqIx98\n\tmCBzF9DqOOnEh9m4L3Tt60T/+sGacLephD9vn7axy9xvlDEI2r+DTzjzuRZUWGz16CbX\n\t6LSanLSLBfPBRr3L3riJue2b/vz6eHqzP1V9f4sxsA+T8RtN+PREXS6S6ACyg91XA6B+\n\t952Q=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"XoEZ0vXd\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=AyI5LqY6Q4C3BH1a4iTRThR6zqsKFaFfbdCrOnV6rjY=;\n\tb=pWmXU8aZHzDis0OInn9sHD+BaVzon+FvRjGxugDtLQMt3ZyreMmNwagwqH2BtprcLz\n\t/chpkqiaXnyKY+OMbIvS34SY0KpUYmByWHFkExdf8up8/zGf9jF2/obTLqqh9zX7tGq6\n\tjsdxaBUSMrmcgVPNL3inUc49tLXadcL/mqw6DDmCWF2HUgcPzOHs/ysCCZ8kXrJLMr9Y\n\ttR0EZ7qeEuohqwG70rEMRSKns092v8RMxAmXcWBKGkFBTNa5T/PxjZi/EsI+ki01fC02\n\tgTeeQp+h8FILFWZTQnW1NhNlnYtTLX7cSFroOqmLXRYkOuQgIMw6xrlk4uTMk0cyW9GE\n\tVhjQ==","X-Gm-Message-State":"AFqh2komJOVIL/9yh3TU0eW5OS+J3jyCLN35fV3co12qc9gxde/V63Vl\n\tnGZ2HzIqwtZ9qgEgKeWgrH53lQB9Tm30wKA5wGLJfgMhvY8HkQ==","X-Google-Smtp-Source":"AMrXdXswI/4VVJsSPkdcMUz2ELpmwUlJThazfnfNNHCMedF8TboiAIgm8IP/Mh6QvF2N3Dtowj7EoPKyhEWtCv4NI8o=","X-Received":"by 2002:a05:6870:e2c9:b0:150:a02f:1505 with SMTP id\n\tw9-20020a056870e2c900b00150a02f1505mr2538744oad.246.1673522342852;\n\tThu, 12 Jan 2023 03:19:02 -0800 (PST)","MIME-Version":"1.0","References":"\n\t<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t\n\t<20221213112251.jnpgyzitusutkqm6@uno.localdomain>\n\t\n\t\n\t<1b0cd37c-75a1-03b7-7df7-cecf0deb0cc7@posteo.de>","In-Reply-To":"<1b0cd37c-75a1-03b7-7df7-cecf0deb0cc7@posteo.de>","Date":"Thu, 12 Jan 2023 11:18:51 +0000","Message-ID":"","To":"Robert Mader , Jacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26254,"web_url":"https://patchwork.libcamera.org/comment/26254/","msgid":"<20230117120945.76xlszfljimhibzw@uno.localdomain>","date":"2023-01-17T12:09:45","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi David,\n cc Sakari for context about the imx258 sensor patches\n\nOn Thu, Jan 12, 2023 at 11:18:51AM +0000, David Plowman wrote:\n> Hi everyone\n>\n> I wanted to give this discussion a little nudge because I don't think\n> it was ever quite concluded, perhaps Christmas got in the way!\n>\n> To summarise a little bit, at least from my point of view, my\n> impression was that the current approach is correct, and the transform\n> field is for the user to request the transform that they want, and may\n> get adjusted to be the transform that they will get. Details of any\n> rotation in the way the sensor is mounted is hidden from the user -\n> they can just say \"I want the image the right way up\" and it\n> \"magically\" happens. (This is exactly how the Raspberry Pi pipeline\n> handler works today.)\n\nI have been able to discuss a bit with others, and I'm here trying to\nsummarize the outcome\n\n* Recap of the assumptions:\n\nFrom the kernel interface we have:\n- Physical mounting rotation (V4L2_CID_CAMERA_SENSOR_ROTATION)\n- Sensor flips capabilities (which depends on the V4L2_CID_[H|F]LIP\n presence and if they are RW)\n\nIt is fair to assume that:\n- Sensors cannot do transpose (flip along the diagonal axis)\n- They can only HVFLIP (ie flip, mirror or 180 degrees rotate)\n\nThe ISP can also do transforms like transpose or even rotate, but\ncompared to acting flips on the sensor this transformation does not\nchange the row/col reading order in the image (the first pixels\nvisible in the will end up being the ones read last) which might have\nan impact for rolling shutter sensors.\n\n* Proposed implementation\n\nlibcamera retrieves the mounting rotation, and if it can fully\ncompensate it by setting sensor's flips (or if flips are already hardcoded to\n1 as it happens in some drivers) it will adjust properties::Rotation\nto report that the image will be rotated upright and application\nshouldn't care. This is what happens today with your implementation of\nthe Transform handling part.\n\nThis also implies that sensor's can correct only 180 degrees rotation\nand everything else will be deferred to\n1) other processing layers\n2) additional transforms in the ISP if the pipeline supports those\n\nIf you're ok with such plan I propose:\n\n- let's land my series of 5 patches that centralize flip handling in\n CameraSensor but does not introduce functional changes\n- let's on top make the CameraSensor class register property::Rotation\n compensated by the sensor (basically only for 180deg correction)\n- let's adjust CameraSensor::setFormat() to apply flips correctly and\n remove transpose handling from there but only handle 180 degrees\n transformations.\n\nFor RPi it means 180deg rotation will be automatically compensated as\nthey're today by sensor's flips. The only difference is that your\nconsumers will see a properties::Rotation = 0 instead of 180, as for\nthem the image is already 'correct'.\n\nFor other devices with different rotations which require transpose\n(as 90/270 deg, typical for smartphones) the whole rotation adjustment\nwill be performed by the application's upper layers.\n\nDoes this sound viable to you ?\n\n>\n> There was a small detail about whether one of the transforms in the\n> code wanted reversing, that's mostly a matter of conventions about\n> using clockwise or anticlockwise rotations (or the direction of the\n> rotation axis).\n>\n> Does anyone have anything else to add here?\n>\n> Thanks!\n> David\n>\n> On Wed, 14 Dec 2022 at 15:55, Robert Mader via libcamera-devel\n> wrote:\n> >\n> > Hi,\n> >\n> > just wanted to drop that\n> >\n> > I'm personally not very opinionated about the API design\n> > the new Pipewire implementation matches the current Raspberry Pi code\n> > I think/agree this patch series is the ideal place to change/stabilize this API\n> > I'll happily adopt the Pipewire implementation as soon as a consensus is found / the implementation lands :)\n> >\n> > Best regards and thanks to everyone joining the discussion!\n> >\n> > Robert\n> >\n> > On 13.12.22 15:59, Naushir Patuck via libcamera-devel wrote:\n> >\n> > Hi all,\n> >\n> >\n> > On Tue, 13 Dec 2022 at 14:30, David Plowman via libcamera-devel wrote:\n> >>\n> >> Hi Jacopo\n> >>\n> >> Thanks for the reply. I won't write a great deal this time round, I\n> >> agree that possibly a call would be a good idea.\n> >>\n> >> On Tue, 13 Dec 2022 at 11:22, Jacopo Mondi wrote:\n> >> >\n> >> > Hi David\n> >> >\n> >> > On Mon, Dec 12, 2022 at 11:46:31AM +0000, David Plowman wrote:\n> >> > > Hi Jacopo\n> >> > >\n> >> > > Thanks for wading into the sea on this question! (Very appropriate now\n> >> > > that the stick person example has been turned into a shark...!)\n> >> > >\n> >> > > On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n> >> > > wrote:\n> >> > > >\n> >> > > > Hi again,\n> >> > > >\n> >> > > > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n> >> > > > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> >> > > > > > Hi Robert\n> >> > > > > >\n> >> > > > > > Thanks for prodding us all on this question!\n> >> > > > >\n> >> > > > > Hi, thanks for the quick reply! And a pleasure :P\n> >> > > > >\n> >> > > > > > I looked back over this again, and found Jacopo's description of the\n> >> > > > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> >> > > > > > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> >> > > > > > is what I stumbled into)\n> >> > > > >\n> >> > > > > The current kernel docu can be found at\n> >> > > > >\n> >> > > > > https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n> >> > >\n> >> > > Thanks for the updated reference. The stick person may have turned\n> >> > > into a shark, but apart from that I think the sense of the\n> >> > > documentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n> >> > > 90 degrees, then your image in memory looks like it has had a 90\n> >> > > degree *clockwise* rotation applied.\n> >> > >\n> >> >\n> >> > Yes!\n> >> >\n> >> > > > >\n> >> > > > > the libcamera one at\n> >> > > > >\n> >> > > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> >> > > > >\n> >> > > > > > From that, and the nice example with the stick person, I deduce that\n> >> > > > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> >> > > > > > degrees, then the image you capture (if nothing else happens to it)\n> >> > > > > > will look like it has had a 90 degree clockwise rotation performed.\n> >> > > > > > Everyone agree with that?\n> >> > > > > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> >> > > > > translate to Transform::Rot90.\n> >> > > >\n> >> > > > Let's here put some definitions in place\n> >> > > >\n> >> > > > V4L2_CID_CAMERA_SENSOR_ROTATION\n> >> > > > This read-only control describes the rotation correction in degrees in\n> >> > > > the counter-clockwise direction to be applied to the captured images\n> >> > > > once captured to memory to compensate for the camera sensor mounting\n> >> > > > rotation.\n> >> > > >\n> >> > > > \\var Transform::Rot270\n> >> > > > Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n> >> > > >\n> >> > > > Let's start by saying we did a sub-optimal (to be nice) job in\n> >> > > > having Transform and V4L2_CID_ROTATION operate in two different\n> >> > > > directions. Before the usage of Transform in libcamera gets widely\n> >> > > > adopted sould we align the two ?\n> >> > >\n> >> > > I don't mind at all... we just have to decide what we want!\n> >> > >\n> >> > > >\n> >> > > > Hence if I'm not mistaken, if your camera has Rotation=90 and an\n> >> > > > application supplies Transform::Identity in\n> >> > > > CameraConfiguration::transform what you should get back is Rot270\n> >> > > >\n> >> > > > Correct ?\n> >> > >\n> >> > > The way things are currently, if the camera says Rotation=90, then\n> >> > > your image in memory looks like it is rotated 90 degrees clockwise, so\n> >> > > the Transform (if your pipeline and sensor can't apply any\n> >> > > flips/rotations at all) will come back as Rot90.\n> >> > >\n> >> > > That is, the only transform you can specify that won't be adjusted, is\n> >> > > a 90 degree clockwise rotation, so transform=Rot90 as things stand\n> >> > > currently.\n> >> > >\n> >> > > If we choose to change the sense of the libcamera.Transform rotation,\n> >> > > then we'd get transform=Rot270 instead.\n> >> > >\n> >> > > (If your pipeline handler can do flips and transposes, it may be able\n> >> > > to do what you ask and so the transform would come back unchanged and\n> >> > > would be \"Valid\".)\n> >> > >\n> >> > > >\n> >> > > > > > So in turn, if you have a camera/ISP that can apply no transforms at\n> >> > > > > > all, then the only value for the user transform that will not come\n> >> > > > > > back as adjusted is \"rot90\", because it looks like it's had a 90\n> >> > > > > > degree clockwise rotation performed (and libcamera transforms count\n> >> > > > > > rotations as being clockwise). Does that also make sense?\n> >> > > >\n> >> > > > It might be the case today, but I wonder if that's desirable.\n> >> > > >\n> >> > > > If an application supplies CameraConfiguration::transform = Identity\n> >> > > > it means it doesn't want any rotation applied by the library. It will\n> >> > > > get back 270 to indicate that that's how the image is rotated and will\n> >> > > > get an Adjusted configuration.\n> >> > >\n> >> > > I'm not quite sure I get that, perhaps it depends how we interpret\n> >> > > \"applied by the library\". I guess I agree if we mean \"applied by the\n> >> > > library together with any rotation of the sensor\"? Maybe a real\n> >> >\n> >> > \"Applied by the library\" == happens inside libcamera, ISP or sensor\n> >> > flips..\n> >> >\n> >> > > example is helpful:\n> >> > >\n> >> > > So for a Raspberry Pi, for example, our sensors are mostly mounted\n> >> > > upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\n> >> > > say the transform they want is transform=Identity i.e. images come\n> >> > > back \"the right way up\". Internally it means that we do apply an extra\n> >> > > h+v flip to undo the 180 degree rotation of the sensor.\n> >> > >\n> >> >\n> >> > Here you go, this is where we disconnect.\n> >> >\n> >> > As I've interpreted the API so far, and the way we designed\n> >> > properties::Rotation was to signal to the application what is the\n> >> > status of the image as it comes from the sensor. Application could then\n> >> > correct that by setting CameraConfiguration::transform, but in most\n> >> > cases the image will be corrected at rendering time by the application\n> >> > itself.\n> >> >\n> >> > IWO I was not expecting auto-correction and Transform::Identity to me\n> >> > means \"give me the images as they come from the sensor\"\n> >> >\n> >> > > >\n> >> > > > If instead an application supplies CameraConfiguration::transform = Rot270\n> >> > > > and your camera -cannot- do any rotation, it will still receive back\n> >> > > > Rot270 and the state is Valid.\n> >> > >\n> >> > > I think that's true if we change the sense of the libcamera.Transform\n> >> > > rotation. Currently, as I said earlier, I think we get Rot90?\n> >> > >\n> >> > > >\n> >> > > > If it asks for CameraConfiguration::transform = Rot270 and the camera\n> >> > > > -can- do that, it will clear CameraConfiguration::transform to\n> >> > > > Identity and return Adjusted ?\n> >> > >\n> >> > > I'm not sure there... The CameraConfiguration::transform is the final\n> >> > > transform you want. If that can be done, it's left alone and you get\n> >> > > back Valid. It's not telling you what else you would need to do\n> >> > > afterwards.\n> >> > >\n> >> > > (To find out what you need to do afterwards you'd work out\n> >> > > \"inverse(what-you-will-get) * what-you-wanted\", or something like\n> >> > > that.)\n> >> > >\n> >> >\n> >> > What I would like is to make as easier as possible for applications\n> >> > to know what they have to do to have the image rotated in the way they\n> >> > expect. There are several \"transform\" steps on the image, the first\n> >> > one happens in libcamera, all the other ones are left to the\n> >> > appplication.\n> >>\n> >> I think this is exactly what I want, namely \"to make it as easy as\n> >> possible for applications\".\n> >>\n> >> On Raspberry Pi - and this has been the case for a couple of years\n> >> now! - if a user wants images the \"normal\" way up then they simply\n> >> pass Transform::Identity. If they want upside down images they pass\n> >> Transform::Rot180. It's as easy as that, there is simply nothing else\n> >> to do.\n> >>\n> >> Requiring all applications to look at the sensor rotation and then\n> >> combine various things to figure out what they have to ask for (and\n> >> possibly get it wrong) seems, well, actually quite unhelpful.\n> >\n> >\n> > FWIW, I do agree that applications should not have to trouble themselves\n> > with munging sensor rotation andrequested rotation to achieve their\n> > desired outcome. This definitely wants to be hidden in the libcamera core\n> > or pipeline handler layers.\n> >\n> > Naush\n> >\n> >\n> >>\n> >>\n> >> But this is indeed what we need to talk about. Let's organise a call!\n> >>\n> >> Thanks\n> >> David\n> >>\n> >> >\n> >> > *) The image \"native\" transform is reported by Rotation (this assumes\n> >> > that any implicit rotation buried in the sensor driver, as per our\n> >> > last discussion with Dave is not taken into account)\n> >> >\n> >> > properties::Rotation = Rot270 = HFlip | Transpose\n> >> >\n> >> > *) Application sets CameraConfiguration::transform and call validate() to\n> >> > check what can be done by libcamera and what should be done by next\n> >> > layers\n> >> >\n> >> > CameraConfigurat::Transform = Rot270\n> >> >\n> >> > *) The camera validate the transform and as it can only flip it sets\n> >> > transform to HFlip, and the state is Adjusted\n> >> >\n> >> > validate() -> The camera can only flip\n> >> > -> CameraConfiguration = HFlip\n> >> >\n> >> > *) Application gets back what the camera can do and instrument the\n> >> > rest of the rendering pipeline to perform the additional transforms\n> >> > the camera cannot do. To get what you have to do I presume you have to\n> >> >\n> >> > additionalTransform = what_you_want & !what_you_get\n> >> >\n> >> > So yes, I was interpreting the usage of CameraConfiguration::transform\n> >> > as a negotiation of what the library can do and what applications have\n> >> > to do by themselves, which if I got you is different than what's\n> >> > currecntly implemented ?\n> >> >\n> >> > > >\n> >> > > > Is this what happens (I'm looking at the above code with David's\n> >> > > > suggestion to use the inverse of roationTransform).\n> >> > > >\n> >> > > >\n> >> > > > > >\n> >> > > > > > So by a slightly arbitrary mixture of conventions, it looks to me as\n> >> > > > > > though the use of the inverse that I queried is indeed wrong, because\n> >> > > > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> >> > > > > > case.\n> >> > > > > >\n> >> > > > > > Does that sound plausible... what do folks think?\n> >> > > > >\n> >> > > > > Ah, I think the question is whether the adjusted value means: \"this is what\n> >> > > > > you'll get\" vs \"this is what you have to do yourself\".\n> >> > > > >\n> >> > > > > Above I argued in favor of the later, but I guess the first one is what the\n> >> > > > > API is supposed to mean. That would imply that the client has compute the\n> >> > > > > difference between requested and adjusted value itself.\n> >> > > >\n> >> > > > I always presumed the latter was meant to happen..\n> >> > >\n> >> > > Yes, I think this is the crux of the matter. I believe it's \"what you\n> >> > > will get\". It's not \"what you need to do after\". (If it was \"what you\n> >> > > need to do after\", it would change every time you try to validate the\n> >> > > configuration!)\n> >> > >\n> >> >\n> >> > On the \"it would change every time you try to validate the\n> >> > configuration\": only if the camera cannot satisfy the transform you\n> >> > have requested. If you ask for a flip and the camera can do it,\n> >> > transform stays un-touched and the state is Valid ?\n> >> >\n> >> > > >\n> >> > > > >\n> >> > > > > In my case that would mean:\n> >> > > > >\n> >> > > > > requested CameraConfiguration::transform: Transform::Identity,\n> >> > > > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> >> > > > > CameraConfiguration::transform: Transform::Rot270\n> >> > > > >\n> >> > > >\n> >> > > > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n> >> > >\n> >> > > Yes, looks right to me!\n> >> > >\n> >> > > >\n> >> > > > > To compute what the client has to do, it would need to do:\n> >> > > > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> >> > > > > that, a 90 degr. rotation clockwise.\n> >> > >\n> >> > > Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> >> > > what-you-wanted\", in this case inverse(rot270) * identity which is\n> >> > > rot90 (in the current clockwise sense).\n> >> > >\n> >> > > > >\n> >> > > > > Does that sound reasonable? In that case the patch here should indeed be\n> >> > > > > `*transform = rotationTransform;`, without the inverse.\n> >> > > > >\n> >> > >\n> >> > > Yes, agree!\n> >> > >\n> >> > > >\n> >> > > > /0\\\n> >> > > >\n> >> > > > before we plumb this into and make assumptions that will be then\n> >> > > > set into stone in our adaption layers, can we take a step back and\n> >> > > > define what kind of API we would like to have ?\n> >> > > >\n> >> > > > Let's start by the definition of CameraConfiguration::transform.\n> >> > > >\n> >> > > > It's a member of CameraConfiguration, and as by definition it is\n> >> > > > filled-in by the application and eventually adjusted by the Camera to\n> >> > > > a value that's acceptable/doable for the current configuration.\n> >> > > >\n> >> > > > 1) Application to Camera = (counter?)clockwise degrees the camera\n> >> > > > system should rotate the image before presenting it to the user\n> >> > >\n> >> > > This may depend on what exactly we mean by \"the camera system\". I'd\n> >> > > explain it as\n> >> > > \"the final resulting transform that the application wants to be\n> >> > > applied to all the output images\". After all, this is actually the\n> >> > > only thing the application knows!\n> >> > >\n> >> >\n> >> > agreed\n> >> >\n> >> > > >\n> >> > > > 2) Camera to application = the rotation the camera can actually do\n> >> > > > (ie. only sensor flips == no Transpose)\n> >> > >\n> >> > > Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n> >> >\n> >> > yes, more correct\n> >> >\n> >> > > because flips aren't rotations, you could in theory have a PH that\n> >> > > could do transposes (even though we probably don't currently).\n> >> > >\n> >> >\n> >> > none currently afaict\n> >> >\n> >> > > >\n> >> > > > The current documentation doesn't explain how it gets adjusted\n> >> > >\n> >> > > This is true, though I also think it's platform dependent. If you had\n> >> > > a PH that can't do transposes but can do flips, you have a choice how\n> >> > > you handle (for example) rot90. You could say \"oh I can't do that, so\n> >> > > in this case I won't do anything at all\" and leave the application to\n> >> > > sort everything out. The Pi tries to say \"well, I'll do everything\n> >> > > except for the transpose, so an application only has to worry about\n> >> > > that one case\". But there is a choice...\n> >> > >\n> >> >\n> >> > I agree it's platform-specific, but the way transform is negotiated with\n> >> > application should be standardized\n> >> >\n> >> > > >\n> >> > > > /**\n> >> > > > * \\var CameraConfiguration::transform\n> >> > > > * \\brief User-specified transform to be applied to the image\n> >> > > > *\n> >> > > > * The transform is a user-specified 2D plane transform that will be applied\n> >> > > > * to the camera images by the processing pipeline before being handed to\n> >> > > > * the application. This is subsequent to any transform that is already\n> >> > > > * required to fix up any platform-defined rotation.\n> >> > > > *\n> >> > > > * The usual 2D plane transforms are allowed here (horizontal/vertical\n> >> > > > * flips, multiple of 90-degree rotations etc.), but the validate() function\n> >> > > > * may adjust this field at its discretion if the selection is not supported.\n> >> > > > */\n> >> > > >\n> >> > > > The last part in particular, \"at its discrection\" it's really too\n> >> > > > generic.\n> >> > >\n> >> > > I guess it's hinting at the fact that it's up to the PH, and we don't\n> >> > > mandate the exact way in which it can fail to do what you asked for!\n> >> > > But it would be fine to have a discussion on that.\n> >> > >\n> >> >\n> >> > That's the point that most concerns me :)\n> >> >\n> >> > > >\n> >> > > > Let's start from the beginning: what application should use transform\n> >> > > > for ? I presume they should:\n> >> > > >\n> >> > > > 1) Inspect propertis::Rotation\n> >> > > > 2) Correct the image for the properties::Rotation amount (actually,\n> >> > > > the inverse because of the clockwise/counter-clockwise thing)\n> >> > > > 3) Add any additional rotation they would like on top of this\n> >> > >\n> >> > > So I think the point of the transform is that you *don't* have to\n> >> > > worry about the properties::Rotation. It takes care of that for you.\n> >> > > You say \"I want upside down images\" and you get them, whatever the\n> >> > > sensor's rotation.\n> >> >\n> >> > That's acceptable as well if we prefer that, but it has to be\n> >> > standardized among pipelines imho\n> >> >\n> >> > >\n> >> > > >\n> >> > > > How should transform be adjusted ?\n> >> > > >\n> >> > > > Identity:\n> >> > > > If application->identity the camera does not have to apply any\n> >> > > > transform. Should the transform be adjutes like it happens today to\n> >> > > > compensate the Rotation (at least that's what I presume happens) ? I\n> >> > > > presume no, applications can infer rotation from properties and if they\n> >> > > > chose Identity either other layers above will do the transform or\n> >> > > > they're fine with images as they are.\n> >> > >\n> >> > > If the application says it wants the \"identity\" transform, then it\n> >> > > should get output images with no transformation applied. But this does\n> >> > > mean it will undo the sensor rotation (this being the normal case on\n> >> > > the Pi).\n> >> > >\n> >> >\n> >> > right, it depends if we decide to account for the sensor rotation\n> >> > inside libcamera or not\n> >> >\n> >> > > >\n> >> > > > A supported/non-supported transform:\n> >> > > > An application asks for a VFLIP, the camera can do it, transform is\n> >> > > > not changed. If the camera cannot flip, the transform is set back to\n> >> > > > Identity and the state is adjusted.\n> >> > >\n> >> > > Yes, mostly, except that the sensor may be mounted with a rotation,\n> >> > > but if it isn't, then I agree with this.\n> >> > >\n> >> > > >\n> >> > > > A partially non-supported transform:\n> >> > > > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> >> > > > only do HFLIP. transform sould be then set to HFLIP and applications\n> >> > > > know that the remaining transformation should be handled elsewhere.\n> >> > >\n> >> > > As I suggested above, I'm not sure whether we want to mandate this. I\n> >> > > agree it seems useful behaviour, which is why the Pi does it, but I'm\n> >> > > open minded about requiring it or not.\n> >> > >\n> >> > > >\n> >> > > > Are we missing anything with such behaviour ? I presume it's rather\n> >> > > > similar to what happens today if not for the fact that the validate()\n> >> > > > implementation adjusts transform to rotation in case it cannot flip\n> >> > > >\n> >> > > > Transform combined = transform * data_->rotationTransform_;\n> >> > > > if (!data_->supportsFlips_ && !!combined)\n> >> > > > transform = -data_->rotationTransform_;\n> >> > > >\n> >> > > > Not really sure I got why.\n> >> > > >\n> >> > > > All of this, but an even more fundamental question: is it too late/is\n> >> > > > it worh it to adjust the Transform definition to adopt the same\n> >> > > > direction as the kernel defined rotation ?\n> >> > > >\n> >> > > > Sorry for the long email :/\n> >> > >\n> >> > > No problem! I like nothing more than discussing transforms, except\n> >> > > possibly colour spaces!\n> >> > >\n> >> >\n> >> > :)\n> >> >\n> >> > > More seriously, here are my key points:\n> >> >\n> >> > I'll try to summarize what I see differently\n> >> >\n> >> > >\n> >> > > 1. The CameraConfiguration::transform field says what the application\n> >> > > wants the final output images to have.\n> >> >\n> >> > 1. The CameraConfiguration::transform field says what transform the\n> >> > library has to apply to images as they arrive from the sensor in its\n> >> > default configuration\n> >> >\n> >> > > 2. It takes account of the sensor rotation. The most obvious example\n> >> > > is the Pi: if the sensor is mounted upside down, and the applications\n> >> > > ask for images \"the right way up\" (i.e. transform = identity), then\n> >> > > the PH will apply h and v flips to compensate for the sensor mounting.\n> >> >\n> >> > 2. It doesn't automatically account for sensor rotation. Application\n> >> > might decide to do that along the rendering pipeline, in example\n> >> >\n> >> > > 3. If the PH can do what the application requests, it leaves the\n> >> > > transform alone and your configuration is Valid.\n> >> >\n> >> > Agreed\n> >> >\n> >> > > 4. If the PH can't do what the application wants, it will set the\n> >> > > value to something that it can do, and the configuration will be\n> >> > > Adjusted.\n> >> >\n> >> > Agreed\n> >> >\n> >> > >\n> >> > > If folks wanted a short conference call to iron any of this out, I'd\n> >> > > be very happy to join in!\n> >> >\n> >> > I've asked others to have a look here, let's see if we need a call,\n> >> > I'm all for it !\n> >> >\n> >> > Thanks\n> >> > j\n> >> >\n> >> > >\n> >> > > Thanks\n> >> > > David\n> >> > >\n> >> > > >\n> >> > > > > > Thanks everyone!\n> >> > > > > > David\n> >> > > > >\n> >> > > > > Thanks!\n> >> > > > >\n> >> > > > > Robert\n> >> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 77964C3240\n\tfor ;\n\tTue, 17 Jan 2023 12:09:51 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C3A31625E4;\n\tTue, 17 Jan 2023 13:09:50 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A9E8E625CE\n\tfor ;\n\tTue, 17 Jan 2023 13:09:49 +0100 (CET)","from ideasonboard.com (93-61-96-190.ip145.fastwebnet.it\n\t[93.61.96.190])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 085A4471;\n\tTue, 17 Jan 2023 13:09:48 +0100 (CET)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1673957390;\n\tbh=/mXsOD1lcY8KuU+a/ntl+eUqtatL9jrswdaJVXb/VUk=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=ACQSlfT/aOqCzWYOtP8ThRzmD/nqSEna2NSguGkXjmwBQaggPMxDmbXmnIBGeTGVM\n\tZy37GbcNA1ZyoGKH7NAIApJz19Ild3We3/a22M945z2Nqi5g95x++8y3YsvP+cKhud\n\ttbxRti/AGkulf2D6oTwYuIUpFnfvLbFxiMOL/aYWTXq1uK2rkdCPllOX/CdD2prEoE\n\t5m/m3xpkHC4eR/9PQZvV2uVt8pvqc74XcIry88S7X3Eiwfr9MLhT/hSKiJJboZHVIj\n\tRfFK7hRyMkYScOCMypQ/8uX/0gU+rSZf5XJqlig2ZHrE5FBh72K4o6IZQtYozxZ5Rh\n\tn/uR2vD5U7bAg==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1673957389;\n\tbh=/mXsOD1lcY8KuU+a/ntl+eUqtatL9jrswdaJVXb/VUk=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=hMOJFo5+06NGb7jdCPSb+JD4xIq6Q8flaqlcnPRdw/XZq7DMNePGOG6Y9ZKgFWnfn\n\tRHu4i8UI07pJ91bSS+TPq/02wUtVhC2KRpU2lyzIbKNDwjoK/GTSdUbnV8vD0JFpuj\n\tUO8he3Qdqnjy1q8zstaauJuzKw7qNZmMef64ed2c="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"hMOJFo5+\"; dkim-atps=neutral","Date":"Tue, 17 Jan 2023 13:09:45 +0100","To":"David Plowman ","Message-ID":"<20230117120945.76xlszfljimhibzw@uno.localdomain>","References":"<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t\n\t<20221213112251.jnpgyzitusutkqm6@uno.localdomain>\n\t\n\t\n\t<1b0cd37c-75a1-03b7-7df7-cecf0deb0cc7@posteo.de>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jacopo Mondi via libcamera-devel ","Reply-To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org,\n\tSakari Ailus ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":26259,"web_url":"https://patchwork.libcamera.org/comment/26259/","msgid":"","date":"2023-01-18T17:22:14","subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Jacopo\n\nThanks for your reply!\n\nOn Tue, 17 Jan 2023 at 12:09, Jacopo Mondi\n wrote:\n>\n> Hi David,\n> cc Sakari for context about the imx258 sensor patches\n>\n> On Thu, Jan 12, 2023 at 11:18:51AM +0000, David Plowman wrote:\n> > Hi everyone\n> >\n> > I wanted to give this discussion a little nudge because I don't think\n> > it was ever quite concluded, perhaps Christmas got in the way!\n> >\n> > To summarise a little bit, at least from my point of view, my\n> > impression was that the current approach is correct, and the transform\n> > field is for the user to request the transform that they want, and may\n> > get adjusted to be the transform that they will get. Details of any\n> > rotation in the way the sensor is mounted is hidden from the user -\n> > they can just say \"I want the image the right way up\" and it\n> > \"magically\" happens. (This is exactly how the Raspberry Pi pipeline\n> > handler works today.)\n>\n> I have been able to discuss a bit with others, and I'm here trying to\n> summarize the outcome\n>\n> * Recap of the assumptions:\n>\n> From the kernel interface we have:\n> - Physical mounting rotation (V4L2_CID_CAMERA_SENSOR_ROTATION)\n> - Sensor flips capabilities (which depends on the V4L2_CID_[H|F]LIP\n> presence and if they are RW)\n>\n> It is fair to assume that:\n> - Sensors cannot do transpose (flip along the diagonal axis)\n> - They can only HVFLIP (ie flip, mirror or 180 degrees rotate)\n>\n> The ISP can also do transforms like transpose or even rotate, but\n> compared to acting flips on the sensor this transformation does not\n> change the row/col reading order in the image (the first pixels\n> visible in the will end up being the ones read last) which might have\n> an impact for rolling shutter sensors.\n>\n> * Proposed implementation\n>\n> libcamera retrieves the mounting rotation, and if it can fully\n> compensate it by setting sensor's flips (or if flips are already hardcoded to\n> 1 as it happens in some drivers) it will adjust properties::Rotation\n> to report that the image will be rotated upright and application\n> shouldn't care. This is what happens today with your implementation of\n> the Transform handling part.\n>\n> This also implies that sensor's can correct only 180 degrees rotation\n> and everything else will be deferred to\n> 1) other processing layers\n> 2) additional transforms in the ISP if the pipeline supports those\n>\n> If you're ok with such plan I propose:\n>\n> - let's land my series of 5 patches that centralize flip handling in\n> CameraSensor but does not introduce functional changes\n> - let's on top make the CameraSensor class register property::Rotation\n> compensated by the sensor (basically only for 180deg correction)\n> - let's adjust CameraSensor::setFormat() to apply flips correctly and\n> remove transpose handling from there but only handle 180 degrees\n> transformations.\n>\n> For RPi it means 180deg rotation will be automatically compensated as\n> they're today by sensor's flips. The only difference is that your\n> consumers will see a properties::Rotation = 0 instead of 180, as for\n> them the image is already 'correct'.\n>\n> For other devices with different rotations which require transpose\n> (as 90/270 deg, typical for smartphones) the whole rotation adjustment\n> will be performed by the application's upper layers.\n>\n> Does this sound viable to you ?\n\nMostly that's sounding good to me. The only thing I have a couple of\nquestions about is what we do if we can't compensate for the camera\nrotation.\n\n* It feels a bit strange to clear the advertised camera mounting\nrotation if we can handle it, but not otherwise. I think we should\nalways advertise what it really is (even if that's in a separate\nproperty). Perhaps there's another way to indicate what transforms a\nPH can handle (a Transform is just a bit field, after all...)?\n\n* I think the code currently tries to compensate for everything except\ntranspose, so that would be the only case you have to implement\noutside libcamera. But obviously there's a choice there. Might an\napplication prefer you to do \"nothing\" instead of \"everything bar the\ntranspose\"? Obviously you could work out what you need to ask for to\nget the \"left over\" transform that you want...\n\nFinally, on the subject of rotation orientations, I think it would be\nnice if you could request transformFromRotation(sensor_rotation) in\nyour configuration, and that would guarantee that no transforms would\nbe applied by libcamera anywhere. That feels \"natural\" to me.\n\nBut otherwise, yes, let's push this forward.\n\nThanks!\nDavid\n\n>\n> >\n> > There was a small detail about whether one of the transforms in the\n> > code wanted reversing, that's mostly a matter of conventions about\n> > using clockwise or anticlockwise rotations (or the direction of the\n> > rotation axis).\n> >\n> > Does anyone have anything else to add here?\n> >\n> > Thanks!\n> > David\n> >\n> > On Wed, 14 Dec 2022 at 15:55, Robert Mader via libcamera-devel\n> > wrote:\n> > >\n> > > Hi,\n> > >\n> > > just wanted to drop that\n> > >\n> > > I'm personally not very opinionated about the API design\n> > > the new Pipewire implementation matches the current Raspberry Pi code\n> > > I think/agree this patch series is the ideal place to change/stabilize this API\n> > > I'll happily adopt the Pipewire implementation as soon as a consensus is found / the implementation lands :)\n> > >\n> > > Best regards and thanks to everyone joining the discussion!\n> > >\n> > > Robert\n> > >\n> > > On 13.12.22 15:59, Naushir Patuck via libcamera-devel wrote:\n> > >\n> > > Hi all,\n> > >\n> > >\n> > > On Tue, 13 Dec 2022 at 14:30, David Plowman via libcamera-devel wrote:\n> > >>\n> > >> Hi Jacopo\n> > >>\n> > >> Thanks for the reply. I won't write a great deal this time round, I\n> > >> agree that possibly a call would be a good idea.\n> > >>\n> > >> On Tue, 13 Dec 2022 at 11:22, Jacopo Mondi wrote:\n> > >> >\n> > >> > Hi David\n> > >> >\n> > >> > On Mon, Dec 12, 2022 at 11:46:31AM +0000, David Plowman wrote:\n> > >> > > Hi Jacopo\n> > >> > >\n> > >> > > Thanks for wading into the sea on this question! (Very appropriate now\n> > >> > > that the stick person example has been turned into a shark...!)\n> > >> > >\n> > >> > > On Fri, 9 Dec 2022 at 12:47, Jacopo Mondi via libcamera-devel\n> > >> > > wrote:\n> > >> > > >\n> > >> > > > Hi again,\n> > >> > > >\n> > >> > > > On Wed, Dec 07, 2022 at 03:48:40PM +0000, Robert Mader via libcamera-devel wrote:\n> > >> > > > > On 07.12.22 15:28, David Plowman via libcamera-devel wrote:\n> > >> > > > > > Hi Robert\n> > >> > > > > >\n> > >> > > > > > Thanks for prodding us all on this question!\n> > >> > > > >\n> > >> > > > > Hi, thanks for the quick reply! And a pleasure :P\n> > >> > > > >\n> > >> > > > > > I looked back over this again, and found Jacopo's description of the\n> > >> > > > > > V4L2_CID_CAMERA_SENSOR_ROTATION control\n> > >> > > > > > (https://patchwork.kernel.org/project/linux-media/patch/20191204091056.4842-5-jacopo@jmondi.org/\n> > >> > > > > > is what I stumbled into)\n> > >> > > > >\n> > >> > > > > The current kernel docu can be found at\n> > >> > > > >\n> > >> > > > > https://github.com/torvalds/linux/blob/8ed710da2873c2aeb3bb805864a699affaf1d03b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst\n> > >> > >\n> > >> > > Thanks for the updated reference. The stick person may have turned\n> > >> > > into a shark, but apart from that I think the sense of the\n> > >> > > documentation is unchanged. So if V4L2_CID_CAMERA_SENSOR_ROTATION says\n> > >> > > 90 degrees, then your image in memory looks like it has had a 90\n> > >> > > degree *clockwise* rotation applied.\n> > >> > >\n> > >> >\n> > >> > Yes!\n> > >> >\n> > >> > > > >\n> > >> > > > > the libcamera one at\n> > >> > > > >\n> > >> > > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/transform.cpp#n54\n> > >> > > > >\n> > >> > > > > > From that, and the nice example with the stick person, I deduce that\n> > >> > > > > > if your camera's V4L2_CID_CAMERA_SENSOR_ROTATION property says 90\n> > >> > > > > > degrees, then the image you capture (if nothing else happens to it)\n> > >> > > > > > will look like it has had a 90 degree clockwise rotation performed.\n> > >> > > > > > Everyone agree with that?\n> > >> > > > > Yep, agreed. That would mean V4L2_CID_CAMERA_SENSOR_ROTATION==90 would\n> > >> > > > > translate to Transform::Rot90.\n> > >> > > >\n> > >> > > > Let's here put some definitions in place\n> > >> > > >\n> > >> > > > V4L2_CID_CAMERA_SENSOR_ROTATION\n> > >> > > > This read-only control describes the rotation correction in degrees in\n> > >> > > > the counter-clockwise direction to be applied to the captured images\n> > >> > > > once captured to memory to compensate for the camera sensor mounting\n> > >> > > > rotation.\n> > >> > > >\n> > >> > > > \\var Transform::Rot270\n> > >> > > > Rotation by 270 degrees clockwise (90 degrees anticlockwise).\n> > >> > > >\n> > >> > > > Let's start by saying we did a sub-optimal (to be nice) job in\n> > >> > > > having Transform and V4L2_CID_ROTATION operate in two different\n> > >> > > > directions. Before the usage of Transform in libcamera gets widely\n> > >> > > > adopted sould we align the two ?\n> > >> > >\n> > >> > > I don't mind at all... we just have to decide what we want!\n> > >> > >\n> > >> > > >\n> > >> > > > Hence if I'm not mistaken, if your camera has Rotation=90 and an\n> > >> > > > application supplies Transform::Identity in\n> > >> > > > CameraConfiguration::transform what you should get back is Rot270\n> > >> > > >\n> > >> > > > Correct ?\n> > >> > >\n> > >> > > The way things are currently, if the camera says Rotation=90, then\n> > >> > > your image in memory looks like it is rotated 90 degrees clockwise, so\n> > >> > > the Transform (if your pipeline and sensor can't apply any\n> > >> > > flips/rotations at all) will come back as Rot90.\n> > >> > >\n> > >> > > That is, the only transform you can specify that won't be adjusted, is\n> > >> > > a 90 degree clockwise rotation, so transform=Rot90 as things stand\n> > >> > > currently.\n> > >> > >\n> > >> > > If we choose to change the sense of the libcamera.Transform rotation,\n> > >> > > then we'd get transform=Rot270 instead.\n> > >> > >\n> > >> > > (If your pipeline handler can do flips and transposes, it may be able\n> > >> > > to do what you ask and so the transform would come back unchanged and\n> > >> > > would be \"Valid\".)\n> > >> > >\n> > >> > > >\n> > >> > > > > > So in turn, if you have a camera/ISP that can apply no transforms at\n> > >> > > > > > all, then the only value for the user transform that will not come\n> > >> > > > > > back as adjusted is \"rot90\", because it looks like it's had a 90\n> > >> > > > > > degree clockwise rotation performed (and libcamera transforms count\n> > >> > > > > > rotations as being clockwise). Does that also make sense?\n> > >> > > >\n> > >> > > > It might be the case today, but I wonder if that's desirable.\n> > >> > > >\n> > >> > > > If an application supplies CameraConfiguration::transform = Identity\n> > >> > > > it means it doesn't want any rotation applied by the library. It will\n> > >> > > > get back 270 to indicate that that's how the image is rotated and will\n> > >> > > > get an Adjusted configuration.\n> > >> > >\n> > >> > > I'm not quite sure I get that, perhaps it depends how we interpret\n> > >> > > \"applied by the library\". I guess I agree if we mean \"applied by the\n> > >> > > library together with any rotation of the sensor\"? Maybe a real\n> > >> >\n> > >> > \"Applied by the library\" == happens inside libcamera, ISP or sensor\n> > >> > flips..\n> > >> >\n> > >> > > example is helpful:\n> > >> > >\n> > >> > > So for a Raspberry Pi, for example, our sensors are mostly mounted\n> > >> > > upside down (V4L2_CID_CAMERA_SENSOR_ROTATION says \"180\"). Our users\n> > >> > > say the transform they want is transform=Identity i.e. images come\n> > >> > > back \"the right way up\". Internally it means that we do apply an extra\n> > >> > > h+v flip to undo the 180 degree rotation of the sensor.\n> > >> > >\n> > >> >\n> > >> > Here you go, this is where we disconnect.\n> > >> >\n> > >> > As I've interpreted the API so far, and the way we designed\n> > >> > properties::Rotation was to signal to the application what is the\n> > >> > status of the image as it comes from the sensor. Application could then\n> > >> > correct that by setting CameraConfiguration::transform, but in most\n> > >> > cases the image will be corrected at rendering time by the application\n> > >> > itself.\n> > >> >\n> > >> > IWO I was not expecting auto-correction and Transform::Identity to me\n> > >> > means \"give me the images as they come from the sensor\"\n> > >> >\n> > >> > > >\n> > >> > > > If instead an application supplies CameraConfiguration::transform = Rot270\n> > >> > > > and your camera -cannot- do any rotation, it will still receive back\n> > >> > > > Rot270 and the state is Valid.\n> > >> > >\n> > >> > > I think that's true if we change the sense of the libcamera.Transform\n> > >> > > rotation. Currently, as I said earlier, I think we get Rot90?\n> > >> > >\n> > >> > > >\n> > >> > > > If it asks for CameraConfiguration::transform = Rot270 and the camera\n> > >> > > > -can- do that, it will clear CameraConfiguration::transform to\n> > >> > > > Identity and return Adjusted ?\n> > >> > >\n> > >> > > I'm not sure there... The CameraConfiguration::transform is the final\n> > >> > > transform you want. If that can be done, it's left alone and you get\n> > >> > > back Valid. It's not telling you what else you would need to do\n> > >> > > afterwards.\n> > >> > >\n> > >> > > (To find out what you need to do afterwards you'd work out\n> > >> > > \"inverse(what-you-will-get) * what-you-wanted\", or something like\n> > >> > > that.)\n> > >> > >\n> > >> >\n> > >> > What I would like is to make as easier as possible for applications\n> > >> > to know what they have to do to have the image rotated in the way they\n> > >> > expect. There are several \"transform\" steps on the image, the first\n> > >> > one happens in libcamera, all the other ones are left to the\n> > >> > appplication.\n> > >>\n> > >> I think this is exactly what I want, namely \"to make it as easy as\n> > >> possible for applications\".\n> > >>\n> > >> On Raspberry Pi - and this has been the case for a couple of years\n> > >> now! - if a user wants images the \"normal\" way up then they simply\n> > >> pass Transform::Identity. If they want upside down images they pass\n> > >> Transform::Rot180. It's as easy as that, there is simply nothing else\n> > >> to do.\n> > >>\n> > >> Requiring all applications to look at the sensor rotation and then\n> > >> combine various things to figure out what they have to ask for (and\n> > >> possibly get it wrong) seems, well, actually quite unhelpful.\n> > >\n> > >\n> > > FWIW, I do agree that applications should not have to trouble themselves\n> > > with munging sensor rotation andrequested rotation to achieve their\n> > > desired outcome. This definitely wants to be hidden in the libcamera core\n> > > or pipeline handler layers.\n> > >\n> > > Naush\n> > >\n> > >\n> > >>\n> > >>\n> > >> But this is indeed what we need to talk about. Let's organise a call!\n> > >>\n> > >> Thanks\n> > >> David\n> > >>\n> > >> >\n> > >> > *) The image \"native\" transform is reported by Rotation (this assumes\n> > >> > that any implicit rotation buried in the sensor driver, as per our\n> > >> > last discussion with Dave is not taken into account)\n> > >> >\n> > >> > properties::Rotation = Rot270 = HFlip | Transpose\n> > >> >\n> > >> > *) Application sets CameraConfiguration::transform and call validate() to\n> > >> > check what can be done by libcamera and what should be done by next\n> > >> > layers\n> > >> >\n> > >> > CameraConfigurat::Transform = Rot270\n> > >> >\n> > >> > *) The camera validate the transform and as it can only flip it sets\n> > >> > transform to HFlip, and the state is Adjusted\n> > >> >\n> > >> > validate() -> The camera can only flip\n> > >> > -> CameraConfiguration = HFlip\n> > >> >\n> > >> > *) Application gets back what the camera can do and instrument the\n> > >> > rest of the rendering pipeline to perform the additional transforms\n> > >> > the camera cannot do. To get what you have to do I presume you have to\n> > >> >\n> > >> > additionalTransform = what_you_want & !what_you_get\n> > >> >\n> > >> > So yes, I was interpreting the usage of CameraConfiguration::transform\n> > >> > as a negotiation of what the library can do and what applications have\n> > >> > to do by themselves, which if I got you is different than what's\n> > >> > currecntly implemented ?\n> > >> >\n> > >> > > >\n> > >> > > > Is this what happens (I'm looking at the above code with David's\n> > >> > > > suggestion to use the inverse of roationTransform).\n> > >> > > >\n> > >> > > >\n> > >> > > > > >\n> > >> > > > > > So by a slightly arbitrary mixture of conventions, it looks to me as\n> > >> > > > > > though the use of the inverse that I queried is indeed wrong, because\n> > >> > > > > > we \"need camera 90 degree rotation\" => \"user transform rot90\" in this\n> > >> > > > > > case.\n> > >> > > > > >\n> > >> > > > > > Does that sound plausible... what do folks think?\n> > >> > > > >\n> > >> > > > > Ah, I think the question is whether the adjusted value means: \"this is what\n> > >> > > > > you'll get\" vs \"this is what you have to do yourself\".\n> > >> > > > >\n> > >> > > > > Above I argued in favor of the later, but I guess the first one is what the\n> > >> > > > > API is supposed to mean. That would imply that the client has compute the\n> > >> > > > > difference between requested and adjusted value itself.\n> > >> > > >\n> > >> > > > I always presumed the latter was meant to happen..\n> > >> > >\n> > >> > > Yes, I think this is the crux of the matter. I believe it's \"what you\n> > >> > > will get\". It's not \"what you need to do after\". (If it was \"what you\n> > >> > > need to do after\", it would change every time you try to validate the\n> > >> > > configuration!)\n> > >> > >\n> > >> >\n> > >> > On the \"it would change every time you try to validate the\n> > >> > configuration\": only if the camera cannot satisfy the transform you\n> > >> > have requested. If you ask for a flip and the camera can do it,\n> > >> > transform stays un-touched and the state is Valid ?\n> > >> >\n> > >> > > >\n> > >> > > > >\n> > >> > > > > In my case that would mean:\n> > >> > > > >\n> > >> > > > > requested CameraConfiguration::transform: Transform::Identity,\n> > >> > > > > V4L2_CID_CAMERA_SENSOR_ROTATION: 270, adjusted\n> > >> > > > > CameraConfiguration::transform: Transform::Rot270\n> > >> > > > >\n> > >> > > >\n> > >> > > > I guess it's Rot90 because of the clockwise/counter-clockwise issue ?\n> > >> > >\n> > >> > > Yes, looks right to me!\n> > >> > >\n> > >> > > >\n> > >> > > > > To compute what the client has to do, it would need to do:\n> > >> > > > > Transform::Identity - Transform::Rot270 = Transform::Rot90. And then apply\n> > >> > > > > that, a 90 degr. rotation clockwise.\n> > >> > >\n> > >> > > Yes, I guess I wrote it above, it's \"inverse(what-you-got) *\n> > >> > > what-you-wanted\", in this case inverse(rot270) * identity which is\n> > >> > > rot90 (in the current clockwise sense).\n> > >> > >\n> > >> > > > >\n> > >> > > > > Does that sound reasonable? In that case the patch here should indeed be\n> > >> > > > > `*transform = rotationTransform;`, without the inverse.\n> > >> > > > >\n> > >> > >\n> > >> > > Yes, agree!\n> > >> > >\n> > >> > > >\n> > >> > > > /0\\\n> > >> > > >\n> > >> > > > before we plumb this into and make assumptions that will be then\n> > >> > > > set into stone in our adaption layers, can we take a step back and\n> > >> > > > define what kind of API we would like to have ?\n> > >> > > >\n> > >> > > > Let's start by the definition of CameraConfiguration::transform.\n> > >> > > >\n> > >> > > > It's a member of CameraConfiguration, and as by definition it is\n> > >> > > > filled-in by the application and eventually adjusted by the Camera to\n> > >> > > > a value that's acceptable/doable for the current configuration.\n> > >> > > >\n> > >> > > > 1) Application to Camera = (counter?)clockwise degrees the camera\n> > >> > > > system should rotate the image before presenting it to the user\n> > >> > >\n> > >> > > This may depend on what exactly we mean by \"the camera system\". I'd\n> > >> > > explain it as\n> > >> > > \"the final resulting transform that the application wants to be\n> > >> > > applied to all the output images\". After all, this is actually the\n> > >> > > only thing the application knows!\n> > >> > >\n> > >> >\n> > >> > agreed\n> > >> >\n> > >> > > >\n> > >> > > > 2) Camera to application = the rotation the camera can actually do\n> > >> > > > (ie. only sensor flips == no Transpose)\n> > >> > >\n> > >> > > Agree, mostly, though I would say \"transform\" rather than \"rotation\"\n> > >> >\n> > >> > yes, more correct\n> > >> >\n> > >> > > because flips aren't rotations, you could in theory have a PH that\n> > >> > > could do transposes (even though we probably don't currently).\n> > >> > >\n> > >> >\n> > >> > none currently afaict\n> > >> >\n> > >> > > >\n> > >> > > > The current documentation doesn't explain how it gets adjusted\n> > >> > >\n> > >> > > This is true, though I also think it's platform dependent. If you had\n> > >> > > a PH that can't do transposes but can do flips, you have a choice how\n> > >> > > you handle (for example) rot90. You could say \"oh I can't do that, so\n> > >> > > in this case I won't do anything at all\" and leave the application to\n> > >> > > sort everything out. The Pi tries to say \"well, I'll do everything\n> > >> > > except for the transpose, so an application only has to worry about\n> > >> > > that one case\". But there is a choice...\n> > >> > >\n> > >> >\n> > >> > I agree it's platform-specific, but the way transform is negotiated with\n> > >> > application should be standardized\n> > >> >\n> > >> > > >\n> > >> > > > /**\n> > >> > > > * \\var CameraConfiguration::transform\n> > >> > > > * \\brief User-specified transform to be applied to the image\n> > >> > > > *\n> > >> > > > * The transform is a user-specified 2D plane transform that will be applied\n> > >> > > > * to the camera images by the processing pipeline before being handed to\n> > >> > > > * the application. This is subsequent to any transform that is already\n> > >> > > > * required to fix up any platform-defined rotation.\n> > >> > > > *\n> > >> > > > * The usual 2D plane transforms are allowed here (horizontal/vertical\n> > >> > > > * flips, multiple of 90-degree rotations etc.), but the validate() function\n> > >> > > > * may adjust this field at its discretion if the selection is not supported.\n> > >> > > > */\n> > >> > > >\n> > >> > > > The last part in particular, \"at its discrection\" it's really too\n> > >> > > > generic.\n> > >> > >\n> > >> > > I guess it's hinting at the fact that it's up to the PH, and we don't\n> > >> > > mandate the exact way in which it can fail to do what you asked for!\n> > >> > > But it would be fine to have a discussion on that.\n> > >> > >\n> > >> >\n> > >> > That's the point that most concerns me :)\n> > >> >\n> > >> > > >\n> > >> > > > Let's start from the beginning: what application should use transform\n> > >> > > > for ? I presume they should:\n> > >> > > >\n> > >> > > > 1) Inspect propertis::Rotation\n> > >> > > > 2) Correct the image for the properties::Rotation amount (actually,\n> > >> > > > the inverse because of the clockwise/counter-clockwise thing)\n> > >> > > > 3) Add any additional rotation they would like on top of this\n> > >> > >\n> > >> > > So I think the point of the transform is that you *don't* have to\n> > >> > > worry about the properties::Rotation. It takes care of that for you.\n> > >> > > You say \"I want upside down images\" and you get them, whatever the\n> > >> > > sensor's rotation.\n> > >> >\n> > >> > That's acceptable as well if we prefer that, but it has to be\n> > >> > standardized among pipelines imho\n> > >> >\n> > >> > >\n> > >> > > >\n> > >> > > > How should transform be adjusted ?\n> > >> > > >\n> > >> > > > Identity:\n> > >> > > > If application->identity the camera does not have to apply any\n> > >> > > > transform. Should the transform be adjutes like it happens today to\n> > >> > > > compensate the Rotation (at least that's what I presume happens) ? I\n> > >> > > > presume no, applications can infer rotation from properties and if they\n> > >> > > > chose Identity either other layers above will do the transform or\n> > >> > > > they're fine with images as they are.\n> > >> > >\n> > >> > > If the application says it wants the \"identity\" transform, then it\n> > >> > > should get output images with no transformation applied. But this does\n> > >> > > mean it will undo the sensor rotation (this being the normal case on\n> > >> > > the Pi).\n> > >> > >\n> > >> >\n> > >> > right, it depends if we decide to account for the sensor rotation\n> > >> > inside libcamera or not\n> > >> >\n> > >> > > >\n> > >> > > > A supported/non-supported transform:\n> > >> > > > An application asks for a VFLIP, the camera can do it, transform is\n> > >> > > > not changed. If the camera cannot flip, the transform is set back to\n> > >> > > > Identity and the state is adjusted.\n> > >> > >\n> > >> > > Yes, mostly, except that the sensor may be mounted with a rotation,\n> > >> > > but if it isn't, then I agree with this.\n> > >> > >\n> > >> > > >\n> > >> > > > A partially non-supported transform:\n> > >> > > > An application asks for Rot270 (Transpose | HFLIP) and the camera can\n> > >> > > > only do HFLIP. transform sould be then set to HFLIP and applications\n> > >> > > > know that the remaining transformation should be handled elsewhere.\n> > >> > >\n> > >> > > As I suggested above, I'm not sure whether we want to mandate this. I\n> > >> > > agree it seems useful behaviour, which is why the Pi does it, but I'm\n> > >> > > open minded about requiring it or not.\n> > >> > >\n> > >> > > >\n> > >> > > > Are we missing anything with such behaviour ? I presume it's rather\n> > >> > > > similar to what happens today if not for the fact that the validate()\n> > >> > > > implementation adjusts transform to rotation in case it cannot flip\n> > >> > > >\n> > >> > > > Transform combined = transform * data_->rotationTransform_;\n> > >> > > > if (!data_->supportsFlips_ && !!combined)\n> > >> > > > transform = -data_->rotationTransform_;\n> > >> > > >\n> > >> > > > Not really sure I got why.\n> > >> > > >\n> > >> > > > All of this, but an even more fundamental question: is it too late/is\n> > >> > > > it worh it to adjust the Transform definition to adopt the same\n> > >> > > > direction as the kernel defined rotation ?\n> > >> > > >\n> > >> > > > Sorry for the long email :/\n> > >> > >\n> > >> > > No problem! I like nothing more than discussing transforms, except\n> > >> > > possibly colour spaces!\n> > >> > >\n> > >> >\n> > >> > :)\n> > >> >\n> > >> > > More seriously, here are my key points:\n> > >> >\n> > >> > I'll try to summarize what I see differently\n> > >> >\n> > >> > >\n> > >> > > 1. The CameraConfiguration::transform field says what the application\n> > >> > > wants the final output images to have.\n> > >> >\n> > >> > 1. The CameraConfiguration::transform field says what transform the\n> > >> > library has to apply to images as they arrive from the sensor in its\n> > >> > default configuration\n> > >> >\n> > >> > > 2. It takes account of the sensor rotation. The most obvious example\n> > >> > > is the Pi: if the sensor is mounted upside down, and the applications\n> > >> > > ask for images \"the right way up\" (i.e. transform = identity), then\n> > >> > > the PH will apply h and v flips to compensate for the sensor mounting.\n> > >> >\n> > >> > 2. It doesn't automatically account for sensor rotation. Application\n> > >> > might decide to do that along the rendering pipeline, in example\n> > >> >\n> > >> > > 3. If the PH can do what the application requests, it leaves the\n> > >> > > transform alone and your configuration is Valid.\n> > >> >\n> > >> > Agreed\n> > >> >\n> > >> > > 4. If the PH can't do what the application wants, it will set the\n> > >> > > value to something that it can do, and the configuration will be\n> > >> > > Adjusted.\n> > >> >\n> > >> > Agreed\n> > >> >\n> > >> > >\n> > >> > > If folks wanted a short conference call to iron any of this out, I'd\n> > >> > > be very happy to join in!\n> > >> >\n> > >> > I've asked others to have a look here, let's see if we need a call,\n> > >> > I'm all for it !\n> > >> >\n> > >> > Thanks\n> > >> > j\n> > >> >\n> > >> > >\n> > >> > > Thanks\n> > >> > > David\n> > >> > >\n> > >> > > >\n> > >> > > > > > Thanks everyone!\n> > >> > > > > > David\n> > >> > > > >\n> > >> > > > > Thanks!\n> > >> > > > >\n> > >> > > > > Robert\n> > >> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 09812BD87C\n\tfor ;\n\tWed, 18 Jan 2023 17:22:29 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6E83F625CC;\n\tWed, 18 Jan 2023 18:22:28 +0100 (CET)","from mail-oi1-x230.google.com (mail-oi1-x230.google.com\n\t[IPv6:2607:f8b0:4864:20::230])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CA597625CC\n\tfor ;\n\tWed, 18 Jan 2023 18:22:26 +0100 (CET)","by mail-oi1-x230.google.com with SMTP id o66so29182362oia.6\n\tfor ;\n\tWed, 18 Jan 2023 09:22:26 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1674062548;\n\tbh=QyUsR/Ydeh70/JjiV0ws64/xeHDTxxIQeZc/3aK7Ieo=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=O7ceklhNTPiILwWmGsee+BdZieeN9T6Ng2ACXTBbMNLKUb7nH/Yrakrl9fD2OOBAF\n\tvaepeqPW047fsZ1k1q+5LDIMst/irMZdgfWBZ6qoqU/OuRUbDF+s4T2tz/kwMfXr/M\n\tNc1SLL2h8KNq9s2uryyempFTnVJYEtWDNP1To0e4WFLqVcRUb1wQQjf0HJpHkMvc9U\n\tpti7jvqZs0ebMAI3026EwXFJsNGapC3U20wd8E711EmUIInpQRvkk9vE0JloOy6gKu\n\ttqQfD/jH0VYKGBy6WPciEJUaRdf47JygWXg7231lCtQQedn99J+fefl1vksW5xxiqp\n\tlD9pn24+oxj9Q==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=CPpofbm668NfkkjQAXO3VY9aSrd4AvCzeoYzuokppZg=;\n\tb=aQNIUoWfDtdOstkT9Zdg2MVs7oo97FGGikdLgx2nAe0wbFA3QSD1kEyG0r8S5HtHfH\n\tFHj16C7uEv43rZjFbsEtUyXTtFc+uSsYdkPvUTFGajBui2BzeZQqyJN4pPFBdC1eZXm7\n\tBz3RVTbX21p3xxLR6BxdYQVJ3PrB9Juj3BffYgvLeUAhBjB2Y6tXSK0oPDH6ZCF8mSZS\n\tTnJefFFJpKwh3AFN177swSVx+2kguxWNnpETCosTBhu/lGAKzTYRd90hIP8TrrajSRLQ\n\tKqNZdhjz/wAFAKGdMuF3KS8JVDiwr1iX7F9HEo6yiOyKfIYIUphjso6a1m5XM76lPRaM\n\tGGlw=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"aQNIUoWf\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=CPpofbm668NfkkjQAXO3VY9aSrd4AvCzeoYzuokppZg=;\n\tb=n18tO9Lt0WTnDNUkh1FY7EMP6kFsTbKidycJK8mc7S+Y/ay8e9JxC27YTMaxK4iLV0\n\tFnIfGad8i31twpFi426TXUmAH4jcRCQvd1vLoot0a4cotUaFc6DxQqUwr5HOgg2MMhyC\n\tfUNCW8pVWZ8YdjiyTGYmWuXwmZy8uoVjeRLfuXhrnY5dOFN6dqc/pVF39CjZbYKNTMYG\n\t1ip6BP9/04HeFuNVRfWTCRBzH9/crqD6ZklriIwxpKmHGxCW1JFBIMBsazGCnwmtUOW7\n\tTC/JF+WNcLbtL5iK8Q71ByELY18SAq5EqgCnEUBsgn7nSuvKNDr5G6lzcYpYgyBvR421\n\tKX/Q==","X-Gm-Message-State":"AFqh2koMzLQG747p0vIdGvQMzAiunITsSv74OKCnM12NLrAH10JiTmrk\n\tldzfXZYYMQN/hvzEbKeXukYLy5VZDP2Y1WOz4Kxj8i2rTy3XGQ==","X-Google-Smtp-Source":"AMrXdXvss1uLyPBVWSq2knycTchm8twAQ+s6tdgD/oeQcV8yGsPC+ivXq3C7Yu3g31Ig6zMvPpy7/Ms8xgr57PYon4Q=","X-Received":"by 2002:aca:5b09:0:b0:354:4cfe:aaad with SMTP id\n\tp9-20020aca5b09000000b003544cfeaaadmr342001oib.246.1674062545016;\n\tWed, 18 Jan 2023 09:22:25 -0800 (PST)","MIME-Version":"1.0","References":"<51ece26c-cc8f-566e-3be6-b77b874f3b3f@collabora.com>\n\t\n\t<2d5edeed-b633-1015-7afa-e2a1a0a383fe@posteo.de>\n\t<20221209124746.r5er7lcsm3pen6r7@uno.localdomain>\n\t\n\t<20221213112251.jnpgyzitusutkqm6@uno.localdomain>\n\t\n\t\n\t<1b0cd37c-75a1-03b7-7df7-cecf0deb0cc7@posteo.de>\n\t\n\t<20230117120945.76xlszfljimhibzw@uno.localdomain>","In-Reply-To":"<20230117120945.76xlszfljimhibzw@uno.localdomain>","Date":"Wed, 18 Jan 2023 17:22:14 +0000","Message-ID":"","To":"Jacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 5/9] libcamera: camera_sensor:\n\tValidate Transform","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org,\n\tSakari Ailus ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]