[{"id":32370,"web_url":"https://patchwork.libcamera.org/comment/32370/","msgid":"<3i57xvgmtl7ey26isg5t7xubghj26pitm37szvyq6q5ingi7wz@d2udte44eqsq>","date":"2024-11-25T19:30:50","subject":"Re: [PATCH v2 7/8] pipeline: rkisp1: Fix ScalerCrop to be in sensor\n\tcoordinates","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"HI Stefan\n\nOn Mon, Nov 25, 2024 at 04:14:16PM +0100, Stefan Klug wrote:\n> ScalerCrop is specified as being in sensor coordinates. The current\n> dewarper implementation on the imx8mp handles ScalerCrop in dewarper\n> coordinates. This leads to unexpected results and an unusable ScalerCrop\n> control in camshark. Fix that by transforming back and forth between\n> sensor coordinates and dewarper coordinates.\n>\n> Signed-off-by: Stefan Klug \n> Reviewed-by: Paul Elder \n>\n> ---\n> Changes in v2:\n> - Initialize dewarperSensorCrop_ to sane defaults\n> - Collected tags\n> ---\n> src/libcamera/pipeline/rkisp1/rkisp1.cpp | 50 +++++++++++++++++++-----\n> 1 file changed, 41 insertions(+), 9 deletions(-)\n>\n> diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> index 1ba416aaa545..0a044b08bc87 100644\n> --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> @@ -204,6 +204,7 @@ private:\n> \tRkISP1SelfPath selfPath_;\n>\n> \tstd::unique_ptr dewarper_;\n> +\tRectangle dewarperSensorCrop_;\n> \tbool useDewarper_;\n>\n> \tstd::optional activeCrop_;\n> @@ -863,6 +864,15 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n> \t\t\t\toutputCfgs.push_back(const_cast(cfg));\n> \t\t\t\tret = dewarper_->configure(cfg, outputCfgs);\n> \t\t\t\tuseDewarper_ = ret ? false : true;\n> +\n> +\t\t\t\t/*\n> +\t\t\t\t * Calculate the crop rectangle of the data\n> +\t\t\t\t * flowing into the dewarper in sensor\n> +\t\t\t\t * coordinates.\n> +\t\t\t\t */\n> +\t\t\t\tdewarperSensorCrop_ =\n> +\t\t\t\t\toutputCrop.transformedBetween(inputCrop,\n> +\t\t\t\t\t\t\t\t sensorInfo.analogCrop);\n\nI have a little trouble with the name dewarperSensorCrop_.\n\nIt represent the maximum crop rectangle in sensor's coordinate, right?\n\nWhy not call it scalerMaxCrop_ and initialize the ScalerCropMaximum\ncontrol as well with it ?\n\n> \t\t\t}\n> \t\t} else if (hasSelfPath_) {\n> \t\t\tret = selfPath_.configure(cfg, format);\n> @@ -1225,10 +1235,19 @@ int PipelineHandlerRkISP1::updateControls(RkISP1CameraData *data)\n> \t\tstd::pair cropLimits =\n> \t\t\tdewarper_->inputCropBounds(&data->mainPathStream_);\n>\n> -\t\tcontrols[&controls::ScalerCrop] = ControlInfo(cropLimits.first,\n> -\t\t\t\t\t\t\t cropLimits.second,\n> -\t\t\t\t\t\t\t cropLimits.second);\n> -\t\tactiveCrop_ = cropLimits.second;\n> +\t\t/*\n> +\t\t * ScalerCrop is specified to be in Sensor coordinates.\n> +\t\t * So we need to transform the limits to sensor coordinates.\n> +\t\t * We can safely assume that the maximum crop limit contains the\n> +\t\t * full fov of the dewarper.\n> +\t\t */\n> +\t\tRectangle min = cropLimits.first.transformedBetween(cropLimits.second,\n> +\t\t\t\t\t\t\t\t dewarperSensorCrop_);\n> +\t\tRectangle max = cropLimits.second.transformedBetween(cropLimits.second,\n> +\t\t\t\t\t\t\t\t dewarperSensorCrop_);\n> +\n> +\t\tcontrols[&controls::ScalerCrop] = ControlInfo(min, max, max);\n> +\t\tactiveCrop_ = max;\n> \t}\n>\n> \t/* Add the IPA registered controls to list of camera controls. */\n> @@ -1256,6 +1275,8 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor)\n> \t/* Initialize the camera properties. */\n> \tdata->properties_ = data->sensor_->properties();\n>\n> +\tdewarperSensorCrop_ = Rectangle(data->sensor_->resolution());\n> +\n> \t/*\n> \t * \\todo Read delay values from the sensor itself or from a\n> \t * a sensor database. For now use generic values taken from\n> @@ -1479,22 +1500,33 @@ void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer)\n> \t/* Handle scaler crop control. */\n> \tconst auto &crop = request->controls().get(controls::ScalerCrop);\n> \tif (crop) {\n> -\t\tRectangle appliedRect = crop.value();\n> +\t\tRectangle rect = crop.value();\n> +\n> +\t\t/*\n> +\t\t * ScalerCrop is specified to be in Sensor coordinates.\n> +\t\t * So we need to transform it into dewarper coordinates.\n> +\t\t * We can safely assume that the maximum crop limit contains the\n> +\t\t * full fov of the dewarper.\n> +\t\t */\n> +\t\tstd::pair cropLimits =\n> +\t\t\tdewarper_->inputCropBounds(&data->mainPathStream_);\n>\n> +\t\trect = rect.transformedBetween(dewarperSensorCrop_, cropLimits.second);\n\nDoes this call for a Rectangle::transformBetween() ?\n\n> \t\tint ret = dewarper_->setInputCrop(&data->mainPathStream_,\n> -\t\t\t\t\t\t &appliedRect);\n> -\t\tif (!ret && appliedRect != crop.value()) {\n> +\t\t\t\t\t\t &rect);\n> +\t\trect = rect.transformedBetween(cropLimits.second, dewarperSensorCrop_);\n> +\t\tif (!ret && rect != crop.value()) {\n> \t\t\t/*\n> \t\t\t * If the rectangle is changed by setInputCrop on the\n> \t\t\t * dewarper, log a debug message and cache the actual\n> \t\t\t * applied rectangle for metadata reporting.\n> \t\t\t */\n> \t\t\tLOG(RkISP1, Debug)\n> -\t\t\t\t<< \"Applied rectangle \" << appliedRect.toString()\n> +\t\t\t\t<< \"Applied rectangle \" << rect.toString()\n> \t\t\t\t<< \" differs from requested \" << crop.value().toString();\n> \t\t}\n>\n> -\t\tactiveCrop_ = appliedRect;\n> +\t\tactiveCrop_ = rect;\n> \t}\n>\n> \t/*\n> --\n> 2.43.0\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 10671C3274\n\tfor ;\n\tMon, 25 Nov 2024 19:30:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4723F66047;\n\tMon, 25 Nov 2024 20:30:56 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5078465F66\n\tfor ;\n\tMon, 25 Nov 2024 20:30:54 +0100 (CET)","from ideasonboard.com (mob-5-90-139-188.net.vodafone.it\n\t[5.90.139.188])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 1C8314AD;\n\tMon, 25 Nov 2024 20:30:32 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"R1nVWvfN\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732563032;\n\tbh=5TSYqnro/LdKwnFjan2KXi2uIA/hTCXjJw2zs11NxjY=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=R1nVWvfNqn8Gs3DHAvMlucOkBTagNksFHR0S6QGdupn5+2m3+e55HA/Vcqp6BN+SV\n\tjQmzWD2JPE9adZsyy1YH1N0Fn85eViksLeqzB3wRJ/9pbLrA6RMSlxl/WwCaB+qnbN\n\t6OGO/erefLI/mAmeJCPIZQeAmigWVnGNkP8RXYRo=","Date":"Mon, 25 Nov 2024 20:30:50 +0100","From":"Jacopo Mondi ","To":"Stefan Klug ","Cc":"libcamera-devel@lists.libcamera.org, \n\tPaul Elder ","Subject":"Re: [PATCH v2 7/8] pipeline: rkisp1: Fix ScalerCrop to be in sensor\n\tcoordinates","Message-ID":"<3i57xvgmtl7ey26isg5t7xubghj26pitm37szvyq6q5ingi7wz@d2udte44eqsq>","References":"<20241125151430.2437285-1-stefan.klug@ideasonboard.com>\n\t<20241125151430.2437285-8-stefan.klug@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20241125151430.2437285-8-stefan.klug@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32428,"web_url":"https://patchwork.libcamera.org/comment/32428/","msgid":"","date":"2024-11-28T13:09:37","subject":"Re: [PATCH v2 7/8] pipeline: rkisp1: Fix ScalerCrop to be in sensor\n\tcoordinates","submitter":{"id":184,"url":"https://patchwork.libcamera.org/api/people/184/","name":"Stefan Klug","email":"stefan.klug@ideasonboard.com"},"content":"Hi Jacopo,\n\nOn Mon, Nov 25, 2024 at 08:30:50PM +0100, Jacopo Mondi wrote:\n> HI Stefan\n> \n> On Mon, Nov 25, 2024 at 04:14:16PM +0100, Stefan Klug wrote:\n> > ScalerCrop is specified as being in sensor coordinates. The current\n> > dewarper implementation on the imx8mp handles ScalerCrop in dewarper\n> > coordinates. This leads to unexpected results and an unusable ScalerCrop\n> > control in camshark. Fix that by transforming back and forth between\n> > sensor coordinates and dewarper coordinates.\n> >\n> > Signed-off-by: Stefan Klug \n> > Reviewed-by: Paul Elder \n> >\n> > ---\n> > Changes in v2:\n> > - Initialize dewarperSensorCrop_ to sane defaults\n> > - Collected tags\n> > ---\n> > src/libcamera/pipeline/rkisp1/rkisp1.cpp | 50 +++++++++++++++++++-----\n> > 1 file changed, 41 insertions(+), 9 deletions(-)\n> >\n> > diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> > index 1ba416aaa545..0a044b08bc87 100644\n> > --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> > +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> > @@ -204,6 +204,7 @@ private:\n> > \tRkISP1SelfPath selfPath_;\n> >\n> > \tstd::unique_ptr dewarper_;\n> > +\tRectangle dewarperSensorCrop_;\n> > \tbool useDewarper_;\n> >\n> > \tstd::optional activeCrop_;\n> > @@ -863,6 +864,15 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n> > \t\t\t\toutputCfgs.push_back(const_cast(cfg));\n> > \t\t\t\tret = dewarper_->configure(cfg, outputCfgs);\n> > \t\t\t\tuseDewarper_ = ret ? false : true;\n> > +\n> > +\t\t\t\t/*\n> > +\t\t\t\t * Calculate the crop rectangle of the data\n> > +\t\t\t\t * flowing into the dewarper in sensor\n> > +\t\t\t\t * coordinates.\n> > +\t\t\t\t */\n> > +\t\t\t\tdewarperSensorCrop_ =\n> > +\t\t\t\t\toutputCrop.transformedBetween(inputCrop,\n> > +\t\t\t\t\t\t\t\t sensorInfo.analogCrop);\n> \n> I have a little trouble with the name dewarperSensorCrop_.\n> \n> It represent the maximum crop rectangle in sensor's coordinate, right?\n> \n> Why not call it scalerMaxCrop_ and initialize the ScalerCropMaximum\n> control as well with it ?\n\nYou're completely right. In this case they are equal. I renamed it to\nscalerMaxCrop_. That variable is only temporary and will vanish again in\nthe upcoming series with full dw100 dewarp/pan/zoom/scale support.\n\n> \n> > \t\t\t}\n> > \t\t} else if (hasSelfPath_) {\n> > \t\t\tret = selfPath_.configure(cfg, format);\n> > @@ -1225,10 +1235,19 @@ int PipelineHandlerRkISP1::updateControls(RkISP1CameraData *data)\n> > \t\tstd::pair cropLimits =\n> > \t\t\tdewarper_->inputCropBounds(&data->mainPathStream_);\n> >\n> > -\t\tcontrols[&controls::ScalerCrop] = ControlInfo(cropLimits.first,\n> > -\t\t\t\t\t\t\t cropLimits.second,\n> > -\t\t\t\t\t\t\t cropLimits.second);\n> > -\t\tactiveCrop_ = cropLimits.second;\n> > +\t\t/*\n> > +\t\t * ScalerCrop is specified to be in Sensor coordinates.\n> > +\t\t * So we need to transform the limits to sensor coordinates.\n> > +\t\t * We can safely assume that the maximum crop limit contains the\n> > +\t\t * full fov of the dewarper.\n> > +\t\t */\n> > +\t\tRectangle min = cropLimits.first.transformedBetween(cropLimits.second,\n> > +\t\t\t\t\t\t\t\t dewarperSensorCrop_);\n> > +\t\tRectangle max = cropLimits.second.transformedBetween(cropLimits.second,\n> > +\t\t\t\t\t\t\t\t dewarperSensorCrop_);\n> > +\n> > +\t\tcontrols[&controls::ScalerCrop] = ControlInfo(min, max, max);\n> > +\t\tactiveCrop_ = max;\n> > \t}\n> >\n> > \t/* Add the IPA registered controls to list of camera controls. */\n> > @@ -1256,6 +1275,8 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor)\n> > \t/* Initialize the camera properties. */\n> > \tdata->properties_ = data->sensor_->properties();\n> >\n> > +\tdewarperSensorCrop_ = Rectangle(data->sensor_->resolution());\n> > +\n> > \t/*\n> > \t * \\todo Read delay values from the sensor itself or from a\n> > \t * a sensor database. For now use generic values taken from\n> > @@ -1479,22 +1500,33 @@ void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer)\n> > \t/* Handle scaler crop control. */\n> > \tconst auto &crop = request->controls().get(controls::ScalerCrop);\n> > \tif (crop) {\n> > -\t\tRectangle appliedRect = crop.value();\n> > +\t\tRectangle rect = crop.value();\n> > +\n> > +\t\t/*\n> > +\t\t * ScalerCrop is specified to be in Sensor coordinates.\n> > +\t\t * So we need to transform it into dewarper coordinates.\n> > +\t\t * We can safely assume that the maximum crop limit contains the\n> > +\t\t * full fov of the dewarper.\n> > +\t\t */\n> > +\t\tstd::pair cropLimits =\n> > +\t\t\tdewarper_->inputCropBounds(&data->mainPathStream_);\n> >\n> > +\t\trect = rect.transformedBetween(dewarperSensorCrop_, cropLimits.second);\n> \n> Does this call for a Rectangle::transformBetween() ?\n\nMaybe :-) Do we generally duplicate every function in geometry? I\nbelieve this line also vanishes in the upcoming series.\n\nCheers,\nStefan\n\n> \n> > \t\tint ret = dewarper_->setInputCrop(&data->mainPathStream_,\n> > -\t\t\t\t\t\t &appliedRect);\n> > -\t\tif (!ret && appliedRect != crop.value()) {\n> > +\t\t\t\t\t\t &rect);\n> > +\t\trect = rect.transformedBetween(cropLimits.second, dewarperSensorCrop_);\n> > +\t\tif (!ret && rect != crop.value()) {\n> > \t\t\t/*\n> > \t\t\t * If the rectangle is changed by setInputCrop on the\n> > \t\t\t * dewarper, log a debug message and cache the actual\n> > \t\t\t * applied rectangle for metadata reporting.\n> > \t\t\t */\n> > \t\t\tLOG(RkISP1, Debug)\n> > -\t\t\t\t<< \"Applied rectangle \" << appliedRect.toString()\n> > +\t\t\t\t<< \"Applied rectangle \" << rect.toString()\n> > \t\t\t\t<< \" differs from requested \" << crop.value().toString();\n> > \t\t}\n> >\n> > -\t\tactiveCrop_ = appliedRect;\n> > +\t\tactiveCrop_ = rect;\n> > \t}\n> >\n> > \t/*\n> > --\n> > 2.43.0\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 89AB4BDE6B\n\tfor ;\n\tThu, 28 Nov 2024 13:09:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 856F565F8E;\n\tThu, 28 Nov 2024 14:09:41 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0CCC165898\n\tfor ;\n\tThu, 28 Nov 2024 14:09:40 +0100 (CET)","from ideasonboard.com (unknown\n\t[IPv6:2a00:6020:448c:6c00:22e0:a94:b035:820])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 2C203526;\n\tThu, 28 Nov 2024 14:09:16 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"olmMIoJS\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732799356;\n\tbh=PkTdZfF1MXQ6Gz9mKBvNF7AV4e+9RnErxBNOqGhiBSg=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=olmMIoJSj9YkpR+cCsHdh76uSjJbCUPWrvvr7oxMRfKZikYilopfAOg8SrAzt2GCC\n\tzhKeQWLrpTJFRBHRFwvEUqMdhzUa4z07o77B/GdlfQQPZJu9rqoBFsCFlLxMRGVMAw\n\tJrZ/eaZ4bkhU9w0/bz1xtCS+CsDBp12RKLDXHxBI=","Date":"Thu, 28 Nov 2024 14:09:37 +0100","From":"Stefan Klug ","To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org, \n\tPaul Elder ","Subject":"Re: [PATCH v2 7/8] pipeline: rkisp1: Fix ScalerCrop to be in sensor\n\tcoordinates","Message-ID":"","References":"<20241125151430.2437285-1-stefan.klug@ideasonboard.com>\n\t<20241125151430.2437285-8-stefan.klug@ideasonboard.com>\n\t<3i57xvgmtl7ey26isg5t7xubghj26pitm37szvyq6q5ingi7wz@d2udte44eqsq>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<3i57xvgmtl7ey26isg5t7xubghj26pitm37szvyq6q5ingi7wz@d2udte44eqsq>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]