[{"id":32368,"web_url":"https://patchwork.libcamera.org/comment/32368/","msgid":"","date":"2024-11-25T19:27:12","subject":"Re: [PATCH v2 2/8] libcamera: rkisp1: Keep aspect ratio on imx8mp","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Stefan\n\nOn Mon, Nov 25, 2024 at 04:14:11PM +0100, Stefan Klug wrote:\n> In the current code, the input stage of the image resizer is used to\n> apply a crop to keep the aspect ratio in cases where the requested\n> output aspect ratio differs from the one of the selected sensor mode. On\n> the imx8mp the resizer hardware is not capable of cropping (for\n> reference see also rkisp1-resizer.c:rkisp1_rsz_set_sink_crop() in the\n> linux kernel v6.10).\n>\n> Therefore apply the necessary cropping on the output of the ISP (on the\n> image stabilization block). The cropping code on the image resizer\n> doesn't need modifications as the requested crop gets ignored by the\n> kernel.\n>\n> While at it, remove a todo comment that is no longer needed.\n>\n> Signed-off-by: Stefan Klug \n> Reviewed-by: Paul Elder \n> Reviewed-by: Jacopo Mondi \n>\n> ---\n>\n> Changes in v2:\n> - Fixed alignment\n> - Improved comments\n> - Removed todo comment\n> - Collected tags\n> ---\n> src/libcamera/pipeline/rkisp1/rkisp1.cpp | 16 ++++++++++++++++\n> src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 13 ++++++++-----\n> 2 files changed, 24 insertions(+), 5 deletions(-)\n>\n> diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> index 5ffcfbce56be..a66b9dda83ab 100644\n> --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> @@ -811,6 +811,22 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n> \tif (!isRaw_)\n> \t\tformat.code = MEDIA_BUS_FMT_YUYV8_2X8;\n>\n> +\t/*\n> +\t * On devices without DUAL_CROP (like the imx8mp) cropping needs to be\n> +\t * done on the ISP/IS output.\n> +\t */\n> +\tif (media_->hwRevision() == RKISP1_V_IMX8MP) {\n> +\t\t/* imx8mp has only a single path. */\n> +\t\tconst auto &cfg = config->at(0);\n> +\t\tSize ispCrop = format.size.boundedToAspectRatio(cfg.size)\n> +\t\t\t\t\t .alignedUpTo(2, 2);\n> +\t\trect = ispCrop.centeredTo(Rectangle(format.size).center());\n> +\t\tif (ispCrop != format.size)\n> +\t\t\tLOG(RkISP1, Info) << \"ISP output needs to be cropped to \"\n\nvery minor nit:\n\nWhat do you think about dropping \"needs to be\" from the Info message.\n\nAlso, this is a fairly typical thing, cropping to the output aspect\nratio before scaling, is it worth a Info message ?\n\nThanks\n j\n\n> +\t\t\t\t\t << rect;\n> +\t\tformat.size = ispCrop;\n> +\t}\n> +\n> \tLOG(RkISP1, Debug)\n> \t\t<< \"Configuring ISP output pad with \" << format\n> \t\t<< \" crop \" << rect;\n> diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> index 236d05af7a2f..eee5b09e2ff0 100644\n> --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> @@ -417,11 +417,14 @@ int RkISP1Path::configure(const StreamConfiguration &config,\n> \t/*\n> \t * Crop on the resizer input to maintain FOV before downscaling.\n> \t *\n> -\t * \\todo The alignment to a multiple of 2 pixels is required but may\n> -\t * change the aspect ratio very slightly. A more advanced algorithm to\n> -\t * compute the resizer input crop rectangle is needed, and it should\n> -\t * also take into account the need to crop away the edge pixels affected\n> -\t * by the ISP processing blocks.\n> +\t * Note that this does not apply to devices without DUAL_CROP support\n> +\t * (like imx8mp) , where the cropping needs to be done on the\n> +\t * ImageStabilizer block on the ISP source pad and therefore is\n> +\t * configured before this stage. For simplicity we still set the crop.\n> +\t * This gets ignored by the kernel driver because the hardware is\n> +\t * missing the capability.\n> +\t *\n> +\t * Alignment to a multiple of 2 pixels is required by the resizer.\n> \t */\n> \tSize ispCrop = inputFormat.size.boundedToAspectRatio(config.size)\n> \t\t\t\t .alignedUpTo(2, 2);\n> --\n> 2.43.0\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 03FDABD160\n\tfor ;\n\tMon, 25 Nov 2024 19:27:19 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4F90965F91;\n\tMon, 25 Nov 2024 20:27:18 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B544665F66\n\tfor ;\n\tMon, 25 Nov 2024 20:27:16 +0100 (CET)","from ideasonboard.com (mob-5-90-139-188.net.vodafone.it\n\t[5.90.139.188])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id AA9754AD;\n\tMon, 25 Nov 2024 20:26:54 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"WkGVaMIL\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732562814;\n\tbh=Z9BpBHLHctvafD3Aho7EDFby2dtHDY8Is9oFNtvXzgo=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=WkGVaMILgPVMEnGO4Pqe1mtjM6zbtAXssu2OGpI6MBazx3xtJ+RPMm90YDZoS83ut\n\tjQ/1kEzQzENXqOqz7YRyZbqqEwv4jDxXQyyvskNVzulevV10eMTTxYsdxStDBksRCr\n\tX/bIcFMdlM6p7OpQZrcm4ChZp/inAyw7EHox6KeQ=","Date":"Mon, 25 Nov 2024 20:27:12 +0100","From":"Jacopo Mondi ","To":"Stefan Klug ","Cc":"libcamera-devel@lists.libcamera.org, \n\tPaul Elder ,\n\tJacopo Mondi ","Subject":"Re: [PATCH v2 2/8] libcamera: rkisp1: Keep aspect ratio on imx8mp","Message-ID":"","References":"<20241125151430.2437285-1-stefan.klug@ideasonboard.com>\n\t<20241125151430.2437285-3-stefan.klug@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20241125151430.2437285-3-stefan.klug@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32423,"web_url":"https://patchwork.libcamera.org/comment/32423/","msgid":"","date":"2024-11-28T12:55:49","subject":"Re: [PATCH v2 2/8] libcamera: rkisp1: Keep aspect ratio on imx8mp","submitter":{"id":184,"url":"https://patchwork.libcamera.org/api/people/184/","name":"Stefan Klug","email":"stefan.klug@ideasonboard.com"},"content":"Hi Jacopo,\n\nOn Mon, Nov 25, 2024 at 08:27:12PM +0100, Jacopo Mondi wrote:\n> Hi Stefan\n> \n> On Mon, Nov 25, 2024 at 04:14:11PM +0100, Stefan Klug wrote:\n> > In the current code, the input stage of the image resizer is used to\n> > apply a crop to keep the aspect ratio in cases where the requested\n> > output aspect ratio differs from the one of the selected sensor mode. On\n> > the imx8mp the resizer hardware is not capable of cropping (for\n> > reference see also rkisp1-resizer.c:rkisp1_rsz_set_sink_crop() in the\n> > linux kernel v6.10).\n> >\n> > Therefore apply the necessary cropping on the output of the ISP (on the\n> > image stabilization block). The cropping code on the image resizer\n> > doesn't need modifications as the requested crop gets ignored by the\n> > kernel.\n> >\n> > While at it, remove a todo comment that is no longer needed.\n> >\n> > Signed-off-by: Stefan Klug \n> > Reviewed-by: Paul Elder \n> > Reviewed-by: Jacopo Mondi \n> >\n> > ---\n> >\n> > Changes in v2:\n> > - Fixed alignment\n> > - Improved comments\n> > - Removed todo comment\n> > - Collected tags\n> > ---\n> > src/libcamera/pipeline/rkisp1/rkisp1.cpp | 16 ++++++++++++++++\n> > src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 13 ++++++++-----\n> > 2 files changed, 24 insertions(+), 5 deletions(-)\n> >\n> > diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> > index 5ffcfbce56be..a66b9dda83ab 100644\n> > --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> > +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n> > @@ -811,6 +811,22 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n> > \tif (!isRaw_)\n> > \t\tformat.code = MEDIA_BUS_FMT_YUYV8_2X8;\n> >\n> > +\t/*\n> > +\t * On devices without DUAL_CROP (like the imx8mp) cropping needs to be\n> > +\t * done on the ISP/IS output.\n> > +\t */\n> > +\tif (media_->hwRevision() == RKISP1_V_IMX8MP) {\n> > +\t\t/* imx8mp has only a single path. */\n> > +\t\tconst auto &cfg = config->at(0);\n> > +\t\tSize ispCrop = format.size.boundedToAspectRatio(cfg.size)\n> > +\t\t\t\t\t .alignedUpTo(2, 2);\n> > +\t\trect = ispCrop.centeredTo(Rectangle(format.size).center());\n> > +\t\tif (ispCrop != format.size)\n> > +\t\t\tLOG(RkISP1, Info) << \"ISP output needs to be cropped to \"\n> \n> very minor nit:\n> \n> What do you think about dropping \"needs to be\" from the Info message.\n> \n> Also, this is a fairly typical thing, cropping to the output aspect\n> ratio before scaling, is it worth a Info message ?\n\nYes you are right. The info is available in the debug logs. I dropped\nthe message for v3.\n\nCheers,\nStefan\n\n> \n> Thanks\n> j\n> \n> > +\t\t\t\t\t << rect;\n> > +\t\tformat.size = ispCrop;\n> > +\t}\n> > +\n> > \tLOG(RkISP1, Debug)\n> > \t\t<< \"Configuring ISP output pad with \" << format\n> > \t\t<< \" crop \" << rect;\n> > diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> > index 236d05af7a2f..eee5b09e2ff0 100644\n> > --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> > +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> > @@ -417,11 +417,14 @@ int RkISP1Path::configure(const StreamConfiguration &config,\n> > \t/*\n> > \t * Crop on the resizer input to maintain FOV before downscaling.\n> > \t *\n> > -\t * \\todo The alignment to a multiple of 2 pixels is required but may\n> > -\t * change the aspect ratio very slightly. A more advanced algorithm to\n> > -\t * compute the resizer input crop rectangle is needed, and it should\n> > -\t * also take into account the need to crop away the edge pixels affected\n> > -\t * by the ISP processing blocks.\n> > +\t * Note that this does not apply to devices without DUAL_CROP support\n> > +\t * (like imx8mp) , where the cropping needs to be done on the\n> > +\t * ImageStabilizer block on the ISP source pad and therefore is\n> > +\t * configured before this stage. For simplicity we still set the crop.\n> > +\t * This gets ignored by the kernel driver because the hardware is\n> > +\t * missing the capability.\n> > +\t *\n> > +\t * Alignment to a multiple of 2 pixels is required by the resizer.\n> > \t */\n> > \tSize ispCrop = inputFormat.size.boundedToAspectRatio(config.size)\n> > \t\t\t\t .alignedUpTo(2, 2);\n> > --\n> > 2.43.0\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id EF141BDE6B\n\tfor ;\n\tThu, 28 Nov 2024 12:55:53 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 26A7165F67;\n\tThu, 28 Nov 2024 13:55:53 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 15F3F65898\n\tfor ;\n\tThu, 28 Nov 2024 13:55:52 +0100 (CET)","from ideasonboard.com (unknown\n\t[IPv6:2a00:6020:448c:6c00:22e0:a94:b035:820])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 4DA9F59D;\n\tThu, 28 Nov 2024 13:55:28 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"oiLxSWcJ\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732798528;\n\tbh=4Od4lK//mY/kTBxE072DYj5eUyyn1Q1ug3HeHFc4c6k=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=oiLxSWcJr1x4qhXQzg4hRtL/+JpH17LcvTaf1diKyJkPJZvwyD0XTwUv/jF1hC1aL\n\tPsuzfLUkA6J1gS24vp2ACyooVQHwP5fBqyIwcfncc/kTf3/ldGeXhDdhp1d9SQ3aih\n\tURujKRZ2ZtpfWmg2VBqh7nB40EX38bMUDdcu8n14=","Date":"Thu, 28 Nov 2024 13:55:49 +0100","From":"Stefan Klug ","To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org, \n\tPaul Elder ","Subject":"Re: [PATCH v2 2/8] libcamera: rkisp1: Keep aspect ratio on imx8mp","Message-ID":"","References":"<20241125151430.2437285-1-stefan.klug@ideasonboard.com>\n\t<20241125151430.2437285-3-stefan.klug@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]