[{"id":31458,"web_url":"https://patchwork.libcamera.org/comment/31458/","msgid":"","date":"2024-09-30T06:08:50","subject":"Re: [PATCH v4] pipeline: rkisp1: Filter out sensor sizes not\n\tsupported by the pipeline","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"On Mon, Sep 30, 2024 at 11:10:25AM GMT, Umang Jain wrote:\n> It is possible that the maximum sensor size (returned by\n> CameraSensor::resolution()) is not supported by the pipeline. In such\n> cases, a filter function is required to filter out resolutions of the\n> camera sensor, which cannot be supported by the pipeline.\n>\n> Introduce the filter function filterSensorResolution() in RkISP1Path\n> class and use it for validate() and generateConfiguration(). The\n> maximum sensor resolution is picked from that filtered set of\n> resolutions instead of CameraSensor::resolution().\n>\n> Signed-off-by: Umang Jain \n> Reviewed-by: Kieran Bingham \n> Tested-by: Kieran Bingham \n> ---\n> Changes in v4:\n> - Include Jacopo's suggestion(v3) for sorting in-place.\n>\n> Changes v3:\n> - None, just a resent over latest master\n> - Split out from [v2,0/2] pipeline: rkisp1: Filter out sensor sizes not supported by the pipeline\n> so that, this can make independent progress.\n> ---\n> src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 42 ++++++++++++++++++-\n> src/libcamera/pipeline/rkisp1/rkisp1_path.h | 8 ++++\n> 2 files changed, 48 insertions(+), 2 deletions(-)\n>\n> diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> index c49017d1..90c49d99 100644\n> --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> @@ -126,12 +126,50 @@ void RkISP1Path::populateFormats()\n> \t}\n> }\n>\n> +/**\n> + * \\brief Filter the sensor resolutions that can be supported\n> + * \\param[in] sensor The camera sensor\n> + *\n> + * This function retrieves all the sizes supported by the sensor and\n> + * filters all the resolutions that can be supported on the pipeline.\n> + * It is possible that the sensor's maximum output resolution is higher\n> + * than the ISP maximum input. In that case, this function filters out all\n> + * the resolution incapable of being supported and returns the maximum\n> + * sensor resolution that can be supported by the pipeline.\n> + *\n> + * \\return Maximum sensor size supported on the pipeline\n> + */\n> +Size RkISP1Path::filterSensorResolution(const CameraSensor *sensor)\n\nStill no sure about the function name and use cases for storing all\nresolutions.\n\nNot a big deal though, let's land this one\nReviewed-by: Jacopo Mondi \n\n> +{\n> +\tauto iter = sensorSizesMap_.find(sensor);\n> +\tif (iter != sensorSizesMap_.end())\n> +\t\treturn iter->second.back();\n> +\n> +\tstd::vector &sizes = sensorSizesMap_[sensor];\n> +\tfor (unsigned int code : sensor->mbusCodes()) {\n> +\t\tfor (const Size &size : sensor->sizes(code)) {\n> +\t\t\tif (size.width > maxResolution_.width ||\n> +\t\t\t size.height > maxResolution_.height)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tsizes.push_back(size);\n> +\t\t}\n> +\t}\n> +\n> +\t/* Sort in increasing order and remove duplicates. */\n> +\tstd::sort(sizes.begin(), sizes.end());\n> +\tauto last = std::unique(sizes.begin(), sizes.end());\n> +\tsizes.erase(last, sizes.end());\n> +\n> +\treturn sizes.back();\n> +}\n> +\n> StreamConfiguration\n> RkISP1Path::generateConfiguration(const CameraSensor *sensor, const Size &size,\n> \t\t\t\t StreamRole role)\n> {\n> \tconst std::vector &mbusCodes = sensor->mbusCodes();\n> -\tconst Size &resolution = sensor->resolution();\n> +\tSize resolution = filterSensorResolution(sensor);\n>\n> \t/* Min and max resolutions to populate the available stream formats. */\n> \tSize maxResolution = maxResolution_.boundedToAspectRatio(resolution)\n> @@ -220,7 +258,7 @@ CameraConfiguration::Status RkISP1Path::validate(const CameraSensor *sensor,\n> \t\t\t\t\t\t StreamConfiguration *cfg)\n> {\n> \tconst std::vector &mbusCodes = sensor->mbusCodes();\n> -\tconst Size &resolution = sensor->resolution();\n> +\tSize resolution = filterSensorResolution(sensor);\n>\n> \tconst StreamConfiguration reqCfg = *cfg;\n> \tCameraConfiguration::Status status = CameraConfiguration::Valid;\n> diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.h b/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n> index 08edefec..9f75fe1f 100644\n> --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n> +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n> @@ -7,6 +7,7 @@\n>\n> #pragma once\n>\n> +#include \n> #include \n> #include \n> #include \n> @@ -63,6 +64,7 @@ public:\n>\n> private:\n> \tvoid populateFormats();\n> +\tSize filterSensorResolution(const CameraSensor *sensor);\n>\n> \tstatic constexpr unsigned int RKISP1_BUFFER_COUNT = 4;\n>\n> @@ -77,6 +79,12 @@ private:\n> \tstd::unique_ptr resizer_;\n> \tstd::unique_ptr video_;\n> \tMediaLink *link_;\n> +\n> +\t/*\n> +\t * Map from camera sensors to the sizes (in increasing order),\n> +\t * which are guaranteed to be supported by the pipeline.\n> +\t */\n> +\tstd::map> sensorSizesMap_;\n> };\n>\n> class RkISP1MainPath : public RkISP1Path\n> --\n> 2.45.0\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5C4BBC3257\n\tfor ;\n\tMon, 30 Sep 2024 06:08:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2B01463512;\n\tMon, 30 Sep 2024 08:08:56 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id F320962C92\n\tfor ;\n\tMon, 30 Sep 2024 08:08:53 +0200 (CEST)","from ideasonboard.com (unknown [95.131.45.214])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DAB2E39F;\n\tMon, 30 Sep 2024 08:07:22 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"D8mQ1xCd\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1727676443;\n\tbh=bm6jREpKuav6KoFd3oTTFdg7M8XWSZwccA6rQdyJ2Cg=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=D8mQ1xCdKBQJxJUD+U5ZBSJ9UQh028V0WoGcOgeexzBJt8GuclEmOhoDR8fnVrG7E\n\toslTKHpCBcQ2S6uAX8tYEENufHlD9HkE+Zw15yEgXugeHkrUy39C2ZHxJX5L4Hwuex\n\txyUoXx4d4OIttxgUidI1/X7VMCDezJc90PitMqdc=","Date":"Mon, 30 Sep 2024 08:08:50 +0200","From":"Jacopo Mondi ","To":"Umang Jain ","Cc":"libcamera-devel@lists.libcamera.org, \n\tJacopo Mondi ,\n\tKieran Bingham ","Subject":"Re: [PATCH v4] pipeline: rkisp1: Filter out sensor sizes not\n\tsupported by the pipeline","Message-ID":"","References":"<20240930054025.5770-1-umang.jain@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20240930054025.5770-1-umang.jain@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":31462,"web_url":"https://patchwork.libcamera.org/comment/31462/","msgid":"<172768568230.532453.9810766782488483822@ping.linuxembedded.co.uk>","date":"2024-09-30T08:41:22","subject":"Re: [PATCH v4] pipeline: rkisp1: Filter out sensor sizes not\n\tsupported by the pipeline","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Quoting Jacopo Mondi (2024-09-30 07:08:50)\n> On Mon, Sep 30, 2024 at 11:10:25AM GMT, Umang Jain wrote:\n> > It is possible that the maximum sensor size (returned by\n> > CameraSensor::resolution()) is not supported by the pipeline. In such\n> > cases, a filter function is required to filter out resolutions of the\n> > camera sensor, which cannot be supported by the pipeline.\n> >\n> > Introduce the filter function filterSensorResolution() in RkISP1Path\n> > class and use it for validate() and generateConfiguration(). The\n> > maximum sensor resolution is picked from that filtered set of\n> > resolutions instead of CameraSensor::resolution().\n> >\n> > Signed-off-by: Umang Jain \n> > Reviewed-by: Kieran Bingham \n> > Tested-by: Kieran Bingham \n> > ---\n> > Changes in v4:\n> > - Include Jacopo's suggestion(v3) for sorting in-place.\n> >\n> > Changes v3:\n> > - None, just a resent over latest master\n> > - Split out from [v2,0/2] pipeline: rkisp1: Filter out sensor sizes not supported by the pipeline\n> > so that, this can make independent progress.\n> > ---\n> > src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 42 ++++++++++++++++++-\n> > src/libcamera/pipeline/rkisp1/rkisp1_path.h | 8 ++++\n> > 2 files changed, 48 insertions(+), 2 deletions(-)\n> >\n> > diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> > index c49017d1..90c49d99 100644\n> > --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> > +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n> > @@ -126,12 +126,50 @@ void RkISP1Path::populateFormats()\n> > }\n> > }\n> >\n> > +/**\n> > + * \\brief Filter the sensor resolutions that can be supported\n> > + * \\param[in] sensor The camera sensor\n> > + *\n> > + * This function retrieves all the sizes supported by the sensor and\n> > + * filters all the resolutions that can be supported on the pipeline.\n> > + * It is possible that the sensor's maximum output resolution is higher\n> > + * than the ISP maximum input. In that case, this function filters out all\n> > + * the resolution incapable of being supported and returns the maximum\n> > + * sensor resolution that can be supported by the pipeline.\n> > + *\n> > + * \\return Maximum sensor size supported on the pipeline\n> > + */\n> > +Size RkISP1Path::filterSensorResolution(const CameraSensor *sensor)\n> \n> Still no sure about the function name and use cases for storing all\n> resolutions.\n> \n> Not a big deal though, let's land this one\n> Reviewed-by: Jacopo Mondi \n\nI don't know where the current output of the raw stream sizes gets\nidentified from, but I assume this is the same list, and should be\ncommon somehow?\n\nUlitmately, we should have a way in the future to report these to the\napplication, along with a representation of what the field of view is\nagainst the sensor native pixel array size.\n\nSo I think this could get reworked when we handle that ... which I hope\nwill be somewhere on the horizon coming into view ;-)\n\n--\nKieran\n\n\n> \n> > +{\n> > + auto iter = sensorSizesMap_.find(sensor);\n> > + if (iter != sensorSizesMap_.end())\n> > + return iter->second.back();\n> > +\n> > + std::vector &sizes = sensorSizesMap_[sensor];\n> > + for (unsigned int code : sensor->mbusCodes()) {\n> > + for (const Size &size : sensor->sizes(code)) {\n> > + if (size.width > maxResolution_.width ||\n> > + size.height > maxResolution_.height)\n> > + continue;\n> > +\n> > + sizes.push_back(size);\n> > + }\n> > + }\n> > +\n> > + /* Sort in increasing order and remove duplicates. */\n> > + std::sort(sizes.begin(), sizes.end());\n> > + auto last = std::unique(sizes.begin(), sizes.end());\n> > + sizes.erase(last, sizes.end());\n> > +\n> > + return sizes.back();\n> > +}\n> > +\n> > StreamConfiguration\n> > RkISP1Path::generateConfiguration(const CameraSensor *sensor, const Size &size,\n> > StreamRole role)\n> > {\n> > const std::vector &mbusCodes = sensor->mbusCodes();\n> > - const Size &resolution = sensor->resolution();\n> > + Size resolution = filterSensorResolution(sensor);\n> >\n> > /* Min and max resolutions to populate the available stream formats. */\n> > Size maxResolution = maxResolution_.boundedToAspectRatio(resolution)\n> > @@ -220,7 +258,7 @@ CameraConfiguration::Status RkISP1Path::validate(const CameraSensor *sensor,\n> > StreamConfiguration *cfg)\n> > {\n> > const std::vector &mbusCodes = sensor->mbusCodes();\n> > - const Size &resolution = sensor->resolution();\n> > + Size resolution = filterSensorResolution(sensor);\n> >\n> > const StreamConfiguration reqCfg = *cfg;\n> > CameraConfiguration::Status status = CameraConfiguration::Valid;\n> > diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.h b/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n> > index 08edefec..9f75fe1f 100644\n> > --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n> > +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n> > @@ -7,6 +7,7 @@\n> >\n> > #pragma once\n> >\n> > +#include \n> > #include \n> > #include \n> > #include \n> > @@ -63,6 +64,7 @@ public:\n> >\n> > private:\n> > void populateFormats();\n> > + Size filterSensorResolution(const CameraSensor *sensor);\n> >\n> > static constexpr unsigned int RKISP1_BUFFER_COUNT = 4;\n> >\n> > @@ -77,6 +79,12 @@ private:\n> > std::unique_ptr resizer_;\n> > std::unique_ptr video_;\n> > MediaLink *link_;\n> > +\n> > + /*\n> > + * Map from camera sensors to the sizes (in increasing order),\n> > + * which are guaranteed to be supported by the pipeline.\n> > + */\n> > + std::map> sensorSizesMap_;\n> > };\n> >\n> > class RkISP1MainPath : public RkISP1Path\n> > --\n> > 2.45.0\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B55D4C0F1B\n\tfor ;\n\tMon, 30 Sep 2024 08:41:27 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 57BAD63516;\n\tMon, 30 Sep 2024 10:41:27 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4FE2463500\n\tfor ;\n\tMon, 30 Sep 2024 10:41:25 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(cpc89244-aztw30-2-0-cust6594.18-1.cable.virginm.net [86.31.185.195])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5A0EC39F;\n\tMon, 30 Sep 2024 10:39:54 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Pr6021cd\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1727685594;\n\tbh=QYJdF0BKkBGIgBPhaaodWzb40AYG3KVlkkjazUabdO0=;\n\th=In-Reply-To:References:Subject:From:Cc:To:Date:From;\n\tb=Pr6021cdegymVRSF4WXpR1dESjZRFfWNR5lyx05gpvbCejL8OE1BU8BGldyLmET8U\n\tKPw9WoMlaF/F6T3E0z1r36mE3hZFI3yc7AMPv712JmBhU/rK1MaaCBsyk6cdoGGIpf\n\t64lGtm24NXkYe6Ge8OpNpEA2roVIJ2LLNsXHSjik=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"","References":"<20240930054025.5770-1-umang.jain@ideasonboard.com>\n\t","Subject":"Re: [PATCH v4] pipeline: rkisp1: Filter out sensor sizes not\n\tsupported by the pipeline","From":"Kieran Bingham ","Cc":"libcamera-devel@lists.libcamera.org,\n\tJacopo Mondi ","To":"Jacopo Mondi ,\n\tUmang Jain ","Date":"Mon, 30 Sep 2024 09:41:22 +0100","Message-ID":"<172768568230.532453.9810766782488483822@ping.linuxembedded.co.uk>","User-Agent":"alot/0.10","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]