{"id":13157,"url":"https://patchwork.libcamera.org/api/patches/13157/?format=json","web_url":"https://patchwork.libcamera.org/patch/13157/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210730082832.152626-4-umang.jain@ideasonboard.com>","date":"2021-07-30T08:28:28","name":"[libcamera-devel,RFC,3/3] ipu3: cio2: Customize sensor format selection logic","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"64a14e93624ffd84a7940010b50028565f9e3a10","submitter":{"id":86,"url":"https://patchwork.libcamera.org/api/people/86/?format=json","name":"Umang Jain","email":"umang.jain@ideasonboard.com"},"delegate":{"id":12,"url":"https://patchwork.libcamera.org/api/users/12/?format=json","username":"uajain","first_name":"Umang","last_name":"Jain","email":"umang.jain@ideasonboard.com"},"mbox":"https://patchwork.libcamera.org/patch/13157/mbox/","series":[{"id":2290,"url":"https://patchwork.libcamera.org/api/series/2290/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2290","date":"2021-07-30T08:28:25","name":"IPU3: Tweak sensor format selection","version":1,"mbox":"https://patchwork.libcamera.org/series/2290/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/13157/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/13157/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id AF0B7C322E\n\tfor ;\n\tFri, 30 Jul 2021 08:28:55 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 61966687C6;\n\tFri, 30 Jul 2021 10:28:55 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8BB74687BD\n\tfor ;\n\tFri, 30 Jul 2021 10:28:53 +0200 (CEST)","from perceval.ideasonboard.com (unknown [103.251.226.16])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 79C5789B;\n\tFri, 30 Jul 2021 10:28:52 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"jd9qs1QS\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1627633733;\n\tbh=6Pk6VqmTnnDdgsN5+EP2HhmtIsGtTJY1s05p3DgZQAY=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=jd9qs1QS5RTry7xMEN+yZOZBNKCIb7CN7rgtoIVeZTO0te1+d3bWXod39fmHjPJ36\n\t9q3Pb8jzr+NrQc7KcYeXuIobiORVIn/IdLaVyGhEuDpyU4yON2kcD1+7vpJo6irRIX\n\tcXvDBHJkj9SMi43ygu/F2yiL9kzB9q3mUmo2PYAc=","From":"Umang Jain ","To":"libcamera-devel@lists.libcamera.org","Date":"Fri, 30 Jul 2021 13:58:28 +0530","Message-Id":"<20210730082832.152626-4-umang.jain@ideasonboard.com>","X-Mailer":"git-send-email 2.31.1","In-Reply-To":"<20210730082832.152626-1-umang.jain@ideasonboard.com>","References":"<20210730082832.152626-1-umang.jain@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [RFC PATCH 3/3] ipu3: cio2: Customize sensor\n\tformat selection logic","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Adapt the sensor format selection logic such that it address\nplatform constraints on Soraka and Nautilus. This is best-effort\nbasis hence, new platforms can bring new constraints in future\nand we will need to adapt accordingly.\n\nCurrently we prioritise sensor format resolutions which closest to the\nFoV of desired output. However, Intel seems to prioritize the selection\nbased on the closest FoV with respect to sensor's maximum resolution.\n\nFor e.g. for a desired output of 1080p, Soraka will select 2112x1568\nsince it's a better FoV match to sensor's maximum resolution\n(4224x3136). It will not match the provided resolution of 2112x1188,\neven if that matches exactly to the desired ratio of 1080p.\n\nOn the other hand, on Nautilus, currently the sensor's maximum\nresolution(4208x3118) is the only selection for any 16:9 desired\nformats, whether it is 640x360 or 1080p or 3840x2160. That means\nthe sensor will run at full bandwidth even for ridiculously smaller\nresolutions so, we probably don't want that either.\n\nThis patch how the sensor format resolution is selected:\n- It prioritizes FoV with respect to sensor's maximum resolution size\n- It shall never return sensor's maximum resolution provided if there\n are other lower resolutions available.\n\n/* DNI: Preliminary testing of getSensorFormat():\n * cam -c1 -swidth=640,height=360,role=raw\n * cam -c1 -swidth=1280,height=720,role=raw\n * cam -c1 -swidth=1920,height=1080,role=raw\n * cam -c1 -swidth=3840,height=2160,role=raw\n */\n\nSigned-off-by: Umang Jain \n---\nOutput of preliminary testing:\n ($) cam -c1 -swidth=640,height=360,role=raw\n INFO IPU3 cio2.cpp:326 Desired Size: 640x360, Found SGRBG10_IPU3 with Size: 2104x1560\n\n ($) cam -c1 -swidth=1280,height=720,role=raw\n INFO IPU3 cio2.cpp:326 Desired Size: 1280x720, Found SGRBG10_IPU3 with Size: 2104x1560\n\n ($) cam -c1 -swidth=1920,height=1080,role=raw\n INFO IPU3 cio2.cpp:326 Desired Size: 1920x1080, Found SGRBG10_IPU3 with Size: 2104x1560\n\n ($) cam -c1 -swidth=3840,height=2160,role=raw\n INFO IPU3 cio2.cpp:326 Desired Size: 3840x2160, Found SGRBG10_IPU3 with Size: 4208x3118\n---\n src/libcamera/pipeline/ipu3/cio2.cpp | 62 +++++++++++++++++-----------\n 1 file changed, 38 insertions(+), 24 deletions(-)","diff":"diff --git a/src/libcamera/pipeline/ipu3/cio2.cpp b/src/libcamera/pipeline/ipu3/cio2.cpp\nindex aef88afd..4fd57ef7 100644\n--- a/src/libcamera/pipeline/ipu3/cio2.cpp\n+++ b/src/libcamera/pipeline/ipu3/cio2.cpp\n@@ -245,20 +245,16 @@ StreamConfiguration CIO2Device::generateConfiguration(Size size) const\n *\n * - The desired \\a size shall fit in the sensor output size to avoid the need\n * to up-scale.\n- * - The sensor output size shall match the desired aspect ratio to avoid the\n- * need to crop the field of view.\n- * - The sensor output size shall be as small as possible to lower the required\n- * bandwidth.\n+ * - The sensor output size will be set to maximum resolution only when there\n+ * are no other available lower sensor resolutions from any of \\a mbusCodes.\n+ * - The sensor output size shall have the closest FoV with respect\n+ * to the sensor's maximum resolution.\n * - The desired \\a size shall be supported by one of the media bus code listed\n * in \\a mbusCodes.\n *\n * When multiple media bus codes can produce the same size, the code at the\n * lowest position in \\a mbusCodes is selected.\n *\n- * The use of this method is optional, as the above criteria may not match the\n- * needs of all pipeline handlers. Pipeline handlers may implement custom\n- * sensor format selection when needed.\n- *\n * The returned sensor output format is guaranteed to be acceptable by the\n * setFormat() method without any modification.\n *\n@@ -268,14 +264,15 @@ StreamConfiguration CIO2Device::generateConfiguration(Size size) const\n V4L2SubdeviceFormat CIO2Device::getSensorFormat(const std::vector &mbusCodes,\n \t\t\t\t\t\tconst Size &size) const\n {\n-\tunsigned int desiredArea = size.width * size.height;\n-\tunsigned int bestArea = UINT_MAX;\n-\tfloat desiredRatio = static_cast(size.width) / size.height;\n-\tfloat bestRatio = FLT_MAX;\n-\tconst Size *bestSize = nullptr;\n+\tstd::map selectedSizes;\n+\tSize bestSize;\n+\tfloat maxResRatio = FLT_MAX;\n+\tfloat bestRatioDiff = -1.0;\n \tuint32_t bestCode = 0;\n \n \tfor (unsigned int code : mbusCodes) {\n+\t\tSize maxSensorResolution;\n+\n \t\tconst auto formats = formats_.find(code);\n \t\tif (formats == formats_.end())\n \t\t\tcontinue;\n@@ -286,33 +283,50 @@ V4L2SubdeviceFormat CIO2Device::getSensorFormat(const std::vector\n \t\t\tif (sz.width < size.width || sz.height < size.height)\n \t\t\t\tcontinue;\n \n+\t\t\tif (sz > maxSensorResolution)\n+\t\t\t\tmaxSensorResolution = sz;\n+\n \t\t\tfloat ratio = static_cast(sz.width) / sz.height;\n-\t\t\tfloat ratioDiff = fabsf(ratio - desiredRatio);\n-\t\t\tunsigned int area = sz.width * sz.height;\n-\t\t\tunsigned int areaDiff = area - desiredArea;\n+\t\t\tselectedSizes.emplace(sz, ratio);\n+\t\t}\n \n-\t\t\tif (ratioDiff > bestRatio)\n-\t\t\t\tcontinue;\n+\t\tif (!selectedSizes.empty()) {\n+\t\t\tmaxResRatio = selectedSizes[maxSensorResolution];\n \n-\t\t\tif (ratioDiff < bestRatio || areaDiff < bestArea) {\n-\t\t\t\tbestRatio = ratioDiff;\n-\t\t\t\tbestArea = areaDiff;\n-\t\t\t\tbestSize = &sz;\n+\t\t\tselectedSizes.erase(maxSensorResolution);\n+\t\t\tif (selectedSizes.empty()) {\n \t\t\t\tbestCode = code;\n+\t\t\t\tbestSize = maxSensorResolution;\n+\t\t\t} else { /* Find the best FoV to sensor's max resolution */\n+\t\t\t\tfor (auto iter : selectedSizes) {\n+\t\t\t\t\tfloat ratioDiff = fabsf(maxResRatio - iter.second);\n+\n+\t\t\t\t\tif (bestRatioDiff < 0 || (ratioDiff < bestRatioDiff)) {\n+\t\t\t\t\t\tbestRatioDiff = ratioDiff;\n+\t\t\t\t\t\tbestCode = code;\n+\t\t\t\t\t\tbestSize = iter.first;\n+\t\t\t\t\t}\n+\t\t\t\t}\n \t\t\t}\n \t\t}\n+\t\tselectedSizes.clear();\n \t}\n \n-\tif (!bestSize) {\n+\tif (bestSize.isNull()) {\n \t\tLOG(IPU3, Debug) << \"No supported format or size found\";\n \t\treturn {};\n \t}\n \n \tV4L2SubdeviceFormat format{\n \t\t.mbus_code = bestCode,\n-\t\t.size = *bestSize,\n+\t\t.size = bestSize,\n \t};\n \n+\tLOG(IPU3, Info)\n+\t\t<< \"Desired Size: \" << size.toString()\n+\t\t<< \", Found \" << mbusCodesToPixelFormat.at(format.mbus_code).toString()\n+\t\t<< \" with Size: \" << format.size.toString();\n+\n \treturn format;\n }\n \n","prefixes":["libcamera-devel","RFC","3/3"]}