{"id":11406,"url":"https://patchwork.libcamera.org/api/covers/11406/?format=json","web_url":"https://patchwork.libcamera.org/cover/11406/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210227180126.37591-1-sebastian.fricke@posteo.net>","date":"2021-02-27T18:01:24","name":"[libcamera-devel,v5,0/2] Fix a format mismatch within the RkISP1 pipeline","submitter":{"id":78,"url":"https://patchwork.libcamera.org/api/people/78/?format=json","name":"Sebastian Fricke","email":"sebastian.fricke@posteo.net"},"mbox":"https://patchwork.libcamera.org/cover/11406/mbox/","series":[{"id":1732,"url":"https://patchwork.libcamera.org/api/series/1732/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1732","date":"2021-02-27T18:01:24","name":"Fix a format mismatch within the RkISP1 pipeline","version":5,"mbox":"https://patchwork.libcamera.org/series/1732/mbox/"}],"comments":"https://patchwork.libcamera.org/api/covers/11406/comments/","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D6E81BD1F1\n\tfor <parsemail@patchwork.libcamera.org>;\n\tSat, 27 Feb 2021 18:01:33 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 21D8B68A62;\n\tSat, 27 Feb 2021 19:01:33 +0100 (CET)","from mout02.posteo.de (mout02.posteo.de [185.67.36.66])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id F1BC7689DD\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tSat, 27 Feb 2021 19:01:30 +0100 (CET)","from submission (posteo.de [89.146.220.130]) \n\tby mout02.posteo.de (Postfix) with ESMTPS id 4A24B2400FB\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tSat, 27 Feb 2021 19:01:30 +0100 (CET)","from customer (localhost [127.0.0.1])\n\tby submission (posteo.de) with ESMTPSA id 4DnvV14H0Mz9rxB;\n\tSat, 27 Feb 2021 19:01:29 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=posteo.net header.i=@posteo.net\n\theader.b=\"AkIFuKuz\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;\n\tt=1614448890; bh=aUiFPujpvyxM6Ctpfw8vv2A6idn15Gzg8h4Brk9COWc=;\n\th=From:To:Cc:Subject:Date:From;\n\tb=AkIFuKuzSQq4X3J3yx2ozjMfCJBFqlY/M16jXnwxTtVjN+HD4AbCcHrYwIxjmAP8n\n\tW+89awE/ujlROi+wmhQFVMhEXVlwlPeop+hvk7cNl9NCT3HQF5qoVUqsFZ9sRXefJG\n\tygy+SVP+qERxtwJNBg2B7txT3tYmu4mpc78kzKi8myCYkMQ/5CtVdH/fPx1XL5y7yc\n\tTJ2hTGdRjK8u/4XjYOSEZYyh4hMdauh23/oZLU3DnXKvhhvguAZrLWZA3UuGHY+JPi\n\tiXd0U/3WX6/7kJQAMpCPmoI6rBySskWN8XqZ97P5sKdgo3ijoZ36ZJ8o1wajJy0TPJ\n\t2z31MAwtlUaoA==","From":"Sebastian Fricke <sebastian.fricke@posteo.net>","To":"libcamera-devel@lists.libcamera.org","Date":"Sat, 27 Feb 2021 19:01:24 +0100","Message-Id":"<20210227180126.37591-1-sebastian.fricke@posteo.net>","X-Mailer":"git-send-email 2.25.1","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v5 0/2] Fix a format mismatch within the\n\tRkISP1 pipeline","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"},"content":"This patch-set fixes a format mismatch, that occurs when a sensor format\nis used that is greater than the maximum resolution for the Image\nSignal Processor. Within the validate method, which is called during the\nconfiguration generation, we request the sensor format.\n\nWe have to check for one special case, that can lead to an oversized sensor\nresolution. The getFormat method takes the smallest sensor resolution,\nwhere the width and height are greater or equal to the input size parameter.\n\nExample for a problematic case:\nSensor resolutions:\n- 2112x1568\n- 4224x3136\nStream configuration size:\n- 1920x1920\n\nIn this case, the requested resolution for the stream is smaller than the\nISP maximum, but the only viable alternative has a height that is\nsmaller than the requested height, so getFormat will pick the oversized\nresolution.\n\nWe resolve this issue by making sure, that the stream configuration\nresolution is smaller than the maximum viable sensor resolution before\nrequesting the best-fit sensor format.\n\nAdditionally, the ISP subdevice has to be shared with the CameraData\nclass, so that we can access that device within a method of the\nRkISP1CameraConfiguration class.\n\nv1: https://lists.libcamera.org/pipermail/libcamera-devel/2020-November/015113.html\nv2: https://lists.libcamera.org/pipermail/libcamera-devel/2020-November/015136.html\nv3: https://patchwork.libcamera.org/patch/10660/\nv4: https://patchwork.libcamera.org/patch/11180/\n\nChangelog:\n\nChanges since v4:\n* Completely rework the design of the fix, adjust the sensor resolution\n  during the validate() method, in order to have a correct sensor format\n  within configure()\n* Share the ISP subdevice between the PipelineHandler and the CameraData\n  class, so that the CameraConfiguration class has access to this device\n\nChanges since v3:\n\n* Introduce a helper method for finding the correct sensor format:\n  `findBestFitFormat`\n* Besides checking for a valid format resolution also check for a\n  matching media bus code\n* Return -EINVAL on an error instead of -1\n* Add documentation for the `findBestFitFormat` method\n\nChanges since v2:\n\n* Replace the act of attempting to set the ISP format before negotiating\n  the actual format for both the ISP input pad and the sensor in order\n  to get the maximum frame size. With a logic that involves enumerating\n  the maximum size directly from the subdevice and using that size for\n  the negotiation process.\n* Improve the log messages\n\nChanges since v1:\n\n* Change snake_case variable names to camelCase\n* Use the request comment style\n* Correct the scope of the newly implemented variables\n* Correct the subject of the debug log for the ISP format configuration\n* Update the comment above the ISP format configuration\n* Check if the original format is not equal to the configured ISP format\n  instead of checking if it is greater, this denies a false positive\n  where the height exceeds the maximum while the width is smaller.\n  If the configured format does not exceed the maximum resolution of the\n  ISP, it will stay untouched so the inequality always means that we\n  have to reconfigure the format.\n* Adjust the comparison of the ISP format size with the available sensor\n  formats, to detect a false-positive were the width is smaller while\n  the height exceeds the maximum\n* Use the standard function `max`\n\n-----\n\nThe following tests were all able to create a working camera pipeline:\n1. Without stream configuration\n2. With a normal stream configuration\n3. With a stream configuration that exceeds the maximum of the ISP\n4. With a very small resolution stream configuration\n5. With a configuration that is closer to the upper than to the lower\nresolution\n\n-----\n\n1. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3\n\n[9:26:53.975708264] [7581] DEBUG Camera camera.cpp:831 streams configuration: (0) 1920x1920-NV12\n[9:26:53.976907882] [7581]  INFO Camera camera.cpp:890 configuring streams: (0) 1920x1920-NV12\n...\n[9:26:53.978132875] [7582] DEBUG RkISP1 rkisp1.cpp:603 Configuring sensor with 2112x1568-SBGGR10_1X10\n[9:26:53.978232332] [7582] DEBUG RkISP1 rkisp1.cpp:609 Sensor configured with 2112x1568-SBGGR10_1X10\n[9:26:53.978358332] [7582] DEBUG RkISP1 rkisp1.cpp:620 ISP input pad configured with 2112x1568-SBGGR10_1X10 crop (0x0)/2112x1568\n[9:26:53.978441748] [7582] DEBUG RkISP1 rkisp1.cpp:626 Configuring ISP output pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:26:53.978548789] [7582] DEBUG RkISP1 rkisp1.cpp:638 ISP output pad configured with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:26:53.978680621] [7582] DEBUG RkISP1 rkisp1_path.cpp:119 Configured main resizer input pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:26:53.978763454] [7582] DEBUG RkISP1 rkisp1_path.cpp:125 Configuring main resizer output pad with 1920x1920-YUYV8_2X8\n[9:26:53.978840454] [7582] DEBUG RkISP1 rkisp1_path.cpp:143 Configured main resizer output pad with 1920x1920-YUYV8_1_5X8\n\n2. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=900,height=600,pixelformat=NV12,role=video\n\n[9:27:40.379117478] [7586] DEBUG Camera camera.cpp:831 streams configuration: (0) 1920x1920-NV12\n[9:27:40.380064806] [7586]  INFO Camera camera.cpp:890 configuring streams: (0) 900x600-NV12\n...\n[9:27:40.381540047] [7587] DEBUG RkISP1 rkisp1.cpp:603 Configuring sensor with 2112x1568-SBGGR10_1X10\n[9:27:40.381681796] [7587] DEBUG RkISP1 rkisp1.cpp:609 Sensor configured with 2112x1568-SBGGR10_1X10\n[9:27:40.381864670] [7587] DEBUG RkISP1 rkisp1.cpp:620 ISP input pad configured with 2112x1568-SBGGR10_1X10 crop (0x0)/2112x1568\n[9:27:40.381985419] [7587] DEBUG RkISP1 rkisp1.cpp:626 Configuring ISP output pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:27:40.382142335] [7587] DEBUG RkISP1 rkisp1.cpp:638 ISP output pad configured with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:27:40.382334251] [7587] DEBUG RkISP1 rkisp1_path.cpp:119 Configured main resizer input pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:27:40.382452667] [7587] DEBUG RkISP1 rkisp1_path.cpp:125 Configuring main resizer output pad with 900x600-YUYV8_2X8\n[9:27:40.382564666] [7587] DEBUG RkISP1 rkisp1_path.cpp:143 Configured main resizer output pad with 900x600-YUYV8_1_5X8\n\n3. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=4500,height=3500,pixelformat=NV12,role=video\n\n[9:28:21.358328098] [7589] DEBUG Camera camera.cpp:831 streams configuration: (0) 1920x1920-NV12\n[9:28:21.358584180] [7589] DEBUG RkISP1 rkisp1_path.cpp:94 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12\n[9:28:21.358701137] [7589] DEBUG RkISP1 rkisp1_path.cpp:94 Adjusting format from 4500x3500-NV12 to 1920x1920-NV12\n[9:28:21.358801470] [7589] DEBUG RkISP1 rkisp1_path.cpp:94 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12\nCamera configuration adjusted\n[9:28:21.359524799] [7589]  INFO Camera camera.cpp:890 configuring streams: (0) 4416x3312-NV12\n...\n[9:28:21.360212587] [7590] DEBUG RkISP1 rkisp1.cpp:603 Configuring sensor with 2112x1568-SBGGR10_1X10\n[9:28:21.360255753] [7590] DEBUG RkISP1 rkisp1.cpp:609 Sensor configured with 2112x1568-SBGGR10_1X10\n[9:28:21.360309419] [7590] DEBUG RkISP1 rkisp1.cpp:620 ISP input pad configured with 2112x1568-SBGGR10_1X10 crop (0x0)/2112x1568\n[9:28:21.360340336] [7590] DEBUG RkISP1 rkisp1.cpp:626 Configuring ISP output pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:28:21.360378252] [7590] DEBUG RkISP1 rkisp1.cpp:638 ISP output pad configured with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:28:21.360430169] [7590] DEBUG RkISP1 rkisp1_path.cpp:119 Configured main resizer input pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:28:21.360459918] [7590] DEBUG RkISP1 rkisp1_path.cpp:125 Configuring main resizer output pad with 4416x3312-YUYV8_2X8\n[9:28:21.360487918] [7590] DEBUG RkISP1 rkisp1_path.cpp:143 Configured main resizer output pad with 4416x3312-YUYV8_1_5X8\n\n4. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=40,height=30,pixelformat=NV12,role=video\n\n[9:29:21.653077053] [7592] DEBUG Camera camera.cpp:831 streams configuration: (0) 1920x1920-NV12\n[9:29:21.653987340] [7592]  INFO Camera camera.cpp:890 configuring streams: (0) 40x30-NV12\n...\n[9:29:21.655460873] [7593] DEBUG RkISP1 rkisp1.cpp:603 Configuring sensor with 2112x1568-SBGGR10_1X10\n[9:29:21.655601455] [7593] DEBUG RkISP1 rkisp1.cpp:609 Sensor configured with 2112x1568-SBGGR10_1X10\n[9:29:21.655780829] [7593] DEBUG RkISP1 rkisp1.cpp:620 ISP input pad configured with 2112x1568-SBGGR10_1X10 crop (0x0)/2112x1568\n[9:29:21.655901578] [7593] DEBUG RkISP1 rkisp1.cpp:626 Configuring ISP output pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:29:21.656136077] [7593] DEBUG RkISP1 rkisp1.cpp:638 ISP output pad configured with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:29:21.656356576] [7593] DEBUG RkISP1 rkisp1_path.cpp:119 Configured main resizer input pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:29:21.656487533] [7593] DEBUG RkISP1 rkisp1_path.cpp:125 Configuring main resizer output pad with 40x30-YUYV8_2X8\n[9:29:21.656608574] [7593] DEBUG RkISP1 rkisp1_path.cpp:143 Configured main resizer output pad with 40x30-YUYV8_1_5X8\n\n5. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=3450,height=2456,pixelformat=NV12,role=video\n\n[9:30:08.027510441] [7595] DEBUG Camera camera.cpp:831 streams configuration: (0) 1920x1920-NV12\n[9:30:08.028457768] [7595]  INFO Camera camera.cpp:890 configuring streams: (0) 3450x2456-NV12\n...\n[9:30:08.029969509] [7596] DEBUG RkISP1 rkisp1.cpp:603 Configuring sensor with 2112x1568-SBGGR10_1X10\n[9:30:08.030114758] [7596] DEBUG RkISP1 rkisp1.cpp:609 Sensor configured with 2112x1568-SBGGR10_1X10\n[9:30:08.030296174] [7596] DEBUG RkISP1 rkisp1.cpp:620 ISP input pad configured with 2112x1568-SBGGR10_1X10 crop (0x0)/2112x1568\n[9:30:08.030422465] [7596] DEBUG RkISP1 rkisp1.cpp:626 Configuring ISP output pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:30:08.030582881] [7596] DEBUG RkISP1 rkisp1.cpp:638 ISP output pad configured with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:30:08.030776546] [7596] DEBUG RkISP1 rkisp1_path.cpp:119 Configured main resizer input pad with 2112x1568-YUYV8_2X8 crop (0x0)/2112x1568\n[9:30:08.030899629] [7596] DEBUG RkISP1 rkisp1_path.cpp:125 Configuring main resizer output pad with 3450x2456-YUYV8_2X8\n[9:30:08.031016586] [7596] DEBUG RkISP1 rkisp1_path.cpp:143 Configured main resizer output pad with 3450x2456-YUYV8_1_5X8\n\nSebastian Fricke (2):\n  pipeline: rkisp1: Share the ISP subdevice\n  pipeline: rkisp1: Fix sensor ISP format mismatch\n\n src/libcamera/pipeline/rkisp1/rkisp1.cpp | 40 +++++++++++++++++++++---\n 1 file changed, 35 insertions(+), 5 deletions(-)"}