{"id":10455,"url":"https://patchwork.libcamera.org/api/covers/10455/?format=json","web_url":"https://patchwork.libcamera.org/cover/10455/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20201120124503.22718-1-sebastian.fricke.linux@gmail.com>","date":"2020-11-20T12:45:02","name":"[libcamera-devel,v2,0/1] pipeline-rkisp1-Fix-sensor-ISP-format-mismatch","submitter":{"id":73,"url":"https://patchwork.libcamera.org/api/people/73/?format=json","name":"Sebastian Fricke","email":"sebastian.fricke.linux@gmail.com"},"mbox":"https://patchwork.libcamera.org/cover/10455/mbox/","series":[{"id":1472,"url":"https://patchwork.libcamera.org/api/series/1472/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1472","date":"2020-11-20T12:45:02","name":"pipeline-rkisp1-Fix-sensor-ISP-format-mismatch","version":2,"mbox":"https://patchwork.libcamera.org/series/1472/mbox/"}],"comments":"https://patchwork.libcamera.org/api/covers/10455/comments/","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id ACD4EBE08A\n\tfor ;\n\tFri, 20 Nov 2020 12:45:31 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 41448615B5;\n\tFri, 20 Nov 2020 13:45:31 +0100 (CET)","from mail-ej1-x629.google.com (mail-ej1-x629.google.com\n\t[IPv6:2a00:1450:4864:20::629])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 825F461564\n\tfor ;\n\tFri, 20 Nov 2020 13:45:30 +0100 (CET)","by mail-ej1-x629.google.com with SMTP id o21so12680125ejb.3\n\tfor ;\n\tFri, 20 Nov 2020 04:45:30 -0800 (PST)","from basti.fritz.box\n\t(p200300d1ff2882007ff0233db4c9f029.dip0.t-ipconnect.de.\n\t[2003:d1:ff28:8200:7ff0:233d:b4c9:f029])\n\tby smtp.gmail.com with ESMTPSA id\n\t22sm1093492ejw.27.2020.11.20.04.45.28\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 20 Nov 2020 04:45:28 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"V29t/kod\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=from:to:cc:subject:date:message-id:mime-version\n\t:content-transfer-encoding;\n\tbh=I6fEgSQM6JyuKxcHbgYyy+0Diosh5nB1OS9tTgey2xw=;\n\tb=V29t/kodOeIgFOJhgf4Fm2X+uW/2XwTiXKxhTJ2qXPLd14NjtcaUJSLEgImJqIoQxd\n\tgkhFPO/Utf04aaEWyCN09/ZpQ2cIgQlYb99V+Z5fD3J07iVkF2Y80S5WSr0OcAS5bjye\n\t9gfWky6bvUM+OS7ymr17Z/QNkonJYTTTuzlTWV90ilLT2MLY8jdeCVBxhxJWz5ZFsh6D\n\tq9HwFhLckVoP/wWGawPShWVM69oKW6MVS9y1+EvFIeX+pIGuxqPooV8i/Z1uorEzoV6q\n\t8B9XZvSGXhRxMxDq1xbwYaVCb9x6LEkMGprliu03B8EgFE4qDglKObfcEklwsOnYRVPx\n\t21DA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version\n\t:content-transfer-encoding;\n\tbh=I6fEgSQM6JyuKxcHbgYyy+0Diosh5nB1OS9tTgey2xw=;\n\tb=mvagxmplTIhEB9Zl67axn44UD3iyNnqT0ctMdNGu8XaHnB7dh1qlI1SzViSQBcSou9\n\tZISMK+HQiLhwcE4I8R7jI4Ps5jeH+xPUi0v+h10WJwD5u+WDEqcb10252/p0ogbbLXDH\n\t6Cz4QVwgrw3LJEszcHK+vUilb6pO9x2sTFsAYGHW9FWiQDy3955eFvFSVFUIrbo3pS7t\n\tT6tLc+ixoM2KmXhHNFvijVJvOHSHuCSDVCdDSdMv4LF9vF0EE/xGm1eexOvyafYeKe5k\n\tF2HXoC3hX75qoT6Tb/XSpq9jEIxLBjYR8iYg+CROs8/bHb36Eroln/WEUXmQDrtGnLOT\n\tf15Q==","X-Gm-Message-State":"AOAM531soGzs6k8B7fjSHRrJjzrkaP8aYWB+EzaP0W+YitJOYkiHfBn8\n\tc6JMuyiFlgi7EMQz2uqoKe5oB4UBCY4=","X-Google-Smtp-Source":"ABdhPJz4NeprrCI9Yt40NPN5ooYsiggXVeRl9u4YTjVfotris88h5v2E0SRRNZGp3mjZ1HGPT7YrGA==","X-Received":"by 2002:a17:907:206b:: with SMTP id\n\tqp11mr4895489ejb.286.1605876329522; \n\tFri, 20 Nov 2020 04:45:29 -0800 (PST)","From":"Sebastian Fricke ","To":"libcamera-devel@lists.libcamera.org","Date":"Fri, 20 Nov 2020 13:45:02 +0100","Message-Id":"<20201120124503.22718-1-sebastian.fricke.linux@gmail.com>","X-Mailer":"git-send-email 2.20.1","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v2 0/1]\n\tpipeline-rkisp1-Fix-sensor-ISP-format-mismatch","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"This patch fixes a mismatch of image formats during the pipeline\ncreation of the RkISP1. The mismatch happens because the current code\ndoes not check if the configured format exceeded the maximum viable\nresolution of the ISP.\nThis patch fixes the issue by first setting the ISP, checking if the\ngiven resolution was taken and searching for another viable format that\nis lower or equal to the ISP maximum.\n\n---\n\nChangelog:\n\nChanges since v1:\n\n* Change snake_case variable names to camelCase\n* Use the request comment style\n* Correct the scope of the newly implemented variables\n* Correct the subject of the debug log for the ISP format configuration\n* Update the comment above the ISP format configuration\n* Check if the original format is not equal to the configured ISP format\n instead of checking if it is greater, this denies a false positive\n where the height exceeds the maximum while the width is smaller.\n If the configured format does not exceed the maximum resolution of the\n ISP, it will stay untouched so the inequality always means that we\n have to reconfigure the format.\n* Adjust the comparison of the ISP format size with the available sensor\n formats, to detect a false-positive were the width is smaller while\n the height exceeds the maximum\n* Use the standard function `max`\n\n---\n\nThe following tests were all able to create a working camera pipeline:\n1. Without stream configuration\n2. With a normal stream configuration\n3. With a stream configuration that exceeds the maximum of the ISP\n4. With a very small resolution stream configuration\n5. With a configuration that is closer to the upper than to the lower\nresolution\n\n----\n\n1. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3\n\n[8:05:49.668381732] [9973] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12\n[8:05:49.669108269] [9973] INFO Camera camera.cpp:830 configuring streams: (0) 1920x1920-NV12\n...\n[8:05:49.670965005] [9974] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10\n[8:05:49.671126004] [9974] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10\n[8:05:49.671233045] [9974] INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format\n[8:05:49.671378294] [9974] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10\n[8:05:49.671510710] [9974] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10\n[8:05:49.671676667] [9974] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10\n[8:05:49.671787791] [9974] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8\n[8:05:49.671894540] [9974] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8\n[8:05:49.672107747] [9974] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8\n[8:05:49.672226163] [9974] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 1920x1920-YUYV8_2X8\n[8:05:49.672341662] [9974] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 1920x1920-YUYV8_1_5X8\n\n2. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=900,height=600,pixelformat=NV12,role=video\n\n[8:07:05.725799205] [9977] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12\n[8:07:05.726242535] [9977] INFO Camera camera.cpp:830 configuring streams: (0) 900x600-NV12\n...\n[8:07:05.727718025] [9978] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10\n[8:07:05.727873482] [9978] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10\n[8:07:05.728056356] [9978] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10\n[8:07:05.728233104] [9978] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10\n[8:07:05.728340437] [9978] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8\n[8:07:05.728448353] [9978] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8\n[8:07:05.728567352] [9978] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8\n[8:07:05.728673809] [9978] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 900x600-YUYV8_2X8\n[8:07:05.728785517] [9978] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 900x600-YUYV8_1_5X8\n\n3. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=4500,height=3500,pixelformat=NV12,role=video\n\n[8:08:02.372820979] [9981] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12\n[8:08:02.373280350] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12\n[8:08:02.373534098] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 1920x1920-NV12\n[8:08:02.373767722] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12\n[8:08:02.374010095] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found\nCamera configuration adjusted\n[8:08:02.374449634] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found\n[8:08:02.374591091] [9981] INFO Camera camera.cpp:830 configuring streams: (0) 4416x3312-NV12\n...\n[8:08:02.376301704] [9982] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-0x0000\n[8:08:02.376469994] [9982] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SRGGB10_1X10\n[8:08:02.376576160] [9982] INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format\n[8:08:02.376710909] [9982] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10\n[8:08:02.376839241] [9982] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10\n[8:08:02.377001115] [9982] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10\n[8:08:02.377104364] [9982] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8\n[8:08:02.377208780] [9982] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8\n[8:08:02.377324863] [9982] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8\n[8:08:02.377425779] [9982] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 4416x3312-YUYV8_2X8\n[8:08:02.377533695] [9982] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 4416x3312-YUYV8_1_5X8\n\n4. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=40,height=30,pixelformat=NV12,role=video\n\n[8:10:41.140561164] [9989] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12\n[8:10:41.141348367] [9989] INFO Camera camera.cpp:830 configuring streams: (0) 40x30-NV12\n...\n[8:10:41.143016688] [9990] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10\n[8:10:41.143161062] [9990] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10\n[8:10:41.143283561] [9990] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10\n[8:10:41.143442810] [9990] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10\n[8:10:41.143544893] [9990] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8\n[8:10:41.143644934] [9990] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8\n[8:10:41.143756058] [9990] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8\n[8:10:41.143854640] [9990] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 40x30-YUYV8_2X8\n[8:10:41.143960223] [9990] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 40x30-YUYV8_1_5X8\n\n5. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=3450,height=2456,pixelformat=NV12,role=video\n\n[8:11:40.321895811] [9992] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12\n[8:11:40.322697014] [9992] INFO Camera camera.cpp:830 configuring streams: (0) 3450x2456-NV12\n...\n[8:11:40.324431543] [9993] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10\n[8:11:40.324579125] [9993] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10\n[8:11:40.324680041] [9993] INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format\n[8:11:40.324815082] [9993] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10\n[8:11:40.324946039] [9993] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10\n[8:11:40.325111121] [9993] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10\n[8:11:40.325216121] [9993] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8\n[8:11:40.325323453] [9993] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8\n[8:11:40.325441286] [9993] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8\n[8:11:40.325547451] [9993] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 3450x2456-YUYV8_2X8\n[8:11:40.325660909] [9993] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 3450x2456-YUYV8_1_5X8\n\n-----\n\nSebastian Fricke (1):\n pipeline: rkisp1: Fix sensor-ISP format mismatch\n\n src/libcamera/pipeline/rkisp1/rkisp1.cpp | 46 ++++++++++++++++++++----\n 1 file changed, 40 insertions(+), 6 deletions(-)"}