[libcamera-devel,0/1] pipeline-rkisp1-Fix-sensor-ISP-format-mismatch
mbox series

Message ID 20201119131331.30847-1-sebastian.fricke.linux@gmail.com
Headers show
Series
  • pipeline-rkisp1-Fix-sensor-ISP-format-mismatch
Related show

Message

Sebastian Fricke Nov. 19, 2020, 1:13 p.m. UTC
This patch fixes a mismatch of image formats during the pipeline
creation of the RkISP1. The mismatch happens because the current code
does not check if the configured format exceeded the maximum viable
resolution of the ISP.
This patch fixes the issue by first setting the ISP, checking if the
given resolution was taken and searching for another viable format that
is lower or equal to the ISP maximum.

Here are a couple of test runs:
1. Without stream configuration
2. With a normal stream configuration
3. With a stream configuration that exceeds the maximum of the ISP
4. With a very small resolution stream configuration
5. With a configuration that is closer to the upper than to the lower
resolution

----

1. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3

[8:05:49.668381732] [9973] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
[8:05:49.669108269] [9973]  INFO Camera camera.cpp:830 configuring streams: (0) 1920x1920-NV12
...
[8:05:49.670965005] [9974] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10
[8:05:49.671126004] [9974] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10
[8:05:49.671233045] [9974]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
[8:05:49.671378294] [9974] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
[8:05:49.671510710] [9974] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
[8:05:49.671676667] [9974] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
[8:05:49.671787791] [9974] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
[8:05:49.671894540] [9974] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
[8:05:49.672107747] [9974] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
[8:05:49.672226163] [9974] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 1920x1920-YUYV8_2X8
[8:05:49.672341662] [9974] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 1920x1920-YUYV8_1_5X8

2. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=900,height=600,pixelformat=NV12,role=video

[8:07:05.725799205] [9977] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
[8:07:05.726242535] [9977]  INFO Camera camera.cpp:830 configuring streams: (0) 900x600-NV12
...
[8:07:05.727718025] [9978] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10
[8:07:05.727873482] [9978] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10
[8:07:05.728056356] [9978] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
[8:07:05.728233104] [9978] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
[8:07:05.728340437] [9978] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
[8:07:05.728448353] [9978] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
[8:07:05.728567352] [9978] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
[8:07:05.728673809] [9978] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 900x600-YUYV8_2X8
[8:07:05.728785517] [9978] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 900x600-YUYV8_1_5X8

3. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=4500,height=3500,pixelformat=NV12,role=video

[8:08:02.372820979] [9981] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
[8:08:02.373280350] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12
[8:08:02.373534098] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 1920x1920-NV12
[8:08:02.373767722] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12
[8:08:02.374010095] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found
Camera configuration adjusted
[8:08:02.374449634] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found
[8:08:02.374591091] [9981]  INFO Camera camera.cpp:830 configuring streams: (0) 4416x3312-NV12
...
[8:08:02.376301704] [9982] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-0x0000
[8:08:02.376469994] [9982] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SRGGB10_1X10
[8:08:02.376576160] [9982]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
[8:08:02.376710909] [9982] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
[8:08:02.376839241] [9982] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
[8:08:02.377001115] [9982] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
[8:08:02.377104364] [9982] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
[8:08:02.377208780] [9982] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
[8:08:02.377324863] [9982] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
[8:08:02.377425779] [9982] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 4416x3312-YUYV8_2X8
[8:08:02.377533695] [9982] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 4416x3312-YUYV8_1_5X8

4. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=40,height=30,pixelformat=NV12,role=video

[8:10:41.140561164] [9989] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
[8:10:41.141348367] [9989]  INFO Camera camera.cpp:830 configuring streams: (0) 40x30-NV12
...
[8:10:41.143016688] [9990] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10
[8:10:41.143161062] [9990] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10
[8:10:41.143283561] [9990] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
[8:10:41.143442810] [9990] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
[8:10:41.143544893] [9990] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
[8:10:41.143644934] [9990] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
[8:10:41.143756058] [9990] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
[8:10:41.143854640] [9990] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 40x30-YUYV8_2X8
[8:10:41.143960223] [9990] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 40x30-YUYV8_1_5X8

5. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=3450,height=2456,pixelformat=NV12,role=video

[8:11:40.321895811] [9992] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
[8:11:40.322697014] [9992]  INFO Camera camera.cpp:830 configuring streams: (0) 3450x2456-NV12
...
[8:11:40.324431543] [9993] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10
[8:11:40.324579125] [9993] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10
[8:11:40.324680041] [9993]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
[8:11:40.324815082] [9993] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
[8:11:40.324946039] [9993] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
[8:11:40.325111121] [9993] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
[8:11:40.325216121] [9993] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
[8:11:40.325323453] [9993] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
[8:11:40.325441286] [9993] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
[8:11:40.325547451] [9993] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 3450x2456-YUYV8_2X8
[8:11:40.325660909] [9993] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 3450x2456-YUYV8_1_5X8

----

Sebastian Fricke (1):
  pipeline: rkisp1: Fix sensor-ISP format mismatch

 src/libcamera/pipeline/rkisp1/rkisp1.cpp | 36 ++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 3 deletions(-)

Comments

Dafna Hirschfeld Nov. 19, 2020, 4:41 p.m. UTC | #1
Hi,
Thanks!

Am 19.11.20 um 14:13 schrieb Sebastian Fricke:
> This patch fixes a mismatch of image formats during the pipeline
> creation of the RkISP1. The mismatch happens because the current code
> does not check if the configured format exceeded the maximum viable
> resolution of the ISP.
> This patch fixes the issue by first setting the ISP, checking if the
> given resolution was taken and searching for another viable format that
> is lower or equal to the ISP maximum.
> 
> Here are a couple of test runs:
> 1. Without stream configuration
> 2. With a normal stream configuration
> 3. With a stream configuration that exceeds the maximum of the ISP
> 4. With a very small resolution stream configuration
> 5. With a configuration that is closer to the upper than to the lower

Are all those commands succeed?
the idea is to make an effort to match the sensor output and isp input
so that streaming succeed as much as possible.

Thanks,
Dafna

> resolution
> 
> ----
> 
> 1. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3
> 
> [8:05:49.668381732] [9973] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
> [8:05:49.669108269] [9973]  INFO Camera camera.cpp:830 configuring streams: (0) 1920x1920-NV12
> ...
> [8:05:49.670965005] [9974] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10
> [8:05:49.671126004] [9974] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10
> [8:05:49.671233045] [9974]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
> [8:05:49.671378294] [9974] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
> [8:05:49.671510710] [9974] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
> [8:05:49.671676667] [9974] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
> [8:05:49.671787791] [9974] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
> [8:05:49.671894540] [9974] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
> [8:05:49.672107747] [9974] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
> [8:05:49.672226163] [9974] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 1920x1920-YUYV8_2X8
> [8:05:49.672341662] [9974] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 1920x1920-YUYV8_1_5X8
> 
> 2. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=900,height=600,pixelformat=NV12,role=video
> 
> [8:07:05.725799205] [9977] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
> [8:07:05.726242535] [9977]  INFO Camera camera.cpp:830 configuring streams: (0) 900x600-NV12
> ...
> [8:07:05.727718025] [9978] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10
> [8:07:05.727873482] [9978] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10
> [8:07:05.728056356] [9978] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
> [8:07:05.728233104] [9978] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
> [8:07:05.728340437] [9978] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
> [8:07:05.728448353] [9978] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
> [8:07:05.728567352] [9978] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
> [8:07:05.728673809] [9978] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 900x600-YUYV8_2X8
> [8:07:05.728785517] [9978] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 900x600-YUYV8_1_5X8
> 
> 3. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=4500,height=3500,pixelformat=NV12,role=video
> 
> [8:08:02.372820979] [9981] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
> [8:08:02.373280350] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12
> [8:08:02.373534098] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 1920x1920-NV12
> [8:08:02.373767722] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12
> [8:08:02.374010095] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found
> Camera configuration adjusted
> [8:08:02.374449634] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found
> [8:08:02.374591091] [9981]  INFO Camera camera.cpp:830 configuring streams: (0) 4416x3312-NV12
> ...
> [8:08:02.376301704] [9982] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-0x0000
> [8:08:02.376469994] [9982] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SRGGB10_1X10
> [8:08:02.376576160] [9982]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
> [8:08:02.376710909] [9982] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
> [8:08:02.376839241] [9982] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
> [8:08:02.377001115] [9982] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
> [8:08:02.377104364] [9982] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
> [8:08:02.377208780] [9982] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
> [8:08:02.377324863] [9982] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
> [8:08:02.377425779] [9982] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 4416x3312-YUYV8_2X8
> [8:08:02.377533695] [9982] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 4416x3312-YUYV8_1_5X8
> 
> 4. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=40,height=30,pixelformat=NV12,role=video
> 
> [8:10:41.140561164] [9989] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
> [8:10:41.141348367] [9989]  INFO Camera camera.cpp:830 configuring streams: (0) 40x30-NV12
> ...
> [8:10:41.143016688] [9990] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10
> [8:10:41.143161062] [9990] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10
> [8:10:41.143283561] [9990] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
> [8:10:41.143442810] [9990] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
> [8:10:41.143544893] [9990] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
> [8:10:41.143644934] [9990] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
> [8:10:41.143756058] [9990] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
> [8:10:41.143854640] [9990] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 40x30-YUYV8_2X8
> [8:10:41.143960223] [9990] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 40x30-YUYV8_1_5X8
> 
> 5. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=3450,height=2456,pixelformat=NV12,role=video
> 
> [8:11:40.321895811] [9992] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
> [8:11:40.322697014] [9992]  INFO Camera camera.cpp:830 configuring streams: (0) 3450x2456-NV12
> ...
> [8:11:40.324431543] [9993] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10
> [8:11:40.324579125] [9993] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10
> [8:11:40.324680041] [9993]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
> [8:11:40.324815082] [9993] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
> [8:11:40.324946039] [9993] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
> [8:11:40.325111121] [9993] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
> [8:11:40.325216121] [9993] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
> [8:11:40.325323453] [9993] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
> [8:11:40.325441286] [9993] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
> [8:11:40.325547451] [9993] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 3450x2456-YUYV8_2X8
> [8:11:40.325660909] [9993] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 3450x2456-YUYV8_1_5X8
> 
> ----
> 
> Sebastian Fricke (1):
>    pipeline: rkisp1: Fix sensor-ISP format mismatch
> 
>   src/libcamera/pipeline/rkisp1/rkisp1.cpp | 36 ++++++++++++++++++++++--
>   1 file changed, 33 insertions(+), 3 deletions(-)
>
Sebastian Fricke Nov. 19, 2020, 6:11 p.m. UTC | #2
On 19.11.2020 17:41, Dafna Hirschfeld wrote:
>Hi,
>Thanks!

Hi Dafna,

>
>Am 19.11.20 um 14:13 schrieb Sebastian Fricke:
>>This patch fixes a mismatch of image formats during the pipeline
>>creation of the RkISP1. The mismatch happens because the current code
>>does not check if the configured format exceeded the maximum viable
>>resolution of the ISP.
>>This patch fixes the issue by first setting the ISP, checking if the
>>given resolution was taken and searching for another viable format that
>>is lower or equal to the ISP maximum.
>>
>>Here are a couple of test runs:
>>1. Without stream configuration
>>2. With a normal stream configuration
>>3. With a stream configuration that exceeds the maximum of the ISP
>>4. With a very small resolution stream configuration
>>5. With a configuration that is closer to the upper than to the lower
>
>Are all those commands succeed?
>the idea is to make an effort to match the sensor output and isp input
>so that streaming succeed as much as possible.

Yes all of those commands worked, but I still have a few warnings
withing the log, which are unrelated to the format matching like:

ERROR V4L2 v4l2_subdevice.cpp:285 'ov13850 1-0010': Unable to get
rectangle 1 on pad 0: Inappropriate ioctl for device
ERROR CameraSensor camera_sensor.cpp:513 'ov13850 1-0010':
Failed to construct camera sensor info: the camera sensor does not report the active area
WARN RkISP1 rkisp1.cpp:936 Camera sensor information not available
WARN Timeline timeline.cpp:120 Action scheduled too late 12:27:38.176723000, run now 12:27:38.192436740
ERROR RkISP1 rkisp1.cpp:394 Parameters not ready on time for frame 5

>
>Thanks,
>Dafna

Thanks for the review!
Sebastian
>
>>resolution
>>
>>----
>>
>>1. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3
>>
>>[8:05:49.668381732] [9973] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
>>[8:05:49.669108269] [9973]  INFO Camera camera.cpp:830 configuring streams: (0) 1920x1920-NV12
>>...
>>[8:05:49.670965005] [9974] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10
>>[8:05:49.671126004] [9974] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10
>>[8:05:49.671233045] [9974]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
>>[8:05:49.671378294] [9974] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
>>[8:05:49.671510710] [9974] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
>>[8:05:49.671676667] [9974] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
>>[8:05:49.671787791] [9974] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
>>[8:05:49.671894540] [9974] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
>>[8:05:49.672107747] [9974] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
>>[8:05:49.672226163] [9974] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 1920x1920-YUYV8_2X8
>>[8:05:49.672341662] [9974] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 1920x1920-YUYV8_1_5X8
>>
>>2. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=900,height=600,pixelformat=NV12,role=video
>>
>>[8:07:05.725799205] [9977] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
>>[8:07:05.726242535] [9977]  INFO Camera camera.cpp:830 configuring streams: (0) 900x600-NV12
>>...
>>[8:07:05.727718025] [9978] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10
>>[8:07:05.727873482] [9978] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10
>>[8:07:05.728056356] [9978] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
>>[8:07:05.728233104] [9978] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
>>[8:07:05.728340437] [9978] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
>>[8:07:05.728448353] [9978] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
>>[8:07:05.728567352] [9978] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
>>[8:07:05.728673809] [9978] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 900x600-YUYV8_2X8
>>[8:07:05.728785517] [9978] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 900x600-YUYV8_1_5X8
>>
>>3. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=4500,height=3500,pixelformat=NV12,role=video
>>
>>[8:08:02.372820979] [9981] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
>>[8:08:02.373280350] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12
>>[8:08:02.373534098] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 1920x1920-NV12
>>[8:08:02.373767722] [9981] DEBUG RkISP1 rkisp1_path.cpp:100 Adjusting format from 4500x3500-NV12 to 4416x3312-NV12
>>[8:08:02.374010095] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found
>>Camera configuration adjusted
>>[8:08:02.374449634] [9981] DEBUG CameraSensor camera_sensor.cpp:408 'ov13850 1-0010': No supported format or size found
>>[8:08:02.374591091] [9981]  INFO Camera camera.cpp:830 configuring streams: (0) 4416x3312-NV12
>>...
>>[8:08:02.376301704] [9982] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-0x0000
>>[8:08:02.376469994] [9982] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SRGGB10_1X10
>>[8:08:02.376576160] [9982]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
>>[8:08:02.376710909] [9982] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
>>[8:08:02.376839241] [9982] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
>>[8:08:02.377001115] [9982] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
>>[8:08:02.377104364] [9982] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
>>[8:08:02.377208780] [9982] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
>>[8:08:02.377324863] [9982] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
>>[8:08:02.377425779] [9982] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 4416x3312-YUYV8_2X8
>>[8:08:02.377533695] [9982] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 4416x3312-YUYV8_1_5X8
>>
>>4. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=40,height=30,pixelformat=NV12,role=video
>>
>>[8:10:41.140561164] [9989] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
>>[8:10:41.141348367] [9989]  INFO Camera camera.cpp:830 configuring streams: (0) 40x30-NV12
>>...
>>[8:10:41.143016688] [9990] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 2112x1568-SBGGR10_1X10
>>[8:10:41.143161062] [9990] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 2112x1568-SBGGR10_1X10
>>[8:10:41.143283561] [9990] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
>>[8:10:41.143442810] [9990] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
>>[8:10:41.143544893] [9990] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
>>[8:10:41.143644934] [9990] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
>>[8:10:41.143756058] [9990] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
>>[8:10:41.143854640] [9990] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 40x30-YUYV8_2X8
>>[8:10:41.143960223] [9990] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 40x30-YUYV8_1_5X8
>>
>>5. LIBCAMERA_LOG_LEVELS=0 cam -c 1 --capture=3 -s width=3450,height=2456,pixelformat=NV12,role=video
>>
>>[8:11:40.321895811] [9992] DEBUG Camera camera.cpp:771 streams configuration: (0) 1920x1920-NV12
>>[8:11:40.322697014] [9992]  INFO Camera camera.cpp:830 configuring streams: (0) 3450x2456-NV12
>>...
>>[8:11:40.324431543] [9993] DEBUG RkISP1 rkisp1.cpp:687 Configuring sensor with 4224x3136-SBGGR10_1X10
>>[8:11:40.324579125] [9993] DEBUG RkISP1 rkisp1.cpp:694 ISP configured with 4032x3024-SBGGR10_1X10
>>[8:11:40.324680041] [9993]  INFO RkISP1 rkisp1.cpp:697 Configured resolution is greater than the maximum resolution for the ISP, trying to re-configure to a smaller valid sensor format
>>[8:11:40.324815082] [9993] DEBUG RkISP1 rkisp1.cpp:716 ISP re-configured with 2112x1568-SBGGR10_1X10
>>[8:11:40.324946039] [9993] DEBUG RkISP1 rkisp1.cpp:724 Sensor configured with 2112x1568-SBGGR10_1X10
>>[8:11:40.325111121] [9993] DEBUG RkISP1 rkisp1.cpp:731 ISP input pad configured with 2112x1568-SBGGR10_1X10
>>[8:11:40.325216121] [9993] DEBUG RkISP1 rkisp1.cpp:735 Configuring ISP output pad with 2112x1568-YUYV8_2X8
>>[8:11:40.325323453] [9993] DEBUG RkISP1 rkisp1.cpp:741 ISP output pad configured with 800x600-YUYV8_2X8
>>[8:11:40.325441286] [9993] DEBUG RkISP1 rkisp1_path.cpp:120 Configured main resizer input pad with 800x600-YUYV8_2X8
>>[8:11:40.325547451] [9993] DEBUG RkISP1 rkisp1_path.cpp:126 Configuring main resizer output pad with 3450x2456-YUYV8_2X8
>>[8:11:40.325660909] [9993] DEBUG RkISP1 rkisp1_path.cpp:144 Configured main resizer output pad with 3450x2456-YUYV8_1_5X8
>>
>>----
>>
>>Sebastian Fricke (1):
>>   pipeline: rkisp1: Fix sensor-ISP format mismatch
>>
>>  src/libcamera/pipeline/rkisp1/rkisp1.cpp | 36 ++++++++++++++++++++++--
>>  1 file changed, 33 insertions(+), 3 deletions(-)
>>