| Message ID | 20261028214307.158415-1-johannes.goede@oss.qualcomm.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series |
|
| Related | show |
Hans de Goede <johannes.goede@oss.qualcomm.com> writes: > Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats > in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) > formats. > > Before this change the simple-pipeline handler would pick whatever format > was enumerated first as capture format in this case. Which on IPU6 boards > comes down to picking unpacked. > > Packed formats use less memory and typically perform better due to reduced > RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with > i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, > 3 consecutive benchmark runs using libcamera's software ISP's builtin > Debayer benchmark gives the following results: > > Unpacked EGL us/frame: 6398 / 7393 / 7524 > CSI2 pack EGL us/frame: 5682 / 5536 / 5634 > > Unpacked CPU us/frame: 7653 / 8147 / 8614 > CSI2 pack CPU us/frame: 7183 / 7063 / 6318 > > Note the EGL comparison is not entirely fair, the unpacked debayer shader > uses the "McGuire" Debayer algorithm where as the packed one uses simpler > linear filtering. > > Modify the simple pipeline handler to prefer the more performant CSI2 > packed format when both formats are available. > > This also fixes a problem with unpacked formats on IPU6 where sometimes at > the start of a stream the IPU6 isys CSI2 receiver may return a corrupt > unpacked frame with high bits set which causes libcamera to crash with > an out of bound array access as the software ISP code expects the high bits > of unpacked formats to always be 0. > > Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 > Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> Reviewed-by: Milan Zamazal <mzamazal@redhat.com> > --- > Changes in v3: > - s/support/supported/ in the comment > > Changes in v2: > - Add note to the commit message about the EGL benchmark comparison not > being fair due to Debayer algorithm diferences > - Extend the comment to mention that this works around a bug on ipu6/7 > so that if the decision is made in the future to prefer unpacked it > will be clear to the person making such a change this might be a problem > --- > src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- > 1 file changed, 23 insertions(+), 1 deletion(-) > > diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp > index 35c29ceca..5f7e21463 100644 > --- a/src/libcamera/pipeline/simple/simple.cpp > +++ b/src/libcamera/pipeline/simple/simple.cpp > @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > * Then pick, among the possible configuration for the pixel format, > * the smallest sensor resolution that can accommodate all streams > * without upscaling. > + * > + * If both packed and unpacked Bayer capture formats are supported > + * prefer packed for lower memory usage and better software ISP > + * performance. > + * > + * Note preferring packed also works around a bug on ipu6/7 setups where > + * the CSI receives sometimes hands over corrupt frames. With unpacked > + * formats this causes OOB array accesses in the software ISP, due to > + * some of the unused high bits being 1 in these corrupt frames. > */ > const SimpleCameraData::Configuration *maxPipeConfig = nullptr; > pipeConfig_ = nullptr; > @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > for (const SimpleCameraData::Configuration *pipeConfig : *configs) { > const Size &captureSize = pipeConfig->captureSize; > const Size &maxOutputSize = pipeConfig->outputSizes.max; > + bool curCfgPacked = false; > + bool newCfgPacked = false; > + > + if (pipeConfig_) { > + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); > + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); > + /* Only compare packed vs unpacked if both formats are Bayer */ > + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { > + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; > + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; > + } > + } > > if (maxOutputSize.width >= maxProcessedStreamSize.width && > maxOutputSize.height >= maxProcessedStreamSize.height && > captureSize.width >= maxRawStreamSize.width && > captureSize.height >= maxRawStreamSize.height) { > - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) > + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || > + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) > pipeConfig_ = pipeConfig; > }
Quoting Hans de Goede (2026-10-28 21:43:07) > Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats > in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) > formats. > > Before this change the simple-pipeline handler would pick whatever format > was enumerated first as capture format in this case. Which on IPU6 boards > comes down to picking unpacked. > > Packed formats use less memory and typically perform better due to reduced > RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with > i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, > 3 consecutive benchmark runs using libcamera's software ISP's builtin > Debayer benchmark gives the following results: > > Unpacked EGL us/frame: 6398 / 7393 / 7524 > CSI2 pack EGL us/frame: 5682 / 5536 / 5634 > > Unpacked CPU us/frame: 7653 / 8147 / 8614 > CSI2 pack CPU us/frame: 7183 / 7063 / 6318 > > Note the EGL comparison is not entirely fair, the unpacked debayer shader > uses the "McGuire" Debayer algorithm where as the packed one uses simpler > linear filtering. For the moment, this looks like a much better solution than 'crashing'. Perhaps later someone might want to consider if it's worth porting the McGuire debayer algorithm into the packed debayer shader if anyone notices issues. > Modify the simple pipeline handler to prefer the more performant CSI2 > packed format when both formats are available. > > This also fixes a problem with unpacked formats on IPU6 where sometimes at > the start of a stream the IPU6 isys CSI2 receiver may return a corrupt > unpacked frame with high bits set which causes libcamera to crash with > an out of bound array access as the software ISP code expects the high bits > of unpacked formats to always be 0. > > Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 > Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> > --- > Changes in v3: > - s/support/supported/ in the comment > > Changes in v2: > - Add note to the commit message about the EGL benchmark comparison not > being fair due to Debayer algorithm diferences > - Extend the comment to mention that this works around a bug on ipu6/7 > so that if the decision is made in the future to prefer unpacked it > will be clear to the person making such a change this might be a problem > --- > src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- > 1 file changed, 23 insertions(+), 1 deletion(-) > > diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp > index 35c29ceca..5f7e21463 100644 > --- a/src/libcamera/pipeline/simple/simple.cpp > +++ b/src/libcamera/pipeline/simple/simple.cpp > @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > * Then pick, among the possible configuration for the pixel format, > * the smallest sensor resolution that can accommodate all streams > * without upscaling. > + * > + * If both packed and unpacked Bayer capture formats are supported > + * prefer packed for lower memory usage and better software ISP > + * performance. > + * > + * Note preferring packed also works around a bug on ipu6/7 setups where > + * the CSI receives sometimes hands over corrupt frames. With unpacked > + * formats this causes OOB array accesses in the software ISP, due to > + * some of the unused high bits being 1 in these corrupt frames. Ack. Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> > */ > const SimpleCameraData::Configuration *maxPipeConfig = nullptr; > pipeConfig_ = nullptr; > @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > for (const SimpleCameraData::Configuration *pipeConfig : *configs) { > const Size &captureSize = pipeConfig->captureSize; > const Size &maxOutputSize = pipeConfig->outputSizes.max; > + bool curCfgPacked = false; > + bool newCfgPacked = false; > + > + if (pipeConfig_) { > + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); > + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); > + /* Only compare packed vs unpacked if both formats are Bayer */ > + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { > + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; > + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; > + } > + } > > if (maxOutputSize.width >= maxProcessedStreamSize.width && > maxOutputSize.height >= maxProcessedStreamSize.height && > captureSize.width >= maxRawStreamSize.width && > captureSize.height >= maxRawStreamSize.height) { > - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) > + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || > + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) > pipeConfig_ = pipeConfig; > } > > -- > 2.56.0 >
Kieran Bingham <kieran.bingham@ideasonboard.com> writes: > Quoting Hans de Goede (2026-10-28 21:43:07) >> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) > >> formats. >> >> Before this change the simple-pipeline handler would pick whatever format >> was enumerated first as capture format in this case. Which on IPU6 boards >> comes down to picking unpacked. >> >> Packed formats use less memory and typically perform better due to reduced >> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >> 3 consecutive benchmark runs using libcamera's software ISP's builtin >> Debayer benchmark gives the following results: >> >> Unpacked EGL us/frame: 6398 / 7393 / 7524 >> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >> >> Unpacked CPU us/frame: 7653 / 8147 / 8614 >> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >> >> Note the EGL comparison is not entirely fair, the unpacked debayer shader >> uses the "McGuire" Debayer algorithm where as the packed one uses simpler >> linear filtering. > > For the moment, this looks like a much better solution than 'crashing'. > > Perhaps later someone might want to consider if it's worth porting the > McGuire debayer algorithm into the packed debayer shader if anyone > notices issues. Bryan's multi-pass uses a common shader / algorithm for both (currently bilinear IIRC); hopefully it resolves that difference sooner or later. > >> Modify the simple pipeline handler to prefer the more performant CSI2 >> packed format when both formats are available. >> >> This also fixes a problem with unpacked formats on IPU6 where sometimes at >> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >> unpacked frame with high bits set which causes libcamera to crash with >> an out of bound array access as the software ISP code expects the high bits >> of unpacked formats to always be 0. >> >> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >> --- >> Changes in v3: >> - s/support/supported/ in the comment >> >> Changes in v2: >> - Add note to the commit message about the EGL benchmark comparison not >> being fair due to Debayer algorithm diferences >> - Extend the comment to mention that this works around a bug on ipu6/7 >> so that if the decision is made in the future to prefer unpacked it >> will be clear to the person making such a change this might be a problem >> --- >> src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- >> 1 file changed, 23 insertions(+), 1 deletion(-) >> >> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >> index 35c29ceca..5f7e21463 100644 >> --- a/src/libcamera/pipeline/simple/simple.cpp >> +++ b/src/libcamera/pipeline/simple/simple.cpp >> @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >> * Then pick, among the possible configuration for the pixel format, >> * the smallest sensor resolution that can accommodate all streams >> * without upscaling. >> + * >> + * If both packed and unpacked Bayer capture formats are supported >> + * prefer packed for lower memory usage and better software ISP >> + * performance. >> + * >> + * Note preferring packed also works around a bug on ipu6/7 setups where >> + * the CSI receives sometimes hands over corrupt frames. With unpacked >> + * formats this causes OOB array accesses in the software ISP, due to >> + * some of the unused high bits being 1 in these corrupt frames. > > Ack. > > Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com> >> */ >> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >> pipeConfig_ = nullptr; >> @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >> const Size &captureSize = pipeConfig->captureSize; >> const Size &maxOutputSize = pipeConfig->outputSizes.max; >> + bool curCfgPacked = false; >> + bool newCfgPacked = false; >> + >> + if (pipeConfig_) { >> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >> + /* Only compare packed vs unpacked if both formats are Bayer */ >> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; >> + } >> + } >> >> if (maxOutputSize.width >= maxProcessedStreamSize.width && >> maxOutputSize.height >= maxProcessedStreamSize.height && >> captureSize.width >= maxRawStreamSize.width && >> captureSize.height >= maxRawStreamSize.height) { >> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) >> pipeConfig_ = pipeConfig; >> } >> >> -- >> 2.56.0 >>
It is with great sadness that I have to report that this makes the camera unusable on a thinkpad x1 yoga gen 7 with ipu6 and ov2740. It is as if the camera is oscillating between present and past frames all the time. It is a cool effect, but I'm afraid people will complain. This happens in cam (sdl renderer), qcam, and firefox (via pipewire) as well. Kernel log: continuous stream of ipu6_isys_csi2_error: 235 callbacks suppressed intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded 2026. 10. 28. 22:43 keltezéssel, Hans de Goede írta: > Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats > in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) > formats. > > Before this change the simple-pipeline handler would pick whatever format > was enumerated first as capture format in this case. Which on IPU6 boards > comes down to picking unpacked. > > Packed formats use less memory and typically perform better due to reduced > RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with > i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, > 3 consecutive benchmark runs using libcamera's software ISP's builtin > Debayer benchmark gives the following results: > > Unpacked EGL us/frame: 6398 / 7393 / 7524 > CSI2 pack EGL us/frame: 5682 / 5536 / 5634 > > Unpacked CPU us/frame: 7653 / 8147 / 8614 > CSI2 pack CPU us/frame: 7183 / 7063 / 6318 > > Note the EGL comparison is not entirely fair, the unpacked debayer shader > uses the "McGuire" Debayer algorithm where as the packed one uses simpler > linear filtering. > > Modify the simple pipeline handler to prefer the more performant CSI2 > packed format when both formats are available. > > This also fixes a problem with unpacked formats on IPU6 where sometimes at > the start of a stream the IPU6 isys CSI2 receiver may return a corrupt > unpacked frame with high bits set which causes libcamera to crash with > an out of bound array access as the software ISP code expects the high bits > of unpacked formats to always be 0. > > Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 > Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> > --- > Changes in v3: > - s/support/supported/ in the comment > > Changes in v2: > - Add note to the commit message about the EGL benchmark comparison not > being fair due to Debayer algorithm diferences > - Extend the comment to mention that this works around a bug on ipu6/7 > so that if the decision is made in the future to prefer unpacked it > will be clear to the person making such a change this might be a problem > --- > src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- > 1 file changed, 23 insertions(+), 1 deletion(-) > > diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp > index 35c29ceca..5f7e21463 100644 > --- a/src/libcamera/pipeline/simple/simple.cpp > +++ b/src/libcamera/pipeline/simple/simple.cpp > @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > * Then pick, among the possible configuration for the pixel format, > * the smallest sensor resolution that can accommodate all streams > * without upscaling. > + * > + * If both packed and unpacked Bayer capture formats are supported > + * prefer packed for lower memory usage and better software ISP > + * performance. > + * > + * Note preferring packed also works around a bug on ipu6/7 setups where > + * the CSI receives sometimes hands over corrupt frames. With unpacked > + * formats this causes OOB array accesses in the software ISP, due to > + * some of the unused high bits being 1 in these corrupt frames. > */ > const SimpleCameraData::Configuration *maxPipeConfig = nullptr; > pipeConfig_ = nullptr; > @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > for (const SimpleCameraData::Configuration *pipeConfig : *configs) { > const Size &captureSize = pipeConfig->captureSize; > const Size &maxOutputSize = pipeConfig->outputSizes.max; > + bool curCfgPacked = false; > + bool newCfgPacked = false; > + > + if (pipeConfig_) { > + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); > + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); > + /* Only compare packed vs unpacked if both formats are Bayer */ > + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { > + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; > + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; > + } > + } > > if (maxOutputSize.width >= maxProcessedStreamSize.width && > maxOutputSize.height >= maxProcessedStreamSize.height && > captureSize.width >= maxRawStreamSize.width && > captureSize.height >= maxRawStreamSize.height) { > - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) > + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || > + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) > pipeConfig_ = pipeConfig; > } >
Hi, On 2-Nov-26 10:44 AM, Barnabás Pőcze wrote: > It is with great sadness that I have to report that this makes the camera unusable > on a thinkpad x1 yoga gen 7 with ipu6 and ov2740. It is as if the camera is > oscillating between present and past frames all the time. It is a cool effect, > but I'm afraid people will complain. > > This happens in cam (sdl renderer), qcam, and firefox (via pipewire) as well. > > Kernel log: continuous stream of > > ipu6_isys_csi2_error: 235 callbacks suppressed > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded > intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded I think this might be causes by the ringbus of the CPU being clocked too low. What happens if you do: md5sum /dev/zero& say 4 or 8 times to cause some load and cause the CPU (and ringbus) to clock up? I've seen these errors before and then it was a too slow ringbus, but we never got around to fixing this. The IPU6/7 driver will need to make some Qos request to the PMU or something to tell it the CSI2 receiver needs more bandwidth on the ringbus. Assuming my guess is that this only happens on an idle system is correct ... Regards, Hans > > > 2026. 10. 28. 22:43 keltezéssel, Hans de Goede írta: >> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) >> formats. >> >> Before this change the simple-pipeline handler would pick whatever format >> was enumerated first as capture format in this case. Which on IPU6 boards >> comes down to picking unpacked. >> >> Packed formats use less memory and typically perform better due to reduced >> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >> 3 consecutive benchmark runs using libcamera's software ISP's builtin >> Debayer benchmark gives the following results: >> >> Unpacked EGL us/frame: 6398 / 7393 / 7524 >> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >> >> Unpacked CPU us/frame: 7653 / 8147 / 8614 >> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >> >> Note the EGL comparison is not entirely fair, the unpacked debayer shader >> uses the "McGuire" Debayer algorithm where as the packed one uses simpler >> linear filtering. >> >> Modify the simple pipeline handler to prefer the more performant CSI2 >> packed format when both formats are available. >> >> This also fixes a problem with unpacked formats on IPU6 where sometimes at >> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >> unpacked frame with high bits set which causes libcamera to crash with >> an out of bound array access as the software ISP code expects the high bits >> of unpacked formats to always be 0. >> >> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >> --- >> Changes in v3: >> - s/support/supported/ in the comment >> >> Changes in v2: >> - Add note to the commit message about the EGL benchmark comparison not >> being fair due to Debayer algorithm diferences >> - Extend the comment to mention that this works around a bug on ipu6/7 >> so that if the decision is made in the future to prefer unpacked it >> will be clear to the person making such a change this might be a problem >> --- >> src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- >> 1 file changed, 23 insertions(+), 1 deletion(-) >> >> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >> index 35c29ceca..5f7e21463 100644 >> --- a/src/libcamera/pipeline/simple/simple.cpp >> +++ b/src/libcamera/pipeline/simple/simple.cpp >> @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >> * Then pick, among the possible configuration for the pixel format, >> * the smallest sensor resolution that can accommodate all streams >> * without upscaling. >> + * >> + * If both packed and unpacked Bayer capture formats are supported >> + * prefer packed for lower memory usage and better software ISP >> + * performance. >> + * >> + * Note preferring packed also works around a bug on ipu6/7 setups where >> + * the CSI receives sometimes hands over corrupt frames. With unpacked >> + * formats this causes OOB array accesses in the software ISP, due to >> + * some of the unused high bits being 1 in these corrupt frames. >> */ >> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >> pipeConfig_ = nullptr; >> @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >> const Size &captureSize = pipeConfig->captureSize; >> const Size &maxOutputSize = pipeConfig->outputSizes.max; >> + bool curCfgPacked = false; >> + bool newCfgPacked = false; >> + >> + if (pipeConfig_) { >> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >> + /* Only compare packed vs unpacked if both formats are Bayer */ >> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; >> + } >> + } >> if (maxOutputSize.width >= maxProcessedStreamSize.width && >> maxOutputSize.height >= maxProcessedStreamSize.height && >> captureSize.width >= maxRawStreamSize.width && >> captureSize.height >= maxRawStreamSize.height) { >> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) >> pipeConfig_ = pipeConfig; >> } >> >
2026. 11. 02. 14:44 keltezéssel, Hans de Goede írta: > Hi, > > On 2-Nov-26 10:44 AM, Barnabás Pőcze wrote: >> It is with great sadness that I have to report that this makes the camera unusable >> on a thinkpad x1 yoga gen 7 with ipu6 and ov2740. It is as if the camera is >> oscillating between present and past frames all the time. It is a cool effect, >> but I'm afraid people will complain. >> >> This happens in cam (sdl renderer), qcam, and firefox (via pipewire) as well. >> >> Kernel log: continuous stream of >> >> ipu6_isys_csi2_error: 235 callbacks suppressed >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded >> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > > I think this might be causes by the ringbus of the CPU being clocked too low. > > What happens if you do: > > md5sum /dev/zero& > > say 4 or 8 times to cause some load and cause the CPU (and ringbus) to clock up? > > I've seen these errors before and then it was a too slow ringbus, but we never > got around to fixing this. The IPU6/7 driver will need to make some Qos request > to the PMU or something to tell it the CSI2 receiver needs more bandwidth on > the ringbus. Assuming my guess is that this only happens on an idle system is > correct ... I started `stress -c ...` in parallel to the streaming, and it appears fix the issue. > > Regards, > > Hans > > > >> >> >> 2026. 10. 28. 22:43 keltezéssel, Hans de Goede írta: >>> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >>> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) >>> formats. >>> >>> Before this change the simple-pipeline handler would pick whatever format >>> was enumerated first as capture format in this case. Which on IPU6 boards >>> comes down to picking unpacked. >>> >>> Packed formats use less memory and typically perform better due to reduced >>> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >>> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >>> 3 consecutive benchmark runs using libcamera's software ISP's builtin >>> Debayer benchmark gives the following results: >>> >>> Unpacked EGL us/frame: 6398 / 7393 / 7524 >>> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >>> >>> Unpacked CPU us/frame: 7653 / 8147 / 8614 >>> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >>> >>> Note the EGL comparison is not entirely fair, the unpacked debayer shader >>> uses the "McGuire" Debayer algorithm where as the packed one uses simpler >>> linear filtering. >>> >>> Modify the simple pipeline handler to prefer the more performant CSI2 >>> packed format when both formats are available. >>> >>> This also fixes a problem with unpacked formats on IPU6 where sometimes at >>> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >>> unpacked frame with high bits set which causes libcamera to crash with >>> an out of bound array access as the software ISP code expects the high bits >>> of unpacked formats to always be 0. >>> >>> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >>> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >>> --- >>> Changes in v3: >>> - s/support/supported/ in the comment >>> >>> Changes in v2: >>> - Add note to the commit message about the EGL benchmark comparison not >>> being fair due to Debayer algorithm diferences >>> - Extend the comment to mention that this works around a bug on ipu6/7 >>> so that if the decision is made in the future to prefer unpacked it >>> will be clear to the person making such a change this might be a problem >>> --- >>> src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- >>> 1 file changed, 23 insertions(+), 1 deletion(-) >>> >>> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >>> index 35c29ceca..5f7e21463 100644 >>> --- a/src/libcamera/pipeline/simple/simple.cpp >>> +++ b/src/libcamera/pipeline/simple/simple.cpp >>> @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>> * Then pick, among the possible configuration for the pixel format, >>> * the smallest sensor resolution that can accommodate all streams >>> * without upscaling. >>> + * >>> + * If both packed and unpacked Bayer capture formats are supported >>> + * prefer packed for lower memory usage and better software ISP >>> + * performance. >>> + * >>> + * Note preferring packed also works around a bug on ipu6/7 setups where >>> + * the CSI receives sometimes hands over corrupt frames. With unpacked >>> + * formats this causes OOB array accesses in the software ISP, due to >>> + * some of the unused high bits being 1 in these corrupt frames. >>> */ >>> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >>> pipeConfig_ = nullptr; >>> @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >>> const Size &captureSize = pipeConfig->captureSize; >>> const Size &maxOutputSize = pipeConfig->outputSizes.max; >>> + bool curCfgPacked = false; >>> + bool newCfgPacked = false; >>> + >>> + if (pipeConfig_) { >>> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >>> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >>> + /* Only compare packed vs unpacked if both formats are Bayer */ >>> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >>> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >>> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; >>> + } >>> + } >>> if (maxOutputSize.width >= maxProcessedStreamSize.width && >>> maxOutputSize.height >= maxProcessedStreamSize.height && >>> captureSize.width >= maxRawStreamSize.width && >>> captureSize.height >= maxRawStreamSize.height) { >>> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >>> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >>> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) >>> pipeConfig_ = pipeConfig; >>> } >>> >> >
Hi, On 2-Nov-26 2:52 PM, Barnabás Pőcze wrote: > 2026. 11. 02. 14:44 keltezéssel, Hans de Goede írta: >> Hi, >> >> On 2-Nov-26 10:44 AM, Barnabás Pőcze wrote: >>> It is with great sadness that I have to report that this makes the camera unusable >>> on a thinkpad x1 yoga gen 7 with ipu6 and ov2740. It is as if the camera is >>> oscillating between present and past frames all the time. It is a cool effect, >>> but I'm afraid people will complain. >>> >>> This happens in cam (sdl renderer), qcam, and firefox (via pipewire) as well. >>> >>> Kernel log: continuous stream of >>> >>> ipu6_isys_csi2_error: 235 callbacks suppressed >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded >> >> I think this might be causes by the ringbus of the CPU being clocked too low. >> >> What happens if you do: >> >> md5sum /dev/zero& >> >> say 4 or 8 times to cause some load and cause the CPU (and ringbus) to clock up? >> >> I've seen these errors before and then it was a too slow ringbus, but we never >> got around to fixing this. The IPU6/7 driver will need to make some Qos request >> to the PMU or something to tell it the CSI2 receiver needs more bandwidth on >> the ringbus. Assuming my guess is that this only happens on an idle system is >> correct ... > > I started `stress -c ...` in parallel to the streaming, and it appears fix the issue. Yes, so this is the ringbus clocking too low to handle the raw sensor bandwidth and the IPU6 isys driver not doing whatever magic it needs to do to request more bandwidth on the ringbus. Regards, Hans >> >> >> >>> >>> >>> 2026. 10. 28. 22:43 keltezéssel, Hans de Goede írta: >>>> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >>>> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) >>>> formats. >>>> >>>> Before this change the simple-pipeline handler would pick whatever format >>>> was enumerated first as capture format in this case. Which on IPU6 boards >>>> comes down to picking unpacked. >>>> >>>> Packed formats use less memory and typically perform better due to reduced >>>> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >>>> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >>>> 3 consecutive benchmark runs using libcamera's software ISP's builtin >>>> Debayer benchmark gives the following results: >>>> >>>> Unpacked EGL us/frame: 6398 / 7393 / 7524 >>>> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >>>> >>>> Unpacked CPU us/frame: 7653 / 8147 / 8614 >>>> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >>>> >>>> Note the EGL comparison is not entirely fair, the unpacked debayer shader >>>> uses the "McGuire" Debayer algorithm where as the packed one uses simpler >>>> linear filtering. >>>> >>>> Modify the simple pipeline handler to prefer the more performant CSI2 >>>> packed format when both formats are available. >>>> >>>> This also fixes a problem with unpacked formats on IPU6 where sometimes at >>>> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >>>> unpacked frame with high bits set which causes libcamera to crash with >>>> an out of bound array access as the software ISP code expects the high bits >>>> of unpacked formats to always be 0. >>>> >>>> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >>>> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >>>> --- >>>> Changes in v3: >>>> - s/support/supported/ in the comment >>>> >>>> Changes in v2: >>>> - Add note to the commit message about the EGL benchmark comparison not >>>> being fair due to Debayer algorithm diferences >>>> - Extend the comment to mention that this works around a bug on ipu6/7 >>>> so that if the decision is made in the future to prefer unpacked it >>>> will be clear to the person making such a change this might be a problem >>>> --- >>>> src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- >>>> 1 file changed, 23 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >>>> index 35c29ceca..5f7e21463 100644 >>>> --- a/src/libcamera/pipeline/simple/simple.cpp >>>> +++ b/src/libcamera/pipeline/simple/simple.cpp >>>> @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>>> * Then pick, among the possible configuration for the pixel format, >>>> * the smallest sensor resolution that can accommodate all streams >>>> * without upscaling. >>>> + * >>>> + * If both packed and unpacked Bayer capture formats are supported >>>> + * prefer packed for lower memory usage and better software ISP >>>> + * performance. >>>> + * >>>> + * Note preferring packed also works around a bug on ipu6/7 setups where >>>> + * the CSI receives sometimes hands over corrupt frames. With unpacked >>>> + * formats this causes OOB array accesses in the software ISP, due to >>>> + * some of the unused high bits being 1 in these corrupt frames. >>>> */ >>>> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >>>> pipeConfig_ = nullptr; >>>> @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>>> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >>>> const Size &captureSize = pipeConfig->captureSize; >>>> const Size &maxOutputSize = pipeConfig->outputSizes.max; >>>> + bool curCfgPacked = false; >>>> + bool newCfgPacked = false; >>>> + >>>> + if (pipeConfig_) { >>>> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >>>> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >>>> + /* Only compare packed vs unpacked if both formats are Bayer */ >>>> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >>>> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >>>> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; >>>> + } >>>> + } >>>> if (maxOutputSize.width >= maxProcessedStreamSize.width && >>>> maxOutputSize.height >= maxProcessedStreamSize.height && >>>> captureSize.width >= maxRawStreamSize.width && >>>> captureSize.height >= maxRawStreamSize.height) { >>>> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >>>> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >>>> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) >>>> pipeConfig_ = pipeConfig; >>>> } >>>> >>> >> >
Quoting Hans de Goede (2026-11-02 15:08:47) > Hi, > > On 2-Nov-26 2:52 PM, Barnabás Pőcze wrote: > > 2026. 11. 02. 14:44 keltezéssel, Hans de Goede írta: > >> Hi, > >> > >> On 2-Nov-26 10:44 AM, Barnabás Pőcze wrote: > >>> It is with great sadness that I have to report that this makes the camera unusable > >>> on a thinkpad x1 yoga gen 7 with ipu6 and ov2740. It is as if the camera is > >>> oscillating between present and past frames all the time. It is a cool effect, > >>> but I'm afraid people will complain. > >>> > >>> This happens in cam (sdl renderer), qcam, and firefox (via pipewire) as well. > >>> > >>> Kernel log: continuous stream of > >>> > >>> ipu6_isys_csi2_error: 235 callbacks suppressed > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Transfer FIFO overflow > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame short packet discarded > >>> intel_ipu6_isys.isys intel_ipu6.isys.40: csi2-1 error: Inter-frame long packet discarded > >> > >> I think this might be causes by the ringbus of the CPU being clocked too low. > >> > >> What happens if you do: > >> > >> md5sum /dev/zero& > >> > >> say 4 or 8 times to cause some load and cause the CPU (and ringbus) to clock up? > >> > >> I've seen these errors before and then it was a too slow ringbus, but we never > >> got around to fixing this. The IPU6/7 driver will need to make some Qos request > >> to the PMU or something to tell it the CSI2 receiver needs more bandwidth on > >> the ringbus. Assuming my guess is that this only happens on an idle system is > >> correct ... > > > > I started `stress -c ...` in parallel to the streaming, and it appears fix the issue. > > Yes, so this is the ringbus clocking too low to handle the raw sensor bandwidth > and the IPU6 isys driver not doing whatever magic it needs to do to request more > bandwidth on the ringbus. Can I infer we call this a kernel bug and I don't need to consider a libcamera-0.8.1 urgent release in the next couple of days ? What's the next step on this issue ? Are there other devices impacted by this ? -- Kieran > > Regards, > > Hans > > > > >> > >> > >> > >>> > >>> > >>> 2026. 10. 28. 22:43 keltezéssel, Hans de Goede írta: > >>>> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats > >>>> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) > >>>> formats. > >>>> > >>>> Before this change the simple-pipeline handler would pick whatever format > >>>> was enumerated first as capture format in this case. Which on IPU6 boards > >>>> comes down to picking unpacked. > >>>> > >>>> Packed formats use less memory and typically perform better due to reduced > >>>> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with > >>>> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, > >>>> 3 consecutive benchmark runs using libcamera's software ISP's builtin > >>>> Debayer benchmark gives the following results: > >>>> > >>>> Unpacked EGL us/frame: 6398 / 7393 / 7524 > >>>> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 > >>>> > >>>> Unpacked CPU us/frame: 7653 / 8147 / 8614 > >>>> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 > >>>> > >>>> Note the EGL comparison is not entirely fair, the unpacked debayer shader > >>>> uses the "McGuire" Debayer algorithm where as the packed one uses simpler > >>>> linear filtering. > >>>> > >>>> Modify the simple pipeline handler to prefer the more performant CSI2 > >>>> packed format when both formats are available. > >>>> > >>>> This also fixes a problem with unpacked formats on IPU6 where sometimes at > >>>> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt > >>>> unpacked frame with high bits set which causes libcamera to crash with > >>>> an out of bound array access as the software ISP code expects the high bits > >>>> of unpacked formats to always be 0. > >>>> > >>>> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 > >>>> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> > >>>> --- > >>>> Changes in v3: > >>>> - s/support/supported/ in the comment > >>>> > >>>> Changes in v2: > >>>> - Add note to the commit message about the EGL benchmark comparison not > >>>> being fair due to Debayer algorithm diferences > >>>> - Extend the comment to mention that this works around a bug on ipu6/7 > >>>> so that if the decision is made in the future to prefer unpacked it > >>>> will be clear to the person making such a change this might be a problem > >>>> --- > >>>> src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- > >>>> 1 file changed, 23 insertions(+), 1 deletion(-) > >>>> > >>>> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp > >>>> index 35c29ceca..5f7e21463 100644 > >>>> --- a/src/libcamera/pipeline/simple/simple.cpp > >>>> +++ b/src/libcamera/pipeline/simple/simple.cpp > >>>> @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > >>>> * Then pick, among the possible configuration for the pixel format, > >>>> * the smallest sensor resolution that can accommodate all streams > >>>> * without upscaling. > >>>> + * > >>>> + * If both packed and unpacked Bayer capture formats are supported > >>>> + * prefer packed for lower memory usage and better software ISP > >>>> + * performance. > >>>> + * > >>>> + * Note preferring packed also works around a bug on ipu6/7 setups where > >>>> + * the CSI receives sometimes hands over corrupt frames. With unpacked > >>>> + * formats this causes OOB array accesses in the software ISP, due to > >>>> + * some of the unused high bits being 1 in these corrupt frames. > >>>> */ > >>>> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; > >>>> pipeConfig_ = nullptr; > >>>> @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > >>>> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { > >>>> const Size &captureSize = pipeConfig->captureSize; > >>>> const Size &maxOutputSize = pipeConfig->outputSizes.max; > >>>> + bool curCfgPacked = false; > >>>> + bool newCfgPacked = false; > >>>> + > >>>> + if (pipeConfig_) { > >>>> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); > >>>> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); > >>>> + /* Only compare packed vs unpacked if both formats are Bayer */ > >>>> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { > >>>> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; > >>>> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; > >>>> + } > >>>> + } > >>>> if (maxOutputSize.width >= maxProcessedStreamSize.width && > >>>> maxOutputSize.height >= maxProcessedStreamSize.height && > >>>> captureSize.width >= maxRawStreamSize.width && > >>>> captureSize.height >= maxRawStreamSize.height) { > >>>> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) > >>>> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || > >>>> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) > >>>> pipeConfig_ = pipeConfig; > >>>> } > >>>> > >>> > >> > > >
diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 35c29ceca..5f7e21463 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1216,6 +1216,15 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() * Then pick, among the possible configuration for the pixel format, * the smallest sensor resolution that can accommodate all streams * without upscaling. + * + * If both packed and unpacked Bayer capture formats are supported + * prefer packed for lower memory usage and better software ISP + * performance. + * + * Note preferring packed also works around a bug on ipu6/7 setups where + * the CSI receives sometimes hands over corrupt frames. With unpacked + * formats this causes OOB array accesses in the software ISP, due to + * some of the unused high bits being 1 in these corrupt frames. */ const SimpleCameraData::Configuration *maxPipeConfig = nullptr; pipeConfig_ = nullptr; @@ -1223,12 +1232,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() for (const SimpleCameraData::Configuration *pipeConfig : *configs) { const Size &captureSize = pipeConfig->captureSize; const Size &maxOutputSize = pipeConfig->outputSizes.max; + bool curCfgPacked = false; + bool newCfgPacked = false; + + if (pipeConfig_) { + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); + /* Only compare packed vs unpacked if both formats are Bayer */ + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; + } + } if (maxOutputSize.width >= maxProcessedStreamSize.width && maxOutputSize.height >= maxProcessedStreamSize.height && captureSize.width >= maxRawStreamSize.width && captureSize.height >= maxRawStreamSize.height) { - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) pipeConfig_ = pipeConfig; }
Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) formats. Before this change the simple-pipeline handler would pick whatever format was enumerated first as capture format in this case. Which on IPU6 boards comes down to picking unpacked. Packed formats use less memory and typically perform better due to reduced RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, 3 consecutive benchmark runs using libcamera's software ISP's builtin Debayer benchmark gives the following results: Unpacked EGL us/frame: 6398 / 7393 / 7524 CSI2 pack EGL us/frame: 5682 / 5536 / 5634 Unpacked CPU us/frame: 7653 / 8147 / 8614 CSI2 pack CPU us/frame: 7183 / 7063 / 6318 Note the EGL comparison is not entirely fair, the unpacked debayer shader uses the "McGuire" Debayer algorithm where as the packed one uses simpler linear filtering. Modify the simple pipeline handler to prefer the more performant CSI2 packed format when both formats are available. This also fixes a problem with unpacked formats on IPU6 where sometimes at the start of a stream the IPU6 isys CSI2 receiver may return a corrupt unpacked frame with high bits set which causes libcamera to crash with an out of bound array access as the software ISP code expects the high bits of unpacked formats to always be 0. Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> --- Changes in v3: - s/support/supported/ in the comment Changes in v2: - Add note to the commit message about the EGL benchmark comparison not being fair due to Debayer algorithm diferences - Extend the comment to mention that this works around a bug on ipu6/7 so that if the decision is made in the future to prefer unpacked it will be clear to the person making such a change this might be a problem --- src/libcamera/pipeline/simple/simple.cpp | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)