| Message ID | 20261025155024.113171-1-johannes.goede@oss.qualcomm.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series |
|
| Related | show |
Hi Hans, thank you for the patch. Hans de Goede <johannes.goede@oss.qualcomm.com> writes: > Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats > in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) > formats. > > Before this change the simple-pipeline handler would pick whatever format > was enumerated first as capture format in this case. Which on IPU6 boards > comes down to picking unpacked. > > Packed formats use less memory and typically perform better due to reduced > RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with > i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, > 3 consecutive benchmark runs using libcamera's software ISP's builtin > Debayer benchmark gives the following results: > > Unpacked EGL us/frame: 6398 / 7393 / 7524 > CSI2 pack EGL us/frame: 5682 / 5536 / 5634 > > Unpacked CPU us/frame: 7653 / 8147 / 8614 > CSI2 pack CPU us/frame: 7183 / 7063 / 6318 > > Modify the simple pipeline handler to prefer the more performant CSI2 > packed format when both formats are available. Please note that packed and unpacked fragment shaders use different debayering algorithms. Packed uses sort of bilinear (despite what the file header says), unpacked McGuire. While the memory argument sounds plausible, the benchmarks cannot tell because of the different algorithms used. In future, there should be a single benchmarking algorithm. I think Bryan's multipass uses the same debayering algorithm for both, so it'd be worth to re-measure there. > > This also fixes a problem with unpacked formats on IPU6 where sometimes at > the start of a stream the IPU6 isys CSI2 receiver may return a corrupt > unpacked frame with high bits set which causes libcamera to crash with > an out of bound array access as the software ISP code expects the high bits > of unpacked formats to always be 0. I'm not sure how much significance such hardware problems should have for selection of the coding, although yes, such a problem can be avoided with packed. I don't say it's not useful to prefer packed but let's get the right benchmarks for the decision if possible. > > Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 > Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> > --- > src/libcamera/pipeline/simple/simple.cpp | 18 +++++++++++++++++- > 1 file changed, 17 insertions(+), 1 deletion(-) > > diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp > index 35c29ceca..3b59087ae 100644 > --- a/src/libcamera/pipeline/simple/simple.cpp > +++ b/src/libcamera/pipeline/simple/simple.cpp > @@ -1216,6 +1216,9 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > * Then pick, among the possible configuration for the pixel format, > * the smallest sensor resolution that can accommodate all streams > * without upscaling. > + * > + * If both packed and unpacked Bayer capture formats are support prefer > + * packed for lower memory usage and better software ISP performance. > */ > const SimpleCameraData::Configuration *maxPipeConfig = nullptr; > pipeConfig_ = nullptr; > @@ -1223,12 +1226,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() > for (const SimpleCameraData::Configuration *pipeConfig : *configs) { > const Size &captureSize = pipeConfig->captureSize; > const Size &maxOutputSize = pipeConfig->outputSizes.max; > + bool curCfgPacked = false; > + bool newCfgPacked = false; > + > + if (pipeConfig_) { > + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); > + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); > + /* Only compare packed vs unpacked if both formats are Bayer */ > + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { > + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; > + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; > + } > + } > > if (maxOutputSize.width >= maxProcessedStreamSize.width && > maxOutputSize.height >= maxProcessedStreamSize.height && > captureSize.width >= maxRawStreamSize.width && > captureSize.height >= maxRawStreamSize.height) { > - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) > + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || > + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) > pipeConfig_ = pipeConfig; > }
Hi Milan, On 26-Oct-26 12:36 PM, Milan Zamazal wrote: > Hi Hans, > > thank you for the patch. > > Hans de Goede <johannes.goede@oss.qualcomm.com> writes: > >> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) >> formats. >> >> Before this change the simple-pipeline handler would pick whatever format >> was enumerated first as capture format in this case. Which on IPU6 boards >> comes down to picking unpacked. >> >> Packed formats use less memory and typically perform better due to reduced >> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >> 3 consecutive benchmark runs using libcamera's software ISP's builtin >> Debayer benchmark gives the following results: >> >> Unpacked EGL us/frame: 6398 / 7393 / 7524 >> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >> >> Unpacked CPU us/frame: 7653 / 8147 / 8614 >> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >> >> Modify the simple pipeline handler to prefer the more performant CSI2 >> packed format when both formats are available. > > Please note that packed and unpacked fragment shaders use different > debayering algorithms. Packed uses sort of bilinear (despite what the > file header says), unpacked McGuire. While the memory argument sounds > plausible, the benchmarks cannot tell because of the different > algorithms used. In future, there should be a single benchmarking > algorithm. > > I think Bryan's multipass uses the same debayering algorithm for both, > so it'd be worth to re-measure there. Well, atm we cannot really do an apples 2 apples comparison for the EGL code-paths since there is no code to do so yet, at least I do not believe Bryan's code for this is really in a stable state ? Note that the CPU based comparison does show a significant win and typically we tend to be more performance constrained when falling back to the CPU. >> This also fixes a problem with unpacked formats on IPU6 where sometimes at >> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >> unpacked frame with high bits set which causes libcamera to crash with >> an out of bound array access as the software ISP code expects the high bits >> of unpacked formats to always be 0. > > I'm not sure how much significance such hardware problems should have > for selection of the coding, although yes, such a problem can be avoided > with packed. The alternatives are: 1. Live with the softISP sometimes crashing (unacceptable IMHO) 2. Masking out the high-bits in the memcpy path in CPU based debayer: https://patchwork.libcamera.org/patch/25007/ where you complained about a 4% performance degradation, while this patch is a win for CPU based decoding + as you point out that patch needs to be expanded to also cover the swstats only case used in the EGL paths, making those somewhat slower too. So even if the performance win for EGL here is limited or 0 when using the same debayer algo in both cases, this still seems like a better way to fix the crashes we are seeing then my initial masking approach. Regards, Hans > >> >> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >> --- >> src/libcamera/pipeline/simple/simple.cpp | 18 +++++++++++++++++- >> 1 file changed, 17 insertions(+), 1 deletion(-) >> >> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >> index 35c29ceca..3b59087ae 100644 >> --- a/src/libcamera/pipeline/simple/simple.cpp >> +++ b/src/libcamera/pipeline/simple/simple.cpp >> @@ -1216,6 +1216,9 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >> * Then pick, among the possible configuration for the pixel format, >> * the smallest sensor resolution that can accommodate all streams >> * without upscaling. >> + * >> + * If both packed and unpacked Bayer capture formats are support prefer >> + * packed for lower memory usage and better software ISP performance. >> */ >> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >> pipeConfig_ = nullptr; >> @@ -1223,12 +1226,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >> const Size &captureSize = pipeConfig->captureSize; >> const Size &maxOutputSize = pipeConfig->outputSizes.max; >> + bool curCfgPacked = false; >> + bool newCfgPacked = false; >> + >> + if (pipeConfig_) { >> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >> + /* Only compare packed vs unpacked if both formats are Bayer */ >> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; >> + } >> + } >> >> if (maxOutputSize.width >= maxProcessedStreamSize.width && >> maxOutputSize.height >= maxProcessedStreamSize.height && >> captureSize.width >= maxRawStreamSize.width && >> captureSize.height >= maxRawStreamSize.height) { >> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) >> pipeConfig_ = pipeConfig; >> } >
Hans de Goede <johannes.goede@oss.qualcomm.com> writes: > Hi Milan, > > On 26-Oct-26 12:36 PM, Milan Zamazal wrote: >> Hi Hans, >> >> thank you for the patch. >> >> Hans de Goede <johannes.goede@oss.qualcomm.com> writes: >> >>> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >>> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) >>> formats. >>> >>> Before this change the simple-pipeline handler would pick whatever format >>> was enumerated first as capture format in this case. Which on IPU6 boards >>> comes down to picking unpacked. >>> >>> Packed formats use less memory and typically perform better due to reduced >>> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >>> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >>> 3 consecutive benchmark runs using libcamera's software ISP's builtin >>> Debayer benchmark gives the following results: >>> >>> Unpacked EGL us/frame: 6398 / 7393 / 7524 >>> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >>> >>> Unpacked CPU us/frame: 7653 / 8147 / 8614 >>> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >>> >>> Modify the simple pipeline handler to prefer the more performant CSI2 >>> packed format when both formats are available. >> >> Please note that packed and unpacked fragment shaders use different >> debayering algorithms. Packed uses sort of bilinear (despite what the >> file header says), unpacked McGuire. While the memory argument sounds >> plausible, the benchmarks cannot tell because of the different >> algorithms used. In future, there should be a single benchmarking >> algorithm. >> >> I think Bryan's multipass uses the same debayering algorithm for both, >> so it'd be worth to re-measure there. > > Well, atm we cannot really do an apples 2 apples comparison for the EGL > code-paths since there is no code to do so yet, at least I do not believe > Bryan's code for this is really in a stable state ? I think the best what we currently have is https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commit/6a69831fbdfd7601cc7b89c8006ed2ff907e2d58 (see *packed*.frag there). I don't know whether the packed case can be optimised but in the current form it has clearly more operations to decode. Without measurements, we can't say what's faster, even less generally faster. I'd test with the branch (0.7.0-multipass-v4) but I don't have access to my environment ATM. > > Note that the CPU based comparison does show a significant win and > typically we tend to be more performance constrained when falling > back to the CPU. I can't say it doesn't matter, I have also platforms where CPU is faster than GPU. But GPU is the primary platform and small differences are relevant and more important than larger CPU differences. Which is a reason itself to prefer one of the encodings and we need a patch like this, the question is which one of the encodings to prefer (putting the IPU6 problem aside). According to your benchmark, with current software ISP, we should apply the patch as it is. > >>> This also fixes a problem with unpacked formats on IPU6 where sometimes at >>> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >>> unpacked frame with high bits set which causes libcamera to crash with >>> an out of bound array access as the software ISP code expects the high bits >>> of unpacked formats to always be 0. >> >> I'm not sure how much significance such hardware problems should have >> for selection of the coding, although yes, such a problem can be avoided >> with packed. > > The alternatives are: > > 1. Live with the softISP sometimes crashing (unacceptable IMHO) > > 2. Masking out the high-bits in the memcpy path in CPU based debayer: > > https://patchwork.libcamera.org/patch/25007/ > > where you complained about a 4% performance degradation, while this patch is > a win for CPU based decoding + as you point out that patch needs to be expanded > to also cover the swstats only case used in the EGL paths, making those somewhat > slower too. > > So even if the performance win for EGL here is limited or 0 when using the same > debayer algo in both cases, this still seems like a better way to fix the crashes > we are seeing then my initial masking approach. The crash workaround should be mentioned in a code comment, in case we decide not to prefer packed in future for whatever reason. > > Regards, > > Hans > > > >> >>> >>> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >>> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >>> --- >>> src/libcamera/pipeline/simple/simple.cpp | 18 +++++++++++++++++- >>> 1 file changed, 17 insertions(+), 1 deletion(-) >>> >>> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >>> index 35c29ceca..3b59087ae 100644 >>> --- a/src/libcamera/pipeline/simple/simple.cpp >>> +++ b/src/libcamera/pipeline/simple/simple.cpp >>> @@ -1216,6 +1216,9 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>> * Then pick, among the possible configuration for the pixel format, >>> * the smallest sensor resolution that can accommodate all streams >>> * without upscaling. >>> + * >>> + * If both packed and unpacked Bayer capture formats are support prefer s/support/supported/ >>> + * packed for lower memory usage and better software ISP performance. >>> */ >>> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >>> pipeConfig_ = nullptr; >>> @@ -1223,12 +1226,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >>> const Size &captureSize = pipeConfig->captureSize; >>> const Size &maxOutputSize = pipeConfig->outputSizes.max; >>> + bool curCfgPacked = false; >>> + bool newCfgPacked = false; How about using just a single boolean, and ... >>> + >>> + if (pipeConfig_) { >>> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >>> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >>> + /* Only compare packed vs unpacked if both formats are Bayer */ >>> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >>> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >>> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; ... evaluating `!curCfgPacked && newCfgPacked' here and ... >>> + } >>> + } >>> >>> if (maxOutputSize.width >= maxProcessedStreamSize.width && >>> maxOutputSize.height >= maxProcessedStreamSize.height && >>> captureSize.width >= maxRawStreamSize.width && >>> captureSize.height >= maxRawStreamSize.height) { >>> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >>> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >>> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) ... using the single boolean here? >>> pipeConfig_ = pipeConfig; >>> } >>
Hi, On 26-Oct-26 2:54 PM, Milan Zamazal wrote: > Hans de Goede <johannes.goede@oss.qualcomm.com> writes: > >> Hi Milan, >> >> On 26-Oct-26 12:36 PM, Milan Zamazal wrote: >>> Hi Hans, >>> >>> thank you for the patch. >>> >>> Hans de Goede <johannes.goede@oss.qualcomm.com> writes: >>> >>>> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >>>> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) >>>> formats. >>>> >>>> Before this change the simple-pipeline handler would pick whatever format >>>> was enumerated first as capture format in this case. Which on IPU6 boards >>>> comes down to picking unpacked. >>>> >>>> Packed formats use less memory and typically perform better due to reduced >>>> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >>>> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >>>> 3 consecutive benchmark runs using libcamera's software ISP's builtin >>>> Debayer benchmark gives the following results: >>>> >>>> Unpacked EGL us/frame: 6398 / 7393 / 7524 >>>> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >>>> >>>> Unpacked CPU us/frame: 7653 / 8147 / 8614 >>>> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >>>> >>>> Modify the simple pipeline handler to prefer the more performant CSI2 >>>> packed format when both formats are available. >>> >>> Please note that packed and unpacked fragment shaders use different >>> debayering algorithms. Packed uses sort of bilinear (despite what the >>> file header says), unpacked McGuire. While the memory argument sounds >>> plausible, the benchmarks cannot tell because of the different >>> algorithms used. In future, there should be a single benchmarking >>> algorithm. >>> >>> I think Bryan's multipass uses the same debayering algorithm for both, >>> so it'd be worth to re-measure there. >> >> Well, atm we cannot really do an apples 2 apples comparison for the EGL >> code-paths since there is no code to do so yet, at least I do not believe >> Bryan's code for this is really in a stable state ? > > I think the best what we currently have is > https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commit/6a69831fbdfd7601cc7b89c8006ed2ff907e2d58 > (see *packed*.frag there). I don't know whether the packed case can be > optimised but in the current form it has clearly more operations to > decode. Without measurements, we can't say what's faster, even less > generally faster. I'd test with the branch (0.7.0-multipass-v4) but I > don't have access to my environment ATM. Ok, I'll give this a test both with/without this patch on top and report results tomorrow. > >> >> Note that the CPU based comparison does show a significant win and >> typically we tend to be more performance constrained when falling >> back to the CPU. > > I can't say it doesn't matter, I have also platforms where CPU is faster > than GPU. But GPU is the primary platform and small differences are > relevant and more important than larger CPU differences. Which is a > reason itself to prefer one of the encodings and we need a patch like > this, the question is which one of the encodings to prefer (putting the > IPU6 problem aside). According to your benchmark, with current software > ISP, we should apply the patch as it is. > >> >>>> This also fixes a problem with unpacked formats on IPU6 where sometimes at >>>> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >>>> unpacked frame with high bits set which causes libcamera to crash with >>>> an out of bound array access as the software ISP code expects the high bits >>>> of unpacked formats to always be 0. >>> >>> I'm not sure how much significance such hardware problems should have >>> for selection of the coding, although yes, such a problem can be avoided >>> with packed. >> >> The alternatives are: >> >> 1. Live with the softISP sometimes crashing (unacceptable IMHO) >> >> 2. Masking out the high-bits in the memcpy path in CPU based debayer: >> >> https://patchwork.libcamera.org/patch/25007/ >> >> where you complained about a 4% performance degradation, while this patch is >> a win for CPU based decoding + as you point out that patch needs to be expanded >> to also cover the swstats only case used in the EGL paths, making those somewhat >> slower too. >> >> So even if the performance win for EGL here is limited or 0 when using the same >> debayer algo in both cases, this still seems like a better way to fix the crashes >> we are seeing then my initial masking approach. > > The crash workaround should be mentioned in a code comment, in case we > decide not to prefer packed in future for whatever reason. Ack, that is a good idea. Regards, Hans >>>> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >>>> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >>>> --- >>>> src/libcamera/pipeline/simple/simple.cpp | 18 +++++++++++++++++- >>>> 1 file changed, 17 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >>>> index 35c29ceca..3b59087ae 100644 >>>> --- a/src/libcamera/pipeline/simple/simple.cpp >>>> +++ b/src/libcamera/pipeline/simple/simple.cpp >>>> @@ -1216,6 +1216,9 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>>> * Then pick, among the possible configuration for the pixel format, >>>> * the smallest sensor resolution that can accommodate all streams >>>> * without upscaling. >>>> + * >>>> + * If both packed and unpacked Bayer capture formats are support prefer > > s/support/supported/ > >>>> + * packed for lower memory usage and better software ISP performance. >>>> */ >>>> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >>>> pipeConfig_ = nullptr; >>>> @@ -1223,12 +1226,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>>> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >>>> const Size &captureSize = pipeConfig->captureSize; >>>> const Size &maxOutputSize = pipeConfig->outputSizes.max; >>>> + bool curCfgPacked = false; >>>> + bool newCfgPacked = false; > > How about using just a single boolean, and ... > >>>> + >>>> + if (pipeConfig_) { >>>> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >>>> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >>>> + /* Only compare packed vs unpacked if both formats are Bayer */ >>>> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >>>> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >>>> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; > > ... evaluating `!curCfgPacked && newCfgPacked' here and ... > >>>> + } >>>> + } >>>> >>>> if (maxOutputSize.width >= maxProcessedStreamSize.width && >>>> maxOutputSize.height >= maxProcessedStreamSize.height && >>>> captureSize.width >= maxRawStreamSize.width && >>>> captureSize.height >= maxRawStreamSize.height) { >>>> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >>>> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >>>> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) > > ... using the single boolean here? > >>>> pipeConfig_ = pipeConfig; >>>> } >>> >
Hi, On 26-Oct-26 14:54, Milan Zamazal wrote: > Hans de Goede <johannes.goede@oss.qualcomm.com> writes: > >> Hi Milan, >> >> On 26-Oct-26 12:36 PM, Milan Zamazal wrote: >>> Hi Hans, >>> >>> thank you for the patch. >>> >>> Hans de Goede <johannes.goede@oss.qualcomm.com> writes: >>> >>>> Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats >>>> in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) >>>> formats. >>>> >>>> Before this change the simple-pipeline handler would pick whatever format >>>> was enumerated first as capture format in this case. Which on IPU6 boards >>>> comes down to picking unpacked. >>>> >>>> Packed formats use less memory and typically perform better due to reduced >>>> RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with >>>> i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, >>>> 3 consecutive benchmark runs using libcamera's software ISP's builtin >>>> Debayer benchmark gives the following results: >>>> >>>> Unpacked EGL us/frame: 6398 / 7393 / 7524 >>>> CSI2 pack EGL us/frame: 5682 / 5536 / 5634 >>>> >>>> Unpacked CPU us/frame: 7653 / 8147 / 8614 >>>> CSI2 pack CPU us/frame: 7183 / 7063 / 6318 >>>> >>>> Modify the simple pipeline handler to prefer the more performant CSI2 >>>> packed format when both formats are available. >>> >>> Please note that packed and unpacked fragment shaders use different >>> debayering algorithms. Packed uses sort of bilinear (despite what the >>> file header says), unpacked McGuire. While the memory argument sounds >>> plausible, the benchmarks cannot tell because of the different >>> algorithms used. In future, there should be a single benchmarking >>> algorithm. >>> >>> I think Bryan's multipass uses the same debayering algorithm for both, >>> so it'd be worth to re-measure there. >> >> Well, atm we cannot really do an apples 2 apples comparison for the EGL >> code-paths since there is no code to do so yet, at least I do not believe >> Bryan's code for this is really in a stable state ? > > I think the best what we currently have is > https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commit/6a69831fbdfd7601cc7b89c8006ed2ff907e2d58 > (see *packed*.frag there). I don't know whether the packed case can be > optimised but in the current form it has clearly more operations to > decode. Without measurements, we can't say what's faster, even less > generally faster. I'd test with the branch (0.7.0-multipass-v4) but I > don't have access to my environment ATM. Ok, so I've tried both the 0.7.0-multipass-v4 and 0.7.2-multipass-v6 branches but neither works on my Raptor Lake IPU6 test laptop. So for v2 I'll keep the old benchmark runs but I'll add a note this is not a straight comparison. >> Note that the CPU based comparison does show a significant win and >> typically we tend to be more performance constrained when falling >> back to the CPU. > > I can't say it doesn't matter, I have also platforms where CPU is faster > than GPU. But GPU is the primary platform and small differences are > relevant and more important than larger CPU differences. Which is a > reason itself to prefer one of the encodings and we need a patch like > this, the question is which one of the encodings to prefer (putting the > IPU6 problem aside). According to your benchmark, with current software > ISP, we should apply the patch as it is. > >> >>>> This also fixes a problem with unpacked formats on IPU6 where sometimes at >>>> the start of a stream the IPU6 isys CSI2 receiver may return a corrupt >>>> unpacked frame with high bits set which causes libcamera to crash with >>>> an out of bound array access as the software ISP code expects the high bits >>>> of unpacked formats to always be 0. >>> >>> I'm not sure how much significance such hardware problems should have >>> for selection of the coding, although yes, such a problem can be avoided >>> with packed. >> >> The alternatives are: >> >> 1. Live with the softISP sometimes crashing (unacceptable IMHO) >> >> 2. Masking out the high-bits in the memcpy path in CPU based debayer: >> >> https://patchwork.libcamera.org/patch/25007/ >> >> where you complained about a 4% performance degradation, while this patch is >> a win for CPU based decoding + as you point out that patch needs to be expanded >> to also cover the swstats only case used in the EGL paths, making those somewhat >> slower too. >> >> So even if the performance win for EGL here is limited or 0 when using the same >> debayer algo in both cases, this still seems like a better way to fix the crashes >> we are seeing then my initial masking approach. > > The crash workaround should be mentioned in a code comment, in case we > decide not to prefer packed in future for whatever reason. Ack. >>>> Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 >>>> Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> >>>> --- >>>> src/libcamera/pipeline/simple/simple.cpp | 18 +++++++++++++++++- >>>> 1 file changed, 17 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp >>>> index 35c29ceca..3b59087ae 100644 >>>> --- a/src/libcamera/pipeline/simple/simple.cpp >>>> +++ b/src/libcamera/pipeline/simple/simple.cpp >>>> @@ -1216,6 +1216,9 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>>> * Then pick, among the possible configuration for the pixel format, >>>> * the smallest sensor resolution that can accommodate all streams >>>> * without upscaling. >>>> + * >>>> + * If both packed and unpacked Bayer capture formats are support prefer > > s/support/supported/ Ack. > >>>> + * packed for lower memory usage and better software ISP performance. >>>> */ >>>> const SimpleCameraData::Configuration *maxPipeConfig = nullptr; >>>> pipeConfig_ = nullptr; >>>> @@ -1223,12 +1226,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() >>>> for (const SimpleCameraData::Configuration *pipeConfig : *configs) { >>>> const Size &captureSize = pipeConfig->captureSize; >>>> const Size &maxOutputSize = pipeConfig->outputSizes.max; >>>> + bool curCfgPacked = false; >>>> + bool newCfgPacked = false; > > How about using just a single boolean, and ... > >>>> + >>>> + if (pipeConfig_) { >>>> + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); >>>> + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); >>>> + /* Only compare packed vs unpacked if both formats are Bayer */ >>>> + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { >>>> + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; >>>> + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; > > ... evaluating `!curCfgPacked && newCfgPacked' here and ... > >>>> + } >>>> + } >>>> >>>> if (maxOutputSize.width >= maxProcessedStreamSize.width && >>>> maxOutputSize.height >= maxProcessedStreamSize.height && >>>> captureSize.width >= maxRawStreamSize.width && >>>> captureSize.height >= maxRawStreamSize.height) { >>>> - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) >>>> + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || >>>> + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) > > ... using the single boolean here? IMHO that makes it much harder to reason / see why we're preferring the new config over the previously selected one. Where as having the 2 bools separately makes it immediately clear when reading *just this if statement* that the new one being packed leads to it being preferred. So I'm going to keep this as is for v2. Regards, Hans
diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 35c29ceca..3b59087ae 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1216,6 +1216,9 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() * Then pick, among the possible configuration for the pixel format, * the smallest sensor resolution that can accommodate all streams * without upscaling. + * + * If both packed and unpacked Bayer capture formats are support prefer + * packed for lower memory usage and better software ISP performance. */ const SimpleCameraData::Configuration *maxPipeConfig = nullptr; pipeConfig_ = nullptr; @@ -1223,12 +1226,25 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() for (const SimpleCameraData::Configuration *pipeConfig : *configs) { const Size &captureSize = pipeConfig->captureSize; const Size &maxOutputSize = pipeConfig->outputSizes.max; + bool curCfgPacked = false; + bool newCfgPacked = false; + + if (pipeConfig_) { + BayerFormat curBayerFmt = BayerFormat::fromPixelFormat(pipeConfig_->captureFormat); + BayerFormat newBayerFmt = BayerFormat::fromPixelFormat(pipeConfig->captureFormat); + /* Only compare packed vs unpacked if both formats are Bayer */ + if (curBayerFmt.isValid() && newBayerFmt.isValid()) { + curCfgPacked = curBayerFmt.packing == BayerFormat::Packing::CSI2; + newCfgPacked = newBayerFmt.packing == BayerFormat::Packing::CSI2; + } + } if (maxOutputSize.width >= maxProcessedStreamSize.width && maxOutputSize.height >= maxProcessedStreamSize.height && captureSize.width >= maxRawStreamSize.width && captureSize.height >= maxRawStreamSize.height) { - if (!pipeConfig_ || captureSize < pipeConfig_->captureSize) + if (!pipeConfig_ || captureSize < pipeConfig_->captureSize || + (captureSize <= pipeConfig_->captureSize && !curCfgPacked && newCfgPacked)) pipeConfig_ = pipeConfig; }
Some CSI2 receivers like the IPU6 isys can store e.g. 10 bpp Bayer formats in either CSI2 packed (5 bytes / 4 pixels) or unpacked (2 bytes / pixel) formats. Before this change the simple-pipeline handler would pick whatever format was enumerated first as capture format in this case. Which on IPU6 boards comes down to picking unpacked. Packed formats use less memory and typically perform better due to reduced RAM bandwidth requirements. E.g. on a Lenovo ThinkPad X1 Yoga Gen 8 with i7-1370P CPU and a ov2740 sensor running at full sensor (FHD) resolution, 3 consecutive benchmark runs using libcamera's software ISP's builtin Debayer benchmark gives the following results: Unpacked EGL us/frame: 6398 / 7393 / 7524 CSI2 pack EGL us/frame: 5682 / 5536 / 5634 Unpacked CPU us/frame: 7653 / 8147 / 8614 CSI2 pack CPU us/frame: 7183 / 7063 / 6318 Modify the simple pipeline handler to prefer the more performant CSI2 packed format when both formats are available. This also fixes a problem with unpacked formats on IPU6 where sometimes at the start of a stream the IPU6 isys CSI2 receiver may return a corrupt unpacked frame with high bits set which causes libcamera to crash with an out of bound array access as the software ISP code expects the high bits of unpacked formats to always be 0. Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/328 Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com> --- src/libcamera/pipeline/simple/simple.cpp | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)