[RFC,0/1] Fix softISP crash on 10/12bpp sparse input frames
mbox series

Message ID 20251112090924.46295-1-johannes.goede@oss.qualcomm.com
Headers show
Series
  • Fix softISP crash on 10/12bpp sparse input frames
Related show

Message

Hans de Goede Nov. 12, 2025, 9:09 a.m. UTC 
Hi All,

As reported here https://bugzilla.redhat.com/show_bug.cgi?id=2402746#c20
there are several places where the swstats / debayer CPU code will
do out of bounds array accesses when processing a corrupt input frame
in 10/12 bpp sparse format. The issue is that these corrupt frames
which store 10 or 12 bpp pixels in 16 bit words may have the high
bits set leading to e.g. pixel values > 1023 for 10 bpp input data,
which in turn leads to out of bounds array accesses.

Here are 2 example backtraces:

#4  0x00007fba4900b084 in std::__glibcxx_assert_fail (file=file@entry=0x7fba495b7344 "/usr/include/c++/15/array", line=line@entry=210, 
    function=function@entry=0x7fba495b5d68 "constexpr std::array<_Tp, _Nm>::value_type& std::array<_Tp, _Nm>::operator[](size_type) [with _Tp = unsigned int; long unsigned int _Nm = 64; reference = unsigned int&; size_type = long unsigned int]", condition=condition@entry=0x7fba495b6df2 "__n < this->size()") at ../../../../../libstdc++-v3/src/c++11/assert_fail.cc:41
#5  0x00007fba49423d15 in std::array<unsigned int, 64ul>::operator[] (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/15/array:210
#6  0x00007fba49423d1b in std::array<unsigned int, 64ul>::operator[] (this=<optimized out>, __n=<optimized out>) at /usr/include/c++/15/array:210
#7  libcamera::SwStatsCpu::statsBGGR10Line0 (this=<optimized out>, src=<optimized out>) at ../src/libcamera/software_isp/swstats_cpu.cpp:219
#8  0x00007fba4951d13b in libcamera::SwStatsCpu::processLine0 (frame=0, y=0, this=<optimized out>, src=0x7fba427f89c0) at ../src/libcamera/software_isp/swstats_cpu.h:63
#9  libcamera::SwStatsCpu::processLine0 (this=<optimized out>, frame=0, y=0, src=0x7fba427f89c0) at ../src/libcamera/software_isp/swstats_cpu.h:54
#10 libcamera::DebayerCpu::process2 (this=this@entry=0x7fba38053c20, frame=frame@entry=0, src=0x7fba48089d08 <error: Cannot access memory at address 0x7fba48089d08>, 

Where swstats_cpu.cpp:219 points SWSTATS_ACCUMULATE_LINE_STATS() which
accesses the yHistogram array.

#4  0x00007f936ca0b084 in std::__glibcxx_assert_fail (file=file@entry=0x7f936cfb727c "/usr/include/c++/15/array", 
    line=line@entry=210, 
    function=function@entry=0x7f936cfb5760 "constexpr std::array<_Tp, _Nm>::value_type& std::array<_Tp, _Nm>::operator[](size_type) [with _Tp = unsigned char; long unsigned int _Nm = 256; reference = unsigned char&; size_type = long unsigned in"..., condition=condition@entry=0x7f936cfb6d2a "__n < this->size()") at ../../../../../libstdc++-v3/src/c++11/assert_fail.cc:41
#5  0x00007f936ce22fd3 in std::array<unsigned char, 256ul>::operator[] (__n=<optimized out>, this=<optimized out>)
    at /usr/include/c++/15/array:210
#6  0x00007f936cf21102 in std::array<unsigned char, 256ul>::operator[] (this=<optimized out>, __n=<optimized out>)
    at ../src/libcamera/software_isp/debayer_cpu.cpp:158
#7  libcamera::DebayerCpu::debayer10_BGBG_BGR888<true, false> (this=<optimized out>, dst=<optimized out>, 
    src=<optimized out>) at ../src/libcamera/software_isp/debayer_cpu.cpp:164

Where debayer_cpu.cpp:164 ends up calling the STORE_PIXEL() macro which
accesses various lookup tables.

The single patch in this series should fix this. This is marked as a RFC
for now because I'm waiting on testing feedback from the reporter.

Regards,

Hans


Hans de Goede (1):
  libcamera: debayer_cpu: Mask out unused bits from > 8bpp non packed
    src data

 src/libcamera/software_isp/debayer_cpu.cpp | 41 ++++++++++++++++++++--
 src/libcamera/software_isp/debayer_cpu.h   |  6 ++++
 2 files changed, 45 insertions(+), 2 deletions(-)