| Message ID | 20240808102346.13065-1-naush@raspberrypi.com |
|---|---|
| Headers | show |
| Series |
|
| Related | show |
Hi all, Gentle ping for any feedback on this please? Thanks, Naush On Thu, 8 Aug 2024 at 11:23, Naushir Patuck <naush@raspberrypi.com> wrote: > > Hi, > > This patch series adds preliminary support for setting individual crop > rectangles for each ISP output stream. The current ScalerCrop control takes a > single rectangle and applies it to all output streams. In this series, we define > a new rpi::ScalerCrops vendor control that takes a span of rectangles, one for > each output stream. The behaviour of the existing ScalerCrop control remains > unchanged. > > One example of the use of this feature is when generating low resolution images > for AI inferencing. Typically, vision neural network models are trained on > square dimension image with 1:1 aspect ratio (e.g. 640x640 or 320x320). If an > application requests, say, a 1920x1080 (16:9) output stream + 640x640 (1:1) > inference stream, then the latter images end up squished and distorted. This > affects the performance of the neural network. By allowing separate crops on > each stream, the inference stream can preserve the pixel aspect ratio for any > requested resolution. > > Note that VC4 based platforms cannot apply different crops to each output, > whereas the PiSP/Pi5 platform can. However, all the infrastructure for this > change lives in the core pipeline handler shared by both platforms. > > Thanks, > Naush > > Naushir Patuck (7): > controls: rpi: Add a vendor rpi::ScalerCrops control > pipeline: rpi: Remove CameraData::scalerCrop_ > pipeline: rpi: Pass crop rectangle as a parameter to > platformSetIspCrop() > pipeline: rpi: Introduce CameraData::CropParams > pipeline: rpi: Track which ISP output is configured for a stream > pipeline: rpi: Pass ISP output index into platformSetIspCrop() > pipeline: rpi: Handler controls::rpi::ScalerCrops > > src/ipa/rpi/common/ipa_base.cpp | 2 + > src/libcamera/control_ids_rpi.yaml | 15 ++++ > .../pipeline/rpi/common/pipeline_base.cpp | 84 +++++++++++++------ > .../pipeline/rpi/common/pipeline_base.h | 26 +++++- > src/libcamera/pipeline/rpi/vc4/vc4.cpp | 18 ++-- > 5 files changed, 112 insertions(+), 33 deletions(-) > > -- > 2.34.1 >
Hi, This patch series adds preliminary support for setting individual crop rectangles for each ISP output stream. The current ScalerCrop control takes a single rectangle and applies it to all output streams. In this series, we define a new rpi::ScalerCrops vendor control that takes a span of rectangles, one for each output stream. The behaviour of the existing ScalerCrop control remains unchanged. One example of the use of this feature is when generating low resolution images for AI inferencing. Typically, vision neural network models are trained on square dimension image with 1:1 aspect ratio (e.g. 640x640 or 320x320). If an application requests, say, a 1920x1080 (16:9) output stream + 640x640 (1:1) inference stream, then the latter images end up squished and distorted. This affects the performance of the neural network. By allowing separate crops on each stream, the inference stream can preserve the pixel aspect ratio for any requested resolution. Note that VC4 based platforms cannot apply different crops to each output, whereas the PiSP/Pi5 platform can. However, all the infrastructure for this change lives in the core pipeline handler shared by both platforms. Thanks, Naush Naushir Patuck (7): controls: rpi: Add a vendor rpi::ScalerCrops control pipeline: rpi: Remove CameraData::scalerCrop_ pipeline: rpi: Pass crop rectangle as a parameter to platformSetIspCrop() pipeline: rpi: Introduce CameraData::CropParams pipeline: rpi: Track which ISP output is configured for a stream pipeline: rpi: Pass ISP output index into platformSetIspCrop() pipeline: rpi: Handler controls::rpi::ScalerCrops src/ipa/rpi/common/ipa_base.cpp | 2 + src/libcamera/control_ids_rpi.yaml | 15 ++++ .../pipeline/rpi/common/pipeline_base.cpp | 84 +++++++++++++------ .../pipeline/rpi/common/pipeline_base.h | 26 +++++- src/libcamera/pipeline/rpi/vc4/vc4.cpp | 18 ++-- 5 files changed, 112 insertions(+), 33 deletions(-)