[libcamera-devel] Suboptimal resolution for digital zoom (controls::ScalerCrop)

18547 diff mbox series

Hi,

I'd like to squeeze out as much resolution as possible from my
Raspberry Pi camera v1
for scanning QR codes. The camera  has a fixed focus at ~50cm from the
lens, so the
resolution gain from moving the QR codes closer is lost to blur from
the loss of focus.

My current strategy is to scan codes at a distance, but use the
considerable resolution
of the sensor (~2500x1900) to gain enough resolution for successful scans.

However, I fail to find any way to achieve that with libcamera. I
tried three options, all
suboptimal:

1. Configure the camera for full resolution and crop to a smaller
section in my program.
That works surprisingly well, except that the white-balance etc.
algorithms run on the full
image and not just the cropped part.
2. Configure the camera for full resolution and set
controls::ScalerCrop to the area
of interest. This seems to fix the white-balance algorithms, but
results in the cropped
area being *upscaled* to full resolution when I receive them. Scanning
for QR codes
at full resolution is way too slow on a RPi Zero.
3. Configure the camera for the same resolution of the area of interest, and set
controls::ScalerCrop. This is efficient, but the sensor is configured
for the low
resolution and thus ruins the resolution gain.

Questions:
1. Is there another way?
2. The following hack works for me,

findBestFormat(data->sensorFormats_, Size{2592,1944}, bitDepth);
        const RPiCameraConfiguration *rpiConfig = static_cast<const
RPiCameraConfiguration *>(config);
        ret = data->sensor_->setFormat(&sensorFormat,
rpiConfig->combinedTransform_);
        if (ret)

Would a similar change that *always* uses the full sensor resolution
be acceptable?
I can't discern any performance issues between full resolution and
lower resolution.

3. If the above fix is unacceptable, what is the right way to fix this
issue in libcamera?

Thanks,
Elias

On Tue, 25 Apr 2023 at 05:24, David Plowman
<david.plowman@raspberrypi.com> wrote:
>
> Hi
>
> It should also be possible to request a raw stream at the full sensor
> resolution, but set the output size to the crop that you really want.
> Initially you'll obviously find that the full image is being
> downscaled to your output resolution. But then set the ScalerCrop to
> match the output size that you asked for, and you should be able to
> get a 1:1 crop from the full resolution image. Does that help?
>
> David

This sounds similar to Laurent's idea below. It worked, thank you.
For future reference, I spent some time figuring out that my dummy raw
stream needed two properties to function as a sensor resolution setting,
only:

  rawConfig.colorSpace = ColorSpace::Raw;
  rawConfig.bufferCount = 0;

Without the colorSpace setting, Camera::configure fails even with a prior
CameraConfiguration::validate. Without the bufferCount setting, you
won't get any request callbacks.

>
> On Tue, 25 Apr 2023 at 07:34, Laurent Pinchart via libcamera-devel
> <libcamera-devel@lists.libcamera.org> wrote:
> >
> > Hi Elias,
> >
> > On Mon, Apr 24, 2023 at 08:04:22PM -0600, Elias Naur via libcamera-devel wrote:
> > > Hi,
> > >
> > > I'd like to squeeze out as much resolution as possible from my Raspberry Pi camera v1
> > > for scanning QR codes. The camera  has a fixed focus at ~50cm from the lens, so the
> > > resolution gain from moving the QR codes closer is lost to blur from the loss of focus.
> > >
> > > My current strategy is to scan codes at a distance, but use the considerable resolution
> > > of the sensor (~2500x1900) to gain enough resolution for successful scans.
> > >
> > > However, I fail to find any way to achieve that with libcamera. I tried three options, all
> > > suboptimal:
> > >
> > > 1. Configure the camera for full resolution and crop to a smaller section in my program.
> > > That works surprisingly well, except that the white-balance etc. algorithms run on the full
> > > image and not just the cropped part.
> > > 2. Configure the camera for full resolution and set controls::ScalerCrop to the area
> > > of interest. This seems to fix the white-balance algorithms, but results in the cropped
> > > area being *upscaled* to full resolution when I receive them. Scanning for QR codes
> > > at full resolution is way too slow on a RPi Zero.
> > > 3. Configure the camera for the same resolution of the area of interest, and set
> > > controls::ScalerCrop. This is efficient, but the sensor is configured for the low
> > > resolution and thus ruins the resolution gain.
> > >
> > > Questions:
> > > 1. Is there another way?
> >
> > How about capturing full resolution frames, without cropping in the
> > capture pipeline, and scanning for the QR code in the area of interest
> > only ?
> >

I believe this is my option 1. above. It works, except that auto-white balance
etc. runs on the full field of view and not just the cropped subimage.

> > > 2. The following hack works for me,
> > >
> > > diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > index 00600441..6ed81dc1 100644
> > > --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > @@ -842,7 +842,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
> > >          * request, and apply it to the sensor with the cached transform, if
> > >          * any.
> > >          */
> > > -       V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize, bitDepth);
> > > +       V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, Size{2592,1944}, bitDepth);
> > >         const RPiCameraConfiguration *rpiConfig = static_cast<const RPiCameraConfiguration *>(config);
> > >         ret = data->sensor_->setFormat(&sensorFormat, rpiConfig->combinedTransform_);
> > >         if (ret)
> > >
> > > Would a similar change that *always* uses the full sensor resolution be acceptable?
> > > I can't discern any performance issues between full resolution and lower resolution.
> >
> > You can't hardcode a specific resolution, as this wouldn't work properly
> > with other sensors. Always selecting the sensor full resolution is also
> > not acceptable, as that will limit the frame rate on high resolution
> > sensors (e.g. the Raspberry Pi HQ camera will be limited to 15fps).
> >

I thought so.

> > > 3. If the above fix is unacceptable, what is the right way to fix this
> > > issue in libcamera?
> >
> > The current way to select the sensor resolution is a bit of a hack. You
> > need to add a raw stream to your camera configuration, set to the full
> > sensor resolution. You don't need to submit buffers for that stream when
> > queuing requests, so you can just capture from the processed YUV stream
> > as you do now.
> >
> > We will likely implement in the future an easier API to select the
> > sensor resolution.
> >

This idea worked, thank you! As you may have expected it took me a
while to figure out
the right incantations, so a more intuitive way of setting the sensor resolution
would be greatly appreciated.

Elias