[libcamera-devel,RFC,0/1] Autofocus controls

15152 mbox series

Hi everyone

I thought it was time for a slightly more formal API proposal for AF
(autofocus).

I've sent a patch listing all the new controls that I think we would
need. These controls cover both the use-cases that I can imagine we
(Raspberry Pi) would expect, and I think it covers all the Android
use-cases too. Nonetheless, I think a few words of additional
discussion and questions are warranted.


1. AfState

This closely follows what Android does, only Android duplicates them
all, once for "passive" (aka. continuous) AF and once for "active"
(user-triggered) AF. I can't really see why duplicating them would be
necessary, it seems pretty clear to me what mode we're in, but of
course we could if there is a good reason.

I wondered a bit about having an "Error" state. Maybe that would get
reported if the AfWindow rectangles are ill-defined?

2. AfWindow

Allowing applications to set the AF windows seems required. For
example, you might be running some kind of object or face detector,
and then you want to feed those results back to the AF algorithm.

I was unsure what units to use for the Rectangles. It seemed to me
that you'd want the rectangles automatically to follow whatever
ScalerCrop you are using. For example, doing digital zoom and leaving
the focus windows in a part of the image you can't see seems
particularly unhelpful!

So anyway, I went with the units being "a proportion of the ScalerCrop
region". Can we think of something better? Would floating point
Rectangles be less annoying?

3. AfMode

As you probably already know, I'm not really seeing the need for an
"off" mode as it seems the same as "auto" to me - so I didn't define
one.

One thing I wondered about was "pausing" CAF (continuous AF). You
might want to let it run, then "pause" for a moment to take some
captures, and then let it re-start from where it was previously. How
to accomplish this?

We could have a separate "pause" control, just for CAF. Or I've
suggested that you could set the mode back to "auto" which would have
the same effect. But then how would you go back to CAF "from where it
left off" as opposed to letting it re-start as though it wasn't
running before? Maybe the "pause" control isn't such a bad idea.

4. LensPosition

I've proposed using the lens driver's units for controlling the lens
position. It's easy and naturally mimics what we tend to do for
exposure/gain.

Another option would be to go a bit more Android-y and insist that
zero means "infinity". Android seems to categorise lenses into
"calibrated" and "uncalibrated" types. For the latter type, it looks
like pretty much the only stipulation is that zero means
"infinity". For "calibrated" lenses the units are formally defined to
be dioptres, but I'm thinking that we don't need to go there right
now.

There's also the question of exactly where we get the lens range from.
It will be different for each module, even ones using the same lens
and/or driver - it will in each case be a different subset of the
range that the driver advertises. But I think that's not a question
for this control API.


Hopefully all that's enough to start taking this forward. There are of
course lots of points of detail to think about here, but clearly the
"big picture" is most important in the first instance.

Thanks!

David

David Plowman (1):
  libcamera: controls: Controls for driving AF (autofocus) algorithms

 src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------
 1 file changed, 167 insertions(+), 60 deletions(-)

Hi David, Thanks for the work!


On Mon, Dec 13, 2021 at 11:22 PM David Plowman
<david.plowman@raspberrypi.com> wrote:
>
> Hi everyone
>
> I thought it was time for a slightly more formal API proposal for AF
> (autofocus).
>
> I've sent a patch listing all the new controls that I think we would
> need. These controls cover both the use-cases that I can imagine we
> (Raspberry Pi) would expect, and I think it covers all the Android
> use-cases too. Nonetheless, I think a few words of additional
> discussion and questions are warranted.
>
>
> 1. AfState
>
> This closely follows what Android does, only Android duplicates them
> all, once for "passive" (aka. continuous) AF and once for "active"
> (user-triggered) AF. I can't really see why duplicating them would be
> necessary, it seems pretty clear to me what mode we're in, but of
> course we could if there is a good reason.
I think it's related to the "pause" control mentioned below.

>
> I wondered a bit about having an "Error" state. Maybe that would get
> reported if the AfWindow rectangles are ill-defined?
>
> 2. AfWindow
>
> Allowing applications to set the AF windows seems required. For
> example, you might be running some kind of object or face detector,
> and then you want to feed those results back to the AF algorithm.
>
> I was unsure what units to use for the Rectangles. It seemed to me
> that you'd want the rectangles automatically to follow whatever
> ScalerCrop you are using. For example, doing digital zoom and leaving
> the focus windows in a part of the image you can't see seems
> particularly unhelpful!
>
> So anyway, I went with the units being "a proportion of the ScalerCrop
> region". Can we think of something better? Would floating point
> Rectangles be less annoying?
>
> 3. AfMode
>
> As you probably already know, I'm not really seeing the need for an
> "off" mode as it seems the same as "auto" to me - so I didn't define
> one.
>
> One thing I wondered about was "pausing" CAF (continuous AF). You
> might want to let it run, then "pause" for a moment to take some
> captures, and then let it re-start from where it was previously. How
> to accomplish this?
>
> We could have a separate "pause" control, just for CAF. Or I've
> suggested that you could set the mode back to "auto" which would have
> the same effect. But then how would you go back to CAF "from where it
> left off" as opposed to letting it re-start as though it wasn't
> running before? Maybe the "pause" control isn't such a bad idea.

I would vote for the "pause" control. I tried to map the combination
of pause and AfState with Android's definition.
It describes the "passive" cases.

PASSIVE_SCAN = CAF + AfStateScanning
PASSIVE_FOCUSED = CAF + AfStateFocused
PASSIVE_UNFOCUSED = CAF + AfStateFailed

ACTIVE_SCAN = CAF + AfStateScanning + pause on
FOCUSED_LOCKED = CAF + AfStateFocused + paused
NOT_FOCUSED_LOCKED = CAF + AfStateFailed + paused

The difference between PASSIVE_FOCUSED and FOCUSED_LOCKED are:
PASSIVE_FOCUSED tells the user "Although it's focused, it may change
anytime due to CAF".
FOCUSED_LOCKED tell the user "It's focused, and won't change unless
the user stop pausing even in CAF mode"

Android use the combination to describe "pause":
CAF + TriggerStart = pause on
CAF + TriggerCancel = pause off

I notice that the TriggerStart is forbidden for CAF in the Patch, but
I think maybe "pause" is a more clear definition for CAF.
And the user doesn't need to do the trick by changing modes.

Another thing I'd like to notice is that:
"CAF + TriggerStart" in Android is not going to abruptly "pause" if
CAF is already scanning, but wait for the scan to finish, and then
pause.

>
> 4. LensPosition
>
> I've proposed using the lens driver's units for controlling the lens
> position. It's easy and naturally mimics what we tend to do for
> exposure/gain.
>
> Another option would be to go a bit more Android-y and insist that
> zero means "infinity". Android seems to categorise lenses into
> "calibrated" and "uncalibrated" types. For the latter type, it looks
> like pretty much the only stipulation is that zero means
> "infinity". For "calibrated" lenses the units are formally defined to
> be dioptres, but I'm thinking that we don't need to go there right
> now.
>
> There's also the question of exactly where we get the lens range from.
> It will be different for each module, even ones using the same lens
> and/or driver - it will in each case be a different subset of the
> range that the driver advertises. But I think that's not a question
> for this control API.
>
>
> Hopefully all that's enough to start taking this forward. There are of
> course lots of points of detail to think about here, but clearly the
> "big picture" is most important in the first instance.
>
> Thanks!
>
> David
>
> David Plowman (1):
>   libcamera: controls: Controls for driving AF (autofocus) algorithms
>
>  src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------
>  1 file changed, 167 insertions(+), 60 deletions(-)
>
> --
> 2.30.2
>