[libcamera-devel,v3,1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms

15689 diff mbox series

This patch describes a series of controls that allow applications to
drive AF algorithms:

AfMode - manual, auto or continuous
AfRange - full, macro or normal
AfSpeed - fast or slow
AfWindows - AF window locations
AfTrigger - start (trigger) an AF scan or cancel
AfPause - pause continuous AF
LensPosition - set or retrieve position of lens
AfState - reports whether scanning/success/failure
AfPauseState - reports whether continuous AF paused or not

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
---
 src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
 1 file changed, 258 insertions(+), 60 deletions(-)

Hi David,
  thanks for this new version

On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> This patch describes a series of controls that allow applications to
> drive AF algorithms:
>
> AfMode - manual, auto or continuous
> AfRange - full, macro or normal
> AfSpeed - fast or slow
> AfWindows - AF window locations
> AfTrigger - start (trigger) an AF scan or cancel
> AfPause - pause continuous AF
> LensPosition - set or retrieve position of lens
> AfState - reports whether scanning/success/failure
> AfPauseState - reports whether continuous AF paused or not
>
> Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> ---
>  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
>  1 file changed, 258 insertions(+), 60 deletions(-)
>
> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> index 9d4638ae..eccf18bc 100644
> --- a/src/libcamera/control_ids.yaml
> +++ b/src/libcamera/control_ids.yaml
> @@ -381,6 +381,264 @@ controls:
>          \todo Define how the sensor timestamp has to be used in the reprocessing
>          use case.
>
> +  - AfMode:
> +      type: int32_t
> +      description: |
> +        Control to set the mode of the AF (autofocus) algorithm. Applications
> +        are allowed to set a new mode, and to send additional controls for
> +        that new mode, in the same request.
> +
> +        An implementation may choose not to implement all the modes.
> +
> +      enum:
> +        - name: AfModeManual
> +          value: 0
> +          description: |
> +            The AF algorithm is in manual mode. In this mode it will never
> +            perform any action nor move the lens of its own accord, but an
> +            application can set controls to move the lens.
> +

If, as I understand it, the lens can only be moved with 'LensPosition'
when in manual mode, I would say so explicitly

               "... but an application can specify the desired lens
               position using the LensPosition control."

> +            In this mode the AfState will always report AfStateReset.

I'm tempted to say AfState is useless and can not be reported at all
in this case. For simplicity (for ph/ipa) I'm fine reporting it
regardless, even if seeing 'reset' might be slightly confusing. I
don't have better alternatives to propose though.

Reading the AfStateReset description

        - name: AfStateReset
          value: 0
          description: |
            The AF algorithm is in manual mode (AfModeManual) or in auto mode
            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
            scan was cancelled.

I wonder if this isn't better named AfStateIdle


> +        - name: AfModeAuto
> +          value: 1
> +          description: |
> +            The AF algorithm is in auto mode. This means that the algorithm
> +            will never move the lens or change state unless the AfTrigger
> +            control is used. The AfTrigger control can be used to initiate a
> +            focus scan, the results of which will be reported by AfState.
> +
> +            If the autofocus algorithm is moved from AfModeAuto to another
> +            mode while a scan is in progress, the scan is cancelled
> +            immediately, without waiting for the scan to finish.
> +
> +            When first entering this mode the AfState will report
> +            AfStateReset. When a trigger control is sent, AfState will
> +            report AfStateScanning for a period before spontaneously
> +            changing to AfStateFocused or AfStateFailed, depending on
> +            the outcome of the scan. It will remain in this state until
> +            another scan is initiated by the AfTrigger control. If a scan is
> +            cancelled (without changing to another mode), AfState will return
> +            to AfStateReset.
> +        - name: AfModeContinuous
> +          value: 2
> +          description: |
> +            The AF algorithm is in continuous mode. This means that the lens
> +            can re-start a scan spontaneously at any moment, without any user
> +            intervention. The AfState still reports whether the algorithm is
> +            currently scanning or not, though the application has no ability
> +            to initiate or cancel scans (though it can "pause" them), nor to
> +            move the lens for itself.
> +
> +            When set to AfModeContinuous, the system will immediately initiate
> +            a scan so AfState will report AfStateScanning, and will settle on
> +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> +
> +  - AfRange:
> +      type: int32_t
> +      description: |
> +        Control to set the range of focus distances that is scanned. An
> +        implementation may choose not to implement all the options here.
> +      enum:
> +        - name: AfRangeNormal
> +          value: 0
> +          description: |
> +            A wide range of focus distances is scanned, all the way from
> +            infinity down to close distances, though depending on the
> +            implementation, possibly not including the very closest macro
> +            positions.
> +        - name: AfRangeMacro
> +          value: 1
> +          description: Only close distances are scanned.
> +        - name: AfRangeFull
> +          value: 2
> +          description: |
> +            The full range of focus distances is scanned just as with
> +            AfRangeNormal but this time including the very closest macro
> +            positions.
> +
> +  - AfSpeed:
> +      type: int32_t
> +      description: |
> +        Control that determines whether the AF algorithm is to move the lens
> +        as quickly as possible or more steadily. For example, during video
> +        recording it may be desirable not to move the lens too abruptly, but
> +        when in a preview mode (waiting for a still capture) it may be
> +        helpful to move the lens as quickly as is reasonably possible.
> +      enum:
> +        - name: AfSpeedNormal
> +          value: 0
> +          description: Move the lens at its usual speed.
> +        - name: AfSpeedFast
> +          value: 1
> +          description: Move the lens more quickly.
> +
> +  - AfWindows:
> +      type: Rectangle
> +      description: |
> +        Sets the focus windows used by the AF algorithm. The units used express
> +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> +        image), as u0.16 format numbers.

I don't like too much the "or if unavailable", as it means that the
reference rectangle will change depending on the platform
capabilities.

I understand your argument in reply to Jean-Michel comment on the
previous version of the series (quoted here for reference)

"The thing that bothers me a bit about the full pixel array is that, if
you use digital zoom, then you will have to update your AfWindows as
they might otherwise lie outside your final image! If it's defined as
being a region of the scaler crop, then it automatically stays the
same (to the application, at least, the pipeline handler probably has
some work to do).

I note that the Android link you give below says "Focus areas are
relative to the current field of view (getZoom())", which I guess is
partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
is easier to work with than the u0.16 format that I specified?"

Let's reason on the IPA point of view. If the AF algorithm works by
estimating the contrast, will it do so before or after the ScalerCrop
rectangle is applied ? If a sensor provides PDAF statistics, will it
do so on the full pixel array size, the portion of if that gets
processed or on the final image ?

Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
rectangle, so that they are applied to the actual FOV before being
passed to the AF algorithm ?

One last point about the unit: is the below representation correct,
when it comes to use a Rectangle to represent a proportion ?
Regardless of the reference rectangle, if we have a 4x3 grid, will the
pipeline report 12 rectangles like the ones below, for applications to
chose from ?

      (0,0)
        +-----+-----+-----+-----+
        |     |     |     |     |
        +-----+-----+-----+-----+
        |     |     |     |     |
        +-----+-----+-----+-----+
        |     |     |     |     |
        +-----+-----+-----+-----+

AfWindows = {
    {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
    {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
    {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
};

My understanding is that the grid is fixed and depends on the platform
capabilities. I wonder if we shouldn't allow applications to select
grids by index.

Also, do we need a property to report the grid structure and allows
application to select them from, or do you see this conveyed through a
ControlInfo ?


> +
> +        In order to be activated, a rectangle must be programmed with non-zero
> +        width and height. If no rectangles are programmed in this way, then the
> +        system will choose its own single default window in the centre of the
> +        image.
> +
> +        The details of how the windows are used are platform dependent. We note
> +        that when there is more than one AF window, a typical implementation
> +        might find the optimal focus position for each one and finally select
> +        the window closest to the camera.

What window is more close to the camera ? My understanding is that
windows apply to the sensor produced frame, hence, when it comes to distance,
they all sit on the same focal plan :)

> +
> +        size: [n]
> +
> +  - AfTrigger:
> +      type: int32_t
> +      description: |
> +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> +        and can also be used to terminate a scan early.
> +
> +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> +
> +      enum:
> +        - name: AfTriggerStart
> +          value: 0
> +          description: Start an AF scan. Ignored if a scan is in progress.
> +        - name: AfTriggerCancel
> +          value: 1
> +          description: Cancel an AF scan. This does not cause the lens to move
> +            anywhere else. Ignored if no scan is in progress.
> +
> +  - AfPause:
> +      type: int32_t
> +      description: |
> +        This control has no effect except when in continuous autofocus mode
> +        (AfModeContinuous). It can be used to pause any lens movements while
> +        (for example) images are captured. The algorithm remains inactive
> +        until it is instructed to resume.
> +
> +      enum:
> +        - name: AfPauseImmediate
> +          value: 0
> +          description: |
> +            Pause the continuous autofocus algorithm immediately, whether or
> +            not any kind of scan is underway. AfPauseState will subsequently
> +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> +            when it received this control.
> +        - name: AfPauseDeferred
> +          value: 1
> +          description: |
> +            This is similar to AfPauseImmediate, and if the AfState is currently
> +            reporting AfStateFocused or AfStateFailed it will remain in that
> +            state and AfPauseState will report AfPauseStatePaused.
> +
> +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> +            will report AfPauseStatePausing until the scan is finished, at which
> +            point AfState will report one of AfStateFocused or AfStateFailed, and
> +            AfPauseState will change to AfPauseStatePaused.

From here on I see lines getting over 80 cols by a few chars. Can we
align it back ?

> +        - name: AfPauseResume
> +          value: 2
> +          description: |
> +            Resume continuous autofocus operation. The algorithm starts again
> +            from exactly where it left off, and AfPauseState will report
> +            AfPauseStateRunning.
> +
> +  - LensPosition:
> +      type: float
> +      description: |
> +        Acts as a control to instruct the lens to move to a particular position
> +        and also reports back the position of the lens for each frame.
> +
> +        The LensPosition control is ignored unless the AfMode is set to
> +        AfModeManual.
> +
> +        The units are dioptres divided by the hyperfocal distance. Non-integer
> +        values are permitted. For example:
> +        0 moves the lens to infinity.
> +        0.5 moves the lens to twice the hyperfocal distance.
> +        1 moves the lens to the hyperfocal position.
> +        And larger values will focus the lens ever closer.
> +

This seems good to me, albeit my understanding of the issue is limited :)

> +  - AfState:
> +      type: int32_t
> +      description: |
> +        Reports the current state of the AF algorithm in conjunction with the
> +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> +        The possible state changes are described below, though we note the following
> +        state transitions that occur when the AfMode is changed.
> +
> +        If the AfMode is set to AfModeManual then the AfState will report
> +        AfStateReset. This action on its own does not cause the lens to move; the
> +        LensPosition control must subsequently be used to achieve this.
> +
> +        If the AfMode is set to AfModeAuto then the AfState will report
> +        AfStateReset, unless AfTriggerStart is sent at the same time in which

s/unless/until ?
s/at the same time// ?

> +        case it will (start a scan and) report AfStateScanning.
> +
> +        If the AfMode is set to AfModeContinuous then the AfState will initially
> +        report AfStateScanning.
> +
> +      enum:
> +        - name: AfStateReset
> +          value: 0
> +          description: |
> +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> +            scan was cancelled.
> +        - name: AfStateScanning
> +          value: 1
> +          description: |
> +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> +            started using the AfTrigger control. The scan can be cancelled by
> +            sending AfTriggerCancel at which point the algorithm will either
> +            move back to AfStateReset or, if the scan actually completes
> +            before the cancel request is processed, to one of
> +            AfStateFocused or AfStateFailed.
> +
> +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)

s/be be/is ?

> +            at which point it may enter this state spontaneously whenever it
> +            determines that a rescan is needed.
> +        - name: AfStateFocused
> +          value: 2
> +          description: |
> +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> +            mode and a scan has completed with the result that the algorithm believes
> +            the image is now in focus.
> +        - name: AfStateFailed
> +          value: 3
> +          description: |
> +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> +            mode and a scan has completed with the result that the algorithm did not
> +            find a good focus position.
> +
> +  - AfPauseState:
> +      type: int32_t
> +      description: |
> +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> +        the algorithm is currently running, paused or pausing (that is, will pause
> +        as soon as any in-progress scan completes).
> +
> +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> +
> +      enum:
> +        - name: AfPauseStateRunning
> +          value: 0
> +          description: |
> +            Continuous AF is running and the algorithm may restart a scan
> +            spontaneously.
> +        - name: AfPauseStatePausing
> +          value: 1
> +          description: |
> +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> +            as soon as any in-progress scan completes (and then report
> +            AfPauseStatePaused). No new scans will be start spontaneously until
> +            the AfPauseResume control is sent.
> +        - name: AfPauseStatePaused
> +          value: 2
> +          description: |
> +            Continuous AF is paused. No further state changes or lens movements
> +            will occur until the AfPauseResume control is sent.
> +

Very nice overall! With a few more clarifications I think this is good
to go!

Thanks
   j

>    # ----------------------------------------------------------------------------
>    # Draft controls section
>
> @@ -406,27 +664,6 @@ controls:
>              The camera will cancel any active or completed metering sequence.
>              The AE algorithm is reset to its initial state.
>
> -  - AfTrigger:
> -      type: int32_t
> -      draft: true
> -      description: |
> -       Control for AF trigger. Currently identical to
> -       ANDROID_CONTROL_AF_TRIGGER.
> -
> -        Whether the camera device will trigger autofocus for this request.
> -      enum:
> -        - name: AfTriggerIdle
> -          value: 0
> -          description: The trigger is idle.
> -        - name: AfTriggerStart
> -          value: 1
> -          description: The AF routine is started by the camera.
> -        - name: AfTriggerCancel
> -          value: 2
> -          description: |
> -            The camera will cancel any active trigger and the AF routine is
> -            reset to its initial state.
> -
>    - NoiseReductionMode:
>        type: int32_t
>        draft: true
> @@ -507,45 +744,6 @@ controls:
>              The AE algorithm has started a pre-capture metering session.
>              \sa AePrecaptureTrigger
>
> -  - AfState:
> -      type: int32_t
> -      draft: true
> -      description: |
> -       Control to report the current AF algorithm state. Currently identical to
> -       ANDROID_CONTROL_AF_STATE.
> -
> -        Current state of the AF algorithm.
> -      enum:
> -        - name: AfStateInactive
> -          value: 0
> -          description: The AF algorithm is inactive.
> -        - name: AfStatePassiveScan
> -          value: 1
> -          description: |
> -            AF is performing a passive scan of the scene in continuous
> -            auto-focus mode.
> -        - name: AfStatePassiveFocused
> -          value: 2
> -          description: |
> -            AF believes the scene is in focus, but might restart scanning.
> -        - name: AfStateActiveScan
> -          value: 3
> -          description: |
> -            AF is performing a scan triggered by an AF trigger request.
> -            \sa AfTrigger
> -        - name: AfStateFocusedLock
> -          value: 4
> -          description: |
> -            AF believes has focused correctly and has locked focus.
> -        - name: AfStateNotFocusedLock
> -          value: 5
> -          description: |
> -            AF has not been able to focus and has locked.
> -        - name: AfStatePassiveUnfocused
> -          value: 6
> -          description: |
> -            AF has completed a passive scan without finding focus.
> -
>    - AwbState:
>        type: int32_t
>        draft: true
> --
> 2.30.2
>

Ah, forgot one thing!

On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> Hi David,
>   thanks for this new version
>
> On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > This patch describes a series of controls that allow applications to
> > drive AF algorithms:
> >
> > AfMode - manual, auto or continuous
> > AfRange - full, macro or normal
> > AfSpeed - fast or slow
> > AfWindows - AF window locations
> > AfTrigger - start (trigger) an AF scan or cancel
> > AfPause - pause continuous AF
> > LensPosition - set or retrieve position of lens
> > AfState - reports whether scanning/success/failure
> > AfPauseState - reports whether continuous AF paused or not
> >
> > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > ---
> >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> >  1 file changed, 258 insertions(+), 60 deletions(-)
> >
> > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > index 9d4638ae..eccf18bc 100644
> > --- a/src/libcamera/control_ids.yaml
> > +++ b/src/libcamera/control_ids.yaml
> > @@ -381,6 +381,264 @@ controls:
> >          \todo Define how the sensor timestamp has to be used in the reprocessing
> >          use case.
> >
> > +  - AfMode:
> > +      type: int32_t
> > +      description: |
> > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > +        are allowed to set a new mode, and to send additional controls for
> > +        that new mode, in the same request.
> > +
> > +        An implementation may choose not to implement all the modes.
> > +
> > +      enum:
> > +        - name: AfModeManual
> > +          value: 0
> > +          description: |
> > +            The AF algorithm is in manual mode. In this mode it will never
> > +            perform any action nor move the lens of its own accord, but an
> > +            application can set controls to move the lens.
> > +
>
> If, as I understand it, the lens can only be moved with 'LensPosition'
> when in manual mode, I would say so explicitly
>
>                "... but an application can specify the desired lens
>                position using the LensPosition control."
>
> > +            In this mode the AfState will always report AfStateReset.
>
> I'm tempted to say AfState is useless and can not be reported at all
> in this case. For simplicity (for ph/ipa) I'm fine reporting it
> regardless, even if seeing 'reset' might be slightly confusing. I
> don't have better alternatives to propose though.
>
> Reading the AfStateReset description
>
>         - name: AfStateReset
>           value: 0
>           description: |
>             The AF algorithm is in manual mode (AfModeManual) or in auto mode
>             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
>             scan was cancelled.
>
> I wonder if this isn't better named AfStateIdle
>
>
> > +        - name: AfModeAuto
> > +          value: 1
> > +          description: |
> > +            The AF algorithm is in auto mode. This means that the algorithm
> > +            will never move the lens or change state unless the AfTrigger
> > +            control is used. The AfTrigger control can be used to initiate a
> > +            focus scan, the results of which will be reported by AfState.
> > +
> > +            If the autofocus algorithm is moved from AfModeAuto to another
> > +            mode while a scan is in progress, the scan is cancelled
> > +            immediately, without waiting for the scan to finish.
> > +
> > +            When first entering this mode the AfState will report
> > +            AfStateReset. When a trigger control is sent, AfState will
> > +            report AfStateScanning for a period before spontaneously
> > +            changing to AfStateFocused or AfStateFailed, depending on
> > +            the outcome of the scan. It will remain in this state until
> > +            another scan is initiated by the AfTrigger control. If a scan is
> > +            cancelled (without changing to another mode), AfState will return
> > +            to AfStateReset.
> > +        - name: AfModeContinuous
> > +          value: 2
> > +          description: |
> > +            The AF algorithm is in continuous mode. This means that the lens
> > +            can re-start a scan spontaneously at any moment, without any user
> > +            intervention. The AfState still reports whether the algorithm is
> > +            currently scanning or not, though the application has no ability
> > +            to initiate or cancel scans (though it can "pause" them), nor to
> > +            move the lens for itself.
> > +
> > +            When set to AfModeContinuous, the system will immediately initiate
> > +            a scan so AfState will report AfStateScanning, and will settle on
> > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > +
> > +  - AfRange:
> > +      type: int32_t
> > +      description: |
> > +        Control to set the range of focus distances that is scanned. An
> > +        implementation may choose not to implement all the options here.
> > +      enum:
> > +        - name: AfRangeNormal
> > +          value: 0
> > +          description: |
> > +            A wide range of focus distances is scanned, all the way from
> > +            infinity down to close distances, though depending on the
> > +            implementation, possibly not including the very closest macro
> > +            positions.
> > +        - name: AfRangeMacro
> > +          value: 1
> > +          description: Only close distances are scanned.
> > +        - name: AfRangeFull
> > +          value: 2
> > +          description: |
> > +            The full range of focus distances is scanned just as with
> > +            AfRangeNormal but this time including the very closest macro
> > +            positions.
> > +
> > +  - AfSpeed:
> > +      type: int32_t
> > +      description: |
> > +        Control that determines whether the AF algorithm is to move the lens
> > +        as quickly as possible or more steadily. For example, during video
> > +        recording it may be desirable not to move the lens too abruptly, but
> > +        when in a preview mode (waiting for a still capture) it may be
> > +        helpful to move the lens as quickly as is reasonably possible.
> > +      enum:
> > +        - name: AfSpeedNormal
> > +          value: 0
> > +          description: Move the lens at its usual speed.
> > +        - name: AfSpeedFast
> > +          value: 1
> > +          description: Move the lens more quickly.
> > +
> > +  - AfWindows:
> > +      type: Rectangle
> > +      description: |
> > +        Sets the focus windows used by the AF algorithm. The units used express
> > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > +        image), as u0.16 format numbers.
>
> I don't like too much the "or if unavailable", as it means that the
> reference rectangle will change depending on the platform
> capabilities.
>
> I understand your argument in reply to Jean-Michel comment on the
> previous version of the series (quoted here for reference)
>
> "The thing that bothers me a bit about the full pixel array is that, if
> you use digital zoom, then you will have to update your AfWindows as
> they might otherwise lie outside your final image! If it's defined as
> being a region of the scaler crop, then it automatically stays the
> same (to the application, at least, the pipeline handler probably has
> some work to do).
>
> I note that the Android link you give below says "Focus areas are
> relative to the current field of view (getZoom())", which I guess is
> partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> is easier to work with than the u0.16 format that I specified?"
>
> Let's reason on the IPA point of view. If the AF algorithm works by
> estimating the contrast, will it do so before or after the ScalerCrop
> rectangle is applied ? If a sensor provides PDAF statistics, will it
> do so on the full pixel array size, the portion of if that gets
> processed or on the final image ?
>
> Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> rectangle, so that they are applied to the actual FOV before being
> passed to the AF algorithm ?
>
> One last point about the unit: is the below representation correct,
> when it comes to use a Rectangle to represent a proportion ?
> Regardless of the reference rectangle, if we have a 4x3 grid, will the
> pipeline report 12 rectangles like the ones below, for applications to
> chose from ?
>
>       (0,0)
>         +-----+-----+-----+-----+
>         |     |     |     |     |
>         +-----+-----+-----+-----+
>         |     |     |     |     |
>         +-----+-----+-----+-----+
>         |     |     |     |     |
>         +-----+-----+-----+-----+
>
> AfWindows = {
>     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
>     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
>     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> };
>
> My understanding is that the grid is fixed and depends on the platform
> capabilities. I wonder if we shouldn't allow applications to select
> grids by index.
>
> Also, do we need a property to report the grid structure and allows
> application to select them from, or do you see this conveyed through a
> ControlInfo ?
>
>
> > +
> > +        In order to be activated, a rectangle must be programmed with non-zero
> > +        width and height. If no rectangles are programmed in this way, then the
> > +        system will choose its own single default window in the centre of the
> > +        image.
> > +
> > +        The details of how the windows are used are platform dependent. We note
> > +        that when there is more than one AF window, a typical implementation
> > +        might find the optimal focus position for each one and finally select
> > +        the window closest to the camera.
>
> What window is more close to the camera ? My understanding is that
> windows apply to the sensor produced frame, hence, when it comes to distance,
> they all sit on the same focal plan :)
>
> > +
> > +        size: [n]
> > +
> > +  - AfTrigger:
> > +      type: int32_t
> > +      description: |
> > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > +        and can also be used to terminate a scan early.
> > +
> > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > +
> > +      enum:
> > +        - name: AfTriggerStart
> > +          value: 0
> > +          description: Start an AF scan. Ignored if a scan is in progress.
> > +        - name: AfTriggerCancel
> > +          value: 1
> > +          description: Cancel an AF scan. This does not cause the lens to move
> > +            anywhere else. Ignored if no scan is in progress.
> > +
> > +  - AfPause:
> > +      type: int32_t
> > +      description: |
> > +        This control has no effect except when in continuous autofocus mode
> > +        (AfModeContinuous). It can be used to pause any lens movements while
> > +        (for example) images are captured. The algorithm remains inactive
> > +        until it is instructed to resume.
> > +
> > +      enum:
> > +        - name: AfPauseImmediate
> > +          value: 0
> > +          description: |
> > +            Pause the continuous autofocus algorithm immediately, whether or
> > +            not any kind of scan is underway. AfPauseState will subsequently
> > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > +            when it received this control.
> > +        - name: AfPauseDeferred
> > +          value: 1
> > +          description: |
> > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > +            state and AfPauseState will report AfPauseStatePaused.
> > +
> > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > +            will report AfPauseStatePausing until the scan is finished, at which
> > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > +            AfPauseState will change to AfPauseStatePaused.
>
> From here on I see lines getting over 80 cols by a few chars. Can we
> align it back ?
>
> > +        - name: AfPauseResume
> > +          value: 2
> > +          description: |
> > +            Resume continuous autofocus operation. The algorithm starts again
> > +            from exactly where it left off, and AfPauseState will report
> > +            AfPauseStateRunning.
> > +
> > +  - LensPosition:
> > +      type: float
> > +      description: |
> > +        Acts as a control to instruct the lens to move to a particular position
> > +        and also reports back the position of the lens for each frame.
> > +
> > +        The LensPosition control is ignored unless the AfMode is set to
> > +        AfModeManual.
> > +
> > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > +        values are permitted. For example:
> > +        0 moves the lens to infinity.
> > +        0.5 moves the lens to twice the hyperfocal distance.
> > +        1 moves the lens to the hyperfocal position.
> > +        And larger values will focus the lens ever closer.
> > +
>
> This seems good to me, albeit my understanding of the issue is limited :)
>

Do we need a property to report the lens hyperfocal distance ?


> > +  - AfState:
> > +      type: int32_t
> > +      description: |
> > +        Reports the current state of the AF algorithm in conjunction with the
> > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > +        The possible state changes are described below, though we note the following
> > +        state transitions that occur when the AfMode is changed.
> > +
> > +        If the AfMode is set to AfModeManual then the AfState will report
> > +        AfStateReset. This action on its own does not cause the lens to move; the
> > +        LensPosition control must subsequently be used to achieve this.
> > +
> > +        If the AfMode is set to AfModeAuto then the AfState will report
> > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
>
> s/unless/until ?
> s/at the same time// ?
>
> > +        case it will (start a scan and) report AfStateScanning.
> > +
> > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > +        report AfStateScanning.
> > +
> > +      enum:
> > +        - name: AfStateReset
> > +          value: 0
> > +          description: |
> > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > +            scan was cancelled.
> > +        - name: AfStateScanning
> > +          value: 1
> > +          description: |
> > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > +            started using the AfTrigger control. The scan can be cancelled by
> > +            sending AfTriggerCancel at which point the algorithm will either
> > +            move back to AfStateReset or, if the scan actually completes
> > +            before the cancel request is processed, to one of
> > +            AfStateFocused or AfStateFailed.
> > +
> > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
>
> s/be be/is ?
>
> > +            at which point it may enter this state spontaneously whenever it
> > +            determines that a rescan is needed.
> > +        - name: AfStateFocused
> > +          value: 2
> > +          description: |
> > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > +            mode and a scan has completed with the result that the algorithm believes
> > +            the image is now in focus.
> > +        - name: AfStateFailed
> > +          value: 3
> > +          description: |
> > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > +            mode and a scan has completed with the result that the algorithm did not
> > +            find a good focus position.
> > +
> > +  - AfPauseState:
> > +      type: int32_t
> > +      description: |
> > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > +        the algorithm is currently running, paused or pausing (that is, will pause
> > +        as soon as any in-progress scan completes).
> > +
> > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > +
> > +      enum:
> > +        - name: AfPauseStateRunning
> > +          value: 0
> > +          description: |
> > +            Continuous AF is running and the algorithm may restart a scan
> > +            spontaneously.
> > +        - name: AfPauseStatePausing
> > +          value: 1
> > +          description: |
> > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > +            as soon as any in-progress scan completes (and then report
> > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > +            the AfPauseResume control is sent.
> > +        - name: AfPauseStatePaused
> > +          value: 2
> > +          description: |
> > +            Continuous AF is paused. No further state changes or lens movements
> > +            will occur until the AfPauseResume control is sent.
> > +
>
> Very nice overall! With a few more clarifications I think this is good
> to go!
>
> Thanks
>    j
>
> >    # ----------------------------------------------------------------------------
> >    # Draft controls section
> >
> > @@ -406,27 +664,6 @@ controls:
> >              The camera will cancel any active or completed metering sequence.
> >              The AE algorithm is reset to its initial state.
> >
> > -  - AfTrigger:
> > -      type: int32_t
> > -      draft: true
> > -      description: |
> > -       Control for AF trigger. Currently identical to
> > -       ANDROID_CONTROL_AF_TRIGGER.
> > -
> > -        Whether the camera device will trigger autofocus for this request.
> > -      enum:
> > -        - name: AfTriggerIdle
> > -          value: 0
> > -          description: The trigger is idle.
> > -        - name: AfTriggerStart
> > -          value: 1
> > -          description: The AF routine is started by the camera.
> > -        - name: AfTriggerCancel
> > -          value: 2
> > -          description: |
> > -            The camera will cancel any active trigger and the AF routine is
> > -            reset to its initial state.
> > -
> >    - NoiseReductionMode:
> >        type: int32_t
> >        draft: true
> > @@ -507,45 +744,6 @@ controls:
> >              The AE algorithm has started a pre-capture metering session.
> >              \sa AePrecaptureTrigger
> >
> > -  - AfState:
> > -      type: int32_t
> > -      draft: true
> > -      description: |
> > -       Control to report the current AF algorithm state. Currently identical to
> > -       ANDROID_CONTROL_AF_STATE.
> > -
> > -        Current state of the AF algorithm.
> > -      enum:
> > -        - name: AfStateInactive
> > -          value: 0
> > -          description: The AF algorithm is inactive.
> > -        - name: AfStatePassiveScan
> > -          value: 1
> > -          description: |
> > -            AF is performing a passive scan of the scene in continuous
> > -            auto-focus mode.
> > -        - name: AfStatePassiveFocused
> > -          value: 2
> > -          description: |
> > -            AF believes the scene is in focus, but might restart scanning.
> > -        - name: AfStateActiveScan
> > -          value: 3
> > -          description: |
> > -            AF is performing a scan triggered by an AF trigger request.
> > -            \sa AfTrigger
> > -        - name: AfStateFocusedLock
> > -          value: 4
> > -          description: |
> > -            AF believes has focused correctly and has locked focus.
> > -        - name: AfStateNotFocusedLock
> > -          value: 5
> > -          description: |
> > -            AF has not been able to focus and has locked.
> > -        - name: AfStatePassiveUnfocused
> > -          value: 6
> > -          description: |
> > -            AF has completed a passive scan without finding focus.
> > -
> >    - AwbState:
> >        type: int32_t
> >        draft: true
> > --
> > 2.30.2
> >

Hi Jacopo

Thanks for the quick reply!

On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
>
> Ah, forgot one thing!
>
> On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > Hi David,
> >   thanks for this new version
> >
> > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > This patch describes a series of controls that allow applications to
> > > drive AF algorithms:
> > >
> > > AfMode - manual, auto or continuous
> > > AfRange - full, macro or normal
> > > AfSpeed - fast or slowwith
> > > AfWindows - AF window locations
> > > AfTrigger - start (trigger) an AF scan or cancel
> > > AfPause - pause continuous AF
> > > LensPosition - set or retrieve position of lens
> > > AfState - reports whether scanning/success/failure
> > > AfPauseState - reports whether continuous AF paused or not
> > >
> > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > ---
> > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > >
> > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > index 9d4638ae..eccf18bc 100644
> > > --- a/src/libcamera/control_ids.yaml
> > > +++ b/src/libcamera/control_ids.yaml
> > > @@ -381,6 +381,264 @@ controls:
> > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > >          use case.
> > >
> > > +  - AfMode:
> > > +      type: int32_t
> > > +      description: |
> > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > +        are allowed to set a new mode, and to send additional controls for
> > > +        that new mode, in the same request.
> > > +
> > > +        An implementation may choose not to implement all the modes.
> > > +
> > > +      enum:
> > > +        - name: AfModeManual
> > > +          value: 0
> > > +          description: |
> > > +            The AF algorithm is in manual mode. In this mode it will never
> > > +            perform any action nor move the lens of its own accord, but an
> > > +            application can set controls to move the lens.
> > > +
> >
> > If, as I understand it, the lens can only be moved with 'LensPosition'
> > when in manual mode, I would say so explicitly
> >
> >                "... but an application can specify the desired lens
> >                position using the LensPosition control."

Yes, will do.

> >
> > > +            In this mode the AfState will always report AfStateReset.
> >
> > I'm tempted to say AfState is useless and can not be reported at all
> > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > regardless, even if seeing 'reset' might be slightly confusing. I
> > don't have better alternatives to propose though.

Don't really mind either. I think reporting stuff all the time might
be slightly easier, at least it's always obvious what to expect...

> >
> > Reading the AfStateReset description
> >
> >         - name: AfStateReset
> >           value: 0
> >           description: |
> >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> >             scan was cancelled.
> >
> > I wonder if this isn't better named AfStateIdle

Yes, also fine with that.

> >
> >
> > > +        - name: AfModeAuto
> > > +          value: 1
> > > +          description: |
> > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > +            will never move the lens or change state unless the AfTrigger
> > > +            control is used. The AfTrigger control can be used to initiate a
> > > +            focus scan, the results of which will be reported by AfState.
> > > +
> > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > +            mode while a scan is in progress, the scan is cancelled
> > > +            immediately, without waiting for the scan to finish.
> > > +
> > > +            When first entering this mode the AfState will report
> > > +            AfStateReset. When a trigger control is sent, AfState will
> > > +            report AfStateScanning for a period before spontaneously
> > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > +            the outcome of the scan. It will remain in this state until
> > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > +            cancelled (without changing to another mode), AfState will return
> > > +            to AfStateReset.
> > > +        - name: AfModeContinuous
> > > +          value: 2
> > > +          description: |
> > > +            The AF algorithm is in continuous mode. This means that the lens
> > > +            can re-start a scan spontaneously at any moment, without any user
> > > +            intervention. The AfState still reports whether the algorithm is
> > > +            currently scanning or not, though the application has no ability
> > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > +            move the lens for itself.
> > > +
> > > +            When set to AfModeContinuous, the system will immediately initiate
> > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > +
> > > +  - AfRange:
> > > +      type: int32_t
> > > +      description: |
> > > +        Control to set the range of focus distances that is scanned. An
> > > +        implementation may choose not to implement all the options here.
> > > +      enum:
> > > +        - name: AfRangeNormal
> > > +          value: 0
> > > +          description: |
> > > +            A wide range of focus distances is scanned, all the way from
> > > +            infinity down to close distances, though depending on the
> > > +            implementation, possibly not including the very closest macro
> > > +            positions.
> > > +        - name: AfRangeMacro
> > > +          value: 1
> > > +          description: Only close distances are scanned.
> > > +        - name: AfRangeFull
> > > +          value: 2
> > > +          description: |
> > > +            The full range of focus distances is scanned just as with
> > > +            AfRangeNormal but this time including the very closest macro
> > > +            positions.
> > > +
> > > +  - AfSpeed:
> > > +      type: int32_t
> > > +      description: |
> > > +        Control that determines whether the AF algorithm is to move the lens
> > > +        as quickly as possible or more steadily. For example, during video
> > > +        recording it may be desirable not to move the lens too abruptly, but
> > > +        when in a preview mode (waiting for a still capture) it may be
> > > +        helpful to move the lens as quickly as is reasonably possible.
> > > +      enum:
> > > +        - name: AfSpeedNormal
> > > +          value: 0
> > > +          description: Move the lens at its usual speed.
> > > +        - name: AfSpeedFast
> > > +          value: 1
> > > +          description: Move the lens more quickly.
> > > +
> > > +  - AfWindows:
> > > +      type: Rectangle
> > > +      description: |
> > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > +        image), as u0.16 format numbers.
> >
> > I don't like too much the "or if unavailable", as it means that the
> > reference rectangle will change depending on the platform
> > capabilities.

I guess what I want is (like in the case of Android below), to specify
AF regions relative to the actual output image. If a platform supports
digital zoom (through ScalerCrop, I assume), the regions must be
relative to that part of the image. If there is no ScalerCrop, then do
we assume the platform doesn't support digital zoom? But the basic
idea is the same, the numbers supplied should be relative to the
output image in this case too.

Is there a better way to explain this, do you think?

> >
> > I understand your argument in reply to Jean-Michel comment on the
> > previous version of the series (quoted here for reference)
> >
> > "The thing that bothers me a bit about the full pixel array is that, if
> > you use digital zoom, then you will have to update your AfWindows as
> > they might otherwise lie outside your final image! If it's defined as
> > being a region of the scaler crop, then it automatically stays the
> > same (to the application, at least, the pipeline handler probably has
> > some work to do).
> >
> > I note that the Android link you give below says "Focus areas are
> > relative to the current field of view (getZoom())", which I guess is
> > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > is easier to work with than the u0.16 format that I specified?"
> >
> > Let's reason on the IPA point of view. If the AF algorithm works by
> > estimating the contrast, will it do so before or after the ScalerCrop
> > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > do so on the full pixel array size, the portion of if that gets
> > processed or on the final image ?

In the case of PDAF, we can presumably use information from whatever
part of the image that the sensor is outputting.

For CDAF (contrast detect AF) I guess it's possible that focus
statistics could be computed either before or after any cropping is
done in the ISP. (Maybe there are some platforms where cropping could
be delegated to the sensor?) So I'd guess that using any focus
statistics from outside the final FoV may actually be impossible,
depending on the platform. But even where you can do this, I don't
believe it's what anyone would want (and Android seems to behave like
this).

> >
> > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > rectangle, so that they are applied to the actual FOV before being
> > passed to the AF algorithm ?

Not quite sure what you meant here. I'd expect the ISP to generate
focus statistics from the raw Bayer pixels that it gets, without too
much processing applied to them. So the job will be to take rectangles
defined in the final FoV and translate them into the image coming
directly from the sensor.

> >
> > One last point about the unit: is the below representation correct,
> > when it comes to use a Rectangle to represent a proportion ?
> > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > pipeline report 12 rectangles like the ones below, for applications to
> > chose from ?
> >
> >       (0,0)
> >         +-----+-----+-----+-----+
> >         |     |     |     |     |
> >         +-----+-----+-----+-----+
> >         |     |     |     |     |
> >         +-----+-----+-----+-----+
> >         |     |     |     |     |
> >         +-----+-----+-----+-----+
> >
> > AfWindows = {
> >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > };
> >
> > My understanding is that the grid is fixed and depends on the platform
> > capabilities. I wonder if we shouldn't allow applications to select
> > grids by index.

I think there is a problem about determining the AF capabilities of a
platform. I could imagine some platforms will only accept a regular
grid of regions, but others might let you put them in arbitrary
places. Or a mixture of both. To be honest, I'm not sure how we handle
that sort of thing. Anyone have any good ideas?

> >
> > Also, do we need a property to report the grid structure and allows
> > application to select them from, or do you see this conveyed through a
> > ControlInfo ?

I wasn't quite sure what the question was here. Are you talking about
selecting the configuration for the AfWindows, or selecting focus
statistics from particular regions? Perhaps an example would help me
here!

> >
> >
> > > +
> > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > +        width and height. If no rectangles are programmed in this way, then the
> > > +        system will choose its own single default window in the centre of the
> > > +        image.
> > > +
> > > +        The details of how the windows are used are platform dependent. We note
> > > +        that when there is more than one AF window, a typical implementation
> > > +        might find the optimal focus position for each one and finally select
> > > +        the window closest to the camera.
> >
> > What window is more close to the camera ? My understanding is that
> > windows apply to the sensor produced frame, hence, when it comes to distance,
> > they all sit on the same focal plan :)

Sorry, by "the window closest to the camera" I mean "the window that
is showing objects in the real world closest to the camera". I'll try
to re-word that a bit.

> >
> > > +
> > > +        size: [n]
> > > +
> > > +  - AfTrigger:
> > > +      type: int32_t
> > > +      description: |
> > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > +        and can also be used to terminate a scan early.
> > > +
> > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > +
> > > +      enum:
> > > +        - name: AfTriggerStart
> > > +          value: 0
> > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > +        - name: AfTriggerCancel
> > > +          value: 1
> > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > +            anywhere else. Ignored if no scan is in progress.
> > > +
> > > +  - AfPause:
> > > +      type: int32_t
> > > +      description: |
> > > +        This control has no effect except when in continuous autofocus mode
> > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > +        (for example) images are captured. The algorithm remains inactive
> > > +        until it is instructed to resume.
> > > +
> > > +      enum:
> > > +        - name: AfPauseImmediate
> > > +          value: 0
> > > +          description: |
> > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > +            when it received this control.
> > > +        - name: AfPauseDeferred
> > > +          value: 1
> > > +          description: |
> > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > +            state and AfPauseState will report AfPauseStatePaused.
> > > +
> > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > +            AfPauseState will change to AfPauseStatePaused.
> >
> > From here on I see lines getting over 80 cols by a few chars. Can we
> > align it back ?

Sure, I'll have a look!

> >
> > > +        - name: AfPauseResume
> > > +          value: 2
> > > +          description: |
> > > +            Resume continuous autofocus operation. The algorithm starts again
> > > +            from exactly where it left off, and AfPauseState will report
> > > +            AfPauseStateRunning.
> > > +
> > > +  - LensPosition:
> > > +      type: float
> > > +      description: |
> > > +        Acts as a control to instruct the lens to move to a particular position
> > > +        and also reports back the position of the lens for each frame.
> > > +
> > > +        The LensPosition control is ignored unless the AfMode is set to
> > > +        AfModeManual.
> > > +
> > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > +        values are permitted. For example:
> > > +        0 moves the lens to infinity.
> > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > +        1 moves the lens to the hyperfocal position.
> > > +        And larger values will focus the lens ever closer.
> > > +
> >
> > This seems good to me, albeit my understanding of the issue is limited :)
> >
>
> Do we need a property to report the lens hyperfocal distance ?

Yes, sometimes. Android distinguishes between calibrated and
uncalibrated cameras. For a calibrated camera an application should be
able to set the focal distance to (for example) 1m, and given the
units I've proposed, you would need to know the hyperfocal distance to
calculate the right number.

For an uncalibrated camera I don't think it matters. Many applications
would simply set the lens position to 1.0 (meaning hyperfocal) and
you'll get the "best overall" focal distance (largest depth of field),
and that's it.

There's also a need to know the allowable range of lens movement. This
will depend on the type of module (knowing the lens driver isn't
enough). But I don't see that applications need to know this, I think
the pipeline handler and AF implementation should take care of it.

>
>
> > > +  - AfState:
> > > +      type: int32_t
> > > +      description: |
> > > +        Reports the current state of the AF algorithm in conjunction with the
> > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > +        The possible state changes are described below, though we note the following
> > > +        state transitions that occur when the AfMode is changed.
> > > +
> > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > +        LensPosition control must subsequently be used to achieve this.
> > > +
> > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> >
> > s/unless/until ?
> > s/at the same time// ?

I'm trying to describe what happens to the AfState when you send
AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
AfTriggerStart). The only occasion when you won't see AfStateIdle will
be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
think this reads OK to me, but I'll see if I can make the wording any
clearer.

> >
> > > +        case it will (start a scan and) report AfStateScanning.
> > > +
> > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > +        report AfStateScanning.
> > > +
> > > +      enum:
> > > +        - name: AfStateReset
> > > +          value: 0
> > > +          description: |
> > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > +            scan was cancelled.
> > > +        - name: AfStateScanning
> > > +          value: 1
> > > +          description: |
> > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > +            started using the AfTrigger control. The scan can be cancelled by
> > > +            sending AfTriggerCancel at which point the algorithm will either
> > > +            move back to AfStateReset or, if the scan actually completes
> > > +            before the cancel request is processed, to one of
> > > +            AfStateFocused or AfStateFailed.
> > > +
> > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> >
> > s/be be/is ?

Yes, thanks!

> >
> > > +            at which point it may enter this state spontaneously whenever it
> > > +            determines that a rescan is needed.
> > > +        - name: AfStateFocused
> > > +          value: 2
> > > +          description: |
> > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > +            mode and a scan has completed with the result that the algorithm believes
> > > +            the image is now in focus.
> > > +        - name: AfStateFailed
> > > +          value: 3
> > > +          description: |
> > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > +            mode and a scan has completed with the result that the algorithm did not
> > > +            find a good focus position.
> > > +
> > > +  - AfPauseState:
> > > +      type: int32_t
> > > +      description: |
> > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > +        as soon as any in-progress scan completes).
> > > +
> > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > +
> > > +      enum:
> > > +        - name: AfPauseStateRunning
> > > +          value: 0
> > > +          description: |
> > > +            Continuous AF is running and the algorithm may restart a scan
> > > +            spontaneously.
> > > +        - name: AfPauseStatePausing
> > > +          value: 1
> > > +          description: |
> > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > +            as soon as any in-progress scan completes (and then report
> > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > +            the AfPauseResume control is sent.
> > > +        - name: AfPauseStatePaused
> > > +          value: 2
> > > +          description: |
> > > +            Continuous AF is paused. No further state changes or lens movements
> > > +            will occur until the AfPauseResume control is sent.
> > > +
> >
> > Very nice overall! With a few more clarifications I think this is good
> > to go!
> >
> > Thanks
> >    j

Thanks!
David

> >
> > >    # ----------------------------------------------------------------------------
> > >    # Draft controls section
> > >
> > > @@ -406,27 +664,6 @@ controls:
> > >              The camera will cancel any active or completed metering sequence.
> > >              The AE algorithm is reset to its initial state.
> > >
> > > -  - AfTrigger:
> > > -      type: int32_t
> > > -      draft: true
> > > -      description: |
> > > -       Control for AF trigger. Currently identical to
> > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > -
> > > -        Whether the camera device will trigger autofocus for this request.
> > > -      enum:
> > > -        - name: AfTriggerIdle
> > > -          value: 0
> > > -          description: The trigger is idle.
> > > -        - name: AfTriggerStart
> > > -          value: 1
> > > -          description: The AF routine is started by the camera.
> > > -        - name: AfTriggerCancel
> > > -          value: 2
> > > -          description: |
> > > -            The camera will cancel any active trigger and the AF routine is
> > > -            reset to its initial state.
> > > -
> > >    - NoiseReductionMode:
> > >        type: int32_t
> > >        draft: true
> > > @@ -507,45 +744,6 @@ controls:
> > >              The AE algorithm has started a pre-capture metering session.
> > >              \sa AePrecaptureTrigger
> > >
> > > -  - AfState:
> > > -      type: int32_t
> > > -      draft: true
> > > -      description: |
> > > -       Control to report the current AF algorithm state. Currently identical to
> > > -       ANDROID_CONTROL_AF_STATE.
> > > -
> > > -        Current state of the AF algorithm.
> > > -      enum:
> > > -        - name: AfStateInactive
> > > -          value: 0
> > > -          description: The AF algorithm is inactive.
> > > -        - name: AfStatePassiveScan
> > > -          value: 1
> > > -          description: |
> > > -            AF is performing a passive scan of the scene in continuous
> > > -            auto-focus mode.
> > > -        - name: AfStatePassiveFocused
> > > -          value: 2
> > > -          description: |
> > > -            AF believes the scene is in focus, but might restart scanning.
> > > -        - name: AfStateActiveScan
> > > -          value: 3
> > > -          description: |
> > > -            AF is performing a scan triggered by an AF trigger request.
> > > -            \sa AfTrigger
> > > -        - name: AfStateFocusedLock
> > > -          value: 4
> > > -          description: |
> > > -            AF believes has focused correctly and has locked focus.
> > > -        - name: AfStateNotFocusedLock
> > > -          value: 5
> > > -          description: |
> > > -            AF has not been able to focus and has locked.
> > > -        - name: AfStatePassiveUnfocused
> > > -          value: 6
> > > -          description: |
> > > -            AF has completed a passive scan without finding focus.
> > > -
> > >    - AwbState:
> > >        type: int32_t
> > >        draft: true
> > > --
> > > 2.30.2
> > >

Hi David,

On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> Hi Jacopo
>
> Thanks for the quick reply!
>
> On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> >
> > Ah, forgot one thing!
> >
> > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > Hi David,
> > >   thanks for this new version
> > >
> > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > This patch describes a series of controls that allow applications to
> > > > drive AF algorithms:
> > > >
> > > > AfMode - manual, auto or continuous
> > > > AfRange - full, macro or normal
> > > > AfSpeed - fast or slowwith
> > > > AfWindows - AF window locations
> > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > AfPause - pause continuous AF
> > > > LensPosition - set or retrieve position of lens
> > > > AfState - reports whether scanning/success/failure
> > > > AfPauseState - reports whether continuous AF paused or not
> > > >
> > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > ---
> > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > >
> > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > index 9d4638ae..eccf18bc 100644
> > > > --- a/src/libcamera/control_ids.yaml
> > > > +++ b/src/libcamera/control_ids.yaml
> > > > @@ -381,6 +381,264 @@ controls:
> > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > >          use case.
> > > >
> > > > +  - AfMode:
> > > > +      type: int32_t
> > > > +      description: |
> > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > +        are allowed to set a new mode, and to send additional controls for
> > > > +        that new mode, in the same request.
> > > > +
> > > > +        An implementation may choose not to implement all the modes.
> > > > +
> > > > +      enum:
> > > > +        - name: AfModeManual
> > > > +          value: 0
> > > > +          description: |
> > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > +            perform any action nor move the lens of its own accord, but an
> > > > +            application can set controls to move the lens.
> > > > +
> > >
> > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > when in manual mode, I would say so explicitly
> > >
> > >                "... but an application can specify the desired lens
> > >                position using the LensPosition control."
>
> Yes, will do.
>
> > >
> > > > +            In this mode the AfState will always report AfStateReset.
> > >
> > > I'm tempted to say AfState is useless and can not be reported at all
> > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > don't have better alternatives to propose though.
>
> Don't really mind either. I think reporting stuff all the time might
> be slightly easier, at least it's always obvious what to expect...
>
> > >
> > > Reading the AfStateReset description
> > >
> > >         - name: AfStateReset
> > >           value: 0
> > >           description: |
> > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > >             scan was cancelled.
> > >
> > > I wonder if this isn't better named AfStateIdle
>
> Yes, also fine with that.
>
> > >
> > >
> > > > +        - name: AfModeAuto
> > > > +          value: 1
> > > > +          description: |
> > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > +            will never move the lens or change state unless the AfTrigger
> > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > +            focus scan, the results of which will be reported by AfState.
> > > > +
> > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > +            mode while a scan is in progress, the scan is cancelled
> > > > +            immediately, without waiting for the scan to finish.
> > > > +
> > > > +            When first entering this mode the AfState will report
> > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > +            report AfStateScanning for a period before spontaneously
> > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > +            the outcome of the scan. It will remain in this state until
> > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > +            cancelled (without changing to another mode), AfState will return
> > > > +            to AfStateReset.
> > > > +        - name: AfModeContinuous
> > > > +          value: 2
> > > > +          description: |
> > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > +            intervention. The AfState still reports whether the algorithm is
> > > > +            currently scanning or not, though the application has no ability
> > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > +            move the lens for itself.
> > > > +
> > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > +
> > > > +  - AfRange:
> > > > +      type: int32_t
> > > > +      description: |
> > > > +        Control to set the range of focus distances that is scanned. An
> > > > +        implementation may choose not to implement all the options here.
> > > > +      enum:
> > > > +        - name: AfRangeNormal
> > > > +          value: 0
> > > > +          description: |
> > > > +            A wide range of focus distances is scanned, all the way from
> > > > +            infinity down to close distances, though depending on the
> > > > +            implementation, possibly not including the very closest macro
> > > > +            positions.
> > > > +        - name: AfRangeMacro
> > > > +          value: 1
> > > > +          description: Only close distances are scanned.
> > > > +        - name: AfRangeFull
> > > > +          value: 2
> > > > +          description: |
> > > > +            The full range of focus distances is scanned just as with
> > > > +            AfRangeNormal but this time including the very closest macro
> > > > +            positions.
> > > > +
> > > > +  - AfSpeed:
> > > > +      type: int32_t
> > > > +      description: |
> > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > +        as quickly as possible or more steadily. For example, during video
> > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > +      enum:
> > > > +        - name: AfSpeedNormal
> > > > +          value: 0
> > > > +          description: Move the lens at its usual speed.
> > > > +        - name: AfSpeedFast
> > > > +          value: 1
> > > > +          description: Move the lens more quickly.
> > > > +
> > > > +  - AfWindows:
> > > > +      type: Rectangle
> > > > +      description: |
> > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > +        image), as u0.16 format numbers.
> > >
> > > I don't like too much the "or if unavailable", as it means that the
> > > reference rectangle will change depending on the platform
> > > capabilities.
>
> I guess what I want is (like in the case of Android below), to specify
> AF regions relative to the actual output image. If a platform supports
> digital zoom (through ScalerCrop, I assume), the regions must be
> relative to that part of the image. If there is no ScalerCrop, then do
> we assume the platform doesn't support digital zoom? But the basic
> idea is the same, the numbers supplied should be relative to the
> output image in this case too.
>

Thing is that there's no "output image size" as two streams produced
from the same camera might have different output sizes ?

> Is there a better way to explain this, do you think?
>
> > >
> > > I understand your argument in reply to Jean-Michel comment on the
> > > previous version of the series (quoted here for reference)
> > >
> > > "The thing that bothers me a bit about the full pixel array is that, if
> > > you use digital zoom, then you will have to update your AfWindows as
> > > they might otherwise lie outside your final image! If it's defined as
> > > being a region of the scaler crop, then it automatically stays the
> > > same (to the application, at least, the pipeline handler probably has
> > > some work to do).
> > >
> > > I note that the Android link you give below says "Focus areas are
> > > relative to the current field of view (getZoom())", which I guess is
> > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > is easier to work with than the u0.16 format that I specified?"
> > >
> > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > estimating the contrast, will it do so before or after the ScalerCrop
> > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > do so on the full pixel array size, the portion of if that gets
> > > processed or on the final image ?
>
> In the case of PDAF, we can presumably use information from whatever
> part of the image that the sensor is outputting.
>
> For CDAF (contrast detect AF) I guess it's possible that focus
> statistics could be computed either before or after any cropping is
> done in the ISP. (Maybe there are some platforms where cropping could
> be delegated to the sensor?) So I'd guess that using any focus
> statistics from outside the final FoV may actually be impossible,
> depending on the platform. But even where you can do this, I don't
> believe it's what anyone would want (and Android seems to behave like
> this).
>
> > >
> > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > rectangle, so that they are applied to the actual FOV before being
> > > passed to the AF algorithm ?
>
> Not quite sure what you meant here. I'd expect the ISP to generate
> focus statistics from the raw Bayer pixels that it gets, without too
> much processing applied to them. So the job will be to take rectangles
> defined in the final FoV and translate them into the image coming
> directly from the sensor.
>

What I meant is: if we specify the AfWindows relative to the full
pixel array, can't we then re-scale it to final cropped output ? or if
no copping on the sensor's output frame size.

This would be similar to what you do with ScalerCrop already ?

        void RPiCameraData::applyScalerCrop(const ControlList &controls) {

		Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);

		/* Create a version of the crop scaled to ISP (camera mode) pixels. */
		Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
		ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
        }

But I guess I'm reasoning in absolute coordinates, not in proportions
like you have proposed here.

> > >
> > > One last point about the unit: is the below representation correct,
> > > when it comes to use a Rectangle to represent a proportion ?
> > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > pipeline report 12 rectangles like the ones below, for applications to
> > > chose from ?
> > >
> > >       (0,0)
> > >         +-----+-----+-----+-----+
> > >         |     |     |     |     |
> > >         +-----+-----+-----+-----+
> > >         |     |     |     |     |
> > >         +-----+-----+-----+-----+
> > >         |     |     |     |     |
> > >         +-----+-----+-----+-----+
> > >
> > > AfWindows = {
> > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > };
> > >
> > > My understanding is that the grid is fixed and depends on the platform
> > > capabilities. I wonder if we shouldn't allow applications to select
> > > grids by index.
>
> I think there is a problem about determining the AF capabilities of a
> platform. I could imagine some platforms will only accept a regular
> grid of regions, but others might let you put them in arbitrary
> places. Or a mixture of both. To be honest, I'm not sure how we handle
> that sort of thing. Anyone have any good ideas?
>

Exactly, that's why having proportions as the ones I tried to sketch
out above doesn't really fit as a model in my head. Each platform
would have its own 'grid', do we need to expose it to applications ?

Can't we let application specify Rectangles with absolute coordinates
(relative to PixelArray or else) and let the IPA/PH map those areas on
their grids ?

> > >
> > > Also, do we need a property to report the grid structure and allows
> > > application to select them from, or do you see this conveyed through a
> > > ControlInfo ?
>
> I wasn't quite sure what the question was here. Are you talking about
> selecting the configuration for the AfWindows, or selecting focus
> statistics from particular regions? Perhaps an example would help me
> here!
>

I'm talking about how would an application know how many spots it can
select from, in other words, how do we let an application know that,
looking at the above example, each grid is 1/4 in width and 1/3 in
height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
4x3 one.

Or maybe I got it all wrong, and you where implying already that the
here specified Rectangles cover arbitrary parts of the image and it's
the IPA that has to match them on their grids.

> > >
> > >
> > > > +
> > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > +        system will choose its own single default window in the centre of the
> > > > +        image.
> > > > +
> > > > +        The details of how the windows are used are platform dependent. We note
> > > > +        that when there is more than one AF window, a typical implementation
> > > > +        might find the optimal focus position for each one and finally select
> > > > +        the window closest to the camera.
> > >
> > > What window is more close to the camera ? My understanding is that
> > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > they all sit on the same focal plan :)
>
> Sorry, by "the window closest to the camera" I mean "the window that
> is showing objects in the real world closest to the camera". I'll try
> to re-word that a bit.
>

Yeah sorry, I could have realized that myself :)

> > >
> > > > +
> > > > +        size: [n]
> > > > +
> > > > +  - AfTrigger:
> > > > +      type: int32_t
> > > > +      description: |
> > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > +        and can also be used to terminate a scan early.
> > > > +
> > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > +
> > > > +      enum:
> > > > +        - name: AfTriggerStart
> > > > +          value: 0
> > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > +        - name: AfTriggerCancel
> > > > +          value: 1
> > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > +            anywhere else. Ignored if no scan is in progress.
> > > > +
> > > > +  - AfPause:
> > > > +      type: int32_t
> > > > +      description: |
> > > > +        This control has no effect except when in continuous autofocus mode
> > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > +        (for example) images are captured. The algorithm remains inactive
> > > > +        until it is instructed to resume.
> > > > +
> > > > +      enum:
> > > > +        - name: AfPauseImmediate
> > > > +          value: 0
> > > > +          description: |
> > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > +            when it received this control.
> > > > +        - name: AfPauseDeferred
> > > > +          value: 1
> > > > +          description: |
> > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > +
> > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > +            AfPauseState will change to AfPauseStatePaused.
> > >
> > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > align it back ?
>
> Sure, I'll have a look!
>
> > >
> > > > +        - name: AfPauseResume
> > > > +          value: 2
> > > > +          description: |
> > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > +            from exactly where it left off, and AfPauseState will report
> > > > +            AfPauseStateRunning.
> > > > +
> > > > +  - LensPosition:
> > > > +      type: float
> > > > +      description: |
> > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > +        and also reports back the position of the lens for each frame.
> > > > +
> > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > +        AfModeManual.
> > > > +
> > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > +        values are permitted. For example:
> > > > +        0 moves the lens to infinity.
> > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > +        1 moves the lens to the hyperfocal position.
> > > > +        And larger values will focus the lens ever closer.
> > > > +
> > >
> > > This seems good to me, albeit my understanding of the issue is limited :)
> > >
> >
> > Do we need a property to report the lens hyperfocal distance ?
>
> Yes, sometimes. Android distinguishes between calibrated and
> uncalibrated cameras. For a calibrated camera an application should be
> able to set the focal distance to (for example) 1m, and given the
> units I've proposed, you would need to know the hyperfocal distance to
> calculate the right number.

Right, so we need a LensHyperfocalDistance or something for calibrated
lenses ?

>
> For an uncalibrated camera I don't think it matters. Many applications
> would simply set the lens position to 1.0 (meaning hyperfocal) and
> you'll get the "best overall" focal distance (largest depth of field),
> and that's it.

I think this part would be relevant to be captured in the control
description.

>
> There's also a need to know the allowable range of lens movement. This
> will depend on the type of module (knowing the lens driver isn't
> enough). But I don't see that applications need to know this, I think
> the pipeline handler and AF implementation should take care of it.
>

I think so. When it comes to the actual lens movement range I don't
think it's relevant for applications. But do the actual lens movement
range maps to a range of LensPosition values that can be specified ? I
guess it's hard to map them on un-calibrated lenses, but for
calibrated ones a LensPositionRanges whose ControlInfo informs
application on the range's min and max might make sense ? (Please note
that this could also be used for the use case we discussed about
wanting to limit the lens range search scope)

> >
> >
> > > > +  - AfState:
> > > > +      type: int32_t
> > > > +      description: |
> > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > +        The possible state changes are described below, though we note the following
> > > > +        state transitions that occur when the AfMode is changed.
> > > > +
> > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > +        LensPosition control must subsequently be used to achieve this.
> > > > +
> > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > >
> > > s/unless/until ?
> > > s/at the same time// ?
>
> I'm trying to describe what happens to the AfState when you send
> AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> AfTriggerStart). The only occasion when you won't see AfStateIdle will
> be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> think this reads OK to me, but I'll see if I can make the wording any
> clearer.

I see. I didn't get it, sorry. If it's fine with you, keep this
version!

>
> > >
> > > > +        case it will (start a scan and) report AfStateScanning.
> > > > +
> > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > +        report AfStateScanning.
> > > > +
> > > > +      enum:
> > > > +        - name: AfStateReset
> > > > +          value: 0
> > > > +          description: |
> > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > +            scan was cancelled.
> > > > +        - name: AfStateScanning
> > > > +          value: 1
> > > > +          description: |
> > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > +            move back to AfStateReset or, if the scan actually completes
> > > > +            before the cancel request is processed, to one of
> > > > +            AfStateFocused or AfStateFailed.
> > > > +
> > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > >
> > > s/be be/is ?
>
> Yes, thanks!
>
> > >
> > > > +            at which point it may enter this state spontaneously whenever it
> > > > +            determines that a rescan is needed.
> > > > +        - name: AfStateFocused
> > > > +          value: 2
> > > > +          description: |
> > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > +            the image is now in focus.
> > > > +        - name: AfStateFailed
> > > > +          value: 3
> > > > +          description: |
> > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > +            find a good focus position.
> > > > +
> > > > +  - AfPauseState:
> > > > +      type: int32_t
> > > > +      description: |
> > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > +        as soon as any in-progress scan completes).
> > > > +
> > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > +
> > > > +      enum:
> > > > +        - name: AfPauseStateRunning
> > > > +          value: 0
> > > > +          description: |
> > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > +            spontaneously.
> > > > +        - name: AfPauseStatePausing
> > > > +          value: 1
> > > > +          description: |
> > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > +            as soon as any in-progress scan completes (and then report
> > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > +            the AfPauseResume control is sent.
> > > > +        - name: AfPauseStatePaused
> > > > +          value: 2
> > > > +          description: |
> > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > +            will occur until the AfPauseResume control is sent.
> > > > +
> > >
> > > Very nice overall! With a few more clarifications I think this is good
> > > to go!
> > >
> > > Thanks
> > >    j
>
> Thanks!
> David
>
> > >
> > > >    # ----------------------------------------------------------------------------
> > > >    # Draft controls section
> > > >
> > > > @@ -406,27 +664,6 @@ controls:
> > > >              The camera will cancel any active or completed metering sequence.
> > > >              The AE algorithm is reset to its initial state.
> > > >
> > > > -  - AfTrigger:
> > > > -      type: int32_t
> > > > -      draft: true
> > > > -      description: |
> > > > -       Control for AF trigger. Currently identical to
> > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > -
> > > > -        Whether the camera device will trigger autofocus for this request.
> > > > -      enum:
> > > > -        - name: AfTriggerIdle
> > > > -          value: 0
> > > > -          description: The trigger is idle.
> > > > -        - name: AfTriggerStart
> > > > -          value: 1
> > > > -          description: The AF routine is started by the camera.
> > > > -        - name: AfTriggerCancel
> > > > -          value: 2
> > > > -          description: |
> > > > -            The camera will cancel any active trigger and the AF routine is
> > > > -            reset to its initial state.
> > > > -
> > > >    - NoiseReductionMode:
> > > >        type: int32_t
> > > >        draft: true
> > > > @@ -507,45 +744,6 @@ controls:
> > > >              The AE algorithm has started a pre-capture metering session.
> > > >              \sa AePrecaptureTrigger
> > > >
> > > > -  - AfState:
> > > > -      type: int32_t
> > > > -      draft: true
> > > > -      description: |
> > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > -       ANDROID_CONTROL_AF_STATE.
> > > > -
> > > > -        Current state of the AF algorithm.
> > > > -      enum:
> > > > -        - name: AfStateInactive
> > > > -          value: 0
> > > > -          description: The AF algorithm is inactive.
> > > > -        - name: AfStatePassiveScan
> > > > -          value: 1
> > > > -          description: |
> > > > -            AF is performing a passive scan of the scene in continuous
> > > > -            auto-focus mode.
> > > > -        - name: AfStatePassiveFocused
> > > > -          value: 2
> > > > -          description: |
> > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > -        - name: AfStateActiveScan
> > > > -          value: 3
> > > > -          description: |
> > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > -            \sa AfTrigger
> > > > -        - name: AfStateFocusedLock
> > > > -          value: 4
> > > > -          description: |
> > > > -            AF believes has focused correctly and has locked focus.
> > > > -        - name: AfStateNotFocusedLock
> > > > -          value: 5
> > > > -          description: |
> > > > -            AF has not been able to focus and has locked.
> > > > -        - name: AfStatePassiveUnfocused
> > > > -          value: 6
> > > > -          description: |
> > > > -            AF has completed a passive scan without finding focus.
> > > > -
> > > >    - AwbState:
> > > >        type: int32_t
> > > >        draft: true
> > > > --
> > > > 2.30.2
> > > >

Hi Jacopo

Thanks for the message. Some replies and more questions below!

On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
>
> Hi David,
>
> On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> > Hi Jacopo
> >
> > Thanks for the quick reply!
> >
> > On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > >
> > > Ah, forgot one thing!
> > >
> > > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > > Hi David,
> > > >   thanks for this new version
> > > >
> > > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > > This patch describes a series of controls that allow applications to
> > > > > drive AF algorithms:
> > > > >
> > > > > AfMode - manual, auto or continuous
> > > > > AfRange - full, macro or normal
> > > > > AfSpeed - fast or slowwith
> > > > > AfWindows - AF window locations
> > > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > > AfPause - pause continuous AF
> > > > > LensPosition - set or retrieve position of lens
> > > > > AfState - reports whether scanning/success/failure
> > > > > AfPauseState - reports whether continuous AF paused or not
> > > > >
> > > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > > ---
> > > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > > >
> > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > > index 9d4638ae..eccf18bc 100644
> > > > > --- a/src/libcamera/control_ids.yaml
> > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > @@ -381,6 +381,264 @@ controls:
> > > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > > >          use case.
> > > > >
> > > > > +  - AfMode:
> > > > > +      type: int32_t
> > > > > +      description: |
> > > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > > +        are allowed to set a new mode, and to send additional controls for
> > > > > +        that new mode, in the same request.
> > > > > +
> > > > > +        An implementation may choose not to implement all the modes.
> > > > > +
> > > > > +      enum:
> > > > > +        - name: AfModeManual
> > > > > +          value: 0
> > > > > +          description: |
> > > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > > +            perform any action nor move the lens of its own accord, but an
> > > > > +            application can set controls to move the lens.
> > > > > +
> > > >
> > > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > > when in manual mode, I would say so explicitly
> > > >
> > > >                "... but an application can specify the desired lens
> > > >                position using the LensPosition control."
> >
> > Yes, will do.
> >
> > > >
> > > > > +            In this mode the AfState will always report AfStateReset.
> > > >
> > > > I'm tempted to say AfState is useless and can not be reported at all
> > > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > > don't have better alternatives to propose though.
> >
> > Don't really mind either. I think reporting stuff all the time might
> > be slightly easier, at least it's always obvious what to expect...
> >
> > > >
> > > > Reading the AfStateReset description
> > > >
> > > >         - name: AfStateReset
> > > >           value: 0
> > > >           description: |
> > > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > >             scan was cancelled.
> > > >
> > > > I wonder if this isn't better named AfStateIdle
> >
> > Yes, also fine with that.
> >
> > > >
> > > >
> > > > > +        - name: AfModeAuto
> > > > > +          value: 1
> > > > > +          description: |
> > > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > > +            will never move the lens or change state unless the AfTrigger
> > > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > > +            focus scan, the results of which will be reported by AfState.
> > > > > +
> > > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > > +            mode while a scan is in progress, the scan is cancelled
> > > > > +            immediately, without waiting for the scan to finish.
> > > > > +
> > > > > +            When first entering this mode the AfState will report
> > > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > > +            report AfStateScanning for a period before spontaneously
> > > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > > +            the outcome of the scan. It will remain in this state until
> > > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > > +            cancelled (without changing to another mode), AfState will return
> > > > > +            to AfStateReset.
> > > > > +        - name: AfModeContinuous
> > > > > +          value: 2
> > > > > +          description: |
> > > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > > +            intervention. The AfState still reports whether the algorithm is
> > > > > +            currently scanning or not, though the application has no ability
> > > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > > +            move the lens for itself.
> > > > > +
> > > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > > +
> > > > > +  - AfRange:
> > > > > +      type: int32_t
> > > > > +      description: |
> > > > > +        Control to set the range of focus distances that is scanned. An
> > > > > +        implementation may choose not to implement all the options here.
> > > > > +      enum:
> > > > > +        - name: AfRangeNormal
> > > > > +          value: 0
> > > > > +          description: |
> > > > > +            A wide range of focus distances is scanned, all the way from
> > > > > +            infinity down to close distances, though depending on the
> > > > > +            implementation, possibly not including the very closest macro
> > > > > +            positions.
> > > > > +        - name: AfRangeMacro
> > > > > +          value: 1
> > > > > +          description: Only close distances are scanned.
> > > > > +        - name: AfRangeFull
> > > > > +          value: 2
> > > > > +          description: |
> > > > > +            The full range of focus distances is scanned just as with
> > > > > +            AfRangeNormal but this time including the very closest macro
> > > > > +            positions.
> > > > > +
> > > > > +  - AfSpeed:
> > > > > +      type: int32_t
> > > > > +      description: |
> > > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > > +        as quickly as possible or more steadily. For example, during video
> > > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > > +      enum:
> > > > > +        - name: AfSpeedNormal
> > > > > +          value: 0
> > > > > +          description: Move the lens at its usual speed.
> > > > > +        - name: AfSpeedFast
> > > > > +          value: 1
> > > > > +          description: Move the lens more quickly.
> > > > > +
> > > > > +  - AfWindows:
> > > > > +      type: Rectangle
> > > > > +      description: |
> > > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > > +        image), as u0.16 format numbers.
> > > >
> > > > I don't like too much the "or if unavailable", as it means that the
> > > > reference rectangle will change depending on the platform
> > > > capabilities.
> >
> > I guess what I want is (like in the case of Android below), to specify
> > AF regions relative to the actual output image. If a platform supports
> > digital zoom (through ScalerCrop, I assume), the regions must be
> > relative to that part of the image. If there is no ScalerCrop, then do
> > we assume the platform doesn't support digital zoom? But the basic
> > idea is the same, the numbers supplied should be relative to the
> > output image in this case too.
> >
>
> Thing is that there's no "output image size" as two streams produced
> from the same camera might have different output sizes ?

You're right that there might be several outputs. So what do I really
mean? I'm sort of looking for the rectangle in the sensor image that
is required to make all the outputs, but I think the ScalerCrop
encapsulates that idea reasonably well. Where there isn't a
ScalerCrop,  I think we use the active pixel area instead.

>
> > Is there a better way to explain this, do you think?
> >
> > > >
> > > > I understand your argument in reply to Jean-Michel comment on the
> > > > previous version of the series (quoted here for reference)
> > > >
> > > > "The thing that bothers me a bit about the full pixel array is that, if
> > > > you use digital zoom, then you will have to update your AfWindows as
> > > > they might otherwise lie outside your final image! If it's defined as
> > > > being a region of the scaler crop, then it automatically stays the
> > > > same (to the application, at least, the pipeline handler probably has
> > > > some work to do).
> > > >
> > > > I note that the Android link you give below says "Focus areas are
> > > > relative to the current field of view (getZoom())", which I guess is
> > > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > > is easier to work with than the u0.16 format that I specified?"
> > > >
> > > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > > estimating the contrast, will it do so before or after the ScalerCrop
> > > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > > do so on the full pixel array size, the portion of if that gets
> > > > processed or on the final image ?
> >
> > In the case of PDAF, we can presumably use information from whatever
> > part of the image that the sensor is outputting.
> >
> > For CDAF (contrast detect AF) I guess it's possible that focus
> > statistics could be computed either before or after any cropping is
> > done in the ISP. (Maybe there are some platforms where cropping could
> > be delegated to the sensor?) So I'd guess that using any focus
> > statistics from outside the final FoV may actually be impossible,
> > depending on the platform. But even where you can do this, I don't
> > believe it's what anyone would want (and Android seems to behave like
> > this).
> >
> > > >
> > > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > > rectangle, so that they are applied to the actual FOV before being
> > > > passed to the AF algorithm ?
> >
> > Not quite sure what you meant here. I'd expect the ISP to generate
> > focus statistics from the raw Bayer pixels that it gets, without too
> > much processing applied to them. So the job will be to take rectangles
> > defined in the final FoV and translate them into the image coming
> > directly from the sensor.
> >
>
> What I meant is: if we specify the AfWindows relative to the full
> pixel array, can't we then re-scale it to final cropped output ? or if
> no copping on the sensor's output frame size.
>
> This would be similar to what you do with ScalerCrop already ?
>
>         void RPiCameraData::applyScalerCrop(const ControlList &controls) {
>
>                 Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
>
>                 /* Create a version of the crop scaled to ISP (camera mode) pixels. */
>                 Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
>                 ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
>         }
>
> But I guess I'm reasoning in absolute coordinates, not in proportions
> like you have proposed here.

I think the question here is whether to use some kind of fractional
coordinates for describing the part of the ScalerCrop that we want AF
to use, or if we simply use pixel coordinates within the ScalerCrop
(or active pixel array), which would then look rather like the code
above.

I suppose I don't particularly mind. If we use actual pixel
coordinates it's maybe a bit less clear that some other scaling
(shrinking to lie within the ScalerCrop) might be necessary, but if
we're happy that this is going to happen anyway then I think that
would work for me.

>
> > > >
> > > > One last point about the unit: is the below representation correct,
> > > > when it comes to use a Rectangle to represent a proportion ?
> > > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > > pipeline report 12 rectangles like the ones below, for applications to
> > > > chose from ?
> > > >
> > > >       (0,0)
> > > >         +-----+-----+-----+-----+
> > > >         |     |     |     |     |
> > > >         +-----+-----+-----+-----+
> > > >         |     |     |     |     |
> > > >         +-----+-----+-----+-----+
> > > >         |     |     |     |     |
> > > >         +-----+-----+-----+-----+
> > > >
> > > > AfWindows = {
> > > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > > };
> > > >
> > > > My understanding is that the grid is fixed and depends on the platform
> > > > capabilities. I wonder if we shouldn't allow applications to select
> > > > grids by index.
> >
> > I think there is a problem about determining the AF capabilities of a
> > platform. I could imagine some platforms will only accept a regular
> > grid of regions, but others might let you put them in arbitrary
> > places. Or a mixture of both. To be honest, I'm not sure how we handle
> > that sort of thing. Anyone have any good ideas?
> >
>
> Exactly, that's why having proportions as the ones I tried to sketch
> out above doesn't really fit as a model in my head. Each platform
> would have its own 'grid', do we need to expose it to applications ?
>
> Can't we let application specify Rectangles with absolute coordinates
> (relative to PixelArray or else) and let the IPA/PH map those areas on
> their grids ?

I really don't know how to deal with this. Some ISPs will have grids
of some sort, some may have windows in arbitrary locations, different
numbers and sizes of each depending on the platform, and some may do
both. Typically I would imagine many applications wanting to use grids
much of the time, but perhaps arbitrary rectangles for following
objects, people and faces.

Perhaps having an AfWindowGrid control and an AfWindows array control
of Rectangles could work? For the former, perhaps we leave the
decisions on the size of the grid to the platform/algorithm - I'd be
worried about having so many parameters that it would be difficult to
use. (You'd have to think about the size of the grid, the spacing of
rectangles in the grid etc... but perhaps Android requires us to
expose this sort of thing?) Perhaps someone knows what Android would
require here?

>
> > > >
> > > > Also, do we need a property to report the grid structure and allows
> > > > application to select them from, or do you see this conveyed through a
> > > > ControlInfo ?
> >
> > I wasn't quite sure what the question was here. Are you talking about
> > selecting the configuration for the AfWindows, or selecting focus
> > statistics from particular regions? Perhaps an example would help me
> > here!
> >
>
> I'm talking about how would an application know how many spots it can
> select from, in other words, how do we let an application know that,
> looking at the above example, each grid is 1/4 in width and 1/3 in
> height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
> 4x3 one.
>
> Or maybe I got it all wrong, and you where implying already that the
> here specified Rectangles cover arbitrary parts of the image and it's
> the IPA that has to match them on their grids.

Or maybe we treat grids as a whole different thing? (as per the
comments just above)

>
> > > >
> > > >
> > > > > +
> > > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > > +        system will choose its own single default window in the centre of the
> > > > > +        image.
> > > > > +
> > > > > +        The details of how the windows are used are platform dependent. We note
> > > > > +        that when there is more than one AF window, a typical implementation
> > > > > +        might find the optimal focus position for each one and finally select
> > > > > +        the window closest to the camera.
> > > >
> > > > What window is more close to the camera ? My understanding is that
> > > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > > they all sit on the same focal plan :)
> >
> > Sorry, by "the window closest to the camera" I mean "the window that
> > is showing objects in the real world closest to the camera". I'll try
> > to re-word that a bit.
> >
>
> Yeah sorry, I could have realized that myself :)
>
> > > >
> > > > > +
> > > > > +        size: [n]
> > > > > +
> > > > > +  - AfTrigger:
> > > > > +      type: int32_t
> > > > > +      description: |
> > > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > > +        and can also be used to terminate a scan early.
> > > > > +
> > > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > > +
> > > > > +      enum:
> > > > > +        - name: AfTriggerStart
> > > > > +          value: 0
> > > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > > +        - name: AfTriggerCancel
> > > > > +          value: 1
> > > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > > +            anywhere else. Ignored if no scan is in progress.
> > > > > +
> > > > > +  - AfPause:
> > > > > +      type: int32_t
> > > > > +      description: |
> > > > > +        This control has no effect except when in continuous autofocus mode
> > > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > > +        (for example) images are captured. The algorithm remains inactive
> > > > > +        until it is instructed to resume.
> > > > > +
> > > > > +      enum:
> > > > > +        - name: AfPauseImmediate
> > > > > +          value: 0
> > > > > +          description: |
> > > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > > +            when it received this control.
> > > > > +        - name: AfPauseDeferred
> > > > > +          value: 1
> > > > > +          description: |
> > > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > > +
> > > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > > +            AfPauseState will change to AfPauseStatePaused.
> > > >
> > > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > > align it back ?
> >
> > Sure, I'll have a look!
> >
> > > >
> > > > > +        - name: AfPauseResume
> > > > > +          value: 2
> > > > > +          description: |
> > > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > > +            from exactly where it left off, and AfPauseState will report
> > > > > +            AfPauseStateRunning.
> > > > > +
> > > > > +  - LensPosition:
> > > > > +      type: float
> > > > > +      description: |
> > > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > > +        and also reports back the position of the lens for each frame.
> > > > > +
> > > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > > +        AfModeManual.
> > > > > +
> > > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > > +        values are permitted. For example:
> > > > > +        0 moves the lens to infinity.
> > > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > > +        1 moves the lens to the hyperfocal position.
> > > > > +        And larger values will focus the lens ever closer.
> > > > > +
> > > >
> > > > This seems good to me, albeit my understanding of the issue is limited :)
> > > >
> > >
> > > Do we need a property to report the lens hyperfocal distance ?
> >
> > Yes, sometimes. Android distinguishes between calibrated and
> > uncalibrated cameras. For a calibrated camera an application should be
> > able to set the focal distance to (for example) 1m, and given the
> > units I've proposed, you would need to know the hyperfocal distance to
> > calculate the right number.
>
> Right, so we need a LensHyperfocalDistance or something for calibrated
> lenses ?

I think so.

>
> >
> > For an uncalibrated camera I don't think it matters. Many applications
> > would simply set the lens position to 1.0 (meaning hyperfocal) and
> > you'll get the "best overall" focal distance (largest depth of field),
> > and that's it.
>
> I think this part would be relevant to be captured in the control
> description.

Yes, I can add something about that.

>
> >
> > There's also a need to know the allowable range of lens movement. This
> > will depend on the type of module (knowing the lens driver isn't
> > enough). But I don't see that applications need to know this, I think
> > the pipeline handler and AF implementation should take care of it.
> >
>
> I think so. When it comes to the actual lens movement range I don't
> think it's relevant for applications. But do the actual lens movement
> range maps to a range of LensPosition values that can be specified ? I
> guess it's hard to map them on un-calibrated lenses, but for
> calibrated ones a LensPositionRanges whose ControlInfo informs
> application on the range's min and max might make sense ? (Please note
> that this could also be used for the use case we discussed about
> wanting to limit the lens range search scope)

I think I would expect a calibrated lens would have to have some kind
of property that says what the nearest and furthest focal distances
are (the furthest usually being zero, meaning infinity). For an
uncalibrated lens things will only ever be "approximately correct".

I could imagine this being used in conjunction with the hypothetical
AfRangeCustom to construct custom search ranges for particular
applications (in future, perhaps).

Thanks again!
David

>
> > >
> > >
> > > > > +  - AfState:
> > > > > +      type: int32_t
> > > > > +      description: |
> > > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > > +        The possible state changes are described below, though we note the following
> > > > > +        state transitions that occur when the AfMode is changed.
> > > > > +
> > > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > > +        LensPosition control must subsequently be used to achieve this.
> > > > > +
> > > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > > >
> > > > s/unless/until ?
> > > > s/at the same time// ?
> >
> > I'm trying to describe what happens to the AfState when you send
> > AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> > AfTriggerStart). The only occasion when you won't see AfStateIdle will
> > be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> > think this reads OK to me, but I'll see if I can make the wording any
> > clearer.
>
> I see. I didn't get it, sorry. If it's fine with you, keep this
> version!
>
> >
> > > >
> > > > > +        case it will (start a scan and) report AfStateScanning.
> > > > > +
> > > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > > +        report AfStateScanning.
> > > > > +
> > > > > +      enum:
> > > > > +        - name: AfStateReset
> > > > > +          value: 0
> > > > > +          description: |
> > > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > +            scan was cancelled.
> > > > > +        - name: AfStateScanning
> > > > > +          value: 1
> > > > > +          description: |
> > > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > > +            move back to AfStateReset or, if the scan actually completes
> > > > > +            before the cancel request is processed, to one of
> > > > > +            AfStateFocused or AfStateFailed.
> > > > > +
> > > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > > >
> > > > s/be be/is ?
> >
> > Yes, thanks!
> >
> > > >
> > > > > +            at which point it may enter this state spontaneously whenever it
> > > > > +            determines that a rescan is needed.
> > > > > +        - name: AfStateFocused
> > > > > +          value: 2
> > > > > +          description: |
> > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > > +            the image is now in focus.
> > > > > +        - name: AfStateFailed
> > > > > +          value: 3
> > > > > +          description: |
> > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > > +            find a good focus position.
> > > > > +
> > > > > +  - AfPauseState:
> > > > > +      type: int32_t
> > > > > +      description: |
> > > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > > +        as soon as any in-progress scan completes).
> > > > > +
> > > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > > +
> > > > > +      enum:
> > > > > +        - name: AfPauseStateRunning
> > > > > +          value: 0
> > > > > +          description: |
> > > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > > +            spontaneously.
> > > > > +        - name: AfPauseStatePausing
> > > > > +          value: 1
> > > > > +          description: |
> > > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > > +            as soon as any in-progress scan completes (and then report
> > > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > > +            the AfPauseResume control is sent.
> > > > > +        - name: AfPauseStatePaused
> > > > > +          value: 2
> > > > > +          description: |
> > > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > > +            will occur until the AfPauseResume control is sent.
> > > > > +
> > > >
> > > > Very nice overall! With a few more clarifications I think this is good
> > > > to go!
> > > >
> > > > Thanks
> > > >    j
> >
> > Thanks!
> > David
> >
> > > >
> > > > >    # ----------------------------------------------------------------------------
> > > > >    # Draft controls section
> > > > >
> > > > > @@ -406,27 +664,6 @@ controls:
> > > > >              The camera will cancel any active or completed metering sequence.
> > > > >              The AE algorithm is reset to its initial state.
> > > > >
> > > > > -  - AfTrigger:
> > > > > -      type: int32_t
> > > > > -      draft: true
> > > > > -      description: |
> > > > > -       Control for AF trigger. Currently identical to
> > > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > > -
> > > > > -        Whether the camera device will trigger autofocus for this request.
> > > > > -      enum:
> > > > > -        - name: AfTriggerIdle
> > > > > -          value: 0
> > > > > -          description: The trigger is idle.
> > > > > -        - name: AfTriggerStart
> > > > > -          value: 1
> > > > > -          description: The AF routine is started by the camera.
> > > > > -        - name: AfTriggerCancel
> > > > > -          value: 2
> > > > > -          description: |
> > > > > -            The camera will cancel any active trigger and the AF routine is
> > > > > -            reset to its initial state.
> > > > > -
> > > > >    - NoiseReductionMode:
> > > > >        type: int32_t
> > > > >        draft: true
> > > > > @@ -507,45 +744,6 @@ controls:
> > > > >              The AE algorithm has started a pre-capture metering session.
> > > > >              \sa AePrecaptureTrigger
> > > > >
> > > > > -  - AfState:
> > > > > -      type: int32_t
> > > > > -      draft: true
> > > > > -      description: |
> > > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > > -       ANDROID_CONTROL_AF_STATE.
> > > > > -
> > > > > -        Current state of the AF algorithm.
> > > > > -      enum:
> > > > > -        - name: AfStateInactive
> > > > > -          value: 0
> > > > > -          description: The AF algorithm is inactive.
> > > > > -        - name: AfStatePassiveScan
> > > > > -          value: 1
> > > > > -          description: |
> > > > > -            AF is performing a passive scan of the scene in continuous
> > > > > -            auto-focus mode.
> > > > > -        - name: AfStatePassiveFocused
> > > > > -          value: 2
> > > > > -          description: |
> > > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > > -        - name: AfStateActiveScan
> > > > > -          value: 3
> > > > > -          description: |
> > > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > > -            \sa AfTrigger
> > > > > -        - name: AfStateFocusedLock
> > > > > -          value: 4
> > > > > -          description: |
> > > > > -            AF believes has focused correctly and has locked focus.
> > > > > -        - name: AfStateNotFocusedLock
> > > > > -          value: 5
> > > > > -          description: |
> > > > > -            AF has not been able to focus and has locked.
> > > > > -        - name: AfStatePassiveUnfocused
> > > > > -          value: 6
> > > > > -          description: |
> > > > > -            AF has completed a passive scan without finding focus.
> > > > > -
> > > > >    - AwbState:
> > > > >        type: int32_t
> > > > >        draft: true
> > > > > --
> > > > > 2.30.2
> > > > >

Hi David,

On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
> Hi Jacopo
>
> Thanks for the message. Some replies and more questions below!
>
> On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
> >
> > Hi David,
> >
> > On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> > > Hi Jacopo
> > >
> > > Thanks for the quick reply!
> > >
> > > On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > >
> > > > Ah, forgot one thing!
> > > >
> > > > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > > > Hi David,
> > > > >   thanks for this new version
> > > > >
> > > > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > > > This patch describes a series of controls that allow applications to
> > > > > > drive AF algorithms:
> > > > > >
> > > > > > AfMode - manual, auto or continuous
> > > > > > AfRange - full, macro or normal
> > > > > > AfSpeed - fast or slowwith
> > > > > > AfWindows - AF window locations
> > > > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > > > AfPause - pause continuous AF
> > > > > > LensPosition - set or retrieve position of lens
> > > > > > AfState - reports whether scanning/success/failure
> > > > > > AfPauseState - reports whether continuous AF paused or not
> > > > > >
> > > > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > > > ---
> > > > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > > > >
> > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > > > index 9d4638ae..eccf18bc 100644
> > > > > > --- a/src/libcamera/control_ids.yaml
> > > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > > @@ -381,6 +381,264 @@ controls:
> > > > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > > > >          use case.
> > > > > >
> > > > > > +  - AfMode:
> > > > > > +      type: int32_t
> > > > > > +      description: |
> > > > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > > > +        are allowed to set a new mode, and to send additional controls for
> > > > > > +        that new mode, in the same request.
> > > > > > +
> > > > > > +        An implementation may choose not to implement all the modes.
> > > > > > +
> > > > > > +      enum:
> > > > > > +        - name: AfModeManual
> > > > > > +          value: 0
> > > > > > +          description: |
> > > > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > > > +            perform any action nor move the lens of its own accord, but an
> > > > > > +            application can set controls to move the lens.
> > > > > > +
> > > > >
> > > > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > > > when in manual mode, I would say so explicitly
> > > > >
> > > > >                "... but an application can specify the desired lens
> > > > >                position using the LensPosition control."
> > >
> > > Yes, will do.
> > >
> > > > >
> > > > > > +            In this mode the AfState will always report AfStateReset.
> > > > >
> > > > > I'm tempted to say AfState is useless and can not be reported at all
> > > > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > > > don't have better alternatives to propose though.
> > >
> > > Don't really mind either. I think reporting stuff all the time might
> > > be slightly easier, at least it's always obvious what to expect...
> > >
> > > > >
> > > > > Reading the AfStateReset description
> > > > >
> > > > >         - name: AfStateReset
> > > > >           value: 0
> > > > >           description: |
> > > > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > >             scan was cancelled.
> > > > >
> > > > > I wonder if this isn't better named AfStateIdle
> > >
> > > Yes, also fine with that.
> > >
> > > > >
> > > > >
> > > > > > +        - name: AfModeAuto
> > > > > > +          value: 1
> > > > > > +          description: |
> > > > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > > > +            will never move the lens or change state unless the AfTrigger
> > > > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > > > +            focus scan, the results of which will be reported by AfState.
> > > > > > +
> > > > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > > > +            mode while a scan is in progress, the scan is cancelled
> > > > > > +            immediately, without waiting for the scan to finish.
> > > > > > +
> > > > > > +            When first entering this mode the AfState will report
> > > > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > > > +            report AfStateScanning for a period before spontaneously
> > > > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > > > +            the outcome of the scan. It will remain in this state until
> > > > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > > > +            cancelled (without changing to another mode), AfState will return
> > > > > > +            to AfStateReset.
> > > > > > +        - name: AfModeContinuous
> > > > > > +          value: 2
> > > > > > +          description: |
> > > > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > > > +            intervention. The AfState still reports whether the algorithm is
> > > > > > +            currently scanning or not, though the application has no ability
> > > > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > > > +            move the lens for itself.
> > > > > > +
> > > > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > > > +
> > > > > > +  - AfRange:
> > > > > > +      type: int32_t
> > > > > > +      description: |
> > > > > > +        Control to set the range of focus distances that is scanned. An
> > > > > > +        implementation may choose not to implement all the options here.
> > > > > > +      enum:
> > > > > > +        - name: AfRangeNormal
> > > > > > +          value: 0
> > > > > > +          description: |
> > > > > > +            A wide range of focus distances is scanned, all the way from
> > > > > > +            infinity down to close distances, though depending on the
> > > > > > +            implementation, possibly not including the very closest macro
> > > > > > +            positions.
> > > > > > +        - name: AfRangeMacro
> > > > > > +          value: 1
> > > > > > +          description: Only close distances are scanned.
> > > > > > +        - name: AfRangeFull
> > > > > > +          value: 2
> > > > > > +          description: |
> > > > > > +            The full range of focus distances is scanned just as with
> > > > > > +            AfRangeNormal but this time including the very closest macro
> > > > > > +            positions.
> > > > > > +
> > > > > > +  - AfSpeed:
> > > > > > +      type: int32_t
> > > > > > +      description: |
> > > > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > > > +        as quickly as possible or more steadily. For example, during video
> > > > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > > > +      enum:
> > > > > > +        - name: AfSpeedNormal
> > > > > > +          value: 0
> > > > > > +          description: Move the lens at its usual speed.
> > > > > > +        - name: AfSpeedFast
> > > > > > +          value: 1
> > > > > > +          description: Move the lens more quickly.
> > > > > > +
> > > > > > +  - AfWindows:
> > > > > > +      type: Rectangle
> > > > > > +      description: |
> > > > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > > > +        image), as u0.16 format numbers.
> > > > >
> > > > > I don't like too much the "or if unavailable", as it means that the
> > > > > reference rectangle will change depending on the platform
> > > > > capabilities.
> > >
> > > I guess what I want is (like in the case of Android below), to specify
> > > AF regions relative to the actual output image. If a platform supports
> > > digital zoom (through ScalerCrop, I assume), the regions must be
> > > relative to that part of the image. If there is no ScalerCrop, then do
> > > we assume the platform doesn't support digital zoom? But the basic
> > > idea is the same, the numbers supplied should be relative to the
> > > output image in this case too.
> > >
> >
> > Thing is that there's no "output image size" as two streams produced
> > from the same camera might have different output sizes ?
>
> You're right that there might be several outputs. So what do I really
> mean? I'm sort of looking for the rectangle in the sensor image that
> is required to make all the outputs, but I think the ScalerCrop
> encapsulates that idea reasonably well. Where there isn't a
> ScalerCrop,  I think we use the active pixel area instead.
>

Gotcha! Indeed scaler crop defines the area of the sensor's output
frame used to generate all streams... Thanks for sticking to it and
help me straight my thoughts.

I still don't like the "if not available part" :(

I'll try to rope in others to get their opinions. What concerns me is
that applications will behave differently on platforms where scaler
crop is available compared to ones where it is not. Maybe I'm just
over-concerned ?


> >
> > > Is there a better way to explain this, do you think?
> > >
> > > > >
> > > > > I understand your argument in reply to Jean-Michel comment on the
> > > > > previous version of the series (quoted here for reference)
> > > > >
> > > > > "The thing that bothers me a bit about the full pixel array is that, if
> > > > > you use digital zoom, then you will have to update your AfWindows as
> > > > > they might otherwise lie outside your final image! If it's defined as
> > > > > being a region of the scaler crop, then it automatically stays the
> > > > > same (to the application, at least, the pipeline handler probably has
> > > > > some work to do).
> > > > >
> > > > > I note that the Android link you give below says "Focus areas are
> > > > > relative to the current field of view (getZoom())", which I guess is
> > > > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > > > is easier to work with than the u0.16 format that I specified?"
> > > > >
> > > > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > > > estimating the contrast, will it do so before or after the ScalerCrop
> > > > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > > > do so on the full pixel array size, the portion of if that gets
> > > > > processed or on the final image ?
> > >
> > > In the case of PDAF, we can presumably use information from whatever
> > > part of the image that the sensor is outputting.
> > >
> > > For CDAF (contrast detect AF) I guess it's possible that focus
> > > statistics could be computed either before or after any cropping is
> > > done in the ISP. (Maybe there are some platforms where cropping could
> > > be delegated to the sensor?) So I'd guess that using any focus
> > > statistics from outside the final FoV may actually be impossible,
> > > depending on the platform. But even where you can do this, I don't
> > > believe it's what anyone would want (and Android seems to behave like
> > > this).
> > >
> > > > >
> > > > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > > > rectangle, so that they are applied to the actual FOV before being
> > > > > passed to the AF algorithm ?
> > >
> > > Not quite sure what you meant here. I'd expect the ISP to generate
> > > focus statistics from the raw Bayer pixels that it gets, without too
> > > much processing applied to them. So the job will be to take rectangles
> > > defined in the final FoV and translate them into the image coming
> > > directly from the sensor.
> > >
> >
> > What I meant is: if we specify the AfWindows relative to the full
> > pixel array, can't we then re-scale it to final cropped output ? or if
> > no copping on the sensor's output frame size.
> >
> > This would be similar to what you do with ScalerCrop already ?
> >
> >         void RPiCameraData::applyScalerCrop(const ControlList &controls) {
> >
> >                 Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
> >
> >                 /* Create a version of the crop scaled to ISP (camera mode) pixels. */
> >                 Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
> >                 ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
> >         }
> >
> > But I guess I'm reasoning in absolute coordinates, not in proportions
> > like you have proposed here.
>
> I think the question here is whether to use some kind of fractional
> coordinates for describing the part of the ScalerCrop that we want AF
> to use, or if we simply use pixel coordinates within the ScalerCrop
> (or active pixel array), which would then look rather like the code
> above.
>
> I suppose I don't particularly mind. If we use actual pixel
> coordinates it's maybe a bit less clear that some other scaling
> (shrinking to lie within the ScalerCrop) might be necessary, but if
> we're happy that this is going to happen anyway then I think that
> would work for me.
>

Let's hear from others as well!

> >
> > > > >
> > > > > One last point about the unit: is the below representation correct,
> > > > > when it comes to use a Rectangle to represent a proportion ?
> > > > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > > > pipeline report 12 rectangles like the ones below, for applications to
> > > > > chose from ?
> > > > >
> > > > >       (0,0)
> > > > >         +-----+-----+-----+-----+
> > > > >         |     |     |     |     |
> > > > >         +-----+-----+-----+-----+
> > > > >         |     |     |     |     |
> > > > >         +-----+-----+-----+-----+
> > > > >         |     |     |     |     |
> > > > >         +-----+-----+-----+-----+
> > > > >
> > > > > AfWindows = {
> > > > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > > > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > > > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > > > };
> > > > >
> > > > > My understanding is that the grid is fixed and depends on the platform
> > > > > capabilities. I wonder if we shouldn't allow applications to select
> > > > > grids by index.
> > >
> > > I think there is a problem about determining the AF capabilities of a
> > > platform. I could imagine some platforms will only accept a regular
> > > grid of regions, but others might let you put them in arbitrary
> > > places. Or a mixture of both. To be honest, I'm not sure how we handle
> > > that sort of thing. Anyone have any good ideas?
> > >
> >
> > Exactly, that's why having proportions as the ones I tried to sketch
> > out above doesn't really fit as a model in my head. Each platform
> > would have its own 'grid', do we need to expose it to applications ?
> >
> > Can't we let application specify Rectangles with absolute coordinates
> > (relative to PixelArray or else) and let the IPA/PH map those areas on
> > their grids ?
>
> I really don't know how to deal with this. Some ISPs will have grids
> of some sort, some may have windows in arbitrary locations, different
> numbers and sizes of each depending on the platform, and some may do
> both. Typically I would imagine many applications wanting to use grids
> much of the time, but perhaps arbitrary rectangles for following
> objects, people and faces.

Indeed. This is an advanced feature for which it might be expected
application to know what the platform can provide maybe ? Describing
all possible options seems very complex, and we'll never be able to
express all of them.

I would be fine stabilizing on a simpler grid-based version of AfWindows
for now and improve on top, maybe ?

>
> Perhaps having an AfWindowGrid control and an AfWindows array control
> of Rectangles could work? For the former, perhaps we leave the
> decisions on the size of the grid to the platform/algorithm - I'd be
> worried about having so many parameters that it would be difficult to
> use. (You'd have to think about the size of the grid, the spacing of
> rectangles in the grid etc... but perhaps Android requires us to
> expose this sort of thing?) Perhaps someone knows what Android would
> require here?
>
> >
> > > > >
> > > > > Also, do we need a property to report the grid structure and allows
> > > > > application to select them from, or do you see this conveyed through a
> > > > > ControlInfo ?
> > >
> > > I wasn't quite sure what the question was here. Are you talking about
> > > selecting the configuration for the AfWindows, or selecting focus
> > > statistics from particular regions? Perhaps an example would help me
> > > here!
> > >
> >
> > I'm talking about how would an application know how many spots it can
> > select from, in other words, how do we let an application know that,
> > looking at the above example, each grid is 1/4 in width and 1/3 in
> > height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
> > 4x3 one.
> >
> > Or maybe I got it all wrong, and you where implying already that the
> > here specified Rectangles cover arbitrary parts of the image and it's
> > the IPA that has to match them on their grids.
>
> Or maybe we treat grids as a whole different thing? (as per the
> comments just above)
>
> >
> > > > >
> > > > >
> > > > > > +
> > > > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > > > +        system will choose its own single default window in the centre of the
> > > > > > +        image.
> > > > > > +
> > > > > > +        The details of how the windows are used are platform dependent. We note
> > > > > > +        that when there is more than one AF window, a typical implementation
> > > > > > +        might find the optimal focus position for each one and finally select
> > > > > > +        the window closest to the camera.
> > > > >
> > > > > What window is more close to the camera ? My understanding is that
> > > > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > > > they all sit on the same focal plan :)
> > >
> > > Sorry, by "the window closest to the camera" I mean "the window that
> > > is showing objects in the real world closest to the camera". I'll try
> > > to re-word that a bit.
> > >
> >
> > Yeah sorry, I could have realized that myself :)
> >
> > > > >
> > > > > > +
> > > > > > +        size: [n]
> > > > > > +
> > > > > > +  - AfTrigger:
> > > > > > +      type: int32_t
> > > > > > +      description: |
> > > > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > > > +        and can also be used to terminate a scan early.
> > > > > > +
> > > > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > > > +
> > > > > > +      enum:
> > > > > > +        - name: AfTriggerStart
> > > > > > +          value: 0
> > > > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > > > +        - name: AfTriggerCancel
> > > > > > +          value: 1
> > > > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > > > +            anywhere else. Ignored if no scan is in progress.
> > > > > > +
> > > > > > +  - AfPause:
> > > > > > +      type: int32_t
> > > > > > +      description: |
> > > > > > +        This control has no effect except when in continuous autofocus mode
> > > > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > > > +        (for example) images are captured. The algorithm remains inactive
> > > > > > +        until it is instructed to resume.
> > > > > > +
> > > > > > +      enum:
> > > > > > +        - name: AfPauseImmediate
> > > > > > +          value: 0
> > > > > > +          description: |
> > > > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > > > +            when it received this control.
> > > > > > +        - name: AfPauseDeferred
> > > > > > +          value: 1
> > > > > > +          description: |
> > > > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > > > +
> > > > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > > > +            AfPauseState will change to AfPauseStatePaused.
> > > > >
> > > > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > > > align it back ?
> > >
> > > Sure, I'll have a look!
> > >
> > > > >
> > > > > > +        - name: AfPauseResume
> > > > > > +          value: 2
> > > > > > +          description: |
> > > > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > > > +            from exactly where it left off, and AfPauseState will report
> > > > > > +            AfPauseStateRunning.
> > > > > > +
> > > > > > +  - LensPosition:
> > > > > > +      type: float
> > > > > > +      description: |
> > > > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > > > +        and also reports back the position of the lens for each frame.
> > > > > > +
> > > > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > > > +        AfModeManual.
> > > > > > +
> > > > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > > > +        values are permitted. For example:
> > > > > > +        0 moves the lens to infinity.
> > > > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > > > +        1 moves the lens to the hyperfocal position.
> > > > > > +        And larger values will focus the lens ever closer.
> > > > > > +
> > > > >
> > > > > This seems good to me, albeit my understanding of the issue is limited :)
> > > > >
> > > >
> > > > Do we need a property to report the lens hyperfocal distance ?
> > >
> > > Yes, sometimes. Android distinguishes between calibrated and
> > > uncalibrated cameras. For a calibrated camera an application should be
> > > able to set the focal distance to (for example) 1m, and given the
> > > units I've proposed, you would need to know the hyperfocal distance to
> > > calculate the right number.
> >
> > Right, so we need a LensHyperfocalDistance or something for calibrated
> > lenses ?
>
> I think so.
>
> >
> > >
> > > For an uncalibrated camera I don't think it matters. Many applications
> > > would simply set the lens position to 1.0 (meaning hyperfocal) and
> > > you'll get the "best overall" focal distance (largest depth of field),
> > > and that's it.
> >
> > I think this part would be relevant to be captured in the control
> > description.
>
> Yes, I can add something about that.
>
> >
> > >
> > > There's also a need to know the allowable range of lens movement. This
> > > will depend on the type of module (knowing the lens driver isn't
> > > enough). But I don't see that applications need to know this, I think
> > > the pipeline handler and AF implementation should take care of it.
> > >
> >
> > I think so. When it comes to the actual lens movement range I don't
> > think it's relevant for applications. But do the actual lens movement
> > range maps to a range of LensPosition values that can be specified ? I
> > guess it's hard to map them on un-calibrated lenses, but for
> > calibrated ones a LensPositionRanges whose ControlInfo informs
> > application on the range's min and max might make sense ? (Please note
> > that this could also be used for the use case we discussed about
> > wanting to limit the lens range search scope)
>
> I think I would expect a calibrated lens would have to have some kind
> of property that says what the nearest and furthest focal distances
> are (the furthest usually being zero, meaning infinity). For an
> uncalibrated lens things will only ever be "approximately correct".


Care to add the property in next version, or can we do it on top with
a separate patch ?

Thanks
   j

>
> I could imagine this being used in conjunction with the hypothetical
> AfRangeCustom to construct custom search ranges for particular
> applications (in future, perhaps).
>
> Thanks again!
> David
>
> >
> > > >
> > > >
> > > > > > +  - AfState:
> > > > > > +      type: int32_t
> > > > > > +      description: |
> > > > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > > > +        The possible state changes are described below, though we note the following
> > > > > > +        state transitions that occur when the AfMode is changed.
> > > > > > +
> > > > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > > > +        LensPosition control must subsequently be used to achieve this.
> > > > > > +
> > > > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > > > >
> > > > > s/unless/until ?
> > > > > s/at the same time// ?
> > >
> > > I'm trying to describe what happens to the AfState when you send
> > > AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> > > AfTriggerStart). The only occasion when you won't see AfStateIdle will
> > > be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> > > think this reads OK to me, but I'll see if I can make the wording any
> > > clearer.
> >
> > I see. I didn't get it, sorry. If it's fine with you, keep this
> > version!
> >
> > >
> > > > >
> > > > > > +        case it will (start a scan and) report AfStateScanning.
> > > > > > +
> > > > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > > > +        report AfStateScanning.
> > > > > > +
> > > > > > +      enum:
> > > > > > +        - name: AfStateReset
> > > > > > +          value: 0
> > > > > > +          description: |
> > > > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > +            scan was cancelled.
> > > > > > +        - name: AfStateScanning
> > > > > > +          value: 1
> > > > > > +          description: |
> > > > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > > > +            move back to AfStateReset or, if the scan actually completes
> > > > > > +            before the cancel request is processed, to one of
> > > > > > +            AfStateFocused or AfStateFailed.
> > > > > > +
> > > > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > > > >
> > > > > s/be be/is ?
> > >
> > > Yes, thanks!
> > >
> > > > >
> > > > > > +            at which point it may enter this state spontaneously whenever it
> > > > > > +            determines that a rescan is needed.
> > > > > > +        - name: AfStateFocused
> > > > > > +          value: 2
> > > > > > +          description: |
> > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > > > +            the image is now in focus.
> > > > > > +        - name: AfStateFailed
> > > > > > +          value: 3
> > > > > > +          description: |
> > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > > > +            find a good focus position.
> > > > > > +
> > > > > > +  - AfPauseState:
> > > > > > +      type: int32_t
> > > > > > +      description: |
> > > > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > > > +        as soon as any in-progress scan completes).
> > > > > > +
> > > > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > > > +
> > > > > > +      enum:
> > > > > > +        - name: AfPauseStateRunning
> > > > > > +          value: 0
> > > > > > +          description: |
> > > > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > > > +            spontaneously.
> > > > > > +        - name: AfPauseStatePausing
> > > > > > +          value: 1
> > > > > > +          description: |
> > > > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > > > +            as soon as any in-progress scan completes (and then report
> > > > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > > > +            the AfPauseResume control is sent.
> > > > > > +        - name: AfPauseStatePaused
> > > > > > +          value: 2
> > > > > > +          description: |
> > > > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > > > +            will occur until the AfPauseResume control is sent.
> > > > > > +
> > > > >
> > > > > Very nice overall! With a few more clarifications I think this is good
> > > > > to go!
> > > > >
> > > > > Thanks
> > > > >    j
> > >
> > > Thanks!
> > > David
> > >
> > > > >
> > > > > >    # ----------------------------------------------------------------------------
> > > > > >    # Draft controls section
> > > > > >
> > > > > > @@ -406,27 +664,6 @@ controls:
> > > > > >              The camera will cancel any active or completed metering sequence.
> > > > > >              The AE algorithm is reset to its initial state.
> > > > > >
> > > > > > -  - AfTrigger:
> > > > > > -      type: int32_t
> > > > > > -      draft: true
> > > > > > -      description: |
> > > > > > -       Control for AF trigger. Currently identical to
> > > > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > > > -
> > > > > > -        Whether the camera device will trigger autofocus for this request.
> > > > > > -      enum:
> > > > > > -        - name: AfTriggerIdle
> > > > > > -          value: 0
> > > > > > -          description: The trigger is idle.
> > > > > > -        - name: AfTriggerStart
> > > > > > -          value: 1
> > > > > > -          description: The AF routine is started by the camera.
> > > > > > -        - name: AfTriggerCancel
> > > > > > -          value: 2
> > > > > > -          description: |
> > > > > > -            The camera will cancel any active trigger and the AF routine is
> > > > > > -            reset to its initial state.
> > > > > > -
> > > > > >    - NoiseReductionMode:
> > > > > >        type: int32_t
> > > > > >        draft: true
> > > > > > @@ -507,45 +744,6 @@ controls:
> > > > > >              The AE algorithm has started a pre-capture metering session.
> > > > > >              \sa AePrecaptureTrigger
> > > > > >
> > > > > > -  - AfState:
> > > > > > -      type: int32_t
> > > > > > -      draft: true
> > > > > > -      description: |
> > > > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > > > -       ANDROID_CONTROL_AF_STATE.
> > > > > > -
> > > > > > -        Current state of the AF algorithm.
> > > > > > -      enum:
> > > > > > -        - name: AfStateInactive
> > > > > > -          value: 0
> > > > > > -          description: The AF algorithm is inactive.
> > > > > > -        - name: AfStatePassiveScan
> > > > > > -          value: 1
> > > > > > -          description: |
> > > > > > -            AF is performing a passive scan of the scene in continuous
> > > > > > -            auto-focus mode.
> > > > > > -        - name: AfStatePassiveFocused
> > > > > > -          value: 2
> > > > > > -          description: |
> > > > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > > > -        - name: AfStateActiveScan
> > > > > > -          value: 3
> > > > > > -          description: |
> > > > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > > > -            \sa AfTrigger
> > > > > > -        - name: AfStateFocusedLock
> > > > > > -          value: 4
> > > > > > -          description: |
> > > > > > -            AF believes has focused correctly and has locked focus.
> > > > > > -        - name: AfStateNotFocusedLock
> > > > > > -          value: 5
> > > > > > -          description: |
> > > > > > -            AF has not been able to focus and has locked.
> > > > > > -        - name: AfStatePassiveUnfocused
> > > > > > -          value: 6
> > > > > > -          description: |
> > > > > > -            AF has completed a passive scan without finding focus.
> > > > > > -
> > > > > >    - AwbState:
> > > > > >        type: int32_t
> > > > > >        draft: true
> > > > > > --
> > > > > > 2.30.2
> > > > > >

Hi everyone,

On Tue, Apr 26, 2022 at 9:16 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
>
> Hi David,
>
> On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
> > Hi Jacopo
> >
> > Thanks for the message. Some replies and more questions below!
> >
> > On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > >
> > > Hi David,
> > >
> > > On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> > > > Hi Jacopo
> > > >
> > > > Thanks for the quick reply!
> > > >
> > > > On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > >
> > > > > Ah, forgot one thing!
> > > > >
> > > > > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > > > > Hi David,
> > > > > >   thanks for this new version
> > > > > >
> > > > > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > > > > This patch describes a series of controls that allow applications to
> > > > > > > drive AF algorithms:
> > > > > > >
> > > > > > > AfMode - manual, auto or continuous
> > > > > > > AfRange - full, macro or normal
> > > > > > > AfSpeed - fast or slowwith
> > > > > > > AfWindows - AF window locations
> > > > > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > > > > AfPause - pause continuous AF
> > > > > > > LensPosition - set or retrieve position of lens
> > > > > > > AfState - reports whether scanning/success/failure
> > > > > > > AfPauseState - reports whether continuous AF paused or not
> > > > > > >
> > > > > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > > > > ---
> > > > > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > > > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > > > > >
> > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > > > > index 9d4638ae..eccf18bc 100644
> > > > > > > --- a/src/libcamera/control_ids.yaml
> > > > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > > > @@ -381,6 +381,264 @@ controls:
> > > > > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > > > > >          use case.
> > > > > > >
> > > > > > > +  - AfMode:
> > > > > > > +      type: int32_t
> > > > > > > +      description: |
> > > > > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > > > > +        are allowed to set a new mode, and to send additional controls for
> > > > > > > +        that new mode, in the same request.
> > > > > > > +
> > > > > > > +        An implementation may choose not to implement all the modes.
> > > > > > > +
> > > > > > > +      enum:
> > > > > > > +        - name: AfModeManual
> > > > > > > +          value: 0
> > > > > > > +          description: |
> > > > > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > > > > +            perform any action nor move the lens of its own accord, but an
> > > > > > > +            application can set controls to move the lens.
> > > > > > > +
> > > > > >
> > > > > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > > > > when in manual mode, I would say so explicitly
> > > > > >
> > > > > >                "... but an application can specify the desired lens
> > > > > >                position using the LensPosition control."
> > > >
> > > > Yes, will do.
> > > >
> > > > > >
> > > > > > > +            In this mode the AfState will always report AfStateReset.
> > > > > >
> > > > > > I'm tempted to say AfState is useless and can not be reported at all
> > > > > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > > > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > > > > don't have better alternatives to propose though.
> > > >
> > > > Don't really mind either. I think reporting stuff all the time might
> > > > be slightly easier, at least it's always obvious what to expect...
> > > >
> > > > > >
> > > > > > Reading the AfStateReset description
> > > > > >
> > > > > >         - name: AfStateReset
> > > > > >           value: 0
> > > > > >           description: |
> > > > > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > >             scan was cancelled.
> > > > > >
> > > > > > I wonder if this isn't better named AfStateIdle
> > > >
> > > > Yes, also fine with that.
> > > >
> > > > > >
> > > > > >
> > > > > > > +        - name: AfModeAuto
> > > > > > > +          value: 1
> > > > > > > +          description: |
> > > > > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > > > > +            will never move the lens or change state unless the AfTrigger
> > > > > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > > > > +            focus scan, the results of which will be reported by AfState.
> > > > > > > +
> > > > > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > > > > +            mode while a scan is in progress, the scan is cancelled
> > > > > > > +            immediately, without waiting for the scan to finish.
> > > > > > > +
> > > > > > > +            When first entering this mode the AfState will report
> > > > > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > > > > +            report AfStateScanning for a period before spontaneously
> > > > > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > > > > +            the outcome of the scan. It will remain in this state until
> > > > > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > > > > +            cancelled (without changing to another mode), AfState will return
> > > > > > > +            to AfStateReset.
> > > > > > > +        - name: AfModeContinuous
> > > > > > > +          value: 2
> > > > > > > +          description: |
> > > > > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > > > > +            intervention. The AfState still reports whether the algorithm is
> > > > > > > +            currently scanning or not, though the application has no ability
> > > > > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > > > > +            move the lens for itself.
> > > > > > > +
> > > > > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > > > > +
> > > > > > > +  - AfRange:
> > > > > > > +      type: int32_t
> > > > > > > +      description: |
> > > > > > > +        Control to set the range of focus distances that is scanned. An
> > > > > > > +        implementation may choose not to implement all the options here.
> > > > > > > +      enum:
> > > > > > > +        - name: AfRangeNormal
> > > > > > > +          value: 0
> > > > > > > +          description: |
> > > > > > > +            A wide range of focus distances is scanned, all the way from
> > > > > > > +            infinity down to close distances, though depending on the
> > > > > > > +            implementation, possibly not including the very closest macro
> > > > > > > +            positions.
> > > > > > > +        - name: AfRangeMacro
> > > > > > > +          value: 1
> > > > > > > +          description: Only close distances are scanned.
> > > > > > > +        - name: AfRangeFull
> > > > > > > +          value: 2
> > > > > > > +          description: |
> > > > > > > +            The full range of focus distances is scanned just as with
> > > > > > > +            AfRangeNormal but this time including the very closest macro
> > > > > > > +            positions.
> > > > > > > +
> > > > > > > +  - AfSpeed:
> > > > > > > +      type: int32_t
> > > > > > > +      description: |
> > > > > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > > > > +        as quickly as possible or more steadily. For example, during video
> > > > > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > > > > +      enum:
> > > > > > > +        - name: AfSpeedNormal
> > > > > > > +          value: 0
> > > > > > > +          description: Move the lens at its usual speed.
> > > > > > > +        - name: AfSpeedFast
> > > > > > > +          value: 1
> > > > > > > +          description: Move the lens more quickly.
> > > > > > > +
> > > > > > > +  - AfWindows:
> > > > > > > +      type: Rectangle
> > > > > > > +      description: |
> > > > > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > > > > +        image), as u0.16 format numbers.
> > > > > >
> > > > > > I don't like too much the "or if unavailable", as it means that the
> > > > > > reference rectangle will change depending on the platform
> > > > > > capabilities.
> > > >
> > > > I guess what I want is (like in the case of Android below), to specify
> > > > AF regions relative to the actual output image. If a platform supports
> > > > digital zoom (through ScalerCrop, I assume), the regions must be
> > > > relative to that part of the image. If there is no ScalerCrop, then do
> > > > we assume the platform doesn't support digital zoom? But the basic
> > > > idea is the same, the numbers supplied should be relative to the
> > > > output image in this case too.
> > > >
> > >
> > > Thing is that there's no "output image size" as two streams produced
> > > from the same camera might have different output sizes ?
> >
> > You're right that there might be several outputs. So what do I really
> > mean? I'm sort of looking for the rectangle in the sensor image that
> > is required to make all the outputs, but I think the ScalerCrop
> > encapsulates that idea reasonably well. Where there isn't a
> > ScalerCrop,  I think we use the active pixel area instead.
> >
>
> Gotcha! Indeed scaler crop defines the area of the sensor's output
> frame used to generate all streams... Thanks for sticking to it and
> help me straight my thoughts.
>
> I still don't like the "if not available part" :(
>
> I'll try to rope in others to get their opinions. What concerns me is
> that applications will behave differently on platforms where scaler
> crop is available compared to ones where it is not. Maybe I'm just
> over-concerned ?
>
>
> > >
> > > > Is there a better way to explain this, do you think?
> > > >
> > > > > >
> > > > > > I understand your argument in reply to Jean-Michel comment on the
> > > > > > previous version of the series (quoted here for reference)
> > > > > >
> > > > > > "The thing that bothers me a bit about the full pixel array is that, if
> > > > > > you use digital zoom, then you will have to update your AfWindows as
> > > > > > they might otherwise lie outside your final image! If it's defined as
> > > > > > being a region of the scaler crop, then it automatically stays the
> > > > > > same (to the application, at least, the pipeline handler probably has
> > > > > > some work to do).
> > > > > >
> > > > > > I note that the Android link you give below says "Focus areas are
> > > > > > relative to the current field of view (getZoom())", which I guess is
> > > > > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > > > > is easier to work with than the u0.16 format that I specified?"
> > > > > >
> > > > > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > > > > estimating the contrast, will it do so before or after the ScalerCrop
> > > > > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > > > > do so on the full pixel array size, the portion of if that gets
> > > > > > processed or on the final image ?
> > > >
> > > > In the case of PDAF, we can presumably use information from whatever
> > > > part of the image that the sensor is outputting.
> > > >
> > > > For CDAF (contrast detect AF) I guess it's possible that focus
> > > > statistics could be computed either before or after any cropping is
> > > > done in the ISP. (Maybe there are some platforms where cropping could
> > > > be delegated to the sensor?) So I'd guess that using any focus
> > > > statistics from outside the final FoV may actually be impossible,
> > > > depending on the platform. But even where you can do this, I don't
> > > > believe it's what anyone would want (and Android seems to behave like
> > > > this).
> > > >
> > > > > >
> > > > > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > > > > rectangle, so that they are applied to the actual FOV before being
> > > > > > passed to the AF algorithm ?
> > > >
> > > > Not quite sure what you meant here. I'd expect the ISP to generate
> > > > focus statistics from the raw Bayer pixels that it gets, without too
> > > > much processing applied to them. So the job will be to take rectangles
> > > > defined in the final FoV and translate them into the image coming
> > > > directly from the sensor.
> > > >
> > >
> > > What I meant is: if we specify the AfWindows relative to the full
> > > pixel array, can't we then re-scale it to final cropped output ? or if
> > > no copping on the sensor's output frame size.
> > >
> > > This would be similar to what you do with ScalerCrop already ?
> > >
> > >         void RPiCameraData::applyScalerCrop(const ControlList &controls) {
> > >
> > >                 Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
> > >
> > >                 /* Create a version of the crop scaled to ISP (camera mode) pixels. */
> > >                 Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
> > >                 ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
> > >         }
> > >
> > > But I guess I'm reasoning in absolute coordinates, not in proportions
> > > like you have proposed here.
> >
> > I think the question here is whether to use some kind of fractional
> > coordinates for describing the part of the ScalerCrop that we want AF
> > to use, or if we simply use pixel coordinates within the ScalerCrop
> > (or active pixel array), which would then look rather like the code
> > above.
> >
> > I suppose I don't particularly mind. If we use actual pixel
> > coordinates it's maybe a bit less clear that some other scaling
> > (shrinking to lie within the ScalerCrop) might be necessary, but if
> > we're happy that this is going to happen anyway then I think that
> > would work for me.
> >
>
> Let's hear from others as well!

I think the focusAreas() is deprecated on Android.
https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas()
Currently it should be AF_REGIONS.
https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AF_REGIONS

It uses the pixel array as the coordinate system of both ScalarCrop
and AFWindows.
For the problem that the AFWindow may be outside of the ScalarCrop
region, it adds an intersection rule:
Only the intersection of AFWindows and ScalarCrop takes effect.

To be honest, the intersection rule is a little weird. I think
limiting the AFWindow to ScalarCrop looks good to me. The Android
adapter can do the intersection.
On the other hand, I think the benefit of using pixel arrays as
coordinate is that the precision can be at pixel level.
Not sure if it matters for any other algorithms.

Another thing I'd like to mention is a rule for ScalarCrop in Android, I quote:
"The crop region is applied after the RAW to other color space (e.g.
YUV) conversion. Since raw streams (e.g. RAW16) don't have the
conversion stage, they are not croppable. The crop region will be
ignored by raw streams."

Since the statistics is usually generated from the RAW, the input for
IPA would be the full pixel array's statistics.
In result, the IPA needs to translate the coordinate to match the
pixel array anyway.
It might be the reason that both ScalarCrop and AFWindows are in the
pixel array coordinate, since they will be in the end.

This is also a weird rule, because it somehow forbids the device to
crop on sensor.

>
> > >
> > > > > >
> > > > > > One last point about the unit: is the below representation correct,
> > > > > > when it comes to use a Rectangle to represent a proportion ?
> > > > > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > > > > pipeline report 12 rectangles like the ones below, for applications to
> > > > > > chose from ?
> > > > > >
> > > > > >       (0,0)
> > > > > >         +-----+-----+-----+-----+
> > > > > >         |     |     |     |     |
> > > > > >         +-----+-----+-----+-----+
> > > > > >         |     |     |     |     |
> > > > > >         +-----+-----+-----+-----+
> > > > > >         |     |     |     |     |
> > > > > >         +-----+-----+-----+-----+
> > > > > >
> > > > > > AfWindows = {
> > > > > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > > > > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > > > > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > > > > };
> > > > > >
> > > > > > My understanding is that the grid is fixed and depends on the platform
> > > > > > capabilities. I wonder if we shouldn't allow applications to select
> > > > > > grids by index.
> > > >
> > > > I think there is a problem about determining the AF capabilities of a
> > > > platform. I could imagine some platforms will only accept a regular
> > > > grid of regions, but others might let you put them in arbitrary
> > > > places. Or a mixture of both. To be honest, I'm not sure how we handle
> > > > that sort of thing. Anyone have any good ideas?
> > > >
> > >
> > > Exactly, that's why having proportions as the ones I tried to sketch
> > > out above doesn't really fit as a model in my head. Each platform
> > > would have its own 'grid', do we need to expose it to applications ?
> > >
> > > Can't we let application specify Rectangles with absolute coordinates
> > > (relative to PixelArray or else) and let the IPA/PH map those areas on
> > > their grids ?
> >
> > I really don't know how to deal with this. Some ISPs will have grids
> > of some sort, some may have windows in arbitrary locations, different
> > numbers and sizes of each depending on the platform, and some may do
> > both. Typically I would imagine many applications wanting to use grids
> > much of the time, but perhaps arbitrary rectangles for following
> > objects, people and faces.

It's actually happening. GCam is using its own face detection
algorithm and passes the region to HAL.
ChromeOS is also integrating face algorithms to choose ROI for 3A.
Although they are for AE mostly, I think the windows for 3A should
have the same coordinate system.
I guess (I'm not sure) this is why the grid based focusAreas() is
deprecated on Android, and moves to pixel array coordinates, since the
face algorithm usually works on pixel coordinates. I think the grid of
ISPs could be left as implementation details for the pipeline handler?

>
> Indeed. This is an advanced feature for which it might be expected
> application to know what the platform can provide maybe ? Describing
> all possible options seems very complex, and we'll never be able to
> express all of them.
>
> I would be fine stabilizing on a simpler grid-based version of AfWindows
> for now and improve on top, maybe ?
>
> >
> > Perhaps having an AfWindowGrid control and an AfWindows array control
> > of Rectangles could work? For the former, perhaps we leave the
> > decisions on the size of the grid to the platform/algorithm - I'd be
> > worried about having so many parameters that it would be difficult to
> > use. (You'd have to think about the size of the grid, the spacing of
> > rectangles in the grid etc... but perhaps Android requires us to
> > expose this sort of thing?) Perhaps someone knows what Android would
> > require here?
> >
> > >
> > > > > >
> > > > > > Also, do we need a property to report the grid structure and allows
> > > > > > application to select them from, or do you see this conveyed through a
> > > > > > ControlInfo ?
> > > >
> > > > I wasn't quite sure what the question was here. Are you talking about
> > > > selecting the configuration for the AfWindows, or selecting focus
> > > > statistics from particular regions? Perhaps an example would help me
> > > > here!
> > > >
> > >
> > > I'm talking about how would an application know how many spots it can
> > > select from, in other words, how do we let an application know that,
> > > looking at the above example, each grid is 1/4 in width and 1/3 in
> > > height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
> > > 4x3 one.
> > >
> > > Or maybe I got it all wrong, and you where implying already that the
> > > here specified Rectangles cover arbitrary parts of the image and it's
> > > the IPA that has to match them on their grids.
> >
> > Or maybe we treat grids as a whole different thing? (as per the
> > comments just above)
> >
> > >
> > > > > >
> > > > > >
> > > > > > > +
> > > > > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > > > > +        system will choose its own single default window in the centre of the
> > > > > > > +        image.
> > > > > > > +
> > > > > > > +        The details of how the windows are used are platform dependent. We note
> > > > > > > +        that when there is more than one AF window, a typical implementation
> > > > > > > +        might find the optimal focus position for each one and finally select
> > > > > > > +        the window closest to the camera.
> > > > > >
> > > > > > What window is more close to the camera ? My understanding is that
> > > > > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > > > > they all sit on the same focal plan :)
> > > >
> > > > Sorry, by "the window closest to the camera" I mean "the window that
> > > > is showing objects in the real world closest to the camera". I'll try
> > > > to re-word that a bit.
> > > >
> > >
> > > Yeah sorry, I could have realized that myself :)
> > >
> > > > > >
> > > > > > > +
> > > > > > > +        size: [n]
> > > > > > > +
> > > > > > > +  - AfTrigger:
> > > > > > > +      type: int32_t
> > > > > > > +      description: |
> > > > > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > > > > +        and can also be used to terminate a scan early.
> > > > > > > +
> > > > > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > > > > +
> > > > > > > +      enum:
> > > > > > > +        - name: AfTriggerStart
> > > > > > > +          value: 0
> > > > > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > > > > +        - name: AfTriggerCancel
> > > > > > > +          value: 1
> > > > > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > > > > +            anywhere else. Ignored if no scan is in progress.
> > > > > > > +
> > > > > > > +  - AfPause:
> > > > > > > +      type: int32_t
> > > > > > > +      description: |
> > > > > > > +        This control has no effect except when in continuous autofocus mode
> > > > > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > > > > +        (for example) images are captured. The algorithm remains inactive
> > > > > > > +        until it is instructed to resume.
> > > > > > > +
> > > > > > > +      enum:
> > > > > > > +        - name: AfPauseImmediate
> > > > > > > +          value: 0
> > > > > > > +          description: |
> > > > > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > > > > +            when it received this control.
> > > > > > > +        - name: AfPauseDeferred
> > > > > > > +          value: 1
> > > > > > > +          description: |
> > > > > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > > > > +
> > > > > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > > > > +            AfPauseState will change to AfPauseStatePaused.
> > > > > >
> > > > > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > > > > align it back ?
> > > >
> > > > Sure, I'll have a look!
> > > >
> > > > > >
> > > > > > > +        - name: AfPauseResume
> > > > > > > +          value: 2
> > > > > > > +          description: |
> > > > > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > > > > +            from exactly where it left off, and AfPauseState will report
> > > > > > > +            AfPauseStateRunning.
> > > > > > > +
> > > > > > > +  - LensPosition:
> > > > > > > +      type: float
> > > > > > > +      description: |
> > > > > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > > > > +        and also reports back the position of the lens for each frame.
> > > > > > > +
> > > > > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > > > > +        AfModeManual.
> > > > > > > +
> > > > > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > > > > +        values are permitted. For example:
> > > > > > > +        0 moves the lens to infinity.
> > > > > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > > > > +        1 moves the lens to the hyperfocal position.
> > > > > > > +        And larger values will focus the lens ever closer.
> > > > > > > +
> > > > > >
> > > > > > This seems good to me, albeit my understanding of the issue is limited :)
> > > > > >
> > > > >
> > > > > Do we need a property to report the lens hyperfocal distance ?
> > > >
> > > > Yes, sometimes. Android distinguishes between calibrated and
> > > > uncalibrated cameras. For a calibrated camera an application should be
> > > > able to set the focal distance to (for example) 1m, and given the
> > > > units I've proposed, you would need to know the hyperfocal distance to
> > > > calculate the right number.
> > >
> > > Right, so we need a LensHyperfocalDistance or something for calibrated
> > > lenses ?
> >
> > I think so.
> >
> > >
> > > >
> > > > For an uncalibrated camera I don't think it matters. Many applications
> > > > would simply set the lens position to 1.0 (meaning hyperfocal) and
> > > > you'll get the "best overall" focal distance (largest depth of field),
> > > > and that's it.
> > >
> > > I think this part would be relevant to be captured in the control
> > > description.
> >
> > Yes, I can add something about that.
> >
> > >
> > > >
> > > > There's also a need to know the allowable range of lens movement. This
> > > > will depend on the type of module (knowing the lens driver isn't
> > > > enough). But I don't see that applications need to know this, I think
> > > > the pipeline handler and AF implementation should take care of it.
> > > >
> > >
> > > I think so. When it comes to the actual lens movement range I don't
> > > think it's relevant for applications. But do the actual lens movement
> > > range maps to a range of LensPosition values that can be specified ? I
> > > guess it's hard to map them on un-calibrated lenses, but for
> > > calibrated ones a LensPositionRanges whose ControlInfo informs
> > > application on the range's min and max might make sense ? (Please note
> > > that this could also be used for the use case we discussed about
> > > wanting to limit the lens range search scope)
> >
> > I think I would expect a calibrated lens would have to have some kind
> > of property that says what the nearest and furthest focal distances
> > are (the furthest usually being zero, meaning infinity). For an
> > uncalibrated lens things will only ever be "approximately correct".
>
>
> Care to add the property in next version, or can we do it on top with
> a separate patch ?
>
> Thanks
>    j
>
> >
> > I could imagine this being used in conjunction with the hypothetical
> > AfRangeCustom to construct custom search ranges for particular
> > applications (in future, perhaps).
> >
> > Thanks again!
> > David
> >
> > >
> > > > >
> > > > >
> > > > > > > +  - AfState:
> > > > > > > +      type: int32_t
> > > > > > > +      description: |
> > > > > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > > > > +        The possible state changes are described below, though we note the following
> > > > > > > +        state transitions that occur when the AfMode is changed.
> > > > > > > +
> > > > > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > > > > +        LensPosition control must subsequently be used to achieve this.
> > > > > > > +
> > > > > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > > > > >
> > > > > > s/unless/until ?
> > > > > > s/at the same time// ?
> > > >
> > > > I'm trying to describe what happens to the AfState when you send
> > > > AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> > > > AfTriggerStart). The only occasion when you won't see AfStateIdle will
> > > > be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> > > > think this reads OK to me, but I'll see if I can make the wording any
> > > > clearer.
> > >
> > > I see. I didn't get it, sorry. If it's fine with you, keep this
> > > version!
> > >
> > > >
> > > > > >
> > > > > > > +        case it will (start a scan and) report AfStateScanning.
> > > > > > > +
> > > > > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > > > > +        report AfStateScanning.
> > > > > > > +
> > > > > > > +      enum:
> > > > > > > +        - name: AfStateReset
> > > > > > > +          value: 0
> > > > > > > +          description: |
> > > > > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > +            scan was cancelled.
> > > > > > > +        - name: AfStateScanning
> > > > > > > +          value: 1
> > > > > > > +          description: |
> > > > > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > > > > +            move back to AfStateReset or, if the scan actually completes
> > > > > > > +            before the cancel request is processed, to one of
> > > > > > > +            AfStateFocused or AfStateFailed.
> > > > > > > +
> > > > > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > > > > >
> > > > > > s/be be/is ?
> > > >
> > > > Yes, thanks!
> > > >
> > > > > >
> > > > > > > +            at which point it may enter this state spontaneously whenever it
> > > > > > > +            determines that a rescan is needed.
> > > > > > > +        - name: AfStateFocused
> > > > > > > +          value: 2
> > > > > > > +          description: |
> > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > > > > +            the image is now in focus.
> > > > > > > +        - name: AfStateFailed
> > > > > > > +          value: 3
> > > > > > > +          description: |
> > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > > > > +            find a good focus position.
> > > > > > > +
> > > > > > > +  - AfPauseState:
> > > > > > > +      type: int32_t
> > > > > > > +      description: |
> > > > > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > > > > +        as soon as any in-progress scan completes).
> > > > > > > +
> > > > > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > > > > +
> > > > > > > +      enum:
> > > > > > > +        - name: AfPauseStateRunning
> > > > > > > +          value: 0
> > > > > > > +          description: |
> > > > > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > > > > +            spontaneously.
> > > > > > > +        - name: AfPauseStatePausing
> > > > > > > +          value: 1
> > > > > > > +          description: |
> > > > > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > > > > +            as soon as any in-progress scan completes (and then report
> > > > > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > > > > +            the AfPauseResume control is sent.
> > > > > > > +        - name: AfPauseStatePaused
> > > > > > > +          value: 2
> > > > > > > +          description: |
> > > > > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > > > > +            will occur until the AfPauseResume control is sent.
> > > > > > > +
> > > > > >
> > > > > > Very nice overall! With a few more clarifications I think this is good
> > > > > > to go!
> > > > > >
> > > > > > Thanks
> > > > > >    j
> > > >
> > > > Thanks!
> > > > David
> > > >
> > > > > >
> > > > > > >    # ----------------------------------------------------------------------------
> > > > > > >    # Draft controls section
> > > > > > >
> > > > > > > @@ -406,27 +664,6 @@ controls:
> > > > > > >              The camera will cancel any active or completed metering sequence.
> > > > > > >              The AE algorithm is reset to its initial state.
> > > > > > >
> > > > > > > -  - AfTrigger:
> > > > > > > -      type: int32_t
> > > > > > > -      draft: true
> > > > > > > -      description: |
> > > > > > > -       Control for AF trigger. Currently identical to
> > > > > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > > > > -
> > > > > > > -        Whether the camera device will trigger autofocus for this request.
> > > > > > > -      enum:
> > > > > > > -        - name: AfTriggerIdle
> > > > > > > -          value: 0
> > > > > > > -          description: The trigger is idle.
> > > > > > > -        - name: AfTriggerStart
> > > > > > > -          value: 1
> > > > > > > -          description: The AF routine is started by the camera.
> > > > > > > -        - name: AfTriggerCancel
> > > > > > > -          value: 2
> > > > > > > -          description: |
> > > > > > > -            The camera will cancel any active trigger and the AF routine is
> > > > > > > -            reset to its initial state.
> > > > > > > -
> > > > > > >    - NoiseReductionMode:
> > > > > > >        type: int32_t
> > > > > > >        draft: true
> > > > > > > @@ -507,45 +744,6 @@ controls:
> > > > > > >              The AE algorithm has started a pre-capture metering session.
> > > > > > >              \sa AePrecaptureTrigger
> > > > > > >
> > > > > > > -  - AfState:
> > > > > > > -      type: int32_t
> > > > > > > -      draft: true
> > > > > > > -      description: |
> > > > > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > > > > -       ANDROID_CONTROL_AF_STATE.
> > > > > > > -
> > > > > > > -        Current state of the AF algorithm.
> > > > > > > -      enum:
> > > > > > > -        - name: AfStateInactive
> > > > > > > -          value: 0
> > > > > > > -          description: The AF algorithm is inactive.
> > > > > > > -        - name: AfStatePassiveScan
> > > > > > > -          value: 1
> > > > > > > -          description: |
> > > > > > > -            AF is performing a passive scan of the scene in continuous
> > > > > > > -            auto-focus mode.
> > > > > > > -        - name: AfStatePassiveFocused
> > > > > > > -          value: 2
> > > > > > > -          description: |
> > > > > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > > > > -        - name: AfStateActiveScan
> > > > > > > -          value: 3
> > > > > > > -          description: |
> > > > > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > > > > -            \sa AfTrigger
> > > > > > > -        - name: AfStateFocusedLock
> > > > > > > -          value: 4
> > > > > > > -          description: |
> > > > > > > -            AF believes has focused correctly and has locked focus.
> > > > > > > -        - name: AfStateNotFocusedLock
> > > > > > > -          value: 5
> > > > > > > -          description: |
> > > > > > > -            AF has not been able to focus and has locked.
> > > > > > > -        - name: AfStatePassiveUnfocused
> > > > > > > -          value: 6
> > > > > > > -          description: |
> > > > > > > -            AF has completed a passive scan without finding focus.
> > > > > > > -
> > > > > > >    - AwbState:
> > > > > > >        type: int32_t
> > > > > > >        draft: true
> > > > > > > --
> > > > > > > 2.30.2
> > > > > > >

Hello Hanlin,

Thanks for your remarks !

On 28/04/2022 14:45, Hanlin Chen wrote:
> Hi everyone,
> 
> On Tue, Apr 26, 2022 at 9:16 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
>>
>> Hi David,
>>
>> On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
>>> Hi Jacopo
>>>
>>> Thanks for the message. Some replies and more questions below!
>>>
>>> On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
>>>>
>>>> Hi David,
>>>>
>>>> On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
>>>>> Hi Jacopo
>>>>>
>>>>> Thanks for the quick reply!
>>>>>
>>>>> On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
>>>>>>
>>>>>> Ah, forgot one thing!
>>>>>>
>>>>>> On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
>>>>>>> Hi David,
>>>>>>>    thanks for this new version
>>>>>>>
>>>>>>> On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
>>>>>>>> This patch describes a series of controls that allow applications to
>>>>>>>> drive AF algorithms:
>>>>>>>>
>>>>>>>> AfMode - manual, auto or continuous
>>>>>>>> AfRange - full, macro or normal
>>>>>>>> AfSpeed - fast or slowwith
>>>>>>>> AfWindows - AF window locations
>>>>>>>> AfTrigger - start (trigger) an AF scan or cancel
>>>>>>>> AfPause - pause continuous AF
>>>>>>>> LensPosition - set or retrieve position of lens
>>>>>>>> AfState - reports whether scanning/success/failure
>>>>>>>> AfPauseState - reports whether continuous AF paused or not
>>>>>>>>
>>>>>>>> Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
>>>>>>>> ---
>>>>>>>>   src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
>>>>>>>>   1 file changed, 258 insertions(+), 60 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
>>>>>>>> index 9d4638ae..eccf18bc 100644
>>>>>>>> --- a/src/libcamera/control_ids.yaml
>>>>>>>> +++ b/src/libcamera/control_ids.yaml
>>>>>>>> @@ -381,6 +381,264 @@ controls:
>>>>>>>>           \todo Define how the sensor timestamp has to be used in the reprocessing
>>>>>>>>           use case.
>>>>>>>>
>>>>>>>> +  - AfMode:
>>>>>>>> +      type: int32_t
>>>>>>>> +      description: |
>>>>>>>> +        Control to set the mode of the AF (autofocus) algorithm. Applications
>>>>>>>> +        are allowed to set a new mode, and to send additional controls for
>>>>>>>> +        that new mode, in the same request.
>>>>>>>> +
>>>>>>>> +        An implementation may choose not to implement all the modes.
>>>>>>>> +
>>>>>>>> +      enum:
>>>>>>>> +        - name: AfModeManual
>>>>>>>> +          value: 0
>>>>>>>> +          description: |
>>>>>>>> +            The AF algorithm is in manual mode. In this mode it will never
>>>>>>>> +            perform any action nor move the lens of its own accord, but an
>>>>>>>> +            application can set controls to move the lens.
>>>>>>>> +
>>>>>>>
>>>>>>> If, as I understand it, the lens can only be moved with 'LensPosition'
>>>>>>> when in manual mode, I would say so explicitly
>>>>>>>
>>>>>>>                 "... but an application can specify the desired lens
>>>>>>>                 position using the LensPosition control."
>>>>>
>>>>> Yes, will do.
>>>>>
>>>>>>>
>>>>>>>> +            In this mode the AfState will always report AfStateReset.
>>>>>>>
>>>>>>> I'm tempted to say AfState is useless and can not be reported at all
>>>>>>> in this case. For simplicity (for ph/ipa) I'm fine reporting it
>>>>>>> regardless, even if seeing 'reset' might be slightly confusing. I
>>>>>>> don't have better alternatives to propose though.
>>>>>
>>>>> Don't really mind either. I think reporting stuff all the time might
>>>>> be slightly easier, at least it's always obvious what to expect...
>>>>>
>>>>>>>
>>>>>>> Reading the AfStateReset description
>>>>>>>
>>>>>>>          - name: AfStateReset
>>>>>>>            value: 0
>>>>>>>            description: |
>>>>>>>              The AF algorithm is in manual mode (AfModeManual) or in auto mode
>>>>>>>              (AfModeAuto) and a scan has not yet been triggered, or an in-progress
>>>>>>>              scan was cancelled.
>>>>>>>
>>>>>>> I wonder if this isn't better named AfStateIdle
>>>>>
>>>>> Yes, also fine with that.
>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> +        - name: AfModeAuto
>>>>>>>> +          value: 1
>>>>>>>> +          description: |
>>>>>>>> +            The AF algorithm is in auto mode. This means that the algorithm
>>>>>>>> +            will never move the lens or change state unless the AfTrigger
>>>>>>>> +            control is used. The AfTrigger control can be used to initiate a
>>>>>>>> +            focus scan, the results of which will be reported by AfState.
>>>>>>>> +
>>>>>>>> +            If the autofocus algorithm is moved from AfModeAuto to another
>>>>>>>> +            mode while a scan is in progress, the scan is cancelled
>>>>>>>> +            immediately, without waiting for the scan to finish.
>>>>>>>> +
>>>>>>>> +            When first entering this mode the AfState will report
>>>>>>>> +            AfStateReset. When a trigger control is sent, AfState will
>>>>>>>> +            report AfStateScanning for a period before spontaneously
>>>>>>>> +            changing to AfStateFocused or AfStateFailed, depending on
>>>>>>>> +            the outcome of the scan. It will remain in this state until
>>>>>>>> +            another scan is initiated by the AfTrigger control. If a scan is
>>>>>>>> +            cancelled (without changing to another mode), AfState will return
>>>>>>>> +            to AfStateReset.
>>>>>>>> +        - name: AfModeContinuous
>>>>>>>> +          value: 2
>>>>>>>> +          description: |
>>>>>>>> +            The AF algorithm is in continuous mode. This means that the lens
>>>>>>>> +            can re-start a scan spontaneously at any moment, without any user
>>>>>>>> +            intervention. The AfState still reports whether the algorithm is
>>>>>>>> +            currently scanning or not, though the application has no ability
>>>>>>>> +            to initiate or cancel scans (though it can "pause" them), nor to
>>>>>>>> +            move the lens for itself.
>>>>>>>> +
>>>>>>>> +            When set to AfModeContinuous, the system will immediately initiate
>>>>>>>> +            a scan so AfState will report AfStateScanning, and will settle on
>>>>>>>> +            one of AfStateFocused or AfStateFailed, depending on the scan result.
>>>>>>>> +
>>>>>>>> +  - AfRange:
>>>>>>>> +      type: int32_t
>>>>>>>> +      description: |
>>>>>>>> +        Control to set the range of focus distances that is scanned. An
>>>>>>>> +        implementation may choose not to implement all the options here.
>>>>>>>> +      enum:
>>>>>>>> +        - name: AfRangeNormal
>>>>>>>> +          value: 0
>>>>>>>> +          description: |
>>>>>>>> +            A wide range of focus distances is scanned, all the way from
>>>>>>>> +            infinity down to close distances, though depending on the
>>>>>>>> +            implementation, possibly not including the very closest macro
>>>>>>>> +            positions.
>>>>>>>> +        - name: AfRangeMacro
>>>>>>>> +          value: 1
>>>>>>>> +          description: Only close distances are scanned.
>>>>>>>> +        - name: AfRangeFull
>>>>>>>> +          value: 2
>>>>>>>> +          description: |
>>>>>>>> +            The full range of focus distances is scanned just as with
>>>>>>>> +            AfRangeNormal but this time including the very closest macro
>>>>>>>> +            positions.
>>>>>>>> +
>>>>>>>> +  - AfSpeed:
>>>>>>>> +      type: int32_t
>>>>>>>> +      description: |
>>>>>>>> +        Control that determines whether the AF algorithm is to move the lens
>>>>>>>> +        as quickly as possible or more steadily. For example, during video
>>>>>>>> +        recording it may be desirable not to move the lens too abruptly, but
>>>>>>>> +        when in a preview mode (waiting for a still capture) it may be
>>>>>>>> +        helpful to move the lens as quickly as is reasonably possible.
>>>>>>>> +      enum:
>>>>>>>> +        - name: AfSpeedNormal
>>>>>>>> +          value: 0
>>>>>>>> +          description: Move the lens at its usual speed.
>>>>>>>> +        - name: AfSpeedFast
>>>>>>>> +          value: 1
>>>>>>>> +          description: Move the lens more quickly.
>>>>>>>> +
>>>>>>>> +  - AfWindows:
>>>>>>>> +      type: Rectangle
>>>>>>>> +      description: |
>>>>>>>> +        Sets the focus windows used by the AF algorithm. The units used express
>>>>>>>> +        a proportion of the ScalerCrop control (or if unavailable, of the entire
>>>>>>>> +        image), as u0.16 format numbers.
>>>>>>>
>>>>>>> I don't like too much the "or if unavailable", as it means that the
>>>>>>> reference rectangle will change depending on the platform
>>>>>>> capabilities.
>>>>>
>>>>> I guess what I want is (like in the case of Android below), to specify
>>>>> AF regions relative to the actual output image. If a platform supports
>>>>> digital zoom (through ScalerCrop, I assume), the regions must be
>>>>> relative to that part of the image. If there is no ScalerCrop, then do
>>>>> we assume the platform doesn't support digital zoom? But the basic
>>>>> idea is the same, the numbers supplied should be relative to the
>>>>> output image in this case too.
>>>>>
>>>>
>>>> Thing is that there's no "output image size" as two streams produced
>>>> from the same camera might have different output sizes ?
>>>
>>> You're right that there might be several outputs. So what do I really
>>> mean? I'm sort of looking for the rectangle in the sensor image that
>>> is required to make all the outputs, but I think the ScalerCrop
>>> encapsulates that idea reasonably well. Where there isn't a
>>> ScalerCrop,  I think we use the active pixel area instead.
>>>
>>
>> Gotcha! Indeed scaler crop defines the area of the sensor's output
>> frame used to generate all streams... Thanks for sticking to it and
>> help me straight my thoughts.
>>
>> I still don't like the "if not available part" :(
>>
>> I'll try to rope in others to get their opinions. What concerns me is
>> that applications will behave differently on platforms where scaler
>> crop is available compared to ones where it is not. Maybe I'm just
>> over-concerned ?
>>
>>
>>>>
>>>>> Is there a better way to explain this, do you think?
>>>>>
>>>>>>>
>>>>>>> I understand your argument in reply to Jean-Michel comment on the
>>>>>>> previous version of the series (quoted here for reference)
>>>>>>>
>>>>>>> "The thing that bothers me a bit about the full pixel array is that, if
>>>>>>> you use digital zoom, then you will have to update your AfWindows as
>>>>>>> they might otherwise lie outside your final image! If it's defined as
>>>>>>> being a region of the scaler crop, then it automatically stays the
>>>>>>> same (to the application, at least, the pipeline handler probably has
>>>>>>> some work to do).
>>>>>>>
>>>>>>> I note that the Android link you give below says "Focus areas are
>>>>>>> relative to the current field of view (getZoom())", which I guess is
>>>>>>> partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
>>>>>>> is easier to work with than the u0.16 format that I specified?"
>>>>>>>
>>>>>>> Let's reason on the IPA point of view. If the AF algorithm works by
>>>>>>> estimating the contrast, will it do so before or after the ScalerCrop
>>>>>>> rectangle is applied ? If a sensor provides PDAF statistics, will it
>>>>>>> do so on the full pixel array size, the portion of if that gets
>>>>>>> processed or on the final image ?
>>>>>
>>>>> In the case of PDAF, we can presumably use information from whatever
>>>>> part of the image that the sensor is outputting.
>>>>>
>>>>> For CDAF (contrast detect AF) I guess it's possible that focus
>>>>> statistics could be computed either before or after any cropping is
>>>>> done in the ISP. (Maybe there are some platforms where cropping could
>>>>> be delegated to the sensor?) So I'd guess that using any focus
>>>>> statistics from outside the final FoV may actually be impossible,
>>>>> depending on the platform. But even where you can do this, I don't
>>>>> believe it's what anyone would want (and Android seems to behave like
>>>>> this).
>>>>>
>>>>>>>
>>>>>>> Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
>>>>>>> rectangle, so that they are applied to the actual FOV before being
>>>>>>> passed to the AF algorithm ?
>>>>>
>>>>> Not quite sure what you meant here. I'd expect the ISP to generate
>>>>> focus statistics from the raw Bayer pixels that it gets, without too
>>>>> much processing applied to them. So the job will be to take rectangles
>>>>> defined in the final FoV and translate them into the image coming
>>>>> directly from the sensor.
>>>>>
>>>>
>>>> What I meant is: if we specify the AfWindows relative to the full
>>>> pixel array, can't we then re-scale it to final cropped output ? or if
>>>> no copping on the sensor's output frame size.
>>>>
>>>> This would be similar to what you do with ScalerCrop already ?
>>>>
>>>>          void RPiCameraData::applyScalerCrop(const ControlList &controls) {
>>>>
>>>>                  Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
>>>>
>>>>                  /* Create a version of the crop scaled to ISP (camera mode) pixels. */
>>>>                  Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
>>>>                  ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
>>>>          }
>>>>
>>>> But I guess I'm reasoning in absolute coordinates, not in proportions
>>>> like you have proposed here.
>>>
>>> I think the question here is whether to use some kind of fractional
>>> coordinates for describing the part of the ScalerCrop that we want AF
>>> to use, or if we simply use pixel coordinates within the ScalerCrop
>>> (or active pixel array), which would then look rather like the code
>>> above.
>>>
>>> I suppose I don't particularly mind. If we use actual pixel
>>> coordinates it's maybe a bit less clear that some other scaling
>>> (shrinking to lie within the ScalerCrop) might be necessary, but if
>>> we're happy that this is going to happen anyway then I think that
>>> would work for me.
>>>
>>
>> Let's hear from others as well!
> 
> I think the focusAreas() is deprecated on Android.
> https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas()
> Currently it should be AF_REGIONS.
> https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AF_REGIONS
> 

Thank you, it is interesting !

> It uses the pixel array as the coordinate system of both ScalarCrop
> and AFWindows.
> For the problem that the AFWindow may be outside of the ScalarCrop
> region, it adds an intersection rule:
> Only the intersection of AFWindows and ScalarCrop takes effect.
> 
> To be honest, the intersection rule is a little weird. I think
> limiting the AFWindow to ScalarCrop looks good to me. The Android
> adapter can do the intersection.
> On the other hand, I think the benefit of using pixel arrays as
> coordinate is that the precision can be at pixel level.
> Not sure if it matters for any other algorithms.
> 
> Another thing I'd like to mention is a rule for ScalarCrop in Android, I quote:
> "The crop region is applied after the RAW to other color space (e.g.
> YUV) conversion. Since raw streams (e.g. RAW16) don't have the
> conversion stage, they are not croppable. The crop region will be
> ignored by raw streams."
> 
> Since the statistics is usually generated from the RAW, the input for
> IPA would be the full pixel array's statistics.
> In result, the IPA needs to translate the coordinate to match the
> pixel array anyway.
> It might be the reason that both ScalarCrop and AFWindows are in the
> pixel array coordinate, since they will be in the end.
> 
> This is also a weird rule, because it somehow forbids the device to
> crop on sensor.
> 
>>
>>>>
>>>>>>>
>>>>>>> One last point about the unit: is the below representation correct,
>>>>>>> when it comes to use a Rectangle to represent a proportion ?
>>>>>>> Regardless of the reference rectangle, if we have a 4x3 grid, will the
>>>>>>> pipeline report 12 rectangles like the ones below, for applications to
>>>>>>> chose from ?
>>>>>>>
>>>>>>>        (0,0)
>>>>>>>          +-----+-----+-----+-----+
>>>>>>>          |     |     |     |     |
>>>>>>>          +-----+-----+-----+-----+
>>>>>>>          |     |     |     |     |
>>>>>>>          +-----+-----+-----+-----+
>>>>>>>          |     |     |     |     |
>>>>>>>          +-----+-----+-----+-----+
>>>>>>>
>>>>>>> AfWindows = {
>>>>>>>      {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
>>>>>>>      {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
>>>>>>>      {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
>>>>>>> };
>>>>>>>
>>>>>>> My understanding is that the grid is fixed and depends on the platform
>>>>>>> capabilities. I wonder if we shouldn't allow applications to select
>>>>>>> grids by index.
>>>>>
>>>>> I think there is a problem about determining the AF capabilities of a
>>>>> platform. I could imagine some platforms will only accept a regular
>>>>> grid of regions, but others might let you put them in arbitrary
>>>>> places. Or a mixture of both. To be honest, I'm not sure how we handle
>>>>> that sort of thing. Anyone have any good ideas?
>>>>>
>>>>
>>>> Exactly, that's why having proportions as the ones I tried to sketch
>>>> out above doesn't really fit as a model in my head. Each platform
>>>> would have its own 'grid', do we need to expose it to applications ?
>>>>
>>>> Can't we let application specify Rectangles with absolute coordinates
>>>> (relative to PixelArray or else) and let the IPA/PH map those areas on
>>>> their grids ?
>>>
>>> I really don't know how to deal with this. Some ISPs will have grids
>>> of some sort, some may have windows in arbitrary locations, different
>>> numbers and sizes of each depending on the platform, and some may do
>>> both. Typically I would imagine many applications wanting to use grids
>>> much of the time, but perhaps arbitrary rectangles for following
>>> objects, people and faces.
> 
> It's actually happening. GCam is using its own face detection
> algorithm and passes the region to HAL.
> ChromeOS is also integrating face algorithms to choose ROI for 3A.
> Although they are for AE mostly, I think the windows for 3A should
> have the same coordinate system.
> I guess (I'm not sure) this is why the grid based focusAreas() is
> deprecated on Android, and moves to pixel array coordinates, since the
> face algorithm usually works on pixel coordinates. I think the grid of
> ISPs could be left as implementation details for the pipeline handler?
> 

I agree with this, we should rely on the pixel array coordinates, and 
let the PH do the conversion it wants to find the intersecting grid 
given its statistics format.
I find the object MeteringRectangle very interesting [1] and I wonder if 
this could be an internal object for 3A algorithms to use (could be 
useful for AGC and AF at least) ?

[1]: 
https://developer.android.com/reference/android/hardware/camera2/params/MeteringRectangle

JM

>>
>> Indeed. This is an advanced feature for which it might be expected
>> application to know what the platform can provide maybe ? Describing
>> all possible options seems very complex, and we'll never be able to
>> express all of them.
>>
>> I would be fine stabilizing on a simpler grid-based version of AfWindows
>> for now and improve on top, maybe ?
>>
>>>
>>> Perhaps having an AfWindowGrid control and an AfWindows array control
>>> of Rectangles could work? For the former, perhaps we leave the
>>> decisions on the size of the grid to the platform/algorithm - I'd be
>>> worried about having so many parameters that it would be difficult to
>>> use. (You'd have to think about the size of the grid, the spacing of
>>> rectangles in the grid etc... but perhaps Android requires us to
>>> expose this sort of thing?) Perhaps someone knows what Android would
>>> require here?
>>>
>>>>
>>>>>>>
>>>>>>> Also, do we need a property to report the grid structure and allows
>>>>>>> application to select them from, or do you see this conveyed through a
>>>>>>> ControlInfo ?
>>>>>
>>>>> I wasn't quite sure what the question was here. Are you talking about
>>>>> selecting the configuration for the AfWindows, or selecting focus
>>>>> statistics from particular regions? Perhaps an example would help me
>>>>> here!
>>>>>
>>>>
>>>> I'm talking about how would an application know how many spots it can
>>>> select from, in other words, how do we let an application know that,
>>>> looking at the above example, each grid is 1/4 in width and 1/3 in
>>>> height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
>>>> 4x3 one.
>>>>
>>>> Or maybe I got it all wrong, and you where implying already that the
>>>> here specified Rectangles cover arbitrary parts of the image and it's
>>>> the IPA that has to match them on their grids.
>>>
>>> Or maybe we treat grids as a whole different thing? (as per the
>>> comments just above)
>>>
>>>>
>>>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> +        In order to be activated, a rectangle must be programmed with non-zero
>>>>>>>> +        width and height. If no rectangles are programmed in this way, then the
>>>>>>>> +        system will choose its own single default window in the centre of the
>>>>>>>> +        image.
>>>>>>>> +
>>>>>>>> +        The details of how the windows are used are platform dependent. We note
>>>>>>>> +        that when there is more than one AF window, a typical implementation
>>>>>>>> +        might find the optimal focus position for each one and finally select
>>>>>>>> +        the window closest to the camera.
>>>>>>>
>>>>>>> What window is more close to the camera ? My understanding is that
>>>>>>> windows apply to the sensor produced frame, hence, when it comes to distance,
>>>>>>> they all sit on the same focal plan :)
>>>>>
>>>>> Sorry, by "the window closest to the camera" I mean "the window that
>>>>> is showing objects in the real world closest to the camera". I'll try
>>>>> to re-word that a bit.
>>>>>
>>>>
>>>> Yeah sorry, I could have realized that myself :)
>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> +        size: [n]
>>>>>>>> +
>>>>>>>> +  - AfTrigger:
>>>>>>>> +      type: int32_t
>>>>>>>> +      description: |
>>>>>>>> +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
>>>>>>>> +        and can also be used to terminate a scan early.
>>>>>>>> +
>>>>>>>> +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
>>>>>>>> +
>>>>>>>> +      enum:
>>>>>>>> +        - name: AfTriggerStart
>>>>>>>> +          value: 0
>>>>>>>> +          description: Start an AF scan. Ignored if a scan is in progress.
>>>>>>>> +        - name: AfTriggerCancel
>>>>>>>> +          value: 1
>>>>>>>> +          description: Cancel an AF scan. This does not cause the lens to move
>>>>>>>> +            anywhere else. Ignored if no scan is in progress.
>>>>>>>> +
>>>>>>>> +  - AfPause:
>>>>>>>> +      type: int32_t
>>>>>>>> +      description: |
>>>>>>>> +        This control has no effect except when in continuous autofocus mode
>>>>>>>> +        (AfModeContinuous). It can be used to pause any lens movements while
>>>>>>>> +        (for example) images are captured. The algorithm remains inactive
>>>>>>>> +        until it is instructed to resume.
>>>>>>>> +
>>>>>>>> +      enum:
>>>>>>>> +        - name: AfPauseImmediate
>>>>>>>> +          value: 0
>>>>>>>> +          description: |
>>>>>>>> +            Pause the continuous autofocus algorithm immediately, whether or
>>>>>>>> +            not any kind of scan is underway. AfPauseState will subsequently
>>>>>>>> +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
>>>>>>>> +            AfStateFocused or AfStateFailed, depending on the algorithm's state
>>>>>>>> +            when it received this control.
>>>>>>>> +        - name: AfPauseDeferred
>>>>>>>> +          value: 1
>>>>>>>> +          description: |
>>>>>>>> +            This is similar to AfPauseImmediate, and if the AfState is currently
>>>>>>>> +            reporting AfStateFocused or AfStateFailed it will remain in that
>>>>>>>> +            state and AfPauseState will report AfPauseStatePaused.
>>>>>>>> +
>>>>>>>> +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
>>>>>>>> +            will report AfPauseStatePausing until the scan is finished, at which
>>>>>>>> +            point AfState will report one of AfStateFocused or AfStateFailed, and
>>>>>>>> +            AfPauseState will change to AfPauseStatePaused.
>>>>>>>
>>>>>>>  From here on I see lines getting over 80 cols by a few chars. Can we
>>>>>>> align it back ?
>>>>>
>>>>> Sure, I'll have a look!
>>>>>
>>>>>>>
>>>>>>>> +        - name: AfPauseResume
>>>>>>>> +          value: 2
>>>>>>>> +          description: |
>>>>>>>> +            Resume continuous autofocus operation. The algorithm starts again
>>>>>>>> +            from exactly where it left off, and AfPauseState will report
>>>>>>>> +            AfPauseStateRunning.
>>>>>>>> +
>>>>>>>> +  - LensPosition:
>>>>>>>> +      type: float
>>>>>>>> +      description: |
>>>>>>>> +        Acts as a control to instruct the lens to move to a particular position
>>>>>>>> +        and also reports back the position of the lens for each frame.
>>>>>>>> +
>>>>>>>> +        The LensPosition control is ignored unless the AfMode is set to
>>>>>>>> +        AfModeManual.
>>>>>>>> +
>>>>>>>> +        The units are dioptres divided by the hyperfocal distance. Non-integer
>>>>>>>> +        values are permitted. For example:
>>>>>>>> +        0 moves the lens to infinity.
>>>>>>>> +        0.5 moves the lens to twice the hyperfocal distance.
>>>>>>>> +        1 moves the lens to the hyperfocal position.
>>>>>>>> +        And larger values will focus the lens ever closer.
>>>>>>>> +
>>>>>>>
>>>>>>> This seems good to me, albeit my understanding of the issue is limited :)
>>>>>>>
>>>>>>
>>>>>> Do we need a property to report the lens hyperfocal distance ?
>>>>>
>>>>> Yes, sometimes. Android distinguishes between calibrated and
>>>>> uncalibrated cameras. For a calibrated camera an application should be
>>>>> able to set the focal distance to (for example) 1m, and given the
>>>>> units I've proposed, you would need to know the hyperfocal distance to
>>>>> calculate the right number.
>>>>
>>>> Right, so we need a LensHyperfocalDistance or something for calibrated
>>>> lenses ?
>>>
>>> I think so.
>>>
>>>>
>>>>>
>>>>> For an uncalibrated camera I don't think it matters. Many applications
>>>>> would simply set the lens position to 1.0 (meaning hyperfocal) and
>>>>> you'll get the "best overall" focal distance (largest depth of field),
>>>>> and that's it.
>>>>
>>>> I think this part would be relevant to be captured in the control
>>>> description.
>>>
>>> Yes, I can add something about that.
>>>
>>>>
>>>>>
>>>>> There's also a need to know the allowable range of lens movement. This
>>>>> will depend on the type of module (knowing the lens driver isn't
>>>>> enough). But I don't see that applications need to know this, I think
>>>>> the pipeline handler and AF implementation should take care of it.
>>>>>
>>>>
>>>> I think so. When it comes to the actual lens movement range I don't
>>>> think it's relevant for applications. But do the actual lens movement
>>>> range maps to a range of LensPosition values that can be specified ? I
>>>> guess it's hard to map them on un-calibrated lenses, but for
>>>> calibrated ones a LensPositionRanges whose ControlInfo informs
>>>> application on the range's min and max might make sense ? (Please note
>>>> that this could also be used for the use case we discussed about
>>>> wanting to limit the lens range search scope)
>>>
>>> I think I would expect a calibrated lens would have to have some kind
>>> of property that says what the nearest and furthest focal distances
>>> are (the furthest usually being zero, meaning infinity). For an
>>> uncalibrated lens things will only ever be "approximately correct".
>>
>>
>> Care to add the property in next version, or can we do it on top with
>> a separate patch ?
>>
>> Thanks
>>     j
>>
>>>
>>> I could imagine this being used in conjunction with the hypothetical
>>> AfRangeCustom to construct custom search ranges for particular
>>> applications (in future, perhaps).
>>>
>>> Thanks again!
>>> David
>>>
>>>>
>>>>>>
>>>>>>
>>>>>>>> +  - AfState:
>>>>>>>> +      type: int32_t
>>>>>>>> +      description: |
>>>>>>>> +        Reports the current state of the AF algorithm in conjunction with the
>>>>>>>> +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
>>>>>>>> +        The possible state changes are described below, though we note the following
>>>>>>>> +        state transitions that occur when the AfMode is changed.
>>>>>>>> +
>>>>>>>> +        If the AfMode is set to AfModeManual then the AfState will report
>>>>>>>> +        AfStateReset. This action on its own does not cause the lens to move; the
>>>>>>>> +        LensPosition control must subsequently be used to achieve this.
>>>>>>>> +
>>>>>>>> +        If the AfMode is set to AfModeAuto then the AfState will report
>>>>>>>> +        AfStateReset, unless AfTriggerStart is sent at the same time in which
>>>>>>>
>>>>>>> s/unless/until ?
>>>>>>> s/at the same time// ?
>>>>>
>>>>> I'm trying to describe what happens to the AfState when you send
>>>>> AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
>>>>> AfTriggerStart). The only occasion when you won't see AfStateIdle will
>>>>> be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
>>>>> think this reads OK to me, but I'll see if I can make the wording any
>>>>> clearer.
>>>>
>>>> I see. I didn't get it, sorry. If it's fine with you, keep this
>>>> version!
>>>>
>>>>>
>>>>>>>
>>>>>>>> +        case it will (start a scan and) report AfStateScanning.
>>>>>>>> +
>>>>>>>> +        If the AfMode is set to AfModeContinuous then the AfState will initially
>>>>>>>> +        report AfStateScanning.
>>>>>>>> +
>>>>>>>> +      enum:
>>>>>>>> +        - name: AfStateReset
>>>>>>>> +          value: 0
>>>>>>>> +          description: |
>>>>>>>> +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
>>>>>>>> +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
>>>>>>>> +            scan was cancelled.
>>>>>>>> +        - name: AfStateScanning
>>>>>>>> +          value: 1
>>>>>>>> +          description: |
>>>>>>>> +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
>>>>>>>> +            started using the AfTrigger control. The scan can be cancelled by
>>>>>>>> +            sending AfTriggerCancel at which point the algorithm will either
>>>>>>>> +            move back to AfStateReset or, if the scan actually completes
>>>>>>>> +            before the cancel request is processed, to one of
>>>>>>>> +            AfStateFocused or AfStateFailed.
>>>>>>>> +
>>>>>>>> +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
>>>>>>>
>>>>>>> s/be be/is ?
>>>>>
>>>>> Yes, thanks!
>>>>>
>>>>>>>
>>>>>>>> +            at which point it may enter this state spontaneously whenever it
>>>>>>>> +            determines that a rescan is needed.
>>>>>>>> +        - name: AfStateFocused
>>>>>>>> +          value: 2
>>>>>>>> +          description: |
>>>>>>>> +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
>>>>>>>> +            mode and a scan has completed with the result that the algorithm believes
>>>>>>>> +            the image is now in focus.
>>>>>>>> +        - name: AfStateFailed
>>>>>>>> +          value: 3
>>>>>>>> +          description: |
>>>>>>>> +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
>>>>>>>> +            mode and a scan has completed with the result that the algorithm did not
>>>>>>>> +            find a good focus position.
>>>>>>>> +
>>>>>>>> +  - AfPauseState:
>>>>>>>> +      type: int32_t
>>>>>>>> +      description: |
>>>>>>>> +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
>>>>>>>> +        the algorithm is currently running, paused or pausing (that is, will pause
>>>>>>>> +        as soon as any in-progress scan completes).
>>>>>>>> +
>>>>>>>> +        Any change to AfMode will cause AfPauseStateRunning to be reported.
>>>>>>>> +
>>>>>>>> +      enum:
>>>>>>>> +        - name: AfPauseStateRunning
>>>>>>>> +          value: 0
>>>>>>>> +          description: |
>>>>>>>> +            Continuous AF is running and the algorithm may restart a scan
>>>>>>>> +            spontaneously.
>>>>>>>> +        - name: AfPauseStatePausing
>>>>>>>> +          value: 1
>>>>>>>> +          description: |
>>>>>>>> +            Continuous AF has been sent an AfPauseDeferred control, and will pause
>>>>>>>> +            as soon as any in-progress scan completes (and then report
>>>>>>>> +            AfPauseStatePaused). No new scans will be start spontaneously until
>>>>>>>> +            the AfPauseResume control is sent.
>>>>>>>> +        - name: AfPauseStatePaused
>>>>>>>> +          value: 2
>>>>>>>> +          description: |
>>>>>>>> +            Continuous AF is paused. No further state changes or lens movements
>>>>>>>> +            will occur until the AfPauseResume control is sent.
>>>>>>>> +
>>>>>>>
>>>>>>> Very nice overall! With a few more clarifications I think this is good
>>>>>>> to go!
>>>>>>>
>>>>>>> Thanks
>>>>>>>     j
>>>>>
>>>>> Thanks!
>>>>> David
>>>>>
>>>>>>>
>>>>>>>>     # ----------------------------------------------------------------------------
>>>>>>>>     # Draft controls section
>>>>>>>>
>>>>>>>> @@ -406,27 +664,6 @@ controls:
>>>>>>>>               The camera will cancel any active or completed metering sequence.
>>>>>>>>               The AE algorithm is reset to its initial state.
>>>>>>>>
>>>>>>>> -  - AfTrigger:
>>>>>>>> -      type: int32_t
>>>>>>>> -      draft: true
>>>>>>>> -      description: |
>>>>>>>> -       Control for AF trigger. Currently identical to
>>>>>>>> -       ANDROID_CONTROL_AF_TRIGGER.
>>>>>>>> -
>>>>>>>> -        Whether the camera device will trigger autofocus for this request.
>>>>>>>> -      enum:
>>>>>>>> -        - name: AfTriggerIdle
>>>>>>>> -          value: 0
>>>>>>>> -          description: The trigger is idle.
>>>>>>>> -        - name: AfTriggerStart
>>>>>>>> -          value: 1
>>>>>>>> -          description: The AF routine is started by the camera.
>>>>>>>> -        - name: AfTriggerCancel
>>>>>>>> -          value: 2
>>>>>>>> -          description: |
>>>>>>>> -            The camera will cancel any active trigger and the AF routine is
>>>>>>>> -            reset to its initial state.
>>>>>>>> -
>>>>>>>>     - NoiseReductionMode:
>>>>>>>>         type: int32_t
>>>>>>>>         draft: true
>>>>>>>> @@ -507,45 +744,6 @@ controls:
>>>>>>>>               The AE algorithm has started a pre-capture metering session.
>>>>>>>>               \sa AePrecaptureTrigger
>>>>>>>>
>>>>>>>> -  - AfState:
>>>>>>>> -      type: int32_t
>>>>>>>> -      draft: true
>>>>>>>> -      description: |
>>>>>>>> -       Control to report the current AF algorithm state. Currently identical to
>>>>>>>> -       ANDROID_CONTROL_AF_STATE.
>>>>>>>> -
>>>>>>>> -        Current state of the AF algorithm.
>>>>>>>> -      enum:
>>>>>>>> -        - name: AfStateInactive
>>>>>>>> -          value: 0
>>>>>>>> -          description: The AF algorithm is inactive.
>>>>>>>> -        - name: AfStatePassiveScan
>>>>>>>> -          value: 1
>>>>>>>> -          description: |
>>>>>>>> -            AF is performing a passive scan of the scene in continuous
>>>>>>>> -            auto-focus mode.
>>>>>>>> -        - name: AfStatePassiveFocused
>>>>>>>> -          value: 2
>>>>>>>> -          description: |
>>>>>>>> -            AF believes the scene is in focus, but might restart scanning.
>>>>>>>> -        - name: AfStateActiveScan
>>>>>>>> -          value: 3
>>>>>>>> -          description: |
>>>>>>>> -            AF is performing a scan triggered by an AF trigger request.
>>>>>>>> -            \sa AfTrigger
>>>>>>>> -        - name: AfStateFocusedLock
>>>>>>>> -          value: 4
>>>>>>>> -          description: |
>>>>>>>> -            AF believes has focused correctly and has locked focus.
>>>>>>>> -        - name: AfStateNotFocusedLock
>>>>>>>> -          value: 5
>>>>>>>> -          description: |
>>>>>>>> -            AF has not been able to focus and has locked.
>>>>>>>> -        - name: AfStatePassiveUnfocused
>>>>>>>> -          value: 6
>>>>>>>> -          description: |
>>>>>>>> -            AF has completed a passive scan without finding focus.
>>>>>>>> -
>>>>>>>>     - AwbState:
>>>>>>>>         type: int32_t
>>>>>>>>         draft: true
>>>>>>>> --
>>>>>>>> 2.30.2
>>>>>>>>

On Thu, Apr 28, 2022 at 8:45 PM Hanlin Chen <hanlinchen@chromium.org> wrote:
>
> Hi everyone,
>
> On Tue, Apr 26, 2022 at 9:16 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
> >
> > Hi David,
> >
> > On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
> > > Hi Jacopo
> > >
> > > Thanks for the message. Some replies and more questions below!
> > >
> > > On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > >
> > > > Hi David,
> > > >
> > > > On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> > > > > Hi Jacopo
> > > > >
> > > > > Thanks for the quick reply!
> > > > >
> > > > > On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > > >
> > > > > > Ah, forgot one thing!
> > > > > >
> > > > > > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > > > > > Hi David,
> > > > > > >   thanks for this new version
> > > > > > >
> > > > > > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > > > > > This patch describes a series of controls that allow applications to
> > > > > > > > drive AF algorithms:
> > > > > > > >
> > > > > > > > AfMode - manual, auto or continuous
> > > > > > > > AfRange - full, macro or normal
> > > > > > > > AfSpeed - fast or slowwith
> > > > > > > > AfWindows - AF window locations
> > > > > > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > > > > > AfPause - pause continuous AF
> > > > > > > > LensPosition - set or retrieve position of lens
> > > > > > > > AfState - reports whether scanning/success/failure
> > > > > > > > AfPauseState - reports whether continuous AF paused or not
> > > > > > > >
> > > > > > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > > > > > ---
> > > > > > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > > > > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > > > > > index 9d4638ae..eccf18bc 100644
> > > > > > > > --- a/src/libcamera/control_ids.yaml
> > > > > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > > > > @@ -381,6 +381,264 @@ controls:
> > > > > > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > > > > > >          use case.
> > > > > > > >
> > > > > > > > +  - AfMode:
> > > > > > > > +      type: int32_t
> > > > > > > > +      description: |
> > > > > > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > > > > > +        are allowed to set a new mode, and to send additional controls for
> > > > > > > > +        that new mode, in the same request.
> > > > > > > > +
> > > > > > > > +        An implementation may choose not to implement all the modes.
> > > > > > > > +
> > > > > > > > +      enum:
> > > > > > > > +        - name: AfModeManual
> > > > > > > > +          value: 0
> > > > > > > > +          description: |
> > > > > > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > > > > > +            perform any action nor move the lens of its own accord, but an
> > > > > > > > +            application can set controls to move the lens.
> > > > > > > > +
> > > > > > >
> > > > > > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > > > > > when in manual mode, I would say so explicitly
> > > > > > >
> > > > > > >                "... but an application can specify the desired lens
> > > > > > >                position using the LensPosition control."
> > > > >
> > > > > Yes, will do.
> > > > >
> > > > > > >
> > > > > > > > +            In this mode the AfState will always report AfStateReset.
> > > > > > >
> > > > > > > I'm tempted to say AfState is useless and can not be reported at all
> > > > > > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > > > > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > > > > > don't have better alternatives to propose though.
> > > > >
> > > > > Don't really mind either. I think reporting stuff all the time might
> > > > > be slightly easier, at least it's always obvious what to expect...
> > > > >
> > > > > > >
> > > > > > > Reading the AfStateReset description
> > > > > > >
> > > > > > >         - name: AfStateReset
> > > > > > >           value: 0
> > > > > > >           description: |
> > > > > > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > >             scan was cancelled.
> > > > > > >
> > > > > > > I wonder if this isn't better named AfStateIdle
> > > > >
> > > > > Yes, also fine with that.
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > > +        - name: AfModeAuto
> > > > > > > > +          value: 1
> > > > > > > > +          description: |
> > > > > > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > > > > > +            will never move the lens or change state unless the AfTrigger
> > > > > > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > > > > > +            focus scan, the results of which will be reported by AfState.
> > > > > > > > +
> > > > > > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > > > > > +            mode while a scan is in progress, the scan is cancelled
> > > > > > > > +            immediately, without waiting for the scan to finish.
> > > > > > > > +
> > > > > > > > +            When first entering this mode the AfState will report
> > > > > > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > > > > > +            report AfStateScanning for a period before spontaneously
> > > > > > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > > > > > +            the outcome of the scan. It will remain in this state until
> > > > > > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > > > > > +            cancelled (without changing to another mode), AfState will return
> > > > > > > > +            to AfStateReset.
> > > > > > > > +        - name: AfModeContinuous
> > > > > > > > +          value: 2
> > > > > > > > +          description: |
> > > > > > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > > > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > > > > > +            intervention. The AfState still reports whether the algorithm is
> > > > > > > > +            currently scanning or not, though the application has no ability
> > > > > > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > > > > > +            move the lens for itself.
> > > > > > > > +
> > > > > > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > > > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > > > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > > > > > +
> > > > > > > > +  - AfRange:
> > > > > > > > +      type: int32_t
> > > > > > > > +      description: |
> > > > > > > > +        Control to set the range of focus distances that is scanned. An
> > > > > > > > +        implementation may choose not to implement all the options here.
> > > > > > > > +      enum:
> > > > > > > > +        - name: AfRangeNormal
> > > > > > > > +          value: 0
> > > > > > > > +          description: |
> > > > > > > > +            A wide range of focus distances is scanned, all the way from
> > > > > > > > +            infinity down to close distances, though depending on the
> > > > > > > > +            implementation, possibly not including the very closest macro
> > > > > > > > +            positions.
> > > > > > > > +        - name: AfRangeMacro
> > > > > > > > +          value: 1
> > > > > > > > +          description: Only close distances are scanned.
> > > > > > > > +        - name: AfRangeFull
> > > > > > > > +          value: 2
> > > > > > > > +          description: |
> > > > > > > > +            The full range of focus distances is scanned just as with
> > > > > > > > +            AfRangeNormal but this time including the very closest macro
> > > > > > > > +            positions.
> > > > > > > > +
> > > > > > > > +  - AfSpeed:
> > > > > > > > +      type: int32_t
> > > > > > > > +      description: |
> > > > > > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > > > > > +        as quickly as possible or more steadily. For example, during video
> > > > > > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > > > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > > > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > > > > > +      enum:
> > > > > > > > +        - name: AfSpeedNormal
> > > > > > > > +          value: 0
> > > > > > > > +          description: Move the lens at its usual speed.
> > > > > > > > +        - name: AfSpeedFast
> > > > > > > > +          value: 1
> > > > > > > > +          description: Move the lens more quickly.
> > > > > > > > +
> > > > > > > > +  - AfWindows:
> > > > > > > > +      type: Rectangle
> > > > > > > > +      description: |
> > > > > > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > > > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > > > > > +        image), as u0.16 format numbers.
> > > > > > >
> > > > > > > I don't like too much the "or if unavailable", as it means that the
> > > > > > > reference rectangle will change depending on the platform
> > > > > > > capabilities.
> > > > >
> > > > > I guess what I want is (like in the case of Android below), to specify
> > > > > AF regions relative to the actual output image. If a platform supports
> > > > > digital zoom (through ScalerCrop, I assume), the regions must be
> > > > > relative to that part of the image. If there is no ScalerCrop, then do
> > > > > we assume the platform doesn't support digital zoom? But the basic
> > > > > idea is the same, the numbers supplied should be relative to the
> > > > > output image in this case too.
> > > > >
> > > >
> > > > Thing is that there's no "output image size" as two streams produced
> > > > from the same camera might have different output sizes ?
> > >
> > > You're right that there might be several outputs. So what do I really
> > > mean? I'm sort of looking for the rectangle in the sensor image that
> > > is required to make all the outputs, but I think the ScalerCrop
> > > encapsulates that idea reasonably well. Where there isn't a
> > > ScalerCrop,  I think we use the active pixel area instead.
> > >
> >
> > Gotcha! Indeed scaler crop defines the area of the sensor's output
> > frame used to generate all streams... Thanks for sticking to it and
> > help me straight my thoughts.
> >
> > I still don't like the "if not available part" :(
> >
> > I'll try to rope in others to get their opinions. What concerns me is
> > that applications will behave differently on platforms where scaler
> > crop is available compared to ones where it is not. Maybe I'm just
> > over-concerned ?
> >
> >
> > > >
> > > > > Is there a better way to explain this, do you think?
> > > > >
> > > > > > >
> > > > > > > I understand your argument in reply to Jean-Michel comment on the
> > > > > > > previous version of the series (quoted here for reference)
> > > > > > >
> > > > > > > "The thing that bothers me a bit about the full pixel array is that, if
> > > > > > > you use digital zoom, then you will have to update your AfWindows as
> > > > > > > they might otherwise lie outside your final image! If it's defined as
> > > > > > > being a region of the scaler crop, then it automatically stays the
> > > > > > > same (to the application, at least, the pipeline handler probably has
> > > > > > > some work to do).
> > > > > > >
> > > > > > > I note that the Android link you give below says "Focus areas are
> > > > > > > relative to the current field of view (getZoom())", which I guess is
> > > > > > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > > > > > is easier to work with than the u0.16 format that I specified?"
> > > > > > >
> > > > > > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > > > > > estimating the contrast, will it do so before or after the ScalerCrop
> > > > > > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > > > > > do so on the full pixel array size, the portion of if that gets
> > > > > > > processed or on the final image ?
> > > > >
> > > > > In the case of PDAF, we can presumably use information from whatever
> > > > > part of the image that the sensor is outputting.
> > > > >
> > > > > For CDAF (contrast detect AF) I guess it's possible that focus
> > > > > statistics could be computed either before or after any cropping is
> > > > > done in the ISP. (Maybe there are some platforms where cropping could
> > > > > be delegated to the sensor?) So I'd guess that using any focus
> > > > > statistics from outside the final FoV may actually be impossible,
> > > > > depending on the platform. But even where you can do this, I don't
> > > > > believe it's what anyone would want (and Android seems to behave like
> > > > > this).
> > > > >
> > > > > > >
> > > > > > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > > > > > rectangle, so that they are applied to the actual FOV before being
> > > > > > > passed to the AF algorithm ?
> > > > >
> > > > > Not quite sure what you meant here. I'd expect the ISP to generate
> > > > > focus statistics from the raw Bayer pixels that it gets, without too
> > > > > much processing applied to them. So the job will be to take rectangles
> > > > > defined in the final FoV and translate them into the image coming
> > > > > directly from the sensor.
> > > > >
> > > >
> > > > What I meant is: if we specify the AfWindows relative to the full
> > > > pixel array, can't we then re-scale it to final cropped output ? or if
> > > > no copping on the sensor's output frame size.
> > > >
> > > > This would be similar to what you do with ScalerCrop already ?
> > > >
> > > >         void RPiCameraData::applyScalerCrop(const ControlList &controls) {
> > > >
> > > >                 Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
> > > >
> > > >                 /* Create a version of the crop scaled to ISP (camera mode) pixels. */
> > > >                 Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
> > > >                 ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
> > > >         }
> > > >
> > > > But I guess I'm reasoning in absolute coordinates, not in proportions
> > > > like you have proposed here.
> > >
> > > I think the question here is whether to use some kind of fractional
> > > coordinates for describing the part of the ScalerCrop that we want AF
> > > to use, or if we simply use pixel coordinates within the ScalerCrop
> > > (or active pixel array), which would then look rather like the code
> > > above.
> > >
> > > I suppose I don't particularly mind. If we use actual pixel
> > > coordinates it's maybe a bit less clear that some other scaling
> > > (shrinking to lie within the ScalerCrop) might be necessary, but if
> > > we're happy that this is going to happen anyway then I think that
> > > would work for me.
> > >
> >
> > Let's hear from others as well!
>
> I think the focusAreas() is deprecated on Android.
> https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas()
> Currently it should be AF_REGIONS.
> https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AF_REGIONS
>
> It uses the pixel array as the coordinate system of both ScalarCrop
> and AFWindows.
> For the problem that the AFWindow may be outside of the ScalarCrop
> region, it adds an intersection rule:
> Only the intersection of AFWindows and ScalarCrop takes effect.
>
> To be honest, the intersection rule is a little weird. I think
> limiting the AFWindow to ScalarCrop looks good to me. The Android
> adapter can do the intersection.
> On the other hand, I think the benefit of using pixel arrays as
> coordinate is that the precision can be at pixel level.
> Not sure if it matters for any other algorithms.
>
> Another thing I'd like to mention is a rule for ScalarCrop in Android, I quote:
> "The crop region is applied after the RAW to other color space (e.g.
> YUV) conversion. Since raw streams (e.g. RAW16) don't have the
> conversion stage, they are not croppable. The crop region will be
> ignored by raw streams."
>
> Since the statistics is usually generated from the RAW, the input for
> IPA would be the full pixel array's statistics.
> In result, the IPA needs to translate the coordinate to match the
> pixel array anyway.
> It might be the reason that both ScalarCrop and AFWindows are in the
> pixel array coordinate, since they will be in the end.
>
> This is also a weird rule, because it somehow forbids the device to
> crop on sensor.
>
> >
> > > >
> > > > > > >
> > > > > > > One last point about the unit: is the below representation correct,
> > > > > > > when it comes to use a Rectangle to represent a proportion ?
> > > > > > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > > > > > pipeline report 12 rectangles like the ones below, for applications to
> > > > > > > chose from ?
> > > > > > >
> > > > > > >       (0,0)
> > > > > > >         +-----+-----+-----+-----+
> > > > > > >         |     |     |     |     |
> > > > > > >         +-----+-----+-----+-----+
> > > > > > >         |     |     |     |     |
> > > > > > >         +-----+-----+-----+-----+
> > > > > > >         |     |     |     |     |
> > > > > > >         +-----+-----+-----+-----+
> > > > > > >
> > > > > > > AfWindows = {
> > > > > > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > > > > > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > > > > > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > > > > > };
> > > > > > >
> > > > > > > My understanding is that the grid is fixed and depends on the platform
> > > > > > > capabilities. I wonder if we shouldn't allow applications to select
> > > > > > > grids by index.
> > > > >
> > > > > I think there is a problem about determining the AF capabilities of a
> > > > > platform. I could imagine some platforms will only accept a regular
> > > > > grid of regions, but others might let you put them in arbitrary
> > > > > places. Or a mixture of both. To be honest, I'm not sure how we handle
> > > > > that sort of thing. Anyone have any good ideas?
> > > > >
> > > >
> > > > Exactly, that's why having proportions as the ones I tried to sketch
> > > > out above doesn't really fit as a model in my head. Each platform
> > > > would have its own 'grid', do we need to expose it to applications ?
> > > >
> > > > Can't we let application specify Rectangles with absolute coordinates
> > > > (relative to PixelArray or else) and let the IPA/PH map those areas on
> > > > their grids ?
> > >
> > > I really don't know how to deal with this. Some ISPs will have grids
> > > of some sort, some may have windows in arbitrary locations, different
> > > numbers and sizes of each depending on the platform, and some may do
> > > both. Typically I would imagine many applications wanting to use grids
> > > much of the time, but perhaps arbitrary rectangles for following
> > > objects, people and faces.
>
> It's actually happening. GCam is using its own face detection
> algorithm and passes the region to HAL.
> ChromeOS is also integrating face algorithms to choose ROI for 3A.
> Although they are for AE mostly, I think the windows for 3A should
> have the same coordinate system.
> I guess (I'm not sure) this is why the grid based focusAreas() is
> deprecated on Android, and moves to pixel array coordinates, since the
> face algorithm usually works on pixel coordinates. I think the grid of
> ISPs could be left as implementation details for the pipeline handler?
>
I was woolly-minded. focusAreas() is not grid based, please ignore the
sentence :P.
Another thing is that AF_CROP_REGION allows windows to be overlapped
and with weight.
I'm not sure if we'd like to allow that. Perhaps this is discussed
already and I missed it?
If there are ML algorithms to recognize the objects in a scene, the
rectangle could be partially overlapped naturally, and may be weighted
by objects, for example face within leaves is more weighted even if
it's not the closet object in the scene. Usually the ML algorithm is
located in the application and outside of HAL.
I guess it's something we can decide later.

> >
> > Indeed. This is an advanced feature for which it might be expected
> > application to know what the platform can provide maybe ? Describing
> > all possible options seems very complex, and we'll never be able to
> > express all of them.
> >
> > I would be fine stabilizing on a simpler grid-based version of AfWindows
> > for now and improve on top, maybe ?
> >
> > >
> > > Perhaps having an AfWindowGrid control and an AfWindows array control
> > > of Rectangles could work? For the former, perhaps we leave the
> > > decisions on the size of the grid to the platform/algorithm - I'd be
> > > worried about having so many parameters that it would be difficult to
> > > use. (You'd have to think about the size of the grid, the spacing of
> > > rectangles in the grid etc... but perhaps Android requires us to
> > > expose this sort of thing?) Perhaps someone knows what Android would
> > > require here?
> > >
> > > >
> > > > > > >
> > > > > > > Also, do we need a property to report the grid structure and allows
> > > > > > > application to select them from, or do you see this conveyed through a
> > > > > > > ControlInfo ?
> > > > >
> > > > > I wasn't quite sure what the question was here. Are you talking about
> > > > > selecting the configuration for the AfWindows, or selecting focus
> > > > > statistics from particular regions? Perhaps an example would help me
> > > > > here!
> > > > >
> > > >
> > > > I'm talking about how would an application know how many spots it can
> > > > select from, in other words, how do we let an application know that,
> > > > looking at the above example, each grid is 1/4 in width and 1/3 in
> > > > height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
> > > > 4x3 one.
> > > >
> > > > Or maybe I got it all wrong, and you where implying already that the
> > > > here specified Rectangles cover arbitrary parts of the image and it's
> > > > the IPA that has to match them on their grids.
> > >
> > > Or maybe we treat grids as a whole different thing? (as per the
> > > comments just above)
> > >
> > > >
> > > > > > >
> > > > > > >
> > > > > > > > +
> > > > > > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > > > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > > > > > +        system will choose its own single default window in the centre of the
> > > > > > > > +        image.
> > > > > > > > +
> > > > > > > > +        The details of how the windows are used are platform dependent. We note
> > > > > > > > +        that when there is more than one AF window, a typical implementation
> > > > > > > > +        might find the optimal focus position for each one and finally select
> > > > > > > > +        the window closest to the camera.
> > > > > > >
> > > > > > > What window is more close to the camera ? My understanding is that
> > > > > > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > > > > > they all sit on the same focal plan :)
> > > > >
> > > > > Sorry, by "the window closest to the camera" I mean "the window that
> > > > > is showing objects in the real world closest to the camera". I'll try
> > > > > to re-word that a bit.
> > > > >
> > > >
> > > > Yeah sorry, I could have realized that myself :)
> > > >
> > > > > > >
> > > > > > > > +
> > > > > > > > +        size: [n]
> > > > > > > > +
> > > > > > > > +  - AfTrigger:
> > > > > > > > +      type: int32_t
> > > > > > > > +      description: |
> > > > > > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > > > > > +        and can also be used to terminate a scan early.
> > > > > > > > +
> > > > > > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > > > > > +
> > > > > > > > +      enum:
> > > > > > > > +        - name: AfTriggerStart
> > > > > > > > +          value: 0
> > > > > > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > > > > > +        - name: AfTriggerCancel
> > > > > > > > +          value: 1
> > > > > > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > > > > > +            anywhere else. Ignored if no scan is in progress.
> > > > > > > > +
> > > > > > > > +  - AfPause:
> > > > > > > > +      type: int32_t
> > > > > > > > +      description: |
> > > > > > > > +        This control has no effect except when in continuous autofocus mode
> > > > > > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > > > > > +        (for example) images are captured. The algorithm remains inactive
> > > > > > > > +        until it is instructed to resume.
> > > > > > > > +
> > > > > > > > +      enum:
> > > > > > > > +        - name: AfPauseImmediate
> > > > > > > > +          value: 0
> > > > > > > > +          description: |
> > > > > > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > > > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > > > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > > > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > > > > > +            when it received this control.
> > > > > > > > +        - name: AfPauseDeferred
> > > > > > > > +          value: 1
> > > > > > > > +          description: |
> > > > > > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > > > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > > > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > > > > > +
> > > > > > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > > > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > > > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > > > > > +            AfPauseState will change to AfPauseStatePaused.
> > > > > > >
> > > > > > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > > > > > align it back ?
> > > > >
> > > > > Sure, I'll have a look!
> > > > >
> > > > > > >
> > > > > > > > +        - name: AfPauseResume
> > > > > > > > +          value: 2
> > > > > > > > +          description: |
> > > > > > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > > > > > +            from exactly where it left off, and AfPauseState will report
> > > > > > > > +            AfPauseStateRunning.
> > > > > > > > +
> > > > > > > > +  - LensPosition:
> > > > > > > > +      type: float
> > > > > > > > +      description: |
> > > > > > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > > > > > +        and also reports back the position of the lens for each frame.
> > > > > > > > +
> > > > > > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > > > > > +        AfModeManual.
> > > > > > > > +
> > > > > > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > > > > > +        values are permitted. For example:
> > > > > > > > +        0 moves the lens to infinity.
> > > > > > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > > > > > +        1 moves the lens to the hyperfocal position.
> > > > > > > > +        And larger values will focus the lens ever closer.
> > > > > > > > +
> > > > > > >
> > > > > > > This seems good to me, albeit my understanding of the issue is limited :)
> > > > > > >
> > > > > >
> > > > > > Do we need a property to report the lens hyperfocal distance ?
> > > > >
> > > > > Yes, sometimes. Android distinguishes between calibrated and
> > > > > uncalibrated cameras. For a calibrated camera an application should be
> > > > > able to set the focal distance to (for example) 1m, and given the
> > > > > units I've proposed, you would need to know the hyperfocal distance to
> > > > > calculate the right number.
> > > >
> > > > Right, so we need a LensHyperfocalDistance or something for calibrated
> > > > lenses ?
> > >
> > > I think so.
> > >
> > > >
> > > > >
> > > > > For an uncalibrated camera I don't think it matters. Many applications
> > > > > would simply set the lens position to 1.0 (meaning hyperfocal) and
> > > > > you'll get the "best overall" focal distance (largest depth of field),
> > > > > and that's it.
> > > >
> > > > I think this part would be relevant to be captured in the control
> > > > description.
> > >
> > > Yes, I can add something about that.
> > >
> > > >
> > > > >
> > > > > There's also a need to know the allowable range of lens movement. This
> > > > > will depend on the type of module (knowing the lens driver isn't
> > > > > enough). But I don't see that applications need to know this, I think
> > > > > the pipeline handler and AF implementation should take care of it.
> > > > >
> > > >
> > > > I think so. When it comes to the actual lens movement range I don't
> > > > think it's relevant for applications. But do the actual lens movement
> > > > range maps to a range of LensPosition values that can be specified ? I
> > > > guess it's hard to map them on un-calibrated lenses, but for
> > > > calibrated ones a LensPositionRanges whose ControlInfo informs
> > > > application on the range's min and max might make sense ? (Please note
> > > > that this could also be used for the use case we discussed about
> > > > wanting to limit the lens range search scope)
> > >
> > > I think I would expect a calibrated lens would have to have some kind
> > > of property that says what the nearest and furthest focal distances
> > > are (the furthest usually being zero, meaning infinity). For an
> > > uncalibrated lens things will only ever be "approximately correct".
> >
> >
> > Care to add the property in next version, or can we do it on top with
> > a separate patch ?
> >
> > Thanks
> >    j
> >
> > >
> > > I could imagine this being used in conjunction with the hypothetical
> > > AfRangeCustom to construct custom search ranges for particular
> > > applications (in future, perhaps).
> > >
> > > Thanks again!
> > > David
> > >
> > > >
> > > > > >
> > > > > >
> > > > > > > > +  - AfState:
> > > > > > > > +      type: int32_t
> > > > > > > > +      description: |
> > > > > > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > > > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > > > > > +        The possible state changes are described below, though we note the following
> > > > > > > > +        state transitions that occur when the AfMode is changed.
> > > > > > > > +
> > > > > > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > > > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > > > > > +        LensPosition control must subsequently be used to achieve this.
> > > > > > > > +
> > > > > > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > > > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > > > > > >
> > > > > > > s/unless/until ?
> > > > > > > s/at the same time// ?
> > > > >
> > > > > I'm trying to describe what happens to the AfState when you send
> > > > > AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> > > > > AfTriggerStart). The only occasion when you won't see AfStateIdle will
> > > > > be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> > > > > think this reads OK to me, but I'll see if I can make the wording any
> > > > > clearer.
> > > >
> > > > I see. I didn't get it, sorry. If it's fine with you, keep this
> > > > version!
> > > >
> > > > >
> > > > > > >
> > > > > > > > +        case it will (start a scan and) report AfStateScanning.
> > > > > > > > +
> > > > > > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > > > > > +        report AfStateScanning.
> > > > > > > > +
> > > > > > > > +      enum:
> > > > > > > > +        - name: AfStateReset
> > > > > > > > +          value: 0
> > > > > > > > +          description: |
> > > > > > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > > +            scan was cancelled.
> > > > > > > > +        - name: AfStateScanning
> > > > > > > > +          value: 1
> > > > > > > > +          description: |
> > > > > > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > > > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > > > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > > > > > +            move back to AfStateReset or, if the scan actually completes
> > > > > > > > +            before the cancel request is processed, to one of
> > > > > > > > +            AfStateFocused or AfStateFailed.
> > > > > > > > +
> > > > > > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > > > > > >
> > > > > > > s/be be/is ?
> > > > >
> > > > > Yes, thanks!
> > > > >
> > > > > > >
> > > > > > > > +            at which point it may enter this state spontaneously whenever it
> > > > > > > > +            determines that a rescan is needed.
> > > > > > > > +        - name: AfStateFocused
> > > > > > > > +          value: 2
> > > > > > > > +          description: |
> > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > > > > > +            the image is now in focus.
> > > > > > > > +        - name: AfStateFailed
> > > > > > > > +          value: 3
> > > > > > > > +          description: |
> > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > > > > > +            find a good focus position.
> > > > > > > > +
> > > > > > > > +  - AfPauseState:
> > > > > > > > +      type: int32_t
> > > > > > > > +      description: |
> > > > > > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > > > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > > > > > +        as soon as any in-progress scan completes).
> > > > > > > > +
> > > > > > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > > > > > +
> > > > > > > > +      enum:
> > > > > > > > +        - name: AfPauseStateRunning
> > > > > > > > +          value: 0
> > > > > > > > +          description: |
> > > > > > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > > > > > +            spontaneously.
> > > > > > > > +        - name: AfPauseStatePausing
> > > > > > > > +          value: 1
> > > > > > > > +          description: |
> > > > > > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > > > > > +            as soon as any in-progress scan completes (and then report
> > > > > > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > > > > > +            the AfPauseResume control is sent.
> > > > > > > > +        - name: AfPauseStatePaused
> > > > > > > > +          value: 2
> > > > > > > > +          description: |
> > > > > > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > > > > > +            will occur until the AfPauseResume control is sent.
> > > > > > > > +
> > > > > > >
> > > > > > > Very nice overall! With a few more clarifications I think this is good
> > > > > > > to go!
> > > > > > >
> > > > > > > Thanks
> > > > > > >    j
> > > > >
> > > > > Thanks!
> > > > > David
> > > > >
> > > > > > >
> > > > > > > >    # ----------------------------------------------------------------------------
> > > > > > > >    # Draft controls section
> > > > > > > >
> > > > > > > > @@ -406,27 +664,6 @@ controls:
> > > > > > > >              The camera will cancel any active or completed metering sequence.
> > > > > > > >              The AE algorithm is reset to its initial state.
> > > > > > > >
> > > > > > > > -  - AfTrigger:
> > > > > > > > -      type: int32_t
> > > > > > > > -      draft: true
> > > > > > > > -      description: |
> > > > > > > > -       Control for AF trigger. Currently identical to
> > > > > > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > > > > > -
> > > > > > > > -        Whether the camera device will trigger autofocus for this request.
> > > > > > > > -      enum:
> > > > > > > > -        - name: AfTriggerIdle
> > > > > > > > -          value: 0
> > > > > > > > -          description: The trigger is idle.
> > > > > > > > -        - name: AfTriggerStart
> > > > > > > > -          value: 1
> > > > > > > > -          description: The AF routine is started by the camera.
> > > > > > > > -        - name: AfTriggerCancel
> > > > > > > > -          value: 2
> > > > > > > > -          description: |
> > > > > > > > -            The camera will cancel any active trigger and the AF routine is
> > > > > > > > -            reset to its initial state.
> > > > > > > > -
> > > > > > > >    - NoiseReductionMode:
> > > > > > > >        type: int32_t
> > > > > > > >        draft: true
> > > > > > > > @@ -507,45 +744,6 @@ controls:
> > > > > > > >              The AE algorithm has started a pre-capture metering session.
> > > > > > > >              \sa AePrecaptureTrigger
> > > > > > > >
> > > > > > > > -  - AfState:
> > > > > > > > -      type: int32_t
> > > > > > > > -      draft: true
> > > > > > > > -      description: |
> > > > > > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > > > > > -       ANDROID_CONTROL_AF_STATE.
> > > > > > > > -
> > > > > > > > -        Current state of the AF algorithm.
> > > > > > > > -      enum:
> > > > > > > > -        - name: AfStateInactive
> > > > > > > > -          value: 0
> > > > > > > > -          description: The AF algorithm is inactive.
> > > > > > > > -        - name: AfStatePassiveScan
> > > > > > > > -          value: 1
> > > > > > > > -          description: |
> > > > > > > > -            AF is performing a passive scan of the scene in continuous
> > > > > > > > -            auto-focus mode.
> > > > > > > > -        - name: AfStatePassiveFocused
> > > > > > > > -          value: 2
> > > > > > > > -          description: |
> > > > > > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > > > > > -        - name: AfStateActiveScan
> > > > > > > > -          value: 3
> > > > > > > > -          description: |
> > > > > > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > > > > > -            \sa AfTrigger
> > > > > > > > -        - name: AfStateFocusedLock
> > > > > > > > -          value: 4
> > > > > > > > -          description: |
> > > > > > > > -            AF believes has focused correctly and has locked focus.
> > > > > > > > -        - name: AfStateNotFocusedLock
> > > > > > > > -          value: 5
> > > > > > > > -          description: |
> > > > > > > > -            AF has not been able to focus and has locked.
> > > > > > > > -        - name: AfStatePassiveUnfocused
> > > > > > > > -          value: 6
> > > > > > > > -          description: |
> > > > > > > > -            AF has completed a passive scan without finding focus.
> > > > > > > > -
> > > > > > > >    - AwbState:
> > > > > > > >        type: int32_t
> > > > > > > >        draft: true
> > > > > > > > --
> > > > > > > > 2.30.2
> > > > > > > >

Hi Han-Lin,
   thanks for the many details!

On Thu, Apr 28, 2022 at 11:44:23PM +0800, Hanlin Chen wrote:
> On Thu, Apr 28, 2022 at 8:45 PM Hanlin Chen <hanlinchen@chromium.org> wrote:
> >
> > Hi everyone,
> >
> > On Tue, Apr 26, 2022 at 9:16 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
> > >
> > > Hi David,
> > >
> > > On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
> > > > Hi Jacopo
> > > >
> > > > Thanks for the message. Some replies and more questions below!
> > > >
> > > > On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > >
> > > > > Hi David,
> > > > >
> > > > > On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> > > > > > Hi Jacopo
> > > > > >
> > > > > > Thanks for the quick reply!
> > > > > >
> > > > > > On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > > > >
> > > > > > > Ah, forgot one thing!
> > > > > > >
> > > > > > > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > Hi David,
> > > > > > > >   thanks for this new version
> > > > > > > >
> > > > > > > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > > > > > > This patch describes a series of controls that allow applications to
> > > > > > > > > drive AF algorithms:
> > > > > > > > >
> > > > > > > > > AfMode - manual, auto or continuous
> > > > > > > > > AfRange - full, macro or normal
> > > > > > > > > AfSpeed - fast or slowwith
> > > > > > > > > AfWindows - AF window locations
> > > > > > > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > > > > > > AfPause - pause continuous AF
> > > > > > > > > LensPosition - set or retrieve position of lens
> > > > > > > > > AfState - reports whether scanning/success/failure
> > > > > > > > > AfPauseState - reports whether continuous AF paused or not
> > > > > > > > >
> > > > > > > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > > > > > > ---
> > > > > > > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > > > > > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > > > > > > index 9d4638ae..eccf18bc 100644
> > > > > > > > > --- a/src/libcamera/control_ids.yaml
> > > > > > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > > > > > @@ -381,6 +381,264 @@ controls:
> > > > > > > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > > > > > > >          use case.
> > > > > > > > >
> > > > > > > > > +  - AfMode:
> > > > > > > > > +      type: int32_t
> > > > > > > > > +      description: |
> > > > > > > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > > > > > > +        are allowed to set a new mode, and to send additional controls for
> > > > > > > > > +        that new mode, in the same request.
> > > > > > > > > +
> > > > > > > > > +        An implementation may choose not to implement all the modes.
> > > > > > > > > +
> > > > > > > > > +      enum:
> > > > > > > > > +        - name: AfModeManual
> > > > > > > > > +          value: 0
> > > > > > > > > +          description: |
> > > > > > > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > > > > > > +            perform any action nor move the lens of its own accord, but an
> > > > > > > > > +            application can set controls to move the lens.
> > > > > > > > > +
> > > > > > > >
> > > > > > > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > > > > > > when in manual mode, I would say so explicitly
> > > > > > > >
> > > > > > > >                "... but an application can specify the desired lens
> > > > > > > >                position using the LensPosition control."
> > > > > >
> > > > > > Yes, will do.
> > > > > >
> > > > > > > >
> > > > > > > > > +            In this mode the AfState will always report AfStateReset.
> > > > > > > >
> > > > > > > > I'm tempted to say AfState is useless and can not be reported at all
> > > > > > > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > > > > > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > > > > > > don't have better alternatives to propose though.
> > > > > >
> > > > > > Don't really mind either. I think reporting stuff all the time might
> > > > > > be slightly easier, at least it's always obvious what to expect...
> > > > > >
> > > > > > > >
> > > > > > > > Reading the AfStateReset description
> > > > > > > >
> > > > > > > >         - name: AfStateReset
> > > > > > > >           value: 0
> > > > > > > >           description: |
> > > > > > > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > >             scan was cancelled.
> > > > > > > >
> > > > > > > > I wonder if this isn't better named AfStateIdle
> > > > > >
> > > > > > Yes, also fine with that.
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > +        - name: AfModeAuto
> > > > > > > > > +          value: 1
> > > > > > > > > +          description: |
> > > > > > > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > > > > > > +            will never move the lens or change state unless the AfTrigger
> > > > > > > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > > > > > > +            focus scan, the results of which will be reported by AfState.
> > > > > > > > > +
> > > > > > > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > > > > > > +            mode while a scan is in progress, the scan is cancelled
> > > > > > > > > +            immediately, without waiting for the scan to finish.
> > > > > > > > > +
> > > > > > > > > +            When first entering this mode the AfState will report
> > > > > > > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > > > > > > +            report AfStateScanning for a period before spontaneously
> > > > > > > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > > > > > > +            the outcome of the scan. It will remain in this state until
> > > > > > > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > > > > > > +            cancelled (without changing to another mode), AfState will return
> > > > > > > > > +            to AfStateReset.
> > > > > > > > > +        - name: AfModeContinuous
> > > > > > > > > +          value: 2
> > > > > > > > > +          description: |
> > > > > > > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > > > > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > > > > > > +            intervention. The AfState still reports whether the algorithm is
> > > > > > > > > +            currently scanning or not, though the application has no ability
> > > > > > > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > > > > > > +            move the lens for itself.
> > > > > > > > > +
> > > > > > > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > > > > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > > > > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > > > > > > +
> > > > > > > > > +  - AfRange:
> > > > > > > > > +      type: int32_t
> > > > > > > > > +      description: |
> > > > > > > > > +        Control to set the range of focus distances that is scanned. An
> > > > > > > > > +        implementation may choose not to implement all the options here.
> > > > > > > > > +      enum:
> > > > > > > > > +        - name: AfRangeNormal
> > > > > > > > > +          value: 0
> > > > > > > > > +          description: |
> > > > > > > > > +            A wide range of focus distances is scanned, all the way from
> > > > > > > > > +            infinity down to close distances, though depending on the
> > > > > > > > > +            implementation, possibly not including the very closest macro
> > > > > > > > > +            positions.
> > > > > > > > > +        - name: AfRangeMacro
> > > > > > > > > +          value: 1
> > > > > > > > > +          description: Only close distances are scanned.
> > > > > > > > > +        - name: AfRangeFull
> > > > > > > > > +          value: 2
> > > > > > > > > +          description: |
> > > > > > > > > +            The full range of focus distances is scanned just as with
> > > > > > > > > +            AfRangeNormal but this time including the very closest macro
> > > > > > > > > +            positions.
> > > > > > > > > +
> > > > > > > > > +  - AfSpeed:
> > > > > > > > > +      type: int32_t
> > > > > > > > > +      description: |
> > > > > > > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > > > > > > +        as quickly as possible or more steadily. For example, during video
> > > > > > > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > > > > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > > > > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > > > > > > +      enum:
> > > > > > > > > +        - name: AfSpeedNormal
> > > > > > > > > +          value: 0
> > > > > > > > > +          description: Move the lens at its usual speed.
> > > > > > > > > +        - name: AfSpeedFast
> > > > > > > > > +          value: 1
> > > > > > > > > +          description: Move the lens more quickly.
> > > > > > > > > +
> > > > > > > > > +  - AfWindows:
> > > > > > > > > +      type: Rectangle
> > > > > > > > > +      description: |
> > > > > > > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > > > > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > > > > > > +        image), as u0.16 format numbers.
> > > > > > > >
> > > > > > > > I don't like too much the "or if unavailable", as it means that the
> > > > > > > > reference rectangle will change depending on the platform
> > > > > > > > capabilities.
> > > > > >
> > > > > > I guess what I want is (like in the case of Android below), to specify
> > > > > > AF regions relative to the actual output image. If a platform supports
> > > > > > digital zoom (through ScalerCrop, I assume), the regions must be
> > > > > > relative to that part of the image. If there is no ScalerCrop, then do
> > > > > > we assume the platform doesn't support digital zoom? But the basic
> > > > > > idea is the same, the numbers supplied should be relative to the
> > > > > > output image in this case too.
> > > > > >
> > > > >
> > > > > Thing is that there's no "output image size" as two streams produced
> > > > > from the same camera might have different output sizes ?
> > > >
> > > > You're right that there might be several outputs. So what do I really
> > > > mean? I'm sort of looking for the rectangle in the sensor image that
> > > > is required to make all the outputs, but I think the ScalerCrop
> > > > encapsulates that idea reasonably well. Where there isn't a
> > > > ScalerCrop,  I think we use the active pixel area instead.
> > > >
> > >
> > > Gotcha! Indeed scaler crop defines the area of the sensor's output
> > > frame used to generate all streams... Thanks for sticking to it and
> > > help me straight my thoughts.
> > >
> > > I still don't like the "if not available part" :(
> > >
> > > I'll try to rope in others to get their opinions. What concerns me is
> > > that applications will behave differently on platforms where scaler
> > > crop is available compared to ones where it is not. Maybe I'm just
> > > over-concerned ?
> > >
> > >
> > > > >
> > > > > > Is there a better way to explain this, do you think?
> > > > > >
> > > > > > > >
> > > > > > > > I understand your argument in reply to Jean-Michel comment on the
> > > > > > > > previous version of the series (quoted here for reference)
> > > > > > > >
> > > > > > > > "The thing that bothers me a bit about the full pixel array is that, if
> > > > > > > > you use digital zoom, then you will have to update your AfWindows as
> > > > > > > > they might otherwise lie outside your final image! If it's defined as
> > > > > > > > being a region of the scaler crop, then it automatically stays the
> > > > > > > > same (to the application, at least, the pipeline handler probably has
> > > > > > > > some work to do).
> > > > > > > >
> > > > > > > > I note that the Android link you give below says "Focus areas are
> > > > > > > > relative to the current field of view (getZoom())", which I guess is
> > > > > > > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > > > > > > is easier to work with than the u0.16 format that I specified?"
> > > > > > > >
> > > > > > > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > > > > > > estimating the contrast, will it do so before or after the ScalerCrop
> > > > > > > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > > > > > > do so on the full pixel array size, the portion of if that gets
> > > > > > > > processed or on the final image ?
> > > > > >
> > > > > > In the case of PDAF, we can presumably use information from whatever
> > > > > > part of the image that the sensor is outputting.
> > > > > >
> > > > > > For CDAF (contrast detect AF) I guess it's possible that focus
> > > > > > statistics could be computed either before or after any cropping is
> > > > > > done in the ISP. (Maybe there are some platforms where cropping could
> > > > > > be delegated to the sensor?) So I'd guess that using any focus
> > > > > > statistics from outside the final FoV may actually be impossible,
> > > > > > depending on the platform. But even where you can do this, I don't
> > > > > > believe it's what anyone would want (and Android seems to behave like
> > > > > > this).
> > > > > >
> > > > > > > >
> > > > > > > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > > > > > > rectangle, so that they are applied to the actual FOV before being
> > > > > > > > passed to the AF algorithm ?
> > > > > >
> > > > > > Not quite sure what you meant here. I'd expect the ISP to generate
> > > > > > focus statistics from the raw Bayer pixels that it gets, without too
> > > > > > much processing applied to them. So the job will be to take rectangles
> > > > > > defined in the final FoV and translate them into the image coming
> > > > > > directly from the sensor.
> > > > > >
> > > > >
> > > > > What I meant is: if we specify the AfWindows relative to the full
> > > > > pixel array, can't we then re-scale it to final cropped output ? or if
> > > > > no copping on the sensor's output frame size.
> > > > >
> > > > > This would be similar to what you do with ScalerCrop already ?
> > > > >
> > > > >         void RPiCameraData::applyScalerCrop(const ControlList &controls) {
> > > > >
> > > > >                 Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
> > > > >
> > > > >                 /* Create a version of the crop scaled to ISP (camera mode) pixels. */
> > > > >                 Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
> > > > >                 ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
> > > > >         }
> > > > >
> > > > > But I guess I'm reasoning in absolute coordinates, not in proportions
> > > > > like you have proposed here.
> > > >
> > > > I think the question here is whether to use some kind of fractional
> > > > coordinates for describing the part of the ScalerCrop that we want AF
> > > > to use, or if we simply use pixel coordinates within the ScalerCrop
> > > > (or active pixel array), which would then look rather like the code
> > > > above.
> > > >
> > > > I suppose I don't particularly mind. If we use actual pixel
> > > > coordinates it's maybe a bit less clear that some other scaling
> > > > (shrinking to lie within the ScalerCrop) might be necessary, but if
> > > > we're happy that this is going to happen anyway then I think that
> > > > would work for me.
> > > >
> > >
> > > Let's hear from others as well!
> >
> > I think the focusAreas() is deprecated on Android.
> > https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas()
> > Currently it should be AF_REGIONS.
> > https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AF_REGIONS
> >
> > It uses the pixel array as the coordinate system of both ScalarCrop
> > and AFWindows.
> > For the problem that the AFWindow may be outside of the ScalarCrop
> > region, it adds an intersection rule:
> > Only the intersection of AFWindows and ScalarCrop takes effect.
> >
> > To be honest, the intersection rule is a little weird. I think
> > limiting the AFWindow to ScalarCrop looks good to me. The Android
> > adapter can do the intersection.
> > On the other hand, I think the benefit of using pixel arrays as
> > coordinate is that the precision can be at pixel level.
> > Not sure if it matters for any other algorithms.
> >


Let me summarize what I read there:

- For devices not supporting android.distortionCorrection.mode
  control, the coordinate system always follows that of
  android.sensor.info.activeArraySize

- For devices supporting android.distortionCorrection.mode control:
  - When the distortion correction mode is OFF, the coordinate system
    follows android.sensor.info.preCorrectionActiveArraySize
  - When the distortion correction mode is not OFF, the coordinate
    system follows android.sensor.info.activeArraySize

- If the metering region is outside the used android.scaler.cropRegion
  returned in capture result metadata, the camera device will ignore
  the sections outside the crop region and output only the
  intersection rectangle as the metering region in the result
  metadata. If the region is entirely outside the crop region, it will
  be ignored and not reported in the result metadata.

However with the introduction of android.control.zoomRatio (API level
30), now android.sensor.info.activeArraySize and
android.sensor.info.preCorrectionActiveArraySize represent
"post-zoomRatio field of view, not pre-zoom field of view."

Which implies "The afRegions coordinates are relative to the
activeArray/preCorrectionActiveArray representing the zoomed field of
view."

Now, we don't have anything like zoomRatio, but this change suggests
that the reference system is now actually the post-zoom field of view,
which for us corresponds to the ScalerCrop rectangle.

Could someone confim my understanding here ?

> > Another thing I'd like to mention is a rule for ScalarCrop in Android, I quote:
> > "The crop region is applied after the RAW to other color space (e.g.
> > YUV) conversion. Since raw streams (e.g. RAW16) don't have the
> > conversion stage, they are not croppable. The crop region will be
> > ignored by raw streams."
> >
> > Since the statistics is usually generated from the RAW, the input for
> > IPA would be the full pixel array's statistics.
> > In result, the IPA needs to translate the coordinate to match the
> > pixel array anyway.
> > It might be the reason that both ScalarCrop and AFWindows are in the
> > pixel array coordinate, since they will be in the end.
> >
> > This is also a weird rule, because it somehow forbids the device to
> > crop on sensor.
> >
> > >
> > > > >
> > > > > > > >
> > > > > > > > One last point about the unit: is the below representation correct,
> > > > > > > > when it comes to use a Rectangle to represent a proportion ?
> > > > > > > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > > > > > > pipeline report 12 rectangles like the ones below, for applications to
> > > > > > > > chose from ?
> > > > > > > >
> > > > > > > >       (0,0)
> > > > > > > >         +-----+-----+-----+-----+
> > > > > > > >         |     |     |     |     |
> > > > > > > >         +-----+-----+-----+-----+
> > > > > > > >         |     |     |     |     |
> > > > > > > >         +-----+-----+-----+-----+
> > > > > > > >         |     |     |     |     |
> > > > > > > >         +-----+-----+-----+-----+
> > > > > > > >
> > > > > > > > AfWindows = {
> > > > > > > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > > > > > > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > > > > > > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > > > > > > };
> > > > > > > >
> > > > > > > > My understanding is that the grid is fixed and depends on the platform
> > > > > > > > capabilities. I wonder if we shouldn't allow applications to select
> > > > > > > > grids by index.
> > > > > >
> > > > > > I think there is a problem about determining the AF capabilities of a
> > > > > > platform. I could imagine some platforms will only accept a regular
> > > > > > grid of regions, but others might let you put them in arbitrary
> > > > > > places. Or a mixture of both. To be honest, I'm not sure how we handle
> > > > > > that sort of thing. Anyone have any good ideas?
> > > > > >
> > > > >
> > > > > Exactly, that's why having proportions as the ones I tried to sketch
> > > > > out above doesn't really fit as a model in my head. Each platform
> > > > > would have its own 'grid', do we need to expose it to applications ?
> > > > >
> > > > > Can't we let application specify Rectangles with absolute coordinates
> > > > > (relative to PixelArray or else) and let the IPA/PH map those areas on
> > > > > their grids ?
> > > >
> > > > I really don't know how to deal with this. Some ISPs will have grids
> > > > of some sort, some may have windows in arbitrary locations, different
> > > > numbers and sizes of each depending on the platform, and some may do
> > > > both. Typically I would imagine many applications wanting to use grids
> > > > much of the time, but perhaps arbitrary rectangles for following
> > > > objects, people and faces.
> >
> > It's actually happening. GCam is using its own face detection
> > algorithm and passes the region to HAL.
> > ChromeOS is also integrating face algorithms to choose ROI for 3A.
> > Although they are for AE mostly, I think the windows for 3A should
> > have the same coordinate system.
> > I guess (I'm not sure) this is why the grid based focusAreas() is
> > deprecated on Android, and moves to pixel array coordinates, since the
> > face algorithm usually works on pixel coordinates. I think the grid of
> > ISPs could be left as implementation details for the pipeline handler?
> >
> I was woolly-minded. focusAreas() is not grid based, please ignore the
> sentence :P.
> Another thing is that AF_CROP_REGION allows windows to be overlapped
> and with weight.
> I'm not sure if we'd like to allow that. Perhaps this is discussed
> already and I missed it?

I think it was briefly mentioned, but no conclusion were taken.

> If there are ML algorithms to recognize the objects in a scene, the
> rectangle could be partially overlapped naturally, and may be weighted
> by objects, for example face within leaves is more weighted even if
> it's not the closet object in the scene. Usually the ML algorithm is
> located in the application and outside of HAL.
> I guess it's something we can decide later.

I agree.

I think what's pressing to decide, for David to be able to send a new
series is:

- Define the AfRegions unit

  There seems to be consensus on using pixel coordinates ?

- Define the AfRegions reference rectangle

  There are multiple reasons to prefer using the ScalerCrop rectangle.
  If my understanding is correct, Android has moved to use the same
  post-zoom reference since the introduction of zoomRatio (see above).

  However I'm still concerned about the fact ScalerCrop might not be
  there and depending on its availability, the reference system might
  change between different platforms.

  There might be way out though. Why don't we define ScalerCrop as a
  mandatory metadata, which is always returned even if the platform
  does not support digital zoom ? If that's the case, ScalerCrop will always
  be equal to the active pixel array. With this, we can say here that
  the reference system is the ScalerCrop rectangle as returned in the
  metadata.

  What do you think ?

- I think we can for now postpone discussing about weights.
  Conceptually it won't be hard to introduce them if we adopt
  something similar to the weighted rectangle android has.

David, are there still points that needs to be clarified to get to a
new version ? Is there anything we can help with ?

Thanks
   j

>
> > >
> > > Indeed. This is an advanced feature for which it might be expected
> > > application to know what the platform can provide maybe ? Describing
> > > all possible options seems very complex, and we'll never be able to
> > > express all of them.
> > >
> > > I would be fine stabilizing on a simpler grid-based version of AfWindows
> > > for now and improve on top, maybe ?
> > >
> > > >
> > > > Perhaps having an AfWindowGrid control and an AfWindows array control
> > > > of Rectangles could work? For the former, perhaps we leave the
> > > > decisions on the size of the grid to the platform/algorithm - I'd be
> > > > worried about having so many parameters that it would be difficult to
> > > > use. (You'd have to think about the size of the grid, the spacing of
> > > > rectangles in the grid etc... but perhaps Android requires us to
> > > > expose this sort of thing?) Perhaps someone knows what Android would
> > > > require here?
> > > >
> > > > >
> > > > > > > >
> > > > > > > > Also, do we need a property to report the grid structure and allows
> > > > > > > > application to select them from, or do you see this conveyed through a
> > > > > > > > ControlInfo ?
> > > > > >
> > > > > > I wasn't quite sure what the question was here. Are you talking about
> > > > > > selecting the configuration for the AfWindows, or selecting focus
> > > > > > statistics from particular regions? Perhaps an example would help me
> > > > > > here!
> > > > > >
> > > > >
> > > > > I'm talking about how would an application know how many spots it can
> > > > > select from, in other words, how do we let an application know that,
> > > > > looking at the above example, each grid is 1/4 in width and 1/3 in
> > > > > height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
> > > > > 4x3 one.
> > > > >
> > > > > Or maybe I got it all wrong, and you where implying already that the
> > > > > here specified Rectangles cover arbitrary parts of the image and it's
> > > > > the IPA that has to match them on their grids.
> > > >
> > > > Or maybe we treat grids as a whole different thing? (as per the
> > > > comments just above)
> > > >
> > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > +
> > > > > > > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > > > > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > > > > > > +        system will choose its own single default window in the centre of the
> > > > > > > > > +        image.
> > > > > > > > > +
> > > > > > > > > +        The details of how the windows are used are platform dependent. We note
> > > > > > > > > +        that when there is more than one AF window, a typical implementation
> > > > > > > > > +        might find the optimal focus position for each one and finally select
> > > > > > > > > +        the window closest to the camera.
> > > > > > > >
> > > > > > > > What window is more close to the camera ? My understanding is that
> > > > > > > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > > > > > > they all sit on the same focal plan :)
> > > > > >
> > > > > > Sorry, by "the window closest to the camera" I mean "the window that
> > > > > > is showing objects in the real world closest to the camera". I'll try
> > > > > > to re-word that a bit.
> > > > > >
> > > > >
> > > > > Yeah sorry, I could have realized that myself :)
> > > > >
> > > > > > > >
> > > > > > > > > +
> > > > > > > > > +        size: [n]
> > > > > > > > > +
> > > > > > > > > +  - AfTrigger:
> > > > > > > > > +      type: int32_t
> > > > > > > > > +      description: |
> > > > > > > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > > > > > > +        and can also be used to terminate a scan early.
> > > > > > > > > +
> > > > > > > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > > > > > > +
> > > > > > > > > +      enum:
> > > > > > > > > +        - name: AfTriggerStart
> > > > > > > > > +          value: 0
> > > > > > > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > > > > > > +        - name: AfTriggerCancel
> > > > > > > > > +          value: 1
> > > > > > > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > > > > > > +            anywhere else. Ignored if no scan is in progress.
> > > > > > > > > +
> > > > > > > > > +  - AfPause:
> > > > > > > > > +      type: int32_t
> > > > > > > > > +      description: |
> > > > > > > > > +        This control has no effect except when in continuous autofocus mode
> > > > > > > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > > > > > > +        (for example) images are captured. The algorithm remains inactive
> > > > > > > > > +        until it is instructed to resume.
> > > > > > > > > +
> > > > > > > > > +      enum:
> > > > > > > > > +        - name: AfPauseImmediate
> > > > > > > > > +          value: 0
> > > > > > > > > +          description: |
> > > > > > > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > > > > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > > > > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > > > > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > > > > > > +            when it received this control.
> > > > > > > > > +        - name: AfPauseDeferred
> > > > > > > > > +          value: 1
> > > > > > > > > +          description: |
> > > > > > > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > > > > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > > > > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > > > > > > +
> > > > > > > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > > > > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > > > > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > > > > > > +            AfPauseState will change to AfPauseStatePaused.
> > > > > > > >
> > > > > > > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > > > > > > align it back ?
> > > > > >
> > > > > > Sure, I'll have a look!
> > > > > >
> > > > > > > >
> > > > > > > > > +        - name: AfPauseResume
> > > > > > > > > +          value: 2
> > > > > > > > > +          description: |
> > > > > > > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > > > > > > +            from exactly where it left off, and AfPauseState will report
> > > > > > > > > +            AfPauseStateRunning.
> > > > > > > > > +
> > > > > > > > > +  - LensPosition:
> > > > > > > > > +      type: float
> > > > > > > > > +      description: |
> > > > > > > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > > > > > > +        and also reports back the position of the lens for each frame.
> > > > > > > > > +
> > > > > > > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > > > > > > +        AfModeManual.
> > > > > > > > > +
> > > > > > > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > > > > > > +        values are permitted. For example:
> > > > > > > > > +        0 moves the lens to infinity.
> > > > > > > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > > > > > > +        1 moves the lens to the hyperfocal position.
> > > > > > > > > +        And larger values will focus the lens ever closer.
> > > > > > > > > +
> > > > > > > >
> > > > > > > > This seems good to me, albeit my understanding of the issue is limited :)
> > > > > > > >
> > > > > > >
> > > > > > > Do we need a property to report the lens hyperfocal distance ?
> > > > > >
> > > > > > Yes, sometimes. Android distinguishes between calibrated and
> > > > > > uncalibrated cameras. For a calibrated camera an application should be
> > > > > > able to set the focal distance to (for example) 1m, and given the
> > > > > > units I've proposed, you would need to know the hyperfocal distance to
> > > > > > calculate the right number.
> > > > >
> > > > > Right, so we need a LensHyperfocalDistance or something for calibrated
> > > > > lenses ?
> > > >
> > > > I think so.
> > > >
> > > > >
> > > > > >
> > > > > > For an uncalibrated camera I don't think it matters. Many applications
> > > > > > would simply set the lens position to 1.0 (meaning hyperfocal) and
> > > > > > you'll get the "best overall" focal distance (largest depth of field),
> > > > > > and that's it.
> > > > >
> > > > > I think this part would be relevant to be captured in the control
> > > > > description.
> > > >
> > > > Yes, I can add something about that.
> > > >
> > > > >
> > > > > >
> > > > > > There's also a need to know the allowable range of lens movement. This
> > > > > > will depend on the type of module (knowing the lens driver isn't
> > > > > > enough). But I don't see that applications need to know this, I think
> > > > > > the pipeline handler and AF implementation should take care of it.
> > > > > >
> > > > >
> > > > > I think so. When it comes to the actual lens movement range I don't
> > > > > think it's relevant for applications. But do the actual lens movement
> > > > > range maps to a range of LensPosition values that can be specified ? I
> > > > > guess it's hard to map them on un-calibrated lenses, but for
> > > > > calibrated ones a LensPositionRanges whose ControlInfo informs
> > > > > application on the range's min and max might make sense ? (Please note
> > > > > that this could also be used for the use case we discussed about
> > > > > wanting to limit the lens range search scope)
> > > >
> > > > I think I would expect a calibrated lens would have to have some kind
> > > > of property that says what the nearest and furthest focal distances
> > > > are (the furthest usually being zero, meaning infinity). For an
> > > > uncalibrated lens things will only ever be "approximately correct".
> > >
> > >
> > > Care to add the property in next version, or can we do it on top with
> > > a separate patch ?
> > >
> > > Thanks
> > >    j
> > >
> > > >
> > > > I could imagine this being used in conjunction with the hypothetical
> > > > AfRangeCustom to construct custom search ranges for particular
> > > > applications (in future, perhaps).
> > > >
> > > > Thanks again!
> > > > David
> > > >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > > > +  - AfState:
> > > > > > > > > +      type: int32_t
> > > > > > > > > +      description: |
> > > > > > > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > > > > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > > > > > > +        The possible state changes are described below, though we note the following
> > > > > > > > > +        state transitions that occur when the AfMode is changed.
> > > > > > > > > +
> > > > > > > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > > > > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > > > > > > +        LensPosition control must subsequently be used to achieve this.
> > > > > > > > > +
> > > > > > > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > > > > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > > > > > > >
> > > > > > > > s/unless/until ?
> > > > > > > > s/at the same time// ?
> > > > > >
> > > > > > I'm trying to describe what happens to the AfState when you send
> > > > > > AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> > > > > > AfTriggerStart). The only occasion when you won't see AfStateIdle will
> > > > > > be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> > > > > > think this reads OK to me, but I'll see if I can make the wording any
> > > > > > clearer.
> > > > >
> > > > > I see. I didn't get it, sorry. If it's fine with you, keep this
> > > > > version!
> > > > >
> > > > > >
> > > > > > > >
> > > > > > > > > +        case it will (start a scan and) report AfStateScanning.
> > > > > > > > > +
> > > > > > > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > > > > > > +        report AfStateScanning.
> > > > > > > > > +
> > > > > > > > > +      enum:
> > > > > > > > > +        - name: AfStateReset
> > > > > > > > > +          value: 0
> > > > > > > > > +          description: |
> > > > > > > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > > > +            scan was cancelled.
> > > > > > > > > +        - name: AfStateScanning
> > > > > > > > > +          value: 1
> > > > > > > > > +          description: |
> > > > > > > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > > > > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > > > > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > > > > > > +            move back to AfStateReset or, if the scan actually completes
> > > > > > > > > +            before the cancel request is processed, to one of
> > > > > > > > > +            AfStateFocused or AfStateFailed.
> > > > > > > > > +
> > > > > > > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > > > > > > >
> > > > > > > > s/be be/is ?
> > > > > >
> > > > > > Yes, thanks!
> > > > > >
> > > > > > > >
> > > > > > > > > +            at which point it may enter this state spontaneously whenever it
> > > > > > > > > +            determines that a rescan is needed.
> > > > > > > > > +        - name: AfStateFocused
> > > > > > > > > +          value: 2
> > > > > > > > > +          description: |
> > > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > > > > > > +            the image is now in focus.
> > > > > > > > > +        - name: AfStateFailed
> > > > > > > > > +          value: 3
> > > > > > > > > +          description: |
> > > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > > > > > > +            find a good focus position.
> > > > > > > > > +
> > > > > > > > > +  - AfPauseState:
> > > > > > > > > +      type: int32_t
> > > > > > > > > +      description: |
> > > > > > > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > > > > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > > > > > > +        as soon as any in-progress scan completes).
> > > > > > > > > +
> > > > > > > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > > > > > > +
> > > > > > > > > +      enum:
> > > > > > > > > +        - name: AfPauseStateRunning
> > > > > > > > > +          value: 0
> > > > > > > > > +          description: |
> > > > > > > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > > > > > > +            spontaneously.
> > > > > > > > > +        - name: AfPauseStatePausing
> > > > > > > > > +          value: 1
> > > > > > > > > +          description: |
> > > > > > > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > > > > > > +            as soon as any in-progress scan completes (and then report
> > > > > > > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > > > > > > +            the AfPauseResume control is sent.
> > > > > > > > > +        - name: AfPauseStatePaused
> > > > > > > > > +          value: 2
> > > > > > > > > +          description: |
> > > > > > > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > > > > > > +            will occur until the AfPauseResume control is sent.
> > > > > > > > > +
> > > > > > > >
> > > > > > > > Very nice overall! With a few more clarifications I think this is good
> > > > > > > > to go!
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >    j
> > > > > >
> > > > > > Thanks!
> > > > > > David
> > > > > >
> > > > > > > >
> > > > > > > > >    # ----------------------------------------------------------------------------
> > > > > > > > >    # Draft controls section
> > > > > > > > >
> > > > > > > > > @@ -406,27 +664,6 @@ controls:
> > > > > > > > >              The camera will cancel any active or completed metering sequence.
> > > > > > > > >              The AE algorithm is reset to its initial state.
> > > > > > > > >
> > > > > > > > > -  - AfTrigger:
> > > > > > > > > -      type: int32_t
> > > > > > > > > -      draft: true
> > > > > > > > > -      description: |
> > > > > > > > > -       Control for AF trigger. Currently identical to
> > > > > > > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > > > > > > -
> > > > > > > > > -        Whether the camera device will trigger autofocus for this request.
> > > > > > > > > -      enum:
> > > > > > > > > -        - name: AfTriggerIdle
> > > > > > > > > -          value: 0
> > > > > > > > > -          description: The trigger is idle.
> > > > > > > > > -        - name: AfTriggerStart
> > > > > > > > > -          value: 1
> > > > > > > > > -          description: The AF routine is started by the camera.
> > > > > > > > > -        - name: AfTriggerCancel
> > > > > > > > > -          value: 2
> > > > > > > > > -          description: |
> > > > > > > > > -            The camera will cancel any active trigger and the AF routine is
> > > > > > > > > -            reset to its initial state.
> > > > > > > > > -
> > > > > > > > >    - NoiseReductionMode:
> > > > > > > > >        type: int32_t
> > > > > > > > >        draft: true
> > > > > > > > > @@ -507,45 +744,6 @@ controls:
> > > > > > > > >              The AE algorithm has started a pre-capture metering session.
> > > > > > > > >              \sa AePrecaptureTrigger
> > > > > > > > >
> > > > > > > > > -  - AfState:
> > > > > > > > > -      type: int32_t
> > > > > > > > > -      draft: true
> > > > > > > > > -      description: |
> > > > > > > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > > > > > > -       ANDROID_CONTROL_AF_STATE.
> > > > > > > > > -
> > > > > > > > > -        Current state of the AF algorithm.
> > > > > > > > > -      enum:
> > > > > > > > > -        - name: AfStateInactive
> > > > > > > > > -          value: 0
> > > > > > > > > -          description: The AF algorithm is inactive.
> > > > > > > > > -        - name: AfStatePassiveScan
> > > > > > > > > -          value: 1
> > > > > > > > > -          description: |
> > > > > > > > > -            AF is performing a passive scan of the scene in continuous
> > > > > > > > > -            auto-focus mode.
> > > > > > > > > -        - name: AfStatePassiveFocused
> > > > > > > > > -          value: 2
> > > > > > > > > -          description: |
> > > > > > > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > > > > > > -        - name: AfStateActiveScan
> > > > > > > > > -          value: 3
> > > > > > > > > -          description: |
> > > > > > > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > > > > > > -            \sa AfTrigger
> > > > > > > > > -        - name: AfStateFocusedLock
> > > > > > > > > -          value: 4
> > > > > > > > > -          description: |
> > > > > > > > > -            AF believes has focused correctly and has locked focus.
> > > > > > > > > -        - name: AfStateNotFocusedLock
> > > > > > > > > -          value: 5
> > > > > > > > > -          description: |
> > > > > > > > > -            AF has not been able to focus and has locked.
> > > > > > > > > -        - name: AfStatePassiveUnfocused
> > > > > > > > > -          value: 6
> > > > > > > > > -          description: |
> > > > > > > > > -            AF has completed a passive scan without finding focus.
> > > > > > > > > -
> > > > > > > > >    - AwbState:
> > > > > > > > >        type: int32_t
> > > > > > > > >        draft: true
> > > > > > > > > --
> > > > > > > > > 2.30.2
> > > > > > > > >

Hi Jacopo,

On Fri, Apr 29, 2022 at 3:54 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
>
> Hi Han-Lin,
>    thanks for the many details!
>
> On Thu, Apr 28, 2022 at 11:44:23PM +0800, Hanlin Chen wrote:
> > On Thu, Apr 28, 2022 at 8:45 PM Hanlin Chen <hanlinchen@chromium.org> wrote:
> > >
> > > Hi everyone,
> > >
> > > On Tue, Apr 26, 2022 at 9:16 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > >
> > > > Hi David,
> > > >
> > > > On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
> > > > > Hi Jacopo
> > > > >
> > > > > Thanks for the message. Some replies and more questions below!
> > > > >
> > > > > On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> > > > > > > Hi Jacopo
> > > > > > >
> > > > > > > Thanks for the quick reply!
> > > > > > >
> > > > > > > On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > > > > >
> > > > > > > > Ah, forgot one thing!
> > > > > > > >
> > > > > > > > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > > Hi David,
> > > > > > > > >   thanks for this new version
> > > > > > > > >
> > > > > > > > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > > > > > > > This patch describes a series of controls that allow applications to
> > > > > > > > > > drive AF algorithms:
> > > > > > > > > >
> > > > > > > > > > AfMode - manual, auto or continuous
> > > > > > > > > > AfRange - full, macro or normal
> > > > > > > > > > AfSpeed - fast or slowwith
> > > > > > > > > > AfWindows - AF window locations
> > > > > > > > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > > > > > > > AfPause - pause continuous AF
> > > > > > > > > > LensPosition - set or retrieve position of lens
> > > > > > > > > > AfState - reports whether scanning/success/failure
> > > > > > > > > > AfPauseState - reports whether continuous AF paused or not
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > > > > > > > ---
> > > > > > > > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > > > > > > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > > > > > > > index 9d4638ae..eccf18bc 100644
> > > > > > > > > > --- a/src/libcamera/control_ids.yaml
> > > > > > > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > > > > > > @@ -381,6 +381,264 @@ controls:
> > > > > > > > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > > > > > > > >          use case.
> > > > > > > > > >
> > > > > > > > > > +  - AfMode:
> > > > > > > > > > +      type: int32_t
> > > > > > > > > > +      description: |
> > > > > > > > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > > > > > > > +        are allowed to set a new mode, and to send additional controls for
> > > > > > > > > > +        that new mode, in the same request.
> > > > > > > > > > +
> > > > > > > > > > +        An implementation may choose not to implement all the modes.
> > > > > > > > > > +
> > > > > > > > > > +      enum:
> > > > > > > > > > +        - name: AfModeManual
> > > > > > > > > > +          value: 0
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > > > > > > > +            perform any action nor move the lens of its own accord, but an
> > > > > > > > > > +            application can set controls to move the lens.
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > > > > > > > when in manual mode, I would say so explicitly
> > > > > > > > >
> > > > > > > > >                "... but an application can specify the desired lens
> > > > > > > > >                position using the LensPosition control."
> > > > > > >
> > > > > > > Yes, will do.
> > > > > > >
> > > > > > > > >
> > > > > > > > > > +            In this mode the AfState will always report AfStateReset.
> > > > > > > > >
> > > > > > > > > I'm tempted to say AfState is useless and can not be reported at all
> > > > > > > > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > > > > > > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > > > > > > > don't have better alternatives to propose though.
> > > > > > >
> > > > > > > Don't really mind either. I think reporting stuff all the time might
> > > > > > > be slightly easier, at least it's always obvious what to expect...
> > > > > > >
> > > > > > > > >
> > > > > > > > > Reading the AfStateReset description
> > > > > > > > >
> > > > > > > > >         - name: AfStateReset
> > > > > > > > >           value: 0
> > > > > > > > >           description: |
> > > > > > > > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > > >             scan was cancelled.
> > > > > > > > >
> > > > > > > > > I wonder if this isn't better named AfStateIdle
> > > > > > >
> > > > > > > Yes, also fine with that.
> > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > +        - name: AfModeAuto
> > > > > > > > > > +          value: 1
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > > > > > > > +            will never move the lens or change state unless the AfTrigger
> > > > > > > > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > > > > > > > +            focus scan, the results of which will be reported by AfState.
> > > > > > > > > > +
> > > > > > > > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > > > > > > > +            mode while a scan is in progress, the scan is cancelled
> > > > > > > > > > +            immediately, without waiting for the scan to finish.
> > > > > > > > > > +
> > > > > > > > > > +            When first entering this mode the AfState will report
> > > > > > > > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > > > > > > > +            report AfStateScanning for a period before spontaneously
> > > > > > > > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > > > > > > > +            the outcome of the scan. It will remain in this state until
> > > > > > > > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > > > > > > > +            cancelled (without changing to another mode), AfState will return
> > > > > > > > > > +            to AfStateReset.
> > > > > > > > > > +        - name: AfModeContinuous
> > > > > > > > > > +          value: 2
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > > > > > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > > > > > > > +            intervention. The AfState still reports whether the algorithm is
> > > > > > > > > > +            currently scanning or not, though the application has no ability
> > > > > > > > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > > > > > > > +            move the lens for itself.
> > > > > > > > > > +
> > > > > > > > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > > > > > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > > > > > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > > > > > > > +
> > > > > > > > > > +  - AfRange:
> > > > > > > > > > +      type: int32_t
> > > > > > > > > > +      description: |
> > > > > > > > > > +        Control to set the range of focus distances that is scanned. An
> > > > > > > > > > +        implementation may choose not to implement all the options here.
> > > > > > > > > > +      enum:
> > > > > > > > > > +        - name: AfRangeNormal
> > > > > > > > > > +          value: 0
> > > > > > > > > > +          description: |
> > > > > > > > > > +            A wide range of focus distances is scanned, all the way from
> > > > > > > > > > +            infinity down to close distances, though depending on the
> > > > > > > > > > +            implementation, possibly not including the very closest macro
> > > > > > > > > > +            positions.
> > > > > > > > > > +        - name: AfRangeMacro
> > > > > > > > > > +          value: 1
> > > > > > > > > > +          description: Only close distances are scanned.
> > > > > > > > > > +        - name: AfRangeFull
> > > > > > > > > > +          value: 2
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The full range of focus distances is scanned just as with
> > > > > > > > > > +            AfRangeNormal but this time including the very closest macro
> > > > > > > > > > +            positions.
> > > > > > > > > > +
> > > > > > > > > > +  - AfSpeed:
> > > > > > > > > > +      type: int32_t
> > > > > > > > > > +      description: |
> > > > > > > > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > > > > > > > +        as quickly as possible or more steadily. For example, during video
> > > > > > > > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > > > > > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > > > > > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > > > > > > > +      enum:
> > > > > > > > > > +        - name: AfSpeedNormal
> > > > > > > > > > +          value: 0
> > > > > > > > > > +          description: Move the lens at its usual speed.
> > > > > > > > > > +        - name: AfSpeedFast
> > > > > > > > > > +          value: 1
> > > > > > > > > > +          description: Move the lens more quickly.
> > > > > > > > > > +
> > > > > > > > > > +  - AfWindows:
> > > > > > > > > > +      type: Rectangle
> > > > > > > > > > +      description: |
> > > > > > > > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > > > > > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > > > > > > > +        image), as u0.16 format numbers.
> > > > > > > > >
> > > > > > > > > I don't like too much the "or if unavailable", as it means that the
> > > > > > > > > reference rectangle will change depending on the platform
> > > > > > > > > capabilities.
> > > > > > >
> > > > > > > I guess what I want is (like in the case of Android below), to specify
> > > > > > > AF regions relative to the actual output image. If a platform supports
> > > > > > > digital zoom (through ScalerCrop, I assume), the regions must be
> > > > > > > relative to that part of the image. If there is no ScalerCrop, then do
> > > > > > > we assume the platform doesn't support digital zoom? But the basic
> > > > > > > idea is the same, the numbers supplied should be relative to the
> > > > > > > output image in this case too.
> > > > > > >
> > > > > >
> > > > > > Thing is that there's no "output image size" as two streams produced
> > > > > > from the same camera might have different output sizes ?
> > > > >
> > > > > You're right that there might be several outputs. So what do I really
> > > > > mean? I'm sort of looking for the rectangle in the sensor image that
> > > > > is required to make all the outputs, but I think the ScalerCrop
> > > > > encapsulates that idea reasonably well. Where there isn't a
> > > > > ScalerCrop,  I think we use the active pixel area instead.
> > > > >
> > > >
> > > > Gotcha! Indeed scaler crop defines the area of the sensor's output
> > > > frame used to generate all streams... Thanks for sticking to it and
> > > > help me straight my thoughts.
> > > >
> > > > I still don't like the "if not available part" :(
> > > >
> > > > I'll try to rope in others to get their opinions. What concerns me is
> > > > that applications will behave differently on platforms where scaler
> > > > crop is available compared to ones where it is not. Maybe I'm just
> > > > over-concerned ?
> > > >
> > > >
> > > > > >
> > > > > > > Is there a better way to explain this, do you think?
> > > > > > >
> > > > > > > > >
> > > > > > > > > I understand your argument in reply to Jean-Michel comment on the
> > > > > > > > > previous version of the series (quoted here for reference)
> > > > > > > > >
> > > > > > > > > "The thing that bothers me a bit about the full pixel array is that, if
> > > > > > > > > you use digital zoom, then you will have to update your AfWindows as
> > > > > > > > > they might otherwise lie outside your final image! If it's defined as
> > > > > > > > > being a region of the scaler crop, then it automatically stays the
> > > > > > > > > same (to the application, at least, the pipeline handler probably has
> > > > > > > > > some work to do).
> > > > > > > > >
> > > > > > > > > I note that the Android link you give below says "Focus areas are
> > > > > > > > > relative to the current field of view (getZoom())", which I guess is
> > > > > > > > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > > > > > > > is easier to work with than the u0.16 format that I specified?"
> > > > > > > > >
> > > > > > > > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > > > > > > > estimating the contrast, will it do so before or after the ScalerCrop
> > > > > > > > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > > > > > > > do so on the full pixel array size, the portion of if that gets
> > > > > > > > > processed or on the final image ?
> > > > > > >
> > > > > > > In the case of PDAF, we can presumably use information from whatever
> > > > > > > part of the image that the sensor is outputting.
> > > > > > >
> > > > > > > For CDAF (contrast detect AF) I guess it's possible that focus
> > > > > > > statistics could be computed either before or after any cropping is
> > > > > > > done in the ISP. (Maybe there are some platforms where cropping could
> > > > > > > be delegated to the sensor?) So I'd guess that using any focus
> > > > > > > statistics from outside the final FoV may actually be impossible,
> > > > > > > depending on the platform. But even where you can do this, I don't
> > > > > > > believe it's what anyone would want (and Android seems to behave like
> > > > > > > this).
> > > > > > >
> > > > > > > > >
> > > > > > > > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > > > > > > > rectangle, so that they are applied to the actual FOV before being
> > > > > > > > > passed to the AF algorithm ?
> > > > > > >
> > > > > > > Not quite sure what you meant here. I'd expect the ISP to generate
> > > > > > > focus statistics from the raw Bayer pixels that it gets, without too
> > > > > > > much processing applied to them. So the job will be to take rectangles
> > > > > > > defined in the final FoV and translate them into the image coming
> > > > > > > directly from the sensor.
> > > > > > >
> > > > > >
> > > > > > What I meant is: if we specify the AfWindows relative to the full
> > > > > > pixel array, can't we then re-scale it to final cropped output ? or if
> > > > > > no copping on the sensor's output frame size.
> > > > > >
> > > > > > This would be similar to what you do with ScalerCrop already ?
> > > > > >
> > > > > >         void RPiCameraData::applyScalerCrop(const ControlList &controls) {
> > > > > >
> > > > > >                 Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
> > > > > >
> > > > > >                 /* Create a version of the crop scaled to ISP (camera mode) pixels. */
> > > > > >                 Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
> > > > > >                 ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
> > > > > >         }
> > > > > >
> > > > > > But I guess I'm reasoning in absolute coordinates, not in proportions
> > > > > > like you have proposed here.
> > > > >
> > > > > I think the question here is whether to use some kind of fractional
> > > > > coordinates for describing the part of the ScalerCrop that we want AF
> > > > > to use, or if we simply use pixel coordinates within the ScalerCrop
> > > > > (or active pixel array), which would then look rather like the code
> > > > > above.
> > > > >
> > > > > I suppose I don't particularly mind. If we use actual pixel
> > > > > coordinates it's maybe a bit less clear that some other scaling
> > > > > (shrinking to lie within the ScalerCrop) might be necessary, but if
> > > > > we're happy that this is going to happen anyway then I think that
> > > > > would work for me.
> > > > >
> > > >
> > > > Let's hear from others as well!
> > >
> > > I think the focusAreas() is deprecated on Android.
> > > https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas()
> > > Currently it should be AF_REGIONS.
> > > https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AF_REGIONS
> > >
> > > It uses the pixel array as the coordinate system of both ScalarCrop
> > > and AFWindows.
> > > For the problem that the AFWindow may be outside of the ScalarCrop
> > > region, it adds an intersection rule:
> > > Only the intersection of AFWindows and ScalarCrop takes effect.
> > >
> > > To be honest, the intersection rule is a little weird. I think
> > > limiting the AFWindow to ScalarCrop looks good to me. The Android
> > > adapter can do the intersection.
> > > On the other hand, I think the benefit of using pixel arrays as
> > > coordinate is that the precision can be at pixel level.
> > > Not sure if it matters for any other algorithms.
> > >
>
>
> Let me summarize what I read there:
>
> - For devices not supporting android.distortionCorrection.mode
>   control, the coordinate system always follows that of
>   android.sensor.info.activeArraySize
>
> - For devices supporting android.distortionCorrection.mode control:
>   - When the distortion correction mode is OFF, the coordinate system
>     follows android.sensor.info.preCorrectionActiveArraySize
>   - When the distortion correction mode is not OFF, the coordinate
>     system follows android.sensor.info.activeArraySize
>
> - If the metering region is outside the used android.scaler.cropRegion
>   returned in capture result metadata, the camera device will ignore
>   the sections outside the crop region and output only the
>   intersection rectangle as the metering region in the result
>   metadata. If the region is entirely outside the crop region, it will
>   be ignored and not reported in the result metadata.
>
> However with the introduction of android.control.zoomRatio (API level
> 30), now android.sensor.info.activeArraySize and
> android.sensor.info.preCorrectionActiveArraySize represent
> "post-zoomRatio field of view, not pre-zoom field of view."
>
> Which implies "The afRegions coordinates are relative to the
> activeArray/preCorrectionActiveArray representing the zoomed field of
> view."
>
> Now, we don't have anything like zoomRatio, but this change suggests
> that the reference system is now actually the post-zoom field of view,
> which for us corresponds to the ScalerCrop rectangle.
It's still the activeArray coordinate system.
>
> Could someone confim my understanding here ?

In my understanding the zoomRatio is defined for optical zoom, but the
digital zoom is still by cropRegion. See:
https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_ZOOM_RATIO

The post-zoom field of view here means the field zoomed by an optical lens.
Because it's set by lens, even if it's zoomed, the frames from the
sensor are still of the rectangle (0, 0, activeArrayWidth,
activeArrayHeight).
The ScalarCrop and afRegions are still defined in the rectangle, i.e.,
the pixel array coordinate system.

I quote an example from the link:
Assuming the pixel array is (0, 0, 2000, 1500). The application can
achieve 2.0x zoom in one of two ways:
1. zoomRatio = 2.0, scaler.cropRegion = (0, 0, 2000, 1500) // By optical zoom
2. zoomRatio = 1.0 (default), scaler.cropRegion = (500, 375, 1500,
1125) // By digital zoom

If the application intends to set aeRegions to be top-left quarter of
the viewfinder field-of-view, the android.control.aeRegions should be
set to (0, 0, 1000, 750) with zoomRatio set to 2.0. Alternatively, the
application can set aeRegions to the equivalent region of (500, 375,
1000, 750) for zoomRatio of 1.0. If the application doesn't explicitly
set android.control.zoomRatio, its value defaults to 1.0.

As above, the aeRegion is always in pixel array coordinate
before/after API 30, no matter how zoomRatio is set, but the
application needs to take zoomRation into consideration.
I think they are defined like this to remain the coordinate behavior
for an legacy application that isn't aware of the new zoomRation API,
and the framework default it to 1.0.

However, since the ScalarCrop is relative to pixelArray, no matter
which one afRegion is relative to, they are interchangeable.
We don't need to stick to that of Android.
>
> > > Another thing I'd like to mention is a rule for ScalarCrop in Android, I quote:
> > > "The crop region is applied after the RAW to other color space (e.g.
> > > YUV) conversion. Since raw streams (e.g. RAW16) don't have the
> > > conversion stage, they are not croppable. The crop region will be
> > > ignored by raw streams."
> > >
> > > Since the statistics is usually generated from the RAW, the input for
> > > IPA would be the full pixel array's statistics.
> > > In result, the IPA needs to translate the coordinate to match the
> > > pixel array anyway.
> > > It might be the reason that both ScalarCrop and AFWindows are in the
> > > pixel array coordinate, since they will be in the end.
> > >
> > > This is also a weird rule, because it somehow forbids the device to
> > > crop on sensor.
> > >
> > > >
> > > > > >
> > > > > > > > >
> > > > > > > > > One last point about the unit: is the below representation correct,
> > > > > > > > > when it comes to use a Rectangle to represent a proportion ?
> > > > > > > > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > > > > > > > pipeline report 12 rectangles like the ones below, for applications to
> > > > > > > > > chose from ?
> > > > > > > > >
> > > > > > > > >       (0,0)
> > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > >         |     |     |     |     |
> > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > >         |     |     |     |     |
> > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > >         |     |     |     |     |
> > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > >
> > > > > > > > > AfWindows = {
> > > > > > > > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > > > > > > > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > > > > > > > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > My understanding is that the grid is fixed and depends on the platform
> > > > > > > > > capabilities. I wonder if we shouldn't allow applications to select
> > > > > > > > > grids by index.
> > > > > > >
> > > > > > > I think there is a problem about determining the AF capabilities of a
> > > > > > > platform. I could imagine some platforms will only accept a regular
> > > > > > > grid of regions, but others might let you put them in arbitrary
> > > > > > > places. Or a mixture of both. To be honest, I'm not sure how we handle
> > > > > > > that sort of thing. Anyone have any good ideas?
> > > > > > >
> > > > > >
> > > > > > Exactly, that's why having proportions as the ones I tried to sketch
> > > > > > out above doesn't really fit as a model in my head. Each platform
> > > > > > would have its own 'grid', do we need to expose it to applications ?
> > > > > >
> > > > > > Can't we let application specify Rectangles with absolute coordinates
> > > > > > (relative to PixelArray or else) and let the IPA/PH map those areas on
> > > > > > their grids ?
> > > > >
> > > > > I really don't know how to deal with this. Some ISPs will have grids
> > > > > of some sort, some may have windows in arbitrary locations, different
> > > > > numbers and sizes of each depending on the platform, and some may do
> > > > > both. Typically I would imagine many applications wanting to use grids
> > > > > much of the time, but perhaps arbitrary rectangles for following
> > > > > objects, people and faces.
> > >
> > > It's actually happening. GCam is using its own face detection
> > > algorithm and passes the region to HAL.
> > > ChromeOS is also integrating face algorithms to choose ROI for 3A.
> > > Although they are for AE mostly, I think the windows for 3A should
> > > have the same coordinate system.
> > > I guess (I'm not sure) this is why the grid based focusAreas() is
> > > deprecated on Android, and moves to pixel array coordinates, since the
> > > face algorithm usually works on pixel coordinates. I think the grid of
> > > ISPs could be left as implementation details for the pipeline handler?
> > >
> > I was woolly-minded. focusAreas() is not grid based, please ignore the
> > sentence :P.
> > Another thing is that AF_CROP_REGION allows windows to be overlapped
> > and with weight.
> > I'm not sure if we'd like to allow that. Perhaps this is discussed
> > already and I missed it?
>
> I think it was briefly mentioned, but no conclusion were taken.
>
> > If there are ML algorithms to recognize the objects in a scene, the
> > rectangle could be partially overlapped naturally, and may be weighted
> > by objects, for example face within leaves is more weighted even if
> > it's not the closet object in the scene. Usually the ML algorithm is
> > located in the application and outside of HAL.
> > I guess it's something we can decide later.
>
> I agree.
>
> I think what's pressing to decide, for David to be able to send a new
> series is:
>
> - Define the AfRegions unit
>
>   There seems to be consensus on using pixel coordinates ?
>
> - Define the AfRegions reference rectangle
>
>   There are multiple reasons to prefer using the ScalerCrop rectangle.
>   If my understanding is correct, Android has moved to use the same
>   post-zoom reference since the introduction of zoomRatio (see above).
>
>   However I'm still concerned about the fact ScalerCrop might not be
>   there and depending on its availability, the reference system might
>   change between different platforms.
>
>   There might be way out though. Why don't we define ScalerCrop as a
>   mandatory metadata, which is always returned even if the platform
>   does not support digital zoom ? If that's the case, ScalerCrop will always
>   be equal to the active pixel array. With this, we can say here that
>   the reference system is the ScalerCrop rectangle as returned in the
>   metadata.
>
>   What do you think ?

I agree.

>
> - I think we can for now postpone discussing about weights.
>   Conceptually it won't be hard to introduce them if we adopt
>   something similar to the weighted rectangle android has.
>
> David, are there still points that needs to be clarified to get to a
> new version ? Is there anything we can help with ?
>
> Thanks
>    j
>
> >
> > > >
> > > > Indeed. This is an advanced feature for which it might be expected
> > > > application to know what the platform can provide maybe ? Describing
> > > > all possible options seems very complex, and we'll never be able to
> > > > express all of them.
> > > >
> > > > I would be fine stabilizing on a simpler grid-based version of AfWindows
> > > > for now and improve on top, maybe ?
> > > >
> > > > >
> > > > > Perhaps having an AfWindowGrid control and an AfWindows array control
> > > > > of Rectangles could work? For the former, perhaps we leave the
> > > > > decisions on the size of the grid to the platform/algorithm - I'd be
> > > > > worried about having so many parameters that it would be difficult to
> > > > > use. (You'd have to think about the size of the grid, the spacing of
> > > > > rectangles in the grid etc... but perhaps Android requires us to
> > > > > expose this sort of thing?) Perhaps someone knows what Android would
> > > > > require here?
> > > > >
> > > > > >
> > > > > > > > >
> > > > > > > > > Also, do we need a property to report the grid structure and allows
> > > > > > > > > application to select them from, or do you see this conveyed through a
> > > > > > > > > ControlInfo ?
> > > > > > >
> > > > > > > I wasn't quite sure what the question was here. Are you talking about
> > > > > > > selecting the configuration for the AfWindows, or selecting focus
> > > > > > > statistics from particular regions? Perhaps an example would help me
> > > > > > > here!
> > > > > > >
> > > > > >
> > > > > > I'm talking about how would an application know how many spots it can
> > > > > > select from, in other words, how do we let an application know that,
> > > > > > looking at the above example, each grid is 1/4 in width and 1/3 in
> > > > > > height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
> > > > > > 4x3 one.
> > > > > >
> > > > > > Or maybe I got it all wrong, and you where implying already that the
> > > > > > here specified Rectangles cover arbitrary parts of the image and it's
> > > > > > the IPA that has to match them on their grids.
> > > > >
> > > > > Or maybe we treat grids as a whole different thing? (as per the
> > > > > comments just above)
> > > > >
> > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > +
> > > > > > > > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > > > > > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > > > > > > > +        system will choose its own single default window in the centre of the
> > > > > > > > > > +        image.
> > > > > > > > > > +
> > > > > > > > > > +        The details of how the windows are used are platform dependent. We note
> > > > > > > > > > +        that when there is more than one AF window, a typical implementation
> > > > > > > > > > +        might find the optimal focus position for each one and finally select
> > > > > > > > > > +        the window closest to the camera.
> > > > > > > > >
> > > > > > > > > What window is more close to the camera ? My understanding is that
> > > > > > > > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > > > > > > > they all sit on the same focal plan :)
> > > > > > >
> > > > > > > Sorry, by "the window closest to the camera" I mean "the window that
> > > > > > > is showing objects in the real world closest to the camera". I'll try
> > > > > > > to re-word that a bit.
> > > > > > >
> > > > > >
> > > > > > Yeah sorry, I could have realized that myself :)
> > > > > >
> > > > > > > > >
> > > > > > > > > > +
> > > > > > > > > > +        size: [n]
> > > > > > > > > > +
> > > > > > > > > > +  - AfTrigger:
> > > > > > > > > > +      type: int32_t
> > > > > > > > > > +      description: |
> > > > > > > > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > > > > > > > +        and can also be used to terminate a scan early.
> > > > > > > > > > +
> > > > > > > > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > > > > > > > +
> > > > > > > > > > +      enum:
> > > > > > > > > > +        - name: AfTriggerStart
> > > > > > > > > > +          value: 0
> > > > > > > > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > > > > > > > +        - name: AfTriggerCancel
> > > > > > > > > > +          value: 1
> > > > > > > > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > > > > > > > +            anywhere else. Ignored if no scan is in progress.
> > > > > > > > > > +
> > > > > > > > > > +  - AfPause:
> > > > > > > > > > +      type: int32_t
> > > > > > > > > > +      description: |
> > > > > > > > > > +        This control has no effect except when in continuous autofocus mode
> > > > > > > > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > > > > > > > +        (for example) images are captured. The algorithm remains inactive
> > > > > > > > > > +        until it is instructed to resume.
> > > > > > > > > > +
> > > > > > > > > > +      enum:
> > > > > > > > > > +        - name: AfPauseImmediate
> > > > > > > > > > +          value: 0
> > > > > > > > > > +          description: |
> > > > > > > > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > > > > > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > > > > > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > > > > > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > > > > > > > +            when it received this control.
> > > > > > > > > > +        - name: AfPauseDeferred
> > > > > > > > > > +          value: 1
> > > > > > > > > > +          description: |
> > > > > > > > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > > > > > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > > > > > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > > > > > > > +
> > > > > > > > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > > > > > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > > > > > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > > > > > > > +            AfPauseState will change to AfPauseStatePaused.
> > > > > > > > >
> > > > > > > > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > > > > > > > align it back ?
> > > > > > >
> > > > > > > Sure, I'll have a look!
> > > > > > >
> > > > > > > > >
> > > > > > > > > > +        - name: AfPauseResume
> > > > > > > > > > +          value: 2
> > > > > > > > > > +          description: |
> > > > > > > > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > > > > > > > +            from exactly where it left off, and AfPauseState will report
> > > > > > > > > > +            AfPauseStateRunning.
> > > > > > > > > > +
> > > > > > > > > > +  - LensPosition:
> > > > > > > > > > +      type: float
> > > > > > > > > > +      description: |
> > > > > > > > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > > > > > > > +        and also reports back the position of the lens for each frame.
> > > > > > > > > > +
> > > > > > > > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > > > > > > > +        AfModeManual.
> > > > > > > > > > +
> > > > > > > > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > > > > > > > +        values are permitted. For example:
> > > > > > > > > > +        0 moves the lens to infinity.
> > > > > > > > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > > > > > > > +        1 moves the lens to the hyperfocal position.
> > > > > > > > > > +        And larger values will focus the lens ever closer.
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > This seems good to me, albeit my understanding of the issue is limited :)
> > > > > > > > >
> > > > > > > >
> > > > > > > > Do we need a property to report the lens hyperfocal distance ?
> > > > > > >
> > > > > > > Yes, sometimes. Android distinguishes between calibrated and
> > > > > > > uncalibrated cameras. For a calibrated camera an application should be
> > > > > > > able to set the focal distance to (for example) 1m, and given the
> > > > > > > units I've proposed, you would need to know the hyperfocal distance to
> > > > > > > calculate the right number.
> > > > > >
> > > > > > Right, so we need a LensHyperfocalDistance or something for calibrated
> > > > > > lenses ?
> > > > >
> > > > > I think so.
> > > > >
> > > > > >
> > > > > > >
> > > > > > > For an uncalibrated camera I don't think it matters. Many applications
> > > > > > > would simply set the lens position to 1.0 (meaning hyperfocal) and
> > > > > > > you'll get the "best overall" focal distance (largest depth of field),
> > > > > > > and that's it.
> > > > > >
> > > > > > I think this part would be relevant to be captured in the control
> > > > > > description.
> > > > >
> > > > > Yes, I can add something about that.
> > > > >
> > > > > >
> > > > > > >
> > > > > > > There's also a need to know the allowable range of lens movement. This
> > > > > > > will depend on the type of module (knowing the lens driver isn't
> > > > > > > enough). But I don't see that applications need to know this, I think
> > > > > > > the pipeline handler and AF implementation should take care of it.
> > > > > > >
> > > > > >
> > > > > > I think so. When it comes to the actual lens movement range I don't
> > > > > > think it's relevant for applications. But do the actual lens movement
> > > > > > range maps to a range of LensPosition values that can be specified ? I
> > > > > > guess it's hard to map them on un-calibrated lenses, but for
> > > > > > calibrated ones a LensPositionRanges whose ControlInfo informs
> > > > > > application on the range's min and max might make sense ? (Please note
> > > > > > that this could also be used for the use case we discussed about
> > > > > > wanting to limit the lens range search scope)
> > > > >
> > > > > I think I would expect a calibrated lens would have to have some kind
> > > > > of property that says what the nearest and furthest focal distances
> > > > > are (the furthest usually being zero, meaning infinity). For an
> > > > > uncalibrated lens things will only ever be "approximately correct".
> > > >
> > > >
> > > > Care to add the property in next version, or can we do it on top with
> > > > a separate patch ?
> > > >
> > > > Thanks
> > > >    j
> > > >
> > > > >
> > > > > I could imagine this being used in conjunction with the hypothetical
> > > > > AfRangeCustom to construct custom search ranges for particular
> > > > > applications (in future, perhaps).
> > > > >
> > > > > Thanks again!
> > > > > David
> > > > >
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > > +  - AfState:
> > > > > > > > > > +      type: int32_t
> > > > > > > > > > +      description: |
> > > > > > > > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > > > > > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > > > > > > > +        The possible state changes are described below, though we note the following
> > > > > > > > > > +        state transitions that occur when the AfMode is changed.
> > > > > > > > > > +
> > > > > > > > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > > > > > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > > > > > > > +        LensPosition control must subsequently be used to achieve this.
> > > > > > > > > > +
> > > > > > > > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > > > > > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > > > > > > > >
> > > > > > > > > s/unless/until ?
> > > > > > > > > s/at the same time// ?
> > > > > > >
> > > > > > > I'm trying to describe what happens to the AfState when you send
> > > > > > > AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> > > > > > > AfTriggerStart). The only occasion when you won't see AfStateIdle will
> > > > > > > be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> > > > > > > think this reads OK to me, but I'll see if I can make the wording any
> > > > > > > clearer.
> > > > > >
> > > > > > I see. I didn't get it, sorry. If it's fine with you, keep this
> > > > > > version!
> > > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > > > +        case it will (start a scan and) report AfStateScanning.
> > > > > > > > > > +
> > > > > > > > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > > > > > > > +        report AfStateScanning.
> > > > > > > > > > +
> > > > > > > > > > +      enum:
> > > > > > > > > > +        - name: AfStateReset
> > > > > > > > > > +          value: 0
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > > > > +            scan was cancelled.
> > > > > > > > > > +        - name: AfStateScanning
> > > > > > > > > > +          value: 1
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > > > > > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > > > > > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > > > > > > > +            move back to AfStateReset or, if the scan actually completes
> > > > > > > > > > +            before the cancel request is processed, to one of
> > > > > > > > > > +            AfStateFocused or AfStateFailed.
> > > > > > > > > > +
> > > > > > > > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > > > > > > > >
> > > > > > > > > s/be be/is ?
> > > > > > >
> > > > > > > Yes, thanks!
> > > > > > >
> > > > > > > > >
> > > > > > > > > > +            at which point it may enter this state spontaneously whenever it
> > > > > > > > > > +            determines that a rescan is needed.
> > > > > > > > > > +        - name: AfStateFocused
> > > > > > > > > > +          value: 2
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > > > > > > > +            the image is now in focus.
> > > > > > > > > > +        - name: AfStateFailed
> > > > > > > > > > +          value: 3
> > > > > > > > > > +          description: |
> > > > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > > > > > > > +            find a good focus position.
> > > > > > > > > > +
> > > > > > > > > > +  - AfPauseState:
> > > > > > > > > > +      type: int32_t
> > > > > > > > > > +      description: |
> > > > > > > > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > > > > > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > > > > > > > +        as soon as any in-progress scan completes).
> > > > > > > > > > +
> > > > > > > > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > > > > > > > +
> > > > > > > > > > +      enum:
> > > > > > > > > > +        - name: AfPauseStateRunning
> > > > > > > > > > +          value: 0
> > > > > > > > > > +          description: |
> > > > > > > > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > > > > > > > +            spontaneously.
> > > > > > > > > > +        - name: AfPauseStatePausing
> > > > > > > > > > +          value: 1
> > > > > > > > > > +          description: |
> > > > > > > > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > > > > > > > +            as soon as any in-progress scan completes (and then report
> > > > > > > > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > > > > > > > +            the AfPauseResume control is sent.
> > > > > > > > > > +        - name: AfPauseStatePaused
> > > > > > > > > > +          value: 2
> > > > > > > > > > +          description: |
> > > > > > > > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > > > > > > > +            will occur until the AfPauseResume control is sent.
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > Very nice overall! With a few more clarifications I think this is good
> > > > > > > > > to go!
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >    j
> > > > > > >
> > > > > > > Thanks!
> > > > > > > David
> > > > > > >
> > > > > > > > >
> > > > > > > > > >    # ----------------------------------------------------------------------------
> > > > > > > > > >    # Draft controls section
> > > > > > > > > >
> > > > > > > > > > @@ -406,27 +664,6 @@ controls:
> > > > > > > > > >              The camera will cancel any active or completed metering sequence.
> > > > > > > > > >              The AE algorithm is reset to its initial state.
> > > > > > > > > >
> > > > > > > > > > -  - AfTrigger:
> > > > > > > > > > -      type: int32_t
> > > > > > > > > > -      draft: true
> > > > > > > > > > -      description: |
> > > > > > > > > > -       Control for AF trigger. Currently identical to
> > > > > > > > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > > > > > > > -
> > > > > > > > > > -        Whether the camera device will trigger autofocus for this request.
> > > > > > > > > > -      enum:
> > > > > > > > > > -        - name: AfTriggerIdle
> > > > > > > > > > -          value: 0
> > > > > > > > > > -          description: The trigger is idle.
> > > > > > > > > > -        - name: AfTriggerStart
> > > > > > > > > > -          value: 1
> > > > > > > > > > -          description: The AF routine is started by the camera.
> > > > > > > > > > -        - name: AfTriggerCancel
> > > > > > > > > > -          value: 2
> > > > > > > > > > -          description: |
> > > > > > > > > > -            The camera will cancel any active trigger and the AF routine is
> > > > > > > > > > -            reset to its initial state.
> > > > > > > > > > -
> > > > > > > > > >    - NoiseReductionMode:
> > > > > > > > > >        type: int32_t
> > > > > > > > > >        draft: true
> > > > > > > > > > @@ -507,45 +744,6 @@ controls:
> > > > > > > > > >              The AE algorithm has started a pre-capture metering session.
> > > > > > > > > >              \sa AePrecaptureTrigger
> > > > > > > > > >
> > > > > > > > > > -  - AfState:
> > > > > > > > > > -      type: int32_t
> > > > > > > > > > -      draft: true
> > > > > > > > > > -      description: |
> > > > > > > > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > > > > > > > -       ANDROID_CONTROL_AF_STATE.
> > > > > > > > > > -
> > > > > > > > > > -        Current state of the AF algorithm.
> > > > > > > > > > -      enum:
> > > > > > > > > > -        - name: AfStateInactive
> > > > > > > > > > -          value: 0
> > > > > > > > > > -          description: The AF algorithm is inactive.
> > > > > > > > > > -        - name: AfStatePassiveScan
> > > > > > > > > > -          value: 1
> > > > > > > > > > -          description: |
> > > > > > > > > > -            AF is performing a passive scan of the scene in continuous
> > > > > > > > > > -            auto-focus mode.
> > > > > > > > > > -        - name: AfStatePassiveFocused
> > > > > > > > > > -          value: 2
> > > > > > > > > > -          description: |
> > > > > > > > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > > > > > > > -        - name: AfStateActiveScan
> > > > > > > > > > -          value: 3
> > > > > > > > > > -          description: |
> > > > > > > > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > > > > > > > -            \sa AfTrigger
> > > > > > > > > > -        - name: AfStateFocusedLock
> > > > > > > > > > -          value: 4
> > > > > > > > > > -          description: |
> > > > > > > > > > -            AF believes has focused correctly and has locked focus.
> > > > > > > > > > -        - name: AfStateNotFocusedLock
> > > > > > > > > > -          value: 5
> > > > > > > > > > -          description: |
> > > > > > > > > > -            AF has not been able to focus and has locked.
> > > > > > > > > > -        - name: AfStatePassiveUnfocused
> > > > > > > > > > -          value: 6
> > > > > > > > > > -          description: |
> > > > > > > > > > -            AF has completed a passive scan without finding focus.
> > > > > > > > > > -
> > > > > > > > > >    - AwbState:
> > > > > > > > > >        type: int32_t
> > > > > > > > > >        draft: true
> > > > > > > > > > --
> > > > > > > > > > 2.30.2
> > > > > > > > > >

Hi everyone

Thanks for all the latest information. Can I perhaps consolidate the
discussion back to the top as I do get a bit lost in the multiple
replies...!

1. Units for the AF windows

I think most people like the idea of using the ScalerCrop region, or
the active pixel area. I think this is mostly fine (see below).

The only slight concern I have is what happens when the ScalerCrop is
updated (i.e. digital zoom).

Do we expect applications to recalculate the AF window coordinates
every time? This feels a bit annoying. If you don't update them then
earlier discussions suggest the PH will perform an intersection
calculation. For this to be sensible then our coordinate system
probably has to be *centred* in the image.

Or do we always use active pixel area units (ignore the ScalerCrop),
and the PH internally scales those numbers to the ScalerCrop (if it
has one)? Is this a more useful behaviour? The coordinate system
wouldn't have to be centred, and indeed ScalerCrop wouldn't need to be
mandatory either.

I don't think optical zoom is too much of an issue as it doesn't
change the size of the image from the sensor, right?

2. AF grids

We did discuss specifying AF windows as a grid. If the ISP actually
supports a grid then getting a list of windows and reverse engineering
a grid sounds annoying. We could have a control that some PHs expose
if they want, though pushing complexity to the application is
generally a bad thing. But I'm not sure what, if anything, we
concluded!

So not sure here.

3. Weights

It's not entirely clear to me how you would use per-window weights.
But I think we could leave these for now.

Anything else that I've forgotten?

Thanks!
David

On Fri, 29 Apr 2022 at 14:58, Hanlin Chen <hanlinchen@chromium.org> wrote:
>
> Hi Jacopo,
>
> On Fri, Apr 29, 2022 at 3:54 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
> >
> > Hi Han-Lin,
> >    thanks for the many details!
> >
> > On Thu, Apr 28, 2022 at 11:44:23PM +0800, Hanlin Chen wrote:
> > > On Thu, Apr 28, 2022 at 8:45 PM Hanlin Chen <hanlinchen@chromium.org> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > On Tue, Apr 26, 2022 at 9:16 PM Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > >
> > > > > Hi David,
> > > > >
> > > > > On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
> > > > > > Hi Jacopo
> > > > > >
> > > > > > Thanks for the message. Some replies and more questions below!
> > > > > >
> > > > > > On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > > > >
> > > > > > > Hi David,
> > > > > > >
> > > > > > > On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
> > > > > > > > Hi Jacopo
> > > > > > > >
> > > > > > > > Thanks for the quick reply!
> > > > > > > >
> > > > > > > > On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo@jmondi.org> wrote:
> > > > > > > > >
> > > > > > > > > Ah, forgot one thing!
> > > > > > > > >
> > > > > > > > > On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > > > > > > > > > Hi David,
> > > > > > > > > >   thanks for this new version
> > > > > > > > > >
> > > > > > > > > > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > > > > > > > > > This patch describes a series of controls that allow applications to
> > > > > > > > > > > drive AF algorithms:
> > > > > > > > > > >
> > > > > > > > > > > AfMode - manual, auto or continuous
> > > > > > > > > > > AfRange - full, macro or normal
> > > > > > > > > > > AfSpeed - fast or slowwith
> > > > > > > > > > > AfWindows - AF window locations
> > > > > > > > > > > AfTrigger - start (trigger) an AF scan or cancel
> > > > > > > > > > > AfPause - pause continuous AF
> > > > > > > > > > > LensPosition - set or retrieve position of lens
> > > > > > > > > > > AfState - reports whether scanning/success/failure
> > > > > > > > > > > AfPauseState - reports whether continuous AF paused or not
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > > > > > > > > > > ---
> > > > > > > > > > >  src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > > > > > > > > >  1 file changed, 258 insertions(+), 60 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > > > > > > > > index 9d4638ae..eccf18bc 100644
> > > > > > > > > > > --- a/src/libcamera/control_ids.yaml
> > > > > > > > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > > > > > > > @@ -381,6 +381,264 @@ controls:
> > > > > > > > > > >          \todo Define how the sensor timestamp has to be used in the reprocessing
> > > > > > > > > > >          use case.
> > > > > > > > > > >
> > > > > > > > > > > +  - AfMode:
> > > > > > > > > > > +      type: int32_t
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > > > > > > > > +        are allowed to set a new mode, and to send additional controls for
> > > > > > > > > > > +        that new mode, in the same request.
> > > > > > > > > > > +
> > > > > > > > > > > +        An implementation may choose not to implement all the modes.
> > > > > > > > > > > +
> > > > > > > > > > > +      enum:
> > > > > > > > > > > +        - name: AfModeManual
> > > > > > > > > > > +          value: 0
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The AF algorithm is in manual mode. In this mode it will never
> > > > > > > > > > > +            perform any action nor move the lens of its own accord, but an
> > > > > > > > > > > +            application can set controls to move the lens.
> > > > > > > > > > > +
> > > > > > > > > >
> > > > > > > > > > If, as I understand it, the lens can only be moved with 'LensPosition'
> > > > > > > > > > when in manual mode, I would say so explicitly
> > > > > > > > > >
> > > > > > > > > >                "... but an application can specify the desired lens
> > > > > > > > > >                position using the LensPosition control."
> > > > > > > >
> > > > > > > > Yes, will do.
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > +            In this mode the AfState will always report AfStateReset.
> > > > > > > > > >
> > > > > > > > > > I'm tempted to say AfState is useless and can not be reported at all
> > > > > > > > > > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > > > > > > > > > regardless, even if seeing 'reset' might be slightly confusing. I
> > > > > > > > > > don't have better alternatives to propose though.
> > > > > > > >
> > > > > > > > Don't really mind either. I think reporting stuff all the time might
> > > > > > > > be slightly easier, at least it's always obvious what to expect...
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Reading the AfStateReset description
> > > > > > > > > >
> > > > > > > > > >         - name: AfStateReset
> > > > > > > > > >           value: 0
> > > > > > > > > >           description: |
> > > > > > > > > >             The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > > > >             (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > > > >             scan was cancelled.
> > > > > > > > > >
> > > > > > > > > > I wonder if this isn't better named AfStateIdle
> > > > > > > >
> > > > > > > > Yes, also fine with that.
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > +        - name: AfModeAuto
> > > > > > > > > > > +          value: 1
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The AF algorithm is in auto mode. This means that the algorithm
> > > > > > > > > > > +            will never move the lens or change state unless the AfTrigger
> > > > > > > > > > > +            control is used. The AfTrigger control can be used to initiate a
> > > > > > > > > > > +            focus scan, the results of which will be reported by AfState.
> > > > > > > > > > > +
> > > > > > > > > > > +            If the autofocus algorithm is moved from AfModeAuto to another
> > > > > > > > > > > +            mode while a scan is in progress, the scan is cancelled
> > > > > > > > > > > +            immediately, without waiting for the scan to finish.
> > > > > > > > > > > +
> > > > > > > > > > > +            When first entering this mode the AfState will report
> > > > > > > > > > > +            AfStateReset. When a trigger control is sent, AfState will
> > > > > > > > > > > +            report AfStateScanning for a period before spontaneously
> > > > > > > > > > > +            changing to AfStateFocused or AfStateFailed, depending on
> > > > > > > > > > > +            the outcome of the scan. It will remain in this state until
> > > > > > > > > > > +            another scan is initiated by the AfTrigger control. If a scan is
> > > > > > > > > > > +            cancelled (without changing to another mode), AfState will return
> > > > > > > > > > > +            to AfStateReset.
> > > > > > > > > > > +        - name: AfModeContinuous
> > > > > > > > > > > +          value: 2
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The AF algorithm is in continuous mode. This means that the lens
> > > > > > > > > > > +            can re-start a scan spontaneously at any moment, without any user
> > > > > > > > > > > +            intervention. The AfState still reports whether the algorithm is
> > > > > > > > > > > +            currently scanning or not, though the application has no ability
> > > > > > > > > > > +            to initiate or cancel scans (though it can "pause" them), nor to
> > > > > > > > > > > +            move the lens for itself.
> > > > > > > > > > > +
> > > > > > > > > > > +            When set to AfModeContinuous, the system will immediately initiate
> > > > > > > > > > > +            a scan so AfState will report AfStateScanning, and will settle on
> > > > > > > > > > > +            one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > > > > > > > > > +
> > > > > > > > > > > +  - AfRange:
> > > > > > > > > > > +      type: int32_t
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        Control to set the range of focus distances that is scanned. An
> > > > > > > > > > > +        implementation may choose not to implement all the options here.
> > > > > > > > > > > +      enum:
> > > > > > > > > > > +        - name: AfRangeNormal
> > > > > > > > > > > +          value: 0
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            A wide range of focus distances is scanned, all the way from
> > > > > > > > > > > +            infinity down to close distances, though depending on the
> > > > > > > > > > > +            implementation, possibly not including the very closest macro
> > > > > > > > > > > +            positions.
> > > > > > > > > > > +        - name: AfRangeMacro
> > > > > > > > > > > +          value: 1
> > > > > > > > > > > +          description: Only close distances are scanned.
> > > > > > > > > > > +        - name: AfRangeFull
> > > > > > > > > > > +          value: 2
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The full range of focus distances is scanned just as with
> > > > > > > > > > > +            AfRangeNormal but this time including the very closest macro
> > > > > > > > > > > +            positions.
> > > > > > > > > > > +
> > > > > > > > > > > +  - AfSpeed:
> > > > > > > > > > > +      type: int32_t
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        Control that determines whether the AF algorithm is to move the lens
> > > > > > > > > > > +        as quickly as possible or more steadily. For example, during video
> > > > > > > > > > > +        recording it may be desirable not to move the lens too abruptly, but
> > > > > > > > > > > +        when in a preview mode (waiting for a still capture) it may be
> > > > > > > > > > > +        helpful to move the lens as quickly as is reasonably possible.
> > > > > > > > > > > +      enum:
> > > > > > > > > > > +        - name: AfSpeedNormal
> > > > > > > > > > > +          value: 0
> > > > > > > > > > > +          description: Move the lens at its usual speed.
> > > > > > > > > > > +        - name: AfSpeedFast
> > > > > > > > > > > +          value: 1
> > > > > > > > > > > +          description: Move the lens more quickly.
> > > > > > > > > > > +
> > > > > > > > > > > +  - AfWindows:
> > > > > > > > > > > +      type: Rectangle
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        Sets the focus windows used by the AF algorithm. The units used express
> > > > > > > > > > > +        a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > > > > > > > > +        image), as u0.16 format numbers.
> > > > > > > > > >
> > > > > > > > > > I don't like too much the "or if unavailable", as it means that the
> > > > > > > > > > reference rectangle will change depending on the platform
> > > > > > > > > > capabilities.
> > > > > > > >
> > > > > > > > I guess what I want is (like in the case of Android below), to specify
> > > > > > > > AF regions relative to the actual output image. If a platform supports
> > > > > > > > digital zoom (through ScalerCrop, I assume), the regions must be
> > > > > > > > relative to that part of the image. If there is no ScalerCrop, then do
> > > > > > > > we assume the platform doesn't support digital zoom? But the basic
> > > > > > > > idea is the same, the numbers supplied should be relative to the
> > > > > > > > output image in this case too.
> > > > > > > >
> > > > > > >
> > > > > > > Thing is that there's no "output image size" as two streams produced
> > > > > > > from the same camera might have different output sizes ?
> > > > > >
> > > > > > You're right that there might be several outputs. So what do I really
> > > > > > mean? I'm sort of looking for the rectangle in the sensor image that
> > > > > > is required to make all the outputs, but I think the ScalerCrop
> > > > > > encapsulates that idea reasonably well. Where there isn't a
> > > > > > ScalerCrop,  I think we use the active pixel area instead.
> > > > > >
> > > > >
> > > > > Gotcha! Indeed scaler crop defines the area of the sensor's output
> > > > > frame used to generate all streams... Thanks for sticking to it and
> > > > > help me straight my thoughts.
> > > > >
> > > > > I still don't like the "if not available part" :(
> > > > >
> > > > > I'll try to rope in others to get their opinions. What concerns me is
> > > > > that applications will behave differently on platforms where scaler
> > > > > crop is available compared to ones where it is not. Maybe I'm just
> > > > > over-concerned ?
> > > > >
> > > > >
> > > > > > >
> > > > > > > > Is there a better way to explain this, do you think?
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I understand your argument in reply to Jean-Michel comment on the
> > > > > > > > > > previous version of the series (quoted here for reference)
> > > > > > > > > >
> > > > > > > > > > "The thing that bothers me a bit about the full pixel array is that, if
> > > > > > > > > > you use digital zoom, then you will have to update your AfWindows as
> > > > > > > > > > they might otherwise lie outside your final image! If it's defined as
> > > > > > > > > > being a region of the scaler crop, then it automatically stays the
> > > > > > > > > > same (to the application, at least, the pipeline handler probably has
> > > > > > > > > > some work to do).
> > > > > > > > > >
> > > > > > > > > > I note that the Android link you give below says "Focus areas are
> > > > > > > > > > relative to the current field of view (getZoom())", which I guess is
> > > > > > > > > > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > > > > > > > > > is easier to work with than the u0.16 format that I specified?"
> > > > > > > > > >
> > > > > > > > > > Let's reason on the IPA point of view. If the AF algorithm works by
> > > > > > > > > > estimating the contrast, will it do so before or after the ScalerCrop
> > > > > > > > > > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > > > > > > > > > do so on the full pixel array size, the portion of if that gets
> > > > > > > > > > processed or on the final image ?
> > > > > > > >
> > > > > > > > In the case of PDAF, we can presumably use information from whatever
> > > > > > > > part of the image that the sensor is outputting.
> > > > > > > >
> > > > > > > > For CDAF (contrast detect AF) I guess it's possible that focus
> > > > > > > > statistics could be computed either before or after any cropping is
> > > > > > > > done in the ISP. (Maybe there are some platforms where cropping could
> > > > > > > > be delegated to the sensor?) So I'd guess that using any focus
> > > > > > > > statistics from outside the final FoV may actually be impossible,
> > > > > > > > depending on the platform. But even where you can do this, I don't
> > > > > > > > believe it's what anyone would want (and Android seems to behave like
> > > > > > > > this).
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > > > > > > > > > rectangle, so that they are applied to the actual FOV before being
> > > > > > > > > > passed to the AF algorithm ?
> > > > > > > >
> > > > > > > > Not quite sure what you meant here. I'd expect the ISP to generate
> > > > > > > > focus statistics from the raw Bayer pixels that it gets, without too
> > > > > > > > much processing applied to them. So the job will be to take rectangles
> > > > > > > > defined in the final FoV and translate them into the image coming
> > > > > > > > directly from the sensor.
> > > > > > > >
> > > > > > >
> > > > > > > What I meant is: if we specify the AfWindows relative to the full
> > > > > > > pixel array, can't we then re-scale it to final cropped output ? or if
> > > > > > > no copping on the sensor's output frame size.
> > > > > > >
> > > > > > > This would be similar to what you do with ScalerCrop already ?
> > > > > > >
> > > > > > >         void RPiCameraData::applyScalerCrop(const ControlList &controls) {
> > > > > > >
> > > > > > >                 Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
> > > > > > >
> > > > > > >                 /* Create a version of the crop scaled to ISP (camera mode) pixels. */
> > > > > > >                 Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
> > > > > > >                 ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
> > > > > > >         }
> > > > > > >
> > > > > > > But I guess I'm reasoning in absolute coordinates, not in proportions
> > > > > > > like you have proposed here.
> > > > > >
> > > > > > I think the question here is whether to use some kind of fractional
> > > > > > coordinates for describing the part of the ScalerCrop that we want AF
> > > > > > to use, or if we simply use pixel coordinates within the ScalerCrop
> > > > > > (or active pixel array), which would then look rather like the code
> > > > > > above.
> > > > > >
> > > > > > I suppose I don't particularly mind. If we use actual pixel
> > > > > > coordinates it's maybe a bit less clear that some other scaling
> > > > > > (shrinking to lie within the ScalerCrop) might be necessary, but if
> > > > > > we're happy that this is going to happen anyway then I think that
> > > > > > would work for me.
> > > > > >
> > > > >
> > > > > Let's hear from others as well!
> > > >
> > > > I think the focusAreas() is deprecated on Android.
> > > > https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas()
> > > > Currently it should be AF_REGIONS.
> > > > https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AF_REGIONS
> > > >
> > > > It uses the pixel array as the coordinate system of both ScalarCrop
> > > > and AFWindows.
> > > > For the problem that the AFWindow may be outside of the ScalarCrop
> > > > region, it adds an intersection rule:
> > > > Only the intersection of AFWindows and ScalarCrop takes effect.
> > > >
> > > > To be honest, the intersection rule is a little weird. I think
> > > > limiting the AFWindow to ScalarCrop looks good to me. The Android
> > > > adapter can do the intersection.
> > > > On the other hand, I think the benefit of using pixel arrays as
> > > > coordinate is that the precision can be at pixel level.
> > > > Not sure if it matters for any other algorithms.
> > > >
> >
> >
> > Let me summarize what I read there:
> >
> > - For devices not supporting android.distortionCorrection.mode
> >   control, the coordinate system always follows that of
> >   android.sensor.info.activeArraySize
> >
> > - For devices supporting android.distortionCorrection.mode control:
> >   - When the distortion correction mode is OFF, the coordinate system
> >     follows android.sensor.info.preCorrectionActiveArraySize
> >   - When the distortion correction mode is not OFF, the coordinate
> >     system follows android.sensor.info.activeArraySize
> >
> > - If the metering region is outside the used android.scaler.cropRegion
> >   returned in capture result metadata, the camera device will ignore
> >   the sections outside the crop region and output only the
> >   intersection rectangle as the metering region in the result
> >   metadata. If the region is entirely outside the crop region, it will
> >   be ignored and not reported in the result metadata.
> >
> > However with the introduction of android.control.zoomRatio (API level
> > 30), now android.sensor.info.activeArraySize and
> > android.sensor.info.preCorrectionActiveArraySize represent
> > "post-zoomRatio field of view, not pre-zoom field of view."
> >
> > Which implies "The afRegions coordinates are relative to the
> > activeArray/preCorrectionActiveArray representing the zoomed field of
> > view."
> >
> > Now, we don't have anything like zoomRatio, but this change suggests
> > that the reference system is now actually the post-zoom field of view,
> > which for us corresponds to the ScalerCrop rectangle.
> It's still the activeArray coordinate system.
> >
> > Could someone confim my understanding here ?
>
> In my understanding the zoomRatio is defined for optical zoom, but the
> digital zoom is still by cropRegion. See:
> https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_ZOOM_RATIO
>
> The post-zoom field of view here means the field zoomed by an optical lens.
> Because it's set by lens, even if it's zoomed, the frames from the
> sensor are still of the rectangle (0, 0, activeArrayWidth,
> activeArrayHeight).
> The ScalarCrop and afRegions are still defined in the rectangle, i.e.,
> the pixel array coordinate system.
>
> I quote an example from the link:
> Assuming the pixel array is (0, 0, 2000, 1500). The application can
> achieve 2.0x zoom in one of two ways:
> 1. zoomRatio = 2.0, scaler.cropRegion = (0, 0, 2000, 1500) // By optical zoom
> 2. zoomRatio = 1.0 (default), scaler.cropRegion = (500, 375, 1500,
> 1125) // By digital zoom
>
> If the application intends to set aeRegions to be top-left quarter of
> the viewfinder field-of-view, the android.control.aeRegions should be
> set to (0, 0, 1000, 750) with zoomRatio set to 2.0. Alternatively, the
> application can set aeRegions to the equivalent region of (500, 375,
> 1000, 750) for zoomRatio of 1.0. If the application doesn't explicitly
> set android.control.zoomRatio, its value defaults to 1.0.
>
> As above, the aeRegion is always in pixel array coordinate
> before/after API 30, no matter how zoomRatio is set, but the
> application needs to take zoomRation into consideration.
> I think they are defined like this to remain the coordinate behavior
> for an legacy application that isn't aware of the new zoomRation API,
> and the framework default it to 1.0.
>
> However, since the ScalarCrop is relative to pixelArray, no matter
> which one afRegion is relative to, they are interchangeable.
> We don't need to stick to that of Android.
> >
> > > > Another thing I'd like to mention is a rule for ScalarCrop in Android, I quote:
> > > > "The crop region is applied after the RAW to other color space (e.g.
> > > > YUV) conversion. Since raw streams (e.g. RAW16) don't have the
> > > > conversion stage, they are not croppable. The crop region will be
> > > > ignored by raw streams."
> > > >
> > > > Since the statistics is usually generated from the RAW, the input for
> > > > IPA would be the full pixel array's statistics.
> > > > In result, the IPA needs to translate the coordinate to match the
> > > > pixel array anyway.
> > > > It might be the reason that both ScalarCrop and AFWindows are in the
> > > > pixel array coordinate, since they will be in the end.
> > > >
> > > > This is also a weird rule, because it somehow forbids the device to
> > > > crop on sensor.
> > > >
> > > > >
> > > > > > >
> > > > > > > > > >
> > > > > > > > > > One last point about the unit: is the below representation correct,
> > > > > > > > > > when it comes to use a Rectangle to represent a proportion ?
> > > > > > > > > > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > > > > > > > > > pipeline report 12 rectangles like the ones below, for applications to
> > > > > > > > > > chose from ?
> > > > > > > > > >
> > > > > > > > > >       (0,0)
> > > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > > >         |     |     |     |     |
> > > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > > >         |     |     |     |     |
> > > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > > >         |     |     |     |     |
> > > > > > > > > >         +-----+-----+-----+-----+
> > > > > > > > > >
> > > > > > > > > > AfWindows = {
> > > > > > > > > >     {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > > > > > > > > >     {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > > > > > > > > >     {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > > My understanding is that the grid is fixed and depends on the platform
> > > > > > > > > > capabilities. I wonder if we shouldn't allow applications to select
> > > > > > > > > > grids by index.
> > > > > > > >
> > > > > > > > I think there is a problem about determining the AF capabilities of a
> > > > > > > > platform. I could imagine some platforms will only accept a regular
> > > > > > > > grid of regions, but others might let you put them in arbitrary
> > > > > > > > places. Or a mixture of both. To be honest, I'm not sure how we handle
> > > > > > > > that sort of thing. Anyone have any good ideas?
> > > > > > > >
> > > > > > >
> > > > > > > Exactly, that's why having proportions as the ones I tried to sketch
> > > > > > > out above doesn't really fit as a model in my head. Each platform
> > > > > > > would have its own 'grid', do we need to expose it to applications ?
> > > > > > >
> > > > > > > Can't we let application specify Rectangles with absolute coordinates
> > > > > > > (relative to PixelArray or else) and let the IPA/PH map those areas on
> > > > > > > their grids ?
> > > > > >
> > > > > > I really don't know how to deal with this. Some ISPs will have grids
> > > > > > of some sort, some may have windows in arbitrary locations, different
> > > > > > numbers and sizes of each depending on the platform, and some may do
> > > > > > both. Typically I would imagine many applications wanting to use grids
> > > > > > much of the time, but perhaps arbitrary rectangles for following
> > > > > > objects, people and faces.
> > > >
> > > > It's actually happening. GCam is using its own face detection
> > > > algorithm and passes the region to HAL.
> > > > ChromeOS is also integrating face algorithms to choose ROI for 3A.
> > > > Although they are for AE mostly, I think the windows for 3A should
> > > > have the same coordinate system.
> > > > I guess (I'm not sure) this is why the grid based focusAreas() is
> > > > deprecated on Android, and moves to pixel array coordinates, since the
> > > > face algorithm usually works on pixel coordinates. I think the grid of
> > > > ISPs could be left as implementation details for the pipeline handler?
> > > >
> > > I was woolly-minded. focusAreas() is not grid based, please ignore the
> > > sentence :P.
> > > Another thing is that AF_CROP_REGION allows windows to be overlapped
> > > and with weight.
> > > I'm not sure if we'd like to allow that. Perhaps this is discussed
> > > already and I missed it?
> >
> > I think it was briefly mentioned, but no conclusion were taken.
> >
> > > If there are ML algorithms to recognize the objects in a scene, the
> > > rectangle could be partially overlapped naturally, and may be weighted
> > > by objects, for example face within leaves is more weighted even if
> > > it's not the closet object in the scene. Usually the ML algorithm is
> > > located in the application and outside of HAL.
> > > I guess it's something we can decide later.
> >
> > I agree.
> >
> > I think what's pressing to decide, for David to be able to send a new
> > series is:
> >
> > - Define the AfRegions unit
> >
> >   There seems to be consensus on using pixel coordinates ?
> >
> > - Define the AfRegions reference rectangle
> >
> >   There are multiple reasons to prefer using the ScalerCrop rectangle.
> >   If my understanding is correct, Android has moved to use the same
> >   post-zoom reference since the introduction of zoomRatio (see above).
> >
> >   However I'm still concerned about the fact ScalerCrop might not be
> >   there and depending on its availability, the reference system might
> >   change between different platforms.
> >
> >   There might be way out though. Why don't we define ScalerCrop as a
> >   mandatory metadata, which is always returned even if the platform
> >   does not support digital zoom ? If that's the case, ScalerCrop will always
> >   be equal to the active pixel array. With this, we can say here that
> >   the reference system is the ScalerCrop rectangle as returned in the
> >   metadata.
> >
> >   What do you think ?
>
> I agree.
>
> >
> > - I think we can for now postpone discussing about weights.
> >   Conceptually it won't be hard to introduce them if we adopt
> >   something similar to the weighted rectangle android has.
> >
> > David, are there still points that needs to be clarified to get to a
> > new version ? Is there anything we can help with ?
> >
> > Thanks
> >    j
> >
> > >
> > > > >
> > > > > Indeed. This is an advanced feature for which it might be expected
> > > > > application to know what the platform can provide maybe ? Describing
> > > > > all possible options seems very complex, and we'll never be able to
> > > > > express all of them.
> > > > >
> > > > > I would be fine stabilizing on a simpler grid-based version of AfWindows
> > > > > for now and improve on top, maybe ?
> > > > >
> > > > > >
> > > > > > Perhaps having an AfWindowGrid control and an AfWindows array control
> > > > > > of Rectangles could work? For the former, perhaps we leave the
> > > > > > decisions on the size of the grid to the platform/algorithm - I'd be
> > > > > > worried about having so many parameters that it would be difficult to
> > > > > > use. (You'd have to think about the size of the grid, the spacing of
> > > > > > rectangles in the grid etc... but perhaps Android requires us to
> > > > > > expose this sort of thing?) Perhaps someone knows what Android would
> > > > > > require here?
> > > > > >
> > > > > > >
> > > > > > > > > >
> > > > > > > > > > Also, do we need a property to report the grid structure and allows
> > > > > > > > > > application to select them from, or do you see this conveyed through a
> > > > > > > > > > ControlInfo ?
> > > > > > > >
> > > > > > > > I wasn't quite sure what the question was here. Are you talking about
> > > > > > > > selecting the configuration for the AfWindows, or selecting focus
> > > > > > > > statistics from particular regions? Perhaps an example would help me
> > > > > > > > here!
> > > > > > > >
> > > > > > >
> > > > > > > I'm talking about how would an application know how many spots it can
> > > > > > > select from, in other words, how do we let an application know that,
> > > > > > > looking at the above example, each grid is 1/4 in width and 1/3 in
> > > > > > > height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
> > > > > > > 4x3 one.
> > > > > > >
> > > > > > > Or maybe I got it all wrong, and you where implying already that the
> > > > > > > here specified Rectangles cover arbitrary parts of the image and it's
> > > > > > > the IPA that has to match them on their grids.
> > > > > >
> > > > > > Or maybe we treat grids as a whole different thing? (as per the
> > > > > > comments just above)
> > > > > >
> > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > +
> > > > > > > > > > > +        In order to be activated, a rectangle must be programmed with non-zero
> > > > > > > > > > > +        width and height. If no rectangles are programmed in this way, then the
> > > > > > > > > > > +        system will choose its own single default window in the centre of the
> > > > > > > > > > > +        image.
> > > > > > > > > > > +
> > > > > > > > > > > +        The details of how the windows are used are platform dependent. We note
> > > > > > > > > > > +        that when there is more than one AF window, a typical implementation
> > > > > > > > > > > +        might find the optimal focus position for each one and finally select
> > > > > > > > > > > +        the window closest to the camera.
> > > > > > > > > >
> > > > > > > > > > What window is more close to the camera ? My understanding is that
> > > > > > > > > > windows apply to the sensor produced frame, hence, when it comes to distance,
> > > > > > > > > > they all sit on the same focal plan :)
> > > > > > > >
> > > > > > > > Sorry, by "the window closest to the camera" I mean "the window that
> > > > > > > > is showing objects in the real world closest to the camera". I'll try
> > > > > > > > to re-word that a bit.
> > > > > > > >
> > > > > > >
> > > > > > > Yeah sorry, I could have realized that myself :)
> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > +
> > > > > > > > > > > +        size: [n]
> > > > > > > > > > > +
> > > > > > > > > > > +  - AfTrigger:
> > > > > > > > > > > +      type: int32_t
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > > > > > > > > +        and can also be used to terminate a scan early.
> > > > > > > > > > > +
> > > > > > > > > > > +        It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > > > > > > > > +
> > > > > > > > > > > +      enum:
> > > > > > > > > > > +        - name: AfTriggerStart
> > > > > > > > > > > +          value: 0
> > > > > > > > > > > +          description: Start an AF scan. Ignored if a scan is in progress.
> > > > > > > > > > > +        - name: AfTriggerCancel
> > > > > > > > > > > +          value: 1
> > > > > > > > > > > +          description: Cancel an AF scan. This does not cause the lens to move
> > > > > > > > > > > +            anywhere else. Ignored if no scan is in progress.
> > > > > > > > > > > +
> > > > > > > > > > > +  - AfPause:
> > > > > > > > > > > +      type: int32_t
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        This control has no effect except when in continuous autofocus mode
> > > > > > > > > > > +        (AfModeContinuous). It can be used to pause any lens movements while
> > > > > > > > > > > +        (for example) images are captured. The algorithm remains inactive
> > > > > > > > > > > +        until it is instructed to resume.
> > > > > > > > > > > +
> > > > > > > > > > > +      enum:
> > > > > > > > > > > +        - name: AfPauseImmediate
> > > > > > > > > > > +          value: 0
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            Pause the continuous autofocus algorithm immediately, whether or
> > > > > > > > > > > +            not any kind of scan is underway. AfPauseState will subsequently
> > > > > > > > > > > +            report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > > > > > > > > > +            AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > > > > > > > > > +            when it received this control.
> > > > > > > > > > > +        - name: AfPauseDeferred
> > > > > > > > > > > +          value: 1
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            This is similar to AfPauseImmediate, and if the AfState is currently
> > > > > > > > > > > +            reporting AfStateFocused or AfStateFailed it will remain in that
> > > > > > > > > > > +            state and AfPauseState will report AfPauseStatePaused.
> > > > > > > > > > > +
> > > > > > > > > > > +            However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > > > > > > > > > +            will report AfPauseStatePausing until the scan is finished, at which
> > > > > > > > > > > +            point AfState will report one of AfStateFocused or AfStateFailed, and
> > > > > > > > > > > +            AfPauseState will change to AfPauseStatePaused.
> > > > > > > > > >
> > > > > > > > > > From here on I see lines getting over 80 cols by a few chars. Can we
> > > > > > > > > > align it back ?
> > > > > > > >
> > > > > > > > Sure, I'll have a look!
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > +        - name: AfPauseResume
> > > > > > > > > > > +          value: 2
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            Resume continuous autofocus operation. The algorithm starts again
> > > > > > > > > > > +            from exactly where it left off, and AfPauseState will report
> > > > > > > > > > > +            AfPauseStateRunning.
> > > > > > > > > > > +
> > > > > > > > > > > +  - LensPosition:
> > > > > > > > > > > +      type: float
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        Acts as a control to instruct the lens to move to a particular position
> > > > > > > > > > > +        and also reports back the position of the lens for each frame.
> > > > > > > > > > > +
> > > > > > > > > > > +        The LensPosition control is ignored unless the AfMode is set to
> > > > > > > > > > > +        AfModeManual.
> > > > > > > > > > > +
> > > > > > > > > > > +        The units are dioptres divided by the hyperfocal distance. Non-integer
> > > > > > > > > > > +        values are permitted. For example:
> > > > > > > > > > > +        0 moves the lens to infinity.
> > > > > > > > > > > +        0.5 moves the lens to twice the hyperfocal distance.
> > > > > > > > > > > +        1 moves the lens to the hyperfocal position.
> > > > > > > > > > > +        And larger values will focus the lens ever closer.
> > > > > > > > > > > +
> > > > > > > > > >
> > > > > > > > > > This seems good to me, albeit my understanding of the issue is limited :)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Do we need a property to report the lens hyperfocal distance ?
> > > > > > > >
> > > > > > > > Yes, sometimes. Android distinguishes between calibrated and
> > > > > > > > uncalibrated cameras. For a calibrated camera an application should be
> > > > > > > > able to set the focal distance to (for example) 1m, and given the
> > > > > > > > units I've proposed, you would need to know the hyperfocal distance to
> > > > > > > > calculate the right number.
> > > > > > >
> > > > > > > Right, so we need a LensHyperfocalDistance or something for calibrated
> > > > > > > lenses ?
> > > > > >
> > > > > > I think so.
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > For an uncalibrated camera I don't think it matters. Many applications
> > > > > > > > would simply set the lens position to 1.0 (meaning hyperfocal) and
> > > > > > > > you'll get the "best overall" focal distance (largest depth of field),
> > > > > > > > and that's it.
> > > > > > >
> > > > > > > I think this part would be relevant to be captured in the control
> > > > > > > description.
> > > > > >
> > > > > > Yes, I can add something about that.
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > There's also a need to know the allowable range of lens movement. This
> > > > > > > > will depend on the type of module (knowing the lens driver isn't
> > > > > > > > enough). But I don't see that applications need to know this, I think
> > > > > > > > the pipeline handler and AF implementation should take care of it.
> > > > > > > >
> > > > > > >
> > > > > > > I think so. When it comes to the actual lens movement range I don't
> > > > > > > think it's relevant for applications. But do the actual lens movement
> > > > > > > range maps to a range of LensPosition values that can be specified ? I
> > > > > > > guess it's hard to map them on un-calibrated lenses, but for
> > > > > > > calibrated ones a LensPositionRanges whose ControlInfo informs
> > > > > > > application on the range's min and max might make sense ? (Please note
> > > > > > > that this could also be used for the use case we discussed about
> > > > > > > wanting to limit the lens range search scope)
> > > > > >
> > > > > > I think I would expect a calibrated lens would have to have some kind
> > > > > > of property that says what the nearest and furthest focal distances
> > > > > > are (the furthest usually being zero, meaning infinity). For an
> > > > > > uncalibrated lens things will only ever be "approximately correct".
> > > > >
> > > > >
> > > > > Care to add the property in next version, or can we do it on top with
> > > > > a separate patch ?
> > > > >
> > > > > Thanks
> > > > >    j
> > > > >
> > > > > >
> > > > > > I could imagine this being used in conjunction with the hypothetical
> > > > > > AfRangeCustom to construct custom search ranges for particular
> > > > > > applications (in future, perhaps).
> > > > > >
> > > > > > Thanks again!
> > > > > > David
> > > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > +  - AfState:
> > > > > > > > > > > +      type: int32_t
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        Reports the current state of the AF algorithm in conjunction with the
> > > > > > > > > > > +        reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > > > > > > > > > +        The possible state changes are described below, though we note the following
> > > > > > > > > > > +        state transitions that occur when the AfMode is changed.
> > > > > > > > > > > +
> > > > > > > > > > > +        If the AfMode is set to AfModeManual then the AfState will report
> > > > > > > > > > > +        AfStateReset. This action on its own does not cause the lens to move; the
> > > > > > > > > > > +        LensPosition control must subsequently be used to achieve this.
> > > > > > > > > > > +
> > > > > > > > > > > +        If the AfMode is set to AfModeAuto then the AfState will report
> > > > > > > > > > > +        AfStateReset, unless AfTriggerStart is sent at the same time in which
> > > > > > > > > >
> > > > > > > > > > s/unless/until ?
> > > > > > > > > > s/at the same time// ?
> > > > > > > >
> > > > > > > > I'm trying to describe what happens to the AfState when you send
> > > > > > > > AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
> > > > > > > > AfTriggerStart). The only occasion when you won't see AfStateIdle will
> > > > > > > > be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
> > > > > > > > think this reads OK to me, but I'll see if I can make the wording any
> > > > > > > > clearer.
> > > > > > >
> > > > > > > I see. I didn't get it, sorry. If it's fine with you, keep this
> > > > > > > version!
> > > > > > >
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > +        case it will (start a scan and) report AfStateScanning.
> > > > > > > > > > > +
> > > > > > > > > > > +        If the AfMode is set to AfModeContinuous then the AfState will initially
> > > > > > > > > > > +        report AfStateScanning.
> > > > > > > > > > > +
> > > > > > > > > > > +      enum:
> > > > > > > > > > > +        - name: AfStateReset
> > > > > > > > > > > +          value: 0
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > > > > > > > > > +            (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > > > > > > > > > +            scan was cancelled.
> > > > > > > > > > > +        - name: AfStateScanning
> > > > > > > > > > > +          value: 1
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > > > > > > > > +            started using the AfTrigger control. The scan can be cancelled by
> > > > > > > > > > > +            sending AfTriggerCancel at which point the algorithm will either
> > > > > > > > > > > +            move back to AfStateReset or, if the scan actually completes
> > > > > > > > > > > +            before the cancel request is processed, to one of
> > > > > > > > > > > +            AfStateFocused or AfStateFailed.
> > > > > > > > > > > +
> > > > > > > > > > > +            Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> > > > > > > > > >
> > > > > > > > > > s/be be/is ?
> > > > > > > >
> > > > > > > > Yes, thanks!
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > +            at which point it may enter this state spontaneously whenever it
> > > > > > > > > > > +            determines that a rescan is needed.
> > > > > > > > > > > +        - name: AfStateFocused
> > > > > > > > > > > +          value: 2
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > > > > +            mode and a scan has completed with the result that the algorithm believes
> > > > > > > > > > > +            the image is now in focus.
> > > > > > > > > > > +        - name: AfStateFailed
> > > > > > > > > > > +          value: 3
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > > > > > > > > > +            mode and a scan has completed with the result that the algorithm did not
> > > > > > > > > > > +            find a good focus position.
> > > > > > > > > > > +
> > > > > > > > > > > +  - AfPauseState:
> > > > > > > > > > > +      type: int32_t
> > > > > > > > > > > +      description: |
> > > > > > > > > > > +        Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > > > > > > > > > +        the algorithm is currently running, paused or pausing (that is, will pause
> > > > > > > > > > > +        as soon as any in-progress scan completes).
> > > > > > > > > > > +
> > > > > > > > > > > +        Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > > > > > > > > > +
> > > > > > > > > > > +      enum:
> > > > > > > > > > > +        - name: AfPauseStateRunning
> > > > > > > > > > > +          value: 0
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            Continuous AF is running and the algorithm may restart a scan
> > > > > > > > > > > +            spontaneously.
> > > > > > > > > > > +        - name: AfPauseStatePausing
> > > > > > > > > > > +          value: 1
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > > > > > > > > > +            as soon as any in-progress scan completes (and then report
> > > > > > > > > > > +            AfPauseStatePaused). No new scans will be start spontaneously until
> > > > > > > > > > > +            the AfPauseResume control is sent.
> > > > > > > > > > > +        - name: AfPauseStatePaused
> > > > > > > > > > > +          value: 2
> > > > > > > > > > > +          description: |
> > > > > > > > > > > +            Continuous AF is paused. No further state changes or lens movements
> > > > > > > > > > > +            will occur until the AfPauseResume control is sent.
> > > > > > > > > > > +
> > > > > > > > > >
> > > > > > > > > > Very nice overall! With a few more clarifications I think this is good
> > > > > > > > > > to go!
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >    j
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > David
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >    # ----------------------------------------------------------------------------
> > > > > > > > > > >    # Draft controls section
> > > > > > > > > > >
> > > > > > > > > > > @@ -406,27 +664,6 @@ controls:
> > > > > > > > > > >              The camera will cancel any active or completed metering sequence.
> > > > > > > > > > >              The AE algorithm is reset to its initial state.
> > > > > > > > > > >
> > > > > > > > > > > -  - AfTrigger:
> > > > > > > > > > > -      type: int32_t
> > > > > > > > > > > -      draft: true
> > > > > > > > > > > -      description: |
> > > > > > > > > > > -       Control for AF trigger. Currently identical to
> > > > > > > > > > > -       ANDROID_CONTROL_AF_TRIGGER.
> > > > > > > > > > > -
> > > > > > > > > > > -        Whether the camera device will trigger autofocus for this request.
> > > > > > > > > > > -      enum:
> > > > > > > > > > > -        - name: AfTriggerIdle
> > > > > > > > > > > -          value: 0
> > > > > > > > > > > -          description: The trigger is idle.
> > > > > > > > > > > -        - name: AfTriggerStart
> > > > > > > > > > > -          value: 1
> > > > > > > > > > > -          description: The AF routine is started by the camera.
> > > > > > > > > > > -        - name: AfTriggerCancel
> > > > > > > > > > > -          value: 2
> > > > > > > > > > > -          description: |
> > > > > > > > > > > -            The camera will cancel any active trigger and the AF routine is
> > > > > > > > > > > -            reset to its initial state.
> > > > > > > > > > > -
> > > > > > > > > > >    - NoiseReductionMode:
> > > > > > > > > > >        type: int32_t
> > > > > > > > > > >        draft: true
> > > > > > > > > > > @@ -507,45 +744,6 @@ controls:
> > > > > > > > > > >              The AE algorithm has started a pre-capture metering session.
> > > > > > > > > > >              \sa AePrecaptureTrigger
> > > > > > > > > > >
> > > > > > > > > > > -  - AfState:
> > > > > > > > > > > -      type: int32_t
> > > > > > > > > > > -      draft: true
> > > > > > > > > > > -      description: |
> > > > > > > > > > > -       Control to report the current AF algorithm state. Currently identical to
> > > > > > > > > > > -       ANDROID_CONTROL_AF_STATE.
> > > > > > > > > > > -
> > > > > > > > > > > -        Current state of the AF algorithm.
> > > > > > > > > > > -      enum:
> > > > > > > > > > > -        - name: AfStateInactive
> > > > > > > > > > > -          value: 0
> > > > > > > > > > > -          description: The AF algorithm is inactive.
> > > > > > > > > > > -        - name: AfStatePassiveScan
> > > > > > > > > > > -          value: 1
> > > > > > > > > > > -          description: |
> > > > > > > > > > > -            AF is performing a passive scan of the scene in continuous
> > > > > > > > > > > -            auto-focus mode.
> > > > > > > > > > > -        - name: AfStatePassiveFocused
> > > > > > > > > > > -          value: 2
> > > > > > > > > > > -          description: |
> > > > > > > > > > > -            AF believes the scene is in focus, but might restart scanning.
> > > > > > > > > > > -        - name: AfStateActiveScan
> > > > > > > > > > > -          value: 3
> > > > > > > > > > > -          description: |
> > > > > > > > > > > -            AF is performing a scan triggered by an AF trigger request.
> > > > > > > > > > > -            \sa AfTrigger
> > > > > > > > > > > -        - name: AfStateFocusedLock
> > > > > > > > > > > -          value: 4
> > > > > > > > > > > -          description: |
> > > > > > > > > > > -            AF believes has focused correctly and has locked focus.
> > > > > > > > > > > -        - name: AfStateNotFocusedLock
> > > > > > > > > > > -          value: 5
> > > > > > > > > > > -          description: |
> > > > > > > > > > > -            AF has not been able to focus and has locked.
> > > > > > > > > > > -        - name: AfStatePassiveUnfocused
> > > > > > > > > > > -          value: 6
> > > > > > > > > > > -          description: |
> > > > > > > > > > > -            AF has completed a passive scan without finding focus.
> > > > > > > > > > > -
> > > > > > > > > > >    - AwbState:
> > > > > > > > > > >        type: int32_t
> > > > > > > > > > >        draft: true
> > > > > > > > > > > --
> > > > > > > > > > > 2.30.2
> > > > > > > > > > >