[v6,02/12] Documentation: design: ae: Document the design for AE controls

22475 diff mbox series

Document the design and rationale for the AE-related controls.
Also add documentation for the controls.

Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

---
No change in v6

Changes in v5:
- improve wording

Changes in v4:
- fix wording
- fix compilation
- remove locked state in explanation

Changes in v3:
- merge the control documentation into the same document (including the
  patch)
  - this is because it was a bit unwieldy to put it in
    control_ids.cpp.in, now that it's used generically to generate
    control ids of all namespaces
---
 Documentation/design/ae.rst | 320 ++++++++++++++++++++++++++++++++++++
 Documentation/index.rst     |   4 +-
 Documentation/meson.build   |   1 +
 3 files changed, 324 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/design/ae.rst

Hi Paul,

Thank you for the patch.

On Wed, Jan 08, 2025 at 06:09:32PM -0600, Paul Elder wrote:
> Document the design and rationale for the AE-related controls.
> Also add documentation for the controls.
> 
> Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> ---
> No change in v6
> 
> Changes in v5:
> - improve wording
> 
> Changes in v4:
> - fix wording
> - fix compilation
> - remove locked state in explanation
> 
> Changes in v3:
> - merge the control documentation into the same document (including the
>   patch)
>   - this is because it was a bit unwieldy to put it in
>     control_ids.cpp.in, now that it's used generically to generate
>     control ids of all namespaces
> ---
>  Documentation/design/ae.rst | 320 ++++++++++++++++++++++++++++++++++++
>  Documentation/index.rst     |   4 +-
>  Documentation/meson.build   |   1 +
>  3 files changed, 324 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/design/ae.rst
> 
> diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst
> new file mode 100644
> index 000000000..f8b2c887c
> --- /dev/null
> +++ b/Documentation/design/ae.rst
> @@ -0,0 +1,320 @@
> +.. SPDX-License-Identifier: CC-BY-SA-4.0
> +
> +Design of Exposure and Gain controls
> +====================================
> +
> +This document explains the design and rationale of the controls related to
> +exposure and gain. This includes the all-encompassing auto-exposure (AE), the
> +manual exposure control, and the manual gain control.
> +
> +Description of the problem
> +--------------------------
> +
> +Sub controls
> +^^^^^^^^^^^^
> +
> +There are more than one control that make up total exposure: exposure time,
> +gain, and aperture (though for now we will not consider aperture). We already
> +had individual controls for setting the values of manual exposure and manual
> +gain, but for switching between auto mode and manual mode we only had a
> +high-level boolean AeEnable control that would set *both* exposure and gain to
> +auto mode or manual mode; we had no way to set one to auto and the other to
> +manual.
> +
> +So, we need to introduce two new controls to act as "levers" to indicate
> +individually for exposure and gain if the value would come from AEGC or if it
> +would come from the manual control value.
> +
> +Aperture priority
> +^^^^^^^^^^^^^^^^^
> +
> +We eventually may need to support aperture, and so whatever our solution is for
> +having only some controls on auto and the others on manual needs to be
> +extensible.
> +
> +Flickering when going from auto to manual
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +When a manual exposure or gain value is requested by the application, it costs
> +a few frames worth of time for them to take effect. This means that during a
> +transition from auto to manual, there would be flickering in the control values
> +and the transition won't be smooth.
> +
> +Take for instance the following flow, where we start on auto exposure (which
> +for the purposes of the example increments by 1 each frame) and we want to
> +switch seamlessly to manual exposure, which involves copying the exposure value
> +computed by the auto exposure algorithm:
> +
> +::
> +
> +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> +                | N   | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
> +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> +
> + Mode requested: Auto    Auto    Auto    Manual  Manual  Manual  Manual
> + Exp requested:  N/A     N/A     N/A     2       2       2       2
> + Set in Frame:   N+2     N+3     N+4     N+5     N+6     N+7     N+8
> +
> + Mode used:      Auto    Auto    Auto    Auto    Auto    Manual  Manual
> + Exp used:       0       1       2       3       4       2       2
> +
> +As we can see, after frame N+2 completes, we copy the exposure value that was
> +used for frame N+2 (which was computed by AE algorithm), and queue that value
> +into request N+3 with manual mode on. However, as it takes two frames for the
> +exposure to be set, the exposure still changes since it is set by AE, and we
> +get a flicker in the exposure during the switch from auto to manual.
> +
> +A solution is to *not submit* any exposure value when manual mode is enabled,
> +and wait until the manual mode as been "applied" before copying the exposure
> +value:
> +
> +::
> +
> +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> +                | N   | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
> +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> +
> + Mode requested: Auto    Auto    Auto    Manual  Manual  Manual  Manual
> + Exp requested:  N/A     N/A     N/A     None    None    None    5
> + Set in Frame:   N+2     N+3     N+4     N+5     N+6     N+7     N+8
> +
> + Mode used:      Auto    Auto    Auto    Auto    Auto    Manual  Manual
> + Exp used:       0       1       2       3       4       5       5
> +
> +In practice, this works. However, libcamera has a policy where once a control
> +is submitted, its value is saved and does not need to be resubmitted. If the
> +manual exposure value was set while auto mode was on, in theory the value would
> +be saved, so when manual mode is enabled, the exposure value that was
> +previously set would immediately be used. Clearly this solution isn't correct,
> +but it can serve as the basis for a proper solution, with some more rigorous
> +rules.
> +
> +Existing solutions
> +------------------
> +
> +Raspberry Pi
> +^^^^^^^^^^^^
> +
> +The Raspberry Pi IPA gets around the lack of individual AeEnable controls for
> +exposure and gain by using magic values. When AeEnable is false, if one of the
> +manual control values was set to 0 then the value computed by AEGC would be
> +used for just that control. This solution isn't desirable, as it prevents
> +that magic value from being used as a valid value.
> +
> +To get around the flickering issue, when AeEnable is false, the Raspberry Pi
> +AEGC simply stops updating the values to be set, without restoring the
> +previously set manual exposure time and gain. This works, but is not a proper
> +solution.
> +
> +Android
> +^^^^^^^
> +
> +The Android HAL specification requires that exposure and gain (sensitivity)
> +must both be manual or both be auto. It cannot be that one is manual while the
> +other is auto, so they simply don't support sub controls.
> +
> +For the flickering issue, the Android HAL has an AeLock control. To transition
> +from auto to manual, the application would keep AE on auto, and turn on the
> +lock. Once the lock has propagated through, then the value can be copied from
> +the result into the request and the lock disabled and the mode set to manual.
> +
> +The problem with this solution is, besides the extra complexity, that it is
> +ambiguous what happens if there is a state transition from manual to locked
> +(even though it's a state transition that doesn't make sense). If locked is
> +defined to "use the last automatically computed values" then it could use the
> +values from the last time it AE was set to auto, or it would be undefined if AE
> +was never auto (eg. it started out as manual), or if AE is implemented to run
> +in the background it could just use the current values that are computed. If
> +locked is defined to "use the last value that was set" there would be less
> +ambiguity. Still, it's better if we can make it impossible to execute this
> +nonsensical state transition, and if we can reduce the complexity of having
> +this extra control or extra setting on a lever.
> +
> +Summary of goals
> +----------------
> +
> +- We need a lock of some sort, to instruct the AEGC to not update output
> +  results
> +
> +- We need manual modes, to override the values computed by the AEGC
> +
> +- We need to support seamless transitions from auto to manual, and do so
> +  without flickering
> +
> +- We need custom minimum values for the manual controls; that is, no magic
> +  values for enabling/disabling auto
> +
> +- All of these need to be done with AE sub-controls (exposure time, analogue
> +  gain) and be extensible to aperture in the future
> +
> +Our solution
> +------------
> +
> +A diagram of our solution:
> +
> +::
> +
> +  +----------------------------+-------------+------------------+-----------------+
> +  |          INPUT             |  ALGORITHM  |     RESULT       |     OUTPUT      |
> +  +----------------------------+-------------+------------------+-----------------+
> +
> +    ExposureTimeMode                                             ExposureTimeMode
> +  ---------------------+----------------------------------------+----------------->
> +    0: Auto            |                                        |
> +    1: Manual          |                                        V
> +                       |                                       |\
> +                       |                                       | \
> +                       |  /----------------------------------> | 1|  ExposureTime
> +                       |  |    +-------------+  exposure time  |  | -------------->
> +                       \--)--> |             | --------------> | 0|
> +    ExposureTime          |    |             |                 | /
> +  ------------------------+--> |             |                 |/
> +                               |             |                       AeState
> +                               |     AEGC    | ----------------------------------->
> +    AnalogueGain               |             |
> +  ------------------------+--> |             |                 |\
> +                          |    |             |                 | \
> +                       /--)--> |             | --------------> | 0|  AnalogueGain
> +                       |  |    +-------------+  analogue gain  |  | -------------->
> +                       |  \----------------------------------> | 1|
> +                       |                                       | /
> +                       |                                       |/
> +                       |                                        ^
> +    AnalogueGainMode   |                                        | AnalogueGainMode
> +  ---------------------+----------------------------------------+----------------->
> +    0: Auto
> +    1: Manual
> +
> +
> +The diagram is divided in four sections horizontally:
> +
> +- Input: The values received from the request controls
> +
> +- Algorithm: The algorithm itself
> +
> +- Result: The values calculated by the algorithm
> +
> +- Output: The values reported in result metadata and applied to the device
> +
> +The four input controls are divided between manual values (ExposureTime and
> +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The
> +former are the manual values, the latter control how they're applied. The two
> +modes are independent from each other, and each can take one of two values:
> +
> +    - Auto (0): The AGC computes the value normally. The AGC result is applied
> +      to the output. The manual value is ignored *and is not retained*.
> +
> +    - Manual (1): The AGC uses the manual value internally. The corresponding
> +      manual control from the request is applied to the output. The AGC result
> +      is ignored.

Too much indentation here, move the '-' marker to the leftmost column.

> +
> +The AeState control reports the state of the unified AEGC block. If both
> +ExposureTimeMode and AnalogueGainMode are set to manual then it will report
> +Idle. If at least one of the two is set to auto, then AeState will report
> +if the AEGC has Converged or not (Searching). This control replaces the old
> +AeLocked control, as it was insufficient for reporting the AE state.
> +
> +There is a caveat to manual mode: the manual control value is not retained if
> +it is set during auto mode. This means that if manual mode is entered without
> +also setting the manual value, then it will enter a state similar to "locked",
> +where the last automatically computed value while the mode was auto will be
> +used. Once the manual value is set, then that will be used and retained as
> +usual.
> +
> +This simulates an auto -> locked -> manual or auto -> manual state transition,
> +and makes it impossible to do the nonsensical manual -> locked state
> +transition.
> +
> +We specifically do not have a "master AE control" like the old AeEnable. This
> +is because we have the individual mode controls, and if we had a master AE
> +control it would be a "control that sets other controls", which could easily
> +get out of control.
> +
> +With this solution, the earlier example would become:
> +
> +::
> +
> +                 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> +                 | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|
> +                 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> + Mode requested:  Auto    Manual  Manual  Manual  Manual  Manual  Manual  Manual  Manual
> + Exp requested:   N/A     None    None    None    None    10      None    10      10
> + Set in Frame:    N+4     N+5     N+6     N+7     N+8     N+9     N+10    N+11    N+12
> +
> + Mode used:       Auto    Auto    Auto    Manual  Manual  Manual  Manual  Manual  Manual
> + Exp used:        2       3       4       5       5       5       5       10      10
> +
> +This example is extended by a few frames to exhibit the simulated "locked"
> +state. At frame N+5 the application has confirmed that the manual mode has been
> +entered, but does not provide a manual value until request N+7. Thus, the value
> +that is used in requests N+5 and N+6 (where the mode is disabled), comes from
> +the last value that was used when the mode was auto, which comes from frame
> +N+4.
> +
> +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for
> +the exposure to be applied. N+8 does not supply a manual value, but the last
> +supplied value is retained, so a manual value of 10 is still used and set in
> +frame N+10.
> +
> +Although this behavior is the same as what we had with waiting for the manual
> +mode to propagate (in the section "Description of the problem"), this time it
> +is correct as we have defined specifically that if a manual value was specified
> +while the mode was auto, it will not be retained.
> +
> +Description of the controls
> +---------------------------
> +
> +As described above, libcamera offers the following controls related to exposure
> +and gain:
> +
> +- AnalogueGain
> +
> +- AnalogueGainMode
> +
> +- ExposureTime
> +
> +- ExposureTimeMode
> +
> +- AeState
> +
> +Auto-exposure and auto-gain can be enabled and disabled separately using the
> +ExposureTimeMode and AnalogueGainMode controls respectively. There is no
> +overarching AeEnable control.
> +
> +When the respective mode is set to auto, the respective value that is computed
> +by the AEGC algorithm is applied to the image sensor. Any value that is
> +supplied in the manual ExposureTime/AnalogueGain control is ignored and not
> +retained. Another way to understand this is that when the mode transitions from
> +auto to manual, the internally stored control value is overwritten with the
> +last value computed by the auto algorithm.
> +
> +This means that when we transition from auto to manual without supplying a
> +manual control value, the last value that was set by the AEGC algorithm will
> +keep be used. This can be used to do a flickerless transition from auto to
> +manual as described earlier. If the camera started out in manual mode and no
> +corresponding value has been supplied yet, then a best-effort default value
> +shall be set.
> +
> +The manual control value can be set in the same request as setting the mode to
> +auto if the desired manual control value is already known.
> +
> +Transitioning from manual to auto shall be implicitly flickerless, as the AEGC
> +algorithms are expected to start running from the last manual value.
> +
> +The AeState metadata reports the state of the AE algorithm. As AE cannot
> +compute exposure and gain separately, the state of the AE component is
> +unified. There are three states: Idle, Searching, and Converged.
> +
> +The state shall be Idle if both ExposureTimeMode and AnalogueGainMode
> +are set to Manual. If the camera only supports one of the two controls,
> +then the state shall be Idle if that one control is set to Manual. If
> +the camera does not support Manual for at least one of the two controls,
> +then the state will never be Idle, as AE will always be running.
> +
> +The state shall be Searching if at least one of exposure or gain calculated
> +by the AE algorithm is used (that is, at least one of the two modes is Auto),
> +*and* the value(s) have not converged yet.
> +
> +The state shall be Converged if at least one of exposure or gain calculated
> +by the AE algorithm is used (that is, at least one of the two modes is Auto),
> +*and* the value(s) have converged.
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index bea406608..251112fbd 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -23,7 +23,9 @@
>     SoftwareISP Benchmarking <software-isp-benchmarking>
>     Tracing guide <guides/tracing>
>  
> +   Design document: AE <design/ae>
> +
>  .. toctree::
>     :hidden:
>  
> -   introduction
> \ No newline at end of file
> +   introduction
> diff --git a/Documentation/meson.build b/Documentation/meson.build
> index 36ffae239..6158320e1 100644
> --- a/Documentation/meson.build
> +++ b/Documentation/meson.build
> @@ -128,6 +128,7 @@ if sphinx.found()
>          'coding-style.rst',
>          'conf.py',
>          'contributing.rst',
> +        'design/ae.rst',
>          'documentation-contents.rst',
>          'environment_variables.rst',
>          'feature_requirements.rst',

On Thu, Jan 09, 2025 at 11:41:59PM +0200, Laurent Pinchart wrote:
> Hi Paul,
> 
> Thank you for the patch.
> 
> On Wed, Jan 08, 2025 at 06:09:32PM -0600, Paul Elder wrote:
> > Document the design and rationale for the AE-related controls.
> > Also add documentation for the controls.
> > 
> > Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
> > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
> > Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > 
> > ---
> > No change in v6
> > 
> > Changes in v5:
> > - improve wording
> > 
> > Changes in v4:
> > - fix wording
> > - fix compilation
> > - remove locked state in explanation
> > 
> > Changes in v3:
> > - merge the control documentation into the same document (including the
> >   patch)
> >   - this is because it was a bit unwieldy to put it in
> >     control_ids.cpp.in, now that it's used generically to generate
> >     control ids of all namespaces
> > ---
> >  Documentation/design/ae.rst | 320 ++++++++++++++++++++++++++++++++++++
> >  Documentation/index.rst     |   4 +-
> >  Documentation/meson.build   |   1 +
> >  3 files changed, 324 insertions(+), 1 deletion(-)
> >  create mode 100644 Documentation/design/ae.rst
> > 
> > diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst
> > new file mode 100644
> > index 000000000..f8b2c887c
> > --- /dev/null
> > +++ b/Documentation/design/ae.rst
> > @@ -0,0 +1,320 @@
> > +.. SPDX-License-Identifier: CC-BY-SA-4.0
> > +
> > +Design of Exposure and Gain controls
> > +====================================
> > +
> > +This document explains the design and rationale of the controls related to
> > +exposure and gain. This includes the all-encompassing auto-exposure (AE), the
> > +manual exposure control, and the manual gain control.
> > +
> > +Description of the problem
> > +--------------------------
> > +
> > +Sub controls
> > +^^^^^^^^^^^^
> > +
> > +There are more than one control that make up total exposure: exposure time,
> > +gain, and aperture (though for now we will not consider aperture). We already
> > +had individual controls for setting the values of manual exposure and manual
> > +gain, but for switching between auto mode and manual mode we only had a
> > +high-level boolean AeEnable control that would set *both* exposure and gain to
> > +auto mode or manual mode; we had no way to set one to auto and the other to
> > +manual.
> > +
> > +So, we need to introduce two new controls to act as "levers" to indicate
> > +individually for exposure and gain if the value would come from AEGC or if it
> > +would come from the manual control value.
> > +
> > +Aperture priority
> > +^^^^^^^^^^^^^^^^^
> > +
> > +We eventually may need to support aperture, and so whatever our solution is for
> > +having only some controls on auto and the others on manual needs to be
> > +extensible.
> > +
> > +Flickering when going from auto to manual
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +When a manual exposure or gain value is requested by the application, it costs
> > +a few frames worth of time for them to take effect. This means that during a
> > +transition from auto to manual, there would be flickering in the control values
> > +and the transition won't be smooth.
> > +
> > +Take for instance the following flow, where we start on auto exposure (which
> > +for the purposes of the example increments by 1 each frame) and we want to
> > +switch seamlessly to manual exposure, which involves copying the exposure value
> > +computed by the auto exposure algorithm:
> > +
> > +::
> > +
> > +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > +                | N   | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
> > +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > +
> > + Mode requested: Auto    Auto    Auto    Manual  Manual  Manual  Manual
> > + Exp requested:  N/A     N/A     N/A     2       2       2       2
> > + Set in Frame:   N+2     N+3     N+4     N+5     N+6     N+7     N+8
> > +
> > + Mode used:      Auto    Auto    Auto    Auto    Auto    Manual  Manual
> > + Exp used:       0       1       2       3       4       2       2
> > +
> > +As we can see, after frame N+2 completes, we copy the exposure value that was
> > +used for frame N+2 (which was computed by AE algorithm), and queue that value
> > +into request N+3 with manual mode on. However, as it takes two frames for the
> > +exposure to be set, the exposure still changes since it is set by AE, and we
> > +get a flicker in the exposure during the switch from auto to manual.
> > +
> > +A solution is to *not submit* any exposure value when manual mode is enabled,
> > +and wait until the manual mode as been "applied" before copying the exposure
> > +value:
> > +
> > +::
> > +
> > +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > +                | N   | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
> > +                +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > +
> > + Mode requested: Auto    Auto    Auto    Manual  Manual  Manual  Manual
> > + Exp requested:  N/A     N/A     N/A     None    None    None    5
> > + Set in Frame:   N+2     N+3     N+4     N+5     N+6     N+7     N+8
> > +
> > + Mode used:      Auto    Auto    Auto    Auto    Auto    Manual  Manual
> > + Exp used:       0       1       2       3       4       5       5
> > +
> > +In practice, this works. However, libcamera has a policy where once a control
> > +is submitted, its value is saved and does not need to be resubmitted. If the
> > +manual exposure value was set while auto mode was on, in theory the value would
> > +be saved, so when manual mode is enabled, the exposure value that was
> > +previously set would immediately be used. Clearly this solution isn't correct,
> > +but it can serve as the basis for a proper solution, with some more rigorous
> > +rules.
> > +
> > +Existing solutions
> > +------------------
> > +
> > +Raspberry Pi
> > +^^^^^^^^^^^^
> > +
> > +The Raspberry Pi IPA gets around the lack of individual AeEnable controls for
> > +exposure and gain by using magic values. When AeEnable is false, if one of the
> > +manual control values was set to 0 then the value computed by AEGC would be
> > +used for just that control. This solution isn't desirable, as it prevents
> > +that magic value from being used as a valid value.
> > +
> > +To get around the flickering issue, when AeEnable is false, the Raspberry Pi
> > +AEGC simply stops updating the values to be set, without restoring the
> > +previously set manual exposure time and gain. This works, but is not a proper
> > +solution.
> > +
> > +Android
> > +^^^^^^^
> > +
> > +The Android HAL specification requires that exposure and gain (sensitivity)
> > +must both be manual or both be auto. It cannot be that one is manual while the
> > +other is auto, so they simply don't support sub controls.
> > +
> > +For the flickering issue, the Android HAL has an AeLock control. To transition
> > +from auto to manual, the application would keep AE on auto, and turn on the
> > +lock. Once the lock has propagated through, then the value can be copied from
> > +the result into the request and the lock disabled and the mode set to manual.
> > +
> > +The problem with this solution is, besides the extra complexity, that it is
> > +ambiguous what happens if there is a state transition from manual to locked
> > +(even though it's a state transition that doesn't make sense). If locked is
> > +defined to "use the last automatically computed values" then it could use the
> > +values from the last time it AE was set to auto, or it would be undefined if AE
> > +was never auto (eg. it started out as manual), or if AE is implemented to run
> > +in the background it could just use the current values that are computed. If
> > +locked is defined to "use the last value that was set" there would be less
> > +ambiguity. Still, it's better if we can make it impossible to execute this
> > +nonsensical state transition, and if we can reduce the complexity of having
> > +this extra control or extra setting on a lever.
> > +
> > +Summary of goals
> > +----------------
> > +
> > +- We need a lock of some sort, to instruct the AEGC to not update output
> > +  results
> > +
> > +- We need manual modes, to override the values computed by the AEGC
> > +
> > +- We need to support seamless transitions from auto to manual, and do so
> > +  without flickering
> > +
> > +- We need custom minimum values for the manual controls; that is, no magic
> > +  values for enabling/disabling auto
> > +
> > +- All of these need to be done with AE sub-controls (exposure time, analogue
> > +  gain) and be extensible to aperture in the future
> > +
> > +Our solution
> > +------------
> > +
> > +A diagram of our solution:
> > +
> > +::
> > +
> > +  +----------------------------+-------------+------------------+-----------------+
> > +  |          INPUT             |  ALGORITHM  |     RESULT       |     OUTPUT      |
> > +  +----------------------------+-------------+------------------+-----------------+
> > +
> > +    ExposureTimeMode                                             ExposureTimeMode
> > +  ---------------------+----------------------------------------+----------------->
> > +    0: Auto            |                                        |
> > +    1: Manual          |                                        V
> > +                       |                                       |\
> > +                       |                                       | \
> > +                       |  /----------------------------------> | 1|  ExposureTime
> > +                       |  |    +-------------+  exposure time  |  | -------------->
> > +                       \--)--> |             | --------------> | 0|
> > +    ExposureTime          |    |             |                 | /
> > +  ------------------------+--> |             |                 |/
> > +                               |             |                       AeState
> > +                               |     AEGC    | ----------------------------------->
> > +    AnalogueGain               |             |
> > +  ------------------------+--> |             |                 |\
> > +                          |    |             |                 | \
> > +                       /--)--> |             | --------------> | 0|  AnalogueGain
> > +                       |  |    +-------------+  analogue gain  |  | -------------->
> > +                       |  \----------------------------------> | 1|
> > +                       |                                       | /
> > +                       |                                       |/
> > +                       |                                        ^
> > +    AnalogueGainMode   |                                        | AnalogueGainMode
> > +  ---------------------+----------------------------------------+----------------->
> > +    0: Auto
> > +    1: Manual
> > +
> > +
> > +The diagram is divided in four sections horizontally:
> > +
> > +- Input: The values received from the request controls
> > +
> > +- Algorithm: The algorithm itself
> > +
> > +- Result: The values calculated by the algorithm
> > +
> > +- Output: The values reported in result metadata and applied to the device
> > +
> > +The four input controls are divided between manual values (ExposureTime and
> > +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The
> > +former are the manual values, the latter control how they're applied. The two
> > +modes are independent from each other, and each can take one of two values:
> > +
> > +    - Auto (0): The AGC computes the value normally. The AGC result is applied
> > +      to the output. The manual value is ignored *and is not retained*.
> > +
> > +    - Manual (1): The AGC uses the manual value internally. The corresponding
> > +      manual control from the request is applied to the output. The AGC result
> > +      is ignored.
> 
> Too much indentation here, move the '-' marker to the leftmost column.
> 

Is... that all?


Paul

> > +
> > +The AeState control reports the state of the unified AEGC block. If both
> > +ExposureTimeMode and AnalogueGainMode are set to manual then it will report
> > +Idle. If at least one of the two is set to auto, then AeState will report
> > +if the AEGC has Converged or not (Searching). This control replaces the old
> > +AeLocked control, as it was insufficient for reporting the AE state.
> > +
> > +There is a caveat to manual mode: the manual control value is not retained if
> > +it is set during auto mode. This means that if manual mode is entered without
> > +also setting the manual value, then it will enter a state similar to "locked",
> > +where the last automatically computed value while the mode was auto will be
> > +used. Once the manual value is set, then that will be used and retained as
> > +usual.
> > +
> > +This simulates an auto -> locked -> manual or auto -> manual state transition,
> > +and makes it impossible to do the nonsensical manual -> locked state
> > +transition.
> > +
> > +We specifically do not have a "master AE control" like the old AeEnable. This
> > +is because we have the individual mode controls, and if we had a master AE
> > +control it would be a "control that sets other controls", which could easily
> > +get out of control.
> > +
> > +With this solution, the earlier example would become:
> > +
> > +::
> > +
> > +                 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > +                 | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|
> > +                 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > + Mode requested:  Auto    Manual  Manual  Manual  Manual  Manual  Manual  Manual  Manual
> > + Exp requested:   N/A     None    None    None    None    10      None    10      10
> > + Set in Frame:    N+4     N+5     N+6     N+7     N+8     N+9     N+10    N+11    N+12
> > +
> > + Mode used:       Auto    Auto    Auto    Manual  Manual  Manual  Manual  Manual  Manual
> > + Exp used:        2       3       4       5       5       5       5       10      10
> > +
> > +This example is extended by a few frames to exhibit the simulated "locked"
> > +state. At frame N+5 the application has confirmed that the manual mode has been
> > +entered, but does not provide a manual value until request N+7. Thus, the value
> > +that is used in requests N+5 and N+6 (where the mode is disabled), comes from
> > +the last value that was used when the mode was auto, which comes from frame
> > +N+4.
> > +
> > +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for
> > +the exposure to be applied. N+8 does not supply a manual value, but the last
> > +supplied value is retained, so a manual value of 10 is still used and set in
> > +frame N+10.
> > +
> > +Although this behavior is the same as what we had with waiting for the manual
> > +mode to propagate (in the section "Description of the problem"), this time it
> > +is correct as we have defined specifically that if a manual value was specified
> > +while the mode was auto, it will not be retained.
> > +
> > +Description of the controls
> > +---------------------------
> > +
> > +As described above, libcamera offers the following controls related to exposure
> > +and gain:
> > +
> > +- AnalogueGain
> > +
> > +- AnalogueGainMode
> > +
> > +- ExposureTime
> > +
> > +- ExposureTimeMode
> > +
> > +- AeState
> > +
> > +Auto-exposure and auto-gain can be enabled and disabled separately using the
> > +ExposureTimeMode and AnalogueGainMode controls respectively. There is no
> > +overarching AeEnable control.
> > +
> > +When the respective mode is set to auto, the respective value that is computed
> > +by the AEGC algorithm is applied to the image sensor. Any value that is
> > +supplied in the manual ExposureTime/AnalogueGain control is ignored and not
> > +retained. Another way to understand this is that when the mode transitions from
> > +auto to manual, the internally stored control value is overwritten with the
> > +last value computed by the auto algorithm.
> > +
> > +This means that when we transition from auto to manual without supplying a
> > +manual control value, the last value that was set by the AEGC algorithm will
> > +keep be used. This can be used to do a flickerless transition from auto to
> > +manual as described earlier. If the camera started out in manual mode and no
> > +corresponding value has been supplied yet, then a best-effort default value
> > +shall be set.
> > +
> > +The manual control value can be set in the same request as setting the mode to
> > +auto if the desired manual control value is already known.
> > +
> > +Transitioning from manual to auto shall be implicitly flickerless, as the AEGC
> > +algorithms are expected to start running from the last manual value.
> > +
> > +The AeState metadata reports the state of the AE algorithm. As AE cannot
> > +compute exposure and gain separately, the state of the AE component is
> > +unified. There are three states: Idle, Searching, and Converged.
> > +
> > +The state shall be Idle if both ExposureTimeMode and AnalogueGainMode
> > +are set to Manual. If the camera only supports one of the two controls,
> > +then the state shall be Idle if that one control is set to Manual. If
> > +the camera does not support Manual for at least one of the two controls,
> > +then the state will never be Idle, as AE will always be running.
> > +
> > +The state shall be Searching if at least one of exposure or gain calculated
> > +by the AE algorithm is used (that is, at least one of the two modes is Auto),
> > +*and* the value(s) have not converged yet.
> > +
> > +The state shall be Converged if at least one of exposure or gain calculated
> > +by the AE algorithm is used (that is, at least one of the two modes is Auto),
> > +*and* the value(s) have converged.
> > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > index bea406608..251112fbd 100644
> > --- a/Documentation/index.rst
> > +++ b/Documentation/index.rst
> > @@ -23,7 +23,9 @@
> >     SoftwareISP Benchmarking <software-isp-benchmarking>
> >     Tracing guide <guides/tracing>
> >  
> > +   Design document: AE <design/ae>
> > +
> >  .. toctree::
> >     :hidden:
> >  
> > -   introduction
> > \ No newline at end of file
> > +   introduction
> > diff --git a/Documentation/meson.build b/Documentation/meson.build
> > index 36ffae239..6158320e1 100644
> > --- a/Documentation/meson.build
> > +++ b/Documentation/meson.build
> > @@ -128,6 +128,7 @@ if sphinx.found()
> >          'coding-style.rst',
> >          'conf.py',
> >          'contributing.rst',
> > +        'design/ae.rst',
> >          'documentation-contents.rst',
> >          'environment_variables.rst',
> >          'feature_requirements.rst',