| Message ID | 20241113131256.3170817-3-paul.elder@ideasonboard.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series |
|
| Related | show |
Hi Paul, Thank you for the patch. On Wed, Nov 13, 2024 at 10:12:50PM +0900, Paul Elder wrote: > Document the design and rationale for the AE-related controls. > Also add documentation for the controls. > > Signed-off-by: Paul Elder <paul.elder@ideasonboard.com> > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org> > > --- > Changes in v3: > - merge the control documentation into the same document (including the > patch) > - this is because it was a bit unwieldy to put it in > control_ids.cpp.in, now that it's used generically to generate > control ids of all namespaces > --- > Documentation/design/ae.rst | 348 ++++++++++++++++++++++++++++++++++++ > 1 file changed, 348 insertions(+) > create mode 100644 Documentation/design/ae.rst > > diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst > new file mode 100644 > index 000000000000..f121afecea5a > --- /dev/null > +++ b/Documentation/design/ae.rst > @@ -0,0 +1,348 @@ > +.. SPDX-License-Identifier: CC-BY-SA-4.0 > + > +Design of Exposure and Gain controls > +==================================== > + > +This document explains the design and rationale of the controls related to > +exposure and gain. This includes the all-encompassing auto-exposure (AE), the > +manual exposure control, and the manual gain control. > + > +Description of the problem > +-------------------------- > + > +Sub controls > +^^^^^^^^^^^^ > + > +There are more than one control that make up exposure: exposure, gain, and > +aperture (though for now we will not consider aperture). We already had > +individual controls for setting the values of manual exposure and manual gain, > +but for switching between auto mode and manual mode we only had a high-level > +boolean AeEnable control that would set *both* exposure and gain to auto mode > +or manual mode; we had no way to set one to auto and the other to manual. > + > +So, we need to introduce two new controls to act as "levers" to indicate > +individually for exposure and gain if the value would come from AEGC or if it > +would come from the manual control value. > + > +Aperture priority > +^^^^^^^^^^^^^^^^^ > + > +We eventually will support aperture, and so whatever our solution is for having > +only some controls on auto and the others on manual needs to be extensible. > + > +Flickering when going from auto to manual > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + > +When a manual exposure or gain value is requested by the application, it costs > +a few frames worth of time for them to take effect. This means that during a > +transition from auto to manual, there would be flickering in the control values > +and the transition won't be smooth. > + > +Take for instance the following flow, where we start on auto exposure (which > +for the purposes of the example increments by 1 each frame) and we want to > +switch seamlessly to manual exposure, which involves copying the exposure value > +computed by the auto exposure algorithm: > + > +:: > + > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + > + Mode requested: Auto Auto Auto Manual Manual Manual Manual > + Exp requested: N/A N/A N/A 2 2 2 2 > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 > + > + Mode used: Auto Auto Auto Auto Auto Manual Manual > + Exp used: 0 1 2 3 4 2 2 > + > +As we can see, after frame N+2 completes, we copy the exposure value that was > +used for frame N+2 (which was computed by AE algorithm), and queue that value > +into request N+3 with manual mode on. However, as it takes two frames for the > +exposure to be set, the exposure still changes since it is set by AE, and we > +get a flicker in the exposure during the switch from auto to manual. > + > +A solution is to *not submit* any exposure value when manual mode is enabled, > +and wait until the manual mode as been "applied" before copying the exposure > +value: > + > +:: > + > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + > + Mode requested: Auto Auto Auto Manual Manual Manual Manual > + Exp requested: N/A N/A N/A None None None 5 > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 > + > + Mode used: Auto Auto Auto Auto Auto Manual Manual > + Exp used: 0 1 2 3 4 5 5 > + > +In practice, this works. However, libcamera has a policy where once a control > +is submitted, its value is saved and does not need to be resubmitted (even > +though this isn't implemented yet). So if the manual exposure value was set > +while auto mode was on, in theory the value would be saved, so when manual mode > +is enabled, the exposure value that was previously set would immediately be > +used. Clearly this solution isn't correct, but it can serve as the basis for a > +proper solution, with some more rigorous rules. > + > +Existing solutions > +------------------ > + > +Raspberry Pi > +^^^^^^^^^^^^ > + > +The raspberry pi IPA got around the lack of individual AeEnable controls for > +exposure and gain by using magic values. When AeEnable was false, if one of the > +manual control values was set to 0 then the value computed by AEGC would be > +used for just that control. This solution isn't desirable, as it prevents > +that magic value from being used as a valid value. > + > +To get around the flickering issue, when AeEnable was false, the raspberry pi > +AEGC would simply stop updating the values to be set. As mentioned above, since > +the value retention mechanism hasn't actually been implemented yet, this > +worked. But, it's not a proper solution. > + > +Android > +^^^^^^^ > + > +The Android HAL specification requires that exposure and gain (sensitivity) > +must both be manual or both be auto. It cannot be that one is manual while the > +other is auto, so they simply don't support sub controls. > + > +For the flickering issue, the Android HAL has an AeLock control. To transition > +from auto to manual, the application would keep AE on auto, and turn on the > +lock. Once the lock has propagated through, then the value can be copied from > +the result into the request and the lock disabled and the mode set to manual. > + > +The problem with this solution is, besides the extra complexity, that it is > +ambiguous what happens if there is a state transition from manual to locked > +(even though it's a state transition that doesn't make sense). If locked is > +defined to "use the last automatically computed values" then it could use the > +values from the last time it AE was set to auto, or it would be undefined if AE > +was never auto (eg. it started out as manual), or if AE is implemented to run > +in the background it could just use the current values that are computed. If > +locked is defined to "use the last value that was set" there would be less > +ambiguity. Still, it's better if we can make it impossible to execute this > +nonsensical state transition, and if we can reduce the complexity of having > +this extra control or extra setting on a lever. > + > +Summary of goals > +---------------- > + > + - We need a lock of some sort, to instruct the AEGC to not update output > + results > + > + - We need manual modes, to override the values computed by the AEGC > + > + - We need to support seamless transitions from auto to manual, and do so > + without flickering > + > + - We need custom minimum values for the manual controls; that is, no magic > + values for enabling/disabling auto > + > + - All of these need to be done with AE sub-controls (exposure time, analogue > + gain) and be extensible to aperture in the future > + > +Our solution > +------------ > + > +A diagram of our solution: > + > +:: > + > + +----------------------------+-------------+------------------+-----------------+ > + | INPUT | ALGORITHM | RESULT | OUTPUT | > + +----------------------------+-------------+------------------+-----------------+ > + > + ExposureTimeMode ExposureTimeMode > + ---------------------+----------------------------------------+-----------------> > + 0: Auto | | > + 1: Manual | V > + | |\ > + | | \ > + | /----------------------------------> | 1| ExposureTime > + | | +-------------+ exposure time | | --------------> > + \--)--> | | --------------> | 0| > + ExposureTime | | | | / > + ------------------------+--> | | |/ > + | | AeState > + | AEGC | -----------------------------------> > + AnalogueGain | | > + ------------------------+--> | | |\ > + | | | | \ > + /--)--> | | --------------> | 0| AnalogueGain > + | | +-------------+ analogue gain | | --------------> > + | \----------------------------------> | 1| > + | | / > + | |/ > + | ^ > + AnalogueGainMode | | AnalogueGainMode > + ---------------------+----------------------------------------+-----------------> > + 0: Auto > + 1: Manual > + > + > +The diagram is divided in four sections horizontally: > + > + - Input: The values received from the request controls > + > + - Algorithm: The algorithm itself > + > + - Result: The values calculated by the algorithm > + > + - Output: The values that sent in result metadata and applied to the device > + > +The four input controls are divided between manual values (ExposureTime and > +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The > +former are the manual values, the latter control how they're applied. The two > +modes are independent from each other, and each can take one of two values: > + > + - Auto (0): The AGC computes the value normally. The AGC result is applied > + to the output. The manual value is ignored *and is not retained*. > + > + - Manual (1): The AGC uses the manual value internally. The corresponding > + manual control from the request is applied to the output. The AGC result > + is ignored. > + > +The AeState control reports the state of the unified AEGC block. If both > +ExposureTimeMode and AnalogueGainMode are set to disabled then it will report > +Idle. If at least one of the two is set to auto, then AeState will report > +if the AEGC has Converged or not (Searching). This control replaces the old > +AeLocked control, as it was insufficient for reporting the AE state. > + > +There is a caveat to the disabled mode: the manual control value is not > +retained if it is set during auto mode. This means that if the disabled mode is > +entered without also setting the manual value, then it will enter a state > +similar to "locked", where the last automatically computed value while the mode > +was auto will be used. Once the manual value is set, then that will be used and > +retained as usual. > + > +This simulates an auto -> locked -> manual or auto -> manual state transition, > +and makes it impossible to do the nonsensical manual -> locked state > +transition. > + > +We specifically do not have a "master AE control" like the old AeEnable. This > +is because we have the individual mode controls, and if we had a master AE > +control it would be a "control that sets other controls", which could easily > +get out of control. > + > +With this solution, the earlier example would become: > + > +:: > + > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10| > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab > + Exp requested: N/A None None None None 10 None 10 10 > + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10 > + > + Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab > + Exp used: 2 3 4 5 5 5 5 10 10 > + > +This example is extended by a few frames to exhibit the simulated "locked" > +state. At frame N+5 the application has confirmed that the auto mode has been > +disabled, but does not provide a manual value until request N+7. Thus, the > +value that is used in requests N+5 and N+6 (where the mode is disabled), comes > +from the last value that was used when the mode was auto, which comes from > +frame N+4. > + > +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for > +the exposure to be applied. N+8 does not supply a manual value, but the last > +supplied value is retained, so a manual value of 10 is still used and set in > +frame N+10. > + > +Although this behavior is the same as what we had with waiting for the manual > +mode to propagate (in the section "Description of the problem"), this time it > +is correct as we have defined specifically that if a manual value was specified > +while the mode was auto, it will not be retained. > + > +Description of the controls > +=========================== > + > + libcamera offers the following controls related to exposure and gain: > + > + - AnalogueGain > + > + - AnalogueGainMode > + > + - ExposureTime > + > + - ExposureTimeMode > + > + - AeState > + > + Auto-exposure and auto-gain can be enabled and disabled separately using the > + ExposureTimeMode and AnalogueGainMode controls respectively. There is no > + overarching AeEnable control. > + > + For each of exposure and gain, we can model it with three states: auto, > + locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only > + have two values, as the locked state is simulated. > + > + :: > + > + /---------------------------------\ > + | | > + V | > + +--------+ +--------+ > + | | ---------------------> | | > + | Auto | +--------+ | Manual | > + | | ----> | Locked | ----> | | > + +--------+ +--------+ +--------+ > + ^ | > + | | > + \----------------/ > + > + Notice from the state diagram that locked to manual is a one-way state > + change, as the reverse direction is nonsensical (see the design document for > + more details on this topic). > + > + The exposure/gain is in the Auto state when > + ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value > + that is computed by the AE algorithm is applied to the image sensor. Any > + value that is supplied in the ExposureTime/AnalogueGain control is ignored > + and is not retained. > + > + If ExposureTimeMode/AnalogueGainMode is set to Manual, it can put us in > + either the Locked or Manual state. The difference is in if > + ExposureTime/AnalogueGain has been supplied. If it has not yet been > + supplied, then we are in the Locked state. If it has been supplied, then we > + are in the Manual state. > + > + In both the Locked state and the Manual state the exposure/gain value does > + not come from the AE algorithm. In the Locked state the value comes from the > + last value computed by the AE algorithm while the state was Auto, or if the > + state was never Auto (e.g. we started in Locked, or the camera doesn't > + support Auto), then the value should be a best-effort default value. In the > + Manual state the value comes from the value supplied in the > + ExposureTime/AnalogueGain control. > + > + To transition from the Locked state to the Manual state, a value needs to be > + submitted in ExposureTime/AnalogueGain. Once the state has transitioned to > + Manual, then this value will be retained, and doesn't need to be resubmitted > + if it doesn't change. > + > + To transition to the Auto state, simply set > + ExposureTimeMode/AnalogueGainMode to Auto. > + > + > + The AeState metadata reports the state of the AE algorithm. As AE cannot > + compute exposure and gain separately, the state of the AE component is > + unified. There are three states: Idle, Searching, and Converged. > + > + The state shall be Idle if both ExposureTimeMode and AnalogueGainMode > + are set to Manual. If the camera only supports one of the two controls, > + then the state shall be Idle if that one control is set to Manual. If > + the camera does not support Manual for at least one of the two controls, > + then the state will never be Idle, as AE will always be running. > + > + The state shall be Searching if at least one of exposure or gain calculated > + by the AE algorithm is used (that is, at least one of the two modes is Auto), > + *and* the value(s) have not converged yet. > + > + The state shall be Converged if at least one of exposure or gain calculated > + by the AE algorithm is used (that is, at least one of the two modes is Auto), > + *and* the value(s) have converged. That is a lot of stuff to go through. I like the model. One thing that came to my mind: What is the reason to use a enum for the ExposureTimeMode control instead of a bool? Are there further extensions planned? GenICam devices often have an additional mode "Once" that runs the control loop until converged and then snaps back to off/manual. But overall I believe we should get that it. Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com> Cheers, Stefan > -- > 2.39.2 >
On Thu, Nov 14, 2024 at 03:36:41PM +0100, Stefan Klug wrote: > Hi Paul, > > Thank you for the patch. > > On Wed, Nov 13, 2024 at 10:12:50PM +0900, Paul Elder wrote: > > Document the design and rationale for the AE-related controls. > > Also add documentation for the controls. > > > > Signed-off-by: Paul Elder <paul.elder@ideasonboard.com> > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org> > > > > --- > > Changes in v3: > > - merge the control documentation into the same document (including the > > patch) > > - this is because it was a bit unwieldy to put it in > > control_ids.cpp.in, now that it's used generically to generate > > control ids of all namespaces > > --- > > Documentation/design/ae.rst | 348 ++++++++++++++++++++++++++++++++++++ > > 1 file changed, 348 insertions(+) > > create mode 100644 Documentation/design/ae.rst > > > > diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst > > new file mode 100644 > > index 000000000000..f121afecea5a > > --- /dev/null > > +++ b/Documentation/design/ae.rst > > @@ -0,0 +1,348 @@ > > +.. SPDX-License-Identifier: CC-BY-SA-4.0 > > + > > +Design of Exposure and Gain controls > > +==================================== > > + > > +This document explains the design and rationale of the controls related to > > +exposure and gain. This includes the all-encompassing auto-exposure (AE), the > > +manual exposure control, and the manual gain control. > > + > > +Description of the problem > > +-------------------------- > > + > > +Sub controls > > +^^^^^^^^^^^^ > > + > > +There are more than one control that make up exposure: exposure, gain, and > > +aperture (though for now we will not consider aperture). We already had > > +individual controls for setting the values of manual exposure and manual gain, > > +but for switching between auto mode and manual mode we only had a high-level > > +boolean AeEnable control that would set *both* exposure and gain to auto mode > > +or manual mode; we had no way to set one to auto and the other to manual. > > + > > +So, we need to introduce two new controls to act as "levers" to indicate > > +individually for exposure and gain if the value would come from AEGC or if it > > +would come from the manual control value. > > + > > +Aperture priority > > +^^^^^^^^^^^^^^^^^ > > + > > +We eventually will support aperture, and so whatever our solution is for having > > +only some controls on auto and the others on manual needs to be extensible. > > + > > +Flickering when going from auto to manual > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > + > > +When a manual exposure or gain value is requested by the application, it costs > > +a few frames worth of time for them to take effect. This means that during a > > +transition from auto to manual, there would be flickering in the control values > > +and the transition won't be smooth. > > + > > +Take for instance the following flow, where we start on auto exposure (which > > +for the purposes of the example increments by 1 each frame) and we want to > > +switch seamlessly to manual exposure, which involves copying the exposure value > > +computed by the auto exposure algorithm: > > + > > +:: > > + > > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | > > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > > + > > + Mode requested: Auto Auto Auto Manual Manual Manual Manual > > + Exp requested: N/A N/A N/A 2 2 2 2 > > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 > > + > > + Mode used: Auto Auto Auto Auto Auto Manual Manual > > + Exp used: 0 1 2 3 4 2 2 > > + > > +As we can see, after frame N+2 completes, we copy the exposure value that was > > +used for frame N+2 (which was computed by AE algorithm), and queue that value > > +into request N+3 with manual mode on. However, as it takes two frames for the > > +exposure to be set, the exposure still changes since it is set by AE, and we > > +get a flicker in the exposure during the switch from auto to manual. > > + > > +A solution is to *not submit* any exposure value when manual mode is enabled, > > +and wait until the manual mode as been "applied" before copying the exposure > > +value: > > + > > +:: > > + > > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | > > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > > + > > + Mode requested: Auto Auto Auto Manual Manual Manual Manual > > + Exp requested: N/A N/A N/A None None None 5 > > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 > > + > > + Mode used: Auto Auto Auto Auto Auto Manual Manual > > + Exp used: 0 1 2 3 4 5 5 > > + > > +In practice, this works. However, libcamera has a policy where once a control > > +is submitted, its value is saved and does not need to be resubmitted (even > > +though this isn't implemented yet). So if the manual exposure value was set > > +while auto mode was on, in theory the value would be saved, so when manual mode > > +is enabled, the exposure value that was previously set would immediately be > > +used. Clearly this solution isn't correct, but it can serve as the basis for a > > +proper solution, with some more rigorous rules. > > + > > +Existing solutions > > +------------------ > > + > > +Raspberry Pi > > +^^^^^^^^^^^^ > > + > > +The raspberry pi IPA got around the lack of individual AeEnable controls for > > +exposure and gain by using magic values. When AeEnable was false, if one of the > > +manual control values was set to 0 then the value computed by AEGC would be > > +used for just that control. This solution isn't desirable, as it prevents > > +that magic value from being used as a valid value. > > + > > +To get around the flickering issue, when AeEnable was false, the raspberry pi > > +AEGC would simply stop updating the values to be set. As mentioned above, since > > +the value retention mechanism hasn't actually been implemented yet, this > > +worked. But, it's not a proper solution. > > + > > +Android > > +^^^^^^^ > > + > > +The Android HAL specification requires that exposure and gain (sensitivity) > > +must both be manual or both be auto. It cannot be that one is manual while the > > +other is auto, so they simply don't support sub controls. > > + > > +For the flickering issue, the Android HAL has an AeLock control. To transition > > +from auto to manual, the application would keep AE on auto, and turn on the > > +lock. Once the lock has propagated through, then the value can be copied from > > +the result into the request and the lock disabled and the mode set to manual. > > + > > +The problem with this solution is, besides the extra complexity, that it is > > +ambiguous what happens if there is a state transition from manual to locked > > +(even though it's a state transition that doesn't make sense). If locked is > > +defined to "use the last automatically computed values" then it could use the > > +values from the last time it AE was set to auto, or it would be undefined if AE > > +was never auto (eg. it started out as manual), or if AE is implemented to run > > +in the background it could just use the current values that are computed. If > > +locked is defined to "use the last value that was set" there would be less > > +ambiguity. Still, it's better if we can make it impossible to execute this > > +nonsensical state transition, and if we can reduce the complexity of having > > +this extra control or extra setting on a lever. > > + > > +Summary of goals > > +---------------- > > + > > + - We need a lock of some sort, to instruct the AEGC to not update output > > + results > > + > > + - We need manual modes, to override the values computed by the AEGC > > + > > + - We need to support seamless transitions from auto to manual, and do so > > + without flickering > > + > > + - We need custom minimum values for the manual controls; that is, no magic > > + values for enabling/disabling auto > > + > > + - All of these need to be done with AE sub-controls (exposure time, analogue > > + gain) and be extensible to aperture in the future > > + > > +Our solution > > +------------ > > + > > +A diagram of our solution: > > + > > +:: > > + > > + +----------------------------+-------------+------------------+-----------------+ > > + | INPUT | ALGORITHM | RESULT | OUTPUT | > > + +----------------------------+-------------+------------------+-----------------+ > > + > > + ExposureTimeMode ExposureTimeMode > > + ---------------------+----------------------------------------+-----------------> > > + 0: Auto | | > > + 1: Manual | V > > + | |\ > > + | | \ > > + | /----------------------------------> | 1| ExposureTime > > + | | +-------------+ exposure time | | --------------> > > + \--)--> | | --------------> | 0| > > + ExposureTime | | | | / > > + ------------------------+--> | | |/ > > + | | AeState > > + | AEGC | -----------------------------------> > > + AnalogueGain | | > > + ------------------------+--> | | |\ > > + | | | | \ > > + /--)--> | | --------------> | 0| AnalogueGain > > + | | +-------------+ analogue gain | | --------------> > > + | \----------------------------------> | 1| > > + | | / > > + | |/ > > + | ^ > > + AnalogueGainMode | | AnalogueGainMode > > + ---------------------+----------------------------------------+-----------------> > > + 0: Auto > > + 1: Manual > > + > > + > > +The diagram is divided in four sections horizontally: > > + > > + - Input: The values received from the request controls > > + > > + - Algorithm: The algorithm itself > > + > > + - Result: The values calculated by the algorithm > > + > > + - Output: The values that sent in result metadata and applied to the device > > + > > +The four input controls are divided between manual values (ExposureTime and > > +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The > > +former are the manual values, the latter control how they're applied. The two > > +modes are independent from each other, and each can take one of two values: > > + > > + - Auto (0): The AGC computes the value normally. The AGC result is applied > > + to the output. The manual value is ignored *and is not retained*. > > + > > + - Manual (1): The AGC uses the manual value internally. The corresponding > > + manual control from the request is applied to the output. The AGC result > > + is ignored. > > + > > +The AeState control reports the state of the unified AEGC block. If both > > +ExposureTimeMode and AnalogueGainMode are set to disabled then it will report > > +Idle. If at least one of the two is set to auto, then AeState will report > > +if the AEGC has Converged or not (Searching). This control replaces the old > > +AeLocked control, as it was insufficient for reporting the AE state. > > + > > +There is a caveat to the disabled mode: the manual control value is not > > +retained if it is set during auto mode. This means that if the disabled mode is > > +entered without also setting the manual value, then it will enter a state > > +similar to "locked", where the last automatically computed value while the mode > > +was auto will be used. Once the manual value is set, then that will be used and > > +retained as usual. > > + > > +This simulates an auto -> locked -> manual or auto -> manual state transition, > > +and makes it impossible to do the nonsensical manual -> locked state > > +transition. > > + > > +We specifically do not have a "master AE control" like the old AeEnable. This > > +is because we have the individual mode controls, and if we had a master AE > > +control it would be a "control that sets other controls", which could easily > > +get out of control. > > + > > +With this solution, the earlier example would become: > > + > > +:: > > + > > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > > + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10| > > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > > + Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab > > + Exp requested: N/A None None None None 10 None 10 10 > > + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10 > > + > > + Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab > > + Exp used: 2 3 4 5 5 5 5 10 10 > > + > > +This example is extended by a few frames to exhibit the simulated "locked" > > +state. At frame N+5 the application has confirmed that the auto mode has been > > +disabled, but does not provide a manual value until request N+7. Thus, the > > +value that is used in requests N+5 and N+6 (where the mode is disabled), comes > > +from the last value that was used when the mode was auto, which comes from > > +frame N+4. > > + > > +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for > > +the exposure to be applied. N+8 does not supply a manual value, but the last > > +supplied value is retained, so a manual value of 10 is still used and set in > > +frame N+10. > > + > > +Although this behavior is the same as what we had with waiting for the manual > > +mode to propagate (in the section "Description of the problem"), this time it > > +is correct as we have defined specifically that if a manual value was specified > > +while the mode was auto, it will not be retained. > > + > > +Description of the controls > > +=========================== > > + > > + libcamera offers the following controls related to exposure and gain: > > + > > + - AnalogueGain > > + > > + - AnalogueGainMode > > + > > + - ExposureTime > > + > > + - ExposureTimeMode > > + > > + - AeState > > + > > + Auto-exposure and auto-gain can be enabled and disabled separately using the > > + ExposureTimeMode and AnalogueGainMode controls respectively. There is no > > + overarching AeEnable control. > > + > > + For each of exposure and gain, we can model it with three states: auto, > > + locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only > > + have two values, as the locked state is simulated. > > + > > + :: > > + > > + /---------------------------------\ > > + | | > > + V | > > + +--------+ +--------+ > > + | | ---------------------> | | > > + | Auto | +--------+ | Manual | > > + | | ----> | Locked | ----> | | > > + +--------+ +--------+ +--------+ > > + ^ | > > + | | > > + \----------------/ > > + > > + Notice from the state diagram that locked to manual is a one-way state > > + change, as the reverse direction is nonsensical (see the design document for > > + more details on this topic). > > + > > + The exposure/gain is in the Auto state when > > + ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value > > + that is computed by the AE algorithm is applied to the image sensor. Any > > + value that is supplied in the ExposureTime/AnalogueGain control is ignored > > + and is not retained. > > + > > + If ExposureTimeMode/AnalogueGainMode is set to Manual, it can put us in > > + either the Locked or Manual state. The difference is in if > > + ExposureTime/AnalogueGain has been supplied. If it has not yet been > > + supplied, then we are in the Locked state. If it has been supplied, then we > > + are in the Manual state. > > + > > + In both the Locked state and the Manual state the exposure/gain value does > > + not come from the AE algorithm. In the Locked state the value comes from the > > + last value computed by the AE algorithm while the state was Auto, or if the > > + state was never Auto (e.g. we started in Locked, or the camera doesn't > > + support Auto), then the value should be a best-effort default value. In the > > + Manual state the value comes from the value supplied in the > > + ExposureTime/AnalogueGain control. > > + > > + To transition from the Locked state to the Manual state, a value needs to be > > + submitted in ExposureTime/AnalogueGain. Once the state has transitioned to > > + Manual, then this value will be retained, and doesn't need to be resubmitted > > + if it doesn't change. > > + > > + To transition to the Auto state, simply set > > + ExposureTimeMode/AnalogueGainMode to Auto. > > + > > + > > + The AeState metadata reports the state of the AE algorithm. As AE cannot > > + compute exposure and gain separately, the state of the AE component is > > + unified. There are three states: Idle, Searching, and Converged. > > + > > + The state shall be Idle if both ExposureTimeMode and AnalogueGainMode > > + are set to Manual. If the camera only supports one of the two controls, > > + then the state shall be Idle if that one control is set to Manual. If > > + the camera does not support Manual for at least one of the two controls, > > + then the state will never be Idle, as AE will always be running. > > + > > + The state shall be Searching if at least one of exposure or gain calculated > > + by the AE algorithm is used (that is, at least one of the two modes is Auto), > > + *and* the value(s) have not converged yet. > > + > > + The state shall be Converged if at least one of exposure or gain calculated > > + by the AE algorithm is used (that is, at least one of the two modes is Auto), > > + *and* the value(s) have converged. > > That is a lot of stuff to go through. I like the model. One thing that \o/ > came to my mind: What is the reason to use a enum for the ExposureTimeMode > control instead of a bool? Are there further extensions planned? GenICam tbh I can't recall any specific reason. Maybe that it's more explicit than AutoExposureTime/AutoAnalogueGain == true/false? > devices often have an additional mode "Once" that runs the control loop > until converged and then snaps back to off/manual. Oh neat! Maybe in the future... > > But overall I believe we should get that it. > Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com> Thanks, Paul
Hi Paul, Thank you for the patch. On Wed, Nov 13, 2024 at 10:12:50PM +0900, Paul Elder wrote: > Document the design and rationale for the AE-related controls. > Also add documentation for the controls. That's a great document, thank you. I'm getting a few build errors though: [518/519] Generating Documentation/documentation with a custom command FAILED: Documentation/html /usr/bin/sphinx-build -D release=v0.3.2+123-1f1977af -q -W -b html Documentation Documentation/html Documentation/design/ae.rst:290: WARNING: Literal block ends without a blank line; unexpected unindent. [docutils] Documentation/design/ae.rst:295: ERROR: Unexpected indentation. [docutils] Documentation/design/ae.rst: WARNING: document isn't included in any toctree > Signed-off-by: Paul Elder <paul.elder@ideasonboard.com> > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org> > > --- > Changes in v3: > - merge the control documentation into the same document (including the > patch) > - this is because it was a bit unwieldy to put it in > control_ids.cpp.in, now that it's used generically to generate > control ids of all namespaces > --- > Documentation/design/ae.rst | 348 ++++++++++++++++++++++++++++++++++++ Don't forget the entry in Documentation/meson.build, in addition to wiring this in the toc tree. > 1 file changed, 348 insertions(+) > create mode 100644 Documentation/design/ae.rst > > diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst > new file mode 100644 > index 000000000000..f121afecea5a > --- /dev/null > +++ b/Documentation/design/ae.rst > @@ -0,0 +1,348 @@ > +.. SPDX-License-Identifier: CC-BY-SA-4.0 > + > +Design of Exposure and Gain controls > +==================================== > + > +This document explains the design and rationale of the controls related to > +exposure and gain. This includes the all-encompassing auto-exposure (AE), the > +manual exposure control, and the manual gain control. > + > +Description of the problem > +-------------------------- > + > +Sub controls > +^^^^^^^^^^^^ > + > +There are more than one control that make up exposure: exposure, gain, and We need to standardize vocabulary. I've sent '[PATCH v2] libcamera: Rename "shutter speed" to "exposure time"' that addresses the low hanging fruit of consistently using "exposure time" when we talk about, well, exposure time. Here that would be There are more than one control that make up exposure: exposure time, gain, and aperture (...). (The first use of "exposure" isn't correct I think, at least according to https://en.wikipedia.org/wiki/Exposure_(photography). I'm not sure what a better term would be. We could also have our own definition of "exposure".) Could you rename "exposure" to "exposure time" through the series when it refers to the exposure time ? We will address the remaining usage of "exposure" separately as I don't want to delay the AE rework. > +aperture (though for now we will not consider aperture). We already had > +individual controls for setting the values of manual exposure and manual gain, > +but for switching between auto mode and manual mode we only had a high-level > +boolean AeEnable control that would set *both* exposure and gain to auto mode > +or manual mode; we had no way to set one to auto and the other to manual. > + > +So, we need to introduce two new controls to act as "levers" to indicate > +individually for exposure and gain if the value would come from AEGC or if it > +would come from the manual control value. > + > +Aperture priority > +^^^^^^^^^^^^^^^^^ > + > +We eventually will support aperture, and so whatever our solution is for having Maybe s/will/may need to/ > +only some controls on auto and the others on manual needs to be extensible. > + > +Flickering when going from auto to manual > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + > +When a manual exposure or gain value is requested by the application, it costs > +a few frames worth of time for them to take effect. This means that during a > +transition from auto to manual, there would be flickering in the control values > +and the transition won't be smooth. > + > +Take for instance the following flow, where we start on auto exposure (which > +for the purposes of the example increments by 1 each frame) and we want to > +switch seamlessly to manual exposure, which involves copying the exposure value > +computed by the auto exposure algorithm: > + > +:: > + > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + > + Mode requested: Auto Auto Auto Manual Manual Manual Manual > + Exp requested: N/A N/A N/A 2 2 2 2 > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 > + > + Mode used: Auto Auto Auto Auto Auto Manual Manual > + Exp used: 0 1 2 3 4 2 2 > + > +As we can see, after frame N+2 completes, we copy the exposure value that was > +used for frame N+2 (which was computed by AE algorithm), and queue that value > +into request N+3 with manual mode on. However, as it takes two frames for the > +exposure to be set, the exposure still changes since it is set by AE, and we > +get a flicker in the exposure during the switch from auto to manual. > + > +A solution is to *not submit* any exposure value when manual mode is enabled, > +and wait until the manual mode as been "applied" before copying the exposure > +value: > + > +:: > + > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + > + Mode requested: Auto Auto Auto Manual Manual Manual Manual > + Exp requested: N/A N/A N/A None None None 5 > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 > + > + Mode used: Auto Auto Auto Auto Auto Manual Manual > + Exp used: 0 1 2 3 4 5 5 > + > +In practice, this works. However, libcamera has a policy where once a control > +is submitted, its value is saved and does not need to be resubmitted (even > +though this isn't implemented yet). So if the manual exposure value was set Is the "not implemented yet" part universally true ? And "yet" implies we plan to implement it. I'd drop the part between parentheses. s/So if/If/ (as there's another "so" further in the sentence) > +while auto mode was on, in theory the value would be saved, so when manual mode > +is enabled, the exposure value that was previously set would immediately be > +used. Clearly this solution isn't correct, but it can serve as the basis for a > +proper solution, with some more rigorous rules. > + > +Existing solutions > +------------------ > + > +Raspberry Pi > +^^^^^^^^^^^^ > + > +The raspberry pi IPA got around the lack of individual AeEnable controls for s/raspberry pi/Raspberry Pi/ > +exposure and gain by using magic values. When AeEnable was false, if one of the Maybe use the present tense (s/was/is/) here as the section is named "Existing solutions" ? > +manual control values was set to 0 then the value computed by AEGC would be > +used for just that control. This solution isn't desirable, as it prevents > +that magic value from being used as a valid value. > + > +To get around the flickering issue, when AeEnable was false, the raspberry pi s/raspberry pi/Raspberry Pi/ > +AEGC would simply stop updating the values to be set. As mentioned above, since > +the value retention mechanism hasn't actually been implemented yet, this Ah, I see why you need the "not implemented yet" above. Maybe To get around the flickering issue, when AeEnable is false, the Raspberry Pi AEGC simply stops updating the values to be set, without restoring the previously set manual exposure time and gain. This works, but is not a proper solution. > +worked. But, it's not a proper solution. > + > +Android > +^^^^^^^ > + > +The Android HAL specification requires that exposure and gain (sensitivity) > +must both be manual or both be auto. It cannot be that one is manual while the > +other is auto, so they simply don't support sub controls. > + > +For the flickering issue, the Android HAL has an AeLock control. To transition > +from auto to manual, the application would keep AE on auto, and turn on the > +lock. Once the lock has propagated through, then the value can be copied from > +the result into the request and the lock disabled and the mode set to manual. > + > +The problem with this solution is, besides the extra complexity, that it is > +ambiguous what happens if there is a state transition from manual to locked > +(even though it's a state transition that doesn't make sense). If locked is > +defined to "use the last automatically computed values" then it could use the > +values from the last time it AE was set to auto, or it would be undefined if AE > +was never auto (eg. it started out as manual), or if AE is implemented to run > +in the background it could just use the current values that are computed. If > +locked is defined to "use the last value that was set" there would be less > +ambiguity. Still, it's better if we can make it impossible to execute this > +nonsensical state transition, and if we can reduce the complexity of having > +this extra control or extra setting on a lever. > + > +Summary of goals > +---------------- > + > + - We need a lock of some sort, to instruct the AEGC to not update output > + results > + > + - We need manual modes, to override the values computed by the AEGC > + > + - We need to support seamless transitions from auto to manual, and do so > + without flickering > + > + - We need custom minimum values for the manual controls; that is, no magic > + values for enabling/disabling auto > + > + - All of these need to be done with AE sub-controls (exposure time, analogue > + gain) and be extensible to aperture in the future You can lower the indentation here. Same below for other lists. > + > +Our solution > +------------ > + > +A diagram of our solution: > + > +:: > + > + +----------------------------+-------------+------------------+-----------------+ > + | INPUT | ALGORITHM | RESULT | OUTPUT | > + +----------------------------+-------------+------------------+-----------------+ > + > + ExposureTimeMode ExposureTimeMode > + ---------------------+----------------------------------------+-----------------> > + 0: Auto | | > + 1: Manual | V There seems to be a small formatting issue here. > + | |\ > + | | \ > + | /----------------------------------> | 1| ExposureTime > + | | +-------------+ exposure time | | --------------> > + \--)--> | | --------------> | 0| > + ExposureTime | | | | / > + ------------------------+--> | | |/ > + | | AeState > + | AEGC | -----------------------------------> > + AnalogueGain | | > + ------------------------+--> | | |\ > + | | | | \ > + /--)--> | | --------------> | 0| AnalogueGain > + | | +-------------+ analogue gain | | --------------> > + | \----------------------------------> | 1| > + | | / > + | |/ > + | ^ > + AnalogueGainMode | | AnalogueGainMode > + ---------------------+----------------------------------------+-----------------> > + 0: Auto > + 1: Manual > + > + > +The diagram is divided in four sections horizontally: > + > + - Input: The values received from the request controls > + > + - Algorithm: The algorithm itself > + > + - Result: The values calculated by the algorithm > + > + - Output: The values that sent in result metadata and applied to the device "that sent" ? > + > +The four input controls are divided between manual values (ExposureTime and > +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The > +former are the manual values, the latter control how they're applied. The two > +modes are independent from each other, and each can take one of two values: > + > + - Auto (0): The AGC computes the value normally. The AGC result is applied > + to the output. The manual value is ignored *and is not retained*. > + > + - Manual (1): The AGC uses the manual value internally. The corresponding > + manual control from the request is applied to the output. The AGC result > + is ignored. > + > +The AeState control reports the state of the unified AEGC block. If both > +ExposureTimeMode and AnalogueGainMode are set to disabled then it will report Here you mention "disabled" but above the controls are defined as "auto" or "manual". Should "disabled" be replaced with "manual", here and below ? > +Idle. If at least one of the two is set to auto, then AeState will report > +if the AEGC has Converged or not (Searching). This control replaces the old > +AeLocked control, as it was insufficient for reporting the AE state. > + > +There is a caveat to the disabled mode: the manual control value is not > +retained if it is set during auto mode. This means that if the disabled mode is > +entered without also setting the manual value, then it will enter a state > +similar to "locked", where the last automatically computed value while the mode > +was auto will be used. Once the manual value is set, then that will be used and > +retained as usual. > + > +This simulates an auto -> locked -> manual or auto -> manual state transition, > +and makes it impossible to do the nonsensical manual -> locked state > +transition. > + > +We specifically do not have a "master AE control" like the old AeEnable. This > +is because we have the individual mode controls, and if we had a master AE > +control it would be a "control that sets other controls", which could easily > +get out of control. No pun intended ? :-) > + > +With this solution, the earlier example would become: > + > +:: > + > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10| > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ > + Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab > + Exp requested: N/A None None None None 10 None 10 10 > + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10 I think the last two N+10 should be N+11 and N+12. > + > + Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab > + Exp used: 2 3 4 5 5 5 5 10 10 > + > +This example is extended by a few frames to exhibit the simulated "locked" > +state. At frame N+5 the application has confirmed that the auto mode has been > +disabled, but does not provide a manual value until request N+7. Thus, the > +value that is used in requests N+5 and N+6 (where the mode is disabled), comes > +from the last value that was used when the mode was auto, which comes from > +frame N+4. > + > +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for > +the exposure to be applied. N+8 does not supply a manual value, but the last > +supplied value is retained, so a manual value of 10 is still used and set in > +frame N+10. > + > +Although this behavior is the same as what we had with waiting for the manual > +mode to propagate (in the section "Description of the problem"), this time it > +is correct as we have defined specifically that if a manual value was specified > +while the mode was auto, it will not be retained. > + > +Description of the controls > +=========================== This uses the same heading level as the top-level title, is that intentional ? > + > + libcamera offers the following controls related to exposure and gain: The indentation is wrong for the whole block, everything should be on the left-most column. > + > + - AnalogueGain > + > + - AnalogueGainMode > + > + - ExposureTime > + > + - ExposureTimeMode > + > + - AeState > + > + Auto-exposure and auto-gain can be enabled and disabled separately using the > + ExposureTimeMode and AnalogueGainMode controls respectively. There is no > + overarching AeEnable control. > + > + For each of exposure and gain, we can model it with three states: auto, > + locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only > + have two values, as the locked state is simulated. > + > + :: > + > + /---------------------------------\ > + | | > + V | > + +--------+ +--------+ > + | | ---------------------> | | > + | Auto | +--------+ | Manual | > + | | ----> | Locked | ----> | | > + +--------+ +--------+ +--------+ > + ^ | > + | | > + \----------------/ > + > + Notice from the state diagram that locked to manual is a one-way state > + change, as the reverse direction is nonsensical (see the design document for > + more details on this topic). Isn't this the design document ? > + > + The exposure/gain is in the Auto state when > + ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value > + that is computed by the AE algorithm is applied to the image sensor. Any > + value that is supplied in the ExposureTime/AnalogueGain control is ignored > + and is not retained. I expect it will be easier to implement this by accumulating the manual exposure time and analog gain controls as we do for all other controls, and override the internal storage with the last value computed by the AGC algorithm when transitioning to manual. That's an implementation detail, the behaviour will be the same from a user point of view, but if we expect AGC to be implemented that way, it may be nicer to also describe it the same way here. > + > + If ExposureTimeMode/AnalogueGainMode is set to Manual, it can put us in > + either the Locked or Manual state. The difference is in if > + ExposureTime/AnalogueGain has been supplied. If it has not yet been > + supplied, then we are in the Locked state. If it has been supplied, then we > + are in the Manual state. > + > + In both the Locked state and the Manual state the exposure/gain value does > + not come from the AE algorithm. In the Locked state the value comes from the > + last value computed by the AE algorithm while the state was Auto, or if the > + state was never Auto (e.g. we started in Locked, or the camera doesn't > + support Auto), then the value should be a best-effort default value. In the Can we start in the locked state, as it's not a real state ? I'd rather say that if the user starts the camera in manual mode without setting a corresponding manual value then a best-effort default value is used. > + Manual state the value comes from the value supplied in the > + ExposureTime/AnalogueGain control. > + > + To transition from the Locked state to the Manual state, a value needs to be > + submitted in ExposureTime/AnalogueGain. Once the state has transitioned to > + Manual, then this value will be retained, and doesn't need to be resubmitted > + if it doesn't change. I understand where all of this come from, but I think the documentation could be simplified if we explained the behaviour as copying the last value computed by the algorithm to the manual value when transitioning from auto to manual (and also mentioning of course that if the same request provides a manual value, it will override the value copied from the algorithm). That way we could avoid introducing a locked pseudo-state, and I think the result would be easier to understand for the reader. Please let me know what you think, I value your opinion. > + > + To transition to the Auto state, simply set > + ExposureTimeMode/AnalogueGainMode to Auto. > + > + Extra blank line. > + The AeState metadata reports the state of the AE algorithm. As AE cannot > + compute exposure and gain separately, the state of the AE component is > + unified. There are three states: Idle, Searching, and Converged. > + > + The state shall be Idle if both ExposureTimeMode and AnalogueGainMode > + are set to Manual. If the camera only supports one of the two controls, > + then the state shall be Idle if that one control is set to Manual. If > + the camera does not support Manual for at least one of the two controls, > + then the state will never be Idle, as AE will always be running. > + > + The state shall be Searching if at least one of exposure or gain calculated > + by the AE algorithm is used (that is, at least one of the two modes is Auto), > + *and* the value(s) have not converged yet. > + > + The state shall be Converged if at least one of exposure or gain calculated > + by the AE algorithm is used (that is, at least one of the two modes is Auto), > + *and* the value(s) have converged.
diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst new file mode 100644 index 000000000000..f121afecea5a --- /dev/null +++ b/Documentation/design/ae.rst @@ -0,0 +1,348 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +Design of Exposure and Gain controls +==================================== + +This document explains the design and rationale of the controls related to +exposure and gain. This includes the all-encompassing auto-exposure (AE), the +manual exposure control, and the manual gain control. + +Description of the problem +-------------------------- + +Sub controls +^^^^^^^^^^^^ + +There are more than one control that make up exposure: exposure, gain, and +aperture (though for now we will not consider aperture). We already had +individual controls for setting the values of manual exposure and manual gain, +but for switching between auto mode and manual mode we only had a high-level +boolean AeEnable control that would set *both* exposure and gain to auto mode +or manual mode; we had no way to set one to auto and the other to manual. + +So, we need to introduce two new controls to act as "levers" to indicate +individually for exposure and gain if the value would come from AEGC or if it +would come from the manual control value. + +Aperture priority +^^^^^^^^^^^^^^^^^ + +We eventually will support aperture, and so whatever our solution is for having +only some controls on auto and the others on manual needs to be extensible. + +Flickering when going from auto to manual +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a manual exposure or gain value is requested by the application, it costs +a few frames worth of time for them to take effect. This means that during a +transition from auto to manual, there would be flickering in the control values +and the transition won't be smooth. + +Take for instance the following flow, where we start on auto exposure (which +for the purposes of the example increments by 1 each frame) and we want to +switch seamlessly to manual exposure, which involves copying the exposure value +computed by the auto exposure algorithm: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A 2 2 2 2 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 2 2 + +As we can see, after frame N+2 completes, we copy the exposure value that was +used for frame N+2 (which was computed by AE algorithm), and queue that value +into request N+3 with manual mode on. However, as it takes two frames for the +exposure to be set, the exposure still changes since it is set by AE, and we +get a flicker in the exposure during the switch from auto to manual. + +A solution is to *not submit* any exposure value when manual mode is enabled, +and wait until the manual mode as been "applied" before copying the exposure +value: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A None None None 5 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 5 5 + +In practice, this works. However, libcamera has a policy where once a control +is submitted, its value is saved and does not need to be resubmitted (even +though this isn't implemented yet). So if the manual exposure value was set +while auto mode was on, in theory the value would be saved, so when manual mode +is enabled, the exposure value that was previously set would immediately be +used. Clearly this solution isn't correct, but it can serve as the basis for a +proper solution, with some more rigorous rules. + +Existing solutions +------------------ + +Raspberry Pi +^^^^^^^^^^^^ + +The raspberry pi IPA got around the lack of individual AeEnable controls for +exposure and gain by using magic values. When AeEnable was false, if one of the +manual control values was set to 0 then the value computed by AEGC would be +used for just that control. This solution isn't desirable, as it prevents +that magic value from being used as a valid value. + +To get around the flickering issue, when AeEnable was false, the raspberry pi +AEGC would simply stop updating the values to be set. As mentioned above, since +the value retention mechanism hasn't actually been implemented yet, this +worked. But, it's not a proper solution. + +Android +^^^^^^^ + +The Android HAL specification requires that exposure and gain (sensitivity) +must both be manual or both be auto. It cannot be that one is manual while the +other is auto, so they simply don't support sub controls. + +For the flickering issue, the Android HAL has an AeLock control. To transition +from auto to manual, the application would keep AE on auto, and turn on the +lock. Once the lock has propagated through, then the value can be copied from +the result into the request and the lock disabled and the mode set to manual. + +The problem with this solution is, besides the extra complexity, that it is +ambiguous what happens if there is a state transition from manual to locked +(even though it's a state transition that doesn't make sense). If locked is +defined to "use the last automatically computed values" then it could use the +values from the last time it AE was set to auto, or it would be undefined if AE +was never auto (eg. it started out as manual), or if AE is implemented to run +in the background it could just use the current values that are computed. If +locked is defined to "use the last value that was set" there would be less +ambiguity. Still, it's better if we can make it impossible to execute this +nonsensical state transition, and if we can reduce the complexity of having +this extra control or extra setting on a lever. + +Summary of goals +---------------- + + - We need a lock of some sort, to instruct the AEGC to not update output + results + + - We need manual modes, to override the values computed by the AEGC + + - We need to support seamless transitions from auto to manual, and do so + without flickering + + - We need custom minimum values for the manual controls; that is, no magic + values for enabling/disabling auto + + - All of these need to be done with AE sub-controls (exposure time, analogue + gain) and be extensible to aperture in the future + +Our solution +------------ + +A diagram of our solution: + +:: + + +----------------------------+-------------+------------------+-----------------+ + | INPUT | ALGORITHM | RESULT | OUTPUT | + +----------------------------+-------------+------------------+-----------------+ + + ExposureTimeMode ExposureTimeMode + ---------------------+----------------------------------------+-----------------> + 0: Auto | | + 1: Manual | V + | |\ + | | \ + | /----------------------------------> | 1| ExposureTime + | | +-------------+ exposure time | | --------------> + \--)--> | | --------------> | 0| + ExposureTime | | | | / + ------------------------+--> | | |/ + | | AeState + | AEGC | -----------------------------------> + AnalogueGain | | + ------------------------+--> | | |\ + | | | | \ + /--)--> | | --------------> | 0| AnalogueGain + | | +-------------+ analogue gain | | --------------> + | \----------------------------------> | 1| + | | / + | |/ + | ^ + AnalogueGainMode | | AnalogueGainMode + ---------------------+----------------------------------------+-----------------> + 0: Auto + 1: Manual + + +The diagram is divided in four sections horizontally: + + - Input: The values received from the request controls + + - Algorithm: The algorithm itself + + - Result: The values calculated by the algorithm + + - Output: The values that sent in result metadata and applied to the device + +The four input controls are divided between manual values (ExposureTime and +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The +former are the manual values, the latter control how they're applied. The two +modes are independent from each other, and each can take one of two values: + + - Auto (0): The AGC computes the value normally. The AGC result is applied + to the output. The manual value is ignored *and is not retained*. + + - Manual (1): The AGC uses the manual value internally. The corresponding + manual control from the request is applied to the output. The AGC result + is ignored. + +The AeState control reports the state of the unified AEGC block. If both +ExposureTimeMode and AnalogueGainMode are set to disabled then it will report +Idle. If at least one of the two is set to auto, then AeState will report +if the AEGC has Converged or not (Searching). This control replaces the old +AeLocked control, as it was insufficient for reporting the AE state. + +There is a caveat to the disabled mode: the manual control value is not +retained if it is set during auto mode. This means that if the disabled mode is +entered without also setting the manual value, then it will enter a state +similar to "locked", where the last automatically computed value while the mode +was auto will be used. Once the manual value is set, then that will be used and +retained as usual. + +This simulates an auto -> locked -> manual or auto -> manual state transition, +and makes it impossible to do the nonsensical manual -> locked state +transition. + +We specifically do not have a "master AE control" like the old AeEnable. This +is because we have the individual mode controls, and if we had a master AE +control it would be a "control that sets other controls", which could easily +get out of control. + +With this solution, the earlier example would become: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10| + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab + Exp requested: N/A None None None None 10 None 10 10 + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10 + + Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab + Exp used: 2 3 4 5 5 5 5 10 10 + +This example is extended by a few frames to exhibit the simulated "locked" +state. At frame N+5 the application has confirmed that the auto mode has been +disabled, but does not provide a manual value until request N+7. Thus, the +value that is used in requests N+5 and N+6 (where the mode is disabled), comes +from the last value that was used when the mode was auto, which comes from +frame N+4. + +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for +the exposure to be applied. N+8 does not supply a manual value, but the last +supplied value is retained, so a manual value of 10 is still used and set in +frame N+10. + +Although this behavior is the same as what we had with waiting for the manual +mode to propagate (in the section "Description of the problem"), this time it +is correct as we have defined specifically that if a manual value was specified +while the mode was auto, it will not be retained. + +Description of the controls +=========================== + + libcamera offers the following controls related to exposure and gain: + + - AnalogueGain + + - AnalogueGainMode + + - ExposureTime + + - ExposureTimeMode + + - AeState + + Auto-exposure and auto-gain can be enabled and disabled separately using the + ExposureTimeMode and AnalogueGainMode controls respectively. There is no + overarching AeEnable control. + + For each of exposure and gain, we can model it with three states: auto, + locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only + have two values, as the locked state is simulated. + + :: + + /---------------------------------\ + | | + V | + +--------+ +--------+ + | | ---------------------> | | + | Auto | +--------+ | Manual | + | | ----> | Locked | ----> | | + +--------+ +--------+ +--------+ + ^ | + | | + \----------------/ + + Notice from the state diagram that locked to manual is a one-way state + change, as the reverse direction is nonsensical (see the design document for + more details on this topic). + + The exposure/gain is in the Auto state when + ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value + that is computed by the AE algorithm is applied to the image sensor. Any + value that is supplied in the ExposureTime/AnalogueGain control is ignored + and is not retained. + + If ExposureTimeMode/AnalogueGainMode is set to Manual, it can put us in + either the Locked or Manual state. The difference is in if + ExposureTime/AnalogueGain has been supplied. If it has not yet been + supplied, then we are in the Locked state. If it has been supplied, then we + are in the Manual state. + + In both the Locked state and the Manual state the exposure/gain value does + not come from the AE algorithm. In the Locked state the value comes from the + last value computed by the AE algorithm while the state was Auto, or if the + state was never Auto (e.g. we started in Locked, or the camera doesn't + support Auto), then the value should be a best-effort default value. In the + Manual state the value comes from the value supplied in the + ExposureTime/AnalogueGain control. + + To transition from the Locked state to the Manual state, a value needs to be + submitted in ExposureTime/AnalogueGain. Once the state has transitioned to + Manual, then this value will be retained, and doesn't need to be resubmitted + if it doesn't change. + + To transition to the Auto state, simply set + ExposureTimeMode/AnalogueGainMode to Auto. + + + The AeState metadata reports the state of the AE algorithm. As AE cannot + compute exposure and gain separately, the state of the AE component is + unified. There are three states: Idle, Searching, and Converged. + + The state shall be Idle if both ExposureTimeMode and AnalogueGainMode + are set to Manual. If the camera only supports one of the two controls, + then the state shall be Idle if that one control is set to Manual. If + the camera does not support Manual for at least one of the two controls, + then the state will never be Idle, as AE will always be running. + + The state shall be Searching if at least one of exposure or gain calculated + by the AE algorithm is used (that is, at least one of the two modes is Auto), + *and* the value(s) have not converged yet. + + The state shall be Converged if at least one of exposure or gain calculated + by the AE algorithm is used (that is, at least one of the two modes is Auto), + *and* the value(s) have converged.