From patchwork Thu Jan 9 00:09:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 22475 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E4CD5C3311 for ; Thu, 9 Jan 2025 00:10:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0CDE868500; Thu, 9 Jan 2025 01:10:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="r4hXAiGL"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 94A3068500 for ; Thu, 9 Jan 2025 01:09:58 +0100 (CET) Received: from pyrite.hamster-moth.ts.net (unknown [IPv6:2604:2d80:9e93:ad00:3d82:7e4f:ab9b:8a]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AF3916F3; Thu, 9 Jan 2025 01:09:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736381345; bh=bG87KFTuvfIXVf0ytUBkGQfGTrJI3PEV9WoDQfPcbj8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r4hXAiGLrexnwg4EGm2h9QgeBIDqtMtiWYwceZQfeM8TGWo2k2vF5v92eKeqA8b5y xNGVu/Aoe4mj5nxzae6kGtzDsq4eDW8OlONrIsAqFDIh0GUMGvIlUW3ohIaCNInxSG KnCV3fxBwquVh0+IfVBdWuQKv9QfmennQJbD7DJo= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , laurent.pinchart@ideasonboard.com, stefan.klug@ideasonboard.com, Jacopo Mondi Subject: [PATCH v6 02/12] Documentation: design: ae: Document the design for AE controls Date: Wed, 8 Jan 2025 18:09:32 -0600 Message-Id: <20250109000942.1616565-3-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250109000942.1616565-1-paul.elder@ideasonboard.com> References: <20250109000942.1616565-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Document the design and rationale for the AE-related controls. Also add documentation for the controls. Signed-off-by: Paul Elder Signed-off-by: Jacopo Mondi Reviewed-by: Stefan Klug Reviewed-by: Laurent Pinchart --- No change in v6 Changes in v5: - improve wording Changes in v4: - fix wording - fix compilation - remove locked state in explanation Changes in v3: - merge the control documentation into the same document (including the patch) - this is because it was a bit unwieldy to put it in control_ids.cpp.in, now that it's used generically to generate control ids of all namespaces --- Documentation/design/ae.rst | 320 ++++++++++++++++++++++++++++++++++++ Documentation/index.rst | 4 +- Documentation/meson.build | 1 + 3 files changed, 324 insertions(+), 1 deletion(-) create mode 100644 Documentation/design/ae.rst diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst new file mode 100644 index 000000000..f8b2c887c --- /dev/null +++ b/Documentation/design/ae.rst @@ -0,0 +1,320 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +Design of Exposure and Gain controls +==================================== + +This document explains the design and rationale of the controls related to +exposure and gain. This includes the all-encompassing auto-exposure (AE), the +manual exposure control, and the manual gain control. + +Description of the problem +-------------------------- + +Sub controls +^^^^^^^^^^^^ + +There are more than one control that make up total exposure: exposure time, +gain, and aperture (though for now we will not consider aperture). We already +had individual controls for setting the values of manual exposure and manual +gain, but for switching between auto mode and manual mode we only had a +high-level boolean AeEnable control that would set *both* exposure and gain to +auto mode or manual mode; we had no way to set one to auto and the other to +manual. + +So, we need to introduce two new controls to act as "levers" to indicate +individually for exposure and gain if the value would come from AEGC or if it +would come from the manual control value. + +Aperture priority +^^^^^^^^^^^^^^^^^ + +We eventually may need to support aperture, and so whatever our solution is for +having only some controls on auto and the others on manual needs to be +extensible. + +Flickering when going from auto to manual +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a manual exposure or gain value is requested by the application, it costs +a few frames worth of time for them to take effect. This means that during a +transition from auto to manual, there would be flickering in the control values +and the transition won't be smooth. + +Take for instance the following flow, where we start on auto exposure (which +for the purposes of the example increments by 1 each frame) and we want to +switch seamlessly to manual exposure, which involves copying the exposure value +computed by the auto exposure algorithm: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A 2 2 2 2 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 2 2 + +As we can see, after frame N+2 completes, we copy the exposure value that was +used for frame N+2 (which was computed by AE algorithm), and queue that value +into request N+3 with manual mode on. However, as it takes two frames for the +exposure to be set, the exposure still changes since it is set by AE, and we +get a flicker in the exposure during the switch from auto to manual. + +A solution is to *not submit* any exposure value when manual mode is enabled, +and wait until the manual mode as been "applied" before copying the exposure +value: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A None None None 5 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 5 5 + +In practice, this works. However, libcamera has a policy where once a control +is submitted, its value is saved and does not need to be resubmitted. If the +manual exposure value was set while auto mode was on, in theory the value would +be saved, so when manual mode is enabled, the exposure value that was +previously set would immediately be used. Clearly this solution isn't correct, +but it can serve as the basis for a proper solution, with some more rigorous +rules. + +Existing solutions +------------------ + +Raspberry Pi +^^^^^^^^^^^^ + +The Raspberry Pi IPA gets around the lack of individual AeEnable controls for +exposure and gain by using magic values. When AeEnable is false, if one of the +manual control values was set to 0 then the value computed by AEGC would be +used for just that control. This solution isn't desirable, as it prevents +that magic value from being used as a valid value. + +To get around the flickering issue, when AeEnable is false, the Raspberry Pi +AEGC simply stops updating the values to be set, without restoring the +previously set manual exposure time and gain. This works, but is not a proper +solution. + +Android +^^^^^^^ + +The Android HAL specification requires that exposure and gain (sensitivity) +must both be manual or both be auto. It cannot be that one is manual while the +other is auto, so they simply don't support sub controls. + +For the flickering issue, the Android HAL has an AeLock control. To transition +from auto to manual, the application would keep AE on auto, and turn on the +lock. Once the lock has propagated through, then the value can be copied from +the result into the request and the lock disabled and the mode set to manual. + +The problem with this solution is, besides the extra complexity, that it is +ambiguous what happens if there is a state transition from manual to locked +(even though it's a state transition that doesn't make sense). If locked is +defined to "use the last automatically computed values" then it could use the +values from the last time it AE was set to auto, or it would be undefined if AE +was never auto (eg. it started out as manual), or if AE is implemented to run +in the background it could just use the current values that are computed. If +locked is defined to "use the last value that was set" there would be less +ambiguity. Still, it's better if we can make it impossible to execute this +nonsensical state transition, and if we can reduce the complexity of having +this extra control or extra setting on a lever. + +Summary of goals +---------------- + +- We need a lock of some sort, to instruct the AEGC to not update output + results + +- We need manual modes, to override the values computed by the AEGC + +- We need to support seamless transitions from auto to manual, and do so + without flickering + +- We need custom minimum values for the manual controls; that is, no magic + values for enabling/disabling auto + +- All of these need to be done with AE sub-controls (exposure time, analogue + gain) and be extensible to aperture in the future + +Our solution +------------ + +A diagram of our solution: + +:: + + +----------------------------+-------------+------------------+-----------------+ + | INPUT | ALGORITHM | RESULT | OUTPUT | + +----------------------------+-------------+------------------+-----------------+ + + ExposureTimeMode ExposureTimeMode + ---------------------+----------------------------------------+-----------------> + 0: Auto | | + 1: Manual | V + | |\ + | | \ + | /----------------------------------> | 1| ExposureTime + | | +-------------+ exposure time | | --------------> + \--)--> | | --------------> | 0| + ExposureTime | | | | / + ------------------------+--> | | |/ + | | AeState + | AEGC | -----------------------------------> + AnalogueGain | | + ------------------------+--> | | |\ + | | | | \ + /--)--> | | --------------> | 0| AnalogueGain + | | +-------------+ analogue gain | | --------------> + | \----------------------------------> | 1| + | | / + | |/ + | ^ + AnalogueGainMode | | AnalogueGainMode + ---------------------+----------------------------------------+-----------------> + 0: Auto + 1: Manual + + +The diagram is divided in four sections horizontally: + +- Input: The values received from the request controls + +- Algorithm: The algorithm itself + +- Result: The values calculated by the algorithm + +- Output: The values reported in result metadata and applied to the device + +The four input controls are divided between manual values (ExposureTime and +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The +former are the manual values, the latter control how they're applied. The two +modes are independent from each other, and each can take one of two values: + + - Auto (0): The AGC computes the value normally. The AGC result is applied + to the output. The manual value is ignored *and is not retained*. + + - Manual (1): The AGC uses the manual value internally. The corresponding + manual control from the request is applied to the output. The AGC result + is ignored. + +The AeState control reports the state of the unified AEGC block. If both +ExposureTimeMode and AnalogueGainMode are set to manual then it will report +Idle. If at least one of the two is set to auto, then AeState will report +if the AEGC has Converged or not (Searching). This control replaces the old +AeLocked control, as it was insufficient for reporting the AE state. + +There is a caveat to manual mode: the manual control value is not retained if +it is set during auto mode. This means that if manual mode is entered without +also setting the manual value, then it will enter a state similar to "locked", +where the last automatically computed value while the mode was auto will be +used. Once the manual value is set, then that will be used and retained as +usual. + +This simulates an auto -> locked -> manual or auto -> manual state transition, +and makes it impossible to do the nonsensical manual -> locked state +transition. + +We specifically do not have a "master AE control" like the old AeEnable. This +is because we have the individual mode controls, and if we had a master AE +control it would be a "control that sets other controls", which could easily +get out of control. + +With this solution, the earlier example would become: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10| + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + Mode requested: Auto Manual Manual Manual Manual Manual Manual Manual Manual + Exp requested: N/A None None None None 10 None 10 10 + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+11 N+12 + + Mode used: Auto Auto Auto Manual Manual Manual Manual Manual Manual + Exp used: 2 3 4 5 5 5 5 10 10 + +This example is extended by a few frames to exhibit the simulated "locked" +state. At frame N+5 the application has confirmed that the manual mode has been +entered, but does not provide a manual value until request N+7. Thus, the value +that is used in requests N+5 and N+6 (where the mode is disabled), comes from +the last value that was used when the mode was auto, which comes from frame +N+4. + +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for +the exposure to be applied. N+8 does not supply a manual value, but the last +supplied value is retained, so a manual value of 10 is still used and set in +frame N+10. + +Although this behavior is the same as what we had with waiting for the manual +mode to propagate (in the section "Description of the problem"), this time it +is correct as we have defined specifically that if a manual value was specified +while the mode was auto, it will not be retained. + +Description of the controls +--------------------------- + +As described above, libcamera offers the following controls related to exposure +and gain: + +- AnalogueGain + +- AnalogueGainMode + +- ExposureTime + +- ExposureTimeMode + +- AeState + +Auto-exposure and auto-gain can be enabled and disabled separately using the +ExposureTimeMode and AnalogueGainMode controls respectively. There is no +overarching AeEnable control. + +When the respective mode is set to auto, the respective value that is computed +by the AEGC algorithm is applied to the image sensor. Any value that is +supplied in the manual ExposureTime/AnalogueGain control is ignored and not +retained. Another way to understand this is that when the mode transitions from +auto to manual, the internally stored control value is overwritten with the +last value computed by the auto algorithm. + +This means that when we transition from auto to manual without supplying a +manual control value, the last value that was set by the AEGC algorithm will +keep be used. This can be used to do a flickerless transition from auto to +manual as described earlier. If the camera started out in manual mode and no +corresponding value has been supplied yet, then a best-effort default value +shall be set. + +The manual control value can be set in the same request as setting the mode to +auto if the desired manual control value is already known. + +Transitioning from manual to auto shall be implicitly flickerless, as the AEGC +algorithms are expected to start running from the last manual value. + +The AeState metadata reports the state of the AE algorithm. As AE cannot +compute exposure and gain separately, the state of the AE component is +unified. There are three states: Idle, Searching, and Converged. + +The state shall be Idle if both ExposureTimeMode and AnalogueGainMode +are set to Manual. If the camera only supports one of the two controls, +then the state shall be Idle if that one control is set to Manual. If +the camera does not support Manual for at least one of the two controls, +then the state will never be Idle, as AE will always be running. + +The state shall be Searching if at least one of exposure or gain calculated +by the AE algorithm is used (that is, at least one of the two modes is Auto), +*and* the value(s) have not converged yet. + +The state shall be Converged if at least one of exposure or gain calculated +by the AE algorithm is used (that is, at least one of the two modes is Auto), +*and* the value(s) have converged. diff --git a/Documentation/index.rst b/Documentation/index.rst index bea406608..251112fbd 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -23,7 +23,9 @@ SoftwareISP Benchmarking Tracing guide + Design document: AE + .. toctree:: :hidden: - introduction \ No newline at end of file + introduction diff --git a/Documentation/meson.build b/Documentation/meson.build index 36ffae239..6158320e1 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -128,6 +128,7 @@ if sphinx.found() 'coding-style.rst', 'conf.py', 'contributing.rst', + 'design/ae.rst', 'documentation-contents.rst', 'environment_variables.rst', 'feature_requirements.rst',