[{"id":32992,"web_url":"https://patchwork.libcamera.org/comment/32992/","msgid":"<20250109214159.GA5235@pendragon.ideasonboard.com>","date":"2025-01-09T21:41:59","subject":"Re: [PATCH v6 02/12] Documentation: design: ae: Document the design\n\tfor AE controls","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Paul,\n\nThank you for the patch.\n\nOn Wed, Jan 08, 2025 at 06:09:32PM -0600, Paul Elder wrote:\n> Document the design and rationale for the AE-related controls.\n> Also add documentation for the controls.\n> \n> Signed-off-by: Paul Elder \n> Signed-off-by: Jacopo Mondi \n> Reviewed-by: Stefan Klug \n> Reviewed-by: Laurent Pinchart \n> \n> ---\n> No change in v6\n> \n> Changes in v5:\n> - improve wording\n> \n> Changes in v4:\n> - fix wording\n> - fix compilation\n> - remove locked state in explanation\n> \n> Changes in v3:\n> - merge the control documentation into the same document (including the\n> patch)\n> - this is because it was a bit unwieldy to put it in\n> control_ids.cpp.in, now that it's used generically to generate\n> control ids of all namespaces\n> ---\n> Documentation/design/ae.rst | 320 ++++++++++++++++++++++++++++++++++++\n> Documentation/index.rst | 4 +-\n> Documentation/meson.build | 1 +\n> 3 files changed, 324 insertions(+), 1 deletion(-)\n> create mode 100644 Documentation/design/ae.rst\n> \n> diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst\n> new file mode 100644\n> index 000000000..f8b2c887c\n> --- /dev/null\n> +++ b/Documentation/design/ae.rst\n> @@ -0,0 +1,320 @@\n> +.. SPDX-License-Identifier: CC-BY-SA-4.0\n> +\n> +Design of Exposure and Gain controls\n> +====================================\n> +\n> +This document explains the design and rationale of the controls related to\n> +exposure and gain. This includes the all-encompassing auto-exposure (AE), the\n> +manual exposure control, and the manual gain control.\n> +\n> +Description of the problem\n> +--------------------------\n> +\n> +Sub controls\n> +^^^^^^^^^^^^\n> +\n> +There are more than one control that make up total exposure: exposure time,\n> +gain, and aperture (though for now we will not consider aperture). We already\n> +had individual controls for setting the values of manual exposure and manual\n> +gain, but for switching between auto mode and manual mode we only had a\n> +high-level boolean AeEnable control that would set *both* exposure and gain to\n> +auto mode or manual mode; we had no way to set one to auto and the other to\n> +manual.\n> +\n> +So, we need to introduce two new controls to act as \"levers\" to indicate\n> +individually for exposure and gain if the value would come from AEGC or if it\n> +would come from the manual control value.\n> +\n> +Aperture priority\n> +^^^^^^^^^^^^^^^^^\n> +\n> +We eventually may need to support aperture, and so whatever our solution is for\n> +having only some controls on auto and the others on manual needs to be\n> +extensible.\n> +\n> +Flickering when going from auto to manual\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> +\n> +When a manual exposure or gain value is requested by the application, it costs\n> +a few frames worth of time for them to take effect. This means that during a\n> +transition from auto to manual, there would be flickering in the control values\n> +and the transition won't be smooth.\n> +\n> +Take for instance the following flow, where we start on auto exposure (which\n> +for the purposes of the example increments by 1 each frame) and we want to\n> +switch seamlessly to manual exposure, which involves copying the exposure value\n> +computed by the auto exposure algorithm:\n> +\n> +::\n> +\n> + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n> + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> +\n> + Mode requested: Auto Auto Auto Manual Manual Manual Manual\n> + Exp requested: N/A N/A N/A 2 2 2 2\n> + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n> +\n> + Mode used: Auto Auto Auto Auto Auto Manual Manual\n> + Exp used: 0 1 2 3 4 2 2\n> +\n> +As we can see, after frame N+2 completes, we copy the exposure value that was\n> +used for frame N+2 (which was computed by AE algorithm), and queue that value\n> +into request N+3 with manual mode on. However, as it takes two frames for the\n> +exposure to be set, the exposure still changes since it is set by AE, and we\n> +get a flicker in the exposure during the switch from auto to manual.\n> +\n> +A solution is to *not submit* any exposure value when manual mode is enabled,\n> +and wait until the manual mode as been \"applied\" before copying the exposure\n> +value:\n> +\n> +::\n> +\n> + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n> + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> +\n> + Mode requested: Auto Auto Auto Manual Manual Manual Manual\n> + Exp requested: N/A N/A N/A None None None 5\n> + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n> +\n> + Mode used: Auto Auto Auto Auto Auto Manual Manual\n> + Exp used: 0 1 2 3 4 5 5\n> +\n> +In practice, this works. However, libcamera has a policy where once a control\n> +is submitted, its value is saved and does not need to be resubmitted. If the\n> +manual exposure value was set while auto mode was on, in theory the value would\n> +be saved, so when manual mode is enabled, the exposure value that was\n> +previously set would immediately be used. Clearly this solution isn't correct,\n> +but it can serve as the basis for a proper solution, with some more rigorous\n> +rules.\n> +\n> +Existing solutions\n> +------------------\n> +\n> +Raspberry Pi\n> +^^^^^^^^^^^^\n> +\n> +The Raspberry Pi IPA gets around the lack of individual AeEnable controls for\n> +exposure and gain by using magic values. When AeEnable is false, if one of the\n> +manual control values was set to 0 then the value computed by AEGC would be\n> +used for just that control. This solution isn't desirable, as it prevents\n> +that magic value from being used as a valid value.\n> +\n> +To get around the flickering issue, when AeEnable is false, the Raspberry Pi\n> +AEGC simply stops updating the values to be set, without restoring the\n> +previously set manual exposure time and gain. This works, but is not a proper\n> +solution.\n> +\n> +Android\n> +^^^^^^^\n> +\n> +The Android HAL specification requires that exposure and gain (sensitivity)\n> +must both be manual or both be auto. It cannot be that one is manual while the\n> +other is auto, so they simply don't support sub controls.\n> +\n> +For the flickering issue, the Android HAL has an AeLock control. To transition\n> +from auto to manual, the application would keep AE on auto, and turn on the\n> +lock. Once the lock has propagated through, then the value can be copied from\n> +the result into the request and the lock disabled and the mode set to manual.\n> +\n> +The problem with this solution is, besides the extra complexity, that it is\n> +ambiguous what happens if there is a state transition from manual to locked\n> +(even though it's a state transition that doesn't make sense). If locked is\n> +defined to \"use the last automatically computed values\" then it could use the\n> +values from the last time it AE was set to auto, or it would be undefined if AE\n> +was never auto (eg. it started out as manual), or if AE is implemented to run\n> +in the background it could just use the current values that are computed. If\n> +locked is defined to \"use the last value that was set\" there would be less\n> +ambiguity. Still, it's better if we can make it impossible to execute this\n> +nonsensical state transition, and if we can reduce the complexity of having\n> +this extra control or extra setting on a lever.\n> +\n> +Summary of goals\n> +----------------\n> +\n> +- We need a lock of some sort, to instruct the AEGC to not update output\n> + results\n> +\n> +- We need manual modes, to override the values computed by the AEGC\n> +\n> +- We need to support seamless transitions from auto to manual, and do so\n> + without flickering\n> +\n> +- We need custom minimum values for the manual controls; that is, no magic\n> + values for enabling/disabling auto\n> +\n> +- All of these need to be done with AE sub-controls (exposure time, analogue\n> + gain) and be extensible to aperture in the future\n> +\n> +Our solution\n> +------------\n> +\n> +A diagram of our solution:\n> +\n> +::\n> +\n> + +----------------------------+-------------+------------------+-----------------+\n> + | INPUT | ALGORITHM | RESULT | OUTPUT |\n> + +----------------------------+-------------+------------------+-----------------+\n> +\n> + ExposureTimeMode ExposureTimeMode\n> + ---------------------+----------------------------------------+----------------->\n> + 0: Auto | |\n> + 1: Manual | V\n> + | |\\\n> + | | \\\n> + | /----------------------------------> | 1| ExposureTime\n> + | | +-------------+ exposure time | | -------------->\n> + \\--)--> | | --------------> | 0|\n> + ExposureTime | | | | /\n> + ------------------------+--> | | |/\n> + | | AeState\n> + | AEGC | ----------------------------------->\n> + AnalogueGain | |\n> + ------------------------+--> | | |\\\n> + | | | | \\\n> + /--)--> | | --------------> | 0| AnalogueGain\n> + | | +-------------+ analogue gain | | -------------->\n> + | \\----------------------------------> | 1|\n> + | | /\n> + | |/\n> + | ^\n> + AnalogueGainMode | | AnalogueGainMode\n> + ---------------------+----------------------------------------+----------------->\n> + 0: Auto\n> + 1: Manual\n> +\n> +\n> +The diagram is divided in four sections horizontally:\n> +\n> +- Input: The values received from the request controls\n> +\n> +- Algorithm: The algorithm itself\n> +\n> +- Result: The values calculated by the algorithm\n> +\n> +- Output: The values reported in result metadata and applied to the device\n> +\n> +The four input controls are divided between manual values (ExposureTime and\n> +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The\n> +former are the manual values, the latter control how they're applied. The two\n> +modes are independent from each other, and each can take one of two values:\n> +\n> + - Auto (0): The AGC computes the value normally. The AGC result is applied\n> + to the output. The manual value is ignored *and is not retained*.\n> +\n> + - Manual (1): The AGC uses the manual value internally. The corresponding\n> + manual control from the request is applied to the output. The AGC result\n> + is ignored.\n\nToo much indentation here, move the '-' marker to the leftmost column.\n\n> +\n> +The AeState control reports the state of the unified AEGC block. If both\n> +ExposureTimeMode and AnalogueGainMode are set to manual then it will report\n> +Idle. If at least one of the two is set to auto, then AeState will report\n> +if the AEGC has Converged or not (Searching). This control replaces the old\n> +AeLocked control, as it was insufficient for reporting the AE state.\n> +\n> +There is a caveat to manual mode: the manual control value is not retained if\n> +it is set during auto mode. This means that if manual mode is entered without\n> +also setting the manual value, then it will enter a state similar to \"locked\",\n> +where the last automatically computed value while the mode was auto will be\n> +used. Once the manual value is set, then that will be used and retained as\n> +usual.\n> +\n> +This simulates an auto -> locked -> manual or auto -> manual state transition,\n> +and makes it impossible to do the nonsensical manual -> locked state\n> +transition.\n> +\n> +We specifically do not have a \"master AE control\" like the old AeEnable. This\n> +is because we have the individual mode controls, and if we had a master AE\n> +control it would be a \"control that sets other controls\", which could easily\n> +get out of control.\n> +\n> +With this solution, the earlier example would become:\n> +\n> +::\n> +\n> + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|\n> + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> + Mode requested: Auto Manual Manual Manual Manual Manual Manual Manual Manual\n> + Exp requested: N/A None None None None 10 None 10 10\n> + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+11 N+12\n> +\n> + Mode used: Auto Auto Auto Manual Manual Manual Manual Manual Manual\n> + Exp used: 2 3 4 5 5 5 5 10 10\n> +\n> +This example is extended by a few frames to exhibit the simulated \"locked\"\n> +state. At frame N+5 the application has confirmed that the manual mode has been\n> +entered, but does not provide a manual value until request N+7. Thus, the value\n> +that is used in requests N+5 and N+6 (where the mode is disabled), comes from\n> +the last value that was used when the mode was auto, which comes from frame\n> +N+4.\n> +\n> +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for\n> +the exposure to be applied. N+8 does not supply a manual value, but the last\n> +supplied value is retained, so a manual value of 10 is still used and set in\n> +frame N+10.\n> +\n> +Although this behavior is the same as what we had with waiting for the manual\n> +mode to propagate (in the section \"Description of the problem\"), this time it\n> +is correct as we have defined specifically that if a manual value was specified\n> +while the mode was auto, it will not be retained.\n> +\n> +Description of the controls\n> +---------------------------\n> +\n> +As described above, libcamera offers the following controls related to exposure\n> +and gain:\n> +\n> +- AnalogueGain\n> +\n> +- AnalogueGainMode\n> +\n> +- ExposureTime\n> +\n> +- ExposureTimeMode\n> +\n> +- AeState\n> +\n> +Auto-exposure and auto-gain can be enabled and disabled separately using the\n> +ExposureTimeMode and AnalogueGainMode controls respectively. There is no\n> +overarching AeEnable control.\n> +\n> +When the respective mode is set to auto, the respective value that is computed\n> +by the AEGC algorithm is applied to the image sensor. Any value that is\n> +supplied in the manual ExposureTime/AnalogueGain control is ignored and not\n> +retained. Another way to understand this is that when the mode transitions from\n> +auto to manual, the internally stored control value is overwritten with the\n> +last value computed by the auto algorithm.\n> +\n> +This means that when we transition from auto to manual without supplying a\n> +manual control value, the last value that was set by the AEGC algorithm will\n> +keep be used. This can be used to do a flickerless transition from auto to\n> +manual as described earlier. If the camera started out in manual mode and no\n> +corresponding value has been supplied yet, then a best-effort default value\n> +shall be set.\n> +\n> +The manual control value can be set in the same request as setting the mode to\n> +auto if the desired manual control value is already known.\n> +\n> +Transitioning from manual to auto shall be implicitly flickerless, as the AEGC\n> +algorithms are expected to start running from the last manual value.\n> +\n> +The AeState metadata reports the state of the AE algorithm. As AE cannot\n> +compute exposure and gain separately, the state of the AE component is\n> +unified. There are three states: Idle, Searching, and Converged.\n> +\n> +The state shall be Idle if both ExposureTimeMode and AnalogueGainMode\n> +are set to Manual. If the camera only supports one of the two controls,\n> +then the state shall be Idle if that one control is set to Manual. If\n> +the camera does not support Manual for at least one of the two controls,\n> +then the state will never be Idle, as AE will always be running.\n> +\n> +The state shall be Searching if at least one of exposure or gain calculated\n> +by the AE algorithm is used (that is, at least one of the two modes is Auto),\n> +*and* the value(s) have not converged yet.\n> +\n> +The state shall be Converged if at least one of exposure or gain calculated\n> +by the AE algorithm is used (that is, at least one of the two modes is Auto),\n> +*and* the value(s) have converged.\n> diff --git a/Documentation/index.rst b/Documentation/index.rst\n> index bea406608..251112fbd 100644\n> --- a/Documentation/index.rst\n> +++ b/Documentation/index.rst\n> @@ -23,7 +23,9 @@\n> SoftwareISP Benchmarking \n> Tracing guide \n> \n> + Design document: AE \n> +\n> .. toctree::\n> :hidden:\n> \n> - introduction\n> \\ No newline at end of file\n> + introduction\n> diff --git a/Documentation/meson.build b/Documentation/meson.build\n> index 36ffae239..6158320e1 100644\n> --- a/Documentation/meson.build\n> +++ b/Documentation/meson.build\n> @@ -128,6 +128,7 @@ if sphinx.found()\n> 'coding-style.rst',\n> 'conf.py',\n> 'contributing.rst',\n> + 'design/ae.rst',\n> 'documentation-contents.rst',\n> 'environment_variables.rst',\n> 'feature_requirements.rst',","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 6EDA8C32EF\n\tfor ;\n\tThu, 9 Jan 2025 21:42:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 22E53684EA;\n\tThu, 9 Jan 2025 22:42:05 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 53E1D61880\n\tfor ;\n\tThu, 9 Jan 2025 22:42:03 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 3D8D06DC;\n\tThu, 9 Jan 2025 22:41:09 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"JMrJW9kx\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1736458869;\n\tbh=OjhWaE6I+MARDifuKqRZw6b9ajD5FIieXfHI4kv2KXQ=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=JMrJW9kxbu2kXOh0woxydD1swfYLkQixA3YzYdrKpTvgqC+heHn7IAoAnc1Cg5zYD\n\tc65GohPCAExBFkChzi5YHOb+GTLwyM3kkmH6tOYnE8ZKFDWyJ5xjpQqNFrCFKiMx+N\n\ti9M/zi/9Nb3xnuBlHPwcy1uUGbEzKCQ2JwNdTI9I=","Date":"Thu, 9 Jan 2025 23:41:59 +0200","From":"Laurent Pinchart ","To":"Paul Elder ","Cc":"libcamera-devel@lists.libcamera.org, stefan.klug@ideasonboard.com,\n\tJacopo Mondi ","Subject":"Re: [PATCH v6 02/12] Documentation: design: ae: Document the design\n\tfor AE controls","Message-ID":"<20250109214159.GA5235@pendragon.ideasonboard.com>","References":"<20250109000942.1616565-1-paul.elder@ideasonboard.com>\n\t<20250109000942.1616565-3-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20250109000942.1616565-3-paul.elder@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":33026,"web_url":"https://patchwork.libcamera.org/comment/33026/","msgid":"","date":"2025-01-10T21:28:24","subject":"Re: [PATCH v6 02/12] Documentation: design: ae: Document the design\n\tfor AE controls","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"On Thu, Jan 09, 2025 at 11:41:59PM +0200, Laurent Pinchart wrote:\n> Hi Paul,\n> \n> Thank you for the patch.\n> \n> On Wed, Jan 08, 2025 at 06:09:32PM -0600, Paul Elder wrote:\n> > Document the design and rationale for the AE-related controls.\n> > Also add documentation for the controls.\n> > \n> > Signed-off-by: Paul Elder \n> > Signed-off-by: Jacopo Mondi \n> > Reviewed-by: Stefan Klug \n> > Reviewed-by: Laurent Pinchart \n> > \n> > ---\n> > No change in v6\n> > \n> > Changes in v5:\n> > - improve wording\n> > \n> > Changes in v4:\n> > - fix wording\n> > - fix compilation\n> > - remove locked state in explanation\n> > \n> > Changes in v3:\n> > - merge the control documentation into the same document (including the\n> > patch)\n> > - this is because it was a bit unwieldy to put it in\n> > control_ids.cpp.in, now that it's used generically to generate\n> > control ids of all namespaces\n> > ---\n> > Documentation/design/ae.rst | 320 ++++++++++++++++++++++++++++++++++++\n> > Documentation/index.rst | 4 +-\n> > Documentation/meson.build | 1 +\n> > 3 files changed, 324 insertions(+), 1 deletion(-)\n> > create mode 100644 Documentation/design/ae.rst\n> > \n> > diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst\n> > new file mode 100644\n> > index 000000000..f8b2c887c\n> > --- /dev/null\n> > +++ b/Documentation/design/ae.rst\n> > @@ -0,0 +1,320 @@\n> > +.. SPDX-License-Identifier: CC-BY-SA-4.0\n> > +\n> > +Design of Exposure and Gain controls\n> > +====================================\n> > +\n> > +This document explains the design and rationale of the controls related to\n> > +exposure and gain. This includes the all-encompassing auto-exposure (AE), the\n> > +manual exposure control, and the manual gain control.\n> > +\n> > +Description of the problem\n> > +--------------------------\n> > +\n> > +Sub controls\n> > +^^^^^^^^^^^^\n> > +\n> > +There are more than one control that make up total exposure: exposure time,\n> > +gain, and aperture (though for now we will not consider aperture). We already\n> > +had individual controls for setting the values of manual exposure and manual\n> > +gain, but for switching between auto mode and manual mode we only had a\n> > +high-level boolean AeEnable control that would set *both* exposure and gain to\n> > +auto mode or manual mode; we had no way to set one to auto and the other to\n> > +manual.\n> > +\n> > +So, we need to introduce two new controls to act as \"levers\" to indicate\n> > +individually for exposure and gain if the value would come from AEGC or if it\n> > +would come from the manual control value.\n> > +\n> > +Aperture priority\n> > +^^^^^^^^^^^^^^^^^\n> > +\n> > +We eventually may need to support aperture, and so whatever our solution is for\n> > +having only some controls on auto and the others on manual needs to be\n> > +extensible.\n> > +\n> > +Flickering when going from auto to manual\n> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> > +\n> > +When a manual exposure or gain value is requested by the application, it costs\n> > +a few frames worth of time for them to take effect. This means that during a\n> > +transition from auto to manual, there would be flickering in the control values\n> > +and the transition won't be smooth.\n> > +\n> > +Take for instance the following flow, where we start on auto exposure (which\n> > +for the purposes of the example increments by 1 each frame) and we want to\n> > +switch seamlessly to manual exposure, which involves copying the exposure value\n> > +computed by the auto exposure algorithm:\n> > +\n> > +::\n> > +\n> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> > +\n> > + Mode requested: Auto Auto Auto Manual Manual Manual Manual\n> > + Exp requested: N/A N/A N/A 2 2 2 2\n> > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n> > +\n> > + Mode used: Auto Auto Auto Auto Auto Manual Manual\n> > + Exp used: 0 1 2 3 4 2 2\n> > +\n> > +As we can see, after frame N+2 completes, we copy the exposure value that was\n> > +used for frame N+2 (which was computed by AE algorithm), and queue that value\n> > +into request N+3 with manual mode on. However, as it takes two frames for the\n> > +exposure to be set, the exposure still changes since it is set by AE, and we\n> > +get a flicker in the exposure during the switch from auto to manual.\n> > +\n> > +A solution is to *not submit* any exposure value when manual mode is enabled,\n> > +and wait until the manual mode as been \"applied\" before copying the exposure\n> > +value:\n> > +\n> > +::\n> > +\n> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> > +\n> > + Mode requested: Auto Auto Auto Manual Manual Manual Manual\n> > + Exp requested: N/A N/A N/A None None None 5\n> > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n> > +\n> > + Mode used: Auto Auto Auto Auto Auto Manual Manual\n> > + Exp used: 0 1 2 3 4 5 5\n> > +\n> > +In practice, this works. However, libcamera has a policy where once a control\n> > +is submitted, its value is saved and does not need to be resubmitted. If the\n> > +manual exposure value was set while auto mode was on, in theory the value would\n> > +be saved, so when manual mode is enabled, the exposure value that was\n> > +previously set would immediately be used. Clearly this solution isn't correct,\n> > +but it can serve as the basis for a proper solution, with some more rigorous\n> > +rules.\n> > +\n> > +Existing solutions\n> > +------------------\n> > +\n> > +Raspberry Pi\n> > +^^^^^^^^^^^^\n> > +\n> > +The Raspberry Pi IPA gets around the lack of individual AeEnable controls for\n> > +exposure and gain by using magic values. When AeEnable is false, if one of the\n> > +manual control values was set to 0 then the value computed by AEGC would be\n> > +used for just that control. This solution isn't desirable, as it prevents\n> > +that magic value from being used as a valid value.\n> > +\n> > +To get around the flickering issue, when AeEnable is false, the Raspberry Pi\n> > +AEGC simply stops updating the values to be set, without restoring the\n> > +previously set manual exposure time and gain. This works, but is not a proper\n> > +solution.\n> > +\n> > +Android\n> > +^^^^^^^\n> > +\n> > +The Android HAL specification requires that exposure and gain (sensitivity)\n> > +must both be manual or both be auto. It cannot be that one is manual while the\n> > +other is auto, so they simply don't support sub controls.\n> > +\n> > +For the flickering issue, the Android HAL has an AeLock control. To transition\n> > +from auto to manual, the application would keep AE on auto, and turn on the\n> > +lock. Once the lock has propagated through, then the value can be copied from\n> > +the result into the request and the lock disabled and the mode set to manual.\n> > +\n> > +The problem with this solution is, besides the extra complexity, that it is\n> > +ambiguous what happens if there is a state transition from manual to locked\n> > +(even though it's a state transition that doesn't make sense). If locked is\n> > +defined to \"use the last automatically computed values\" then it could use the\n> > +values from the last time it AE was set to auto, or it would be undefined if AE\n> > +was never auto (eg. it started out as manual), or if AE is implemented to run\n> > +in the background it could just use the current values that are computed. If\n> > +locked is defined to \"use the last value that was set\" there would be less\n> > +ambiguity. Still, it's better if we can make it impossible to execute this\n> > +nonsensical state transition, and if we can reduce the complexity of having\n> > +this extra control or extra setting on a lever.\n> > +\n> > +Summary of goals\n> > +----------------\n> > +\n> > +- We need a lock of some sort, to instruct the AEGC to not update output\n> > + results\n> > +\n> > +- We need manual modes, to override the values computed by the AEGC\n> > +\n> > +- We need to support seamless transitions from auto to manual, and do so\n> > + without flickering\n> > +\n> > +- We need custom minimum values for the manual controls; that is, no magic\n> > + values for enabling/disabling auto\n> > +\n> > +- All of these need to be done with AE sub-controls (exposure time, analogue\n> > + gain) and be extensible to aperture in the future\n> > +\n> > +Our solution\n> > +------------\n> > +\n> > +A diagram of our solution:\n> > +\n> > +::\n> > +\n> > + +----------------------------+-------------+------------------+-----------------+\n> > + | INPUT | ALGORITHM | RESULT | OUTPUT |\n> > + +----------------------------+-------------+------------------+-----------------+\n> > +\n> > + ExposureTimeMode ExposureTimeMode\n> > + ---------------------+----------------------------------------+----------------->\n> > + 0: Auto | |\n> > + 1: Manual | V\n> > + | |\\\n> > + | | \\\n> > + | /----------------------------------> | 1| ExposureTime\n> > + | | +-------------+ exposure time | | -------------->\n> > + \\--)--> | | --------------> | 0|\n> > + ExposureTime | | | | /\n> > + ------------------------+--> | | |/\n> > + | | AeState\n> > + | AEGC | ----------------------------------->\n> > + AnalogueGain | |\n> > + ------------------------+--> | | |\\\n> > + | | | | \\\n> > + /--)--> | | --------------> | 0| AnalogueGain\n> > + | | +-------------+ analogue gain | | -------------->\n> > + | \\----------------------------------> | 1|\n> > + | | /\n> > + | |/\n> > + | ^\n> > + AnalogueGainMode | | AnalogueGainMode\n> > + ---------------------+----------------------------------------+----------------->\n> > + 0: Auto\n> > + 1: Manual\n> > +\n> > +\n> > +The diagram is divided in four sections horizontally:\n> > +\n> > +- Input: The values received from the request controls\n> > +\n> > +- Algorithm: The algorithm itself\n> > +\n> > +- Result: The values calculated by the algorithm\n> > +\n> > +- Output: The values reported in result metadata and applied to the device\n> > +\n> > +The four input controls are divided between manual values (ExposureTime and\n> > +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The\n> > +former are the manual values, the latter control how they're applied. The two\n> > +modes are independent from each other, and each can take one of two values:\n> > +\n> > + - Auto (0): The AGC computes the value normally. The AGC result is applied\n> > + to the output. The manual value is ignored *and is not retained*.\n> > +\n> > + - Manual (1): The AGC uses the manual value internally. The corresponding\n> > + manual control from the request is applied to the output. The AGC result\n> > + is ignored.\n> \n> Too much indentation here, move the '-' marker to the leftmost column.\n> \n\nIs... that all?\n\n\nPaul\n\n> > +\n> > +The AeState control reports the state of the unified AEGC block. If both\n> > +ExposureTimeMode and AnalogueGainMode are set to manual then it will report\n> > +Idle. If at least one of the two is set to auto, then AeState will report\n> > +if the AEGC has Converged or not (Searching). This control replaces the old\n> > +AeLocked control, as it was insufficient for reporting the AE state.\n> > +\n> > +There is a caveat to manual mode: the manual control value is not retained if\n> > +it is set during auto mode. This means that if manual mode is entered without\n> > +also setting the manual value, then it will enter a state similar to \"locked\",\n> > +where the last automatically computed value while the mode was auto will be\n> > +used. Once the manual value is set, then that will be used and retained as\n> > +usual.\n> > +\n> > +This simulates an auto -> locked -> manual or auto -> manual state transition,\n> > +and makes it impossible to do the nonsensical manual -> locked state\n> > +transition.\n> > +\n> > +We specifically do not have a \"master AE control\" like the old AeEnable. This\n> > +is because we have the individual mode controls, and if we had a master AE\n> > +control it would be a \"control that sets other controls\", which could easily\n> > +get out of control.\n> > +\n> > +With this solution, the earlier example would become:\n> > +\n> > +::\n> > +\n> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> > + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|\n> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n> > + Mode requested: Auto Manual Manual Manual Manual Manual Manual Manual Manual\n> > + Exp requested: N/A None None None None 10 None 10 10\n> > + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+11 N+12\n> > +\n> > + Mode used: Auto Auto Auto Manual Manual Manual Manual Manual Manual\n> > + Exp used: 2 3 4 5 5 5 5 10 10\n> > +\n> > +This example is extended by a few frames to exhibit the simulated \"locked\"\n> > +state. At frame N+5 the application has confirmed that the manual mode has been\n> > +entered, but does not provide a manual value until request N+7. Thus, the value\n> > +that is used in requests N+5 and N+6 (where the mode is disabled), comes from\n> > +the last value that was used when the mode was auto, which comes from frame\n> > +N+4.\n> > +\n> > +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for\n> > +the exposure to be applied. N+8 does not supply a manual value, but the last\n> > +supplied value is retained, so a manual value of 10 is still used and set in\n> > +frame N+10.\n> > +\n> > +Although this behavior is the same as what we had with waiting for the manual\n> > +mode to propagate (in the section \"Description of the problem\"), this time it\n> > +is correct as we have defined specifically that if a manual value was specified\n> > +while the mode was auto, it will not be retained.\n> > +\n> > +Description of the controls\n> > +---------------------------\n> > +\n> > +As described above, libcamera offers the following controls related to exposure\n> > +and gain:\n> > +\n> > +- AnalogueGain\n> > +\n> > +- AnalogueGainMode\n> > +\n> > +- ExposureTime\n> > +\n> > +- ExposureTimeMode\n> > +\n> > +- AeState\n> > +\n> > +Auto-exposure and auto-gain can be enabled and disabled separately using the\n> > +ExposureTimeMode and AnalogueGainMode controls respectively. There is no\n> > +overarching AeEnable control.\n> > +\n> > +When the respective mode is set to auto, the respective value that is computed\n> > +by the AEGC algorithm is applied to the image sensor. Any value that is\n> > +supplied in the manual ExposureTime/AnalogueGain control is ignored and not\n> > +retained. Another way to understand this is that when the mode transitions from\n> > +auto to manual, the internally stored control value is overwritten with the\n> > +last value computed by the auto algorithm.\n> > +\n> > +This means that when we transition from auto to manual without supplying a\n> > +manual control value, the last value that was set by the AEGC algorithm will\n> > +keep be used. This can be used to do a flickerless transition from auto to\n> > +manual as described earlier. If the camera started out in manual mode and no\n> > +corresponding value has been supplied yet, then a best-effort default value\n> > +shall be set.\n> > +\n> > +The manual control value can be set in the same request as setting the mode to\n> > +auto if the desired manual control value is already known.\n> > +\n> > +Transitioning from manual to auto shall be implicitly flickerless, as the AEGC\n> > +algorithms are expected to start running from the last manual value.\n> > +\n> > +The AeState metadata reports the state of the AE algorithm. As AE cannot\n> > +compute exposure and gain separately, the state of the AE component is\n> > +unified. There are three states: Idle, Searching, and Converged.\n> > +\n> > +The state shall be Idle if both ExposureTimeMode and AnalogueGainMode\n> > +are set to Manual. If the camera only supports one of the two controls,\n> > +then the state shall be Idle if that one control is set to Manual. If\n> > +the camera does not support Manual for at least one of the two controls,\n> > +then the state will never be Idle, as AE will always be running.\n> > +\n> > +The state shall be Searching if at least one of exposure or gain calculated\n> > +by the AE algorithm is used (that is, at least one of the two modes is Auto),\n> > +*and* the value(s) have not converged yet.\n> > +\n> > +The state shall be Converged if at least one of exposure or gain calculated\n> > +by the AE algorithm is used (that is, at least one of the two modes is Auto),\n> > +*and* the value(s) have converged.\n> > diff --git a/Documentation/index.rst b/Documentation/index.rst\n> > index bea406608..251112fbd 100644\n> > --- a/Documentation/index.rst\n> > +++ b/Documentation/index.rst\n> > @@ -23,7 +23,9 @@\n> > SoftwareISP Benchmarking \n> > Tracing guide \n> > \n> > + Design document: AE \n> > +\n> > .. toctree::\n> > :hidden:\n> > \n> > - introduction\n> > \\ No newline at end of file\n> > + introduction\n> > diff --git a/Documentation/meson.build b/Documentation/meson.build\n> > index 36ffae239..6158320e1 100644\n> > --- a/Documentation/meson.build\n> > +++ b/Documentation/meson.build\n> > @@ -128,6 +128,7 @@ if sphinx.found()\n> > 'coding-style.rst',\n> > 'conf.py',\n> > 'contributing.rst',\n> > + 'design/ae.rst',\n> > 'documentation-contents.rst',\n> > 'environment_variables.rst',\n> > 'feature_requirements.rst',","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1A555C32F5\n\tfor ;\n\tFri, 10 Jan 2025 21:28:35 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 315B1684E7;\n\tFri, 10 Jan 2025 22:28:34 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7AB5A608AA\n\tfor ;\n\tFri, 10 Jan 2025 22:28:32 +0100 (CET)","from pyrite.rasen.tech (h175-177-049-030.catv02.itscom.jp\n\t[175.177.49.30])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 4A73D161;\n\tFri, 10 Jan 2025 22:27:36 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"VCSt+KTM\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1736544457;\n\tbh=qOB1rZa5EaTEIUUXjezqp+3/baaEy5/2M/ga5XCkiaQ=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=VCSt+KTMGp2c8DhiHkAkhX4+kQEGnDMc6Ns3m7dJA2BiTt24EiNlR2IHujBlXAGzl\n\tRrzlPQBYFeWBicMrcvnWdcO+1vbUWDhQdweGOBxyPONa/YL4a/61s4WTUD5UhnfrTg\n\t4lnZSZdxnCcBeXnbjTMxyeDUQ5lg0u48CTlx67Uw=","Date":"Sat, 11 Jan 2025 06:28:24 +0900","From":"Paul Elder ","To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org, stefan.klug@ideasonboard.com,\n\tJacopo Mondi ","Subject":"Re: [PATCH v6 02/12] Documentation: design: ae: Document the design\n\tfor AE controls","Message-ID":"","References":"<20250109000942.1616565-1-paul.elder@ideasonboard.com>\n\t<20250109000942.1616565-3-paul.elder@ideasonboard.com>\n\t<20250109214159.GA5235@pendragon.ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<20250109214159.GA5235@pendragon.ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]