Patch Detail
Show a patch.
GET /api/patches/21885/?format=api
{ "id": 21885, "url": "https://patchwork.libcamera.org/api/patches/21885/?format=api", "web_url": "https://patchwork.libcamera.org/patch/21885/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20241113131256.3170817-3-paul.elder@ideasonboard.com>", "date": "2024-11-13T13:12:50", "name": "[v3,2/8] Documentation: design: ae: Document the design for AE controls", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": false, "hash": "493f1efa45745cc02941d84e50295139a8a0e051", "submitter": { "id": 17, "url": "https://patchwork.libcamera.org/api/people/17/?format=api", "name": "Paul Elder", "email": "paul.elder@ideasonboard.com" }, "delegate": null, "mbox": "https://patchwork.libcamera.org/patch/21885/mbox/", "series": [ { "id": 4790, "url": "https://patchwork.libcamera.org/api/series/4790/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=4790", "date": "2024-11-13T13:12:48", "name": "AEGC controls", "version": 3, "mbox": "https://patchwork.libcamera.org/series/4790/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/patches/21885/comments/", "check": "pending", "checks": "https://patchwork.libcamera.org/api/patches/21885/checks/", "tags": {}, "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 43A2EBE173\n\tfor <parsemail@patchwork.libcamera.org>;\n\tWed, 13 Nov 2024 13:13:25 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EEED66581E;\n\tWed, 13 Nov 2024 14:13:24 +0100 (CET)", "from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7A36065816\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed, 13 Nov 2024 14:13:23 +0100 (CET)", "from neptunite.flets-east.jp (unknown\n\t[IPv6:2404:7a81:160:2100:eb25:7aa0:7da7:eb24])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 343FF1306;\n\tWed, 13 Nov 2024 14:13:07 +0100 (CET)" ], "Authentication-Results": "lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"TNOtL0+V\"; dkim-atps=neutral", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1731503590;\n\tbh=HkBAWMufoWK21R/4mNX+Yn9tKT9evsJ9f/yELTctYGY=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=TNOtL0+VAdBOuVApDX2c8SRzUaNQdOXIlvt4PjTYgHOfjyphQvHq6P8hJyj/+to8n\n\t10mw3kd9st1deut0Ds5VmC0dsjkdpOBIkK3k9cEFBkrLIszIL9tf1rHK4AajWgefBk\n\tXYCQH9lGmc0+34CLKNx2MMFoM7Ia8OlA4QpT7o8w=", "From": "Paul Elder <paul.elder@ideasonboard.com>", "To": "libcamera-devel@lists.libcamera.org", "Cc": "laurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com,\n\tnaush@raspberrypi.com, david.plowman@raspberrypi.com,\n\tPaul Elder <paul.elder@ideasonboard.com>,\n\tJacopo Mondi <jacopo@jmondi.org>", "Subject": "[PATCH v3 2/8] Documentation: design: ae: Document the design for AE\n\tcontrols", "Date": "Wed, 13 Nov 2024 22:12:50 +0900", "Message-Id": "<20241113131256.3170817-3-paul.elder@ideasonboard.com>", "X-Mailer": "git-send-email 2.39.2", "In-Reply-To": "<20241113131256.3170817-1-paul.elder@ideasonboard.com>", "References": "<20241113131256.3170817-1-paul.elder@ideasonboard.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "Document the design and rationale for the AE-related controls.\nAlso add documentation for the controls.\n\nSigned-off-by: Paul Elder <paul.elder@ideasonboard.com>\nSigned-off-by: Jacopo Mondi <jacopo@jmondi.org>\n\n---\nChanges in v3:\n- merge the control documentation into the same document (including the\n patch)\n - this is because it was a bit unwieldy to put it in\n control_ids.cpp.in, now that it's used generically to generate\n control ids of all namespaces\n---\n Documentation/design/ae.rst | 348 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 348 insertions(+)\n create mode 100644 Documentation/design/ae.rst", "diff": "diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst\nnew file mode 100644\nindex 000000000000..f121afecea5a\n--- /dev/null\n+++ b/Documentation/design/ae.rst\n@@ -0,0 +1,348 @@\n+.. SPDX-License-Identifier: CC-BY-SA-4.0\n+\n+Design of Exposure and Gain controls\n+====================================\n+\n+This document explains the design and rationale of the controls related to\n+exposure and gain. This includes the all-encompassing auto-exposure (AE), the\n+manual exposure control, and the manual gain control.\n+\n+Description of the problem\n+--------------------------\n+\n+Sub controls\n+^^^^^^^^^^^^\n+\n+There are more than one control that make up exposure: exposure, gain, and\n+aperture (though for now we will not consider aperture). We already had\n+individual controls for setting the values of manual exposure and manual gain,\n+but for switching between auto mode and manual mode we only had a high-level\n+boolean AeEnable control that would set *both* exposure and gain to auto mode\n+or manual mode; we had no way to set one to auto and the other to manual.\n+\n+So, we need to introduce two new controls to act as \"levers\" to indicate\n+individually for exposure and gain if the value would come from AEGC or if it\n+would come from the manual control value.\n+\n+Aperture priority\n+^^^^^^^^^^^^^^^^^\n+\n+We eventually will support aperture, and so whatever our solution is for having\n+only some controls on auto and the others on manual needs to be extensible.\n+\n+Flickering when going from auto to manual\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+When a manual exposure or gain value is requested by the application, it costs\n+a few frames worth of time for them to take effect. This means that during a\n+transition from auto to manual, there would be flickering in the control values\n+and the transition won't be smooth.\n+\n+Take for instance the following flow, where we start on auto exposure (which\n+for the purposes of the example increments by 1 each frame) and we want to\n+switch seamlessly to manual exposure, which involves copying the exposure value\n+computed by the auto exposure algorithm:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual\n+ Exp requested: N/A N/A N/A 2 2 2 2\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 2 2\n+\n+As we can see, after frame N+2 completes, we copy the exposure value that was\n+used for frame N+2 (which was computed by AE algorithm), and queue that value\n+into request N+3 with manual mode on. However, as it takes two frames for the\n+exposure to be set, the exposure still changes since it is set by AE, and we\n+get a flicker in the exposure during the switch from auto to manual.\n+\n+A solution is to *not submit* any exposure value when manual mode is enabled,\n+and wait until the manual mode as been \"applied\" before copying the exposure\n+value:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual\n+ Exp requested: N/A N/A N/A None None None 5\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 5 5\n+\n+In practice, this works. However, libcamera has a policy where once a control\n+is submitted, its value is saved and does not need to be resubmitted (even\n+though this isn't implemented yet). So if the manual exposure value was set\n+while auto mode was on, in theory the value would be saved, so when manual mode\n+is enabled, the exposure value that was previously set would immediately be\n+used. Clearly this solution isn't correct, but it can serve as the basis for a\n+proper solution, with some more rigorous rules.\n+\n+Existing solutions\n+------------------\n+\n+Raspberry Pi\n+^^^^^^^^^^^^\n+\n+The raspberry pi IPA got around the lack of individual AeEnable controls for\n+exposure and gain by using magic values. When AeEnable was false, if one of the\n+manual control values was set to 0 then the value computed by AEGC would be\n+used for just that control. This solution isn't desirable, as it prevents\n+that magic value from being used as a valid value.\n+\n+To get around the flickering issue, when AeEnable was false, the raspberry pi\n+AEGC would simply stop updating the values to be set. As mentioned above, since\n+the value retention mechanism hasn't actually been implemented yet, this\n+worked. But, it's not a proper solution.\n+\n+Android\n+^^^^^^^\n+\n+The Android HAL specification requires that exposure and gain (sensitivity)\n+must both be manual or both be auto. It cannot be that one is manual while the\n+other is auto, so they simply don't support sub controls.\n+\n+For the flickering issue, the Android HAL has an AeLock control. To transition\n+from auto to manual, the application would keep AE on auto, and turn on the\n+lock. Once the lock has propagated through, then the value can be copied from\n+the result into the request and the lock disabled and the mode set to manual.\n+\n+The problem with this solution is, besides the extra complexity, that it is\n+ambiguous what happens if there is a state transition from manual to locked\n+(even though it's a state transition that doesn't make sense). If locked is\n+defined to \"use the last automatically computed values\" then it could use the\n+values from the last time it AE was set to auto, or it would be undefined if AE\n+was never auto (eg. it started out as manual), or if AE is implemented to run\n+in the background it could just use the current values that are computed. If\n+locked is defined to \"use the last value that was set\" there would be less\n+ambiguity. Still, it's better if we can make it impossible to execute this\n+nonsensical state transition, and if we can reduce the complexity of having\n+this extra control or extra setting on a lever.\n+\n+Summary of goals\n+----------------\n+\n+ - We need a lock of some sort, to instruct the AEGC to not update output\n+ results\n+\n+ - We need manual modes, to override the values computed by the AEGC\n+\n+ - We need to support seamless transitions from auto to manual, and do so\n+ without flickering\n+\n+ - We need custom minimum values for the manual controls; that is, no magic\n+ values for enabling/disabling auto\n+\n+ - All of these need to be done with AE sub-controls (exposure time, analogue\n+ gain) and be extensible to aperture in the future\n+\n+Our solution\n+------------\n+\n+A diagram of our solution:\n+\n+::\n+\n+ +----------------------------+-------------+------------------+-----------------+\n+ | INPUT | ALGORITHM | RESULT | OUTPUT |\n+ +----------------------------+-------------+------------------+-----------------+\n+\n+ ExposureTimeMode ExposureTimeMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto | |\n+ 1: Manual | V\n+ | |\\\n+ | | \\\n+ | /----------------------------------> | 1| ExposureTime\n+ | | +-------------+ exposure time | | -------------->\n+ \\--)--> | | --------------> | 0|\n+ ExposureTime | | | | /\n+ ------------------------+--> | | |/\n+ | | AeState\n+ | AEGC | ----------------------------------->\n+ AnalogueGain | |\n+ ------------------------+--> | | |\\\n+ | | | | \\\n+ /--)--> | | --------------> | 0| AnalogueGain\n+ | | +-------------+ analogue gain | | -------------->\n+ | \\----------------------------------> | 1|\n+ | | /\n+ | |/\n+ | ^\n+ AnalogueGainMode | | AnalogueGainMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto\n+ 1: Manual\n+\n+\n+The diagram is divided in four sections horizontally:\n+\n+ - Input: The values received from the request controls\n+\n+ - Algorithm: The algorithm itself\n+\n+ - Result: The values calculated by the algorithm\n+\n+ - Output: The values that sent in result metadata and applied to the device\n+\n+The four input controls are divided between manual values (ExposureTime and\n+AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The\n+former are the manual values, the latter control how they're applied. The two\n+modes are independent from each other, and each can take one of two values:\n+\n+ - Auto (0): The AGC computes the value normally. The AGC result is applied\n+ to the output. The manual value is ignored *and is not retained*.\n+\n+ - Manual (1): The AGC uses the manual value internally. The corresponding\n+ manual control from the request is applied to the output. The AGC result\n+ is ignored.\n+\n+The AeState control reports the state of the unified AEGC block. If both\n+ExposureTimeMode and AnalogueGainMode are set to disabled then it will report\n+Idle. If at least one of the two is set to auto, then AeState will report\n+if the AEGC has Converged or not (Searching). This control replaces the old\n+AeLocked control, as it was insufficient for reporting the AE state.\n+\n+There is a caveat to the disabled mode: the manual control value is not\n+retained if it is set during auto mode. This means that if the disabled mode is\n+entered without also setting the manual value, then it will enter a state\n+similar to \"locked\", where the last automatically computed value while the mode\n+was auto will be used. Once the manual value is set, then that will be used and\n+retained as usual.\n+\n+This simulates an auto -> locked -> manual or auto -> manual state transition,\n+and makes it impossible to do the nonsensical manual -> locked state\n+transition.\n+\n+We specifically do not have a \"master AE control\" like the old AeEnable. This\n+is because we have the individual mode controls, and if we had a master AE\n+control it would be a \"control that sets other controls\", which could easily\n+get out of control.\n+\n+With this solution, the earlier example would become:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab\n+ Exp requested: N/A None None None None 10 None 10 10\n+ Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10\n+\n+ Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab\n+ Exp used: 2 3 4 5 5 5 5 10 10\n+\n+This example is extended by a few frames to exhibit the simulated \"locked\"\n+state. At frame N+5 the application has confirmed that the auto mode has been\n+disabled, but does not provide a manual value until request N+7. Thus, the\n+value that is used in requests N+5 and N+6 (where the mode is disabled), comes\n+from the last value that was used when the mode was auto, which comes from\n+frame N+4.\n+\n+Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for\n+the exposure to be applied. N+8 does not supply a manual value, but the last\n+supplied value is retained, so a manual value of 10 is still used and set in\n+frame N+10.\n+\n+Although this behavior is the same as what we had with waiting for the manual\n+mode to propagate (in the section \"Description of the problem\"), this time it\n+is correct as we have defined specifically that if a manual value was specified\n+while the mode was auto, it will not be retained.\n+\n+Description of the controls\n+===========================\n+\n+ libcamera offers the following controls related to exposure and gain:\n+\n+ - AnalogueGain\n+\n+ - AnalogueGainMode\n+\n+ - ExposureTime\n+\n+ - ExposureTimeMode\n+\n+ - AeState\n+\n+ Auto-exposure and auto-gain can be enabled and disabled separately using the\n+ ExposureTimeMode and AnalogueGainMode controls respectively. There is no\n+ overarching AeEnable control.\n+\n+ For each of exposure and gain, we can model it with three states: auto,\n+ locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only\n+ have two values, as the locked state is simulated.\n+\n+ ::\n+\n+ /---------------------------------\\\n+ | |\n+ V |\n+ +--------+ +--------+\n+ | | ---------------------> | |\n+ | Auto | +--------+ | Manual |\n+ | | ----> | Locked | ----> | |\n+ +--------+ +--------+ +--------+\n+ ^ |\n+ | |\n+ \\----------------/\n+\n+ Notice from the state diagram that locked to manual is a one-way state\n+ change, as the reverse direction is nonsensical (see the design document for\n+ more details on this topic).\n+\n+ The exposure/gain is in the Auto state when\n+ ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value\n+ that is computed by the AE algorithm is applied to the image sensor. Any\n+ value that is supplied in the ExposureTime/AnalogueGain control is ignored\n+ and is not retained.\n+\n+ If ExposureTimeMode/AnalogueGainMode is set to Manual, it can put us in\n+ either the Locked or Manual state. The difference is in if\n+ ExposureTime/AnalogueGain has been supplied. If it has not yet been\n+ supplied, then we are in the Locked state. If it has been supplied, then we\n+ are in the Manual state.\n+\n+ In both the Locked state and the Manual state the exposure/gain value does\n+ not come from the AE algorithm. In the Locked state the value comes from the\n+ last value computed by the AE algorithm while the state was Auto, or if the\n+ state was never Auto (e.g. we started in Locked, or the camera doesn't\n+ support Auto), then the value should be a best-effort default value. In the\n+ Manual state the value comes from the value supplied in the\n+ ExposureTime/AnalogueGain control.\n+\n+ To transition from the Locked state to the Manual state, a value needs to be\n+ submitted in ExposureTime/AnalogueGain. Once the state has transitioned to\n+ Manual, then this value will be retained, and doesn't need to be resubmitted\n+ if it doesn't change.\n+\n+ To transition to the Auto state, simply set\n+ ExposureTimeMode/AnalogueGainMode to Auto.\n+\n+\n+ The AeState metadata reports the state of the AE algorithm. As AE cannot\n+ compute exposure and gain separately, the state of the AE component is\n+ unified. There are three states: Idle, Searching, and Converged.\n+\n+ The state shall be Idle if both ExposureTimeMode and AnalogueGainMode\n+ are set to Manual. If the camera only supports one of the two controls,\n+ then the state shall be Idle if that one control is set to Manual. If\n+ the camera does not support Manual for at least one of the two controls,\n+ then the state will never be Idle, as AE will always be running.\n+\n+ The state shall be Searching if at least one of exposure or gain calculated\n+ by the AE algorithm is used (that is, at least one of the two modes is Auto),\n+ *and* the value(s) have not converged yet.\n+\n+ The state shall be Converged if at least one of exposure or gain calculated\n+ by the AE algorithm is used (that is, at least one of the two modes is Auto),\n+ *and* the value(s) have converged.\n", "prefixes": [ "v3", "2/8" ] }