Patch Detail
Show a patch.
GET /api/1.1/patches/17389/?format=api
{ "id": 17389, "url": "https://patchwork.libcamera.org/api/1.1/patches/17389/?format=api", "web_url": "https://patchwork.libcamera.org/patch/17389/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/1.1/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20220923085222.7759-1-jacopo@jmondi.org>", "date": "2022-09-23T08:52:21", "name": "[libcamera-devel,7/6] Documentation: design: ae: Document the design for AE controls", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": false, "hash": "0da9fe30c0839681842742edef6c0238537a388f", "submitter": { "id": 3, "url": "https://patchwork.libcamera.org/api/1.1/people/3/?format=api", "name": "Jacopo Mondi", "email": "jacopo@jmondi.org" }, "delegate": { "id": 15, "url": "https://patchwork.libcamera.org/api/1.1/users/15/?format=api", "username": "jmondi", "first_name": "Jacopo", "last_name": "Mondi", "email": "jacopo@jmondi.org" }, "mbox": "https://patchwork.libcamera.org/patch/17389/mbox/", "series": [ { "id": 3404, "url": "https://patchwork.libcamera.org/api/1.1/series/3404/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=3404", "date": "2022-08-11T15:02:13", "name": "AEGC controls", "version": 2, "mbox": "https://patchwork.libcamera.org/series/3404/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/patches/17389/comments/", "check": "pending", "checks": "https://patchwork.libcamera.org/api/patches/17389/checks/", "tags": {}, "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E3A53C0DA4\n\tfor <parsemail@patchwork.libcamera.org>;\n\tFri, 23 Sep 2022 08:52:40 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4B2E46222C;\n\tFri, 23 Sep 2022 10:52:40 +0200 (CEST)", "from relay12.mail.gandi.net (relay12.mail.gandi.net\n\t[IPv6:2001:4b98:dc4:8::232])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CCD52621BC\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 23 Sep 2022 10:52:38 +0200 (CEST)", "(Authenticated sender: jacopo@jmondi.org)\n\tby mail.gandi.net (Postfix) with ESMTPSA id 90C07200005;\n\tFri, 23 Sep 2022 08:52:37 +0000 (UTC)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1663923160;\n\tbh=+Hr2MDQTRcmktC4am1RnCysmVamhKenRq/kVjVwA7ZE=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=I/eJ9mfQQavVKSYH7bf01UQFLctNC56DvhtvociHKG99OHmUtmBQGBdPWrFqzF4Bh\n\tGeSgTKIev4gXbyihzQu1fsdop6XVz4G/Ge9vVIvT4ahsYKJN6+OA2Vd7zzKzH30cFO\n\toD4a25z3Wxm7jOp0uqNqX58c+/YLqDg/q0gR9BBR2AOlMt7Yjn2XbhW2wCn1YCKY+6\n\t25qxS796A3miFCrCltHLTBlu1BFLVswxGR8kyMZjh5bOEXey5jDOBWLoH3qStV6hX+\n\t+sCTBfYc3pHwwNjtKbumMNMcR8b9tjjaI8NKbN2so1LQZomOvAnM30tO5V3H7onkoL\n\t+1M+OmUVRYaUA==", "To": "libcamera-devel@lists.libcamera.org", "Date": "Fri, 23 Sep 2022 10:52:21 +0200", "Message-Id": "<20220923085222.7759-1-jacopo@jmondi.org>", "X-Mailer": "git-send-email 2.37.3", "In-Reply-To": "<20220811150219.62066-1-jacopo@jmondi.org>", "References": "<20220811150219.62066-1-jacopo@jmondi.org>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "Subject": "[libcamera-devel] [PATCH 7/6] Documentation: design: ae: Document\n\tthe design for AE controls", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "From": "Jacopo Mondi via libcamera-devel <libcamera-devel@lists.libcamera.org>", "Reply-To": "Jacopo Mondi <jacopo@jmondi.org>", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "From: Paul Elder <paul.elder@ideasonboard.com>\n\nDocument the design and rationale for the AE-related controls.\n\nSigned-off-by: Paul Elder <paul.elder@ideasonboard.com>\nSigned-off-by: Jacopo Mondi <jacopo@jmondi.org>\n---\nv1->v2:\n- s/Inactive/Idle\n- s/Disabled/Manual\n---\n Documentation/design/ae.rst | 260 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 260 insertions(+)\n create mode 100644 Documentation/design/ae.rst\n\n--\n2.37.3", "diff": "diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst\nnew file mode 100644\nindex 000000000000..ecef301ba3df\n--- /dev/null\n+++ b/Documentation/design/ae.rst\n@@ -0,0 +1,260 @@\n+.. SPDX-License-Identifier: CC-BY-SA-4.0\n+\n+Design of Exposure and Gain controls\n+====================================\n+\n+This document explains the design and rationale of the controls related to\n+exposure and gain. This includes the all-encompassing auto-exposure (AE), the\n+manual exposure control, and the manual gain control.\n+\n+Description of the problem\n+--------------------------\n+\n+Sub controls\n+^^^^^^^^^^^^\n+\n+There are more than one control that make up exposure: exposure, gain, and\n+aperture (though for now we will not consider aperture). We already had\n+individual controls for setting the values of manual exposure and manual gain,\n+but for switching between auto mode and manual mode we only had a high-level\n+boolean AeEnable control that would set *both* exposure and gain to auto mode\n+or manual mode; we had no way to set one to auto and the other to manual.\n+\n+So, we need to introduce two new controls to act as \"levers\" to indicate\n+individually for exposure and gain if the value would come from AEGC or if it\n+would come from the manual control value.\n+\n+Aperture priority\n+^^^^^^^^^^^^^^^^^\n+\n+We eventually will support aperture, and so whatever our solution is for having\n+only some controls on auto and the others on manual needs to be extensible.\n+\n+Flickering when going from auto to manual\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+When a manual exposure or gain value is requested by the application, it costs\n+a few frames worth of time for them to take effect. This means that during a\n+transition from auto to manual, there would be flickering in the control values\n+and the transition won't be smooth.\n+\n+Take for instance the following flow, where we start on auto exposure (which\n+for the purposes of the example increments by 1 each frame) and we want to\n+switch seamlessly to manual exposure, which involves copying the exposure value\n+computed by the auto exposure algorithm:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual\n+ Exp requested: N/A N/A N/A 2 2 2 2\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 2 2\n+\n+As we can see, after frame N+2 completes, we copy the exposure value that was\n+used for frame N+2 (which was computed by AE algorithm), and queue that value\n+into request N+3 with manual mode on. However, as it takes two frames for the\n+exposure to be set, the exposure still changes since it is set by AE, and we\n+get a flicker in the exposure during the switch from auto to manual.\n+\n+A solution is to *not submit* any exposure value when manual mode is enabled,\n+and wait until the manual mode as been \"applied\" before copying the exposure\n+value:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual\n+ Exp requested: N/A N/A N/A None None None 5\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 5 5\n+\n+In practice, this works. However, libcamera has a policy where once a control\n+is submitted, its value is saved and does not need to be resubmitted (even\n+though this isn't implemented yet). So if the manual exposure value was set\n+while auto mode was on, in theory the value would be saved, so when manual mode\n+is enabled, the exposure value that was previously set would immediately be\n+used. Clearly this solution isn't correct, but it can serve as the basis for a\n+proper solution, with some more rigorous rules.\n+\n+Existing solutions\n+------------------\n+\n+Raspberry Pi\n+^^^^^^^^^^^^\n+\n+The raspberry pi IPA got around the lack of individual AeEnable controls for\n+exposure and gain by using magic values. When AeEnable was false, if one of the\n+manual control values was set to 0 then the value computed by AEGC would be\n+used for just that control. This solution isn't desirable, as it prevents\n+that magic value from being used as a valid value.\n+\n+To get around the flickering issue, when AeEnable was false, the raspberry pi\n+AEGC would simply stop updating the values to be set. As mentioned above, since\n+the value retention mechanism hasn't actually been implemented yet, this\n+worked. But, it's not a proper solution.\n+\n+Android\n+^^^^^^^\n+\n+The Android HAL specification requires that exposure and gain (sensitivity)\n+must both be manual or both be auto. It cannot be that one is manual while the\n+other is auto, so they simply don't support sub controls.\n+\n+For the flickering issue, the Android HAL has an AeLock control. To transition\n+from auto to manual, the application would keep AE on auto, and turn on the\n+lock. Once the lock has propagated through, then the value can be copied from\n+the result into the request and the lock disabled and the mode set to manual.\n+\n+The problem with this solution is, besides the extra complexity, that it is\n+ambiguous what happens if there is a state transition from manual to locked\n+(even though it's a state transition that doesn't make sense). If locked is\n+defined to \"use the last automatically computed values\" then it could use the\n+values from the last time it AE was set to auto, or it would be undefined if AE\n+was never auto (eg. it started out as manual), or if AE is implemented to run\n+in the background it could just use the current values that are computed. If\n+locked is defined to \"use the last value that was set\" there would be less\n+ambiguity. Still, it's better if we can make it impossible to execute this\n+nonsensical state transition, and if we can reduce the complexity of having\n+this extra control or extra setting on a lever.\n+\n+Summary of goals\n+----------------\n+\n+ - We need a lock of some sort, to instruct the AEGC to not update output\n+ results\n+\n+ - We need manual modes, to override the values computed by the AEGC\n+\n+ - We need to support seamless transitions from auto to manual, and do so\n+ without flickering\n+\n+ - We need custom minimum values for the manual controls; that is, no magic\n+ values for enabling/disabling auto\n+\n+ - All of these need to be done with AE sub-controls (exposure time, analogue\n+ gain) and be extensible to aperture in the future\n+\n+Our solution\n+------------\n+\n+A diagram of our solution:\n+\n+::\n+\n+ +----------------------------+-------------+------------------+-----------------+\n+ | INPUT | ALGORITHM | RESULT | OUTPUT |\n+ +----------------------------+-------------+------------------+-----------------+\n+\n+ ExposureTimeMode ExposureTimeMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto | |\n+ 1: Manual | V\n+ | |\\\n+ | | \\\n+ | /----------------------------------> | 1| ExposureTime\n+ | | +-------------+ exposure time | | -------------->\n+ \\--)--> | | --------------> | 0|\n+ ExposureTime | | | | /\n+ ------------------------+--> | | |/\n+ | | AeState\n+ | AEGC | ----------------------------------->\n+ AnalogueGain | |\n+ ------------------------+--> | | |\\\n+ | | | | \\\n+ /--)--> | | --------------> | 0| AnalogueGain\n+ | | +-------------+ analogue gain | | -------------->\n+ | \\----------------------------------> | 1|\n+ | | /\n+ | |/\n+ | ^\n+ AnalogueGainMode | | AnalogueGainMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto\n+ 1: Manual\n+\n+\n+The diagram is divided in four sections horizontally:\n+\n+ - Input: The values received from the request controls\n+\n+ - Algorithm: The algorithm itself\n+\n+ - Result: The values calculated by the algorithm\n+\n+ - Output: The values that sent in result metadata and applied to the device\n+\n+The four input controls are divided between manual values (ExposureTime and\n+AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The\n+former are the manual values, the latter control how they're applied. The two\n+modes are independent from each other, and each can take one of two values:\n+\n+ - Auto (0): The AGC computes the value normally. The AGC result is applied\n+ to the output. The manual value is ignored *and is not retained*.\n+\n+ - Manual (1): The AGC uses the manual value internally. The corresponding\n+ manual control from the request is applied to the output. The AGC result\n+ is ignored.\n+\n+The AeState control reports the state of the unified AEGC block. If both\n+ExposureTimeMode and AnalogueGainMode are set to disabled then it will report\n+Idle. If at least one of the two is set to auto, then AeState will report\n+if the AEGC has Converged or not (Searching). This control replaces the old\n+AeLocked control, as it was insufficient for reporting the AE state.\n+\n+There is a caveat to the disabled mode: the manual control value is not\n+retained if it is set during auto mode. This means that if the disabled mode is\n+entered without also setting the manual value, then it will enter a state\n+similar to \"locked\", where the last automatically computed value while the mode\n+was auto will be used. Once the manual value is set, then that will be used and\n+retained as usual.\n+\n+This simulates an auto -> locked -> manual or auto -> manual state transition,\n+and makes it impossible to do the nonsensical manual -> locked state\n+transition.\n+\n+We specifically do not have a \"master AE control\" like the old AeEnable. This\n+is because we have the individual mode controls, and if we had a master AE\n+control it would be a \"control that sets other controls\", which could easily\n+get out of control.\n+\n+With this solution, the earlier example would become:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab\n+ Exp requested: N/A None None None None 10 None 10 10\n+ Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10\n+\n+ Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab\n+ Exp used: 2 3 4 5 5 5 5 10 10\n+\n+This example is extended by a few frames to exhibit the simulated \"locked\"\n+state. At frame N+5 the application has confirmed that the auto mode has been\n+disabled, but does not provide a manual value until request N+7. Thus, the\n+value that is used in requests N+5 and N+6 (where the mode is disabled), comes\n+from the last value that was used when the mode was auto, which comes from\n+frame N+4.\n+\n+Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for\n+the exposure to be applied. N+8 does not supply a manual value, but the last\n+supplied value is retained, so a manual value of 10 is still used and set in\n+frame N+10.\n+\n+Although this behavior is the same as what we had with waiting for the manual\n+mode to propagate (in the section \"Description of the problem\"), this time it\n+is correct as we have defined specifically that if a manual value was specified\n+while the mode was auto, it will not be retained.\n", "prefixes": [ "libcamera-devel", "7/6" ] }