{"id":15971,"url":"https://patchwork.libcamera.org/api/patches/15971/?format=json","web_url":"https://patchwork.libcamera.org/patch/15971/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220518134728.777709-3-paul.elder@ideasonboard.com>","date":"2022-05-18T13:47:27","name":"[libcamera-devel,2/3] Documentation: design: ae: Document the design for AE controls","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"a5026a3ff60b1b00e38f4979e1356be308b4f95a","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/?format=json","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/15971/mbox/","series":[{"id":3123,"url":"https://patchwork.libcamera.org/api/series/3123/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3123","date":"2022-05-18T13:47:25","name":"AE controls","version":1,"mbox":"https://patchwork.libcamera.org/series/3123/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/15971/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/15971/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0649AC326C\n\tfor ;\n\tWed, 18 May 2022 13:47:49 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5A3E76565D;\n\tWed, 18 May 2022 15:47:49 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CBCA065659\n\tfor ;\n\tWed, 18 May 2022 15:47:44 +0200 (CEST)","from localhost.localdomain (unknown [45.131.31.124])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5D273E50;\n\tWed, 18 May 2022 15:47:44 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1652881669;\n\tbh=40PKtqYJq2vAzT03Flwq9s8stKjhU/o0uvoNy8Xckrw=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=X+jwu590/aRTqy0p2+HhCMVyTS8mxrJ5B+739Zjz8M2QNWPvx4Nd7QyMAyCZyXJzh\n\tas+UcJUOwho2rc8KNzhkB/A+kHBzrCQLhFZesOVMpuMsHWcvXONVDcw2gW6UNx3tpX\n\t0zPw2LApUwq6dMpxP1Jn/wMeoFBX8LFL0kM2Nm4XRlqYqB3kYlUj+bO329och2lrHA\n\tHXPtjXwaV5+4pOoBar3uLFD6XgCXBDL7aSagLk3wRMh02sFhf7ueG8tLEX1OHbilM5\n\tRZNBDnWrngcvtAZtmFzf/eOkpzkVv8RmP6XC5eR2zb+FsA2YG7yOVe6asVyUBL/B+Z\n\tByyMOKivYcJFQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1652881664;\n\tbh=40PKtqYJq2vAzT03Flwq9s8stKjhU/o0uvoNy8Xckrw=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=NQyjAyeSNJfvOSB1bnqduXv53ps0qnUOH3DYCjUJASMK7gmItxIDchVWN4OkKPl1g\n\tdtCDpRGqAK4UhhFznYe6O+x7vIYC87Xukk7kWhFMJSBJR7bPch/sRS2ZdqKYfvsJOX\n\tt82i51OsXef4wlYcvG9IfPAxsw+0z6jahDpn5M3M="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"NQyjAyeS\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Wed, 18 May 2022 15:47:27 +0200","Message-Id":"<20220518134728.777709-3-paul.elder@ideasonboard.com>","X-Mailer":"git-send-email 2.30.2","In-Reply-To":"<20220518134728.777709-1-paul.elder@ideasonboard.com>","References":"<20220518134728.777709-1-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 2/3] Documentation: design: ae: Document\n\tthe design for AE controls","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Paul Elder via libcamera-devel ","Reply-To":"Paul Elder ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Document the design and rationale for the AE-related controls.\n\nSigned-off-by: Paul Elder \n---\n Documentation/design/ae.rst | 260 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 260 insertions(+)\n create mode 100644 Documentation/design/ae.rst","diff":"diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst\nnew file mode 100644\nindex 00000000..09ca2ee7\n--- /dev/null\n+++ b/Documentation/design/ae.rst\n@@ -0,0 +1,260 @@\n+.. SPDX-License-Identifier: CC-BY-SA-4.0\n+\n+Design of Exposure and Gain controls\n+====================================\n+\n+This document explains the design and rationale of the controls related to\n+exposure and gain. This includes the all-encompassing auto-exposure (AE), the\n+manual exposure control, and the manual gain control.\n+\n+Description of the problem\n+--------------------------\n+\n+Sub controls\n+^^^^^^^^^^^^\n+\n+There are more than one control that make up exposure: exposure, gain, and\n+aperture (though for now we will not consider aperture). We already had\n+individual controls for setting the values of manual exposure and manual gain,\n+but for switching between auto mode and manual mode we only had a high-level\n+boolean AeEnable control that would set *both* exposure and gain to auto mode\n+or manual mode; we had no way to set one to auto and the other to manual.\n+\n+So, we need to introduce two new controls to act as \"levers\" to indicate\n+individually for exposure and gain if the value would come from AEGC or if it\n+would come from the manual control value.\n+\n+Aperture priority\n+^^^^^^^^^^^^^^^^^\n+\n+We eventually will support aperture, and so whatever our solution is for having\n+only some controls on auto and the others on manual needs to be extensible.\n+\n+Flickering when going from auto to manual\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+When a manual exposure or gain value is requested by the application, it costs\n+a few frames worth of time for them to take effect. This means that during a\n+transition from auto to manual, there would be flickering in the control values\n+and the transition won't be smooth.\n+\n+Take for instance the following flow, where we start on auto exposure (which\n+for the purposes of the example increments by 1 each frame) and we want to\n+switch seamlessly to manual exposure, which involves copying the exposure value\n+computed by the auto exposure algorithm:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual \n+ Exp requested: N/A N/A N/A 2 2 2 2\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 2 2\n+\n+As we can see, after frame N+2 completes, we copy the exposure value that was\n+used for frame N+2 (which was computed by AE algorithm), and queue that value\n+into request N+3 with manual mode on. However, as it takes two frames for the\n+exposure to be set, the exposure still changes since it is set by AE, and we\n+get a flicker in the exposure during the switch from auto to manual.\n+\n+A solution is to *not submit* any exposure value when manual mode is enabled,\n+and wait until the manual mode as been \"applied\" before copying the exposure\n+value:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual \n+ Exp requested: N/A N/A N/A None None None 5\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 5 5\n+\n+In practice, this works. However, libcamera has a policy where once a control\n+is submitted, its value is saved and does not need to be resubmitted (even\n+though this isn't implemented yet). So if the manual exposure value was set\n+while auto mode was on, in theory the value would be saved, so when manual mode\n+is enabled, the exposure value that was previously set would immediately be\n+used. Clearly this solution isn't correct, but it can serve as the basis for a\n+proper solution, with some more rigorous rules.\n+\n+Existing solutions\n+------------------\n+\n+Raspberry Pi\n+^^^^^^^^^^^^\n+\n+The raspberry pi IPA got around the lack of individual AeEnable controls for\n+exposure and gain by using magic values. When AeEnable was false, if one of the\n+manual control values was set to 0 then the value computed by AEGC would be\n+used for just that control. This solution isn't desirable, as it prevents\n+that magic value from being used as a valid value.\n+\n+To get around the flickering issue, when AeEnable was false, the raspberry pi\n+AEGC would simply stop updating the values to be set. As mentioned above, since\n+the value retention mechanism hasn't actually been implemented yet, this\n+worked. But, it's not a proper solution.\n+\n+Android\n+^^^^^^^\n+\n+The Android HAL specification requires that exposure and gain (sensitivity)\n+must both be manual or both be auto. It cannot be that one is manual while the\n+other is auto, so they simply don't support sub controls.\n+\n+For the flickering issue, the Android HAL has an AeLock control. To transition\n+from auto to manual, the application would keep AE on auto, and turn on the\n+lock. Once the lock has propagated through, then the value can be copied from\n+the result into the request and the lock disabled and the mode set to manual.\n+\n+The problem with this solution is, besides the extra complexity, that it is\n+ambiguous what happens if there is a state transition from manual to locked\n+(even though it's a state transition that doesn't make sense). If locked is\n+defined to \"use the last automatically computed values\" then it could use the\n+values from the last time it AE was set to auto, or it would be undefined if AE\n+was never auto (eg. it started out as manual), or if AE is implemented to run\n+in the background it could just use the current values that are computed. If\n+locked is defined to \"use the last value that was set\" there would be less\n+ambiguity. Still, it's better if we can make it impossible to execute this\n+nonsensical state transition, and if we can reduce the complexity of having\n+this extra control or extra setting on a lever.\n+\n+Summary of goals\n+----------------\n+\n+ - We need a lock of some sort, to instruct the AEGC to not update output\n+ results\n+\n+ - We need manual modes, to override the values computed by the AEGC\n+\n+ - We need to support seamless transitions from auto to manual, and do so\n+ without flickering\n+\n+ - We need custom minimum values for the manual controls; that is, no magic\n+ values for enabling/disabling auto\n+\n+ - All of these need to be done with AE sub-controls (exposure time, analogue\n+ gain) and be extensible to aperture in the future\n+\n+Our solution\n+------------\n+\n+A diagram of our solution:\n+\n+::\n+\n+ +----------------------------+-------------+------------------+-----------------+\n+ | INPUT | ALGORITHM | RESULT | OUTPUT |\n+ +----------------------------+-------------+------------------+-----------------+\n+\n+ ExposureTimeMode ExposureTimeMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto | |\n+ 1: Disabled | V\n+ | |\\\n+ | | \\\n+ | /----------------------------------> | 1| ExposureTime\n+ | | +-------------+ exposure time | | -------------->\n+ \\--)--> | | --------------> | 0|\n+ ExposureTime | | | | /\n+ ------------------------+--> | | |/\n+ | | AeState\n+ | AEGC | ----------------------------------->\n+ AnalogueGain | |\n+ ------------------------+--> | | |\\\n+ | | | | \\\n+ /--)--> | | --------------> | 0| AnalogueGain\n+ | | +-------------+ analogue gain | | -------------->\n+ | \\----------------------------------> | 1|\n+ | | /\n+ | |/\n+ | ^\n+ AnalogueGainMode | | AnalogueGainMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto\n+ 1: Disabled\n+\n+\n+The diagram is divided in four sections horizontally:\n+\n+ - Input: The values received from the request controls\n+\n+ - Algorithm: The algorithm itself\n+\n+ - Result: The values calculated by the algorithm\n+\n+ - Output: The values that sent in result metadata and applied to the device\n+\n+The four input controls are divided between manual values (ExposureTime and\n+AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The\n+former are the manual values, the latter control how they're applied. The two\n+modes are independent from each other, and each can take one of two values:\n+\n+ - Auto (0): The AGC computes the value normally. The AGC result is applied\n+ to the output. The manual value is ignored *and is not retained*.\n+\n+ - Disabled (1): The AGC uses the manual value internally. The corresponding\n+ manual control from the request is applied to the output. The AGC result\n+ is ignored.\n+\n+The AeState control reports the state of the unified AEGC block. If both\n+ExposureTimeMode and AnalogueGainMode are set to disabled then it will report\n+Inactive. If at least one of the two is set to auto, then AeState will report\n+if the AEGC has Converged or not (Searching). This control replaces the old\n+AeLocked control, as it was insufficient for reporting the AE state.\n+\n+There is a caveat to the disabled mode: the manual control value is not\n+retained if it is set during auto mode. This means that if the disabled mode is\n+entered without also setting the manual value, then it will enter a state\n+similar to \"locked\", where the last automatically computed value while the mode\n+was auto will be used. Once the manual value is set, then that will be used and\n+retained as usual.\n+\n+This simulates an auto -> locked -> manual or auto -> manual state transition,\n+and makes it impossible to do the nonsensical manual -> locked state\n+transition.\n+\n+We specifically do not have a \"master AE control\" like the old AeEnable. This\n+is because we have the individual mode controls, and if we had a master AE\n+control it would be a \"control that sets other controls\", which could easily\n+get out of control.\n+\n+With this solution, the earlier example would become:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab\n+ Exp requested: N/A None None None None 10 None 10 10\n+ Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10\n+\n+ Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab\n+ Exp used: 2 3 4 5 5 5 5 10 10\n+\n+This example is extended by a few frames to exhibit the simulated \"locked\"\n+state. At frame N+5 the application has confirmed that the auto mode has been\n+disabled, but does not provide a manual value until request N+7. Thus, the\n+value that is used in requests N+5 and N+6 (where the mode is disabled), comes\n+from the last value that was used when the mode was auto, which comes from\n+frame N+4.\n+\n+Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for\n+the exposure to be applied. N+8 does not supply a manual value, but the last\n+supplied value is retained, so a manual value of 10 is still used and set in\n+frame N+10.\n+\n+Although this behavior is the same as what we had with waiting for the manual\n+mode to propagate (in the section \"Description of the problem\"), this time it\n+is correct as we have defined specifically that if a manual value was specified\n+while the mode was auto, it will not be retained.\n","prefixes":["libcamera-devel","2/3"]}