Patch Detail
Show a patch.
GET /api/1.1/patches/22333/?format=api
{ "id": 22333, "url": "https://patchwork.libcamera.org/api/1.1/patches/22333/?format=api", "web_url": "https://patchwork.libcamera.org/patch/22333/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/1.1/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20241216043954.3506855-3-paul.elder@ideasonboard.com>", "date": "2024-12-16T04:39:48", "name": "[v5,2/8] Documentation: design: ae: Document the design for AE controls", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": false, "hash": "e9e34d4f9f641fdae1fd4c09da3eec2b650d1a60", "submitter": { "id": 17, "url": "https://patchwork.libcamera.org/api/1.1/people/17/?format=api", "name": "Paul Elder", "email": "paul.elder@ideasonboard.com" }, "delegate": null, "mbox": "https://patchwork.libcamera.org/patch/22333/mbox/", "series": [ { "id": 4892, "url": "https://patchwork.libcamera.org/api/1.1/series/4892/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=4892", "date": "2024-12-16T04:39:46", "name": "AEGC controls", "version": 5, "mbox": "https://patchwork.libcamera.org/series/4892/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/patches/22333/comments/", "check": "pending", "checks": "https://patchwork.libcamera.org/api/patches/22333/checks/", "tags": {}, "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B8909C32F6\n\tfor <parsemail@patchwork.libcamera.org>;\n\tMon, 16 Dec 2024 04:40:25 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5D74B67F45;\n\tMon, 16 Dec 2024 05:40:25 +0100 (CET)", "from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 20EBD67F33\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tMon, 16 Dec 2024 05:40:22 +0100 (CET)", "from neptunite.flets-east.jp (unknown\n\t[IPv6:2404:7a81:160:2100:c11a:e451:861f:3d32])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 26F7713C;\n\tMon, 16 Dec 2024 05:39:43 +0100 (CET)" ], "Authentication-Results": "lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"LFOM3jP1\"; dkim-atps=neutral", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1734323985;\n\tbh=DFBKz3hbNdKyjkyQmA1E81djg04q2js74FlUxqygvYQ=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=LFOM3jP1G68CmaDT7JNnqHpQvEBAUBrKSEltUjHI/M9Pb6cgrjcEdJ0OwoDdBBvSI\n\thwtcSxmlyl52jKPlyYr4A9iGDjC72deh7UHuF+pp/3ICfo5fTXDpcPL5+d+dDJnN/C\n\tBG/H9IgNdTPmN7E2of4rJ2vd1yNwJkP6R15NOoGY=", "From": "Paul Elder <paul.elder@ideasonboard.com>", "To": "libcamera-devel@lists.libcamera.org", "Cc": "Paul Elder <paul.elder@ideasonboard.com>,\n\tJacopo Mondi <jacopo@jmondi.org>, \n\tStefan Klug <stefan.klug@ideasonboard.com>,\n\tLaurent Pinchart <laurent.pinchart@ideasonboard.com>", "Subject": "[PATCH v5 2/8] Documentation: design: ae: Document the design for AE\n\tcontrols", "Date": "Mon, 16 Dec 2024 13:39:48 +0900", "Message-Id": "<20241216043954.3506855-3-paul.elder@ideasonboard.com>", "X-Mailer": "git-send-email 2.39.2", "In-Reply-To": "<20241216043954.3506855-1-paul.elder@ideasonboard.com>", "References": "<20241216043954.3506855-1-paul.elder@ideasonboard.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "Document the design and rationale for the AE-related controls.\nAlso add documentation for the controls.\n\nSigned-off-by: Paul Elder <paul.elder@ideasonboard.com>\nSigned-off-by: Jacopo Mondi <jacopo@jmondi.org>\nReviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>\nReviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>\n\n---\nChanges in v5:\n- improve wording\n\nChanges in v4:\n- fix wording\n- fix compilation\n- remove locked state in explanation\n\nChanges in v3:\n- merge the control documentation into the same document (including the\n patch)\n - this is because it was a bit unwieldy to put it in\n control_ids.cpp.in, now that it's used generically to generate\n control ids of all namespaces\n---\n Documentation/design/ae.rst | 320 ++++++++++++++++++++++++++++++++++++\n Documentation/index.rst | 4 +-\n Documentation/meson.build | 1 +\n 3 files changed, 324 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/design/ae.rst", "diff": "diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst\nnew file mode 100644\nindex 000000000000..f8b2c887c2dd\n--- /dev/null\n+++ b/Documentation/design/ae.rst\n@@ -0,0 +1,320 @@\n+.. SPDX-License-Identifier: CC-BY-SA-4.0\n+\n+Design of Exposure and Gain controls\n+====================================\n+\n+This document explains the design and rationale of the controls related to\n+exposure and gain. This includes the all-encompassing auto-exposure (AE), the\n+manual exposure control, and the manual gain control.\n+\n+Description of the problem\n+--------------------------\n+\n+Sub controls\n+^^^^^^^^^^^^\n+\n+There are more than one control that make up total exposure: exposure time,\n+gain, and aperture (though for now we will not consider aperture). We already\n+had individual controls for setting the values of manual exposure and manual\n+gain, but for switching between auto mode and manual mode we only had a\n+high-level boolean AeEnable control that would set *both* exposure and gain to\n+auto mode or manual mode; we had no way to set one to auto and the other to\n+manual.\n+\n+So, we need to introduce two new controls to act as \"levers\" to indicate\n+individually for exposure and gain if the value would come from AEGC or if it\n+would come from the manual control value.\n+\n+Aperture priority\n+^^^^^^^^^^^^^^^^^\n+\n+We eventually may need to support aperture, and so whatever our solution is for\n+having only some controls on auto and the others on manual needs to be\n+extensible.\n+\n+Flickering when going from auto to manual\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+When a manual exposure or gain value is requested by the application, it costs\n+a few frames worth of time for them to take effect. This means that during a\n+transition from auto to manual, there would be flickering in the control values\n+and the transition won't be smooth.\n+\n+Take for instance the following flow, where we start on auto exposure (which\n+for the purposes of the example increments by 1 each frame) and we want to\n+switch seamlessly to manual exposure, which involves copying the exposure value\n+computed by the auto exposure algorithm:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual\n+ Exp requested: N/A N/A N/A 2 2 2 2\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 2 2\n+\n+As we can see, after frame N+2 completes, we copy the exposure value that was\n+used for frame N+2 (which was computed by AE algorithm), and queue that value\n+into request N+3 with manual mode on. However, as it takes two frames for the\n+exposure to be set, the exposure still changes since it is set by AE, and we\n+get a flicker in the exposure during the switch from auto to manual.\n+\n+A solution is to *not submit* any exposure value when manual mode is enabled,\n+and wait until the manual mode as been \"applied\" before copying the exposure\n+value:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+\n+ Mode requested: Auto Auto Auto Manual Manual Manual Manual\n+ Exp requested: N/A N/A N/A None None None 5\n+ Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8\n+\n+ Mode used: Auto Auto Auto Auto Auto Manual Manual\n+ Exp used: 0 1 2 3 4 5 5\n+\n+In practice, this works. However, libcamera has a policy where once a control\n+is submitted, its value is saved and does not need to be resubmitted. If the\n+manual exposure value was set while auto mode was on, in theory the value would\n+be saved, so when manual mode is enabled, the exposure value that was\n+previously set would immediately be used. Clearly this solution isn't correct,\n+but it can serve as the basis for a proper solution, with some more rigorous\n+rules.\n+\n+Existing solutions\n+------------------\n+\n+Raspberry Pi\n+^^^^^^^^^^^^\n+\n+The Raspberry Pi IPA gets around the lack of individual AeEnable controls for\n+exposure and gain by using magic values. When AeEnable is false, if one of the\n+manual control values was set to 0 then the value computed by AEGC would be\n+used for just that control. This solution isn't desirable, as it prevents\n+that magic value from being used as a valid value.\n+\n+To get around the flickering issue, when AeEnable is false, the Raspberry Pi\n+AEGC simply stops updating the values to be set, without restoring the\n+previously set manual exposure time and gain. This works, but is not a proper\n+solution.\n+\n+Android\n+^^^^^^^\n+\n+The Android HAL specification requires that exposure and gain (sensitivity)\n+must both be manual or both be auto. It cannot be that one is manual while the\n+other is auto, so they simply don't support sub controls.\n+\n+For the flickering issue, the Android HAL has an AeLock control. To transition\n+from auto to manual, the application would keep AE on auto, and turn on the\n+lock. Once the lock has propagated through, then the value can be copied from\n+the result into the request and the lock disabled and the mode set to manual.\n+\n+The problem with this solution is, besides the extra complexity, that it is\n+ambiguous what happens if there is a state transition from manual to locked\n+(even though it's a state transition that doesn't make sense). If locked is\n+defined to \"use the last automatically computed values\" then it could use the\n+values from the last time it AE was set to auto, or it would be undefined if AE\n+was never auto (eg. it started out as manual), or if AE is implemented to run\n+in the background it could just use the current values that are computed. If\n+locked is defined to \"use the last value that was set\" there would be less\n+ambiguity. Still, it's better if we can make it impossible to execute this\n+nonsensical state transition, and if we can reduce the complexity of having\n+this extra control or extra setting on a lever.\n+\n+Summary of goals\n+----------------\n+\n+- We need a lock of some sort, to instruct the AEGC to not update output\n+ results\n+\n+- We need manual modes, to override the values computed by the AEGC\n+\n+- We need to support seamless transitions from auto to manual, and do so\n+ without flickering\n+\n+- We need custom minimum values for the manual controls; that is, no magic\n+ values for enabling/disabling auto\n+\n+- All of these need to be done with AE sub-controls (exposure time, analogue\n+ gain) and be extensible to aperture in the future\n+\n+Our solution\n+------------\n+\n+A diagram of our solution:\n+\n+::\n+\n+ +----------------------------+-------------+------------------+-----------------+\n+ | INPUT | ALGORITHM | RESULT | OUTPUT |\n+ +----------------------------+-------------+------------------+-----------------+\n+\n+ ExposureTimeMode ExposureTimeMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto | |\n+ 1: Manual | V\n+ | |\\\n+ | | \\\n+ | /----------------------------------> | 1| ExposureTime\n+ | | +-------------+ exposure time | | -------------->\n+ \\--)--> | | --------------> | 0|\n+ ExposureTime | | | | /\n+ ------------------------+--> | | |/\n+ | | AeState\n+ | AEGC | ----------------------------------->\n+ AnalogueGain | |\n+ ------------------------+--> | | |\\\n+ | | | | \\\n+ /--)--> | | --------------> | 0| AnalogueGain\n+ | | +-------------+ analogue gain | | -------------->\n+ | \\----------------------------------> | 1|\n+ | | /\n+ | |/\n+ | ^\n+ AnalogueGainMode | | AnalogueGainMode\n+ ---------------------+----------------------------------------+----------------->\n+ 0: Auto\n+ 1: Manual\n+\n+\n+The diagram is divided in four sections horizontally:\n+\n+- Input: The values received from the request controls\n+\n+- Algorithm: The algorithm itself\n+\n+- Result: The values calculated by the algorithm\n+\n+- Output: The values reported in result metadata and applied to the device\n+\n+The four input controls are divided between manual values (ExposureTime and\n+AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The\n+former are the manual values, the latter control how they're applied. The two\n+modes are independent from each other, and each can take one of two values:\n+\n+ - Auto (0): The AGC computes the value normally. The AGC result is applied\n+ to the output. The manual value is ignored *and is not retained*.\n+\n+ - Manual (1): The AGC uses the manual value internally. The corresponding\n+ manual control from the request is applied to the output. The AGC result\n+ is ignored.\n+\n+The AeState control reports the state of the unified AEGC block. If both\n+ExposureTimeMode and AnalogueGainMode are set to manual then it will report\n+Idle. If at least one of the two is set to auto, then AeState will report\n+if the AEGC has Converged or not (Searching). This control replaces the old\n+AeLocked control, as it was insufficient for reporting the AE state.\n+\n+There is a caveat to manual mode: the manual control value is not retained if\n+it is set during auto mode. This means that if manual mode is entered without\n+also setting the manual value, then it will enter a state similar to \"locked\",\n+where the last automatically computed value while the mode was auto will be\n+used. Once the manual value is set, then that will be used and retained as\n+usual.\n+\n+This simulates an auto -> locked -> manual or auto -> manual state transition,\n+and makes it impossible to do the nonsensical manual -> locked state\n+transition.\n+\n+We specifically do not have a \"master AE control\" like the old AeEnable. This\n+is because we have the individual mode controls, and if we had a master AE\n+control it would be a \"control that sets other controls\", which could easily\n+get out of control.\n+\n+With this solution, the earlier example would become:\n+\n+::\n+\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|\n+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+\n+ Mode requested: Auto Manual Manual Manual Manual Manual Manual Manual Manual\n+ Exp requested: N/A None None None None 10 None 10 10\n+ Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+11 N+12\n+\n+ Mode used: Auto Auto Auto Manual Manual Manual Manual Manual Manual\n+ Exp used: 2 3 4 5 5 5 5 10 10\n+\n+This example is extended by a few frames to exhibit the simulated \"locked\"\n+state. At frame N+5 the application has confirmed that the manual mode has been\n+entered, but does not provide a manual value until request N+7. Thus, the value\n+that is used in requests N+5 and N+6 (where the mode is disabled), comes from\n+the last value that was used when the mode was auto, which comes from frame\n+N+4.\n+\n+Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for\n+the exposure to be applied. N+8 does not supply a manual value, but the last\n+supplied value is retained, so a manual value of 10 is still used and set in\n+frame N+10.\n+\n+Although this behavior is the same as what we had with waiting for the manual\n+mode to propagate (in the section \"Description of the problem\"), this time it\n+is correct as we have defined specifically that if a manual value was specified\n+while the mode was auto, it will not be retained.\n+\n+Description of the controls\n+---------------------------\n+\n+As described above, libcamera offers the following controls related to exposure\n+and gain:\n+\n+- AnalogueGain\n+\n+- AnalogueGainMode\n+\n+- ExposureTime\n+\n+- ExposureTimeMode\n+\n+- AeState\n+\n+Auto-exposure and auto-gain can be enabled and disabled separately using the\n+ExposureTimeMode and AnalogueGainMode controls respectively. There is no\n+overarching AeEnable control.\n+\n+When the respective mode is set to auto, the respective value that is computed\n+by the AEGC algorithm is applied to the image sensor. Any value that is\n+supplied in the manual ExposureTime/AnalogueGain control is ignored and not\n+retained. Another way to understand this is that when the mode transitions from\n+auto to manual, the internally stored control value is overwritten with the\n+last value computed by the auto algorithm.\n+\n+This means that when we transition from auto to manual without supplying a\n+manual control value, the last value that was set by the AEGC algorithm will\n+keep be used. This can be used to do a flickerless transition from auto to\n+manual as described earlier. If the camera started out in manual mode and no\n+corresponding value has been supplied yet, then a best-effort default value\n+shall be set.\n+\n+The manual control value can be set in the same request as setting the mode to\n+auto if the desired manual control value is already known.\n+\n+Transitioning from manual to auto shall be implicitly flickerless, as the AEGC\n+algorithms are expected to start running from the last manual value.\n+\n+The AeState metadata reports the state of the AE algorithm. As AE cannot\n+compute exposure and gain separately, the state of the AE component is\n+unified. There are three states: Idle, Searching, and Converged.\n+\n+The state shall be Idle if both ExposureTimeMode and AnalogueGainMode\n+are set to Manual. If the camera only supports one of the two controls,\n+then the state shall be Idle if that one control is set to Manual. If\n+the camera does not support Manual for at least one of the two controls,\n+then the state will never be Idle, as AE will always be running.\n+\n+The state shall be Searching if at least one of exposure or gain calculated\n+by the AE algorithm is used (that is, at least one of the two modes is Auto),\n+*and* the value(s) have not converged yet.\n+\n+The state shall be Converged if at least one of exposure or gain calculated\n+by the AE algorithm is used (that is, at least one of the two modes is Auto),\n+*and* the value(s) have converged.\ndiff --git a/Documentation/index.rst b/Documentation/index.rst\nindex bea406608477..251112fbdf5f 100644\n--- a/Documentation/index.rst\n+++ b/Documentation/index.rst\n@@ -23,7 +23,9 @@\n SoftwareISP Benchmarking <software-isp-benchmarking>\n Tracing guide <guides/tracing>\n \n+ Design document: AE <design/ae>\n+\n .. toctree::\n :hidden:\n \n- introduction\n\\ No newline at end of file\n+ introduction\ndiff --git a/Documentation/meson.build b/Documentation/meson.build\nindex 36ffae239353..6158320e177c 100644\n--- a/Documentation/meson.build\n+++ b/Documentation/meson.build\n@@ -128,6 +128,7 @@ if sphinx.found()\n 'coding-style.rst',\n 'conf.py',\n 'contributing.rst',\n+ 'design/ae.rst',\n 'documentation-contents.rst',\n 'environment_variables.rst',\n 'feature_requirements.rst',\n", "prefixes": [ "v5", "2/8" ] }