[{"id":32771,"web_url":"https://patchwork.libcamera.org/comment/32771/","msgid":"<20241216142044.GU32204@pendragon.ideasonboard.com>","date":"2024-12-16T14:20:44","subject":"Re: [PATCH v5 1/8] controls: Introduce AEGC-related controls","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Paul,\n\nThank you for the patch.\n\nOn Mon, Dec 16, 2024 at 01:39:47PM +0900, Paul Elder wrote:\n> Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> to model the AEGC algorithm block.\n> \n> The three controls allow applications to select the exposure time and\n> analogue gain computation calculation mode (auto vs manual)\n> independently from one another, while the AeState control reports the\n> global state for the AEGC algorithm.\n> \n> The new controls are meant to replace the existing AeEnable and AeLocked\n> controls, which are momentarily kept not to break compilation of\n> platforms making use of them.\n> \n> Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> Signed-off-by: Paul Elder \n> Signed-off-by: Jacopo Mondi \n\nReviewed-by: Laurent Pinchart \n\n> ---\n> Changes in v5:\n> - improve wordings\n> \n> Changes in v4:\n> - fix wordings\n> - add mention on transition from manual to auto\n> - add rules regarding the allowed combinations of control info\n> \n> Changes in v3:\n> - recovered from 2-year-old bitrot\n> ---\n> src/libcamera/control_ids_core.yaml | 262 +++++++++++++++++++++++++--\n> src/libcamera/control_ids_draft.yaml | 29 ---\n> 2 files changed, 242 insertions(+), 49 deletions(-)\n> \n> diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> index d45cf8e56187..a64d567791c3 100644\n> --- a/src/libcamera/control_ids_core.yaml\n> +++ b/src/libcamera/control_ids_core.yaml\n> @@ -26,6 +26,70 @@ controls:\n> \n> \\sa AeEnable\n> \n> + - AeState:\n> + type: int32_t\n> + description: |\n> + Report the AEGC algorithm state.\n> +\n> + The AEGC algorithm computes the exposure time and the analogue gain\n> + to be applied to the image sensor.\n> +\n> + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> + AnalogueGainMode controls, which allow applications to decide how\n> + the exposure time and gain are computed, in Auto or Manual mode,\n> + independently from one another.\n> +\n> + The AeState control reports the AEGC algorithm state through a single\n> + value and describes it as a single computation block which computes\n> + both the exposure time and the analogue gain values.\n> +\n> + When both the exposure time and analogue gain values are configured to\n> + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> + compute any value and the AeState control will report AeStateIdle.\n> +\n> + When at least the exposure time or analogue gain are configured to be\n> + computed by the AEGC algorithm, the AeState control will report if the\n> + algorithm has converged to stable values for all of the controls set\n> + to be computed in Auto mode.\n> +\n> + \\sa AnalogueGainMode\n> + \\sa ExposureTimeMode\n> +\n> + enum:\n> + - name: AeStateIdle\n> + value: 0\n> + description: |\n> + The AEGC algorithm is inactive.\n> +\n> + This state is returned when both AnalogueGainMode and\n> + ExposureTimeMode are set to Manual and the algorithm is not\n> + actively computing any value.\n> + - name: AeStateSearching\n> + value: 1\n> + description: |\n> + The AEGC algorithm is actively computing new values, for either the\n> + exposure time or the analogue gain, but has not converged to a\n> + stable result yet.\n> +\n> + This state is returned if at least one of AnalogueGainMode or\n> + ExposureTimeMode is auto and the algorithm hasn't converged yet.\n> +\n> + The AEGC algorithm converges once stable values are computed for\n> + all of the controls set to be computed in Auto mode. Once the\n> + algorithm converges the state is moved to AeStateConverged.\n> + - name: AeStateConverged\n> + value: 2\n> + description: |\n> + The AEGC algorithm has converged.\n> +\n> + This state is returned if at least one of AnalogueGainMode or\n> + ExposureTimeMode is Auto, and the AEGC algorithm has converged to a\n> + stable value.\n> +\n> + If the measurements move too far away from the convergence point\n> + then the AEGC algorithm might start adjusting again, in which case\n> + the state is moved to AeStateSearching.\n> +\n> # AeMeteringMode needs further attention:\n> # - Auto-generate max enum value.\n> # - Better handling of custom types.\n> @@ -104,6 +168,13 @@ controls:\n> The exposure modes specify how the desired total exposure is divided\n> between the exposure time and the sensor's analogue gain. They are\n> platform specific, and not all exposure modes may be supported.\n> +\n> + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> + the fixed values will override any choices made by AeExposureMode.\n> +\n> + \\sa AnalogueGainMode\n> + \\sa ExposureTimeMode\n> +\n> enum:\n> - name: ExposureNormal\n> value: 0\n> @@ -124,13 +195,15 @@ controls:\n> Specify an Exposure Value (EV) parameter.\n> \n> The EV parameter will only be applied if the AE algorithm is currently\n> - enabled.\n> + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> + are in Auto mode.\n> \n> By convention EV adjusts the exposure as log2. For example\n> EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> \n> - \\sa AeEnable\n> + \\sa AnalogueGainMode\n> + \\sa ExposureTimeMode\n> \n> - ExposureTime:\n> type: int32_t\n> @@ -139,17 +212,107 @@ controls:\n> \n> This value is specified in micro-seconds.\n> \n> - Setting this value means that it is now fixed and the AE algorithm may\n> - not change it. Setting it back to zero returns it to the control of the\n> - AE algorithm.\n> + This control will only take effect if ExposureTimeMode is Manual. If\n> + this control is set when ExposureTimeMode is Auto, the value will be\n> + ignored and will not be retained.\n> +\n> + When reported in metadata, this control indicates what exposure time\n> + was used for the current frame, regardless of ExposureTimeMode.\n> + ExposureTimeMode will indicate the source of the exposure time value,\n> + whether it came from the AE algorithm or not.\n> +\n> + \\sa AnalogueGain\n> + \\sa ExposureTimeMode\n> +\n> + - ExposureTimeMode:\n> + type: int32_t\n> + description: |\n> + Controls the source of the exposure time that is applied to the image\n> + sensor.\n> +\n> + When set to Auto, the AE algorithm computes the exposure time and\n> + configures the image sensor accordingly. When set to Manual, the value\n> + of the ExposureTime control is used.\n> +\n> + When transitioning from Auto to Manual mode and no ExposureTime control\n> + is provided by the application, the last value computed by the AE\n> + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> + was never set to Auto (either because the camera started in Manual mode,\n> + or Auto is not supported by the camera), the camera should use a\n> + best-effort default value.\n> \n> - \\sa AnalogueGain AeEnable\n> + If ExposureTimeModeManual is supported, the ExposureTime control must\n> + also be supported.\n> \n> - \\todo Document the interactions between AeEnable and setting a fixed\n> - value for this control. Consider interactions with other AE features,\n> - such as aperture and aperture/shutter priority mode, and decide if\n> - control of which features should be automatically adjusted shouldn't\n> - better be handled through a separate AE mode control.\n> + Cameras that support manual control of the sensor shall support manual\n> + mode for both ExposureTimeMode and AnalogueGainMode, and shall expose\n> + the ExposureTime and AnalogueGain controls. If the camera also has an\n> + AEGC implementation, both ExposureTimeMode and AnalogueGainMode shall\n> + support both manual and auto mode. If auto mode is available, it shall\n> + be the default mode. These rules do not apply to black box cameras\n> + such as UVC cameras, where the available gain and exposure modes are\n> + completely dependent on what the device exposes.\n> +\n> + \\par Flickerless exposure mode transitions\n> +\n> + Applications that wish to transition from ExposureTimeModeAuto to direct\n> + control of the exposure time without causing extra flicker can do so by\n> + selecting an ExposureTime value as close as possible to the last value\n> + computed by the auto exposure algorithm in order to avoid any visible\n> + flickering.\n> +\n> + To select the correct value to use as ExposureTime value, applications\n> + should accommodate the natural delay in applying controls caused by the\n> + capture pipeline frame depth.\n> +\n> + When switching to manual exposure mode, applications should not\n> + immediately specify an ExposureTime value in the same request where\n> + ExposureTimeMode is set to Manual. They should instead wait for the\n> + first Request where ExposureTimeMode is reported as\n> + ExposureTimeModeManual in the Request metadata, and use the reported\n> + ExposureTime to populate the control value in the next Request to be\n> + queued to the Camera.\n> +\n> + The implementation of the auto-exposure algorithm should equally try to\n> + minimize flickering and when transitioning from manual exposure mode to\n> + auto exposure use the last value provided by the application as starting\n> + point.\n> +\n> + 1. Start with ExposureTimeMode set to Auto\n> +\n> + 2. Set ExposureTimeMode to Manual\n> +\n> + 3. Wait for the first completed request that has ExposureTimeMode\n> + set to Manual\n> +\n> + 4. Copy the value reported in ExposureTime into a new request, and\n> + submit it\n> +\n> + 5. Proceed to run manual exposure time as desired\n> +\n> + \\sa ExposureTime\n> + enum:\n> + - name: ExposureTimeModeAuto\n> + value: 0\n> + description: |\n> + The exposure time will be calculated automatically and set by the\n> + AE algorithm.\n> +\n> + If ExposureTime is set while this mode is active, it will be\n> + ignored, and its value will not be retained.\n> +\n> + When transitioning from Manual to Auto mode, the AEGC should start\n> + its adjustments based on the last set manual ExposureTime value.\n> + - name: ExposureTimeModeManual\n> + value: 1\n> + description: |\n> + The exposure time will not be updated by the AE algorithm.\n> +\n> + When transitioning from Auto to Manual mode, the last computed\n> + exposure value is used until a new value is specified through the\n> + ExposureTime control. If an ExposureTime value is specified in the\n> + same request where the ExposureTimeMode is changed from Auto to\n> + Manual, the provided ExposureTime is applied immediately.\n> \n> - AnalogueGain:\n> type: float\n> @@ -159,17 +322,76 @@ controls:\n> The value of the control specifies the gain multiplier applied to all\n> colour channels. This value cannot be lower than 1.0.\n> \n> - Setting this value means that it is now fixed and the AE algorithm may\n> - not change it. Setting it back to zero returns it to the control of the\n> - AE algorithm.\n> + This control will only take effect if AnalogueGainMode is Manual. If\n> + this control is set when AnalogueGainMode is Auto, the value will be\n> + ignored and will not be retained.\n> +\n> + When reported in metadata, this control indicates what analogue gain\n> + was used for the current request, regardless of AnalogueGainMode.\n> + AnalogueGainMode will indicate the source of the analogue gain value,\n> + whether it came from the AEGC algorithm or not.\n> +\n> + \\sa ExposureTime\n> + \\sa AnalogueGainMode\n> +\n> + - AnalogueGainMode:\n> + type: int32_t\n> + description: |\n> + Controls the source of the analogue gain that is applied to the image\n> + sensor.\n> +\n> + When set to Auto, the AEGC algorithm computes the analogue gain and\n> + configures the image sensor accordingly. When set to Manual, the value\n> + of the AnalogueGain control is used.\n> +\n> + When transitioning from Auto to Manual mode and no AnalogueGain control\n> + is provided by the application, the last value computed by the AEGC\n> + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> + was never set to Auto (either because the camera started in Manual mode,\n> + or Auto is not supported by the camera), the camera should use a\n> + best-effort default value.\n> +\n> + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> + also be supported.\n> +\n> + For cameras where we have control over the ISP, both ExposureTimeMode\n> + and AnalogueGainMode are expected to support manual mode, and both\n> + controls (as well as ExposureTimeMode and AnalogueGain) are expected to\n> + be present. If the camera also has an AEGC implementation, both\n> + ExposureTimeMode and AnalogueGainMode shall support both manual and\n> + auto mode. If auto mode is available, it shall be the default mode.\n> + These rules do not apply to black box cameras such as UVC cameras,\n> + where the available gain and exposure modes are completely dependent on\n> + what the hardware exposes.\n> +\n> + The same procedure described for performing flickerless transitions in\n> + the ExposureTimeMode control documentation can be applied to analogue\n> + gain.\n> +\n> + \\sa ExposureTimeMode\n> + \\sa AnalogueGain\n> + enum:\n> + - name: AnalogueGainModeAuto\n> + value: 0\n> + description: |\n> + The analogue gain will be calculated automatically and set by the\n> + AEGC algorithm.\n> +\n> + If AnalogueGain is set while this mode is active, it will be\n> + ignored, and it will also not be retained.\n> \n> - \\sa ExposureTime AeEnable\n> + When transitioning from Manual to Auto mode, the AEGC should start\n> + its adjustments based on the last set manual AnalogueGain value.\n> + - name: AnalogueGainModeManual\n> + value: 1\n> + description: |\n> + The analogue gain will not be updated by the AEGC algorithm.\n> \n> - \\todo Document the interactions between AeEnable and setting a fixed\n> - value for this control. Consider interactions with other AE features,\n> - such as aperture and aperture/shutter priority mode, and decide if\n> - control of which features should be automatically adjusted shouldn't\n> - better be handled through a separate AE mode control.\n> + When transitioning from Auto to Manual mode, the last computed\n> + gain value is used until a new value is specified through the\n> + AnalogueGain control. If an AnalogueGain value is specified in the\n> + same request where the AnalogueGainMode is changed from Auto to\n> + Manual, the provided AnalogueGain is applied immediately.\n> \n> - AeFlickerMode:\n> type: int32_t\n> diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> index 1b284257f601..32c2b3feaf65 100644\n> --- a/src/libcamera/control_ids_draft.yaml\n> +++ b/src/libcamera/control_ids_draft.yaml\n> @@ -77,35 +77,6 @@ controls:\n> High quality aberration correction which might reduce the frame\n> rate.\n> \n> - - AeState:\n> - type: int32_t\n> - description: |\n> - Control to report the current AE algorithm state. Currently identical to\n> - ANDROID_CONTROL_AE_STATE.\n> -\n> - Current state of the AE algorithm.\n> - enum:\n> - - name: AeStateInactive\n> - value: 0\n> - description: The AE algorithm is inactive.\n> - - name: AeStateSearching\n> - value: 1\n> - description: The AE algorithm has not converged yet.\n> - - name: AeStateConverged\n> - value: 2\n> - description: The AE algorithm has converged.\n> - - name: AeStateLocked\n> - value: 3\n> - description: The AE algorithm is locked.\n> - - name: AeStateFlashRequired\n> - value: 4\n> - description: The AE algorithm would need a flash for good results\n> - - name: AeStatePrecapture\n> - value: 5\n> - description: |\n> - The AE algorithm has started a pre-capture metering session.\n> - \\sa AePrecaptureTrigger\n> -\n> - AwbState:\n> type: int32_t\n> description: |","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3F77BC32F6\n\tfor ;\n\tMon, 16 Dec 2024 14:21:04 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 684CA67F6B;\n\tMon, 16 Dec 2024 15:21:03 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A51A167F55\n\tfor ;\n\tMon, 16 Dec 2024 15:21:01 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id B46B8160;\n\tMon, 16 Dec 2024 15:20:24 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"BS9Uj6a/\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1734358824;\n\tbh=ThaW6orHhp298EanmutmaNFMyyfiMTn2B/XuPDGimiU=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=BS9Uj6a//KYMcb5DwvXfUYx3SlM3TDR4n+nstN7JHTbKrHpwr77Op3rSjEIUNq8V/\n\tAk/IJlT8U2l8IVSGv5Z/WpZylhaBHqhXlQ4AnpgZ6TeECSCbwuFtFwY75rE2wcRST7\n\tO+BnBQ0OvlSS/QFreMdHP7fU6QScyBIlI8s3CGAg=","Date":"Mon, 16 Dec 2024 16:20:44 +0200","From":"Laurent Pinchart ","To":"Paul Elder ","Cc":"libcamera-devel@lists.libcamera.org, Jacopo Mondi ","Subject":"Re: [PATCH v5 1/8] controls: Introduce AEGC-related controls","Message-ID":"<20241216142044.GU32204@pendragon.ideasonboard.com>","References":"<20241216043954.3506855-1-paul.elder@ideasonboard.com>\n\t<20241216043954.3506855-2-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20241216043954.3506855-2-paul.elder@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]