[{"id":32148,"web_url":"https://patchwork.libcamera.org/comment/32148/","msgid":"","date":"2024-11-13T14:18:26","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Paul\n\nThanks for posting all this!\n\nOn Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n>\n> Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> to model the AEGC algorithm block.\n>\n> The three controls allow applications to select the exposure time and\n> analogue gain computation calculation mode (auto vs manual)\n> independently from one another, while the AeState control reports the\n> global state for the AEGC algorithm.\n>\n> The new controls are meant to replace the existing AeEnable and AeLocked\n> controls, which are momentarily kept not to break compilation of\n> platforms making use of them.\n>\n> Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> Signed-off-by: Paul Elder \n> Signed-off-by: Jacopo Mondi \n>\n> ---\n> Changes in v3:\n> - recovered from 2-year-old bitrot\n\nIndeed, I think I remember discussing all this _years_ ago!!\n\n> ---\n> src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> src/libcamera/control_ids_draft.yaml | 29 ----\n> 2 files changed, 223 insertions(+), 49 deletions(-)\n>\n> diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> index 1b1bd9507d25..98ef0736aa1b 100644\n> --- a/src/libcamera/control_ids_core.yaml\n> +++ b/src/libcamera/control_ids_core.yaml\n> @@ -26,6 +26,75 @@ controls:\n>\n> \\sa AeEnable\n>\n> + - AeState:\n> + type: int32_t\n> + description: |\n> + Control to report the AEGC algorithm state.\n> +\n> + The AEGC algorithm computes the exposure time and the analogue gain\n> + values to be applied to the image sensor.\n> +\n> + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> + AnalogueGainMode controls, which allow applications to decide how\n> + the exposure time and gain are computed, in Auto or Manual mode,\n> + independently from one another.\n> +\n> + The AeState control reports the AEGC algorithm state through a single\n> + value and describes it as a single computation block which computes\n> + both the exposure time and the analogue gain values.\n> +\n> + When both the exposure time and analogue gain values are configured to\n> + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> + compute any value and the AeState control will report AeStateIdle.\n> +\n> + When at least the exposure time or analogue gain are configured to be\n> + computed by the AEGC algorithm, the AeState control will report if the\n> + algorithm has converged to stable values for all of the controls set\n> + to be computed in Auto mode.\n> +\n> + \\sa AnalogueGainMode\n> + \\sa ExposureTimeMode\n> +\n> + enum:\n> + - name: AeStateIdle\n> + value: 0\n> + description: |\n> + The AEGC algorithm is inactive.\n> +\n> + This state is returned when both AnalogueGainMode and\n> + ExposureTimeMode are set to Manual and the algorithm is not\n> + actively computing any value.\n> +\n> + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> + AEGC algorithm might spontaneously initiate a new scan, in which\n\nMaybe \"might spontaneously start adjusting again\"? The word \"scan\"\nisn't really wrong, just sounds a bit like it was copy-pasted from an\nautofocus description! But I'm not too bothered.\n\n> + case the AeState control is moved to AeStateSearching.\n> + - name: AeStateSearching\n> + value: 1\n> + description: |\n> + The AEGC algorithm is actively computing new values, for either the\n> + exposure time or the analogue gain, but has not converged to a\n> + stable result yet.\n> +\n> + This state is returned if at least one of AnalogueGainMode\n> + or ExposureTimeMode is set to auto and the algorithm hasn't\n> + converged yet.\n> +\n> + The AEGC algorithm converges once stable values are computed for\n> + any of the controls set to be computed in Auto mode.\n> +\n> + Once the algorithm converges the state is moved to AeStateConverged.\n> + - name: AeStateConverged\n> + value: 2\n> + description: |\n> + The AEGC algorithm has converged.\n> +\n> + This state is returned if at least one of AnalogueGainMode\n> + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> + converged to stable value.\n> +\n> + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> + which case the state is moved to AeStateSearching.\n> +\n> # AeMeteringMode needs further attention:\n> # - Auto-generate max enum value.\n> # - Better handling of custom types.\n> @@ -104,6 +173,13 @@ controls:\n> The exposure modes specify how the desired total exposure is divided\n> between the shutter time and the sensor's analogue gain. They are\n> platform specific, and not all exposure modes may be supported.\n> +\n> + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> + the fixed values will override any choices made by AeExposureMode.\n> +\n> + \\sa AnalogueGainMode\n> + \\sa ExposureTimeMode\n> +\n> enum:\n> - name: ExposureNormal\n> value: 0\n> @@ -124,13 +200,15 @@ controls:\n> Specify an Exposure Value (EV) parameter.\n>\n> The EV parameter will only be applied if the AE algorithm is currently\n> - enabled.\n> + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> + are in Auto mode.\n>\n> By convention EV adjusts the exposure as log2. For example\n> EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n>\n> - \\sa AeEnable\n> + \\sa AnalogueGainMode\n> + \\sa ExposureTimeMode\n>\n> - ExposureTime:\n> type: int32_t\n> @@ -140,17 +218,95 @@ controls:\n>\n> This value is specified in micro-seconds.\n>\n> - Setting this value means that it is now fixed and the AE algorithm may\n> - not change it. Setting it back to zero returns it to the control of the\n> - AE algorithm.\n> + This control will only take effect if ExposureTimeMode is Manual. If\n> + this control is set when ExposureTimeMode is Auto, the value will be\n> + ignored and will not be retained.\n\nI'm not sure about this, it seems like a change of API behaviour which\nwill require users to be re-educated and application code to be hunted\ndown and changed. Previously you could just set the exposure time to X\nms, and it \"automatically went to manual mode\". Auto focus behaves the\nsame, I think, with respect to setting the lens position. It's just\nmore convenient.\n\n[There's also the question of how confident you are that pipeline\nhandlers correctly deal with two controls like this that are\neffectively ordered...]\n\n> +\n> + When reported in metadata, this control indicates what exposure time\n> + was used for the current request, regardless of ExposureTimeMode.\n> + ExposureTimeMode will indicate the source of the exposure time value,\n> + whether it came from the AE algorithm or not.\n> +\n> + \\sa AnalogueGain\n> + \\sa ExposureTimeMode\n> +\n> + - ExposureTimeMode:\n> + type: int32_t\n> + description: |\n> + Controls the source of the exposure time that is applied to the image\n> + sensor. When set to Auto, the AE algorithm computes the exposure time\n> + and configures the image sensor accordingly. When set to Manual,\n> + the value of the ExposureTime control is used.\n> +\n> + When transitioning from Auto to Manual mode and no ExposureTime control\n> + is provided by the application, the last value computed by the AE\n> + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> + was never set to Auto (either because the camera started in Manual mode,\n> + or Auto is not supported by the camera), the camera should use a\n> + best-effort default value.\n\nYes, I think this sounds OK. So setting it to manual mode is what\nsetting AeEnable to zero used to do, except that it can be done\nseparately for exposure time and gain now.\n\n> +\n> + If ExposureTimeModeManual is supported, the ExposureTime control must\n> + also be supported.\n> +\n> + The set of ExposureTimeMode modes that are supported by the camera must\n> + have an intersection with the supported set of AnalogueGainMode modes.\n\nThat had me scratching my head for a moment! Maybe\n\n\"At least one of the modes, auto or manual, must be supported by both\nExposureTimeMode and AnalogueGainMode.\"\n\nIs that clearer? (Also, is it what you meant?)\n\n> +\n> + Flickerless exposure mode transitions\n> +\n> + Applications that transition from ExposureTimeModeAuto to the direct\n> + control of the exposure time should aim to do so by selecting an\n> + ExposureTime value as close as possible to the last value computed by\n> + the auto exposure algorithm in order to avoid any visible flickering.\n\nSounds a bit like you _have_ to do this. Maybe\n\n\"Applications that wish to transition from ..Auto to direct control of\nthe exposure time without causing extra flicker, should do so by\nselecting an ExposureTime value as close as possible to the last value\ncomputed by the auto exposure algorithm.\"\n\nDon't know if that's better or not.\n\n> +\n> + To select the correct value to use as ExposureTime value, applications\n> + should accommodate the natural delay in applying controls caused by the\n> + capture pipeline frame depth.\n> +\n> + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> + immediately specify an ExposureTime value in the same request where\n> + ExposureTimeMode is set to Manual. They should instead wait for the\n> + first Request where ExposureTimeMode is reported as\n> + ExposureTimeModeManual in the Request metadata, and use the reported\n> + ExposureTime to populate the control value in the next Request to be\n> + queued to the Camera.\n> +\n> + The implementation of the auto-exposure algorithm should equally try to\n> + minimize flickering and when transitioning from manual exposure mode to\n> + auto exposure use the last value provided by the application as starting\n> + point.\n> +\n> + 1. Start with ExposureTimeMode set to Auto\n> +\n> + 2. Set ExposureTimeMode to Manual\n> +\n> + 3. Wait for the first completed request that has ExposureTimeMode\n> + set to Manual\n> +\n> + 4. Copy the value reported in ExposureTime into a new request, and\n> + submit it\n> +\n> + 5. Proceed to run manual exposure time\n> +\n> + \\sa ExposureTime\n> + enum:\n> + - name: ExposureTimeModeAuto\n> + value: 0\n> + description: |\n> + The exposure time will be calculated automatically and set by the\n> + AE algorithm.\n>\n> - \\sa AnalogueGain AeEnable\n> + If ExposureTime is set while this mode is active, it will be\n> + ignored, and it will also not be retained.\n> + - name: ExposureTimeModeManual\n> + value: 1\n> + description: |\n> + The exposure time will not be updated by the AE algorithm.\n>\n> - \\todo Document the interactions between AeEnable and setting a fixed\n> - value for this control. Consider interactions with other AE features,\n> - such as aperture and aperture/shutter priority mode, and decide if\n> - control of which features should be automatically adjusted shouldn't\n> - better be handled through a separate AE mode control.\n> + When transitioning from Auto to Manual mode, the last computed\n> + exposure value is used until a new value is specified through the\n> + ExposureTime control. If an ExposureTime value is specified in the\n> + same request where the ExposureTimeMode is changed from Auto to\n> + Manual, the provided ExposureTime is applied immediately.\n\nDid we say anywhere what happens when you go from manual back to auto?\nI'm assuming the adjustments restart from the manual values that you\ncurrently have.\n\n>\n> - AnalogueGain:\n> type: float\n> @@ -160,17 +316,64 @@ controls:\n> The value of the control specifies the gain multiplier applied to all\n> colour channels. This value cannot be lower than 1.0.\n>\n> - Setting this value means that it is now fixed and the AE algorithm may\n> - not change it. Setting it back to zero returns it to the control of the\n> - AE algorithm.\n> + This control will only take effect if AnalogueGainMode is Manual. If\n> + this control is set when AnalogueGainMode is Auto, the value will be\n> + ignored and will not be retained.\n\nAs above. It would be user-friendly to switch automatically to manual, I think.\n\n> +\n> + When reported in metadata, this control indicates what analogue gain\n> + was used for the current request, regardless of AnalogueGainMode.\n> + AnalogueGainMode will indicate the source of the analogue gain value,\n> + whether it came from the AEGC algorithm or not.\n> +\n> + \\sa ExposureTime\n> + \\sa AnalogueGainMode\n> +\n> + - AnalogueGainMode:\n> + type: int32_t\n> + description: |\n> + Controls the source of the analogue gain that is applied to the image\n> + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> + and configures the image sensor accordingly. When set to Manual,\n> + the value of the AnalogueGain control is used.\n> +\n> + When transitioning from Auto to Manual mode and no AnalogueGain control\n> + is provided by the application, the last value computed by the AEGC\n> + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> + was never set to Auto (either because the camera started in Manual mode,\n> + or Auto is not supported by the camera), the camera should use a\n> + best-effort default value.\n> +\n> + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> + also be supported.\n> +\n> + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> + have an intersection with the supported set of ExposureTimeMode modes.\n\nAs above!\n\nThanks again\n\nDavid\n\n> +\n> + The same procedure described for performing flickerless transitions in\n> + the ExposureTimeMode control documentation should be applied to\n> + analogue gain.\n> +\n> + \\sa ExposureTimeMode\n> + \\sa AnalogueGain\n> + enum:\n> + - name: AnalogueGainModeAuto\n> + value: 0\n> + description: |\n> + The analogue gain will be calculated automatically and set by the\n> + AEGC algorithm.\n>\n> - \\sa ExposureTime AeEnable\n> + If AnalogueGain is set while this mode is active, it will be\n> + ignored, and it will also not be retained.\n> + - name: AnalogueGainModeManual\n> + value: 1\n> + description: |\n> + The analogue gain will not be updated by the AEGC algorithm.\n>\n> - \\todo Document the interactions between AeEnable and setting a fixed\n> - value for this control. Consider interactions with other AE features,\n> - such as aperture and aperture/shutter priority mode, and decide if\n> - control of which features should be automatically adjusted shouldn't\n> - better be handled through a separate AE mode control.\n> + When transitioning from Auto to Manual mode, the last computed\n> + gain value is used until a new value is specified through the\n> + AnalogueGain control. If an AnalogueGain value is specified in the\n> + same request where the AnalogueGainMode is changed from Auto to\n> + Manual, the provided AnalogueGain is applied immediately.\n>\n> - AeFlickerMode:\n> type: int32_t\n> diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> index 1b284257f601..32c2b3feaf65 100644\n> --- a/src/libcamera/control_ids_draft.yaml\n> +++ b/src/libcamera/control_ids_draft.yaml\n> @@ -77,35 +77,6 @@ controls:\n> High quality aberration correction which might reduce the frame\n> rate.\n>\n> - - AeState:\n> - type: int32_t\n> - description: |\n> - Control to report the current AE algorithm state. Currently identical to\n> - ANDROID_CONTROL_AE_STATE.\n> -\n> - Current state of the AE algorithm.\n> - enum:\n> - - name: AeStateInactive\n> - value: 0\n> - description: The AE algorithm is inactive.\n> - - name: AeStateSearching\n> - value: 1\n> - description: The AE algorithm has not converged yet.\n> - - name: AeStateConverged\n> - value: 2\n> - description: The AE algorithm has converged.\n> - - name: AeStateLocked\n> - value: 3\n> - description: The AE algorithm is locked.\n> - - name: AeStateFlashRequired\n> - value: 4\n> - description: The AE algorithm would need a flash for good results\n> - - name: AeStatePrecapture\n> - value: 5\n> - description: |\n> - The AE algorithm has started a pre-capture metering session.\n> - \\sa AePrecaptureTrigger\n> -\n> - AwbState:\n> type: int32_t\n> description: |\n> --\n> 2.39.2\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2FF28BE173\n\tfor ;\n\tWed, 13 Nov 2024 14:18:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3F34B657CF;\n\tWed, 13 Nov 2024 15:18:41 +0100 (CET)","from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com\n\t[IPv6:2607:f8b0:4864:20::72d])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C2209657CF\n\tfor ;\n\tWed, 13 Nov 2024 15:18:38 +0100 (CET)","by mail-qk1-x72d.google.com with SMTP id\n\taf79cd13be357-7b15d330ce1so516561885a.1\n\tfor ;\n\tWed, 13 Nov 2024 06:18:38 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"k84G+tY2\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1731507517; x=1732112317;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=nGbVFKWOwiqpwYlIFe8LQNJaxbnhWBflRB5qmdQEgjc=;\n\tb=k84G+tY2rn+IwExQCi/kODyi+cNqmohksUJWnbQtC+46+HRoEZqVoDD9bwgrZcSF1O\n\tSxUMiaNrqq0zm5Y+yt59NvjDqVOA9xKTGPBoPHMpmUKzDtiv/Sf7tlKZ6j43kluagGIV\n\tzlbV1cO2afMA8ms/kjdz8mwvQOE4QX7644+qj25xgSTMF6kiw5rSsq5IEl2v4EO0y6ZA\n\t5zkFiC3dQyx+ZAxgQ639iaadfPc8DjD+gI8HNZjGxVI+OKbvvL16dUNeiYmcFM3Z1rv8\n\tQ4lb6B46yZsOKi5cfHT4iJvP7xDaQRj9hM1XXUOxl0casT9HG5+ocDToIHkO05OuQ2aM\n\tb+0A==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1731507517; x=1732112317;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=nGbVFKWOwiqpwYlIFe8LQNJaxbnhWBflRB5qmdQEgjc=;\n\tb=SQw4mB/Kc9F208Al/DVoYGDc6j9L17ux4BAsKosmaPEeuRjLy2B0lAsSmRFV4y3BKQ\n\tIdYa+SOsoC76y9U23kEA1mDWUK8YLu6Y/R/OpMO5iGcSPfTc5m99838fqzYbvdjmB1RE\n\tL0Sg7SrauIhV0Dv1miLRg+9rDUAWO4NZ0UEB6nBj/nR5PGFALx6xlGIDOCnSbor7yb2E\n\tGY2i0E0eIrw+a2/W3xcmq/fnRVIz8idtWLKdxhLV2f81WGVKqeAit4rywZ65/OyA65k0\n\tpeQ3PEQN59h2QJYugcqoruSBkwKnB4YUbGz+sRHMYwKIV4h4dXqJxw2pEXbeazUOJe/X\n\tix+w==","X-Gm-Message-State":"AOJu0YxXiINqOY80R0KJ3o3qBpHMGR1H5DOSDvcP/wDLNZX64EznI+CG\n\tri9SaBhiQEfPz/25mQ3y3Sw5OGAYtbp/YI58qNx/+xwTjOFHBx+1jzNzLHQzkFtrCHdeUe31k0u\n\ttw/yeoe1I593uFf8vuZfo4M5hsyUux2Cjl1qhtelOfJV4CiY0YLE=","X-Google-Smtp-Source":"AGHT+IGONRus11qh2kOtkbbq9Q5N+KBADXtWprV9zUATqg2aYiagfFXkA5CUanXXdEshoJZ+rGjB9FkPPTAesfxXcII=","X-Received":"by 2002:a05:620a:3704:b0:7b1:4a0b:891 with SMTP id\n\taf79cd13be357-7b3528aaademr343660185a.3.1731507517433;\n\tWed, 13 Nov 2024 06:18:37 -0800 (PST)","MIME-Version":"1.0","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>","In-Reply-To":"<20241113131256.3170817-2-paul.elder@ideasonboard.com>","From":"David Plowman ","Date":"Wed, 13 Nov 2024 14:18:26 +0000","Message-ID":"","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","To":"Paul Elder ","Cc":"libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com, \n\tjacopo.mondi@ideasonboard.com, naush@raspberrypi.com, \n\tJacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32172,"web_url":"https://patchwork.libcamera.org/comment/32172/","msgid":"","date":"2024-11-14T14:16:20","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":184,"url":"https://patchwork.libcamera.org/api/people/184/","name":"Stefan Klug","email":"stefan.klug@ideasonboard.com"},"content":"On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\nHi Paul,\n\nThank you for the patch. \n\n> Hi Paul\n> \n> Thanks for posting all this!\n> \n> On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> >\n> > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > to model the AEGC algorithm block.\n> >\n> > The three controls allow applications to select the exposure time and\n> > analogue gain computation calculation mode (auto vs manual)\n> > independently from one another, while the AeState control reports the\n> > global state for the AEGC algorithm.\n> >\n> > The new controls are meant to replace the existing AeEnable and AeLocked\n> > controls, which are momentarily kept not to break compilation of\n> > platforms making use of them.\n> >\n> > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > Signed-off-by: Paul Elder \n> > Signed-off-by: Jacopo Mondi \n> >\n> > ---\n> > Changes in v3:\n> > - recovered from 2-year-old bitrot\n> \n> Indeed, I think I remember discussing all this _years_ ago!!\n> \n> > ---\n> > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > src/libcamera/control_ids_draft.yaml | 29 ----\n> > 2 files changed, 223 insertions(+), 49 deletions(-)\n> >\n> > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > index 1b1bd9507d25..98ef0736aa1b 100644\n> > --- a/src/libcamera/control_ids_core.yaml\n> > +++ b/src/libcamera/control_ids_core.yaml\n> > @@ -26,6 +26,75 @@ controls:\n> >\n> > \\sa AeEnable\n> >\n> > + - AeState:\n> > + type: int32_t\n> > + description: |\n> > + Control to report the AEGC algorithm state.\n> > +\n> > + The AEGC algorithm computes the exposure time and the analogue gain\n> > + values to be applied to the image sensor.\n> > +\n> > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > + AnalogueGainMode controls, which allow applications to decide how\n> > + the exposure time and gain are computed, in Auto or Manual mode,\n> > + independently from one another.\n> > +\n> > + The AeState control reports the AEGC algorithm state through a single\n> > + value and describes it as a single computation block which computes\n> > + both the exposure time and the analogue gain values.\n> > +\n> > + When both the exposure time and analogue gain values are configured to\n> > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > + compute any value and the AeState control will report AeStateIdle.\n> > +\n> > + When at least the exposure time or analogue gain are configured to be\n> > + computed by the AEGC algorithm, the AeState control will report if the\n> > + algorithm has converged to stable values for all of the controls set\n> > + to be computed in Auto mode.\n> > +\n> > + \\sa AnalogueGainMode\n> > + \\sa ExposureTimeMode\n> > +\n> > + enum:\n> > + - name: AeStateIdle\n> > + value: 0\n> > + description: |\n> > + The AEGC algorithm is inactive.\n> > +\n> > + This state is returned when both AnalogueGainMode and\n> > + ExposureTimeMode are set to Manual and the algorithm is not\n> > + actively computing any value.\n> > +\n> > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > + AEGC algorithm might spontaneously initiate a new scan, in which\n> \n> Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> isn't really wrong, just sounds a bit like it was copy-pasted from an\n> autofocus description! But I'm not too bothered.\n> \n> > + case the AeState control is moved to AeStateSearching.\n> > + - name: AeStateSearching\n\nA bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\nwell be because I'm not a native speaker). How does \"regulating\" or\n\"controlling\" sound to native ears? In my mind a search doesn't really\nconverge, but a regulation does.\n\n> > + value: 1\n> > + description: |\n> > + The AEGC algorithm is actively computing new values, for either the\n> > + exposure time or the analogue gain, but has not converged to a\n> > + stable result yet.\n> > +\n> > + This state is returned if at least one of AnalogueGainMode\n> > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > + converged yet.\n> > +\n> > + The AEGC algorithm converges once stable values are computed for\n> > + any of the controls set to be computed in Auto mode.\n\nShould that be \"all of the\"? \n\n> > +\n> > + Once the algorithm converges the state is moved to AeStateConverged.\n> > + - name: AeStateConverged\n> > + value: 2\n> > + description: |\n> > + The AEGC algorithm has converged.\n> > +\n> > + This state is returned if at least one of AnalogueGainMode\n> > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > + converged to stable value.\n> > +\n> > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > + which case the state is moved to AeStateSearching.\n\nThis sounds like it is based on a random event. What about something\nlike: When the statistics/measurements move too far away from the\nconvergence point the AEGC algo automatically restarts\nregulation/scanning... \n\n> > +\n> > # AeMeteringMode needs further attention:\n> > # - Auto-generate max enum value.\n> > # - Better handling of custom types.\n> > @@ -104,6 +173,13 @@ controls:\n> > The exposure modes specify how the desired total exposure is divided\n> > between the shutter time and the sensor's analogue gain. They are\n> > platform specific, and not all exposure modes may be supported.\n> > +\n> > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > + the fixed values will override any choices made by AeExposureMode.\n> > +\n> > + \\sa AnalogueGainMode\n> > + \\sa ExposureTimeMode\n> > +\n> > enum:\n> > - name: ExposureNormal\n> > value: 0\n> > @@ -124,13 +200,15 @@ controls:\n> > Specify an Exposure Value (EV) parameter.\n> >\n> > The EV parameter will only be applied if the AE algorithm is currently\n> > - enabled.\n> > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > + are in Auto mode.\n> >\n> > By convention EV adjusts the exposure as log2. For example\n> > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> >\n> > - \\sa AeEnable\n> > + \\sa AnalogueGainMode\n> > + \\sa ExposureTimeMode\n> >\n> > - ExposureTime:\n> > type: int32_t\n> > @@ -140,17 +218,95 @@ controls:\n> >\n> > This value is specified in micro-seconds.\n> >\n> > - Setting this value means that it is now fixed and the AE algorithm may\n> > - not change it. Setting it back to zero returns it to the control of the\n> > - AE algorithm.\n> > + This control will only take effect if ExposureTimeMode is Manual. If\n> > + this control is set when ExposureTimeMode is Auto, the value will be\n> > + ignored and will not be retained.\n> \n> I'm not sure about this, it seems like a change of API behaviour which\n> will require users to be re-educated and application code to be hunted\n> down and changed. Previously you could just set the exposure time to X\n> ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> same, I think, with respect to setting the lens position. It's just\n> more convenient.\n\nThis is an interesting discussion. I wasn't aware of the old\nspecification. I see that just setting one value and deducing what the\nuser wants is convenient. On the other hand one control changing the\nvalue of another control feels quite difficult to explain and in my\nopinion should be prevented whenever possible. I believe it is easier to\neducate the user and have an explicit interface than to have hidden\nmagic whenever possible.\n\n> \n> [There's also the question of how confident you are that pipeline\n> handlers correctly deal with two controls like this that are\n> effectively ordered...]\n> \n> > +\n> > + When reported in metadata, this control indicates what exposure time\n> > + was used for the current request, regardless of ExposureTimeMode.\n> > + ExposureTimeMode will indicate the source of the exposure time value,\n> > + whether it came from the AE algorithm or not.\n> > +\n> > + \\sa AnalogueGain\n> > + \\sa ExposureTimeMode\n> > +\n> > + - ExposureTimeMode:\n> > + type: int32_t\n> > + description: |\n> > + Controls the source of the exposure time that is applied to the image\n> > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > + and configures the image sensor accordingly. When set to Manual,\n> > + the value of the ExposureTime control is used.\n> > +\n> > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > + is provided by the application, the last value computed by the AE\n> > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > + was never set to Auto (either because the camera started in Manual mode,\n> > + or Auto is not supported by the camera), the camera should use a\n> > + best-effort default value.\n> \n> Yes, I think this sounds OK. So setting it to manual mode is what\n> setting AeEnable to zero used to do, except that it can be done\n> separately for exposure time and gain now.\n> \n> > +\n> > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > + also be supported.\n> > +\n> > + The set of ExposureTimeMode modes that are supported by the camera must\n> > + have an intersection with the supported set of AnalogueGainMode modes.\n> \n> That had me scratching my head for a moment! Maybe\n> \n> \"At least one of the modes, auto or manual, must be supported by both\n> ExposureTimeMode and AnalogueGainMode.\"\n> \n> Is that clearer? (Also, is it what you meant?)\n> \n> > +\n> > + Flickerless exposure mode transitions\n> > +\n> > + Applications that transition from ExposureTimeModeAuto to the direct\n> > + control of the exposure time should aim to do so by selecting an\n> > + ExposureTime value as close as possible to the last value computed by\n> > + the auto exposure algorithm in order to avoid any visible flickering.\n> \n> Sounds a bit like you _have_ to do this. Maybe\n> \n> \"Applications that wish to transition from ..Auto to direct control of\n> the exposure time without causing extra flicker, should do so by\n> selecting an ExposureTime value as close as possible to the last value\n> computed by the auto exposure algorithm.\"\n> \n> Don't know if that's better or not.\n> \n> > +\n> > + To select the correct value to use as ExposureTime value, applications\n> > + should accommodate the natural delay in applying controls caused by the\n> > + capture pipeline frame depth.\n> > +\n> > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > + immediately specify an ExposureTime value in the same request where\n> > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > + first Request where ExposureTimeMode is reported as\n> > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > + ExposureTime to populate the control value in the next Request to be\n> > + queued to the Camera.\n> > +\n> > + The implementation of the auto-exposure algorithm should equally try to\n> > + minimize flickering and when transitioning from manual exposure mode to\n> > + auto exposure use the last value provided by the application as starting\n> > + point.\n> > +\n> > + 1. Start with ExposureTimeMode set to Auto\n> > +\n> > + 2. Set ExposureTimeMode to Manual\n> > +\n> > + 3. Wait for the first completed request that has ExposureTimeMode\n> > + set to Manual\n> > +\n> > + 4. Copy the value reported in ExposureTime into a new request, and\n> > + submit it\n> > +\n> > + 5. Proceed to run manual exposure time\n> > +\n> > + \\sa ExposureTime\n> > + enum:\n> > + - name: ExposureTimeModeAuto\n> > + value: 0\n> > + description: |\n> > + The exposure time will be calculated automatically and set by the\n> > + AE algorithm.\n> >\n> > - \\sa AnalogueGain AeEnable\n> > + If ExposureTime is set while this mode is active, it will be\n> > + ignored, and it will also not be retained.\n> > + - name: ExposureTimeModeManual\n> > + value: 1\n> > + description: |\n> > + The exposure time will not be updated by the AE algorithm.\n> >\n> > - \\todo Document the interactions between AeEnable and setting a fixed\n> > - value for this control. Consider interactions with other AE features,\n> > - such as aperture and aperture/shutter priority mode, and decide if\n> > - control of which features should be automatically adjusted shouldn't\n> > - better be handled through a separate AE mode control.\n> > + When transitioning from Auto to Manual mode, the last computed\n> > + exposure value is used until a new value is specified through the\n> > + ExposureTime control. If an ExposureTime value is specified in the\n> > + same request where the ExposureTimeMode is changed from Auto to\n> > + Manual, the provided ExposureTime is applied immediately.\n> \n> Did we say anywhere what happens when you go from manual back to auto?\n> I'm assuming the adjustments restart from the manual values that you\n> currently have.\n> \n> >\n> > - AnalogueGain:\n> > type: float\n> > @@ -160,17 +316,64 @@ controls:\n> > The value of the control specifies the gain multiplier applied to all\n> > colour channels. This value cannot be lower than 1.0.\n> >\n> > - Setting this value means that it is now fixed and the AE algorithm may\n> > - not change it. Setting it back to zero returns it to the control of the\n> > - AE algorithm.\n> > + This control will only take effect if AnalogueGainMode is Manual. If\n> > + this control is set when AnalogueGainMode is Auto, the value will be\n> > + ignored and will not be retained.\n> \n> As above. It would be user-friendly to switch automatically to manual, I think.\n> \n> > +\n> > + When reported in metadata, this control indicates what analogue gain\n> > + was used for the current request, regardless of AnalogueGainMode.\n> > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > + whether it came from the AEGC algorithm or not.\n> > +\n> > + \\sa ExposureTime\n> > + \\sa AnalogueGainMode\n> > +\n> > + - AnalogueGainMode:\n> > + type: int32_t\n> > + description: |\n> > + Controls the source of the analogue gain that is applied to the image\n> > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > + and configures the image sensor accordingly. When set to Manual,\n> > + the value of the AnalogueGain control is used.\n> > +\n> > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > + is provided by the application, the last value computed by the AEGC\n> > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > + was never set to Auto (either because the camera started in Manual mode,\n> > + or Auto is not supported by the camera), the camera should use a\n> > + best-effort default value.\n> > +\n> > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > + also be supported.\n> > +\n> > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > + have an intersection with the supported set of ExposureTimeMode modes.\n> \n> As above!\n> \n> Thanks again\n> \n> David\n> \n> > +\n> > + The same procedure described for performing flickerless transitions in\n> > + the ExposureTimeMode control documentation should be applied to\n> > + analogue gain.\n> > +\n> > + \\sa ExposureTimeMode\n> > + \\sa AnalogueGain\n> > + enum:\n> > + - name: AnalogueGainModeAuto\n> > + value: 0\n> > + description: |\n> > + The analogue gain will be calculated automatically and set by the\n> > + AEGC algorithm.\n> >\n> > - \\sa ExposureTime AeEnable\n> > + If AnalogueGain is set while this mode is active, it will be\n> > + ignored, and it will also not be retained.\n> > + - name: AnalogueGainModeManual\n> > + value: 1\n> > + description: |\n> > + The analogue gain will not be updated by the AEGC algorithm.\n> >\n> > - \\todo Document the interactions between AeEnable and setting a fixed\n> > - value for this control. Consider interactions with other AE features,\n> > - such as aperture and aperture/shutter priority mode, and decide if\n> > - control of which features should be automatically adjusted shouldn't\n> > - better be handled through a separate AE mode control.\n> > + When transitioning from Auto to Manual mode, the last computed\n> > + gain value is used until a new value is specified through the\n> > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > + same request where the AnalogueGainMode is changed from Auto to\n> > + Manual, the provided AnalogueGain is applied immediately.\n> >\n> > - AeFlickerMode:\n> > type: int32_t\n> > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > index 1b284257f601..32c2b3feaf65 100644\n> > --- a/src/libcamera/control_ids_draft.yaml\n> > +++ b/src/libcamera/control_ids_draft.yaml\n> > @@ -77,35 +77,6 @@ controls:\n> > High quality aberration correction which might reduce the frame\n> > rate.\n> >\n> > - - AeState:\n> > - type: int32_t\n> > - description: |\n> > - Control to report the current AE algorithm state. Currently identical to\n> > - ANDROID_CONTROL_AE_STATE.\n> > -\n> > - Current state of the AE algorithm.\n> > - enum:\n> > - - name: AeStateInactive\n> > - value: 0\n> > - description: The AE algorithm is inactive.\n> > - - name: AeStateSearching\n> > - value: 1\n> > - description: The AE algorithm has not converged yet.\n> > - - name: AeStateConverged\n> > - value: 2\n> > - description: The AE algorithm has converged.\n> > - - name: AeStateLocked\n> > - value: 3\n> > - description: The AE algorithm is locked.\n> > - - name: AeStateFlashRequired\n> > - value: 4\n> > - description: The AE algorithm would need a flash for good results\n> > - - name: AeStatePrecapture\n> > - value: 5\n> > - description: |\n> > - The AE algorithm has started a pre-capture metering session.\n> > - \\sa AePrecaptureTrigger\n> > -\n> > - AwbState:\n> > type: int32_t\n> > description: |\n> > --\n> > 2.39.2\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id CB4A9C0F1B\n\tfor ;\n\tThu, 14 Nov 2024 14:16:26 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0503365844;\n\tThu, 14 Nov 2024 15:16:26 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4499B657E0\n\tfor ;\n\tThu, 14 Nov 2024 15:16:24 +0100 (CET)","from ideasonboard.com (unknown\n\t[IPv6:2a00:6020:448c:6c00:cd64:b5ef:1d95:ef1c])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id E1F21827;\n\tThu, 14 Nov 2024 15:16:09 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"D3E2qCDC\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1731593770;\n\tbh=4jIAi5RXCvinksHvq2rviqQRfZo9te5EonxElh/QRDg=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=D3E2qCDCzw6S6CEKMC0fcdXyELwL0AyS8+LJy5LnfUpXXgVOvSGBqbPBtq5EqDoA7\n\tXWkID2GIqUrwFASFe/JMhF5BNGL8V0Sr+BaSSpsudbpssWiLB1NuJxss3VyMT+kAFX\n\tVemLzvkitcXRzUct5hrE1CJT3i8jvkokhNHj2Eok=","Date":"Thu, 14 Nov 2024 15:16:20 +0100","From":"Stefan Klug ","To":"David Plowman ","Cc":"Paul Elder , \n\tlibcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com,\n\tjacopo.mondi@ideasonboard.com, \n\tnaush@raspberrypi.com, Jacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32187,"web_url":"https://patchwork.libcamera.org/comment/32187/","msgid":"","date":"2024-11-15T13:01:37","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"Hi Naush,\n\nThanks for the input!\n\nOn Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> Hi Paul\n> \n> Thanks for posting all this!\n> \n> On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> >\n> > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > to model the AEGC algorithm block.\n> >\n> > The three controls allow applications to select the exposure time and\n> > analogue gain computation calculation mode (auto vs manual)\n> > independently from one another, while the AeState control reports the\n> > global state for the AEGC algorithm.\n> >\n> > The new controls are meant to replace the existing AeEnable and AeLocked\n> > controls, which are momentarily kept not to break compilation of\n> > platforms making use of them.\n> >\n> > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > Signed-off-by: Paul Elder \n> > Signed-off-by: Jacopo Mondi \n> >\n> > ---\n> > Changes in v3:\n> > - recovered from 2-year-old bitrot\n> \n> Indeed, I think I remember discussing all this _years_ ago!!\n> \n> > ---\n> > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > src/libcamera/control_ids_draft.yaml | 29 ----\n> > 2 files changed, 223 insertions(+), 49 deletions(-)\n> >\n> > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > index 1b1bd9507d25..98ef0736aa1b 100644\n> > --- a/src/libcamera/control_ids_core.yaml\n> > +++ b/src/libcamera/control_ids_core.yaml\n> > @@ -26,6 +26,75 @@ controls:\n> >\n> > \\sa AeEnable\n> >\n> > + - AeState:\n> > + type: int32_t\n> > + description: |\n> > + Control to report the AEGC algorithm state.\n> > +\n> > + The AEGC algorithm computes the exposure time and the analogue gain\n> > + values to be applied to the image sensor.\n> > +\n> > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > + AnalogueGainMode controls, which allow applications to decide how\n> > + the exposure time and gain are computed, in Auto or Manual mode,\n> > + independently from one another.\n> > +\n> > + The AeState control reports the AEGC algorithm state through a single\n> > + value and describes it as a single computation block which computes\n> > + both the exposure time and the analogue gain values.\n> > +\n> > + When both the exposure time and analogue gain values are configured to\n> > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > + compute any value and the AeState control will report AeStateIdle.\n> > +\n> > + When at least the exposure time or analogue gain are configured to be\n> > + computed by the AEGC algorithm, the AeState control will report if the\n> > + algorithm has converged to stable values for all of the controls set\n> > + to be computed in Auto mode.\n> > +\n> > + \\sa AnalogueGainMode\n> > + \\sa ExposureTimeMode\n> > +\n> > + enum:\n> > + - name: AeStateIdle\n> > + value: 0\n> > + description: |\n> > + The AEGC algorithm is inactive.\n> > +\n> > + This state is returned when both AnalogueGainMode and\n> > + ExposureTimeMode are set to Manual and the algorithm is not\n> > + actively computing any value.\n> > +\n> > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > + AEGC algorithm might spontaneously initiate a new scan, in which\n> \n> Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> isn't really wrong, just sounds a bit like it was copy-pasted from an\n> autofocus description! But I'm not too bothered.\n\nYeah that probably reads better. (it's been 2 years I don't remember\nwhere the wording came from)\n\n> \n> > + case the AeState control is moved to AeStateSearching.\n> > + - name: AeStateSearching\n> > + value: 1\n> > + description: |\n> > + The AEGC algorithm is actively computing new values, for either the\n> > + exposure time or the analogue gain, but has not converged to a\n> > + stable result yet.\n> > +\n> > + This state is returned if at least one of AnalogueGainMode\n> > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > + converged yet.\n> > +\n> > + The AEGC algorithm converges once stable values are computed for\n> > + any of the controls set to be computed in Auto mode.\n> > +\n> > + Once the algorithm converges the state is moved to AeStateConverged.\n> > + - name: AeStateConverged\n> > + value: 2\n> > + description: |\n> > + The AEGC algorithm has converged.\n> > +\n> > + This state is returned if at least one of AnalogueGainMode\n> > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > + converged to stable value.\n> > +\n> > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > + which case the state is moved to AeStateSearching.\n> > +\n> > # AeMeteringMode needs further attention:\n> > # - Auto-generate max enum value.\n> > # - Better handling of custom types.\n> > @@ -104,6 +173,13 @@ controls:\n> > The exposure modes specify how the desired total exposure is divided\n> > between the shutter time and the sensor's analogue gain. They are\n> > platform specific, and not all exposure modes may be supported.\n> > +\n> > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > + the fixed values will override any choices made by AeExposureMode.\n> > +\n> > + \\sa AnalogueGainMode\n> > + \\sa ExposureTimeMode\n> > +\n> > enum:\n> > - name: ExposureNormal\n> > value: 0\n> > @@ -124,13 +200,15 @@ controls:\n> > Specify an Exposure Value (EV) parameter.\n> >\n> > The EV parameter will only be applied if the AE algorithm is currently\n> > - enabled.\n> > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > + are in Auto mode.\n> >\n> > By convention EV adjusts the exposure as log2. For example\n> > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> >\n> > - \\sa AeEnable\n> > + \\sa AnalogueGainMode\n> > + \\sa ExposureTimeMode\n> >\n> > - ExposureTime:\n> > type: int32_t\n> > @@ -140,17 +218,95 @@ controls:\n> >\n> > This value is specified in micro-seconds.\n> >\n> > - Setting this value means that it is now fixed and the AE algorithm may\n> > - not change it. Setting it back to zero returns it to the control of the\n> > - AE algorithm.\n> > + This control will only take effect if ExposureTimeMode is Manual. If\n> > + this control is set when ExposureTimeMode is Auto, the value will be\n> > + ignored and will not be retained.\n> \n> I'm not sure about this, it seems like a change of API behaviour which\n> will require users to be re-educated and application code to be hunted\n> down and changed. Previously you could just set the exposure time to X\n> ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> same, I think, with respect to setting the lens position. It's just\n> more convenient.\n\nIt does indeed change the API, but I was always of the camp that setting\n0 to enable/disable auto mode isn't a very good API :/\n\n> \n> [There's also the question of how confident you are that pipeline\n> handlers correctly deal with two controls like this that are\n> effectively ordered...]\n\nWell at the moment they don't :)\nI was thinking about having these enforced by lc-compliance.\n\n> \n> > +\n> > + When reported in metadata, this control indicates what exposure time\n> > + was used for the current request, regardless of ExposureTimeMode.\n> > + ExposureTimeMode will indicate the source of the exposure time value,\n> > + whether it came from the AE algorithm or not.\n> > +\n> > + \\sa AnalogueGain\n> > + \\sa ExposureTimeMode\n> > +\n> > + - ExposureTimeMode:\n> > + type: int32_t\n> > + description: |\n> > + Controls the source of the exposure time that is applied to the image\n> > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > + and configures the image sensor accordingly. When set to Manual,\n> > + the value of the ExposureTime control is used.\n> > +\n> > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > + is provided by the application, the last value computed by the AE\n> > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > + was never set to Auto (either because the camera started in Manual mode,\n> > + or Auto is not supported by the camera), the camera should use a\n> > + best-effort default value.\n> \n> Yes, I think this sounds OK. So setting it to manual mode is what\n> setting AeEnable to zero used to do, except that it can be done\n> separately for exposure time and gain now.\n\n\\o/\n\n> \n> > +\n> > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > + also be supported.\n> > +\n> > + The set of ExposureTimeMode modes that are supported by the camera must\n> > + have an intersection with the supported set of AnalogueGainMode modes.\n> \n> That had me scratching my head for a moment! Maybe\n> \n> \"At least one of the modes, auto or manual, must be supported by both\n> ExposureTimeMode and AnalogueGainMode.\"\n> \n> Is that clearer? (Also, is it what you meant?)\n\nNo, that's not what I meant.\n\nI suppose an explicit listing of all valid combinations would be clear\nto at least illustrate what I meant.\n\nValid combinations of supported modes:\n\n- exposure: { }, gain: { }\n- exposure: { auto }, gain: { auto }\n- exposure: { manual }, gain: { manual }\n- exposure: { auto, manual }, gain: { auto, manual }\n\nInvalid combinations of supported modes:\n\n- exposure: { auto }, gain: { manual }\n- exposure: { manual }, gain: { auto }\n- exposure: { }, gain: { auto }\n- exposure: { }, gain: { manual }\n- exposure: { auto }, gain: { }\n- exposure: { manual }, gain: { }\n\nOk now that I've written them out I'm not certain about the last four\ncases, but anyway this is what I had in mind.\n\n\"The intersection of the sets of supported modes must not be empty\" :)\n\n(ok technically the first valid combination is empty but)\n\n> \n> > +\n> > + Flickerless exposure mode transitions\n> > +\n> > + Applications that transition from ExposureTimeModeAuto to the direct\n> > + control of the exposure time should aim to do so by selecting an\n> > + ExposureTime value as close as possible to the last value computed by\n> > + the auto exposure algorithm in order to avoid any visible flickering.\n> \n> Sounds a bit like you _have_ to do this. Maybe\n\nYou're right, it kind of does.\n\n> \n> \"Applications that wish to transition from ..Auto to direct control of\n> the exposure time without causing extra flicker, should do so by\n> selecting an ExposureTime value as close as possible to the last value\n> computed by the auto exposure algorithm.\"\n> \n> Don't know if that's better or not.\n\nI think it's better, thanks.\n\n> \n> > +\n> > + To select the correct value to use as ExposureTime value, applications\n> > + should accommodate the natural delay in applying controls caused by the\n> > + capture pipeline frame depth.\n> > +\n> > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > + immediately specify an ExposureTime value in the same request where\n> > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > + first Request where ExposureTimeMode is reported as\n> > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > + ExposureTime to populate the control value in the next Request to be\n> > + queued to the Camera.\n> > +\n> > + The implementation of the auto-exposure algorithm should equally try to\n> > + minimize flickering and when transitioning from manual exposure mode to\n> > + auto exposure use the last value provided by the application as starting\n> > + point.\n> > +\n> > + 1. Start with ExposureTimeMode set to Auto\n> > +\n> > + 2. Set ExposureTimeMode to Manual\n> > +\n> > + 3. Wait for the first completed request that has ExposureTimeMode\n> > + set to Manual\n> > +\n> > + 4. Copy the value reported in ExposureTime into a new request, and\n> > + submit it\n> > +\n> > + 5. Proceed to run manual exposure time\n> > +\n> > + \\sa ExposureTime\n> > + enum:\n> > + - name: ExposureTimeModeAuto\n> > + value: 0\n> > + description: |\n> > + The exposure time will be calculated automatically and set by the\n> > + AE algorithm.\n> >\n> > - \\sa AnalogueGain AeEnable\n> > + If ExposureTime is set while this mode is active, it will be\n> > + ignored, and it will also not be retained.\n> > + - name: ExposureTimeModeManual\n> > + value: 1\n> > + description: |\n> > + The exposure time will not be updated by the AE algorithm.\n> >\n> > - \\todo Document the interactions between AeEnable and setting a fixed\n> > - value for this control. Consider interactions with other AE features,\n> > - such as aperture and aperture/shutter priority mode, and decide if\n> > - control of which features should be automatically adjusted shouldn't\n> > - better be handled through a separate AE mode control.\n> > + When transitioning from Auto to Manual mode, the last computed\n> > + exposure value is used until a new value is specified through the\n> > + ExposureTime control. If an ExposureTime value is specified in the\n> > + same request where the ExposureTimeMode is changed from Auto to\n> > + Manual, the provided ExposureTime is applied immediately.\n> \n> Did we say anywhere what happens when you go from manual back to auto?\n> I'm assuming the adjustments restart from the manual values that you\n> currently have.\n\nI don't remember saying anywhere what happens. It should restart from\nthe manual values yeah. This needs to be enforced by lc-compliance too.\n\n> \n> >\n> > - AnalogueGain:\n> > type: float\n> > @@ -160,17 +316,64 @@ controls:\n> > The value of the control specifies the gain multiplier applied to all\n> > colour channels. This value cannot be lower than 1.0.\n> >\n> > - Setting this value means that it is now fixed and the AE algorithm may\n> > - not change it. Setting it back to zero returns it to the control of the\n> > - AE algorithm.\n> > + This control will only take effect if AnalogueGainMode is Manual. If\n> > + this control is set when AnalogueGainMode is Auto, the value will be\n> > + ignored and will not be retained.\n> \n> As above. It would be user-friendly to switch automatically to manual, I think.\n\nIf you do that then the procedure that I described above for flickerless\nswitching from auto to manual won't work and we'd need another\nmechanism/design to implement flickerless switching.\n\n\nThanks,\n\nPaul\n\n> \n> > +\n> > + When reported in metadata, this control indicates what analogue gain\n> > + was used for the current request, regardless of AnalogueGainMode.\n> > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > + whether it came from the AEGC algorithm or not.\n> > +\n> > + \\sa ExposureTime\n> > + \\sa AnalogueGainMode\n> > +\n> > + - AnalogueGainMode:\n> > + type: int32_t\n> > + description: |\n> > + Controls the source of the analogue gain that is applied to the image\n> > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > + and configures the image sensor accordingly. When set to Manual,\n> > + the value of the AnalogueGain control is used.\n> > +\n> > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > + is provided by the application, the last value computed by the AEGC\n> > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > + was never set to Auto (either because the camera started in Manual mode,\n> > + or Auto is not supported by the camera), the camera should use a\n> > + best-effort default value.\n> > +\n> > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > + also be supported.\n> > +\n> > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > + have an intersection with the supported set of ExposureTimeMode modes.\n> \n> As above!\n> \n> Thanks again\n> \n> David\n> \n> > +\n> > + The same procedure described for performing flickerless transitions in\n> > + the ExposureTimeMode control documentation should be applied to\n> > + analogue gain.\n> > +\n> > + \\sa ExposureTimeMode\n> > + \\sa AnalogueGain\n> > + enum:\n> > + - name: AnalogueGainModeAuto\n> > + value: 0\n> > + description: |\n> > + The analogue gain will be calculated automatically and set by the\n> > + AEGC algorithm.\n> >\n> > - \\sa ExposureTime AeEnable\n> > + If AnalogueGain is set while this mode is active, it will be\n> > + ignored, and it will also not be retained.\n> > + - name: AnalogueGainModeManual\n> > + value: 1\n> > + description: |\n> > + The analogue gain will not be updated by the AEGC algorithm.\n> >\n> > - \\todo Document the interactions between AeEnable and setting a fixed\n> > - value for this control. Consider interactions with other AE features,\n> > - such as aperture and aperture/shutter priority mode, and decide if\n> > - control of which features should be automatically adjusted shouldn't\n> > - better be handled through a separate AE mode control.\n> > + When transitioning from Auto to Manual mode, the last computed\n> > + gain value is used until a new value is specified through the\n> > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > + same request where the AnalogueGainMode is changed from Auto to\n> > + Manual, the provided AnalogueGain is applied immediately.\n> >\n> > - AeFlickerMode:\n> > type: int32_t\n> > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > index 1b284257f601..32c2b3feaf65 100644\n> > --- a/src/libcamera/control_ids_draft.yaml\n> > +++ b/src/libcamera/control_ids_draft.yaml\n> > @@ -77,35 +77,6 @@ controls:\n> > High quality aberration correction which might reduce the frame\n> > rate.\n> >\n> > - - AeState:\n> > - type: int32_t\n> > - description: |\n> > - Control to report the current AE algorithm state. Currently identical to\n> > - ANDROID_CONTROL_AE_STATE.\n> > -\n> > - Current state of the AE algorithm.\n> > - enum:\n> > - - name: AeStateInactive\n> > - value: 0\n> > - description: The AE algorithm is inactive.\n> > - - name: AeStateSearching\n> > - value: 1\n> > - description: The AE algorithm has not converged yet.\n> > - - name: AeStateConverged\n> > - value: 2\n> > - description: The AE algorithm has converged.\n> > - - name: AeStateLocked\n> > - value: 3\n> > - description: The AE algorithm is locked.\n> > - - name: AeStateFlashRequired\n> > - value: 4\n> > - description: The AE algorithm would need a flash for good results\n> > - - name: AeStatePrecapture\n> > - value: 5\n> > - description: |\n> > - The AE algorithm has started a pre-capture metering session.\n> > - \\sa AePrecaptureTrigger\n> > -\n> > - AwbState:\n> > type: int32_t\n> > description: |\n> > --\n> > 2.39.2\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 95535C0F1B\n\tfor ;\n\tFri, 15 Nov 2024 13:01:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9B4C56588B;\n\tFri, 15 Nov 2024 14:01:49 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2B9C86580A\n\tfor ;\n\tFri, 15 Nov 2024 14:01:47 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2404:7a81:160:2100:bd6c:1638:cb26:1bfa])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id F3EA7291;\n\tFri, 15 Nov 2024 14:01:29 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"fbrQU9Ha\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1731675692;\n\tbh=i1w60w5j2bOWsWwpMTaBOtm03eZaewVGoawm0s4TTCc=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=fbrQU9Hao8ojG9KqWw3EKnOVuVZ49HiU5EL8nqn23cFNRwU6DIS7fKQHOSEa+vEsW\n\t6YexfOjdfwm9YRBH2QBQwwlke4Gg7uG+3uAsMsIxjjY8i0uZVL8XvWeXCi9Ix1c1dN\n\tdrEEf0w8tcQ2pwTigH1IYz5kY3w0Rvef0wzHQqQA=","Date":"Fri, 15 Nov 2024 22:01:37 +0900","From":"Paul Elder ","To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com, \n\tjacopo.mondi@ideasonboard.com, naush@raspberrypi.com,\n\tJacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32192,"web_url":"https://patchwork.libcamera.org/comment/32192/","msgid":"","date":"2024-11-15T13:41:47","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> Hi Paul,\n> \n> Thank you for the patch. \n> \n> > Hi Paul\n> > \n> > Thanks for posting all this!\n> > \n> > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > >\n> > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > to model the AEGC algorithm block.\n> > >\n> > > The three controls allow applications to select the exposure time and\n> > > analogue gain computation calculation mode (auto vs manual)\n> > > independently from one another, while the AeState control reports the\n> > > global state for the AEGC algorithm.\n> > >\n> > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > controls, which are momentarily kept not to break compilation of\n> > > platforms making use of them.\n> > >\n> > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > Signed-off-by: Paul Elder \n> > > Signed-off-by: Jacopo Mondi \n> > >\n> > > ---\n> > > Changes in v3:\n> > > - recovered from 2-year-old bitrot\n> > \n> > Indeed, I think I remember discussing all this _years_ ago!!\n> > \n> > > ---\n> > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > >\n> > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > --- a/src/libcamera/control_ids_core.yaml\n> > > +++ b/src/libcamera/control_ids_core.yaml\n> > > @@ -26,6 +26,75 @@ controls:\n> > >\n> > > \\sa AeEnable\n> > >\n> > > + - AeState:\n> > > + type: int32_t\n> > > + description: |\n> > > + Control to report the AEGC algorithm state.\n> > > +\n> > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > + values to be applied to the image sensor.\n> > > +\n> > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > + AnalogueGainMode controls, which allow applications to decide how\n> > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > + independently from one another.\n> > > +\n> > > + The AeState control reports the AEGC algorithm state through a single\n> > > + value and describes it as a single computation block which computes\n> > > + both the exposure time and the analogue gain values.\n> > > +\n> > > + When both the exposure time and analogue gain values are configured to\n> > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > + compute any value and the AeState control will report AeStateIdle.\n> > > +\n> > > + When at least the exposure time or analogue gain are configured to be\n> > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > + algorithm has converged to stable values for all of the controls set\n> > > + to be computed in Auto mode.\n> > > +\n> > > + \\sa AnalogueGainMode\n> > > + \\sa ExposureTimeMode\n> > > +\n> > > + enum:\n> > > + - name: AeStateIdle\n> > > + value: 0\n> > > + description: |\n> > > + The AEGC algorithm is inactive.\n> > > +\n> > > + This state is returned when both AnalogueGainMode and\n> > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > + actively computing any value.\n> > > +\n> > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > \n> > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > autofocus description! But I'm not too bothered.\n> > \n> > > + case the AeState control is moved to AeStateSearching.\n> > > + - name: AeStateSearching\n> \n> A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> well be because I'm not a native speaker). How does \"regulating\" or\n> \"controlling\" sound to native ears? In my mind a search doesn't really\n> converge, but a regulation does.\n\nHm to be \"regulating\" and \"controlling\" aren't the right words either...\nSearching I think works fine because you're searching in a numerical\nspace for the value that maximizes the score.\n\n> \n> > > + value: 1\n> > > + description: |\n> > > + The AEGC algorithm is actively computing new values, for either the\n> > > + exposure time or the analogue gain, but has not converged to a\n> > > + stable result yet.\n> > > +\n> > > + This state is returned if at least one of AnalogueGainMode\n> > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > + converged yet.\n> > > +\n> > > + The AEGC algorithm converges once stable values are computed for\n> > > + any of the controls set to be computed in Auto mode.\n> \n> Should that be \"all of the\"? \n\nOh yeah it should be.\n\n> \n> > > +\n> > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > + - name: AeStateConverged\n> > > + value: 2\n> > > + description: |\n> > > + The AEGC algorithm has converged.\n> > > +\n> > > + This state is returned if at least one of AnalogueGainMode\n> > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > + converged to stable value.\n> > > +\n> > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > + which case the state is moved to AeStateSearching.\n> \n> This sounds like it is based on a random event. What about something\n> like: When the statistics/measurements move too far away from the\n> convergence point the AEGC algo automatically restarts\n> regulation/scanning... \n\nIn a way I think it's spontaneous to the user :)\n\nBut indeed your description would be more accurate.\n\n> \n> > > +\n> > > # AeMeteringMode needs further attention:\n> > > # - Auto-generate max enum value.\n> > > # - Better handling of custom types.\n> > > @@ -104,6 +173,13 @@ controls:\n> > > The exposure modes specify how the desired total exposure is divided\n> > > between the shutter time and the sensor's analogue gain. They are\n> > > platform specific, and not all exposure modes may be supported.\n> > > +\n> > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > + the fixed values will override any choices made by AeExposureMode.\n> > > +\n> > > + \\sa AnalogueGainMode\n> > > + \\sa ExposureTimeMode\n> > > +\n> > > enum:\n> > > - name: ExposureNormal\n> > > value: 0\n> > > @@ -124,13 +200,15 @@ controls:\n> > > Specify an Exposure Value (EV) parameter.\n> > >\n> > > The EV parameter will only be applied if the AE algorithm is currently\n> > > - enabled.\n> > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > + are in Auto mode.\n> > >\n> > > By convention EV adjusts the exposure as log2. For example\n> > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > >\n> > > - \\sa AeEnable\n> > > + \\sa AnalogueGainMode\n> > > + \\sa ExposureTimeMode\n> > >\n> > > - ExposureTime:\n> > > type: int32_t\n> > > @@ -140,17 +218,95 @@ controls:\n> > >\n> > > This value is specified in micro-seconds.\n> > >\n> > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > - not change it. Setting it back to zero returns it to the control of the\n> > > - AE algorithm.\n> > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > + ignored and will not be retained.\n> > \n> > I'm not sure about this, it seems like a change of API behaviour which\n> > will require users to be re-educated and application code to be hunted\n> > down and changed. Previously you could just set the exposure time to X\n> > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > same, I think, with respect to setting the lens position. It's just\n> > more convenient.\n> \n> This is an interesting discussion. I wasn't aware of the old\n> specification. I see that just setting one value and deducing what the\n> user wants is convenient. On the other hand one control changing the\n> value of another control feels quite difficult to explain and in my\n> opinion should be prevented whenever possible. I believe it is easier to\n> educate the user and have an explicit interface than to have hidden\n> magic whenever possible.\n\nYeah that was my thought process too when I designed this.\n\n\nThanks,\n\nPaul\n\n> \n> > \n> > [There's also the question of how confident you are that pipeline\n> > handlers correctly deal with two controls like this that are\n> > effectively ordered...]\n> > \n> > > +\n> > > + When reported in metadata, this control indicates what exposure time\n> > > + was used for the current request, regardless of ExposureTimeMode.\n> > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > + whether it came from the AE algorithm or not.\n> > > +\n> > > + \\sa AnalogueGain\n> > > + \\sa ExposureTimeMode\n> > > +\n> > > + - ExposureTimeMode:\n> > > + type: int32_t\n> > > + description: |\n> > > + Controls the source of the exposure time that is applied to the image\n> > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > + and configures the image sensor accordingly. When set to Manual,\n> > > + the value of the ExposureTime control is used.\n> > > +\n> > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > + is provided by the application, the last value computed by the AE\n> > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > + was never set to Auto (either because the camera started in Manual mode,\n> > > + or Auto is not supported by the camera), the camera should use a\n> > > + best-effort default value.\n> > \n> > Yes, I think this sounds OK. So setting it to manual mode is what\n> > setting AeEnable to zero used to do, except that it can be done\n> > separately for exposure time and gain now.\n> > \n> > > +\n> > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > + also be supported.\n> > > +\n> > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > \n> > That had me scratching my head for a moment! Maybe\n> > \n> > \"At least one of the modes, auto or manual, must be supported by both\n> > ExposureTimeMode and AnalogueGainMode.\"\n> > \n> > Is that clearer? (Also, is it what you meant?)\n> > \n> > > +\n> > > + Flickerless exposure mode transitions\n> > > +\n> > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > + control of the exposure time should aim to do so by selecting an\n> > > + ExposureTime value as close as possible to the last value computed by\n> > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > \n> > Sounds a bit like you _have_ to do this. Maybe\n> > \n> > \"Applications that wish to transition from ..Auto to direct control of\n> > the exposure time without causing extra flicker, should do so by\n> > selecting an ExposureTime value as close as possible to the last value\n> > computed by the auto exposure algorithm.\"\n> > \n> > Don't know if that's better or not.\n> > \n> > > +\n> > > + To select the correct value to use as ExposureTime value, applications\n> > > + should accommodate the natural delay in applying controls caused by the\n> > > + capture pipeline frame depth.\n> > > +\n> > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > + immediately specify an ExposureTime value in the same request where\n> > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > + first Request where ExposureTimeMode is reported as\n> > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > + ExposureTime to populate the control value in the next Request to be\n> > > + queued to the Camera.\n> > > +\n> > > + The implementation of the auto-exposure algorithm should equally try to\n> > > + minimize flickering and when transitioning from manual exposure mode to\n> > > + auto exposure use the last value provided by the application as starting\n> > > + point.\n> > > +\n> > > + 1. Start with ExposureTimeMode set to Auto\n> > > +\n> > > + 2. Set ExposureTimeMode to Manual\n> > > +\n> > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > + set to Manual\n> > > +\n> > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > + submit it\n> > > +\n> > > + 5. Proceed to run manual exposure time\n> > > +\n> > > + \\sa ExposureTime\n> > > + enum:\n> > > + - name: ExposureTimeModeAuto\n> > > + value: 0\n> > > + description: |\n> > > + The exposure time will be calculated automatically and set by the\n> > > + AE algorithm.\n> > >\n> > > - \\sa AnalogueGain AeEnable\n> > > + If ExposureTime is set while this mode is active, it will be\n> > > + ignored, and it will also not be retained.\n> > > + - name: ExposureTimeModeManual\n> > > + value: 1\n> > > + description: |\n> > > + The exposure time will not be updated by the AE algorithm.\n> > >\n> > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > - value for this control. Consider interactions with other AE features,\n> > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > - control of which features should be automatically adjusted shouldn't\n> > > - better be handled through a separate AE mode control.\n> > > + When transitioning from Auto to Manual mode, the last computed\n> > > + exposure value is used until a new value is specified through the\n> > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > + same request where the ExposureTimeMode is changed from Auto to\n> > > + Manual, the provided ExposureTime is applied immediately.\n> > \n> > Did we say anywhere what happens when you go from manual back to auto?\n> > I'm assuming the adjustments restart from the manual values that you\n> > currently have.\n> > \n> > >\n> > > - AnalogueGain:\n> > > type: float\n> > > @@ -160,17 +316,64 @@ controls:\n> > > The value of the control specifies the gain multiplier applied to all\n> > > colour channels. This value cannot be lower than 1.0.\n> > >\n> > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > - not change it. Setting it back to zero returns it to the control of the\n> > > - AE algorithm.\n> > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > + ignored and will not be retained.\n> > \n> > As above. It would be user-friendly to switch automatically to manual, I think.\n> > \n> > > +\n> > > + When reported in metadata, this control indicates what analogue gain\n> > > + was used for the current request, regardless of AnalogueGainMode.\n> > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > + whether it came from the AEGC algorithm or not.\n> > > +\n> > > + \\sa ExposureTime\n> > > + \\sa AnalogueGainMode\n> > > +\n> > > + - AnalogueGainMode:\n> > > + type: int32_t\n> > > + description: |\n> > > + Controls the source of the analogue gain that is applied to the image\n> > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > + and configures the image sensor accordingly. When set to Manual,\n> > > + the value of the AnalogueGain control is used.\n> > > +\n> > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > + is provided by the application, the last value computed by the AEGC\n> > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > + was never set to Auto (either because the camera started in Manual mode,\n> > > + or Auto is not supported by the camera), the camera should use a\n> > > + best-effort default value.\n> > > +\n> > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > + also be supported.\n> > > +\n> > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > \n> > As above!\n> > \n> > Thanks again\n> > \n> > David\n> > \n> > > +\n> > > + The same procedure described for performing flickerless transitions in\n> > > + the ExposureTimeMode control documentation should be applied to\n> > > + analogue gain.\n> > > +\n> > > + \\sa ExposureTimeMode\n> > > + \\sa AnalogueGain\n> > > + enum:\n> > > + - name: AnalogueGainModeAuto\n> > > + value: 0\n> > > + description: |\n> > > + The analogue gain will be calculated automatically and set by the\n> > > + AEGC algorithm.\n> > >\n> > > - \\sa ExposureTime AeEnable\n> > > + If AnalogueGain is set while this mode is active, it will be\n> > > + ignored, and it will also not be retained.\n> > > + - name: AnalogueGainModeManual\n> > > + value: 1\n> > > + description: |\n> > > + The analogue gain will not be updated by the AEGC algorithm.\n> > >\n> > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > - value for this control. Consider interactions with other AE features,\n> > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > - control of which features should be automatically adjusted shouldn't\n> > > - better be handled through a separate AE mode control.\n> > > + When transitioning from Auto to Manual mode, the last computed\n> > > + gain value is used until a new value is specified through the\n> > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > + same request where the AnalogueGainMode is changed from Auto to\n> > > + Manual, the provided AnalogueGain is applied immediately.\n> > >\n> > > - AeFlickerMode:\n> > > type: int32_t\n> > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > index 1b284257f601..32c2b3feaf65 100644\n> > > --- a/src/libcamera/control_ids_draft.yaml\n> > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > @@ -77,35 +77,6 @@ controls:\n> > > High quality aberration correction which might reduce the frame\n> > > rate.\n> > >\n> > > - - AeState:\n> > > - type: int32_t\n> > > - description: |\n> > > - Control to report the current AE algorithm state. Currently identical to\n> > > - ANDROID_CONTROL_AE_STATE.\n> > > -\n> > > - Current state of the AE algorithm.\n> > > - enum:\n> > > - - name: AeStateInactive\n> > > - value: 0\n> > > - description: The AE algorithm is inactive.\n> > > - - name: AeStateSearching\n> > > - value: 1\n> > > - description: The AE algorithm has not converged yet.\n> > > - - name: AeStateConverged\n> > > - value: 2\n> > > - description: The AE algorithm has converged.\n> > > - - name: AeStateLocked\n> > > - value: 3\n> > > - description: The AE algorithm is locked.\n> > > - - name: AeStateFlashRequired\n> > > - value: 4\n> > > - description: The AE algorithm would need a flash for good results\n> > > - - name: AeStatePrecapture\n> > > - value: 5\n> > > - description: |\n> > > - The AE algorithm has started a pre-capture metering session.\n> > > - \\sa AePrecaptureTrigger\n> > > -\n> > > - AwbState:\n> > > type: int32_t\n> > > description: |\n> > > --\n> > > 2.39.2\n> > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BC56FC0F1B\n\tfor ;\n\tFri, 15 Nov 2024 13:41:58 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id F13DB65892;\n\tFri, 15 Nov 2024 14:41:57 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AD7316588D\n\tfor ;\n\tFri, 15 Nov 2024 14:41:56 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2404:7a81:160:2100:bd6c:1638:cb26:1bfa])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 33F25496;\n\tFri, 15 Nov 2024 14:41:38 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"dA809BCn\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1731678101;\n\tbh=070cVZ3pvoFYObLTfP/WYOetMvnA/O6bpCzcZFm7MVA=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=dA809BCnyP334XgRsd2yNICZIS9d1t8mUUGjYs1ojOOnR/OLuRaPG0EX8UoD+KiCp\n\tqBymE6UX1vYUNiP+oN/3Uwyfe+CI3cd/xbpf1mpVmqMdXCcFdbaXVcqyLgtjUNGWmV\n\t5xJ15b37WKErL4cctGF9TyTVxC/CXclDcPjg2F5A=","Date":"Fri, 15 Nov 2024 22:41:47 +0900","From":"Paul Elder ","To":"Stefan Klug ","Cc":"David Plowman ,\n\tlibcamera-devel@lists.libcamera.org,\n\tlaurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com,\n\tnaush@raspberrypi.com, Jacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32194,"web_url":"https://patchwork.libcamera.org/comment/32194/","msgid":"","date":"2024-11-15T13:51:00","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n>\n> On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > Hi Paul,\n> >\n> > Thank you for the patch.\n> >\n> > > Hi Paul\n> > >\n> > > Thanks for posting all this!\n> > >\n> > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > >\n> > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > to model the AEGC algorithm block.\n> > > >\n> > > > The three controls allow applications to select the exposure time and\n> > > > analogue gain computation calculation mode (auto vs manual)\n> > > > independently from one another, while the AeState control reports the\n> > > > global state for the AEGC algorithm.\n> > > >\n> > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > controls, which are momentarily kept not to break compilation of\n> > > > platforms making use of them.\n> > > >\n> > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > Signed-off-by: Paul Elder \n> > > > Signed-off-by: Jacopo Mondi \n> > > >\n> > > > ---\n> > > > Changes in v3:\n> > > > - recovered from 2-year-old bitrot\n> > >\n> > > Indeed, I think I remember discussing all this _years_ ago!!\n> > >\n> > > > ---\n> > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > >\n> > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > @@ -26,6 +26,75 @@ controls:\n> > > >\n> > > > \\sa AeEnable\n> > > >\n> > > > + - AeState:\n> > > > + type: int32_t\n> > > > + description: |\n> > > > + Control to report the AEGC algorithm state.\n> > > > +\n> > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > + values to be applied to the image sensor.\n> > > > +\n> > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > + independently from one another.\n> > > > +\n> > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > + value and describes it as a single computation block which computes\n> > > > + both the exposure time and the analogue gain values.\n> > > > +\n> > > > + When both the exposure time and analogue gain values are configured to\n> > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > +\n> > > > + When at least the exposure time or analogue gain are configured to be\n> > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > + algorithm has converged to stable values for all of the controls set\n> > > > + to be computed in Auto mode.\n> > > > +\n> > > > + \\sa AnalogueGainMode\n> > > > + \\sa ExposureTimeMode\n> > > > +\n> > > > + enum:\n> > > > + - name: AeStateIdle\n> > > > + value: 0\n> > > > + description: |\n> > > > + The AEGC algorithm is inactive.\n> > > > +\n> > > > + This state is returned when both AnalogueGainMode and\n> > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > + actively computing any value.\n> > > > +\n> > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > >\n> > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > autofocus description! But I'm not too bothered.\n> > >\n> > > > + case the AeState control is moved to AeStateSearching.\n> > > > + - name: AeStateSearching\n> >\n> > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > well be because I'm not a native speaker). How does \"regulating\" or\n> > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > converge, but a regulation does.\n>\n> Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> Searching I think works fine because you're searching in a numerical\n> space for the value that maximizes the score.\n>\n> >\n> > > > + value: 1\n> > > > + description: |\n> > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > + exposure time or the analogue gain, but has not converged to a\n> > > > + stable result yet.\n> > > > +\n> > > > + This state is returned if at least one of AnalogueGainMode\n> > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > + converged yet.\n> > > > +\n> > > > + The AEGC algorithm converges once stable values are computed for\n> > > > + any of the controls set to be computed in Auto mode.\n> >\n> > Should that be \"all of the\"?\n>\n> Oh yeah it should be.\n>\n> >\n> > > > +\n> > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > + - name: AeStateConverged\n> > > > + value: 2\n> > > > + description: |\n> > > > + The AEGC algorithm has converged.\n> > > > +\n> > > > + This state is returned if at least one of AnalogueGainMode\n> > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > + converged to stable value.\n> > > > +\n> > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > + which case the state is moved to AeStateSearching.\n> >\n> > This sounds like it is based on a random event. What about something\n> > like: When the statistics/measurements move too far away from the\n> > convergence point the AEGC algo automatically restarts\n> > regulation/scanning...\n>\n> In a way I think it's spontaneous to the user :)\n>\n> But indeed your description would be more accurate.\n>\n> >\n> > > > +\n> > > > # AeMeteringMode needs further attention:\n> > > > # - Auto-generate max enum value.\n> > > > # - Better handling of custom types.\n> > > > @@ -104,6 +173,13 @@ controls:\n> > > > The exposure modes specify how the desired total exposure is divided\n> > > > between the shutter time and the sensor's analogue gain. They are\n> > > > platform specific, and not all exposure modes may be supported.\n> > > > +\n> > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > +\n> > > > + \\sa AnalogueGainMode\n> > > > + \\sa ExposureTimeMode\n> > > > +\n> > > > enum:\n> > > > - name: ExposureNormal\n> > > > value: 0\n> > > > @@ -124,13 +200,15 @@ controls:\n> > > > Specify an Exposure Value (EV) parameter.\n> > > >\n> > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > - enabled.\n> > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > + are in Auto mode.\n> > > >\n> > > > By convention EV adjusts the exposure as log2. For example\n> > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > >\n> > > > - \\sa AeEnable\n> > > > + \\sa AnalogueGainMode\n> > > > + \\sa ExposureTimeMode\n> > > >\n> > > > - ExposureTime:\n> > > > type: int32_t\n> > > > @@ -140,17 +218,95 @@ controls:\n> > > >\n> > > > This value is specified in micro-seconds.\n> > > >\n> > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > - AE algorithm.\n> > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > + ignored and will not be retained.\n> > >\n> > > I'm not sure about this, it seems like a change of API behaviour which\n> > > will require users to be re-educated and application code to be hunted\n> > > down and changed. Previously you could just set the exposure time to X\n> > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > same, I think, with respect to setting the lens position. It's just\n> > > more convenient.\n> >\n> > This is an interesting discussion. I wasn't aware of the old\n> > specification. I see that just setting one value and deducing what the\n> > user wants is convenient. On the other hand one control changing the\n> > value of another control feels quite difficult to explain and in my\n> > opinion should be prevented whenever possible. I believe it is easier to\n> > educate the user and have an explicit interface than to have hidden\n> > magic whenever possible.\n>\n> Yeah that was my thought process too when I designed this.\n\nThere is one minor annoyance this change will introduce because of the\nlack of ordering in the ControlList map.\n\nGoing from, say, auto control into manual will require 2 controls, and\nthe presence of the AnalogueGainModeManual enum + the AnalogueGain\nvalue to use. Since Controls are not ordered, if you iterate through\nthe ControlList, you will have to check for the presence of one, then\nthe other programmatically.\n\n>\n>\n> Thanks,\n>\n> Paul\n>\n> >\n> > >\n> > > [There's also the question of how confident you are that pipeline\n> > > handlers correctly deal with two controls like this that are\n> > > effectively ordered...]\n> > >\n> > > > +\n> > > > + When reported in metadata, this control indicates what exposure time\n> > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > + whether it came from the AE algorithm or not.\n> > > > +\n> > > > + \\sa AnalogueGain\n> > > > + \\sa ExposureTimeMode\n> > > > +\n> > > > + - ExposureTimeMode:\n> > > > + type: int32_t\n> > > > + description: |\n> > > > + Controls the source of the exposure time that is applied to the image\n> > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > + the value of the ExposureTime control is used.\n> > > > +\n> > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > + is provided by the application, the last value computed by the AE\n> > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > + or Auto is not supported by the camera), the camera should use a\n> > > > + best-effort default value.\n> > >\n> > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > setting AeEnable to zero used to do, except that it can be done\n> > > separately for exposure time and gain now.\n> > >\n> > > > +\n> > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > + also be supported.\n> > > > +\n> > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > >\n> > > That had me scratching my head for a moment! Maybe\n> > >\n> > > \"At least one of the modes, auto or manual, must be supported by both\n> > > ExposureTimeMode and AnalogueGainMode.\"\n> > >\n> > > Is that clearer? (Also, is it what you meant?)\n> > >\n> > > > +\n> > > > + Flickerless exposure mode transitions\n> > > > +\n> > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > + control of the exposure time should aim to do so by selecting an\n> > > > + ExposureTime value as close as possible to the last value computed by\n> > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > >\n> > > Sounds a bit like you _have_ to do this. Maybe\n> > >\n> > > \"Applications that wish to transition from ..Auto to direct control of\n> > > the exposure time without causing extra flicker, should do so by\n> > > selecting an ExposureTime value as close as possible to the last value\n> > > computed by the auto exposure algorithm.\"\n> > >\n> > > Don't know if that's better or not.\n> > >\n> > > > +\n> > > > + To select the correct value to use as ExposureTime value, applications\n> > > > + should accommodate the natural delay in applying controls caused by the\n> > > > + capture pipeline frame depth.\n> > > > +\n> > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > + immediately specify an ExposureTime value in the same request where\n> > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > + first Request where ExposureTimeMode is reported as\n> > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > + ExposureTime to populate the control value in the next Request to be\n> > > > + queued to the Camera.\n> > > > +\n> > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > + auto exposure use the last value provided by the application as starting\n> > > > + point.\n> > > > +\n> > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > +\n> > > > + 2. Set ExposureTimeMode to Manual\n> > > > +\n> > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > + set to Manual\n> > > > +\n> > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > + submit it\n> > > > +\n> > > > + 5. Proceed to run manual exposure time\n> > > > +\n> > > > + \\sa ExposureTime\n> > > > + enum:\n> > > > + - name: ExposureTimeModeAuto\n> > > > + value: 0\n> > > > + description: |\n> > > > + The exposure time will be calculated automatically and set by the\n> > > > + AE algorithm.\n> > > >\n> > > > - \\sa AnalogueGain AeEnable\n> > > > + If ExposureTime is set while this mode is active, it will be\n> > > > + ignored, and it will also not be retained.\n> > > > + - name: ExposureTimeModeManual\n> > > > + value: 1\n> > > > + description: |\n> > > > + The exposure time will not be updated by the AE algorithm.\n> > > >\n> > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > - value for this control. Consider interactions with other AE features,\n> > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > - control of which features should be automatically adjusted shouldn't\n> > > > - better be handled through a separate AE mode control.\n> > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > + exposure value is used until a new value is specified through the\n> > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > + Manual, the provided ExposureTime is applied immediately.\n> > >\n> > > Did we say anywhere what happens when you go from manual back to auto?\n> > > I'm assuming the adjustments restart from the manual values that you\n> > > currently have.\n> > >\n> > > >\n> > > > - AnalogueGain:\n> > > > type: float\n> > > > @@ -160,17 +316,64 @@ controls:\n> > > > The value of the control specifies the gain multiplier applied to all\n> > > > colour channels. This value cannot be lower than 1.0.\n> > > >\n> > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > - AE algorithm.\n> > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > + ignored and will not be retained.\n> > >\n> > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > >\n> > > > +\n> > > > + When reported in metadata, this control indicates what analogue gain\n> > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > + whether it came from the AEGC algorithm or not.\n> > > > +\n> > > > + \\sa ExposureTime\n> > > > + \\sa AnalogueGainMode\n> > > > +\n> > > > + - AnalogueGainMode:\n> > > > + type: int32_t\n> > > > + description: |\n> > > > + Controls the source of the analogue gain that is applied to the image\n> > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > + the value of the AnalogueGain control is used.\n> > > > +\n> > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > + is provided by the application, the last value computed by the AEGC\n> > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > + or Auto is not supported by the camera), the camera should use a\n> > > > + best-effort default value.\n> > > > +\n> > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > + also be supported.\n> > > > +\n> > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > >\n> > > As above!\n> > >\n> > > Thanks again\n> > >\n> > > David\n> > >\n> > > > +\n> > > > + The same procedure described for performing flickerless transitions in\n> > > > + the ExposureTimeMode control documentation should be applied to\n> > > > + analogue gain.\n> > > > +\n> > > > + \\sa ExposureTimeMode\n> > > > + \\sa AnalogueGain\n> > > > + enum:\n> > > > + - name: AnalogueGainModeAuto\n> > > > + value: 0\n> > > > + description: |\n> > > > + The analogue gain will be calculated automatically and set by the\n> > > > + AEGC algorithm.\n> > > >\n> > > > - \\sa ExposureTime AeEnable\n> > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > + ignored, and it will also not be retained.\n> > > > + - name: AnalogueGainModeManual\n> > > > + value: 1\n> > > > + description: |\n> > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > >\n> > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > - value for this control. Consider interactions with other AE features,\n> > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > - control of which features should be automatically adjusted shouldn't\n> > > > - better be handled through a separate AE mode control.\n> > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > + gain value is used until a new value is specified through the\n> > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > >\n> > > > - AeFlickerMode:\n> > > > type: int32_t\n> > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > @@ -77,35 +77,6 @@ controls:\n> > > > High quality aberration correction which might reduce the frame\n> > > > rate.\n> > > >\n> > > > - - AeState:\n> > > > - type: int32_t\n> > > > - description: |\n> > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > - ANDROID_CONTROL_AE_STATE.\n> > > > -\n> > > > - Current state of the AE algorithm.\n> > > > - enum:\n> > > > - - name: AeStateInactive\n> > > > - value: 0\n> > > > - description: The AE algorithm is inactive.\n> > > > - - name: AeStateSearching\n> > > > - value: 1\n> > > > - description: The AE algorithm has not converged yet.\n> > > > - - name: AeStateConverged\n> > > > - value: 2\n> > > > - description: The AE algorithm has converged.\n> > > > - - name: AeStateLocked\n> > > > - value: 3\n> > > > - description: The AE algorithm is locked.\n> > > > - - name: AeStateFlashRequired\n> > > > - value: 4\n> > > > - description: The AE algorithm would need a flash for good results\n> > > > - - name: AeStatePrecapture\n> > > > - value: 5\n> > > > - description: |\n> > > > - The AE algorithm has started a pre-capture metering session.\n> > > > - \\sa AePrecaptureTrigger\n> > > > -\n> > > > - AwbState:\n> > > > type: int32_t\n> > > > description: |\n> > > > --\n> > > > 2.39.2\n> > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DA911C0F1B\n\tfor ;\n\tFri, 15 Nov 2024 13:51:39 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2EB3B65892;\n\tFri, 15 Nov 2024 14:51:39 +0100 (CET)","from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com\n\t[IPv6:2607:f8b0:4864:20::112d])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1F2B265882\n\tfor ;\n\tFri, 15 Nov 2024 14:51:37 +0100 (CET)","by mail-yw1-x112d.google.com with SMTP id\n\t00721157ae682-6ea34be9b20so1664657b3.3\n\tfor ;\n\tFri, 15 Nov 2024 05:51:37 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"Ou2yo7Uv\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1731678696; x=1732283496;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=6jH8u46e7tGXWmKAu+dhftsvy/32BTPE0MpxHTsLX9M=;\n\tb=Ou2yo7Uv4pbKTAZIRzkul5jYZ4JfCO614CITIrEe6LjojpF+u8fBnPqN62S06M53kT\n\t7kWp09cwx3HkkA6eiWL4sxaIHB/4AYDV9AA24/GZpdg+Njrj+bgGC5kpMsPogYkSVCTa\n\tmii41j4M4+fDkNUvmeznRuIkMDeS1lwwtDysqduHhC3IaMvK3FRQLiLcYQ8igSGGaNLj\n\tuthkr6VVTJ9WjPiNPAvSfISXQUGf/MljkU7ofdOYQ1HoN55gL9iUMk9yg8YVQcWHYhnt\n\tf90Bw/mh6DD8wIXiy5kAcQ8MrV8oquIVSI4k0eSkGuiZI+5D71iE5Km6lqeXLHlabiQk\n\tEmwg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1731678696; x=1732283496;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=6jH8u46e7tGXWmKAu+dhftsvy/32BTPE0MpxHTsLX9M=;\n\tb=mdklHSVOTlQzy+QhVi0MCxMs84dDVf2yJpUXT/sktRClqJhCSgNFzAh23JXnrqbDld\n\tkOD1MV31x/Kr8Mg3BupzMwIEWTakAb+bwV5SAF5kYvD25Y9n8riniz2kPHQKRIwDrMDm\n\tL/V4qpbxzr6lzWfUOtRKT2Xf0fPuGZ6Azvg5soIXvhu6Y5o9o7XzD80qiEnDgop8sNLM\n\t5xVeyxDKrRHzWuUgrXzPn3gUbZklHb85Xy9ayZShEOmo7+ZMngFQEEuzI6qUzWpl3F2d\n\tEK/EGnR3QQkBllyygNOheGQYyl8Pq9PEVc45PUhtYXzkXRVBTUTSlDmt6MNzeD48Z/n1\n\tbmUA==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCUwnqpRZ+ZhadS9EVgCtmV/xncapoUPcFoZx4j3JLkBqdT9VwmVBKJfRk7b65qI6xthEH/CvoRlcg3KL0PhHgM=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YwJCGlLf0ksfLQyjuH/nTfE+nXby/onzQVEnR9UgkAjP+X56EP6\n\t56OlgO2VGkYd7xjxhuknmOnJkuY+8LwQfuYtKh7NvOqQJVX/wNtLBHGolVERY2xfAtoW1qMK1na\n\tEIvcEEN6V8m/01BWdWQPz3oOnsHKj3sjN+vjTdQ==","X-Gm-Gg":"ASbGncuW5tWEEJUghCVhsz2MqadnQxbicFVJ+TZHknKqF+RquyZiceKCOa2AJhdpD4H\n\tG1D4ElpCzLaTQQrzC/hmtVf7yMDV6qMZZDaO0wAlm+ocsUjOd3K+t19iCpEoA","X-Google-Smtp-Source":"AGHT+IEe7m0ZAOBMFJjJB6wQgHtld2pvYg2qQJO7swMf7hzUhJbePYUp3LXf+GXmpvE/2x1bvcB2yJR4iHC3NCjjNgY=","X-Received":"by 2002:a05:690c:46c4:b0:6d1:f545:3c8b with SMTP id\n\t00721157ae682-6ee55bf0908mr11210297b3.4.1731678695712;\n\tFri, 15 Nov 2024 05:51:35 -0800 (PST)","MIME-Version":"1.0","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t","In-Reply-To":"","From":"Naushir Patuck ","Date":"Fri, 15 Nov 2024 13:51:00 +0000","Message-ID":"","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","To":"Paul Elder ","Cc":"Stefan Klug , \n\tDavid Plowman ,\n\tlibcamera-devel@lists.libcamera.org, \n\tlaurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com, \n\tJacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32210,"web_url":"https://patchwork.libcamera.org/comment/32210/","msgid":"","date":"2024-11-18T08:30:39","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"On Fri, Nov 15, 2024 at 01:51:00PM +0000, Naushir Patuck wrote:\n> On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n> >\n> > On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > > Hi Paul,\n> > >\n> > > Thank you for the patch.\n> > >\n> > > > Hi Paul\n> > > >\n> > > > Thanks for posting all this!\n> > > >\n> > > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > > >\n> > > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > > to model the AEGC algorithm block.\n> > > > >\n> > > > > The three controls allow applications to select the exposure time and\n> > > > > analogue gain computation calculation mode (auto vs manual)\n> > > > > independently from one another, while the AeState control reports the\n> > > > > global state for the AEGC algorithm.\n> > > > >\n> > > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > > controls, which are momentarily kept not to break compilation of\n> > > > > platforms making use of them.\n> > > > >\n> > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > > Signed-off-by: Paul Elder \n> > > > > Signed-off-by: Jacopo Mondi \n> > > > >\n> > > > > ---\n> > > > > Changes in v3:\n> > > > > - recovered from 2-year-old bitrot\n> > > >\n> > > > Indeed, I think I remember discussing all this _years_ ago!!\n> > > >\n> > > > > ---\n> > > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > > >\n> > > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > > @@ -26,6 +26,75 @@ controls:\n> > > > >\n> > > > > \\sa AeEnable\n> > > > >\n> > > > > + - AeState:\n> > > > > + type: int32_t\n> > > > > + description: |\n> > > > > + Control to report the AEGC algorithm state.\n> > > > > +\n> > > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > > + values to be applied to the image sensor.\n> > > > > +\n> > > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > > + independently from one another.\n> > > > > +\n> > > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > > + value and describes it as a single computation block which computes\n> > > > > + both the exposure time and the analogue gain values.\n> > > > > +\n> > > > > + When both the exposure time and analogue gain values are configured to\n> > > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > > +\n> > > > > + When at least the exposure time or analogue gain are configured to be\n> > > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > > + algorithm has converged to stable values for all of the controls set\n> > > > > + to be computed in Auto mode.\n> > > > > +\n> > > > > + \\sa AnalogueGainMode\n> > > > > + \\sa ExposureTimeMode\n> > > > > +\n> > > > > + enum:\n> > > > > + - name: AeStateIdle\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The AEGC algorithm is inactive.\n> > > > > +\n> > > > > + This state is returned when both AnalogueGainMode and\n> > > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > > + actively computing any value.\n> > > > > +\n> > > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > > >\n> > > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > > autofocus description! But I'm not too bothered.\n> > > >\n> > > > > + case the AeState control is moved to AeStateSearching.\n> > > > > + - name: AeStateSearching\n> > >\n> > > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > > well be because I'm not a native speaker). How does \"regulating\" or\n> > > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > > converge, but a regulation does.\n> >\n> > Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> > Searching I think works fine because you're searching in a numerical\n> > space for the value that maximizes the score.\n> >\n> > >\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > > + exposure time or the analogue gain, but has not converged to a\n> > > > > + stable result yet.\n> > > > > +\n> > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > > + converged yet.\n> > > > > +\n> > > > > + The AEGC algorithm converges once stable values are computed for\n> > > > > + any of the controls set to be computed in Auto mode.\n> > >\n> > > Should that be \"all of the\"?\n> >\n> > Oh yeah it should be.\n> >\n> > >\n> > > > > +\n> > > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > > + - name: AeStateConverged\n> > > > > + value: 2\n> > > > > + description: |\n> > > > > + The AEGC algorithm has converged.\n> > > > > +\n> > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > > + converged to stable value.\n> > > > > +\n> > > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > > + which case the state is moved to AeStateSearching.\n> > >\n> > > This sounds like it is based on a random event. What about something\n> > > like: When the statistics/measurements move too far away from the\n> > > convergence point the AEGC algo automatically restarts\n> > > regulation/scanning...\n> >\n> > In a way I think it's spontaneous to the user :)\n> >\n> > But indeed your description would be more accurate.\n> >\n> > >\n> > > > > +\n> > > > > # AeMeteringMode needs further attention:\n> > > > > # - Auto-generate max enum value.\n> > > > > # - Better handling of custom types.\n> > > > > @@ -104,6 +173,13 @@ controls:\n> > > > > The exposure modes specify how the desired total exposure is divided\n> > > > > between the shutter time and the sensor's analogue gain. They are\n> > > > > platform specific, and not all exposure modes may be supported.\n> > > > > +\n> > > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > > +\n> > > > > + \\sa AnalogueGainMode\n> > > > > + \\sa ExposureTimeMode\n> > > > > +\n> > > > > enum:\n> > > > > - name: ExposureNormal\n> > > > > value: 0\n> > > > > @@ -124,13 +200,15 @@ controls:\n> > > > > Specify an Exposure Value (EV) parameter.\n> > > > >\n> > > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > > - enabled.\n> > > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > > + are in Auto mode.\n> > > > >\n> > > > > By convention EV adjusts the exposure as log2. For example\n> > > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > > >\n> > > > > - \\sa AeEnable\n> > > > > + \\sa AnalogueGainMode\n> > > > > + \\sa ExposureTimeMode\n> > > > >\n> > > > > - ExposureTime:\n> > > > > type: int32_t\n> > > > > @@ -140,17 +218,95 @@ controls:\n> > > > >\n> > > > > This value is specified in micro-seconds.\n> > > > >\n> > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > - AE algorithm.\n> > > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > > + ignored and will not be retained.\n> > > >\n> > > > I'm not sure about this, it seems like a change of API behaviour which\n> > > > will require users to be re-educated and application code to be hunted\n> > > > down and changed. Previously you could just set the exposure time to X\n> > > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > > same, I think, with respect to setting the lens position. It's just\n> > > > more convenient.\n> > >\n> > > This is an interesting discussion. I wasn't aware of the old\n> > > specification. I see that just setting one value and deducing what the\n> > > user wants is convenient. On the other hand one control changing the\n> > > value of another control feels quite difficult to explain and in my\n> > > opinion should be prevented whenever possible. I believe it is easier to\n> > > educate the user and have an explicit interface than to have hidden\n> > > magic whenever possible.\n> >\n> > Yeah that was my thought process too when I designed this.\n> \n> There is one minor annoyance this change will introduce because of the\n> lack of ordering in the ControlList map.\n> \n> Going from, say, auto control into manual will require 2 controls, and\n> the presence of the AnalogueGainModeManual enum + the AnalogueGain\n> value to use. Since Controls are not ordered, if you iterate through\n> the ControlList, you will have to check for the presence of one, then\n> the other programmatically.\n> \n\nOh good point, I totally forgot about that. Back to the drawing board...\n\n\nPaul\n\n> >\n> > >\n> > > >\n> > > > [There's also the question of how confident you are that pipeline\n> > > > handlers correctly deal with two controls like this that are\n> > > > effectively ordered...]\n> > > >\n> > > > > +\n> > > > > + When reported in metadata, this control indicates what exposure time\n> > > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > > + whether it came from the AE algorithm or not.\n> > > > > +\n> > > > > + \\sa AnalogueGain\n> > > > > + \\sa ExposureTimeMode\n> > > > > +\n> > > > > + - ExposureTimeMode:\n> > > > > + type: int32_t\n> > > > > + description: |\n> > > > > + Controls the source of the exposure time that is applied to the image\n> > > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > + the value of the ExposureTime control is used.\n> > > > > +\n> > > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > > + is provided by the application, the last value computed by the AE\n> > > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > + best-effort default value.\n> > > >\n> > > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > > setting AeEnable to zero used to do, except that it can be done\n> > > > separately for exposure time and gain now.\n> > > >\n> > > > > +\n> > > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > > + also be supported.\n> > > > > +\n> > > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > > >\n> > > > That had me scratching my head for a moment! Maybe\n> > > >\n> > > > \"At least one of the modes, auto or manual, must be supported by both\n> > > > ExposureTimeMode and AnalogueGainMode.\"\n> > > >\n> > > > Is that clearer? (Also, is it what you meant?)\n> > > >\n> > > > > +\n> > > > > + Flickerless exposure mode transitions\n> > > > > +\n> > > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > > + control of the exposure time should aim to do so by selecting an\n> > > > > + ExposureTime value as close as possible to the last value computed by\n> > > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > > >\n> > > > Sounds a bit like you _have_ to do this. Maybe\n> > > >\n> > > > \"Applications that wish to transition from ..Auto to direct control of\n> > > > the exposure time without causing extra flicker, should do so by\n> > > > selecting an ExposureTime value as close as possible to the last value\n> > > > computed by the auto exposure algorithm.\"\n> > > >\n> > > > Don't know if that's better or not.\n> > > >\n> > > > > +\n> > > > > + To select the correct value to use as ExposureTime value, applications\n> > > > > + should accommodate the natural delay in applying controls caused by the\n> > > > > + capture pipeline frame depth.\n> > > > > +\n> > > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > > + immediately specify an ExposureTime value in the same request where\n> > > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > > + first Request where ExposureTimeMode is reported as\n> > > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > > + ExposureTime to populate the control value in the next Request to be\n> > > > > + queued to the Camera.\n> > > > > +\n> > > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > > + auto exposure use the last value provided by the application as starting\n> > > > > + point.\n> > > > > +\n> > > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > > +\n> > > > > + 2. Set ExposureTimeMode to Manual\n> > > > > +\n> > > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > > + set to Manual\n> > > > > +\n> > > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > > + submit it\n> > > > > +\n> > > > > + 5. Proceed to run manual exposure time\n> > > > > +\n> > > > > + \\sa ExposureTime\n> > > > > + enum:\n> > > > > + - name: ExposureTimeModeAuto\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The exposure time will be calculated automatically and set by the\n> > > > > + AE algorithm.\n> > > > >\n> > > > > - \\sa AnalogueGain AeEnable\n> > > > > + If ExposureTime is set while this mode is active, it will be\n> > > > > + ignored, and it will also not be retained.\n> > > > > + - name: ExposureTimeModeManual\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The exposure time will not be updated by the AE algorithm.\n> > > > >\n> > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > - value for this control. Consider interactions with other AE features,\n> > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > - better be handled through a separate AE mode control.\n> > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > + exposure value is used until a new value is specified through the\n> > > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > > + Manual, the provided ExposureTime is applied immediately.\n> > > >\n> > > > Did we say anywhere what happens when you go from manual back to auto?\n> > > > I'm assuming the adjustments restart from the manual values that you\n> > > > currently have.\n> > > >\n> > > > >\n> > > > > - AnalogueGain:\n> > > > > type: float\n> > > > > @@ -160,17 +316,64 @@ controls:\n> > > > > The value of the control specifies the gain multiplier applied to all\n> > > > > colour channels. This value cannot be lower than 1.0.\n> > > > >\n> > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > - AE algorithm.\n> > > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > > + ignored and will not be retained.\n> > > >\n> > > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > > >\n> > > > > +\n> > > > > + When reported in metadata, this control indicates what analogue gain\n> > > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > > + whether it came from the AEGC algorithm or not.\n> > > > > +\n> > > > > + \\sa ExposureTime\n> > > > > + \\sa AnalogueGainMode\n> > > > > +\n> > > > > + - AnalogueGainMode:\n> > > > > + type: int32_t\n> > > > > + description: |\n> > > > > + Controls the source of the analogue gain that is applied to the image\n> > > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > + the value of the AnalogueGain control is used.\n> > > > > +\n> > > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > > + is provided by the application, the last value computed by the AEGC\n> > > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > + best-effort default value.\n> > > > > +\n> > > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > > + also be supported.\n> > > > > +\n> > > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > > >\n> > > > As above!\n> > > >\n> > > > Thanks again\n> > > >\n> > > > David\n> > > >\n> > > > > +\n> > > > > + The same procedure described for performing flickerless transitions in\n> > > > > + the ExposureTimeMode control documentation should be applied to\n> > > > > + analogue gain.\n> > > > > +\n> > > > > + \\sa ExposureTimeMode\n> > > > > + \\sa AnalogueGain\n> > > > > + enum:\n> > > > > + - name: AnalogueGainModeAuto\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The analogue gain will be calculated automatically and set by the\n> > > > > + AEGC algorithm.\n> > > > >\n> > > > > - \\sa ExposureTime AeEnable\n> > > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > > + ignored, and it will also not be retained.\n> > > > > + - name: AnalogueGainModeManual\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > > >\n> > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > - value for this control. Consider interactions with other AE features,\n> > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > - better be handled through a separate AE mode control.\n> > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > + gain value is used until a new value is specified through the\n> > > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > > >\n> > > > > - AeFlickerMode:\n> > > > > type: int32_t\n> > > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > > @@ -77,35 +77,6 @@ controls:\n> > > > > High quality aberration correction which might reduce the frame\n> > > > > rate.\n> > > > >\n> > > > > - - AeState:\n> > > > > - type: int32_t\n> > > > > - description: |\n> > > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > > - ANDROID_CONTROL_AE_STATE.\n> > > > > -\n> > > > > - Current state of the AE algorithm.\n> > > > > - enum:\n> > > > > - - name: AeStateInactive\n> > > > > - value: 0\n> > > > > - description: The AE algorithm is inactive.\n> > > > > - - name: AeStateSearching\n> > > > > - value: 1\n> > > > > - description: The AE algorithm has not converged yet.\n> > > > > - - name: AeStateConverged\n> > > > > - value: 2\n> > > > > - description: The AE algorithm has converged.\n> > > > > - - name: AeStateLocked\n> > > > > - value: 3\n> > > > > - description: The AE algorithm is locked.\n> > > > > - - name: AeStateFlashRequired\n> > > > > - value: 4\n> > > > > - description: The AE algorithm would need a flash for good results\n> > > > > - - name: AeStatePrecapture\n> > > > > - value: 5\n> > > > > - description: |\n> > > > > - The AE algorithm has started a pre-capture metering session.\n> > > > > - \\sa AePrecaptureTrigger\n> > > > > -\n> > > > > - AwbState:\n> > > > > type: int32_t\n> > > > > description: |\n> > > > > --\n> > > > > 2.39.2\n> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BB10BC32EA\n\tfor ;\n\tMon, 18 Nov 2024 08:30:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EDFC8658C1;\n\tMon, 18 Nov 2024 09:30:49 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 816FD618BC\n\tfor ;\n\tMon, 18 Nov 2024 09:30:48 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2404:7a81:160:2100:fb8e:b30c:9a4a:eed0])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 71ED08FA;\n\tMon, 18 Nov 2024 09:30:29 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"fuMwyP0k\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1731918631;\n\tbh=DOvlyHP9iZ8TUYXc5rD92cXUETiP8IFBllnR52GRAY8=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=fuMwyP0kRC/nGm1dp4/dBEoIWbSc8Zt2haS56Y8VWBLPYi+iHS0htyXkM99PTOlO0\n\tKcmU1JcRBsOoXz7aZTdl5rB+6IDirMEDHpM5rc65Ja028VAsjoCSvha/G+Lg9hgxck\n\tr4lgQyVuQKLmHO/2I7U2vCrasbKBvwer1J2xo+kk=","Date":"Mon, 18 Nov 2024 17:30:39 +0900","From":"Paul Elder ","To":"Naushir Patuck ","Cc":"Stefan Klug ,\n\tDavid Plowman ,\n\tlibcamera-devel@lists.libcamera.org,\n\tlaurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com,\n\tJacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32225,"web_url":"https://patchwork.libcamera.org/comment/32225/","msgid":"","date":"2024-11-18T13:51:02","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"On Mon, Nov 18, 2024 at 05:30:39PM +0900, Paul Elder wrote:\n> On Fri, Nov 15, 2024 at 01:51:00PM +0000, Naushir Patuck wrote:\n> > On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n> > >\n> > > On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > > > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > > > Hi Paul,\n> > > >\n> > > > Thank you for the patch.\n> > > >\n> > > > > Hi Paul\n> > > > >\n> > > > > Thanks for posting all this!\n> > > > >\n> > > > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > > > >\n> > > > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > > > to model the AEGC algorithm block.\n> > > > > >\n> > > > > > The three controls allow applications to select the exposure time and\n> > > > > > analogue gain computation calculation mode (auto vs manual)\n> > > > > > independently from one another, while the AeState control reports the\n> > > > > > global state for the AEGC algorithm.\n> > > > > >\n> > > > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > > > controls, which are momentarily kept not to break compilation of\n> > > > > > platforms making use of them.\n> > > > > >\n> > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > > > Signed-off-by: Paul Elder \n> > > > > > Signed-off-by: Jacopo Mondi \n> > > > > >\n> > > > > > ---\n> > > > > > Changes in v3:\n> > > > > > - recovered from 2-year-old bitrot\n> > > > >\n> > > > > Indeed, I think I remember discussing all this _years_ ago!!\n> > > > >\n> > > > > > ---\n> > > > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > > > >\n> > > > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > > > @@ -26,6 +26,75 @@ controls:\n> > > > > >\n> > > > > > \\sa AeEnable\n> > > > > >\n> > > > > > + - AeState:\n> > > > > > + type: int32_t\n> > > > > > + description: |\n> > > > > > + Control to report the AEGC algorithm state.\n> > > > > > +\n> > > > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > > > + values to be applied to the image sensor.\n> > > > > > +\n> > > > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > > > + independently from one another.\n> > > > > > +\n> > > > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > > > + value and describes it as a single computation block which computes\n> > > > > > + both the exposure time and the analogue gain values.\n> > > > > > +\n> > > > > > + When both the exposure time and analogue gain values are configured to\n> > > > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > > > +\n> > > > > > + When at least the exposure time or analogue gain are configured to be\n> > > > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > > > + algorithm has converged to stable values for all of the controls set\n> > > > > > + to be computed in Auto mode.\n> > > > > > +\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > +\n> > > > > > + enum:\n> > > > > > + - name: AeStateIdle\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The AEGC algorithm is inactive.\n> > > > > > +\n> > > > > > + This state is returned when both AnalogueGainMode and\n> > > > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > > > + actively computing any value.\n> > > > > > +\n> > > > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > > > >\n> > > > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > > > autofocus description! But I'm not too bothered.\n> > > > >\n> > > > > > + case the AeState control is moved to AeStateSearching.\n> > > > > > + - name: AeStateSearching\n> > > >\n> > > > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > > > well be because I'm not a native speaker). How does \"regulating\" or\n> > > > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > > > converge, but a regulation does.\n> > >\n> > > Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> > > Searching I think works fine because you're searching in a numerical\n> > > space for the value that maximizes the score.\n> > >\n> > > >\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > > > + exposure time or the analogue gain, but has not converged to a\n> > > > > > + stable result yet.\n> > > > > > +\n> > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > > > + converged yet.\n> > > > > > +\n> > > > > > + The AEGC algorithm converges once stable values are computed for\n> > > > > > + any of the controls set to be computed in Auto mode.\n> > > >\n> > > > Should that be \"all of the\"?\n> > >\n> > > Oh yeah it should be.\n> > >\n> > > >\n> > > > > > +\n> > > > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > > > + - name: AeStateConverged\n> > > > > > + value: 2\n> > > > > > + description: |\n> > > > > > + The AEGC algorithm has converged.\n> > > > > > +\n> > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > > > + converged to stable value.\n> > > > > > +\n> > > > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > > > + which case the state is moved to AeStateSearching.\n> > > >\n> > > > This sounds like it is based on a random event. What about something\n> > > > like: When the statistics/measurements move too far away from the\n> > > > convergence point the AEGC algo automatically restarts\n> > > > regulation/scanning...\n> > >\n> > > In a way I think it's spontaneous to the user :)\n> > >\n> > > But indeed your description would be more accurate.\n> > >\n> > > >\n> > > > > > +\n> > > > > > # AeMeteringMode needs further attention:\n> > > > > > # - Auto-generate max enum value.\n> > > > > > # - Better handling of custom types.\n> > > > > > @@ -104,6 +173,13 @@ controls:\n> > > > > > The exposure modes specify how the desired total exposure is divided\n> > > > > > between the shutter time and the sensor's analogue gain. They are\n> > > > > > platform specific, and not all exposure modes may be supported.\n> > > > > > +\n> > > > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > > > +\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > +\n> > > > > > enum:\n> > > > > > - name: ExposureNormal\n> > > > > > value: 0\n> > > > > > @@ -124,13 +200,15 @@ controls:\n> > > > > > Specify an Exposure Value (EV) parameter.\n> > > > > >\n> > > > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > > > - enabled.\n> > > > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > > > + are in Auto mode.\n> > > > > >\n> > > > > > By convention EV adjusts the exposure as log2. For example\n> > > > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > > > >\n> > > > > > - \\sa AeEnable\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > + \\sa ExposureTimeMode\n> > > > > >\n> > > > > > - ExposureTime:\n> > > > > > type: int32_t\n> > > > > > @@ -140,17 +218,95 @@ controls:\n> > > > > >\n> > > > > > This value is specified in micro-seconds.\n> > > > > >\n> > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > - AE algorithm.\n> > > > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > > > + ignored and will not be retained.\n> > > > >\n> > > > > I'm not sure about this, it seems like a change of API behaviour which\n> > > > > will require users to be re-educated and application code to be hunted\n> > > > > down and changed. Previously you could just set the exposure time to X\n> > > > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > > > same, I think, with respect to setting the lens position. It's just\n> > > > > more convenient.\n> > > >\n> > > > This is an interesting discussion. I wasn't aware of the old\n> > > > specification. I see that just setting one value and deducing what the\n> > > > user wants is convenient. On the other hand one control changing the\n> > > > value of another control feels quite difficult to explain and in my\n> > > > opinion should be prevented whenever possible. I believe it is easier to\n> > > > educate the user and have an explicit interface than to have hidden\n> > > > magic whenever possible.\n> > >\n> > > Yeah that was my thought process too when I designed this.\n> > \n> > There is one minor annoyance this change will introduce because of the\n> > lack of ordering in the ControlList map.\n> > \n> > Going from, say, auto control into manual will require 2 controls, and\n> > the presence of the AnalogueGainModeManual enum + the AnalogueGain\n> > value to use. Since Controls are not ordered, if you iterate through\n> > the ControlList, you will have to check for the presence of one, then\n> > the other programmatically.\n> > \n> \n> Oh good point, I totally forgot about that. Back to the drawing board...\n\nOk I'm back from the drawing board.\n\nAt the moment I see two easy options/solutions. One is to check each the\nprescence of individual control instead of iterating over the control\nlist, like what rkisp1 does. The other option is to directly check the\nprescence of the enable control when handling the manual control in the\niteration. Something like (using what's in the Raspberry Pi IPA):\n\nfor (auto const &ctrl : controls) {\n\tswitch(ctrl.first) {\n\tcase controls::EXPOSURE_TIME_MODE: {\n\t\texposureEnableFlag_ =\n\t\t\t(*exposureEnable == controls::ExposureTimeModeAuto);\n\t}\n\n\tcase controls::EXPOSURE_TIME: {\n\t\tconst auto &exposureEnable =\n\t\t\tcontrols.get(controls::ExposureTimeMode);\n\t\tif (exposureEnable &&\n\t\t (*exposureEnable == controls::ExposureTimeModeAuto))\n\t\t exposureEnableFlag_ = true;\n\t\t\t\n\t\tif (!exposureEnable && !exposureEnableFlag_)\n\t\t\tcontinue;\n\t}\n\t}\n}\n\nI suppose they are indeed minor annoyances, but I think it's a necessary\nbyproduct of consistency and correctness that the new controls provide\nover what we had previously.\n\n\nPaul\n\n\n> > >\n> > > >\n> > > > >\n> > > > > [There's also the question of how confident you are that pipeline\n> > > > > handlers correctly deal with two controls like this that are\n> > > > > effectively ordered...]\n> > > > >\n> > > > > > +\n> > > > > > + When reported in metadata, this control indicates what exposure time\n> > > > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > > > + whether it came from the AE algorithm or not.\n> > > > > > +\n> > > > > > + \\sa AnalogueGain\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > +\n> > > > > > + - ExposureTimeMode:\n> > > > > > + type: int32_t\n> > > > > > + description: |\n> > > > > > + Controls the source of the exposure time that is applied to the image\n> > > > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > + the value of the ExposureTime control is used.\n> > > > > > +\n> > > > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > > > + is provided by the application, the last value computed by the AE\n> > > > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > + best-effort default value.\n> > > > >\n> > > > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > > > setting AeEnable to zero used to do, except that it can be done\n> > > > > separately for exposure time and gain now.\n> > > > >\n> > > > > > +\n> > > > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > > > + also be supported.\n> > > > > > +\n> > > > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > > > >\n> > > > > That had me scratching my head for a moment! Maybe\n> > > > >\n> > > > > \"At least one of the modes, auto or manual, must be supported by both\n> > > > > ExposureTimeMode and AnalogueGainMode.\"\n> > > > >\n> > > > > Is that clearer? (Also, is it what you meant?)\n> > > > >\n> > > > > > +\n> > > > > > + Flickerless exposure mode transitions\n> > > > > > +\n> > > > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > > > + control of the exposure time should aim to do so by selecting an\n> > > > > > + ExposureTime value as close as possible to the last value computed by\n> > > > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > > > >\n> > > > > Sounds a bit like you _have_ to do this. Maybe\n> > > > >\n> > > > > \"Applications that wish to transition from ..Auto to direct control of\n> > > > > the exposure time without causing extra flicker, should do so by\n> > > > > selecting an ExposureTime value as close as possible to the last value\n> > > > > computed by the auto exposure algorithm.\"\n> > > > >\n> > > > > Don't know if that's better or not.\n> > > > >\n> > > > > > +\n> > > > > > + To select the correct value to use as ExposureTime value, applications\n> > > > > > + should accommodate the natural delay in applying controls caused by the\n> > > > > > + capture pipeline frame depth.\n> > > > > > +\n> > > > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > > > + immediately specify an ExposureTime value in the same request where\n> > > > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > > > + first Request where ExposureTimeMode is reported as\n> > > > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > > > + ExposureTime to populate the control value in the next Request to be\n> > > > > > + queued to the Camera.\n> > > > > > +\n> > > > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > > > + auto exposure use the last value provided by the application as starting\n> > > > > > + point.\n> > > > > > +\n> > > > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > > > +\n> > > > > > + 2. Set ExposureTimeMode to Manual\n> > > > > > +\n> > > > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > > > + set to Manual\n> > > > > > +\n> > > > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > > > + submit it\n> > > > > > +\n> > > > > > + 5. Proceed to run manual exposure time\n> > > > > > +\n> > > > > > + \\sa ExposureTime\n> > > > > > + enum:\n> > > > > > + - name: ExposureTimeModeAuto\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The exposure time will be calculated automatically and set by the\n> > > > > > + AE algorithm.\n> > > > > >\n> > > > > > - \\sa AnalogueGain AeEnable\n> > > > > > + If ExposureTime is set while this mode is active, it will be\n> > > > > > + ignored, and it will also not be retained.\n> > > > > > + - name: ExposureTimeModeManual\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + The exposure time will not be updated by the AE algorithm.\n> > > > > >\n> > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > - better be handled through a separate AE mode control.\n> > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > + exposure value is used until a new value is specified through the\n> > > > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > > > + Manual, the provided ExposureTime is applied immediately.\n> > > > >\n> > > > > Did we say anywhere what happens when you go from manual back to auto?\n> > > > > I'm assuming the adjustments restart from the manual values that you\n> > > > > currently have.\n> > > > >\n> > > > > >\n> > > > > > - AnalogueGain:\n> > > > > > type: float\n> > > > > > @@ -160,17 +316,64 @@ controls:\n> > > > > > The value of the control specifies the gain multiplier applied to all\n> > > > > > colour channels. This value cannot be lower than 1.0.\n> > > > > >\n> > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > - AE algorithm.\n> > > > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > > > + ignored and will not be retained.\n> > > > >\n> > > > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > > > >\n> > > > > > +\n> > > > > > + When reported in metadata, this control indicates what analogue gain\n> > > > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > > > + whether it came from the AEGC algorithm or not.\n> > > > > > +\n> > > > > > + \\sa ExposureTime\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > +\n> > > > > > + - AnalogueGainMode:\n> > > > > > + type: int32_t\n> > > > > > + description: |\n> > > > > > + Controls the source of the analogue gain that is applied to the image\n> > > > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > + the value of the AnalogueGain control is used.\n> > > > > > +\n> > > > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > > > + is provided by the application, the last value computed by the AEGC\n> > > > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > + best-effort default value.\n> > > > > > +\n> > > > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > > > + also be supported.\n> > > > > > +\n> > > > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > > > >\n> > > > > As above!\n> > > > >\n> > > > > Thanks again\n> > > > >\n> > > > > David\n> > > > >\n> > > > > > +\n> > > > > > + The same procedure described for performing flickerless transitions in\n> > > > > > + the ExposureTimeMode control documentation should be applied to\n> > > > > > + analogue gain.\n> > > > > > +\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > + \\sa AnalogueGain\n> > > > > > + enum:\n> > > > > > + - name: AnalogueGainModeAuto\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The analogue gain will be calculated automatically and set by the\n> > > > > > + AEGC algorithm.\n> > > > > >\n> > > > > > - \\sa ExposureTime AeEnable\n> > > > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > > > + ignored, and it will also not be retained.\n> > > > > > + - name: AnalogueGainModeManual\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > > > >\n> > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > - better be handled through a separate AE mode control.\n> > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > + gain value is used until a new value is specified through the\n> > > > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > > > >\n> > > > > > - AeFlickerMode:\n> > > > > > type: int32_t\n> > > > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > > > @@ -77,35 +77,6 @@ controls:\n> > > > > > High quality aberration correction which might reduce the frame\n> > > > > > rate.\n> > > > > >\n> > > > > > - - AeState:\n> > > > > > - type: int32_t\n> > > > > > - description: |\n> > > > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > > > - ANDROID_CONTROL_AE_STATE.\n> > > > > > -\n> > > > > > - Current state of the AE algorithm.\n> > > > > > - enum:\n> > > > > > - - name: AeStateInactive\n> > > > > > - value: 0\n> > > > > > - description: The AE algorithm is inactive.\n> > > > > > - - name: AeStateSearching\n> > > > > > - value: 1\n> > > > > > - description: The AE algorithm has not converged yet.\n> > > > > > - - name: AeStateConverged\n> > > > > > - value: 2\n> > > > > > - description: The AE algorithm has converged.\n> > > > > > - - name: AeStateLocked\n> > > > > > - value: 3\n> > > > > > - description: The AE algorithm is locked.\n> > > > > > - - name: AeStateFlashRequired\n> > > > > > - value: 4\n> > > > > > - description: The AE algorithm would need a flash for good results\n> > > > > > - - name: AeStatePrecapture\n> > > > > > - value: 5\n> > > > > > - description: |\n> > > > > > - The AE algorithm has started a pre-capture metering session.\n> > > > > > - \\sa AePrecaptureTrigger\n> > > > > > -\n> > > > > > - AwbState:\n> > > > > > type: int32_t\n> > > > > > description: |\n> > > > > > --\n> > > > > > 2.39.2\n> > > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 89F42C32DD\n\tfor ;\n\tMon, 18 Nov 2024 13:51:12 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 432B2658D9;\n\tMon, 18 Nov 2024 14:51:12 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A6B5E60532\n\tfor ;\n\tMon, 18 Nov 2024 14:51:10 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2404:7a81:160:2100:fb8e:b30c:9a4a:eed0])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 00D47316;\n\tMon, 18 Nov 2024 14:50:51 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"MN+FujYo\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1731937854;\n\tbh=3zFfbOvsa9twhf6j2O/VJH9y5kdaJKTJ4W//pXBFapE=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=MN+FujYoTuLmmZ5pVolHmtcN78xnCNXE49woHoivYxQtpp9jpgukgshcDdBl08JXc\n\tdFDMfvc41aI3tdk3TfbLXVu92zRyP7IaFuUa3cIROoc+GecudvEhVMMYhN94FCwtF5\n\tyqdzhkBXJirwqITQzVS9jBaKXxi7ZO+sJhzMylqI=","Date":"Mon, 18 Nov 2024 22:51:02 +0900","From":"Paul Elder ","To":"Naushir Patuck ","Cc":"Stefan Klug ,\n\tDavid Plowman ,\n\tlibcamera-devel@lists.libcamera.org,\n\tlaurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com,\n\tJacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32232,"web_url":"https://patchwork.libcamera.org/comment/32232/","msgid":"","date":"2024-11-18T14:29:15","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Paul\n\nI would just like to make sure we're looking after some of our (many)\nless-expert Raspberry Pi users, and don't want to cause code they're\nusing to break with minimal warning or explanation. (These are not\npeople who read libcamera mailing lists!)\n\nFor example, it could be helpful to us to maintain some degree of\nbackward compatibility. Maybe switching to \"manual mode\" automatically\nwould help. As I commented previously, AF lens position works like\nthis, so it would be more consistent.\n\nI understand that folks don't like the \"zero means auto\" value, on the\nother hand I'm not totally persuaded that breaking a single control\ninto two, now with the risk of a race hazard, is so very much better.\nMaybe we could have a switchover period where the old method works but\nissues a warning?\n\nAnother thought would be for us to \"translate\" control lists from the\nolder API to the newer one at runtime. It's hardly a big deal, but\nfeels a teeny weeny bit inelegant!\n\nThanks!\nDavid\n\nOn Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n>\n> On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > Hi Paul,\n> >\n> > Thank you for the patch.\n> >\n> > > Hi Paul\n> > >\n> > > Thanks for posting all this!\n> > >\n> > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > >\n> > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > to model the AEGC algorithm block.\n> > > >\n> > > > The three controls allow applications to select the exposure time and\n> > > > analogue gain computation calculation mode (auto vs manual)\n> > > > independently from one another, while the AeState control reports the\n> > > > global state for the AEGC algorithm.\n> > > >\n> > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > controls, which are momentarily kept not to break compilation of\n> > > > platforms making use of them.\n> > > >\n> > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > Signed-off-by: Paul Elder \n> > > > Signed-off-by: Jacopo Mondi \n> > > >\n> > > > ---\n> > > > Changes in v3:\n> > > > - recovered from 2-year-old bitrot\n> > >\n> > > Indeed, I think I remember discussing all this _years_ ago!!\n> > >\n> > > > ---\n> > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > >\n> > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > @@ -26,6 +26,75 @@ controls:\n> > > >\n> > > > \\sa AeEnable\n> > > >\n> > > > + - AeState:\n> > > > + type: int32_t\n> > > > + description: |\n> > > > + Control to report the AEGC algorithm state.\n> > > > +\n> > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > + values to be applied to the image sensor.\n> > > > +\n> > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > + independently from one another.\n> > > > +\n> > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > + value and describes it as a single computation block which computes\n> > > > + both the exposure time and the analogue gain values.\n> > > > +\n> > > > + When both the exposure time and analogue gain values are configured to\n> > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > +\n> > > > + When at least the exposure time or analogue gain are configured to be\n> > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > + algorithm has converged to stable values for all of the controls set\n> > > > + to be computed in Auto mode.\n> > > > +\n> > > > + \\sa AnalogueGainMode\n> > > > + \\sa ExposureTimeMode\n> > > > +\n> > > > + enum:\n> > > > + - name: AeStateIdle\n> > > > + value: 0\n> > > > + description: |\n> > > > + The AEGC algorithm is inactive.\n> > > > +\n> > > > + This state is returned when both AnalogueGainMode and\n> > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > + actively computing any value.\n> > > > +\n> > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > >\n> > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > autofocus description! But I'm not too bothered.\n> > >\n> > > > + case the AeState control is moved to AeStateSearching.\n> > > > + - name: AeStateSearching\n> >\n> > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > well be because I'm not a native speaker). How does \"regulating\" or\n> > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > converge, but a regulation does.\n>\n> Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> Searching I think works fine because you're searching in a numerical\n> space for the value that maximizes the score.\n>\n> >\n> > > > + value: 1\n> > > > + description: |\n> > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > + exposure time or the analogue gain, but has not converged to a\n> > > > + stable result yet.\n> > > > +\n> > > > + This state is returned if at least one of AnalogueGainMode\n> > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > + converged yet.\n> > > > +\n> > > > + The AEGC algorithm converges once stable values are computed for\n> > > > + any of the controls set to be computed in Auto mode.\n> >\n> > Should that be \"all of the\"?\n>\n> Oh yeah it should be.\n>\n> >\n> > > > +\n> > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > + - name: AeStateConverged\n> > > > + value: 2\n> > > > + description: |\n> > > > + The AEGC algorithm has converged.\n> > > > +\n> > > > + This state is returned if at least one of AnalogueGainMode\n> > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > + converged to stable value.\n> > > > +\n> > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > + which case the state is moved to AeStateSearching.\n> >\n> > This sounds like it is based on a random event. What about something\n> > like: When the statistics/measurements move too far away from the\n> > convergence point the AEGC algo automatically restarts\n> > regulation/scanning...\n>\n> In a way I think it's spontaneous to the user :)\n>\n> But indeed your description would be more accurate.\n>\n> >\n> > > > +\n> > > > # AeMeteringMode needs further attention:\n> > > > # - Auto-generate max enum value.\n> > > > # - Better handling of custom types.\n> > > > @@ -104,6 +173,13 @@ controls:\n> > > > The exposure modes specify how the desired total exposure is divided\n> > > > between the shutter time and the sensor's analogue gain. They are\n> > > > platform specific, and not all exposure modes may be supported.\n> > > > +\n> > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > +\n> > > > + \\sa AnalogueGainMode\n> > > > + \\sa ExposureTimeMode\n> > > > +\n> > > > enum:\n> > > > - name: ExposureNormal\n> > > > value: 0\n> > > > @@ -124,13 +200,15 @@ controls:\n> > > > Specify an Exposure Value (EV) parameter.\n> > > >\n> > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > - enabled.\n> > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > + are in Auto mode.\n> > > >\n> > > > By convention EV adjusts the exposure as log2. For example\n> > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > >\n> > > > - \\sa AeEnable\n> > > > + \\sa AnalogueGainMode\n> > > > + \\sa ExposureTimeMode\n> > > >\n> > > > - ExposureTime:\n> > > > type: int32_t\n> > > > @@ -140,17 +218,95 @@ controls:\n> > > >\n> > > > This value is specified in micro-seconds.\n> > > >\n> > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > - AE algorithm.\n> > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > + ignored and will not be retained.\n> > >\n> > > I'm not sure about this, it seems like a change of API behaviour which\n> > > will require users to be re-educated and application code to be hunted\n> > > down and changed. Previously you could just set the exposure time to X\n> > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > same, I think, with respect to setting the lens position. It's just\n> > > more convenient.\n> >\n> > This is an interesting discussion. I wasn't aware of the old\n> > specification. I see that just setting one value and deducing what the\n> > user wants is convenient. On the other hand one control changing the\n> > value of another control feels quite difficult to explain and in my\n> > opinion should be prevented whenever possible. I believe it is easier to\n> > educate the user and have an explicit interface than to have hidden\n> > magic whenever possible.\n>\n> Yeah that was my thought process too when I designed this.\n>\n>\n> Thanks,\n>\n> Paul\n>\n> >\n> > >\n> > > [There's also the question of how confident you are that pipeline\n> > > handlers correctly deal with two controls like this that are\n> > > effectively ordered...]\n> > >\n> > > > +\n> > > > + When reported in metadata, this control indicates what exposure time\n> > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > + whether it came from the AE algorithm or not.\n> > > > +\n> > > > + \\sa AnalogueGain\n> > > > + \\sa ExposureTimeMode\n> > > > +\n> > > > + - ExposureTimeMode:\n> > > > + type: int32_t\n> > > > + description: |\n> > > > + Controls the source of the exposure time that is applied to the image\n> > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > + the value of the ExposureTime control is used.\n> > > > +\n> > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > + is provided by the application, the last value computed by the AE\n> > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > + or Auto is not supported by the camera), the camera should use a\n> > > > + best-effort default value.\n> > >\n> > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > setting AeEnable to zero used to do, except that it can be done\n> > > separately for exposure time and gain now.\n> > >\n> > > > +\n> > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > + also be supported.\n> > > > +\n> > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > >\n> > > That had me scratching my head for a moment! Maybe\n> > >\n> > > \"At least one of the modes, auto or manual, must be supported by both\n> > > ExposureTimeMode and AnalogueGainMode.\"\n> > >\n> > > Is that clearer? (Also, is it what you meant?)\n> > >\n> > > > +\n> > > > + Flickerless exposure mode transitions\n> > > > +\n> > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > + control of the exposure time should aim to do so by selecting an\n> > > > + ExposureTime value as close as possible to the last value computed by\n> > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > >\n> > > Sounds a bit like you _have_ to do this. Maybe\n> > >\n> > > \"Applications that wish to transition from ..Auto to direct control of\n> > > the exposure time without causing extra flicker, should do so by\n> > > selecting an ExposureTime value as close as possible to the last value\n> > > computed by the auto exposure algorithm.\"\n> > >\n> > > Don't know if that's better or not.\n> > >\n> > > > +\n> > > > + To select the correct value to use as ExposureTime value, applications\n> > > > + should accommodate the natural delay in applying controls caused by the\n> > > > + capture pipeline frame depth.\n> > > > +\n> > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > + immediately specify an ExposureTime value in the same request where\n> > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > + first Request where ExposureTimeMode is reported as\n> > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > + ExposureTime to populate the control value in the next Request to be\n> > > > + queued to the Camera.\n> > > > +\n> > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > + auto exposure use the last value provided by the application as starting\n> > > > + point.\n> > > > +\n> > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > +\n> > > > + 2. Set ExposureTimeMode to Manual\n> > > > +\n> > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > + set to Manual\n> > > > +\n> > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > + submit it\n> > > > +\n> > > > + 5. Proceed to run manual exposure time\n> > > > +\n> > > > + \\sa ExposureTime\n> > > > + enum:\n> > > > + - name: ExposureTimeModeAuto\n> > > > + value: 0\n> > > > + description: |\n> > > > + The exposure time will be calculated automatically and set by the\n> > > > + AE algorithm.\n> > > >\n> > > > - \\sa AnalogueGain AeEnable\n> > > > + If ExposureTime is set while this mode is active, it will be\n> > > > + ignored, and it will also not be retained.\n> > > > + - name: ExposureTimeModeManual\n> > > > + value: 1\n> > > > + description: |\n> > > > + The exposure time will not be updated by the AE algorithm.\n> > > >\n> > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > - value for this control. Consider interactions with other AE features,\n> > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > - control of which features should be automatically adjusted shouldn't\n> > > > - better be handled through a separate AE mode control.\n> > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > + exposure value is used until a new value is specified through the\n> > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > + Manual, the provided ExposureTime is applied immediately.\n> > >\n> > > Did we say anywhere what happens when you go from manual back to auto?\n> > > I'm assuming the adjustments restart from the manual values that you\n> > > currently have.\n> > >\n> > > >\n> > > > - AnalogueGain:\n> > > > type: float\n> > > > @@ -160,17 +316,64 @@ controls:\n> > > > The value of the control specifies the gain multiplier applied to all\n> > > > colour channels. This value cannot be lower than 1.0.\n> > > >\n> > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > - AE algorithm.\n> > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > + ignored and will not be retained.\n> > >\n> > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > >\n> > > > +\n> > > > + When reported in metadata, this control indicates what analogue gain\n> > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > + whether it came from the AEGC algorithm or not.\n> > > > +\n> > > > + \\sa ExposureTime\n> > > > + \\sa AnalogueGainMode\n> > > > +\n> > > > + - AnalogueGainMode:\n> > > > + type: int32_t\n> > > > + description: |\n> > > > + Controls the source of the analogue gain that is applied to the image\n> > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > + the value of the AnalogueGain control is used.\n> > > > +\n> > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > + is provided by the application, the last value computed by the AEGC\n> > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > + or Auto is not supported by the camera), the camera should use a\n> > > > + best-effort default value.\n> > > > +\n> > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > + also be supported.\n> > > > +\n> > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > >\n> > > As above!\n> > >\n> > > Thanks again\n> > >\n> > > David\n> > >\n> > > > +\n> > > > + The same procedure described for performing flickerless transitions in\n> > > > + the ExposureTimeMode control documentation should be applied to\n> > > > + analogue gain.\n> > > > +\n> > > > + \\sa ExposureTimeMode\n> > > > + \\sa AnalogueGain\n> > > > + enum:\n> > > > + - name: AnalogueGainModeAuto\n> > > > + value: 0\n> > > > + description: |\n> > > > + The analogue gain will be calculated automatically and set by the\n> > > > + AEGC algorithm.\n> > > >\n> > > > - \\sa ExposureTime AeEnable\n> > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > + ignored, and it will also not be retained.\n> > > > + - name: AnalogueGainModeManual\n> > > > + value: 1\n> > > > + description: |\n> > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > >\n> > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > - value for this control. Consider interactions with other AE features,\n> > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > - control of which features should be automatically adjusted shouldn't\n> > > > - better be handled through a separate AE mode control.\n> > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > + gain value is used until a new value is specified through the\n> > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > >\n> > > > - AeFlickerMode:\n> > > > type: int32_t\n> > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > @@ -77,35 +77,6 @@ controls:\n> > > > High quality aberration correction which might reduce the frame\n> > > > rate.\n> > > >\n> > > > - - AeState:\n> > > > - type: int32_t\n> > > > - description: |\n> > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > - ANDROID_CONTROL_AE_STATE.\n> > > > -\n> > > > - Current state of the AE algorithm.\n> > > > - enum:\n> > > > - - name: AeStateInactive\n> > > > - value: 0\n> > > > - description: The AE algorithm is inactive.\n> > > > - - name: AeStateSearching\n> > > > - value: 1\n> > > > - description: The AE algorithm has not converged yet.\n> > > > - - name: AeStateConverged\n> > > > - value: 2\n> > > > - description: The AE algorithm has converged.\n> > > > - - name: AeStateLocked\n> > > > - value: 3\n> > > > - description: The AE algorithm is locked.\n> > > > - - name: AeStateFlashRequired\n> > > > - value: 4\n> > > > - description: The AE algorithm would need a flash for good results\n> > > > - - name: AeStatePrecapture\n> > > > - value: 5\n> > > > - description: |\n> > > > - The AE algorithm has started a pre-capture metering session.\n> > > > - \\sa AePrecaptureTrigger\n> > > > -\n> > > > - AwbState:\n> > > > type: int32_t\n> > > > description: |\n> > > > --\n> > > > 2.39.2\n> > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 6C297C32DD\n\tfor ;\n\tMon, 18 Nov 2024 14:29:31 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 36942658DA;\n\tMon, 18 Nov 2024 15:29:30 +0100 (CET)","from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com\n\t[IPv6:2607:f8b0:4864:20::f2a])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EBE8F60532\n\tfor ;\n\tMon, 18 Nov 2024 15:29:27 +0100 (CET)","by mail-qv1-xf2a.google.com with SMTP id\n\t6a1803df08f44-6d41864d745so8768256d6.2\n\tfor ;\n\tMon, 18 Nov 2024 06:29:27 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"PPxMMbod\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1731940167; x=1732544967;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=DxK7tzKgbb+gsVjuL3P5kfRR1yEUE2JtDCNX6SskFoo=;\n\tb=PPxMMbodk1NaSc6xERvY+ogBxAEvTznA8nR9A+DoBtPgXLPnxYonkWEvbRwUvkU0yh\n\tidXYqq1U8crVqm0ZtOZHwwKOCfklgK8JcC9TRoc84RPrg25x9o7MwfLYiDiZhjAQA5N+\n\tqrIBm5Il/gXPmpwSMRKuYwDi/lDkq1/x7TfnLjnUJC/L7DZNs8U81JIBvfoLXI0uC9vh\n\t10mlqztVItpzUApQI54NKsDTTRYXspWBs8XLEABBBpVs1jdLFnNJX4J1B72tz+XDbdRr\n\tpNArzqaJduVQESXYegVl7vQJI1xCsJX4FHqtnd2+VnEL2VLhmpg3eoAHal1SmpJZTlRq\n\tSltQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1731940167; x=1732544967;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=DxK7tzKgbb+gsVjuL3P5kfRR1yEUE2JtDCNX6SskFoo=;\n\tb=i/AAh31IR8b2n672xZo2cjY3JR64nDUf5Ws/m+q21XZsXAby8So9SBG+wyRnXOv78m\n\tm4AMqmF20lCDs1xi2JspPdA7C5snyB1pMLNaYni/UiyHNzWN27OYsEIlmUIbqMVHYiS5\n\tPeAce3KSrZ8H7riOLHWFOfktlXXxz9Y6H0S1bVhPLgtOdZzSXevLpYMugXMWU4CYKS6i\n\teRwbOdIfdX9MwJsatM0h4aI0dbVwOGCveHy+qKQboQ4z4yon6TRDM7GYImxbZQ37Ho8U\n\tPmwaQnHJiBbIKIlmM6ZnBtHwzcdvGgckrmZpFn13Mdo8+61CVNG4yRmdyEBwWhRgDpkc\n\tOLBw==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCX6GtRuVw4GW+/RFaJQzadcFRP5e5LUfui279gaY98w+xyNUaQr16MHfTV5jw0PThMM32mELg4PlecAbwijRUQ=@lists.libcamera.org","X-Gm-Message-State":"AOJu0Yy+JXageGFVTELPvE/GEgx9/AGJc/Mfz5pS7d79oWQbMIVn6kZQ\n\twEUn/YuHtRlbP51cBkDqbu0jN5LuHKkSdbLIqf4dH6dQel/zsKGSqg1cUXwTul7UOnoMlXsKsMv\n\tgOiet0hS53W0XnvpqvQ9VWigf7vtsk+9kN2k1sxmgIoXlmifn","X-Google-Smtp-Source":"AGHT+IEukFHTpL6XdS0CIHlCDbO1yZtSGqgRdBvqLMlgsOuDUfHldh1YESun4u5SzlipUcAPskMKfSrfTUsMRbAohu8=","X-Received":"by 2002:a05:6214:5342:b0:6cb:ee7b:7ac4 with SMTP id\n\t6a1803df08f44-6d3fb7b0466mr163604296d6.3.1731940166632;\n\tMon, 18 Nov 2024 06:29:26 -0800 (PST)","MIME-Version":"1.0","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t","In-Reply-To":"","From":"David Plowman ","Date":"Mon, 18 Nov 2024 14:29:15 +0000","Message-ID":"","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","To":"Paul Elder ","Cc":"Stefan Klug ,\n\tlibcamera-devel@lists.libcamera.org, \n\tlaurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com, \n\tnaush@raspberrypi.com, Jacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32234,"web_url":"https://patchwork.libcamera.org/comment/32234/","msgid":"","date":"2024-11-18T14:43:45","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi David\n\nOn Mon, Nov 18, 2024 at 02:29:15PM +0000, David Plowman wrote:\n> Hi Paul\n>\n> I would just like to make sure we're looking after some of our (many)\n> less-expert Raspberry Pi users, and don't want to cause code they're\n> using to break with minimal warning or explanation. (These are not\n> people who read libcamera mailing lists!)\n>\n\nOne clarification here: are you concerned about pi users that has\ncustom applications interfacing directly with libcamera (either\nthrough the C++ or Python APIs) ?\n\n> For example, it could be helpful to us to maintain some degree of\n> backward compatibility. Maybe switching to \"manual mode\" automatically\n> would help. As I commented previously, AF lens position works like\n> this, so it would be more consistent.\n>\n> I understand that folks don't like the \"zero means auto\" value, on the\n> other hand I'm not totally persuaded that breaking a single control\n> into two, now with the risk of a race hazard, is so very much better.\n> Maybe we could have a switchover period where the old method works but\n> issues a warning?\n>\n> Another thought would be for us to \"translate\" control lists from the\n> older API to the newer one at runtime. It's hardly a big deal, but\n> feels a teeny weeny bit inelegant!\n>\n\nBecause this seems to suggest you could handle the translation, I\nthough you were considering doing that in your applications, but\nnow I wonder if you're suggesting to do that in the IPA\n\nThanks\n j\n\n> Thanks!\n> David\n>\n> On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n> >\n> > On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > > Hi Paul,\n> > >\n> > > Thank you for the patch.\n> > >\n> > > > Hi Paul\n> > > >\n> > > > Thanks for posting all this!\n> > > >\n> > > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > > >\n> > > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > > to model the AEGC algorithm block.\n> > > > >\n> > > > > The three controls allow applications to select the exposure time and\n> > > > > analogue gain computation calculation mode (auto vs manual)\n> > > > > independently from one another, while the AeState control reports the\n> > > > > global state for the AEGC algorithm.\n> > > > >\n> > > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > > controls, which are momentarily kept not to break compilation of\n> > > > > platforms making use of them.\n> > > > >\n> > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > > Signed-off-by: Paul Elder \n> > > > > Signed-off-by: Jacopo Mondi \n> > > > >\n> > > > > ---\n> > > > > Changes in v3:\n> > > > > - recovered from 2-year-old bitrot\n> > > >\n> > > > Indeed, I think I remember discussing all this _years_ ago!!\n> > > >\n> > > > > ---\n> > > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > > >\n> > > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > > @@ -26,6 +26,75 @@ controls:\n> > > > >\n> > > > > \\sa AeEnable\n> > > > >\n> > > > > + - AeState:\n> > > > > + type: int32_t\n> > > > > + description: |\n> > > > > + Control to report the AEGC algorithm state.\n> > > > > +\n> > > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > > + values to be applied to the image sensor.\n> > > > > +\n> > > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > > + independently from one another.\n> > > > > +\n> > > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > > + value and describes it as a single computation block which computes\n> > > > > + both the exposure time and the analogue gain values.\n> > > > > +\n> > > > > + When both the exposure time and analogue gain values are configured to\n> > > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > > +\n> > > > > + When at least the exposure time or analogue gain are configured to be\n> > > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > > + algorithm has converged to stable values for all of the controls set\n> > > > > + to be computed in Auto mode.\n> > > > > +\n> > > > > + \\sa AnalogueGainMode\n> > > > > + \\sa ExposureTimeMode\n> > > > > +\n> > > > > + enum:\n> > > > > + - name: AeStateIdle\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The AEGC algorithm is inactive.\n> > > > > +\n> > > > > + This state is returned when both AnalogueGainMode and\n> > > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > > + actively computing any value.\n> > > > > +\n> > > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > > >\n> > > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > > autofocus description! But I'm not too bothered.\n> > > >\n> > > > > + case the AeState control is moved to AeStateSearching.\n> > > > > + - name: AeStateSearching\n> > >\n> > > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > > well be because I'm not a native speaker). How does \"regulating\" or\n> > > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > > converge, but a regulation does.\n> >\n> > Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> > Searching I think works fine because you're searching in a numerical\n> > space for the value that maximizes the score.\n> >\n> > >\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > > + exposure time or the analogue gain, but has not converged to a\n> > > > > + stable result yet.\n> > > > > +\n> > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > > + converged yet.\n> > > > > +\n> > > > > + The AEGC algorithm converges once stable values are computed for\n> > > > > + any of the controls set to be computed in Auto mode.\n> > >\n> > > Should that be \"all of the\"?\n> >\n> > Oh yeah it should be.\n> >\n> > >\n> > > > > +\n> > > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > > + - name: AeStateConverged\n> > > > > + value: 2\n> > > > > + description: |\n> > > > > + The AEGC algorithm has converged.\n> > > > > +\n> > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > > + converged to stable value.\n> > > > > +\n> > > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > > + which case the state is moved to AeStateSearching.\n> > >\n> > > This sounds like it is based on a random event. What about something\n> > > like: When the statistics/measurements move too far away from the\n> > > convergence point the AEGC algo automatically restarts\n> > > regulation/scanning...\n> >\n> > In a way I think it's spontaneous to the user :)\n> >\n> > But indeed your description would be more accurate.\n> >\n> > >\n> > > > > +\n> > > > > # AeMeteringMode needs further attention:\n> > > > > # - Auto-generate max enum value.\n> > > > > # - Better handling of custom types.\n> > > > > @@ -104,6 +173,13 @@ controls:\n> > > > > The exposure modes specify how the desired total exposure is divided\n> > > > > between the shutter time and the sensor's analogue gain. They are\n> > > > > platform specific, and not all exposure modes may be supported.\n> > > > > +\n> > > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > > +\n> > > > > + \\sa AnalogueGainMode\n> > > > > + \\sa ExposureTimeMode\n> > > > > +\n> > > > > enum:\n> > > > > - name: ExposureNormal\n> > > > > value: 0\n> > > > > @@ -124,13 +200,15 @@ controls:\n> > > > > Specify an Exposure Value (EV) parameter.\n> > > > >\n> > > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > > - enabled.\n> > > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > > + are in Auto mode.\n> > > > >\n> > > > > By convention EV adjusts the exposure as log2. For example\n> > > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > > >\n> > > > > - \\sa AeEnable\n> > > > > + \\sa AnalogueGainMode\n> > > > > + \\sa ExposureTimeMode\n> > > > >\n> > > > > - ExposureTime:\n> > > > > type: int32_t\n> > > > > @@ -140,17 +218,95 @@ controls:\n> > > > >\n> > > > > This value is specified in micro-seconds.\n> > > > >\n> > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > - AE algorithm.\n> > > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > > + ignored and will not be retained.\n> > > >\n> > > > I'm not sure about this, it seems like a change of API behaviour which\n> > > > will require users to be re-educated and application code to be hunted\n> > > > down and changed. Previously you could just set the exposure time to X\n> > > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > > same, I think, with respect to setting the lens position. It's just\n> > > > more convenient.\n> > >\n> > > This is an interesting discussion. I wasn't aware of the old\n> > > specification. I see that just setting one value and deducing what the\n> > > user wants is convenient. On the other hand one control changing the\n> > > value of another control feels quite difficult to explain and in my\n> > > opinion should be prevented whenever possible. I believe it is easier to\n> > > educate the user and have an explicit interface than to have hidden\n> > > magic whenever possible.\n> >\n> > Yeah that was my thought process too when I designed this.\n> >\n> >\n> > Thanks,\n> >\n> > Paul\n> >\n> > >\n> > > >\n> > > > [There's also the question of how confident you are that pipeline\n> > > > handlers correctly deal with two controls like this that are\n> > > > effectively ordered...]\n> > > >\n> > > > > +\n> > > > > + When reported in metadata, this control indicates what exposure time\n> > > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > > + whether it came from the AE algorithm or not.\n> > > > > +\n> > > > > + \\sa AnalogueGain\n> > > > > + \\sa ExposureTimeMode\n> > > > > +\n> > > > > + - ExposureTimeMode:\n> > > > > + type: int32_t\n> > > > > + description: |\n> > > > > + Controls the source of the exposure time that is applied to the image\n> > > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > + the value of the ExposureTime control is used.\n> > > > > +\n> > > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > > + is provided by the application, the last value computed by the AE\n> > > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > + best-effort default value.\n> > > >\n> > > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > > setting AeEnable to zero used to do, except that it can be done\n> > > > separately for exposure time and gain now.\n> > > >\n> > > > > +\n> > > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > > + also be supported.\n> > > > > +\n> > > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > > >\n> > > > That had me scratching my head for a moment! Maybe\n> > > >\n> > > > \"At least one of the modes, auto or manual, must be supported by both\n> > > > ExposureTimeMode and AnalogueGainMode.\"\n> > > >\n> > > > Is that clearer? (Also, is it what you meant?)\n> > > >\n> > > > > +\n> > > > > + Flickerless exposure mode transitions\n> > > > > +\n> > > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > > + control of the exposure time should aim to do so by selecting an\n> > > > > + ExposureTime value as close as possible to the last value computed by\n> > > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > > >\n> > > > Sounds a bit like you _have_ to do this. Maybe\n> > > >\n> > > > \"Applications that wish to transition from ..Auto to direct control of\n> > > > the exposure time without causing extra flicker, should do so by\n> > > > selecting an ExposureTime value as close as possible to the last value\n> > > > computed by the auto exposure algorithm.\"\n> > > >\n> > > > Don't know if that's better or not.\n> > > >\n> > > > > +\n> > > > > + To select the correct value to use as ExposureTime value, applications\n> > > > > + should accommodate the natural delay in applying controls caused by the\n> > > > > + capture pipeline frame depth.\n> > > > > +\n> > > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > > + immediately specify an ExposureTime value in the same request where\n> > > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > > + first Request where ExposureTimeMode is reported as\n> > > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > > + ExposureTime to populate the control value in the next Request to be\n> > > > > + queued to the Camera.\n> > > > > +\n> > > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > > + auto exposure use the last value provided by the application as starting\n> > > > > + point.\n> > > > > +\n> > > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > > +\n> > > > > + 2. Set ExposureTimeMode to Manual\n> > > > > +\n> > > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > > + set to Manual\n> > > > > +\n> > > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > > + submit it\n> > > > > +\n> > > > > + 5. Proceed to run manual exposure time\n> > > > > +\n> > > > > + \\sa ExposureTime\n> > > > > + enum:\n> > > > > + - name: ExposureTimeModeAuto\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The exposure time will be calculated automatically and set by the\n> > > > > + AE algorithm.\n> > > > >\n> > > > > - \\sa AnalogueGain AeEnable\n> > > > > + If ExposureTime is set while this mode is active, it will be\n> > > > > + ignored, and it will also not be retained.\n> > > > > + - name: ExposureTimeModeManual\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The exposure time will not be updated by the AE algorithm.\n> > > > >\n> > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > - value for this control. Consider interactions with other AE features,\n> > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > - better be handled through a separate AE mode control.\n> > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > + exposure value is used until a new value is specified through the\n> > > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > > + Manual, the provided ExposureTime is applied immediately.\n> > > >\n> > > > Did we say anywhere what happens when you go from manual back to auto?\n> > > > I'm assuming the adjustments restart from the manual values that you\n> > > > currently have.\n> > > >\n> > > > >\n> > > > > - AnalogueGain:\n> > > > > type: float\n> > > > > @@ -160,17 +316,64 @@ controls:\n> > > > > The value of the control specifies the gain multiplier applied to all\n> > > > > colour channels. This value cannot be lower than 1.0.\n> > > > >\n> > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > - AE algorithm.\n> > > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > > + ignored and will not be retained.\n> > > >\n> > > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > > >\n> > > > > +\n> > > > > + When reported in metadata, this control indicates what analogue gain\n> > > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > > + whether it came from the AEGC algorithm or not.\n> > > > > +\n> > > > > + \\sa ExposureTime\n> > > > > + \\sa AnalogueGainMode\n> > > > > +\n> > > > > + - AnalogueGainMode:\n> > > > > + type: int32_t\n> > > > > + description: |\n> > > > > + Controls the source of the analogue gain that is applied to the image\n> > > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > + the value of the AnalogueGain control is used.\n> > > > > +\n> > > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > > + is provided by the application, the last value computed by the AEGC\n> > > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > + best-effort default value.\n> > > > > +\n> > > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > > + also be supported.\n> > > > > +\n> > > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > > >\n> > > > As above!\n> > > >\n> > > > Thanks again\n> > > >\n> > > > David\n> > > >\n> > > > > +\n> > > > > + The same procedure described for performing flickerless transitions in\n> > > > > + the ExposureTimeMode control documentation should be applied to\n> > > > > + analogue gain.\n> > > > > +\n> > > > > + \\sa ExposureTimeMode\n> > > > > + \\sa AnalogueGain\n> > > > > + enum:\n> > > > > + - name: AnalogueGainModeAuto\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The analogue gain will be calculated automatically and set by the\n> > > > > + AEGC algorithm.\n> > > > >\n> > > > > - \\sa ExposureTime AeEnable\n> > > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > > + ignored, and it will also not be retained.\n> > > > > + - name: AnalogueGainModeManual\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > > >\n> > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > - value for this control. Consider interactions with other AE features,\n> > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > - better be handled through a separate AE mode control.\n> > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > + gain value is used until a new value is specified through the\n> > > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > > >\n> > > > > - AeFlickerMode:\n> > > > > type: int32_t\n> > > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > > @@ -77,35 +77,6 @@ controls:\n> > > > > High quality aberration correction which might reduce the frame\n> > > > > rate.\n> > > > >\n> > > > > - - AeState:\n> > > > > - type: int32_t\n> > > > > - description: |\n> > > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > > - ANDROID_CONTROL_AE_STATE.\n> > > > > -\n> > > > > - Current state of the AE algorithm.\n> > > > > - enum:\n> > > > > - - name: AeStateInactive\n> > > > > - value: 0\n> > > > > - description: The AE algorithm is inactive.\n> > > > > - - name: AeStateSearching\n> > > > > - value: 1\n> > > > > - description: The AE algorithm has not converged yet.\n> > > > > - - name: AeStateConverged\n> > > > > - value: 2\n> > > > > - description: The AE algorithm has converged.\n> > > > > - - name: AeStateLocked\n> > > > > - value: 3\n> > > > > - description: The AE algorithm is locked.\n> > > > > - - name: AeStateFlashRequired\n> > > > > - value: 4\n> > > > > - description: The AE algorithm would need a flash for good results\n> > > > > - - name: AeStatePrecapture\n> > > > > - value: 5\n> > > > > - description: |\n> > > > > - The AE algorithm has started a pre-capture metering session.\n> > > > > - \\sa AePrecaptureTrigger\n> > > > > -\n> > > > > - AwbState:\n> > > > > type: int32_t\n> > > > > description: |\n> > > > > --\n> > > > > 2.39.2\n> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8EDB2C32DD\n\tfor ;\n\tMon, 18 Nov 2024 14:43:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A6D16658E1;\n\tMon, 18 Nov 2024 15:43:51 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 35375658D9\n\tfor ;\n\tMon, 18 Nov 2024 15:43:50 +0100 (CET)","from ideasonboard.com (net-93-150-224-74.cust.vodafonedsl.it\n\t[93.150.224.74])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A890AA57;\n\tMon, 18 Nov 2024 15:43:32 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"pfEf1z3T\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1731941013;\n\tbh=RP9XN8ZTxALIzaTtXwHDNnwZwUJroqVnOiXPkJSy9Hw=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=pfEf1z3Td/Rny9vFRpqCNz7YWqF6Fgxy4jf7df5soO+SZSy0apf3ZEtPSY3PzsXpM\n\tUYRbE8kXNVCLr4Tpoj6IHlOipolcmhqMSGq8bXIvwmpApqLwpdp2eJ1ijMmJ0PagCg\n\tcHIDtXoWOBRdGXiw9jtNq/mQSEyGRhwFY+7LTuzU=","Date":"Mon, 18 Nov 2024 15:43:45 +0100","From":"Jacopo Mondi ","To":"David Plowman ","Cc":"Paul Elder , \n\tStefan Klug ,\n\tlibcamera-devel@lists.libcamera.org, \n\tlaurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com,\n\tnaush@raspberrypi.com, Jacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32283,"web_url":"https://patchwork.libcamera.org/comment/32283/","msgid":"","date":"2024-11-19T13:44:13","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Jacopo\n\nOn Mon, 18 Nov 2024 at 14:43, Jacopo Mondi\n wrote:\n>\n> Hi David\n>\n> On Mon, Nov 18, 2024 at 02:29:15PM +0000, David Plowman wrote:\n> > Hi Paul\n> >\n> > I would just like to make sure we're looking after some of our (many)\n> > less-expert Raspberry Pi users, and don't want to cause code they're\n> > using to break with minimal warning or explanation. (These are not\n> > people who read libcamera mailing lists!)\n> >\n>\n> One clarification here: are you concerned about pi users that has\n> custom applications interfacing directly with libcamera (either\n> through the C++ or Python APIs) ?\n\nI'm not particularly worried about anyone using C++. It's the Python\nusers, who may be using simple scripts they've written or been given,\nwho could get confused and end up clogging the Raspberry Pi forums!\n\n>\n> > For example, it could be helpful to us to maintain some degree of\n> > backward compatibility. Maybe switching to \"manual mode\" automatically\n> > would help. As I commented previously, AF lens position works like\n> > this, so it would be more consistent.\n> >\n> > I understand that folks don't like the \"zero means auto\" value, on the\n> > other hand I'm not totally persuaded that breaking a single control\n> > into two, now with the risk of a race hazard, is so very much better.\n> > Maybe we could have a switchover period where the old method works but\n> > issues a warning?\n> >\n> > Another thought would be for us to \"translate\" control lists from the\n> > older API to the newer one at runtime. It's hardly a big deal, but\n> > feels a teeny weeny bit inelegant!\n> >\n>\n> Because this seems to suggest you could handle the translation, I\n> though you were considering doing that in your applications, but\n> now I wonder if you're suggesting to do that in the IPA\n\nYes, I can handle the translation for Python users easily enough. It's\nnot lovely, but I think that's probably the least bad alternative for\nus.\n\nSorry if I've been sounding a bit grumpy about this!!\n\nDavid\n\n>\n> Thanks\n> j\n>\n> > Thanks!\n> > David\n> >\n> > On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n> > >\n> > > On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > > > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > > > Hi Paul,\n> > > >\n> > > > Thank you for the patch.\n> > > >\n> > > > > Hi Paul\n> > > > >\n> > > > > Thanks for posting all this!\n> > > > >\n> > > > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > > > >\n> > > > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > > > to model the AEGC algorithm block.\n> > > > > >\n> > > > > > The three controls allow applications to select the exposure time and\n> > > > > > analogue gain computation calculation mode (auto vs manual)\n> > > > > > independently from one another, while the AeState control reports the\n> > > > > > global state for the AEGC algorithm.\n> > > > > >\n> > > > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > > > controls, which are momentarily kept not to break compilation of\n> > > > > > platforms making use of them.\n> > > > > >\n> > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > > > Signed-off-by: Paul Elder \n> > > > > > Signed-off-by: Jacopo Mondi \n> > > > > >\n> > > > > > ---\n> > > > > > Changes in v3:\n> > > > > > - recovered from 2-year-old bitrot\n> > > > >\n> > > > > Indeed, I think I remember discussing all this _years_ ago!!\n> > > > >\n> > > > > > ---\n> > > > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > > > >\n> > > > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > > > @@ -26,6 +26,75 @@ controls:\n> > > > > >\n> > > > > > \\sa AeEnable\n> > > > > >\n> > > > > > + - AeState:\n> > > > > > + type: int32_t\n> > > > > > + description: |\n> > > > > > + Control to report the AEGC algorithm state.\n> > > > > > +\n> > > > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > > > + values to be applied to the image sensor.\n> > > > > > +\n> > > > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > > > + independently from one another.\n> > > > > > +\n> > > > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > > > + value and describes it as a single computation block which computes\n> > > > > > + both the exposure time and the analogue gain values.\n> > > > > > +\n> > > > > > + When both the exposure time and analogue gain values are configured to\n> > > > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > > > +\n> > > > > > + When at least the exposure time or analogue gain are configured to be\n> > > > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > > > + algorithm has converged to stable values for all of the controls set\n> > > > > > + to be computed in Auto mode.\n> > > > > > +\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > +\n> > > > > > + enum:\n> > > > > > + - name: AeStateIdle\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The AEGC algorithm is inactive.\n> > > > > > +\n> > > > > > + This state is returned when both AnalogueGainMode and\n> > > > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > > > + actively computing any value.\n> > > > > > +\n> > > > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > > > >\n> > > > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > > > autofocus description! But I'm not too bothered.\n> > > > >\n> > > > > > + case the AeState control is moved to AeStateSearching.\n> > > > > > + - name: AeStateSearching\n> > > >\n> > > > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > > > well be because I'm not a native speaker). How does \"regulating\" or\n> > > > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > > > converge, but a regulation does.\n> > >\n> > > Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> > > Searching I think works fine because you're searching in a numerical\n> > > space for the value that maximizes the score.\n> > >\n> > > >\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > > > + exposure time or the analogue gain, but has not converged to a\n> > > > > > + stable result yet.\n> > > > > > +\n> > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > > > + converged yet.\n> > > > > > +\n> > > > > > + The AEGC algorithm converges once stable values are computed for\n> > > > > > + any of the controls set to be computed in Auto mode.\n> > > >\n> > > > Should that be \"all of the\"?\n> > >\n> > > Oh yeah it should be.\n> > >\n> > > >\n> > > > > > +\n> > > > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > > > + - name: AeStateConverged\n> > > > > > + value: 2\n> > > > > > + description: |\n> > > > > > + The AEGC algorithm has converged.\n> > > > > > +\n> > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > > > + converged to stable value.\n> > > > > > +\n> > > > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > > > + which case the state is moved to AeStateSearching.\n> > > >\n> > > > This sounds like it is based on a random event. What about something\n> > > > like: When the statistics/measurements move too far away from the\n> > > > convergence point the AEGC algo automatically restarts\n> > > > regulation/scanning...\n> > >\n> > > In a way I think it's spontaneous to the user :)\n> > >\n> > > But indeed your description would be more accurate.\n> > >\n> > > >\n> > > > > > +\n> > > > > > # AeMeteringMode needs further attention:\n> > > > > > # - Auto-generate max enum value.\n> > > > > > # - Better handling of custom types.\n> > > > > > @@ -104,6 +173,13 @@ controls:\n> > > > > > The exposure modes specify how the desired total exposure is divided\n> > > > > > between the shutter time and the sensor's analogue gain. They are\n> > > > > > platform specific, and not all exposure modes may be supported.\n> > > > > > +\n> > > > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > > > +\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > +\n> > > > > > enum:\n> > > > > > - name: ExposureNormal\n> > > > > > value: 0\n> > > > > > @@ -124,13 +200,15 @@ controls:\n> > > > > > Specify an Exposure Value (EV) parameter.\n> > > > > >\n> > > > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > > > - enabled.\n> > > > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > > > + are in Auto mode.\n> > > > > >\n> > > > > > By convention EV adjusts the exposure as log2. For example\n> > > > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > > > >\n> > > > > > - \\sa AeEnable\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > + \\sa ExposureTimeMode\n> > > > > >\n> > > > > > - ExposureTime:\n> > > > > > type: int32_t\n> > > > > > @@ -140,17 +218,95 @@ controls:\n> > > > > >\n> > > > > > This value is specified in micro-seconds.\n> > > > > >\n> > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > - AE algorithm.\n> > > > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > > > + ignored and will not be retained.\n> > > > >\n> > > > > I'm not sure about this, it seems like a change of API behaviour which\n> > > > > will require users to be re-educated and application code to be hunted\n> > > > > down and changed. Previously you could just set the exposure time to X\n> > > > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > > > same, I think, with respect to setting the lens position. It's just\n> > > > > more convenient.\n> > > >\n> > > > This is an interesting discussion. I wasn't aware of the old\n> > > > specification. I see that just setting one value and deducing what the\n> > > > user wants is convenient. On the other hand one control changing the\n> > > > value of another control feels quite difficult to explain and in my\n> > > > opinion should be prevented whenever possible. I believe it is easier to\n> > > > educate the user and have an explicit interface than to have hidden\n> > > > magic whenever possible.\n> > >\n> > > Yeah that was my thought process too when I designed this.\n> > >\n> > >\n> > > Thanks,\n> > >\n> > > Paul\n> > >\n> > > >\n> > > > >\n> > > > > [There's also the question of how confident you are that pipeline\n> > > > > handlers correctly deal with two controls like this that are\n> > > > > effectively ordered...]\n> > > > >\n> > > > > > +\n> > > > > > + When reported in metadata, this control indicates what exposure time\n> > > > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > > > + whether it came from the AE algorithm or not.\n> > > > > > +\n> > > > > > + \\sa AnalogueGain\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > +\n> > > > > > + - ExposureTimeMode:\n> > > > > > + type: int32_t\n> > > > > > + description: |\n> > > > > > + Controls the source of the exposure time that is applied to the image\n> > > > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > + the value of the ExposureTime control is used.\n> > > > > > +\n> > > > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > > > + is provided by the application, the last value computed by the AE\n> > > > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > + best-effort default value.\n> > > > >\n> > > > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > > > setting AeEnable to zero used to do, except that it can be done\n> > > > > separately for exposure time and gain now.\n> > > > >\n> > > > > > +\n> > > > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > > > + also be supported.\n> > > > > > +\n> > > > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > > > >\n> > > > > That had me scratching my head for a moment! Maybe\n> > > > >\n> > > > > \"At least one of the modes, auto or manual, must be supported by both\n> > > > > ExposureTimeMode and AnalogueGainMode.\"\n> > > > >\n> > > > > Is that clearer? (Also, is it what you meant?)\n> > > > >\n> > > > > > +\n> > > > > > + Flickerless exposure mode transitions\n> > > > > > +\n> > > > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > > > + control of the exposure time should aim to do so by selecting an\n> > > > > > + ExposureTime value as close as possible to the last value computed by\n> > > > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > > > >\n> > > > > Sounds a bit like you _have_ to do this. Maybe\n> > > > >\n> > > > > \"Applications that wish to transition from ..Auto to direct control of\n> > > > > the exposure time without causing extra flicker, should do so by\n> > > > > selecting an ExposureTime value as close as possible to the last value\n> > > > > computed by the auto exposure algorithm.\"\n> > > > >\n> > > > > Don't know if that's better or not.\n> > > > >\n> > > > > > +\n> > > > > > + To select the correct value to use as ExposureTime value, applications\n> > > > > > + should accommodate the natural delay in applying controls caused by the\n> > > > > > + capture pipeline frame depth.\n> > > > > > +\n> > > > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > > > + immediately specify an ExposureTime value in the same request where\n> > > > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > > > + first Request where ExposureTimeMode is reported as\n> > > > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > > > + ExposureTime to populate the control value in the next Request to be\n> > > > > > + queued to the Camera.\n> > > > > > +\n> > > > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > > > + auto exposure use the last value provided by the application as starting\n> > > > > > + point.\n> > > > > > +\n> > > > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > > > +\n> > > > > > + 2. Set ExposureTimeMode to Manual\n> > > > > > +\n> > > > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > > > + set to Manual\n> > > > > > +\n> > > > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > > > + submit it\n> > > > > > +\n> > > > > > + 5. Proceed to run manual exposure time\n> > > > > > +\n> > > > > > + \\sa ExposureTime\n> > > > > > + enum:\n> > > > > > + - name: ExposureTimeModeAuto\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The exposure time will be calculated automatically and set by the\n> > > > > > + AE algorithm.\n> > > > > >\n> > > > > > - \\sa AnalogueGain AeEnable\n> > > > > > + If ExposureTime is set while this mode is active, it will be\n> > > > > > + ignored, and it will also not be retained.\n> > > > > > + - name: ExposureTimeModeManual\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + The exposure time will not be updated by the AE algorithm.\n> > > > > >\n> > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > - better be handled through a separate AE mode control.\n> > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > + exposure value is used until a new value is specified through the\n> > > > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > > > + Manual, the provided ExposureTime is applied immediately.\n> > > > >\n> > > > > Did we say anywhere what happens when you go from manual back to auto?\n> > > > > I'm assuming the adjustments restart from the manual values that you\n> > > > > currently have.\n> > > > >\n> > > > > >\n> > > > > > - AnalogueGain:\n> > > > > > type: float\n> > > > > > @@ -160,17 +316,64 @@ controls:\n> > > > > > The value of the control specifies the gain multiplier applied to all\n> > > > > > colour channels. This value cannot be lower than 1.0.\n> > > > > >\n> > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > - AE algorithm.\n> > > > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > > > + ignored and will not be retained.\n> > > > >\n> > > > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > > > >\n> > > > > > +\n> > > > > > + When reported in metadata, this control indicates what analogue gain\n> > > > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > > > + whether it came from the AEGC algorithm or not.\n> > > > > > +\n> > > > > > + \\sa ExposureTime\n> > > > > > + \\sa AnalogueGainMode\n> > > > > > +\n> > > > > > + - AnalogueGainMode:\n> > > > > > + type: int32_t\n> > > > > > + description: |\n> > > > > > + Controls the source of the analogue gain that is applied to the image\n> > > > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > + the value of the AnalogueGain control is used.\n> > > > > > +\n> > > > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > > > + is provided by the application, the last value computed by the AEGC\n> > > > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > + best-effort default value.\n> > > > > > +\n> > > > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > > > + also be supported.\n> > > > > > +\n> > > > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > > > >\n> > > > > As above!\n> > > > >\n> > > > > Thanks again\n> > > > >\n> > > > > David\n> > > > >\n> > > > > > +\n> > > > > > + The same procedure described for performing flickerless transitions in\n> > > > > > + the ExposureTimeMode control documentation should be applied to\n> > > > > > + analogue gain.\n> > > > > > +\n> > > > > > + \\sa ExposureTimeMode\n> > > > > > + \\sa AnalogueGain\n> > > > > > + enum:\n> > > > > > + - name: AnalogueGainModeAuto\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The analogue gain will be calculated automatically and set by the\n> > > > > > + AEGC algorithm.\n> > > > > >\n> > > > > > - \\sa ExposureTime AeEnable\n> > > > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > > > + ignored, and it will also not be retained.\n> > > > > > + - name: AnalogueGainModeManual\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > > > >\n> > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > - better be handled through a separate AE mode control.\n> > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > + gain value is used until a new value is specified through the\n> > > > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > > > >\n> > > > > > - AeFlickerMode:\n> > > > > > type: int32_t\n> > > > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > > > @@ -77,35 +77,6 @@ controls:\n> > > > > > High quality aberration correction which might reduce the frame\n> > > > > > rate.\n> > > > > >\n> > > > > > - - AeState:\n> > > > > > - type: int32_t\n> > > > > > - description: |\n> > > > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > > > - ANDROID_CONTROL_AE_STATE.\n> > > > > > -\n> > > > > > - Current state of the AE algorithm.\n> > > > > > - enum:\n> > > > > > - - name: AeStateInactive\n> > > > > > - value: 0\n> > > > > > - description: The AE algorithm is inactive.\n> > > > > > - - name: AeStateSearching\n> > > > > > - value: 1\n> > > > > > - description: The AE algorithm has not converged yet.\n> > > > > > - - name: AeStateConverged\n> > > > > > - value: 2\n> > > > > > - description: The AE algorithm has converged.\n> > > > > > - - name: AeStateLocked\n> > > > > > - value: 3\n> > > > > > - description: The AE algorithm is locked.\n> > > > > > - - name: AeStateFlashRequired\n> > > > > > - value: 4\n> > > > > > - description: The AE algorithm would need a flash for good results\n> > > > > > - - name: AeStatePrecapture\n> > > > > > - value: 5\n> > > > > > - description: |\n> > > > > > - The AE algorithm has started a pre-capture metering session.\n> > > > > > - \\sa AePrecaptureTrigger\n> > > > > > -\n> > > > > > - AwbState:\n> > > > > > type: int32_t\n> > > > > > description: |\n> > > > > > --\n> > > > > > 2.39.2\n> > > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BAEEBC326B\n\tfor ;\n\tTue, 19 Nov 2024 13:44:29 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E4A6E65F17;\n\tTue, 19 Nov 2024 14:44:28 +0100 (CET)","from mail-oa1-x32.google.com (mail-oa1-x32.google.com\n\t[IPv6:2001:4860:4864:20::32])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 672F9658A2\n\tfor ;\n\tTue, 19 Nov 2024 14:44:26 +0100 (CET)","by mail-oa1-x32.google.com with SMTP id\n\t586e51a60fabf-2969ae2c99fso596574fac.3\n\tfor ;\n\tTue, 19 Nov 2024 05:44:26 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"KiES7O5T\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1732023865; x=1732628665;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=idwOi4u5jmSs6BN9LEq4nrxTmBHMZeLeufv29HHxA1s=;\n\tb=KiES7O5Tih4bYEPY4/2Ai5FixsejgKq5oO+NZQcN//p1NI8VXM4oerQIcd3tVchIxg\n\tzi9wLOdvtEOsJ/7r+Aylls109eW15CKzyeeW6cyXzlhJIPpU0gQuMbSfcd4hQaVbiwq6\n\tEQaPBTWTsjOtot93r7nBdH6pDY9C1/sjLG82c4xopd7BRaFGaEjl8ePM5hQOj9ytmGCQ\n\tuPLMGS+Z3LtS+jjdSbKEAbMuDW795aTLNPEWNsl1LYTZjJ+x7kl9Z6BP9xZo7rl/JQvQ\n\tA5O/3FhSlIgVmOJ0M2RfOdPluiZW5QSO6+6typ2+ILuZ1Ls/fcW/9OEXejEeIia2iISA\n\tidSg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1732023865; x=1732628665;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=idwOi4u5jmSs6BN9LEq4nrxTmBHMZeLeufv29HHxA1s=;\n\tb=wg5bjtpPWsnnxqx3CRlZ9kRDNcCysu/UaQXxt/AIdyilgYu+o/oxSDWc9vyNxTakoo\n\tiFq5UpOTmKcdJHN5Zt3VZLWYNEs4bqCK4Vkg6qT6VS8wi3SttrJp+RJ/oMgri5izZKmi\n\t7SUZBnoIo4rkm95R+gxey/givw6lbUkVeNFnIIuPJRDEE2scU2hHUnCNzw3dUogTFoR3\n\tjkAyye7fCOZYTTBRicDlxaYKr3szvrja1tGfmBOwzk9THyX7C9mp66A2WThYCLOozfX6\n\tLijSXexfjQBA41a26Sqx6tGmFZtKGYKD/PX7u7SbFaWcd7tMW14abAmtP3xHMTEIvf6e\n\t8+MA==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCUspL1sb8lo2xe9urLnFjXF3do8nSa5HKA2eqqbGGzZcw1bw5L1y9nyvxyGaxjZzo8eA0I88n/BHxJ8LQm2H/Q=@lists.libcamera.org","X-Gm-Message-State":"AOJu0Yw/McAVBG27yNz+B2jGVdfI6Yx6rH2xkIGMNb5wFIkSiYVK5gAP\n\tneoKYfeovtC8PUDv8PEbUj030/aQQeJJjTL7kjGcEX8/JzuAydpXiyoa87DHwdrVgd7Pavg6MiR\n\tK1R4FpSB3+TCcdKkE5rd9rVXEQCuEPu2WsMCLO0eX0rR5TsQz","X-Google-Smtp-Source":"AGHT+IFoiQ2ucPtJH4ZT5/U0cN1DDLi7dq5zrq4CG88U65FCvfkso5Zlpo3m0a1ZvlRAiXBgoJFJTgfqY+0SgiRoRFk=","X-Received":"by 2002:a05:6359:4295:b0:1b8:211a:e7b3 with SMTP id\n\te5c5f4694b2df-1c6cd205737mr655425355d.5.1732023864833;\n\tTue, 19 Nov 2024 05:44:24 -0800 (PST)","MIME-Version":"1.0","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t","In-Reply-To":"","From":"David Plowman ","Date":"Tue, 19 Nov 2024 13:44:13 +0000","Message-ID":"","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","To":"Jacopo Mondi ","Cc":"Paul Elder ,\n\tStefan Klug , \n\tlibcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com,\n\tnaush@raspberrypi.com, Jacopo Mondi ","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32284,"web_url":"https://patchwork.libcamera.org/comment/32284/","msgid":"<6kmhlxidabqwf2a5raw3kv3biv6ci274c7tjicht56m2tt3pgn@bo3yaavi6re7>","date":"2024-11-19T13:51:59","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi David\n\nOn Tue, Nov 19, 2024 at 01:44:13PM +0000, David Plowman wrote:\n> Hi Jacopo\n>\n> On Mon, 18 Nov 2024 at 14:43, Jacopo Mondi\n> wrote:\n> >\n> > Hi David\n> >\n> > On Mon, Nov 18, 2024 at 02:29:15PM +0000, David Plowman wrote:\n> > > Hi Paul\n> > >\n> > > I would just like to make sure we're looking after some of our (many)\n> > > less-expert Raspberry Pi users, and don't want to cause code they're\n> > > using to break with minimal warning or explanation. (These are not\n> > > people who read libcamera mailing lists!)\n> > >\n> >\n> > One clarification here: are you concerned about pi users that has\n> > custom applications interfacing directly with libcamera (either\n> > through the C++ or Python APIs) ?\n>\n> I'm not particularly worried about anyone using C++. It's the Python\n> users, who may be using simple scripts they've written or been given,\n> who could get confused and end up clogging the Raspberry Pi forums!\n>\n> >\n> > > For example, it could be helpful to us to maintain some degree of\n> > > backward compatibility. Maybe switching to \"manual mode\" automatically\n> > > would help. As I commented previously, AF lens position works like\n> > > this, so it would be more consistent.\n> > >\n> > > I understand that folks don't like the \"zero means auto\" value, on the\n> > > other hand I'm not totally persuaded that breaking a single control\n> > > into two, now with the risk of a race hazard, is so very much better.\n> > > Maybe we could have a switchover period where the old method works but\n> > > issues a warning?\n> > >\n> > > Another thought would be for us to \"translate\" control lists from the\n> > > older API to the newer one at runtime. It's hardly a big deal, but\n> > > feels a teeny weeny bit inelegant!\n> > >\n> >\n> > Because this seems to suggest you could handle the translation, I\n> > though you were considering doing that in your applications, but\n> > now I wonder if you're suggesting to do that in the IPA\n>\n> Yes, I can handle the translation for Python users easily enough. It's\n> not lovely, but I think that's probably the least bad alternative for\n> us.\n>\n> Sorry if I've been sounding a bit grumpy about this!!\n>\n\nAbsolutely not, I was just not sure if you were suggesting the\ntranslation to happen in the applications or in the IPA.\n\nThanks for the clarification\n\n> David\n>\n> >\n> > Thanks\n> > j\n> >\n> > > Thanks!\n> > > David\n> > >\n> > > On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n> > > >\n> > > > On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > > > > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > > > > Hi Paul,\n> > > > >\n> > > > > Thank you for the patch.\n> > > > >\n> > > > > > Hi Paul\n> > > > > >\n> > > > > > Thanks for posting all this!\n> > > > > >\n> > > > > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > > > > >\n> > > > > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > > > > to model the AEGC algorithm block.\n> > > > > > >\n> > > > > > > The three controls allow applications to select the exposure time and\n> > > > > > > analogue gain computation calculation mode (auto vs manual)\n> > > > > > > independently from one another, while the AeState control reports the\n> > > > > > > global state for the AEGC algorithm.\n> > > > > > >\n> > > > > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > > > > controls, which are momentarily kept not to break compilation of\n> > > > > > > platforms making use of them.\n> > > > > > >\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > > > > Signed-off-by: Paul Elder \n> > > > > > > Signed-off-by: Jacopo Mondi \n> > > > > > >\n> > > > > > > ---\n> > > > > > > Changes in v3:\n> > > > > > > - recovered from 2-year-old bitrot\n> > > > > >\n> > > > > > Indeed, I think I remember discussing all this _years_ ago!!\n> > > > > >\n> > > > > > > ---\n> > > > > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > > > > >\n> > > > > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > > > > @@ -26,6 +26,75 @@ controls:\n> > > > > > >\n> > > > > > > \\sa AeEnable\n> > > > > > >\n> > > > > > > + - AeState:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Control to report the AEGC algorithm state.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > > > > + values to be applied to the image sensor.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > > > > + independently from one another.\n> > > > > > > +\n> > > > > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > > > > + value and describes it as a single computation block which computes\n> > > > > > > + both the exposure time and the analogue gain values.\n> > > > > > > +\n> > > > > > > + When both the exposure time and analogue gain values are configured to\n> > > > > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > > > > +\n> > > > > > > + When at least the exposure time or analogue gain are configured to be\n> > > > > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > > > > + algorithm has converged to stable values for all of the controls set\n> > > > > > > + to be computed in Auto mode.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > + enum:\n> > > > > > > + - name: AeStateIdle\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm is inactive.\n> > > > > > > +\n> > > > > > > + This state is returned when both AnalogueGainMode and\n> > > > > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > > > > + actively computing any value.\n> > > > > > > +\n> > > > > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > > > > >\n> > > > > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > > > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > > > > autofocus description! But I'm not too bothered.\n> > > > > >\n> > > > > > > + case the AeState control is moved to AeStateSearching.\n> > > > > > > + - name: AeStateSearching\n> > > > >\n> > > > > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > > > > well be because I'm not a native speaker). How does \"regulating\" or\n> > > > > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > > > > converge, but a regulation does.\n> > > >\n> > > > Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> > > > Searching I think works fine because you're searching in a numerical\n> > > > space for the value that maximizes the score.\n> > > >\n> > > > >\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > > > > + exposure time or the analogue gain, but has not converged to a\n> > > > > > > + stable result yet.\n> > > > > > > +\n> > > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > > > > + converged yet.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm converges once stable values are computed for\n> > > > > > > + any of the controls set to be computed in Auto mode.\n> > > > >\n> > > > > Should that be \"all of the\"?\n> > > >\n> > > > Oh yeah it should be.\n> > > >\n> > > > >\n> > > > > > > +\n> > > > > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > > > > + - name: AeStateConverged\n> > > > > > > + value: 2\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm has converged.\n> > > > > > > +\n> > > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > > > > + converged to stable value.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > > > > + which case the state is moved to AeStateSearching.\n> > > > >\n> > > > > This sounds like it is based on a random event. What about something\n> > > > > like: When the statistics/measurements move too far away from the\n> > > > > convergence point the AEGC algo automatically restarts\n> > > > > regulation/scanning...\n> > > >\n> > > > In a way I think it's spontaneous to the user :)\n> > > >\n> > > > But indeed your description would be more accurate.\n> > > >\n> > > > >\n> > > > > > > +\n> > > > > > > # AeMeteringMode needs further attention:\n> > > > > > > # - Auto-generate max enum value.\n> > > > > > > # - Better handling of custom types.\n> > > > > > > @@ -104,6 +173,13 @@ controls:\n> > > > > > > The exposure modes specify how the desired total exposure is divided\n> > > > > > > between the shutter time and the sensor's analogue gain. They are\n> > > > > > > platform specific, and not all exposure modes may be supported.\n> > > > > > > +\n> > > > > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > enum:\n> > > > > > > - name: ExposureNormal\n> > > > > > > value: 0\n> > > > > > > @@ -124,13 +200,15 @@ controls:\n> > > > > > > Specify an Exposure Value (EV) parameter.\n> > > > > > >\n> > > > > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > > > > - enabled.\n> > > > > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > > > > + are in Auto mode.\n> > > > > > >\n> > > > > > > By convention EV adjusts the exposure as log2. For example\n> > > > > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > > > > >\n> > > > > > > - \\sa AeEnable\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > >\n> > > > > > > - ExposureTime:\n> > > > > > > type: int32_t\n> > > > > > > @@ -140,17 +218,95 @@ controls:\n> > > > > > >\n> > > > > > > This value is specified in micro-seconds.\n> > > > > > >\n> > > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > > - AE algorithm.\n> > > > > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > > > > + ignored and will not be retained.\n> > > > > >\n> > > > > > I'm not sure about this, it seems like a change of API behaviour which\n> > > > > > will require users to be re-educated and application code to be hunted\n> > > > > > down and changed. Previously you could just set the exposure time to X\n> > > > > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > > > > same, I think, with respect to setting the lens position. It's just\n> > > > > > more convenient.\n> > > > >\n> > > > > This is an interesting discussion. I wasn't aware of the old\n> > > > > specification. I see that just setting one value and deducing what the\n> > > > > user wants is convenient. On the other hand one control changing the\n> > > > > value of another control feels quite difficult to explain and in my\n> > > > > opinion should be prevented whenever possible. I believe it is easier to\n> > > > > educate the user and have an explicit interface than to have hidden\n> > > > > magic whenever possible.\n> > > >\n> > > > Yeah that was my thought process too when I designed this.\n> > > >\n> > > >\n> > > > Thanks,\n> > > >\n> > > > Paul\n> > > >\n> > > > >\n> > > > > >\n> > > > > > [There's also the question of how confident you are that pipeline\n> > > > > > handlers correctly deal with two controls like this that are\n> > > > > > effectively ordered...]\n> > > > > >\n> > > > > > > +\n> > > > > > > + When reported in metadata, this control indicates what exposure time\n> > > > > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > > > > + whether it came from the AE algorithm or not.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGain\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > + - ExposureTimeMode:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Controls the source of the exposure time that is applied to the image\n> > > > > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > > + the value of the ExposureTime control is used.\n> > > > > > > +\n> > > > > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > > > > + is provided by the application, the last value computed by the AE\n> > > > > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > > + best-effort default value.\n> > > > > >\n> > > > > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > > > > setting AeEnable to zero used to do, except that it can be done\n> > > > > > separately for exposure time and gain now.\n> > > > > >\n> > > > > > > +\n> > > > > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > > > > + also be supported.\n> > > > > > > +\n> > > > > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > > > > >\n> > > > > > That had me scratching my head for a moment! Maybe\n> > > > > >\n> > > > > > \"At least one of the modes, auto or manual, must be supported by both\n> > > > > > ExposureTimeMode and AnalogueGainMode.\"\n> > > > > >\n> > > > > > Is that clearer? (Also, is it what you meant?)\n> > > > > >\n> > > > > > > +\n> > > > > > > + Flickerless exposure mode transitions\n> > > > > > > +\n> > > > > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > > > > + control of the exposure time should aim to do so by selecting an\n> > > > > > > + ExposureTime value as close as possible to the last value computed by\n> > > > > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > > > > >\n> > > > > > Sounds a bit like you _have_ to do this. Maybe\n> > > > > >\n> > > > > > \"Applications that wish to transition from ..Auto to direct control of\n> > > > > > the exposure time without causing extra flicker, should do so by\n> > > > > > selecting an ExposureTime value as close as possible to the last value\n> > > > > > computed by the auto exposure algorithm.\"\n> > > > > >\n> > > > > > Don't know if that's better or not.\n> > > > > >\n> > > > > > > +\n> > > > > > > + To select the correct value to use as ExposureTime value, applications\n> > > > > > > + should accommodate the natural delay in applying controls caused by the\n> > > > > > > + capture pipeline frame depth.\n> > > > > > > +\n> > > > > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > > > > + immediately specify an ExposureTime value in the same request where\n> > > > > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > > > > + first Request where ExposureTimeMode is reported as\n> > > > > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > > > > + ExposureTime to populate the control value in the next Request to be\n> > > > > > > + queued to the Camera.\n> > > > > > > +\n> > > > > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > > > > + auto exposure use the last value provided by the application as starting\n> > > > > > > + point.\n> > > > > > > +\n> > > > > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > > > > +\n> > > > > > > + 2. Set ExposureTimeMode to Manual\n> > > > > > > +\n> > > > > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > > > > + set to Manual\n> > > > > > > +\n> > > > > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > > > > + submit it\n> > > > > > > +\n> > > > > > > + 5. Proceed to run manual exposure time\n> > > > > > > +\n> > > > > > > + \\sa ExposureTime\n> > > > > > > + enum:\n> > > > > > > + - name: ExposureTimeModeAuto\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The exposure time will be calculated automatically and set by the\n> > > > > > > + AE algorithm.\n> > > > > > >\n> > > > > > > - \\sa AnalogueGain AeEnable\n> > > > > > > + If ExposureTime is set while this mode is active, it will be\n> > > > > > > + ignored, and it will also not be retained.\n> > > > > > > + - name: ExposureTimeModeManual\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The exposure time will not be updated by the AE algorithm.\n> > > > > > >\n> > > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > > - better be handled through a separate AE mode control.\n> > > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > > + exposure value is used until a new value is specified through the\n> > > > > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > > > > + Manual, the provided ExposureTime is applied immediately.\n> > > > > >\n> > > > > > Did we say anywhere what happens when you go from manual back to auto?\n> > > > > > I'm assuming the adjustments restart from the manual values that you\n> > > > > > currently have.\n> > > > > >\n> > > > > > >\n> > > > > > > - AnalogueGain:\n> > > > > > > type: float\n> > > > > > > @@ -160,17 +316,64 @@ controls:\n> > > > > > > The value of the control specifies the gain multiplier applied to all\n> > > > > > > colour channels. This value cannot be lower than 1.0.\n> > > > > > >\n> > > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > > - AE algorithm.\n> > > > > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > > > > + ignored and will not be retained.\n> > > > > >\n> > > > > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > > > > >\n> > > > > > > +\n> > > > > > > + When reported in metadata, this control indicates what analogue gain\n> > > > > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > > > > + whether it came from the AEGC algorithm or not.\n> > > > > > > +\n> > > > > > > + \\sa ExposureTime\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > +\n> > > > > > > + - AnalogueGainMode:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Controls the source of the analogue gain that is applied to the image\n> > > > > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > > + the value of the AnalogueGain control is used.\n> > > > > > > +\n> > > > > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > > > > + is provided by the application, the last value computed by the AEGC\n> > > > > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > > + best-effort default value.\n> > > > > > > +\n> > > > > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > > > > + also be supported.\n> > > > > > > +\n> > > > > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > > > > >\n> > > > > > As above!\n> > > > > >\n> > > > > > Thanks again\n> > > > > >\n> > > > > > David\n> > > > > >\n> > > > > > > +\n> > > > > > > + The same procedure described for performing flickerless transitions in\n> > > > > > > + the ExposureTimeMode control documentation should be applied to\n> > > > > > > + analogue gain.\n> > > > > > > +\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > + \\sa AnalogueGain\n> > > > > > > + enum:\n> > > > > > > + - name: AnalogueGainModeAuto\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The analogue gain will be calculated automatically and set by the\n> > > > > > > + AEGC algorithm.\n> > > > > > >\n> > > > > > > - \\sa ExposureTime AeEnable\n> > > > > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > > > > + ignored, and it will also not be retained.\n> > > > > > > + - name: AnalogueGainModeManual\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > > > > >\n> > > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > > - better be handled through a separate AE mode control.\n> > > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > > + gain value is used until a new value is specified through the\n> > > > > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > > > > >\n> > > > > > > - AeFlickerMode:\n> > > > > > > type: int32_t\n> > > > > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > > > > @@ -77,35 +77,6 @@ controls:\n> > > > > > > High quality aberration correction which might reduce the frame\n> > > > > > > rate.\n> > > > > > >\n> > > > > > > - - AeState:\n> > > > > > > - type: int32_t\n> > > > > > > - description: |\n> > > > > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > > > > - ANDROID_CONTROL_AE_STATE.\n> > > > > > > -\n> > > > > > > - Current state of the AE algorithm.\n> > > > > > > - enum:\n> > > > > > > - - name: AeStateInactive\n> > > > > > > - value: 0\n> > > > > > > - description: The AE algorithm is inactive.\n> > > > > > > - - name: AeStateSearching\n> > > > > > > - value: 1\n> > > > > > > - description: The AE algorithm has not converged yet.\n> > > > > > > - - name: AeStateConverged\n> > > > > > > - value: 2\n> > > > > > > - description: The AE algorithm has converged.\n> > > > > > > - - name: AeStateLocked\n> > > > > > > - value: 3\n> > > > > > > - description: The AE algorithm is locked.\n> > > > > > > - - name: AeStateFlashRequired\n> > > > > > > - value: 4\n> > > > > > > - description: The AE algorithm would need a flash for good results\n> > > > > > > - - name: AeStatePrecapture\n> > > > > > > - value: 5\n> > > > > > > - description: |\n> > > > > > > - The AE algorithm has started a pre-capture metering session.\n> > > > > > > - \\sa AePrecaptureTrigger\n> > > > > > > -\n> > > > > > > - AwbState:\n> > > > > > > type: int32_t\n> > > > > > > description: |\n> > > > > > > --\n> > > > > > > 2.39.2\n> > > > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0FA68C32F1\n\tfor ;\n\tTue, 19 Nov 2024 13:52:09 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1B5EF65F1A;\n\tTue, 19 Nov 2024 14:52:08 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AC58A658A2\n\tfor ;\n\tTue, 19 Nov 2024 14:52:05 +0100 (CET)","from ideasonboard.com (93-61-96-190.ip145.fastwebnet.it\n\t[93.61.96.190])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 22144B3;\n\tTue, 19 Nov 2024 14:51:46 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"WDEGypJ/\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732024306;\n\tbh=oS8+b+LaWG81IT6IbEkMzhkWKnJ+A0LDl1aX9PYePcU=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=WDEGypJ/MIl678WTtj0m45Aaa2mgpsaAX+nPFU5XRYcbjSZA4QhoaerejxdBIXPrH\n\t8NmoLFZCLR5HxGJ59rtvLIJCuZNxuCawopU0/CGxEtA5fLzJMUs2Id2VN5BhzhYv9E\n\ttF+TbPD5GqDwh/TtB9RI3+XF476CNN7dRf5VNbc0=","Date":"Tue, 19 Nov 2024 14:51:59 +0100","From":"Jacopo Mondi ","To":"David Plowman ","Cc":"Jacopo Mondi , \n\tPaul Elder ,\n\tStefan Klug , \n\tlibcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com,\n\tnaush@raspberrypi.com, Jacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"<6kmhlxidabqwf2a5raw3kv3biv6ci274c7tjicht56m2tt3pgn@bo3yaavi6re7>","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32317,"web_url":"https://patchwork.libcamera.org/comment/32317/","msgid":"<20241121021013.GZ12409@pendragon.ideasonboard.com>","date":"2024-11-21T02:10:13","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Fri, Nov 15, 2024 at 10:01:37PM +0900, Paul Elder wrote:\n> On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > >\n> > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > to model the AEGC algorithm block.\n> > >\n> > > The three controls allow applications to select the exposure time and\n> > > analogue gain computation calculation mode (auto vs manual)\n> > > independently from one another, while the AeState control reports the\n> > > global state for the AEGC algorithm.\n> > >\n> > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > controls, which are momentarily kept not to break compilation of\n> > > platforms making use of them.\n> > >\n> > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > Signed-off-by: Paul Elder \n> > > Signed-off-by: Jacopo Mondi \n> > >\n> > > ---\n> > > Changes in v3:\n> > > - recovered from 2-year-old bitrot\n> > \n> > Indeed, I think I remember discussing all this _years_ ago!!\n> > \n> > > ---\n> > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > >\n> > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > --- a/src/libcamera/control_ids_core.yaml\n> > > +++ b/src/libcamera/control_ids_core.yaml\n> > > @@ -26,6 +26,75 @@ controls:\n> > >\n> > > \\sa AeEnable\n> > >\n> > > + - AeState:\n> > > + type: int32_t\n> > > + description: |\n> > > + Control to report the AEGC algorithm state.\n> > > +\n> > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > + values to be applied to the image sensor.\n> > > +\n> > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > + AnalogueGainMode controls, which allow applications to decide how\n> > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > + independently from one another.\n> > > +\n> > > + The AeState control reports the AEGC algorithm state through a single\n> > > + value and describes it as a single computation block which computes\n> > > + both the exposure time and the analogue gain values.\n> > > +\n> > > + When both the exposure time and analogue gain values are configured to\n> > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > + compute any value and the AeState control will report AeStateIdle.\n> > > +\n> > > + When at least the exposure time or analogue gain are configured to be\n> > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > + algorithm has converged to stable values for all of the controls set\n> > > + to be computed in Auto mode.\n> > > +\n> > > + \\sa AnalogueGainMode\n> > > + \\sa ExposureTimeMode\n> > > +\n> > > + enum:\n> > > + - name: AeStateIdle\n> > > + value: 0\n> > > + description: |\n> > > + The AEGC algorithm is inactive.\n> > > +\n> > > + This state is returned when both AnalogueGainMode and\n> > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > + actively computing any value.\n> > > +\n> > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > \n> > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > autofocus description! But I'm not too bothered.\n> \n> Yeah that probably reads better. (it's been 2 years I don't remember\n> where the wording came from)\n\nSounds good.\n\n> > > + case the AeState control is moved to AeStateSearching.\n> > > + - name: AeStateSearching\n> > > + value: 1\n> > > + description: |\n> > > + The AEGC algorithm is actively computing new values, for either the\n> > > + exposure time or the analogue gain, but has not converged to a\n> > > + stable result yet.\n> > > +\n> > > + This state is returned if at least one of AnalogueGainMode\n> > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > + converged yet.\n> > > +\n> > > + The AEGC algorithm converges once stable values are computed for\n> > > + any of the controls set to be computed in Auto mode.\n> > > +\n> > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > + - name: AeStateConverged\n> > > + value: 2\n> > > + description: |\n> > > + The AEGC algorithm has converged.\n> > > +\n> > > + This state is returned if at least one of AnalogueGainMode\n> > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > + converged to stable value.\n> > > +\n> > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > + which case the state is moved to AeStateSearching.\n> > > +\n> > > # AeMeteringMode needs further attention:\n> > > # - Auto-generate max enum value.\n> > > # - Better handling of custom types.\n> > > @@ -104,6 +173,13 @@ controls:\n> > > The exposure modes specify how the desired total exposure is divided\n> > > between the shutter time and the sensor's analogue gain. They are\n> > > platform specific, and not all exposure modes may be supported.\n> > > +\n> > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > + the fixed values will override any choices made by AeExposureMode.\n> > > +\n> > > + \\sa AnalogueGainMode\n> > > + \\sa ExposureTimeMode\n> > > +\n> > > enum:\n> > > - name: ExposureNormal\n> > > value: 0\n> > > @@ -124,13 +200,15 @@ controls:\n> > > Specify an Exposure Value (EV) parameter.\n> > >\n> > > The EV parameter will only be applied if the AE algorithm is currently\n> > > - enabled.\n> > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > + are in Auto mode.\n> > >\n> > > By convention EV adjusts the exposure as log2. For example\n> > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > >\n> > > - \\sa AeEnable\n> > > + \\sa AnalogueGainMode\n> > > + \\sa ExposureTimeMode\n> > >\n> > > - ExposureTime:\n> > > type: int32_t\n> > > @@ -140,17 +218,95 @@ controls:\n> > >\n> > > This value is specified in micro-seconds.\n> > >\n> > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > - not change it. Setting it back to zero returns it to the control of the\n> > > - AE algorithm.\n> > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > + ignored and will not be retained.\n> > \n> > I'm not sure about this, it seems like a change of API behaviour which\n> > will require users to be re-educated and application code to be hunted\n> > down and changed. Previously you could just set the exposure time to X\n> > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > same, I think, with respect to setting the lens position. It's just\n> > more convenient.\n> \n> It does indeed change the API, but I was always of the camp that setting\n> 0 to enable/disable auto mode isn't a very good API :/\n\nWhile we don't want to change APIs in a way that will require changes in\napplications just for the sake of it, we have lots of controls that will\nneed to change. We want to minimize disturbances for users, but they're\nunavoidable until we freeze the ABI.\n\n> > [There's also the question of how confident you are that pipeline\n> > handlers correctly deal with two controls like this that are\n> > effectively ordered...]\n> \n> Well at the moment they don't :)\n> I was thinking about having these enforced by lc-compliance.\n\nlc-compliance is the right tool for this job. It hasn't seen lots of\npublic activity recently, but that should change.\n\n> > > +\n> > > + When reported in metadata, this control indicates what exposure time\n> > > + was used for the current request, regardless of ExposureTimeMode.\n> > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > + whether it came from the AE algorithm or not.\n> > > +\n> > > + \\sa AnalogueGain\n> > > + \\sa ExposureTimeMode\n> > > +\n> > > + - ExposureTimeMode:\n> > > + type: int32_t\n> > > + description: |\n> > > + Controls the source of the exposure time that is applied to the image\n> > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > + and configures the image sensor accordingly. When set to Manual,\n> > > + the value of the ExposureTime control is used.\n> > > +\n> > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > + is provided by the application, the last value computed by the AE\n> > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > + was never set to Auto (either because the camera started in Manual mode,\n> > > + or Auto is not supported by the camera), the camera should use a\n> > > + best-effort default value.\n> > \n> > Yes, I think this sounds OK. So setting it to manual mode is what\n> > setting AeEnable to zero used to do, except that it can be done\n> > separately for exposure time and gain now.\n> \n> \\o/\n> \n> > > +\n> > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > + also be supported.\n> > > +\n> > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > \n> > That had me scratching my head for a moment! Maybe\n\nAgree. Understanding the libcamera documentation shouldn't require a\nmaster in theoretical mathematics :-) (OK, I'm exagerating a little bit\nhere, but I think the current wording could be made clearer.)\n\n> > \"At least one of the modes, auto or manual, must be supported by both\n> > ExposureTimeMode and AnalogueGainMode.\"\n> > \n> > Is that clearer? (Also, is it what you meant?)\n> \n> No, that's not what I meant.\n> \n> I suppose an explicit listing of all valid combinations would be clear\n> to at least illustrate what I meant.\n> \n> Valid combinations of supported modes:\n> \n> - exposure: { }, gain: { }\n\nTechnically speaking the intersection of empty sets is empty.\n\nIf no modes are supported, I would expect the ExposureTimeMode control\nnot to be exposed. Is my assumption incorrect ?\n\n> - exposure: { auto }, gain: { auto }\n\nDo you foresee a use case for this ? \"Auto only\" means a camera where we\nhave no control whatsoever over the ISP, as otherwise we could implement\nmanual mode. I suppose we could, in theory, have a UVC camera that\ndoesn't support manual mode :-/\n\nShould we disallow auto-only for the time being, until we find a camera\nthat can't support manual mode ? Or would that be disturbing for\napplications to introduce this later ?\n\nIf we have control over the ISP, I think this should be disallowed, but\nthat's likely not in scope for the control documentation.\n\n> - exposure: { manual }, gain: { manual }\n\nSimilarly here, when we have control of the ISP, I think this shouldn't\nbe allowed.\n\n> - exposure: { auto, manual }, gain: { auto, manual }\n> \n> Invalid combinations of supported modes:\n> \n> - exposure: { auto }, gain: { manual }\n> - exposure: { manual }, gain: { auto }\n> - exposure: { }, gain: { auto }\n> - exposure: { }, gain: { manual }\n> - exposure: { auto }, gain: { }\n> - exposure: { manual }, gain: { }\n> \n> Ok now that I've written them out I'm not certain about the last four\n> cases, but anyway this is what I had in mind.\n> \n> \"The intersection of the sets of supported modes must not be empty\" :)\n\nDoes this mean the following is also valid ?\n\n- exposure: { auto }, gain: { auto, manual }\n- exposure: { manual }, gain: { auto, manual }\n- exposure: { auto, manual }, gain: { auto }\n- exposure: { auto, manual }, gain: { manual }\n\nIf we have control over the ISP, I don't see a reason to allow those\ncombinations. Unless there would be a camera sensor that exposes control\nof exposure time but has no analogue gain ? I think that's a bit\nfar-fetched.\n\nUVC cameras are also annoying here. The UVC specification includes the\nfollowing related controls:\n\n- CT_AE_MODE_CONTROL: Manual, Auto, Shutter Priority (manual exposure\n and auto iris), Aperture Priority (auto exposure and manual iris)\n\n- CT_AE_PRIORITY_CONTROL: Indicates whether or not auto-exposure can\n change the frame rate\n\n- CT_EXPOSURE_TIME_ABSOLUTE_CONTROL, CT_EXPOSURE_TIME_RELATIVE_CONTROL:\n Manual exposure control\n\n- CT_IRIS_ABSOLUTE_CONTROL, CT_IRIS_RELATIVE_CONTROL: Manual iris\n (aperture) control\n\n- PU_GAIN_CONTROL: Manual gain control\n\nI have over time collected UVC descriptor dumps from 440 devices. Here's\nsome findings.\n\n240 devices do not exposure any AE mode or exposure time control. We can\nonly assume they always operate in auto-exposure mode. That's more than\nhalf of them.\n\n19 devices expose the CT_AE_MODE_CONTROL control without exposing\nCT_EXPOSURE_TIME_ABSOLUTE_CONTROL or CT_EXPOSURE_TIME_RELATIVE_CONTROL.\nI don't know what modes they report as supported for CT_AE_MODE_CONTROL,\nbut I can imagine that some of them would only support the Auto mode.\n\n10 devices report the CT_EXPOSURE_TIME_ABSOLUTE_CONTROL control without\nreporting CT_AE_MODE_CONTROL. 6 of them are industrial cameras, so it's\nplausible they would support manual mode only. Some other devices are\nwebcam and I can only assume their descriptors are somehow wrong.\n\n2 devices report CT_EXPOSURE_TIME_RELATIVE_CONTROL without\nCT_EXPOSURE_TIME_ABSOLUTE_CONTROL. That's annoying as the relative\nexposure control doesn't have a unit, unlike the absolute exposure\ncontrol. We can probably ignore manual mode in that case.\n\nPU_GAIN_CONTROL is \"interesting\", in the sense that it's listed as a\nProcessing Unit control, not an Input Terminal (= camera sensor)\ncontrol. I expect it maps to digital gain, with analogue gain being\ncontrolled under the hood by auto-exposure.\n\n372 devices report the PU_GAIN_CONTROL. Among those, 52 do not report\nthe CT_AE_MODE_CONTROL control.\n\n> (ok technically the first valid combination is empty but)\n> \n> > > +\n> > > + Flickerless exposure mode transitions\n> > > +\n> > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > + control of the exposure time should aim to do so by selecting an\n> > > + ExposureTime value as close as possible to the last value computed by\n> > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > \n> > Sounds a bit like you _have_ to do this. Maybe\n> \n> You're right, it kind of does.\n> \n> > \n> > \"Applications that wish to transition from ..Auto to direct control of\n> > the exposure time without causing extra flicker, should do so by\n> > selecting an ExposureTime value as close as possible to the last value\n> > computed by the auto exposure algorithm.\"\n> > \n> > Don't know if that's better or not.\n> \n> I think it's better, thanks.\n> \n> > > +\n> > > + To select the correct value to use as ExposureTime value, applications\n> > > + should accommodate the natural delay in applying controls caused by the\n> > > + capture pipeline frame depth.\n> > > +\n> > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > + immediately specify an ExposureTime value in the same request where\n> > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > + first Request where ExposureTimeMode is reported as\n> > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > + ExposureTime to populate the control value in the next Request to be\n> > > + queued to the Camera.\n> > > +\n> > > + The implementation of the auto-exposure algorithm should equally try to\n> > > + minimize flickering and when transitioning from manual exposure mode to\n> > > + auto exposure use the last value provided by the application as starting\n> > > + point.\n> > > +\n> > > + 1. Start with ExposureTimeMode set to Auto\n> > > +\n> > > + 2. Set ExposureTimeMode to Manual\n> > > +\n> > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > + set to Manual\n> > > +\n> > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > + submit it\n> > > +\n> > > + 5. Proceed to run manual exposure time\n> > > +\n> > > + \\sa ExposureTime\n> > > + enum:\n> > > + - name: ExposureTimeModeAuto\n> > > + value: 0\n> > > + description: |\n> > > + The exposure time will be calculated automatically and set by the\n> > > + AE algorithm.\n> > >\n> > > - \\sa AnalogueGain AeEnable\n> > > + If ExposureTime is set while this mode is active, it will be\n> > > + ignored, and it will also not be retained.\n> > > + - name: ExposureTimeModeManual\n> > > + value: 1\n> > > + description: |\n> > > + The exposure time will not be updated by the AE algorithm.\n> > >\n> > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > - value for this control. Consider interactions with other AE features,\n> > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > - control of which features should be automatically adjusted shouldn't\n> > > - better be handled through a separate AE mode control.\n> > > + When transitioning from Auto to Manual mode, the last computed\n> > > + exposure value is used until a new value is specified through the\n> > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > + same request where the ExposureTimeMode is changed from Auto to\n> > > + Manual, the provided ExposureTime is applied immediately.\n> > \n> > Did we say anywhere what happens when you go from manual back to auto?\n> > I'm assuming the adjustments restart from the manual values that you\n> > currently have.\n> \n> I don't remember saying anywhere what happens. It should restart from\n> the manual values yeah. This needs to be enforced by lc-compliance too.\n\nHow would we check that though ? The camera won't report what AE will\nstart with, will it ? Or is it about the exposure time and gain not\nbeing changed until the AE algorithm will have a useful value to set ?\nIf the request queue is full enough, I expect the frame where AE gets\nreenabled to have an exposure time and gain computed by the AE\nalgorithm.\n\n> > >\n> > > - AnalogueGain:\n> > > type: float\n> > > @@ -160,17 +316,64 @@ controls:\n> > > The value of the control specifies the gain multiplier applied to all\n> > > colour channels. This value cannot be lower than 1.0.\n> > >\n> > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > - not change it. Setting it back to zero returns it to the control of the\n> > > - AE algorithm.\n> > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > + ignored and will not be retained.\n> > \n> > As above. It would be user-friendly to switch automatically to manual, I think.\n> \n> If you do that then the procedure that I described above for flickerless\n> switching from auto to manual won't work and we'd need another\n> mechanism/design to implement flickerless switching.\n> \n> > > +\n> > > + When reported in metadata, this control indicates what analogue gain\n> > > + was used for the current request, regardless of AnalogueGainMode.\n> > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > + whether it came from the AEGC algorithm or not.\n> > > +\n> > > + \\sa ExposureTime\n> > > + \\sa AnalogueGainMode\n> > > +\n> > > + - AnalogueGainMode:\n> > > + type: int32_t\n> > > + description: |\n> > > + Controls the source of the analogue gain that is applied to the image\n> > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > + and configures the image sensor accordingly. When set to Manual,\n> > > + the value of the AnalogueGain control is used.\n> > > +\n> > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > + is provided by the application, the last value computed by the AEGC\n> > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > + was never set to Auto (either because the camera started in Manual mode,\n> > > + or Auto is not supported by the camera), the camera should use a\n> > > + best-effort default value.\n> > > +\n> > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > + also be supported.\n> > > +\n> > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > \n> > As above!\n> > \n> > Thanks again\n> > \n> > David\n> > \n> > > +\n> > > + The same procedure described for performing flickerless transitions in\n> > > + the ExposureTimeMode control documentation should be applied to\n> > > + analogue gain.\n> > > +\n> > > + \\sa ExposureTimeMode\n> > > + \\sa AnalogueGain\n> > > + enum:\n> > > + - name: AnalogueGainModeAuto\n> > > + value: 0\n> > > + description: |\n> > > + The analogue gain will be calculated automatically and set by the\n> > > + AEGC algorithm.\n> > >\n> > > - \\sa ExposureTime AeEnable\n> > > + If AnalogueGain is set while this mode is active, it will be\n> > > + ignored, and it will also not be retained.\n> > > + - name: AnalogueGainModeManual\n> > > + value: 1\n> > > + description: |\n> > > + The analogue gain will not be updated by the AEGC algorithm.\n> > >\n> > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > - value for this control. Consider interactions with other AE features,\n> > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > - control of which features should be automatically adjusted shouldn't\n> > > - better be handled through a separate AE mode control.\n> > > + When transitioning from Auto to Manual mode, the last computed\n> > > + gain value is used until a new value is specified through the\n> > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > + same request where the AnalogueGainMode is changed from Auto to\n> > > + Manual, the provided AnalogueGain is applied immediately.\n> > >\n> > > - AeFlickerMode:\n> > > type: int32_t\n> > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > index 1b284257f601..32c2b3feaf65 100644\n> > > --- a/src/libcamera/control_ids_draft.yaml\n> > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > @@ -77,35 +77,6 @@ controls:\n> > > High quality aberration correction which might reduce the frame\n> > > rate.\n> > >\n> > > - - AeState:\n> > > - type: int32_t\n> > > - description: |\n> > > - Control to report the current AE algorithm state. Currently identical to\n> > > - ANDROID_CONTROL_AE_STATE.\n> > > -\n> > > - Current state of the AE algorithm.\n> > > - enum:\n> > > - - name: AeStateInactive\n> > > - value: 0\n> > > - description: The AE algorithm is inactive.\n> > > - - name: AeStateSearching\n> > > - value: 1\n> > > - description: The AE algorithm has not converged yet.\n> > > - - name: AeStateConverged\n> > > - value: 2\n> > > - description: The AE algorithm has converged.\n> > > - - name: AeStateLocked\n> > > - value: 3\n> > > - description: The AE algorithm is locked.\n> > > - - name: AeStateFlashRequired\n> > > - value: 4\n> > > - description: The AE algorithm would need a flash for good results\n> > > - - name: AeStatePrecapture\n> > > - value: 5\n> > > - description: |\n> > > - The AE algorithm has started a pre-capture metering session.\n> > > - \\sa AePrecaptureTrigger\n> > > -\n> > > - AwbState:\n> > > type: int32_t\n> > > description: |","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 61EFFC32F5\n\tfor ;\n\tThu, 21 Nov 2024 02:10:27 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8F61565F96;\n\tThu, 21 Nov 2024 03:10:26 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 202C965F54\n\tfor ;\n\tThu, 21 Nov 2024 03:10:24 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id D2707670;\n\tThu, 21 Nov 2024 03:10:04 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"ZgCdhH3q\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732155005;\n\tbh=JaiE861SpBhqKHnwYFFRwy0qK+2cV4y01uq7Cz69reQ=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=ZgCdhH3qFsYdZnBtepWOsvS5g96GTjZUjWW9JCbWt3Fc308P47zqpO9bgUKvVcX8k\n\toZEYqZvK5yYPpPHY1JL7nNJqU28szE/fHMXQMYRTYiIclV+WZOZ6/cIntFrhkLvn2G\n\t60oTs1rs6EVvSlp5zoBiShD+WJQXIi1n5dKzrQ4s=","Date":"Thu, 21 Nov 2024 04:10:13 +0200","From":"Laurent Pinchart ","To":"Paul Elder ","Cc":"David Plowman ,\n\tlibcamera-devel@lists.libcamera.org, jacopo.mondi@ideasonboard.com,\n\tnaush@raspberrypi.com, Jacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"<20241121021013.GZ12409@pendragon.ideasonboard.com>","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32318,"web_url":"https://patchwork.libcamera.org/comment/32318/","msgid":"<20241121022802.GA12409@pendragon.ideasonboard.com>","date":"2024-11-21T02:28:02","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Mon, Nov 18, 2024 at 10:51:02PM +0900, Paul Elder wrote:\n> On Mon, Nov 18, 2024 at 05:30:39PM +0900, Paul Elder wrote:\n> > On Fri, Nov 15, 2024 at 01:51:00PM +0000, Naushir Patuck wrote:\n> > > On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n> > > > On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > > > > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > > > > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > > > > >\n> > > > > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > > > > to model the AEGC algorithm block.\n> > > > > > >\n> > > > > > > The three controls allow applications to select the exposure time and\n> > > > > > > analogue gain computation calculation mode (auto vs manual)\n> > > > > > > independently from one another, while the AeState control reports the\n> > > > > > > global state for the AEGC algorithm.\n> > > > > > >\n> > > > > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > > > > controls, which are momentarily kept not to break compilation of\n> > > > > > > platforms making use of them.\n> > > > > > >\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > > > > Signed-off-by: Paul Elder \n> > > > > > > Signed-off-by: Jacopo Mondi \n> > > > > > >\n> > > > > > > ---\n> > > > > > > Changes in v3:\n> > > > > > > - recovered from 2-year-old bitrot\n> > > > > >\n> > > > > > Indeed, I think I remember discussing all this _years_ ago!!\n> > > > > >\n> > > > > > > ---\n> > > > > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > > > > >\n> > > > > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > > > > @@ -26,6 +26,75 @@ controls:\n> > > > > > >\n> > > > > > > \\sa AeEnable\n> > > > > > >\n> > > > > > > + - AeState:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Control to report the AEGC algorithm state.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > > > > + values to be applied to the image sensor.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > > > > + independently from one another.\n> > > > > > > +\n> > > > > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > > > > + value and describes it as a single computation block which computes\n> > > > > > > + both the exposure time and the analogue gain values.\n> > > > > > > +\n> > > > > > > + When both the exposure time and analogue gain values are configured to\n> > > > > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > > > > +\n> > > > > > > + When at least the exposure time or analogue gain are configured to be\n> > > > > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > > > > + algorithm has converged to stable values for all of the controls set\n> > > > > > > + to be computed in Auto mode.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > + enum:\n> > > > > > > + - name: AeStateIdle\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm is inactive.\n> > > > > > > +\n> > > > > > > + This state is returned when both AnalogueGainMode and\n> > > > > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > > > > + actively computing any value.\n> > > > > > > +\n> > > > > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > > > > >\n> > > > > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > > > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > > > > autofocus description! But I'm not too bothered.\n> > > > > >\n> > > > > > > + case the AeState control is moved to AeStateSearching.\n> > > > > > > + - name: AeStateSearching\n> > > > >\n> > > > > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > > > > well be because I'm not a native speaker). How does \"regulating\" or\n> > > > > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > > > > converge, but a regulation does.\n> > > >\n> > > > Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> > > > Searching I think works fine because you're searching in a numerical\n> > > > space for the value that maximizes the score.\n> > > >\n> > > > >\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > > > > + exposure time or the analogue gain, but has not converged to a\n> > > > > > > + stable result yet.\n> > > > > > > +\n> > > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > > > > + converged yet.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm converges once stable values are computed for\n> > > > > > > + any of the controls set to be computed in Auto mode.\n> > > > >\n> > > > > Should that be \"all of the\"?\n> > > >\n> > > > Oh yeah it should be.\n> > > >\n> > > > >\n> > > > > > > +\n> > > > > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > > > > + - name: AeStateConverged\n> > > > > > > + value: 2\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm has converged.\n> > > > > > > +\n> > > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > > > > + converged to stable value.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > > > > + which case the state is moved to AeStateSearching.\n> > > > >\n> > > > > This sounds like it is based on a random event. What about something\n> > > > > like: When the statistics/measurements move too far away from the\n> > > > > convergence point the AEGC algo automatically restarts\n> > > > > regulation/scanning...\n> > > >\n> > > > In a way I think it's spontaneous to the user :)\n> > > >\n> > > > But indeed your description would be more accurate.\n> > > >\n> > > > >\n> > > > > > > +\n> > > > > > > # AeMeteringMode needs further attention:\n> > > > > > > # - Auto-generate max enum value.\n> > > > > > > # - Better handling of custom types.\n> > > > > > > @@ -104,6 +173,13 @@ controls:\n> > > > > > > The exposure modes specify how the desired total exposure is divided\n> > > > > > > between the shutter time and the sensor's analogue gain. They are\n> > > > > > > platform specific, and not all exposure modes may be supported.\n> > > > > > > +\n> > > > > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > enum:\n> > > > > > > - name: ExposureNormal\n> > > > > > > value: 0\n> > > > > > > @@ -124,13 +200,15 @@ controls:\n> > > > > > > Specify an Exposure Value (EV) parameter.\n> > > > > > >\n> > > > > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > > > > - enabled.\n> > > > > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > > > > + are in Auto mode.\n> > > > > > >\n> > > > > > > By convention EV adjusts the exposure as log2. For example\n> > > > > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > > > > >\n> > > > > > > - \\sa AeEnable\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > >\n> > > > > > > - ExposureTime:\n> > > > > > > type: int32_t\n> > > > > > > @@ -140,17 +218,95 @@ controls:\n> > > > > > >\n> > > > > > > This value is specified in micro-seconds.\n> > > > > > >\n> > > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > > - AE algorithm.\n> > > > > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > > > > + ignored and will not be retained.\n> > > > > >\n> > > > > > I'm not sure about this, it seems like a change of API behaviour which\n> > > > > > will require users to be re-educated and application code to be hunted\n> > > > > > down and changed. Previously you could just set the exposure time to X\n> > > > > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > > > > same, I think, with respect to setting the lens position. It's just\n> > > > > > more convenient.\n> > > > >\n> > > > > This is an interesting discussion. I wasn't aware of the old\n> > > > > specification. I see that just setting one value and deducing what the\n> > > > > user wants is convenient. On the other hand one control changing the\n> > > > > value of another control feels quite difficult to explain and in my\n> > > > > opinion should be prevented whenever possible. I believe it is easier to\n> > > > > educate the user and have an explicit interface than to have hidden\n> > > > > magic whenever possible.\n> > > >\n> > > > Yeah that was my thought process too when I designed this.\n> > > \n> > > There is one minor annoyance this change will introduce because of the\n> > > lack of ordering in the ControlList map.\n> > > \n> > > Going from, say, auto control into manual will require 2 controls, and\n> > > the presence of the AnalogueGainModeManual enum + the AnalogueGain\n> > > value to use. Since Controls are not ordered, if you iterate through\n> > > the ControlList, you will have to check for the presence of one, then\n> > > the other programmatically.\n> > > \n> > \n> > Oh good point, I totally forgot about that. Back to the drawing board...\n> \n> Ok I'm back from the drawing board.\n> \n> At the moment I see two easy options/solutions. One is to check each the\n> prescence of individual control instead of iterating over the control\n> list, like what rkisp1 does. The other option is to directly check the\n> prescence of the enable control when handling the manual control in the\n> iteration. Something like (using what's in the Raspberry Pi IPA):\n> \n> for (auto const &ctrl : controls) {\n> \tswitch(ctrl.first) {\n> \tcase controls::EXPOSURE_TIME_MODE: {\n> \t\texposureEnableFlag_ =\n> \t\t\t(*exposureEnable == controls::ExposureTimeModeAuto);\n> \t}\n> \n> \tcase controls::EXPOSURE_TIME: {\n> \t\tconst auto &exposureEnable =\n> \t\t\tcontrols.get(controls::ExposureTimeMode);\n> \t\tif (exposureEnable &&\n> \t\t (*exposureEnable == controls::ExposureTimeModeAuto))\n> \t\t exposureEnableFlag_ = true;\n> \t\t\t\n> \t\tif (!exposureEnable && !exposureEnableFlag_)\n> \t\t\tcontinue;\n> \t}\n> \t}\n> }\n> \n> I suppose they are indeed minor annoyances, but I think it's a necessary\n> byproduct of consistency and correctness that the new controls provide\n> over what we had previously.\n\nAnother option is to split processing of controls in algorithms in two\nparts, first processing the controls from the ControlList, and then\napplying them. The AGC algorithm would in that case just record the\nvalues from the control list when the IPA iterates over controls, and a\nsubsequent apply step would occur after the iteration.\n\n> > > > > >\n> > > > > > [There's also the question of how confident you are that pipeline\n> > > > > > handlers correctly deal with two controls like this that are\n> > > > > > effectively ordered...]\n> > > > > >\n> > > > > > > +\n> > > > > > > + When reported in metadata, this control indicates what exposure time\n> > > > > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > > > > + whether it came from the AE algorithm or not.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGain\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > + - ExposureTimeMode:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Controls the source of the exposure time that is applied to the image\n> > > > > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > > + the value of the ExposureTime control is used.\n> > > > > > > +\n> > > > > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > > > > + is provided by the application, the last value computed by the AE\n> > > > > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > > + best-effort default value.\n> > > > > >\n> > > > > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > > > > setting AeEnable to zero used to do, except that it can be done\n> > > > > > separately for exposure time and gain now.\n> > > > > >\n> > > > > > > +\n> > > > > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > > > > + also be supported.\n> > > > > > > +\n> > > > > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > > > > >\n> > > > > > That had me scratching my head for a moment! Maybe\n> > > > > >\n> > > > > > \"At least one of the modes, auto or manual, must be supported by both\n> > > > > > ExposureTimeMode and AnalogueGainMode.\"\n> > > > > >\n> > > > > > Is that clearer? (Also, is it what you meant?)\n> > > > > >\n> > > > > > > +\n> > > > > > > + Flickerless exposure mode transitions\n> > > > > > > +\n> > > > > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > > > > + control of the exposure time should aim to do so by selecting an\n> > > > > > > + ExposureTime value as close as possible to the last value computed by\n> > > > > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > > > > >\n> > > > > > Sounds a bit like you _have_ to do this. Maybe\n> > > > > >\n> > > > > > \"Applications that wish to transition from ..Auto to direct control of\n> > > > > > the exposure time without causing extra flicker, should do so by\n> > > > > > selecting an ExposureTime value as close as possible to the last value\n> > > > > > computed by the auto exposure algorithm.\"\n> > > > > >\n> > > > > > Don't know if that's better or not.\n> > > > > >\n> > > > > > > +\n> > > > > > > + To select the correct value to use as ExposureTime value, applications\n> > > > > > > + should accommodate the natural delay in applying controls caused by the\n> > > > > > > + capture pipeline frame depth.\n> > > > > > > +\n> > > > > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > > > > + immediately specify an ExposureTime value in the same request where\n> > > > > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > > > > + first Request where ExposureTimeMode is reported as\n> > > > > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > > > > + ExposureTime to populate the control value in the next Request to be\n> > > > > > > + queued to the Camera.\n> > > > > > > +\n> > > > > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > > > > + auto exposure use the last value provided by the application as starting\n> > > > > > > + point.\n> > > > > > > +\n> > > > > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > > > > +\n> > > > > > > + 2. Set ExposureTimeMode to Manual\n> > > > > > > +\n> > > > > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > > > > + set to Manual\n> > > > > > > +\n> > > > > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > > > > + submit it\n> > > > > > > +\n> > > > > > > + 5. Proceed to run manual exposure time\n> > > > > > > +\n> > > > > > > + \\sa ExposureTime\n> > > > > > > + enum:\n> > > > > > > + - name: ExposureTimeModeAuto\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The exposure time will be calculated automatically and set by the\n> > > > > > > + AE algorithm.\n> > > > > > >\n> > > > > > > - \\sa AnalogueGain AeEnable\n> > > > > > > + If ExposureTime is set while this mode is active, it will be\n> > > > > > > + ignored, and it will also not be retained.\n> > > > > > > + - name: ExposureTimeModeManual\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The exposure time will not be updated by the AE algorithm.\n> > > > > > >\n> > > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > > - better be handled through a separate AE mode control.\n> > > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > > + exposure value is used until a new value is specified through the\n> > > > > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > > > > + Manual, the provided ExposureTime is applied immediately.\n> > > > > >\n> > > > > > Did we say anywhere what happens when you go from manual back to auto?\n> > > > > > I'm assuming the adjustments restart from the manual values that you\n> > > > > > currently have.\n> > > > > >\n> > > > > > >\n> > > > > > > - AnalogueGain:\n> > > > > > > type: float\n> > > > > > > @@ -160,17 +316,64 @@ controls:\n> > > > > > > The value of the control specifies the gain multiplier applied to all\n> > > > > > > colour channels. This value cannot be lower than 1.0.\n> > > > > > >\n> > > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > > - AE algorithm.\n> > > > > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > > > > + ignored and will not be retained.\n> > > > > >\n> > > > > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > > > > >\n> > > > > > > +\n> > > > > > > + When reported in metadata, this control indicates what analogue gain\n> > > > > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > > > > + whether it came from the AEGC algorithm or not.\n> > > > > > > +\n> > > > > > > + \\sa ExposureTime\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > +\n> > > > > > > + - AnalogueGainMode:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Controls the source of the analogue gain that is applied to the image\n> > > > > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > > + the value of the AnalogueGain control is used.\n> > > > > > > +\n> > > > > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > > > > + is provided by the application, the last value computed by the AEGC\n> > > > > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > > + best-effort default value.\n> > > > > > > +\n> > > > > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > > > > + also be supported.\n> > > > > > > +\n> > > > > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > > > > >\n> > > > > > As above!\n> > > > > >\n> > > > > > Thanks again\n> > > > > >\n> > > > > > David\n> > > > > >\n> > > > > > > +\n> > > > > > > + The same procedure described for performing flickerless transitions in\n> > > > > > > + the ExposureTimeMode control documentation should be applied to\n> > > > > > > + analogue gain.\n> > > > > > > +\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > + \\sa AnalogueGain\n> > > > > > > + enum:\n> > > > > > > + - name: AnalogueGainModeAuto\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The analogue gain will be calculated automatically and set by the\n> > > > > > > + AEGC algorithm.\n> > > > > > >\n> > > > > > > - \\sa ExposureTime AeEnable\n> > > > > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > > > > + ignored, and it will also not be retained.\n> > > > > > > + - name: AnalogueGainModeManual\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > > > > >\n> > > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > > - better be handled through a separate AE mode control.\n> > > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > > + gain value is used until a new value is specified through the\n> > > > > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > > > > >\n> > > > > > > - AeFlickerMode:\n> > > > > > > type: int32_t\n> > > > > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > > > > @@ -77,35 +77,6 @@ controls:\n> > > > > > > High quality aberration correction which might reduce the frame\n> > > > > > > rate.\n> > > > > > >\n> > > > > > > - - AeState:\n> > > > > > > - type: int32_t\n> > > > > > > - description: |\n> > > > > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > > > > - ANDROID_CONTROL_AE_STATE.\n> > > > > > > -\n> > > > > > > - Current state of the AE algorithm.\n> > > > > > > - enum:\n> > > > > > > - - name: AeStateInactive\n> > > > > > > - value: 0\n> > > > > > > - description: The AE algorithm is inactive.\n> > > > > > > - - name: AeStateSearching\n> > > > > > > - value: 1\n> > > > > > > - description: The AE algorithm has not converged yet.\n> > > > > > > - - name: AeStateConverged\n> > > > > > > - value: 2\n> > > > > > > - description: The AE algorithm has converged.\n> > > > > > > - - name: AeStateLocked\n> > > > > > > - value: 3\n> > > > > > > - description: The AE algorithm is locked.\n> > > > > > > - - name: AeStateFlashRequired\n> > > > > > > - value: 4\n> > > > > > > - description: The AE algorithm would need a flash for good results\n> > > > > > > - - name: AeStatePrecapture\n> > > > > > > - value: 5\n> > > > > > > - description: |\n> > > > > > > - The AE algorithm has started a pre-capture metering session.\n> > > > > > > - \\sa AePrecaptureTrigger\n> > > > > > > -\n> > > > > > > - AwbState:\n> > > > > > > type: int32_t\n> > > > > > > description: |","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C20F6C32F9\n\tfor ;\n\tThu, 21 Nov 2024 02:28:14 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1B7BD65F96;\n\tThu, 21 Nov 2024 03:28:14 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A8C9B65F54\n\tfor ;\n\tThu, 21 Nov 2024 03:28:12 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id AEF96670;\n\tThu, 21 Nov 2024 03:27:53 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"NtiQRyiS\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732156073;\n\tbh=rRHx3yFC1uLuoKj/ryB0b/pdBLAo8caaQvXmYxfEddU=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=NtiQRyiS+igEcefQdvAqFO8w/kfLIItxNH++FyfN6a+IWAXrMBgzs4AlD6SYqiMG3\n\ty077Q40lIDbo7+vEMEWfRYupfMbEwVe5lLubohVFVs/Wz5zx593eKeqig5+5pjZ1nD\n\t3q/npQh7odp2dO/s+5e+iIyW8JNbGnVsz3x2fkwg=","Date":"Thu, 21 Nov 2024 04:28:02 +0200","From":"Laurent Pinchart ","To":"Paul Elder ","Cc":"Naushir Patuck ,\n\tStefan Klug ,\n\tDavid Plowman ,\n\tlibcamera-devel@lists.libcamera.org, jacopo.mondi@ideasonboard.com,\n\tJacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"<20241121022802.GA12409@pendragon.ideasonboard.com>","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32319,"web_url":"https://patchwork.libcamera.org/comment/32319/","msgid":"<20241121030034.GB12409@pendragon.ideasonboard.com>","date":"2024-11-21T03:00:34","subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi David,\n\nOn Tue, Nov 19, 2024 at 01:44:13PM +0000, David Plowman wrote:\n> On Mon, 18 Nov 2024 at 14:43, Jacopo Mondi wrote:\n> > On Mon, Nov 18, 2024 at 02:29:15PM +0000, David Plowman wrote:\n> > > Hi Paul\n> > >\n> > > I would just like to make sure we're looking after some of our (many)\n> > > less-expert Raspberry Pi users, and don't want to cause code they're\n> > > using to break with minimal warning or explanation. (These are not\n> > > people who read libcamera mailing lists!)\n> >\n> > One clarification here: are you concerned about pi users that has\n> > custom applications interfacing directly with libcamera (either\n> > through the C++ or Python APIs) ?\n> \n> I'm not particularly worried about anyone using C++. It's the Python\n> users, who may be using simple scripts they've written or been given,\n> who could get confused and end up clogging the Raspberry Pi forums!\n> \n> > > For example, it could be helpful to us to maintain some degree of\n> > > backward compatibility. Maybe switching to \"manual mode\" automatically\n> > > would help. As I commented previously, AF lens position works like\n> > > this, so it would be more consistent.\n> > >\n> > > I understand that folks don't like the \"zero means auto\" value, on the\n> > > other hand I'm not totally persuaded that breaking a single control\n> > > into two, now with the risk of a race hazard, is so very much better.\n> > > Maybe we could have a switchover period where the old method works but\n> > > issues a warning?\n> > >\n> > > Another thought would be for us to \"translate\" control lists from the\n> > > older API to the newer one at runtime. It's hardly a big deal, but\n> > > feels a teeny weeny bit inelegant!\n> >\n> > Because this seems to suggest you could handle the translation, I\n> > though you were considering doing that in your applications, but\n> > now I wonder if you're suggesting to do that in the IPA\n> \n> Yes, I can handle the translation for Python users easily enough. It's\n> not lovely, but I think that's probably the least bad alternative for\n> us.\n> \n> Sorry if I've been sounding a bit grumpy about this!!\n\nYou haven't, quite the contrary. Thanks for your constructive input in\nthis discussion.\n\nWe don't want to make life more difficult than it needs to be for any\nuser, and neither do we want to turn your support forums into living\nhell for you. We could have a transition period where the RPi IPA module\nperforms the translation, but handling it on the Python side could be\nbetter. How long would you expect that transition period to last ?\n\n> > > On Fri, 15 Nov 2024 at 13:41, Paul Elder wrote:\n> > > > On Thu, Nov 14, 2024 at 03:16:20PM +0100, Stefan Klug wrote:\n> > > > > On Wed, Nov 13, 2024 at 02:18:26PM +0000, David Plowman wrote:\n> > > > > Hi Paul,\n> > > > >\n> > > > > Thank you for the patch.\n> > > > >\n> > > > > > Hi Paul\n> > > > > >\n> > > > > > Thanks for posting all this!\n> > > > > >\n> > > > > > On Wed, 13 Nov 2024 at 13:13, Paul Elder wrote:\n> > > > > > >\n> > > > > > > Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls\n> > > > > > > to model the AEGC algorithm block.\n> > > > > > >\n> > > > > > > The three controls allow applications to select the exposure time and\n> > > > > > > analogue gain computation calculation mode (auto vs manual)\n> > > > > > > independently from one another, while the AeState control reports the\n> > > > > > > global state for the AEGC algorithm.\n> > > > > > >\n> > > > > > > The new controls are meant to replace the existing AeEnable and AeLocked\n> > > > > > > controls, which are momentarily kept not to break compilation of\n> > > > > > > platforms making use of them.\n> > > > > > >\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=42\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=43\n> > > > > > > Bug: https://bugs.libcamera.org/show_bug.cgi?id=47\n> > > > > > > Signed-off-by: Paul Elder \n> > > > > > > Signed-off-by: Jacopo Mondi \n> > > > > > >\n> > > > > > > ---\n> > > > > > > Changes in v3:\n> > > > > > > - recovered from 2-year-old bitrot\n> > > > > >\n> > > > > > Indeed, I think I remember discussing all this _years_ ago!!\n> > > > > >\n> > > > > > > ---\n> > > > > > > src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++---\n> > > > > > > src/libcamera/control_ids_draft.yaml | 29 ----\n> > > > > > > 2 files changed, 223 insertions(+), 49 deletions(-)\n> > > > > > >\n> > > > > > > diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml\n> > > > > > > index 1b1bd9507d25..98ef0736aa1b 100644\n> > > > > > > --- a/src/libcamera/control_ids_core.yaml\n> > > > > > > +++ b/src/libcamera/control_ids_core.yaml\n> > > > > > > @@ -26,6 +26,75 @@ controls:\n> > > > > > >\n> > > > > > > \\sa AeEnable\n> > > > > > >\n> > > > > > > + - AeState:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Control to report the AEGC algorithm state.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm computes the exposure time and the analogue gain\n> > > > > > > + values to be applied to the image sensor.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and\n> > > > > > > + AnalogueGainMode controls, which allow applications to decide how\n> > > > > > > + the exposure time and gain are computed, in Auto or Manual mode,\n> > > > > > > + independently from one another.\n> > > > > > > +\n> > > > > > > + The AeState control reports the AEGC algorithm state through a single\n> > > > > > > + value and describes it as a single computation block which computes\n> > > > > > > + both the exposure time and the analogue gain values.\n> > > > > > > +\n> > > > > > > + When both the exposure time and analogue gain values are configured to\n> > > > > > > + be in Manual mode, the AEGC algorithm is quiescent and does not actively\n> > > > > > > + compute any value and the AeState control will report AeStateIdle.\n> > > > > > > +\n> > > > > > > + When at least the exposure time or analogue gain are configured to be\n> > > > > > > + computed by the AEGC algorithm, the AeState control will report if the\n> > > > > > > + algorithm has converged to stable values for all of the controls set\n> > > > > > > + to be computed in Auto mode.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > + enum:\n> > > > > > > + - name: AeStateIdle\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm is inactive.\n> > > > > > > +\n> > > > > > > + This state is returned when both AnalogueGainMode and\n> > > > > > > + ExposureTimeMode are set to Manual and the algorithm is not\n> > > > > > > + actively computing any value.\n> > > > > > > +\n> > > > > > > + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the\n> > > > > > > + AEGC algorithm might spontaneously initiate a new scan, in which\n> > > > > >\n> > > > > > Maybe \"might spontaneously start adjusting again\"? The word \"scan\"\n> > > > > > isn't really wrong, just sounds a bit like it was copy-pasted from an\n> > > > > > autofocus description! But I'm not too bothered.\n> > > > > >\n> > > > > > > + case the AeState control is moved to AeStateSearching.\n> > > > > > > + - name: AeStateSearching\n> > > > >\n> > > > > A bit of bikeshedding here. Somehow \"searching\" sound strange to me (may\n> > > > > well be because I'm not a native speaker). How does \"regulating\" or\n> > > > > \"controlling\" sound to native ears? In my mind a search doesn't really\n> > > > > converge, but a regulation does.\n> > > >\n> > > > Hm to be \"regulating\" and \"controlling\" aren't the right words either...\n> > > > Searching I think works fine because you're searching in a numerical\n> > > > space for the value that maximizes the score.\n> > > >\n> > > > >\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm is actively computing new values, for either the\n> > > > > > > + exposure time or the analogue gain, but has not converged to a\n> > > > > > > + stable result yet.\n> > > > > > > +\n> > > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > > + or ExposureTimeMode is set to auto and the algorithm hasn't\n> > > > > > > + converged yet.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm converges once stable values are computed for\n> > > > > > > + any of the controls set to be computed in Auto mode.\n> > > > >\n> > > > > Should that be \"all of the\"?\n> > > >\n> > > > Oh yeah it should be.\n> > > >\n> > > > >\n> > > > > > > +\n> > > > > > > + Once the algorithm converges the state is moved to AeStateConverged.\n> > > > > > > + - name: AeStateConverged\n> > > > > > > + value: 2\n> > > > > > > + description: |\n> > > > > > > + The AEGC algorithm has converged.\n> > > > > > > +\n> > > > > > > + This state is returned if at least one of AnalogueGainMode\n> > > > > > > + or ExposureTimeMode is set to Auto, and the AEGC algorithm has\n> > > > > > > + converged to stable value.\n> > > > > > > +\n> > > > > > > + The AEGC algorithm might spontaneously re-initiate an AE scan, in\n> > > > > > > + which case the state is moved to AeStateSearching.\n> > > > >\n> > > > > This sounds like it is based on a random event. What about something\n> > > > > like: When the statistics/measurements move too far away from the\n> > > > > convergence point the AEGC algo automatically restarts\n> > > > > regulation/scanning...\n> > > >\n> > > > In a way I think it's spontaneous to the user :)\n> > > >\n> > > > But indeed your description would be more accurate.\n> > > >\n> > > > >\n> > > > > > > +\n> > > > > > > # AeMeteringMode needs further attention:\n> > > > > > > # - Auto-generate max enum value.\n> > > > > > > # - Better handling of custom types.\n> > > > > > > @@ -104,6 +173,13 @@ controls:\n> > > > > > > The exposure modes specify how the desired total exposure is divided\n> > > > > > > between the shutter time and the sensor's analogue gain. They are\n> > > > > > > platform specific, and not all exposure modes may be supported.\n> > > > > > > +\n> > > > > > > + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,\n> > > > > > > + the fixed values will override any choices made by AeExposureMode.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > enum:\n> > > > > > > - name: ExposureNormal\n> > > > > > > value: 0\n> > > > > > > @@ -124,13 +200,15 @@ controls:\n> > > > > > > Specify an Exposure Value (EV) parameter.\n> > > > > > >\n> > > > > > > The EV parameter will only be applied if the AE algorithm is currently\n> > > > > > > - enabled.\n> > > > > > > + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode\n> > > > > > > + are in Auto mode.\n> > > > > > >\n> > > > > > > By convention EV adjusts the exposure as log2. For example\n> > > > > > > EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment\n> > > > > > > of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].\n> > > > > > >\n> > > > > > > - \\sa AeEnable\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > >\n> > > > > > > - ExposureTime:\n> > > > > > > type: int32_t\n> > > > > > > @@ -140,17 +218,95 @@ controls:\n> > > > > > >\n> > > > > > > This value is specified in micro-seconds.\n> > > > > > >\n> > > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > > - AE algorithm.\n> > > > > > > + This control will only take effect if ExposureTimeMode is Manual. If\n> > > > > > > + this control is set when ExposureTimeMode is Auto, the value will be\n> > > > > > > + ignored and will not be retained.\n> > > > > >\n> > > > > > I'm not sure about this, it seems like a change of API behaviour which\n> > > > > > will require users to be re-educated and application code to be hunted\n> > > > > > down and changed. Previously you could just set the exposure time to X\n> > > > > > ms, and it \"automatically went to manual mode\". Auto focus behaves the\n> > > > > > same, I think, with respect to setting the lens position. It's just\n> > > > > > more convenient.\n> > > > >\n> > > > > This is an interesting discussion. I wasn't aware of the old\n> > > > > specification. I see that just setting one value and deducing what the\n> > > > > user wants is convenient. On the other hand one control changing the\n> > > > > value of another control feels quite difficult to explain and in my\n> > > > > opinion should be prevented whenever possible. I believe it is easier to\n> > > > > educate the user and have an explicit interface than to have hidden\n> > > > > magic whenever possible.\n> > > >\n> > > > Yeah that was my thought process too when I designed this.\n> > > >\n> > > >\n> > > > Thanks,\n> > > >\n> > > > Paul\n> > > >\n> > > > >\n> > > > > >\n> > > > > > [There's also the question of how confident you are that pipeline\n> > > > > > handlers correctly deal with two controls like this that are\n> > > > > > effectively ordered...]\n> > > > > >\n> > > > > > > +\n> > > > > > > + When reported in metadata, this control indicates what exposure time\n> > > > > > > + was used for the current request, regardless of ExposureTimeMode.\n> > > > > > > + ExposureTimeMode will indicate the source of the exposure time value,\n> > > > > > > + whether it came from the AE algorithm or not.\n> > > > > > > +\n> > > > > > > + \\sa AnalogueGain\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > +\n> > > > > > > + - ExposureTimeMode:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Controls the source of the exposure time that is applied to the image\n> > > > > > > + sensor. When set to Auto, the AE algorithm computes the exposure time\n> > > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > > + the value of the ExposureTime control is used.\n> > > > > > > +\n> > > > > > > + When transitioning from Auto to Manual mode and no ExposureTime control\n> > > > > > > + is provided by the application, the last value computed by the AE\n> > > > > > > + algorithm when the mode was Auto will be used. If the ExposureTimeMode\n> > > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > > + best-effort default value.\n> > > > > >\n> > > > > > Yes, I think this sounds OK. So setting it to manual mode is what\n> > > > > > setting AeEnable to zero used to do, except that it can be done\n> > > > > > separately for exposure time and gain now.\n> > > > > >\n> > > > > > > +\n> > > > > > > + If ExposureTimeModeManual is supported, the ExposureTime control must\n> > > > > > > + also be supported.\n> > > > > > > +\n> > > > > > > + The set of ExposureTimeMode modes that are supported by the camera must\n> > > > > > > + have an intersection with the supported set of AnalogueGainMode modes.\n> > > > > >\n> > > > > > That had me scratching my head for a moment! Maybe\n> > > > > >\n> > > > > > \"At least one of the modes, auto or manual, must be supported by both\n> > > > > > ExposureTimeMode and AnalogueGainMode.\"\n> > > > > >\n> > > > > > Is that clearer? (Also, is it what you meant?)\n> > > > > >\n> > > > > > > +\n> > > > > > > + Flickerless exposure mode transitions\n> > > > > > > +\n> > > > > > > + Applications that transition from ExposureTimeModeAuto to the direct\n> > > > > > > + control of the exposure time should aim to do so by selecting an\n> > > > > > > + ExposureTime value as close as possible to the last value computed by\n> > > > > > > + the auto exposure algorithm in order to avoid any visible flickering.\n> > > > > >\n> > > > > > Sounds a bit like you _have_ to do this. Maybe\n> > > > > >\n> > > > > > \"Applications that wish to transition from ..Auto to direct control of\n> > > > > > the exposure time without causing extra flicker, should do so by\n> > > > > > selecting an ExposureTime value as close as possible to the last value\n> > > > > > computed by the auto exposure algorithm.\"\n> > > > > >\n> > > > > > Don't know if that's better or not.\n> > > > > >\n> > > > > > > +\n> > > > > > > + To select the correct value to use as ExposureTime value, applications\n> > > > > > > + should accommodate the natural delay in applying controls caused by the\n> > > > > > > + capture pipeline frame depth.\n> > > > > > > +\n> > > > > > > + When switching to manual exposure mode, . I'm struggling a bit with that wording!!applications should not\n> > > > > > > + immediately specify an ExposureTime value in the same request where\n> > > > > > > + ExposureTimeMode is set to Manual. They should instead wait for the\n> > > > > > > + first Request where ExposureTimeMode is reported as\n> > > > > > > + ExposureTimeModeManual in the Request metadata, and use the reported\n> > > > > > > + ExposureTime to populate the control value in the next Request to be\n> > > > > > > + queued to the Camera.\n> > > > > > > +\n> > > > > > > + The implementation of the auto-exposure algorithm should equally try to\n> > > > > > > + minimize flickering and when transitioning from manual exposure mode to\n> > > > > > > + auto exposure use the last value provided by the application as starting\n> > > > > > > + point.\n> > > > > > > +\n> > > > > > > + 1. Start with ExposureTimeMode set to Auto\n> > > > > > > +\n> > > > > > > + 2. Set ExposureTimeMode to Manual\n> > > > > > > +\n> > > > > > > + 3. Wait for the first completed request that has ExposureTimeMode\n> > > > > > > + set to Manual\n> > > > > > > +\n> > > > > > > + 4. Copy the value reported in ExposureTime into a new request, and\n> > > > > > > + submit it\n> > > > > > > +\n> > > > > > > + 5. Proceed to run manual exposure time\n> > > > > > > +\n> > > > > > > + \\sa ExposureTime\n> > > > > > > + enum:\n> > > > > > > + - name: ExposureTimeModeAuto\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The exposure time will be calculated automatically and set by the\n> > > > > > > + AE algorithm.\n> > > > > > >\n> > > > > > > - \\sa AnalogueGain AeEnable\n> > > > > > > + If ExposureTime is set while this mode is active, it will be\n> > > > > > > + ignored, and it will also not be retained.\n> > > > > > > + - name: ExposureTimeModeManual\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The exposure time will not be updated by the AE algorithm.\n> > > > > > >\n> > > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > > - better be handled through a separate AE mode control.\n> > > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > > + exposure value is used until a new value is specified through the\n> > > > > > > + ExposureTime control. If an ExposureTime value is specified in the\n> > > > > > > + same request where the ExposureTimeMode is changed from Auto to\n> > > > > > > + Manual, the provided ExposureTime is applied immediately.\n> > > > > >\n> > > > > > Did we say anywhere what happens when you go from manual back to auto?\n> > > > > > I'm assuming the adjustments restart from the manual values that you\n> > > > > > currently have.\n> > > > > >\n> > > > > > >\n> > > > > > > - AnalogueGain:\n> > > > > > > type: float\n> > > > > > > @@ -160,17 +316,64 @@ controls:\n> > > > > > > The value of the control specifies the gain multiplier applied to all\n> > > > > > > colour channels. This value cannot be lower than 1.0.\n> > > > > > >\n> > > > > > > - Setting this value means that it is now fixed and the AE algorithm may\n> > > > > > > - not change it. Setting it back to zero returns it to the control of the\n> > > > > > > - AE algorithm.\n> > > > > > > + This control will only take effect if AnalogueGainMode is Manual. If\n> > > > > > > + this control is set when AnalogueGainMode is Auto, the value will be\n> > > > > > > + ignored and will not be retained.\n> > > > > >\n> > > > > > As above. It would be user-friendly to switch automatically to manual, I think.\n> > > > > >\n> > > > > > > +\n> > > > > > > + When reported in metadata, this control indicates what analogue gain\n> > > > > > > + was used for the current request, regardless of AnalogueGainMode.\n> > > > > > > + AnalogueGainMode will indicate the source of the analogue gain value,\n> > > > > > > + whether it came from the AEGC algorithm or not.\n> > > > > > > +\n> > > > > > > + \\sa ExposureTime\n> > > > > > > + \\sa AnalogueGainMode\n> > > > > > > +\n> > > > > > > + - AnalogueGainMode:\n> > > > > > > + type: int32_t\n> > > > > > > + description: |\n> > > > > > > + Controls the source of the analogue gain that is applied to the image\n> > > > > > > + sensor. When set to Auto, the AEGC algorithm computes the analogue gain\n> > > > > > > + and configures the image sensor accordingly. When set to Manual,\n> > > > > > > + the value of the AnalogueGain control is used.\n> > > > > > > +\n> > > > > > > + When transitioning from Auto to Manual mode and no AnalogueGain control\n> > > > > > > + is provided by the application, the last value computed by the AEGC\n> > > > > > > + algorithm when the mode was Auto will be used. If the AnalogueGainMode\n> > > > > > > + was never set to Auto (either because the camera started in Manual mode,\n> > > > > > > + or Auto is not supported by the camera), the camera should use a\n> > > > > > > + best-effort default value.\n> > > > > > > +\n> > > > > > > + If AnalogueGainModeManual is supported, the AnalogueGain control must\n> > > > > > > + also be supported.\n> > > > > > > +\n> > > > > > > + The set of AnalogueGainMode modes that are . I'm struggling a bit with that wording!!supported by the camera must\n> > > > > > > + have an intersection with the supported set of ExposureTimeMode modes.\n> > > > > >\n> > > > > > As above!\n> > > > > >\n> > > > > > Thanks again\n> > > > > >\n> > > > > > David\n> > > > > >\n> > > > > > > +\n> > > > > > > + The same procedure described for performing flickerless transitions in\n> > > > > > > + the ExposureTimeMode control documentation should be applied to\n> > > > > > > + analogue gain.\n> > > > > > > +\n> > > > > > > + \\sa ExposureTimeMode\n> > > > > > > + \\sa AnalogueGain\n> > > > > > > + enum:\n> > > > > > > + - name: AnalogueGainModeAuto\n> > > > > > > + value: 0\n> > > > > > > + description: |\n> > > > > > > + The analogue gain will be calculated automatically and set by the\n> > > > > > > + AEGC algorithm.\n> > > > > > >\n> > > > > > > - \\sa ExposureTime AeEnable\n> > > > > > > + If AnalogueGain is set while this mode is active, it will be\n> > > > > > > + ignored, and it will also not be retained.\n> > > > > > > + - name: AnalogueGainModeManual\n> > > > > > > + value: 1\n> > > > > > > + description: |\n> > > > > > > + The analogue gain will not be updated by the AEGC algorithm.\n> > > > > > >\n> > > > > > > - \\todo Document the interactions between AeEnable and setting a fixed\n> > > > > > > - value for this control. Consider interactions with other AE features,\n> > > > > > > - such as aperture and aperture/shutter priority mode, and decide if\n> > > > > > > - control of which features should be automatically adjusted shouldn't\n> > > > > > > - better be handled through a separate AE mode control.\n> > > > > > > + When transitioning from Auto to Manual mode, the last computed\n> > > > > > > + gain value is used until a new value is specified through the\n> > > > > > > + AnalogueGain control. If an AnalogueGain value is specified in the\n> > > > > > > + same request where the AnalogueGainMode is changed from Auto to\n> > > > > > > + Manual, the provided AnalogueGain is applied immediately.\n> > > > > > >\n> > > > > > > - AeFlickerMode:\n> > > > > > > type: int32_t\n> > > > > > > diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml\n> > > > > > > index 1b284257f601..32c2b3feaf65 100644\n> > > > > > > --- a/src/libcamera/control_ids_draft.yaml\n> > > > > > > +++ b/src/libcamera/control_ids_draft.yaml\n> > > > > > > @@ -77,35 +77,6 @@ controls:\n> > > > > > > High quality aberration correction which might reduce the frame\n> > > > > > > rate.\n> > > > > > >\n> > > > > > > - - AeState:\n> > > > > > > - type: int32_t\n> > > > > > > - description: |\n> > > > > > > - Control to report the current AE algorithm state. Currently identical to\n> > > > > > > - ANDROID_CONTROL_AE_STATE.\n> > > > > > > -\n> > > > > > > - Current state of the AE algorithm.\n> > > > > > > - enum:\n> > > > > > > - - name: AeStateInactive\n> > > > > > > - value: 0\n> > > > > > > - description: The AE algorithm is inactive.\n> > > > > > > - - name: AeStateSearching\n> > > > > > > - value: 1\n> > > > > > > - description: The AE algorithm has not converged yet.\n> > > > > > > - - name: AeStateConverged\n> > > > > > > - value: 2\n> > > > > > > - description: The AE algorithm has converged.\n> > > > > > > - - name: AeStateLocked\n> > > > > > > - value: 3\n> > > > > > > - description: The AE algorithm is locked.\n> > > > > > > - - name: AeStateFlashRequired\n> > > > > > > - value: 4\n> > > > > > > - description: The AE algorithm would need a flash for good results\n> > > > > > > - - name: AeStatePrecapture\n> > > > > > > - value: 5\n> > > > > > > - description: |\n> > > > > > > - The AE algorithm has started a pre-capture metering session.\n> > > > > > > - \\sa AePrecaptureTrigger\n> > > > > > > -\n> > > > > > > - AwbState:\n> > > > > > > type: int32_t\n> > > > > > > description: |\n> > > > > > > --\n> > > > > > > 2.39.2\n> > > > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E8ABEC32F5\n\tfor ;\n\tThu, 21 Nov 2024 03:00:46 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0FB4965F9A;\n\tThu, 21 Nov 2024 04:00:46 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 63D3865F54\n\tfor ;\n\tThu, 21 Nov 2024 04:00:44 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 3206B670;\n\tThu, 21 Nov 2024 04:00:25 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"TW5I+aJn\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1732158025;\n\tbh=JMqqnPOCWjZRrXeH4j0GZfHKqkh2LSPgxoqf2pzXI2I=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=TW5I+aJnDpoqGCpCCpnSlC5BWdXvERLhPQHZ4Q+/W+xti9vDZcMEib/8PzSCNbCSn\n\tNUasBawtnh0BgTsaeNj2Nv/088Ce8yx/aHsvupCHbYsVYr1GXRB9kCZdVE1890r48i\n\t2U2oks3wYuFEV2UaYIi364SddTFCbcHQQ9KVkHCA=","Date":"Thu, 21 Nov 2024 05:00:34 +0200","From":"Laurent Pinchart ","To":"David Plowman ","Cc":"Jacopo Mondi ,\n\tPaul Elder ,\n\tStefan Klug ,\n\tlibcamera-devel@lists.libcamera.org, naush@raspberrypi.com,\n\tJacopo Mondi ","Subject":"Re: [PATCH v3 1/8] controls: Introduce AEGC-related controls","Message-ID":"<20241121030034.GB12409@pendragon.ideasonboard.com>","References":"<20241113131256.3170817-1-paul.elder@ideasonboard.com>\n\t<20241113131256.3170817-2-paul.elder@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]