From patchwork Wed Nov 13 13:12:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 21884 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3F066BE173 for ; Wed, 13 Nov 2024 13:13:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CF0A66581A; Wed, 13 Nov 2024 14:13:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pRLMndBG"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A89B5657CF for ; Wed, 13 Nov 2024 14:13:20 +0100 (CET) Received: from neptunite.flets-east.jp (unknown [IPv6:2404:7a81:160:2100:eb25:7aa0:7da7:eb24]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5A0021054; Wed, 13 Nov 2024 14:13:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1731503587; bh=0crekWeiI+7OO3f+VEASj+1KuSHPD3P4S9rF18zAkI8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pRLMndBGUmbAitU43kytvpZfATPIGVuQif9IZYoSY//QrppIYtErOOuHiWE2b+MS4 9m/kVeNcun/Sces4hAgbGmr2enCTjDVWUh3tHC/xB0yzOZd7suaYQQH4HBgwIMwTc6 2aUOSXFQetTlhI67LbPAl1FDMW2coizKo88a6RIA= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: laurent.pinchart@ideasonboard.com, jacopo.mondi@ideasonboard.com, naush@raspberrypi.com, david.plowman@raspberrypi.com, Paul Elder , Jacopo Mondi Subject: [PATCH v3 1/8] controls: Introduce AEGC-related controls Date: Wed, 13 Nov 2024 22:12:49 +0900 Message-Id: <20241113131256.3170817-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241113131256.3170817-1-paul.elder@ideasonboard.com> References: <20241113131256.3170817-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls to model the AEGC algorithm block. The three controls allow applications to select the exposure time and analogue gain computation calculation mode (auto vs manual) independently from one another, while the AeState control reports the global state for the AEGC algorithm. The new controls are meant to replace the existing AeEnable and AeLocked controls, which are momentarily kept not to break compilation of platforms making use of them. Bug: https://bugs.libcamera.org/show_bug.cgi?id=42 Bug: https://bugs.libcamera.org/show_bug.cgi?id=43 Bug: https://bugs.libcamera.org/show_bug.cgi?id=47 Signed-off-by: Paul Elder Signed-off-by: Jacopo Mondi --- Changes in v3: - recovered from 2-year-old bitrot --- src/libcamera/control_ids_core.yaml | 243 ++++++++++++++++++++++++--- src/libcamera/control_ids_draft.yaml | 29 ---- 2 files changed, 223 insertions(+), 49 deletions(-) diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml index 1b1bd9507d25..98ef0736aa1b 100644 --- a/src/libcamera/control_ids_core.yaml +++ b/src/libcamera/control_ids_core.yaml @@ -26,6 +26,75 @@ controls: \sa AeEnable + - AeState: + type: int32_t + description: | + Control to report the AEGC algorithm state. + + The AEGC algorithm computes the exposure time and the analogue gain + values to be applied to the image sensor. + + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and + AnalogueGainMode controls, which allow applications to decide how + the exposure time and gain are computed, in Auto or Manual mode, + independently from one another. + + The AeState control reports the AEGC algorithm state through a single + value and describes it as a single computation block which computes + both the exposure time and the analogue gain values. + + When both the exposure time and analogue gain values are configured to + be in Manual mode, the AEGC algorithm is quiescent and does not actively + compute any value and the AeState control will report AeStateIdle. + + When at least the exposure time or analogue gain are configured to be + computed by the AEGC algorithm, the AeState control will report if the + algorithm has converged to stable values for all of the controls set + to be computed in Auto mode. + + \sa AnalogueGainMode + \sa ExposureTimeMode + + enum: + - name: AeStateIdle + value: 0 + description: | + The AEGC algorithm is inactive. + + This state is returned when both AnalogueGainMode and + ExposureTimeMode are set to Manual and the algorithm is not + actively computing any value. + + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the + AEGC algorithm might spontaneously initiate a new scan, in which + case the AeState control is moved to AeStateSearching. + - name: AeStateSearching + value: 1 + description: | + The AEGC algorithm is actively computing new values, for either the + exposure time or the analogue gain, but has not converged to a + stable result yet. + + This state is returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to auto and the algorithm hasn't + converged yet. + + The AEGC algorithm converges once stable values are computed for + any of the controls set to be computed in Auto mode. + + Once the algorithm converges the state is moved to AeStateConverged. + - name: AeStateConverged + value: 2 + description: | + The AEGC algorithm has converged. + + This state is returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to Auto, and the AEGC algorithm has + converged to stable value. + + The AEGC algorithm might spontaneously re-initiate an AE scan, in + which case the state is moved to AeStateSearching. + # AeMeteringMode needs further attention: # - Auto-generate max enum value. # - Better handling of custom types. @@ -104,6 +173,13 @@ controls: The exposure modes specify how the desired total exposure is divided between the shutter time and the sensor's analogue gain. They are platform specific, and not all exposure modes may be supported. + + When one of AnalogueGainMode or ExposureTimeMode is set to Manual, + the fixed values will override any choices made by AeExposureMode. + + \sa AnalogueGainMode + \sa ExposureTimeMode + enum: - name: ExposureNormal value: 0 @@ -124,13 +200,15 @@ controls: Specify an Exposure Value (EV) parameter. The EV parameter will only be applied if the AE algorithm is currently - enabled. + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode + are in Auto mode. By convention EV adjusts the exposure as log2. For example EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x]. - \sa AeEnable + \sa AnalogueGainMode + \sa ExposureTimeMode - ExposureTime: type: int32_t @@ -140,17 +218,95 @@ controls: This value is specified in micro-seconds. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if ExposureTimeMode is Manual. If + this control is set when ExposureTimeMode is Auto, the value will be + ignored and will not be retained. + + When reported in metadata, this control indicates what exposure time + was used for the current request, regardless of ExposureTimeMode. + ExposureTimeMode will indicate the source of the exposure time value, + whether it came from the AE algorithm or not. + + \sa AnalogueGain + \sa ExposureTimeMode + + - ExposureTimeMode: + type: int32_t + description: | + Controls the source of the exposure time that is applied to the image + sensor. When set to Auto, the AE algorithm computes the exposure time + and configures the image sensor accordingly. When set to Manual, + the value of the ExposureTime control is used. + + When transitioning from Auto to Manual mode and no ExposureTime control + is provided by the application, the last value computed by the AE + algorithm when the mode was Auto will be used. If the ExposureTimeMode + was never set to Auto (either because the camera started in Manual mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + If ExposureTimeModeManual is supported, the ExposureTime control must + also be supported. + + The set of ExposureTimeMode modes that are supported by the camera must + have an intersection with the supported set of AnalogueGainMode modes. + + Flickerless exposure mode transitions + + Applications that transition from ExposureTimeModeAuto to the direct + control of the exposure time should aim to do so by selecting an + ExposureTime value as close as possible to the last value computed by + the auto exposure algorithm in order to avoid any visible flickering. + + To select the correct value to use as ExposureTime value, applications + should accommodate the natural delay in applying controls caused by the + capture pipeline frame depth. + + When switching to manual exposure mode, applications should not + immediately specify an ExposureTime value in the same request where + ExposureTimeMode is set to Manual. They should instead wait for the + first Request where ExposureTimeMode is reported as + ExposureTimeModeManual in the Request metadata, and use the reported + ExposureTime to populate the control value in the next Request to be + queued to the Camera. + + The implementation of the auto-exposure algorithm should equally try to + minimize flickering and when transitioning from manual exposure mode to + auto exposure use the last value provided by the application as starting + point. + + 1. Start with ExposureTimeMode set to Auto + + 2. Set ExposureTimeMode to Manual + + 3. Wait for the first completed request that has ExposureTimeMode + set to Manual + + 4. Copy the value reported in ExposureTime into a new request, and + submit it + + 5. Proceed to run manual exposure time + + \sa ExposureTime + enum: + - name: ExposureTimeModeAuto + value: 0 + description: | + The exposure time will be calculated automatically and set by the + AE algorithm. - \sa AnalogueGain AeEnable + If ExposureTime is set while this mode is active, it will be + ignored, and it will also not be retained. + - name: ExposureTimeModeManual + value: 1 + description: | + The exposure time will not be updated by the AE algorithm. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + When transitioning from Auto to Manual mode, the last computed + exposure value is used until a new value is specified through the + ExposureTime control. If an ExposureTime value is specified in the + same request where the ExposureTimeMode is changed from Auto to + Manual, the provided ExposureTime is applied immediately. - AnalogueGain: type: float @@ -160,17 +316,64 @@ controls: The value of the control specifies the gain multiplier applied to all colour channels. This value cannot be lower than 1.0. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if AnalogueGainMode is Manual. If + this control is set when AnalogueGainMode is Auto, the value will be + ignored and will not be retained. + + When reported in metadata, this control indicates what analogue gain + was used for the current request, regardless of AnalogueGainMode. + AnalogueGainMode will indicate the source of the analogue gain value, + whether it came from the AEGC algorithm or not. + + \sa ExposureTime + \sa AnalogueGainMode + + - AnalogueGainMode: + type: int32_t + description: | + Controls the source of the analogue gain that is applied to the image + sensor. When set to Auto, the AEGC algorithm computes the analogue gain + and configures the image sensor accordingly. When set to Manual, + the value of the AnalogueGain control is used. + + When transitioning from Auto to Manual mode and no AnalogueGain control + is provided by the application, the last value computed by the AEGC + algorithm when the mode was Auto will be used. If the AnalogueGainMode + was never set to Auto (either because the camera started in Manual mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + If AnalogueGainModeManual is supported, the AnalogueGain control must + also be supported. + + The set of AnalogueGainMode modes that are supported by the camera must + have an intersection with the supported set of ExposureTimeMode modes. + + The same procedure described for performing flickerless transitions in + the ExposureTimeMode control documentation should be applied to + analogue gain. + + \sa ExposureTimeMode + \sa AnalogueGain + enum: + - name: AnalogueGainModeAuto + value: 0 + description: | + The analogue gain will be calculated automatically and set by the + AEGC algorithm. - \sa ExposureTime AeEnable + If AnalogueGain is set while this mode is active, it will be + ignored, and it will also not be retained. + - name: AnalogueGainModeManual + value: 1 + description: | + The analogue gain will not be updated by the AEGC algorithm. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + When transitioning from Auto to Manual mode, the last computed + gain value is used until a new value is specified through the + AnalogueGain control. If an AnalogueGain value is specified in the + same request where the AnalogueGainMode is changed from Auto to + Manual, the provided AnalogueGain is applied immediately. - AeFlickerMode: type: int32_t diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml index 1b284257f601..32c2b3feaf65 100644 --- a/src/libcamera/control_ids_draft.yaml +++ b/src/libcamera/control_ids_draft.yaml @@ -77,35 +77,6 @@ controls: High quality aberration correction which might reduce the frame rate. - - AeState: - type: int32_t - description: | - Control to report the current AE algorithm state. Currently identical to - ANDROID_CONTROL_AE_STATE. - - Current state of the AE algorithm. - enum: - - name: AeStateInactive - value: 0 - description: The AE algorithm is inactive. - - name: AeStateSearching - value: 1 - description: The AE algorithm has not converged yet. - - name: AeStateConverged - value: 2 - description: The AE algorithm has converged. - - name: AeStateLocked - value: 3 - description: The AE algorithm is locked. - - name: AeStateFlashRequired - value: 4 - description: The AE algorithm would need a flash for good results - - name: AeStatePrecapture - value: 5 - description: | - The AE algorithm has started a pre-capture metering session. - \sa AePrecaptureTrigger - - AwbState: type: int32_t description: |