From patchwork Wed May 18 13:47:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 15970 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E2464C3256 for ; Wed, 18 May 2022 13:47:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 995F965659; Wed, 18 May 2022 15:47:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652881668; bh=MURPECdij0YNP5QMgjT3RzzshKDOJhKkkHRBQlRxJ1Y=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ySvnytWjHBi4jFAUxOhYRQ1jnXnwJEruu6ssLleGgupZIEr84NKCEwtn9OOTJUCUn ePihpA5588db7+xNtglxtT3bxHimCntKyTWTRZeo2lK2UeF8ZZHtRc2MfH4Zbv94zE RX1IeKaxKnwr3VUKVkDmy8GXRo3qb3hxPjYcRXAuaH1VvR6Ut4/h5gdd9Tnwn+F/QK wUCBFEte+EbQg5VMPnxfGAKBRVGd9o5s8rwyeXSWJMuA28NoAoQP+nr+meca/4uUPG a+0ao31OfDUnrQJRLP5ciXdpf98utPUiEdMNrnl0MJhudTQdMGrlNdkKXHOoKlW3zL PbDCpdBJsAYdA== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 60C7665657 for ; Wed, 18 May 2022 15:47:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rc6jZQRA"; dkim-atps=neutral Received: from localhost.localdomain (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E402B1027; Wed, 18 May 2022 15:47:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652881664; bh=MURPECdij0YNP5QMgjT3RzzshKDOJhKkkHRBQlRxJ1Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rc6jZQRAqzRi8kv0Mx9tBcI8LeX0HSYedaty2eUqnF0Dhi5A1qTYyuI/vIYEQZwno TGVFUjHqk2HkNQDg9rbNSeDCGQBAzzvJsRGDZTvZR74yJD1r3uqWWue5OjyQdKE9cd guLCx9cTYhWKRsfymnyPxGH2vuE0YVFHAn0/4N7o= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:47:26 +0200 Message-Id: <20220518134728.777709-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220518134728.777709-1-paul.elder@ideasonboard.com> References: <20220518134728.777709-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 1/3] controls: Reorganize the AE-related controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Paul Elder via libcamera-devel From: Paul Elder Reply-To: Paul Elder Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We have multiple goals: - we need a lock of some sort, to instruct the AEGC to not update output results - we need manual modes, to override the values computed by the AEGC - we need to support seamless transitions from auto -> manual, and do so without flickering - we need custom minimum values for the manual controls, that is no magic values for enabling/disabling auto - all of these need to be done with AE sub-controls (exposure time, analogue gain) To achieve these goals, we introduce mode controls for the AE sub-controls: ExposureTimeMode and AnalogueGainMode. These have an auto state, and a disabled state. The disabled state has an internal one-way state change from locked to manual, triggered by the presence of the value-controls (ExposureTime and AnalogueGain). We then remove the AeEnable control, as it is a redundant control in the face of these two mode controls. We also remove AeLocked, as it is insufficient for reporting the AE state, and we promote AeState to non-draft to fill its role. Notably, the locked state is removed, since this information can be obtained from the aforementioned mode controls. Bug: https://bugs.libcamera.org/show_bug.cgi?id=42 Bug: https://bugs.libcamera.org/show_bug.cgi?id=43 Bug: https://bugs.libcamera.org/show_bug.cgi?id=47 Signed-off-by: Paul Elder --- Changes in v4: - remove FlashRequired and Precapture from AeState - upgrade documentation of all the controls Changes in v3: - improve wording of the control descriptions - make more succinct and clear - add description of how to do a flickerless transition Changes in v2: - No changes, just resubmitting at the head of this series so that it's together and so that /people will actually see it/ Initial version: Still RFC as I haven't updated the users of the control yet, and I want to check that these are the controls and docs that we want. We've decided that the "master AE control" will be implemented by a helper... but looking at uvcvideo and the V4L2 controls I'm wondering if such helper should come earlier than later? --- src/libcamera/control_ids.yaml | 262 +++++++++++++++++++++++++-------- 1 file changed, 200 insertions(+), 62 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 9d4638ae..9f5ce5e8 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -7,23 +7,46 @@ # Unless otherwise stated, all controls are bi-directional, i.e. they can be # set through Request::controls() and returned out through Request::metadata(). controls: - - AeEnable: - type: bool + - AeState: + type: int32_t description: | - Enable or disable the AE. + Control to report the AE algorithm state associated with the capture + result. - \sa ExposureTime AnalogueGain + The state is still reported even if ExposureTimeMode or + AnalogueGainMode is set to Disabled. - - AeLocked: - type: bool - description: | - Report the lock status of a running AE algorithm. + \sa AnalogueGain + \sa AnalogueGainMode + \sa ExposureTime + \sa ExposureTimeMode - If the AE algorithm is locked the value shall be set to true, if it's - converging it shall be set to false. If the AE algorithm is not - running the control shall not be present in the metadata control list. + enum: + - name: AeStateInactive + value: 0 + description: | + The AE algorithm is inactive. - \sa AeEnable + This state should be returned if both AnalogueGainMode and + ExposureTimeMode are set to disabled (or one, if the camera only + supports one of the two controls). + - name: AeStateSearching + value: 1 + description: | + The AE algorithm has not converged yet. + + This state should be returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to auto, and the AE algorithm hasn't + converged yet. If the AE algorithm converges, the state shall go to + AeStateConverged. + - name: AeStateConverged + value: 2 + description: | + The AE algorithm has converged. + + This state should be returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to auto, and the AE algorithm has + converged. # AeMeteringMode needs further attention: # - Auto-generate max enum value. @@ -93,6 +116,13 @@ controls: how the desired total exposure is divided between the shutter time and the sensor's analogue gain. The exposure modes are platform specific, and not all exposure modes may be supported. + + When one of AnalogueGainMode or ExposureTimeMode is set to Disabled, + the fixed values will override any choices made by AeExposureMode. + + \sa AnalogueGainMode + \sa ExposureTimeMode + enum: - name: ExposureNormal value: 0 @@ -111,13 +141,15 @@ controls: type: float description: | Specify an Exposure Value (EV) parameter. The EV parameter will only be - applied if the AE algorithm is currently enabled. + applied if the AE algorithm is currently enabled, that is, at least one + of AnalogueGainMode and ExposureTimeMode are auto. By convention EV adjusts the exposure as log2. For example EV = [-2, -1, 0.5, 0, 0.5, 1, 2] results in an exposure adjustment of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x]. - \sa AeEnable + \sa AnalogueGainMode + \sa ExposureTimeMode - ExposureTime: type: int32_t @@ -125,17 +157,85 @@ controls: Exposure time (shutter speed) for the frame applied in the sensor device. This value is specified in micro-seconds. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if ExposureTimeMode is Disabled. If + this control is set when ExposureTimeMode is Auto, the value will be + ignored and will not be retained. + + When reported in metadata, this control indicates what exposure time + was used for the current request, regardless of ExposureTimeMode. + ExposureTimeMode will indicate the source of the exposure time value, + whether it came from the AE algorithm or not. + + \sa AnalogueGain + \sa ExposureTimeMode + + - ExposureTimeMode: + type: int32_t + description: | + Controls the source of the exposure time that is applied to the image + sensor. When set to Auto, the AE algorithm computes the exposure time + and configures the image sensor accordingly. When set to Disabled, + exposure time specified in ExposureTime is applied to the image sensor. + If ExposureTime is not set, then the value last computed by the AE + algorithm when the mode was Auto will be used. + + If ExposureTime is not set and the mode is ExposureTimeModeDisabled and + AE was never Auto (either because the camera started in Disabled mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + When ExposureTimeMode is set Auto, the value set in ExposureTime is + ignored and is not retained. This means that if ExposureTimeMode is set + to Disabled and ExposureTime is not also set, the exposure time that + was last computed by the AE algorithm while the mode was Auto will be + applied to the sensor. + + If ExposureTimeModeDisabled is supported, the ExposureTime control must + also be supported. + + The set of ExposureTimeMode modes that are supported by the camera must + have an intersection with the supported set of AnalogueGainMode modes. - \sa AnalogueGain AeEnable + As it takes a few frames to apply the exposure time, there is a period of + time between submitting a request with ExposureTimeMode set to Disabled + and the exposure time component of the AE actually being disabled, + during which the AE algorithm can still update the exposure time. If an + application is switching from automatic and manual control and wishes + to eliminate any flicker during the switch, the following procedure is + recommended. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + 1. Start with ExposureTimeMode set to Auto + + 2. Set ExposureTimeMode to Disabled + + 3. Wait for the first request to be output that has ExposureTimeMode + set to Disabled + + 4. Copy the value reported in ExposureTime into a new request, and + submit it + + 5. Proceed to run manual exposure time + + \sa ExposureTime + enum: + - name: ExposureTimeModeAuto + value: 0 + description: | + The exposure time will be calculated automatically and set by the + AE algorithm. If ExposureTime is set while this mode is active, it + will be ignored, and it will also not be retained. + - name: ExposureTimeModeDisabled + value: 1 + description: | + The exposure time will not be updated by the AE algorithm. It will + come from the last calculated value when the mode was Auto, or from + the value specified in ExposureTime. + + When transitioning from Auto to Disabled mode, the last computed + exposure value is used until a new value is specified through the + ExposureTime control. If an ExposureTime value is specified in the + same request where the ExposureTimeMode is changed from Auto to + Disabled, the provided ExposureTime is applied. - AnalogueGain: type: float @@ -144,17 +244,85 @@ controls: The value of the control specifies the gain multiplier applied to all colour channels. This value cannot be lower than 1.0. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if AnalogueGainMode is Disabled. If + this control is set when AnalogueGainMode is Auto, the value will be + ignored and will not be retained. + + When reported in metadata, this control indicates what analogue gain + was used for the current request, regardless of AnalogueGainMode. + AnalogueGainMode will indicate the source of the analogue gain value, + whether it came from the AE algorithm or not. + + \sa ExposureTime + \sa AnalogueGainMode + + - AnalogueGainMode: + type: int32_t + description: | + Controls the source of the analogue gain that is applied to the image + sensor. When set to Auto, the AE algorithm computes the analogue gain + and configures the image sensor accordingly. When set to Disabled, + analogue gain specified in AnalogueGain is applied to the image sensor. + If AnalogueGain is not set, then the value last computed by the AE + algorithm when the mode was Auto will be used. + + If AnalogueGain is not set and the mode is AnalogueGainModeDisabled and + AE was never Auto (either because the camera started in Disabled mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + When AnalogueGainMode is set Auto, the value set in AnalogueGain is + ignored and is not retained. This means that if AnalogueGainMode is set + to Disabled and AnalogueGain is not also set, the analogue gain that + was last computed by the AE algorithm while the mode was Auto will be + applied to the sensor. - \sa ExposureTime AeEnable + If AnalogueGainModeDisabled is supported, the AnalogueGain control must + also be supported. + + The set of AnalogueGainMode modes that are supported by the camera must + have an intersection with the supported set of ExposureTimeMode modes. + + As it takes a few frames to apply the analogue gain, there is a period of + time between submitting a request with AnalogueGainMode set to Disabled + and the analogue gain component of the AE actually being disabled, + during which the AE algorithm can still update the analogue gain. If an + application is switching from automatic and manual control and wishes + to eliminate any flicker during the switch, the following procedure is + recommended. + + 1. Start with AnalogueGainMode set to Auto + + 2. Set AnalogueGainMode to Disabled + + 3. Wait for the first request to be output that has AnalogueGainMode + set to Disabled + + 4. Copy the value reported in AnalogueGain into a new request, and + submit it + + 5. Proceed to run manual analogue gain + + \sa AnalogueGain + enum: + - name: AnalogueGainModeAuto + value: 0 + description: | + The analogue gain will be calculated automatically and set by the + AE algorithm. If AnalogueGain is set while this mode is active, it + will be ignored, and it will also not be retained. + - name: AnalogueGainModeDisabled + value: 1 + description: | + The analogue gain will not be updated by the AE algorithm. It will + come from the last calculated value when the mode was Auto, or from + the value specified in AnalogueGain. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + When transitioning from Auto to Disabled mode the last computed + gain value is used until a new value is specified through the + AnalogueGain control. If an AnalogueGain value is specified in the + same request where the AnalogueGainMode is set to Disabled, the + provided AnalogueGain is applied. - Brightness: type: float @@ -477,36 +645,6 @@ controls: High quality aberration correction which might reduce the frame rate. - - AeState: - type: int32_t - draft: true - description: | - Control to report the current AE algorithm state. Currently identical to - ANDROID_CONTROL_AE_STATE. - - Current state of the AE algorithm. - enum: - - name: AeStateInactive - value: 0 - description: The AE algorithm is inactive. - - name: AeStateSearching - value: 1 - description: The AE algorithm has not converged yet. - - name: AeStateConverged - value: 2 - description: The AE algorithm has converged. - - name: AeStateLocked - value: 3 - description: The AE algorithm is locked. - - name: AeStateFlashRequired - value: 4 - description: The AE algorithm would need a flash for good results - - name: AeStatePrecapture - value: 5 - description: | - The AE algorithm has started a pre-capture metering session. - \sa AePrecaptureTrigger - - AfState: type: int32_t draft: true