From patchwork Wed May 18 13:47:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 15970 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E2464C3256 for ; Wed, 18 May 2022 13:47:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 995F965659; Wed, 18 May 2022 15:47:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652881668; bh=MURPECdij0YNP5QMgjT3RzzshKDOJhKkkHRBQlRxJ1Y=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ySvnytWjHBi4jFAUxOhYRQ1jnXnwJEruu6ssLleGgupZIEr84NKCEwtn9OOTJUCUn ePihpA5588db7+xNtglxtT3bxHimCntKyTWTRZeo2lK2UeF8ZZHtRc2MfH4Zbv94zE RX1IeKaxKnwr3VUKVkDmy8GXRo3qb3hxPjYcRXAuaH1VvR6Ut4/h5gdd9Tnwn+F/QK wUCBFEte+EbQg5VMPnxfGAKBRVGd9o5s8rwyeXSWJMuA28NoAoQP+nr+meca/4uUPG a+0ao31OfDUnrQJRLP5ciXdpf98utPUiEdMNrnl0MJhudTQdMGrlNdkKXHOoKlW3zL PbDCpdBJsAYdA== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 60C7665657 for ; Wed, 18 May 2022 15:47:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rc6jZQRA"; dkim-atps=neutral Received: from localhost.localdomain (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E402B1027; Wed, 18 May 2022 15:47:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652881664; bh=MURPECdij0YNP5QMgjT3RzzshKDOJhKkkHRBQlRxJ1Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rc6jZQRAqzRi8kv0Mx9tBcI8LeX0HSYedaty2eUqnF0Dhi5A1qTYyuI/vIYEQZwno TGVFUjHqk2HkNQDg9rbNSeDCGQBAzzvJsRGDZTvZR74yJD1r3uqWWue5OjyQdKE9cd guLCx9cTYhWKRsfymnyPxGH2vuE0YVFHAn0/4N7o= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:47:26 +0200 Message-Id: <20220518134728.777709-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220518134728.777709-1-paul.elder@ideasonboard.com> References: <20220518134728.777709-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 1/3] controls: Reorganize the AE-related controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Paul Elder via libcamera-devel From: Paul Elder Reply-To: Paul Elder Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We have multiple goals: - we need a lock of some sort, to instruct the AEGC to not update output results - we need manual modes, to override the values computed by the AEGC - we need to support seamless transitions from auto -> manual, and do so without flickering - we need custom minimum values for the manual controls, that is no magic values for enabling/disabling auto - all of these need to be done with AE sub-controls (exposure time, analogue gain) To achieve these goals, we introduce mode controls for the AE sub-controls: ExposureTimeMode and AnalogueGainMode. These have an auto state, and a disabled state. The disabled state has an internal one-way state change from locked to manual, triggered by the presence of the value-controls (ExposureTime and AnalogueGain). We then remove the AeEnable control, as it is a redundant control in the face of these two mode controls. We also remove AeLocked, as it is insufficient for reporting the AE state, and we promote AeState to non-draft to fill its role. Notably, the locked state is removed, since this information can be obtained from the aforementioned mode controls. Bug: https://bugs.libcamera.org/show_bug.cgi?id=42 Bug: https://bugs.libcamera.org/show_bug.cgi?id=43 Bug: https://bugs.libcamera.org/show_bug.cgi?id=47 Signed-off-by: Paul Elder --- Changes in v4: - remove FlashRequired and Precapture from AeState - upgrade documentation of all the controls Changes in v3: - improve wording of the control descriptions - make more succinct and clear - add description of how to do a flickerless transition Changes in v2: - No changes, just resubmitting at the head of this series so that it's together and so that /people will actually see it/ Initial version: Still RFC as I haven't updated the users of the control yet, and I want to check that these are the controls and docs that we want. We've decided that the "master AE control" will be implemented by a helper... but looking at uvcvideo and the V4L2 controls I'm wondering if such helper should come earlier than later? --- src/libcamera/control_ids.yaml | 262 +++++++++++++++++++++++++-------- 1 file changed, 200 insertions(+), 62 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 9d4638ae..9f5ce5e8 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -7,23 +7,46 @@ # Unless otherwise stated, all controls are bi-directional, i.e. they can be # set through Request::controls() and returned out through Request::metadata(). controls: - - AeEnable: - type: bool + - AeState: + type: int32_t description: | - Enable or disable the AE. + Control to report the AE algorithm state associated with the capture + result. - \sa ExposureTime AnalogueGain + The state is still reported even if ExposureTimeMode or + AnalogueGainMode is set to Disabled. - - AeLocked: - type: bool - description: | - Report the lock status of a running AE algorithm. + \sa AnalogueGain + \sa AnalogueGainMode + \sa ExposureTime + \sa ExposureTimeMode - If the AE algorithm is locked the value shall be set to true, if it's - converging it shall be set to false. If the AE algorithm is not - running the control shall not be present in the metadata control list. + enum: + - name: AeStateInactive + value: 0 + description: | + The AE algorithm is inactive. - \sa AeEnable + This state should be returned if both AnalogueGainMode and + ExposureTimeMode are set to disabled (or one, if the camera only + supports one of the two controls). + - name: AeStateSearching + value: 1 + description: | + The AE algorithm has not converged yet. + + This state should be returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to auto, and the AE algorithm hasn't + converged yet. If the AE algorithm converges, the state shall go to + AeStateConverged. + - name: AeStateConverged + value: 2 + description: | + The AE algorithm has converged. + + This state should be returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to auto, and the AE algorithm has + converged. # AeMeteringMode needs further attention: # - Auto-generate max enum value. @@ -93,6 +116,13 @@ controls: how the desired total exposure is divided between the shutter time and the sensor's analogue gain. The exposure modes are platform specific, and not all exposure modes may be supported. + + When one of AnalogueGainMode or ExposureTimeMode is set to Disabled, + the fixed values will override any choices made by AeExposureMode. + + \sa AnalogueGainMode + \sa ExposureTimeMode + enum: - name: ExposureNormal value: 0 @@ -111,13 +141,15 @@ controls: type: float description: | Specify an Exposure Value (EV) parameter. The EV parameter will only be - applied if the AE algorithm is currently enabled. + applied if the AE algorithm is currently enabled, that is, at least one + of AnalogueGainMode and ExposureTimeMode are auto. By convention EV adjusts the exposure as log2. For example EV = [-2, -1, 0.5, 0, 0.5, 1, 2] results in an exposure adjustment of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x]. - \sa AeEnable + \sa AnalogueGainMode + \sa ExposureTimeMode - ExposureTime: type: int32_t @@ -125,17 +157,85 @@ controls: Exposure time (shutter speed) for the frame applied in the sensor device. This value is specified in micro-seconds. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if ExposureTimeMode is Disabled. If + this control is set when ExposureTimeMode is Auto, the value will be + ignored and will not be retained. + + When reported in metadata, this control indicates what exposure time + was used for the current request, regardless of ExposureTimeMode. + ExposureTimeMode will indicate the source of the exposure time value, + whether it came from the AE algorithm or not. + + \sa AnalogueGain + \sa ExposureTimeMode + + - ExposureTimeMode: + type: int32_t + description: | + Controls the source of the exposure time that is applied to the image + sensor. When set to Auto, the AE algorithm computes the exposure time + and configures the image sensor accordingly. When set to Disabled, + exposure time specified in ExposureTime is applied to the image sensor. + If ExposureTime is not set, then the value last computed by the AE + algorithm when the mode was Auto will be used. + + If ExposureTime is not set and the mode is ExposureTimeModeDisabled and + AE was never Auto (either because the camera started in Disabled mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + When ExposureTimeMode is set Auto, the value set in ExposureTime is + ignored and is not retained. This means that if ExposureTimeMode is set + to Disabled and ExposureTime is not also set, the exposure time that + was last computed by the AE algorithm while the mode was Auto will be + applied to the sensor. + + If ExposureTimeModeDisabled is supported, the ExposureTime control must + also be supported. + + The set of ExposureTimeMode modes that are supported by the camera must + have an intersection with the supported set of AnalogueGainMode modes. - \sa AnalogueGain AeEnable + As it takes a few frames to apply the exposure time, there is a period of + time between submitting a request with ExposureTimeMode set to Disabled + and the exposure time component of the AE actually being disabled, + during which the AE algorithm can still update the exposure time. If an + application is switching from automatic and manual control and wishes + to eliminate any flicker during the switch, the following procedure is + recommended. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + 1. Start with ExposureTimeMode set to Auto + + 2. Set ExposureTimeMode to Disabled + + 3. Wait for the first request to be output that has ExposureTimeMode + set to Disabled + + 4. Copy the value reported in ExposureTime into a new request, and + submit it + + 5. Proceed to run manual exposure time + + \sa ExposureTime + enum: + - name: ExposureTimeModeAuto + value: 0 + description: | + The exposure time will be calculated automatically and set by the + AE algorithm. If ExposureTime is set while this mode is active, it + will be ignored, and it will also not be retained. + - name: ExposureTimeModeDisabled + value: 1 + description: | + The exposure time will not be updated by the AE algorithm. It will + come from the last calculated value when the mode was Auto, or from + the value specified in ExposureTime. + + When transitioning from Auto to Disabled mode, the last computed + exposure value is used until a new value is specified through the + ExposureTime control. If an ExposureTime value is specified in the + same request where the ExposureTimeMode is changed from Auto to + Disabled, the provided ExposureTime is applied. - AnalogueGain: type: float @@ -144,17 +244,85 @@ controls: The value of the control specifies the gain multiplier applied to all colour channels. This value cannot be lower than 1.0. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if AnalogueGainMode is Disabled. If + this control is set when AnalogueGainMode is Auto, the value will be + ignored and will not be retained. + + When reported in metadata, this control indicates what analogue gain + was used for the current request, regardless of AnalogueGainMode. + AnalogueGainMode will indicate the source of the analogue gain value, + whether it came from the AE algorithm or not. + + \sa ExposureTime + \sa AnalogueGainMode + + - AnalogueGainMode: + type: int32_t + description: | + Controls the source of the analogue gain that is applied to the image + sensor. When set to Auto, the AE algorithm computes the analogue gain + and configures the image sensor accordingly. When set to Disabled, + analogue gain specified in AnalogueGain is applied to the image sensor. + If AnalogueGain is not set, then the value last computed by the AE + algorithm when the mode was Auto will be used. + + If AnalogueGain is not set and the mode is AnalogueGainModeDisabled and + AE was never Auto (either because the camera started in Disabled mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + When AnalogueGainMode is set Auto, the value set in AnalogueGain is + ignored and is not retained. This means that if AnalogueGainMode is set + to Disabled and AnalogueGain is not also set, the analogue gain that + was last computed by the AE algorithm while the mode was Auto will be + applied to the sensor. - \sa ExposureTime AeEnable + If AnalogueGainModeDisabled is supported, the AnalogueGain control must + also be supported. + + The set of AnalogueGainMode modes that are supported by the camera must + have an intersection with the supported set of ExposureTimeMode modes. + + As it takes a few frames to apply the analogue gain, there is a period of + time between submitting a request with AnalogueGainMode set to Disabled + and the analogue gain component of the AE actually being disabled, + during which the AE algorithm can still update the analogue gain. If an + application is switching from automatic and manual control and wishes + to eliminate any flicker during the switch, the following procedure is + recommended. + + 1. Start with AnalogueGainMode set to Auto + + 2. Set AnalogueGainMode to Disabled + + 3. Wait for the first request to be output that has AnalogueGainMode + set to Disabled + + 4. Copy the value reported in AnalogueGain into a new request, and + submit it + + 5. Proceed to run manual analogue gain + + \sa AnalogueGain + enum: + - name: AnalogueGainModeAuto + value: 0 + description: | + The analogue gain will be calculated automatically and set by the + AE algorithm. If AnalogueGain is set while this mode is active, it + will be ignored, and it will also not be retained. + - name: AnalogueGainModeDisabled + value: 1 + description: | + The analogue gain will not be updated by the AE algorithm. It will + come from the last calculated value when the mode was Auto, or from + the value specified in AnalogueGain. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + When transitioning from Auto to Disabled mode the last computed + gain value is used until a new value is specified through the + AnalogueGain control. If an AnalogueGain value is specified in the + same request where the AnalogueGainMode is set to Disabled, the + provided AnalogueGain is applied. - Brightness: type: float @@ -477,36 +645,6 @@ controls: High quality aberration correction which might reduce the frame rate. - - AeState: - type: int32_t - draft: true - description: | - Control to report the current AE algorithm state. Currently identical to - ANDROID_CONTROL_AE_STATE. - - Current state of the AE algorithm. - enum: - - name: AeStateInactive - value: 0 - description: The AE algorithm is inactive. - - name: AeStateSearching - value: 1 - description: The AE algorithm has not converged yet. - - name: AeStateConverged - value: 2 - description: The AE algorithm has converged. - - name: AeStateLocked - value: 3 - description: The AE algorithm is locked. - - name: AeStateFlashRequired - value: 4 - description: The AE algorithm would need a flash for good results - - name: AeStatePrecapture - value: 5 - description: | - The AE algorithm has started a pre-capture metering session. - \sa AePrecaptureTrigger - - AfState: type: int32_t draft: true From patchwork Wed May 18 13:47:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 15971 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0649AC326C for ; Wed, 18 May 2022 13:47:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5A3E76565D; Wed, 18 May 2022 15:47:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652881669; bh=40PKtqYJq2vAzT03Flwq9s8stKjhU/o0uvoNy8Xckrw=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=X+jwu590/aRTqy0p2+HhCMVyTS8mxrJ5B+739Zjz8M2QNWPvx4Nd7QyMAyCZyXJzh as+UcJUOwho2rc8KNzhkB/A+kHBzrCQLhFZesOVMpuMsHWcvXONVDcw2gW6UNx3tpX 0zPw2LApUwq6dMpxP1Jn/wMeoFBX8LFL0kM2Nm4XRlqYqB3kYlUj+bO329och2lrHA HXPtjXwaV5+4pOoBar3uLFD6XgCXBDL7aSagLk3wRMh02sFhf7ueG8tLEX1OHbilM5 RZNBDnWrngcvtAZtmFzf/eOkpzkVv8RmP6XC5eR2zb+FsA2YG7yOVe6asVyUBL/B+Z ByyMOKivYcJFQ== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CBCA065659 for ; Wed, 18 May 2022 15:47:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NQyjAyeS"; dkim-atps=neutral Received: from localhost.localdomain (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5D273E50; Wed, 18 May 2022 15:47:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652881664; bh=40PKtqYJq2vAzT03Flwq9s8stKjhU/o0uvoNy8Xckrw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NQyjAyeSNJfvOSB1bnqduXv53ps0qnUOH3DYCjUJASMK7gmItxIDchVWN4OkKPl1g dtCDpRGqAK4UhhFznYe6O+x7vIYC87Xukk7kWhFMJSBJR7bPch/sRS2ZdqKYfvsJOX t82i51OsXef4wlYcvG9IfPAxsw+0z6jahDpn5M3M= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:47:27 +0200 Message-Id: <20220518134728.777709-3-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220518134728.777709-1-paul.elder@ideasonboard.com> References: <20220518134728.777709-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/3] Documentation: design: ae: Document the design for AE controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Paul Elder via libcamera-devel From: Paul Elder Reply-To: Paul Elder Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Document the design and rationale for the AE-related controls. Signed-off-by: Paul Elder --- Documentation/design/ae.rst | 260 ++++++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 Documentation/design/ae.rst diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst new file mode 100644 index 00000000..09ca2ee7 --- /dev/null +++ b/Documentation/design/ae.rst @@ -0,0 +1,260 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +Design of Exposure and Gain controls +==================================== + +This document explains the design and rationale of the controls related to +exposure and gain. This includes the all-encompassing auto-exposure (AE), the +manual exposure control, and the manual gain control. + +Description of the problem +-------------------------- + +Sub controls +^^^^^^^^^^^^ + +There are more than one control that make up exposure: exposure, gain, and +aperture (though for now we will not consider aperture). We already had +individual controls for setting the values of manual exposure and manual gain, +but for switching between auto mode and manual mode we only had a high-level +boolean AeEnable control that would set *both* exposure and gain to auto mode +or manual mode; we had no way to set one to auto and the other to manual. + +So, we need to introduce two new controls to act as "levers" to indicate +individually for exposure and gain if the value would come from AEGC or if it +would come from the manual control value. + +Aperture priority +^^^^^^^^^^^^^^^^^ + +We eventually will support aperture, and so whatever our solution is for having +only some controls on auto and the others on manual needs to be extensible. + +Flickering when going from auto to manual +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a manual exposure or gain value is requested by the application, it costs +a few frames worth of time for them to take effect. This means that during a +transition from auto to manual, there would be flickering in the control values +and the transition won't be smooth. + +Take for instance the following flow, where we start on auto exposure (which +for the purposes of the example increments by 1 each frame) and we want to +switch seamlessly to manual exposure, which involves copying the exposure value +computed by the auto exposure algorithm: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A 2 2 2 2 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 2 2 + +As we can see, after frame N+2 completes, we copy the exposure value that was +used for frame N+2 (which was computed by AE algorithm), and queue that value +into request N+3 with manual mode on. However, as it takes two frames for the +exposure to be set, the exposure still changes since it is set by AE, and we +get a flicker in the exposure during the switch from auto to manual. + +A solution is to *not submit* any exposure value when manual mode is enabled, +and wait until the manual mode as been "applied" before copying the exposure +value: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A None None None 5 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 5 5 + +In practice, this works. However, libcamera has a policy where once a control +is submitted, its value is saved and does not need to be resubmitted (even +though this isn't implemented yet). So if the manual exposure value was set +while auto mode was on, in theory the value would be saved, so when manual mode +is enabled, the exposure value that was previously set would immediately be +used. Clearly this solution isn't correct, but it can serve as the basis for a +proper solution, with some more rigorous rules. + +Existing solutions +------------------ + +Raspberry Pi +^^^^^^^^^^^^ + +The raspberry pi IPA got around the lack of individual AeEnable controls for +exposure and gain by using magic values. When AeEnable was false, if one of the +manual control values was set to 0 then the value computed by AEGC would be +used for just that control. This solution isn't desirable, as it prevents +that magic value from being used as a valid value. + +To get around the flickering issue, when AeEnable was false, the raspberry pi +AEGC would simply stop updating the values to be set. As mentioned above, since +the value retention mechanism hasn't actually been implemented yet, this +worked. But, it's not a proper solution. + +Android +^^^^^^^ + +The Android HAL specification requires that exposure and gain (sensitivity) +must both be manual or both be auto. It cannot be that one is manual while the +other is auto, so they simply don't support sub controls. + +For the flickering issue, the Android HAL has an AeLock control. To transition +from auto to manual, the application would keep AE on auto, and turn on the +lock. Once the lock has propagated through, then the value can be copied from +the result into the request and the lock disabled and the mode set to manual. + +The problem with this solution is, besides the extra complexity, that it is +ambiguous what happens if there is a state transition from manual to locked +(even though it's a state transition that doesn't make sense). If locked is +defined to "use the last automatically computed values" then it could use the +values from the last time it AE was set to auto, or it would be undefined if AE +was never auto (eg. it started out as manual), or if AE is implemented to run +in the background it could just use the current values that are computed. If +locked is defined to "use the last value that was set" there would be less +ambiguity. Still, it's better if we can make it impossible to execute this +nonsensical state transition, and if we can reduce the complexity of having +this extra control or extra setting on a lever. + +Summary of goals +---------------- + + - We need a lock of some sort, to instruct the AEGC to not update output + results + + - We need manual modes, to override the values computed by the AEGC + + - We need to support seamless transitions from auto to manual, and do so + without flickering + + - We need custom minimum values for the manual controls; that is, no magic + values for enabling/disabling auto + + - All of these need to be done with AE sub-controls (exposure time, analogue + gain) and be extensible to aperture in the future + +Our solution +------------ + +A diagram of our solution: + +:: + + +----------------------------+-------------+------------------+-----------------+ + | INPUT | ALGORITHM | RESULT | OUTPUT | + +----------------------------+-------------+------------------+-----------------+ + + ExposureTimeMode ExposureTimeMode + ---------------------+----------------------------------------+-----------------> + 0: Auto | | + 1: Disabled | V + | |\ + | | \ + | /----------------------------------> | 1| ExposureTime + | | +-------------+ exposure time | | --------------> + \--)--> | | --------------> | 0| + ExposureTime | | | | / + ------------------------+--> | | |/ + | | AeState + | AEGC | -----------------------------------> + AnalogueGain | | + ------------------------+--> | | |\ + | | | | \ + /--)--> | | --------------> | 0| AnalogueGain + | | +-------------+ analogue gain | | --------------> + | \----------------------------------> | 1| + | | / + | |/ + | ^ + AnalogueGainMode | | AnalogueGainMode + ---------------------+----------------------------------------+-----------------> + 0: Auto + 1: Disabled + + +The diagram is divided in four sections horizontally: + + - Input: The values received from the request controls + + - Algorithm: The algorithm itself + + - Result: The values calculated by the algorithm + + - Output: The values that sent in result metadata and applied to the device + +The four input controls are divided between manual values (ExposureTime and +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The +former are the manual values, the latter control how they're applied. The two +modes are independent from each other, and each can take one of two values: + + - Auto (0): The AGC computes the value normally. The AGC result is applied + to the output. The manual value is ignored *and is not retained*. + + - Disabled (1): The AGC uses the manual value internally. The corresponding + manual control from the request is applied to the output. The AGC result + is ignored. + +The AeState control reports the state of the unified AEGC block. If both +ExposureTimeMode and AnalogueGainMode are set to disabled then it will report +Inactive. If at least one of the two is set to auto, then AeState will report +if the AEGC has Converged or not (Searching). This control replaces the old +AeLocked control, as it was insufficient for reporting the AE state. + +There is a caveat to the disabled mode: the manual control value is not +retained if it is set during auto mode. This means that if the disabled mode is +entered without also setting the manual value, then it will enter a state +similar to "locked", where the last automatically computed value while the mode +was auto will be used. Once the manual value is set, then that will be used and +retained as usual. + +This simulates an auto -> locked -> manual or auto -> manual state transition, +and makes it impossible to do the nonsensical manual -> locked state +transition. + +We specifically do not have a "master AE control" like the old AeEnable. This +is because we have the individual mode controls, and if we had a master AE +control it would be a "control that sets other controls", which could easily +get out of control. + +With this solution, the earlier example would become: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10| + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab + Exp requested: N/A None None None None 10 None 10 10 + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10 + + Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab + Exp used: 2 3 4 5 5 5 5 10 10 + +This example is extended by a few frames to exhibit the simulated "locked" +state. At frame N+5 the application has confirmed that the auto mode has been +disabled, but does not provide a manual value until request N+7. Thus, the +value that is used in requests N+5 and N+6 (where the mode is disabled), comes +from the last value that was used when the mode was auto, which comes from +frame N+4. + +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for +the exposure to be applied. N+8 does not supply a manual value, but the last +supplied value is retained, so a manual value of 10 is still used and set in +frame N+10. + +Although this behavior is the same as what we had with waiting for the manual +mode to propagate (in the section "Description of the problem"), this time it +is correct as we have defined specifically that if a manual value was specified +while the mode was auto, it will not be retained. From patchwork Wed May 18 13:47:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 15972 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D70CBC3256 for ; Wed, 18 May 2022 13:47:50 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CB2046565E; Wed, 18 May 2022 15:47:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652881669; bh=qstfj7d2bg2uIa4x6d1yL9Wn39GJh6nHjxaqXi69Oo4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=a5JgNyn0G/qLPxfpTpU8qxRIMJtNnkVceD8rhz4/CfvNQATA0ZB0eqdQZkr7DZRPK jRhbaVHQmdpNzK/x1tpgWIdmCvYDThHWBoNdyr/I0PMFd5zfz3/5rBRBDmoeucDYyP 7dmahzWoJ55Q3zERYuARin7ABi7EEHy7Qv+u0ibYmdd7/r2+ZkwrRkY2DncauNR356 KZ4qwrOwjKwVrJP0fxD5p7pA2vNzgPV2kRq4fsq+T8+YjDGuW/ahh1TIbBW70MQwqX w7O3K9/hoZqpTxo1NI6Bs+VqS8aR3MMbbGZSISVK7qH41AxceD6NDGtDNngdxIKlfe En7G2e3LSuMaQ== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4358865658 for ; Wed, 18 May 2022 15:47:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="EutlDkle"; dkim-atps=neutral Received: from localhost.localdomain (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id CCB5B1027; Wed, 18 May 2022 15:47:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652881665; bh=qstfj7d2bg2uIa4x6d1yL9Wn39GJh6nHjxaqXi69Oo4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EutlDkleSPGlwUSlnVqUd/Rpku+rnooWyyiED0GNDDhBWypQfRs48gsXi0r6MAi52 A8Fbk6/PHy6DDHD5pLahOeuKvhDistUoB06lnd7e+ygwtz/KkZ/xl9qHknLi8cTaC1 GRX4r51MxpmzGymbF1z8J6aEuRyNHst36t/gGJdI= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:47:28 +0200 Message-Id: <20220518134728.777709-4-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220518134728.777709-1-paul.elder@ideasonboard.com> References: <20220518134728.777709-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/3] libcamera: control_ids: Add high-level documentation for AE X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Paul Elder via libcamera-devel From: Paul Elder Reply-To: Paul Elder Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add high-level documentation for AE, describing the states and the transitions and how one would use the AE controls. Signed-off-by: Paul Elder --- src/libcamera/control_ids.cpp.in | 82 ++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/src/libcamera/control_ids.cpp.in b/src/libcamera/control_ids.cpp.in index 5fb1c2c3..62451d7f 100644 --- a/src/libcamera/control_ids.cpp.in +++ b/src/libcamera/control_ids.cpp.in @@ -10,6 +10,88 @@ #include #include +/** + * \page AE Auto-exposure Support + * + * libcamera offers the following controls related to exposure and gain: + * - AnalogueGain + * - AnalogueGainMode + * - ExposureTime + * - ExposureTimeMode + * - AeState + * + * Auto-exposure and auto-gain can be enabled and disabled separately using the + * ExposureTimeMode and AnalogueGainMode controls respectively. There is no + * overarching AeEnable control. + * + * For each of exposure and gain, we can model it with three states: auto, + * locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only + * have two values, as the locked state is simulated. + * + * /---------------------------------\ + * | | + * V | + * +--------+ +--------+ + * | | ---------------------> | | + * | Auto | +--------+ | Manual | + * | | ----> | Locked | ----> | | + * +--------+ +--------+ +--------+ + * ^ | + * | | + * \----------------/ + * + * Notice from the state diagram that locked to manual is a one-way state + * change, as the reverse direction is nonsensical (see the design document for + * more details on this topic). + * + * The exposure/gain is in the Auto state when + * ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value + * that is computed by the AE algorithm is applied to the image sensor. Any + * value that is supplied in the ExposureTime/AnalogueGain control is ignored + * and is not retained. + * + * If ExposureTimeMode/AnalogueGainMode is set to Disabled, it can put us in + * either the Locked or Manual state. The difference is in if + * ExposureTime/AnalogueGain has been supplied. If it has not yet been + * supplied, then we are in the Locked state. If it has been supplied, then we + * are in the Manual state. + * + * In both the Locked state and the Manual state the exposure/gain value does + * not come from the AE algorithm. In the Locked state the value comes from the + * last value computed by the AE algorithm while the state was Auto, or if the + * state was never Auto (e.g. we started in Locked, or the camera doesn't + * support Auto), then the value should be a best-effort default value. In the + * Manual state the value comes from the value supplied in the + * ExposureTime/AnalogueGain control. + * + * To transition from the Locked state to the Manual state, a value needs to be + * submitted in ExposureTime/AnalogueGain. Once the state has transitioned to + * Manual, then this value will be retained, and doesn't need to be resubmitted + * if it doesn't change. + * + * To transition to the Auto state, simply set + * ExposureTimeMode/AnalogueGainMode to Auto. + * + * + * The AeState metadata reports the state of the AE algorithm. As AE cannot + * compute exposure and gain separately, the state of the AE component is + * unified. There are three states: Inactive, Searching, and Converged. + * + * The state shall be Inactive if both ExposureTimeMode and AnalogueGainMode + * are set to Disabled. If the camera only supports one of the two controls, + * then the state shall be Inactive if that one control is set to Disabled. If + * the camera does not support Disabled for at least one of the two controls, + * then the state will never be Inactive, as AE will always be running. + * + * The state shall be Searching if at least one of exposure or gain calculated + * by the AE algorithm is used (that is, at least one of the two modes is Auto), + * *and* the value(s) have not converged yet. + * + * The state shall be Converged if at least one of exposure or gain calculated + * by the AE algorithm is used (that is, at least one of the two modes is Auto), + * *and* the value(s) have converged. + */ + /** * \file control_ids.h * \brief Camera control identifiers