From patchwork Tue Dec 21 04:36:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 15179 X-Patchwork-Delegate: paul.elder@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C071EBE080 for ; Tue, 21 Dec 2021 04:36:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 361DC608F9; Tue, 21 Dec 2021 05:36:23 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="krON6oLD"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C438E60592 for ; Tue, 21 Dec 2021 05:36:20 +0100 (CET) Received: from pyrite.mediacom.info (unknown [IPv6:2604:2d80:ad90:fb00:96fd:8874:873:6c16]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C2223FD2; Tue, 21 Dec 2021 05:36:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1640061380; bh=g52XoS8OdmFvRJ7mJBhylmfhTKgyygElo9/q5i1W9B0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=krON6oLDzmsnrcmvRec9CPTs6g29dAX2F2bF59m/HGq5vTVqsyCA0sJyMiwGHXmWO RSQFkOkE6zwHN31Mm5DT1AenpQN4635TbhuqY1wESX8UWxUs5e2B05BA+oviUtmnpg fVla4cb92gjxWLQ+FvzjLy0umRM/PhoQFCXNNsik= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Dec 2021 22:36:03 -0600 Message-Id: <20211221043610.2512334-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211221043610.2512334-1-paul.elder@ideasonboard.com> References: <20211221043610.2512334-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 1/8] controls: Reorganize the AE-related controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We have multiple goals: - we need a lock of some sort, to instruct the AEGC to not update output results - we need manual modes, to override the values computed by the AEGC - we need to support seamless transitions from auto -> manual, and do so without flickering - we need custom minimum values for the manual controls, that is no magic values for enabling/disabling auto - all of these need to be done with AE sub-controls (exposure time, analogue gain) To achieve these goals, we introduce mode controls for the AE sub-controls: ExposureTimeMode and AnalogueGainMode. These have an auto state, and a disabled state. The disabled state has an internal one-way state change from locked to manual, triggered by the presence of the value-controls (ExposureTime and AnalogueGain). We then remove the AeEnable control, as it is a redundant control in the face of these two mode controls. We also remove AeLocked, as it is insufficient for reporting the AE state, and we promote AeState to non-draft to fill its role. Notably, the locked state is removed, since this information can be obtained from the aforementioned mode controls. Bug: https://bugs.libcamera.org/show_bug.cgi?id=42 Bug: https://bugs.libcamera.org/show_bug.cgi?id=43 Bug: https://bugs.libcamera.org/show_bug.cgi?id=47 Signed-off-by: Paul Elder --- Changes in v3: - improve wording of the control descriptions - make more succinct and clear - add description of how to do a flickerless transition Changes in v2: - No changes, just resubmitting at the head of this series so that it's together and so that /people will actually see it/ Initial version: Still RFC as I haven't updated the users of the control yet, and I want to check that these are the controls and docs that we want. We've decided that the "master AE control" will be implemented by a helper... but looking at uvcvideo and the V4L2 controls I'm wondering if such helper should come earlier than later? --- src/libcamera/control_ids.yaml | 239 ++++++++++++++++++++++++--------- 1 file changed, 177 insertions(+), 62 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 9d4638ae..84679317 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -7,23 +7,63 @@ # Unless otherwise stated, all controls are bi-directional, i.e. they can be # set through Request::controls() and returned out through Request::metadata(). controls: - - AeEnable: - type: bool + - AeState: + type: int32_t description: | - Enable or disable the AE. + Control to report the current AE algorithm state. The camera device can do + several state transitions between two results, if it is allowed by the + state transition table. For example, AeStateInactive may never + actually be seen in a result. - \sa ExposureTime AnalogueGain + The state in the result is the state for this request. If AE state + becomes AeStateConverged, then the image data associated with the + result should be good to use. - - AeLocked: - type: bool - description: | - Report the lock status of a running AE algorithm. + The state is still reported even if ExposureTimeMode or + AnalogueGainMode is set to Disabled. - If the AE algorithm is locked the value shall be set to true, if it's - converging it shall be set to false. If the AE algorithm is not - running the control shall not be present in the metadata control list. + \sa AnalogueGain + \sa AnalogueGainMode + \sa ExposureTime + \sa ExposureTimeMode - \sa AeEnable + enum: + - name: AeStateInactive + value: 0 + description: | + The AE algorithm is inactive. + If the camera initiates an AE scan, the state shall go to + AeStateSearching. + + Unlike the analogous control in Android, enabling or disabling any + of the AE-related mode controls (eg. AnalogueGainMode, + ExposureTimeMode) does not require the state to be reset to to + AeStateInactive. + - name: AeStateSearching + value: 1 + description: | + The AE algorithm has not converged yet. + If the camera finishes an AE scan, the state shall go to + AeStateConverged. If the camera finishes an AE scan, but flash is + required, the state shall go to AeStateFlashRequired. + - name: AeStateConverged + value: 2 + description: | + The AE algorithm has converged. + If the camera initiates an AE scan, the state shall go to + AeStateSearching. + - name: AeStateFlashRequired + value: 3 + description: | + The AE algorithm would need a flash for good results. + If the camera initiates an AE scan, the state shall go to + AeStateSearching. + - name: AeStatePrecapture + value: 4 + description: | + The AE algorithm has started a pre-capture metering session. + After the sequence is finished, the state shall go to AeStateConverged if + \sa AePrecaptureTrigger # AeMeteringMode needs further attention: # - Auto-generate max enum value. @@ -93,6 +133,13 @@ controls: how the desired total exposure is divided between the shutter time and the sensor's analogue gain. The exposure modes are platform specific, and not all exposure modes may be supported. + + When one of AnalogueGainMode or ExposureTimeMode is set to Disabled, + the fixed values will override any choices made by AeExposureMode. + + \sa AnalogueGainMode + \sa ExposureTimeMode + enum: - name: ExposureNormal value: 0 @@ -111,13 +158,15 @@ controls: type: float description: | Specify an Exposure Value (EV) parameter. The EV parameter will only be - applied if the AE algorithm is currently enabled. + applied if the AE algorithm is currently enabled, that is, at least one + of AnalogueGainMode and ExposureTimeMode are auto. By convention EV adjusts the exposure as log2. For example EV = [-2, -1, 0.5, 0, 0.5, 1, 2] results in an exposure adjustment of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x]. - \sa AeEnable + \sa AnalogueGainMode + \sa ExposureTimeMode - ExposureTime: type: int32_t @@ -125,17 +174,65 @@ controls: Exposure time (shutter speed) for the frame applied in the sensor device. This value is specified in micro-seconds. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if ExposureTimeMode is Disabled. Its + presence in a request acts as a trigger to switch to the internal + manual mode within ExposureTimeModeDisabled. + + When reported in metadata, this control indicates what exposure time + was used for the current request, regardless of ExposureTimeMode. + ExposureTimeMode will indicate the source of the exposure time value, + whether it came from the AE algorithm or not. + + \sa AnalogueGain + \sa ExposureTimeMode - \sa AnalogueGain AeEnable + - ExposureTimeMode: + type: int32_t + description: | + Controls how the frame exposure time is computed. When set to Auto, the + AE algorithm computes the exposure time of the next frame and + configures the image sensor accordingly. When set to Disabled, the AE + algorithm stops updating the exposure time. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + As the camera device has a pipeline of in-flight requests, there is a + period of time between submitting a request with ExposureTimeMode set + to Disabled and the exposure time component of the AE actually being + disabled, during which the AE algorithm can still update the exposure + time. If an application is switching from automatic and manual control + and wishes to eliminate any flicker during the switch, the following + procedure is recommended. + + 1. Start with ExposureTimeMode set to Auto + + 2. Set ExposureTimeMode to Disabled + + 3. Wait for the first request to be output that has ExposureTimeMode + set to Disabled + + 4. Copy the value reported in ExposureTime into a new request, and + submit it + + 5. Proceed to run manual exposure time + + \sa ExposureTime + enum: + - name: ExposureTimeModeAuto + value: 0 + description: | + The exposure time will be calculated automatically and set by the + AE algorithm. + - name: ExposureTimeModeDisabled + value: 1 + description: | + The exposure time will not be updated by the AE algorithm. It will + come from the last calculated value when the mode was Auto, or from + the value specified in ExposureTime. + + When transitioning from Auto to Disabled mode the last computed + exposure value is used until a new value is specified through the + ExposureTime control. If an ExposureTime value is specified in the + same request where the ExposureTimeMode is set to Disabled, the + provided ExposureTime is applied immediately. - AnalogueGain: type: float @@ -144,17 +241,65 @@ controls: The value of the control specifies the gain multiplier applied to all colour channels. This value cannot be lower than 1.0. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if ExposureTimeMode is Disabled. Its + presence in a request acts as a trigger to switch to the internal + manual mode within ExposureTimeModeDisabled. + + When reported in metadata, this control indicates what exposure time + was used for the current request, regardless of ExposureTimeMode. + ExposureTimeMode will indicate the source of the exposure time value, + whether it came from the AE algorithm or not. + + \sa ExposureTime + \sa AnalogueGainMode + + - AnalogueGainMode: + type: int32_t + description: | + Controls how the analogue gain is computed. When set to Auto, the + AE algorithm computes the analogue gain of the next frame and + configures the image sensor accordingly. When set to Disabled, the AE + algorithm stops updating the analogue gain. + + As the camera device has a pipeline of in-flight requests, there is a + period of time between submitting a request with AnalogueGainMode set + to Disabled and the analogue gain component of the AE actually being + disabled, during which the AE algorithm can still update the analogue + gain. If an application is switching from automatic and manual control + and wishes to eliminate any flicker during the switch, the following + procedure is recommended. + + 1. Start with AnalogueGainMode set to Auto - \sa ExposureTime AeEnable + 2. Set AnalogueGainMode to Disabled - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + 3. Wait for the first request to be output that has AnalogueGainMode + set to Disabled + + 4. Copy the value reported in AnalogueGain into a new request, and + submit it + + 5. Proceed to run manual analogue gain + + \sa AnalogueGain + enum: + - name: AnalogueGainModeAuto + value: 0 + description: | + The analogue gain will be calculated automatically and set by the + AE algorithm. + - name: AnalogueGainModeDisabled + value: 1 + description: | + The analogue gain will not be updated by the AE algorithm. It will + come from the last calculated value when the mode was Auto, or from + the value specified in AnalogueGain. + + When transitioning from Auto to Disabled mode the last computed + gain value is used until a new value is specified through the + AnalogueGain control. If an AnalogueGain value is specified in the + same request where the AnalogueGainMode is set to Disabled, the + provided AnalogueGain is applied immediately. - Brightness: type: float @@ -477,36 +622,6 @@ controls: High quality aberration correction which might reduce the frame rate. - - AeState: - type: int32_t - draft: true - description: | - Control to report the current AE algorithm state. Currently identical to - ANDROID_CONTROL_AE_STATE. - - Current state of the AE algorithm. - enum: - - name: AeStateInactive - value: 0 - description: The AE algorithm is inactive. - - name: AeStateSearching - value: 1 - description: The AE algorithm has not converged yet. - - name: AeStateConverged - value: 2 - description: The AE algorithm has converged. - - name: AeStateLocked - value: 3 - description: The AE algorithm is locked. - - name: AeStateFlashRequired - value: 4 - description: The AE algorithm would need a flash for good results - - name: AeStatePrecapture - value: 5 - description: | - The AE algorithm has started a pre-capture metering session. - \sa AePrecaptureTrigger - - AfState: type: int32_t draft: true