| Message ID | 20220331151747.19458-2-david.plowman@raspberrypi.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series |
|
| Related | show |
Hi David, Thanks for your patch ! On 31/03/2022 17:17, David Plowman wrote: > This patch describes a series of controls that allow applications to > drive AF algorithms: > > AfMode - manual, auto or continuous > AfLensPosition - set lens position to macro/hyperfocal/infinity > AfRange - full, macro or normal > AfSpeed - fast or slow > AfWindows - AF window locations > AfTrigger - start (trigger an AF scan) or cancel > AfPause - pause continuous AF > LensPosition - position of lens from lens driver > AfState - reports the mode, whether scanning/success/failure > --- > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > 1 file changed, 313 insertions(+), 60 deletions(-) > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > index 9d4638ae..23607368 100644 > --- a/src/libcamera/control_ids.yaml > +++ b/src/libcamera/control_ids.yaml > @@ -406,27 +406,6 @@ controls: > The camera will cancel any active or completed metering sequence. > The AE algorithm is reset to its initial state. > > - - AfTrigger: > - type: int32_t > - draft: true > - description: | > - Control for AF trigger. Currently identical to > - ANDROID_CONTROL_AF_TRIGGER. > - > - Whether the camera device will trigger autofocus for this request. > - enum: > - - name: AfTriggerIdle > - value: 0 > - description: The trigger is idle. > - - name: AfTriggerStart > - value: 1 > - description: The AF routine is started by the camera. > - - name: AfTriggerCancel > - value: 2 > - description: | > - The camera will cancel any active trigger and the AF routine is > - reset to its initial state. > - > - NoiseReductionMode: > type: int32_t > draft: true > @@ -507,45 +486,6 @@ controls: > The AE algorithm has started a pre-capture metering session. > \sa AePrecaptureTrigger > > - - AfState: > - type: int32_t > - draft: true > - description: | > - Control to report the current AF algorithm state. Currently identical to > - ANDROID_CONTROL_AF_STATE. > - > - Current state of the AF algorithm. > - enum: > - - name: AfStateInactive > - value: 0 > - description: The AF algorithm is inactive. > - - name: AfStatePassiveScan > - value: 1 > - description: | > - AF is performing a passive scan of the scene in continuous > - auto-focus mode. > - - name: AfStatePassiveFocused > - value: 2 > - description: | > - AF believes the scene is in focus, but might restart scanning. > - - name: AfStateActiveScan > - value: 3 > - description: | > - AF is performing a scan triggered by an AF trigger request. > - \sa AfTrigger > - - name: AfStateFocusedLock > - value: 4 > - description: | > - AF believes has focused correctly and has locked focus. > - - name: AfStateNotFocusedLock > - value: 5 > - description: | > - AF has not been able to focus and has locked. > - - name: AfStatePassiveUnfocused > - value: 6 > - description: | > - AF has completed a passive scan without finding focus. > - > - AwbState: > type: int32_t > draft: true > @@ -690,4 +630,317 @@ controls: > value. All of the custom test patterns will be static (that is the > raw image must not vary from frame to frame). > > + - AfMode: > + type: int32_t > + draft: true > + description: | > + Control to set the mode of the AF (autofocus) algorithm. Applications > + are allowed to set a new mode, and to send additional controls for > + that new mode, in the same request. > + > + An implementation may choose not to implement all the modes. > + > + enum: > + - name: AfModeManual > + value: 0 > + description: | > + The AF algorithm is in manual mode. In this mode it will never > + perform any action nor move the lens of its own accord, but an > + application can set controls to move the lens. > + > + In this mode the AfState will always report AfStateManual. > + - name: AfModeAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode. This means that the algorithm > + will never move the lens or change state unless the AfTrigger > + control is used. The AfTrigger control can be used to initiate a > + focus scan, the results of which will also be reported by AfState. > + > + If the autofocus algorithm is moved from AfModeAuto to another > + mode while a scan is in progress, the scan is cancelled > + immediately, without waiting for the scan to finish. > + > + When first entering this mode the AfState will report > + AfStateAuto. When a trigger control is sent, AfState will > + report AfStateAutoScanning for a period before spontaneously > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > + the outcome of the scan. It will remain in this state until > + another scan is initiated by the AfTrigger control. If a scan is > + cancelled (without changing to another mode), AfState will return > + to AfStateAuto. > + - name: AfModeContinuous > + value: 2 > + description: | > + The AF algorithm is in continuous mode. This means that the lens > + can re-start a scan spontaneously at any moment, without any user > + intervention. The AfState still reports whether the algorithm is > + currently scanning or not, though the application has no ability > + to initiate or cancel scans (though it can "pause" them), nor to > + move the lens for itself. > + > + When set to AfModeContinuous, the system will immediately initiate > + a scan so AfState will report AfStateContinuousScanning, and will > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > + depending on the scan result. > + > + - AfLensPosition: > + type: int32_t > + draft: true > + description: | > + Control to set the position of the lens to one of the following > + predefined locations. This control only has any effect when the AfMode > + is set to AfModeManual. > + > + This control is distinct from the LensPosition control, which sets the > + lens position using the lens driver's units. > + enum: > + - name: AfLensPositionMacro > + value: 0 > + description: The closest focal position of the lens. > + - name: AfLensPositionHyperfocal > + value: 1 > + description: Hyperfocal position. > + - name: AfLensPositionInfinity > + value: 2 > + description: The furthest focal position (usually infinity). This is a clear call for VCMHelper classes and vcm tuning files :-). > + > + - AfRange: > + type: int32_t > + draft: true > + description: | > + Control to set the range of focus distances that is scanned. An > + implementation may choose not to implement all the options here. > + enum: > + - name: AfRangeNormal > + value: 0 > + description: | > + A wide range of focus distances is scanned, all the way from > + infinity down to close distances, though depending on the > + implementation, possibly not including the very closest macro > + positions. > + - name: AfRangeMacro > + value: 1 > + description: Only close distances are scanned. > + - name: AfRangeFull > + value: 2 > + description: | > + The full range of focus distances is scanned just as with > + AfRangeNormal but this time including the very closest macro > + positions. > + > + - AfSpeed: > + type: int32_t > + draft: true > + description: | > + Control that determines whether the AF algorithm is to move the lens > + as quickly as possible or more steadily. For example, during video > + recording it may be desirable not to move the lens too abruptly, but > + when in a preview mode (waiting for a still capture) it may be > + helpful to move the lens as quickly as is reasonably possible. > + enum: > + - name: AfSpeedNormal > + value: 0 > + description: Move the lens at its usual speed. > + - name: AfSpeedFast > + value: 1 > + description: Move the lens more quickly. > + > + - AfWindows: > + type: Rectangle > + draft: true > + description: | > + Sets the focus windows used by the AF algorithm. The units used express > + a proportion of the ScalerCrop control (or if unavailable, of the entire > + image), as u0.16 format numbers. > + > + In order to be activated, a rectangle must be programmed with non-zero > + width and height. If no rectangles are programmed in this way, then the > + system will choose its own single default window in the centre of the > + image. > + > + The details of how the windows are used are platform dependent. We note > + that when there is more than one AF window, a typical implementation > + might find the optimal focus position for each one and finally select > + the window closest to the camera. > + > + size: [platform dependent] > + > + - AfTrigger: > + type: int32_t > + draft: true > + description: | > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > + and can also be used to terminate a scan early. > + > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > + > + enum: > + - name: AfTriggerStart > + value: 0 > + description: Start an AF scan. Ignored if a scan is in progress. > + - name: AfTriggerCancel > + value: 1 > + description: Cancel an AF scan. This does not cause the lens to move > + anywhere else. Ignored if no scan is in progress. > + > + - AfPause: > + type: int32_t > + draft: true > + description: | > + This control has no effect except when in continuous autofocus mode > + (AfModeContinuous). It can be used to pause any lens movements while > + (for example) images are captured. The algorithm remains inactive > + until it is instructed to resume. > + > + enum: > + - name: AfPauseImmediate > + value: 0 > + description: | > + Pause the continuous autofocus algorithm immediately, whether or > + not any kind of scan is underway. If the AfState was previously > + reporting AfStateContinuousScanning it will now change to > + AfStateContinuousScanningPaused, and similarly for > + AfStateContinuousFocused and AfStateContinuousFailed. > + - name AfPauseDeferred > + value: 1 I get an error when compiling: yaml.scanner.ScannerError: mapping values are not allowed here in "<byte string>", line 806, column 16: value: 1 And indeed, you missed a colon: s/name AfPauseDeferred/name: AfPauseDeferred/ > + description: | > + Pause the continuous autofocus algorithm as soon as it is no longer > + scanning. If the AfState is currently reporting > + AfStateContinuousFocused is will change to > + AfStateContinuousFocusedPaused, and similarly in the case of > + AfStateContinuousFailed. > + > + If AfState reports AfStateContinuousScanning it will change to > + AfStateContinuousScanningPausing, and then move to one of > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > + when the scan completes. > + - name: AfPauseResume > + value: 2 > + description: | > + Resume continous autofocus operation. The algorithm starts again s/continous/continuous > + from exactly where it left off, and the AfState will drop the > + 'Paused' or 'Pausing' part of the state. > + > + - LensPosition: > + type: int32_t > + draft: true > + description: | > + Acts as a control to instruct the lens to move to a particular position > + and also reports back the position of the lens for each frame. > + > + The units are determined by the lens driver. > + > + The LensPosition control is ignored unless the AfMode is set to > + AfModeManual. > + > + Note that this control is distinct from AfLensPosition, which allows > + the lens to be moved to its macro/hyperfocal/infinity position, rather > + than using lens driver units. > + > + - AfState: > + type: int32_t > + draft: true > + description: | > + Reports the current state of the AF algorithm. The possible state changes > + are described below, although we add that the following general state > + changes are also allowed. > + > + In any of the AfStateManual or AfStateContinuous states, the AfMode > + may be set to AfModeAuto and the algorithm will move to the > + AfStateAuto state. If AfTriggerStart is sent at the same time > + then the algorithm will move to AfStateAutoScanning state. > + > + In any of the AfStateAuto or AfStateManual states, the AfMode may > + be set to AfModeContinuous and the algorithm will move to > + AfStateContinuousScanning. > + > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > + be set to AfModeManual and the algorithm will move to > + AfStateManual. The lens will not be moved and will be left where > + it was at that moment. > + > + enum: > + - name: AfStateManual > + value: 0 > + description: | > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > + and AfLensPosition controls can be used directly to move the lens. > + - name: AfStateAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and has either just > + been moved into that state, or a scan that was in progress has been > + cancelled. > + - name: AfStateAutoScanning > + value: 2 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > + started using the AfTrigger control. The scan can be cancelled by > + sending AfTriggerCancel at which point the algorithm will either > + move back to AfStateAuto or, if the scan actually completes > + before the cancel request is processed, to one of > + AfStateAutoFocused or AfStateAutoFailed. > + - name: AfStateAutoFocused > + value: 3 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm believes the image is > + now in focus. > + - name: AfStateAutoFailed > + value: 4 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm did not find a good > + focus position. > + - name: AfStateContinuousScanning > + value: 5 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + is currently scanning for a good focus position. This occurs when > + the mode is first set to AfModeContinuous, or may happen > + spontaneously when the algorithm believes a re-scan is needed. > + - name: AfStateContinuousScanningPausing > + value: 6 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseDeferred was sent. The scan will complete > + at which point the algorithm moves to the > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > + state. > + - name: AfStateContinuousScanningPaused > + value: 7 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseImmediate was sent. The scan will not > + complete and the algorithm will remain in this state. The scan > + may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFocused > + value: 8 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed with the algorithm believing it has found a > + good focus position. > + - name: AfStateContinuousFocusedPaused > + value: 9 > + description: | > + The AF algorithm was in the AfStateContinuousFocused state and > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and the > + scan has completed successfully. The algorithm will now remain > + in this state, and may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFailed > + value: 10 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed without finding a good focus position. > + - name: AfStateContinuousFailedPaused > + value: 11 > + description: | > + The AF algorithm was in the AfStateContinuousFailed state and > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and the > + scan has completed unsuccessfully. The algorithm will now remain > + in this state, and may be resumed by sending AfPauseResume. > + > ... I think this is a very important step, and I would like it integrated as soon as possible. With the fixes: Reviewed-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com> Thanks ! JM
Hi David, thanks for the patch! Reviewed-by: Han-Lin Chen <hanlinchen@chromium.org> On Thu, Mar 31, 2022 at 11:17 PM David Plowman < david.plowman@raspberrypi.com> wrote: > This patch describes a series of controls that allow applications to > drive AF algorithms: > > AfMode - manual, auto or continuous > AfLensPosition - set lens position to macro/hyperfocal/infinity > AfRange - full, macro or normal > AfSpeed - fast or slow > AfWindows - AF window locations > AfTrigger - start (trigger an AF scan) or cancel > AfPause - pause continuous AF > LensPosition - position of lens from lens driver > AfState - reports the mode, whether scanning/success/failure > --- > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > 1 file changed, 313 insertions(+), 60 deletions(-) > > diff --git a/src/libcamera/control_ids.yaml > b/src/libcamera/control_ids.yaml > index 9d4638ae..23607368 100644 > --- a/src/libcamera/control_ids.yaml > +++ b/src/libcamera/control_ids.yaml > @@ -406,27 +406,6 @@ controls: > The camera will cancel any active or completed metering > sequence. > The AE algorithm is reset to its initial state. > > - - AfTrigger: > - type: int32_t > - draft: true > - description: | > - Control for AF trigger. Currently identical to > - ANDROID_CONTROL_AF_TRIGGER. > - > - Whether the camera device will trigger autofocus for this request. > - enum: > - - name: AfTriggerIdle > - value: 0 > - description: The trigger is idle. > - - name: AfTriggerStart > - value: 1 > - description: The AF routine is started by the camera. > - - name: AfTriggerCancel > - value: 2 > - description: | > - The camera will cancel any active trigger and the AF routine > is > - reset to its initial state. > - > - NoiseReductionMode: > type: int32_t > draft: true > @@ -507,45 +486,6 @@ controls: > The AE algorithm has started a pre-capture metering session. > \sa AePrecaptureTrigger > > - - AfState: > - type: int32_t > - draft: true > - description: | > - Control to report the current AF algorithm state. Currently > identical to > - ANDROID_CONTROL_AF_STATE. > - > - Current state of the AF algorithm. > - enum: > - - name: AfStateInactive > - value: 0 > - description: The AF algorithm is inactive. > - - name: AfStatePassiveScan > - value: 1 > - description: | > - AF is performing a passive scan of the scene in continuous > - auto-focus mode. > - - name: AfStatePassiveFocused > - value: 2 > - description: | > - AF believes the scene is in focus, but might restart scanning. > - - name: AfStateActiveScan > - value: 3 > - description: | > - AF is performing a scan triggered by an AF trigger request. > - \sa AfTrigger > - - name: AfStateFocusedLock > - value: 4 > - description: | > - AF believes has focused correctly and has locked focus. > - - name: AfStateNotFocusedLock > - value: 5 > - description: | > - AF has not been able to focus and has locked. > - - name: AfStatePassiveUnfocused > - value: 6 > - description: | > - AF has completed a passive scan without finding focus. > - > - AwbState: > type: int32_t > draft: true > @@ -690,4 +630,317 @@ controls: > value. All of the custom test patterns will be static (that > is the > raw image must not vary from frame to frame). > > + - AfMode: > + type: int32_t > + draft: true > + description: | > + Control to set the mode of the AF (autofocus) algorithm. > Applications > + are allowed to set a new mode, and to send additional controls for > + that new mode, in the same request. > + > + An implementation may choose not to implement all the modes. > + > + enum: > + - name: AfModeManual > + value: 0 > + description: | > + The AF algorithm is in manual mode. In this mode it will never > + perform any action nor move the lens of its own accord, but an > + application can set controls to move the lens. > + > + In this mode the AfState will always report AfStateManual. > + - name: AfModeAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode. This means that the > algorithm > + will never move the lens or change state unless the AfTrigger > + control is used. The AfTrigger control can be used to > initiate a > + focus scan, the results of which will also be reported by > AfState. > + > + If the autofocus algorithm is moved from AfModeAuto to another > + mode while a scan is in progress, the scan is cancelled > + immediately, without waiting for the scan to finish. > + > + When first entering this mode the AfState will report > + AfStateAuto. When a trigger control is sent, AfState will > + report AfStateAutoScanning for a period before spontaneously > + changing to AfStateAutoFocused or AfStateAutoFailed, > depending on > + the outcome of the scan. It will remain in this state until > + another scan is initiated by the AfTrigger control. If a scan > is > + cancelled (without changing to another mode), AfState will > return > + to AfStateAuto. > + - name: AfModeContinuous > + value: 2 > + description: | > + The AF algorithm is in continuous mode. This means that the > lens > + can re-start a scan spontaneously at any moment, without any > user > + intervention. The AfState still reports whether the algorithm > is > + currently scanning or not, though the application has no > ability > + to initiate or cancel scans (though it can "pause" them), nor > to > + move the lens for itself. > + > + When set to AfModeContinuous, the system will immediately > initiate > + a scan so AfState will report AfStateContinuousScanning, and > will > + settle on one of AfStateContinuousFocused or > AfStateContinuousFailed, > + depending on the scan result. > + > + - AfLensPosition: > + type: int32_t > + draft: true > + description: | > + Control to set the position of the lens to one of the following > + predefined locations. This control only has any effect when the > AfMode > + is set to AfModeManual. > + > + This control is distinct from the LensPosition control, which > sets the > + lens position using the lens driver's units. > + enum: > + - name: AfLensPositionMacro > + value: 0 > + description: The closest focal position of the lens. > + - name: AfLensPositionHyperfocal > + value: 1 > + description: Hyperfocal position. > + - name: AfLensPositionInfinity > + value: 2 > + description: The furthest focal position (usually infinity). > + > + - AfRange: > + type: int32_t > + draft: true > + description: | > + Control to set the range of focus distances that is scanned. An > + implementation may choose not to implement all the options here. > + enum: > + - name: AfRangeNormal > + value: 0 > + description: | > + A wide range of focus distances is scanned, all the way from > + infinity down to close distances, though depending on the > + implementation, possibly not including the very closest macro > + positions. > + - name: AfRangeMacro > + value: 1 > + description: Only close distances are scanned. > + - name: AfRangeFull > + value: 2 > + description: | > + The full range of focus distances is scanned just as with > + AfRangeNormal but this time including the very closest macro > + positions. > + > + - AfSpeed: > + type: int32_t > + draft: true > + description: | > + Control that determines whether the AF algorithm is to move the > lens > + as quickly as possible or more steadily. For example, during video > + recording it may be desirable not to move the lens too abruptly, > but > + when in a preview mode (waiting for a still capture) it may be > + helpful to move the lens as quickly as is reasonably possible. > + enum: > + - name: AfSpeedNormal > + value: 0 > + description: Move the lens at its usual speed. > + - name: AfSpeedFast > + value: 1 > + description: Move the lens more quickly. > + > + - AfWindows: > + type: Rectangle > + draft: true > + description: | > + Sets the focus windows used by the AF algorithm. The units used > express > + a proportion of the ScalerCrop control (or if unavailable, of the > entire > + image), as u0.16 format numbers. > + > + In order to be activated, a rectangle must be programmed with > non-zero > + width and height. If no rectangles are programmed in this way, > then the > + system will choose its own single default window in the centre of > the > + image. > + > + The details of how the windows are used are platform dependent. > We note > + that when there is more than one AF window, a typical > implementation > + might find the optimal focus position for each one and finally > select > + the window closest to the camera. > + > + size: [platform dependent] > + > + - AfTrigger: > + type: int32_t > + draft: true > + description: | > + This control starts an autofocus scan when AfMode is set to > AfModeAuto, > + and can also be used to terminate a scan early. > + > + It is ignored if AfMode is set to AfModeManual or > AfModeContinuous. > + > + enum: > + - name: AfTriggerStart > + value: 0 > + description: Start an AF scan. Ignored if a scan is in progress. > + - name: AfTriggerCancel > + value: 1 > + description: Cancel an AF scan. This does not cause the lens to > move > + anywhere else. Ignored if no scan is in progress. > + > + - AfPause: > + type: int32_t > + draft: true > + description: | > + This control has no effect except when in continuous autofocus > mode > + (AfModeContinuous). It can be used to pause any lens movements > while > + (for example) images are captured. The algorithm remains inactive > + until it is instructed to resume. > + > + enum: > + - name: AfPauseImmediate > + value: 0 > + description: | > + Pause the continuous autofocus algorithm immediately, whether > or > + not any kind of scan is underway. If the AfState was > previously > + reporting AfStateContinuousScanning it will now change to > + AfStateContinuousScanningPaused, and similarly for > + AfStateContinuousFocused and AfStateContinuousFailed. > + - name AfPauseDeferred > + value: 1 > + description: | > + Pause the continuous autofocus algorithm as soon as it is no > longer > + scanning. If the AfState is currently reporting > + AfStateContinuousFocused is will change to > + AfStateContinuousFocusedPaused, and similarly in the case of > + AfStateContinuousFailed. > + > + If AfState reports AfStateContinuousScanning it will change to > + AfStateContinuousScanningPausing, and then move to one of > + AfStateContinuousFocusedPaused or > AfStateContinuousFailedPaused > + when the scan completes. > + - name: AfPauseResume > + value: 2 > + description: | > + Resume continous autofocus operation. The algorithm starts > again > + from exactly where it left off, and the AfState will drop the > + 'Paused' or 'Pausing' part of the state. > + > + - LensPosition: > + type: int32_t > + draft: true > + description: | > + Acts as a control to instruct the lens to move to a particular > position > + and also reports back the position of the lens for each frame. > + > + The units are determined by the lens driver. > + > + The LensPosition control is ignored unless the AfMode is set to > + AfModeManual. > + > + Note that this control is distinct from AfLensPosition, which > allows > + the lens to be moved to its macro/hyperfocal/infinity position, > rather > + than using lens driver units. > + > + - AfState: > + type: int32_t > + draft: true > + description: | > + Reports the current state of the AF algorithm. The possible state > changes > + are described below, although we add that the following general > state > + changes are also allowed. > + > + In any of the AfStateManual or AfStateContinuous states, the > AfMode > + may be set to AfModeAuto and the algorithm will move to the > + AfStateAuto state. If AfTriggerStart is sent at the same time > + then the algorithm will move to AfStateAutoScanning state. > + > + In any of the AfStateAuto or AfStateManual states, the AfMode may > + be set to AfModeContinuous and the algorithm will move to > + AfStateContinuousScanning. > + > + In any of the AfStateAuto or AfStateContinuous states, the AfMode > may > + be set to AfModeManual and the algorithm will move to > + AfStateManual. The lens will not be moved and will be left where > + it was at that moment. > + > + enum: > + - name: AfStateManual > + value: 0 > + description: | > + The AF algorithm is in manual mode (AfModeManual). The > LensPosition > + and AfLensPosition controls can be used directly to move the > lens. > + - name: AfStateAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and has either > just > + been moved into that state, or a scan that was in progress > has been > + cancelled. > + - name: AfStateAutoScanning > + value: 2 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > been > + started using the AfTrigger control. The scan can be > cancelled by > + sending AfTriggerCancel at which point the algorithm will > either > + move back to AfStateAuto or, if the scan actually completes > + before the cancel request is processed, to one of > + AfStateAutoFocused or AfStateAutoFailed. > + - name: AfStateAutoFocused > + value: 3 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm believes the > image is > + now in focus. > + - name: AfStateAutoFailed > + value: 4 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm did not find a > good > + focus position. > + - name: AfStateContinuousScanning > + value: 5 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + is currently scanning for a good focus position. This occurs > when > + the mode is first set to AfModeContinuous, or may happen > + spontaneously when the algorithm believes a re-scan is needed. > + - name: AfStateContinuousScanningPausing > + value: 6 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseDeferred was sent. The scan will > complete > + at which point the algorithm moves to the > + AfStateContinuousFocusedPaused or > AfStateContinuousFailedPaused > + state. > + - name: AfStateContinuousScanningPaused > + value: 7 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseImmediate was sent. The scan will not > + complete and the algorithm will remain in this state. The scan > + may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFocused > + value: 8 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed with the algorithm believing it has > found a > + good focus position. > + - name: AfStateContinuousFocusedPaused > + value: 9 > + description: | > + The AF algorithm was in the AfStateContinuousFocused state and > + has been paused (by either AfPauseImmediate or > AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and > the > + scan has completed successfully. The algorithm will now remain > + in this state, and may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFailed > + value: 10 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed without finding a good focus position. > + - name: AfStateContinuousFailedPaused > + value: 11 > + description: | > + The AF algorithm was in the AfStateContinuousFailed state and > + has been paused (by either AfPauseImmediate or > AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and > the > + scan has completed unsuccessfully. The algorithm will now > remain > + in this state, and may be resumed by sending AfPauseResume. > + > ... > -- > 2.30.2 > >
Hi Thanks to those who have had a chance to look at this! On Fri, 1 Apr 2022 at 10:37, Hanlin Chen <hanlinchen@chromium.org> wrote: > > Hi David, thanks for the patch! > Reviewed-by: Han-Lin Chen <hanlinchen@chromium.org> > > On Thu, Mar 31, 2022 at 11:17 PM David Plowman <david.plowman@raspberrypi.com> wrote: >> >> This patch describes a series of controls that allow applications to >> drive AF algorithms: >> >> AfMode - manual, auto or continuous >> AfLensPosition - set lens position to macro/hyperfocal/infinity >> AfRange - full, macro or normal >> AfSpeed - fast or slow >> AfWindows - AF window locations >> AfTrigger - start (trigger an AF scan) or cancel >> AfPause - pause continuous AF >> LensPosition - position of lens from lens driver >> AfState - reports the mode, whether scanning/success/failure >> --- >> src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ >> 1 file changed, 313 insertions(+), 60 deletions(-) >> >> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml >> index 9d4638ae..23607368 100644 >> --- a/src/libcamera/control_ids.yaml >> +++ b/src/libcamera/control_ids.yaml >> @@ -406,27 +406,6 @@ controls: >> The camera will cancel any active or completed metering sequence. >> The AE algorithm is reset to its initial state. >> >> - - AfTrigger: >> - type: int32_t >> - draft: true >> - description: | >> - Control for AF trigger. Currently identical to >> - ANDROID_CONTROL_AF_TRIGGER. >> - >> - Whether the camera device will trigger autofocus for this request. >> - enum: >> - - name: AfTriggerIdle >> - value: 0 >> - description: The trigger is idle. >> - - name: AfTriggerStart >> - value: 1 >> - description: The AF routine is started by the camera. >> - - name: AfTriggerCancel >> - value: 2 >> - description: | >> - The camera will cancel any active trigger and the AF routine is >> - reset to its initial state. >> - >> - NoiseReductionMode: >> type: int32_t >> draft: true >> @@ -507,45 +486,6 @@ controls: >> The AE algorithm has started a pre-capture metering session. >> \sa AePrecaptureTrigger >> >> - - AfState: >> - type: int32_t >> - draft: true >> - description: | >> - Control to report the current AF algorithm state. Currently identical to >> - ANDROID_CONTROL_AF_STATE. >> - >> - Current state of the AF algorithm. >> - enum: >> - - name: AfStateInactive >> - value: 0 >> - description: The AF algorithm is inactive. >> - - name: AfStatePassiveScan >> - value: 1 >> - description: | >> - AF is performing a passive scan of the scene in continuous >> - auto-focus mode. >> - - name: AfStatePassiveFocused >> - value: 2 >> - description: | >> - AF believes the scene is in focus, but might restart scanning. >> - - name: AfStateActiveScan >> - value: 3 >> - description: | >> - AF is performing a scan triggered by an AF trigger request. >> - \sa AfTrigger >> - - name: AfStateFocusedLock >> - value: 4 >> - description: | >> - AF believes has focused correctly and has locked focus. >> - - name: AfStateNotFocusedLock >> - value: 5 >> - description: | >> - AF has not been able to focus and has locked. >> - - name: AfStatePassiveUnfocused >> - value: 6 >> - description: | >> - AF has completed a passive scan without finding focus. >> - >> - AwbState: >> type: int32_t >> draft: true >> @@ -690,4 +630,317 @@ controls: >> value. All of the custom test patterns will be static (that is the >> raw image must not vary from frame to frame). >> >> + - AfMode: >> + type: int32_t >> + draft: true >> + description: | >> + Control to set the mode of the AF (autofocus) algorithm. Applications >> + are allowed to set a new mode, and to send additional controls for >> + that new mode, in the same request. >> + >> + An implementation may choose not to implement all the modes. >> + >> + enum: >> + - name: AfModeManual >> + value: 0 >> + description: | >> + The AF algorithm is in manual mode. In this mode it will never >> + perform any action nor move the lens of its own accord, but an >> + application can set controls to move the lens. >> + >> + In this mode the AfState will always report AfStateManual. >> + - name: AfModeAuto >> + value: 1 >> + description: | >> + The AF algorithm is in auto mode. This means that the algorithm >> + will never move the lens or change state unless the AfTrigger >> + control is used. The AfTrigger control can be used to initiate a >> + focus scan, the results of which will also be reported by AfState. >> + >> + If the autofocus algorithm is moved from AfModeAuto to another >> + mode while a scan is in progress, the scan is cancelled >> + immediately, without waiting for the scan to finish. >> + >> + When first entering this mode the AfState will report >> + AfStateAuto. When a trigger control is sent, AfState will >> + report AfStateAutoScanning for a period before spontaneously >> + changing to AfStateAutoFocused or AfStateAutoFailed, depending on >> + the outcome of the scan. It will remain in this state until >> + another scan is initiated by the AfTrigger control. If a scan is >> + cancelled (without changing to another mode), AfState will return >> + to AfStateAuto. >> + - name: AfModeContinuous >> + value: 2 >> + description: | >> + The AF algorithm is in continuous mode. This means that the lens >> + can re-start a scan spontaneously at any moment, without any user >> + intervention. The AfState still reports whether the algorithm is >> + currently scanning or not, though the application has no ability >> + to initiate or cancel scans (though it can "pause" them), nor to >> + move the lens for itself. >> + >> + When set to AfModeContinuous, the system will immediately initiate >> + a scan so AfState will report AfStateContinuousScanning, and will >> + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, >> + depending on the scan result. >> + >> + - AfLensPosition: >> + type: int32_t >> + draft: true >> + description: | >> + Control to set the position of the lens to one of the following >> + predefined locations. This control only has any effect when the AfMode >> + is set to AfModeManual. >> + >> + This control is distinct from the LensPosition control, which sets the >> + lens position using the lens driver's units. >> + enum: >> + - name: AfLensPositionMacro >> + value: 0 >> + description: The closest focal position of the lens. >> + - name: AfLensPositionHyperfocal >> + value: 1 >> + description: Hyperfocal position. >> + - name: AfLensPositionInfinity >> + value: 2 >> + description: The furthest focal position (usually infinity). >> + >> + - AfRange: >> + type: int32_t >> + draft: true >> + description: | >> + Control to set the range of focus distances that is scanned. An >> + implementation may choose not to implement all the options here. >> + enum: >> + - name: AfRangeNormal >> + value: 0 >> + description: | >> + A wide range of focus distances is scanned, all the way from >> + infinity down to close distances, though depending on the >> + implementation, possibly not including the very closest macro >> + positions. >> + - name: AfRangeMacro >> + value: 1 >> + description: Only close distances are scanned. >> + - name: AfRangeFull >> + value: 2 >> + description: | >> + The full range of focus distances is scanned just as with >> + AfRangeNormal but this time including the very closest macro >> + positions. >> + >> + - AfSpeed: >> + type: int32_t >> + draft: true >> + description: | >> + Control that determines whether the AF algorithm is to move the lens >> + as quickly as possible or more steadily. For example, during video >> + recording it may be desirable not to move the lens too abruptly, but >> + when in a preview mode (waiting for a still capture) it may be >> + helpful to move the lens as quickly as is reasonably possible. >> + enum: >> + - name: AfSpeedNormal >> + value: 0 >> + description: Move the lens at its usual speed. >> + - name: AfSpeedFast >> + value: 1 >> + description: Move the lens more quickly. >> + >> + - AfWindows: >> + type: Rectangle >> + draft: true >> + description: | >> + Sets the focus windows used by the AF algorithm. The units used express >> + a proportion of the ScalerCrop control (or if unavailable, of the entire >> + image), as u0.16 format numbers. >> + >> + In order to be activated, a rectangle must be programmed with non-zero >> + width and height. If no rectangles are programmed in this way, then the >> + system will choose its own single default window in the centre of the >> + image. >> + >> + The details of how the windows are used are platform dependent. We note >> + that when there is more than one AF window, a typical implementation >> + might find the optimal focus position for each one and finally select >> + the window closest to the camera. >> + >> + size: [platform dependent] >> + >> + - AfTrigger: >> + type: int32_t >> + draft: true >> + description: | >> + This control starts an autofocus scan when AfMode is set to AfModeAuto, >> + and can also be used to terminate a scan early. >> + >> + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. >> + >> + enum: >> + - name: AfTriggerStart >> + value: 0 >> + description: Start an AF scan. Ignored if a scan is in progress. >> + - name: AfTriggerCancel >> + value: 1 >> + description: Cancel an AF scan. This does not cause the lens to move >> + anywhere else. Ignored if no scan is in progress. >> + >> + - AfPause: >> + type: int32_t >> + draft: true >> + description: | >> + This control has no effect except when in continuous autofocus mode >> + (AfModeContinuous). It can be used to pause any lens movements while >> + (for example) images are captured. The algorithm remains inactive >> + until it is instructed to resume. >> + >> + enum: >> + - name: AfPauseImmediate >> + value: 0 >> + description: | >> + Pause the continuous autofocus algorithm immediately, whether or >> + not any kind of scan is underway. If the AfState was previously >> + reporting AfStateContinuousScanning it will now change to >> + AfStateContinuousScanningPaused, and similarly for >> + AfStateContinuousFocused and AfStateContinuousFailed. >> + - name AfPauseDeferred Thanks for pointing out the typo here, Jean-Michel. I will fix it in the next revision (I'll just wait a bit and see if any other reviews come in..). >> + value: 1 >> + description: | >> + Pause the continuous autofocus algorithm as soon as it is no longer >> + scanning. If the AfState is currently reporting >> + AfStateContinuousFocused is will change to >> + AfStateContinuousFocusedPaused, and similarly in the case of >> + AfStateContinuousFailed. >> + >> + If AfState reports AfStateContinuousScanning it will change to >> + AfStateContinuousScanningPausing, and then move to one of >> + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused >> + when the scan completes. >> + - name: AfPauseResume >> + value: 2 >> + description: | >> + Resume continous autofocus operation. The algorithm starts again >> + from exactly where it left off, and the AfState will drop the >> + 'Paused' or 'Pausing' part of the state. >> + >> + - LensPosition: >> + type: int32_t >> + draft: true >> + description: | >> + Acts as a control to instruct the lens to move to a particular position >> + and also reports back the position of the lens for each frame. >> + >> + The units are determined by the lens driver. >> + >> + The LensPosition control is ignored unless the AfMode is set to >> + AfModeManual. >> + >> + Note that this control is distinct from AfLensPosition, which allows >> + the lens to be moved to its macro/hyperfocal/infinity position, rather >> + than using lens driver units. >> + >> + - AfState: >> + type: int32_t >> + draft: true >> + description: | >> + Reports the current state of the AF algorithm. The possible state changes >> + are described below, although we add that the following general state >> + changes are also allowed. >> + >> + In any of the AfStateManual or AfStateContinuous states, the AfMode >> + may be set to AfModeAuto and the algorithm will move to the >> + AfStateAuto state. If AfTriggerStart is sent at the same time >> + then the algorithm will move to AfStateAutoScanning state. >> + >> + In any of the AfStateAuto or AfStateManual states, the AfMode may >> + be set to AfModeContinuous and the algorithm will move to >> + AfStateContinuousScanning. >> + >> + In any of the AfStateAuto or AfStateContinuous states, the AfMode may >> + be set to AfModeManual and the algorithm will move to >> + AfStateManual. The lens will not be moved and will be left where >> + it was at that moment. >> + >> + enum: >> + - name: AfStateManual >> + value: 0 >> + description: | >> + The AF algorithm is in manual mode (AfModeManual). The LensPosition >> + and AfLensPosition controls can be used directly to move the lens. >> + - name: AfStateAuto >> + value: 1 >> + description: | >> + The AF algorithm is in auto mode (AfModeAuto), and has either just >> + been moved into that state, or a scan that was in progress has been >> + cancelled. >> + - name: AfStateAutoScanning >> + value: 2 >> + description: | >> + The AF algorithm is in auto mode (AfModeAuto), and a scan has been >> + started using the AfTrigger control. The scan can be cancelled by >> + sending AfTriggerCancel at which point the algorithm will either >> + move back to AfStateAuto or, if the scan actually completes >> + before the cancel request is processed, to one of >> + AfStateAutoFocused or AfStateAutoFailed. >> + - name: AfStateAutoFocused >> + value: 3 >> + description: | >> + The AF algorithm is in auto mode (AfModeAuto), and a scan has >> + completed with the result that the algorithm believes the image is >> + now in focus. >> + - name: AfStateAutoFailed >> + value: 4 >> + description: | >> + The AF algorithm is in auto mode (AfModeAuto), and a scan has >> + completed with the result that the algorithm did not find a good >> + focus position. >> + - name: AfStateContinuousScanning >> + value: 5 >> + description: | >> + The AF algorithm is in continuous mode (AfModeContinuous) and >> + is currently scanning for a good focus position. This occurs when >> + the mode is first set to AfModeContinuous, or may happen >> + spontaneously when the algorithm believes a re-scan is needed. >> + - name: AfStateContinuousScanningPausing >> + value: 6 >> + description: | >> + The AF algorithm is in continuous mode (AfModeContinuous) and >> + was scanning when AfPauseDeferred was sent. The scan will complete >> + at which point the algorithm moves to the >> + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused >> + state. >> + - name: AfStateContinuousScanningPaused >> + value: 7 >> + description: | >> + The AF algorithm is in continuous mode (AfModeContinuous) and >> + was scanning when AfPauseImmediate was sent. The scan will not >> + complete and the algorithm will remain in this state. The scan >> + may be resumed by sending AfPauseResume. >> + - name: AfStateContinuousFocused >> + value: 8 >> + description: | >> + The AF algorithm is in continuous mode (AfModeContinuous) and >> + a scan has completed with the algorithm believing it has found a >> + good focus position. >> + - name: AfStateContinuousFocusedPaused >> + value: 9 >> + description: | >> + The AF algorithm was in the AfStateContinuousFocused state and >> + has been paused (by either AfPauseImmediate or AfPauseDeferred), >> + or it was in the AfStateContinuousScanningPausing state and the >> + scan has completed successfully. The algorithm will now remain >> + in this state, and may be resumed by sending AfPauseResume. >> + - name: AfStateContinuousFailed >> + value: 10 >> + description: | >> + The AF algorithm is in continuous mode (AfModeContinuous) and >> + a scan has completed without finding a good focus position. >> + - name: AfStateContinuousFailedPaused >> + value: 11 >> + description: | >> + The AF algorithm was in the AfStateContinuousFailed state and >> + has been paused (by either AfPauseImmediate or AfPauseDeferred), >> + or it was in the AfStateContinuousScanningPausing state and the >> + scan has completed unsuccessfully. The algorithm will now remain >> + in this state, and may be resumed by sending AfPauseResume. >> + >> ... >> -- >> 2.30.2 >> I was also wondering if there were any comments on the state machine diagram or if folks are happy with it. I'm not wild about the "paused/pausing" thing, but I guess if we want that feature then we have to distinguish these states. I haven't allowed for transitions between "AfStateContinuousScanningPausing" and "AfStateContinuousScanningPaused" which you could imagine might exist, are people OK with that? Thanks again David
Hi David, sorry for being late :) On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote: > This patch describes a series of controls that allow applications to > drive AF algorithms: > > AfMode - manual, auto or continuous > AfLensPosition - set lens position to macro/hyperfocal/infinity > AfRange - full, macro or normal > AfSpeed - fast or slow > AfWindows - AF window locations > AfTrigger - start (trigger an AF scan) or cancel > AfPause - pause continuous AF > LensPosition - position of lens from lens driver > AfState - reports the mode, whether scanning/success/failure > --- > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > 1 file changed, 313 insertions(+), 60 deletions(-) > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > index 9d4638ae..23607368 100644 > --- a/src/libcamera/control_ids.yaml > +++ b/src/libcamera/control_ids.yaml > @@ -406,27 +406,6 @@ controls: > The camera will cancel any active or completed metering sequence. > The AE algorithm is reset to its initial state. > > - - AfTrigger: > - type: int32_t > - draft: true > - description: | > - Control for AF trigger. Currently identical to > - ANDROID_CONTROL_AF_TRIGGER. > - > - Whether the camera device will trigger autofocus for this request. > - enum: > - - name: AfTriggerIdle > - value: 0 > - description: The trigger is idle. > - - name: AfTriggerStart > - value: 1 > - description: The AF routine is started by the camera. > - - name: AfTriggerCancel > - value: 2 > - description: | > - The camera will cancel any active trigger and the AF routine is > - reset to its initial state. > - > - NoiseReductionMode: > type: int32_t > draft: true > @@ -507,45 +486,6 @@ controls: > The AE algorithm has started a pre-capture metering session. > \sa AePrecaptureTrigger > > - - AfState: > - type: int32_t > - draft: true > - description: | > - Control to report the current AF algorithm state. Currently identical to > - ANDROID_CONTROL_AF_STATE. > - > - Current state of the AF algorithm. > - enum: > - - name: AfStateInactive > - value: 0 > - description: The AF algorithm is inactive. > - - name: AfStatePassiveScan > - value: 1 > - description: | > - AF is performing a passive scan of the scene in continuous > - auto-focus mode. > - - name: AfStatePassiveFocused > - value: 2 > - description: | > - AF believes the scene is in focus, but might restart scanning. > - - name: AfStateActiveScan > - value: 3 > - description: | > - AF is performing a scan triggered by an AF trigger request. > - \sa AfTrigger > - - name: AfStateFocusedLock > - value: 4 > - description: | > - AF believes has focused correctly and has locked focus. > - - name: AfStateNotFocusedLock > - value: 5 > - description: | > - AF has not been able to focus and has locked. > - - name: AfStatePassiveUnfocused > - value: 6 > - description: | > - AF has completed a passive scan without finding focus. > - > - AwbState: > type: int32_t > draft: true > @@ -690,4 +630,317 @@ controls: > value. All of the custom test patterns will be static (that is the > raw image must not vary from frame to frame). > > + - AfMode: > + type: int32_t > + draft: true Was having all these controls as draft intentional ? The previous definitions where set to drafts as they mapped to the android definition, but now that we're defining our own ones... > + description: | > + Control to set the mode of the AF (autofocus) algorithm. Applications > + are allowed to set a new mode, and to send additional controls for > + that new mode, in the same request. > + > + An implementation may choose not to implement all the modes. > + > + enum: > + - name: AfModeManual > + value: 0 > + description: | > + The AF algorithm is in manual mode. In this mode it will never > + perform any action nor move the lens of its own accord, but an > + application can set controls to move the lens. > + > + In this mode the AfState will always report AfStateManual. > + - name: AfModeAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode. This means that the algorithm > + will never move the lens or change state unless the AfTrigger > + control is used. The AfTrigger control can be used to initiate a > + focus scan, the results of which will also be reported by AfState. > + > + If the autofocus algorithm is moved from AfModeAuto to another > + mode while a scan is in progress, the scan is cancelled > + immediately, without waiting for the scan to finish. > + > + When first entering this mode the AfState will report > + AfStateAuto. When a trigger control is sent, AfState will > + report AfStateAutoScanning for a period before spontaneously > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > + the outcome of the scan. It will remain in this state until > + another scan is initiated by the AfTrigger control. If a scan is > + cancelled (without changing to another mode), AfState will return > + to AfStateAuto. > + - name: AfModeContinuous > + value: 2 > + description: | > + The AF algorithm is in continuous mode. This means that the lens > + can re-start a scan spontaneously at any moment, without any user > + intervention. The AfState still reports whether the algorithm is > + currently scanning or not, though the application has no ability > + to initiate or cancel scans (though it can "pause" them), nor to > + move the lens for itself. > + > + When set to AfModeContinuous, the system will immediately initiate > + a scan so AfState will report AfStateContinuousScanning, and will > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > + depending on the scan result. > + > + - AfLensPosition: > + type: int32_t > + draft: true > + description: | > + Control to set the position of the lens to one of the following > + predefined locations. This control only has any effect when the AfMode > + is set to AfModeManual. > + > + This control is distinct from the LensPosition control, which sets the > + lens position using the lens driver's units. > + enum: > + - name: AfLensPositionMacro > + value: 0 > + description: The closest focal position of the lens. > + - name: AfLensPositionHyperfocal > + value: 1 > + description: Hyperfocal position. > + - name: AfLensPositionInfinity > + value: 2 > + description: The furthest focal position (usually infinity). Interesting, bikeshedding on the name apart, is this some 'simplified' LensState ? Do you think there is a use case for this ? Why would anyone set the AF to auto mode then use some coarse grained control like this one ? Wouldn't it make more sense as an 'hint' to the AF algorithm to restrict the search space ? (Btw I have a use case for supplying a range of lens positions, which has been previously estimated according to some pre-defined use cases, to the AF algorithm in order restrict the AF search range to increase the speed of a scan. Do you think it's a valid use case and it's worth a control ?) > + > + - AfRange: > + type: int32_t > + draft: true > + description: | > + Control to set the range of focus distances that is scanned. An > + implementation may choose not to implement all the options here. Only valid in Auto and Continuous mode, right ? (This makes me think that we could easily model Continuous by adding a TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of AfPause... well, it's quite a redesign so I'll stop here unless someone thinks it's a really good idea :) > + enum: > + - name: AfRangeNormal > + value: 0 > + description: | > + A wide range of focus distances is scanned, all the way from > + infinity down to close distances, though depending on the > + implementation, possibly not including the very closest macro > + positions. > + - name: AfRangeMacro > + value: 1 > + description: Only close distances are scanned. > + - name: AfRangeFull > + value: 2 > + description: | > + The full range of focus distances is scanned just as with > + AfRangeNormal but this time including the very closest macro > + positions. > + > + - AfSpeed: > + type: int32_t > + draft: true > + description: | > + Control that determines whether the AF algorithm is to move the lens > + as quickly as possible or more steadily. For example, during video > + recording it may be desirable not to move the lens too abruptly, but > + when in a preview mode (waiting for a still capture) it may be > + helpful to move the lens as quickly as is reasonably possible. > + enum: > + - name: AfSpeedNormal > + value: 0 > + description: Move the lens at its usual speed. > + - name: AfSpeedFast > + value: 1 > + description: Move the lens more quickly. > + > + - AfWindows: > + type: Rectangle > + draft: true > + description: | > + Sets the focus windows used by the AF algorithm. The units used express > + a proportion of the ScalerCrop control (or if unavailable, of the entire > + image), as u0.16 format numbers. > + > + In order to be activated, a rectangle must be programmed with non-zero > + width and height. If no rectangles are programmed in this way, then the > + system will choose its own single default window in the centre of the > + image. > + > + The details of how the windows are used are platform dependent. We note > + that when there is more than one AF window, a typical implementation > + might find the optimal focus position for each one and finally select > + the window closest to the camera. > + > + size: [platform dependent] I think this should just be size: [n] > + > + - AfTrigger: > + type: int32_t > + draft: true > + description: | > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > + and can also be used to terminate a scan early. > + > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > + > + enum: > + - name: AfTriggerStart > + value: 0 > + description: Start an AF scan. Ignored if a scan is in progress. > + - name: AfTriggerCancel > + value: 1 > + description: Cancel an AF scan. This does not cause the lens to move > + anywhere else. Ignored if no scan is in progress. > + > + - AfPause: > + type: int32_t > + draft: true > + description: | > + This control has no effect except when in continuous autofocus mode > + (AfModeContinuous). It can be used to pause any lens movements while > + (for example) images are captured. The algorithm remains inactive > + until it is instructed to resume. > + > + enum: > + - name: AfPauseImmediate > + value: 0 > + description: | > + Pause the continuous autofocus algorithm immediately, whether or > + not any kind of scan is underway. If the AfState was previously > + reporting AfStateContinuousScanning it will now change to > + AfStateContinuousScanningPaused, and similarly for > + AfStateContinuousFocused and AfStateContinuousFailed. > + - name AfPauseDeferred > + value: 1 > + description: | > + Pause the continuous autofocus algorithm as soon as it is no longer > + scanning. If the AfState is currently reporting > + AfStateContinuousFocused is will change to > + AfStateContinuousFocusedPaused, and similarly in the case of > + AfStateContinuousFailed. > + > + If AfState reports AfStateContinuousScanning it will change to > + AfStateContinuousScanningPausing, and then move to one of > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > + when the scan completes. > + - name: AfPauseResume > + value: 2 > + description: | > + Resume continous autofocus operation. The algorithm starts again > + from exactly where it left off, and the AfState will drop the > + 'Paused' or 'Pausing' part of the state. > + > + - LensPosition: > + type: int32_t > + draft: true > + description: | > + Acts as a control to instruct the lens to move to a particular position > + and also reports back the position of the lens for each frame. > + > + The units are determined by the lens driver. Ugh, we cannot do this in the control definition. Driver specificities cannot surface to the controls definition, which should be platform independent. I think we'll have to establish a numerical range (go ahead if you have any idea) and translate to the actual driver values in the CameraLens class ? > + > + The LensPosition control is ignored unless the AfMode is set to > + AfModeManual. > + > + Note that this control is distinct from AfLensPosition, which allows > + the lens to be moved to its macro/hyperfocal/infinity position, rather > + than using lens driver units. > + > + - AfState: > + type: int32_t > + draft: true > + description: | > + Reports the current state of the AF algorithm. The possible state changes > + are described below, although we add that the following general state > + changes are also allowed. wow, there is a lot of states here :) Let me copy them here - name: AfStateManual - name: AfStateAuto - name: AfStateAutoScanning - name: AfStateAutoFocused - name: AfStateAutoFailed - name: AfStateContinuousScanning - name: AfStateContinuousScanningPausing - name: AfStateContinuousScanningPaused - name: AfStateContinuousFocused - name: AfStateContinuousFocusedPaused - name: AfStateContinuousFailed - name: AfStateContinuousFailedPaused (please note that AfStateAuto seems mis-aligned, unless I messed up something while cutting&paste. Please check the compiled doc to see how it looks like...) So we have three macro groups - Manual - Auto - Scanning - Focus - Failed - Continuous - Scanning - Pausing - Paused - Focused - Paused - Failed - Paused The first group is AfMode (Manual, Auto, Continuous) and can be deduced by inspecting the AfMode metadata. The second group is the actual state (Scannng, Focused, Failed) The third group is specific to continuous mode as it accepts a Pause trigger. I understand why you have this, as expressing "The AfState is set to $something when in AfModeManual" etc etc might get cumbersome, but I think it might be worth a try 1) Manual mode (the bad) As we all know there is a delay between when a lens is instructed to move and when it reaches its final state. In Manual mode the lens is moved with LensPosition and AfLensPosition. To know the current state: - With LensPosition metadata can be inspected until it doesn't reach the desired value. - With AfLensPosition it is not clear to me how to know when the lens as reched one of the pre-defined positions. Is it so wrong to report either (Scanning,Focus,Failed) even for Manual mode ? 2) AutoMode (the good) This seems the easy one, as (Scanning,Focus,Failed) are already there 3) Continous (the ugly) Continuous mode can be paused. That's a shame :) Would that be that bad keeping a Paused or Pausing state which can only be returned for Continuous mode. All in all, can we reduce the number of states to - Scanning The AF algorithm is scanning and has not yet reached a stable result. When running in Auto mode an AF scan is started by using the AfTrigger control. When in Manual mode and a new lens position is requested either by sending a LensPosition or AfLensPosition control, the AF state will report "Scanning" until the lens doesn't reach the desired position. Similarly, when running in continuous mode and the AF algorithm decides to start a scan spontaneously the AfState will report Scanning until a new stable state is reached. - Focus The AF algorithm has reached a stable state and the image is now focused. When running in Manual mode the lens has reached the position requested using either LensPosition and AfLensPosition. - Failed The AF algorithm has completed a scan and without finding a good focus position. When running in manual mode, the camera has failed moving the lens to the desired position. - Paused (copying your below text, please see comments below) The AF algorithm is in continuous mode (AfModeContinuous) and was scanning when AfPauseImmediate was sent. The scan will not complete and the algorithm will remain in this state. - Pausing The AF algorithm is in continuous mode (AfModeContinuous) and was scanning when AfPauseDeferred was sent. The scan will complete at which point the algorithm moves to the AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused state. > + > + In any of the AfStateManual or AfStateContinuous states, the AfMode > + may be set to AfModeAuto and the algorithm will move to the > + AfStateAuto state. If AfTriggerStart is sent at the same time > + then the algorithm will move to AfStateAutoScanning state. > + > + In any of the AfStateAuto or AfStateManual states, the AfMode may > + be set to AfModeContinuous and the algorithm will move to > + AfStateContinuousScanning. > + > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > + be set to AfModeManual and the algorithm will move to > + AfStateManual. The lens will not be moved and will be left where > + it was at that moment. > + > + enum: > + - name: AfStateManual > + value: 0 > + description: | > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > + and AfLensPosition controls can be used directly to move the lens. > + - name: AfStateAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and has either just > + been moved into that state, or a scan that was in progress has been > + cancelled. > + - name: AfStateAutoScanning > + value: 2 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > + started using the AfTrigger control. The scan can be cancelled by > + sending AfTriggerCancel at which point the algorithm will either > + move back to AfStateAuto or, if the scan actually completes > + before the cancel request is processed, to one of > + AfStateAutoFocused or AfStateAutoFailed. > + - name: AfStateAutoFocused > + value: 3 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm believes the image is > + now in focus. > + - name: AfStateAutoFailed > + value: 4 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm did not find a good > + focus position. > + - name: AfStateContinuousScanning > + value: 5 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + is currently scanning for a good focus position. This occurs when > + the mode is first set to AfModeContinuous, or may happen > + spontaneously when the algorithm believes a re-scan is needed. > + - name: AfStateContinuousScanningPausing > + value: 6 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseDeferred was sent. The scan will complete > + at which point the algorithm moves to the > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > + state. > + - name: AfStateContinuousScanningPaused > + value: 7 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseImmediate was sent. The scan will not Why can't the algorithm be paused when it has reached a stable state but only when scanning ? Won't a pause 'freeze' the algorithm in its current state no matter what it is ? I understand with "AfPauseDeferred" that it has to wait for AF to be locked before stopping it, but with "Immediate" doesn't "pause" apply to all states ? Thanks j > + complete and the algorithm will remain in this state. The scan > + may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFocused > + value: 8 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed with the algorithm believing it has found a > + good focus position. > + - name: AfStateContinuousFocusedPaused > + value: 9 > + description: | > + The AF algorithm was in the AfStateContinuousFocused state and > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and the > + scan has completed successfully. The algorithm will now remain > + in this state, and may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFailed > + value: 10 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed without finding a good focus position. > + - name: AfStateContinuousFailedPaused > + value: 11 > + description: | > + The AF algorithm was in the AfStateContinuousFailed state and > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and the > + scan has completed unsuccessfully. The algorithm will now remain > + in this state, and may be resumed by sending AfPauseResume. > + > ... > -- > 2.30.2 >
Hi Jacopo Thanks for taking the time to read all this! On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <jacopo@jmondi.org> wrote: > > Hi David, > sorry for being late :) > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote: > > This patch describes a series of controls that allow applications to > > drive AF algorithms: > > > > AfMode - manual, auto or continuous > > AfLensPosition - set lens position to macro/hyperfocal/infinity > > AfRange - full, macro or normal > > AfSpeed - fast or slow > > AfWindows - AF window locations > > AfTrigger - start (trigger an AF scan) or cancel > > AfPause - pause continuous AF > > LensPosition - position of lens from lens driver > > AfState - reports the mode, whether scanning/success/failure > > --- > > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > > 1 file changed, 313 insertions(+), 60 deletions(-) > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > > index 9d4638ae..23607368 100644 > > --- a/src/libcamera/control_ids.yaml > > +++ b/src/libcamera/control_ids.yaml > > @@ -406,27 +406,6 @@ controls: > > The camera will cancel any active or completed metering sequence. > > The AE algorithm is reset to its initial state. > > > > - - AfTrigger: > > - type: int32_t > > - draft: true > > - description: | > > - Control for AF trigger. Currently identical to > > - ANDROID_CONTROL_AF_TRIGGER. > > - > > - Whether the camera device will trigger autofocus for this request. > > - enum: > > - - name: AfTriggerIdle > > - value: 0 > > - description: The trigger is idle. > > - - name: AfTriggerStart > > - value: 1 > > - description: The AF routine is started by the camera. > > - - name: AfTriggerCancel > > - value: 2 > > - description: | > > - The camera will cancel any active trigger and the AF routine is > > - reset to its initial state. > > - > > - NoiseReductionMode: > > type: int32_t > > draft: true > > @@ -507,45 +486,6 @@ controls: > > The AE algorithm has started a pre-capture metering session. > > \sa AePrecaptureTrigger > > > > - - AfState: > > - type: int32_t > > - draft: true > > - description: | > > - Control to report the current AF algorithm state. Currently identical to > > - ANDROID_CONTROL_AF_STATE. > > - > > - Current state of the AF algorithm. > > - enum: > > - - name: AfStateInactive > > - value: 0 > > - description: The AF algorithm is inactive. > > - - name: AfStatePassiveScan > > - value: 1 > > - description: | > > - AF is performing a passive scan of the scene in continuous > > - auto-focus mode. > > - - name: AfStatePassiveFocused > > - value: 2 > > - description: | > > - AF believes the scene is in focus, but might restart scanning. > > - - name: AfStateActiveScan > > - value: 3 > > - description: | > > - AF is performing a scan triggered by an AF trigger request. > > - \sa AfTrigger > > - - name: AfStateFocusedLock > > - value: 4 > > - description: | > > - AF believes has focused correctly and has locked focus. > > - - name: AfStateNotFocusedLock > > - value: 5 > > - description: | > > - AF has not been able to focus and has locked. > > - - name: AfStatePassiveUnfocused > > - value: 6 > > - description: | > > - AF has completed a passive scan without finding focus. > > - > > - AwbState: > > type: int32_t > > draft: true > > @@ -690,4 +630,317 @@ controls: > > value. All of the custom test patterns will be static (that is the > > raw image must not vary from frame to frame). > > > > + - AfMode: > > + type: int32_t > > + draft: true > > Was having all these controls as draft intentional ? The previous > definitions where set to drafts as they mapped to the android > definition, but now that we're defining our own ones... Not really... I only made them "draft" because it seemed like everything was "draft" at first! But happy to change that. Are you saying that these definitely should *not* be draft, then? > > > + description: | > > + Control to set the mode of the AF (autofocus) algorithm. Applications > > + are allowed to set a new mode, and to send additional controls for > > + that new mode, in the same request. > > + > > + An implementation may choose not to implement all the modes. > > + > > + enum: > > + - name: AfModeManual > > + value: 0 > > + description: | > > + The AF algorithm is in manual mode. In this mode it will never > > + perform any action nor move the lens of its own accord, but an > > + application can set controls to move the lens. > > + > > + In this mode the AfState will always report AfStateManual. > > + - name: AfModeAuto > > + value: 1 > > + description: | > > + The AF algorithm is in auto mode. This means that the algorithm > > + will never move the lens or change state unless the AfTrigger > > + control is used. The AfTrigger control can be used to initiate a > > + focus scan, the results of which will also be reported by AfState. > > + > > + If the autofocus algorithm is moved from AfModeAuto to another > > + mode while a scan is in progress, the scan is cancelled > > + immediately, without waiting for the scan to finish. > > + > > + When first entering this mode the AfState will report > > + AfStateAuto. When a trigger control is sent, AfState will > > + report AfStateAutoScanning for a period before spontaneously > > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > > + the outcome of the scan. It will remain in this state until > > + another scan is initiated by the AfTrigger control. If a scan is > > + cancelled (without changing to another mode), AfState will return > > + to AfStateAuto. > > + - name: AfModeContinuous > > + value: 2 > > + description: | > > + The AF algorithm is in continuous mode. This means that the lens > > + can re-start a scan spontaneously at any moment, without any user > > + intervention. The AfState still reports whether the algorithm is > > + currently scanning or not, though the application has no ability > > + to initiate or cancel scans (though it can "pause" them), nor to > > + move the lens for itself. > > + > > + When set to AfModeContinuous, the system will immediately initiate > > + a scan so AfState will report AfStateContinuousScanning, and will > > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > > + depending on the scan result. > > + > > + - AfLensPosition: > > + type: int32_t > > + draft: true > > + description: | > > + Control to set the position of the lens to one of the following > > + predefined locations. This control only has any effect when the AfMode > > + is set to AfModeManual. > > + > > + This control is distinct from the LensPosition control, which sets the > > + lens position using the lens driver's units. > > + enum: > > + - name: AfLensPositionMacro > > + value: 0 > > + description: The closest focal position of the lens. > > + - name: AfLensPositionHyperfocal > > + value: 1 > > + description: Hyperfocal position. > > + - name: AfLensPositionInfinity > > + value: 2 > > + description: The furthest focal position (usually infinity). > > Interesting, bikeshedding on the name apart, is this some 'simplified' > LensState ? Do you think there is a use case for this ? Why would > anyone set the AF to auto mode then use some coarse grained control > like this one ? Wouldn't it make more sense as an 'hint' to the AF > algorithm to restrict the search space ? In the light of your comments below about the LensPosition not using driver units, then maybe this could be deleted, and we could select some kind of canonical range for the lens? The most basic use case would be following on from the capture of an image after an AF scan: - capture image - AfMode to manual - Lens to hyperfocal - AfMode to auto - now wait for the shutter button to be pressed before you start another AF scan. If your UI has a "macro" capture mode of some kind, maybe it's "Lens to macro" instead of "Lens to hyperfocal". But as noted, if LensPosition allows us to select hyperfocal/macro directly, then we can do without this one. > > (Btw I have a use case for supplying a range of lens positions, which > has been previously estimated according to some pre-defined use cases, > to the AF algorithm in order restrict the AF search range to increase > the speed of a scan. Do you think it's a valid use case and it's worth a > control ?) I'd imagined doing this kind of thing with the AfSpeed and AfRange controls and by having the AF algorithm tuned for the module. I'm not convinced about passing in lens positions from outside the camera system. Where would they come from? Shouldn't the camera system know what positions to use? In the Raspberry Pi world we would of course store this kind of information in the tuning file. I've noticed some discussion elsewhere about supplying configuration to the IPAs - it seems like this might be a case in point? I think that for real camera systems, in the end, there is no escape from "configuration" (aka. "tuning"). > > > + > > + - AfRange: > > + type: int32_t > > + draft: true > > + description: | > > + Control to set the range of focus distances that is scanned. An > > + implementation may choose not to implement all the options here. > > Only valid in Auto and Continuous mode, right ? Generally I've been expecting that you could change settings like this in Manual mode too, though it obviously can't have any effect until you go back to Auto/Continuous. Does that sound reasonable? > > (This makes me think that we could easily model Continuous by adding a > TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of > AfPause... well, it's quite a redesign so I'll stop here unless > someone thinks it's a really good idea :) Actually I'm quite strongly against anything like this. I think Auto and Continuous should be quite separate. The reason is that Auto is (relatively) easy to implement, you do a scan over some range and choose the best spot. But Continuous AF is an absolute nightmare to do well. You have all these problems: - The algo doesn't realise it needs to rescan and so seems to be "stuck". (Watch people waving their hands in front of the lens to try and unstick it.) So you add ever more tests and arbitrary thresholds on different statistics to tell if a rescan is needed, and then the whole thing becomes untunable. - The algo rescans too often. A car in the distance goes by and the whole image see-saws in and out while a rescan happens, ending back at the same place. Yet more tests, thresholds... - You can never make all your tests and thresholds work at the same time but in different conditions. So you make them variable depending on things like exposure and gain. Now you are truly doomed. - The algo is generally not robust to camera shake, which frequently causes it to end up "failing" with a completely out-of-focus image. And then you can't get it to rescan... cue more hand-waving in front of the lens. You implement more special tests and timeouts... At this point you will wish for the earth to swallow you up. - When the algo rescans it starts off going the wrong way causing extra see-sawing. - You try to solve the last problem by doing really small initial steps. Only then it ends up statistically unstable. From the above you may detect that we have been there and have several T-shirts. Speaking for myself, I have no plans to implement Continuous AF without PDAF sensors, as PDAF is quite a big help. > > > + enum: > > + - name: AfRangeNormal > > + value: 0 > > + description: | > > + A wide range of focus distances is scanned, all the way from > > + infinity down to close distances, though depending on the > > + implementation, possibly not including the very closest macro > > + positions. > > + - name: AfRangeMacro > > + value: 1 > > + description: Only close distances are scanned. > > + - name: AfRangeFull > > + value: 2 > > + description: | > > + The full range of focus distances is scanned just as with > > + AfRangeNormal but this time including the very closest macro > > + positions. > > + > > + - AfSpeed: > > + type: int32_t > > + draft: true > > + description: | > > + Control that determines whether the AF algorithm is to move the lens > > + as quickly as possible or more steadily. For example, during video > > + recording it may be desirable not to move the lens too abruptly, but > > + when in a preview mode (waiting for a still capture) it may be > > + helpful to move the lens as quickly as is reasonably possible. > > + enum: > > + - name: AfSpeedNormal > > + value: 0 > > + description: Move the lens at its usual speed. > > + - name: AfSpeedFast > > + value: 1 > > + description: Move the lens more quickly. > > + > > + - AfWindows: > > + type: Rectangle > > + draft: true > > + description: | > > + Sets the focus windows used by the AF algorithm. The units used express > > + a proportion of the ScalerCrop control (or if unavailable, of the entire > > + image), as u0.16 format numbers. > > + > > + In order to be activated, a rectangle must be programmed with non-zero > > + width and height. If no rectangles are programmed in this way, then the > > + system will choose its own single default window in the centre of the > > + image. > > + > > + The details of how the windows are used are platform dependent. We note > > + that when there is more than one AF window, a typical implementation > > + might find the optimal focus position for each one and finally select > > + the window closest to the camera. > > + > > + size: [platform dependent] > > I think this should just be > > size: [n] OK! > > + > > + - AfTrigger: > > + type: int32_t > > + draft: true > > + description: | > > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > > + and can also be used to terminate a scan early. > > + > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > > + > > + enum: > > + - name: AfTriggerStart > > + value: 0 > > + description: Start an AF scan. Ignored if a scan is in progress. > > + - name: AfTriggerCancel > > + value: 1 > > + description: Cancel an AF scan. This does not cause the lens to move > > + anywhere else. Ignored if no scan is in progress. > > + > > + - AfPause: > > + type: int32_t > > + draft: true > > + description: | > > + This control has no effect except when in continuous autofocus mode > > + (AfModeContinuous). It can be used to pause any lens movements while > > + (for example) images are captured. The algorithm remains inactive > > + until it is instructed to resume. > > + > > + enum: > > + - name: AfPauseImmediate > > + value: 0 > > + description: | > > + Pause the continuous autofocus algorithm immediately, whether or > > + not any kind of scan is underway. If the AfState was previously > > + reporting AfStateContinuousScanning it will now change to > > + AfStateContinuousScanningPaused, and similarly for > > + AfStateContinuousFocused and AfStateContinuousFailed. > > + - name AfPauseDeferred > > + value: 1 > > + description: | > > + Pause the continuous autofocus algorithm as soon as it is no longer > > + scanning. If the AfState is currently reporting > > + AfStateContinuousFocused is will change to > > + AfStateContinuousFocusedPaused, and similarly in the case of > > + AfStateContinuousFailed. > > + > > + If AfState reports AfStateContinuousScanning it will change to > > + AfStateContinuousScanningPausing, and then move to one of > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > + when the scan completes. > > + - name: AfPauseResume > > + value: 2 > > + description: | > > + Resume continous autofocus operation. The algorithm starts again > > + from exactly where it left off, and the AfState will drop the > > + 'Paused' or 'Pausing' part of the state. > > + > > + - LensPosition: > > + type: int32_t > > + draft: true > > + description: | > > + Acts as a control to instruct the lens to move to a particular position > > + and also reports back the position of the lens for each frame. > > + > > + The units are determined by the lens driver. > > Ugh, we cannot do this in the control definition. Driver specificities > cannot surface to the controls definition, which should be platform > independent. > > I think we'll have to establish a numerical range (go ahead if you > have any idea) and translate to the actual driver values in the > CameraLens class ? I wasn't aware of this, but maybe it's a good thing. The question is what kind of range to use, and how do you know what limits apply for any given camera module? How about a dioptre-like scale, normalised for the hyperfocal distance H. It would go like this: The value n would mean being focused at distance H/n (n does not have to be an integer). So: The value 1 would mean hyperfocal position (focal distance H/1). The value 0 would mean infinity (focal distance H/0). Larger values would get progressively closer. Setting a control value outside the allowed range will clamp it to that end of the range. In the Raspberry Pi world it would be sufficient for the tuning file to know the allowable range of values. More generally, I could imagine an application might want to say "focus at precisely 1m" in which case it would have to know the value of H from a property or something, though providing such information would be optional (some lenses simply aren't precisely calibrated). Perhaps an application might want to get the allowable range of (maybe approximate) focal distances too? > > > + > > + The LensPosition control is ignored unless the AfMode is set to > > + AfModeManual. > > + > > + Note that this control is distinct from AfLensPosition, which allows > > + the lens to be moved to its macro/hyperfocal/infinity position, rather > > + than using lens driver units. > > + > > + - AfState: > > + type: int32_t > > + draft: true > > + description: | > > + Reports the current state of the AF algorithm. The possible state changes > > + are described below, although we add that the following general state > > + changes are also allowed. > > wow, there is a lot of states here :) Did you follow the link to the state transition diagram? I would hope that should go some way to explaining it! Here's the link again for convenience: https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing > > Let me copy them here > > - name: AfStateManual > - name: AfStateAuto > - name: AfStateAutoScanning > - name: AfStateAutoFocused > - name: AfStateAutoFailed > - name: AfStateContinuousScanning > - name: AfStateContinuousScanningPausing > - name: AfStateContinuousScanningPaused > - name: AfStateContinuousFocused > - name: AfStateContinuousFocusedPaused > - name: AfStateContinuousFailed > - name: AfStateContinuousFailedPaused I "multiplied out" all the different parts of the state to make a single long list. Actually I think that makes it more Android-like. But we could reduce this number if, for example, we're happy to report the state of AfMode separately on every frame. I don't mind - do we have a general preference for this kind of thing? > > (please note that AfStateAuto seems mis-aligned, unless I messed up > something while cutting&paste. Please check the compiled doc to see > how it looks like...) I will check. > > So we have three macro groups > > - Manual > > - Auto > - Scanning > - Focus > - Failed > > - Continuous > > - Scanning > - Pausing > - Paused > - Focused > - Paused > - Failed > - Paused > > The first group is AfMode (Manual, Auto, Continuous) and can be > deduced by inspecting the AfMode metadata. > > The second group is the actual state (Scannng, Focused, Failed) > > The third group is specific to continuous mode as it accepts a Pause > trigger. > > I understand why you have this, as expressing "The AfState is set to > $something when in AfModeManual" etc etc might get cumbersome, but I > think it might be worth a try > > 1) Manual mode (the bad) > As we all know there is a delay between when a lens is instructed > to move and when it reaches its final state. > > In Manual mode the lens is moved with LensPosition and > AfLensPosition. > > To know the current state: > - With LensPosition metadata can be inspected until it doesn't reach the > desired value. > - With AfLensPosition it is not clear to me how to know when the lens > as reched one of the pre-defined positions. It sounds like we're deleting AfLensPosition. > > Is it so wrong to report either (Scanning,Focus,Failed) even for > Manual mode ? I don't think it's wrong. If we have a single long list of states then it just makes for more states. If we don't then it's harmless, though I'm not aware of a use for it. > > 2) AutoMode (the good) > This seems the easy one, as (Scanning,Focus,Failed) are already > there > > 3) Continous (the ugly) > Continuous mode can be paused. That's a shame :) > Would that be that bad keeping a Paused or Pausing state which can > only be returned for Continuous mode. The whole paused/pausing thing rather complicates things, but it does seem to me that it's necessary. If you were in Continuous mode, you would rather the lens didn't set off again just as you want to capture a sequence of pictures. > > All in all, can we reduce the number of states to > > - Scanning > > The AF algorithm is scanning and has not yet reached a > stable result. When running in Auto mode an AF scan is > started by using the AfTrigger control. When in Manual mode > and a new lens position is requested either by sending a > LensPosition or AfLensPosition control, the AF state will > report "Scanning" until the lens doesn't reach the desired > position. Similarly, when running in continuous mode and the > AF algorithm decides to start a scan spontaneously the > AfState will report Scanning until a new stable state is > reached. > > - Focus > > The AF algorithm has reached a stable state and the image is > now focused. When running in Manual mode the lens has > reached the position requested using either LensPosition and > AfLensPosition. > > - Failed > > The AF algorithm has completed a scan and without finding a > good focus position. When running in manual mode, the camera > has failed moving the lens to the desired position. > > - Paused (copying your below text, please see comments below) > > The AF algorithm is in continuous mode (AfModeContinuous) and > was scanning when AfPauseImmediate was sent. The scan will not > complete and the algorithm will remain in this state. > > - Pausing > > The AF algorithm is in continuous mode (AfModeContinuous) and > was scanning when AfPauseDeferred was sent. The scan will complete > at which point the algorithm moves to the > AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > state. I think we could get by if we report the following for every frame: AfMode - Manual, Auto or Continuous AfState - Scanning, Focused or Failed AfPause - Resume (which means it's running), Immediate or Deferred (these states only apply during Continuous; the Deferred value can only occur when also Scanning). Or we could wrap AfState and AfPause together: AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused, FocusedPaused, FailedPaused. That makes some of the interaction between pausing and scanning clearer, but then it's more states again. I don't know... > > > > + > > + In any of the AfStateManual or AfStateContinuous states, the AfMode > > + may be set to AfModeAuto and the algorithm will move to the > > + AfStateAuto state. If AfTriggerStart is sent at the same time > > + then the algorithm will move to AfStateAutoScanning state. > > + > > + In any of the AfStateAuto or AfStateManual states, the AfMode may > > + be set to AfModeContinuous and the algorithm will move to > > + AfStateContinuousScanning. > > + > > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > > + be set to AfModeManual and the algorithm will move to > > + AfStateManual. The lens will not be moved and will be left where > > + it was at that moment. > > + > > + enum: > > + - name: AfStateManual > > + value: 0 > > + description: | > > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > > + and AfLensPosition controls can be used directly to move the lens. > > + - name: AfStateAuto > > + value: 1 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and has either just > > + been moved into that state, or a scan that was in progress has been > > + cancelled. > > + - name: AfStateAutoScanning > > + value: 2 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > > + started using the AfTrigger control. The scan can be cancelled by > > + sending AfTriggerCancel at which point the algorithm will either > > + move back to AfStateAuto or, if the scan actually completes > > + before the cancel request is processed, to one of > > + AfStateAutoFocused or AfStateAutoFailed. > > + - name: AfStateAutoFocused > > + value: 3 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > + completed with the result that the algorithm believes the image is > > + now in focus. > > + - name: AfStateAutoFailed > > + value: 4 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > + completed with the result that the algorithm did not find a good > > + focus position. > > + - name: AfStateContinuousScanning > > + value: 5 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + is currently scanning for a good focus position. This occurs when > > + the mode is first set to AfModeContinuous, or may happen > > + spontaneously when the algorithm believes a re-scan is needed. > > + - name: AfStateContinuousScanningPausing > > + value: 6 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + was scanning when AfPauseDeferred was sent. The scan will complete > > + at which point the algorithm moves to the > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > + state. > > + - name: AfStateContinuousScanningPaused > > + value: 7 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + was scanning when AfPauseImmediate was sent. The scan will not > > Why can't the algorithm be paused when it has reached a stable state > but only when scanning ? Won't a pause 'freeze' the algorithm in its > current state no matter what it is ? I understand with > "AfPauseDeferred" that it has to wait for AF to be locked before > stopping it, but with "Immediate" doesn't "pause" apply to all states ? Wasn't sure I understood this. It can be paused once it's reached Focused or Failed, and then both PauseImmediate and PauseDeferred will cause it to report FocusedPaused or FailedPaused. Was that the question? Anyway, thanks very much! David > > Thanks > j > > > + complete and the algorithm will remain in this state. The scan > > + may be resumed by sending AfPauseResume. > > + - name: AfStateContinuousFocused > > + value: 8 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + a scan has completed with the algorithm believing it has found a > > + good focus position. > > + - name: AfStateContinuousFocusedPaused > > + value: 9 > > + description: | > > + The AF algorithm was in the AfStateContinuousFocused state and > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > + or it was in the AfStateContinuousScanningPausing state and the > > + scan has completed successfully. The algorithm will now remain > > + in this state, and may be resumed by sending AfPauseResume. > > + - name: AfStateContinuousFailed > > + value: 10 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + a scan has completed without finding a good focus position. > > + - name: AfStateContinuousFailedPaused > > + value: 11 > > + description: | > > + The AF algorithm was in the AfStateContinuousFailed state and > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > + or it was in the AfStateContinuousScanningPausing state and the > > + scan has completed unsuccessfully. The algorithm will now remain > > + in this state, and may be resumed by sending AfPauseResume. > > + > > ... > > -- > > 2.30.2 > >
Hi David, + Han-lin for an Android question on metadata On Wed, Apr 06, 2022 at 02:05:20PM +0100, David Plowman wrote: > Hi Jacopo > > Thanks for taking the time to read all this! > > On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <jacopo@jmondi.org> wrote: > > > > Hi David, > > sorry for being late :) > > > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote: > > > This patch describes a series of controls that allow applications to > > > drive AF algorithms: > > > > > > AfMode - manual, auto or continuous > > > AfLensPosition - set lens position to macro/hyperfocal/infinity > > > AfRange - full, macro or normal > > > AfSpeed - fast or slow > > > AfWindows - AF window locations > > > AfTrigger - start (trigger an AF scan) or cancel > > > AfPause - pause continuous AF > > > LensPosition - position of lens from lens driver > > > AfState - reports the mode, whether scanning/success/failure > > > --- > > > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > > > 1 file changed, 313 insertions(+), 60 deletions(-) > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > > > index 9d4638ae..23607368 100644 > > > --- a/src/libcamera/control_ids.yaml > > > +++ b/src/libcamera/control_ids.yaml > > > @@ -406,27 +406,6 @@ controls: > > > The camera will cancel any active or completed metering sequence. > > > The AE algorithm is reset to its initial state. > > > > > > - - AfTrigger: > > > - type: int32_t > > > - draft: true > > > - description: | > > > - Control for AF trigger. Currently identical to > > > - ANDROID_CONTROL_AF_TRIGGER. > > > - > > > - Whether the camera device will trigger autofocus for this request. > > > - enum: > > > - - name: AfTriggerIdle > > > - value: 0 > > > - description: The trigger is idle. > > > - - name: AfTriggerStart > > > - value: 1 > > > - description: The AF routine is started by the camera. > > > - - name: AfTriggerCancel > > > - value: 2 > > > - description: | > > > - The camera will cancel any active trigger and the AF routine is > > > - reset to its initial state. > > > - > > > - NoiseReductionMode: > > > type: int32_t > > > draft: true > > > @@ -507,45 +486,6 @@ controls: > > > The AE algorithm has started a pre-capture metering session. > > > \sa AePrecaptureTrigger > > > > > > - - AfState: > > > - type: int32_t > > > - draft: true > > > - description: | > > > - Control to report the current AF algorithm state. Currently identical to > > > - ANDROID_CONTROL_AF_STATE. > > > - > > > - Current state of the AF algorithm. > > > - enum: > > > - - name: AfStateInactive > > > - value: 0 > > > - description: The AF algorithm is inactive. > > > - - name: AfStatePassiveScan > > > - value: 1 > > > - description: | > > > - AF is performing a passive scan of the scene in continuous > > > - auto-focus mode. > > > - - name: AfStatePassiveFocused > > > - value: 2 > > > - description: | > > > - AF believes the scene is in focus, but might restart scanning. > > > - - name: AfStateActiveScan > > > - value: 3 > > > - description: | > > > - AF is performing a scan triggered by an AF trigger request. > > > - \sa AfTrigger > > > - - name: AfStateFocusedLock > > > - value: 4 > > > - description: | > > > - AF believes has focused correctly and has locked focus. > > > - - name: AfStateNotFocusedLock > > > - value: 5 > > > - description: | > > > - AF has not been able to focus and has locked. > > > - - name: AfStatePassiveUnfocused > > > - value: 6 > > > - description: | > > > - AF has completed a passive scan without finding focus. > > > - > > > - AwbState: > > > type: int32_t > > > draft: true > > > @@ -690,4 +630,317 @@ controls: > > > value. All of the custom test patterns will be static (that is the > > > raw image must not vary from frame to frame). > > > > > > + - AfMode: > > > + type: int32_t > > > + draft: true > > > > Was having all these controls as draft intentional ? The previous > > definitions where set to drafts as they mapped to the android > > definition, but now that we're defining our own ones... > > Not really... I only made them "draft" because it seemed like > everything was "draft" at first! But happy to change that. Are you > saying that these definitely should *not* be draft, then? > Yeah, "draft" was used to import controls from Android as they are for sake of CTS compliance. Let's drop draft from the ones we're defining > > > > > + description: | > > > + Control to set the mode of the AF (autofocus) algorithm. Applications > > > + are allowed to set a new mode, and to send additional controls for > > > + that new mode, in the same request. > > > + > > > + An implementation may choose not to implement all the modes. > > > + > > > + enum: > > > + - name: AfModeManual > > > + value: 0 > > > + description: | > > > + The AF algorithm is in manual mode. In this mode it will never > > > + perform any action nor move the lens of its own accord, but an > > > + application can set controls to move the lens. > > > + > > > + In this mode the AfState will always report AfStateManual. > > > + - name: AfModeAuto > > > + value: 1 > > > + description: | > > > + The AF algorithm is in auto mode. This means that the algorithm > > > + will never move the lens or change state unless the AfTrigger > > > + control is used. The AfTrigger control can be used to initiate a > > > + focus scan, the results of which will also be reported by AfState. > > > + > > > + If the autofocus algorithm is moved from AfModeAuto to another > > > + mode while a scan is in progress, the scan is cancelled > > > + immediately, without waiting for the scan to finish. > > > + > > > + When first entering this mode the AfState will report > > > + AfStateAuto. When a trigger control is sent, AfState will > > > + report AfStateAutoScanning for a period before spontaneously > > > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > > > + the outcome of the scan. It will remain in this state until > > > + another scan is initiated by the AfTrigger control. If a scan is > > > + cancelled (without changing to another mode), AfState will return > > > + to AfStateAuto. > > > + - name: AfModeContinuous > > > + value: 2 > > > + description: | > > > + The AF algorithm is in continuous mode. This means that the lens > > > + can re-start a scan spontaneously at any moment, without any user > > > + intervention. The AfState still reports whether the algorithm is > > > + currently scanning or not, though the application has no ability > > > + to initiate or cancel scans (though it can "pause" them), nor to > > > + move the lens for itself. > > > + > > > + When set to AfModeContinuous, the system will immediately initiate > > > + a scan so AfState will report AfStateContinuousScanning, and will > > > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > > > + depending on the scan result. > > > + > > > + - AfLensPosition: > > > + type: int32_t > > > + draft: true > > > + description: | > > > + Control to set the position of the lens to one of the following > > > + predefined locations. This control only has any effect when the AfMode > > > + is set to AfModeManual. > > > + > > > + This control is distinct from the LensPosition control, which sets the > > > + lens position using the lens driver's units. > > > + enum: > > > + - name: AfLensPositionMacro > > > + value: 0 > > > + description: The closest focal position of the lens. > > > + - name: AfLensPositionHyperfocal > > > + value: 1 > > > + description: Hyperfocal position. > > > + - name: AfLensPositionInfinity > > > + value: 2 > > > + description: The furthest focal position (usually infinity). > > > > Interesting, bikeshedding on the name apart, is this some 'simplified' > > LensState ? Do you think there is a use case for this ? Why would > > anyone set the AF to auto mode then use some coarse grained control > > like this one ? Wouldn't it make more sense as an 'hint' to the AF > > algorithm to restrict the search space ? > > In the light of your comments below about the LensPosition not using > driver units, then maybe this could be deleted, and we could select > some kind of canonical range for the lens? > Ah wait. What I was suggesting ("An "hint" to the AF algorithm to restrict the search space") is basically AfRange. > The most basic use case would be following on from the capture of an > image after an AF scan: > > - capture image > - AfMode to manual > - Lens to hyperfocal > - AfMode to auto > - now wait for the shutter button to be pressed before you start > another AF scan. Naive question: Wouldn't setting the lens to hyperfocal render the preview unusable ? > > If your UI has a "macro" capture mode of some kind, maybe it's "Lens > to macro" instead of "Lens to hyperfocal". Why is this different than starting an AF scan (without going through Manual) with an AfRange that suggests in which range to scan ? Is this an optimization to forcefully move the lens closer to where it will likely end being moved to ? > > But as noted, if LensPosition allows us to select hyperfocal/macro > directly, then we can do without this one. > Providing my above understanding is correct, it would be a bit less intuitive for applications to use a numerical value instead of some easier "Macro"/"Hyperfocal". Of course, wrappers like your one can simplify this for their consumers. > > > > (Btw I have a use case for supplying a range of lens positions, which > > has been previously estimated according to some pre-defined use cases, > > to the AF algorithm in order restrict the AF search range to increase > > the speed of a scan. Do you think it's a valid use case and it's worth a > > control ?) > > I'd imagined doing this kind of thing with the AfSpeed and AfRange > controls and by having the AF algorithm tuned for the module. I'm not > convinced about passing in lens positions from outside the camera > system. Where would they come from? Shouldn't the camera system know > what positions to use? > A possible use case, for controlled environments with low latency requirements, is to run tests beforehand and for each 'scene' define what is the ideal lens movement range. When the application starts an AF scan it would tell the AF algorithm about the 'preferred' range. I understand this works backwards, as the lens position should be the result of the AF algorithm run not one of its inputs :) > In the Raspberry Pi world we would of course store this kind of > information in the tuning file. I've noticed some discussion elsewhere > about supplying configuration to the IPAs - it seems like this might > be a case in point? I think that for real camera systems, in the end, > there is no escape from "configuration" (aka. "tuning"). > > > > > > + > > > + - AfRange: > > > + type: int32_t > > > + draft: true > > > + description: | > > > + Control to set the range of focus distances that is scanned. An > > > + implementation may choose not to implement all the options here. > > > > Only valid in Auto and Continuous mode, right ? > > Generally I've been expecting that you could change settings like this > in Manual mode too, though it obviously can't have any effect until > you go back to Auto/Continuous. Does that sound reasonable? I would rather assume thaat if a control is not valid in the current context is simply ignored and not cached to be re-applied when the context change ? > > > > > (This makes me think that we could easily model Continuous by adding a > > TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of > > AfPause... well, it's quite a redesign so I'll stop here unless > > someone thinks it's a really good idea :) > > Actually I'm quite strongly against anything like this. I think Auto > and Continuous should be quite separate. The reason is that Auto is > (relatively) easy to implement, you do a scan over some range and > choose the best spot. But Continuous AF is an absolute nightmare to do > well. You have all these problems: > > - The algo doesn't realise it needs to rescan and so seems to be > "stuck". (Watch people waving their hands in front of the lens to try > and unstick it.) So you add ever more tests and arbitrary thresholds > on different statistics to tell if a rescan is needed, and then the > whole thing becomes untunable. > - The algo rescans too often. A car in the distance goes by and the > whole image see-saws in and out while a rescan happens, ending back at > the same place. Yet more tests, thresholds... > - You can never make all your tests and thresholds work at the same > time but in different conditions. So you make them variable depending > on things like exposure and gain. Now you are truly doomed. > - The algo is generally not robust to camera shake, which frequently > causes it to end up "failing" with a completely out-of-focus image. > And then you can't get it to rescan... cue more hand-waving in front > of the lens. You implement more special tests and timeouts... At this > point you will wish for the earth to swallow you up. > - When the algo rescans it starts off going the wrong way causing > extra see-sawing. > - You try to solve the last problem by doing really small initial > steps. Only then it ends up statistically unstable. > > From the above you may detect that we have been there and have several > T-shirts. Speaking for myself, I have no plans to implement Continuous > AF without PDAF sensors, as PDAF is quite a big help. > Yes, it feels like you know what you're talking about :) I understand you would like be cautious with CAF, but if one doesn't want to implement CAF can't she simply not report TriggerContinuous ? Anyway, that was just an idea thrown in the mixer, don't worry about it. > > > > > + enum: > > > + - name: AfRangeNormal > > > + value: 0 > > > + description: | > > > + A wide range of focus distances is scanned, all the way from > > > + infinity down to close distances, though depending on the > > > + implementation, possibly not including the very closest macro > > > + positions. > > > + - name: AfRangeMacro > > > + value: 1 > > > + description: Only close distances are scanned. > > > + - name: AfRangeFull > > > + value: 2 > > > + description: | > > > + The full range of focus distances is scanned just as with > > > + AfRangeNormal but this time including the very closest macro > > > + positions. > > > + > > > + - AfSpeed: > > > + type: int32_t > > > + draft: true > > > + description: | > > > + Control that determines whether the AF algorithm is to move the lens > > > + as quickly as possible or more steadily. For example, during video > > > + recording it may be desirable not to move the lens too abruptly, but > > > + when in a preview mode (waiting for a still capture) it may be > > > + helpful to move the lens as quickly as is reasonably possible. > > > + enum: > > > + - name: AfSpeedNormal > > > + value: 0 > > > + description: Move the lens at its usual speed. > > > + - name: AfSpeedFast > > > + value: 1 > > > + description: Move the lens more quickly. > > > + > > > + - AfWindows: > > > + type: Rectangle > > > + draft: true > > > + description: | > > > + Sets the focus windows used by the AF algorithm. The units used express > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire > > > + image), as u0.16 format numbers. > > > + > > > + In order to be activated, a rectangle must be programmed with non-zero > > > + width and height. If no rectangles are programmed in this way, then the > > > + system will choose its own single default window in the centre of the > > > + image. > > > + > > > + The details of how the windows are used are platform dependent. We note > > > + that when there is more than one AF window, a typical implementation > > > + might find the optimal focus position for each one and finally select > > > + the window closest to the camera. > > > + > > > + size: [platform dependent] > > > > I think this should just be > > > > size: [n] > > OK! > > > > + > > > + - AfTrigger: > > > + type: int32_t > > > + draft: true > > > + description: | > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > > > + and can also be used to terminate a scan early. > > > + > > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > > > + > > > + enum: > > > + - name: AfTriggerStart > > > + value: 0 > > > + description: Start an AF scan. Ignored if a scan is in progress. > > > + - name: AfTriggerCancel > > > + value: 1 > > > + description: Cancel an AF scan. This does not cause the lens to move > > > + anywhere else. Ignored if no scan is in progress. > > > + > > > + - AfPause: > > > + type: int32_t > > > + draft: true > > > + description: | > > > + This control has no effect except when in continuous autofocus mode > > > + (AfModeContinuous). It can be used to pause any lens movements while > > > + (for example) images are captured. The algorithm remains inactive > > > + until it is instructed to resume. > > > + > > > + enum: > > > + - name: AfPauseImmediate > > > + value: 0 > > > + description: | > > > + Pause the continuous autofocus algorithm immediately, whether or > > > + not any kind of scan is underway. If the AfState was previously > > > + reporting AfStateContinuousScanning it will now change to > > > + AfStateContinuousScanningPaused, and similarly for > > > + AfStateContinuousFocused and AfStateContinuousFailed. > > > + - name AfPauseDeferred > > > + value: 1 > > > + description: | > > > + Pause the continuous autofocus algorithm as soon as it is no longer > > > + scanning. If the AfState is currently reporting > > > + AfStateContinuousFocused is will change to > > > + AfStateContinuousFocusedPaused, and similarly in the case of > > > + AfStateContinuousFailed. > > > + > > > + If AfState reports AfStateContinuousScanning it will change to > > > + AfStateContinuousScanningPausing, and then move to one of > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > + when the scan completes. > > > + - name: AfPauseResume > > > + value: 2 > > > + description: | > > > + Resume continous autofocus operation. The algorithm starts again > > > + from exactly where it left off, and the AfState will drop the > > > + 'Paused' or 'Pausing' part of the state. > > > + > > > + - LensPosition: > > > + type: int32_t > > > + draft: true > > > + description: | > > > + Acts as a control to instruct the lens to move to a particular position > > > + and also reports back the position of the lens for each frame. > > > + > > > + The units are determined by the lens driver. > > > > Ugh, we cannot do this in the control definition. Driver specificities > > cannot surface to the controls definition, which should be platform > > independent. > > > > I think we'll have to establish a numerical range (go ahead if you > > have any idea) and translate to the actual driver values in the > > CameraLens class ? > > I wasn't aware of this, but maybe it's a good thing. The question is Well, isn't it the same with "Brightness", for example ? > what kind of range to use, and how do you know what limits apply for > any given camera module? How about a dioptre-like scale, normalised > for the hyperfocal distance H. It would go like this: > > The value n would mean being focused at distance H/n (n does not have > to be an integer). So: > > The value 1 would mean hyperfocal position (focal distance H/1). > The value 0 would mean infinity (focal distance H/0). > Larger values would get progressively closer. > > Setting a control value outside the allowed range will clamp it to > that end of the range. I sincerely don't know enough about this to tell if it's a good idea or not. Hence, it seems like a good idea to me :) > > In the Raspberry Pi world it would be sufficient for the tuning file > to know the allowable range of values. More generally, I could imagine > an application might want to say "focus at precisely 1m" in which case > it would have to know the value of H from a property or something, > though providing such information would be optional (some lenses > simply aren't precisely calibrated). Perhaps an application might want > to get the allowable range of (maybe approximate) focal distances too? > > > > > > + > > > + The LensPosition control is ignored unless the AfMode is set to > > > + AfModeManual. > > > + > > > + Note that this control is distinct from AfLensPosition, which allows > > > + the lens to be moved to its macro/hyperfocal/infinity position, rather > > > + than using lens driver units. > > > + > > > + - AfState: > > > + type: int32_t > > > + draft: true > > > + description: | > > > + Reports the current state of the AF algorithm. The possible state changes > > > + are described below, although we add that the following general state > > > + changes are also allowed. > > > > wow, there is a lot of states here :) > > Did you follow the link to the state transition diagram? I would hope > that should go some way to explaining it! Here's the link again for > convenience: > > https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing > Missed it. Thanks for providing it. > > > > Let me copy them here > > > > - name: AfStateManual > > - name: AfStateAuto > > - name: AfStateAutoScanning > > - name: AfStateAutoFocused > > - name: AfStateAutoFailed > > - name: AfStateContinuousScanning > > - name: AfStateContinuousScanningPausing > > - name: AfStateContinuousScanningPaused > > - name: AfStateContinuousFocused > > - name: AfStateContinuousFocusedPaused > > - name: AfStateContinuousFailed > > - name: AfStateContinuousFailedPaused > > I "multiplied out" all the different parts of the state to make a > single long list. Actually I think that makes it more Android-like. > But we could reduce this number if, for example, we're happy to report > the state of AfMode separately on every frame. I don't mind - do we > have a general preference for this kind of thing? > Good question. More generally we need to establish a policy for metadata. For controls we know that controls should be attached to a request only when their value changes. If a control is not supplied with new values, then the old (or default) one will stay valid. This implies the PH/IPA state machine is then stateful and 'remembers' what was the last value a control was set to. What about metadata instead ? Can we assume the same stateful-ness in applications too, or is it a burden we cannot impose on them ? Han-lin, do you know how does that work with Android ? Does it report metadata even when their values do not change or do they assume that, like controls, a metadata is reported only when its value has changed. What would the preference be for the RPi applications ? > > > > (please note that AfStateAuto seems mis-aligned, unless I messed up > > something while cutting&paste. Please check the compiled doc to see > > how it looks like...) > > I will check. > > > > > So we have three macro groups > > > > - Manual > > > > - Auto > > - Scanning > > - Focus > > - Failed > > > > - Continuous > > > > - Scanning > > - Pausing > > - Paused > > - Focused > > - Paused > > - Failed > > - Paused > > > > The first group is AfMode (Manual, Auto, Continuous) and can be > > deduced by inspecting the AfMode metadata. > > > > The second group is the actual state (Scannng, Focused, Failed) > > > > The third group is specific to continuous mode as it accepts a Pause > > trigger. > > > > I understand why you have this, as expressing "The AfState is set to > > $something when in AfModeManual" etc etc might get cumbersome, but I > > think it might be worth a try > > > > 1) Manual mode (the bad) > > As we all know there is a delay between when a lens is instructed > > to move and when it reaches its final state. > > > > In Manual mode the lens is moved with LensPosition and > > AfLensPosition. > > > > To know the current state: > > - With LensPosition metadata can be inspected until it doesn't reach the > > desired value. > > - With AfLensPosition it is not clear to me how to know when the lens > > as reched one of the pre-defined positions. > > It sounds like we're deleting AfLensPosition. > > > > > Is it so wrong to report either (Scanning,Focus,Failed) even for > > Manual mode ? > > I don't think it's wrong. If we have a single long list of states then > it just makes for more states. If we don't then it's harmless, though > I'm not aware of a use for it. > > > > > 2) AutoMode (the good) > > This seems the easy one, as (Scanning,Focus,Failed) are already > > there > > > > 3) Continous (the ugly) > > Continuous mode can be paused. That's a shame :) > > Would that be that bad keeping a Paused or Pausing state which can > > only be returned for Continuous mode. > > The whole paused/pausing thing rather complicates things, but it does > seem to me that it's necessary. If you were in Continuous mode, you > would rather the lens didn't set off again just as you want to capture > a sequence of pictures. > > > > > All in all, can we reduce the number of states to > > > > - Scanning > > > > The AF algorithm is scanning and has not yet reached a > > stable result. When running in Auto mode an AF scan is > > started by using the AfTrigger control. When in Manual mode > > and a new lens position is requested either by sending a > > LensPosition or AfLensPosition control, the AF state will > > report "Scanning" until the lens doesn't reach the desired > > position. Similarly, when running in continuous mode and the > > AF algorithm decides to start a scan spontaneously the > > AfState will report Scanning until a new stable state is > > reached. > > > > - Focus > > > > The AF algorithm has reached a stable state and the image is > > now focused. When running in Manual mode the lens has > > reached the position requested using either LensPosition and > > AfLensPosition. > > > > - Failed > > > > The AF algorithm has completed a scan and without finding a > > good focus position. When running in manual mode, the camera > > has failed moving the lens to the desired position. > > > > - Paused (copying your below text, please see comments below) > > > > The AF algorithm is in continuous mode (AfModeContinuous) and > > was scanning when AfPauseImmediate was sent. The scan will not > > complete and the algorithm will remain in this state. > > > > - Pausing > > > > The AF algorithm is in continuous mode (AfModeContinuous) and > > was scanning when AfPauseDeferred was sent. The scan will complete > > at which point the algorithm moves to the > > AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > state. > > I think we could get by if we report the following for every frame: > > AfMode - Manual, Auto or Continuous > AfState - Scanning, Focused or Failed > AfPause - Resume (which means it's running), Immediate or Deferred > (these states only apply during Continuous; the Deferred value can > only occur when also Scanning). > > Or we could wrap AfState and AfPause together: > > AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused, > FocusedPaused, FailedPaused. > > That makes some of the interaction between pausing and scanning > clearer, but then it's more states again. I don't know... > Let's clarify what the policy on metadata is first. > > > > > > > + > > > + In any of the AfStateManual or AfStateContinuous states, the AfMode > > > + may be set to AfModeAuto and the algorithm will move to the > > > + AfStateAuto state. If AfTriggerStart is sent at the same time > > > + then the algorithm will move to AfStateAutoScanning state. > > > + > > > + In any of the AfStateAuto or AfStateManual states, the AfMode may > > > + be set to AfModeContinuous and the algorithm will move to > > > + AfStateContinuousScanning. > > > + > > > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > > > + be set to AfModeManual and the algorithm will move to > > > + AfStateManual. The lens will not be moved and will be left where > > > + it was at that moment. > > > + > > > + enum: > > > + - name: AfStateManual > > > + value: 0 > > > + description: | > > > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > > > + and AfLensPosition controls can be used directly to move the lens. > > > + - name: AfStateAuto > > > + value: 1 > > > + description: | > > > + The AF algorithm is in auto mode (AfModeAuto), and has either just > > > + been moved into that state, or a scan that was in progress has been > > > + cancelled. > > > + - name: AfStateAutoScanning > > > + value: 2 > > > + description: | > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > > > + started using the AfTrigger control. The scan can be cancelled by > > > + sending AfTriggerCancel at which point the algorithm will either > > > + move back to AfStateAuto or, if the scan actually completes > > > + before the cancel request is processed, to one of > > > + AfStateAutoFocused or AfStateAutoFailed. > > > + - name: AfStateAutoFocused > > > + value: 3 > > > + description: | > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > > + completed with the result that the algorithm believes the image is > > > + now in focus. > > > + - name: AfStateAutoFailed > > > + value: 4 > > > + description: | > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > > + completed with the result that the algorithm did not find a good > > > + focus position. > > > + - name: AfStateContinuousScanning > > > + value: 5 > > > + description: | > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > + is currently scanning for a good focus position. This occurs when > > > + the mode is first set to AfModeContinuous, or may happen > > > + spontaneously when the algorithm believes a re-scan is needed. > > > + - name: AfStateContinuousScanningPausing > > > + value: 6 > > > + description: | > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > + was scanning when AfPauseDeferred was sent. The scan will complete > > > + at which point the algorithm moves to the > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > + state. > > > + - name: AfStateContinuousScanningPaused > > > + value: 7 > > > + description: | > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > + was scanning when AfPauseImmediate was sent. The scan will not > > > > Why can't the algorithm be paused when it has reached a stable state > > but only when scanning ? Won't a pause 'freeze' the algorithm in its > > current state no matter what it is ? I understand with > > "AfPauseDeferred" that it has to wait for AF to be locked before > > stopping it, but with "Immediate" doesn't "pause" apply to all states ? > > Wasn't sure I understood this. It can be paused once it's reached > Focused or Failed, and then both PauseImmediate and PauseDeferred will > cause it to report FocusedPaused or FailedPaused. Was that the > question? You're right, I have probably missed FocusedPaused and FailedPaused. :) Thanks j > > Anyway, thanks very much! > > David > > > > > Thanks > > j > > > > > + complete and the algorithm will remain in this state. The scan > > > + may be resumed by sending AfPauseResume. > > > + - name: AfStateContinuousFocused > > > + value: 8 > > > + description: | > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > + a scan has completed with the algorithm believing it has found a > > > + good focus position. > > > + - name: AfStateContinuousFocusedPaused > > > + value: 9 > > > + description: | > > > + The AF algorithm was in the AfStateContinuousFocused state and > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > > + or it was in the AfStateContinuousScanningPausing state and the > > > + scan has completed successfully. The algorithm will now remain > > > + in this state, and may be resumed by sending AfPauseResume. > > > + - name: AfStateContinuousFailed > > > + value: 10 > > > + description: | > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > + a scan has completed without finding a good focus position. > > > + - name: AfStateContinuousFailedPaused > > > + value: 11 > > > + description: | > > > + The AF algorithm was in the AfStateContinuousFailed state and > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > > + or it was in the AfStateContinuousScanningPausing state and the > > > + scan has completed unsuccessfully. The algorithm will now remain > > > + in this state, and may be resumed by sending AfPauseResume. > > > + > > > ... > > > -- > > > 2.30.2 > > >
Hi Jacopo Thanks for the reply! On Wed, 6 Apr 2022 at 17:34, Jacopo Mondi <jacopo@jmondi.org> wrote: > > Hi David, > + Han-lin for an Android question on metadata > > On Wed, Apr 06, 2022 at 02:05:20PM +0100, David Plowman wrote: > > Hi Jacopo > > > > Thanks for taking the time to read all this! > > > > On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <jacopo@jmondi.org> wrote: > > > > > > Hi David, > > > sorry for being late :) > > > > > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote: > > > > This patch describes a series of controls that allow applications to > > > > drive AF algorithms: > > > > > > > > AfMode - manual, auto or continuous > > > > AfLensPosition - set lens position to macro/hyperfocal/infinity > > > > AfRange - full, macro or normal > > > > AfSpeed - fast or slow > > > > AfWindows - AF window locations > > > > AfTrigger - start (trigger an AF scan) or cancel > > > > AfPause - pause continuous AF > > > > LensPosition - position of lens from lens driver > > > > AfState - reports the mode, whether scanning/success/failure > > > > --- > > > > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > > > > 1 file changed, 313 insertions(+), 60 deletions(-) > > > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > > > > index 9d4638ae..23607368 100644 > > > > --- a/src/libcamera/control_ids.yaml > > > > +++ b/src/libcamera/control_ids.yaml > > > > @@ -406,27 +406,6 @@ controls: > > > > The camera will cancel any active or completed metering sequence. > > > > The AE algorithm is reset to its initial state. > > > > > > > > - - AfTrigger: > > > > - type: int32_t > > > > - draft: true > > > > - description: | > > > > - Control for AF trigger. Currently identical to > > > > - ANDROID_CONTROL_AF_TRIGGER. > > > > - > > > > - Whether the camera device will trigger autofocus for this request. > > > > - enum: > > > > - - name: AfTriggerIdle > > > > - value: 0 > > > > - description: The trigger is idle. > > > > - - name: AfTriggerStart > > > > - value: 1 > > > > - description: The AF routine is started by the camera. > > > > - - name: AfTriggerCancel > > > > - value: 2 > > > > - description: | > > > > - The camera will cancel any active trigger and the AF routine is > > > > - reset to its initial state. > > > > - > > > > - NoiseReductionMode: > > > > type: int32_t > > > > draft: true > > > > @@ -507,45 +486,6 @@ controls: > > > > The AE algorithm has started a pre-capture metering session. > > > > \sa AePrecaptureTrigger > > > > > > > > - - AfState: > > > > - type: int32_t > > > > - draft: true > > > > - description: | > > > > - Control to report the current AF algorithm state. Currently identical to > > > > - ANDROID_CONTROL_AF_STATE. > > > > - > > > > - Current state of the AF algorithm. > > > > - enum: > > > > - - name: AfStateInactive > > > > - value: 0 > > > > - description: The AF algorithm is inactive. > > > > - - name: AfStatePassiveScan > > > > - value: 1 > > > > - description: | > > > > - AF is performing a passive scan of the scene in continuous > > > > - auto-focus mode. > > > > - - name: AfStatePassiveFocused > > > > - value: 2 > > > > - description: | > > > > - AF believes the scene is in focus, but might restart scanning. > > > > - - name: AfStateActiveScan > > > > - value: 3 > > > > - description: | > > > > - AF is performing a scan triggered by an AF trigger request. > > > > - \sa AfTrigger > > > > - - name: AfStateFocusedLock > > > > - value: 4 > > > > - description: | > > > > - AF believes has focused correctly and has locked focus. > > > > - - name: AfStateNotFocusedLock > > > > - value: 5 > > > > - description: | > > > > - AF has not been able to focus and has locked. > > > > - - name: AfStatePassiveUnfocused > > > > - value: 6 > > > > - description: | > > > > - AF has completed a passive scan without finding focus. > > > > - > > > > - AwbState: > > > > type: int32_t > > > > draft: true > > > > @@ -690,4 +630,317 @@ controls: > > > > value. All of the custom test patterns will be static (that is the > > > > raw image must not vary from frame to frame). > > > > > > > > + - AfMode: > > > > + type: int32_t > > > > + draft: true > > > > > > Was having all these controls as draft intentional ? The previous > > > definitions where set to drafts as they mapped to the android > > > definition, but now that we're defining our own ones... > > > > Not really... I only made them "draft" because it seemed like > > everything was "draft" at first! But happy to change that. Are you > > saying that these definitely should *not* be draft, then? > > > > Yeah, "draft" was used to import controls from Android as they are for > sake of CTS compliance. Let's drop draft from the ones we're defining Will do! > > > > > > > > + description: | > > > > + Control to set the mode of the AF (autofocus) algorithm. Applications > > > > + are allowed to set a new mode, and to send additional controls for > > > > + that new mode, in the same request. > > > > + > > > > + An implementation may choose not to implement all the modes. > > > > + > > > > + enum: > > > > + - name: AfModeManual > > > > + value: 0 > > > > + description: | > > > > + The AF algorithm is in manual mode. In this mode it will never > > > > + perform any action nor move the lens of its own accord, but an > > > > + application can set controls to move the lens. > > > > + > > > > + In this mode the AfState will always report AfStateManual. > > > > + - name: AfModeAuto > > > > + value: 1 > > > > + description: | > > > > + The AF algorithm is in auto mode. This means that the algorithm > > > > + will never move the lens or change state unless the AfTrigger > > > > + control is used. The AfTrigger control can be used to initiate a > > > > + focus scan, the results of which will also be reported by AfState. > > > > + > > > > + If the autofocus algorithm is moved from AfModeAuto to another > > > > + mode while a scan is in progress, the scan is cancelled > > > > + immediately, without waiting for the scan to finish. > > > > + > > > > + When first entering this mode the AfState will report > > > > + AfStateAuto. When a trigger control is sent, AfState will > > > > + report AfStateAutoScanning for a period before spontaneously > > > > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > > > > + the outcome of the scan. It will remain in this state until > > > > + another scan is initiated by the AfTrigger control. If a scan is > > > > + cancelled (without changing to another mode), AfState will return > > > > + to AfStateAuto. > > > > + - name: AfModeContinuous > > > > + value: 2 > > > > + description: | > > > > + The AF algorithm is in continuous mode. This means that the lens > > > > + can re-start a scan spontaneously at any moment, without any user > > > > + intervention. The AfState still reports whether the algorithm is > > > > + currently scanning or not, though the application has no ability > > > > + to initiate or cancel scans (though it can "pause" them), nor to > > > > + move the lens for itself. > > > > + > > > > + When set to AfModeContinuous, the system will immediately initiate > > > > + a scan so AfState will report AfStateContinuousScanning, and will > > > > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > > > > + depending on the scan result. > > > > + > > > > + - AfLensPosition: > > > > + type: int32_t > > > > + draft: true > > > > + description: | > > > > + Control to set the position of the lens to one of the following > > > > + predefined locations. This control only has any effect when the AfMode > > > > + is set to AfModeManual. > > > > + > > > > + This control is distinct from the LensPosition control, which sets the > > > > + lens position using the lens driver's units. > > > > + enum: > > > > + - name: AfLensPositionMacro > > > > + value: 0 > > > > + description: The closest focal position of the lens. > > > > + - name: AfLensPositionHyperfocal > > > > + value: 1 > > > > + description: Hyperfocal position. > > > > + - name: AfLensPositionInfinity > > > > + value: 2 > > > > + description: The furthest focal position (usually infinity). > > > > > > Interesting, bikeshedding on the name apart, is this some 'simplified' > > > LensState ? Do you think there is a use case for this ? Why would > > > anyone set the AF to auto mode then use some coarse grained control > > > like this one ? Wouldn't it make more sense as an 'hint' to the AF > > > algorithm to restrict the search space ? > > > > In the light of your comments below about the LensPosition not using > > driver units, then maybe this could be deleted, and we could select > > some kind of canonical range for the lens? > > > > Ah wait. What I was suggesting ("An "hint" to the AF algorithm to > restrict the search space") is basically AfRange. I wonder if there's a case for a "custom" value for AfRange, and we could then even have a control that sets the custom range? > > > The most basic use case would be following on from the capture of an > > image after an AF scan: > > > > - capture image > > - AfMode to manual > > - Lens to hyperfocal > > - AfMode to auto > > - now wait for the shutter button to be pressed before you start > > another AF scan. > > Naive question: > Wouldn't setting the lens to hyperfocal render the preview unusable ? Hyperfocal is the place where "most stuff" is normally in focus (strictly speaking, has the greatest depth of field), so in the absence of other information it's normally the best place to put the lens. It's just for the preview, you'd typically run another scan for a capture. (Some systems run Continuous AF during preview, of course. But you know my anxiety about CAF algorithms... in my opinion just putting the lens to hyperfocal is way less annoying than a not-very-good CAF implementation!) > > > > > If your UI has a "macro" capture mode of some kind, maybe it's "Lens > > to macro" instead of "Lens to hyperfocal". > > Why is this different than starting an AF scan (without going through > Manual) with an AfRange that suggests in which range to scan ? Is this > an optimization to forcefully move the lens closer to where it will > likely end being moved to ? If you know you're doing a macro capture it would be helpful for the preview to put the lens somewhere where macro things are likely to be more in focus, instead of using hyperfocal which will likely make close objects very blurry. > > > > > But as noted, if LensPosition allows us to select hyperfocal/macro > > directly, then we can do without this one. > > > > Providing my above understanding is correct, it would be a bit less intuitive > for applications to use a numerical value instead of some easier > "Macro"/"Hyperfocal". Of course, wrappers like your one can simplify this > for their consumers. True, though I'm now quite liking the "dioptre-like" canonical range described below, where 1 = hyperfocal, 0 = infinity, large numbers = close > > > > > > > (Btw I have a use case for supplying a range of lens positions, which > > > has been previously estimated according to some pre-defined use cases, > > > to the AF algorithm in order restrict the AF search range to increase > > > the speed of a scan. Do you think it's a valid use case and it's worth a > > > control ?) > > > > I'd imagined doing this kind of thing with the AfSpeed and AfRange > > controls and by having the AF algorithm tuned for the module. I'm not > > convinced about passing in lens positions from outside the camera > > system. Where would they come from? Shouldn't the camera system know > > what positions to use? > > > > A possible use case, for controlled environments with low latency > requirements, is to run tests beforehand and for each 'scene' define > what is the ideal lens movement range. When the application starts an > AF scan it would tell the AF algorithm about the 'preferred' range. > > I understand this works backwards, as the lens position should be the > result of the AF algorithm run not one of its inputs :) Indeed, but as noted above, maybe there's an argument for an AfRangeCustom which could be user-settable? We've added "Custom" values for other controls in some places, IIRC. > > > In the Raspberry Pi world we would of course store this kind of > > information in the tuning file. I've noticed some discussion elsewhere > > about supplying configuration to the IPAs - it seems like this might > > be a case in point? I think that for real camera systems, in the end, > > there is no escape from "configuration" (aka. "tuning"). > > > > > > > > > + > > > > + - AfRange: > > > > + type: int32_t > > > > + draft: true > > > > + description: | > > > > + Control to set the range of focus distances that is scanned. An > > > > + implementation may choose not to implement all the options here. > > > > > > Only valid in Auto and Continuous mode, right ? > > > > Generally I've been expecting that you could change settings like this > > in Manual mode too, though it obviously can't have any effect until > > you go back to Auto/Continuous. Does that sound reasonable? > > I would rather assume thaat if a control is not valid in the current > context is simply ignored and not cached to be re-applied when the > context change ? Possibly, though I'm slightly nervous about how you'd do things like: - You're in Manual mode, but want to go to Auto and set the Range to Normal. - You're in Auto mode but were using a different range. You want to reset it to Normal before switching to Manual. Can you send the AfRange change and the new mode at the same time in both cases or not? Or do you need two separate requests to send the changes? It's not clear to me exactly what would work. If you say that the AfRange always gets updated, then the behaviour is obvious, and you can send both AfRange and AfMode in the same request which would be easier to use. > > > > > > > > > (This makes me think that we could easily model Continuous by adding a > > > TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of > > > AfPause... well, it's quite a redesign so I'll stop here unless > > > someone thinks it's a really good idea :) > > > > Actually I'm quite strongly against anything like this. I think Auto > > and Continuous should be quite separate. The reason is that Auto is > > (relatively) easy to implement, you do a scan over some range and > > choose the best spot. But Continuous AF is an absolute nightmare to do > > well. You have all these problems: > > > > - The algo doesn't realise it needs to rescan and so seems to be > > "stuck". (Watch people waving their hands in front of the lens to try > > and unstick it.) So you add ever more tests and arbitrary thresholds > > on different statistics to tell if a rescan is needed, and then the > > whole thing becomes untunable. > > - The algo rescans too often. A car in the distance goes by and the > > whole image see-saws in and out while a rescan happens, ending back at > > the same place. Yet more tests, thresholds... > > - You can never make all your tests and thresholds work at the same > > time but in different conditions. So you make them variable depending > > on things like exposure and gain. Now you are truly doomed. > > - The algo is generally not robust to camera shake, which frequently > > causes it to end up "failing" with a completely out-of-focus image. > > And then you can't get it to rescan... cue more hand-waving in front > > of the lens. You implement more special tests and timeouts... At this > > point you will wish for the earth to swallow you up. > > - When the algo rescans it starts off going the wrong way causing > > extra see-sawing. > > - You try to solve the last problem by doing really small initial > > steps. Only then it ends up statistically unstable. > > > > From the above you may detect that we have been there and have several > > T-shirts. Speaking for myself, I have no plans to implement Continuous > > AF without PDAF sensors, as PDAF is quite a big help. > > > > Yes, it feels like you know what you're talking about :) > > I understand you would like be cautious with CAF, but if one doesn't > want to implement CAF can't she simply not report TriggerContinuous ? > > Anyway, that was just an idea thrown in the mixer, don't worry about > it. > > > > > > > > + enum: > > > > + - name: AfRangeNormal > > > > + value: 0 > > > > + description: | > > > > + A wide range of focus distances is scanned, all the way from > > > > + infinity down to close distances, though depending on the > > > > + implementation, possibly not including the very closest macro > > > > + positions. > > > > + - name: AfRangeMacro > > > > + value: 1 > > > > + description: Only close distances are scanned. > > > > + - name: AfRangeFull > > > > + value: 2 > > > > + description: | > > > > + The full range of focus distances is scanned just as with > > > > + AfRangeNormal but this time including the very closest macro > > > > + positions. > > > > + > > > > + - AfSpeed: > > > > + type: int32_t > > > > + draft: true > > > > + description: | > > > > + Control that determines whether the AF algorithm is to move the lens > > > > + as quickly as possible or more steadily. For example, during video > > > > + recording it may be desirable not to move the lens too abruptly, but > > > > + when in a preview mode (waiting for a still capture) it may be > > > > + helpful to move the lens as quickly as is reasonably possible. > > > > + enum: > > > > + - name: AfSpeedNormal > > > > + value: 0 > > > > + description: Move the lens at its usual speed. > > > > + - name: AfSpeedFast > > > > + value: 1 > > > > + description: Move the lens more quickly. > > > > + > > > > + - AfWindows: > > > > + type: Rectangle > > > > + draft: true > > > > + description: | > > > > + Sets the focus windows used by the AF algorithm. The units used express > > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire > > > > + image), as u0.16 format numbers. > > > > + > > > > + In order to be activated, a rectangle must be programmed with non-zero > > > > + width and height. If no rectangles are programmed in this way, then the > > > > + system will choose its own single default window in the centre of the > > > > + image. > > > > + > > > > + The details of how the windows are used are platform dependent. We note > > > > + that when there is more than one AF window, a typical implementation > > > > + might find the optimal focus position for each one and finally select > > > > + the window closest to the camera. > > > > + > > > > + size: [platform dependent] > > > > > > I think this should just be > > > > > > size: [n] > > > > OK! > > > > > > + > > > > + - AfTrigger: > > > > + type: int32_t > > > > + draft: true > > > > + description: | > > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > > > > + and can also be used to terminate a scan early. > > > > + > > > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > > > > + > > > > + enum: > > > > + - name: AfTriggerStart > > > > + value: 0 > > > > + description: Start an AF scan. Ignored if a scan is in progress. > > > > + - name: AfTriggerCancel > > > > + value: 1 > > > > + description: Cancel an AF scan. This does not cause the lens to move > > > > + anywhere else. Ignored if no scan is in progress. > > > > + > > > > + - AfPause: > > > > + type: int32_t > > > > + draft: true > > > > + description: | > > > > + This control has no effect except when in continuous autofocus mode > > > > + (AfModeContinuous). It can be used to pause any lens movements while > > > > + (for example) images are captured. The algorithm remains inactive > > > > + until it is instructed to resume. > > > > + > > > > + enum: > > > > + - name: AfPauseImmediate > > > > + value: 0 > > > > + description: | > > > > + Pause the continuous autofocus algorithm immediately, whether or > > > > + not any kind of scan is underway. If the AfState was previously > > > > + reporting AfStateContinuousScanning it will now change to > > > > + AfStateContinuousScanningPaused, and similarly for > > > > + AfStateContinuousFocused and AfStateContinuousFailed. > > > > + - name AfPauseDeferred > > > > + value: 1 > > > > + description: | > > > > + Pause the continuous autofocus algorithm as soon as it is no longer > > > > + scanning. If the AfState is currently reporting > > > > + AfStateContinuousFocused is will change to > > > > + AfStateContinuousFocusedPaused, and similarly in the case of > > > > + AfStateContinuousFailed. > > > > + > > > > + If AfState reports AfStateContinuousScanning it will change to > > > > + AfStateContinuousScanningPausing, and then move to one of > > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > > + when the scan completes. > > > > + - name: AfPauseResume > > > > + value: 2 > > > > + description: | > > > > + Resume continous autofocus operation. The algorithm starts again > > > > + from exactly where it left off, and the AfState will drop the > > > > + 'Paused' or 'Pausing' part of the state. > > > > + > > > > + - LensPosition: > > > > + type: int32_t > > > > + draft: true > > > > + description: | > > > > + Acts as a control to instruct the lens to move to a particular position > > > > + and also reports back the position of the lens for each frame. > > > > + > > > > + The units are determined by the lens driver. > > > > > > Ugh, we cannot do this in the control definition. Driver specificities > > > cannot surface to the controls definition, which should be platform > > > independent. > > > > > > I think we'll have to establish a numerical range (go ahead if you > > > have any idea) and translate to the actual driver values in the > > > CameraLens class ? > > > > I wasn't aware of this, but maybe it's a good thing. The question is > > Well, isn't it the same with "Brightness", for example ? Indeed, or ExposureTime where we explicitly want microseconds, not driver units. > > > what kind of range to use, and how do you know what limits apply for > > any given camera module? How about a dioptre-like scale, normalised > > for the hyperfocal distance H. It would go like this: > > > > The value n would mean being focused at distance H/n (n does not have > > to be an integer). So: > > > > The value 1 would mean hyperfocal position (focal distance H/1). > > The value 0 would mean infinity (focal distance H/0). > > Larger values would get progressively closer. > > > > Setting a control value outside the allowed range will clamp it to > > that end of the range. > > I sincerely don't know enough about this to tell if it's a good idea > or not. Hence, it seems like a good idea to me :) > > > > > In the Raspberry Pi world it would be sufficient for the tuning file > > to know the allowable range of values. More generally, I could imagine > > an application might want to say "focus at precisely 1m" in which case > > it would have to know the value of H from a property or something, > > though providing such information would be optional (some lenses > > simply aren't precisely calibrated). Perhaps an application might want > > to get the allowable range of (maybe approximate) focal distances too? > > > > > > > > > + > > > > + The LensPosition control is ignored unless the AfMode is set to > > > > + AfModeManual. > > > > + > > > > + Note that this control is distinct from AfLensPosition, which allows > > > > + the lens to be moved to its macro/hyperfocal/infinity position, rather > > > > + than using lens driver units. > > > > + > > > > + - AfState: > > > > + type: int32_t > > > > + draft: true > > > > + description: | > > > > + Reports the current state of the AF algorithm. The possible state changes > > > > + are described below, although we add that the following general state > > > > + changes are also allowed. > > > > > > wow, there is a lot of states here :) > > > > Did you follow the link to the state transition diagram? I would hope > > that should go some way to explaining it! Here's the link again for > > convenience: > > > > https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing > > > > Missed it. Thanks for providing it. > > > > > > > Let me copy them here > > > > > > - name: AfStateManual > > > - name: AfStateAuto > > > - name: AfStateAutoScanning > > > - name: AfStateAutoFocused > > > - name: AfStateAutoFailed > > > - name: AfStateContinuousScanning > > > - name: AfStateContinuousScanningPausing > > > - name: AfStateContinuousScanningPaused > > > - name: AfStateContinuousFocused > > > - name: AfStateContinuousFocusedPaused > > > - name: AfStateContinuousFailed > > > - name: AfStateContinuousFailedPaused > > > > I "multiplied out" all the different parts of the state to make a > > single long list. Actually I think that makes it more Android-like. > > But we could reduce this number if, for example, we're happy to report > > the state of AfMode separately on every frame. I don't mind - do we > > have a general preference for this kind of thing? > > > > Good question. > > More generally we need to establish a policy for metadata. For > controls we know that controls should be attached to a request only > when their value changes. If a control is not supplied with new > values, then the old (or default) one will stay valid. > > This implies the PH/IPA state machine is then stateful and 'remembers' > what was the last value a control was set to. > > What about metadata instead ? Can we assume the same stateful-ness in > applications too, or is it a burden we cannot impose on them ? > > Han-lin, do you know how does that work with Android ? Does it report > metadata even when their values do not change or do they assume that, > like controls, a metadata is reported only when its value has changed. > > What would the preference be for the RPi applications ? I don't think I have a very strong preference. I'm happy with either: - Report everything in the AfState on every frame. - Report the state in multiple metadata items (say, AfState and AfMode) on every frame. I'm less keen on having applications have to remember if they sent something and then wait for the request where it actually happened, it seems to me that would simply add annoying boilerplate code to all applications. What do you think? > > > > > > > (please note that AfStateAuto seems mis-aligned, unless I messed up > > > something while cutting&paste. Please check the compiled doc to see > > > how it looks like...) > > > > I will check. > > > > > > > > So we have three macro groups > > > > > > - Manual > > > > > > - Auto > > > - Scanning > > > - Focus > > > - Failed > > > > > > - Continuous > > > > > > - Scanning > > > - Pausing > > > - Paused > > > - Focused > > > - Paused > > > - Failed > > > - Paused > > > > > > The first group is AfMode (Manual, Auto, Continuous) and can be > > > deduced by inspecting the AfMode metadata. > > > > > > The second group is the actual state (Scannng, Focused, Failed) > > > > > > The third group is specific to continuous mode as it accepts a Pause > > > trigger. > > > > > > I understand why you have this, as expressing "The AfState is set to > > > $something when in AfModeManual" etc etc might get cumbersome, but I > > > think it might be worth a try > > > > > > 1) Manual mode (the bad) > > > As we all know there is a delay between when a lens is instructed > > > to move and when it reaches its final state. > > > > > > In Manual mode the lens is moved with LensPosition and > > > AfLensPosition. > > > > > > To know the current state: > > > - With LensPosition metadata can be inspected until it doesn't reach the > > > desired value. > > > - With AfLensPosition it is not clear to me how to know when the lens > > > as reched one of the pre-defined positions. > > > > It sounds like we're deleting AfLensPosition. > > > > > > > > Is it so wrong to report either (Scanning,Focus,Failed) even for > > > Manual mode ? > > > > I don't think it's wrong. If we have a single long list of states then > > it just makes for more states. If we don't then it's harmless, though > > I'm not aware of a use for it. > > > > > > > > 2) AutoMode (the good) > > > This seems the easy one, as (Scanning,Focus,Failed) are already > > > there > > > > > > 3) Continous (the ugly) > > > Continuous mode can be paused. That's a shame :) > > > Would that be that bad keeping a Paused or Pausing state which can > > > only be returned for Continuous mode. > > > > The whole paused/pausing thing rather complicates things, but it does > > seem to me that it's necessary. If you were in Continuous mode, you > > would rather the lens didn't set off again just as you want to capture > > a sequence of pictures. > > > > > > > > All in all, can we reduce the number of states to > > > > > > - Scanning > > > > > > The AF algorithm is scanning and has not yet reached a > > > stable result. When running in Auto mode an AF scan is > > > started by using the AfTrigger control. When in Manual mode > > > and a new lens position is requested either by sending a > > > LensPosition or AfLensPosition control, the AF state will > > > report "Scanning" until the lens doesn't reach the desired > > > position. Similarly, when running in continuous mode and the > > > AF algorithm decides to start a scan spontaneously the > > > AfState will report Scanning until a new stable state is > > > reached. > > > > > > - Focus > > > > > > The AF algorithm has reached a stable state and the image is > > > now focused. When running in Manual mode the lens has > > > reached the position requested using either LensPosition and > > > AfLensPosition. > > > > > > - Failed > > > > > > The AF algorithm has completed a scan and without finding a > > > good focus position. When running in manual mode, the camera > > > has failed moving the lens to the desired position. > > > > > > - Paused (copying your below text, please see comments below) > > > > > > The AF algorithm is in continuous mode (AfModeContinuous) and > > > was scanning when AfPauseImmediate was sent. The scan will not > > > complete and the algorithm will remain in this state. > > > > > > - Pausing > > > > > > The AF algorithm is in continuous mode (AfModeContinuous) and > > > was scanning when AfPauseDeferred was sent. The scan will complete > > > at which point the algorithm moves to the > > > AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > state. > > > > I think we could get by if we report the following for every frame: > > > > AfMode - Manual, Auto or Continuous > > AfState - Scanning, Focused or Failed > > AfPause - Resume (which means it's running), Immediate or Deferred > > (these states only apply during Continuous; the Deferred value can > > only occur when also Scanning). > > > > Or we could wrap AfState and AfPause together: > > > > AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused, > > FocusedPaused, FailedPaused. > > > > That makes some of the interaction between pausing and scanning > > clearer, but then it's more states again. I don't know... > > > > Let's clarify what the policy on metadata is first. Agree! > > > > > > > > > > > + > > > > + In any of the AfStateManual or AfStateContinuous states, the AfMode > > > > + may be set to AfModeAuto and the algorithm will move to the > > > > + AfStateAuto state. If AfTriggerStart is sent at the same time > > > > + then the algorithm will move to AfStateAutoScanning state. > > > > + > > > > + In any of the AfStateAuto or AfStateManual states, the AfMode may > > > > + be set to AfModeContinuous and the algorithm will move to > > > > + AfStateContinuousScanning. > > > > + > > > > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > > > > + be set to AfModeManual and the algorithm will move to > > > > + AfStateManual. The lens will not be moved and will be left where > > > > + it was at that moment. > > > > + > > > > + enum: > > > > + - name: AfStateManual > > > > + value: 0 > > > > + description: | > > > > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > > > > + and AfLensPosition controls can be used directly to move the lens. > > > > + - name: AfStateAuto > > > > + value: 1 > > > > + description: | > > > > + The AF algorithm is in auto mode (AfModeAuto), and has either just > > > > + been moved into that state, or a scan that was in progress has been > > > > + cancelled. > > > > + - name: AfStateAutoScanning > > > > + value: 2 > > > > + description: | > > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > > > > + started using the AfTrigger control. The scan can be cancelled by > > > > + sending AfTriggerCancel at which point the algorithm will either > > > > + move back to AfStateAuto or, if the scan actually completes > > > > + before the cancel request is processed, to one of > > > > + AfStateAutoFocused or AfStateAutoFailed. > > > > + - name: AfStateAutoFocused > > > > + value: 3 > > > > + description: | > > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > > > + completed with the result that the algorithm believes the image is > > > > + now in focus. > > > > + - name: AfStateAutoFailed > > > > + value: 4 > > > > + description: | > > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > > > + completed with the result that the algorithm did not find a good > > > > + focus position. > > > > + - name: AfStateContinuousScanning > > > > + value: 5 > > > > + description: | > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > + is currently scanning for a good focus position. This occurs when > > > > + the mode is first set to AfModeContinuous, or may happen > > > > + spontaneously when the algorithm believes a re-scan is needed. > > > > + - name: AfStateContinuousScanningPausing > > > > + value: 6 > > > > + description: | > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > + was scanning when AfPauseDeferred was sent. The scan will complete > > > > + at which point the algorithm moves to the > > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > > + state. > > > > + - name: AfStateContinuousScanningPaused > > > > + value: 7 > > > > + description: | > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > + was scanning when AfPauseImmediate was sent. The scan will not > > > > > > Why can't the algorithm be paused when it has reached a stable state > > > but only when scanning ? Won't a pause 'freeze' the algorithm in its > > > current state no matter what it is ? I understand with > > > "AfPauseDeferred" that it has to wait for AF to be locked before > > > stopping it, but with "Immediate" doesn't "pause" apply to all states ? > > > > Wasn't sure I understood this. It can be paused once it's reached > > Focused or Failed, and then both PauseImmediate and PauseDeferred will > > cause it to report FocusedPaused or FailedPaused. Was that the > > question? > > You're right, I have probably missed FocusedPaused and FailedPaused. > :) > > Thanks > j Thanks for all the discussion! David > > > > > Anyway, thanks very much! > > > > David > > > > > > > > Thanks > > > j > > > > > > > + complete and the algorithm will remain in this state. The scan > > > > + may be resumed by sending AfPauseResume. > > > > + - name: AfStateContinuousFocused > > > > + value: 8 > > > > + description: | > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > + a scan has completed with the algorithm believing it has found a > > > > + good focus position. > > > > + - name: AfStateContinuousFocusedPaused > > > > + value: 9 > > > > + description: | > > > > + The AF algorithm was in the AfStateContinuousFocused state and > > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > > > + or it was in the AfStateContinuousScanningPausing state and the > > > > + scan has completed successfully. The algorithm will now remain > > > > + in this state, and may be resumed by sending AfPauseResume. > > > > + - name: AfStateContinuousFailed > > > > + value: 10 > > > > + description: | > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > + a scan has completed without finding a good focus position. > > > > + - name: AfStateContinuousFailedPaused > > > > + value: 11 > > > > + description: | > > > > + The AF algorithm was in the AfStateContinuousFailed state and > > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > > > + or it was in the AfStateContinuousScanningPausing state and the > > > > + scan has completed unsuccessfully. The algorithm will now remain > > > > + in this state, and may be resumed by sending AfPauseResume. > > > > + > > > > ... > > > > -- > > > > 2.30.2 > > > >
Hi David and Jacopo, On Thu, Apr 7, 2022 at 5:05 PM David Plowman <david.plowman@raspberrypi.com> wrote: > Hi Jacopo > > Thanks for the reply! > > On Wed, 6 Apr 2022 at 17:34, Jacopo Mondi <jacopo@jmondi.org> wrote: > > > > Hi David, > > + Han-lin for an Android question on metadata > > > > On Wed, Apr 06, 2022 at 02:05:20PM +0100, David Plowman wrote: > > > Hi Jacopo > > > > > > Thanks for taking the time to read all this! > > > > > > On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <jacopo@jmondi.org> wrote: > > > > > > > > Hi David, > > > > sorry for being late :) > > > > > > > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via > libcamera-devel wrote: > > > > > This patch describes a series of controls that allow applications > to > > > > > drive AF algorithms: > > > > > > > > > > AfMode - manual, auto or continuous > > > > > AfLensPosition - set lens position to macro/hyperfocal/infinity > > > > > AfRange - full, macro or normal > > > > > AfSpeed - fast or slow > > > > > AfWindows - AF window locations > > > > > AfTrigger - start (trigger an AF scan) or cancel > > > > > AfPause - pause continuous AF > > > > > LensPosition - position of lens from lens driver > > > > > AfState - reports the mode, whether scanning/success/failure > > > > > --- > > > > > src/libcamera/control_ids.yaml | 373 > +++++++++++++++++++++++++++------ > > > > > 1 file changed, 313 insertions(+), 60 deletions(-) > > > > > > > > > > diff --git a/src/libcamera/control_ids.yaml > b/src/libcamera/control_ids.yaml > > > > > index 9d4638ae..23607368 100644 > > > > > --- a/src/libcamera/control_ids.yaml > > > > > +++ b/src/libcamera/control_ids.yaml > > > > > @@ -406,27 +406,6 @@ controls: > > > > > The camera will cancel any active or completed > metering sequence. > > > > > The AE algorithm is reset to its initial state. > > > > > > > > > > - - AfTrigger: > > > > > - type: int32_t > > > > > - draft: true > > > > > - description: | > > > > > - Control for AF trigger. Currently identical to > > > > > - ANDROID_CONTROL_AF_TRIGGER. > > > > > - > > > > > - Whether the camera device will trigger autofocus for this > request. > > > > > - enum: > > > > > - - name: AfTriggerIdle > > > > > - value: 0 > > > > > - description: The trigger is idle. > > > > > - - name: AfTriggerStart > > > > > - value: 1 > > > > > - description: The AF routine is started by the camera. > > > > > - - name: AfTriggerCancel > > > > > - value: 2 > > > > > - description: | > > > > > - The camera will cancel any active trigger and the AF > routine is > > > > > - reset to its initial state. > > > > > - > > > > > - NoiseReductionMode: > > > > > type: int32_t > > > > > draft: true > > > > > @@ -507,45 +486,6 @@ controls: > > > > > The AE algorithm has started a pre-capture metering > session. > > > > > \sa AePrecaptureTrigger > > > > > > > > > > - - AfState: > > > > > - type: int32_t > > > > > - draft: true > > > > > - description: | > > > > > - Control to report the current AF algorithm state. > Currently identical to > > > > > - ANDROID_CONTROL_AF_STATE. > > > > > - > > > > > - Current state of the AF algorithm. > > > > > - enum: > > > > > - - name: AfStateInactive > > > > > - value: 0 > > > > > - description: The AF algorithm is inactive. > > > > > - - name: AfStatePassiveScan > > > > > - value: 1 > > > > > - description: | > > > > > - AF is performing a passive scan of the scene in > continuous > > > > > - auto-focus mode. > > > > > - - name: AfStatePassiveFocused > > > > > - value: 2 > > > > > - description: | > > > > > - AF believes the scene is in focus, but might restart > scanning. > > > > > - - name: AfStateActiveScan > > > > > - value: 3 > > > > > - description: | > > > > > - AF is performing a scan triggered by an AF trigger > request. > > > > > - \sa AfTrigger > > > > > - - name: AfStateFocusedLock > > > > > - value: 4 > > > > > - description: | > > > > > - AF believes has focused correctly and has locked > focus. > > > > > - - name: AfStateNotFocusedLock > > > > > - value: 5 > > > > > - description: | > > > > > - AF has not been able to focus and has locked. > > > > > - - name: AfStatePassiveUnfocused > > > > > - value: 6 > > > > > - description: | > > > > > - AF has completed a passive scan without finding focus. > > > > > - > > > > > - AwbState: > > > > > type: int32_t > > > > > draft: true > > > > > @@ -690,4 +630,317 @@ controls: > > > > > value. All of the custom test patterns will be static > (that is the > > > > > raw image must not vary from frame to frame). > > > > > > > > > > + - AfMode: > > > > > + type: int32_t > > > > > + draft: true > > > > > > > > Was having all these controls as draft intentional ? The previous > > > > definitions where set to drafts as they mapped to the android > > > > definition, but now that we're defining our own ones... > > > > > > Not really... I only made them "draft" because it seemed like > > > everything was "draft" at first! But happy to change that. Are you > > > saying that these definitely should *not* be draft, then? > > > > > > > Yeah, "draft" was used to import controls from Android as they are for > > sake of CTS compliance. Let's drop draft from the ones we're defining > > Will do! > > > > > > > > > > > > + description: | > > > > > + Control to set the mode of the AF (autofocus) algorithm. > Applications > > > > > + are allowed to set a new mode, and to send additional > controls for > > > > > + that new mode, in the same request. > > > > > + > > > > > + An implementation may choose not to implement all the > modes. > > > > > + > > > > > + enum: > > > > > + - name: AfModeManual > > > > > + value: 0 > > > > > + description: | > > > > > + The AF algorithm is in manual mode. In this mode it > will never > > > > > + perform any action nor move the lens of its own > accord, but an > > > > > + application can set controls to move the lens. > > > > > + > > > > > + In this mode the AfState will always report > AfStateManual. > > > > > + - name: AfModeAuto > > > > > + value: 1 > > > > > + description: | > > > > > + The AF algorithm is in auto mode. This means that the > algorithm > > > > > + will never move the lens or change state unless the > AfTrigger > > > > > + control is used. The AfTrigger control can be used to > initiate a > > > > > + focus scan, the results of which will also be > reported by AfState. > > > > > + > > > > > + If the autofocus algorithm is moved from AfModeAuto > to another > > > > > + mode while a scan is in progress, the scan is > cancelled > > > > > + immediately, without waiting for the scan to finish. > > > > > + > > > > > + When first entering this mode the AfState will report > > > > > + AfStateAuto. When a trigger control is sent, AfState > will > > > > > + report AfStateAutoScanning for a period before > spontaneously > > > > > + changing to AfStateAutoFocused or AfStateAutoFailed, > depending on > > > > > + the outcome of the scan. It will remain in this state > until > > > > > + another scan is initiated by the AfTrigger control. > If a scan is > > > > > + cancelled (without changing to another mode), AfState > will return > > > > > + to AfStateAuto. > > > > > + - name: AfModeContinuous > > > > > + value: 2 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode. This means > that the lens > > > > > + can re-start a scan spontaneously at any moment, > without any user > > > > > + intervention. The AfState still reports whether the > algorithm is > > > > > + currently scanning or not, though the application has > no ability > > > > > + to initiate or cancel scans (though it can "pause" > them), nor to > > > > > + move the lens for itself. > > > > > + > > > > > + When set to AfModeContinuous, the system will > immediately initiate > > > > > + a scan so AfState will report > AfStateContinuousScanning, and will > > > > > + settle on one of AfStateContinuousFocused or > AfStateContinuousFailed, > > > > > + depending on the scan result. > > > > > + > > > > > + - AfLensPosition: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Control to set the position of the lens to one of the > following > > > > > + predefined locations. This control only has any effect > when the AfMode > > > > > + is set to AfModeManual. > > > > > + > > > > > + This control is distinct from the LensPosition control, > which sets the > > > > > + lens position using the lens driver's units. > > > > > + enum: > > > > > + - name: AfLensPositionMacro > > > > > + value: 0 > > > > > + description: The closest focal position of the lens. > > > > > + - name: AfLensPositionHyperfocal > > > > > + value: 1 > > > > > + description: Hyperfocal position. > > > > > + - name: AfLensPositionInfinity > > > > > + value: 2 > > > > > + description: The furthest focal position (usually > infinity). > > > > > > > > Interesting, bikeshedding on the name apart, is this some > 'simplified' > > > > LensState ? Do you think there is a use case for this ? Why would > > > > anyone set the AF to auto mode then use some coarse grained control > > > > like this one ? Wouldn't it make more sense as an 'hint' to the AF > > > > algorithm to restrict the search space ? > > > > > > In the light of your comments below about the LensPosition not using > > > driver units, then maybe this could be deleted, and we could select > > > some kind of canonical range for the lens? > > > > > > > Ah wait. What I was suggesting ("An "hint" to the AF algorithm to > > restrict the search space") is basically AfRange. > > I wonder if there's a case for a "custom" value for AfRange, and we > could then even have a control that sets the custom range? > > > > > > The most basic use case would be following on from the capture of an > > > image after an AF scan: > > > > > > - capture image > > > - AfMode to manual > > > - Lens to hyperfocal > > > - AfMode to auto > > > - now wait for the shutter button to be pressed before you start > > > another AF scan. > > > > Naive question: > > Wouldn't setting the lens to hyperfocal render the preview unusable ? > > Hyperfocal is the place where "most stuff" is normally in focus > (strictly speaking, has the greatest depth of field), so in the > absence of other information it's normally the best place to put the > lens. It's just for the preview, you'd typically run another scan for > a capture. > > (Some systems run Continuous AF during preview, of course. But you > know my anxiety about CAF algorithms... in my opinion just putting the > lens to hyperfocal is way less annoying than a not-very-good CAF > implementation!) > > > > > > > > > If your UI has a "macro" capture mode of some kind, maybe it's "Lens > > > to macro" instead of "Lens to hyperfocal". > > > > Why is this different than starting an AF scan (without going through > > Manual) with an AfRange that suggests in which range to scan ? Is this > > an optimization to forcefully move the lens closer to where it will > > likely end being moved to ? > > If you know you're doing a macro capture it would be helpful for the > preview to put the lens somewhere where macro things are likely to be > more in focus, instead of using hyperfocal which will likely make > close objects very blurry. > > > > > > > > > But as noted, if LensPosition allows us to select hyperfocal/macro > > > directly, then we can do without this one. > > > > > > > Providing my above understanding is correct, it would be a bit less > intuitive > > for applications to use a numerical value instead of some easier > > "Macro"/"Hyperfocal". Of course, wrappers like your one can simplify this > > for their consumers. > > True, though I'm now quite liking the "dioptre-like" canonical range > described below, where 1 = hyperfocal, 0 = infinity, large numbers = > close > > > > > > > > > > > (Btw I have a use case for supplying a range of lens positions, which > > > > has been previously estimated according to some pre-defined use > cases, > > > > to the AF algorithm in order restrict the AF search range to increase > > > > the speed of a scan. Do you think it's a valid use case and it's > worth a > > > > control ?) > > > > > > I'd imagined doing this kind of thing with the AfSpeed and AfRange > > > controls and by having the AF algorithm tuned for the module. I'm not > > > convinced about passing in lens positions from outside the camera > > > system. Where would they come from? Shouldn't the camera system know > > > what positions to use? > > > > > > > A possible use case, for controlled environments with low latency > > requirements, is to run tests beforehand and for each 'scene' define > > what is the ideal lens movement range. When the application starts an > > AF scan it would tell the AF algorithm about the 'preferred' range. > > > > I understand this works backwards, as the lens position should be the > > result of the AF algorithm run not one of its inputs :) > > Indeed, but as noted above, maybe there's an argument for an > AfRangeCustom which could be user-settable? We've added "Custom" > values for other controls in some places, IIRC. > > > > > > In the Raspberry Pi world we would of course store this kind of > > > information in the tuning file. I've noticed some discussion elsewhere > > > about supplying configuration to the IPAs - it seems like this might > > > be a case in point? I think that for real camera systems, in the end, > > > there is no escape from "configuration" (aka. "tuning"). > > > > > > > > > > > > + > > > > > + - AfRange: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Control to set the range of focus distances that is > scanned. An > > > > > + implementation may choose not to implement all the > options here. > > > > > > > > Only valid in Auto and Continuous mode, right ? > > > > > > Generally I've been expecting that you could change settings like this > > > in Manual mode too, though it obviously can't have any effect until > > > you go back to Auto/Continuous. Does that sound reasonable? > > > > I would rather assume thaat if a control is not valid in the current > > context is simply ignored and not cached to be re-applied when the > > context change ? > > Possibly, though I'm slightly nervous about how you'd do things like: > > - You're in Manual mode, but want to go to Auto and set the Range to > Normal. > - You're in Auto mode but were using a different range. You want to > reset it to Normal before switching to Manual. > > Can you send the AfRange change and the new mode at the same time in > both cases or not? Or do you need two separate requests to send the > changes? It's not clear to me exactly what would work. If you say that > the AfRange always gets updated, then the behaviour is obvious, and > you can send both AfRange and AfMode in the same request which would > be easier to use. > > > > > > > > > > > > > > (This makes me think that we could easily model Continuous by adding > a > > > > TriggerContinuous when in Auto mode. With a TriggerPause we could > get rid of > > > > AfPause... well, it's quite a redesign so I'll stop here unless > > > > someone thinks it's a really good idea :) > > > > > > Actually I'm quite strongly against anything like this. I think Auto > > > and Continuous should be quite separate. The reason is that Auto is > > > (relatively) easy to implement, you do a scan over some range and > > > choose the best spot. But Continuous AF is an absolute nightmare to do > > > well. You have all these problems: > > > > > > - The algo doesn't realise it needs to rescan and so seems to be > > > "stuck". (Watch people waving their hands in front of the lens to try > > > and unstick it.) So you add ever more tests and arbitrary thresholds > > > on different statistics to tell if a rescan is needed, and then the > > > whole thing becomes untunable. > > > - The algo rescans too often. A car in the distance goes by and the > > > whole image see-saws in and out while a rescan happens, ending back at > > > the same place. Yet more tests, thresholds... > > > - You can never make all your tests and thresholds work at the same > > > time but in different conditions. So you make them variable depending > > > on things like exposure and gain. Now you are truly doomed. > > > - The algo is generally not robust to camera shake, which frequently > > > causes it to end up "failing" with a completely out-of-focus image. > > > And then you can't get it to rescan... cue more hand-waving in front > > > of the lens. You implement more special tests and timeouts... At this > > > point you will wish for the earth to swallow you up. > > > - When the algo rescans it starts off going the wrong way causing > > > extra see-sawing. > > > - You try to solve the last problem by doing really small initial > > > steps. Only then it ends up statistically unstable. > > > > > > From the above you may detect that we have been there and have several > > > T-shirts. Speaking for myself, I have no plans to implement Continuous > > > AF without PDAF sensors, as PDAF is quite a big help. > > > > > > > Yes, it feels like you know what you're talking about :) > > > > I understand you would like be cautious with CAF, but if one doesn't > > want to implement CAF can't she simply not report TriggerContinuous ? > > > > Anyway, that was just an idea thrown in the mixer, don't worry about > > it. > > > > > > > > > > > + enum: > > > > > + - name: AfRangeNormal > > > > > + value: 0 > > > > > + description: | > > > > > + A wide range of focus distances is scanned, all the > way from > > > > > + infinity down to close distances, though depending on > the > > > > > + implementation, possibly not including the very > closest macro > > > > > + positions. > > > > > + - name: AfRangeMacro > > > > > + value: 1 > > > > > + description: Only close distances are scanned. > > > > > + - name: AfRangeFull > > > > > + value: 2 > > > > > + description: | > > > > > + The full range of focus distances is scanned just as > with > > > > > + AfRangeNormal but this time including the very > closest macro > > > > > + positions. > > > > > + > > > > > + - AfSpeed: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Control that determines whether the AF algorithm is to > move the lens > > > > > + as quickly as possible or more steadily. For example, > during video > > > > > + recording it may be desirable not to move the lens too > abruptly, but > > > > > + when in a preview mode (waiting for a still capture) it > may be > > > > > + helpful to move the lens as quickly as is reasonably > possible. > > > > > + enum: > > > > > + - name: AfSpeedNormal > > > > > + value: 0 > > > > > + description: Move the lens at its usual speed. > > > > > + - name: AfSpeedFast > > > > > + value: 1 > > > > > + description: Move the lens more quickly. > > > > > + > > > > > + - AfWindows: > > > > > + type: Rectangle > > > > > + draft: true > > > > > + description: | > > > > > + Sets the focus windows used by the AF algorithm. The > units used express > > > > > + a proportion of the ScalerCrop control (or if > unavailable, of the entire > > > > > + image), as u0.16 format numbers. > > > > > + > > > > > + In order to be activated, a rectangle must be programmed > with non-zero > > > > > + width and height. If no rectangles are programmed in this > way, then the > > > > > + system will choose its own single default window in the > centre of the > > > > > + image. > > > > > + > > > > > + The details of how the windows are used are platform > dependent. We note > > > > > + that when there is more than one AF window, a typical > implementation > > > > > + might find the optimal focus position for each one and > finally select > > > > > + the window closest to the camera. > > > > > + > > > > > + size: [platform dependent] > > > > > > > > I think this should just be > > > > > > > > size: [n] > > > > > > OK! > > > > > > > > + > > > > > + - AfTrigger: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + This control starts an autofocus scan when AfMode is set > to AfModeAuto, > > > > > + and can also be used to terminate a scan early. > > > > > + > > > > > + It is ignored if AfMode is set to AfModeManual or > AfModeContinuous. > > > > > + > > > > > + enum: > > > > > + - name: AfTriggerStart > > > > > + value: 0 > > > > > + description: Start an AF scan. Ignored if a scan is in > progress. > > > > > + - name: AfTriggerCancel > > > > > + value: 1 > > > > > + description: Cancel an AF scan. This does not cause the > lens to move > > > > > + anywhere else. Ignored if no scan is in progress. > > > > > + > > > > > + - AfPause: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + This control has no effect except when in continuous > autofocus mode > > > > > + (AfModeContinuous). It can be used to pause any lens > movements while > > > > > + (for example) images are captured. The algorithm remains > inactive > > > > > + until it is instructed to resume. > > > > > + > > > > > + enum: > > > > > + - name: AfPauseImmediate > > > > > + value: 0 > > > > > + description: | > > > > > + Pause the continuous autofocus algorithm immediately, > whether or > > > > > + not any kind of scan is underway. If the AfState was > previously > > > > > + reporting AfStateContinuousScanning it will now > change to > > > > > + AfStateContinuousScanningPaused, and similarly for > > > > > + AfStateContinuousFocused and AfStateContinuousFailed. > > > > > + - name AfPauseDeferred > > > > > + value: 1 > > > > > + description: | > > > > > + Pause the continuous autofocus algorithm as soon as > it is no longer > > > > > + scanning. If the AfState is currently reporting > > > > > + AfStateContinuousFocused is will change to > > > > > + AfStateContinuousFocusedPaused, and similarly in the > case of > > > > > + AfStateContinuousFailed. > > > > > + > > > > > + If AfState reports AfStateContinuousScanning it will > change to > > > > > + AfStateContinuousScanningPausing, and then move to > one of > > > > > + AfStateContinuousFocusedPaused or > AfStateContinuousFailedPaused > > > > > + when the scan completes. > > > > > + - name: AfPauseResume > > > > > + value: 2 > > > > > + description: | > > > > > + Resume continous autofocus operation. The algorithm > starts again > > > > > + from exactly where it left off, and the AfState will > drop the > > > > > + 'Paused' or 'Pausing' part of the state. > > > > > + > > > > > + - LensPosition: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Acts as a control to instruct the lens to move to a > particular position > > > > > + and also reports back the position of the lens for each > frame. > > > > > + > > > > > + The units are determined by the lens driver. > > > > > > > > Ugh, we cannot do this in the control definition. Driver > specificities > > > > cannot surface to the controls definition, which should be platform > > > > independent. > > > > > > > > I think we'll have to establish a numerical range (go ahead if you > > > > have any idea) and translate to the actual driver values in the > > > > CameraLens class ? > > > > > > I wasn't aware of this, but maybe it's a good thing. The question is > > > > Well, isn't it the same with "Brightness", for example ? > > Indeed, or ExposureTime where we explicitly want microseconds, not driver > units. > > > > > > what kind of range to use, and how do you know what limits apply for > > > any given camera module? How about a dioptre-like scale, normalised > > > for the hyperfocal distance H. It would go like this: > > > > > > The value n would mean being focused at distance H/n (n does not have > > > to be an integer). So: > > > > > > The value 1 would mean hyperfocal position (focal distance H/1). > > > The value 0 would mean infinity (focal distance H/0). > > > Larger values would get progressively closer. > > > > > > Setting a control value outside the allowed range will clamp it to > > > that end of the range. > > > > I sincerely don't know enough about this to tell if it's a good idea > > or not. Hence, it seems like a good idea to me :) > > > > > > > > In the Raspberry Pi world it would be sufficient for the tuning file > > > to know the allowable range of values. More generally, I could imagine > > > an application might want to say "focus at precisely 1m" in which case > > > it would have to know the value of H from a property or something, > > > though providing such information would be optional (some lenses > > > simply aren't precisely calibrated). Perhaps an application might want > > > to get the allowable range of (maybe approximate) focal distances too? > > > > > > > > > > > > + > > > > > + The LensPosition control is ignored unless the AfMode is > set to > > > > > + AfModeManual. > > > > > + > > > > > + Note that this control is distinct from AfLensPosition, > which allows > > > > > + the lens to be moved to its macro/hyperfocal/infinity > position, rather > > > > > + than using lens driver units. > > > > > + > > > > > + - AfState: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Reports the current state of the AF algorithm. The > possible state changes > > > > > + are described below, although we add that the following > general state > > > > > + changes are also allowed. > > > > > > > > wow, there is a lot of states here :) > > > > > > Did you follow the link to the state transition diagram? I would hope > > > that should go some way to explaining it! Here's the link again for > > > convenience: > > > > > > > https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing > > > > > > > Missed it. Thanks for providing it. > > > > > > > > > > Let me copy them here > > > > > > > > - name: AfStateManual > > > > - name: AfStateAuto > > > > - name: AfStateAutoScanning > > > > - name: AfStateAutoFocused > > > > - name: AfStateAutoFailed > > > > - name: AfStateContinuousScanning > > > > - name: AfStateContinuousScanningPausing > > > > - name: AfStateContinuousScanningPaused > > > > - name: AfStateContinuousFocused > > > > - name: AfStateContinuousFocusedPaused > > > > - name: AfStateContinuousFailed > > > > - name: AfStateContinuousFailedPaused > > > > > > I "multiplied out" all the different parts of the state to make a > > > single long list. Actually I think that makes it more Android-like. > > > But we could reduce this number if, for example, we're happy to report > > > the state of AfMode separately on every frame. I don't mind - do we > > > have a general preference for this kind of thing? > > > > > > > Good question. > > > > More generally we need to establish a policy for metadata. For > > controls we know that controls should be attached to a request only > > when their value changes. If a control is not supplied with new > > values, then the old (or default) one will stay valid. > > > > This implies the PH/IPA state machine is then stateful and 'remembers' > > what was the last value a control was set to. > > > > What about metadata instead ? Can we assume the same stateful-ness in > > applications too, or is it a burden we cannot impose on them ? > > > > Han-lin, do you know how does that work with Android ? Does it report > > metadata even when their values do not change or do they assume that, > > like controls, a metadata is reported only when its value has changed. > > > I think Android metadata doesn't have the assumption, and is supposed to return with each frame. In fact, with partial results supported, Android doesn't enforce a strict order of returning metadata among requests. It's more like each metadata is independent of the previous one. > What would the preference be for the RPi applications ? > > I don't think I have a very strong preference. I'm happy with either: > > - Report everything in the AfState on every frame. > - Report the state in multiple metadata items (say, AfState and > AfMode) on every frame. > > I'm less keen on having applications have to remember if they sent > something and then wait for the request where it actually happened, it > seems to me that would simply add annoying boilerplate code to all > applications. What do you think? > > > > > > > > > (please note that AfStateAuto seems mis-aligned, unless I messed up > > > > something while cutting&paste. Please check the compiled doc to see > > > > how it looks like...) > > > > > > I will check. > > > > > > > > > > > So we have three macro groups > > > > > > > > - Manual > > > > > > > > - Auto > > > > - Scanning > > > > - Focus > > > > - Failed > > > > > > > > - Continuous > > > > > > > > - Scanning > > > > - Pausing > > > > - Paused > > > > - Focused > > > > - Paused > > > > - Failed > > > > - Paused > > > > > > > > The first group is AfMode (Manual, Auto, Continuous) and can be > > > > deduced by inspecting the AfMode metadata. > > > > > > > > The second group is the actual state (Scannng, Focused, Failed) > > > > > > > > The third group is specific to continuous mode as it accepts a Pause > > > > trigger. > > > > > > > > I understand why you have this, as expressing "The AfState is set to > > > > $something when in AfModeManual" etc etc might get cumbersome, but I > > > > think it might be worth a try > > > > > > > > 1) Manual mode (the bad) > > > > As we all know there is a delay between when a lens is instructed > > > > to move and when it reaches its final state. > > > > > > > > In Manual mode the lens is moved with LensPosition and > > > > AfLensPosition. > > > > > > > > To know the current state: > > > > - With LensPosition metadata can be inspected until it doesn't > reach the > > > > desired value. > > > > - With AfLensPosition it is not clear to me how to know when the > lens > > > > as reched one of the pre-defined positions. > > > > > > It sounds like we're deleting AfLensPosition. > > > > > > > > > > > Is it so wrong to report either (Scanning,Focus,Failed) even for > > > > Manual mode ? > > > > > > I don't think it's wrong. If we have a single long list of states then > > > it just makes for more states. If we don't then it's harmless, though > > > I'm not aware of a use for it. > > > > > > > > > > > 2) AutoMode (the good) > > > > This seems the easy one, as (Scanning,Focus,Failed) are already > > > > there > > > > > > > > 3) Continous (the ugly) > > > > Continuous mode can be paused. That's a shame :) > > > > Would that be that bad keeping a Paused or Pausing state which can > > > > only be returned for Continuous mode. > > > > > > The whole paused/pausing thing rather complicates things, but it does > > > seem to me that it's necessary. If you were in Continuous mode, you > > > would rather the lens didn't set off again just as you want to capture > > > a sequence of pictures. > > > > > > > > > > > All in all, can we reduce the number of states to > > > > > > > > - Scanning > > > > > > > > The AF algorithm is scanning and has not yet reached a > > > > stable result. When running in Auto mode an AF scan is > > > > started by using the AfTrigger control. When in Manual mode > > > > and a new lens position is requested either by sending a > > > > LensPosition or AfLensPosition control, the AF state will > > > > report "Scanning" until the lens doesn't reach the desired > > > > position. Similarly, when running in continuous mode and > the > > > > AF algorithm decides to start a scan spontaneously the > > > > AfState will report Scanning until a new stable state is > > > > reached. > > > > > > > > - Focus > > > > > > > > The AF algorithm has reached a stable state and the image > is > > > > now focused. When running in Manual mode the lens has > > > > reached the position requested using either LensPosition > and > > > > AfLensPosition. > > > > > > > > - Failed > > > > > > > > The AF algorithm has completed a scan and without finding a > > > > good focus position. When running in manual mode, the > camera > > > > has failed moving the lens to the desired position. > > > > > > > > - Paused (copying your below text, please see comments below) > > > > > > > > The AF algorithm is in continuous mode (AfModeContinuous) > and > > > > was scanning when AfPauseImmediate was sent. The scan will > not > > > > complete and the algorithm will remain in this state. > > > > > > > > - Pausing > > > > > > > > The AF algorithm is in continuous mode (AfModeContinuous) > and > > > > was scanning when AfPauseDeferred was sent. The scan will > complete > > > > at which point the algorithm moves to the > > > > AfStateContinuousFocusedPaused or > AfStateContinuousFailedPaused > > > > state. > > > > > > I think we could get by if we report the following for every frame: > > > > > > AfMode - Manual, Auto or Continuous > > > AfState - Scanning, Focused or Failed > > > AfPause - Resume (which means it's running), Immediate or Deferred > > > (these states only apply during Continuous; the Deferred value can > > > only occur when also Scanning). > > > > > > Or we could wrap AfState and AfPause together: > > > > > > AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused, > > > FocusedPaused, FailedPaused. > > > > > > That makes some of the interaction between pausing and scanning > > > clearer, but then it's more states again. I don't know... > > > > > > > Let's clarify what the policy on metadata is first. > > Agree! > > > > > > > > > > > > > > > > + > > > > > + In any of the AfStateManual or AfStateContinuous states, > the AfMode > > > > > + may be set to AfModeAuto and the algorithm will move to > the > > > > > + AfStateAuto state. If AfTriggerStart is sent at the same > time > > > > > + then the algorithm will move to AfStateAutoScanning state. > > > > > + > > > > > + In any of the AfStateAuto or AfStateManual states, the > AfMode may > > > > > + be set to AfModeContinuous and the algorithm will move to > > > > > + AfStateContinuousScanning. > > > > > + > > > > > + In any of the AfStateAuto or AfStateContinuous states, > the AfMode may > > > > > + be set to AfModeManual and the algorithm will move to > > > > > + AfStateManual. The lens will not be moved and will be > left where > > > > > + it was at that moment. > > > > > + > > > > > + enum: > > > > > + - name: AfStateManual > > > > > + value: 0 > > > > > + description: | > > > > > + The AF algorithm is in manual mode (AfModeManual). > The LensPosition > > > > > + and AfLensPosition controls can be used directly to > move the lens. > > > > > + - name: AfStateAuto > > > > > + value: 1 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and > has either just > > > > > + been moved into that state, or a scan that was in > progress has been > > > > > + cancelled. > > > > > + - name: AfStateAutoScanning > > > > > + value: 2 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and a > scan has been > > > > > + started using the AfTrigger control. The scan can be > cancelled by > > > > > + sending AfTriggerCancel at which point the algorithm > will either > > > > > + move back to AfStateAuto or, if the scan actually > completes > > > > > + before the cancel request is processed, to one of > > > > > + AfStateAutoFocused or AfStateAutoFailed. > > > > > + - name: AfStateAutoFocused > > > > > + value: 3 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and a > scan has > > > > > + completed with the result that the algorithm believes > the image is > > > > > + now in focus. > > > > > + - name: AfStateAutoFailed > > > > > + value: 4 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and a > scan has > > > > > + completed with the result that the algorithm did not > find a good > > > > > + focus position. > > > > > + - name: AfStateContinuousScanning > > > > > + value: 5 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode > (AfModeContinuous) and > > > > > + is currently scanning for a good focus position. This > occurs when > > > > > + the mode is first set to AfModeContinuous, or may > happen > > > > > + spontaneously when the algorithm believes a re-scan > is needed. > > > > > + - name: AfStateContinuousScanningPausing > > > > > + value: 6 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode > (AfModeContinuous) and > > > > > + was scanning when AfPauseDeferred was sent. The scan > will complete > > > > > + at which point the algorithm moves to the > > > > > + AfStateContinuousFocusedPaused or > AfStateContinuousFailedPaused > > > > > + state. > > > > > + - name: AfStateContinuousScanningPaused > > > > > + value: 7 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode > (AfModeContinuous) and > > > > > + was scanning when AfPauseImmediate was sent. The scan > will not > > > > > > > > Why can't the algorithm be paused when it has reached a stable state > > > > but only when scanning ? Won't a pause 'freeze' the algorithm in its > > > > current state no matter what it is ? I understand with > > > > "AfPauseDeferred" that it has to wait for AF to be locked before > > > > stopping it, but with "Immediate" doesn't "pause" apply to all > states ? > > > > > > Wasn't sure I understood this. It can be paused once it's reached > > > Focused or Failed, and then both PauseImmediate and PauseDeferred will > > > cause it to report FocusedPaused or FailedPaused. Was that the > > > question? > > > > You're right, I have probably missed FocusedPaused and FailedPaused. > > :) > > > > Thanks > > j > > Thanks for all the discussion! > > David > > > > > > > > > Anyway, thanks very much! > > > > > > David > > > > > > > > > > > Thanks > > > > j > > > > > > > > > + complete and the algorithm will remain in this state. > The scan > > > > > + may be resumed by sending AfPauseResume. > > > > > + - name: AfStateContinuousFocused > > > > > + value: 8 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode > (AfModeContinuous) and > > > > > + a scan has completed with the algorithm believing it > has found a > > > > > + good focus position. > > > > > + - name: AfStateContinuousFocusedPaused > > > > > + value: 9 > > > > > + description: | > > > > > + The AF algorithm was in the AfStateContinuousFocused > state and > > > > > + has been paused (by either AfPauseImmediate or > AfPauseDeferred), > > > > > + or it was in the AfStateContinuousScanningPausing > state and the > > > > > + scan has completed successfully. The algorithm will > now remain > > > > > + in this state, and may be resumed by sending > AfPauseResume. > > > > > + - name: AfStateContinuousFailed > > > > > + value: 10 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode > (AfModeContinuous) and > > > > > + a scan has completed without finding a good focus > position. > > > > > + - name: AfStateContinuousFailedPaused > > > > > + value: 11 > > > > > + description: | > > > > > + The AF algorithm was in the AfStateContinuousFailed > state and > > > > > + has been paused (by either AfPauseImmediate or > AfPauseDeferred), > > > > > + or it was in the AfStateContinuousScanningPausing > state and the > > > > > + scan has completed unsuccessfully. The algorithm will > now remain > > > > > + in this state, and may be resumed by sending > AfPauseResume. > > > > > + > > > > > ... > > > > > -- > > > > > 2.30.2 > > > > > >
Hi David, On Thu, Apr 07, 2022 at 10:05:32AM +0100, David Plowman wrote: > Hi Jacopo > > Thanks for the reply! > > On Wed, 6 Apr 2022 at 17:34, Jacopo Mondi <jacopo@jmondi.org> wrote: > > > > Hi David, > > + Han-lin for an Android question on metadata > > > > On Wed, Apr 06, 2022 at 02:05:20PM +0100, David Plowman wrote: > > > Hi Jacopo > > > > > > Thanks for taking the time to read all this! > > > > > > On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <jacopo@jmondi.org> wrote: > > > > > > > > Hi David, > > > > sorry for being late :) > > > > > > > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote: > > > > > This patch describes a series of controls that allow applications to > > > > > drive AF algorithms: > > > > > > > > > > AfMode - manual, auto or continuous > > > > > AfLensPosition - set lens position to macro/hyperfocal/infinity > > > > > AfRange - full, macro or normal > > > > > AfSpeed - fast or slow > > > > > AfWindows - AF window locations > > > > > AfTrigger - start (trigger an AF scan) or cancel > > > > > AfPause - pause continuous AF > > > > > LensPosition - position of lens from lens driver > > > > > AfState - reports the mode, whether scanning/success/failure > > > > > --- > > > > > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > > > > > 1 file changed, 313 insertions(+), 60 deletions(-) > > > > > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > > > > > index 9d4638ae..23607368 100644 > > > > > --- a/src/libcamera/control_ids.yaml > > > > > +++ b/src/libcamera/control_ids.yaml > > > > > @@ -406,27 +406,6 @@ controls: > > > > > The camera will cancel any active or completed metering sequence. > > > > > The AE algorithm is reset to its initial state. > > > > > > > > > > - - AfTrigger: > > > > > - type: int32_t > > > > > - draft: true > > > > > - description: | > > > > > - Control for AF trigger. Currently identical to > > > > > - ANDROID_CONTROL_AF_TRIGGER. > > > > > - > > > > > - Whether the camera device will trigger autofocus for this request. > > > > > - enum: > > > > > - - name: AfTriggerIdle > > > > > - value: 0 > > > > > - description: The trigger is idle. > > > > > - - name: AfTriggerStart > > > > > - value: 1 > > > > > - description: The AF routine is started by the camera. > > > > > - - name: AfTriggerCancel > > > > > - value: 2 > > > > > - description: | > > > > > - The camera will cancel any active trigger and the AF routine is > > > > > - reset to its initial state. > > > > > - > > > > > - NoiseReductionMode: > > > > > type: int32_t > > > > > draft: true > > > > > @@ -507,45 +486,6 @@ controls: > > > > > The AE algorithm has started a pre-capture metering session. > > > > > \sa AePrecaptureTrigger > > > > > > > > > > - - AfState: > > > > > - type: int32_t > > > > > - draft: true > > > > > - description: | > > > > > - Control to report the current AF algorithm state. Currently identical to > > > > > - ANDROID_CONTROL_AF_STATE. > > > > > - > > > > > - Current state of the AF algorithm. > > > > > - enum: > > > > > - - name: AfStateInactive > > > > > - value: 0 > > > > > - description: The AF algorithm is inactive. > > > > > - - name: AfStatePassiveScan > > > > > - value: 1 > > > > > - description: | > > > > > - AF is performing a passive scan of the scene in continuous > > > > > - auto-focus mode. > > > > > - - name: AfStatePassiveFocused > > > > > - value: 2 > > > > > - description: | > > > > > - AF believes the scene is in focus, but might restart scanning. > > > > > - - name: AfStateActiveScan > > > > > - value: 3 > > > > > - description: | > > > > > - AF is performing a scan triggered by an AF trigger request. > > > > > - \sa AfTrigger > > > > > - - name: AfStateFocusedLock > > > > > - value: 4 > > > > > - description: | > > > > > - AF believes has focused correctly and has locked focus. > > > > > - - name: AfStateNotFocusedLock > > > > > - value: 5 > > > > > - description: | > > > > > - AF has not been able to focus and has locked. > > > > > - - name: AfStatePassiveUnfocused > > > > > - value: 6 > > > > > - description: | > > > > > - AF has completed a passive scan without finding focus. > > > > > - > > > > > - AwbState: > > > > > type: int32_t > > > > > draft: true > > > > > @@ -690,4 +630,317 @@ controls: > > > > > value. All of the custom test patterns will be static (that is the > > > > > raw image must not vary from frame to frame). > > > > > > > > > > + - AfMode: > > > > > + type: int32_t > > > > > + draft: true > > > > > > > > Was having all these controls as draft intentional ? The previous > > > > definitions where set to drafts as they mapped to the android > > > > definition, but now that we're defining our own ones... > > > > > > Not really... I only made them "draft" because it seemed like > > > everything was "draft" at first! But happy to change that. Are you > > > saying that these definitely should *not* be draft, then? > > > > > > > Yeah, "draft" was used to import controls from Android as they are for > > sake of CTS compliance. Let's drop draft from the ones we're defining > > Will do! > > > > > > > > > > > > + description: | > > > > > + Control to set the mode of the AF (autofocus) algorithm. Applications > > > > > + are allowed to set a new mode, and to send additional controls for > > > > > + that new mode, in the same request. > > > > > + > > > > > + An implementation may choose not to implement all the modes. > > > > > + > > > > > + enum: > > > > > + - name: AfModeManual > > > > > + value: 0 > > > > > + description: | > > > > > + The AF algorithm is in manual mode. In this mode it will never > > > > > + perform any action nor move the lens of its own accord, but an > > > > > + application can set controls to move the lens. > > > > > + > > > > > + In this mode the AfState will always report AfStateManual. > > > > > + - name: AfModeAuto > > > > > + value: 1 > > > > > + description: | > > > > > + The AF algorithm is in auto mode. This means that the algorithm > > > > > + will never move the lens or change state unless the AfTrigger > > > > > + control is used. The AfTrigger control can be used to initiate a > > > > > + focus scan, the results of which will also be reported by AfState. > > > > > + > > > > > + If the autofocus algorithm is moved from AfModeAuto to another > > > > > + mode while a scan is in progress, the scan is cancelled > > > > > + immediately, without waiting for the scan to finish. > > > > > + > > > > > + When first entering this mode the AfState will report > > > > > + AfStateAuto. When a trigger control is sent, AfState will > > > > > + report AfStateAutoScanning for a period before spontaneously > > > > > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > > > > > + the outcome of the scan. It will remain in this state until > > > > > + another scan is initiated by the AfTrigger control. If a scan is > > > > > + cancelled (without changing to another mode), AfState will return > > > > > + to AfStateAuto. > > > > > + - name: AfModeContinuous > > > > > + value: 2 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode. This means that the lens > > > > > + can re-start a scan spontaneously at any moment, without any user > > > > > + intervention. The AfState still reports whether the algorithm is > > > > > + currently scanning or not, though the application has no ability > > > > > + to initiate or cancel scans (though it can "pause" them), nor to > > > > > + move the lens for itself. > > > > > + > > > > > + When set to AfModeContinuous, the system will immediately initiate > > > > > + a scan so AfState will report AfStateContinuousScanning, and will > > > > > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > > > > > + depending on the scan result. > > > > > + > > > > > + - AfLensPosition: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Control to set the position of the lens to one of the following > > > > > + predefined locations. This control only has any effect when the AfMode > > > > > + is set to AfModeManual. > > > > > + > > > > > + This control is distinct from the LensPosition control, which sets the > > > > > + lens position using the lens driver's units. > > > > > + enum: > > > > > + - name: AfLensPositionMacro > > > > > + value: 0 > > > > > + description: The closest focal position of the lens. > > > > > + - name: AfLensPositionHyperfocal > > > > > + value: 1 > > > > > + description: Hyperfocal position. > > > > > + - name: AfLensPositionInfinity > > > > > + value: 2 > > > > > + description: The furthest focal position (usually infinity). > > > > > > > > Interesting, bikeshedding on the name apart, is this some 'simplified' > > > > LensState ? Do you think there is a use case for this ? Why would > > > > anyone set the AF to auto mode then use some coarse grained control > > > > like this one ? Wouldn't it make more sense as an 'hint' to the AF > > > > algorithm to restrict the search space ? > > > > > > In the light of your comments below about the LensPosition not using > > > driver units, then maybe this could be deleted, and we could select > > > some kind of canonical range for the lens? > > > > > > > Ah wait. What I was suggesting ("An "hint" to the AF algorithm to > > restrict the search space") is basically AfRange. > > I wonder if there's a case for a "custom" value for AfRange, and we > could then even have a control that sets the custom range? > Let's start with fixed positions then eventually extend it ? > > > > > The most basic use case would be following on from the capture of an > > > image after an AF scan: > > > > > > - capture image > > > - AfMode to manual > > > - Lens to hyperfocal > > > - AfMode to auto > > > - now wait for the shutter button to be pressed before you start > > > another AF scan. > > > > Naive question: > > Wouldn't setting the lens to hyperfocal render the preview unusable ? > > Hyperfocal is the place where "most stuff" is normally in focus > (strictly speaking, has the greatest depth of field), so in the > absence of other information it's normally the best place to put the > lens. It's just for the preview, you'd typically run another scan for > a capture. > I see, thank for the clrification > (Some systems run Continuous AF during preview, of course. But you > know my anxiety about CAF algorithms... in my opinion just putting the > lens to hyperfocal is way less annoying than a not-very-good CAF > implementation!) > > > > > > > > > If your UI has a "macro" capture mode of some kind, maybe it's "Lens > > > to macro" instead of "Lens to hyperfocal". > > > > Why is this different than starting an AF scan (without going through > > Manual) with an AfRange that suggests in which range to scan ? Is this > > an optimization to forcefully move the lens closer to where it will > > likely end being moved to ? > > If you know you're doing a macro capture it would be helpful for the > preview to put the lens somewhere where macro things are likely to be > more in focus, instead of using hyperfocal which will likely make > close objects very blurry. > > > > > > > > > But as noted, if LensPosition allows us to select hyperfocal/macro > > > directly, then we can do without this one. > > > > > > > Providing my above understanding is correct, it would be a bit less intuitive > > for applications to use a numerical value instead of some easier > > "Macro"/"Hyperfocal". Of course, wrappers like your one can simplify this > > for their consumers. > > True, though I'm now quite liking the "dioptre-like" canonical range > described below, where 1 = hyperfocal, 0 = infinity, large numbers = > close > > > > > > > > > > > (Btw I have a use case for supplying a range of lens positions, which > > > > has been previously estimated according to some pre-defined use cases, > > > > to the AF algorithm in order restrict the AF search range to increase > > > > the speed of a scan. Do you think it's a valid use case and it's worth a > > > > control ?) > > > > > > I'd imagined doing this kind of thing with the AfSpeed and AfRange > > > controls and by having the AF algorithm tuned for the module. I'm not > > > convinced about passing in lens positions from outside the camera > > > system. Where would they come from? Shouldn't the camera system know > > > what positions to use? > > > > > > > A possible use case, for controlled environments with low latency > > requirements, is to run tests beforehand and for each 'scene' define > > what is the ideal lens movement range. When the application starts an > > AF scan it would tell the AF algorithm about the 'preferred' range. > > > > I understand this works backwards, as the lens position should be the > > result of the AF algorithm run not one of its inputs :) > > Indeed, but as noted above, maybe there's an argument for an > AfRangeCustom which could be user-settable? We've added "Custom" > values for other controls in some places, IIRC. > > > > > > In the Raspberry Pi world we would of course store this kind of > > > information in the tuning file. I've noticed some discussion elsewhere > > > about supplying configuration to the IPAs - it seems like this might > > > be a case in point? I think that for real camera systems, in the end, > > > there is no escape from "configuration" (aka. "tuning"). > > > > > > > > > > > > + > > > > > + - AfRange: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Control to set the range of focus distances that is scanned. An > > > > > + implementation may choose not to implement all the options here. > > > > > > > > Only valid in Auto and Continuous mode, right ? > > > > > > Generally I've been expecting that you could change settings like this > > > in Manual mode too, though it obviously can't have any effect until > > > you go back to Auto/Continuous. Does that sound reasonable? > > > > I would rather assume thaat if a control is not valid in the current > > context is simply ignored and not cached to be re-applied when the > > context change ? > > Possibly, though I'm slightly nervous about how you'd do things like: > > - You're in Manual mode, but want to go to Auto and set the Range to Normal. > - You're in Auto mode but were using a different range. You want to > reset it to Normal before switching to Manual. > > Can you send the AfRange change and the new mode at the same time in > both cases or not? Or do you need two separate requests to send the > changes? It's not clear to me exactly what would work. If you say that > the AfRange always gets updated, then the behaviour is obvious, and > you can send both AfRange and AfMode in the same request which would > be easier to use. > I don't see why both controls cannot be specified together. Even if we allow AfRange to be specified in Manual mode and cache it to apply it when switching to Auto, AfRange will only be applied when AfMode==Auto is received, hence I don't see any optimization in specifying it when in Manual mode ? > > > > > > > > > > > > > (This makes me think that we could easily model Continuous by adding a > > > > TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of > > > > AfPause... well, it's quite a redesign so I'll stop here unless > > > > someone thinks it's a really good idea :) > > > > > > Actually I'm quite strongly against anything like this. I think Auto > > > and Continuous should be quite separate. The reason is that Auto is > > > (relatively) easy to implement, you do a scan over some range and > > > choose the best spot. But Continuous AF is an absolute nightmare to do > > > well. You have all these problems: > > > > > > - The algo doesn't realise it needs to rescan and so seems to be > > > "stuck". (Watch people waving their hands in front of the lens to try > > > and unstick it.) So you add ever more tests and arbitrary thresholds > > > on different statistics to tell if a rescan is needed, and then the > > > whole thing becomes untunable. > > > - The algo rescans too often. A car in the distance goes by and the > > > whole image see-saws in and out while a rescan happens, ending back at > > > the same place. Yet more tests, thresholds... > > > - You can never make all your tests and thresholds work at the same > > > time but in different conditions. So you make them variable depending > > > on things like exposure and gain. Now you are truly doomed. > > > - The algo is generally not robust to camera shake, which frequently > > > causes it to end up "failing" with a completely out-of-focus image. > > > And then you can't get it to rescan... cue more hand-waving in front > > > of the lens. You implement more special tests and timeouts... At this > > > point you will wish for the earth to swallow you up. > > > - When the algo rescans it starts off going the wrong way causing > > > extra see-sawing. > > > - You try to solve the last problem by doing really small initial > > > steps. Only then it ends up statistically unstable. > > > > > > From the above you may detect that we have been there and have several > > > T-shirts. Speaking for myself, I have no plans to implement Continuous > > > AF without PDAF sensors, as PDAF is quite a big help. > > > > > > > Yes, it feels like you know what you're talking about :) > > > > I understand you would like be cautious with CAF, but if one doesn't > > want to implement CAF can't she simply not report TriggerContinuous ? > > > > Anyway, that was just an idea thrown in the mixer, don't worry about > > it. > > > > > > > > > > > + enum: > > > > > + - name: AfRangeNormal > > > > > + value: 0 > > > > > + description: | > > > > > + A wide range of focus distances is scanned, all the way from > > > > > + infinity down to close distances, though depending on the > > > > > + implementation, possibly not including the very closest macro > > > > > + positions. > > > > > + - name: AfRangeMacro > > > > > + value: 1 > > > > > + description: Only close distances are scanned. > > > > > + - name: AfRangeFull > > > > > + value: 2 > > > > > + description: | > > > > > + The full range of focus distances is scanned just as with > > > > > + AfRangeNormal but this time including the very closest macro > > > > > + positions. > > > > > + > > > > > + - AfSpeed: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Control that determines whether the AF algorithm is to move the lens > > > > > + as quickly as possible or more steadily. For example, during video > > > > > + recording it may be desirable not to move the lens too abruptly, but > > > > > + when in a preview mode (waiting for a still capture) it may be > > > > > + helpful to move the lens as quickly as is reasonably possible. > > > > > + enum: > > > > > + - name: AfSpeedNormal > > > > > + value: 0 > > > > > + description: Move the lens at its usual speed. > > > > > + - name: AfSpeedFast > > > > > + value: 1 > > > > > + description: Move the lens more quickly. > > > > > + > > > > > + - AfWindows: > > > > > + type: Rectangle > > > > > + draft: true > > > > > + description: | > > > > > + Sets the focus windows used by the AF algorithm. The units used express > > > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire > > > > > + image), as u0.16 format numbers. > > > > > + > > > > > + In order to be activated, a rectangle must be programmed with non-zero > > > > > + width and height. If no rectangles are programmed in this way, then the > > > > > + system will choose its own single default window in the centre of the > > > > > + image. > > > > > + > > > > > + The details of how the windows are used are platform dependent. We note > > > > > + that when there is more than one AF window, a typical implementation > > > > > + might find the optimal focus position for each one and finally select > > > > > + the window closest to the camera. > > > > > + > > > > > + size: [platform dependent] > > > > > > > > I think this should just be > > > > > > > > size: [n] > > > > > > OK! > > > > > > > > + > > > > > + - AfTrigger: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > > > > > + and can also be used to terminate a scan early. > > > > > + > > > > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > > > > > + > > > > > + enum: > > > > > + - name: AfTriggerStart > > > > > + value: 0 > > > > > + description: Start an AF scan. Ignored if a scan is in progress. > > > > > + - name: AfTriggerCancel > > > > > + value: 1 > > > > > + description: Cancel an AF scan. This does not cause the lens to move > > > > > + anywhere else. Ignored if no scan is in progress. > > > > > + > > > > > + - AfPause: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + This control has no effect except when in continuous autofocus mode > > > > > + (AfModeContinuous). It can be used to pause any lens movements while > > > > > + (for example) images are captured. The algorithm remains inactive > > > > > + until it is instructed to resume. > > > > > + > > > > > + enum: > > > > > + - name: AfPauseImmediate > > > > > + value: 0 > > > > > + description: | > > > > > + Pause the continuous autofocus algorithm immediately, whether or > > > > > + not any kind of scan is underway. If the AfState was previously > > > > > + reporting AfStateContinuousScanning it will now change to > > > > > + AfStateContinuousScanningPaused, and similarly for > > > > > + AfStateContinuousFocused and AfStateContinuousFailed. > > > > > + - name AfPauseDeferred > > > > > + value: 1 > > > > > + description: | > > > > > + Pause the continuous autofocus algorithm as soon as it is no longer > > > > > + scanning. If the AfState is currently reporting > > > > > + AfStateContinuousFocused is will change to > > > > > + AfStateContinuousFocusedPaused, and similarly in the case of > > > > > + AfStateContinuousFailed. > > > > > + > > > > > + If AfState reports AfStateContinuousScanning it will change to > > > > > + AfStateContinuousScanningPausing, and then move to one of > > > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > > > + when the scan completes. > > > > > + - name: AfPauseResume > > > > > + value: 2 > > > > > + description: | > > > > > + Resume continous autofocus operation. The algorithm starts again > > > > > + from exactly where it left off, and the AfState will drop the > > > > > + 'Paused' or 'Pausing' part of the state. > > > > > + > > > > > + - LensPosition: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Acts as a control to instruct the lens to move to a particular position > > > > > + and also reports back the position of the lens for each frame. > > > > > + > > > > > + The units are determined by the lens driver. > > > > > > > > Ugh, we cannot do this in the control definition. Driver specificities > > > > cannot surface to the controls definition, which should be platform > > > > independent. > > > > > > > > I think we'll have to establish a numerical range (go ahead if you > > > > have any idea) and translate to the actual driver values in the > > > > CameraLens class ? > > > > > > I wasn't aware of this, but maybe it's a good thing. The question is > > > > Well, isn't it the same with "Brightness", for example ? > > Indeed, or ExposureTime where we explicitly want microseconds, not driver units. > > > > > > what kind of range to use, and how do you know what limits apply for > > > any given camera module? How about a dioptre-like scale, normalised > > > for the hyperfocal distance H. It would go like this: > > > > > > The value n would mean being focused at distance H/n (n does not have > > > to be an integer). So: > > > > > > The value 1 would mean hyperfocal position (focal distance H/1). > > > The value 0 would mean infinity (focal distance H/0). > > > Larger values would get progressively closer. > > > > > > Setting a control value outside the allowed range will clamp it to > > > that end of the range. > > > > I sincerely don't know enough about this to tell if it's a good idea > > or not. Hence, it seems like a good idea to me :) > > > > > > > > In the Raspberry Pi world it would be sufficient for the tuning file > > > to know the allowable range of values. More generally, I could imagine > > > an application might want to say "focus at precisely 1m" in which case > > > it would have to know the value of H from a property or something, > > > though providing such information would be optional (some lenses > > > simply aren't precisely calibrated). Perhaps an application might want > > > to get the allowable range of (maybe approximate) focal distances too? > > > So this is a lens construction parameter that would need to be recorded in the tuning file. In your experience, is this parameter easily available or can it be easily estimated ? > > > > > > > > > + > > > > > + The LensPosition control is ignored unless the AfMode is set to > > > > > + AfModeManual. > > > > > + > > > > > + Note that this control is distinct from AfLensPosition, which allows > > > > > + the lens to be moved to its macro/hyperfocal/infinity position, rather > > > > > + than using lens driver units. > > > > > + > > > > > + - AfState: > > > > > + type: int32_t > > > > > + draft: true > > > > > + description: | > > > > > + Reports the current state of the AF algorithm. The possible state changes > > > > > + are described below, although we add that the following general state > > > > > + changes are also allowed. > > > > > > > > wow, there is a lot of states here :) > > > > > > Did you follow the link to the state transition diagram? I would hope > > > that should go some way to explaining it! Here's the link again for > > > convenience: > > > > > > https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing > > > > > > > Missed it. Thanks for providing it. > > > > > > > > > > Let me copy them here > > > > > > > > - name: AfStateManual > > > > - name: AfStateAuto > > > > - name: AfStateAutoScanning > > > > - name: AfStateAutoFocused > > > > - name: AfStateAutoFailed > > > > - name: AfStateContinuousScanning > > > > - name: AfStateContinuousScanningPausing > > > > - name: AfStateContinuousScanningPaused > > > > - name: AfStateContinuousFocused > > > > - name: AfStateContinuousFocusedPaused > > > > - name: AfStateContinuousFailed > > > > - name: AfStateContinuousFailedPaused > > > > > > I "multiplied out" all the different parts of the state to make a > > > single long list. Actually I think that makes it more Android-like. > > > But we could reduce this number if, for example, we're happy to report > > > the state of AfMode separately on every frame. I don't mind - do we > > > have a general preference for this kind of thing? > > > > > > > Good question. > > > > More generally we need to establish a policy for metadata. For > > controls we know that controls should be attached to a request only > > when their value changes. If a control is not supplied with new > > values, then the old (or default) one will stay valid. > > > > This implies the PH/IPA state machine is then stateful and 'remembers' > > what was the last value a control was set to. > > > > What about metadata instead ? Can we assume the same stateful-ness in > > applications too, or is it a burden we cannot impose on them ? > > > > Han-lin, do you know how does that work with Android ? Does it report > > metadata even when their values do not change or do they assume that, > > like controls, a metadata is reported only when its value has changed. > > > > What would the preference be for the RPi applications ? > > I don't think I have a very strong preference. I'm happy with either: > > - Report everything in the AfState on every frame. > - Report the state in multiple metadata items (say, AfState and > AfMode) on every frame. According to Han-Lin reply, also Android seems to require to have metadata in each completed request, hence we could have AfMode reported separately and AfState simplified. The only drawback is that to reconstruct the actual state (mode + state) application have to inspect two metadata instead of one, but I think it's acceptable and simplifies things on the IPA side. > > I'm less keen on having applications have to remember if they sent > something and then wait for the request where it actually happened, it > seems to me that would simply add annoying boilerplate code to all > applications. What do you think? > Well, that's how it generally works. If you set your algorithm to auto or manual mode, you have to wait for a metadata that tells you when the mode has actually been changed, being it reported through a separate AfMode metadata or through a more expressive AfState which incorporates the mode. > > > > > > > > > > (please note that AfStateAuto seems mis-aligned, unless I messed up > > > > something while cutting&paste. Please check the compiled doc to see > > > > how it looks like...) > > > > > > I will check. > > > > > > > > > > > So we have three macro groups > > > > > > > > - Manual > > > > > > > > - Auto > > > > - Scanning > > > > - Focus > > > > - Failed > > > > > > > > - Continuous > > > > > > > > - Scanning > > > > - Pausing > > > > - Paused > > > > - Focused > > > > - Paused > > > > - Failed > > > > - Paused > > > > > > > > The first group is AfMode (Manual, Auto, Continuous) and can be > > > > deduced by inspecting the AfMode metadata. > > > > > > > > The second group is the actual state (Scannng, Focused, Failed) > > > > > > > > The third group is specific to continuous mode as it accepts a Pause > > > > trigger. > > > > > > > > I understand why you have this, as expressing "The AfState is set to > > > > $something when in AfModeManual" etc etc might get cumbersome, but I > > > > think it might be worth a try > > > > > > > > 1) Manual mode (the bad) > > > > As we all know there is a delay between when a lens is instructed > > > > to move and when it reaches its final state. > > > > > > > > In Manual mode the lens is moved with LensPosition and > > > > AfLensPosition. > > > > > > > > To know the current state: > > > > - With LensPosition metadata can be inspected until it doesn't reach the > > > > desired value. > > > > - With AfLensPosition it is not clear to me how to know when the lens > > > > as reched one of the pre-defined positions. > > > > > > It sounds like we're deleting AfLensPosition. > > > > > > > > > > > Is it so wrong to report either (Scanning,Focus,Failed) even for > > > > Manual mode ? > > > > > > I don't think it's wrong. If we have a single long list of states then > > > it just makes for more states. If we don't then it's harmless, though > > > I'm not aware of a use for it. > > > > > > > > > > > 2) AutoMode (the good) > > > > This seems the easy one, as (Scanning,Focus,Failed) are already > > > > there > > > > > > > > 3) Continous (the ugly) > > > > Continuous mode can be paused. That's a shame :) > > > > Would that be that bad keeping a Paused or Pausing state which can > > > > only be returned for Continuous mode. > > > > > > The whole paused/pausing thing rather complicates things, but it does > > > seem to me that it's necessary. If you were in Continuous mode, you > > > would rather the lens didn't set off again just as you want to capture > > > a sequence of pictures. > > > > > > > > > > > All in all, can we reduce the number of states to > > > > > > > > - Scanning > > > > > > > > The AF algorithm is scanning and has not yet reached a > > > > stable result. When running in Auto mode an AF scan is > > > > started by using the AfTrigger control. When in Manual mode > > > > and a new lens position is requested either by sending a > > > > LensPosition or AfLensPosition control, the AF state will > > > > report "Scanning" until the lens doesn't reach the desired > > > > position. Similarly, when running in continuous mode and the > > > > AF algorithm decides to start a scan spontaneously the > > > > AfState will report Scanning until a new stable state is > > > > reached. > > > > > > > > - Focus > > > > > > > > The AF algorithm has reached a stable state and the image is > > > > now focused. When running in Manual mode the lens has > > > > reached the position requested using either LensPosition and > > > > AfLensPosition. > > > > > > > > - Failed > > > > > > > > The AF algorithm has completed a scan and without finding a > > > > good focus position. When running in manual mode, the camera > > > > has failed moving the lens to the desired position. > > > > > > > > - Paused (copying your below text, please see comments below) > > > > > > > > The AF algorithm is in continuous mode (AfModeContinuous) and > > > > was scanning when AfPauseImmediate was sent. The scan will not > > > > complete and the algorithm will remain in this state. > > > > > > > > - Pausing > > > > > > > > The AF algorithm is in continuous mode (AfModeContinuous) and > > > > was scanning when AfPauseDeferred was sent. The scan will complete > > > > at which point the algorithm moves to the > > > > AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > > state. > > > > > > I think we could get by if we report the following for every frame: > > > > > > AfMode - Manual, Auto or Continuous > > > AfState - Scanning, Focused or Failed > > > AfPause - Resume (which means it's running), Immediate or Deferred > > > (these states only apply during Continuous; the Deferred value can > > > only occur when also Scanning). > > > > > > Or we could wrap AfState and AfPause together: > > > > > > AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused, > > > FocusedPaused, FailedPaused. > > > > > > That makes some of the interaction between pausing and scanning > > > clearer, but then it's more states again. I don't know... > > > > > > > Let's clarify what the policy on metadata is first. > > Agree! My preference would be to have three metadata, although it would require a bit more work on the application side and maybe also on the IPA side as correctness of the metadata combinations should be guaranteed (ie AfPause::Deferred can only be returned when AfState::Scanning according to your above comment). But it would simplify the states greatly and it's worth it in my opinion. It's a gut feeling, so if anyone feels there are reasons not to go this way feel free to express it. Thanks j > > > > > > > > > > > > > > > > + > > > > > + In any of the AfStateManual or AfStateContinuous states, the AfMode > > > > > + may be set to AfModeAuto and the algorithm will move to the > > > > > + AfStateAuto state. If AfTriggerStart is sent at the same time > > > > > + then the algorithm will move to AfStateAutoScanning state. > > > > > + > > > > > + In any of the AfStateAuto or AfStateManual states, the AfMode may > > > > > + be set to AfModeContinuous and the algorithm will move to > > > > > + AfStateContinuousScanning. > > > > > + > > > > > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > > > > > + be set to AfModeManual and the algorithm will move to > > > > > + AfStateManual. The lens will not be moved and will be left where > > > > > + it was at that moment. > > > > > + > > > > > + enum: > > > > > + - name: AfStateManual > > > > > + value: 0 > > > > > + description: | > > > > > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > > > > > + and AfLensPosition controls can be used directly to move the lens. > > > > > + - name: AfStateAuto > > > > > + value: 1 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and has either just > > > > > + been moved into that state, or a scan that was in progress has been > > > > > + cancelled. > > > > > + - name: AfStateAutoScanning > > > > > + value: 2 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > > > > > + started using the AfTrigger control. The scan can be cancelled by > > > > > + sending AfTriggerCancel at which point the algorithm will either > > > > > + move back to AfStateAuto or, if the scan actually completes > > > > > + before the cancel request is processed, to one of > > > > > + AfStateAutoFocused or AfStateAutoFailed. > > > > > + - name: AfStateAutoFocused > > > > > + value: 3 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > > > > + completed with the result that the algorithm believes the image is > > > > > + now in focus. > > > > > + - name: AfStateAutoFailed > > > > > + value: 4 > > > > > + description: | > > > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > > > > + completed with the result that the algorithm did not find a good > > > > > + focus position. > > > > > + - name: AfStateContinuousScanning > > > > > + value: 5 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > > + is currently scanning for a good focus position. This occurs when > > > > > + the mode is first set to AfModeContinuous, or may happen > > > > > + spontaneously when the algorithm believes a re-scan is needed. > > > > > + - name: AfStateContinuousScanningPausing > > > > > + value: 6 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > > + was scanning when AfPauseDeferred was sent. The scan will complete > > > > > + at which point the algorithm moves to the > > > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > > > > + state. > > > > > + - name: AfStateContinuousScanningPaused > > > > > + value: 7 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > > + was scanning when AfPauseImmediate was sent. The scan will not > > > > > > > > Why can't the algorithm be paused when it has reached a stable state > > > > but only when scanning ? Won't a pause 'freeze' the algorithm in its > > > > current state no matter what it is ? I understand with > > > > "AfPauseDeferred" that it has to wait for AF to be locked before > > > > stopping it, but with "Immediate" doesn't "pause" apply to all states ? > > > > > > Wasn't sure I understood this. It can be paused once it's reached > > > Focused or Failed, and then both PauseImmediate and PauseDeferred will > > > cause it to report FocusedPaused or FailedPaused. Was that the > > > question? > > > > You're right, I have probably missed FocusedPaused and FailedPaused. > > :) > > > > Thanks > > j > > Thanks for all the discussion! > > David > > > > > > > > > Anyway, thanks very much! > > > > > > David > > > > > > > > > > > Thanks > > > > j > > > > > > > > > + complete and the algorithm will remain in this state. The scan > > > > > + may be resumed by sending AfPauseResume. > > > > > + - name: AfStateContinuousFocused > > > > > + value: 8 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > > + a scan has completed with the algorithm believing it has found a > > > > > + good focus position. > > > > > + - name: AfStateContinuousFocusedPaused > > > > > + value: 9 > > > > > + description: | > > > > > + The AF algorithm was in the AfStateContinuousFocused state and > > > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > > > > + or it was in the AfStateContinuousScanningPausing state and the > > > > > + scan has completed successfully. The algorithm will now remain > > > > > + in this state, and may be resumed by sending AfPauseResume. > > > > > + - name: AfStateContinuousFailed > > > > > + value: 10 > > > > > + description: | > > > > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > > > > + a scan has completed without finding a good focus position. > > > > > + - name: AfStateContinuousFailedPaused > > > > > + value: 11 > > > > > + description: | > > > > > + The AF algorithm was in the AfStateContinuousFailed state and > > > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > > > > + or it was in the AfStateContinuousScanningPausing state and the > > > > > + scan has completed unsuccessfully. The algorithm will now remain > > > > > + in this state, and may be resumed by sending AfPauseResume. > > > > > + > > > > > ... > > > > > -- > > > > > 2.30.2 > > > > >
Hello David, Now that I'm starting to play a bit with it, I see a few things we might want to improve. On 31/03/2022 17:17, David Plowman wrote: > This patch describes a series of controls that allow applications to > drive AF algorithms: > > AfMode - manual, auto or continuous > AfLensPosition - set lens position to macro/hyperfocal/infinity > AfRange - full, macro or normal > AfSpeed - fast or slow > AfWindows - AF window locations > AfTrigger - start (trigger an AF scan) or cancel > AfPause - pause continuous AF > LensPosition - position of lens from lens driver > AfState - reports the mode, whether scanning/success/failure > --- > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > 1 file changed, 313 insertions(+), 60 deletions(-) > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > index 9d4638ae..23607368 100644 > --- a/src/libcamera/control_ids.yaml > +++ b/src/libcamera/control_ids.yaml > @@ -406,27 +406,6 @@ controls: > The camera will cancel any active or completed metering sequence. > The AE algorithm is reset to its initial state. > > - - AfTrigger: > - type: int32_t > - draft: true > - description: | > - Control for AF trigger. Currently identical to > - ANDROID_CONTROL_AF_TRIGGER. > - > - Whether the camera device will trigger autofocus for this request. > - enum: > - - name: AfTriggerIdle > - value: 0 > - description: The trigger is idle. > - - name: AfTriggerStart > - value: 1 > - description: The AF routine is started by the camera. > - - name: AfTriggerCancel > - value: 2 > - description: | > - The camera will cancel any active trigger and the AF routine is > - reset to its initial state. > - > - NoiseReductionMode: > type: int32_t > draft: true > @@ -507,45 +486,6 @@ controls: > The AE algorithm has started a pre-capture metering session. > \sa AePrecaptureTrigger > > - - AfState: > - type: int32_t > - draft: true > - description: | > - Control to report the current AF algorithm state. Currently identical to > - ANDROID_CONTROL_AF_STATE. > - > - Current state of the AF algorithm. > - enum: > - - name: AfStateInactive > - value: 0 > - description: The AF algorithm is inactive. > - - name: AfStatePassiveScan > - value: 1 > - description: | > - AF is performing a passive scan of the scene in continuous > - auto-focus mode. > - - name: AfStatePassiveFocused > - value: 2 > - description: | > - AF believes the scene is in focus, but might restart scanning. > - - name: AfStateActiveScan > - value: 3 > - description: | > - AF is performing a scan triggered by an AF trigger request. > - \sa AfTrigger > - - name: AfStateFocusedLock > - value: 4 > - description: | > - AF believes has focused correctly and has locked focus. > - - name: AfStateNotFocusedLock > - value: 5 > - description: | > - AF has not been able to focus and has locked. > - - name: AfStatePassiveUnfocused > - value: 6 > - description: | > - AF has completed a passive scan without finding focus. > - > - AwbState: > type: int32_t > draft: true > @@ -690,4 +630,317 @@ controls: > value. All of the custom test patterns will be static (that is the > raw image must not vary from frame to frame). > > + - AfMode: > + type: int32_t > + draft: true > + description: | > + Control to set the mode of the AF (autofocus) algorithm. Applications > + are allowed to set a new mode, and to send additional controls for > + that new mode, in the same request. > + > + An implementation may choose not to implement all the modes. > + > + enum: > + - name: AfModeManual > + value: 0 > + description: | > + The AF algorithm is in manual mode. In this mode it will never > + perform any action nor move the lens of its own accord, but an > + application can set controls to move the lens. > + > + In this mode the AfState will always report AfStateManual. > + - name: AfModeAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode. This means that the algorithm > + will never move the lens or change state unless the AfTrigger > + control is used. The AfTrigger control can be used to initiate a > + focus scan, the results of which will also be reported by AfState. > + > + If the autofocus algorithm is moved from AfModeAuto to another > + mode while a scan is in progress, the scan is cancelled > + immediately, without waiting for the scan to finish. > + > + When first entering this mode the AfState will report > + AfStateAuto. When a trigger control is sent, AfState will > + report AfStateAutoScanning for a period before spontaneously > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > + the outcome of the scan. It will remain in this state until > + another scan is initiated by the AfTrigger control. If a scan is > + cancelled (without changing to another mode), AfState will return > + to AfStateAuto. > + - name: AfModeContinuous > + value: 2 > + description: | > + The AF algorithm is in continuous mode. This means that the lens > + can re-start a scan spontaneously at any moment, without any user > + intervention. The AfState still reports whether the algorithm is > + currently scanning or not, though the application has no ability > + to initiate or cancel scans (though it can "pause" them), nor to > + move the lens for itself. > + > + When set to AfModeContinuous, the system will immediately initiate > + a scan so AfState will report AfStateContinuousScanning, and will > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > + depending on the scan result. > + > + - AfLensPosition: > + type: int32_t > + draft: true > + description: | > + Control to set the position of the lens to one of the following > + predefined locations. This control only has any effect when the AfMode > + is set to AfModeManual. > + > + This control is distinct from the LensPosition control, which sets the > + lens position using the lens driver's units. > + enum: > + - name: AfLensPositionMacro > + value: 0 > + description: The closest focal position of the lens. > + - name: AfLensPositionHyperfocal > + value: 1 > + description: Hyperfocal position. > + - name: AfLensPositionInfinity > + value: 2 > + description: The furthest focal position (usually infinity). > + > + - AfRange: > + type: int32_t > + draft: true > + description: | > + Control to set the range of focus distances that is scanned. An > + implementation may choose not to implement all the options here. > + enum: > + - name: AfRangeNormal > + value: 0 > + description: | > + A wide range of focus distances is scanned, all the way from > + infinity down to close distances, though depending on the > + implementation, possibly not including the very closest macro > + positions. > + - name: AfRangeMacro > + value: 1 > + description: Only close distances are scanned. > + - name: AfRangeFull > + value: 2 > + description: | > + The full range of focus distances is scanned just as with > + AfRangeNormal but this time including the very closest macro > + positions. > + > + - AfSpeed: > + type: int32_t > + draft: true > + description: | > + Control that determines whether the AF algorithm is to move the lens > + as quickly as possible or more steadily. For example, during video > + recording it may be desirable not to move the lens too abruptly, but > + when in a preview mode (waiting for a still capture) it may be > + helpful to move the lens as quickly as is reasonably possible. > + enum: > + - name: AfSpeedNormal > + value: 0 > + description: Move the lens at its usual speed. > + - name: AfSpeedFast > + value: 1 > + description: Move the lens more quickly. > + > + - AfWindows: > + type: Rectangle > + draft: true > + description: | > + Sets the focus windows used by the AF algorithm. The units used express > + a proportion of the ScalerCrop control (or if unavailable, of the entire > + image), as u0.16 format numbers. I don't mind using ScalerCrop, but I think it would probably make more sense to use the full pixel array ? > + > + In order to be activated, a rectangle must be programmed with non-zero > + width and height. If no rectangles are programmed in this way, then the > + system will choose its own single default window in the centre of the > + image. > + > + The details of how the windows are used are platform dependent. We note > + that when there is more than one AF window, a typical implementation > + might find the optimal focus position for each one and finally select > + the window closest to the camera. > + > + size: [platform dependent] And I would like to point you to this one (you probably already now it): https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas() I really like the number of areas variable, and even more the weight per area. It makes sense, you might want to have more weight in the center, but still have a larger region. I don't get why the corrdinates are between [-1000, 1000] though but... not that important. What do you think ? JM > + > + - AfTrigger: > + type: int32_t > + draft: true > + description: | > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > + and can also be used to terminate a scan early. > + > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > + > + enum: > + - name: AfTriggerStart > + value: 0 > + description: Start an AF scan. Ignored if a scan is in progress. > + - name: AfTriggerCancel > + value: 1 > + description: Cancel an AF scan. This does not cause the lens to move > + anywhere else. Ignored if no scan is in progress. > + > + - AfPause: > + type: int32_t > + draft: true > + description: | > + This control has no effect except when in continuous autofocus mode > + (AfModeContinuous). It can be used to pause any lens movements while > + (for example) images are captured. The algorithm remains inactive > + until it is instructed to resume. > + > + enum: > + - name: AfPauseImmediate > + value: 0 > + description: | > + Pause the continuous autofocus algorithm immediately, whether or > + not any kind of scan is underway. If the AfState was previously > + reporting AfStateContinuousScanning it will now change to > + AfStateContinuousScanningPaused, and similarly for > + AfStateContinuousFocused and AfStateContinuousFailed. > + - name AfPauseDeferred > + value: 1 > + description: | > + Pause the continuous autofocus algorithm as soon as it is no longer > + scanning. If the AfState is currently reporting > + AfStateContinuousFocused is will change to > + AfStateContinuousFocusedPaused, and similarly in the case of > + AfStateContinuousFailed. > + > + If AfState reports AfStateContinuousScanning it will change to > + AfStateContinuousScanningPausing, and then move to one of > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > + when the scan completes. > + - name: AfPauseResume > + value: 2 > + description: | > + Resume continous autofocus operation. The algorithm starts again > + from exactly where it left off, and the AfState will drop the > + 'Paused' or 'Pausing' part of the state. > + > + - LensPosition: > + type: int32_t > + draft: true > + description: | > + Acts as a control to instruct the lens to move to a particular position > + and also reports back the position of the lens for each frame. > + > + The units are determined by the lens driver. > + > + The LensPosition control is ignored unless the AfMode is set to > + AfModeManual. > + > + Note that this control is distinct from AfLensPosition, which allows > + the lens to be moved to its macro/hyperfocal/infinity position, rather > + than using lens driver units. > + > + - AfState: > + type: int32_t > + draft: true > + description: | > + Reports the current state of the AF algorithm. The possible state changes > + are described below, although we add that the following general state > + changes are also allowed. > + > + In any of the AfStateManual or AfStateContinuous states, the AfMode > + may be set to AfModeAuto and the algorithm will move to the > + AfStateAuto state. If AfTriggerStart is sent at the same time > + then the algorithm will move to AfStateAutoScanning state. > + > + In any of the AfStateAuto or AfStateManual states, the AfMode may > + be set to AfModeContinuous and the algorithm will move to > + AfStateContinuousScanning. > + > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > + be set to AfModeManual and the algorithm will move to > + AfStateManual. The lens will not be moved and will be left where > + it was at that moment. > + > + enum: > + - name: AfStateManual > + value: 0 > + description: | > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > + and AfLensPosition controls can be used directly to move the lens. > + - name: AfStateAuto > + value: 1 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and has either just > + been moved into that state, or a scan that was in progress has been > + cancelled. > + - name: AfStateAutoScanning > + value: 2 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > + started using the AfTrigger control. The scan can be cancelled by > + sending AfTriggerCancel at which point the algorithm will either > + move back to AfStateAuto or, if the scan actually completes > + before the cancel request is processed, to one of > + AfStateAutoFocused or AfStateAutoFailed. > + - name: AfStateAutoFocused > + value: 3 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm believes the image is > + now in focus. > + - name: AfStateAutoFailed > + value: 4 > + description: | > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > + completed with the result that the algorithm did not find a good > + focus position. > + - name: AfStateContinuousScanning > + value: 5 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + is currently scanning for a good focus position. This occurs when > + the mode is first set to AfModeContinuous, or may happen > + spontaneously when the algorithm believes a re-scan is needed. > + - name: AfStateContinuousScanningPausing > + value: 6 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseDeferred was sent. The scan will complete > + at which point the algorithm moves to the > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > + state. > + - name: AfStateContinuousScanningPaused > + value: 7 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + was scanning when AfPauseImmediate was sent. The scan will not > + complete and the algorithm will remain in this state. The scan > + may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFocused > + value: 8 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed with the algorithm believing it has found a > + good focus position. > + - name: AfStateContinuousFocusedPaused > + value: 9 > + description: | > + The AF algorithm was in the AfStateContinuousFocused state and > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and the > + scan has completed successfully. The algorithm will now remain > + in this state, and may be resumed by sending AfPauseResume. > + - name: AfStateContinuousFailed > + value: 10 > + description: | > + The AF algorithm is in continuous mode (AfModeContinuous) and > + a scan has completed without finding a good focus position. > + - name: AfStateContinuousFailedPaused > + value: 11 > + description: | > + The AF algorithm was in the AfStateContinuousFailed state and > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > + or it was in the AfStateContinuousScanningPausing state and the > + scan has completed unsuccessfully. The algorithm will now remain > + in this state, and may be resumed by sending AfPauseResume. > + > ...
Hi Jean-Michel Thanks for raising these points, I think there are some things to figure out here! On Fri, 15 Apr 2022 at 09:22, Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com> wrote: > > Hello David, > > Now that I'm starting to play a bit with it, I see a few things we might > want to improve. > > On 31/03/2022 17:17, David Plowman wrote: > > This patch describes a series of controls that allow applications to > > drive AF algorithms: > > > > AfMode - manual, auto or continuous > > AfLensPosition - set lens position to macro/hyperfocal/infinity > > AfRange - full, macro or normal > > AfSpeed - fast or slow > > AfWindows - AF window locations > > AfTrigger - start (trigger an AF scan) or cancel > > AfPause - pause continuous AF > > LensPosition - position of lens from lens driver > > AfState - reports the mode, whether scanning/success/failure > > --- > > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------ > > 1 file changed, 313 insertions(+), 60 deletions(-) > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml > > index 9d4638ae..23607368 100644 > > --- a/src/libcamera/control_ids.yaml > > +++ b/src/libcamera/control_ids.yaml > > @@ -406,27 +406,6 @@ controls: > > The camera will cancel any active or completed metering sequence. > > The AE algorithm is reset to its initial state. > > > > - - AfTrigger: > > - type: int32_t > > - draft: true > > - description: | > > - Control for AF trigger. Currently identical to > > - ANDROID_CONTROL_AF_TRIGGER. > > - > > - Whether the camera device will trigger autofocus for this request. > > - enum: > > - - name: AfTriggerIdle > > - value: 0 > > - description: The trigger is idle. > > - - name: AfTriggerStart > > - value: 1 > > - description: The AF routine is started by the camera. > > - - name: AfTriggerCancel > > - value: 2 > > - description: | > > - The camera will cancel any active trigger and the AF routine is > > - reset to its initial state. > > - > > - NoiseReductionMode: > > type: int32_t > > draft: true > > @@ -507,45 +486,6 @@ controls: > > The AE algorithm has started a pre-capture metering session. > > \sa AePrecaptureTrigger > > > > - - AfState: > > - type: int32_t > > - draft: true > > - description: | > > - Control to report the current AF algorithm state. Currently identical to > > - ANDROID_CONTROL_AF_STATE. > > - > > - Current state of the AF algorithm. > > - enum: > > - - name: AfStateInactive > > - value: 0 > > - description: The AF algorithm is inactive. > > - - name: AfStatePassiveScan > > - value: 1 > > - description: | > > - AF is performing a passive scan of the scene in continuous > > - auto-focus mode. > > - - name: AfStatePassiveFocused > > - value: 2 > > - description: | > > - AF believes the scene is in focus, but might restart scanning. > > - - name: AfStateActiveScan > > - value: 3 > > - description: | > > - AF is performing a scan triggered by an AF trigger request. > > - \sa AfTrigger > > - - name: AfStateFocusedLock > > - value: 4 > > - description: | > > - AF believes has focused correctly and has locked focus. > > - - name: AfStateNotFocusedLock > > - value: 5 > > - description: | > > - AF has not been able to focus and has locked. > > - - name: AfStatePassiveUnfocused > > - value: 6 > > - description: | > > - AF has completed a passive scan without finding focus. > > - > > - AwbState: > > type: int32_t > > draft: true > > @@ -690,4 +630,317 @@ controls: > > value. All of the custom test patterns will be static (that is the > > raw image must not vary from frame to frame). > > > > + - AfMode: > > + type: int32_t > > + draft: true > > + description: | > > + Control to set the mode of the AF (autofocus) algorithm. Applications > > + are allowed to set a new mode, and to send additional controls for > > + that new mode, in the same request. > > + > > + An implementation may choose not to implement all the modes. > > + > > + enum: > > + - name: AfModeManual > > + value: 0 > > + description: | > > + The AF algorithm is in manual mode. In this mode it will never > > + perform any action nor move the lens of its own accord, but an > > + application can set controls to move the lens. > > + > > + In this mode the AfState will always report AfStateManual. > > + - name: AfModeAuto > > + value: 1 > > + description: | > > + The AF algorithm is in auto mode. This means that the algorithm > > + will never move the lens or change state unless the AfTrigger > > + control is used. The AfTrigger control can be used to initiate a > > + focus scan, the results of which will also be reported by AfState. > > + > > + If the autofocus algorithm is moved from AfModeAuto to another > > + mode while a scan is in progress, the scan is cancelled > > + immediately, without waiting for the scan to finish. > > + > > + When first entering this mode the AfState will report > > + AfStateAuto. When a trigger control is sent, AfState will > > + report AfStateAutoScanning for a period before spontaneously > > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on > > + the outcome of the scan. It will remain in this state until > > + another scan is initiated by the AfTrigger control. If a scan is > > + cancelled (without changing to another mode), AfState will return > > + to AfStateAuto. > > + - name: AfModeContinuous > > + value: 2 > > + description: | > > + The AF algorithm is in continuous mode. This means that the lens > > + can re-start a scan spontaneously at any moment, without any user > > + intervention. The AfState still reports whether the algorithm is > > + currently scanning or not, though the application has no ability > > + to initiate or cancel scans (though it can "pause" them), nor to > > + move the lens for itself. > > + > > + When set to AfModeContinuous, the system will immediately initiate > > + a scan so AfState will report AfStateContinuousScanning, and will > > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, > > + depending on the scan result. > > + > > + - AfLensPosition: > > + type: int32_t > > + draft: true > > + description: | > > + Control to set the position of the lens to one of the following > > + predefined locations. This control only has any effect when the AfMode > > + is set to AfModeManual. > > + > > + This control is distinct from the LensPosition control, which sets the > > + lens position using the lens driver's units. > > + enum: > > + - name: AfLensPositionMacro > > + value: 0 > > + description: The closest focal position of the lens. > > + - name: AfLensPositionHyperfocal > > + value: 1 > > + description: Hyperfocal position. > > + - name: AfLensPositionInfinity > > + value: 2 > > + description: The furthest focal position (usually infinity). > > + > > + - AfRange: > > + type: int32_t > > + draft: true > > + description: | > > + Control to set the range of focus distances that is scanned. An > > + implementation may choose not to implement all the options here. > > + enum: > > + - name: AfRangeNormal > > + value: 0 > > + description: | > > + A wide range of focus distances is scanned, all the way from > > + infinity down to close distances, though depending on the > > + implementation, possibly not including the very closest macro > > + positions. > > + - name: AfRangeMacro > > + value: 1 > > + description: Only close distances are scanned. > > + - name: AfRangeFull > > + value: 2 > > + description: | > > + The full range of focus distances is scanned just as with > > + AfRangeNormal but this time including the very closest macro > > + positions. > > + > > + - AfSpeed: > > + type: int32_t > > + draft: true > > + description: | > > + Control that determines whether the AF algorithm is to move the lens > > + as quickly as possible or more steadily. For example, during video > > + recording it may be desirable not to move the lens too abruptly, but > > + when in a preview mode (waiting for a still capture) it may be > > + helpful to move the lens as quickly as is reasonably possible. > > + enum: > > + - name: AfSpeedNormal > > + value: 0 > > + description: Move the lens at its usual speed. > > + - name: AfSpeedFast > > + value: 1 > > + description: Move the lens more quickly. > > + > > + - AfWindows: > > + type: Rectangle > > + draft: true > > + description: | > > + Sets the focus windows used by the AF algorithm. The units used express > > + a proportion of the ScalerCrop control (or if unavailable, of the entire > > + image), as u0.16 format numbers. > > I don't mind using ScalerCrop, but I think it would probably make more > sense to use the full pixel array ? The thing that bothers me a bit about the full pixel array is that, if you use digital zoom, then you will have to update your AfWindows as they might otherwise lie outside your final image! If it's defined as being a region of the scaler crop, then it automatically stays the same (to the application, at least, the pipeline handler probably has some work to do). I note that the Android link you give below says "Focus areas are relative to the current field of view (getZoom())", which I guess is partly why they use that slightly arbitrary +/-1000 range. Maybe 1000 is easier to work with than the u0.16 format that I specified? > > > + > > + In order to be activated, a rectangle must be programmed with non-zero > > + width and height. If no rectangles are programmed in this way, then the > > + system will choose its own single default window in the centre of the > > + image. > > + > > + The details of how the windows are used are platform dependent. We note > > + that when there is more than one AF window, a typical implementation > > + might find the optimal focus position for each one and finally select > > + the window closest to the camera. > > + > > + size: [platform dependent] > > And I would like to point you to this one (you probably already now it): > https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas() > > I really like the number of areas variable, and even more the weight per > area. It makes sense, you might want to have more weight in the center, > but still have a larger region. > > I don't get why the corrdinates are between [-1000, 1000] though but... > not that important. > > What do you think ? Interesting, I'm not quite sure what to make of it. It feels rather like they don't support traditional multi-spot AF where you focus on the closest of the regions, not on some average of them. The former (traditional multi-spot) certainly seems more useful to me. (In fact I think "averaging" focus regions is a bit misguided, because you have no a priori information how different the amount of detail might be in different regions. So a slightly featureless face in the foreground would get totally overwhelmed by that bush lurking in the background, pretty much irrespective of the weights you specify.) In my original proposal there was an AfMethod control. Perhaps we need to bring it back (with values AfMethodAverage and AfMethodMultiSpot) to give us both options? In the former case we'd obviously need to add a per-region weight. Actually, I think we might be able to conjure up an interesting interpretation for the weights in the multi-spot case, but perhaps we leave that as "implementation defined" for the moment. Thoughts? I'm trying to put together the next version of the patch, but opinions on this in the meantime would be helpful. Thanks! David > JM > > > + > > + - AfTrigger: > > + type: int32_t > > + draft: true > > + description: | > > + This control starts an autofocus scan when AfMode is set to AfModeAuto, > > + and can also be used to terminate a scan early. > > + > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. > > + > > + enum: > > + - name: AfTriggerStart > > + value: 0 > > + description: Start an AF scan. Ignored if a scan is in progress. > > + - name: AfTriggerCancel > > + value: 1 > > + description: Cancel an AF scan. This does not cause the lens to move > > + anywhere else. Ignored if no scan is in progress. > > + > > + - AfPause: > > + type: int32_t > > + draft: true > > + description: | > > + This control has no effect except when in continuous autofocus mode > > + (AfModeContinuous). It can be used to pause any lens movements while > > + (for example) images are captured. The algorithm remains inactive > > + until it is instructed to resume. > > + > > + enum: > > + - name: AfPauseImmediate > > + value: 0 > > + description: | > > + Pause the continuous autofocus algorithm immediately, whether or > > + not any kind of scan is underway. If the AfState was previously > > + reporting AfStateContinuousScanning it will now change to > > + AfStateContinuousScanningPaused, and similarly for > > + AfStateContinuousFocused and AfStateContinuousFailed. > > + - name AfPauseDeferred > > + value: 1 > > + description: | > > + Pause the continuous autofocus algorithm as soon as it is no longer > > + scanning. If the AfState is currently reporting > > + AfStateContinuousFocused is will change to > > + AfStateContinuousFocusedPaused, and similarly in the case of > > + AfStateContinuousFailed. > > + > > + If AfState reports AfStateContinuousScanning it will change to > > + AfStateContinuousScanningPausing, and then move to one of > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > + when the scan completes. > > + - name: AfPauseResume > > + value: 2 > > + description: | > > + Resume continous autofocus operation. The algorithm starts again > > + from exactly where it left off, and the AfState will drop the > > + 'Paused' or 'Pausing' part of the state. > > + > > + - LensPosition: > > + type: int32_t > > + draft: true > > + description: | > > + Acts as a control to instruct the lens to move to a particular position > > + and also reports back the position of the lens for each frame. > > + > > + The units are determined by the lens driver. > > + > > + The LensPosition control is ignored unless the AfMode is set to > > + AfModeManual. > > + > > + Note that this control is distinct from AfLensPosition, which allows > > + the lens to be moved to its macro/hyperfocal/infinity position, rather > > + than using lens driver units. > > + > > + - AfState: > > + type: int32_t > > + draft: true > > + description: | > > + Reports the current state of the AF algorithm. The possible state changes > > + are described below, although we add that the following general state > > + changes are also allowed. > > + > > + In any of the AfStateManual or AfStateContinuous states, the AfMode > > + may be set to AfModeAuto and the algorithm will move to the > > + AfStateAuto state. If AfTriggerStart is sent at the same time > > + then the algorithm will move to AfStateAutoScanning state. > > + > > + In any of the AfStateAuto or AfStateManual states, the AfMode may > > + be set to AfModeContinuous and the algorithm will move to > > + AfStateContinuousScanning. > > + > > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may > > + be set to AfModeManual and the algorithm will move to > > + AfStateManual. The lens will not be moved and will be left where > > + it was at that moment. > > + > > + enum: > > + - name: AfStateManual > > + value: 0 > > + description: | > > + The AF algorithm is in manual mode (AfModeManual). The LensPosition > > + and AfLensPosition controls can be used directly to move the lens. > > + - name: AfStateAuto > > + value: 1 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and has either just > > + been moved into that state, or a scan that was in progress has been > > + cancelled. > > + - name: AfStateAutoScanning > > + value: 2 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been > > + started using the AfTrigger control. The scan can be cancelled by > > + sending AfTriggerCancel at which point the algorithm will either > > + move back to AfStateAuto or, if the scan actually completes > > + before the cancel request is processed, to one of > > + AfStateAutoFocused or AfStateAutoFailed. > > + - name: AfStateAutoFocused > > + value: 3 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > + completed with the result that the algorithm believes the image is > > + now in focus. > > + - name: AfStateAutoFailed > > + value: 4 > > + description: | > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has > > + completed with the result that the algorithm did not find a good > > + focus position. > > + - name: AfStateContinuousScanning > > + value: 5 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + is currently scanning for a good focus position. This occurs when > > + the mode is first set to AfModeContinuous, or may happen > > + spontaneously when the algorithm believes a re-scan is needed. > > + - name: AfStateContinuousScanningPausing > > + value: 6 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + was scanning when AfPauseDeferred was sent. The scan will complete > > + at which point the algorithm moves to the > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused > > + state. > > + - name: AfStateContinuousScanningPaused > > + value: 7 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + was scanning when AfPauseImmediate was sent. The scan will not > > + complete and the algorithm will remain in this state. The scan > > + may be resumed by sending AfPauseResume. > > + - name: AfStateContinuousFocused > > + value: 8 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + a scan has completed with the algorithm believing it has found a > > + good focus position. > > + - name: AfStateContinuousFocusedPaused > > + value: 9 > > + description: | > > + The AF algorithm was in the AfStateContinuousFocused state and > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > + or it was in the AfStateContinuousScanningPausing state and the > > + scan has completed successfully. The algorithm will now remain > > + in this state, and may be resumed by sending AfPauseResume. > > + - name: AfStateContinuousFailed > > + value: 10 > > + description: | > > + The AF algorithm is in continuous mode (AfModeContinuous) and > > + a scan has completed without finding a good focus position. > > + - name: AfStateContinuousFailedPaused > > + value: 11 > > + description: | > > + The AF algorithm was in the AfStateContinuousFailed state and > > + has been paused (by either AfPauseImmediate or AfPauseDeferred), > > + or it was in the AfStateContinuousScanningPausing state and the > > + scan has completed unsuccessfully. The algorithm will now remain > > + in this state, and may be resumed by sending AfPauseResume. > > + > > ...
diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 9d4638ae..23607368 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -406,27 +406,6 @@ controls: The camera will cancel any active or completed metering sequence. The AE algorithm is reset to its initial state. - - AfTrigger: - type: int32_t - draft: true - description: | - Control for AF trigger. Currently identical to - ANDROID_CONTROL_AF_TRIGGER. - - Whether the camera device will trigger autofocus for this request. - enum: - - name: AfTriggerIdle - value: 0 - description: The trigger is idle. - - name: AfTriggerStart - value: 1 - description: The AF routine is started by the camera. - - name: AfTriggerCancel - value: 2 - description: | - The camera will cancel any active trigger and the AF routine is - reset to its initial state. - - NoiseReductionMode: type: int32_t draft: true @@ -507,45 +486,6 @@ controls: The AE algorithm has started a pre-capture metering session. \sa AePrecaptureTrigger - - AfState: - type: int32_t - draft: true - description: | - Control to report the current AF algorithm state. Currently identical to - ANDROID_CONTROL_AF_STATE. - - Current state of the AF algorithm. - enum: - - name: AfStateInactive - value: 0 - description: The AF algorithm is inactive. - - name: AfStatePassiveScan - value: 1 - description: | - AF is performing a passive scan of the scene in continuous - auto-focus mode. - - name: AfStatePassiveFocused - value: 2 - description: | - AF believes the scene is in focus, but might restart scanning. - - name: AfStateActiveScan - value: 3 - description: | - AF is performing a scan triggered by an AF trigger request. - \sa AfTrigger - - name: AfStateFocusedLock - value: 4 - description: | - AF believes has focused correctly and has locked focus. - - name: AfStateNotFocusedLock - value: 5 - description: | - AF has not been able to focus and has locked. - - name: AfStatePassiveUnfocused - value: 6 - description: | - AF has completed a passive scan without finding focus. - - AwbState: type: int32_t draft: true @@ -690,4 +630,317 @@ controls: value. All of the custom test patterns will be static (that is the raw image must not vary from frame to frame). + - AfMode: + type: int32_t + draft: true + description: | + Control to set the mode of the AF (autofocus) algorithm. Applications + are allowed to set a new mode, and to send additional controls for + that new mode, in the same request. + + An implementation may choose not to implement all the modes. + + enum: + - name: AfModeManual + value: 0 + description: | + The AF algorithm is in manual mode. In this mode it will never + perform any action nor move the lens of its own accord, but an + application can set controls to move the lens. + + In this mode the AfState will always report AfStateManual. + - name: AfModeAuto + value: 1 + description: | + The AF algorithm is in auto mode. This means that the algorithm + will never move the lens or change state unless the AfTrigger + control is used. The AfTrigger control can be used to initiate a + focus scan, the results of which will also be reported by AfState. + + If the autofocus algorithm is moved from AfModeAuto to another + mode while a scan is in progress, the scan is cancelled + immediately, without waiting for the scan to finish. + + When first entering this mode the AfState will report + AfStateAuto. When a trigger control is sent, AfState will + report AfStateAutoScanning for a period before spontaneously + changing to AfStateAutoFocused or AfStateAutoFailed, depending on + the outcome of the scan. It will remain in this state until + another scan is initiated by the AfTrigger control. If a scan is + cancelled (without changing to another mode), AfState will return + to AfStateAuto. + - name: AfModeContinuous + value: 2 + description: | + The AF algorithm is in continuous mode. This means that the lens + can re-start a scan spontaneously at any moment, without any user + intervention. The AfState still reports whether the algorithm is + currently scanning or not, though the application has no ability + to initiate or cancel scans (though it can "pause" them), nor to + move the lens for itself. + + When set to AfModeContinuous, the system will immediately initiate + a scan so AfState will report AfStateContinuousScanning, and will + settle on one of AfStateContinuousFocused or AfStateContinuousFailed, + depending on the scan result. + + - AfLensPosition: + type: int32_t + draft: true + description: | + Control to set the position of the lens to one of the following + predefined locations. This control only has any effect when the AfMode + is set to AfModeManual. + + This control is distinct from the LensPosition control, which sets the + lens position using the lens driver's units. + enum: + - name: AfLensPositionMacro + value: 0 + description: The closest focal position of the lens. + - name: AfLensPositionHyperfocal + value: 1 + description: Hyperfocal position. + - name: AfLensPositionInfinity + value: 2 + description: The furthest focal position (usually infinity). + + - AfRange: + type: int32_t + draft: true + description: | + Control to set the range of focus distances that is scanned. An + implementation may choose not to implement all the options here. + enum: + - name: AfRangeNormal + value: 0 + description: | + A wide range of focus distances is scanned, all the way from + infinity down to close distances, though depending on the + implementation, possibly not including the very closest macro + positions. + - name: AfRangeMacro + value: 1 + description: Only close distances are scanned. + - name: AfRangeFull + value: 2 + description: | + The full range of focus distances is scanned just as with + AfRangeNormal but this time including the very closest macro + positions. + + - AfSpeed: + type: int32_t + draft: true + description: | + Control that determines whether the AF algorithm is to move the lens + as quickly as possible or more steadily. For example, during video + recording it may be desirable not to move the lens too abruptly, but + when in a preview mode (waiting for a still capture) it may be + helpful to move the lens as quickly as is reasonably possible. + enum: + - name: AfSpeedNormal + value: 0 + description: Move the lens at its usual speed. + - name: AfSpeedFast + value: 1 + description: Move the lens more quickly. + + - AfWindows: + type: Rectangle + draft: true + description: | + Sets the focus windows used by the AF algorithm. The units used express + a proportion of the ScalerCrop control (or if unavailable, of the entire + image), as u0.16 format numbers. + + In order to be activated, a rectangle must be programmed with non-zero + width and height. If no rectangles are programmed in this way, then the + system will choose its own single default window in the centre of the + image. + + The details of how the windows are used are platform dependent. We note + that when there is more than one AF window, a typical implementation + might find the optimal focus position for each one and finally select + the window closest to the camera. + + size: [platform dependent] + + - AfTrigger: + type: int32_t + draft: true + description: | + This control starts an autofocus scan when AfMode is set to AfModeAuto, + and can also be used to terminate a scan early. + + It is ignored if AfMode is set to AfModeManual or AfModeContinuous. + + enum: + - name: AfTriggerStart + value: 0 + description: Start an AF scan. Ignored if a scan is in progress. + - name: AfTriggerCancel + value: 1 + description: Cancel an AF scan. This does not cause the lens to move + anywhere else. Ignored if no scan is in progress. + + - AfPause: + type: int32_t + draft: true + description: | + This control has no effect except when in continuous autofocus mode + (AfModeContinuous). It can be used to pause any lens movements while + (for example) images are captured. The algorithm remains inactive + until it is instructed to resume. + + enum: + - name: AfPauseImmediate + value: 0 + description: | + Pause the continuous autofocus algorithm immediately, whether or + not any kind of scan is underway. If the AfState was previously + reporting AfStateContinuousScanning it will now change to + AfStateContinuousScanningPaused, and similarly for + AfStateContinuousFocused and AfStateContinuousFailed. + - name AfPauseDeferred + value: 1 + description: | + Pause the continuous autofocus algorithm as soon as it is no longer + scanning. If the AfState is currently reporting + AfStateContinuousFocused is will change to + AfStateContinuousFocusedPaused, and similarly in the case of + AfStateContinuousFailed. + + If AfState reports AfStateContinuousScanning it will change to + AfStateContinuousScanningPausing, and then move to one of + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused + when the scan completes. + - name: AfPauseResume + value: 2 + description: | + Resume continous autofocus operation. The algorithm starts again + from exactly where it left off, and the AfState will drop the + 'Paused' or 'Pausing' part of the state. + + - LensPosition: + type: int32_t + draft: true + description: | + Acts as a control to instruct the lens to move to a particular position + and also reports back the position of the lens for each frame. + + The units are determined by the lens driver. + + The LensPosition control is ignored unless the AfMode is set to + AfModeManual. + + Note that this control is distinct from AfLensPosition, which allows + the lens to be moved to its macro/hyperfocal/infinity position, rather + than using lens driver units. + + - AfState: + type: int32_t + draft: true + description: | + Reports the current state of the AF algorithm. The possible state changes + are described below, although we add that the following general state + changes are also allowed. + + In any of the AfStateManual or AfStateContinuous states, the AfMode + may be set to AfModeAuto and the algorithm will move to the + AfStateAuto state. If AfTriggerStart is sent at the same time + then the algorithm will move to AfStateAutoScanning state. + + In any of the AfStateAuto or AfStateManual states, the AfMode may + be set to AfModeContinuous and the algorithm will move to + AfStateContinuousScanning. + + In any of the AfStateAuto or AfStateContinuous states, the AfMode may + be set to AfModeManual and the algorithm will move to + AfStateManual. The lens will not be moved and will be left where + it was at that moment. + + enum: + - name: AfStateManual + value: 0 + description: | + The AF algorithm is in manual mode (AfModeManual). The LensPosition + and AfLensPosition controls can be used directly to move the lens. + - name: AfStateAuto + value: 1 + description: | + The AF algorithm is in auto mode (AfModeAuto), and has either just + been moved into that state, or a scan that was in progress has been + cancelled. + - name: AfStateAutoScanning + value: 2 + description: | + The AF algorithm is in auto mode (AfModeAuto), and a scan has been + started using the AfTrigger control. The scan can be cancelled by + sending AfTriggerCancel at which point the algorithm will either + move back to AfStateAuto or, if the scan actually completes + before the cancel request is processed, to one of + AfStateAutoFocused or AfStateAutoFailed. + - name: AfStateAutoFocused + value: 3 + description: | + The AF algorithm is in auto mode (AfModeAuto), and a scan has + completed with the result that the algorithm believes the image is + now in focus. + - name: AfStateAutoFailed + value: 4 + description: | + The AF algorithm is in auto mode (AfModeAuto), and a scan has + completed with the result that the algorithm did not find a good + focus position. + - name: AfStateContinuousScanning + value: 5 + description: | + The AF algorithm is in continuous mode (AfModeContinuous) and + is currently scanning for a good focus position. This occurs when + the mode is first set to AfModeContinuous, or may happen + spontaneously when the algorithm believes a re-scan is needed. + - name: AfStateContinuousScanningPausing + value: 6 + description: | + The AF algorithm is in continuous mode (AfModeContinuous) and + was scanning when AfPauseDeferred was sent. The scan will complete + at which point the algorithm moves to the + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused + state. + - name: AfStateContinuousScanningPaused + value: 7 + description: | + The AF algorithm is in continuous mode (AfModeContinuous) and + was scanning when AfPauseImmediate was sent. The scan will not + complete and the algorithm will remain in this state. The scan + may be resumed by sending AfPauseResume. + - name: AfStateContinuousFocused + value: 8 + description: | + The AF algorithm is in continuous mode (AfModeContinuous) and + a scan has completed with the algorithm believing it has found a + good focus position. + - name: AfStateContinuousFocusedPaused + value: 9 + description: | + The AF algorithm was in the AfStateContinuousFocused state and + has been paused (by either AfPauseImmediate or AfPauseDeferred), + or it was in the AfStateContinuousScanningPausing state and the + scan has completed successfully. The algorithm will now remain + in this state, and may be resumed by sending AfPauseResume. + - name: AfStateContinuousFailed + value: 10 + description: | + The AF algorithm is in continuous mode (AfModeContinuous) and + a scan has completed without finding a good focus position. + - name: AfStateContinuousFailedPaused + value: 11 + description: | + The AF algorithm was in the AfStateContinuousFailed state and + has been paused (by either AfPauseImmediate or AfPauseDeferred), + or it was in the AfStateContinuousScanningPausing state and the + scan has completed unsuccessfully. The algorithm will now remain + in this state, and may be resumed by sending AfPauseResume. + ...