[{"id":21777,"web_url":"https://patchwork.libcamera.org/comment/21777/","msgid":"","date":"2021-12-14T13:44:41","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi David,\n\nThank you for this work. I'll kick off with a few questions, but I'm sure\nthere will be more to come :)\n\nOn Mon, 13 Dec 2021 at 15:22, David Plowman \nwrote:\n\n> This patch describes a series of controls that allow applications to\n> drive AF algorithms:\n>\n> AfMode - auto or continuous\n> AfRange - full, macro or normal\n> AfSpeed - fast or slow\n> AfMethod - single or multi-spot\n> AfWindow - AF window locations\n> AfTrigger - start (trigger an AF scan) or cancel\n> LensPosition - position of lens from lens driver\n> AfState - reset, scanning, focused or failed\n> ---\n> src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> 1 file changed, 167 insertions(+), 60 deletions(-)\n>\n> diff --git a/src/libcamera/control_ids.yaml\n> b/src/libcamera/control_ids.yaml\n> index 9d4638ae..e579f7b7 100644\n> --- a/src/libcamera/control_ids.yaml\n> +++ b/src/libcamera/control_ids.yaml\n> @@ -406,27 +406,6 @@ controls:\n> The camera will cancel any active or completed metering\n> sequence.\n> The AE algorithm is reset to its initial state.\n>\n> - - AfTrigger:\n> - type: int32_t\n> - draft: true\n> - description: |\n> - Control for AF trigger. Currently identical to\n> - ANDROID_CONTROL_AF_TRIGGER.\n> -\n> - Whether the camera device will trigger autofocus for this request.\n> - enum:\n> - - name: AfTriggerIdle\n> - value: 0\n> - description: The trigger is idle.\n> - - name: AfTriggerStart\n> - value: 1\n> - description: The AF routine is started by the camera.\n> - - name: AfTriggerCancel\n> - value: 2\n> - description: |\n> - The camera will cancel any active trigger and the AF routine\n> is\n> - reset to its initial state.\n> -\n> - NoiseReductionMode:\n> type: int32_t\n> draft: true\n> @@ -507,45 +486,6 @@ controls:\n> The AE algorithm has started a pre-capture metering session.\n> \\sa AePrecaptureTrigger\n>\n> - - AfState:\n> - type: int32_t\n> - draft: true\n> - description: |\n> - Control to report the current AF algorithm state. Currently\n> identical to\n> - ANDROID_CONTROL_AF_STATE.\n> -\n> - Current state of the AF algorithm.\n> - enum:\n> - - name: AfStateInactive\n> - value: 0\n> - description: The AF algorithm is inactive.\n> - - name: AfStatePassiveScan\n> - value: 1\n> - description: |\n> - AF is performing a passive scan of the scene in continuous\n> - auto-focus mode.\n> - - name: AfStatePassiveFocused\n> - value: 2\n> - description: |\n> - AF believes the scene is in focus, but might restart scanning.\n> - - name: AfStateActiveScan\n> - value: 3\n> - description: |\n> - AF is performing a scan triggered by an AF trigger request.\n> - \\sa AfTrigger\n> - - name: AfStateFocusedLock\n> - value: 4\n> - description: |\n> - AF believes has focused correctly and has locked focus.\n> - - name: AfStateNotFocusedLock\n> - value: 5\n> - description: |\n> - AF has not been able to focus and has locked.\n> - - name: AfStatePassiveUnfocused\n> - value: 6\n> - description: |\n> - AF has completed a passive scan without finding focus.\n> -\n> - AwbState:\n> type: int32_t\n> draft: true\n> @@ -690,4 +630,171 @@ controls:\n> value. All of the custom test patterns will be static (that\n> is the\n> raw image must not vary from frame to frame).\n>\n> + - AfMode:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control to set the mode of the AF (autofocus) algorithm.\n> + enum:\n> + - name: AfModeAuto\n> + value: 0\n> + description: |\n> + The AF algorithm is in auto mode. This means that the\n> algorithm\n> + will never move the lens or change state unless the AfTrigger\n> + control is used. The AfTrigger control can be used to\n> initiate a\n> + focus scan, the results of which will be reported by the\n> AfState.\n> +\n> + In this mode, an application can move the lens for itself\n> using\n> + the LensPosition control. If the algorithm was scanning when\n> the\n> + lens is moved in this way, then the scan is implicitly\n> cancelled.\n> + - name: AfModeContinuous\n> + value: 1\n> + description: |\n> + The AF algorithm is in continuous mode. This means that the\n> lens\n> + can re-start a scan spontaneously at any moment, without any\n> user\n> + intervention. The AfState still reports whether the algorithm\n> is\n> + currently scanning or not, though the application has no\n> ability\n> + to initiate or cancel scans, nor move the lens for itself.\n> +\n> + The mode can be set to AfModeAuto which has the effect of\n> + \"pausing\" any continuous AF activity, after which it could\n> then\n> + be moved back to AfModeContinuous to resume operation.\n>\n\nI wonder if we need a method to tell the lens to move to a \"default\"\nposition?\nFor example, cancel AF or CAF, then move the lens to hyperfocal. With the\nabove description, we can stop AF, and use the LensPosition control to move\nto a specified position, but how do we know what position corresponds to\nhyperfocal? If the hyperfocal position is defined in the tuning file, the\napplication\nwould not have access to the value. Perhaps an explicit mode to do this?\n\n\n> +\n> + - AfRange:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control to set the range of focus distances that is scanned.\n> + enum:\n> + - name: AfRangeFull\n> + value: 0\n> + description: The full range of focus distances is scanned.\n> + - name: AfRangeMacro\n> + value: 1\n> + description: Only close distances are scanned.\n> + - name: AfRangeNormal\n> + value: 2\n> + description: |\n> + The full range of focus distances is scanned except for some\n> of\n> + the closest macro positions.\n>\n\nIs the \"closest macro position\" a property of the module, or a user\npreference?\n\n\n> +\n> + - AfSpeed:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control that determines whether the AF algorithm is to move the\n> lens\n> + as quickly as possible or more steadily. For example, during video\n> + recording it may be desirable not to move the lens too abruptly,\n> but\n> + when in a preview mode (waiting for a still capture) it may be\n> + helpful to move the lens as quickly as is reasonably possible.\n> + enum:\n> + - name: AfSpeedFast\n> + value: 0\n> + description: Move the lens quickly.\n> + - name: AfSpeedSlow\n> + value: 0\n> + description: Move the lens more steadily.\n> +\n> + - AfMethod:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control whether the AF algorithm uses a single window in the\n> image to\n> + determine the best focus position, or multiple windows\n> simultaneously.\n> + enum:\n> + - name: AfMethodSingle\n> + value: 0\n> + description: |\n> + A single window within the image, defaulting to the centre,\n> is used\n> + to select the best focus distance.\n> + - name: AfMethodMultiSpot\n> + value: 0\n> + description: |\n> + Multiple windows within the image are used to select the best\n> focus\n> + distance. The best focus distance is found for each one of the\n> + windows, and then the distance that is closest to the camera\n> is\n> + selected.\n>\n\nWould AFfMethod be used in continuous focus modes as well? if yes, then\nperhaps\ns/AF algorithm/focus algorithms/? Ditto for AfWindow.\n\n\n> +\n> + - AfWindow:\n> + type: Rectangle\n> + draft: true\n> + description: |\n> + Sets the focus windows used by the AF algorithm. The units used\n> express\n> + a proportion of the ScalerCrop control (or if unavailable, of\n> the entire\n> + image), as u0.16 format numbers.\n> +\n> + In order to be activated, a rectangle must be programmed with\n> non-zero\n> + width and height. If no rectangles are programmed in this way,\n> then the\n> + system will choose its own single default window in the centre\n> of the\n> + image.\n> +\n> + If AfMethod is set to AfMethodSingle, then only the first\n> Rectangle in\n> + this list is used (or the system default one if it is\n> unprogrammed).\n> +\n> + If AfMethod is set to AfMethodMultiSpot then all the valid\n> Rectangles in\n> + this list are used. The size of the control indicates how many\n> such\n> + windows can be programmed and will vary between different\n> platforms.\n> +\n> + size: [platform dependent]\n> +\n> + - AfTrigger:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + This control starts an autofocus scan when AfMode is set to\n> AfModeAuto,\n> + and can also be used to terminate a scan early.\n> +\n> + It is ignored if AfMode is set to AfModeContinuous.\n> +\n> + enum:\n> + - name: AfTriggerStart\n> + value: 0\n> + description: Start an AF scan. Ignored if a scan is in progress.\n> + - name: AfTriggerCancel\n> + value: 1\n> + description: Cancel an AF scan. Ingored if no scan is in\n> progress.\n> +\n> + - LensPosition:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Acts as a control to instruct the lens to move to a particular\n> position\n> + and also reports back the position of the lens for each frame.\n> +\n> + The units are determined by the lens driver.\n>\n\nThis would mean applications might not see the same/consistent behavior with\ndifferent lens modules. Should we define a libcamera range (e.g. 0 ->\nInfinity,\n1023 -> Macro) and convert to/from lens driver units in the IPA or pipeline\nhandler?\n\n\n> +\n> + If the LensPosition control is used while an AF scan is in\n> progress,\n> + the scan is implicitly cancelled and the lens is then moved to\n> the\n> + desired location. It is ignored if AfMode is set to\n> AfModeContinuous.\n> +\n> + - AfState:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Reports the current state of the AF algorithm.\n> + enum:\n> + - name: AfStateReset\n> + value: 0\n> + description: |\n> + The AF algorithm reports this state when:\n> + * The camera system has just been started.\n> + * A scan has been cancelled.\n> + * The lens has been moved by the LensPosition control.\n> + - name: AfStateScanning\n> + value: 1\n> + description: |\n> + AF is performing a scan. This state can be entered\n> spontaneously\n> + if AfMode is set to AfModeContinuous, otherwise it requires\n> the\n> + application to use the AfTrigger control to start the scan.\n> + - name: AfStateFocused\n> + value: 2\n> + description: |\n> + An AF scan has been performed and the algorithm believes the\n> + scene is in focus.\n> + - name: AfStateFailed\n> + value: 3\n> + description: |\n> + An AF scan has been performed but the algorithm has not\n> been able\n> + to find the best focus position.\n> +\n>\n\nOne last point, we currently have a FocusFoM control that returns a focus\nmeasure\nto the application. In light of the new AfWindow measure, should that\nperhaps be\nredefined?\n\nRegards,\nNaush\n\n\n\n> ...\n> --\n> 2.30.2\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C9D52BDB13\n\tfor ;\n\tTue, 14 Dec 2021 13:45:01 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EA0F960894;\n\tTue, 14 Dec 2021 14:45:00 +0100 (CET)","from mail-lf1-x136.google.com (mail-lf1-x136.google.com\n\t[IPv6:2a00:1450:4864:20::136])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D017460117\n\tfor ;\n\tTue, 14 Dec 2021 14:44:58 +0100 (CET)","by mail-lf1-x136.google.com with SMTP id t26so36896123lfk.9\n\tfor ;\n\tTue, 14 Dec 2021 05:44:58 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"XjDBJH6E\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=8noi/uQfvGBM9gQP+0AuBPUzWY5Pi/qjZFUHEETc3NI=;\n\tb=XjDBJH6EKB9zKX4sDocRivxRqfu6R0MqH6+c1ePOLON5Vqshzmw3mBxpyc/I9jPAkQ\n\tZvbKGTFxAe3+Php0Gs3aM+O/RmrM1GbRwq9Cd1zXBSYg5/e/17QLgcNZWTpSWOxE0srh\n\tyF7pf/GgQNrw2U7HpLN3l8bpnQ9Jvl9aoYjFMutVhUNpgbyhjxuyF8tySemPppVZFg7H\n\tak9xwiHRv/vgj4pfnxRM1nMiI2TFPYEAomIBeVZXbviOJdYsEkcrVshY/xjHjBGv1r+I\n\tB0eSaKDs7SB6Vt7yuBoQc4ozr2HpwUt+i3K5qV6jQ04zdon5alDHMsjxnBYutJkaeQiw\n\tzfnQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=8noi/uQfvGBM9gQP+0AuBPUzWY5Pi/qjZFUHEETc3NI=;\n\tb=4mhgT4ljW0x/0Jy2MYJLDX1gCFNc1sML5F9fy76yMy2K/bAQ/w77VMnCtk64KekS8P\n\ts0ET6uiJ217o7OscS2qsYFE3TUeH3DpYOHzRPPDpeT09u00Nmv1aG8nPGKdoKkkDmSGb\n\ti0tmFJv/pxVHI2AMpq7SrdIILmQ0OyI+0bYPSOF+WoWwxJ4qB5Xtdajqh7hyl/keIY/g\n\t0jPEYERfcfhRHqvLl2xIuw1F8EuDAj0MXFzWdZc7XW5BQ0qZ1f/Pdika25RArK13kaAE\n\tEEx5o8ZEKcxvqD4/UBC8LdrLq9St1ac0re0KmbFV2Jtez2vqSxwm3WOpnoqagBp8F+fm\n\tBZLw==","X-Gm-Message-State":"AOAM530k/M77Dt62OKaZTLgyOTMlE3qNI2xudFd7Sx2VhL0qm3x2kYJe\n\tWRUcMGekhGtq8n5cfa//hEY62qig+DyY1lnF7Zsklg==","X-Google-Smtp-Source":"ABdhPJyBO9JpbnpjIo5AkjimJT+6hkx0B7gA2ityY6fCb5bid61/p1Zn6NNVCDR2+hZaEX3yKJFEW5TxTMMTxv8aXko=","X-Received":"by 2002:a05:6512:2204:: with SMTP id\n\th4mr4891881lfu.687.1639489497619; \n\tTue, 14 Dec 2021 05:44:57 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>","In-Reply-To":"<20211213152215.17584-2-david.plowman@raspberrypi.com>","From":"Naushir Patuck ","Date":"Tue, 14 Dec 2021 13:44:41 +0000","Message-ID":"","To":"David Plowman ","Content-Type":"multipart/alternative; boundary=\"0000000000001e7ab905d31b65d2\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21807,"web_url":"https://patchwork.libcamera.org/comment/21807/","msgid":"","date":"2021-12-16T11:18:39","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":98,"url":"https://patchwork.libcamera.org/api/people/98/","name":"Hanlin Chen","email":"hanlinchen@chromium.org"},"content":"Hi David, thanks for this work!\n\nOn Mon, Dec 13, 2021 at 11:22 PM David Plowman\n wrote:\n>\n> This patch describes a series of controls that allow applications to\n> drive AF algorithms:\n>\n> AfMode - auto or continuous\n> AfRange - full, macro or normal\n> AfSpeed - fast or slow\n> AfMethod - single or multi-spot\n> AfWindow - AF window locations\n> AfTrigger - start (trigger an AF scan) or cancel\n> LensPosition - position of lens from lens driver\n> AfState - reset, scanning, focused or failed\n> ---\n> src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> 1 file changed, 167 insertions(+), 60 deletions(-)\n>\n> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> index 9d4638ae..e579f7b7 100644\n> --- a/src/libcamera/control_ids.yaml\n> +++ b/src/libcamera/control_ids.yaml\n> @@ -406,27 +406,6 @@ controls:\n> The camera will cancel any active or completed metering sequence.\n> The AE algorithm is reset to its initial state.\n>\n> - - AfTrigger:\n> - type: int32_t\n> - draft: true\n> - description: |\n> - Control for AF trigger. Currently identical to\n> - ANDROID_CONTROL_AF_TRIGGER.\n> -\n> - Whether the camera device will trigger autofocus for this request.\n> - enum:\n> - - name: AfTriggerIdle\n> - value: 0\n> - description: The trigger is idle.\n> - - name: AfTriggerStart\n> - value: 1\n> - description: The AF routine is started by the camera.\n> - - name: AfTriggerCancel\n> - value: 2\n> - description: |\n> - The camera will cancel any active trigger and the AF routine is\n> - reset to its initial state.\n> -\n> - NoiseReductionMode:\n> type: int32_t\n> draft: true\n> @@ -507,45 +486,6 @@ controls:\n> The AE algorithm has started a pre-capture metering session.\n> \\sa AePrecaptureTrigger\n>\n> - - AfState:\n> - type: int32_t\n> - draft: true\n> - description: |\n> - Control to report the current AF algorithm state. Currently identical to\n> - ANDROID_CONTROL_AF_STATE.\n> -\n> - Current state of the AF algorithm.\n> - enum:\n> - - name: AfStateInactive\n> - value: 0\n> - description: The AF algorithm is inactive.\n> - - name: AfStatePassiveScan\n> - value: 1\n> - description: |\n> - AF is performing a passive scan of the scene in continuous\n> - auto-focus mode.\n> - - name: AfStatePassiveFocused\n> - value: 2\n> - description: |\n> - AF believes the scene is in focus, but might restart scanning.\n> - - name: AfStateActiveScan\n> - value: 3\n> - description: |\n> - AF is performing a scan triggered by an AF trigger request.\n> - \\sa AfTrigger\n> - - name: AfStateFocusedLock\n> - value: 4\n> - description: |\n> - AF believes has focused correctly and has locked focus.\n> - - name: AfStateNotFocusedLock\n> - value: 5\n> - description: |\n> - AF has not been able to focus and has locked.\n> - - name: AfStatePassiveUnfocused\n> - value: 6\n> - description: |\n> - AF has completed a passive scan without finding focus.\n> -\n> - AwbState:\n> type: int32_t\n> draft: true\n> @@ -690,4 +630,171 @@ controls:\n> value. All of the custom test patterns will be static (that is the\n> raw image must not vary from frame to frame).\n>\n> + - AfMode:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control to set the mode of the AF (autofocus) algorithm.\n> + enum:\n> + - name: AfModeAuto\n> + value: 0\n> + description: |\n> + The AF algorithm is in auto mode. This means that the algorithm\n> + will never move the lens or change state unless the AfTrigger\n> + control is used. The AfTrigger control can be used to initiate a\n> + focus scan, the results of which will be reported by the AfState.\n> +\n> + In this mode, an application can move the lens for itself using\n> + the LensPosition control. If the algorithm was scanning when the\n> + lens is moved in this way, then the scan is implicitly cancelled.\n\nI would suggest splitting the Auto mode into AfModeOnDemand and\nAfModeManual (The naming could be re-considered).\nThe reason is that the sub-controls for each AfModes could be distinct.\n1. AfModeManual:\nLensPosition is valid only for the mode, and AfState is always AfStateReset.\n2. AfModeOnDemand\nAfTrigger and AfRange are valid only for the mode, and AfState reports\naccordingly. The scan is implicitly canceled when changing from the\nmode.\n3. AfModeContinuous\nAfSpeed and \"pause\" are valid only for the mode, and AfState reports\naccordingly. The scan is implicitly canceled when changing from the\nmode.\n\n> + - name: AfModeContinuous\n> + value: 1\n> + description: |\n> + The AF algorithm is in continuous mode. This means that the lens\n> + can re-start a scan spontaneously at any moment, without any user\n> + intervention. The AfState still reports whether the algorithm is\n> + currently scanning or not, though the application has no ability\n> + to initiate or cancel scans, nor move the lens for itself.\n> +\n> + The mode can be set to AfModeAuto which has the effect of\n> + \"pausing\" any continuous AF activity, after which it could then\n> + be moved back to AfModeContinuous to resume operation.\n> +\n> + - AfRange:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control to set the range of focus distances that is scanned.\n> + enum:\n> + - name: AfRangeFull\n> + value: 0\n> + description: The full range of focus distances is scanned.\n> + - name: AfRangeMacro\n> + value: 1\n> + description: Only close distances are scanned.\n> + - name: AfRangeNormal\n> + value: 2\n> + description: |\n> + The full range of focus distances is scanned except for some of\n> + the closest macro positions.\n> +\n> + - AfSpeed:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control that determines whether the AF algorithm is to move the lens\n> + as quickly as possible or more steadily. For example, during video\n> + recording it may be desirable not to move the lens too abruptly, but\n> + when in a preview mode (waiting for a still capture) it may be\n> + helpful to move the lens as quickly as is reasonably possible.\n> + enum:\n> + - name: AfSpeedFast\n> + value: 0\n> + description: Move the lens quickly.\n> + - name: AfSpeedSlow\n> + value: 0\n> + description: Move the lens more steadily.\nHow about\ns/Fast/Aggressively\ns/Slow/Steadily\n> +\n> + - AfMethod:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Control whether the AF algorithm uses a single window in the image to\n> + determine the best focus position, or multiple windows simultaneously.\n> + enum:\n> + - name: AfMethodSingle\n> + value: 0\n> + description: |\n> + A single window within the image, defaulting to the centre, is used\n> + to select the best focus distance.\n> + - name: AfMethodMultiSpot\n> + value: 0\n> + description: |\n> + Multiple windows within the image are used to select the best focus\n> + distance. The best focus distance is found for each one of the\n> + windows, and then the distance that is closest to the camera is\n> + selected.\n> +\n> + - AfWindow:\n> + type: Rectangle\n> + draft: true\n> + description: |\n> + Sets the focus windows used by the AF algorithm. The units used express\n> + a proportion of the ScalerCrop control (or if unavailable, of the entire\n> + image), as u0.16 format numbers.\n> +\n> + In order to be activated, a rectangle must be programmed with non-zero\n> + width and height. If no rectangles are programmed in this way, then the\n> + system will choose its own single default window in the centre of the\n> + image.\n> +\n> + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n> + this list is used (or the system default one if it is unprogrammed).\n> +\n> + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n> + this list are used. The size of the control indicates how many such\n> + windows can be programmed and will vary between different platforms.\n> +\n> + size: [platform dependent]\n> +\n> + - AfTrigger:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n> + and can also be used to terminate a scan early.\n> +\n> + It is ignored if AfMode is set to AfModeContinuous.\n> +\n> + enum:\n> + - name: AfTriggerStart\n> + value: 0\n> + description: Start an AF scan. Ignored if a scan is in progress.\n> + - name: AfTriggerCancel\n> + value: 1\n> + description: Cancel an AF scan. Ingored if no scan is in progress.\n> +\n> + - LensPosition:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Acts as a control to instruct the lens to move to a particular position\n> + and also reports back the position of the lens for each frame.\n> +\n> + The units are determined by the lens driver.\n> +\n> + If the LensPosition control is used while an AF scan is in progress,\n> + the scan is implicitly cancelled and the lens is then moved to the\n> + desired location. It is ignored if AfMode is set to AfModeContinuous.\n> +\n> + - AfState:\n> + type: int32_t\n> + draft: true\n> + description: |\n> + Reports the current state of the AF algorithm.\n> + enum:\n> + - name: AfStateReset\n> + value: 0\n> + description: |\n> + The AF algorithm reports this state when:\n> + * The camera system has just been started.\n> + * A scan has been cancelled.\n> + * The lens has been moved by the LensPosition control.\n> + - name: AfStateScanning\n> + value: 1\n> + description: |\n> + AF is performing a scan. This state can be entered spontaneously\n> + if AfMode is set to AfModeContinuous, otherwise it requires the\n> + application to use the AfTrigger control to start the scan.\n> + - name: AfStateFocused\n> + value: 2\n> + description: |\n> + An AF scan has been performed and the algorithm believes the\n> + scene is in focus.\n> + - name: AfStateFailed\n> + value: 3\n> + description: |\n> + An AF scan has been performed but the algorithm has not been able\n> + to find the best focus position.\n> +\n> ...\n> --\n> 2.30.2\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 65D2DBDB13\n\tfor ;\n\tThu, 16 Dec 2021 11:18:53 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9CE09608A4;\n\tThu, 16 Dec 2021 12:18:52 +0100 (CET)","from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com\n\t[IPv6:2607:f8b0:4864:20::22c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 34E2C60868\n\tfor ;\n\tThu, 16 Dec 2021 12:18:51 +0100 (CET)","by mail-oi1-x22c.google.com with SMTP id t23so36025223oiw.3\n\tfor ;\n\tThu, 16 Dec 2021 03:18:51 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"V8I80nxS\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=ClB5xfT4wVfTz5zai42GI6FFKQkuew38I3mwCULo0mo=;\n\tb=V8I80nxSM0OII5eyCPqvqcClxLyCoAV1DNwSZb2Cjx49jCcQgm0NJLuOT8RFakMgBP\n\tHpHMqVrc9lSWxaVfOgD/9qYmqhN0XJZ84Gm0X8sWTP90qmelt5vWvWvIy5bBM0pbSMAh\n\tkxpynTIEUT5isokA5eQPY2btjISfCbWM3Pb4E=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=ClB5xfT4wVfTz5zai42GI6FFKQkuew38I3mwCULo0mo=;\n\tb=t3m98XjBlgkPEcaKqmRG6/5fXvmK9TylRKJD052md4WzrhYBVnJ4BgwM0rFigDcq01\n\tAqIC4fydl5l+KmNOA+me5cwzI+aIaLFDZ/zekc3RGXGdhSz3k7uN0ofjLSEr/Le1RcES\n\tz7JB2be/wOpj4g1WWfoQBzftLgriApZsTZiT/yeU7QfN0gHQBjb5imIxWs3DRFZmD1E/\n\tLANuTiNUemtNcsQa/4hEW6qAkFONsray6pUF3axnQOAkspe2T36necUu5ZyJW798pNOm\n\tIRsZRZ4rRZAMLeoN7gXNY7OdnR6f67W8D5Uay8tIy++cjFrtlCBaS8GyNPUX5PaLB/IX\n\t3DDw==","X-Gm-Message-State":"AOAM533P7lgDn+b23WqkyhgdW8hd01f6eJ4LEk6z6xNe5f7mDinJ0qLV\n\tLiZ1LJAEsTgU5QGBdPZ+f00YmxPfL2iiayyNhW8FrMAz2ThVDg==","X-Google-Smtp-Source":"ABdhPJzU+L83Rk7cqH57JIf81BzlbjXKApqB769D07pZa8HI6igUWV+kZ/T5HCnse5jPH9wtzNBlIFCCCsVvVfDzHVc=","X-Received":"by 2002:a54:4595:: with SMTP id\n\tz21mr3504404oib.169.1639653529937; \n\tThu, 16 Dec 2021 03:18:49 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>","In-Reply-To":"<20211213152215.17584-2-david.plowman@raspberrypi.com>","From":"Hanlin Chen ","Date":"Thu, 16 Dec 2021 19:18:39 +0800","Message-ID":"","To":"David Plowman ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21808,"web_url":"https://patchwork.libcamera.org/comment/21808/","msgid":"","date":"2021-12-16T11:50:07","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":98,"url":"https://patchwork.libcamera.org/api/people/98/","name":"Hanlin Chen","email":"hanlinchen@chromium.org"},"content":"On Tue, Dec 14, 2021 at 9:45 PM Naushir Patuck wrote:\n>\n> Hi David,\n>\n> Thank you for this work. I'll kick off with a few questions, but I'm sure there will be more to come :)\n>\n> On Mon, 13 Dec 2021 at 15:22, David Plowman wrote:\n>>\n>> This patch describes a series of controls that allow applications to\n>> drive AF algorithms:\n>>\n>> AfMode - auto or continuous\n>> AfRange - full, macro or normal\n>> AfSpeed - fast or slow\n>> AfMethod - single or multi-spot\n>> AfWindow - AF window locations\n>> AfTrigger - start (trigger an AF scan) or cancel\n>> LensPosition - position of lens from lens driver\n>> AfState - reset, scanning, focused or failed\n>> ---\n>> src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n>> 1 file changed, 167 insertions(+), 60 deletions(-)\n>>\n>> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n>> index 9d4638ae..e579f7b7 100644\n>> --- a/src/libcamera/control_ids.yaml\n>> +++ b/src/libcamera/control_ids.yaml\n>> @@ -406,27 +406,6 @@ controls:\n>> The camera will cancel any active or completed metering sequence.\n>> The AE algorithm is reset to its initial state.\n>>\n>> - - AfTrigger:\n>> - type: int32_t\n>> - draft: true\n>> - description: |\n>> - Control for AF trigger. Currently identical to\n>> - ANDROID_CONTROL_AF_TRIGGER.\n>> -\n>> - Whether the camera device will trigger autofocus for this request.\n>> - enum:\n>> - - name: AfTriggerIdle\n>> - value: 0\n>> - description: The trigger is idle.\n>> - - name: AfTriggerStart\n>> - value: 1\n>> - description: The AF routine is started by the camera.\n>> - - name: AfTriggerCancel\n>> - value: 2\n>> - description: |\n>> - The camera will cancel any active trigger and the AF routine is\n>> - reset to its initial state.\n>> -\n>> - NoiseReductionMode:\n>> type: int32_t\n>> draft: true\n>> @@ -507,45 +486,6 @@ controls:\n>> The AE algorithm has started a pre-capture metering session.\n>> \\sa AePrecaptureTrigger\n>>\n>> - - AfState:\n>> - type: int32_t\n>> - draft: true\n>> - description: |\n>> - Control to report the current AF algorithm state. Currently identical to\n>> - ANDROID_CONTROL_AF_STATE.\n>> -\n>> - Current state of the AF algorithm.\n>> - enum:\n>> - - name: AfStateInactive\n>> - value: 0\n>> - description: The AF algorithm is inactive.\n>> - - name: AfStatePassiveScan\n>> - value: 1\n>> - description: |\n>> - AF is performing a passive scan of the scene in continuous\n>> - auto-focus mode.\n>> - - name: AfStatePassiveFocused\n>> - value: 2\n>> - description: |\n>> - AF believes the scene is in focus, but might restart scanning.\n>> - - name: AfStateActiveScan\n>> - value: 3\n>> - description: |\n>> - AF is performing a scan triggered by an AF trigger request.\n>> - \\sa AfTrigger\n>> - - name: AfStateFocusedLock\n>> - value: 4\n>> - description: |\n>> - AF believes has focused correctly and has locked focus.\n>> - - name: AfStateNotFocusedLock\n>> - value: 5\n>> - description: |\n>> - AF has not been able to focus and has locked.\n>> - - name: AfStatePassiveUnfocused\n>> - value: 6\n>> - description: |\n>> - AF has completed a passive scan without finding focus.\n>> -\n>> - AwbState:\n>> type: int32_t\n>> draft: true\n>> @@ -690,4 +630,171 @@ controls:\n>> value. All of the custom test patterns will be static (that is the\n>> raw image must not vary from frame to frame).\n>>\n>> + - AfMode:\n>> + type: int32_t\n>> + draft: true\n>> + description: |\n>> + Control to set the mode of the AF (autofocus) algorithm.\n>> + enum:\n>> + - name: AfModeAuto\n>> + value: 0\n>> + description: |\n>> + The AF algorithm is in auto mode. This means that the algorithm\n>> + will never move the lens or change state unless the AfTrigger\n>> + control is used. The AfTrigger control can be used to initiate a\n>> + focus scan, the results of which will be reported by the AfState.\n>> +\n>> + In this mode, an application can move the lens for itself using\n>> + the LensPosition control. If the algorithm was scanning when the\n>> + lens is moved in this way, then the scan is implicitly cancelled.\n>> + - name: AfModeContinuous\n>> + value: 1\n>> + description: |\n>> + The AF algorithm is in continuous mode. This means that the lens\n>> + can re-start a scan spontaneously at any moment, without any user\n>> + intervention. The AfState still reports whether the algorithm is\n>> + currently scanning or not, though the application has no ability\n>> + to initiate or cancel scans, nor move the lens for itself.\n>> +\n>> + The mode can be set to AfModeAuto which has the effect of\n>> + \"pausing\" any continuous AF activity, after which it could then\n>> + be moved back to AfModeContinuous to resume operation.\n>\n>\n> I wonder if we need a method to tell the lens to move to a \"default\" position?\n> For example, cancel AF or CAF, then move the lens to hyperfocal. With the\n> above description, we can stop AF, and use the LensPosition control to move\n> to a specified position, but how do we know what position corresponds to\n> hyperfocal? If the hyperfocal position is defined in the tuning file, the application\n> would not have access to the value. Perhaps an explicit mode to do this?\nCould we set the default value (and min/max) for LensPosition as the\nhyperfocal? As the definition of Sharpness.\nAnd the user can move to the hyperfocal position manually, or other\nposition within the range.\n>\n>>\n>> +\n>> + - AfRange:\n>> + type: int32_t\n>> + draft: true\n>> + description: |\n>> + Control to set the range of focus distances that is scanned.\n>> + enum:\n>> + - name: AfRangeFull\n>> + value: 0\n>> + description: The full range of focus distances is scanned.\n>> + - name: AfRangeMacro\n>> + value: 1\n>> + description: Only close distances are scanned.\n>> + - name: AfRangeNormal\n>> + value: 2\n>> + description: |\n>> + The full range of focus distances is scanned except for some of\n>> + the closest macro positions.\n>\n>\n> Is the \"closest macro position\" a property of the module, or a user preference?\n>\n>>\n>> +\n>> + - AfSpeed:\n>> + type: int32_t\n>> + draft: true\n>> + description: |\n>> + Control that determines whether the AF algorithm is to move the lens\n>> + as quickly as possible or more steadily. For example, during video\n>> + recording it may be desirable not to move the lens too abruptly, but\n>> + when in a preview mode (waiting for a still capture) it may be\n>> + helpful to move the lens as quickly as is reasonably possible.\n>> + enum:\n>> + - name: AfSpeedFast\n>> + value: 0\n>> + description: Move the lens quickly.\n>> + - name: AfSpeedSlow\n>> + value: 0\n>> + description: Move the lens more steadily.\n>> +\n>> + - AfMethod:\n>> + type: int32_t\n>> + draft: true\n>> + description: |\n>> + Control whether the AF algorithm uses a single window in the image to\n>> + determine the best focus position, or multiple windows simultaneously.\n>> + enum:\n>> + - name: AfMethodSingle\n>> + value: 0\n>> + description: |\n>> + A single window within the image, defaulting to the centre, is used\n>> + to select the best focus distance.\n>> + - name: AfMethodMultiSpot\n>> + value: 0\n>> + description: |\n>> + Multiple windows within the image are used to select the best focus\n>> + distance. The best focus distance is found for each one of the\n>> + windows, and then the distance that is closest to the camera is\n>> + selected.\n>\n>\n> Would AFfMethod be used in continuous focus modes as well? if yes, then perhaps\n> s/AF algorithm/focus algorithms/? Ditto for AfWindow.\n>\n>>\n>> +\n>> + - AfWindow:\n>> + type: Rectangle\n>> + draft: true\n>> + description: |\n>> + Sets the focus windows used by the AF algorithm. The units used express\n>> + a proportion of the ScalerCrop control (or if unavailable, of the entire\n>> + image), as u0.16 format numbers.\n>> +\n>> + In order to be activated, a rectangle must be programmed with non-zero\n>> + width and height. If no rectangles are programmed in this way, then the\n>> + system will choose its own single default window in the centre of the\n>> + image.\n>> +\n>> + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n>> + this list is used (or the system default one if it is unprogrammed).\n>> +\n>> + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n>> + this list are used. The size of the control indicates how many such\n>> + windows can be programmed and will vary between different platforms.\n>> +\n>> + size: [platform dependent]\n>> +\n>> + - AfTrigger:\n>> + type: int32_t\n>> + draft: true\n>> + description: |\n>> + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n>> + and can also be used to terminate a scan early.\n>> +\n>> + It is ignored if AfMode is set to AfModeContinuous.\n>> +\n>> + enum:\n>> + - name: AfTriggerStart\n>> + value: 0\n>> + description: Start an AF scan. Ignored if a scan is in progress.\n>> + - name: AfTriggerCancel\n>> + value: 1\n>> + description: Cancel an AF scan. Ingored if no scan is in progress.\n>> +\n>> + - LensPosition:\n>> + type: int32_t\n>> + draft: true\n>> + description: |\n>> + Acts as a control to instruct the lens to move to a particular position\n>> + and also reports back the position of the lens for each frame.\n>> +\n>> + The units are determined by the lens driver.\n>\n>\n> This would mean applications might not see the same/consistent behavior with\n> different lens modules. Should we define a libcamera range (e.g. 0 -> Infinity,\n> 1023 -> Macro) and convert to/from lens driver units in the IPA or pipeline handler?\n>\n>>\n>> +\n>> + If the LensPosition control is used while an AF scan is in progress,\n>> + the scan is implicitly cancelled and the lens is then moved to the\n>> + desired location. It is ignored if AfMode is set to AfModeContinuous.\n>> +\n>> + - AfState:\n>> + type: int32_t\n>> + draft: true\n>> + description: |\n>> + Reports the current state of the AF algorithm.\n>> + enum:\n>> + - name: AfStateReset\n>> + value: 0\n>> + description: |\n>> + The AF algorithm reports this state when:\n>> + * The camera system has just been started.\n>> + * A scan has been cancelled.\n>> + * The lens has been moved by the LensPosition control.\n>> + - name: AfStateScanning\n>> + value: 1\n>> + description: |\n>> + AF is performing a scan. This state can be entered spontaneously\n>> + if AfMode is set to AfModeContinuous, otherwise it requires the\n>> + application to use the AfTrigger control to start the scan.\n>> + - name: AfStateFocused\n>> + value: 2\n>> + description: |\n>> + An AF scan has been performed and the algorithm believes the\n>> + scene is in focus.\n>> + - name: AfStateFailed\n>> + value: 3\n>> + description: |\n>> + An AF scan has been performed but the algorithm has not been able\n>> + to find the best focus position.\n>> +\n>\n>\n> One last point, we currently have a FocusFoM control that returns a focus measure\n> to the application. In light of the new AfWindow measure, should that perhaps be\n> redefined?\n>\n> Regards,\n> Naush\n>\n>\n>>\n>> ...\n>> --\n>> 2.30.2\n>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 90B8FBF415\n\tfor ;\n\tThu, 16 Dec 2021 11:50:21 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C1F36608E3;\n\tThu, 16 Dec 2021 12:50:20 +0100 (CET)","from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com\n\t[IPv6:2607:f8b0:4864:20::22c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 756E760868\n\tfor ;\n\tThu, 16 Dec 2021 12:50:19 +0100 (CET)","by mail-oi1-x22c.google.com with SMTP id s139so36028967oie.13\n\tfor ;\n\tThu, 16 Dec 2021 03:50:19 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"T8eOQzbO\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=vNiiYdsRUl+hsFKbuCXWNsMxW9YSDjhgqqd49RAwkgg=;\n\tb=T8eOQzbOXNUy1t3oZCsG/VS9L61SH+MynxfPdC5rDYjg2iEEECF7TDRg/cvUJC5Y/m\n\tbE29HXbd8t0bVTnJBvvBrJ+hfzH3TXHykAeGUi2I2YXQaSgGjkgWmbc6JI/UFHwp3yDe\n\tQvjuZGdeDTUqs06bERqaEUbzeVYFl3VVDrmic=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=vNiiYdsRUl+hsFKbuCXWNsMxW9YSDjhgqqd49RAwkgg=;\n\tb=4YJ3WIukQ9rcRa6eb1QDUO+W9Rh9lNSukgkhouvUc5iAhQldwK+BObbxSS+ly4SZKy\n\tvPlxVLTyD1v7thT+2ARZ40vUPX8TAIZjplHDy9wa1h/XE8eBaQ6VBc2CrLdQKdQssMfL\n\t2SYivKOqkvxYvPn1U8b40K/xUtk0D672xqAVaWAQB+FZfDbPtZtU/lmmSyDNnE+Cs6iQ\n\tbGA9ZKPl6hMqVspsV19r6kjKFCGT3zyz1PVbkNnvRIHQBX2L/nZrAxFejRjPzcAFRQ/i\n\tg0+vCOF7uZkD5Mv7/6popRM3XrG1V9G+daOUX8SqHn+YXQ0iBIeiTA4Mr/PLELXyNWsY\n\tpO+Q==","X-Gm-Message-State":"AOAM533DPeFtCi9Becgp7Ed6Q33QXrTbBdDx9EFAqKUEqX/tJHVPTUpN\n\tygnzKXg80pJ5ljOvxruiH9NiVNUUhIsBOINu7NUNRwB5UrXsbA==","X-Google-Smtp-Source":"ABdhPJwmC55G9TWL0kZ2OAsgO+UyviDVc1Z1dR8/zdImylPYNn14SQT+/yYkUY/lzgMbM3Pul9+2dxaYYIsyXXerxrk=","X-Received":"by 2002:a54:4595:: with SMTP id\n\tz21mr3582709oib.169.1639655418250; \n\tThu, 16 Dec 2021 03:50:18 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t","In-Reply-To":"","From":"Hanlin Chen ","Date":"Thu, 16 Dec 2021 19:50:07 +0800","Message-ID":"","To":"Naushir Patuck ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21814,"web_url":"https://patchwork.libcamera.org/comment/21814/","msgid":"","date":"2021-12-17T08:40:29","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Hanlin, Naush\n\nThanks for the comments. Let me try and answer a few of them below.\n\nOn Thu, 16 Dec 2021 at 11:50, Hanlin Chen wrote:\n>\n> On Tue, Dec 14, 2021 at 9:45 PM Naushir Patuck wrote:\n> >\n> > Hi David,\n> >\n> > Thank you for this work. I'll kick off with a few questions, but I'm sure there will be more to come :)\n> >\n> > On Mon, 13 Dec 2021 at 15:22, David Plowman wrote:\n> >>\n> >> This patch describes a series of controls that allow applications to\n> >> drive AF algorithms:\n> >>\n> >> AfMode - auto or continuous\n> >> AfRange - full, macro or normal\n> >> AfSpeed - fast or slow\n> >> AfMethod - single or multi-spot\n> >> AfWindow - AF window locations\n> >> AfTrigger - start (trigger an AF scan) or cancel\n> >> LensPosition - position of lens from lens driver\n> >> AfState - reset, scanning, focused or failed\n> >> ---\n> >> src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> >> 1 file changed, 167 insertions(+), 60 deletions(-)\n> >>\n> >> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> >> index 9d4638ae..e579f7b7 100644\n> >> --- a/src/libcamera/control_ids.yaml\n> >> +++ b/src/libcamera/control_ids.yaml\n> >> @@ -406,27 +406,6 @@ controls:\n> >> The camera will cancel any active or completed metering sequence.\n> >> The AE algorithm is reset to its initial state.\n> >>\n> >> - - AfTrigger:\n> >> - type: int32_t\n> >> - draft: true\n> >> - description: |\n> >> - Control for AF trigger. Currently identical to\n> >> - ANDROID_CONTROL_AF_TRIGGER.\n> >> -\n> >> - Whether the camera device will trigger autofocus for this request.\n> >> - enum:\n> >> - - name: AfTriggerIdle\n> >> - value: 0\n> >> - description: The trigger is idle.\n> >> - - name: AfTriggerStart\n> >> - value: 1\n> >> - description: The AF routine is started by the camera.\n> >> - - name: AfTriggerCancel\n> >> - value: 2\n> >> - description: |\n> >> - The camera will cancel any active trigger and the AF routine is\n> >> - reset to its initial state.\n> >> -\n> >> - NoiseReductionMode:\n> >> type: int32_t\n> >> draft: true\n> >> @@ -507,45 +486,6 @@ controls:\n> >> The AE algorithm has started a pre-capture metering session.\n> >> \\sa AePrecaptureTrigger\n> >>\n> >> - - AfState:\n> >> - type: int32_t\n> >> - draft: true\n> >> - description: |\n> >> - Control to report the current AF algorithm state. Currently identical to\n> >> - ANDROID_CONTROL_AF_STATE.\n> >> -\n> >> - Current state of the AF algorithm.\n> >> - enum:\n> >> - - name: AfStateInactive\n> >> - value: 0\n> >> - description: The AF algorithm is inactive.\n> >> - - name: AfStatePassiveScan\n> >> - value: 1\n> >> - description: |\n> >> - AF is performing a passive scan of the scene in continuous\n> >> - auto-focus mode.\n> >> - - name: AfStatePassiveFocused\n> >> - value: 2\n> >> - description: |\n> >> - AF believes the scene is in focus, but might restart scanning.\n> >> - - name: AfStateActiveScan\n> >> - value: 3\n> >> - description: |\n> >> - AF is performing a scan triggered by an AF trigger request.\n> >> - \\sa AfTrigger\n> >> - - name: AfStateFocusedLock\n> >> - value: 4\n> >> - description: |\n> >> - AF believes has focused correctly and has locked focus.\n> >> - - name: AfStateNotFocusedLock\n> >> - value: 5\n> >> - description: |\n> >> - AF has not been able to focus and has locked.\n> >> - - name: AfStatePassiveUnfocused\n> >> - value: 6\n> >> - description: |\n> >> - AF has completed a passive scan without finding focus.\n> >> -\n> >> - AwbState:\n> >> type: int32_t\n> >> draft: true\n> >> @@ -690,4 +630,171 @@ controls:\n> >> value. All of the custom test patterns will be static (that is the\n> >> raw image must not vary from frame to frame).\n> >>\n> >> + - AfMode:\n> >> + type: int32_t\n> >> + draft: true\n> >> + description: |\n> >> + Control to set the mode of the AF (autofocus) algorithm.\n> >> + enum:\n> >> + - name: AfModeAuto\n> >> + value: 0\n> >> + description: |\n> >> + The AF algorithm is in auto mode. This means that the algorithm\n> >> + will never move the lens or change state unless the AfTrigger\n> >> + control is used. The AfTrigger control can be used to initiate a\n> >> + focus scan, the results of which will be reported by the AfState.\n> >> +\n> >> + In this mode, an application can move the lens for itself using\n> >> + the LensPosition control. If the algorithm was scanning when the\n> >> + lens is moved in this way, then the scan is implicitly cancelled.\n> >> + - name: AfModeContinuous\n> >> + value: 1\n> >> + description: |\n> >> + The AF algorithm is in continuous mode. This means that the lens\n> >> + can re-start a scan spontaneously at any moment, without any user\n> >> + intervention. The AfState still reports whether the algorithm is\n> >> + currently scanning or not, though the application has no ability\n> >> + to initiate or cancel scans, nor move the lens for itself.\n> >> +\n> >> + The mode can be set to AfModeAuto which has the effect of\n> >> + \"pausing\" any continuous AF activity, after which it could then\n> >> + be moved back to AfModeContinuous to resume operation.\n> >\n> >\n> > I wonder if we need a method to tell the lens to move to a \"default\" position?\n> > For example, cancel AF or CAF, then move the lens to hyperfocal. With the\n> > above description, we can stop AF, and use the LensPosition control to move\n> > to a specified position, but how do we know what position corresponds to\n> > hyperfocal? If the hyperfocal position is defined in the tuning file, the application\n> > would not have access to the value. Perhaps an explicit mode to do this?\n> Could we set the default value (and min/max) for LensPosition as the\n> hyperfocal? As the definition of Sharpness.\n> And the user can move to the hyperfocal position manually, or other\n> position within the range.\n\nI think I'd like the default value of the LensPosition control to be\ndefined to be hyperfocal. Would that work for everyone? The only catch\nis that this value will have to come from a database or a tuning file\nas the lens driver will not know it. Or possibly it could be in the\ndevice tree.\n\n> >\n> >>\n> >> +\n> >> + - AfRange:\n> >> + type: int32_t\n> >> + draft: true\n> >> + description: |\n> >> + Control to set the range of focus distances that is scanned.\n> >> + enum:\n> >> + - name: AfRangeFull\n> >> + value: 0\n> >> + description: The full range of focus distances is scanned.\n> >> + - name: AfRangeMacro\n> >> + value: 1\n> >> + description: Only close distances are scanned.\n> >> + - name: AfRangeNormal\n> >> + value: 2\n> >> + description: |\n> >> + The full range of focus distances is scanned except for some of\n> >> + the closest macro positions.\n> >\n> >\n> > Is the \"closest macro position\" a property of the module, or a user preference?\n\nWell, both I suppose. I suppose each module will have a \"closest macro\nposition\" but I think it quite likely that applications may not always\nwant to use it and would prefer to \"tune\" a different value.\n\n> >\n> >>\n> >> +\n> >> + - AfSpeed:\n> >> + type: int32_t\n> >> + draft: true\n> >> + description: |\n> >> + Control that determines whether the AF algorithm is to move the lens\n> >> + as quickly as possible or more steadily. For example, during video\n> >> + recording it may be desirable not to move the lens too abruptly, but\n> >> + when in a preview mode (waiting for a still capture) it may be\n> >> + helpful to move the lens as quickly as is reasonably possible.\n> >> + enum:\n> >> + - name: AfSpeedFast\n> >> + value: 0\n> >> + description: Move the lens quickly.\n> >> + - name: AfSpeedSlow\n> >> + value: 0\n> >> + description: Move the lens more steadily.\n> >> +\n> >> + - AfMethod:\n> >> + type: int32_t\n> >> + draft: true\n> >> + description: |\n> >> + Control whether the AF algorithm uses a single window in the image to\n> >> + determine the best focus position, or multiple windows simultaneously.\n> >> + enum:\n> >> + - name: AfMethodSingle\n> >> + value: 0\n> >> + description: |\n> >> + A single window within the image, defaulting to the centre, is used\n> >> + to select the best focus distance.\n> >> + - name: AfMethodMultiSpot\n> >> + value: 0\n> >> + description: |\n> >> + Multiple windows within the image are used to select the best focus\n> >> + distance. The best focus distance is found for each one of the\n> >> + windows, and then the distance that is closest to the camera is\n> >> + selected.\n> >\n> >\n> > Would AFfMethod be used in continuous focus modes as well? if yes, then perhaps\n> > s/AF algorithm/focus algorithms/? Ditto for AfWindow.\n\nGood point. I don't see how AfMethod could work for CAF so it would be\nfor \"auto\" AF only. I think setting the (single) window size for CAF\nis something you might still want to do.\n\n> >\n> >>\n> >> +\n> >> + - AfWindow:\n> >> + type: Rectangle\n> >> + draft: true\n> >> + description: |\n> >> + Sets the focus windows used by the AF algorithm. The units used express\n> >> + a proportion of the ScalerCrop control (or if unavailable, of the entire\n> >> + image), as u0.16 format numbers.\n> >> +\n> >> + In order to be activated, a rectangle must be programmed with non-zero\n> >> + width and height. If no rectangles are programmed in this way, then the\n> >> + system will choose its own single default window in the centre of the\n> >> + image.\n> >> +\n> >> + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n> >> + this list is used (or the system default one if it is unprogrammed).\n> >> +\n> >> + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n> >> + this list are used. The size of the control indicates how many such\n> >> + windows can be programmed and will vary between different platforms.\n> >> +\n> >> + size: [platform dependent]\n> >> +\n> >> + - AfTrigger:\n> >> + type: int32_t\n> >> + draft: true\n> >> + description: |\n> >> + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n> >> + and can also be used to terminate a scan early.\n> >> +\n> >> + It is ignored if AfMode is set to AfModeContinuous.\n> >> +\n> >> + enum:\n> >> + - name: AfTriggerStart\n> >> + value: 0\n> >> + description: Start an AF scan. Ignored if a scan is in progress.\n> >> + - name: AfTriggerCancel\n> >> + value: 1\n> >> + description: Cancel an AF scan. Ingored if no scan is in progress.\n> >> +\n> >> + - LensPosition:\n> >> + type: int32_t\n> >> + draft: true\n> >> + description: |\n> >> + Acts as a control to instruct the lens to move to a particular position\n> >> + and also reports back the position of the lens for each frame.\n> >> +\n> >> + The units are determined by the lens driver.\n> >\n> >\n> > This would mean applications might not see the same/consistent behavior with\n> > different lens modules. Should we define a libcamera range (e.g. 0 -> Infinity,\n> > 1023 -> Macro) and convert to/from lens driver units in the IPA or pipeline handler?\n\nGood question. The answer isn't really clear to me. Would we also want\nto define a fixed value for hyperfocal? On the other hand, if the lens\nposition control min/default/max advertises all these values, maybe\nthat's enough? One thing I'm not clear on is whether you could tell\nwhich end of the driver's range is macro and which is infinity...\nmaybe there does have to be some database/tuning magic that tells us\nthis/\n\n> >\n> >>\n> >> +\n> >> + If the LensPosition control is used while an AF scan is in progress,\n> >> + the scan is implicitly cancelled and the lens is then moved to the\n> >> + desired location. It is ignored if AfMode is set to AfModeContinuous.\n> >> +\n> >> + - AfState:\n> >> + type: int32_t\n> >> + draft: true\n> >> + description: |\n> >> + Reports the current state of the AF algorithm.\n> >> + enum:\n> >> + - name: AfStateReset\n> >> + value: 0\n> >> + description: |\n> >> + The AF algorithm reports this state when:\n> >> + * The camera system has just been started.\n> >> + * A scan has been cancelled.\n> >> + * The lens has been moved by the LensPosition control.\n> >> + - name: AfStateScanning\n> >> + value: 1\n> >> + description: |\n> >> + AF is performing a scan. This state can be entered spontaneously\n> >> + if AfMode is set to AfModeContinuous, otherwise it requires the\n> >> + application to use the AfTrigger control to start the scan.\n> >> + - name: AfStateFocused\n> >> + value: 2\n> >> + description: |\n> >> + An AF scan has been performed and the algorithm believes the\n> >> + scene is in focus.\n> >> + - name: AfStateFailed\n> >> + value: 3\n> >> + description: |\n> >> + An AF scan has been performed but the algorithm has not been able\n> >> + to find the best focus position.\n> >> +\n> >\n> >\n> > One last point, we currently have a FocusFoM control that returns a focus measure\n> > to the application. In light of the new AfWindow measure, should that perhaps be\n> > redefined?\n\nProbably!\n\nThanks!\n\nDavid\n\n> >\n> > Regards,\n> > Naush\n> >\n> >\n> >>\n> >> ...\n> >> --\n> >> 2.30.2\n> >>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C57DEBDB13\n\tfor ;\n\tFri, 17 Dec 2021 08:40:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C9C9260894;\n\tFri, 17 Dec 2021 09:40:42 +0100 (CET)","from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com\n\t[IPv6:2a00:1450:4864:20::42c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C57F760223\n\tfor ;\n\tFri, 17 Dec 2021 09:40:40 +0100 (CET)","by mail-wr1-x42c.google.com with SMTP id q16so2598854wrg.7\n\tfor ;\n\tFri, 17 Dec 2021 00:40:40 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"WQRDa6wV\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=TxvHGLLTntxRiWOce4SAd4r1NFrFQvcMtfHfC+06QkY=;\n\tb=WQRDa6wV5q8hPzY5tlQtS9a8uSGeYZKOX50MKB3lOehzD+Fv5FHmUhsryDgSU5eUMi\n\trBHEF2vtzACXpoI4e1tuQ0CKIzZa0M4llT4ozlK7VBVFJ6P9r614e10tO1R/r0Zcx4rO\n\trXVaj5yt4MbXUCK0s4Q5VL19tH90qDgVVcVytLr5QZvtQKIOZgF95pX+nTLPoqHrXUxW\n\tbHagK2zaW532NrsppnExUfKEhXLQS/+YjZtDZ/9s8sZqNKmRXkvaRQjFZ3Fa8ON9XWlv\n\tddR/YPa5EC5TxeSfAmGCUpQPhUNxwLGUWbN+JaUvmbMdH8FkKyh2sz2B7kgPKiHR3kio\n\tzpxA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=TxvHGLLTntxRiWOce4SAd4r1NFrFQvcMtfHfC+06QkY=;\n\tb=VOA1+6UN3g9AyA3FGXNbbDtJ1AAIjjlDHRj4XpHSSQLBtfdinlgTcmjebloaAcLPlh\n\t1Rgd3Tmfk3Zg3cbYxnabM8eJwHIHCVCCfe6kBYruqSO/ifdRlJUmT3LhP/YpSWy6btK8\n\t2v2YcC+1hfKapqq66Ta7qxaIZBVPdhJ+LPU7RSL4dAb9HtbLcWg/4/1mHaPRAUZkSc0I\n\tFwfZsy8bwyherfYp1E/RYiLZMXO6LBItUcx9CxYAsKzHSIW0qidERGMQ1aw6wb8GnlNO\n\tCA/76wbWrLhOMZZMG5bxHYxH86LWQdo+/XORSRXJZhiPeWp3hjyF+omJqIW+uZQ0Ds4a\n\tD23Q==","X-Gm-Message-State":"AOAM530WrZrRIH+aCIMIRqhYuNBNh2dOgPgOGWP5sc3GS6ls/qEabywQ\n\t8iuerfwThAhxfGdBF0dDPSXX+EiPJc3R0HZmWLIFNA==","X-Google-Smtp-Source":"ABdhPJxQoXNGUXu7p4hd5MxD/zVHh/ogAaR4yJu5s+LCXRbbB05UtoB6qJTk2+p9925ENQoEg1yueE4FyHSR+K3rAKU=","X-Received":"by 2002:adf:f801:: with SMTP id s1mr1582881wrp.3.1639730440099; \n\tFri, 17 Dec 2021 00:40:40 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t\n\t","In-Reply-To":"","From":"David Plowman ","Date":"Fri, 17 Dec 2021 08:40:29 +0000","Message-ID":"","To":"Hanlin Chen ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21815,"web_url":"https://patchwork.libcamera.org/comment/21815/","msgid":"","date":"2021-12-17T08:51:37","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi again\n\nThanks for the comments!\n\nOn Thu, 16 Dec 2021 at 11:18, Hanlin Chen wrote:\n>\n> Hi David, thanks for this work!\n>\n> On Mon, Dec 13, 2021 at 11:22 PM David Plowman\n> wrote:\n> >\n> > This patch describes a series of controls that allow applications to\n> > drive AF algorithms:\n> >\n> > AfMode - auto or continuous\n> > AfRange - full, macro or normal\n> > AfSpeed - fast or slow\n> > AfMethod - single or multi-spot\n> > AfWindow - AF window locations\n> > AfTrigger - start (trigger an AF scan) or cancel\n> > LensPosition - position of lens from lens driver\n> > AfState - reset, scanning, focused or failed\n> > ---\n> > src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> > 1 file changed, 167 insertions(+), 60 deletions(-)\n> >\n> > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > index 9d4638ae..e579f7b7 100644\n> > --- a/src/libcamera/control_ids.yaml\n> > +++ b/src/libcamera/control_ids.yaml\n> > @@ -406,27 +406,6 @@ controls:\n> > The camera will cancel any active or completed metering sequence.\n> > The AE algorithm is reset to its initial state.\n> >\n> > - - AfTrigger:\n> > - type: int32_t\n> > - draft: true\n> > - description: |\n> > - Control for AF trigger. Currently identical to\n> > - ANDROID_CONTROL_AF_TRIGGER.\n> > -\n> > - Whether the camera device will trigger autofocus for this request.\n> > - enum:\n> > - - name: AfTriggerIdle\n> > - value: 0\n> > - description: The trigger is idle.\n> > - - name: AfTriggerStart\n> > - value: 1\n> > - description: The AF routine is started by the camera.\n> > - - name: AfTriggerCancel\n> > - value: 2\n> > - description: |\n> > - The camera will cancel any active trigger and the AF routine is\n> > - reset to its initial state.\n> > -\n> > - NoiseReductionMode:\n> > type: int32_t\n> > draft: true\n> > @@ -507,45 +486,6 @@ controls:\n> > The AE algorithm has started a pre-capture metering session.\n> > \\sa AePrecaptureTrigger\n> >\n> > - - AfState:\n> > - type: int32_t\n> > - draft: true\n> > - description: |\n> > - Control to report the current AF algorithm state. Currently identical to\n> > - ANDROID_CONTROL_AF_STATE.\n> > -\n> > - Current state of the AF algorithm.\n> > - enum:\n> > - - name: AfStateInactive\n> > - value: 0\n> > - description: The AF algorithm is inactive.\n> > - - name: AfStatePassiveScan\n> > - value: 1\n> > - description: |\n> > - AF is performing a passive scan of the scene in continuous\n> > - auto-focus mode.\n> > - - name: AfStatePassiveFocused\n> > - value: 2\n> > - description: |\n> > - AF believes the scene is in focus, but might restart scanning.\n> > - - name: AfStateActiveScan\n> > - value: 3\n> > - description: |\n> > - AF is performing a scan triggered by an AF trigger request.\n> > - \\sa AfTrigger\n> > - - name: AfStateFocusedLock\n> > - value: 4\n> > - description: |\n> > - AF believes has focused correctly and has locked focus.\n> > - - name: AfStateNotFocusedLock\n> > - value: 5\n> > - description: |\n> > - AF has not been able to focus and has locked.\n> > - - name: AfStatePassiveUnfocused\n> > - value: 6\n> > - description: |\n> > - AF has completed a passive scan without finding focus.\n> > -\n> > - AwbState:\n> > type: int32_t\n> > draft: true\n> > @@ -690,4 +630,171 @@ controls:\n> > value. All of the custom test patterns will be static (that is the\n> > raw image must not vary from frame to frame).\n> >\n> > + - AfMode:\n> > + type: int32_t\n> > + draft: true\n> > + description: |\n> > + Control to set the mode of the AF (autofocus) algorithm.\n> > + enum:\n> > + - name: AfModeAuto\n> > + value: 0\n> > + description: |\n> > + The AF algorithm is in auto mode. This means that the algorithm\n> > + will never move the lens or change state unless the AfTrigger\n> > + control is used. The AfTrigger control can be used to initiate a\n> > + focus scan, the results of which will be reported by the AfState.\n> > +\n> > + In this mode, an application can move the lens for itself using\n> > + the LensPosition control. If the algorithm was scanning when the\n> > + lens is moved in this way, then the scan is implicitly cancelled.\n>\n> I would suggest splitting the Auto mode into AfModeOnDemand and\n> AfModeManual (The naming could be re-considered).\n> The reason is that the sub-controls for each AfModes could be distinct.\n> 1. AfModeManual:\n> LensPosition is valid only for the mode, and AfState is always AfStateReset.\n> 2. AfModeOnDemand\n> AfTrigger and AfRange are valid only for the mode, and AfState reports\n> accordingly. The scan is implicitly canceled when changing from the\n> mode.\n\nI'm a little bit nervous about having a mode where the lens position\ncontrol doesn't work. In my earlier email I described performing an AF\nscan, capturing an image, then moving the lens back to hyperfocal. It\nseems inconvenient to have to put the AF algorithm into a separate\nmode in order to move the lens, only to put it back in the mode where\na scan is once again allowed.\n\nI don't think anything is lost by doing it this way - if an\napplication doesn't want to perform an AF scan it should simply not\nrequest one.\n\nI think AfRange applies to CAF as well. I see no reason why you might\nnot want to restrict the range of searches for just the same reasons\nas you might do with \"regular\" AF.\n\n> 3. AfModeContinuous\n> AfSpeed and \"pause\" are valid only for the mode, and AfState reports\n> accordingly. The scan is implicitly canceled when changing from the\n> mode.\n\nI'm not sure AfSpeed is *necessarily* only for CAF. I think we've done\nAF systems in the past where even regular AF has a \"I really want to\ntake the picture quickly\" mode. But it could be a platform choice?\n\n>\n> > + - name: AfModeContinuous\n> > + value: 1\n> > + description: |\n> > + The AF algorithm is in continuous mode. This means that the lens\n> > + can re-start a scan spontaneously at any moment, without any user\n> > + intervention. The AfState still reports whether the algorithm is\n> > + currently scanning or not, though the application has no ability\n> > + to initiate or cancel scans, nor move the lens for itself.\n> > +\n> > + The mode can be set to AfModeAuto which has the effect of\n> > + \"pausing\" any continuous AF activity, after which it could then\n> > + be moved back to AfModeContinuous to resume operation.\n> > +\n> > + - AfRange:\n> > + type: int32_t\n> > + draft: true\n> > + description: |\n> > + Control to set the range of focus distances that is scanned.\n> > + enum:\n> > + - name: AfRangeFull\n> > + value: 0\n> > + description: The full range of focus distances is scanned.\n> > + - name: AfRangeMacro\n> > + value: 1\n> > + description: Only close distances are scanned.\n> > + - name: AfRangeNormal\n> > + value: 2\n> > + description: |\n> > + The full range of focus distances is scanned except for some of\n> > + the closest macro positions.\n> > +\n> > + - AfSpeed:\n> > + type: int32_t\n> > + draft: true\n> > + description: |\n> > + Control that determines whether the AF algorithm is to move the lens\n> > + as quickly as possible or more steadily. For example, during video\n> > + recording it may be desirable not to move the lens too abruptly, but\n> > + when in a preview mode (waiting for a still capture) it may be\n> > + helpful to move the lens as quickly as is reasonably possible.\n> > + enum:\n> > + - name: AfSpeedFast\n> > + value: 0\n> > + description: Move the lens quickly.\n> > + - name: AfSpeedSlow\n> > + value: 0\n> > + description: Move the lens more steadily.\n> How about\n> s/Fast/Aggressively\n> s/Slow/Steadily\n\nI agree the choice is tricky here. I think I would regard what I\ncalled \"Slow\" to be the \"usual\" choice, so perhaps \"AfSpeedNormal\" is\nbetter (and maybe make that value 0). I still prefer \"AfModeFast\" to\n\"AfModeAggressive\" - \"aggressive\" sounds just a bit too... aggressive?\n:)\n\nBut thanks for all the discussion. Glad we're making progress on all this!\n\nBest regards\nDavid\n\n> > +\n> > + - AfMethod:\n> > + type: int32_t\n> > + draft: true\n> > + description: |\n> > + Control whether the AF algorithm uses a single window in the image to\n> > + determine the best focus position, or multiple windows simultaneously.\n> > + enum:\n> > + - name: AfMethodSingle\n> > + value: 0\n> > + description: |\n> > + A single window within the image, defaulting to the centre, is used\n> > + to select the best focus distance.\n> > + - name: AfMethodMultiSpot\n> > + value: 0\n> > + description: |\n> > + Multiple windows within the image are used to select the best focus\n> > + distance. The best focus distance is found for each one of the\n> > + windows, and then the distance that is closest to the camera is\n> > + selected.\n> > +\n> > + - AfWindow:\n> > + type: Rectangle\n> > + draft: true\n> > + description: |\n> > + Sets the focus windows used by the AF algorithm. The units used express\n> > + a proportion of the ScalerCrop control (or if unavailable, of the entire\n> > + image), as u0.16 format numbers.\n> > +\n> > + In order to be activated, a rectangle must be programmed with non-zero\n> > + width and height. If no rectangles are programmed in this way, then the\n> > + system will choose its own single default window in the centre of the\n> > + image.\n> > +\n> > + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n> > + this list is used (or the system default one if it is unprogrammed).\n> > +\n> > + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n> > + this list are used. The size of the control indicates how many such\n> > + windows can be programmed and will vary between different platforms.\n> > +\n> > + size: [platform dependent]\n> > +\n> > + - AfTrigger:\n> > + type: int32_t\n> > + draft: true\n> > + description: |\n> > + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n> > + and can also be used to terminate a scan early.\n> > +\n> > + It is ignored if AfMode is set to AfModeContinuous.\n> > +\n> > + enum:\n> > + - name: AfTriggerStart\n> > + value: 0\n> > + description: Start an AF scan. Ignored if a scan is in progress.\n> > + - name: AfTriggerCancel\n> > + value: 1\n> > + description: Cancel an AF scan. Ingored if no scan is in progress.\n> > +\n> > + - LensPosition:\n> > + type: int32_t\n> > + draft: true\n> > + description: |\n> > + Acts as a control to instruct the lens to move to a particular position\n> > + and also reports back the position of the lens for each frame.\n> > +\n> > + The units are determined by the lens driver.\n> > +\n> > + If the LensPosition control is used while an AF scan is in progress,\n> > + the scan is implicitly cancelled and the lens is then moved to the\n> > + desired location. It is ignored if AfMode is set to AfModeContinuous.\n> > +\n> > + - AfState:\n> > + type: int32_t\n> > + draft: true\n> > + description: |\n> > + Reports the current state of the AF algorithm.\n> > + enum:\n> > + - name: AfStateReset\n> > + value: 0\n> > + description: |\n> > + The AF algorithm reports this state when:\n> > + * The camera system has just been started.\n> > + * A scan has been cancelled.\n> > + * The lens has been moved by the LensPosition control.\n> > + - name: AfStateScanning\n> > + value: 1\n> > + description: |\n> > + AF is performing a scan. This state can be entered spontaneously\n> > + if AfMode is set to AfModeContinuous, otherwise it requires the\n> > + application to use the AfTrigger control to start the scan.\n> > + - name: AfStateFocused\n> > + value: 2\n> > + description: |\n> > + An AF scan has been performed and the algorithm believes the\n> > + scene is in focus.\n> > + - name: AfStateFailed\n> > + value: 3\n> > + description: |\n> > + An AF scan has been performed but the algorithm has not been able\n> > + to find the best focus position.\n> > +\n> > ...\n> > --\n> > 2.30.2\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 46CE8BF415\n\tfor ;\n\tFri, 17 Dec 2021 08:51:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9C365608A2;\n\tFri, 17 Dec 2021 09:51:49 +0100 (CET)","from mail-wr1-x432.google.com (mail-wr1-x432.google.com\n\t[IPv6:2a00:1450:4864:20::432])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AE13B60223\n\tfor ;\n\tFri, 17 Dec 2021 09:51:48 +0100 (CET)","by mail-wr1-x432.google.com with SMTP id j18so2696288wrd.2\n\tfor ;\n\tFri, 17 Dec 2021 00:51:48 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"gkrpyN66\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=53XbqZ9gQpzBxePk3uhZzLv8Eb405PZ/rvL1osIqM6A=;\n\tb=gkrpyN662M0rrQ4UiQ7KdFDjPaYWpUygWI/dgtLa1YmkL7UCfaQUXcuQEupWwJQcXm\n\tYEtAGQoCq7SXXcJrZwj5/9moQKS8kAZsixfU+TRON/SH7oE6Mjnan21d0/MIDceI2h07\n\tdY7xxMXEKhRYOnCVfMOXbDP9FqebROW+QvLNM09jCf41A08ye/E6ZQU44m6wmLJbERwb\n\tux/OeD+LvP37mfBBzDaHKjyxilFLic27pLkD7UDBQSMyiLBOVNxDgcA8C0xRZNAUp/44\n\tAiGN9H8fh+GfEVu8FLfBB3hxaxkHcq8WcpClZrSrwacu76LVi8lhEn3M00V71hgTmc4K\n\tPYow==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=53XbqZ9gQpzBxePk3uhZzLv8Eb405PZ/rvL1osIqM6A=;\n\tb=0WV9H+kiJuwMdFKdgrC9jXQxbkvQi3sEVFo1QikKJH4aEd98B17VjlilNVmzuWNr6t\n\tk4CSByTvkS+tW0QRwZVAcfrwgmFLfq1ix3U4FY5Qh0yBmiZ22cZNcUv5Vw9ih9uJM2qo\n\tZT/M3PrGWn/1ssojnuBOUjcCIes1ABRJ1ZaKr+WfCtLSADlgPa6DXmtzvHx4CK3QntdQ\n\tK8XgVrZlZbpWygsjxjjurLEa1MbaPFqtfElrUtt9TjlPLhsgIJ3URKcWv25zNFFztt0o\n\t8k/MzA8tTLLkYo05ZNBg0qgqi+izW00ADmby3vKqUc3KAoe6qcRlSNa3ZR5Fk129/Uhb\n\tAdaA==","X-Gm-Message-State":"AOAM532Qeta7/gdKki9uSPJptnvcH6ExTrmEiqTTPAIyalYlD+JGrk/H\n\tfGnrERZRxtTBk7SoaQrGiw+iY4lJeo+Nk1xVCmGaTsltkjR+qw==","X-Google-Smtp-Source":"ABdhPJwcmzlN+Yw6JuCzasyCsr+flk66R+2KDZeZfOIkEzUa9IHmdK2kX2EcwGJeFT9q86WqJpeFk8taiRTHRMM2FPA=","X-Received":"by 2002:adf:f801:: with SMTP id s1mr1620162wrp.3.1639731108237; \n\tFri, 17 Dec 2021 00:51:48 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t","In-Reply-To":"","From":"David Plowman ","Date":"Fri, 17 Dec 2021 08:51:37 +0000","Message-ID":"","To":"Hanlin Chen ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21842,"web_url":"https://patchwork.libcamera.org/comment/21842/","msgid":"","date":"2021-12-21T10:20:49","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":98,"url":"https://patchwork.libcamera.org/api/people/98/","name":"Hanlin Chen","email":"hanlinchen@chromium.org"},"content":"Hi David, thanks for the comments.\n\nOn Fri, Dec 17, 2021 at 4:51 PM David Plowman\n wrote:\n>\n> Hi again\n>\n> Thanks for the comments!\n>\n> On Thu, 16 Dec 2021 at 11:18, Hanlin Chen wrote:\n> >\n> > Hi David, thanks for this work!\n> >\n> > On Mon, Dec 13, 2021 at 11:22 PM David Plowman\n> > wrote:\n> > >\n> > > This patch describes a series of controls that allow applications to\n> > > drive AF algorithms:\n> > >\n> > > AfMode - auto or continuous\n> > > AfRange - full, macro or normal\n> > > AfSpeed - fast or slow\n> > > AfMethod - single or multi-spot\n> > > AfWindow - AF window locations\n> > > AfTrigger - start (trigger an AF scan) or cancel\n> > > LensPosition - position of lens from lens driver\n> > > AfState - reset, scanning, focused or failed\n> > > ---\n> > > src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> > > 1 file changed, 167 insertions(+), 60 deletions(-)\n> > >\n> > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > index 9d4638ae..e579f7b7 100644\n> > > --- a/src/libcamera/control_ids.yaml\n> > > +++ b/src/libcamera/control_ids.yaml\n> > > @@ -406,27 +406,6 @@ controls:\n> > > The camera will cancel any active or completed metering sequence.\n> > > The AE algorithm is reset to its initial state.\n> > >\n> > > - - AfTrigger:\n> > > - type: int32_t\n> > > - draft: true\n> > > - description: |\n> > > - Control for AF trigger. Currently identical to\n> > > - ANDROID_CONTROL_AF_TRIGGER.\n> > > -\n> > > - Whether the camera device will trigger autofocus for this request.\n> > > - enum:\n> > > - - name: AfTriggerIdle\n> > > - value: 0\n> > > - description: The trigger is idle.\n> > > - - name: AfTriggerStart\n> > > - value: 1\n> > > - description: The AF routine is started by the camera.\n> > > - - name: AfTriggerCancel\n> > > - value: 2\n> > > - description: |\n> > > - The camera will cancel any active trigger and the AF routine is\n> > > - reset to its initial state.\n> > > -\n> > > - NoiseReductionMode:\n> > > type: int32_t\n> > > draft: true\n> > > @@ -507,45 +486,6 @@ controls:\n> > > The AE algorithm has started a pre-capture metering session.\n> > > \\sa AePrecaptureTrigger\n> > >\n> > > - - AfState:\n> > > - type: int32_t\n> > > - draft: true\n> > > - description: |\n> > > - Control to report the current AF algorithm state. Currently identical to\n> > > - ANDROID_CONTROL_AF_STATE.\n> > > -\n> > > - Current state of the AF algorithm.\n> > > - enum:\n> > > - - name: AfStateInactive\n> > > - value: 0\n> > > - description: The AF algorithm is inactive.\n> > > - - name: AfStatePassiveScan\n> > > - value: 1\n> > > - description: |\n> > > - AF is performing a passive scan of the scene in continuous\n> > > - auto-focus mode.\n> > > - - name: AfStatePassiveFocused\n> > > - value: 2\n> > > - description: |\n> > > - AF believes the scene is in focus, but might restart scanning.\n> > > - - name: AfStateActiveScan\n> > > - value: 3\n> > > - description: |\n> > > - AF is performing a scan triggered by an AF trigger request.\n> > > - \\sa AfTrigger\n> > > - - name: AfStateFocusedLock\n> > > - value: 4\n> > > - description: |\n> > > - AF believes has focused correctly and has locked focus.\n> > > - - name: AfStateNotFocusedLock\n> > > - value: 5\n> > > - description: |\n> > > - AF has not been able to focus and has locked.\n> > > - - name: AfStatePassiveUnfocused\n> > > - value: 6\n> > > - description: |\n> > > - AF has completed a passive scan without finding focus.\n> > > -\n> > > - AwbState:\n> > > type: int32_t\n> > > draft: true\n> > > @@ -690,4 +630,171 @@ controls:\n> > > value. All of the custom test patterns will be static (that is the\n> > > raw image must not vary from frame to frame).\n> > >\n> > > + - AfMode:\n> > > + type: int32_t\n> > > + draft: true\n> > > + description: |\n> > > + Control to set the mode of the AF (autofocus) algorithm.\n> > > + enum:\n> > > + - name: AfModeAuto\n> > > + value: 0\n> > > + description: |\n> > > + The AF algorithm is in auto mode. This means that the algorithm\n> > > + will never move the lens or change state unless the AfTrigger\n> > > + control is used. The AfTrigger control can be used to initiate a\n> > > + focus scan, the results of which will be reported by the AfState.\n> > > +\n> > > + In this mode, an application can move the lens for itself using\n> > > + the LensPosition control. If the algorithm was scanning when the\n> > > + lens is moved in this way, then the scan is implicitly cancelled.\n> >\n> > I would suggest splitting the Auto mode into AfModeOnDemand and\n> > AfModeManual (The naming could be re-considered).\n> > The reason is that the sub-controls for each AfModes could be distinct.\n> > 1. AfModeManual:\n> > LensPosition is valid only for the mode, and AfState is always AfStateReset.\n> > 2. AfModeOnDemand\n> > AfTrigger and AfRange are valid only for the mode, and AfState reports\n> > accordingly. The scan is implicitly canceled when changing from the\n> > mode.\n>\n> I'm a little bit nervous about having a mode where the lens position\n> control doesn't work. In my earlier email I described performing an AF\n> scan, capturing an image, then moving the lens back to hyperfocal. It\n> seems inconvenient to have to put the AF algorithm into a separate\n> mode in order to move the lens, only to put it back in the mode where\n> a scan is once again allowed.\n>\n> I don't think anything is lost by doing it this way - if an\n> application doesn't want to perform an AF scan it should simply not\n> request one.\nI agreed that it does cover the controls from Android, i.e., the\nAndroid request can be translated to the API.\nMy concern is from another direction: translation from libcamera to\nAndroid Camera3.\nFrom the Android to libcamera:\nANDROID_CONTROL_AF_MODE_OFF + ANDROID_LENS_FOCUS_DISTANCE ==>\nAfModeAuto + lensPosition\nANDROID_CONTROL_AF_MODE_CONTINUOUS_VIDEO +\nANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_START ==> AfModeAuto +\nAfTrigger\nFrom libcamera to Android:\nAfModeAuto + lensPostion (Assuming current position is reported by\nlibcamera) ==> ??\nIt could be CAF, AUTO or OFF mode.\nStates from libcamera Android:\nAfModeAuto + AfTrigger + focused ==>\nANDROID_CONTROL_AF_STATE_FOCUSED_LOCKED? or\nANDROID_CONTROL_AF_STATE_PASSIVE_FOCUSED?\n\nFor the Android adapter layer, above might be workarounded by caching\nthe requests and do reference and find out current context.\nA more subtle case might arise when integrating other platforms.\nFor example, Qualcomm closed 3A lib follows the modes and states\naccording to Android (Partly true for Intel and MTK).\nIn the case, the translation flow will like:\nAndroid API -> libcamera API -> Android like API for 3A lib ->\nlibcamera API -> Android API\n \n \nIt's hard to predict how to work around all corner cases.\nThis makes me believe that if we could have a clear one to one mapping\nbetween Android and libcamera API, it could be beneficial later.\nHowever, it will make the API more verbose.\n\nAnother perspective is that Android defines required state machine transition:\nhttps://source.android.com/devices/camera/camera3_3Amodes\n\nFor IPA to implement the state transition satisfying Android's\nrequirement, the IPA should know exactly how the libcamera states\nmapping back to the Android states.\nAbove is not satisfying because we will be somehow restricted by\nAndroid's definition ;-|, However, I think it could be beneficial if\nthe API cam map libcamera states/modes to all Android states/modes\ndirectly without extra context/call order information.\n\n>\n> I think AfRange applies to CAF as well. I see no reason why you might\n> not want to restrict the range of searches for just the same reasons\n> as you might do with \"regular\" AF.\nAgreed.\n>\n> > 3. AfModeContinuous\n> > AfSpeed and \"pause\" are valid only for the mode, and AfState reports\n> > accordingly. The scan is implicitly canceled when changing from the\n> > mode.\n>\n> I'm not sure AfSpeed is *necessarily* only for CAF. I think we've done\n> AF systems in the past where even regular AF has a \"I really want to\n> take the picture quickly\" mode. But it could be a platform choice?\nAgreed.\n>\n> >\n> > > + - name: AfModeContinuous\n> > > + value: 1\n> > > + description: |\n> > > + The AF algorithm is in continuous mode. This means that the lens\n> > > + can re-start a scan spontaneously at any moment, without any user\n> > > + intervention. The AfState still reports whether the algorithm is\n> > > + currently scanning or not, though the application has no ability\n> > > + to initiate or cancel scans, nor move the lens for itself.\n> > > +\n> > > + The mode can be set to AfModeAuto which has the effect of\n> > > + \"pausing\" any continuous AF activity, after which it could then\n> > > + be moved back to AfModeContinuous to resume operation.\n> > > +\n> > > + - AfRange:\n> > > + type: int32_t\n> > > + draft: true\n> > > + description: |\n> > > + Control to set the range of focus distances that is scanned.\n> > > + enum:\n> > > + - name: AfRangeFull\n> > > + value: 0\n> > > + description: The full range of focus distances is scanned.\n> > > + - name: AfRangeMacro\n> > > + value: 1\n> > > + description: Only close distances are scanned.\n> > > + - name: AfRangeNormal\n> > > + value: 2\n> > > + description: |\n> > > + The full range of focus distances is scanned except for some of\n> > > + the closest macro positions.\n> > > +\n> > > + - AfSpeed:\n> > > + type: int32_t\n> > > + draft: true\n> > > + description: |\n> > > + Control that determines whether the AF algorithm is to move the lens\n> > > + as quickly as possible or more steadily. For example, during video\n> > > + recording it may be desirable not to move the lens too abruptly, but\n> > > + when in a preview mode (waiting for a still capture) it may be\n> > > + helpful to move the lens as quickly as is reasonably possible.\n> > > + enum:\n> > > + - name: AfSpeedFast\n> > > + value: 0\n> > > + description: Move the lens quickly.\n> > > + - name: AfSpeedSlow\n> > > + value: 0\n> > > + description: Move the lens more steadily.\n> > How about\n> > s/Fast/Aggressively\n> > s/Slow/Steadily\n>\n> I agree the choice is tricky here. I think I would regard what I\n> called \"Slow\" to be the \"usual\" choice, so perhaps \"AfSpeedNormal\" is\n> better (and maybe make that value 0). I still prefer \"AfModeFast\" to\n> \"AfModeAggressive\" - \"aggressive\" sounds just a bit too... aggressive?\n> :)\nLGTM.\n>\n> But thanks for all the discussion. Glad we're making progress on all this!\n>\n> Best regards\n> David\n>\n> > > +\n> > > + - AfMethod:\n> > > + type: int32_t\n> > > + draft: true\n> > > + description: |\n> > > + Control whether the AF algorithm uses a single window in the image to\n> > > + determine the best focus position, or multiple windows simultaneously.\n> > > + enum:\n> > > + - name: AfMethodSingle\n> > > + value: 0\n> > > + description: |\n> > > + A single window within the image, defaulting to the centre, is used\n> > > + to select the best focus distance.\n> > > + - name: AfMethodMultiSpot\n> > > + value: 0\n> > > + description: |\n> > > + Multiple windows within the image are used to select the best focus\n> > > + distance. The best focus distance is found for each one of the\n> > > + windows, and then the distance that is closest to the camera is\n> > > + selected.\n> > > +\n> > > + - AfWindow:\n> > > + type: Rectangle\n> > > + draft: true\n> > > + description: |\n> > > + Sets the focus windows used by the AF algorithm. The units used express\n> > > + a proportion of the ScalerCrop control (or if unavailable, of the entire\n> > > + image), as u0.16 format numbers.\n> > > +\n> > > + In order to be activated, a rectangle must be programmed with non-zero\n> > > + width and height. If no rectangles are programmed in this way, then the\n> > > + system will choose its own single default window in the centre of the\n> > > + image.\n> > > +\n> > > + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n> > > + this list is used (or the system default one if it is unprogrammed).\n> > > +\n> > > + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n> > > + this list are used. The size of the control indicates how many such\n> > > + windows can be programmed and will vary between different platforms.\n> > > +\n> > > + size: [platform dependent]\n> > > +\n> > > + - AfTrigger:\n> > > + type: int32_t\n> > > + draft: true\n> > > + description: |\n> > > + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n> > > + and can also be used to terminate a scan early.\n> > > +\n> > > + It is ignored if AfMode is set to AfModeContinuous.\n> > > +\n> > > + enum:\n> > > + - name: AfTriggerStart\n> > > + value: 0\n> > > + description: Start an AF scan. Ignored if a scan is in progress.\n> > > + - name: AfTriggerCancel\n> > > + value: 1\n> > > + description: Cancel an AF scan. Ingored if no scan is in progress.\n> > > +\n> > > + - LensPosition:\n> > > + type: int32_t\n> > > + draft: true\n> > > + description: |\n> > > + Acts as a control to instruct the lens to move to a particular position\n> > > + and also reports back the position of the lens for each frame.\n> > > +\n> > > + The units are determined by the lens driver.\n> > > +\n> > > + If the LensPosition control is used while an AF scan is in progress,\n> > > + the scan is implicitly cancelled and the lens is then moved to the\n> > > + desired location. It is ignored if AfMode is set to AfModeContinuous.\n> > > +\n> > > + - AfState:\n> > > + type: int32_t\n> > > + draft: true\n> > > + description: |\n> > > + Reports the current state of the AF algorithm.\n> > > + enum:\n> > > + - name: AfStateReset\n> > > + value: 0\n> > > + description: |\n> > > + The AF algorithm reports this state when:\n> > > + * The camera system has just been started.\n> > > + * A scan has been cancelled.\n> > > + * The lens has been moved by the LensPosition control.\n> > > + - name: AfStateScanning\n> > > + value: 1\n> > > + description: |\n> > > + AF is performing a scan. This state can be entered spontaneously\n> > > + if AfMode is set to AfModeContinuous, otherwise it requires the\n> > > + application to use the AfTrigger control to start the scan.\n> > > + - name: AfStateFocused\n> > > + value: 2\n> > > + description: |\n> > > + An AF scan has been performed and the algorithm believes the\n> > > + scene is in focus.\n> > > + - name: AfStateFailed\n> > > + value: 3\n> > > + description: |\n> > > + An AF scan has been performed but the algorithm has not been able\n> > > + to find the best focus position.\n> > > +\n> > > ...\n> > > --\n> > > 2.30.2\n> > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id AC5E4BF415\n\tfor ;\n\tTue, 21 Dec 2021 10:21:04 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D67E0608A2;\n\tTue, 21 Dec 2021 11:21:03 +0100 (CET)","from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com\n\t[IPv6:2607:f8b0:4864:20::32e])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4129360113\n\tfor ;\n\tTue, 21 Dec 2021 11:21:02 +0100 (CET)","by mail-ot1-x32e.google.com with SMTP id\n\tv15-20020a9d604f000000b0056cdb373b82so16070591otj.7\n\tfor ;\n\tTue, 21 Dec 2021 02:21:02 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"clSnDFUR\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=rvu7lSvZuJb13JkhSwnHs9Bql4utCzgWLJRnsagzMEY=;\n\tb=clSnDFURxK3GO6crX6yg1Wz5rsotQ+MRpU7RQ8g2XJtWf40GiXRad4VfxkPSp+g26p\n\t9mppOQvhWPtxfNSj+IbMdfdpNzlzgcgQb6C66XMiLahwKOkEwIdKoCtPCztZipYFsAfE\n\tzXPVQhO+Jkki3XsWiVKM3mMe/7VR3ZVRkDX6Y=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=rvu7lSvZuJb13JkhSwnHs9Bql4utCzgWLJRnsagzMEY=;\n\tb=fQFQkBg4uDFO1HOBWGBXZMnwRxq9yr0Up24gb7x9yN65l3E5xis+l6ZskdrOAxACte\n\tUHIXVTUG+b1aP3ruOhPkIGJ+z3ndSkM1Z8Ue91CfpJLD2/e9bagrVqtltsxcozyKJd8P\n\tHXcd6mnijyXDuHU+KZl8JNktRMqfJ9N6fCgtgRX3jo8qK0NQKnLwK6oNuYg2CerfQNOA\n\tIwsUdIVxiB/n1xhrhp954pehC0tfRFHPmvZITt8DS6XPi+8RRx9eQFvm7k70iDenjkgk\n\tn3sd77PPVpRTl8hTDSvCbPmmFoDfdbqf6AQ8iZpoaUer4xYYe5bmIylOxuDtJbGW502K\n\tJQBg==","X-Gm-Message-State":"AOAM531PpYYxKGHBebMFujwAMGVZ1V5RLz3EDg3XVmvc4boXrbq2z20m\n\tfVNo3WiCUz9dt4lhlPyApCyGigV/egTIDjicSw0DUA==","X-Google-Smtp-Source":"ABdhPJw3dw9cHp1Rg/qQBZmswUuT1DH2kOvTqP/Q3yz3layhYGMV1vMCkF6s3650I6j4b/j9COdQuU+myGFEW+tvT7E=","X-Received":"by 2002:a9d:1726:: with SMTP id\n\ti38mr1641142ota.176.1640082060606; \n\tTue, 21 Dec 2021 02:21:00 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t\n\t","In-Reply-To":"","From":"Hanlin Chen ","Date":"Tue, 21 Dec 2021 18:20:49 +0800","Message-ID":"","To":"David Plowman ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22027,"web_url":"https://patchwork.libcamera.org/comment/22027/","msgid":"","date":"2022-01-13T17:54:20","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Hanlin\n\nSorry for taking some time to get back to this!\n\nOn Tue, 21 Dec 2021 at 10:21, Hanlin Chen wrote:\n>\n> Hi David, thanks for the comments.\n>\n> On Fri, Dec 17, 2021 at 4:51 PM David Plowman\n> wrote:\n> >\n> > Hi again\n> >\n> > Thanks for the comments!\n> >\n> > On Thu, 16 Dec 2021 at 11:18, Hanlin Chen wrote:\n> > >\n> > > Hi David, thanks for this work!\n> > >\n> > > On Mon, Dec 13, 2021 at 11:22 PM David Plowman\n> > > wrote:\n> > > >\n> > > > This patch describes a series of controls that allow applications to\n> > > > drive AF algorithms:\n> > > >\n> > > > AfMode - auto or continuous\n> > > > AfRange - full, macro or normal\n> > > > AfSpeed - fast or slow\n> > > > AfMethod - single or multi-spot\n> > > > AfWindow - AF window locations\n> > > > AfTrigger - start (trigger an AF scan) or cancel\n> > > > LensPosition - position of lens from lens driver\n> > > > AfState - reset, scanning, focused or failed\n> > > > ---\n> > > > src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> > > > 1 file changed, 167 insertions(+), 60 deletions(-)\n> > > >\n> > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > index 9d4638ae..e579f7b7 100644\n> > > > --- a/src/libcamera/control_ids.yaml\n> > > > +++ b/src/libcamera/control_ids.yaml\n> > > > @@ -406,27 +406,6 @@ controls:\n> > > > The camera will cancel any active or completed metering sequence.\n> > > > The AE algorithm is reset to its initial state.\n> > > >\n> > > > - - AfTrigger:\n> > > > - type: int32_t\n> > > > - draft: true\n> > > > - description: |\n> > > > - Control for AF trigger. Currently identical to\n> > > > - ANDROID_CONTROL_AF_TRIGGER.\n> > > > -\n> > > > - Whether the camera device will trigger autofocus for this request.\n> > > > - enum:\n> > > > - - name: AfTriggerIdle\n> > > > - value: 0\n> > > > - description: The trigger is idle.\n> > > > - - name: AfTriggerStart\n> > > > - value: 1\n> > > > - description: The AF routine is started by the camera.\n> > > > - - name: AfTriggerCancel\n> > > > - value: 2\n> > > > - description: |\n> > > > - The camera will cancel any active trigger and the AF routine is\n> > > > - reset to its initial state.\n> > > > -\n> > > > - NoiseReductionMode:\n> > > > type: int32_t\n> > > > draft: true\n> > > > @@ -507,45 +486,6 @@ controls:\n> > > > The AE algorithm has started a pre-capture metering session.\n> > > > \\sa AePrecaptureTrigger\n> > > >\n> > > > - - AfState:\n> > > > - type: int32_t\n> > > > - draft: true\n> > > > - description: |\n> > > > - Control to report the current AF algorithm state. Currently identical to\n> > > > - ANDROID_CONTROL_AF_STATE.\n> > > > -\n> > > > - Current state of the AF algorithm.\n> > > > - enum:\n> > > > - - name: AfStateInactive\n> > > > - value: 0\n> > > > - description: The AF algorithm is inactive.\n> > > > - - name: AfStatePassiveScan\n> > > > - value: 1\n> > > > - description: |\n> > > > - AF is performing a passive scan of the scene in continuous\n> > > > - auto-focus mode.\n> > > > - - name: AfStatePassiveFocused\n> > > > - value: 2\n> > > > - description: |\n> > > > - AF believes the scene is in focus, but might restart scanning.\n> > > > - - name: AfStateActiveScan\n> > > > - value: 3\n> > > > - description: |\n> > > > - AF is performing a scan triggered by an AF trigger request.\n> > > > - \\sa AfTrigger\n> > > > - - name: AfStateFocusedLock\n> > > > - value: 4\n> > > > - description: |\n> > > > - AF believes has focused correctly and has locked focus.\n> > > > - - name: AfStateNotFocusedLock\n> > > > - value: 5\n> > > > - description: |\n> > > > - AF has not been able to focus and has locked.\n> > > > - - name: AfStatePassiveUnfocused\n> > > > - value: 6\n> > > > - description: |\n> > > > - AF has completed a passive scan without finding focus.\n> > > > -\n> > > > - AwbState:\n> > > > type: int32_t\n> > > > draft: true\n> > > > @@ -690,4 +630,171 @@ controls:\n> > > > value. All of the custom test patterns will be static (that is the\n> > > > raw image must not vary from frame to frame).\n> > > >\n> > > > + - AfMode:\n> > > > + type: int32_t\n> > > > + draft: true\n> > > > + description: |\n> > > > + Control to set the mode of the AF (autofocus) algorithm.\n> > > > + enum:\n> > > > + - name: AfModeAuto\n> > > > + value: 0\n> > > > + description: |\n> > > > + The AF algorithm is in auto mode. This means that the algorithm\n> > > > + will never move the lens or change state unless the AfTrigger\n> > > > + control is used. The AfTrigger control can be used to initiate a\n> > > > + focus scan, the results of which will be reported by the AfState.\n> > > > +\n> > > > + In this mode, an application can move the lens for itself using\n> > > > + the LensPosition control. If the algorithm was scanning when the\n> > > > + lens is moved in this way, then the scan is implicitly cancelled.\n> > >\n> > > I would suggest splitting the Auto mode into AfModeOnDemand and\n> > > AfModeManual (The naming could be re-considered).\n> > > The reason is that the sub-controls for each AfModes could be distinct.\n> > > 1. AfModeManual:\n> > > LensPosition is valid only for the mode, and AfState is always AfStateReset.\n> > > 2. AfModeOnDemand\n> > > AfTrigger and AfRange are valid only for the mode, and AfState reports\n> > > accordingly. The scan is implicitly canceled when changing from the\n> > > mode.\n> >\n> > I'm a little bit nervous about having a mode where the lens position\n> > control doesn't work. In my earlier email I described performing an AF\n> > scan, capturing an image, then moving the lens back to hyperfocal. It\n> > seems inconvenient to have to put the AF algorithm into a separate\n> > mode in order to move the lens, only to put it back in the mode where\n> > a scan is once again allowed.\n> >\n> > I don't think anything is lost by doing it this way - if an\n> > application doesn't want to perform an AF scan it should simply not\n> > request one.\n> I agreed that it does cover the controls from Android, i.e., the\n> Android request can be translated to the API.\n> My concern is from another direction: translation from libcamera to\n> Android Camera3.\n> From the Android to libcamera:\n> ANDROID_CONTROL_AF_MODE_OFF + ANDROID_LENS_FOCUS_DISTANCE ==>\n> AfModeAuto + lensPosition\n> ANDROID_CONTROL_AF_MODE_CONTINUOUS_VIDEO +\n> ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_START ==> AfModeAuto +\n> AfTrigger\n\nI don't quite understand how an AE_PRECAPTURE_TRIGGER_START affects\nthe AF state machine, isn't that something we'd send to the AE\nalgorithm (if it had such a control)? And then we'd wait for the AE\nstate to say it was finished?\n\n> From libcamera to Android:\n> AfModeAuto + lensPostion (Assuming current position is reported by\n> libcamera) ==> ??\n> It could be CAF, AUTO or OFF mode.\n\nAs I understand it, Android only lets you set the lens position when\nAF mode is \"OFF\". So in this case we'd have to set the Android AF to\n\"OFF\" along with the lens position, right?\n\nIf the next thing libcamera wants to do is something other than move\nthe lens, then we'll have to set Android back to the state\ncorresponding to libcamera's. Does that work?\n\n> States from libcamera Android:\n> AfModeAuto + AfTrigger + focused ==>\n> ANDROID_CONTROL_AF_STATE_FOCUSED_LOCKED? or\n> ANDROID_CONTROL_AF_STATE_PASSIVE_FOCUSED?\n\nIf there was a trigger, then I believe the \"focused\" state needs to\ncorrespond to AF_STATE_FOCUSED_LOCKED. In fact, is it the case that in\nauto mode, Android can never report AF_STATE_PASSIVE_FOCUSED?\n\n>\n> For the Android adapter layer, above might be workarounded by caching\n> the requests and do reference and find out current context.\n> A more subtle case might arise when integrating other platforms.\n> For example, Qualcomm closed 3A lib follows the modes and states\n> according to Android (Partly true for Intel and MTK).\n> In the case, the translation flow will like:\n> Android API -> libcamera API -> Android like API for 3A lib ->\n> libcamera API -> Android API\n> \n> \n> It's hard to predict how to work around all corner cases.\n> This makes me believe that if we could have a clear one to one mapping\n> between Android and libcamera API, it could be beneficial later.\n> However, it will make the API more verbose.\n>\n> Another perspective is that Android defines required state machine transition:\n> https://source.android.com/devices/camera/camera3_3Amodes\n>\n> For IPA to implement the state transition satisfying Android's\n> requirement, the IPA should know exactly how the libcamera states\n> mapping back to the Android states.\n> Above is not satisfying because we will be somehow restricted by\n> Android's definition ;-|, However, I think it could be beneficial if\n> the API cam map libcamera states/modes to all Android states/modes\n> directly without extra context/call order information.\n\nI'm really in two minds about the question of an \"off\" mode in\nlibcamera. Either we make libcamera application code handle the\ntransition to \"off\" and back to \"auto\", or a libcamera-to-Android\ntranslation layer would have to do it. Does anyone else out there feel\nstrongly?\n\n>\n> >\n> > I think AfRange applies to CAF as well. I see no reason why you might\n> > not want to restrict the range of searches for just the same reasons\n> > as you might do with \"regular\" AF.\n> Agreed.\n> >\n> > > 3. AfModeContinuous\n> > > AfSpeed and \"pause\" are valid only for the mode, and AfState reports\n> > > accordingly. The scan is implicitly canceled when changing from the\n> > > mode.\n> >\n> > I'm not sure AfSpeed is *necessarily* only for CAF. I think we've done\n> > AF systems in the past where even regular AF has a \"I really want to\n> > take the picture quickly\" mode. But it could be a platform choice?\n> Agreed.\n> >\n> > >\n> > > > + - name: AfModeContinuous\n> > > > + value: 1\n> > > > + description: |\n> > > > + The AF algorithm is in continuous mode. This means that the lens\n> > > > + can re-start a scan spontaneously at any moment, without any user\n> > > > + intervention. The AfState still reports whether the algorithm is\n> > > > + currently scanning or not, though the application has no ability\n> > > > + to initiate or cancel scans, nor move the lens for itself.\n> > > > +\n> > > > + The mode can be set to AfModeAuto which has the effect of\n> > > > + \"pausing\" any continuous AF activity, after which it could then\n> > > > + be moved back to AfModeContinuous to resume operation.\n> > > > +\n> > > > + - AfRange:\n> > > > + type: int32_t\n> > > > + draft: true\n> > > > + description: |\n> > > > + Control to set the range of focus distances that is scanned.\n> > > > + enum:\n> > > > + - name: AfRangeFull\n> > > > + value: 0\n> > > > + description: The full range of focus distances is scanned.\n> > > > + - name: AfRangeMacro\n> > > > + value: 1\n> > > > + description: Only close distances are scanned.\n> > > > + - name: AfRangeNormal\n> > > > + value: 2\n> > > > + description: |\n> > > > + The full range of focus distances is scanned except for some of\n> > > > + the closest macro positions.\n> > > > +\n> > > > + - AfSpeed:\n> > > > + type: int32_t\n> > > > + draft: true\n> > > > + description: |\n> > > > + Control that determines whether the AF algorithm is to move the lens\n> > > > + as quickly as possible or more steadily. For example, during video\n> > > > + recording it may be desirable not to move the lens too abruptly, but\n> > > > + when in a preview mode (waiting for a still capture) it may be\n> > > > + helpful to move the lens as quickly as is reasonably possible.\n> > > > + enum:\n> > > > + - name: AfSpeedFast\n> > > > + value: 0\n> > > > + description: Move the lens quickly.\n> > > > + - name: AfSpeedSlow\n> > > > + value: 0\n> > > > + description: Move the lens more steadily.\n> > > How about\n> > > s/Fast/Aggressively\n> > > s/Slow/Steadily\n> >\n> > I agree the choice is tricky here. I think I would regard what I\n> > called \"Slow\" to be the \"usual\" choice, so perhaps \"AfSpeedNormal\" is\n> > better (and maybe make that value 0). I still prefer \"AfModeFast\" to\n> > \"AfModeAggressive\" - \"aggressive\" sounds just a bit too... aggressive?\n> > :)\n> LGTM.\n> >\n> > But thanks for all the discussion. Glad we're making progress on all this!\n> >\n> > Best regards\n> > David\n> >\n> > > > +\n> > > > + - AfMethod:\n> > > > + type: int32_t\n> > > > + draft: true\n> > > > + description: |\n> > > > + Control whether the AF algorithm uses a single window in the image to\n> > > > + determine the best focus position, or multiple windows simultaneously.\n> > > > + enum:\n> > > > + - name: AfMethodSingle\n> > > > + value: 0\n> > > > + description: |\n> > > > + A single window within the image, defaulting to the centre, is used\n> > > > + to select the best focus distance.\n> > > > + - name: AfMethodMultiSpot\n> > > > + value: 0\n> > > > + description: |\n> > > > + Multiple windows within the image are used to select the best focus\n> > > > + distance. The best focus distance is found for each one of the\n> > > > + windows, and then the distance that is closest to the camera is\n> > > > + selected.\n> > > > +\n> > > > + - AfWindow:\n> > > > + type: Rectangle\n> > > > + draft: true\n> > > > + description: |\n> > > > + Sets the focus windows used by the AF algorithm. The units used express\n> > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire\n> > > > + image), as u0.16 format numbers.\n> > > > +\n> > > > + In order to be activated, a rectangle must be programmed with non-zero\n> > > > + width and height. If no rectangles are programmed in this way, then the\n> > > > + system will choose its own single default window in the centre of the\n> > > > + image.\n> > > > +\n> > > > + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n> > > > + this list is used (or the system default one if it is unprogrammed).\n> > > > +\n> > > > + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n> > > > + this list are used. The size of the control indicates how many such\n> > > > + windows can be programmed and will vary between different platforms.\n> > > > +\n> > > > + size: [platform dependent]\n> > > > +\n> > > > + - AfTrigger:\n> > > > + type: int32_t\n> > > > + draft: true\n> > > > + description: |\n> > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n> > > > + and can also be used to terminate a scan early.\n> > > > +\n> > > > + It is ignored if AfMode is set to AfModeContinuous.\n> > > > +\n> > > > + enum:\n> > > > + - name: AfTriggerStart\n> > > > + value: 0\n> > > > + description: Start an AF scan. Ignored if a scan is in progress.\n> > > > + - name: AfTriggerCancel\n> > > > + value: 1\n> > > > + description: Cancel an AF scan. Ingored if no scan is in progress.\n\nFrom one of your other emails, and checking the Android documentation,\nI see that \"Trigger\" does mean something in CAF mode though, as you\nsay, a \"pause\" control captures the meaning a bit better. I think\nAndroid uses \"trigger start\" to mean:\n\n- In \"video\" CAF mode: pause immediately, whatever is happening.\n- In \"picture\" CAF mode: pause when any current (passive) scan is finished.\n\nPerhaps a pause control would have 3 values: \"pause immediately\",\n\"pause when ready\" and \"resume\".\n\nThanks and best regards\n\nDavid\n\n> > > > +\n> > > > + - LensPosition:\n> > > > + type: int32_t\n> > > > + draft: true\n> > > > + description: |\n> > > > + Acts as a control to instruct the lens to move to a particular position\n> > > > + and also reports back the position of the lens for each frame.\n> > > > +\n> > > > + The units are determined by the lens driver.\n> > > > +\n> > > > + If the LensPosition control is used while an AF scan is in progress,\n> > > > + the scan is implicitly cancelled and the lens is then moved to the\n> > > > + desired location. It is ignored if AfMode is set to AfModeContinuous.\n> > > > +\n> > > > + - AfState:\n> > > > + type: int32_t\n> > > > + draft: true\n> > > > + description: |\n> > > > + Reports the current state of the AF algorithm.\n> > > > + enum:\n> > > > + - name: AfStateReset\n> > > > + value: 0\n> > > > + description: |\n> > > > + The AF algorithm reports this state when:\n> > > > + * The camera system has just been started.\n> > > > + * A scan has been cancelled.\n> > > > + * The lens has been moved by the LensPosition control.\n> > > > + - name: AfStateScanning\n> > > > + value: 1\n> > > > + description: |\n> > > > + AF is performing a scan. This state can be entered spontaneously\n> > > > + if AfMode is set to AfModeContinuous, otherwise it requires the\n> > > > + application to use the AfTrigger control to start the scan.\n> > > > + - name: AfStateFocused\n> > > > + value: 2\n> > > > + description: |\n> > > > + An AF scan has been performed and the algorithm believes the\n> > > > + scene is in focus.\n> > > > + - name: AfStateFailed\n> > > > + value: 3\n> > > > + description: |\n> > > > + An AF scan has been performed but the algorithm has not been able\n> > > > + to find the best focus position.\n> > > > +\n> > > > ...\n> > > > --\n> > > > 2.30.2\n> > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E54ECBF415\n\tfor ;\n\tThu, 13 Jan 2022 17:54:35 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0419760931;\n\tThu, 13 Jan 2022 18:54:35 +0100 (CET)","from mail-wr1-x431.google.com (mail-wr1-x431.google.com\n\t[IPv6:2a00:1450:4864:20::431])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D4803604F4\n\tfor ;\n\tThu, 13 Jan 2022 18:54:33 +0100 (CET)","by mail-wr1-x431.google.com with SMTP id v6so11538096wra.8\n\tfor ;\n\tThu, 13 Jan 2022 09:54:33 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"czqktL3b\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=oei04rrD63VAPF0cO408xMF27szLHcId8yLpuJvgahI=;\n\tb=czqktL3bHNqAb0MhYBHYBUKm+ysBocs3U3U4C5l0F/b4kjB19QAfO3l00qSD4k27ZY\n\thyG/nwrCRWzsBpv8uUhSM0vkgzvYrIUo8toBCM+/eUIQ4CjNUbE/VFxumcOVFspS07+v\n\tL1dJTf8LmuccgCZ/iXN0jXkpJ+n1QAQCSYvfNsxx1CTr6d3eEuGfIX4MQ44fjTAOQbjF\n\tzFvCw9cpoXu9+YepH9A+5SLruuKq31yl+uQ8styOzLI7UMT0zkzt4ijERUN/+Lro4Wf7\n\t/OCQalxzTx9o/AYKiDcpBXJFkeoKD3DcEEU76IMy0qJlFCNmrVFqsbu7c0EGXniuesGY\n\tvHUA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=oei04rrD63VAPF0cO408xMF27szLHcId8yLpuJvgahI=;\n\tb=akLGAxTMEFEKOfp3G7lE2cX/oyAfMkWBuDM0YqrvQxdBlOkvkowYdOfib1EczLqOXJ\n\tUlu/dO9j0TGK4eMlwmtFpdGAO53hFagdbfCvREVRvYjtmo/KrcqTsIoEUh6Lp38OTpnG\n\twLRYeWglplQMJcQhj35+NLoPUCEyPydT8/SwEJOFH6k2MwvnT7GUObTSykpPHICAgNUK\n\tjxZFio7h8BY6P7yOWHkPDABpYaghh82jXEYdpbVmj+9wzmYX7OD+oiiNkFDMgr/u7qPj\n\tKy8ULdeUw4tx2tgaVahSLLnLBR1oQNIwkW9VOCJVIvhwNYmMqb1I13LobcJOV+pqVc60\n\t3D2Q==","X-Gm-Message-State":"AOAM5307mh1uzwQP27odauG01zLHDXb8kUmlogMBRTtCJKS6PsTPlG41\n\tniInsbXYhFyiZ6u+kZIriPRG/P7S6TnuM5NTCy41cA==","X-Google-Smtp-Source":"ABdhPJzbdQyvJDY5vzLg7xhZuSKsQPycMvUNqYvFiKeJ8ol3CTFcREWExHdPmjlYZboB9iFm+9StLqE7AbbuilLtNDk=","X-Received":"by 2002:adf:d1c8:: with SMTP id b8mr3728640wrd.317.1642096472956;\n\tThu, 13 Jan 2022 09:54:32 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t\n\t\n\t","In-Reply-To":"","From":"David Plowman ","Date":"Thu, 13 Jan 2022 17:54:20 +0000","Message-ID":"","To":"Hanlin Chen ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22040,"web_url":"https://patchwork.libcamera.org/comment/22040/","msgid":"","date":"2022-01-17T09:31:25","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":97,"url":"https://patchwork.libcamera.org/api/people/97/","name":"Nicolas Dufresne via libcamera-devel","email":"libcamera-devel@lists.libcamera.org"},"content":"Hi David,\nThanks for the comments.\n\nOn Fri, Jan 14, 2022 at 1:54 AM David Plowman\n wrote:\n>\n> Hi Hanlin\n>\n> Sorry for taking some time to get back to this!\nNo worries. We all work asynchronously :)\n\n> From one of your other emails, and checking the Android documentation,\n> I see that \"Trigger\" does mean something in CAF mode though, as you\n> say, a \"pause\" control captures the meaning a bit better. I think\n> Android uses \"trigger start\" to mean:\n>\n> - In \"video\" CAF mode: pause immediately, whatever is happening.\n> - In \"picture\" CAF mode: pause when any current (passive) scan is finished.\n>\n> Perhaps a pause control would have 3 values: \"pause immediately\",\n> \"pause when ready\" and \"resume\".\nI moved the comment up, so I can comment on it earlier.\nYes, I think it is more clear, although the algorithm may need to\nimplement it for both video and picture CAF.\nI suppose it's fine?\n\n>\n> On Tue, 21 Dec 2021 at 10:21, Hanlin Chen wrote:\n> >\n> > Hi David, thanks for the comments.\n> >\n> > On Fri, Dec 17, 2021 at 4:51 PM David Plowman\n> > wrote:\n> > >\n> > > Hi again\n> > >\n> > > Thanks for the comments!\n> > >\n> > > On Thu, 16 Dec 2021 at 11:18, Hanlin Chen wrote:\n> > > >\n> > > > Hi David, thanks for this work!\n> > > >\n> > > > On Mon, Dec 13, 2021 at 11:22 PM David Plowman\n> > > > wrote:\n> > > > >\n> > > > > This patch describes a series of controls that allow applications to\n> > > > > drive AF algorithms:\n> > > > >\n> > > > > AfMode - auto or continuous\n> > > > > AfRange - full, macro or normal\n> > > > > AfSpeed - fast or slow\n> > > > > AfMethod - single or multi-spot\n> > > > > AfWindow - AF window locations\n> > > > > AfTrigger - start (trigger an AF scan) or cancel\n> > > > > LensPosition - position of lens from lens driver\n> > > > > AfState - reset, scanning, focused or failed\n> > > > > ---\n> > > > > src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> > > > > 1 file changed, 167 insertions(+), 60 deletions(-)\n> > > > >\n> > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > index 9d4638ae..e579f7b7 100644\n> > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > @@ -406,27 +406,6 @@ controls:\n> > > > > The camera will cancel any active or completed metering sequence.\n> > > > > The AE algorithm is reset to its initial state.\n> > > > >\n> > > > > - - AfTrigger:\n> > > > > - type: int32_t\n> > > > > - draft: true\n> > > > > - description: |\n> > > > > - Control for AF trigger. Currently identical to\n> > > > > - ANDROID_CONTROL_AF_TRIGGER.\n> > > > > -\n> > > > > - Whether the camera device will trigger autofocus for this request.\n> > > > > - enum:\n> > > > > - - name: AfTriggerIdle\n> > > > > - value: 0\n> > > > > - description: The trigger is idle.\n> > > > > - - name: AfTriggerStart\n> > > > > - value: 1\n> > > > > - description: The AF routine is started by the camera.\n> > > > > - - name: AfTriggerCancel\n> > > > > - value: 2\n> > > > > - description: |\n> > > > > - The camera will cancel any active trigger and the AF routine is\n> > > > > - reset to its initial state.\n> > > > > -\n> > > > > - NoiseReductionMode:\n> > > > > type: int32_t\n> > > > > draft: true\n> > > > > @@ -507,45 +486,6 @@ controls:\n> > > > > The AE algorithm has started a pre-capture metering session.\n> > > > > \\sa AePrecaptureTrigger\n> > > > >\n> > > > > - - AfState:\n> > > > > - type: int32_t\n> > > > > - draft: true\n> > > > > - description: |\n> > > > > - Control to report the current AF algorithm state. Currently identical to\n> > > > > - ANDROID_CONTROL_AF_STATE.\n> > > > > -\n> > > > > - Current state of the AF algorithm.\n> > > > > - enum:\n> > > > > - - name: AfStateInactive\n> > > > > - value: 0\n> > > > > - description: The AF algorithm is inactive.\n> > > > > - - name: AfStatePassiveScan\n> > > > > - value: 1\n> > > > > - description: |\n> > > > > - AF is performing a passive scan of the scene in continuous\n> > > > > - auto-focus mode.\n> > > > > - - name: AfStatePassiveFocused\n> > > > > - value: 2\n> > > > > - description: |\n> > > > > - AF believes the scene is in focus, but might restart scanning.\n> > > > > - - name: AfStateActiveScan\n> > > > > - value: 3\n> > > > > - description: |\n> > > > > - AF is performing a scan triggered by an AF trigger request.\n> > > > > - \\sa AfTrigger\n> > > > > - - name: AfStateFocusedLock\n> > > > > - value: 4\n> > > > > - description: |\n> > > > > - AF believes has focused correctly and has locked focus.\n> > > > > - - name: AfStateNotFocusedLock\n> > > > > - value: 5\n> > > > > - description: |\n> > > > > - AF has not been able to focus and has locked.\n> > > > > - - name: AfStatePassiveUnfocused\n> > > > > - value: 6\n> > > > > - description: |\n> > > > > - AF has completed a passive scan without finding focus.\n> > > > > -\n> > > > > - AwbState:\n> > > > > type: int32_t\n> > > > > draft: true\n> > > > > @@ -690,4 +630,171 @@ controls:\n> > > > > value. All of the custom test patterns will be static (that is the\n> > > > > raw image must not vary from frame to frame).\n> > > > >\n> > > > > + - AfMode:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control to set the mode of the AF (autofocus) algorithm.\n> > > > > + enum:\n> > > > > + - name: AfModeAuto\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The AF algorithm is in auto mode. This means that the algorithm\n> > > > > + will never move the lens or change state unless the AfTrigger\n> > > > > + control is used. The AfTrigger control can be used to initiate a\n> > > > > + focus scan, the results of which will be reported by the AfState.\n> > > > > +\n> > > > > + In this mode, an application can move the lens for itself using\n> > > > > + the LensPosition control. If the algorithm was scanning when the\n> > > > > + lens is moved in this way, then the scan is implicitly cancelled.\n> > > >\n> > > > I would suggest splitting the Auto mode into AfModeOnDemand and\n> > > > AfModeManual (The naming could be re-considered).\n> > > > The reason is that the sub-controls for each AfModes could be distinct.\n> > > > 1. AfModeManual:\n> > > > LensPosition is valid only for the mode, and AfState is always AfStateReset.\n> > > > 2. AfModeOnDemand\n> > > > AfTrigger and AfRange are valid only for the mode, and AfState reports\n> > > > accordingly. The scan is implicitly canceled when changing from the\n> > > > mode.\n> > >\n> > > I'm a little bit nervous about having a mode where the lens position\n> > > control doesn't work. In my earlier email I described performing an AF\n> > > scan, capturing an image, then moving the lens back to hyperfocal. It\n> > > seems inconvenient to have to put the AF algorithm into a separate\n> > > mode in order to move the lens, only to put it back in the mode where\n> > > a scan is once again allowed.\n> > >\n> > > I don't think anything is lost by doing it this way - if an\n> > > application doesn't want to perform an AF scan it should simply not\n> > > request one.\n> > I agreed that it does cover the controls from Android, i.e., the\n> > Android request can be translated to the API.\n> > My concern is from another direction: translation from libcamera to\n> > Android Camera3.\n> > From the Android to libcamera:\n> > ANDROID_CONTROL_AF_MODE_OFF + ANDROID_LENS_FOCUS_DISTANCE ==>\n> > AfModeAuto + lensPosition\n> > ANDROID_CONTROL_AF_MODE_CONTINUOUS_VIDEO +\n> > ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_START ==> AfModeAuto +\n> > AfTrigger\n>\n> I don't quite understand how an AE_PRECAPTURE_TRIGGER_START affects\n> the AF state machine, isn't that something we'd send to the AE\n> algorithm (if it had such a control)? And then we'd wait for the AE\n> state to say it was finished?\nAh... It's a copy error, I simply mean AF_TRIGGER here.\n\n>\n> > From libcamera to Android:\n> > AfModeAuto + lensPostion (Assuming current position is reported by\n> > libcamera) ==> ??\n> > It could be CAF, AUTO or OFF mode.\n>\n> As I understand it, Android only lets you set the lens position when\n> AF mode is \"OFF\". So in this case we'd have to set the Android AF to\n> \"OFF\" along with the lens position, right?\n>\n> If the next thing libcamera wants to do is something other than move\n> the lens, then we'll have to set Android back to the state\n> corresponding to libcamera's. Does that work?\n\nThe context here is that I seem to assume that if we don't have\n\"pause\" in CAF, we need to mimic it by changing Auto with\nlensPosition.\nThus, when translating back to Android, it's not clear that it's Auto\nor CAF with mimicking \"pause\".\nMaybe my assumption is wrong?\n\nHowever, if we have \"pause\", then the problem disappears:\nwith lensPosition == OFF, without == AUTO.\n>\n> > States from libcamera Android:\n> > AfModeAuto + AfTrigger + focused ==>\n> > ANDROID_CONTROL_AF_STATE_FOCUSED_LOCKED? or\n> > ANDROID_CONTROL_AF_STATE_PASSIVE_FOCUSED?\n>\n> If there was a trigger, then I believe the \"focused\" state needs to\n> correspond to AF_STATE_FOCUSED_LOCKED. In fact, is it the case that in\n> auto mode, Android can never report AF_STATE_PASSIVE_FOCUSED?\nSame above.\n>\n> >\n> > For the Android adapter layer, above might be workarounded by caching\n> > the requests and do reference and find out current context.\n> > A more subtle case might arise when integrating other platforms.\n> > For example, Qualcomm closed 3A lib follows the modes and states\n> > according to Android (Partly true for Intel and MTK).\n> > In the case, the translation flow will like:\n> > Android API -> libcamera API -> Android like API for 3A lib ->\n> > libcamera API -> Android API\n> > \n> > \n> > It's hard to predict how to work around all corner cases.\n> > This makes me believe that if we could have a clear one to one mapping\n> > between Android and libcamera API, it could be beneficial later.\n> > However, it will make the API more verbose.\n> >\n> > Another perspective is that Android defines required state machine transition:\n> > https://source.android.com/devices/camera/camera3_3Amodes\n> >\n> > For IPA to implement the state transition satisfying Android's\n> > requirement, the IPA should know exactly how the libcamera states\n> > mapping back to the Android states.\n> > Above is not satisfying because we will be somehow restricted by\n> > Android's definition ;-|, However, I think it could be beneficial if\n> > the API cam map libcamera states/modes to all Android states/modes\n> > directly without extra context/call order information.\n>\n> I'm really in two minds about the question of an \"off\" mode in\n> libcamera. Either we make libcamera application code handle the\n> transition to \"off\" and back to \"auto\", or a libcamera-to-Android\n> translation layer would have to do it. Does anyone else out there feel\n> strongly?\nI suppose we could align it with AE (or let AE align with AF :P). AE\nhas a similar case for AnalogueGainMode and ExposureTimeMode.\nhttps://patchwork.libcamera.org/patch/15179/\nWhere Paul seems to use \"Auto\" and \"Disabled\" for the case.\ncc'ed Paul here.\n\n>\n> >\n> > >\n> > > I think AfRange applies to CAF as well. I see no reason why you might\n> > > not want to restrict the range of searches for just the same reasons\n> > > as you might do with \"regular\" AF.\n> > Agreed.\n> > >\n> > > > 3. AfModeContinuous\n> > > > AfSpeed and \"pause\" are valid only for the mode, and AfState reports\n> > > > accordingly. The scan is implicitly canceled when changing from the\n> > > > mode.\n> > >\n> > > I'm not sure AfSpeed is *necessarily* only for CAF. I think we've done\n> > > AF systems in the past where even regular AF has a \"I really want to\n> > > take the picture quickly\" mode. But it could be a platform choice?\n> > Agreed.\n> > >\n> > > >\n> > > > > + - name: AfModeContinuous\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The AF algorithm is in continuous mode. This means that the lens\n> > > > > + can re-start a scan spontaneously at any moment, without any user\n> > > > > + intervention. The AfState still reports whether the algorithm is\n> > > > > + currently scanning or not, though the application has no ability\n> > > > > + to initiate or cancel scans, nor move the lens for itself.\n> > > > > +\n> > > > > + The mode can be set to AfModeAuto which has the effect of\n> > > > > + \"pausing\" any continuous AF activity, after which it could then\n> > > > > + be moved back to AfModeContinuous to resume operation.\n> > > > > +\n> > > > > + - AfRange:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control to set the range of focus distances that is scanned.\n> > > > > + enum:\n> > > > > + - name: AfRangeFull\n> > > > > + value: 0\n> > > > > + description: The full range of focus distances is scanned.\n> > > > > + - name: AfRangeMacro\n> > > > > + value: 1\n> > > > > + description: Only close distances are scanned.\n> > > > > + - name: AfRangeNormal\n> > > > > + value: 2\n> > > > > + description: |\n> > > > > + The full range of focus distances is scanned except for some of\n> > > > > + the closest macro positions.\n> > > > > +\n> > > > > + - AfSpeed:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control that determines whether the AF algorithm is to move the lens\n> > > > > + as quickly as possible or more steadily. For example, during video\n> > > > > + recording it may be desirable not to move the lens too abruptly, but\n> > > > > + when in a preview mode (waiting for a still capture) it may be\n> > > > > + helpful to move the lens as quickly as is reasonably possible.\n> > > > > + enum:\n> > > > > + - name: AfSpeedFast\n> > > > > + value: 0\n> > > > > + description: Move the lens quickly.\n> > > > > + - name: AfSpeedSlow\n> > > > > + value: 0\n> > > > > + description: Move the lens more steadily.\n> > > > How about\n> > > > s/Fast/Aggressively\n> > > > s/Slow/Steadily\n> > >\n> > > I agree the choice is tricky here. I think I would regard what I\n> > > called \"Slow\" to be the \"usual\" choice, so perhaps \"AfSpeedNormal\" is\n> > > better (and maybe make that value 0). I still prefer \"AfModeFast\" to\n> > > \"AfModeAggressive\" - \"aggressive\" sounds just a bit too... aggressive?\n> > > :)\n> > LGTM.\n> > >\n> > > But thanks for all the discussion. Glad we're making progress on all this!\n> > >\n> > > Best regards\n> > > David\n> > >\n> > > > > +\n> > > > > + - AfMethod:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control whether the AF algorithm uses a single window in the image to\n> > > > > + determine the best focus position, or multiple windows simultaneously.\n> > > > > + enum:\n> > > > > + - name: AfMethodSingle\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + A single window within the image, defaulting to the centre, is used\n> > > > > + to select the best focus distance.\n> > > > > + - name: AfMethodMultiSpot\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + Multiple windows within the image are used to select the best focus\n> > > > > + distance. The best focus distance is found for each one of the\n> > > > > + windows, and then the distance that is closest to the camera is\n> > > > > + selected.\n> > > > > +\n> > > > > + - AfWindow:\n> > > > > + type: Rectangle\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Sets the focus windows used by the AF algorithm. The units used express\n> > > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire\n> > > > > + image), as u0.16 format numbers.\n> > > > > +\n> > > > > + In order to be activated, a rectangle must be programmed with non-zero\n> > > > > + width and height. If no rectangles are programmed in this way, then the\n> > > > > + system will choose its own single default window in the centre of the\n> > > > > + image.\n> > > > > +\n> > > > > + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n> > > > > + this list is used (or the system default one if it is unprogrammed).\n> > > > > +\n> > > > > + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n> > > > > + this list are used. The size of the control indicates how many such\n> > > > > + windows can be programmed and will vary between different platforms.\n> > > > > +\n> > > > > + size: [platform dependent]\n> > > > > +\n> > > > > + - AfTrigger:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n> > > > > + and can also be used to terminate a scan early.\n> > > > > +\n> > > > > + It is ignored if AfMode is set to AfModeContinuous.\n> > > > > +\n> > > > > + enum:\n> > > > > + - name: AfTriggerStart\n> > > > > + value: 0\n> > > > > + description: Start an AF scan. Ignored if a scan is in progress.\n> > > > > + - name: AfTriggerCancel\n> > > > > + value: 1\n> > > > > + description: Cancel an AF scan. Ingored if no scan is in progress.\n>\n> From one of your other emails, and checking the Android documentation,\n> I see that \"Trigger\" does mean something in CAF mode though, as you\n> say, a \"pause\" control captures the meaning a bit better. I think\n> Android uses \"trigger start\" to mean:\n>\n> - In \"video\" CAF mode: pause immediately, whatever is happening.\n> - In \"picture\" CAF mode: pause when any current (passive) scan is finished.\n>\n> Perhaps a pause control would have 3 values: \"pause immediately\",\n> \"pause when ready\" and \"resume\".\n>\n> Thanks and best regards\n>\n> David\n>\n> > > > > +\n> > > > > + - LensPosition:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Acts as a control to instruct the lens to move to a particular position\n> > > > > + and also reports back the position of the lens for each frame.\n> > > > > +\n> > > > > + The units are determined by the lens driver.\n> > > > > +\n> > > > > + If the LensPosition control is used while an AF scan is in progress,\n> > > > > + the scan is implicitly cancelled and the lens is then moved to the\n> > > > > + desired location. It is ignored if AfMode is set to AfModeContinuous.\n> > > > > +\n> > > > > + - AfState:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Reports the current state of the AF algorithm.\n> > > > > + enum:\n> > > > > + - name: AfStateReset\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The AF algorithm reports this state when:\n> > > > > + * The camera system has just been started.\n> > > > > + * A scan has been cancelled.\n> > > > > + * The lens has been moved by the LensPosition control.\n> > > > > + - name: AfStateScanning\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + AF is performing a scan. This state can be entered spontaneously\n> > > > > + if AfMode is set to AfModeContinuous, otherwise it requires the\n> > > > > + application to use the AfTrigger control to start the scan.\n> > > > > + - name: AfStateFocused\n> > > > > + value: 2\n> > > > > + description: |\n> > > > > + An AF scan has been performed and the algorithm believes the\n> > > > > + scene is in focus.\n> > > > > + - name: AfStateFailed\n> > > > > + value: 3\n> > > > > + description: |\n> > > > > + An AF scan has been performed but the algorithm has not been able\n> > > > > + to find the best focus position.\n> > > > > +\n> > > > > ...\n> > > > > --\n> > > > > 2.30.2\n> > > > >\n\n\n\n--\nCheers.\nHanlin Chen","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 95A72BE086\n\tfor ;\n\tMon, 17 Jan 2022 09:31:40 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D0F056021A;\n\tMon, 17 Jan 2022 10:31:39 +0100 (CET)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1642411899;\n\tbh=VRHZUsHbRqzL3NNfeaL389NTmPVZIEAZ0qXrJxjduAY=;\n\th=References:In-Reply-To:Date:To:List-Id:List-Post:From:Cc:\n\tList-Subscribe:List-Unsubscribe:List-Archive:Reply-To:List-Help:\n\tSubject:From;\n\tb=XuXGIV3lNgUq2wxDkEbwInMwm+oAAnAQvwlD4q58HSUtbnpDbhoAspUhnYGUcCUL7\n\t6QV83Hn3A0HtT9jpj4+IantITfSaebZM0bzvR1l9MCLBd/Qq9QcFiLWmnxWNoQwywH\n\tKl8YWvVwULdVlw2ak9cyvWf/H4PLiVGxqHN8mdUp6UbSXdelMycSM/fkJmGEuLq+n5\n\tlfyRJcvqFmxeHPzA1bTF20Fdq144L7NCmK6vaUbUDwEc1hTgUiNgnclRhDxC+QO/fr\n\t0J8exKKVEAgDDOveT6ltGp8j2niujpY3vGAXfbvn1pjGd27XvRMyNfQ8IJ2jOo8QYw\n\tsJOzyxInIn5Sg==","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t\n\t\n\t\n\t","In-Reply-To":"","Date":"Mon, 17 Jan 2022 17:31:25 +0800","To":"David Plowman ","MIME-Version":"1.0","Message-ID":"","List-Id":"","List-Post":"","From":"Han-lin Chen via libcamera-devel ","Precedence":"list","Cc":"libcamera devel ","X-Mailman-Version":"2.1.29","X-BeenThere":"libcamera-devel@lists.libcamera.org","List-Subscribe":",\n\t","List-Unsubscribe":",\n\t","List-Archive":"","Reply-To":"Han-lin Chen ","List-Help":"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","Content-Type":"message/rfc822","Content-Disposition":"inline","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22041,"web_url":"https://patchwork.libcamera.org/comment/22041/","msgid":"","date":"2022-01-17T10:13:20","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":98,"url":"https://patchwork.libcamera.org/api/people/98/","name":"Hanlin Chen","email":"hanlinchen@chromium.org"},"content":"The mail is re-sent by another account to avoid DMARC rejection. Sorry\nfor the inconvenience.\n\nHi David,\nThanks for the comments.\n\nOn Fri, Jan 14, 2022 at 1:54 AM David Plowman\n wrote:\n>\n> Hi Hanlin\n>\n> Sorry for taking some time to get back to this!\nNo worries. We all work asynchronously :)\n\n> From one of your other emails, and checking the Android documentation,\n> I see that \"Trigger\" does mean something in CAF mode though, as you\n> say, a \"pause\" control captures the meaning a bit better. I think\n> Android uses \"trigger start\" to mean:\n>\n> - In \"video\" CAF mode: pause immediately, whatever is happening.\n> - In \"picture\" CAF mode: pause when any current (passive) scan is finished.\n>\n> Perhaps a pause control would have 3 values: \"pause immediately\",\n> \"pause when ready\" and \"resume\".\nI moved the comment up, so I can comment on it earlier.\nYes, I think it is more clear, although the algorithm may need to\nimplement it for both video and picture CAF.\nI suppose it's fine?\n>\n> On Tue, 21 Dec 2021 at 10:21, Hanlin Chen wrote:\n> >\n> > Hi David, thanks for the comments.\n> >\n> > On Fri, Dec 17, 2021 at 4:51 PM David Plowman\n> > wrote:\n> > >\n> > > Hi again\n> > >\n> > > Thanks for the comments!\n> > >\n> > > On Thu, 16 Dec 2021 at 11:18, Hanlin Chen wrote:\n> > > >\n> > > > Hi David, thanks for this work!\n> > > >\n> > > > On Mon, Dec 13, 2021 at 11:22 PM David Plowman\n> > > > wrote:\n> > > > >\n> > > > > This patch describes a series of controls that allow applications to\n> > > > > drive AF algorithms:\n> > > > >\n> > > > > AfMode - auto or continuous\n> > > > > AfRange - full, macro or normal\n> > > > > AfSpeed - fast or slow\n> > > > > AfMethod - single or multi-spot\n> > > > > AfWindow - AF window locations\n> > > > > AfTrigger - start (trigger an AF scan) or cancel\n> > > > > LensPosition - position of lens from lens driver\n> > > > > AfState - reset, scanning, focused or failed\n> > > > > ---\n> > > > > src/libcamera/control_ids.yaml | 227 ++++++++++++++++++++++++---------\n> > > > > 1 file changed, 167 insertions(+), 60 deletions(-)\n> > > > >\n> > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > index 9d4638ae..e579f7b7 100644\n> > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > @@ -406,27 +406,6 @@ controls:\n> > > > > The camera will cancel any active or completed metering sequence.\n> > > > > The AE algorithm is reset to its initial state.\n> > > > >\n> > > > > - - AfTrigger:\n> > > > > - type: int32_t\n> > > > > - draft: true\n> > > > > - description: |\n> > > > > - Control for AF trigger. Currently identical to\n> > > > > - ANDROID_CONTROL_AF_TRIGGER.\n> > > > > -\n> > > > > - Whether the camera device will trigger autofocus for this request.\n> > > > > - enum:\n> > > > > - - name: AfTriggerIdle\n> > > > > - value: 0\n> > > > > - description: The trigger is idle.\n> > > > > - - name: AfTriggerStart\n> > > > > - value: 1\n> > > > > - description: The AF routine is started by the camera.\n> > > > > - - name: AfTriggerCancel\n> > > > > - value: 2\n> > > > > - description: |\n> > > > > - The camera will cancel any active trigger and the AF routine is\n> > > > > - reset to its initial state.\n> > > > > -\n> > > > > - NoiseReductionMode:\n> > > > > type: int32_t\n> > > > > draft: true\n> > > > > @@ -507,45 +486,6 @@ controls:\n> > > > > The AE algorithm has started a pre-capture metering session.\n> > > > > \\sa AePrecaptureTrigger\n> > > > >\n> > > > > - - AfState:\n> > > > > - type: int32_t\n> > > > > - draft: true\n> > > > > - description: |\n> > > > > - Control to report the current AF algorithm state. Currently identical to\n> > > > > - ANDROID_CONTROL_AF_STATE.\n> > > > > -\n> > > > > - Current state of the AF algorithm.\n> > > > > - enum:\n> > > > > - - name: AfStateInactive\n> > > > > - value: 0\n> > > > > - description: The AF algorithm is inactive.\n> > > > > - - name: AfStatePassiveScan\n> > > > > - value: 1\n> > > > > - description: |\n> > > > > - AF is performing a passive scan of the scene in continuous\n> > > > > - auto-focus mode.\n> > > > > - - name: AfStatePassiveFocused\n> > > > > - value: 2\n> > > > > - description: |\n> > > > > - AF believes the scene is in focus, but might restart scanning.\n> > > > > - - name: AfStateActiveScan\n> > > > > - value: 3\n> > > > > - description: |\n> > > > > - AF is performing a scan triggered by an AF trigger request.\n> > > > > - \\sa AfTrigger\n> > > > > - - name: AfStateFocusedLock\n> > > > > - value: 4\n> > > > > - description: |\n> > > > > - AF believes has focused correctly and has locked focus.\n> > > > > - - name: AfStateNotFocusedLock\n> > > > > - value: 5\n> > > > > - description: |\n> > > > > - AF has not been able to focus and has locked.\n> > > > > - - name: AfStatePassiveUnfocused\n> > > > > - value: 6\n> > > > > - description: |\n> > > > > - AF has completed a passive scan without finding focus.\n> > > > > -\n> > > > > - AwbState:\n> > > > > type: int32_t\n> > > > > draft: true\n> > > > > @@ -690,4 +630,171 @@ controls:\n> > > > > value. All of the custom test patterns will be static (that is the\n> > > > > raw image must not vary from frame to frame).\n> > > > >\n> > > > > + - AfMode:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control to set the mode of the AF (autofocus) algorithm.\n> > > > > + enum:\n> > > > > + - name: AfModeAuto\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The AF algorithm is in auto mode. This means that the algorithm\n> > > > > + will never move the lens or change state unless the AfTrigger\n> > > > > + control is used. The AfTrigger control can be used to initiate a\n> > > > > + focus scan, the results of which will be reported by the AfState.\n> > > > > +\n> > > > > + In this mode, an application can move the lens for itself using\n> > > > > + the LensPosition control. If the algorithm was scanning when the\n> > > > > + lens is moved in this way, then the scan is implicitly cancelled.\n> > > >\n> > > > I would suggest splitting the Auto mode into AfModeOnDemand and\n> > > > AfModeManual (The naming could be re-considered).\n> > > > The reason is that the sub-controls for each AfModes could be distinct.\n> > > > 1. AfModeManual:\n> > > > LensPosition is valid only for the mode, and AfState is always AfStateReset.\n> > > > 2. AfModeOnDemand\n> > > > AfTrigger and AfRange are valid only for the mode, and AfState reports\n> > > > accordingly. The scan is implicitly canceled when changing from the\n> > > > mode.\n> > >\n> > > I'm a little bit nervous about having a mode where the lens position\n> > > control doesn't work. In my earlier email I described performing an AF\n> > > scan, capturing an image, then moving the lens back to hyperfocal. It\n> > > seems inconvenient to have to put the AF algorithm into a separate\n> > > mode in order to move the lens, only to put it back in the mode where\n> > > a scan is once again allowed.\n> > >\n> > > I don't think anything is lost by doing it this way - if an\n> > > application doesn't want to perform an AF scan it should simply not\n> > > request one.\n> > I agreed that it does cover the controls from Android, i.e., the\n> > Android request can be translated to the API.\n> > My concern is from another direction: translation from libcamera to\n> > Android Camera3.\n> > From the Android to libcamera:\n> > ANDROID_CONTROL_AF_MODE_OFF + ANDROID_LENS_FOCUS_DISTANCE ==>\n> > AfModeAuto + lensPosition\n> > ANDROID_CONTROL_AF_MODE_CONTINUOUS_VIDEO +\n> > ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_START ==> AfModeAuto +\n> > AfTrigger\n>\n> I don't quite understand how an AE_PRECAPTURE_TRIGGER_START affects\n> the AF state machine, isn't that something we'd send to the AE\n> algorithm (if it had such a control)? And then we'd wait for the AE\n> state to say it was finished?\n\nAh... It's a copy error, I simply mean AF_TRIGGER here.\n>\n> > From libcamera to Android:\n> > AfModeAuto + lensPostion (Assuming current position is reported by\n> > libcamera) ==> ??\n> > It could be CAF, AUTO or OFF mode.\n>\n> As I understand it, Android only lets you set the lens position when\n> AF mode is \"OFF\". So in this case we'd have to set the Android AF to\n> \"OFF\" along with the lens position, right?\n>\n> If the next thing libcamera wants to do is something other than move\n> the lens, then we'll have to set Android back to the state\n> corresponding to libcamera's. Does that work?\n>\nThe context here is that I seem to assume that if we don't have\n\"pause\" in CAF, we need to mimic it by changing Auto with\nlensPosition.\nThus, when translating back to Android, it's not clear that it's Auto\nor CAF with mimicking \"pause\".\nMaybe my assumption is wrong?\n\nHowever, if we have \"pause\", then the problem disappears:\nwith lensPosition == OFF, without == AUTO.\n\n> > States from libcamera Android:\n> > AfModeAuto + AfTrigger + focused ==>\n> > ANDROID_CONTROL_AF_STATE_FOCUSED_LOCKED? or\n> > ANDROID_CONTROL_AF_STATE_PASSIVE_FOCUSED?\n>\n> If there was a trigger, then I believe the \"focused\" state needs to\n> correspond to AF_STATE_FOCUSED_LOCKED. In fact, is it the case that in\n> auto mode, Android can never report AF_STATE_PASSIVE_FOCUSED?\nSame above.\nYes, if it's AF_MODE_AUTO, it never report AF_STATE_PASSIVE_FOCUSED.\n>\n> >\n> > For the Android adapter layer, above might be workarounded by caching\n> > the requests and do reference and find out current context.\n> > A more subtle case might arise when integrating other platforms.\n> > For example, Qualcomm closed 3A lib follows the modes and states\n> > according to Android (Partly true for Intel and MTK).\n> > In the case, the translation flow will like:\n> > Android API -> libcamera API -> Android like API for 3A lib ->\n> > libcamera API -> Android API\n> > \n> > \n> > It's hard to predict how to work around all corner cases.\n> > This makes me believe that if we could have a clear one to one mapping\n> > between Android and libcamera API, it could be beneficial later.\n> > However, it will make the API more verbose.\n> >\n> > Another perspective is that Android defines required state machine transition:\n> > https://source.android.com/devices/camera/camera3_3Amodes\n> >\n> > For IPA to implement the state transition satisfying Android's\n> > requirement, the IPA should know exactly how the libcamera states\n> > mapping back to the Android states.\n> > Above is not satisfying because we will be somehow restricted by\n> > Android's definition ;-|, However, I think it could be beneficial if\n> > the API cam map libcamera states/modes to all Android states/modes\n> > directly without extra context/call order information.\n>\n> I'm really in two minds about the question of an \"off\" mode in\n> libcamera. Either we make libcamera application code handle the\n> transition to \"off\" and back to \"auto\", or a libcamera-to-Android\n> translation layer would have to do it. Does anyone else out there feel\n> strongly?\n\nI suppose we could align it with AE (or let AE align with AF :P). AE\nhas a similar case for AnalogueGainMode and ExposureTimeMode.\nhttps://patchwork.libcamera.org/patch/15179/\nWhere Paul seems to use \"Auto\" and \"Disabled\" for the case.\ncc'ed Paul.\n\n>\n> >\n> > >\n> > > I think AfRange applies to CAF as well. I see no reason why you might\n> > > not want to restrict the range of searches for just the same reasons\n> > > as you might do with \"regular\" AF.\n> > Agreed.\n> > >\n> > > > 3. AfModeContinuous\n> > > > AfSpeed and \"pause\" are valid only for the mode, and AfState reports\n> > > > accordingly. The scan is implicitly canceled when changing from the\n> > > > mode.\n> > >\n> > > I'm not sure AfSpeed is *necessarily* only for CAF. I think we've done\n> > > AF systems in the past where even regular AF has a \"I really want to\n> > > take the picture quickly\" mode. But it could be a platform choice?\n> > Agreed.\n> > >\n> > > >\n> > > > > + - name: AfModeContinuous\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + The AF algorithm is in continuous mode. This means that the lens\n> > > > > + can re-start a scan spontaneously at any moment, without any user\n> > > > > + intervention. The AfState still reports whether the algorithm is\n> > > > > + currently scanning or not, though the application has no ability\n> > > > > + to initiate or cancel scans, nor move the lens for itself.\n> > > > > +\n> > > > > + The mode can be set to AfModeAuto which has the effect of\n> > > > > + \"pausing\" any continuous AF activity, after which it could then\n> > > > > + be moved back to AfModeContinuous to resume operation.\n> > > > > +\n> > > > > + - AfRange:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control to set the range of focus distances that is scanned.\n> > > > > + enum:\n> > > > > + - name: AfRangeFull\n> > > > > + value: 0\n> > > > > + description: The full range of focus distances is scanned.\n> > > > > + - name: AfRangeMacro\n> > > > > + value: 1\n> > > > > + description: Only close distances are scanned.\n> > > > > + - name: AfRangeNormal\n> > > > > + value: 2\n> > > > > + description: |\n> > > > > + The full range of focus distances is scanned except for some of\n> > > > > + the closest macro positions.\n> > > > > +\n> > > > > + - AfSpeed:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control that determines whether the AF algorithm is to move the lens\n> > > > > + as quickly as possible or more steadily. For example, during video\n> > > > > + recording it may be desirable not to move the lens too abruptly, but\n> > > > > + when in a preview mode (waiting for a still capture) it may be\n> > > > > + helpful to move the lens as quickly as is reasonably possible.\n> > > > > + enum:\n> > > > > + - name: AfSpeedFast\n> > > > > + value: 0\n> > > > > + description: Move the lens quickly.\n> > > > > + - name: AfSpeedSlow\n> > > > > + value: 0\n> > > > > + description: Move the lens more steadily.\n> > > > How about\n> > > > s/Fast/Aggressively\n> > > > s/Slow/Steadily\n> > >\n> > > I agree the choice is tricky here. I think I would regard what I\n> > > called \"Slow\" to be the \"usual\" choice, so perhaps \"AfSpeedNormal\" is\n> > > better (and maybe make that value 0). I still prefer \"AfModeFast\" to\n> > > \"AfModeAggressive\" - \"aggressive\" sounds just a bit too... aggressive?\n> > > :)\n> > LGTM.\n> > >\n> > > But thanks for all the discussion. Glad we're making progress on all this!\n> > >\n> > > Best regards\n> > > David\n> > >\n> > > > > +\n> > > > > + - AfMethod:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Control whether the AF algorithm uses a single window in the image to\n> > > > > + determine the best focus position, or multiple windows simultaneously.\n> > > > > + enum:\n> > > > > + - name: AfMethodSingle\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + A single window within the image, defaulting to the centre, is used\n> > > > > + to select the best focus distance.\n> > > > > + - name: AfMethodMultiSpot\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + Multiple windows within the image are used to select the best focus\n> > > > > + distance. The best focus distance is found for each one of the\n> > > > > + windows, and then the distance that is closest to the camera is\n> > > > > + selected.\n> > > > > +\n> > > > > + - AfWindow:\n> > > > > + type: Rectangle\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Sets the focus windows used by the AF algorithm. The units used express\n> > > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire\n> > > > > + image), as u0.16 format numbers.\n> > > > > +\n> > > > > + In order to be activated, a rectangle must be programmed with non-zero\n> > > > > + width and height. If no rectangles are programmed in this way, then the\n> > > > > + system will choose its own single default window in the centre of the\n> > > > > + image.\n> > > > > +\n> > > > > + If AfMethod is set to AfMethodSingle, then only the first Rectangle in\n> > > > > + this list is used (or the system default one if it is unprogrammed).\n> > > > > +\n> > > > > + If AfMethod is set to AfMethodMultiSpot then all the valid Rectangles in\n> > > > > + this list are used. The size of the control indicates how many such\n> > > > > + windows can be programmed and will vary between different platforms.\n> > > > > +\n> > > > > + size: [platform dependent]\n> > > > > +\n> > > > > + - AfTrigger:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto,\n> > > > > + and can also be used to terminate a scan early.\n> > > > > +\n> > > > > + It is ignored if AfMode is set to AfModeContinuous.\n> > > > > +\n> > > > > + enum:\n> > > > > + - name: AfTriggerStart\n> > > > > + value: 0\n> > > > > + description: Start an AF scan. Ignored if a scan is in progress.\n> > > > > + - name: AfTriggerCancel\n> > > > > + value: 1\n> > > > > + description: Cancel an AF scan. Ingored if no scan is in progress.\n>\n> From one of your other emails, and checking the Android documentation,\n> I see that \"Trigger\" does mean something in CAF mode though, as you\n> say, a \"pause\" control captures the meaning a bit better. I think\n> Android uses \"trigger start\" to mean:\n>\n> - In \"video\" CAF mode: pause immediately, whatever is happening.\n> - In \"picture\" CAF mode: pause when any current (passive) scan is finished.\n>\n> Perhaps a pause control would have 3 values: \"pause immediately\",\n> \"pause when ready\" and \"resume\".\n>\n> Thanks and best regards\n>\n> David\n>\n> > > > > +\n> > > > > + - LensPosition:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Acts as a control to instruct the lens to move to a particular position\n> > > > > + and also reports back the position of the lens for each frame.\n> > > > > +\n> > > > > + The units are determined by the lens driver.\n> > > > > +\n> > > > > + If the LensPosition control is used while an AF scan is in progress,\n> > > > > + the scan is implicitly cancelled and the lens is then moved to the\n> > > > > + desired location. It is ignored if AfMode is set to AfModeContinuous.\n> > > > > +\n> > > > > + - AfState:\n> > > > > + type: int32_t\n> > > > > + draft: true\n> > > > > + description: |\n> > > > > + Reports the current state of the AF algorithm.\n> > > > > + enum:\n> > > > > + - name: AfStateReset\n> > > > > + value: 0\n> > > > > + description: |\n> > > > > + The AF algorithm reports this state when:\n> > > > > + * The camera system has just been started.\n> > > > > + * A scan has been cancelled.\n> > > > > + * The lens has been moved by the LensPosition control.\n> > > > > + - name: AfStateScanning\n> > > > > + value: 1\n> > > > > + description: |\n> > > > > + AF is performing a scan. This state can be entered spontaneously\n> > > > > + if AfMode is set to AfModeContinuous, otherwise it requires the\n> > > > > + application to use the AfTrigger control to start the scan.\n> > > > > + - name: AfStateFocused\n> > > > > + value: 2\n> > > > > + description: |\n> > > > > + An AF scan has been performed and the algorithm believes the\n> > > > > + scene is in focus.\n> > > > > + - name: AfStateFailed\n> > > > > + value: 3\n> > > > > + description: |\n> > > > > + An AF scan has been performed but the algorithm has not been able\n> > > > > + to find the best focus position.\n> > > > > +\n> > > > > ...\n> > > > > --\n> > > > > 2.30.2\n> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2AB8ABF415\n\tfor ;\n\tMon, 17 Jan 2022 10:13:35 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6AC79604F8;\n\tMon, 17 Jan 2022 11:13:34 +0100 (CET)","from mail-ot1-x329.google.com (mail-ot1-x329.google.com\n\t[IPv6:2607:f8b0:4864:20::329])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2968F60213\n\tfor ;\n\tMon, 17 Jan 2022 11:13:33 +0100 (CET)","by mail-ot1-x329.google.com with SMTP id\n\tz25-20020a0568301db900b005946f536d85so10139821oti.9\n\tfor ;\n\tMon, 17 Jan 2022 02:13:33 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"J744zBhx\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=AgjQzMnWf8ygdAG/OAm0+LKm1ky5V/10RTZ7vfblEd4=;\n\tb=J744zBhxYRyPDRht79DfEfsu7nrJyRFgZ4lJZWWLiAY4jXxGkNcHFGvkojAMzcKZoJ\n\tZA5sgjAvjLTxbQ81JFh5e+AZKjSSlEuuCLfgM3Tz7PDhTWW/3G/pWy5U8QmhFmWjff+v\n\tDrjocDo4ABFwIyGbAI+QaYoDkD/UpW/yopKpQ=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=AgjQzMnWf8ygdAG/OAm0+LKm1ky5V/10RTZ7vfblEd4=;\n\tb=Gz6CNyP1BbJydq+EGQnO7RHOcIGLEwKSZ+RAImhA6J3835Sg42O1GeWRQ5jAV6UxMQ\n\ta02FcJXLbv+DU5gClXO6bIxz5hpUCPaKoeFHtNnTRiGv36v32WnGpRbaokC9xSjYtIij\n\t7mW0YT4abS5cSgoUaOHt6Pg1HS1pntlMbEfjfta2rBJCWLhdQt5yvFv6yrdo4VuQbdiw\n\tSN2ttsQE5TG0ZiFBE1GRPBFgNTRyMhiT8vIs3J+s+tqarrBw1fQKTMUE4Pl4ldj2pHZj\n\tGRANCgk1JtPnVFL5FQYlQouXjQR/ElJ4KejKERVrmj8UUVToGr+dTNmACZCj6KgR+6av\n\tXN3A==","X-Gm-Message-State":"AOAM533uOkWreJP6e+QMDO253ZkgS8P3AndEWd7WuB32joTy58xUM+DG\n\tWNkRSX0vnotlGkWuWnYdVbhUWKbluxNRDARtqDmx6w==","X-Google-Smtp-Source":"ABdhPJzmykQZPybQszfPCz+uFCboMUpB5Q5lHunzrE67EdbM2MB546V8xQGiixo95mtco+xlaBKK+WfijdU0P6F0eyU=","X-Received":"by 2002:a05:6830:19ee:: with SMTP id\n\tt14mr14644002ott.176.1642414411472; \n\tMon, 17 Jan 2022 02:13:31 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t\n\t\n\t\n\t","In-Reply-To":"","From":"Hanlin Chen ","Date":"Mon, 17 Jan 2022 18:13:20 +0800","Message-ID":"","To":"David Plowman ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22047,"web_url":"https://patchwork.libcamera.org/comment/22047/","msgid":"","date":"2022-01-17T15:48:55","subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi again everyone\n\nThanks for the latest replies and all the discussions. I think the best\nthing is probably if I get on and produce a v2 of this patch. I think the\nmain differences will be: a \"pause\" control for CAF, and I might add an\n\"off\" state in which the lens can be moved. Then we can resume the\ndiscussion once more!\n\nBest regards\n\nDavid\n\nOn Mon, 17 Jan 2022 at 10:13, Hanlin Chen wrote:\n\n> The mail is re-sent by another account to avoid DMARC rejection. Sorry\n> for the inconvenience.\n>\n> Hi David,\n> Thanks for the comments.\n>\n> On Fri, Jan 14, 2022 at 1:54 AM David Plowman\n> wrote:\n> >\n> > Hi Hanlin\n> >\n> > Sorry for taking some time to get back to this!\n> No worries. We all work asynchronously :)\n>\n> > From one of your other emails, and checking the Android documentation,\n> > I see that \"Trigger\" does mean something in CAF mode though, as you\n> > say, a \"pause\" control captures the meaning a bit better. I think\n> > Android uses \"trigger start\" to mean:\n> >\n> > - In \"video\" CAF mode: pause immediately, whatever is happening.\n> > - In \"picture\" CAF mode: pause when any current (passive) scan is\n> finished.\n> >\n> > Perhaps a pause control would have 3 values: \"pause immediately\",\n> > \"pause when ready\" and \"resume\".\n> I moved the comment up, so I can comment on it earlier.\n> Yes, I think it is more clear, although the algorithm may need to\n> implement it for both video and picture CAF.\n> I suppose it's fine?\n> >\n> > On Tue, 21 Dec 2021 at 10:21, Hanlin Chen \n> wrote:\n> > >\n> > > Hi David, thanks for the comments.\n> > >\n> > > On Fri, Dec 17, 2021 at 4:51 PM David Plowman\n> > > wrote:\n> > > >\n> > > > Hi again\n> > > >\n> > > > Thanks for the comments!\n> > > >\n> > > > On Thu, 16 Dec 2021 at 11:18, Hanlin Chen \n> wrote:\n> > > > >\n> > > > > Hi David, thanks for this work!\n> > > > >\n> > > > > On Mon, Dec 13, 2021 at 11:22 PM David Plowman\n> > > > > wrote:\n> > > > > >\n> > > > > > This patch describes a series of controls that allow\n> applications to\n> > > > > > drive AF algorithms:\n> > > > > >\n> > > > > > AfMode - auto or continuous\n> > > > > > AfRange - full, macro or normal\n> > > > > > AfSpeed - fast or slow\n> > > > > > AfMethod - single or multi-spot\n> > > > > > AfWindow - AF window locations\n> > > > > > AfTrigger - start (trigger an AF scan) or cancel\n> > > > > > LensPosition - position of lens from lens driver\n> > > > > > AfState - reset, scanning, focused or failed\n> > > > > > ---\n> > > > > > src/libcamera/control_ids.yaml | 227\n> ++++++++++++++++++++++++---------\n> > > > > > 1 file changed, 167 insertions(+), 60 deletions(-)\n> > > > > >\n> > > > > > diff --git a/src/libcamera/control_ids.yaml\n> b/src/libcamera/control_ids.yaml\n> > > > > > index 9d4638ae..e579f7b7 100644\n> > > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > > @@ -406,27 +406,6 @@ controls:\n> > > > > > The camera will cancel any active or completed\n> metering sequence.\n> > > > > > The AE algorithm is reset to its initial state.\n> > > > > >\n> > > > > > - - AfTrigger:\n> > > > > > - type: int32_t\n> > > > > > - draft: true\n> > > > > > - description: |\n> > > > > > - Control for AF trigger. Currently identical to\n> > > > > > - ANDROID_CONTROL_AF_TRIGGER.\n> > > > > > -\n> > > > > > - Whether the camera device will trigger autofocus for\n> this request.\n> > > > > > - enum:\n> > > > > > - - name: AfTriggerIdle\n> > > > > > - value: 0\n> > > > > > - description: The trigger is idle.\n> > > > > > - - name: AfTriggerStart\n> > > > > > - value: 1\n> > > > > > - description: The AF routine is started by the camera.\n> > > > > > - - name: AfTriggerCancel\n> > > > > > - value: 2\n> > > > > > - description: |\n> > > > > > - The camera will cancel any active trigger and the\n> AF routine is\n> > > > > > - reset to its initial state.\n> > > > > > -\n> > > > > > - NoiseReductionMode:\n> > > > > > type: int32_t\n> > > > > > draft: true\n> > > > > > @@ -507,45 +486,6 @@ controls:\n> > > > > > The AE algorithm has started a pre-capture metering\n> session.\n> > > > > > \\sa AePrecaptureTrigger\n> > > > > >\n> > > > > > - - AfState:\n> > > > > > - type: int32_t\n> > > > > > - draft: true\n> > > > > > - description: |\n> > > > > > - Control to report the current AF algorithm state.\n> Currently identical to\n> > > > > > - ANDROID_CONTROL_AF_STATE.\n> > > > > > -\n> > > > > > - Current state of the AF algorithm.\n> > > > > > - enum:\n> > > > > > - - name: AfStateInactive\n> > > > > > - value: 0\n> > > > > > - description: The AF algorithm is inactive.\n> > > > > > - - name: AfStatePassiveScan\n> > > > > > - value: 1\n> > > > > > - description: |\n> > > > > > - AF is performing a passive scan of the scene in\n> continuous\n> > > > > > - auto-focus mode.\n> > > > > > - - name: AfStatePassiveFocused\n> > > > > > - value: 2\n> > > > > > - description: |\n> > > > > > - AF believes the scene is in focus, but might\n> restart scanning.\n> > > > > > - - name: AfStateActiveScan\n> > > > > > - value: 3\n> > > > > > - description: |\n> > > > > > - AF is performing a scan triggered by an AF trigger\n> request.\n> > > > > > - \\sa AfTrigger\n> > > > > > - - name: AfStateFocusedLock\n> > > > > > - value: 4\n> > > > > > - description: |\n> > > > > > - AF believes has focused correctly and has locked\n> focus.\n> > > > > > - - name: AfStateNotFocusedLock\n> > > > > > - value: 5\n> > > > > > - description: |\n> > > > > > - AF has not been able to focus and has locked.\n> > > > > > - - name: AfStatePassiveUnfocused\n> > > > > > - value: 6\n> > > > > > - description: |\n> > > > > > - AF has completed a passive scan without finding\n> focus.\n> > > > > > -\n> > > > > > - AwbState:\n> > > > > > type: int32_t\n> > > > > > draft: true\n> > > > > > @@ -690,4 +630,171 @@ controls:\n> > > > > > value. All of the custom test patterns will be\n> static (that is the\n> > > > > > raw image must not vary from frame to frame).\n> > > > > >\n> > > > > > + - AfMode:\n> > > > > > + type: int32_t\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + Control to set the mode of the AF (autofocus) algorithm.\n> > > > > > + enum:\n> > > > > > + - name: AfModeAuto\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The AF algorithm is in auto mode. This means that\n> the algorithm\n> > > > > > + will never move the lens or change state unless the\n> AfTrigger\n> > > > > > + control is used. The AfTrigger control can be used\n> to initiate a\n> > > > > > + focus scan, the results of which will be reported\n> by the AfState.\n> > > > > > +\n> > > > > > + In this mode, an application can move the lens for\n> itself using\n> > > > > > + the LensPosition control. If the algorithm was\n> scanning when the\n> > > > > > + lens is moved in this way, then the scan is\n> implicitly cancelled.\n> > > > >\n> > > > > I would suggest splitting the Auto mode into AfModeOnDemand and\n> > > > > AfModeManual (The naming could be re-considered).\n> > > > > The reason is that the sub-controls for each AfModes could be\n> distinct.\n> > > > > 1. AfModeManual:\n> > > > > LensPosition is valid only for the mode, and AfState is always\n> AfStateReset.\n> > > > > 2. AfModeOnDemand\n> > > > > AfTrigger and AfRange are valid only for the mode, and AfState\n> reports\n> > > > > accordingly. The scan is implicitly canceled when changing from the\n> > > > > mode.\n> > > >\n> > > > I'm a little bit nervous about having a mode where the lens position\n> > > > control doesn't work. In my earlier email I described performing an\n> AF\n> > > > scan, capturing an image, then moving the lens back to hyperfocal. It\n> > > > seems inconvenient to have to put the AF algorithm into a separate\n> > > > mode in order to move the lens, only to put it back in the mode where\n> > > > a scan is once again allowed.\n> > > >\n> > > > I don't think anything is lost by doing it this way - if an\n> > > > application doesn't want to perform an AF scan it should simply not\n> > > > request one.\n> > > I agreed that it does cover the controls from Android, i.e., the\n> > > Android request can be translated to the API.\n> > > My concern is from another direction: translation from libcamera to\n> > > Android Camera3.\n> > > From the Android to libcamera:\n> > > ANDROID_CONTROL_AF_MODE_OFF + ANDROID_LENS_FOCUS_DISTANCE ==>\n> > > AfModeAuto + lensPosition\n> > > ANDROID_CONTROL_AF_MODE_CONTINUOUS_VIDEO +\n> > > ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_START ==> AfModeAuto +\n> > > AfTrigger\n> >\n> > I don't quite understand how an AE_PRECAPTURE_TRIGGER_START affects\n> > the AF state machine, isn't that something we'd send to the AE\n> > algorithm (if it had such a control)? And then we'd wait for the AE\n> > state to say it was finished?\n>\n> Ah... It's a copy error, I simply mean AF_TRIGGER here.\n> >\n> > > From libcamera to Android:\n> > > AfModeAuto + lensPostion (Assuming current position is reported by\n> > > libcamera) ==> ??\n> > > It could be CAF, AUTO or OFF mode.\n> >\n> > As I understand it, Android only lets you set the lens position when\n> > AF mode is \"OFF\". So in this case we'd have to set the Android AF to\n> > \"OFF\" along with the lens position, right?\n> >\n> > If the next thing libcamera wants to do is something other than move\n> > the lens, then we'll have to set Android back to the state\n> > corresponding to libcamera's. Does that work?\n> >\n> The context here is that I seem to assume that if we don't have\n> \"pause\" in CAF, we need to mimic it by changing Auto with\n> lensPosition.\n> Thus, when translating back to Android, it's not clear that it's Auto\n> or CAF with mimicking \"pause\".\n> Maybe my assumption is wrong?\n>\n> However, if we have \"pause\", then the problem disappears:\n> with lensPosition == OFF, without == AUTO.\n>\n> > > States from libcamera Android:\n> > > AfModeAuto + AfTrigger + focused ==>\n> > > ANDROID_CONTROL_AF_STATE_FOCUSED_LOCKED? or\n> > > ANDROID_CONTROL_AF_STATE_PASSIVE_FOCUSED?\n> >\n> > If there was a trigger, then I believe the \"focused\" state needs to\n> > correspond to AF_STATE_FOCUSED_LOCKED. In fact, is it the case that in\n> > auto mode, Android can never report AF_STATE_PASSIVE_FOCUSED?\n> Same above.\n> Yes, if it's AF_MODE_AUTO, it never report AF_STATE_PASSIVE_FOCUSED.\n> >\n> > >\n> > > For the Android adapter layer, above might be workarounded by caching\n> > > the requests and do reference and find out current context.\n> > > A more subtle case might arise when integrating other platforms.\n> > > For example, Qualcomm closed 3A lib follows the modes and states\n> > > according to Android (Partly true for Intel and MTK).\n> > > In the case, the translation flow will like:\n> > > Android API -> libcamera API -> Android like API for 3A lib ->\n> > > libcamera API -> Android API\n> > > \n> > > \n> > > It's hard to predict how to work around all corner cases.\n> > > This makes me believe that if we could have a clear one to one mapping\n> > > between Android and libcamera API, it could be beneficial later.\n> > > However, it will make the API more verbose.\n> > >\n> > > Another perspective is that Android defines required state machine\n> transition:\n> > > https://source.android.com/devices/camera/camera3_3Amodes\n> > >\n> > > For IPA to implement the state transition satisfying Android's\n> > > requirement, the IPA should know exactly how the libcamera states\n> > > mapping back to the Android states.\n> > > Above is not satisfying because we will be somehow restricted by\n> > > Android's definition ;-|, However, I think it could be beneficial if\n> > > the API cam map libcamera states/modes to all Android states/modes\n> > > directly without extra context/call order information.\n> >\n> > I'm really in two minds about the question of an \"off\" mode in\n> > libcamera. Either we make libcamera application code handle the\n> > transition to \"off\" and back to \"auto\", or a libcamera-to-Android\n> > translation layer would have to do it. Does anyone else out there feel\n> > strongly?\n>\n> I suppose we could align it with AE (or let AE align with AF :P). AE\n> has a similar case for AnalogueGainMode and ExposureTimeMode.\n> https://patchwork.libcamera.org/patch/15179/\n> Where Paul seems to use \"Auto\" and \"Disabled\" for the case.\n> cc'ed Paul.\n>\n> >\n> > >\n> > > >\n> > > > I think AfRange applies to CAF as well. I see no reason why you might\n> > > > not want to restrict the range of searches for just the same reasons\n> > > > as you might do with \"regular\" AF.\n> > > Agreed.\n> > > >\n> > > > > 3. AfModeContinuous\n> > > > > AfSpeed and \"pause\" are valid only for the mode, and AfState\n> reports\n> > > > > accordingly. The scan is implicitly canceled when changing from the\n> > > > > mode.\n> > > >\n> > > > I'm not sure AfSpeed is *necessarily* only for CAF. I think we've\n> done\n> > > > AF systems in the past where even regular AF has a \"I really want to\n> > > > take the picture quickly\" mode. But it could be a platform choice?\n> > > Agreed.\n> > > >\n> > > > >\n> > > > > > + - name: AfModeContinuous\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + The AF algorithm is in continuous mode. This means\n> that the lens\n> > > > > > + can re-start a scan spontaneously at any moment,\n> without any user\n> > > > > > + intervention. The AfState still reports whether the\n> algorithm is\n> > > > > > + currently scanning or not, though the application\n> has no ability\n> > > > > > + to initiate or cancel scans, nor move the lens for\n> itself.\n> > > > > > +\n> > > > > > + The mode can be set to AfModeAuto which has the\n> effect of\n> > > > > > + \"pausing\" any continuous AF activity, after which\n> it could then\n> > > > > > + be moved back to AfModeContinuous to resume\n> operation.\n> > > > > > +\n> > > > > > + - AfRange:\n> > > > > > + type: int32_t\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + Control to set the range of focus distances that is\n> scanned.\n> > > > > > + enum:\n> > > > > > + - name: AfRangeFull\n> > > > > > + value: 0\n> > > > > > + description: The full range of focus distances is\n> scanned.\n> > > > > > + - name: AfRangeMacro\n> > > > > > + value: 1\n> > > > > > + description: Only close distances are scanned.\n> > > > > > + - name: AfRangeNormal\n> > > > > > + value: 2\n> > > > > > + description: |\n> > > > > > + The full range of focus distances is scanned except\n> for some of\n> > > > > > + the closest macro positions.\n> > > > > > +\n> > > > > > + - AfSpeed:\n> > > > > > + type: int32_t\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + Control that determines whether the AF algorithm is to\n> move the lens\n> > > > > > + as quickly as possible or more steadily. For example,\n> during video\n> > > > > > + recording it may be desirable not to move the lens too\n> abruptly, but\n> > > > > > + when in a preview mode (waiting for a still capture) it\n> may be\n> > > > > > + helpful to move the lens as quickly as is reasonably\n> possible.\n> > > > > > + enum:\n> > > > > > + - name: AfSpeedFast\n> > > > > > + value: 0\n> > > > > > + description: Move the lens quickly.\n> > > > > > + - name: AfSpeedSlow\n> > > > > > + value: 0\n> > > > > > + description: Move the lens more steadily.\n> > > > > How about\n> > > > > s/Fast/Aggressively\n> > > > > s/Slow/Steadily\n> > > >\n> > > > I agree the choice is tricky here. I think I would regard what I\n> > > > called \"Slow\" to be the \"usual\" choice, so perhaps \"AfSpeedNormal\" is\n> > > > better (and maybe make that value 0). I still prefer \"AfModeFast\" to\n> > > > \"AfModeAggressive\" - \"aggressive\" sounds just a bit too...\n> aggressive?\n> > > > :)\n> > > LGTM.\n> > > >\n> > > > But thanks for all the discussion. Glad we're making progress on all\n> this!\n> > > >\n> > > > Best regards\n> > > > David\n> > > >\n> > > > > > +\n> > > > > > + - AfMethod:\n> > > > > > + type: int32_t\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + Control whether the AF algorithm uses a single window\n> in the image to\n> > > > > > + determine the best focus position, or multiple windows\n> simultaneously.\n> > > > > > + enum:\n> > > > > > + - name: AfMethodSingle\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + A single window within the image, defaulting to the\n> centre, is used\n> > > > > > + to select the best focus distance.\n> > > > > > + - name: AfMethodMultiSpot\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + Multiple windows within the image are used to\n> select the best focus\n> > > > > > + distance. The best focus distance is found for each\n> one of the\n> > > > > > + windows, and then the distance that is closest to\n> the camera is\n> > > > > > + selected.\n> > > > > > +\n> > > > > > + - AfWindow:\n> > > > > > + type: Rectangle\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + Sets the focus windows used by the AF algorithm. The\n> units used express\n> > > > > > + a proportion of the ScalerCrop control (or if\n> unavailable, of the entire\n> > > > > > + image), as u0.16 format numbers.\n> > > > > > +\n> > > > > > + In order to be activated, a rectangle must be\n> programmed with non-zero\n> > > > > > + width and height. If no rectangles are programmed in\n> this way, then the\n> > > > > > + system will choose its own single default window in\n> the centre of the\n> > > > > > + image.\n> > > > > > +\n> > > > > > + If AfMethod is set to AfMethodSingle, then only the\n> first Rectangle in\n> > > > > > + this list is used (or the system default one if it is\n> unprogrammed).\n> > > > > > +\n> > > > > > + If AfMethod is set to AfMethodMultiSpot then all the\n> valid Rectangles in\n> > > > > > + this list are used. The size of the control indicates\n> how many such\n> > > > > > + windows can be programmed and will vary between\n> different platforms.\n> > > > > > +\n> > > > > > + size: [platform dependent]\n> > > > > > +\n> > > > > > + - AfTrigger:\n> > > > > > + type: int32_t\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + This control starts an autofocus scan when AfMode is\n> set to AfModeAuto,\n> > > > > > + and can also be used to terminate a scan early.\n> > > > > > +\n> > > > > > + It is ignored if AfMode is set to AfModeContinuous.\n> > > > > > +\n> > > > > > + enum:\n> > > > > > + - name: AfTriggerStart\n> > > > > > + value: 0\n> > > > > > + description: Start an AF scan. Ignored if a scan is\n> in progress.\n> > > > > > + - name: AfTriggerCancel\n> > > > > > + value: 1\n> > > > > > + description: Cancel an AF scan. Ingored if no scan is\n> in progress.\n> >\n> > From one of your other emails, and checking the Android documentation,\n> > I see that \"Trigger\" does mean something in CAF mode though, as you\n> > say, a \"pause\" control captures the meaning a bit better. I think\n> > Android uses \"trigger start\" to mean:\n> >\n> > - In \"video\" CAF mode: pause immediately, whatever is happening.\n> > - In \"picture\" CAF mode: pause when any current (passive) scan is\n> finished.\n> >\n> > Perhaps a pause control would have 3 values: \"pause immediately\",\n> > \"pause when ready\" and \"resume\".\n> >\n> > Thanks and best regards\n> >\n> > David\n> >\n> > > > > > +\n> > > > > > + - LensPosition:\n> > > > > > + type: int32_t\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + Acts as a control to instruct the lens to move to a\n> particular position\n> > > > > > + and also reports back the position of the lens for\n> each frame.\n> > > > > > +\n> > > > > > + The units are determined by the lens driver.\n> > > > > > +\n> > > > > > + If the LensPosition control is used while an AF scan\n> is in progress,\n> > > > > > + the scan is implicitly cancelled and the lens is then\n> moved to the\n> > > > > > + desired location. It is ignored if AfMode is set to\n> AfModeContinuous.\n> > > > > > +\n> > > > > > + - AfState:\n> > > > > > + type: int32_t\n> > > > > > + draft: true\n> > > > > > + description: |\n> > > > > > + Reports the current state of the AF algorithm.\n> > > > > > + enum:\n> > > > > > + - name: AfStateReset\n> > > > > > + value: 0\n> > > > > > + description: |\n> > > > > > + The AF algorithm reports this state when:\n> > > > > > + * The camera system has just been started.\n> > > > > > + * A scan has been cancelled.\n> > > > > > + * The lens has been moved by the LensPosition\n> control.\n> > > > > > + - name: AfStateScanning\n> > > > > > + value: 1\n> > > > > > + description: |\n> > > > > > + AF is performing a scan. This state can be\n> entered spontaneously\n> > > > > > + if AfMode is set to AfModeContinuous, otherwise\n> it requires the\n> > > > > > + application to use the AfTrigger control to start\n> the scan.\n> > > > > > + - name: AfStateFocused\n> > > > > > + value: 2\n> > > > > > + description: |\n> > > > > > + An AF scan has been performed and the algorithm\n> believes the\n> > > > > > + scene is in focus.\n> > > > > > + - name: AfStateFailed\n> > > > > > + value: 3\n> > > > > > + description: |\n> > > > > > + An AF scan has been performed but the algorithm\n> has not been able\n> > > > > > + to find the best focus position.\n> > > > > > +\n> > > > > > ...\n> > > > > > --\n> > > > > > 2.30.2\n> > > > > >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id F00CFBF415\n\tfor ;\n\tMon, 17 Jan 2022 15:49:09 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D847060940;\n\tMon, 17 Jan 2022 16:49:08 +0100 (CET)","from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com\n\t[IPv6:2a00:1450:4864:20::32a])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7AE346021A\n\tfor ;\n\tMon, 17 Jan 2022 16:49:07 +0100 (CET)","by mail-wm1-x32a.google.com with SMTP id p18so21305403wmg.4\n\tfor ;\n\tMon, 17 Jan 2022 07:49:07 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"sRLsijr+\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=c2dkRcRQT5IdDeqVsz6ThaCyGUDDSQxrYmotccmsZGo=;\n\tb=sRLsijr+4b6RaagAGz1NaQbVK+ONRSCJsNtUn+bQfhxvvHndrq6s/OQObz+GfmxYFx\n\t0tZ2ue9n0D6LdE1c/8Sz5O80bhsG+VyYvRHYllP5XSneNZ8mBoKD7dYMeNfcrfcyPa4J\n\tzOj6mLiV/3tCcWToLPi9iZgYd3MB6FNTm4YNqQimlZUYSGti6Jm04xqCYGDejYsWYW7H\n\tf1l/AHCSWqWJIyTio0PdGxxW+M32zTsmaeQBjG4721WzlVio/hqHJmdInVjFaJD8T2h3\n\tkYlYvn8JT/z8ADnX/T13XkPugjJk3E6Q17kL57nBClPt36j5PohJcHlYJAfAiOiLnsE+\n\tUXrw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=c2dkRcRQT5IdDeqVsz6ThaCyGUDDSQxrYmotccmsZGo=;\n\tb=2mKnmRFYb3OGfkFDhPvHW4gscl+5TILD7WE1HXK0iN/ZZ/4W140TOeJ1w6yk5+gkKO\n\tZ3/s59dm6gzrfFeRO4wZA6n9u2pOf4oPZ/uayOTbpSViObMF92aIGKwECKDGQUlKqWU2\n\t5Qatesopuwl/MqVb0q4BsDO48yQzDM7yUyxxxvIYiAy9Zk6VR48ScfzFSMBnQCLoqqZQ\n\tLZQDQVBk7MSB5Wh0dzUCpvgk67moyYRdcHLSjvMGwgD7tTceVwuOW78zaWvPchbz3/lh\n\tV0m2Ny9yj5z5kqN7Fis6oCravoU4qEpzch2Bc1dVgeqjJkIlbRZdFPWCfc/iEltUF4VH\n\to2VA==","X-Gm-Message-State":"AOAM532z/vdEIfGXqFZcb0ljN/mGiwS6p/2eM1AnanN9UnhwCsdg2cQb\n\tbg7G/KBDPokxNRn3cyA5/olauW0GRCn+AY4dc1jAQA==","X-Google-Smtp-Source":"ABdhPJydhZEsDmizofTxzP5pTLe5nCbZ6nEfQ8H1m/84CQXQYPBecc272RG4n6Bect9XP/1UN0w6yetO2fDkF1zP7ZU=","X-Received":"by 2002:a5d:64c8:: with SMTP id f8mr6291551wri.3.1642434547034; \n\tMon, 17 Jan 2022 07:49:07 -0800 (PST)","MIME-Version":"1.0","References":"<20211213152215.17584-1-david.plowman@raspberrypi.com>\n\t<20211213152215.17584-2-david.plowman@raspberrypi.com>\n\t\n\t\n\t\n\t\n\t","In-Reply-To":"","From":"David Plowman ","Date":"Mon, 17 Jan 2022 15:48:55 +0000","Message-ID":"","To":"Hanlin Chen ","Content-Type":"multipart/alternative; boundary=\"000000000000be41ba05d5c917db\"","Subject":"Re: [libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls\n\tfor driving AF (autofocus) algorithms","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]