[{"id":22355,"web_url":"https://patchwork.libcamera.org/comment/22355/","msgid":"","date":"2022-03-22T14:01:42","subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate !\n\nOn 22/03/2022 03:41, Kate Hsuan via libcamera-devel wrote:\n> The floating results of AWB and AE significantly impact the AF\n> performance and the AF scene variance estimation. If the exposure value\n> and color settings are changed by the algorithm, the AF scene variance will\n> be affected and the algorithm may find an incorrect local maximum value.\n> Consequently, a wrong lens position is determined and we could only get\n> a blurred image.\n\nI agree with you, at least on one assumption: AWB can have an impact on \nAF. AE is more debatable I think :-).\nI think we should consider something a bit different from what you did, \nif you don't mind, as I don't think AWB should know anything out of its \nown scope, same for AGC, same for AF, same for... you get it :-).\nThis should be taken into account at the IPA level, the controller, \nwhich could set a state for an algorithm.\n\nI think that, while autofocus is starting, or in coarse search mode, we \nshould not change the awb gains. When is gets into the fine search mode, \nthen AWB could be started as it should not interfere anymore. So, I \nthink AF could have a 'status' field in its frameContext, and the \nalgorithms (AWB, AGC, probably AF) should be \"pausable\". David sent a \npatch lately for RPi [1] and this is something we should consider.\n\nThat beeing said, and I hope I am clear in my explanations, before \ncalling the process() call for the algorithms, the IPU3IPU could set the \nstatus of AWB and AGC based on the AF status. I would start AF after AGC \nis stable enough (could be based on a number of frames for a first try) \nand I would enable AWB while AGC is starting.\n\nOnce AGC is stable, AF is entering the coarse search, and in this phase \nAWB is paused. Once it enters the fine search, we can resume AWB.\n\nI think it should be enough, as we are not zooming, so illumination is \nnot changing a lot with focus change AFAIK.\n\nI don't really have time to test it so if you want/can I would greatly \nappreciate :-).\n\nThanks !\nJM\n\n[1]: https://patchwork.libcamera.org/patch/15343/\n\n> \n> This patch proposed an AWB and AE suspension mechanism while AF is\n> searching for the focus. A lock flag is used to identify that AWB and\n> AE should be suspended or not. AGC and AE algorithm test the flag and\n> determine that they have to perform the algorithm or suspend itself.\n> \n> Kate Hsuan (4):\n> ipa: ipu3: af: Introduce AWB and AE lock when AF scanning\n> ipa: ipu3: af: AE and AWE lock request flow when performing AF\n> ipa: ipu3: awb: AWB lock for AF scan\n> ipa: ipu3: agc: AE lock for AF scanning\n> \n> src/ipa/ipu3/algorithms/af.cpp | 25 +++++++++++++++++++++++++\n> src/ipa/ipu3/algorithms/af.h | 2 ++\n> src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++++++++++\n> src/ipa/ipu3/algorithms/agc.h | 1 +\n> src/ipa/ipu3/algorithms/awb.cpp | 13 +++++++++++++\n> src/ipa/ipu3/algorithms/awb.h | 1 +\n> src/ipa/ipu3/ipa_context.h | 1 +\n> 7 files changed, 57 insertions(+)\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D8FC6BD80A\n\tfor ;\n\tTue, 22 Mar 2022 14:01:47 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 27B43604DB;\n\tTue, 22 Mar 2022 15:01:47 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2520F604C5\n\tfor ;\n\tTue, 22 Mar 2022 15:01:46 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:fde5:4bb:7d84:65f8] (unknown\n\t[IPv6:2a01:e0a:169:7140:fde5:4bb:7d84:65f8])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id C0F9DDFA;\n\tTue, 22 Mar 2022 15:01:45 +0100 (CET)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1647957707;\n\tbh=Gw9sW+PbSVi9SdXhYR9mtmH9m7TzbiXPBmi6fDKvNsI=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=MHj95FxKcneRgoKoOHBxc1YQ27+86Quly8glCCXWFvd/eDeMYi9Dz88b9NdOZlbFL\n\tX/LqE7qU5BUrwPHhYNWE8CqkJQoq+LMjtPdaWo8helkTiZqgAV1l2tQlM5Evm/+ODy\n\ti93WWFuSYeOQV9XvkG0DMSm+lz/ZyKvp6RUTJSIiZuOmxvkt7Yt07AycCPHyNUIub0\n\tHgM9Sy0QpuQhr8sWF5ud2SvuohyJbUptX/e3I+6m0KJTSEvOpz1wwApVex8HppOH4v\n\tyCZ9GcbAxjtQAY55QYkutzT1AMu3RQxeaofxZN2xY8MxoriQFWLcsZ1Y3xRFrK+7xw\n\tt0pNw62b9FIqg==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1647957705;\n\tbh=Gw9sW+PbSVi9SdXhYR9mtmH9m7TzbiXPBmi6fDKvNsI=;\n\th=Date:Subject:To:References:From:In-Reply-To:From;\n\tb=ujsaiNeRIcWGCwE0d6agWDdmt94pvHO8NK3vY2fs6DGOOQEX2m2Hirm9FDJMylRZe\n\tHn5G4a0M5E+xQ+s/X4nOTxWAP6kesjiNKYNnqSvbaz5LmimMqi8Jukz4f4Mh4/kpFx\n\tjCSPHNZqVhyMEQlxzoh5yZYB7K86yicV7+wOZIes="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"ujsaiNeR\"; dkim-atps=neutral","Message-ID":"","Date":"Tue, 22 Mar 2022 15:01:42 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.5.0","Content-Language":"en-US","To":"Kate Hsuan ,\n\tlibcamera devel ","References":"<20220322024155.6528-1-hpa@redhat.com>","In-Reply-To":"<20220322024155.6528-1-hpa@redhat.com>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jean-Michel Hautbois via libcamera-devel\n\t","Reply-To":"Jean-Michel Hautbois ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22406,"web_url":"https://patchwork.libcamera.org/comment/22406/","msgid":"","date":"2022-03-23T13:30:42","subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","submitter":{"id":102,"url":"https://patchwork.libcamera.org/api/people/102/","name":"Hans de Goede","email":"hdegoede@redhat.com"},"content":"Hi,\n\nOn 3/22/22 15:01, Jean-Michel Hautbois via libcamera-devel wrote:\n> Hi Kate !\n> \n> On 22/03/2022 03:41, Kate Hsuan via libcamera-devel wrote:\n>> The floating results of AWB and AE significantly impact the AF\n>> performance and the AF scene variance estimation. If the exposure value\n>> and color settings are changed by the algorithm, the AF scene variance will\n>> be affected and the algorithm may find an incorrect local maximum value.\n>> Consequently, a wrong lens position is determined and we could only get\n>> a blurred image.\n> \n> I agree with you, at least on one assumption: AWB can have an impact on AF. AE is more debatable I think :-).\n\nInteresting I would actually expect the importance to be the otherway\naround. If AE is way off, so way overexposed or much to dark, there\nis not much on which the focus algorithm can actually focus. Where as\nif an image is e.g. to \"green\" I would not expect that to make a lot\nof difference for the contrast of the image.\n\n> I think we should consider something a bit different from what you did, if you don't mind, as I don't think AWB should know anything out of its own scope, same for AGC, same for AF, same for... you get it :-).\n> This should be taken into account at the IPA level, the controller, which could set a state for an algorithm.\n\nI agree, but I think we want this on other IPU-s too, so maybe we need to do this\nat an even higher level, assuming that is possible (I still need\nto familiarize myself with libcamera more) ?\n\n> I think that, while autofocus is starting, or in coarse search mode, we should not change the awb gains. When is gets into the fine search mode, then AWB could be started as it should not interfere anymore. So, I think AF could have a 'status' field in its frameContext, and the algorithms (AWB, AGC, probably AF) should be \"pausable\". David sent a patch lately for RPi [1] and this is something we should consider.\n> \n> That beeing said, and I hope I am clear in my explanations, before calling the process() call for the algorithms, the IPU3IPU could set the status of AWB and AGC based on the AF status. I would start AF after AGC is stable enough (could be based on a number of frames for a first try) and I would enable AWB while AGC is starting.\n> \n> Once AGC is stable, AF is entering the coarse search, and in this phase AWB is paused. Once it enters the fine search, we can resume AWB.\n\nAgreed I think we also need some sort of AGC \"done\" bit because as\nsaid I don't think doing auto-focus before we at least have the\nexposure setup somewhat correct makes a lot of sense.\n\n> I think it should be enough, as we are not zooming, so illumination is not changing a lot with focus change AFAIK.\n\nRegards,\n\nHans\n\n\n\n> \n> I don't really have time to test it so if you want/can I would greatly appreciate :-).\n> \n> Thanks !\n> JM\n> \n> [1]: https://patchwork.libcamera.org/patch/15343/\n> \n>>\n>> This patch proposed an AWB and AE suspension mechanism while AF is\n>> searching for the focus. A lock flag is used to identify that AWB and\n>> AE should be suspended or not. AGC and AE algorithm test the flag and\n>> determine that they have to perform the algorithm or suspend itself.\n>>\n>> Kate Hsuan (4):\n>>    ipa: ipu3: af: Introduce AWB and AE lock when AF scanning\n>>    ipa: ipu3: af: AE and AWE lock request flow when performing AF\n>>    ipa: ipu3: awb: AWB lock for AF scan\n>>    ipa: ipu3: agc: AE lock for AF scanning\n>>\n>>   src/ipa/ipu3/algorithms/af.cpp  | 25 +++++++++++++++++++++++++\n>>   src/ipa/ipu3/algorithms/af.h    |  2 ++\n>>   src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++++++++++\n>>   src/ipa/ipu3/algorithms/agc.h   |  1 +\n>>   src/ipa/ipu3/algorithms/awb.cpp | 13 +++++++++++++\n>>   src/ipa/ipu3/algorithms/awb.h   |  1 +\n>>   src/ipa/ipu3/ipa_context.h      |  1 +\n>>   7 files changed, 57 insertions(+)\n>>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D7A4DBD80A\n\tfor ;\n\tWed, 23 Mar 2022 13:30:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 14D13604DC;\n\tWed, 23 Mar 2022 14:30:52 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 563DD604C4\n\tfor ;\n\tWed, 23 Mar 2022 14:30:50 +0100 (CET)","from mail-ed1-f69.google.com (mail-ed1-f69.google.com\n\t[209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-553-XrGC-AtdPFGTrhCYbt8pAQ-1; Wed, 23 Mar 2022 09:30:44 -0400","by mail-ed1-f69.google.com with SMTP id\n\tb71-20020a509f4d000000b00418d658e9d1so1023105edf.19\n\tfor ;\n\tWed, 23 Mar 2022 06:30:44 -0700 (PDT)","from ?IPV6:2001:1c00:c1e:bf00:cdb2:2781:c55:5db0?\n\t(2001-1c00-0c1e-bf00-cdb2-2781-0c55-5db0.cable.dynamic.v6.ziggo.nl.\n\t[2001:1c00:c1e:bf00:cdb2:2781:c55:5db0])\n\tby smtp.gmail.com with ESMTPSA id\n\tk3-20020a05640212c300b0041605b2d9c1sm11385197edx.58.2022.03.23.06.30.42\n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tWed, 23 Mar 2022 06:30:42 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1648042252;\n\tbh=BnSorwM/XFsXCZPozoAquNAeLkBsBDuEuVEFwPmrP5E=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=XVVKTOCbudoof82nkCXS3Ku+Mi3OorfOtb0OunAOAQc9Yxj1unPY3sJalS7DKf57N\n\tjBVpdAF7/AgK2Ca6az7UVekARdqp5t87UdmPwLyz11EzWc/TGYwya1EQei78lzPSML\n\tzQuiwZFYG78Ze0K6OMwC78+7gmRWqqg5L3WuG51eGqLGHs+NiQ7FOBP+o8jsqTQDlH\n\t0BVv6J7T6WQ4+EivhK3SOCEdwvA4naNZ0vE7ly+CrxdH4hgTvBPxb7K27fEqiDD5vl\n\tcJ3DzuLpBC0GEEVZuoEJlV9S77timy23hA87H+EoGO7a9A2GpnkU9ynhNejtwQWocW\n\tSM+X0y4Uk00rg==","v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1648042248;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=R3FDAaLRnID8P0CzVE6Bj6DK8MLEDg3hvFQsflQAvLU=;\n\tb=WxrgzqJ2nXUlDA5SmfTBN3Q+amgMs+R8whjqaQSmOPqeJ5GycygNZqCGA7db/lmtYhh12g\n\tKSqPpQj9UWmO+TbCg8/0ZVUcmspcmgGxpiXHsyIkiEeFXnSsgkjRk5JuTXdI8tiCTxat94\n\tCqhqmwiSESR2b6iKa7v2Ao8Q2A8VBxk="],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=redhat.com\n\theader.i=@redhat.com header.b=\"WxrgzqJ2\"; \n\tdkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hdegoede@redhat.com"],"X-MC-Unique":"XrGC-AtdPFGTrhCYbt8pAQ-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:message-id:date:mime-version:user-agent:subject\n\t:content-language:to:references:from:in-reply-to\n\t:content-transfer-encoding;\n\tbh=R3FDAaLRnID8P0CzVE6Bj6DK8MLEDg3hvFQsflQAvLU=;\n\tb=t08FGqyGrwHbkPsJiWT1rl6TuP6uBLagLscgXqAWQO+KNg0Tm9s3PKO09z0VcLZnL+\n\t777iZD6Yw6ul+0uZcbv4U4qMgkuGy8Y9qZ2GLFaU23YYGJMk9Q61cCHLKaZb5iWU7knX\n\tU6cZ9EQCxakL6anLKNRbn1p3yZid9hLrCKOsBxe+a2cH4cAPRjOEymyTNlD+NF77NxPx\n\tO5SPr7KGDOlG8MgZJEUtsizPu46/ACRsYRPdR4dAol9Susg1w6zbWX5i/HBW7nn8z6wU\n\tY7j/baX2ye61nincGRrnOxowoN8Aq1QHI4TFU3VjlLp0iPjDGVHp++V0FYaGoV65KvnG\n\tQdMg==","X-Gm-Message-State":"AOAM532ZsP63cpbAKLuU94pAyHTxyqwsKud5G1LotbCuloMrfAVZWsn7\n\tDBmdpDw2mrVF82Hg1XmbLm2+85CQI8cGX0Lb27zoWvxDZDpvA+YNoTQhJ9T1BPdOsGi6iZMuEjx\n\t39fCVShegThZJuYWrzfJ/HbjJ8SScE9TevQ==","X-Received":["by 2002:a17:906:2646:b0:6d5:d889:c92b with SMTP id\n\ti6-20020a170906264600b006d5d889c92bmr32829563ejc.696.1648042243669; \n\tWed, 23 Mar 2022 06:30:43 -0700 (PDT)","by 2002:a17:906:2646:b0:6d5:d889:c92b with SMTP id\n\ti6-20020a170906264600b006d5d889c92bmr32829527ejc.696.1648042243340; \n\tWed, 23 Mar 2022 06:30:43 -0700 (PDT)"],"X-Google-Smtp-Source":"ABdhPJwnoN/pDrJQmh0tvc2STOxsCWA6YjLXYlr3HTJoYfeJpL8YzW5eGK+Tuy9qwp+4yBgn9iKilw==","Message-ID":"","Date":"Wed, 23 Mar 2022 14:30:42 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.4.0","To":"Jean-Michel Hautbois ,\n\tKate Hsuan ,\n\tlibcamera devel ","References":"<20220322024155.6528-1-hpa@redhat.com>\n\t","In-Reply-To":"","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Language":"en-US","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Hans de Goede via libcamera-devel ","Reply-To":"Hans de Goede ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22408,"web_url":"https://patchwork.libcamera.org/comment/22408/","msgid":"<6eece3df-f384-4374-9dc0-fdddbe767474@ideasonboard.com>","date":"2022-03-23T13:52:14","subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hello Hans,\n\nOn 23/03/2022 14:30, Hans de Goede wrote:\n> Hi,\n> \n> On 3/22/22 15:01, Jean-Michel Hautbois via libcamera-devel wrote:\n>> Hi Kate !\n>>\n>> On 22/03/2022 03:41, Kate Hsuan via libcamera-devel wrote:\n>>> The floating results of AWB and AE significantly impact the AF\n>>> performance and the AF scene variance estimation. If the exposure value\n>>> and color settings are changed by the algorithm, the AF scene variance will\n>>> be affected and the algorithm may find an incorrect local maximum value.\n>>> Consequently, a wrong lens position is determined and we could only get\n>>> a blurred image.\n>>\n>> I agree with you, at least on one assumption: AWB can have an impact on AF. AE is more debatable I think :-).\n> \n> Interesting I would actually expect the importance to be the otherway\n> around. If AE is way off, so way overexposed or much to dark, there\n> is not much on which the focus algorithm can actually focus. Where as\n> if an image is e.g. to \"green\" I would not expect that to make a lot\n> of difference for the contrast of the image.\n\nWell, indeed, if your scene is too dark or too bright, you will have big \nissues. But in case you are flickering around a somehow not that bad \nvalue, then shouldn't really change the edges. On the contrary if you \nchange only the red channel for instance, you may experience weird \nthings... I am not sure that I am clear here :-). Maybe should we \nconduct experiments extracting the filters output in various conditions \nfor a given scene ?\n\n> \n>> I think we should consider something a bit different from what you did, if you don't mind, as I don't think AWB should know anything out of its own scope, same for AGC, same for AF, same for... you get it :-).\n>> This should be taken into account at the IPA level, the controller, which could set a state for an algorithm.\n> \n> I agree, but I think we want this on other IPU-s too, so maybe we need to do this\n> at an even higher level, assuming that is possible (I still need\n> to familiarize myself with libcamera more) ?\n> \n>> I think that, while autofocus is starting, or in coarse search mode, we should not change the awb gains. When is gets into the fine search mode, then AWB could be started as it should not interfere anymore. So, I think AF could have a 'status' field in its frameContext, and the algorithms (AWB, AGC, probably AF) should be \"pausable\". David sent a patch lately for RPi [1] and this is something we should consider.\n>>\n>> That beeing said, and I hope I am clear in my explanations, before calling the process() call for the algorithms, the IPU3IPU could set the status of AWB and AGC based on the AF status. I would start AF after AGC is stable enough (could be based on a number of frames for a first try) and I would enable AWB while AGC is starting.\n>>\n>> Once AGC is stable, AF is entering the coarse search, and in this phase AWB is paused. Once it enters the fine search, we can resume AWB.\n> \n> Agreed I think we also need some sort of AGC \"done\" bit because as\n> said I don't think doing auto-focus before we at least have the\n> exposure setup somewhat correct makes a lot of sense.\n> \n>> I think it should be enough, as we are not zooming, so illumination is not changing a lot with focus change AFAIK.\n> \n> Regards,\n> \n> Hans\n> \n> \n> \n>>\n>> I don't really have time to test it so if you want/can I would greatly appreciate :-).\n>>\n>> Thanks !\n>> JM\n>>\n>> [1]: https://patchwork.libcamera.org/patch/15343/\n>>\n>>>\n>>> This patch proposed an AWB and AE suspension mechanism while AF is\n>>> searching for the focus. A lock flag is used to identify that AWB and\n>>> AE should be suspended or not. AGC and AE algorithm test the flag and\n>>> determine that they have to perform the algorithm or suspend itself.\n>>>\n>>> Kate Hsuan (4):\n>>>    ipa: ipu3: af: Introduce AWB and AE lock when AF scanning\n>>>    ipa: ipu3: af: AE and AWE lock request flow when performing AF\n>>>    ipa: ipu3: awb: AWB lock for AF scan\n>>>    ipa: ipu3: agc: AE lock for AF scanning\n>>>\n>>>   src/ipa/ipu3/algorithms/af.cpp  | 25 +++++++++++++++++++++++++\n>>>   src/ipa/ipu3/algorithms/af.h    |  2 ++\n>>>   src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++++++++++\n>>>   src/ipa/ipu3/algorithms/agc.h   |  1 +\n>>>   src/ipa/ipu3/algorithms/awb.cpp | 13 +++++++++++++\n>>>   src/ipa/ipu3/algorithms/awb.h   |  1 +\n>>>   src/ipa/ipu3/ipa_context.h      |  1 +\n>>>   7 files changed, 57 insertions(+)\n>>>\n>>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7109FBD80A\n\tfor ;\n\tWed, 23 Mar 2022 13:52:19 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9FD40604DA;\n\tWed, 23 Mar 2022 14:52:18 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2A101604C4\n\tfor ;\n\tWed, 23 Mar 2022 14:52:17 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:2e05:b10:1542:d51b] (unknown\n\t[IPv6:2a01:e0a:169:7140:2e05:b10:1542:d51b])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id BD9C39DE;\n\tWed, 23 Mar 2022 14:52:16 +0100 (CET)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1648043538;\n\tbh=iHZCbNaLDTWXlHQmLjOo+9u9qhO/cyYBYbF+9PT4lB4=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=P2Xs8lOUZCQayNoFWJFkhC8acg5Q8ZMr1siAKdnNdt9Cj0Spba5eSIj7L147pPTXc\n\t5RmFW72hWLoiXPKYsywENkIUU+U+Qv+p+mLLe2rknWTz+3O1CO3JFNtj4hD9C2g5PF\n\tBoFthRxfXxNdMezfB5W+14I8GvqrLYCyiUKttYIfZum4AyBdJW3S0EhKnd1u7SX3xF\n\tH2y1TaXsCkUzmp6QkKCThC51o9/tew9q/rCucDtG5Pv0eMGVSDuHSyK/IhNqWMuT9U\n\tnjz0H1JIIYfDVxP0N19sVtmixISQlsPx3JmQreVPpq1r3OFtuHgLnZ0X49aD+/LZU8\n\tcq/VmMX6SERCg==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1648043536;\n\tbh=iHZCbNaLDTWXlHQmLjOo+9u9qhO/cyYBYbF+9PT4lB4=;\n\th=Date:Subject:To:References:From:In-Reply-To:From;\n\tb=lcgQ13RBeCSy2MDa3PQb3/7hazJ3I3NYUuoPbt+ncPS5MKjF8CYrj/cL6OTwafo5d\n\tWfeKLX56qzHOe2csxVVbcQk1+JIGocSQWCJgSwr5t6dpAVuQ6QBSAfwbLvLIE/O2Bk\n\tNIYSvNEHVJfilfEKuYWGT/lGaityETEY73FSWoNY="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"lcgQ13RB\"; dkim-atps=neutral","Message-ID":"<6eece3df-f384-4374-9dc0-fdddbe767474@ideasonboard.com>","Date":"Wed, 23 Mar 2022 14:52:14 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.5.0","Content-Language":"en-US","To":"Hans de Goede , Kate Hsuan ,\n\tlibcamera devel ","References":"<20220322024155.6528-1-hpa@redhat.com>\n\t\n\t","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jean-Michel Hautbois via libcamera-devel\n\t","Reply-To":"Jean-Michel Hautbois ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22429,"web_url":"https://patchwork.libcamera.org/comment/22429/","msgid":"","date":"2022-03-24T10:36:49","subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel and Hans,\n\nOn Wed, Mar 23, 2022 at 9:52 PM Jean-Michel Hautbois\n wrote:\n>\n> Hello Hans,\n>\n> On 23/03/2022 14:30, Hans de Goede wrote:\n> > Hi,\n> >\n> > On 3/22/22 15:01, Jean-Michel Hautbois via libcamera-devel wrote:\n> >> Hi Kate !\n> >>\n> >> On 22/03/2022 03:41, Kate Hsuan via libcamera-devel wrote:\n> >>> The floating results of AWB and AE significantly impact the AF\n> >>> performance and the AF scene variance estimation. If the exposure value\n> >>> and color settings are changed by the algorithm, the AF scene variance will\n> >>> be affected and the algorithm may find an incorrect local maximum value.\n> >>> Consequently, a wrong lens position is determined and we could only get\n> >>> a blurred image.\n> >>\n> >> I agree with you, at least on one assumption: AWB can have an impact on AF. AE is more debatable I think :-).\n> >\n> > Interesting I would actually expect the importance to be the otherway\n> > around. If AE is way off, so way overexposed or much to dark, there\n> > is not much on which the focus algorithm can actually focus. Where as\n> > if an image is e.g. to \"green\" I would not expect that to make a lot\n> > of difference for the contrast of the image.\n>\n> Well, indeed, if your scene is too dark or too bright, you will have big\n> issues. But in case you are flickering around a somehow not that bad\n> value, then shouldn't really change the edges. On the contrary if you\n> change only the red channel for instance, you may experience weird\n> things... I am not sure that I am clear here :-). Maybe should we\n> conduct experiments extracting the filters output in various conditions\n> for a given scene ?\n\nI observed that the AF variance changed dramatically when AE or AWB\ncondition changed. Even under the AF stable state, if the light was\nturned to dark or lighter, the variance was changed and triggered the\nAF scanning. I can export some of the filtered output for some\nconditions. Could we list the conditions which you are interested in?\nThank you :)\n\n>\n> >\n> >> I think we should consider something a bit different from what you did, if you don't mind, as I don't think AWB should know anything out of its own scope, same for AGC, same for AF, same for... you get it :-).\n> >> This should be taken into account at the IPA level, the controller, which could set a state for an algorithm.\n> >\n> > I agree, but I think we want this on other IPU-s too, so maybe we need to do this\n> > at an even higher level, assuming that is possible (I still need\n> > to familiarize myself with libcamera more) ?\n> >\n> >> I think that, while autofocus is starting, or in coarse search mode, we should not change the awb gains. When is gets into the fine search mode, then AWB could be started as it should not interfere anymore. So, I think AF could have a 'status' field in its frameContext, and the algorithms (AWB, AGC, probably AF) should be \"pausable\". David sent a patch lately for RPi [1] and this is something we should consider.\n> >>\n> >> That beeing said, and I hope I am clear in my explanations, before calling the process() call for the algorithms, the IPU3IPU could set the status of AWB and AGC based on the AF status. I would start AF after AGC is stable enough (could be based on a number of frames for a first try) and I would enable AWB while AGC is starting.\n> >>\n> >> Once AGC is stable, AF is entering the coarse search, and in this phase AWB is paused. Once it enters the fine search, we can resume AWB.\n> >\n> > Agreed I think we also need some sort of AGC \"done\" bit because as\n> > said I don't think doing auto-focus before we at least have the\n> > exposure setup somewhat correct makes a lot of sense.\n> >\n> >> I think it should be enough, as we are not zooming, so illumination is not changing a lot with focus change AFAIK.\n> >\n> > Regards,\n> >\n> > Hans\n> >\n> >\n> >\n> >>\n> >> I don't really have time to test it so if you want/can I would greatly appreciate :-).\n> >>\n> >> Thanks !\n> >> JM\n> >>\n> >> [1]: https://patchwork.libcamera.org/patch/15343/\n> >>\n> >>>\n> >>> This patch proposed an AWB and AE suspension mechanism while AF is\n> >>> searching for the focus. A lock flag is used to identify that AWB and\n> >>> AE should be suspended or not. AGC and AE algorithm test the flag and\n> >>> determine that they have to perform the algorithm or suspend itself.\n> >>>\n> >>> Kate Hsuan (4):\n> >>> ipa: ipu3: af: Introduce AWB and AE lock when AF scanning\n> >>> ipa: ipu3: af: AE and AWE lock request flow when performing AF\n> >>> ipa: ipu3: awb: AWB lock for AF scan\n> >>> ipa: ipu3: agc: AE lock for AF scanning\n> >>>\n> >>> src/ipa/ipu3/algorithms/af.cpp | 25 +++++++++++++++++++++++++\n> >>> src/ipa/ipu3/algorithms/af.h | 2 ++\n> >>> src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++++++++++\n> >>> src/ipa/ipu3/algorithms/agc.h | 1 +\n> >>> src/ipa/ipu3/algorithms/awb.cpp | 13 +++++++++++++\n> >>> src/ipa/ipu3/algorithms/awb.h | 1 +\n> >>> src/ipa/ipu3/ipa_context.h | 1 +\n> >>> 7 files changed, 57 insertions(+)\n> >>>\n> >>\n> >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3F05CC0F1B\n\tfor ;\n\tThu, 24 Mar 2022 10:37:10 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 792F5604D5;\n\tThu, 24 Mar 2022 11:37:09 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E7280601F5\n\tfor ;\n\tThu, 24 Mar 2022 11:37:07 +0100 (CET)","from mail-lj1-f197.google.com (mail-lj1-f197.google.com\n\t[209.85.208.197]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-97-YEQBvC9QNemM5dmGH5y9-g-1; Thu, 24 Mar 2022 06:37:02 -0400","by mail-lj1-f197.google.com with SMTP id\n\tv8-20020a2e7a08000000b002498273eb20so1663616ljc.7\n\tfor ;\n\tThu, 24 Mar 2022 03:37:02 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1648118229;\n\tbh=4MgQDaZcdk+087CmPh4RJ/+vbWFTmmLixBoYEoM0voE=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=vyoCzN6zdcYx5YRRsoT5Dgn71YK02UkYgAh7M86ztPVCa40tR+q5ANWeyMwpO6RPP\n\tNlqUI/lR+xYJKVyODa4+EII6/hoRtytdgZWmIUpOTzEDIUtlzCQENooqqwcyNfRCh+\n\trJ0mSaj/5P2eMPYNvlx5ROzx4JfIn583a6zSEbbD8R1ZHfGrG1pTVPeyj2dwiq6GDD\n\tYSgmPY0PS8hhRusXqfZXaYlif1sreK7doX7MkcyLMVhxoxRNWqV+9vvZ0E5hTFXU+u\n\tx4jewxxlt2jFEPEH68gttS1cBdunC5xgV5XOZndxR/IwEfjd4rsoKhFE0KxeBfDGCY\n\tPqEchg8+iLMuA==","v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1648118226;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=MRBoMCk7zE6ikV3eQdUUwtAGDUxpWjnRSqC7HdVttSA=;\n\tb=dq08DcQ/y5f5z+8dwmr4Am8nXVobvqwWjN0ng0W9yP2Mx/ZCvD8c+M96AIe+RCITGZjlSs\n\tZHmlJIBUyw2izWVsNWRdsA74wSBDrW98vm9B3f2O4UKBYYj46G09Jn1y8MXy2/OgfaAzuk\n\teu++gmUSEv3UdLmYwkbP2OZtp+h4ego="],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=redhat.com\n\theader.i=@redhat.com header.b=\"dq08DcQ/\"; \n\tdkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"X-MC-Unique":"YEQBvC9QNemM5dmGH5y9-g-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=MRBoMCk7zE6ikV3eQdUUwtAGDUxpWjnRSqC7HdVttSA=;\n\tb=jhO4owQ/0pHuIyIb14GGlpyCdboCsO4hppqGsmQ7u/L+NnMO6L14cYFm4OCRFb/pba\n\tm2bmiVc8GWM5s7zHkJXAhyDiXpQMYZyL97IOADp78xWZKhpW/x3kv+GQ50jIhBhnRoEm\n\t3zbvix2VQDHVMAm2kx9OOnMnm3bOjE5IsdiELUI2/7CGkgH/4+I7LHAfT+qr2f77XKG+\n\t11JaGU1JKVx/IzMvlIAjcz/jNp6y5+8UsSDouHtjGa7RIgENL+IbEkvVCD2BTVOJMQnP\n\t2i7yeB12nJQF3a4JOSB+x9FK16UAI9SV5vBT43oYWVeNOj44q8cfzSxmRUTA3OsGULjF\n\t4wJg==","X-Gm-Message-State":"AOAM5316J/GVxjgZQllvagcBP7zKq1sQHSLWzln5OpyePTw3ug0eyRoa\n\tA5qU9mlhRnUrgnQ6Tr3FKNTQppmmGmMieNhdZkHHOMb7RwgPVwbiTbWcpXOLgpo0umXelpeWG78\n\tqLBiJMC4Ip4ilp+Q2D80/wauWcl0rFQ+bQsHMw8zPOrS8fq3jsA==","X-Received":["by 2002:a2e:a796:0:b0:249:86ac:f836 with SMTP id\n\tc22-20020a2ea796000000b0024986acf836mr3537815ljf.104.1648118220818; \n\tThu, 24 Mar 2022 03:37:00 -0700 (PDT)","by 2002:a2e:a796:0:b0:249:86ac:f836 with SMTP id\n\tc22-20020a2ea796000000b0024986acf836mr3537795ljf.104.1648118220458;\n\tThu, 24 Mar 2022 03:37:00 -0700 (PDT)"],"X-Google-Smtp-Source":"ABdhPJzAcMeP4BQbnQ9es2Nl4fQOfX2W4AcNQJbw5Wn9la0FdcqyGzCwUJMRPrcfx9KEAx+buDJmsgEScJIjOkLXhj0=","MIME-Version":"1.0","References":"<20220322024155.6528-1-hpa@redhat.com>\n\t\n\t\n\t<6eece3df-f384-4374-9dc0-fdddbe767474@ideasonboard.com>","In-Reply-To":"<6eece3df-f384-4374-9dc0-fdddbe767474@ideasonboard.com>","Date":"Thu, 24 Mar 2022 18:36:49 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","Subject":"Re: [libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily\n\tsuspend AWB and AE during AF scanning","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Kate Hsuan via libcamera-devel ","Reply-To":"Kate Hsuan ","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]