[{"id":20348,"web_url":"https://patchwork.libcamera.org/comment/20348/","msgid":"","date":"2021-10-21T02:35:39","subject":"Re: [libcamera-devel] [PATCH v2 11/13] ipa: ipu3: agc: Remove\n\tcondition on exposure correction","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Jean-Michel,\n\nThank you for the patch.\n\nOn Wed, Oct 20, 2021 at 05:46:05PM +0200, Jean-Michel Hautbois wrote:\n> Until now, we can't know when the exposure and gains applied in the\n> IPAIPU3::setControls() are really applied (it can be several frames). We\n> don't want to use the values calculated as if they are already applied,\n> and this is done by testing the frame number.\n> \n> When the exposure is estimated, we verify if it changed enough for\n> exposure and gain to be updated.\n> \n> There is no need for that because we are now filtering the value with\n> the previous one correctly, so if the change is very small the exposure\n> and analogue gain my evolve a bit but it should not be visible to the\n> user.\n\nRepeating my comment from v1,\n\nI fail to see why a small change won't change the exposure and analogue\ngain. As far as I understand, this removes a hysteresis, which I believe\nwill cause oscillations.\n\nI'd rather work on dropping the kFrameSkipCount and using the correct\nexposure time and gain values.\n\n> Signed-off-by: Jean-Michel Hautbois \n> Reviewed-by: Kieran Bingham \n> ---\n> src/ipa/ipu3/algorithms/agc.cpp | 78 ++++++++++++++++-----------------\n> 1 file changed, 37 insertions(+), 41 deletions(-)\n> \n> diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp\n> index 19f3a420..0417dc99 100644\n> --- a/src/ipa/ipu3/algorithms/agc.cpp\n> +++ b/src/ipa/ipu3/algorithms/agc.cpp\n> @@ -138,53 +138,49 @@ void Agc::lockExposureGain(uint32_t &exposure, double &analogueGain)\n> \tif ((frameCount_ < kInitialFrameMinAECount) || (frameCount_ - lastFrame_ < kFrameSkipCount))\n> \t\treturn;\n> \n> -\t/* Are we correctly exposed ? */\n> -\tif (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) {\n> -\t\tLOG(IPU3Agc, Debug) << \"!!! Good exposure with iqMean = \" << iqMean_;\n> -\t} else {\n> -\t\tdouble evGain = kEvGainTarget * knumHistogramBins / iqMean_;\n> +\tdouble evGain = kEvGainTarget * knumHistogramBins / iqMean_;\n> \n> -\t\t/* extracted from Rpi::Agc::computeTargetExposure */\n> -\t\tutils::Duration currentShutter = exposure * lineDuration_;\n> -\t\tcurrentExposureNoDg_ = currentShutter * analogueGain;\n> -\t\tLOG(IPU3Agc, Debug) << \"Actual total exposure \" << currentExposureNoDg_\n> -\t\t\t\t << \" Shutter speed \" << currentShutter\n> -\t\t\t\t << \" Gain \" << analogueGain\n> -\t\t\t\t << \" Needed ev gain \" << evGain;\n> +\t/* extracted from Rpi::Agc::computeTargetExposure */\n> +\tutils::Duration currentShutter = exposure * lineDuration_;\n> +\tcurrentExposureNoDg_ = currentShutter * analogueGain;\n> +\tLOG(IPU3Agc, Debug) << \"Actual total exposure \" << currentExposureNoDg_\n> +\t\t\t << \" Shutter speed \" << currentShutter\n> +\t\t\t << \" Gain \" << analogueGain\n> +\t\t\t << \" Needed ev gain \" << evGain;\n> \n> -\t\tcurrentExposure_ = prevExposureValue_ * evGain;\n> -\t\tutils::Duration minShutterSpeed = minExposureLines_ * lineDuration_;\n> -\t\tutils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_;\n> +\tcurrentExposure_ = prevExposureValue_ * evGain;\n> +\tutils::Duration minShutterSpeed = minExposureLines_ * lineDuration_;\n> +\tutils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_;\n> \n> -\t\tutils::Duration maxTotalExposure = maxShutterSpeed * kMaxGain;\n> -\t\tcurrentExposure_ = std::min(currentExposure_, maxTotalExposure);\n> -\t\tLOG(IPU3Agc, Debug) << \"Target total exposure \" << currentExposure_\n> -\t\t\t\t << \", maximum is \" << maxTotalExposure;\n> +\tutils::Duration maxTotalExposure = maxShutterSpeed * kMaxGain;\n> +\tcurrentExposure_ = std::min(currentExposure_, maxTotalExposure);\n> +\tLOG(IPU3Agc, Debug) << \"Target total exposure \" << currentExposure_\n> +\t\t\t << \", maximum is \" << maxTotalExposure;\n> \n> -\t\t/* \\todo: estimate if we need to desaturate */\n> -\t\tfilterExposure();\n> +\t/* \\todo: estimate if we need to desaturate */\n> +\tfilterExposure();\n> \n> -\t\tutils::Duration exposureValue = filteredExposure_;\n> -\t\tutils::Duration shutterTime = minShutterSpeed;\n> +\tutils::Duration exposureValue = filteredExposure_;\n> +\tutils::Duration shutterTime = minShutterSpeed;\n> +\n> +\t/*\n> +\t* Push the shutter time up to the maximum first, and only then\n> +\t* increase the gain.\n> +\t*/\n> +\tshutterTime = std::clamp(exposureValue / kMinGain,\n> +\t\t\t\t\t\t minShutterSpeed, maxShutterSpeed);\n> +\tdouble stepGain = std::clamp(exposureValue / shutterTime,\n> +\t\t\t\t kMinGain, kMaxGain);\n> +\tLOG(IPU3Agc, Debug) << \"Divided up shutter and gain are \"\n> +\t\t\t << shutterTime << \" and \"\n> +\t\t\t << stepGain;\n> +\n> +\texposure = shutterTime / lineDuration_;\n> +\tanalogueGain = stepGain;\n> +\n> +\t/* Update the exposure value for the next process call */\n> +\tprevExposureValue_ = shutterTime * analogueGain;\n> \n> -\t\t/*\n> -\t\t * Push the shutter time up to the maximum first, and only then\n> -\t\t * increase the gain.\n> -\t\t */\n> -\t\tshutterTime = std::clamp(exposureValue / kMinGain,\n> -\t\t\t\t\t\t\t minShutterSpeed, maxShutterSpeed);\n> -\t\tdouble stepGain = std::clamp(exposureValue / shutterTime,\n> -\t\t\t\t\t kMinGain, kMaxGain);\n> -\t\tLOG(IPU3Agc, Debug) << \"Divided up shutter and gain are \"\n> -\t\t\t\t << shutterTime << \" and \"\n> -\t\t\t\t << stepGain;\n> -\n> -\t\texposure = shutterTime / lineDuration_;\n> -\t\tanalogueGain = stepGain;\n> -\n> -\t\t/* Update the exposure value for the next process call */\n> -\t\tprevExposureValue_ = shutterTime * analogueGain;\n> -\t}\n> \tlastFrame_ = frameCount_;\n> }\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1EEF0BDB1C\n\tfor ;\n\tThu, 21 Oct 2021 02:36:02 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 78E5168F58;\n\tThu, 21 Oct 2021 04:36:01 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6C0F260124\n\tfor ;\n\tThu, 21 Oct 2021 04:35:59 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id E05DD2BA;\n\tThu, 21 Oct 2021 04:35:58 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"WNAV2MV7\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1634783759;\n\tbh=mWN+N8P9jophIZKK3fIwtoCbzoDFeh6/yZZpU5FfqnU=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=WNAV2MV7NiRzCQzP91fOgfgaxYAz5+7ODl5CgVUjbuoYscpoeyDufe+nJcCABGOZg\n\tdaqAp6fDuDeNywNmUJ+Wr9bsZ/7agQhpEeJWKAgzWrsSvnU+FiLoKjW9iHrv8MBAO5\n\t0OtRnu3svAlN3rztc6jOtUu9+1wSYO1lnoQF73hw=","Date":"Thu, 21 Oct 2021 05:35:39 +0300","From":"Laurent Pinchart ","To":"Jean-Michel Hautbois ","Message-ID":"","References":"<20211020154607.180161-1-jeanmichel.hautbois@ideasonboard.com>\n\t<20211020154607.180161-12-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20211020154607.180161-12-jeanmichel.hautbois@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v2 11/13] ipa: ipu3: agc: Remove\n\tcondition on exposure correction","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":20355,"web_url":"https://patchwork.libcamera.org/comment/20355/","msgid":"","date":"2021-10-21T07:01:02","subject":"Re: [libcamera-devel] [PATCH v2 11/13] ipa: ipu3: agc: Remove\n\tcondition on exposure correction","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Laurent,\n\nOn 21/10/2021 04:35, Laurent Pinchart wrote:\n> Hi Jean-Michel,\n> \n> Thank you for the patch.\n> \n> On Wed, Oct 20, 2021 at 05:46:05PM +0200, Jean-Michel Hautbois wrote:\n>> Until now, we can't know when the exposure and gains applied in the\n>> IPAIPU3::setControls() are really applied (it can be several frames). We\n>> don't want to use the values calculated as if they are already applied,\n>> and this is done by testing the frame number.\n>>\n>> When the exposure is estimated, we verify if it changed enough for\n>> exposure and gain to be updated.\n>>\n>> There is no need for that because we are now filtering the value with\n>> the previous one correctly, so if the change is very small the exposure\n>> and analogue gain my evolve a bit but it should not be visible to the\n>> user.\n> \n> Repeating my comment from v1,\n> \n> I fail to see why a small change won't change the exposure and analogue\n> gain. As far as I understand, this removes a hysteresis, which I believe\n> will cause oscillations.\n\nYes, that's why I changed the commit message a bit. You may have \noscillations, but those would be very small, and given the sensitivity \nof the sensors, you may not even notice it (we have a low-pass filter).\n\n> \n> I'd rather work on dropping the kFrameSkipCount and using the correct\n> exposure time and gain values.\n\nYes, we need to do that, certainly based on DelayedControls, as we don't \nhave embedded data in all sensors. I am not sure on how to do it \nefficiently, so any input will be appreciated ;-).\n\n> \n>> Signed-off-by: Jean-Michel Hautbois \n>> Reviewed-by: Kieran Bingham \n>> ---\n>> src/ipa/ipu3/algorithms/agc.cpp | 78 ++++++++++++++++-----------------\n>> 1 file changed, 37 insertions(+), 41 deletions(-)\n>>\n>> diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp\n>> index 19f3a420..0417dc99 100644\n>> --- a/src/ipa/ipu3/algorithms/agc.cpp\n>> +++ b/src/ipa/ipu3/algorithms/agc.cpp\n>> @@ -138,53 +138,49 @@ void Agc::lockExposureGain(uint32_t &exposure, double &analogueGain)\n>> \tif ((frameCount_ < kInitialFrameMinAECount) || (frameCount_ - lastFrame_ < kFrameSkipCount))\n>> \t\treturn;\n>> \n>> -\t/* Are we correctly exposed ? */\n>> -\tif (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) {\n>> -\t\tLOG(IPU3Agc, Debug) << \"!!! Good exposure with iqMean = \" << iqMean_;\n>> -\t} else {\n>> -\t\tdouble evGain = kEvGainTarget * knumHistogramBins / iqMean_;\n>> +\tdouble evGain = kEvGainTarget * knumHistogramBins / iqMean_;\n>> \n>> -\t\t/* extracted from Rpi::Agc::computeTargetExposure */\n>> -\t\tutils::Duration currentShutter = exposure * lineDuration_;\n>> -\t\tcurrentExposureNoDg_ = currentShutter * analogueGain;\n>> -\t\tLOG(IPU3Agc, Debug) << \"Actual total exposure \" << currentExposureNoDg_\n>> -\t\t\t\t << \" Shutter speed \" << currentShutter\n>> -\t\t\t\t << \" Gain \" << analogueGain\n>> -\t\t\t\t << \" Needed ev gain \" << evGain;\n>> +\t/* extracted from Rpi::Agc::computeTargetExposure */\n>> +\tutils::Duration currentShutter = exposure * lineDuration_;\n>> +\tcurrentExposureNoDg_ = currentShutter * analogueGain;\n>> +\tLOG(IPU3Agc, Debug) << \"Actual total exposure \" << currentExposureNoDg_\n>> +\t\t\t << \" Shutter speed \" << currentShutter\n>> +\t\t\t << \" Gain \" << analogueGain\n>> +\t\t\t << \" Needed ev gain \" << evGain;\n>> \n>> -\t\tcurrentExposure_ = prevExposureValue_ * evGain;\n>> -\t\tutils::Duration minShutterSpeed = minExposureLines_ * lineDuration_;\n>> -\t\tutils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_;\n>> +\tcurrentExposure_ = prevExposureValue_ * evGain;\n>> +\tutils::Duration minShutterSpeed = minExposureLines_ * lineDuration_;\n>> +\tutils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_;\n>> \n>> -\t\tutils::Duration maxTotalExposure = maxShutterSpeed * kMaxGain;\n>> -\t\tcurrentExposure_ = std::min(currentExposure_, maxTotalExposure);\n>> -\t\tLOG(IPU3Agc, Debug) << \"Target total exposure \" << currentExposure_\n>> -\t\t\t\t << \", maximum is \" << maxTotalExposure;\n>> +\tutils::Duration maxTotalExposure = maxShutterSpeed * kMaxGain;\n>> +\tcurrentExposure_ = std::min(currentExposure_, maxTotalExposure);\n>> +\tLOG(IPU3Agc, Debug) << \"Target total exposure \" << currentExposure_\n>> +\t\t\t << \", maximum is \" << maxTotalExposure;\n>> \n>> -\t\t/* \\todo: estimate if we need to desaturate */\n>> -\t\tfilterExposure();\n>> +\t/* \\todo: estimate if we need to desaturate */\n>> +\tfilterExposure();\n>> \n>> -\t\tutils::Duration exposureValue = filteredExposure_;\n>> -\t\tutils::Duration shutterTime = minShutterSpeed;\n>> +\tutils::Duration exposureValue = filteredExposure_;\n>> +\tutils::Duration shutterTime = minShutterSpeed;\n>> +\n>> +\t/*\n>> +\t* Push the shutter time up to the maximum first, and only then\n>> +\t* increase the gain.\n>> +\t*/\n>> +\tshutterTime = std::clamp(exposureValue / kMinGain,\n>> +\t\t\t\t\t\t minShutterSpeed, maxShutterSpeed);\n>> +\tdouble stepGain = std::clamp(exposureValue / shutterTime,\n>> +\t\t\t\t kMinGain, kMaxGain);\n>> +\tLOG(IPU3Agc, Debug) << \"Divided up shutter and gain are \"\n>> +\t\t\t << shutterTime << \" and \"\n>> +\t\t\t << stepGain;\n>> +\n>> +\texposure = shutterTime / lineDuration_;\n>> +\tanalogueGain = stepGain;\n>> +\n>> +\t/* Update the exposure value for the next process call */\n>> +\tprevExposureValue_ = shutterTime * analogueGain;\n>> \n>> -\t\t/*\n>> -\t\t * Push the shutter time up to the maximum first, and only then\n>> -\t\t * increase the gain.\n>> -\t\t */\n>> -\t\tshutterTime = std::clamp(exposureValue / kMinGain,\n>> -\t\t\t\t\t\t\t minShutterSpeed, maxShutterSpeed);\n>> -\t\tdouble stepGain = std::clamp(exposureValue / shutterTime,\n>> -\t\t\t\t\t kMinGain, kMaxGain);\n>> -\t\tLOG(IPU3Agc, Debug) << \"Divided up shutter and gain are \"\n>> -\t\t\t\t << shutterTime << \" and \"\n>> -\t\t\t\t << stepGain;\n>> -\n>> -\t\texposure = shutterTime / lineDuration_;\n>> -\t\tanalogueGain = stepGain;\n>> -\n>> -\t\t/* Update the exposure value for the next process call */\n>> -\t\tprevExposureValue_ = shutterTime * analogueGain;\n>> -\t}\n>> \tlastFrame_ = frameCount_;\n>> }\n>> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 020DEBDB1C\n\tfor ;\n\tThu, 21 Oct 2021 07:01:07 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 51960604FE;\n\tThu, 21 Oct 2021 09:01:06 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D354B60129\n\tfor ;\n\tThu, 21 Oct 2021 09:01:04 +0200 (CEST)","from [IPV6:2a01:e0a:169:7140:f9d:5926:ad90:4996] (unknown\n\t[IPv6:2a01:e0a:169:7140:f9d:5926:ad90:4996])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 835FF276;\n\tThu, 21 Oct 2021 09:01:04 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"fr7drQ9d\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1634799664;\n\tbh=8Z0AngxTTM3pU48uYgBhtROMZ1jHNf3TWmHYN31sajc=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=fr7drQ9dpd85aVCaiDyL8Kf94U82oZkRFJV1yNEaK4IyKHMNP6So11bQjUlXUqr0U\n\tc/bG1qXgmWScEXJWJzN0blZl9BpECO9qEg2hNai7vdVKdG3oN7Ic6jarNVeKBOsdTq\n\tZV9NPJggpHXeZvHius7e4aaja7PcnRpq6HMtJOoA=","Message-ID":"","Date":"Thu, 21 Oct 2021 09:01:02 +0200","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.1.2","Content-Language":"en-US","To":"Laurent Pinchart ","References":"<20211020154607.180161-1-jeanmichel.hautbois@ideasonboard.com>\n\t<20211020154607.180161-12-jeanmichel.hautbois@ideasonboard.com>\n\t","From":"Jean-Michel Hautbois ","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Subject":"Re: [libcamera-devel] [PATCH v2 11/13] ipa: ipu3: agc: Remove\n\tcondition on exposure correction","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]