From patchwork Fri May 6 05:18:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kate Hsuan X-Patchwork-Id: 15804 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 61B78C326C for ; Fri, 6 May 2022 05:19:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2EE6C6564B; Fri, 6 May 2022 07:19:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1651814364; bh=PrnElUlrXByTDH+8FOVKAfnnBLgnt5I5IVpGBLLSkpA=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=lau1G3wTgt9GTw5V19v0Afmx9Z1/gpMhNyfJMSKiH95bnUt7gA6uqfEHzf6O0jktW yQws+cNoLtZ9xiDEJahdBS7HeolzNUioSg5Pd6GxodS4ns3QFuRubHQcQ+wFNiDldw Q/1EFTCeQkFHCzDtFUk9g/WKMItkeRyjo34zXC4wI8lrSwXE+ai88gemjX3pN/6egF WcuprRqGva93WrJ/B7DVdbxk85rf8oLCJyl6/nBU6DlBKxcT4evfON1XnCir5V/4nV IIWTWoJShBrtt7Vdbvad60hJhwbHQ6D6i8X7Jgsg3RwvDZQtp/UTvBBBE9Pyqkl/LI l1VRAyTpxnm6g== Received: from us-smtp-delivery-74.mimecast.com (us-smtp-delivery-74.mimecast.com [170.10.133.74]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2CADD6563F for ; Fri, 6 May 2022 07:19:20 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="e4zLU3RF"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651814359; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bvBQL51J0KorVN3sj4FomrergGz9rjyduVp/Kvi3Tcw=; b=e4zLU3RFxsN8qPDEDXOacdmcBpdJ01EgejEvtyndUkwDG/4R/ex+0++xxmteU4RIeAMUae zUeQqeK2MWZUljaArU2i5O0y8cxmwR1fO5XyT5d6mWPK55A1wgaBIlwZgf9NCpBakapyOo urX/xJqnJ0tyMjxjOyi2vMTz27nsz7I= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-206-OlBpATyHN2uXM9U1M9iWdA-1; Fri, 06 May 2022 01:19:16 -0400 X-MC-Unique: OlBpATyHN2uXM9U1M9iWdA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 851C6800B21; Fri, 6 May 2022 05:19:16 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 117782166B17; Fri, 6 May 2022 05:19:13 +0000 (UTC) To: libcamera devel , Jean-Michel Hautbois , Hans De Goede Date: Fri, 6 May 2022 13:18:39 +0800 Message-Id: <20220506051841.70792-2-hpa@redhat.com> In-Reply-To: <20220506051841.70792-1-hpa@redhat.com> References: <20220506051841.70792-1-hpa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [PATCH v2 1/3] ipa: ipu3: agc: AGC exposure stability test X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kate Hsuan via libcamera-devel From: Kate Hsuan Reply-To: Kate Hsuan Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The average exposure value is determined to be a baseline for the stability testing and is formed by the moving average. If the latest estimated exposure value is in the 90% confidence interval of average exposure, the stable status will be set to true. Signed-off-by: Kate Hsuan --- src/ipa/ipu3/algorithms/agc.cpp | 22 +++++++++++++++++++++- src/ipa/ipu3/algorithms/agc.h | 3 +++ src/ipa/ipu3/ipa_context.h | 1 + 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 7d4b3503..c115f3bc 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -72,7 +72,7 @@ static constexpr double kRelativeLuminanceTarget = 0.16; Agc::Agc() : frameCount_(0), minShutterSpeed_(0s), - maxShutterSpeed_(0s), filteredExposure_(0s) + maxShutterSpeed_(0s), filteredExposure_(0s), avgExposure_(0) { } @@ -102,6 +102,8 @@ int Agc::configure(IPAContext &context, frameContext.agc.gain = std::max(minAnalogueGain_, kMinAnalogueGain); frameContext.agc.exposure = 10ms / configuration.sensor.lineDuration; + frameContext.agc.stable = false; + frameCount_ = 0; return 0; } @@ -314,6 +316,23 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, return ySum / (grid.height * grid.width) / 255; } +void Agc::evaluateStable(IPAContext &context) +{ + uint32_t latestExposure = context.frameContext.agc.exposure; + uint32_t upperEx = avgExposure_ + (avgExposure_ * 0.1); + uint32_t lowerEx = avgExposure_ - (avgExposure_ * 0.1); + + if (context.frameContext.af.stable) { + avgExposure_ = (0.2 * avgExposure_) + ((1 - 0.2) * latestExposure); + if (latestExposure > lowerEx && latestExposure < upperEx) + context.frameContext.agc.stable = true; + else { + if (context.frameContext.af.stable) + context.frameContext.agc.stable = false; + } + } +} + /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -359,6 +378,7 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) computeExposure(context, yGain, iqMeanGain); frameCount_++; + evaluateStable(context); } } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index ad705605..56ef2c10 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -40,6 +40,7 @@ private: const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); + void evaluateStable(IPAContext &context); uint64_t frameCount_; @@ -51,6 +52,8 @@ private: utils::Duration filteredExposure_; + uint32_t avgExposure_; + uint32_t stride_; }; diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 103498ef..ddef4e74 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -52,6 +52,7 @@ struct IPAFrameContext { struct { uint32_t exposure; double gain; + bool stable; } agc; struct { From patchwork Fri May 6 05:18:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kate Hsuan X-Patchwork-Id: 15805 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 95C6EC3256 for ; Fri, 6 May 2022 05:19:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E551565650; Fri, 6 May 2022 07:19:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1651814366; bh=ZGem10q5WiPOC/d2uCOAfYfrTHlzxaLoBKgTMKV7uL0=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=GEJnfKaIo1kCG6T3JyzNjV3nZfID2izfk9ciYshWI3l4ALZnEfIccm23bwJp9Ohic 4J/sZW49VfftCjL/dHpf1X8ZxAi4XCzIcA5RtxF+wLGVu0gWLjq7AOAxnXxHfnFgUL mKyn0n3aKYcbW0gbooQakPKPQLVHepV3OLFqbgJIeZ2SUzllhrp6tF7+32BNhgCv+j xhRcmkqCS8PHOmWi8lb7Jf8M6GvCYNr3CRptCUb7WZpd/tSojAN1oVln9eHaVl6z3J 6RMuk/uEk+tymtC/q7RKJCw3FxyKGOzM3ChIOS8zIGhgPojDLbJaiq9ygaTY8o+Ov3 VQgBUSKv0ZufQ== Received: from us-smtp-delivery-74.mimecast.com (us-smtp-delivery-74.mimecast.com [170.10.133.74]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 829A865649 for ; Fri, 6 May 2022 07:19:24 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="DHPWrOAK"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651814363; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cQ9ZrM0EBcsJIxhSzclD3wJkVaHwjYhhItKbzmRLEzo=; b=DHPWrOAKCqNe8riYrpxptNzOhJjj6IAPP/g+0arV1/AN92NApQSAf9cpttjH6KNfKIZ6jS FMRHgNAOc8Sk/7SfB5epU3F2JN8q+tl0oidjpPGu41JCaRH/CMy971YZ1wZpGetxfXoi8j +o+bWHSm57/cNcsXDurWChAwGjcYLjw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-394-1tJWtS3lOHK6ZPPmyIZnfQ-1; Fri, 06 May 2022 01:19:22 -0400 X-MC-Unique: 1tJWtS3lOHK6ZPPmyIZnfQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5D11F8001EA; Fri, 6 May 2022 05:19:22 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD8402166B1A; Fri, 6 May 2022 05:19:19 +0000 (UTC) To: libcamera devel , Jean-Michel Hautbois , Hans De Goede Date: Fri, 6 May 2022 13:18:40 +0800 Message-Id: <20220506051841.70792-3-hpa@redhat.com> In-Reply-To: <20220506051841.70792-1-hpa@redhat.com> References: <20220506051841.70792-1-hpa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [PATCH v2 2/3] ipa: ipu3: af: Trigger AF after AGC is stable X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kate Hsuan via libcamera-devel From: Kate Hsuan Reply-To: Kate Hsuan Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" AGC stable status is used to determine that AF is triggered or paused. If AGC is ready, AF can start to evaluate the lens position. Signed-off-by: Kate Hsuan --- src/ipa/ipu3/algorithms/af.cpp | 12 ++++++++++++ src/ipa/ipu3/algorithms/af.h | 2 ++ 2 files changed, 14 insertions(+) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index f700b01f..8cabd064 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -403,6 +403,16 @@ bool Af::afIsOutOfFocus(IPAContext context) return false; } +void Af::pause(IPAContext &context) +{ + context.frameContext.af.focus = focus_; +} + +bool Af::isAgcStable(IPAContext context) +{ + return context.frameContext.agc.stable; +} + /** * \brief Determine the max contrast image and lens position. * \param[in] context The IPA context. @@ -421,6 +431,8 @@ bool Af::afIsOutOfFocus(IPAContext context) */ void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { + if (!isAgcStable(context)) + pause(context); /* Evaluate the AF buffer length */ uint32_t afRawBufferLen = context.configuration.af.afGrid.width * context.configuration.af.afGrid.height; diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h index b85cf941..c1c9d39e 100644 --- a/src/ipa/ipu3/algorithms/af.h +++ b/src/ipa/ipu3/algorithms/af.h @@ -44,6 +44,8 @@ private: double afEstimateVariance(Span y_items, bool isY1); bool afIsOutOfFocus(IPAContext context); + bool isAgcStable(IPAContext context); + void pause(IPAContext &context); /* VCM step configuration. It is the current setting of the VCM step. */ uint32_t focus_; From patchwork Fri May 6 05:18:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kate Hsuan X-Patchwork-Id: 15806 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CA78EC3256 for ; Fri, 6 May 2022 05:19:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8C8B665640; Fri, 6 May 2022 07:19:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1651814371; bh=C5/uYU927iuwA4pj2bU6uI3pimNAYm1DCPWq50b9Qd4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Z7U5FxLvWUXATq3E8slzihbL4wZpneqWBqu4kCWtwcSaT8wgHReah3a7Q5cNA3dh6 Qz7RVJ+m+rr/vRQulZWuXzt7Refb7IoA28TvcozbcvPBjx0PfBIqfTtW6yjgj/7NjV Xh6pmZFU8B9YClyAPLRxIqwnKLfoyDYFFlRrQb8o++jVF67uGys0n2WjRckLpU4HkT K+as9Pwc2414+uvq0DmsGaEajsnE9A+Sx01Z/PajAHS1p+Sgoc/0mnybIm1X+1RqXl BN1cMaGzoImJByktIojfYt5azCf7xrWVfMAmqx7Hf46oGkZepcV2WY4nSEGdFpauyV GVJOIacAFgjPA== Received: from us-smtp-delivery-74.mimecast.com (us-smtp-delivery-74.mimecast.com [170.10.129.74]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6489D6563F for ; Fri, 6 May 2022 07:19:29 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="gZk6mPb1"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651814368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=omsIT3200jmOpd8UmbMKOoQ2PBpF6lw563RcNDag/9M=; b=gZk6mPb1Dtl6N5otkgZJq6gXdbcIWd5oyDv2GYdPfkFOuvEzQvmXana3Q7wV5WGyTP9hgo q7awky3RvyX7/ak/TuJ9TGAcpWxf6j2PixOQ3RRHEbY/ykQhxL1CXN8Ap1CuGkNddo1CGC IYLhItgidMwuviUASur+eXYGe6N1gtQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-471-br-AjkGxMWGOPtjBw1FSiQ-1; Fri, 06 May 2022 01:19:27 -0400 X-MC-Unique: br-AjkGxMWGOPtjBw1FSiQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 480F03C0CD43; Fri, 6 May 2022 05:19:27 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id C92D22166B17; Fri, 6 May 2022 05:19:24 +0000 (UTC) To: libcamera devel , Jean-Michel Hautbois , Hans De Goede Date: Fri, 6 May 2022 13:18:41 +0800 Message-Id: <20220506051841.70792-4-hpa@redhat.com> In-Reply-To: <20220506051841.70792-1-hpa@redhat.com> References: <20220506051841.70792-1-hpa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [PATCH v2 3/3] ipa: ipu3: awb: Suspend AWB when AF is working X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kate Hsuan via libcamera-devel From: Kate Hsuan Reply-To: Kate Hsuan Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" An unstable AWB results in AF performance degradation. AWB is properly paused during AF scanning to improve the accuracy of focus. Signed-off-by: Kate Hsuan --- src/ipa/ipu3/algorithms/awb.cpp | 8 +++++++- src/ipa/ipu3/algorithms/awb.h | 1 + 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 87a6cc7a..3c924fc9 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -384,12 +384,18 @@ void Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats) } } +bool Awb::isAfStable(IPAContext context) +{ + return context.frameContext.af.stable; +} + /** * \copydoc libcamera::ipa::Algorithm::process */ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { - calculateWBGains(stats); + if (!isAfStable(context)) + calculateWBGains(stats); /* * Gains are only recalculated if enough zones were detected. diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index ab4b0a33..131a51e4 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -72,6 +72,7 @@ private: void awbGreyWorld(); uint32_t estimateCCT(double red, double green, double blue); static constexpr uint16_t threshold(float value); + bool isAfStable(IPAContext context); std::vector zones_; Accumulator awbStats_[kAwbStatsSizeX * kAwbStatsSizeY];