From patchwork Tue Jul 18 14:43:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 18853 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E18BCC324D for ; Tue, 18 Jul 2023 14:43:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 546E8628C2; Tue, 18 Jul 2023 16:43:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1689691388; bh=odvmMP1mfbDW4IpPqhAZBKs1y1ZjTvXNU7NEBQ+IzuE=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=1KIqlWUlbDfdLyO6lYhxIPwc4lp6azcyra92Mnnh9KOTCAfG1OkJxX2HvSBpOgBZT NpGAJHLdIiz1MZC/JpKpLZmbtEh83DFLQdRcY4V7sLI6xX8pHa4K4Qrhf837hUWNST xdamPrSUKA5/7Iy7+fmCZs2jtPLiJIJNJchGB5flMwelEXfax+PS+wndv5mRy9STG1 APL5Tay9MxkFdQ0PSCyiIZsEzAl4Ji8HmmuPVXjcKh8RTHU2QljukpXnH46oZPMdJL k6axyfHKiEctlZAfGKdZVcnFY4Oznle/L639ydI9/4ViQ9dJm8ha8hfF0BRmYaM/P7 0gaOWWDOxl2Uw== Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4922961E2B for ; Tue, 18 Jul 2023 16:43:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="D1Kn0pH6"; dkim-atps=neutral Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-3fbf1b82de7so37130105e9.1 for ; Tue, 18 Jul 2023 07:43:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1689691386; x=1692283386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+lbhbtS+PWJC1jWNapWf2Uo+tHj0pyzFh1gsmDnPZcE=; b=D1Kn0pH61oHm4F3obNbUHfot8CVuN/ktRfNLMwWiUaMmBAYiZyTOlT/fGYkGLRia1Z IpoHzl+aMNz+yf8JzeVOsTXfXqNOlZRNovTMVqI8h4nqPhJVvx40FFhCD47IkMygc59J 9OUeBXZ5r++Ck3WHkaDKN+YbbHlG4CmDf6at3AuTTzU6wPVirepJocwM2L5iOykM9Xk2 1fc4OMAtTCc3OJsHMRi3Kl4tQ3sL4Ow7gpxlqaY59Sr5a7GfaqXwABRm91lDxxaPUngh WdJI+sqMBPWaDZh5Lg/VmYwRRE1kWJ+yhtW5/iH3tFpVHasqK/d5qglrVzAvwuVwgk5T dYGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689691386; x=1692283386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+lbhbtS+PWJC1jWNapWf2Uo+tHj0pyzFh1gsmDnPZcE=; b=A5r7pV9M7sGDCE7lV15sZR3NJZ0iROB2jFkdsUkARWoQxpCaBtNK3Rqt7iatVxW5r9 g2guFGBnj8As8cQ1qSUTkiIPPC468JO0Qzi75YiFfEQWYXDb0tcMeh22En/45LEO1g83 JjXgCDwFoTiTAfXzMFEu96GFqUp9rnq1y8/YNyQB3zPH/22DXm1lYcqmDLuhepnroHo0 JwfIUyl1et4fI6TmBgHPe4KgJeb/O+WxXO5AVO7H2xEU4uinQpnEaeGAIUGU/mfZNptM 6ch+HJu87/QLecRmTWPAnt2DKnEWly78/6lwR2O+KA3E3Nrsd918KlGleDbXyN+E0SBe whAQ== X-Gm-Message-State: ABy/qLZu3E29hqbejmx75HI0hJgmmuO4y3UlVt29EKw3t0k+QgShwzYT hk284MhrJjdkego6H1WPUi6ZPOzlmS06YFUv2Eo= X-Google-Smtp-Source: APBJJlGcnbflMaL+r8GcnQyMHcSZ7kYa+HwkS/aF0fZ9P52QQfaFY9g+D9g5Kg6GhN1prH2DwegcFQ== X-Received: by 2002:a05:600c:3c9e:b0:3fa:ef97:1fa5 with SMTP id bg30-20020a05600c3c9e00b003faef971fa5mr2335549wmb.6.1689691385591; Tue, 18 Jul 2023 07:43:05 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:2bce:64d6:1a5c:49a2]) by smtp.gmail.com with ESMTPSA id 12-20020a05600c230c00b003f90b9b2c31sm10469686wmo.28.2023.07.18.07.43.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Jul 2023 07:43:05 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Tue, 18 Jul 2023 15:43:00 +0100 Message-Id: <20230718144301.3612-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230718144301.3612-1-david.plowman@raspberrypi.com> References: <20230718144301.3612-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 1/2] libcamera: controls: Add controls for AEC/AGC flicker avoidance X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: David Plowman via libcamera-devel From: David Plowman Reply-To: David Plowman Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Flicker is the term used to describe brightness banding or oscillation of images caused typically by artificial lighting driven by a 50 or 60Hz mains supply. We add three controls intended to be used by AEC/AGC algorithms: AeFlickerMode to enable flicker avoidance. AeFlickerPeriod to set the flicker period "manually". AeFlickerDetected to report any flicker that is currently detected. Signed-off-by: David Plowman Reviewed-by: Jacopo Mondi Reviewed-by: Umang Jain --- src/libcamera/control_ids.yaml | 90 +++++++++++++++++++++++++++------- 1 file changed, 73 insertions(+), 17 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 056886e6..f2e542f4 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -156,6 +156,79 @@ controls: control of which features should be automatically adjusted shouldn't better be handled through a separate AE mode control. + - AeFlickerMode: + type: int32_t + description: | + Set the flicker mode, which determines whether, and how, the AGC/AEC + algorithm attempts to hide flicker effects caused by the duty cycle of + artificial lighting. + + Although implementation dependent, many algorithms for "flicker + avoidance" work by restricting this exposure time to integer multiples + of the cycle period, wherever possible. + + Implementations may not support all of the flicker modes listed below. + + By default the system will start in FlickerAuto mode if this is + supported, otherwise the flicker mode will be set to FlickerOff. + + enum: + - name: FlickerOff + value: 0 + description: No flicker avoidance is performed. + - name: FlickerManual + value: 1 + description: Manual flicker avoidance. + Suppress flicker effects caused by lighting running with a period + specified by the AeFlickerPeriod control. + \sa AeFlickerPeriod + - name: FlickerAuto + value: 2 + description: Automatic flicker period detection and avoidance. + The system will automatically determine the most likely value of + flicker period, and avoid flicker of this frequency. Once flicker + is being corrected, it is implementation dependent whether the + system is still able to detect a change in the flicker period. + \sa AeFlickerDetected + + - AeFlickerPeriod: + type: int32_t + description: Manual flicker period in microseconds. + This value sets the current flicker period to avoid. It is used when + AeFlickerMode is set to FlickerManual. + + To cancel 50Hz mains flicker, this should be set to 10000 (corresponding + to 100Hz), or 8333 (120Hz) for 60Hz mains. + + Setting the mode to FlickerManual when no AeFlickerPeriod has ever been + set means that no flicker cancellation occurs (until the value of this + control is updated). + + Switching to modes other than FlickerManual has no effect on the + value of the AeFlickerPeriod control. + + \sa AeFlickerMode + + - AeFlickerDetected: + type: int32_t + description: Flicker period detected in microseconds. + The value reported here indicates the currently detected flicker + period, or zero if no flicker at all is detected. + + When AeFlickerMode is set to FlickerAuto, there may be a period during + which the value reported here remains zero. Once a non-zero value is + reported, then this is the flicker period that has been detected and is + now being cancelled. + + In the case of 50Hz mains flicker, the value would be 10000 + (corresponding to 100Hz), or 8333 (120Hz) for 60Hz mains flicker. + + It is implementation dependent whether the system can continue to detect + flicker of different periods when another frequency is already being + cancelled. + + \sa AeFlickerMode + - Brightness: type: float description: | @@ -850,23 +923,6 @@ controls: value: 1 description: The lens shading map mode is available. - - SceneFlicker: - type: int32_t - draft: true - description: | - Control to report the detected scene light frequency. Currently - identical to ANDROID_STATISTICS_SCENE_FLICKER. - enum: - - name: SceneFickerOff - value: 0 - description: No flickering detected. - - name: SceneFicker50Hz - value: 1 - description: 50Hz flickering detected. - - name: SceneFicker60Hz - value: 2 - description: 60Hz flickering detected. - - PipelineDepth: type: int32_t draft: true From patchwork Tue Jul 18 14:43:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 18854 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D4F6AC32AA for ; Tue, 18 Jul 2023 14:43:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C1251628C8; Tue, 18 Jul 2023 16:43:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1689691388; bh=KbG1U3RdatzlRHTz88vkr9smMIdXLOH3J7WBupWg8FA=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=TlhYjVne1QJqzx+9/nb3g/4s39XRqrAMlPUS1ZL+rOc/wbjp+AiEO7oO3rRiGNuad wCqapVs9csD4JsD9PVTBtMgzpjqS4ygXdZcw1ZDx3BDf301Ie1LtmXc1Rz4yG/Cphk M67O8UWZSVy6tDcKW/rk1mvFZhf1NwzzpeymMwfWgcQZMgVZ0cVvZ+85yvCull7QNl 4uIaOuO2lUgtRVSATt/pmHV826PxnAFjDd0t9CfKUqEp59sHb8DvRrzeZ2jNJkKFeU Cbiyu60vfgkbKDO2mPuyasKEEy2sCLYaRrtXGFzYmvZ+YYklBFb2GHFJJVP+6QtgA2 f+Jd4h1ob7tfA== Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 04AB9628C2 for ; Tue, 18 Jul 2023 16:43:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="hxnvc8vk"; dkim-atps=neutral Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-3fbf7fbe722so60072215e9.3 for ; Tue, 18 Jul 2023 07:43:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1689691386; x=1692283386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AZEm6gA229Srb7s98kzWNEYtc3dFuABlq3BByQqRhLs=; b=hxnvc8vkwnpuLqDOkx5XjMAV8IZGXdx2g0QC2iw7xCEEaYqprrqddauN0sjTqq8wfF ihleFuMt55oLCpSzW7UT14gxEYkv5jrCZOfFtwtKWhNolQGXV2P2piGI0KwAEwCwFdzi 9y2khfBkVHydX21wjkeTPl67rh3Z7vZcfHpjgckJBKVdnmLFIH/AYkUPIvStuXpXTf1Q rf4awhTvwOuQoHx0aYQn828pQq3theipaeVY1aHJOMmbsSYnUIcSp9ZrRKrJNmIAgVB+ byQFNhtQShwT+hQTCJIKcKvG7jKsZ93XHo0FBZBFoa0P52kUa4VnKWxdA/8CAROmnjCJ ivoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689691386; x=1692283386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AZEm6gA229Srb7s98kzWNEYtc3dFuABlq3BByQqRhLs=; b=VM49quz7wf9NGTv8jDlrNxewaQPxgl98r8BBv3zJDlYD1paHa8Pt6t+lcAvwpy7k9B WhZ6fwgTBj4g19LbMLE6CnTFBgfmdpDdHfMKYfCSbBHMPsVBFDnaTt23Jjwq8RnzM/CA 1XgZ2jaqDwUdck0QYUoS8NjmBla8IprNzbgHChv0/kR/IE1Qr3VFzsPZ/C3VNXg9GoX8 EYzdwLmBcHDe8iw4eXb7I5qc5JbKpwHoDhkjhfWOv59KtgxsMnkQ5VyqctbxNv/NYbAM xbnhPYNCKULlNkp7jWy97NjueHAeYVOxJZa7yPFdjT6ctKmLRLHyiRxeO0WCgtgtuGta kqrw== X-Gm-Message-State: ABy/qLYn0jmAKk8BWuOLfryTiDslLmcIYPujLFjsN3vRt5lOaf+GMrA4 2BeRVyS5+mvEh9kmDDU7SRbXFkRkeKj64hz94ZU= X-Google-Smtp-Source: APBJJlFbDvIxRhZzCHGxJ+z3mh9q7Z81KPGiiVF/FdtdCaHvmPHltoI4MP8j5Pk59126WIFIdILnKQ== X-Received: by 2002:a05:600c:ac9:b0:3fb:4053:a9d5 with SMTP id c9-20020a05600c0ac900b003fb4053a9d5mr2298320wmr.25.1689691386166; Tue, 18 Jul 2023 07:43:06 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:2bce:64d6:1a5c:49a2]) by smtp.gmail.com with ESMTPSA id 12-20020a05600c230c00b003f90b9b2c31sm10469686wmo.28.2023.07.18.07.43.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Jul 2023 07:43:05 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Tue, 18 Jul 2023 15:43:01 +0100 Message-Id: <20230718144301.3612-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230718144301.3612-1-david.plowman@raspberrypi.com> References: <20230718144301.3612-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 2/2] ipa: rpi: common: Handle AEC/AGC flicker controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: David Plowman via libcamera-devel From: David Plowman Reply-To: David Plowman Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We handle the flicker modes by passing the correct period to the AEC/AGC algorithm which already contains the necessary code. The "Auto" mode, as well as reporting the detected flicker period via the "AeFlickerDetected" metadata, are unsupported for now. Signed-off-by: David Plowman Reviewed-by: Jacopo Mondi Reviewed-by: Umang Jain --- src/ipa/rpi/common/ipa_base.cpp | 64 ++++++++++++++++++++++++++++++++- src/ipa/rpi/common/ipa_base.h | 6 ++++ 2 files changed, 69 insertions(+), 1 deletion(-) diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index f40f2e71..4438ecd9 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -61,6 +61,10 @@ const ControlInfoMap::Map ipaControls{ { &controls::AeConstraintMode, ControlInfo(controls::AeConstraintModeValues) }, { &controls::AeExposureMode, ControlInfo(controls::AeExposureModeValues) }, { &controls::ExposureValue, ControlInfo(-8.0f, 8.0f, 0.0f) }, + { &controls::AeFlickerMode, ControlInfo(static_cast(controls::FlickerOff), + static_cast(controls::FlickerManual), + static_cast(controls::FlickerOff)) }, + { &controls::AeFlickerPeriod, ControlInfo(100, 1000000) }, { &controls::Brightness, ControlInfo(-1.0f, 1.0f, 0.0f) }, { &controls::Contrast, ControlInfo(0.0f, 32.0f, 1.0f) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, @@ -97,7 +101,7 @@ namespace ipa::RPi { IpaBase::IpaBase() : controller_(), frameCount_(0), mistrustCount_(0), lastRunTimestamp_(0), - firstStart_(true) + firstStart_(true), flickerState_({ 0, 0s }) { } @@ -812,6 +816,64 @@ void IpaBase::applyControls(const ControlList &controls) break; } + case controls::AE_FLICKER_MODE: { + RPiController::AgcAlgorithm *agc = dynamic_cast( + controller_.getAlgorithm("agc")); + if (!agc) { + LOG(IPARPI, Warning) + << "Could not set AeFlickerMode - no AGC algorithm"; + break; + } + + int32_t mode = ctrl.second.get(); + bool modeValid = true; + + switch (mode) { + case controls::FlickerOff: + agc->setFlickerPeriod(0us); + + break; + + case controls::FlickerManual: + agc->setFlickerPeriod(flickerState_.manualPeriod); + + break; + + default: + LOG(IPARPI, Error) << "Flicker mode " << mode << " is not supported"; + modeValid = false; + + break; + } + + if (modeValid) + flickerState_.mode = mode; + + break; + } + + case controls::AE_FLICKER_PERIOD: { + RPiController::AgcAlgorithm *agc = dynamic_cast( + controller_.getAlgorithm("agc")); + if (!agc) { + LOG(IPARPI, Warning) + << "Could not set AeFlickerPeriod - no AGC algorithm"; + break; + } + + uint32_t manualPeriod = ctrl.second.get(); + flickerState_.manualPeriod = manualPeriod * 1.0us; + + /* + * We note that it makes no difference if the mode gets set to "manual" + * first, and the period updated after, or vice versa. + */ + if (flickerState_.mode == controls::FlickerManual) + agc->setFlickerPeriod(flickerState_.manualPeriod); + + break; + } + case controls::AWB_ENABLE: { /* Silently ignore this control for a mono sensor. */ if (monoSensor_) diff --git a/src/ipa/rpi/common/ipa_base.h b/src/ipa/rpi/common/ipa_base.h index 39d00760..097f436a 100644 --- a/src/ipa/rpi/common/ipa_base.h +++ b/src/ipa/rpi/common/ipa_base.h @@ -116,6 +116,12 @@ private: /* Frame duration (1/fps) limits. */ utils::Duration minFrameDuration_; utils::Duration maxFrameDuration_; + + /* The current state of flicker avoidance. */ + struct FlickerState { + int32_t mode; + utils::Duration manualPeriod; + } flickerState_; }; } /* namespace ipa::RPi */