From patchwork Thu Jul 13 15:12:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 18820 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8C9C9C3243 for ; Thu, 13 Jul 2023 15:12:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 11327628C6; Thu, 13 Jul 2023 17:12:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1689261144; bh=V/gSmuMH38IPCacjZgb56tS70tY0C7M2x0mnRwta7xQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=O+J5UiDr0p2/beJHScs+1LjQBSYCHZuDliqN7xvxAF5cea+hvANP5/W2As8VKu8QB k33hbn5U4IQ+s/i+Rw7OriJaYtxE9gvOzIjQsWBmg35jJXzJi01ibaL32A5T+BKQP3 ZQOaQ0oU8GgS4y3yZtfSvN956n9oZ/6zODhNTcGm8wwf9zIozo5s9mQJtC7MChEeUf KgWtu/ls+M02UU7ySVj4oebtYq26HImDTM5xvqFb1w1z3baCsu8VZtABXyesx3z6R3 kpxaGOHhGNX39IWvMKPrR2vSW6TythsW9RUSkpU8AQjKPR9z2aZ9zLo40BKsEG3U/I hLVrBpwo+OawA== Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 33E9E628BC for ; Thu, 13 Jul 2023 17:12:22 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="tBhAzCmZ"; dkim-atps=neutral Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3fbc6ab5ff5so7690615e9.1 for ; Thu, 13 Jul 2023 08:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1689261141; x=1691853141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KwGuxCULk77E/Yw9MPkmolm5Nod6d9LR09UZXbDa/Kg=; b=tBhAzCmZn2AqQuL6GjLlq+HQguo7N/EFbadRUIA7od9VGqROC7WndsBu7oVy4EYLvy 2u2xfNDC2wLkyuWg9eflmet6Cb/Jmugtzp7xwBqMgKy1prcPtxdd0jd5yz1qLOMXwqXd Y6rxZBbJ7myUR6cH0uHqV3PTl9GuetSv5DIfdr5zBtRt1hdpvwKTlGY66g6S98lMiF9d ejndtKDdMeojokEatwgIv/7NJmGBVV/4tZhGUIEz1IyMmdHmYbYuDhSi5lmG0BO/g5tK o/JPD6xjzn9X9gHmwAe0LK9ka+c+vkHouSeVX15KzSwQybofAcItgSd3wbHbSDua3wmO oX1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689261141; x=1691853141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KwGuxCULk77E/Yw9MPkmolm5Nod6d9LR09UZXbDa/Kg=; b=UD7qns77S/NfQsooGqm/qCrCKJRxYlVbOQigH4HxAJ1gVVYsy8rRJxD9tR3QI6FCFh bQc55oqxfXoSKrRWuQOdgGckmUfeHocrRpAa4dlrYVeFfAgfo9MJ1QZX3Rnna5QSKPm9 bLjTqfsUdWv7fGmoKEO/xWi0IzWTwYWoyONzezsAdYbidJmpSEHWs8KqGIzl9KMBTgTc agvOkOfEX30+Mh7ye0zqkQSIpIzoyHprQnQllJDe8BznzPmPAmOe2jLQ9P6eRzXf6oAW +aVJXw0U4HgUbW/FovSK3yPThRhxC8+/I1ZJ/b+yafASCfhfySwoFOFOi0dyyY03FjIM qSgw== X-Gm-Message-State: ABy/qLbKsUqirBP+s0+f6lSb8uSmPBHq1GwDl5nhTUuaK+BuISIFK5Sw v3kx1vYLN4PzFdRTVsn5TorYwDO2FS8aAgSohWk= X-Google-Smtp-Source: APBJJlGI/4HzBUTHoLAAM8Gty2D3msCDJdhQK9pDwlocLuj34R0ej8O0pfWVKQOq73PuFWNSrh3K4Q== X-Received: by 2002:a7b:ca4a:0:b0:3fc:240:cecc with SMTP id m10-20020a7bca4a000000b003fc0240ceccmr1716316wml.11.1689261141441; Thu, 13 Jul 2023 08:12:21 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:2bce:64d6:1a5c:49a2]) by smtp.gmail.com with ESMTPSA id f11-20020a7bc8cb000000b003fa98908014sm18771086wml.8.2023.07.13.08.12.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 08:12:21 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Thu, 13 Jul 2023 16:12:17 +0100 Message-Id: <20230713151218.26045-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230713151218.26045-1-david.plowman@raspberrypi.com> References: <20230713151218.26045-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 1/2] libcamera: controls: Add controls for AEC/AGC flicker avoidance X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: David Plowman via libcamera-devel From: David Plowman Reply-To: David Plowman Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Flicker is the term used to describe brightness banding or oscillation of images caused typically by artificial lighting driven by a 50 or 60Hz mains supply. We add three controls intended to be used by AEC/AGC algorithms: AeFlickerMode to enable flicker avoidance. AeFlickerCustom to set custom flicker periods. AeFlickerDetected to report any flicker that is currently detected. Signed-off-by: David Plowman --- src/libcamera/control_ids.yaml | 79 ++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 17 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 056886e6..f1629b89 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -156,6 +156,68 @@ controls: control of which features should be automatically adjusted shouldn't better be handled through a separate AE mode control. + - AeFlickerMode: + type: int32_t + description: | + Set the flicker mode, which determines whether, and how, the AGC/AEC + algorithm attempts to hide flicker effects caused by the duty cycle of + artificial lighting. + + Although implementation dependent, many algorithms for "flicker + avoidance" work by restricting this exposure time to integer multiples + of the cycle period, wherever possible. + + Implementations may not support all of the flicker modes listed below. + + By default the system will start in FlickerAuto mode if this is + supported, otherwise the flicker mode will be set to FlickerOff. + + enum: + - name: FlickerOff + value: 0 + description: No flicker avoidance is performed. + - name: FlickerCustom + value: 1 + description: Custom flicker avoidance. + Suppress flicker effects caused by lighting running with a period + specified by the AeFlickerCustom control. + \sa AeFlickerCustom + - name: FlickerAuto + value: 2 + description: Automatic flicker period detection and avoidance. + The system will automatically determine the most likely value of + flicker period, and avoid flicker of this frequency. Once flicker + is being corrected, it is implementation dependent whether the + system is still able to detect a change in the flicker period. + + - AeFlickerCustom: + type: int32_t + description: Custom flicker period in microseconds. + This value sets the current flicker period to avoid. It is used when + AeFlickerMode is set to FlickerCustom. + + To cancel 50Hz mains flicker, this should be set to 10000 (corresponding + to 100Hz), or 8333 (120Hz) for 60Hz mains. + + If this control is not available, then the setting of custom flicker + periods is not supported. + + \sa AeFlickerMode + + - AeFlickerDetected: + type: int32_t + description: Flicker period detected in microseconds. + The value reported here indicates the currently detected flicker + period, or zero if no flicker at all is detected. + + In the case of 50Hz mains flicker, the value would be 10000 + (corresponding to 100Hz), or 8333 (120Hz) for 60Hz mains flicker. + + It is implementation dependent exactly when the system is able + to detect and report the flicker period. + + \sa AeFlickerMode + - Brightness: type: float description: | @@ -850,23 +912,6 @@ controls: value: 1 description: The lens shading map mode is available. - - SceneFlicker: - type: int32_t - draft: true - description: | - Control to report the detected scene light frequency. Currently - identical to ANDROID_STATISTICS_SCENE_FLICKER. - enum: - - name: SceneFickerOff - value: 0 - description: No flickering detected. - - name: SceneFicker50Hz - value: 1 - description: 50Hz flickering detected. - - name: SceneFicker60Hz - value: 2 - description: 60Hz flickering detected. - - PipelineDepth: type: int32_t draft: true From patchwork Thu Jul 13 15:12:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 18821 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4F7DFBEFBE for ; Thu, 13 Jul 2023 15:12:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C302A628BE; Thu, 13 Jul 2023 17:12:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1689261144; bh=LjLquS0H3O5FIw/g+yfCCv2S++a1b3mowkz+YB+h3wQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Ad8Z4/y2woG90VWP/RMwdvsILAd0dpzJRZc0SuDqAFoEzcb4YWIJESMCNT3mB6PRo 5/5twXa7O4sZvGLmMH6hqeaYdDYDYTVNOzVe1273RP/j8vsPbMbtpFRgD22Zj/CkQd LVncmELj/JU44ZlvXi3UBgDke2tJ05C5qNVtn8Iiu4NNK54FMoktPfYYT8sH0Gufkw 49EtwZp/Iqn6y0tJSxo5u0uBT1pP0BAroArVW6Dxd1g+d5S4moikfIg8NvSdW5bfnp JKi+Z+hSMo6aayVBq4J01xkRUkThZ8KmL57wLv8os5G8VS5lEiYXe25kkfPE3V6DCz ZUJ1xYj4GwSNg== Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 95715628BC for ; Thu, 13 Jul 2023 17:12:22 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="cb45i4zs"; dkim-atps=neutral Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-3fbc0609cd6so7664805e9.1 for ; Thu, 13 Jul 2023 08:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1689261142; x=1691853142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XzmlYLkxgEHn2L3BZQy6j1gfEvkwbGXBFfztjzabUdo=; b=cb45i4zsSCLy64uMXMHGrK3LJhPwHVNFQFeTO+yX3wAD9n+Abik+Hz/O01WG352doy HTeGDWMFe2clNY2osHFiDsUykbKZzzBTJKE2Yiqgyh/Tj0q8tXD9IMG0Q8Woz7Q/sGNs aX0ArMvkwOazpTxlqDu5H6B5+L+8U1UN7BovhFy6luIdqxtatxbBPJ808JEiKmZShV+u NXmkvJ3tairMKV5TTFIWlfy/9Rjk05UGLm7TDtEJPLcJ0ZIhPToYukQmJb+UYLo9CUuf WHjojL2yvKb0iVPk/ZlnAks/ygUU3L+PPiQmIicpjDzhX2WE5SILByWxjPzl0rqS3FZ4 gT7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689261142; x=1691853142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XzmlYLkxgEHn2L3BZQy6j1gfEvkwbGXBFfztjzabUdo=; b=BL+wj/f2+NfU1KR3UEFrK+NpBjZ19SG9rudX5X10ro2gsZNBpIC59yEd8gwYJWXuk0 J+dOtNy6YXyo06jz6GR80WzfHbHLIHJZSY1jjIUZl1iFk4B0Bxsgnrf6fYdZYbVMDI8A UPaULkLnSoqdfLMI1lDVjcgzjeoLJeNe9pP7DfIfUiGUDlaOSeC3GoJ5XhYSrkWNfkKL L/o+FqfQ/Kyabey01b4FMrPaewjbJUJrUzOLcvTYsvLpPO8blxBIf/0irad/2QAsQx8o E0gsB7BF5GPg6qLTukeysyPKSsEAK6m5jpMdV8SIHk3QD+zZMpQM98Wh/pn3U6VAb3Uf BgDw== X-Gm-Message-State: ABy/qLYc4iH5Z8ZD9MNN7g8Xpm0IXENZ9TFsGHJeJKGPb7fEt594EJnw l7w454v2hOMhN/C2+MXZ2WH68b2CXSScLw2oc5E= X-Google-Smtp-Source: APBJJlHNdIJR5nVKtVOIZRTZCiWDiwZ2lg2Cl0IoY8FltDwhmWMIa33i/blVZgqTH4EzPKue6CQTkw== X-Received: by 2002:a1c:4b11:0:b0:3fa:962e:e8f4 with SMTP id y17-20020a1c4b11000000b003fa962ee8f4mr1595101wma.26.1689261142058; Thu, 13 Jul 2023 08:12:22 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:2bce:64d6:1a5c:49a2]) by smtp.gmail.com with ESMTPSA id f11-20020a7bc8cb000000b003fa98908014sm18771086wml.8.2023.07.13.08.12.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jul 2023 08:12:21 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Thu, 13 Jul 2023 16:12:18 +0100 Message-Id: <20230713151218.26045-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230713151218.26045-1-david.plowman@raspberrypi.com> References: <20230713151218.26045-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 2/2] ipa: rpi: common: Handle AEC/AGC flicker controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: David Plowman via libcamera-devel From: David Plowman Reply-To: David Plowman Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We handle the flicker modes by passing the correct period to the AEC/AGC algorithm which already contains the necessary code. The "Auto" mode, as well as reporting the detected flicker period via the "AeFlickerDetected" metadata, are unsupported for now. Signed-off-by: David Plowman --- src/ipa/rpi/common/ipa_base.cpp | 64 ++++++++++++++++++++++++++++++++- src/ipa/rpi/common/ipa_base.h | 6 ++++ 2 files changed, 69 insertions(+), 1 deletion(-) diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index f40f2e71..81d65ea5 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -61,6 +61,8 @@ const ControlInfoMap::Map ipaControls{ { &controls::AeConstraintMode, ControlInfo(controls::AeConstraintModeValues) }, { &controls::AeExposureMode, ControlInfo(controls::AeExposureModeValues) }, { &controls::ExposureValue, ControlInfo(-8.0f, 8.0f, 0.0f) }, + { &controls::AeFlickerMode, ControlInfo(controls::AeFlickerModeValues) }, + { &controls::AeFlickerCustom, ControlInfo(100, 1000000) }, { &controls::Brightness, ControlInfo(-1.0f, 1.0f, 0.0f) }, { &controls::Contrast, ControlInfo(0.0f, 32.0f, 1.0f) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, @@ -97,7 +99,7 @@ namespace ipa::RPi { IpaBase::IpaBase() : controller_(), frameCount_(0), mistrustCount_(0), lastRunTimestamp_(0), - firstStart_(true) + firstStart_(true), flickerState_({ 0, 0s }) { } @@ -812,6 +814,66 @@ void IpaBase::applyControls(const ControlList &controls) break; } + case controls::AE_FLICKER_MODE: { + RPiController::AgcAlgorithm *agc = dynamic_cast( + controller_.getAlgorithm("agc")); + if (!agc) { + LOG(IPARPI, Warning) + << "Could not set AeFlickerMode - no AGC algorithm"; + break; + } + + int32_t mode = ctrl.second.get(); + bool modeValid = true; + + switch (mode) { + case controls::FlickerOff: { + agc->setFlickerPeriod(0us); + + break; + } + + case controls::FlickerCustom: { + agc->setFlickerPeriod(flickerState_.customPeriod); + + break; + } + + default: + LOG(IPARPI, Error) << "Flicker mode " << mode << " is not supported"; + modeValid = false; + + break; + } + + if (modeValid) + flickerState_.mode = mode; + + break; + } + + case controls::AE_FLICKER_CUSTOM: { + RPiController::AgcAlgorithm *agc = dynamic_cast( + controller_.getAlgorithm("agc")); + if (!agc) { + LOG(IPARPI, Warning) + << "Could not set AeFlickerCustom - no AGC algorithm"; + break; + } + + uint32_t customPeriod = ctrl.second.get(); + flickerState_.customPeriod = customPeriod * 1.0us; + + /* + * We note that it makes no difference if the mode gets set to "custom" + * first, and the period updated after, or vice versa. + */ + if (flickerState_.mode == controls::FlickerCustom) + agc->setFlickerPeriod(flickerState_.customPeriod); + + break; + } + case controls::AWB_ENABLE: { /* Silently ignore this control for a mono sensor. */ if (monoSensor_) diff --git a/src/ipa/rpi/common/ipa_base.h b/src/ipa/rpi/common/ipa_base.h index 39d00760..22823176 100644 --- a/src/ipa/rpi/common/ipa_base.h +++ b/src/ipa/rpi/common/ipa_base.h @@ -116,6 +116,12 @@ private: /* Frame duration (1/fps) limits. */ utils::Duration minFrameDuration_; utils::Duration maxFrameDuration_; + + /* The current state of flicker avoidance. */ + struct FlickerState { + int32_t mode; + utils::Duration customPeriod; + } flickerState_; }; } /* namespace ipa::RPi */