{"id":19019,"url":"https://patchwork.libcamera.org/api/patches/19019/?format=json","web_url":"https://patchwork.libcamera.org/patch/19019/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20230915122954.5231-6-david.plowman@raspberrypi.com>","date":"2023-09-15T12:29:54","name":"[libcamera-devel,v4,5/5] ipa: rpi: agc: Use channel constraints in the AGC algorithm","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"4ff5b4525e7c8dc460acea649acc2232bd746c97","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/?format=json","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/19019/mbox/","series":[{"id":4024,"url":"https://patchwork.libcamera.org/api/series/4024/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4024","date":"2023-09-15T12:29:49","name":"Multi-channel AGC","version":4,"mbox":"https://patchwork.libcamera.org/series/4024/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/19019/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/19019/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 94C22C32B0\n\tfor ;\n\tFri, 15 Sep 2023 12:30:27 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4847262926;\n\tFri, 15 Sep 2023 14:30:27 +0200 (CEST)","from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com\n\t[IPv6:2a00:1450:4864:20::32d])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9DB3362921\n\tfor ;\n\tFri, 15 Sep 2023 14:30:24 +0200 (CEST)","by mail-wm1-x32d.google.com with SMTP id\n\t5b1f17b1804b1-404c023ef5eso608205e9.2\n\tfor ;\n\tFri, 15 Sep 2023 05:30:24 -0700 (PDT)","from localhost.localdomain ([195.180.61.40])\n\tby smtp.gmail.com with ESMTPSA id\n\ty14-20020a7bcd8e000000b004030e8ff964sm7435009wmj.34.2023.09.15.05.30.22\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 15 Sep 2023 05:30:22 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1694781027;\n\tbh=PFRHo7ZjisGDzWh1gjMquuN4xevwKE6p6hdi0bQ7CSw=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=3LEWej7XO63I1Pf1ZStDcVsIX+5uGGda1O0znh2lEF0WbzzhGxoLoblXatuqQU+Mj\n\tXIihUeuUAde1Rli0g15dnzsEX2cAeSW6IVrVu9guMcccC4lsrqvCD2LTzQ3PabuPFB\n\t5uH4/Na8z1GgwGNSQs4pqSuPRPtL0TMpEQRPckCfcUwpHAyfQpJVz8ehkSgLJHxBpX\n\tCn9gE9JbhRjDihjYyOpU9TfkF/c0nMHf/DU9r2DWT0T77WSjHih4dX3z2JAfswGW9K\n\tb0HIjaOUpWXvXo/p5kq4UeU7CNmOrnhH4AfCwx1VOwENt/yojJs8qvKx76A5sDCuC/\n\tnFlLO6zoz9fNA==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1694781024; x=1695385824;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=AFKDyVvaVCZIc4fPNYqVnvCQYOh7SjNAKjO928Hakpk=;\n\tb=SDjslvxQ1447CQd3tnGDI6NFj9/bliqUENoxw2RBSPbVIEuAgjKpTyn61CIqtlyDuu\n\tqIuIcJtFyMDU76g37Ns21UkQSSKresJGLdBB/+J6ryc9yEwt1F1bCX7C+5WOErRo9Hc7\n\tHkhqqMTSAnuDqBNetqRIPtJQVzg5bS8orWnuDM2+Wo/VWSNIpJzqgirOYu6dbKl92Z9K\n\te2Gm1mmiJ/V2a1KY86X+FLyEbrUcDo60MqcgmcyjFIDAE9/e1SucJcqSbY6OoyZGfq0Z\n\tflKloyW3mXCJtHmac+Ez7JbrkaWl9dR/T8XI5EO90H3Cb93X4/bAf/lRE2lHOHDgoIbH\n\tp4lQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"SDjslvxQ\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1694781024; x=1695385824;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=AFKDyVvaVCZIc4fPNYqVnvCQYOh7SjNAKjO928Hakpk=;\n\tb=n6tCDZm5IJKX9yaV27F4iQcZlpnl84ikcfs1oj+E1f4gpQWZGEDCrFtyLwj231rWKk\n\tpnYuR1JB2MvJkTwToedIbtsvoi2YEQ1KD4B0pUt+1ZwBi1n7ybfcdjo8Whbnx/dTFSgG\n\tKMVSvZOwQUupSoHVOXNM0oAHVC4EmVAFAbA17q6USXvvVUx9ivolPGBfSkQcJzVkXrZf\n\t5BJGhq/xXbFfnRP86zOIu/fvusw60egLLNZ4BtNombHJTtg+Qj1GY+U+OyVOblWC9ofy\n\tAF9gU0EFmW/dpiZh5wIi005jEl9nxFCO5rk37/Lu52lj/2apzj8dzEGwqFmS9fVgaaBl\n\t5aGw==","X-Gm-Message-State":"AOJu0YyAiwGhryHwrHqZrR+1hRpr7UvC2EEMD/8m0r5TkpQaD6OFXltS\n\tmtbzBBLErtmPgt/QlWPHbTAHUMxPYrKZsjGQR58=","X-Google-Smtp-Source":"AGHT+IGtGViaZYX7OFSUn6KlG49IN4I6OGHFGv592c5e2jGLzjxi20QSQt8nUdMFhhYUqKjRlMlzkA==","X-Received":"by 2002:a1c:ed08:0:b0:402:cd60:8aa9 with SMTP id\n\tl8-20020a1ced08000000b00402cd608aa9mr1354879wmh.20.1694781023718; \n\tFri, 15 Sep 2023 05:30:23 -0700 (PDT)","To":"libcamera-devel@lists.libcamera.org","Date":"Fri, 15 Sep 2023 13:29:54 +0100","Message-Id":"<20230915122954.5231-6-david.plowman@raspberrypi.com>","X-Mailer":"git-send-email 2.30.2","In-Reply-To":"<20230915122954.5231-1-david.plowman@raspberrypi.com>","References":"<20230915122954.5231-1-david.plowman@raspberrypi.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v4 5/5] ipa: rpi: agc: Use channel\n\tconstraints in the AGC algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"Jacopo Mondi ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Whenever we run Agc::process(), we store the most recent total\nexposure requested for each channel.\n\nWith these values we can apply the channel constraints after\ntime-filtering the requested total exposure, but before working out\nhow much digital gain is needed.\n\nSigned-off-by: David Plowman \nReviewed-by: Naushir Patuck \nReviewed-by: Jacopo Mondi \n---\n src/ipa/rpi/controller/rpi/agc.cpp | 22 ++++++--\n src/ipa/rpi/controller/rpi/agc.h | 1 +\n src/ipa/rpi/controller/rpi/agc_channel.cpp | 58 +++++++++++++++++++---\n src/ipa/rpi/controller/rpi/agc_channel.h | 8 ++-\n 4 files changed, 75 insertions(+), 14 deletions(-)","diff":"diff --git a/src/ipa/rpi/controller/rpi/agc.cpp b/src/ipa/rpi/controller/rpi/agc.cpp\nindex 8b9d2026..870cb431 100644\n--- a/src/ipa/rpi/controller/rpi/agc.cpp\n+++ b/src/ipa/rpi/controller/rpi/agc.cpp\n@@ -39,6 +39,7 @@ int Agc::read(const libcamera::YamlObject ¶ms)\n \t */\n \tif (!params.contains(\"channels\")) {\n \t\tLOG(RPiAgc, Debug) << \"Single channel only\";\n+\t\tchannelTotalExposures_.resize(1, 0s);\n \t\tchannelData_.emplace_back();\n \t\treturn channelData_.back().channel.read(params, getHardwareConfig());\n \t}\n@@ -58,6 +59,8 @@ int Agc::read(const libcamera::YamlObject ¶ms)\n \t\treturn -1;\n \t}\n \n+\tchannelTotalExposures_.resize(channelData_.size(), 0s);\n+\n \treturn 0;\n }\n \n@@ -236,16 +239,22 @@ static void getDelayedChannelIndex(Metadata *metadata, const char *message, unsi\n \t}\n }\n \n-static void setCurrentChannelIndex(Metadata *metadata, const char *message, unsigned int channelIndex)\n+static libcamera::utils::Duration\n+setCurrentChannelIndexGetExposure(Metadata *metadata, const char *message, unsigned int channelIndex)\n {\n \tstd::unique_lock lock(*metadata);\n \tAgcStatus *status = metadata->getLocked(\"agc.status\");\n-\tif (status)\n+\tlibcamera::utils::Duration dur = 0s;\n+\n+\tif (status) {\n \t\tstatus->channel = channelIndex;\n-\telse {\n+\t\tdur = status->totalExposureValue;\n+\t} else {\n \t\t/* This does happen at startup, otherwise it would be a Warning or Error. */\n \t\tLOG(RPiAgc, Debug) << message;\n \t}\n+\n+\treturn dur;\n }\n \n void Agc::prepare(Metadata *imageMetadata)\n@@ -310,8 +319,11 @@ void Agc::process(StatisticsPtr &stats, Metadata *imageMetadata)\n \t\tLOG(RPiAgc, Debug) << \"process: channel \" << channelIndex << \" not seen yet\";\n \t}\n \n-\tchannelData.channel.process(stats, deviceStatus, imageMetadata);\n-\tsetCurrentChannelIndex(imageMetadata, \"process: no AGC status found\", channelIndex);\n+\tchannelData.channel.process(stats, deviceStatus, imageMetadata, channelTotalExposures_);\n+\tauto dur = setCurrentChannelIndexGetExposure(imageMetadata, \"process: no AGC status found\",\n+\t\t\t\t\t\t channelIndex);\n+\tif (dur)\n+\t\tchannelTotalExposures_[channelIndex] = dur;\n \n \t/* And onto the next channel for the next call. */\n \tindex_ = (index_ + 1) % activeChannels_.size();\ndiff --git a/src/ipa/rpi/controller/rpi/agc.h b/src/ipa/rpi/controller/rpi/agc.h\nindex ee85c693..90890439 100644\n--- a/src/ipa/rpi/controller/rpi/agc.h\n+++ b/src/ipa/rpi/controller/rpi/agc.h\n@@ -55,6 +55,7 @@ private:\n \tstd::vector channelData_;\n \tstd::vector activeChannels_;\n \tunsigned int index_; /* index into the activeChannels_ */\n+\tAgcChannelTotalExposures channelTotalExposures_;\n };\n \n } /* namespace RPiController */\ndiff --git a/src/ipa/rpi/controller/rpi/agc_channel.cpp b/src/ipa/rpi/controller/rpi/agc_channel.cpp\nindex afa06686..93174a4b 100644\n--- a/src/ipa/rpi/controller/rpi/agc_channel.cpp\n+++ b/src/ipa/rpi/controller/rpi/agc_channel.cpp\n@@ -493,7 +493,9 @@ void AgcChannel::prepare(Metadata *imageMetadata)\n \t}\n }\n \n-void AgcChannel::process(StatisticsPtr &stats, DeviceStatus const &deviceStatus, Metadata *imageMetadata)\n+void AgcChannel::process(StatisticsPtr &stats, DeviceStatus const &deviceStatus,\n+\t\t\t Metadata *imageMetadata,\n+\t\t\t const AgcChannelTotalExposures &channelTotalExposures)\n {\n \tframeCount_++;\n \t/*\n@@ -512,12 +514,17 @@ void AgcChannel::process(StatisticsPtr &stats, DeviceStatus const &deviceStatus,\n \tcomputeTargetExposure(gain);\n \t/* The results have to be filtered so as not to change too rapidly. */\n \tfilterExposure();\n+\t/*\n+\t * We may be asked to limit the exposure using other channels. If another channel\n+\t * determines our upper bound we may want to know this later.\n+\t */\n+\tbool channelBound = applyChannelConstraints(channelTotalExposures);\n \t/*\n \t * Some of the exposure has to be applied as digital gain, so work out\n-\t * what that is. This function also tells us whether it's decided to\n-\t * \"desaturate\" the image more quickly.\n+\t * what that is. It also tells us whether it's trying to desaturate the image\n+\t * more quickly, which can only happen when another channel is not limiting us.\n \t */\n-\tbool desaturate = applyDigitalGain(gain, targetY);\n+\tbool desaturate = applyDigitalGain(gain, targetY, channelBound);\n \t/*\n \t * The last thing is to divide up the exposure value into a shutter time\n \t * and analogue gain, according to the current exposure mode.\n@@ -794,7 +801,44 @@ void AgcChannel::computeTargetExposure(double gain)\n \tLOG(RPiAgc, Debug) << \"Target totalExposure \" << target_.totalExposure;\n }\n \n-bool AgcChannel::applyDigitalGain(double gain, double targetY)\n+bool AgcChannel::applyChannelConstraints(const AgcChannelTotalExposures &channelTotalExposures)\n+{\n+\tbool channelBound = false;\n+\tLOG(RPiAgc, Debug)\n+\t\t<< \"Total exposure before channel constraints \" << filtered_.totalExposure;\n+\n+\tfor (const auto &constraint : config_.channelConstraints) {\n+\t\tLOG(RPiAgc, Debug)\n+\t\t\t<< \"Check constraint: channel \" << constraint.channel << \" bound \"\n+\t\t\t<< (constraint.bound == AgcChannelConstraint::Bound::UPPER ? \"UPPER\" : \"LOWER\")\n+\t\t\t<< \" factor \" << constraint.factor;\n+\t\tif (constraint.channel >= channelTotalExposures.size() ||\n+\t\t !channelTotalExposures[constraint.channel]) {\n+\t\t\tLOG(RPiAgc, Debug) << \"no such channel or no exposure available- skipped\";\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tlibcamera::utils::Duration limitExposure =\n+\t\t\tchannelTotalExposures[constraint.channel] * constraint.factor;\n+\t\tLOG(RPiAgc, Debug) << \"Limit exposure \" << limitExposure;\n+\t\tif ((constraint.bound == AgcChannelConstraint::Bound::UPPER &&\n+\t\t filtered_.totalExposure > limitExposure) ||\n+\t\t (constraint.bound == AgcChannelConstraint::Bound::LOWER &&\n+\t\t filtered_.totalExposure < limitExposure)) {\n+\t\t\tfiltered_.totalExposure = limitExposure;\n+\t\t\tLOG(RPiAgc, Debug) << \"Constraint applies\";\n+\t\t\tchannelBound = true;\n+\t\t} else\n+\t\t\tLOG(RPiAgc, Debug) << \"Constraint does not apply\";\n+\t}\n+\n+\tLOG(RPiAgc, Debug)\n+\t\t<< \"Total exposure after channel constraints \" << filtered_.totalExposure;\n+\n+\treturn channelBound;\n+}\n+\n+bool AgcChannel::applyDigitalGain(double gain, double targetY, bool channelBound)\n {\n \tdouble minColourGain = std::min({ awb_.gainR, awb_.gainG, awb_.gainB, 1.0 });\n \tASSERT(minColourGain != 0.0);\n@@ -814,8 +858,8 @@ bool AgcChannel::applyDigitalGain(double gain, double targetY)\n \t * quickly (and we then approach the correct value more quickly from\n \t * below).\n \t */\n-\tbool desaturate = targetY > config_.fastReduceThreshold &&\n-\t\t\t gain < sqrt(targetY);\n+\tbool desaturate = !channelBound &&\n+\t\t\t targetY > config_.fastReduceThreshold && gain < sqrt(targetY);\n \tif (desaturate)\n \t\tdg /= config_.fastReduceThreshold;\n \tLOG(RPiAgc, Debug) << \"Digital gain \" << dg << \" desaturate? \" << desaturate;\ndiff --git a/src/ipa/rpi/controller/rpi/agc_channel.h b/src/ipa/rpi/controller/rpi/agc_channel.h\nindex ceaab12f..ae826fa8 100644\n--- a/src/ipa/rpi/controller/rpi/agc_channel.h\n+++ b/src/ipa/rpi/controller/rpi/agc_channel.h\n@@ -21,6 +21,8 @@\n \n namespace RPiController {\n \n+using AgcChannelTotalExposures = std::vector;\n+\n struct AgcMeteringMode {\n \tstd::vector weights;\n \tint read(const libcamera::YamlObject ¶ms);\n@@ -95,7 +97,8 @@ public:\n \tvoid disableAuto();\n \tvoid switchMode(CameraMode const &cameraMode, Metadata *metadata);\n \tvoid prepare(Metadata *imageMetadata);\n-\tvoid process(StatisticsPtr &stats, DeviceStatus const &deviceStatus, Metadata *imageMetadata);\n+\tvoid process(StatisticsPtr &stats, DeviceStatus const &deviceStatus, Metadata *imageMetadata,\n+\t\t const AgcChannelTotalExposures &channelTotalExposures);\n \n private:\n \tbool updateLockStatus(DeviceStatus const &deviceStatus);\n@@ -107,7 +110,8 @@ private:\n \t\t\t double &gain, double &targetY);\n \tvoid computeTargetExposure(double gain);\n \tvoid filterExposure();\n-\tbool applyDigitalGain(double gain, double targetY);\n+\tbool applyChannelConstraints(const AgcChannelTotalExposures &channelTotalExposures);\n+\tbool applyDigitalGain(double gain, double targetY, bool channelBound);\n \tvoid divideUpExposure();\n \tvoid writeAndFinish(Metadata *imageMetadata, bool desaturate);\n \tlibcamera::utils::Duration limitShutter(libcamera::utils::Duration shutter);\n","prefixes":["libcamera-devel","v4","5/5"]}