{"id":19038,"url":"https://patchwork.libcamera.org/api/1.1/patches/19038/?format=json","web_url":"https://patchwork.libcamera.org/patch/19038/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20230915155844.2388-6-david.plowman@raspberrypi.com>","date":"2023-09-15T15:58:44","name":"[libcamera-devel,v5,5/5] ipa: rpi: agc: Use channel constraints in the AGC algorithm","commit_ref":"7127954aaa5fe9938743cbc41a328027697aba8a","pull_url":null,"state":"accepted","archived":false,"hash":"4ff5b4525e7c8dc460acea649acc2232bd746c97","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/1.1/people/42/?format=json","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/19038/mbox/","series":[{"id":4026,"url":"https://patchwork.libcamera.org/api/1.1/series/4026/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4026","date":"2023-09-15T15:58:39","name":"Multi-channel AGC","version":5,"mbox":"https://patchwork.libcamera.org/series/4026/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/19038/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/19038/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0A949C32B0\n\tfor ;\n\tFri, 15 Sep 2023 15:59:00 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A196C6291C;\n\tFri, 15 Sep 2023 17:58:59 +0200 (CEST)","from mail-wm1-x336.google.com (mail-wm1-x336.google.com\n\t[IPv6:2a00:1450:4864:20::336])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 74A7762916\n\tfor ;\n\tFri, 15 Sep 2023 17:58:56 +0200 (CEST)","by mail-wm1-x336.google.com with SMTP id\n\t5b1f17b1804b1-403012f276dso25196005e9.0\n\tfor ;\n\tFri, 15 Sep 2023 08:58:56 -0700 (PDT)","from localhost.localdomain ([195.180.61.40])\n\tby smtp.gmail.com with ESMTPSA id\n\tl13-20020a7bc44d000000b00401c595fcc7sm7810618wmi.11.2023.09.15.08.58.54\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 15 Sep 2023 08:58:55 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1694793539;\n\tbh=Eo/H8q5i3+FT5oPUdT+DXN8pB6mxy+TQNhCh9E9Fw0k=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=L0xe4GKgpiCl4KHX/1ozarwgQtsm+z4a/rEtv6mCOq1N3b74HT0cyAHHBDNxBeBJ/\n\tMzR1IJWZa4+mFMzdU0/8gtn3El/HrrELtTSx3qe0ppdSNt+gz65/IiW38IPpi7uWCe\n\t3HO8f0YuYbclsS4k87YneIK7o7IzkL3YzFbC3D29anc3zfK6Nm0vvkcSEVl5VH8OLm\n\tuoA8EnM6BZ1kDqlk9pwW2bXSRCv2ohQmCgIhRcVLo/zphW/bFLqaiyYiixPmuyt/Mu\n\tx58tzuqwe6hdhgylKdUtp8nzPIOuR8Ia3wPqCmuTaIqm3v42rKcNFWfWAWBKlTMHYf\n\t0Cdry3XaomF+g==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1694793536; x=1695398336;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=CWSOLadTJtKvYCIdSC1vIHqtxyxalOIgM1mgM1CtEyU=;\n\tb=necXUJas2zjLc9ffC2RyvFtrPr7wzxrG+zssWAlo8v7q/hp6Il4LVsFPlvlzEE9fGy\n\tojxpw3fjK480ki/w/+EOdBZ2dOmSWsEPgp6kIk3bqCDYxq0O52H4Dm9DJDB9aIO/IZA+\n\tE0Cy1H6E2If3zYsmGzkg9VpjGnbVMeqcVlfOTKyEEzKXpxin8ozj7uHKaTtXQsI2hPI6\n\tP8wluqVjG5YI3cK1oADtpnWO7XSkcIiTtILO5EeS3nbUMSvXVCPQiDjlvwcRhp5G4gdS\n\t20oLZh8eFe6fEaPukZHaMrmu1ZtCPU0xBDdg7mKI5cxmkSEPIUetiW22ddLsyfFT9Hp2\n\tO8UA=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"necXUJas\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1694793536; x=1695398336;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=CWSOLadTJtKvYCIdSC1vIHqtxyxalOIgM1mgM1CtEyU=;\n\tb=Y3v3DaUwvkCqp04RcVBqz/rF9uO50eWPpOKLXsVXtUX4O5FvYOFRt2cEffB+fO7wZh\n\t+Bg+8gE2LPqspAK4JHdwQGWTktYXMbAt6jUiiTvDTMZofUIYjho+6TZ0Sz2RVpTrG/6c\n\t5+pdMIXYza4WnuEUr+6LKl+jOp7RbdKNZaNpaBYwlJjG2retfoTttcesWJMQn15420fF\n\t3h7yxXGHwqq1uKlAC886XOr7+nPJOVluucGWqhoLB7C9imQF+ondkb/BZsXfytu0mKzL\n\tRPdWzy47I16mwnK/l5FRTbuzx3BCewjZKIO5dl3ZD3RmdDvG75hazb5Jp58qBxKPANoM\n\t2j8A==","X-Gm-Message-State":"AOJu0YyeMShsxp2XMhHzDZMBPPtnXkI+QZRwweXxdJnrrOyfra4lEEx/\n\thE6ZtA1AMeLx4xlhfp1/vYTZLas3bwYKeSSVHhI=","X-Google-Smtp-Source":"AGHT+IGKSnYOg6OYaFozE1b/GNHwX42q7UjAGTjEM0ZApSFAWbe0g6XmZA4pD7nSYfYn77KVX3R7QQ==","X-Received":"by 2002:a05:600c:2209:b0:404:746e:c601 with SMTP id\n\tz9-20020a05600c220900b00404746ec601mr2104418wml.9.1694793535672; \n\tFri, 15 Sep 2023 08:58:55 -0700 (PDT)","To":"libcamera-devel@lists.libcamera.org","Date":"Fri, 15 Sep 2023 16:58:44 +0100","Message-Id":"<20230915155844.2388-6-david.plowman@raspberrypi.com>","X-Mailer":"git-send-email 2.30.2","In-Reply-To":"<20230915155844.2388-1-david.plowman@raspberrypi.com>","References":"<20230915155844.2388-1-david.plowman@raspberrypi.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v5 5/5] ipa: rpi: agc: Use channel\n\tconstraints in the AGC algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"Jacopo Mondi ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Whenever we run Agc::process(), we store the most recent total\nexposure requested for each channel.\n\nWith these values we can apply the channel constraints after\ntime-filtering the requested total exposure, but before working out\nhow much digital gain is needed.\n\nSigned-off-by: David Plowman \nReviewed-by: Naushir Patuck \nReviewed-by: Jacopo Mondi \n---\n src/ipa/rpi/controller/rpi/agc.cpp | 22 ++++++--\n src/ipa/rpi/controller/rpi/agc.h | 1 +\n src/ipa/rpi/controller/rpi/agc_channel.cpp | 58 +++++++++++++++++++---\n src/ipa/rpi/controller/rpi/agc_channel.h | 8 ++-\n 4 files changed, 75 insertions(+), 14 deletions(-)","diff":"diff --git a/src/ipa/rpi/controller/rpi/agc.cpp b/src/ipa/rpi/controller/rpi/agc.cpp\nindex 8b9d2026..870cb431 100644\n--- a/src/ipa/rpi/controller/rpi/agc.cpp\n+++ b/src/ipa/rpi/controller/rpi/agc.cpp\n@@ -39,6 +39,7 @@ int Agc::read(const libcamera::YamlObject ¶ms)\n \t */\n \tif (!params.contains(\"channels\")) {\n \t\tLOG(RPiAgc, Debug) << \"Single channel only\";\n+\t\tchannelTotalExposures_.resize(1, 0s);\n \t\tchannelData_.emplace_back();\n \t\treturn channelData_.back().channel.read(params, getHardwareConfig());\n \t}\n@@ -58,6 +59,8 @@ int Agc::read(const libcamera::YamlObject ¶ms)\n \t\treturn -1;\n \t}\n \n+\tchannelTotalExposures_.resize(channelData_.size(), 0s);\n+\n \treturn 0;\n }\n \n@@ -236,16 +239,22 @@ static void getDelayedChannelIndex(Metadata *metadata, const char *message, unsi\n \t}\n }\n \n-static void setCurrentChannelIndex(Metadata *metadata, const char *message, unsigned int channelIndex)\n+static libcamera::utils::Duration\n+setCurrentChannelIndexGetExposure(Metadata *metadata, const char *message, unsigned int channelIndex)\n {\n \tstd::unique_lock lock(*metadata);\n \tAgcStatus *status = metadata->getLocked(\"agc.status\");\n-\tif (status)\n+\tlibcamera::utils::Duration dur = 0s;\n+\n+\tif (status) {\n \t\tstatus->channel = channelIndex;\n-\telse {\n+\t\tdur = status->totalExposureValue;\n+\t} else {\n \t\t/* This does happen at startup, otherwise it would be a Warning or Error. */\n \t\tLOG(RPiAgc, Debug) << message;\n \t}\n+\n+\treturn dur;\n }\n \n void Agc::prepare(Metadata *imageMetadata)\n@@ -310,8 +319,11 @@ void Agc::process(StatisticsPtr &stats, Metadata *imageMetadata)\n \t\tLOG(RPiAgc, Debug) << \"process: channel \" << channelIndex << \" not seen yet\";\n \t}\n \n-\tchannelData.channel.process(stats, deviceStatus, imageMetadata);\n-\tsetCurrentChannelIndex(imageMetadata, \"process: no AGC status found\", channelIndex);\n+\tchannelData.channel.process(stats, deviceStatus, imageMetadata, channelTotalExposures_);\n+\tauto dur = setCurrentChannelIndexGetExposure(imageMetadata, \"process: no AGC status found\",\n+\t\t\t\t\t\t channelIndex);\n+\tif (dur)\n+\t\tchannelTotalExposures_[channelIndex] = dur;\n \n \t/* And onto the next channel for the next call. */\n \tindex_ = (index_ + 1) % activeChannels_.size();\ndiff --git a/src/ipa/rpi/controller/rpi/agc.h b/src/ipa/rpi/controller/rpi/agc.h\nindex ee85c693..90890439 100644\n--- a/src/ipa/rpi/controller/rpi/agc.h\n+++ b/src/ipa/rpi/controller/rpi/agc.h\n@@ -55,6 +55,7 @@ private:\n \tstd::vector channelData_;\n \tstd::vector activeChannels_;\n \tunsigned int index_; /* index into the activeChannels_ */\n+\tAgcChannelTotalExposures channelTotalExposures_;\n };\n \n } /* namespace RPiController */\ndiff --git a/src/ipa/rpi/controller/rpi/agc_channel.cpp b/src/ipa/rpi/controller/rpi/agc_channel.cpp\nindex dfe363ff..3957dbc3 100644\n--- a/src/ipa/rpi/controller/rpi/agc_channel.cpp\n+++ b/src/ipa/rpi/controller/rpi/agc_channel.cpp\n@@ -493,7 +493,9 @@ void AgcChannel::prepare(Metadata *imageMetadata)\n \t}\n }\n \n-void AgcChannel::process(StatisticsPtr &stats, DeviceStatus const &deviceStatus, Metadata *imageMetadata)\n+void AgcChannel::process(StatisticsPtr &stats, DeviceStatus const &deviceStatus,\n+\t\t\t Metadata *imageMetadata,\n+\t\t\t const AgcChannelTotalExposures &channelTotalExposures)\n {\n \tframeCount_++;\n \t/*\n@@ -512,12 +514,17 @@ void AgcChannel::process(StatisticsPtr &stats, DeviceStatus const &deviceStatus,\n \tcomputeTargetExposure(gain);\n \t/* The results have to be filtered so as not to change too rapidly. */\n \tfilterExposure();\n+\t/*\n+\t * We may be asked to limit the exposure using other channels. If another channel\n+\t * determines our upper bound we may want to know this later.\n+\t */\n+\tbool channelBound = applyChannelConstraints(channelTotalExposures);\n \t/*\n \t * Some of the exposure has to be applied as digital gain, so work out\n-\t * what that is. This function also tells us whether it's decided to\n-\t * \"desaturate\" the image more quickly.\n+\t * what that is. It also tells us whether it's trying to desaturate the image\n+\t * more quickly, which can only happen when another channel is not limiting us.\n \t */\n-\tbool desaturate = applyDigitalGain(gain, targetY);\n+\tbool desaturate = applyDigitalGain(gain, targetY, channelBound);\n \t/*\n \t * The last thing is to divide up the exposure value into a shutter time\n \t * and analogue gain, according to the current exposure mode.\n@@ -794,7 +801,44 @@ void AgcChannel::computeTargetExposure(double gain)\n \tLOG(RPiAgc, Debug) << \"Target totalExposure \" << target_.totalExposure;\n }\n \n-bool AgcChannel::applyDigitalGain(double gain, double targetY)\n+bool AgcChannel::applyChannelConstraints(const AgcChannelTotalExposures &channelTotalExposures)\n+{\n+\tbool channelBound = false;\n+\tLOG(RPiAgc, Debug)\n+\t\t<< \"Total exposure before channel constraints \" << filtered_.totalExposure;\n+\n+\tfor (const auto &constraint : config_.channelConstraints) {\n+\t\tLOG(RPiAgc, Debug)\n+\t\t\t<< \"Check constraint: channel \" << constraint.channel << \" bound \"\n+\t\t\t<< (constraint.bound == AgcChannelConstraint::Bound::UPPER ? \"UPPER\" : \"LOWER\")\n+\t\t\t<< \" factor \" << constraint.factor;\n+\t\tif (constraint.channel >= channelTotalExposures.size() ||\n+\t\t !channelTotalExposures[constraint.channel]) {\n+\t\t\tLOG(RPiAgc, Debug) << \"no such channel or no exposure available- skipped\";\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tlibcamera::utils::Duration limitExposure =\n+\t\t\tchannelTotalExposures[constraint.channel] * constraint.factor;\n+\t\tLOG(RPiAgc, Debug) << \"Limit exposure \" << limitExposure;\n+\t\tif ((constraint.bound == AgcChannelConstraint::Bound::UPPER &&\n+\t\t filtered_.totalExposure > limitExposure) ||\n+\t\t (constraint.bound == AgcChannelConstraint::Bound::LOWER &&\n+\t\t filtered_.totalExposure < limitExposure)) {\n+\t\t\tfiltered_.totalExposure = limitExposure;\n+\t\t\tLOG(RPiAgc, Debug) << \"Constraint applies\";\n+\t\t\tchannelBound = true;\n+\t\t} else\n+\t\t\tLOG(RPiAgc, Debug) << \"Constraint does not apply\";\n+\t}\n+\n+\tLOG(RPiAgc, Debug)\n+\t\t<< \"Total exposure after channel constraints \" << filtered_.totalExposure;\n+\n+\treturn channelBound;\n+}\n+\n+bool AgcChannel::applyDigitalGain(double gain, double targetY, bool channelBound)\n {\n \tdouble minColourGain = std::min({ awb_.gainR, awb_.gainG, awb_.gainB, 1.0 });\n \tASSERT(minColourGain != 0.0);\n@@ -814,8 +858,8 @@ bool AgcChannel::applyDigitalGain(double gain, double targetY)\n \t * quickly (and we then approach the correct value more quickly from\n \t * below).\n \t */\n-\tbool desaturate = targetY > config_.fastReduceThreshold &&\n-\t\t\t gain < sqrt(targetY);\n+\tbool desaturate = !channelBound &&\n+\t\t\t targetY > config_.fastReduceThreshold && gain < sqrt(targetY);\n \tif (desaturate)\n \t\tdg /= config_.fastReduceThreshold;\n \tLOG(RPiAgc, Debug) << \"Digital gain \" << dg << \" desaturate? \" << desaturate;\ndiff --git a/src/ipa/rpi/controller/rpi/agc_channel.h b/src/ipa/rpi/controller/rpi/agc_channel.h\nindex ceaab12f..ae826fa8 100644\n--- a/src/ipa/rpi/controller/rpi/agc_channel.h\n+++ b/src/ipa/rpi/controller/rpi/agc_channel.h\n@@ -21,6 +21,8 @@\n \n namespace RPiController {\n \n+using AgcChannelTotalExposures = std::vector;\n+\n struct AgcMeteringMode {\n \tstd::vector weights;\n \tint read(const libcamera::YamlObject ¶ms);\n@@ -95,7 +97,8 @@ public:\n \tvoid disableAuto();\n \tvoid switchMode(CameraMode const &cameraMode, Metadata *metadata);\n \tvoid prepare(Metadata *imageMetadata);\n-\tvoid process(StatisticsPtr &stats, DeviceStatus const &deviceStatus, Metadata *imageMetadata);\n+\tvoid process(StatisticsPtr &stats, DeviceStatus const &deviceStatus, Metadata *imageMetadata,\n+\t\t const AgcChannelTotalExposures &channelTotalExposures);\n \n private:\n \tbool updateLockStatus(DeviceStatus const &deviceStatus);\n@@ -107,7 +110,8 @@ private:\n \t\t\t double &gain, double &targetY);\n \tvoid computeTargetExposure(double gain);\n \tvoid filterExposure();\n-\tbool applyDigitalGain(double gain, double targetY);\n+\tbool applyChannelConstraints(const AgcChannelTotalExposures &channelTotalExposures);\n+\tbool applyDigitalGain(double gain, double targetY, bool channelBound);\n \tvoid divideUpExposure();\n \tvoid writeAndFinish(Metadata *imageMetadata, bool desaturate);\n \tlibcamera::utils::Duration limitShutter(libcamera::utils::Duration shutter);\n","prefixes":["libcamera-devel","v5","5/5"]}