From patchwork Mon Jul 31 09:46:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 18909 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BDD3FBDB13 for ; Mon, 31 Jul 2023 09:46:54 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 59DFD627F5; Mon, 31 Jul 2023 11:46:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1690796814; bh=/uTwRHVyrFT2HTsBf8wu4vdEWf1dVb1Bcnb02Sh6A5o=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=yD74CfNRbAW6vXPAbdK6Cm0h1faX8gi7EmFkKU1x+h5Cn7VXL1EOzpp7MnIFS//jM +tIq/orQxiTB1cCTYUSgvefQqPZf20ZqXiEMq3p+HBehxLo3fQ6YOwjRr4QT8rdGpW WQGhBPfJUh3bmu84Rh/lDmluMiM9xB3O7R8TgNwBFW7HTGdBERPLgIcLLQ2TOxou+9 OuXwMlUepwJ9FhJiThDusPqTNE1C5HNzuqtNncsGk5ujqNpOQTY2P8W/uHPugrBQWt kWyF1o/VptwO2Tp8rBnkgGH78gdI5NhDH9kD2KCL4syBxBGFAhEQR4muE3BZKC3oI/ D0v5M4qWV82tA== Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7FB4C627F4 for ; Mon, 31 Jul 2023 11:46:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="p3QU2A9K"; dkim-atps=neutral Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-3fbc63c2e84so48072575e9.3 for ; Mon, 31 Jul 2023 02:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1690796808; x=1691401608; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4rXIM8Dmbucq8jQtDzcDP9vwhEVUIMbE0QaT2+4DNc4=; b=p3QU2A9KbeSr6QkEWyNc2YMZktFrGADj36Et2ZP9Mka7wPKvkhUWSaDzDH+bE14S4l QL6oaWwfPhWmH8w8SWW5dlmlXnkRFZ9g8TSnsuzqtpwcDBheW18S6k3Oj/KVTRBEgtUb ZTxcvojbcs3xHg8o85EIqo6C7TvHA4eZrOLG1O9SfBnUwAURw+WtqcjIKeT3q+D47X42 RdAh41XGZgolcDg4ev3yb3WzDsnfsUB11iC7fmTljTCmEWcmzfu7aF8vSJmZ0p6Q/B1M o/qvQQAA6ZS8SBtxTC7Ni3kY5RDrt3Fim3D3iGV4PEsWRKKSIyTiYL27gunKFLaYiPUC AMsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690796808; x=1691401608; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4rXIM8Dmbucq8jQtDzcDP9vwhEVUIMbE0QaT2+4DNc4=; b=VsHeVO/3kyXodFgybR5zaY+UPtFV8eZxG+5YZLPTv6BVB1XFgN4kmEczkojusSUIWw ly03/SonGZsU86VEBQmlCj7+mx922ZxXg3sozwp/10/MVq8UJm6EM3BhqSPEOPErO2AT qWkj0WEb/fYjBThLqBKGX/u3Udfx98SGhr0YdaSTFFVOFPquYmgRcbiUJ8E4TKcq60YG El4c+ceW1qQD/e/zfzAUBtnN3I3ePLTGGRgg0TwmmrZOWCY0J5NV8WU92HOoS0MSHBAP GPF4UXgOb61WkFMCUwLjE2EjS/KV7Mbz6aEZ22NQmoQraxxlSGlOHmkTfmyA9MaFnm5U 4y6g== X-Gm-Message-State: ABy/qLabunypinXJicO0KAH/7WrwOU4hQ2yZPlFx944IKhYrOE4Vba3F dFksrW5ekKk6vdpdRYuefSRbdREv5HP109cirdk= X-Google-Smtp-Source: APBJJlHgTehi+Q+xi1vtwdEQPJBcl1MwlPNr9OJfrlRNwva6Z0veOvlS0iFWhCgp5mMSorXMpjsRhA== X-Received: by 2002:a1c:e915:0:b0:3fc:2e8:ea8b with SMTP id q21-20020a1ce915000000b003fc02e8ea8bmr7767293wmc.28.1690796807862; Mon, 31 Jul 2023 02:46:47 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:2bce:64d6:1a5c:49a2]) by smtp.gmail.com with ESMTPSA id 9-20020a05600c240900b003fa98908014sm13612838wmp.8.2023.07.31.02.46.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Jul 2023 02:46:47 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Mon, 31 Jul 2023 10:46:41 +0100 Message-Id: <20230731094641.73646-6-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230731094641.73646-1-david.plowman@raspberrypi.com> References: <20230731094641.73646-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 5/5] ipa: rpi: agc: Use channel constraints in the AGC algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: David Plowman via libcamera-devel From: David Plowman Reply-To: David Plowman Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Whenever we run Agc::process(), we store the most recent total exposure requested for each channel. With these values we can apply the channel constraints after time-filtering the requested total exposure, but before working out how much digital gain is needed. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck --- src/ipa/rpi/controller/rpi/agc.cpp | 21 ++++-- src/ipa/rpi/controller/rpi/agc.h | 1 + src/ipa/rpi/controller/rpi/agc_channel.cpp | 76 ++++++++++++++++++---- src/ipa/rpi/controller/rpi/agc_channel.h | 8 ++- src/ipa/rpi/vc4/data/imx477.json | 3 +- 5 files changed, 90 insertions(+), 19 deletions(-) diff --git a/src/ipa/rpi/controller/rpi/agc.cpp b/src/ipa/rpi/controller/rpi/agc.cpp index 7e627bba..7077fbff 100644 --- a/src/ipa/rpi/controller/rpi/agc.cpp +++ b/src/ipa/rpi/controller/rpi/agc.cpp @@ -40,6 +40,7 @@ int Agc::read(const libcamera::YamlObject ¶ms) */ if (!params.contains("channels")) { LOG(RPiAgc, Debug) << "Single channel only"; + channelResults_.resize(1, 0s); channelData_.emplace_back(); return channelData_.back().channel.read(params, getHardwareConfig()); } @@ -59,6 +60,8 @@ int Agc::read(const libcamera::YamlObject ¶ms) return -1; } + channelResults_.resize(channelData_.size(), 0s); + return 0; } @@ -234,15 +237,21 @@ static void getChannelIndex(Metadata *metadata, const char *message, unsigned in LOG(RPiAgc, Debug) << message; } -static void setChannelIndex(Metadata *metadata, const char *message, unsigned int channelIndex) +static libcamera::utils::Duration +setChannelIndex(Metadata *metadata, const char *message, unsigned int channelIndex) { std::unique_lock lock(*metadata); AgcStatus *status = metadata->getLocked("agc.status"); - if (status) + libcamera::utils::Duration dur = 0s; + + if (status) { status->channel = channelIndex; - else + dur = status->totalExposureValue; + } else /* This does happen at startup, otherwise it would be a Warning or Error. */ LOG(RPiAgc, Debug) << message; + + return dur; } void Agc::prepare(Metadata *imageMetadata) @@ -306,8 +315,10 @@ void Agc::process(StatisticsPtr &stats, Metadata *imageMetadata) /* Can also happen when new channels start. */ LOG(RPiAgc, Debug) << "process: channel " << channelIndex << " not seen yet"; - channelData.channel.process(stats, &deviceStatus, imageMetadata); - setChannelIndex(imageMetadata, "process: no AGC status found", channelIndex); + channelData.channel.process(stats, &deviceStatus, imageMetadata, channelResults_); + auto dur = setChannelIndex(imageMetadata, "process: no AGC status found", channelIndex); + if (dur) + channelResults_[channelIndex] = dur; /* And onto the next channel for the next call. */ index_ = (index_ + 1) % activeChannels_.size(); diff --git a/src/ipa/rpi/controller/rpi/agc.h b/src/ipa/rpi/controller/rpi/agc.h index 2eed2bab..acd6dc2f 100644 --- a/src/ipa/rpi/controller/rpi/agc.h +++ b/src/ipa/rpi/controller/rpi/agc.h @@ -54,6 +54,7 @@ private: std::vector channelData_; std::vector activeChannels_; unsigned int index_; /* index into the activeChannels_ */ + AgcChannelResults channelResults_; }; } /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/rpi/agc_channel.cpp b/src/ipa/rpi/controller/rpi/agc_channel.cpp index ed8a3b30..4e60802c 100644 --- a/src/ipa/rpi/controller/rpi/agc_channel.cpp +++ b/src/ipa/rpi/controller/rpi/agc_channel.cpp @@ -175,6 +175,11 @@ readConstraintModes(std::map &constraintModes, int AgcChannelConstraint::read(const libcamera::YamlObject ¶ms) { + auto channelValue = params["channel"].get(); + if (!channelValue) + return -EINVAL; + channel = *channelValue; + std::string boundString = params["bound"].get(""); transform(boundString.begin(), boundString.end(), boundString.begin(), ::toupper); @@ -184,15 +189,15 @@ int AgcChannelConstraint::read(const libcamera::YamlObject ¶ms) } bound = boundString == "UPPER" ? Bound::UPPER : Bound::LOWER; - auto value = params["factor"].get(); - if (!value) + auto factorValue = params["factor"].get(); + if (!factorValue) return -EINVAL; - factor = *value; + factor = *factorValue; return 0; } -int readChannelConstraints(std::vector channelConstraints, +int readChannelConstraints(std::vector &channelConstraints, const libcamera::YamlObject ¶ms) { int ret = 0; @@ -231,6 +236,7 @@ int AgcConfig::read(const libcamera::YamlObject ¶ms) ret = readChannelConstraints(channelConstraints, params["channel_constraints"]); if (ret) return ret; + LOG(RPiAgc, Info) << "Read " << channelConstraints.size() << " channel constraints"; } ret = yTarget.read(params["y_target"]); @@ -489,7 +495,8 @@ void AgcChannel::prepare(Metadata *imageMetadata) } } -void AgcChannel::process(StatisticsPtr &stats, const DeviceStatus *deviceStatus, Metadata *imageMetadata) +void AgcChannel::process(StatisticsPtr &stats, const DeviceStatus *deviceStatus, + Metadata *imageMetadata, const AgcChannelResults &channelResults) { frameCount_++; /* @@ -508,12 +515,17 @@ void AgcChannel::process(StatisticsPtr &stats, const DeviceStatus *deviceStatus, computeTargetExposure(gain); /* The results have to be filtered so as not to change too rapidly. */ filterExposure(); + /* + * We may be asked to limit the exposure using other channels. If another channel + * determines our upper bound we may want to know this later. + */ + bool channelBound = applyChannelConstraints(channelResults); /* * Some of the exposure has to be applied as digital gain, so work out - * what that is. This function also tells us whether it's decided to - * "desaturate" the image more quickly. + * what that is. It also tells us whether it's trying to desaturate the image + * more quickly, which can only happen when another channel is not limiting us. */ - bool desaturate = applyDigitalGain(gain, targetY); + bool desaturate = applyDigitalGain(gain, targetY, channelBound); /* * The last thing is to divide up the exposure value into a shutter time * and analogue gain, according to the current exposure mode. @@ -792,7 +804,49 @@ void AgcChannel::computeTargetExposure(double gain) LOG(RPiAgc, Debug) << "Target totalExposure " << target_.totalExposure; } -bool AgcChannel::applyDigitalGain(double gain, double targetY) +bool AgcChannel::applyChannelConstraints(const AgcChannelResults &channelResults) +{ + bool channelBound = false; + LOG(RPiAgc, Debug) + << "Total exposure before channel constraints " << filtered_.totalExposure; + + for (const auto &constraint : config_.channelConstraints) { + LOG(RPiAgc, Debug) + << "Check constraint: channel " << constraint.channel << " bound " + << (constraint.bound == AgcChannelConstraint::Bound::UPPER ? "UPPER" : "LOWER") + << " factor " << constraint.factor; + if (constraint.channel >= channelResults.size() || + !channelResults[constraint.channel]) { + LOG(RPiAgc, Debug) << "no such channel - skipped"; + continue; + } + + if (channelResults[constraint.channel] == 0s) { + LOG(RPiAgc, Debug) << "zero exposure - skipped"; + continue; + } + + libcamera::utils::Duration limitExposure = + channelResults[constraint.channel] * constraint.factor; + LOG(RPiAgc, Debug) << "Limit exposure " << limitExposure; + if ((constraint.bound == AgcChannelConstraint::Bound::UPPER && + filtered_.totalExposure > limitExposure) || + (constraint.bound == AgcChannelConstraint::Bound::LOWER && + filtered_.totalExposure < limitExposure)) { + filtered_.totalExposure = limitExposure; + LOG(RPiAgc, Debug) << "Applies"; + channelBound = true; + } else + LOG(RPiAgc, Debug) << "Does not apply"; + } + + LOG(RPiAgc, Debug) + << "Total exposure after channel constraints " << filtered_.totalExposure; + + return channelBound; +} + +bool AgcChannel::applyDigitalGain(double gain, double targetY, bool channelBound) { double minColourGain = std::min({ awb_.gainR, awb_.gainG, awb_.gainB, 1.0 }); ASSERT(minColourGain != 0.0); @@ -812,8 +866,8 @@ bool AgcChannel::applyDigitalGain(double gain, double targetY) * quickly (and we then approach the correct value more quickly from * below). */ - bool desaturate = targetY > config_.fastReduceThreshold && - gain < sqrt(targetY); + bool desaturate = !channelBound && + targetY > config_.fastReduceThreshold && gain < sqrt(targetY); if (desaturate) dg /= config_.fastReduceThreshold; LOG(RPiAgc, Debug) << "Digital gain " << dg << " desaturate? " << desaturate; diff --git a/src/ipa/rpi/controller/rpi/agc_channel.h b/src/ipa/rpi/controller/rpi/agc_channel.h index 446125ef..7ce34360 100644 --- a/src/ipa/rpi/controller/rpi/agc_channel.h +++ b/src/ipa/rpi/controller/rpi/agc_channel.h @@ -19,6 +19,8 @@ namespace RPiController { +using AgcChannelResults = std::vector; + struct AgcMeteringMode { std::vector weights; int read(const libcamera::YamlObject ¶ms); @@ -93,7 +95,8 @@ public: void disableAuto(); void switchMode(CameraMode const &cameraMode, Metadata *metadata); void prepare(Metadata *imageMetadata); - void process(StatisticsPtr &stats, const DeviceStatus *deviceStatus, Metadata *imageMetadata); + void process(StatisticsPtr &stats, const DeviceStatus *deviceStatus, Metadata *imageMetadata, + const AgcChannelResults &channelResults); private: bool updateLockStatus(DeviceStatus const &deviceStatus); @@ -105,7 +108,8 @@ private: double &gain, double &targetY); void computeTargetExposure(double gain); void filterExposure(); - bool applyDigitalGain(double gain, double targetY); + bool applyChannelConstraints(const AgcChannelResults &channelResults); + bool applyDigitalGain(double gain, double targetY, bool channelBound); void divideUpExposure(); void writeAndFinish(Metadata *imageMetadata, bool desaturate); libcamera::utils::Duration limitShutter(libcamera::utils::Duration shutter); diff --git a/src/ipa/rpi/vc4/data/imx477.json b/src/ipa/rpi/vc4/data/imx477.json index daffc268..bbe6da0d 100644 --- a/src/ipa/rpi/vc4/data/imx477.json +++ b/src/ipa/rpi/vc4/data/imx477.json @@ -515,4 +515,5 @@ "rpi.sharpen": { } } ] -} \ No newline at end of file +} +