From patchwork Fri Jun 14 07:42:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 20310 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 08A32C3237 for ; Fri, 14 Jun 2024 07:42:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AB93365490; Fri, 14 Jun 2024 09:42:30 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="vb4jH4t/"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EA4F065489 for ; Fri, 14 Jun 2024 09:42:28 +0200 (CEST) Received: from neptunite.hamster-moth.ts.net (h175-177-049-156.catv02.itscom.jp [175.177.49.156]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C10DA10C4; Fri, 14 Jun 2024 09:42:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1718350934; bh=C6ldniQmD5q1I7bDiZm2NJLLxayQYyaP6l57y/MXzww=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vb4jH4t/elYDc7AYoXrlsSx0k7WWRwAAOlxywUMVpm4SlWBzrXjSOey3qsAvq9iPe 4evM8TQ9SzKeuFA2uTsMyl1SsO/OLKidWhEptkIHcL88+ARL2QJ3vJtANJ3nGcqw40 58E4p8HY914HFi3rPIiecks6EhyrG3Zd0aDo7I9Q= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , Stefan Klug , Daniel Scally Subject: [PATCH v6 1/2] ipa: rkisp1: agc: Read histogram weights from tuning file Date: Fri, 14 Jun 2024 16:42:13 +0900 Message-Id: <20240614074214.3600996-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240614074214.3600996-1-paul.elder@ideasonboard.com> References: <20240614074214.3600996-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add support to the rkisp1 AGC to read histogram weights from the tuning file. As controls for selecting the metering mode are not yet supported, for now hardcode the matrix metering mode, which is the same as what the AGC previously hardcoded. Signed-off-by: Paul Elder Reviewed-by: Stefan Klug Reviewed-by: Daniel Scally --- Changes in v6: - fix predivider calculation - misc cleanups - remove support for v12 (technically it depends on what the tuning file provides) No change in v5 Changes in v4: - add debug print when parsing the yaml file Changes in v3: - support the new tuning file layout for interpolated matrices - support both v10 and v12 Changes in v2: - add default metering mode if none are read successfully from the tuning file - compute the predivider instead of using a table --- src/ipa/rkisp1/algorithms/agc.cpp | 100 +++++++++++++++++++++++++++++- src/ipa/rkisp1/algorithms/agc.h | 6 ++ 2 files changed, 103 insertions(+), 3 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 50e0690fe146..5a9c6be118d9 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -17,6 +17,8 @@ #include #include +#include "libcamera/internal/yaml_parser.h" + #include "libipa/histogram.h" /** @@ -36,6 +38,85 @@ namespace ipa::rkisp1::algorithms { LOG_DEFINE_CATEGORY(RkISP1Agc) +int Agc::parseMeteringModes(IPAContext &context, const YamlObject &tuningData) +{ + if (!tuningData.isDictionary()) { + LOG(RkISP1Agc, Error) + << "'AeMeteringMode' parameter not found in tuning file"; + return -EINVAL; + } + + for (const auto &[key, value] : tuningData.asDict()) { + if (controls::AeMeteringModeNameValueMap.find(key) == + controls::AeMeteringModeNameValueMap.end()) { + LOG(RkISP1Agc, Warning) + << "Skipping unknown metering mode '" << key << "'"; + continue; + } + + std::vector weights = + value.getList().value_or(std::vector{}); + if (weights.size() != context.hw->numHistogramWeights) { + LOG(RkISP1Agc, Warning) + << "Failed to read metering mode'" << key << "'"; + continue; + } + + meteringModes_[controls::AeMeteringModeNameValueMap.at(key)] = weights; + } + + if (meteringModes_.empty()) { + LOG(RkISP1Agc, Warning) + << "No metering modes read from tuning file; defaulting to matrix"; + int32_t meteringModeId = controls::AeMeteringModeNameValueMap.at("MeteringMatrix"); + std::vector weights(context.hw->numHistogramWeights, 1); + + meteringModes_[meteringModeId] = weights; + } + + return 0; +} + +uint8_t Agc::computeHistogramPredivider(Size &size, enum rkisp1_cif_isp_histogram_mode mode) +{ + /* + * The maximum number of pixels that could potentially be in one bin is + * if all the pixels of the image are in it, multiplied by 3 for the + * three color channels. The counter for each bin is 16 bits wide, so + * `factor` thus contains the number of times we'd wrap around. This is + * obviously the number of pixels that we need to skip to make sure + * that we don't wrap around, but we compute the square root of it + * instead, as the skip that we need to program is for both the x and y + * directions. + * + * Even though it looks like dividing into a counter of 65536 would + * overflow by 1, this is apparently fine according to the hardware + * documentation, and this successfully gets the expected documented + * predivider size for cases where: + * (width / predivider) * (height / predivider) * 3 == 65536. + * + * There's a bit of extra rounding math to make sure the rounding goes + * the correct direction so that the square of the step is big enough + * to encompass the `factor` number of pixels that we need to skip. + * + * \todo Take into account weights. That is, if the weights are low + * enough we can potentially reduce the predivider to increase + * precision. This needs some investigation however, as this hardware + * behavior is undocumented and is only an educated guess. + */ + int count = mode == RKISP1_CIF_ISP_HISTOGRAM_MODE_RGB_COMBINED ? 3 : 1; + double factor = size.width * size.height * count / 65536.0; + double root = std::sqrt(factor); + uint8_t predivider; + + if (std::pow(std::floor(root), 2) < factor) + predivider = static_cast(std::ceil(root)); + else + predivider = static_cast(std::floor(root)); + + return std::clamp(predivider, 3, 127); +} + Agc::Agc() { supportsRaw_ = true; @@ -59,6 +140,11 @@ int Agc::init(IPAContext &context, const YamlObject &tuningData) if (ret) return ret; + const YamlObject &yamlMeteringModes = tuningData["AeMeteringMode"]; + ret = parseMeteringModes(context, yamlMeteringModes); + if (ret) + return ret; + context.ctrlMap.merge(controls()); return 0; @@ -160,6 +246,7 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, frameContext.agc.gain = context.activeState.agc.automatic.gain; } + /* \todo Remove this when we can set the below with controls */ if (frame > 0) return; @@ -178,14 +265,21 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, params->meas.hst_config.meas_window = context.configuration.agc.measureWindow; /* Produce the luminance histogram. */ params->meas.hst_config.mode = RKISP1_CIF_ISP_HISTOGRAM_MODE_Y_HISTOGRAM; + /* Set an average weighted histogram. */ Span weights{ params->meas.hst_config.hist_weight, context.hw->numHistogramWeights }; - std::fill(weights.begin(), weights.end(), 1); - /* Step size can't be less than 3. */ - params->meas.hst_config.histogram_predivider = 4; + /* \todo Get this from control */ + std::vector &modeWeights = meteringModes_.at(controls::MeteringMatrix); + std::copy(modeWeights.begin(), modeWeights.end(), weights.begin()); + + struct rkisp1_cif_isp_window window = params->meas.hst_config.meas_window; + Size windowSize = { window.h_size, window.v_size }; + params->meas.hst_config.histogram_predivider = + computeHistogramPredivider(windowSize, + static_cast(params->meas.hst_config.mode)); /* Update the configuration for histogram. */ params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_HST; diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index 04b3247e1276..996fea71755c 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -44,11 +44,17 @@ public: ControlList &metadata) override; private: + int parseMeteringModes(IPAContext &context, const YamlObject &tuningData); + uint8_t computeHistogramPredivider(Size &size, + enum rkisp1_cif_isp_histogram_mode mode); + void fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata); double estimateLuminance(double gain) const override; Span expMeans_; + + std::map> meteringModes_; }; } /* namespace ipa::rkisp1::algorithms */ From patchwork Fri Jun 14 07:42:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 20311 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9DA51C3237 for ; Fri, 14 Jun 2024 07:42:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5D78565489; Fri, 14 Jun 2024 09:42:34 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="vC2gMbNx"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 076126548D for ; Fri, 14 Jun 2024 09:42:32 +0200 (CEST) Received: from neptunite.hamster-moth.ts.net (h175-177-049-156.catv02.itscom.jp [175.177.49.156]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 56ACF10C4; Fri, 14 Jun 2024 09:42:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1718350937; bh=+oRxh4XOdw5+JqwcGsdMmvKdL/4hppRKFcsHT22TKVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vC2gMbNxNkI3kEW+QCqlxJEW89T+5/5VCGBc143jdaUWFA4Tn7ub71DekzW+ge86N t44d0f+f6P2aHFhTsm1W4hceU41avT5/yjlEPfOVF0D1IAsmYi30RxiYXvRlXWPMbN ajMW1kSibB5WoFcFXCAsPfo3a3INyWzDdVPldpXs= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , Stefan Klug Subject: [PATCH v6 2/2] ipa: rkisp1: agc: Plumb mode-selection and frame duration controls Date: Fri, 14 Jun 2024 16:42:14 +0900 Message-Id: <20240614074214.3600996-3-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240614074214.3600996-1-paul.elder@ideasonboard.com> References: <20240614074214.3600996-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Plumb controls for setting metering mode, exposure mode, constraint mode, and frame duration limits. Also report them as available controls, as well as in metadata. While at it, add the missing #include for tuple, as a std::tie is used. Signed-off-by: Paul Elder Reviewed-by: Stefan Klug Reviewed-by: Kieran Bingham --- Changes in v6: - remove the extra Algorithm::controls_ - remove the intermediate variable for accumulating the metering modes - only update parameters when they are changed - change the types of the control modes in the IPA cotext from ints to enums - add the appropriate casts when used - initialize maxShutterSpeed at configure time to avoid the awkward extra check to ensure that it isn't zero - add a todo that FrameDurationLimits should be initialized in agc and not in the IPA No change in v5 No change in v4 Changes in v3: - prevent maxShutterSpeed (for setLimits) from choosing 0 shutter speed if one of the inputs std::min() is not set (this fixes an assertion failure in the exposure mode helper) Changes in v2: - add #include - don't overwrite metering/exposure/constraint mode controls to default if no control is supplied, and retain the old value instead (same for frame duration limits) --- src/ipa/rkisp1/algorithms/agc.cpp | 76 ++++++++++++++++++++++++++++--- src/ipa/rkisp1/ipa_context.h | 12 ++++- 2 files changed, 79 insertions(+), 9 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 5a9c6be118d9..8702145187c7 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -10,6 +10,8 @@ #include #include #include +#include +#include #include #include @@ -74,6 +76,13 @@ int Agc::parseMeteringModes(IPAContext &context, const YamlObject &tuningData) meteringModes_[meteringModeId] = weights; } + std::vector meteringModes; + std::vector meteringModeKeys = utils::map_keys(meteringModes_); + std::transform(meteringModeKeys.begin(), meteringModeKeys.end(), + std::back_inserter(meteringModes), + [](int x) { return ControlValue(x); }); + context.ctrlMap[&controls::AeMeteringMode] = ControlInfo(meteringModes); + return 0; } @@ -167,8 +176,19 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.activeState.agc.manual.exposure = context.activeState.agc.automatic.exposure; context.activeState.agc.autoEnabled = !context.configuration.raw; - context.activeState.agc.constraintMode = constraintModes().begin()->first; - context.activeState.agc.exposureMode = exposureModeHelpers().begin()->first; + context.activeState.agc.constraintMode = + static_cast(constraintModes().begin()->first); + context.activeState.agc.exposureMode = + static_cast(exposureModeHelpers().begin()->first); + context.activeState.agc.meteringMode = + static_cast(meteringModes_.begin()->first); + + /* + * \todo This should probably come from FrameDurationLimits instead, + * except it's computed in the IPA and not here so we'd have to + * recompute it. + */ + context.activeState.agc.maxShutterSpeed = context.configuration.sensor.maxShutterSpeed; /* * Define the measurement window for AGC as a centered rectangle @@ -179,7 +199,6 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.configuration.agc.measureWindow.h_size = 3 * configInfo.outputSize.width / 4; context.configuration.agc.measureWindow.v_size = 3 * configInfo.outputSize.height / 4; - /* \todo Run this again when FrameDurationLimits is passed in */ setLimits(context.configuration.sensor.minShutterSpeed, context.configuration.sensor.maxShutterSpeed, context.configuration.sensor.minAnalogueGain, @@ -233,6 +252,39 @@ void Agc::queueRequest(IPAContext &context, frameContext.agc.exposure = agc.manual.exposure; frameContext.agc.gain = agc.manual.gain; } + + const auto &meteringMode = controls.get(controls::AeMeteringMode); + if (meteringMode) { + frameContext.agc.update = agc.meteringMode != *meteringMode; + agc.meteringMode = + static_cast(*meteringMode); + } + frameContext.agc.meteringMode = agc.meteringMode; + + const auto &exposureMode = controls.get(controls::AeExposureMode); + if (exposureMode) { + frameContext.agc.update = agc.exposureMode != *exposureMode; + agc.exposureMode = + static_cast(*exposureMode); + } + frameContext.agc.exposureMode = agc.exposureMode; + + const auto &constraintMode = controls.get(controls::AeConstraintMode); + if (constraintMode) { + frameContext.agc.update = agc.constraintMode != *constraintMode; + agc.constraintMode = + static_cast(*constraintMode); + } + frameContext.agc.constraintMode = agc.constraintMode; + + const auto &frameDurationLimits = controls.get(controls::FrameDurationLimits); + if (frameDurationLimits) { + utils::Duration maxShutterSpeed = + std::chrono::milliseconds((*frameDurationLimits).back()); + frameContext.agc.update = agc.maxShutterSpeed != maxShutterSpeed; + agc.maxShutterSpeed = maxShutterSpeed; + } + frameContext.agc.maxShutterSpeed = agc.maxShutterSpeed; } /** @@ -246,8 +298,7 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, frameContext.agc.gain = context.activeState.agc.automatic.gain; } - /* \todo Remove this when we can set the below with controls */ - if (frame > 0) + if (frame > 0 && !frameContext.agc.update) return; /* Configure the measurement window. */ @@ -271,8 +322,7 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, params->meas.hst_config.hist_weight, context.hw->numHistogramWeights }; - /* \todo Get this from control */ - std::vector &modeWeights = meteringModes_.at(controls::MeteringMatrix); + std::vector &modeWeights = meteringModes_.at(frameContext.agc.meteringMode); std::copy(modeWeights.begin(), modeWeights.end(), weights.begin()); struct rkisp1_cif_isp_window window = params->meas.hst_config.meas_window; @@ -295,6 +345,7 @@ void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, * frameContext.sensor.exposure; metadata.set(controls::AnalogueGain, frameContext.sensor.gain); metadata.set(controls::ExposureTime, exposureTime.get()); + metadata.set(controls::AeEnable, frameContext.agc.autoEnabled); /* \todo Use VBlank value calculated from each frame exposure. */ uint32_t vTotal = context.configuration.sensor.size.height @@ -302,6 +353,10 @@ void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, utils::Duration frameDuration = context.configuration.sensor.lineDuration * vTotal; metadata.set(controls::FrameDuration, frameDuration.get()); + + metadata.set(controls::AeMeteringMode, frameContext.agc.meteringMode); + metadata.set(controls::AeExposureMode, frameContext.agc.exposureMode); + metadata.set(controls::AeConstraintMode, frameContext.agc.constraintMode); } /** @@ -376,6 +431,13 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [](uint32_t x) { return x >> 4; }); expMeans_ = { params->ae.exp_mean, context.hw->numAeCells }; + utils::Duration maxShutterSpeed = std::min(context.configuration.sensor.maxShutterSpeed, + frameContext.agc.maxShutterSpeed); + setLimits(context.configuration.sensor.minShutterSpeed, + maxShutterSpeed, + context.configuration.sensor.minAnalogueGain, + context.configuration.sensor.maxAnalogueGain); + /* * The Agc algorithm needs to know the effective exposure value that was * applied to the sensor when the statistics were collected. diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 2a994d81ae41..6022480d4fd2 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -12,6 +12,7 @@ #include +#include #include #include @@ -68,8 +69,10 @@ struct IPAActiveState { } automatic; bool autoEnabled; - uint32_t constraintMode; - uint32_t exposureMode; + controls::AeConstraintModeEnum constraintMode; + controls::AeExposureModeEnum exposureMode; + controls::AeMeteringModeEnum meteringMode; + utils::Duration maxShutterSpeed; } agc; struct { @@ -115,6 +118,11 @@ struct IPAFrameContext : public FrameContext { uint32_t exposure; double gain; bool autoEnabled; + controls::AeConstraintModeEnum constraintMode; + controls::AeExposureModeEnum exposureMode; + controls::AeMeteringModeEnum meteringMode; + utils::Duration maxShutterSpeed; + bool update; } agc; struct {