From patchwork Fri May 6 09:53:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15808 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DE669C3256 for ; Fri, 6 May 2022 09:53:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 95B0A65649; Fri, 6 May 2022 11:53:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1651830804; bh=AHE+nswpanWOjh8KZ4PctKywXT3tnbYVn08WqG0emfU=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=BnqE4RBfH24qcw25RLg/EKTqvD4QkU/N31xjB6BZkm1RGRxKOhucWjoBSGdDMxpoN eENRTwGszWLXvekb0JJAsng94OXO9jVzsBBmRhRt8x8lfuhvaXqTDWdjIzB/hP4EIy fHeJXdqXMqXwQT+uZijp9G9E+ScslXF69xMSn6ahDWYP8J4i2PcQr8fSjvvf+JgUKv Fvxa95FD1necGIywtKurtKlcSVA81GwiFmgVNd5UhIw1nBwbyZduTwyt4x9Q96v+BD 5J8BvOgyrbsMsZXLZpJTsWTSi7L5v51eA/Nt5u6X7Nh7xdL6EVSTOVjT0GW1CRburt Ucva+nQgcX8HA== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7B69660421 for ; Fri, 6 May 2022 11:53:23 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="oUiC77/G"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [103.251.226.91]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1C6B4487; Fri, 6 May 2022 11:53:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1651830803; bh=AHE+nswpanWOjh8KZ4PctKywXT3tnbYVn08WqG0emfU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oUiC77/GlcA49r9PnM6Di9jgg3cmUncdDxPV5953/1mCQVTLMr9wn7+neMtekl71E TbK43C7y79+VSuc3koW1BCJXKoVwWtOvOb4xR85E2A/4ws1uu+qvY4eS6tgXGJ6sKr B7vBLTSmNP7iyNEPeemBSjSnHzDtToQCjExwa08c= To: libcamera-devel@lists.libcamera.org Date: Fri, 6 May 2022 15:23:05 +0530 Message-Id: <20220506095307.78370-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220506095307.78370-1-umang.jain@ideasonboard.com> References: <20220506095307.78370-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/3] ipa: ipu3: Add a frameCompleted() helper X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" IPAIPU3::frameCompleted helper will consolidate all the book-keeping required by the IPA when it has finished processing a particular frame. Signed-off-by: Umang Jain --- src/ipa/ipu3/ipu3.cpp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index dd6cfd79..061dc9f5 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -161,6 +161,8 @@ private: void setControls(unsigned int frame); void calculateBdsGrid(const Size &bdsOutputSize); + void frameCompleted(const uint32_t frame); + std::map buffers_; ControlInfoMap sensorCtrls_; @@ -505,6 +507,10 @@ void IPAIPU3::unmapBuffers(const std::vector &ids) } } +void IPAIPU3::frameCompleted([[maybe_unused]] const uint32_t frame) +{ +} + /** * \brief Fill and return a buffer with ISP processing parameters for a frame * \param[in] frame The frame number @@ -598,6 +604,8 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, */ metadataReady.emit(frame, ctrls); + + frameCompleted(frame); } /** From patchwork Fri May 6 09:53:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15809 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6D635C3256 for ; Fri, 6 May 2022 09:53:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1EF376564B; Fri, 6 May 2022 11:53:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1651830807; bh=jvdRq5HZ7YI+bwk+a+RCiArlSW0QqaLKuMNI8EpBta8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=aJIQBQiVCAdgcDGLd3D1EbaYDnpD7AxzZCX+ERNDuq5196lVfdBqkavfQM7YUoWW1 zG+U27nAwUAJA+iJS6RNvP41rdPmIgLiAKI4QABfec/2qekGkIIohIes/jGnm5SOeY D8hemNpjxc99U6il+3uQ5eaAEFpb0Z5OrL+gyk0lp4kK5iVXKzmlfKy3Pu44i7XfUf q9zAaIblcBqTkuTyjiEa4MfrM/mi88J50P4eP0lPFu7sB0nHja+QPROFPC2xkpOUEt ea+1GstSnLFRzrxXczrCjjqTKcQVgMNKm8ofwIMoNlls113Q6wX6uhnPur5cE2VBo+ I/WJ1xbNkVs4A== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 05F7365643 for ; Fri, 6 May 2022 11:53:26 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="P2KMD6P0"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [103.251.226.91]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A9833487; Fri, 6 May 2022 11:53:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1651830805; bh=jvdRq5HZ7YI+bwk+a+RCiArlSW0QqaLKuMNI8EpBta8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P2KMD6P0LfhZBKqxoscYzVzMphMC4HK14PVsJWYMvdlczh5pXeprGY8t67p/Lb+Eo mHKYc2cTfDDKPWS+OBv1/1TAfE/W2OdVhlHdNFvj+Vs8QFLykbl8IR2KH6m1ur1qnJ nj+DFKNQn8mi57EHA3QSByZTw4kURb+CLaYtJVww= To: libcamera-devel@lists.libcamera.org Date: Fri, 6 May 2022 15:23:06 +0530 Message-Id: <20220506095307.78370-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220506095307.78370-1-umang.jain@ideasonboard.com> References: <20220506095307.78370-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/3] ipa: ipu3: Rework IPAFrameContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Currently, IPAFrameContext consolidates the values computed by the active state of the algorithms, along with the values applied on the sensor. Moving ahead, we want to have a frame context associated with each incoming request (or frame to be captured). This not necessarily should be tied to the "active state" of the algorithms hence: - Rename current IPAFrameContext -> IPAActiveState This will now reflect the latest active state of the algorithms and has nothing to do with any frame-related ops/values. - Re-instate IPAFrameContext with a sub-structure 'sensor' currently storing the exposure and gain value. Adapt the various access to the frame context to the new changes as described above. In subsequent patch, the re-instated IPAFrameContext will reflect a frame number stored in it and will be used in conjunction with a queue. Signed-off-by: Umang Jain --- src/ipa/ipu3/algorithms/af.cpp | 42 ++++++------ src/ipa/ipu3/algorithms/agc.cpp | 21 +++--- src/ipa/ipu3/algorithms/agc.h | 2 +- src/ipa/ipu3/algorithms/awb.cpp | 16 ++--- src/ipa/ipu3/algorithms/tone_mapping.cpp | 10 +-- src/ipa/ipu3/ipa_context.cpp | 84 ++++++++++++++---------- src/ipa/ipu3/ipa_context.h | 16 +++-- src/ipa/ipu3/ipu3.cpp | 11 ++-- 8 files changed, 110 insertions(+), 92 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index f700b01f..8a5a6b1a 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -185,11 +185,11 @@ int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo) afIgnoreFrameReset(); /* Initial focus value */ - context.frameContext.af.focus = 0; + context.activeState.af.focus = 0; /* Maximum variance of the AF statistics */ - context.frameContext.af.maxVariance = 0; + context.activeState.af.maxVariance = 0; /* The stable AF value flag. if it is true, the AF should be in a stable state. */ - context.frameContext.af.stable = false; + context.activeState.af.stable = false; return 0; } @@ -211,10 +211,10 @@ void Af::afCoarseScan(IPAContext &context) if (afScan(context, kCoarseSearchStep)) { coarseCompleted_ = true; - context.frameContext.af.maxVariance = 0; - focus_ = context.frameContext.af.focus - - (context.frameContext.af.focus * kFineRange); - context.frameContext.af.focus = focus_; + context.activeState.af.maxVariance = 0; + focus_ = context.activeState.af.focus - + (context.activeState.af.focus * kFineRange); + context.activeState.af.focus = focus_; previousVariance_ = 0; maxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)), 0U, kMaxFocusSteps); @@ -237,7 +237,7 @@ void Af::afFineScan(IPAContext &context) return; if (afScan(context, kFineSearchStep)) { - context.frameContext.af.stable = true; + context.activeState.af.stable = true; fineCompleted_ = true; } } @@ -254,10 +254,10 @@ void Af::afReset(IPAContext &context) if (afNeedIgnoreFrame()) return; - context.frameContext.af.maxVariance = 0; - context.frameContext.af.focus = 0; + context.activeState.af.maxVariance = 0; + context.activeState.af.focus = 0; focus_ = 0; - context.frameContext.af.stable = false; + context.activeState.af.stable = false; ignoreCounter_ = kIgnoreFrame; previousVariance_ = 0.0; coarseCompleted_ = false; @@ -280,7 +280,7 @@ bool Af::afScan(IPAContext &context, int min_step) { if (focus_ > maxStep_) { /* If reach the max step, move lens to the position. */ - context.frameContext.af.focus = bestFocus_; + context.activeState.af.focus = bestFocus_; return true; } else { /* @@ -288,8 +288,8 @@ bool Af::afScan(IPAContext &context, int min_step) * derivative. If the direction changes, it means we have * passed a maximum one step before. */ - if ((currentVariance_ - context.frameContext.af.maxVariance) >= - -(context.frameContext.af.maxVariance * 0.1)) { + if ((currentVariance_ - context.activeState.af.maxVariance) >= + -(context.activeState.af.maxVariance * 0.1)) { /* * Positive and zero derivative: * The variance is still increasing. The focus could be @@ -298,8 +298,8 @@ bool Af::afScan(IPAContext &context, int min_step) */ bestFocus_ = focus_; focus_ += min_step; - context.frameContext.af.focus = focus_; - context.frameContext.af.maxVariance = currentVariance_; + context.activeState.af.focus = focus_; + context.activeState.af.maxVariance = currentVariance_; } else { /* * Negative derivative: @@ -307,7 +307,7 @@ bool Af::afScan(IPAContext &context, int min_step) * variance is found. Set focus step to previous good one * then return immediately. */ - context.frameContext.af.focus = bestFocus_; + context.activeState.af.focus = bestFocus_; return true; } } @@ -389,13 +389,13 @@ double Af::afEstimateVariance(Span y_items, bool isY1) bool Af::afIsOutOfFocus(IPAContext context) { const uint32_t diff_var = std::abs(currentVariance_ - - context.frameContext.af.maxVariance); - const double var_ratio = diff_var / context.frameContext.af.maxVariance; + context.activeState.af.maxVariance); + const double var_ratio = diff_var / context.activeState.af.maxVariance; LOG(IPU3Af, Debug) << "Variance change rate: " << var_ratio << " Current VCM step: " - << context.frameContext.af.focus; + << context.activeState.af.focus; if (var_ratio > kMaxChange) return true; @@ -437,7 +437,7 @@ void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) */ currentVariance_ = afEstimateVariance(y_items, !coarseCompleted_); - if (!context.frameContext.af.stable) { + if (!context.activeState.af.stable) { afCoarseScan(context); afFineScan(context); } else { diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 7d4b3503..fdeec09d 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -87,7 +87,7 @@ int Agc::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { const IPASessionConfiguration &configuration = context.configuration; - IPAFrameContext &frameContext = context.frameContext; + IPAActiveState &activeState = context.activeState; stride_ = configuration.grid.stride; @@ -99,8 +99,8 @@ int Agc::configure(IPAContext &context, maxAnalogueGain_ = std::min(configuration.agc.maxAnalogueGain, kMaxAnalogueGain); /* Configure the default exposure and gain. */ - frameContext.agc.gain = std::max(minAnalogueGain_, kMinAnalogueGain); - frameContext.agc.exposure = 10ms / configuration.sensor.lineDuration; + activeState.agc.gain = std::max(minAnalogueGain_, kMinAnalogueGain); + activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; frameCount_ = 0; return 0; @@ -249,9 +249,10 @@ void Agc::computeExposure(IPAContext &context, double yGain, << shutterTime << " and " << stepGain; + IPAActiveState &activeState = context.activeState; /* Update the estimated exposure and gain. */ - frameContext.agc.exposure = shutterTime / configuration.sensor.lineDuration; - frameContext.agc.gain = stepGain; + activeState.agc.exposure = shutterTime / configuration.sensor.lineDuration; + activeState.agc.gain = stepGain; } /** @@ -279,7 +280,7 @@ void Agc::computeExposure(IPAContext &context, double yGain, * More detailed information can be found in: * https://en.wikipedia.org/wiki/Relative_luminance */ -double Agc::estimateLuminance(IPAFrameContext &frameContext, +double Agc::estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain) @@ -307,9 +308,9 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, * Apply the AWB gains to approximate colours correctly, use the Rec. * 601 formula to calculate the relative luminance, and normalize it. */ - double ySum = redSum * frameContext.awb.gains.red * 0.299 - + greenSum * frameContext.awb.gains.green * 0.587 - + blueSum * frameContext.awb.gains.blue * 0.114; + double ySum = redSum * activeState.awb.gains.red * 0.299 + + greenSum * activeState.awb.gains.green * 0.587 + + blueSum * activeState.awb.gains.blue * 0.114; return ySum / (grid.height * grid.width) / 255; } @@ -344,7 +345,7 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) double yTarget = kRelativeLuminanceTarget; for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(context.frameContext, + double yValue = estimateLuminance(context.activeState, context.configuration.grid.bdsGrid, stats, yGain); double extraGain = std::min(10.0, yTarget / (yValue + .001)); diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index ad705605..31420841 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -36,7 +36,7 @@ private: utils::Duration filterExposure(utils::Duration currentExposure); void computeExposure(IPAContext &context, double yGain, double iqMeanGain); - double estimateLuminance(IPAFrameContext &frameContext, + double estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 87a6cc7a..ab6924eb 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -396,10 +396,10 @@ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) * The results are cached, so if no results were calculated, we set the * cached values from asyncResults_ here. */ - context.frameContext.awb.gains.blue = asyncResults_.blueGain; - context.frameContext.awb.gains.green = asyncResults_.greenGain; - context.frameContext.awb.gains.red = asyncResults_.redGain; - context.frameContext.awb.temperatureK = asyncResults_.temperatureK; + context.activeState.awb.gains.blue = asyncResults_.blueGain; + context.activeState.awb.gains.green = asyncResults_.greenGain; + context.activeState.awb.gains.red = asyncResults_.redGain; + context.activeState.awb.temperatureK = asyncResults_.temperatureK; } constexpr uint16_t Awb::threshold(float value) @@ -450,10 +450,10 @@ void Awb::prepare(IPAContext &context, ipu3_uapi_params *params) params->acc_param.bnr.opt_center_sqr.y_sqr_reset = params->acc_param.bnr.opt_center.y_reset * params->acc_param.bnr.opt_center.y_reset; /* Convert to u3.13 fixed point values */ - params->acc_param.bnr.wb_gains.gr = 8192 * context.frameContext.awb.gains.green; - params->acc_param.bnr.wb_gains.r = 8192 * context.frameContext.awb.gains.red; - params->acc_param.bnr.wb_gains.b = 8192 * context.frameContext.awb.gains.blue; - params->acc_param.bnr.wb_gains.gb = 8192 * context.frameContext.awb.gains.green; + params->acc_param.bnr.wb_gains.gr = 8192 * context.activeState.awb.gains.green; + params->acc_param.bnr.wb_gains.r = 8192 * context.activeState.awb.gains.red; + params->acc_param.bnr.wb_gains.b = 8192 * context.activeState.awb.gains.blue; + params->acc_param.bnr.wb_gains.gb = 8192 * context.activeState.awb.gains.green; LOG(IPU3Awb, Debug) << "Color temperature estimated: " << asyncResults_.temperatureK; diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 2040eda5..7c78d0d9 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -42,7 +42,7 @@ int ToneMapping::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { /* Initialise tone mapping gamma value. */ - context.frameContext.toneMapping.gamma = 0.0; + context.activeState.toneMapping.gamma = 0.0; return 0; } @@ -60,7 +60,7 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, { /* Copy the calculated LUT into the parameters buffer. */ memcpy(params->acc_param.gamma.gc_lut.lut, - context.frameContext.toneMapping.gammaCorrection.lut, + context.activeState.toneMapping.gammaCorrection.lut, IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES * sizeof(params->acc_param.gamma.gc_lut.lut[0])); @@ -87,11 +87,11 @@ void ToneMapping::process(IPAContext &context, */ gamma_ = 1.1; - if (context.frameContext.toneMapping.gamma == gamma_) + if (context.activeState.toneMapping.gamma == gamma_) return; struct ipu3_uapi_gamma_corr_lut &lut = - context.frameContext.toneMapping.gammaCorrection; + context.activeState.toneMapping.gammaCorrection; for (uint32_t i = 0; i < std::size(lut.lut); i++) { double j = static_cast(i) / (std::size(lut.lut) - 1); @@ -101,7 +101,7 @@ void ToneMapping::process(IPAContext &context, lut.lut[i] = gamma * 8191; } - context.frameContext.toneMapping.gamma = gamma_; + context.activeState.toneMapping.gamma = gamma_; } } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index b1570dde..06eb2776 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -24,9 +24,20 @@ namespace libcamera::ipa::ipu3 { * may also be updated in the start() operation. */ +/** + * \struct IPAActiveState + * \brief The active state of the IPA algorithms + * + * The IPA is fed with the statistics generated from the latest frame captured + * by the hardware. The statistics are then processed by the IPA algorithms to + * compute ISP parameters required for the next frame capture. The current state + * of the algorithms is reflected through IPAActiveState. The latest computed + * values by the IPA algorithms are stored in this structure. + */ + /** * \struct IPAFrameContext - * \brief Per-frame context for algorithms + * \brief Context for a frame * * The frame context stores data specific to a single frame processed by the * IPA. Each frame processed by the IPA has a context associated with it, @@ -34,9 +45,10 @@ namespace libcamera::ipa::ipu3 { * * \todo Detail how to access contexts for a particular frame * - * Each of the fields in the frame context belongs to either a specific - * algorithm, or to the top-level IPA module. A field may be read by any - * algorithm, but should only be written by its owner. + * Fields in the frame context should reflect the values with which the frame + * was processed on the hardware ISP. A field may be read by any algorithm as + * and when required, or can be used to prepare the metadata container of the + * frame. */ /** @@ -49,10 +61,10 @@ namespace libcamera::ipa::ipu3 { * \var IPAContext::frameContext * \brief The frame context for the frame being processed * - * \todo While the frame context is supposed to be per-frame, this - * single frame context stores data related to both the current frame - * and the previous frames, with fields being updated as the algorithms - * are run. This needs to be turned into real per-frame data storage. + * \var IPAContext::activeState + * \brief The current state of IPA algorithms + * + * \todo The frame context needs to be turned into real per-frame data storage. */ /** @@ -78,17 +90,17 @@ namespace libcamera::ipa::ipu3 { */ /** - * \var IPAFrameContext::af + * \var IPAActiveState::af * \brief Context for the Automatic Focus algorithm * - * \struct IPAFrameContext::af - * \var IPAFrameContext::af.focus + * \struct IPAActiveState::af + * \var IPAActiveState::af.focus * \brief Current position of the lens * - * \var IPAFrameContext::af.maxVariance + * \var IPAActiveState::af.maxVariance * \brief The maximum variance of the current image. * - * \var IPAFrameContext::af.stable + * \var IPAActiveState::af.stable * \brief It is set to true, if the best focus is found. */ @@ -121,64 +133,64 @@ namespace libcamera::ipa::ipu3 { */ /** - * \var IPAFrameContext::agc + * \var IPAActiveState::agc * \brief Context for the Automatic Gain Control algorithm * * The exposure and gain determined are expected to be applied to the sensor * at the earliest opportunity. * - * \var IPAFrameContext::agc.exposure + * \var IPAActiveState::agc.exposure * \brief Exposure time expressed as a number of lines * - * \var IPAFrameContext::agc.gain + * \var IPAActiveState::agc.gain * \brief Analogue gain multiplier * * The gain should be adapted to the sensor specific gain code before applying. */ /** - * \var IPAFrameContext::awb + * \var IPAActiveState::awb * \brief Context for the Automatic White Balance algorithm * - * \struct IPAFrameContext::awb.gains + * \struct IPAActiveState::awb.gains * \brief White balance gains * - * \var IPAFrameContext::awb.gains.red + * \var IPAActiveState::awb.gains.red * \brief White balance gain for R channel * - * \var IPAFrameContext::awb.gains.green + * \var IPAActiveState::awb.gains.green * \brief White balance gain for G channel * - * \var IPAFrameContext::awb.gains.blue + * \var IPAActiveState::awb.gains.blue * \brief White balance gain for B channel * - * \var IPAFrameContext::awb.temperatureK + * \var IPAActiveState::awb.temperatureK * \brief Estimated color temperature */ /** - * \var IPAFrameContext::sensor - * \brief Effective sensor values - * - * \var IPAFrameContext::sensor.exposure - * \brief Exposure time expressed as a number of lines - * - * \var IPAFrameContext::sensor.gain - * \brief Analogue gain multiplier - */ - -/** - * \var IPAFrameContext::toneMapping + * \var IPAActiveState::toneMapping * \brief Context for ToneMapping and Gamma control * - * \var IPAFrameContext::toneMapping.gamma + * \var IPAActiveState::toneMapping.gamma * \brief Gamma value for the LUT * - * \var IPAFrameContext::toneMapping.gammaCorrection + * \var IPAActiveState::toneMapping.gammaCorrection * \brief Per-pixel tone mapping implemented as a LUT * * The LUT structure is defined by the IPU3 kernel interface. See * struct ipu3_uapi_gamma_corr_lut for further details. */ +/** + * \var IPAFrameContext::sensor + * \brief Effective sensor values that were applied for the frame + * + * \var IPAFrameContext::sensor.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::sensor.gain + * \brief Analogue gain multiplier + */ + } /* namespace libcamera::ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 103498ef..8d681131 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -42,7 +42,7 @@ struct IPASessionConfiguration { } sensor; }; -struct IPAFrameContext { +struct IPAActiveState { struct { uint32_t focus; double maxVariance; @@ -64,19 +64,23 @@ struct IPAFrameContext { double temperatureK; } awb; - struct { - uint32_t exposure; - double gain; - } sensor; - struct { double gamma; struct ipu3_uapi_gamma_corr_lut gammaCorrection; } toneMapping; }; +struct IPAFrameContext { + struct { + uint32_t exposure; + double gain; + } sensor; +}; + struct IPAContext { IPASessionConfiguration configuration; + IPAActiveState activeState; + IPAFrameContext frameContext; }; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 061dc9f5..268c8f61 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -456,7 +456,8 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, calculateBdsGrid(configInfo.bdsOutputSize); - /* Clean frameContext at each reconfiguration. */ + /* Clean IPAActiveState at each reconfiguration. */ + context_.activeState = {}; context_.frameContext = {}; if (!validateSensorControls()) { @@ -591,7 +592,7 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, ctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain); - ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration); @@ -631,8 +632,8 @@ void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame, */ void IPAIPU3::setControls(unsigned int frame) { - int32_t exposure = context_.frameContext.agc.exposure; - int32_t gain = camHelper_->gainCode(context_.frameContext.agc.gain); + int32_t exposure = context_.activeState.agc.exposure; + int32_t gain = camHelper_->gainCode(context_.activeState.agc.gain); ControlList ctrls(sensorCtrls_); ctrls.set(V4L2_CID_EXPOSURE, exposure); @@ -640,7 +641,7 @@ void IPAIPU3::setControls(unsigned int frame) ControlList lensCtrls(lensCtrls_); lensCtrls.set(V4L2_CID_FOCUS_ABSOLUTE, - static_cast(context_.frameContext.af.focus)); + static_cast(context_.activeState.af.focus)); setSensorControls.emit(frame, ctrls, lensCtrls); } From patchwork Fri May 6 09:53:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15810 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 430E4C3256 for ; Fri, 6 May 2022 09:53:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0462C65643; Fri, 6 May 2022 11:53:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1651830809; bh=IFeCJ3yN3MrRoQpo1uDUprI+hmvYjx9F9SXDZ9enxK8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=BdrtPQ9EfktgGLgboJZEmx+YOSViVlKS7dc+4oHYLCIBALm6DOmbzY84V6AcxwYiF JUcvTfmtK0ZWsrs6KHNZNlNwPrxzDyeSYyjJBchfZ9MVsmOeyWVRM/eH/dHR6zgDHQ 3E4f0N1HTztuQYIadi5vpAfhnwFRcZXzlaP9npFeWaRmJQw3Y2535baKbZQ70oZQAc nMI+K6etYO9nPYSacG/USNEWTCSYcoHGnvTZ8ozjIhf/mIdZlBHR2UkOhWOLpoU9W4 9Rc4aVzYwH++BvCe74NqeK1VaQHzyT67b+CsFA/srPPPI629cfzdDHuAiPHPxPgJmU X3pRA4ER2+Y+Q== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1EE7C65643 for ; Fri, 6 May 2022 11:53:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tgZaxrir"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [103.251.226.91]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E21A5487; Fri, 6 May 2022 11:53:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1651830807; bh=IFeCJ3yN3MrRoQpo1uDUprI+hmvYjx9F9SXDZ9enxK8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tgZaxrir2jwUnv9UCiKwKJc6oZG9Tt436i5gMFv4aK/MuIHakfx4PVN/1nmgdrroy oVlXZ2NzLsunc+pzhuK53F3xYOgnjaFcsM087rNhYIjJJdUrFxpvSe4qc+TYGsqQFB DI1Rx8V6KkRFEznPZICHtMy9bCxQFpESA0RZhp8c= To: libcamera-devel@lists.libcamera.org Date: Fri, 6 May 2022 15:23:07 +0530 Message-Id: <20220506095307.78370-4-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220506095307.78370-1-umang.jain@ideasonboard.com> References: <20220506095307.78370-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 3/3] ipa: ipu3: Introduce a IPAFrameContext queue X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Introduce a queue of IPAFrameContext which shall maintain the data required to track each frame. Currently, it is storing only the sensor controls values applied for the frame but new structures can be introduced in IPAFrameContext to extend any frame-related contexts (either in-processing or new frames being queued). For example, this patch provides the foundation for the extension of IPAFrameContext to store incoming request controls. Signed-off-by: Umang Jain --- src/ipa/ipu3/algorithms/agc.cpp | 2 +- src/ipa/ipu3/ipa_context.cpp | 16 ++++++++++++++-- src/ipa/ipu3/ipa_context.h | 8 +++++++- src/ipa/ipu3/ipu3.cpp | 31 ++++++++++++++++++++++++++----- 4 files changed, 48 insertions(+), 9 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index fdeec09d..4784af00 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -187,7 +187,7 @@ void Agc::computeExposure(IPAContext &context, double yGain, double iqMeanGain) { const IPASessionConfiguration &configuration = context.configuration; - IPAFrameContext &frameContext = context.frameContext; + IPAFrameContext &frameContext = context.frameContextQueue.front(); /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; double analogueGain = frameContext.sensor.gain; diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 06eb2776..fb48bc9b 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -58,8 +58,8 @@ namespace libcamera::ipa::ipu3 { * \var IPAContext::configuration * \brief The IPA session configuration, immutable during the session * - * \var IPAContext::frameContext - * \brief The frame context for the frame being processed + * \var IPAContext::frameContextQueue + * \brief FIFO container of IPAFrameContext for all the frames being queued * * \var IPAContext::activeState * \brief The current state of IPA algorithms @@ -182,6 +182,15 @@ namespace libcamera::ipa::ipu3 { * struct ipu3_uapi_gamma_corr_lut for further details. */ +/** + * \brief Construct a IPAFrameContext instance + */ +IPAFrameContext::IPAFrameContext(uint32_t frame) + : frame(frame) +{ + sensor = {}; +} + /** * \var IPAFrameContext::sensor * \brief Effective sensor values that were applied for the frame @@ -191,6 +200,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPAFrameContext::sensor.gain * \brief Analogue gain multiplier + * + * \var IPAFrameContext::frame + * \brief The frame number associated with this IPAFrameContext */ } /* namespace libcamera::ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 8d681131..20cccc97 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -8,6 +8,8 @@ #pragma once +#include + #include #include @@ -71,17 +73,21 @@ struct IPAActiveState { }; struct IPAFrameContext { + IPAFrameContext(uint32_t frame); + struct { uint32_t exposure; double gain; } sensor; + + uint32_t frame; }; struct IPAContext { IPASessionConfiguration configuration; IPAActiveState activeState; - IPAFrameContext frameContext; + std::queue frameContextQueue; }; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 268c8f61..1b566f14 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -350,6 +350,8 @@ int IPAIPU3::start() */ void IPAIPU3::stop() { + while (!context_.frameContextQueue.empty()) + context_.frameContextQueue.pop(); } /** @@ -458,7 +460,7 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, /* Clean IPAActiveState at each reconfiguration. */ context_.activeState = {}; - context_.frameContext = {}; + context_.frameContextQueue = {}; if (!validateSensorControls()) { LOG(IPAIPU3, Error) << "Sensor control validation failed."; @@ -510,6 +512,13 @@ void IPAIPU3::unmapBuffers(const std::vector &ids) void IPAIPU3::frameCompleted([[maybe_unused]] const uint32_t frame) { + while (!context_.frameContextQueue.empty()) { + auto &fc = context_.frameContextQueue.front(); + if (fc.frame <= frame) + context_.frameContextQueue.pop(); + else + break; + } } /** @@ -574,8 +583,18 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); - context_.frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); - context_.frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + auto &frameContext = context_.frameContextQueue.front(); + + /* + * An assert might be too harsh here. We want to know the cases + * where the front of the queue (implies the current frame in processing, + * diverges from the frame parameter of this function + * \todo Identify those cases - e.g. frame drop? + */ + ASSERT(context_.frameContextQueue.front().frame == frame); + + frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); + frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); double lineDuration = context_.configuration.sensor.lineDuration.get(); int32_t vBlank = context_.configuration.sensor.defVBlank; @@ -590,11 +609,11 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, int64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration; ctrls.set(controls::FrameDuration, frameDuration); - ctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain); + ctrls.set(controls::AnalogueGain, frameContext.sensor.gain); ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); - ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration); + ctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); /* * \todo The Metadata provides a path to getting extended data @@ -621,6 +640,8 @@ void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame, [[maybe_unused]] const ControlList &controls) { /* \todo Start processing for 'frame' based on 'controls'. */ + + context_.frameContextQueue.emplace(frame); } /**