From patchwork Wed May 18 13:10:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15947 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D6469C3256 for ; Wed, 18 May 2022 13:10:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E95BB6565C; Wed, 18 May 2022 15:10:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652879442; bh=+LeCO4mftC3kxvlv2pt8aAm5z9+wKzVWlqJrEAjT11k=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=WVtEg9HTRvhgpXc6UxEc3efF9N9UFztTgyk384iVWzvhCWkUCpXm/WNSG6uZpsagj w5dFbgJWOb4r9lE/Vnxzm07RhI6vHhPizdYms46M7NOxwvQb1tliLppiU9txoVmxbo zla5LljNqh5eX+f9QDeHOwBjPJ4/jqQLmaC6EhTFLWqcZWuuTWBSH8iBsEsXZZNBJM ketrjbjhZWhll1OYWKPOvqGd5EyGZy89A2OFuqdY25QQl5dD3rz6min+p0LPBoaVHC vUMmWhU5P3JsNfb9LHR1uqDvxgRDXJXgNuZ5ubDj6tP4U8N8UD8fsD+u9Aw5TqSFN7 BcKoRxPd2xweg== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 85B0A65656 for ; Wed, 18 May 2022 15:10:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="bZyPAmwx"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id BA483E50; Wed, 18 May 2022 15:10:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652879439; bh=+LeCO4mftC3kxvlv2pt8aAm5z9+wKzVWlqJrEAjT11k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bZyPAmwxViHBa2zts4uKDLNgJJwFpM3rFYZjTatknmTo4gC2YStHRjXfJmU3orLx/ HCIu5t96cI0JGxA6ewuXNZzPIr/gAVbe23ZetTSOnQB3RbGJ6J1kpgkl+PQOVhrih6 LQ1XwUGHu4LhefcaA3nIHOlnmm+FxBLUsUQJu3VA= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:10:28 +0200 Message-Id: <20220518131030.421225-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220518131030.421225-1-umang.jain@ideasonboard.com> References: <20220518131030.421225-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 1/3] ipa: ipu3: Rework IPAFrameContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Currently, IPAFrameContext consolidates the values computed by the active state of the algorithms, along with the values applied on the sensor. Moving ahead, we want to have a frame context associated with each incoming request (or frame to be captured). This shouldn't be necessarily be tied to "active state" of the algorithms hence: - Rename current IPAFrameContext -> IPAActiveState This will now reflect the latest active state of the algorithms and has nothing to do with any frame-related ops/values. - Re-instate IPAFrameContext with a sub-structure 'sensor' currently storing the exposure and gain value. Adapt the various access to the frame context to the new changes as described above. Subsequently, the re-instated IPAFrameContext will be extended to contain a frame number and ControlList to remember the incoming request controls provided by the application. A ring-buffer will be introduced to store these frame contexts for a certain number of frames. Signed-off-by: Umang Jain Reviewed-by: Jacopo Mondi Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/af.cpp | 42 +++++------ src/ipa/ipu3/algorithms/agc.cpp | 21 +++--- src/ipa/ipu3/algorithms/agc.h | 2 +- src/ipa/ipu3/algorithms/awb.cpp | 16 ++--- src/ipa/ipu3/algorithms/tone_mapping.cpp | 10 +-- src/ipa/ipu3/ipa_context.cpp | 88 ++++++++++++++---------- src/ipa/ipu3/ipa_context.h | 16 +++-- src/ipa/ipu3/ipu3.cpp | 11 +-- 8 files changed, 112 insertions(+), 94 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index f700b01f..8a5a6b1a 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -185,11 +185,11 @@ int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo) afIgnoreFrameReset(); /* Initial focus value */ - context.frameContext.af.focus = 0; + context.activeState.af.focus = 0; /* Maximum variance of the AF statistics */ - context.frameContext.af.maxVariance = 0; + context.activeState.af.maxVariance = 0; /* The stable AF value flag. if it is true, the AF should be in a stable state. */ - context.frameContext.af.stable = false; + context.activeState.af.stable = false; return 0; } @@ -211,10 +211,10 @@ void Af::afCoarseScan(IPAContext &context) if (afScan(context, kCoarseSearchStep)) { coarseCompleted_ = true; - context.frameContext.af.maxVariance = 0; - focus_ = context.frameContext.af.focus - - (context.frameContext.af.focus * kFineRange); - context.frameContext.af.focus = focus_; + context.activeState.af.maxVariance = 0; + focus_ = context.activeState.af.focus - + (context.activeState.af.focus * kFineRange); + context.activeState.af.focus = focus_; previousVariance_ = 0; maxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)), 0U, kMaxFocusSteps); @@ -237,7 +237,7 @@ void Af::afFineScan(IPAContext &context) return; if (afScan(context, kFineSearchStep)) { - context.frameContext.af.stable = true; + context.activeState.af.stable = true; fineCompleted_ = true; } } @@ -254,10 +254,10 @@ void Af::afReset(IPAContext &context) if (afNeedIgnoreFrame()) return; - context.frameContext.af.maxVariance = 0; - context.frameContext.af.focus = 0; + context.activeState.af.maxVariance = 0; + context.activeState.af.focus = 0; focus_ = 0; - context.frameContext.af.stable = false; + context.activeState.af.stable = false; ignoreCounter_ = kIgnoreFrame; previousVariance_ = 0.0; coarseCompleted_ = false; @@ -280,7 +280,7 @@ bool Af::afScan(IPAContext &context, int min_step) { if (focus_ > maxStep_) { /* If reach the max step, move lens to the position. */ - context.frameContext.af.focus = bestFocus_; + context.activeState.af.focus = bestFocus_; return true; } else { /* @@ -288,8 +288,8 @@ bool Af::afScan(IPAContext &context, int min_step) * derivative. If the direction changes, it means we have * passed a maximum one step before. */ - if ((currentVariance_ - context.frameContext.af.maxVariance) >= - -(context.frameContext.af.maxVariance * 0.1)) { + if ((currentVariance_ - context.activeState.af.maxVariance) >= + -(context.activeState.af.maxVariance * 0.1)) { /* * Positive and zero derivative: * The variance is still increasing. The focus could be @@ -298,8 +298,8 @@ bool Af::afScan(IPAContext &context, int min_step) */ bestFocus_ = focus_; focus_ += min_step; - context.frameContext.af.focus = focus_; - context.frameContext.af.maxVariance = currentVariance_; + context.activeState.af.focus = focus_; + context.activeState.af.maxVariance = currentVariance_; } else { /* * Negative derivative: @@ -307,7 +307,7 @@ bool Af::afScan(IPAContext &context, int min_step) * variance is found. Set focus step to previous good one * then return immediately. */ - context.frameContext.af.focus = bestFocus_; + context.activeState.af.focus = bestFocus_; return true; } } @@ -389,13 +389,13 @@ double Af::afEstimateVariance(Span y_items, bool isY1) bool Af::afIsOutOfFocus(IPAContext context) { const uint32_t diff_var = std::abs(currentVariance_ - - context.frameContext.af.maxVariance); - const double var_ratio = diff_var / context.frameContext.af.maxVariance; + context.activeState.af.maxVariance); + const double var_ratio = diff_var / context.activeState.af.maxVariance; LOG(IPU3Af, Debug) << "Variance change rate: " << var_ratio << " Current VCM step: " - << context.frameContext.af.focus; + << context.activeState.af.focus; if (var_ratio > kMaxChange) return true; @@ -437,7 +437,7 @@ void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) */ currentVariance_ = afEstimateVariance(y_items, !coarseCompleted_); - if (!context.frameContext.af.stable) { + if (!context.activeState.af.stable) { afCoarseScan(context); afFineScan(context); } else { diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 7d4b3503..fdeec09d 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -87,7 +87,7 @@ int Agc::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { const IPASessionConfiguration &configuration = context.configuration; - IPAFrameContext &frameContext = context.frameContext; + IPAActiveState &activeState = context.activeState; stride_ = configuration.grid.stride; @@ -99,8 +99,8 @@ int Agc::configure(IPAContext &context, maxAnalogueGain_ = std::min(configuration.agc.maxAnalogueGain, kMaxAnalogueGain); /* Configure the default exposure and gain. */ - frameContext.agc.gain = std::max(minAnalogueGain_, kMinAnalogueGain); - frameContext.agc.exposure = 10ms / configuration.sensor.lineDuration; + activeState.agc.gain = std::max(minAnalogueGain_, kMinAnalogueGain); + activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; frameCount_ = 0; return 0; @@ -249,9 +249,10 @@ void Agc::computeExposure(IPAContext &context, double yGain, << shutterTime << " and " << stepGain; + IPAActiveState &activeState = context.activeState; /* Update the estimated exposure and gain. */ - frameContext.agc.exposure = shutterTime / configuration.sensor.lineDuration; - frameContext.agc.gain = stepGain; + activeState.agc.exposure = shutterTime / configuration.sensor.lineDuration; + activeState.agc.gain = stepGain; } /** @@ -279,7 +280,7 @@ void Agc::computeExposure(IPAContext &context, double yGain, * More detailed information can be found in: * https://en.wikipedia.org/wiki/Relative_luminance */ -double Agc::estimateLuminance(IPAFrameContext &frameContext, +double Agc::estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain) @@ -307,9 +308,9 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, * Apply the AWB gains to approximate colours correctly, use the Rec. * 601 formula to calculate the relative luminance, and normalize it. */ - double ySum = redSum * frameContext.awb.gains.red * 0.299 - + greenSum * frameContext.awb.gains.green * 0.587 - + blueSum * frameContext.awb.gains.blue * 0.114; + double ySum = redSum * activeState.awb.gains.red * 0.299 + + greenSum * activeState.awb.gains.green * 0.587 + + blueSum * activeState.awb.gains.blue * 0.114; return ySum / (grid.height * grid.width) / 255; } @@ -344,7 +345,7 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) double yTarget = kRelativeLuminanceTarget; for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(context.frameContext, + double yValue = estimateLuminance(context.activeState, context.configuration.grid.bdsGrid, stats, yGain); double extraGain = std::min(10.0, yTarget / (yValue + .001)); diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index ad705605..31420841 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -36,7 +36,7 @@ private: utils::Duration filterExposure(utils::Duration currentExposure); void computeExposure(IPAContext &context, double yGain, double iqMeanGain); - double estimateLuminance(IPAFrameContext &frameContext, + double estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 87a6cc7a..ab6924eb 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -396,10 +396,10 @@ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) * The results are cached, so if no results were calculated, we set the * cached values from asyncResults_ here. */ - context.frameContext.awb.gains.blue = asyncResults_.blueGain; - context.frameContext.awb.gains.green = asyncResults_.greenGain; - context.frameContext.awb.gains.red = asyncResults_.redGain; - context.frameContext.awb.temperatureK = asyncResults_.temperatureK; + context.activeState.awb.gains.blue = asyncResults_.blueGain; + context.activeState.awb.gains.green = asyncResults_.greenGain; + context.activeState.awb.gains.red = asyncResults_.redGain; + context.activeState.awb.temperatureK = asyncResults_.temperatureK; } constexpr uint16_t Awb::threshold(float value) @@ -450,10 +450,10 @@ void Awb::prepare(IPAContext &context, ipu3_uapi_params *params) params->acc_param.bnr.opt_center_sqr.y_sqr_reset = params->acc_param.bnr.opt_center.y_reset * params->acc_param.bnr.opt_center.y_reset; /* Convert to u3.13 fixed point values */ - params->acc_param.bnr.wb_gains.gr = 8192 * context.frameContext.awb.gains.green; - params->acc_param.bnr.wb_gains.r = 8192 * context.frameContext.awb.gains.red; - params->acc_param.bnr.wb_gains.b = 8192 * context.frameContext.awb.gains.blue; - params->acc_param.bnr.wb_gains.gb = 8192 * context.frameContext.awb.gains.green; + params->acc_param.bnr.wb_gains.gr = 8192 * context.activeState.awb.gains.green; + params->acc_param.bnr.wb_gains.r = 8192 * context.activeState.awb.gains.red; + params->acc_param.bnr.wb_gains.b = 8192 * context.activeState.awb.gains.blue; + params->acc_param.bnr.wb_gains.gb = 8192 * context.activeState.awb.gains.green; LOG(IPU3Awb, Debug) << "Color temperature estimated: " << asyncResults_.temperatureK; diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 2040eda5..7c78d0d9 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -42,7 +42,7 @@ int ToneMapping::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { /* Initialise tone mapping gamma value. */ - context.frameContext.toneMapping.gamma = 0.0; + context.activeState.toneMapping.gamma = 0.0; return 0; } @@ -60,7 +60,7 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, { /* Copy the calculated LUT into the parameters buffer. */ memcpy(params->acc_param.gamma.gc_lut.lut, - context.frameContext.toneMapping.gammaCorrection.lut, + context.activeState.toneMapping.gammaCorrection.lut, IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES * sizeof(params->acc_param.gamma.gc_lut.lut[0])); @@ -87,11 +87,11 @@ void ToneMapping::process(IPAContext &context, */ gamma_ = 1.1; - if (context.frameContext.toneMapping.gamma == gamma_) + if (context.activeState.toneMapping.gamma == gamma_) return; struct ipu3_uapi_gamma_corr_lut &lut = - context.frameContext.toneMapping.gammaCorrection; + context.activeState.toneMapping.gammaCorrection; for (uint32_t i = 0; i < std::size(lut.lut); i++) { double j = static_cast(i) / (std::size(lut.lut) - 1); @@ -101,7 +101,7 @@ void ToneMapping::process(IPAContext &context, lut.lut[i] = gamma * 8191; } - context.frameContext.toneMapping.gamma = gamma_; + context.activeState.toneMapping.gamma = gamma_; } } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index b1570dde..383c2e37 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -24,19 +24,31 @@ namespace libcamera::ipa::ipu3 { * may also be updated in the start() operation. */ +/** + * \struct IPAActiveState + * \brief The active state of the IPA algorithms + * + * The IPA is fed with the statistics generated from the latest frame captured + * by the hardware. The statistics are then processed by the IPA algorithms to + * compute ISP parameters required for the next frame capture. The current state + * of the algorithms is reflected through the IPAActiveState to store the values + * most recently computed by the IPA algorithms. + */ + /** * \struct IPAFrameContext - * \brief Per-frame context for algorithms + * \brief Context for a frame * * The frame context stores data specific to a single frame processed by the * IPA. Each frame processed by the IPA has a context associated with it, * accessible through the IPAContext structure. * - * \todo Detail how to access contexts for a particular frame - * - * Each of the fields in the frame context belongs to either a specific - * algorithm, or to the top-level IPA module. A field may be read by any - * algorithm, but should only be written by its owner. + * Fields in the frame context should reflect values and controls + * associated with the specific frame as requested by the application, and + * as configured by the hardware. Fields can be read by algorithms to + * determine if they should update any specific action for this frame, and + * finally to update the metadata control lists when the frame is fully + * completed. */ /** @@ -49,10 +61,10 @@ namespace libcamera::ipa::ipu3 { * \var IPAContext::frameContext * \brief The frame context for the frame being processed * - * \todo While the frame context is supposed to be per-frame, this - * single frame context stores data related to both the current frame - * and the previous frames, with fields being updated as the algorithms - * are run. This needs to be turned into real per-frame data storage. + * \var IPAContext::activeState + * \brief The current state of IPA algorithms + * + * \todo The frame context needs to be turned into real per-frame data storage. */ /** @@ -78,17 +90,17 @@ namespace libcamera::ipa::ipu3 { */ /** - * \var IPAFrameContext::af + * \var IPAActiveState::af * \brief Context for the Automatic Focus algorithm * - * \struct IPAFrameContext::af - * \var IPAFrameContext::af.focus + * \struct IPAActiveState::af + * \var IPAActiveState::af.focus * \brief Current position of the lens * - * \var IPAFrameContext::af.maxVariance + * \var IPAActiveState::af.maxVariance * \brief The maximum variance of the current image. * - * \var IPAFrameContext::af.stable + * \var IPAActiveState::af.stable * \brief It is set to true, if the best focus is found. */ @@ -121,64 +133,64 @@ namespace libcamera::ipa::ipu3 { */ /** - * \var IPAFrameContext::agc + * \var IPAActiveState::agc * \brief Context for the Automatic Gain Control algorithm * * The exposure and gain determined are expected to be applied to the sensor * at the earliest opportunity. * - * \var IPAFrameContext::agc.exposure + * \var IPAActiveState::agc.exposure * \brief Exposure time expressed as a number of lines * - * \var IPAFrameContext::agc.gain + * \var IPAActiveState::agc.gain * \brief Analogue gain multiplier * * The gain should be adapted to the sensor specific gain code before applying. */ /** - * \var IPAFrameContext::awb + * \var IPAActiveState::awb * \brief Context for the Automatic White Balance algorithm * - * \struct IPAFrameContext::awb.gains + * \struct IPAActiveState::awb.gains * \brief White balance gains * - * \var IPAFrameContext::awb.gains.red + * \var IPAActiveState::awb.gains.red * \brief White balance gain for R channel * - * \var IPAFrameContext::awb.gains.green + * \var IPAActiveState::awb.gains.green * \brief White balance gain for G channel * - * \var IPAFrameContext::awb.gains.blue + * \var IPAActiveState::awb.gains.blue * \brief White balance gain for B channel * - * \var IPAFrameContext::awb.temperatureK + * \var IPAActiveState::awb.temperatureK * \brief Estimated color temperature */ /** - * \var IPAFrameContext::sensor - * \brief Effective sensor values - * - * \var IPAFrameContext::sensor.exposure - * \brief Exposure time expressed as a number of lines - * - * \var IPAFrameContext::sensor.gain - * \brief Analogue gain multiplier - */ - -/** - * \var IPAFrameContext::toneMapping + * \var IPAActiveState::toneMapping * \brief Context for ToneMapping and Gamma control * - * \var IPAFrameContext::toneMapping.gamma + * \var IPAActiveState::toneMapping.gamma * \brief Gamma value for the LUT * - * \var IPAFrameContext::toneMapping.gammaCorrection + * \var IPAActiveState::toneMapping.gammaCorrection * \brief Per-pixel tone mapping implemented as a LUT * * The LUT structure is defined by the IPU3 kernel interface. See * struct ipu3_uapi_gamma_corr_lut for further details. */ +/** + * \var IPAFrameContext::sensor + * \brief Effective sensor values that were applied for the frame + * + * \var IPAFrameContext::sensor.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::sensor.gain + * \brief Analogue gain multiplier + */ + } /* namespace libcamera::ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 103498ef..8d681131 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -42,7 +42,7 @@ struct IPASessionConfiguration { } sensor; }; -struct IPAFrameContext { +struct IPAActiveState { struct { uint32_t focus; double maxVariance; @@ -64,19 +64,23 @@ struct IPAFrameContext { double temperatureK; } awb; - struct { - uint32_t exposure; - double gain; - } sensor; - struct { double gamma; struct ipu3_uapi_gamma_corr_lut gammaCorrection; } toneMapping; }; +struct IPAFrameContext { + struct { + uint32_t exposure; + double gain; + } sensor; +}; + struct IPAContext { IPASessionConfiguration configuration; + IPAActiveState activeState; + IPAFrameContext frameContext; }; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index dd6cfd79..3b4fc911 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -454,7 +454,8 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, calculateBdsGrid(configInfo.bdsOutputSize); - /* Clean frameContext at each reconfiguration. */ + /* Clean IPAActiveState at each reconfiguration. */ + context_.activeState = {}; context_.frameContext = {}; if (!validateSensorControls()) { @@ -585,7 +586,7 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, ctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain); - ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration); @@ -623,8 +624,8 @@ void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame, */ void IPAIPU3::setControls(unsigned int frame) { - int32_t exposure = context_.frameContext.agc.exposure; - int32_t gain = camHelper_->gainCode(context_.frameContext.agc.gain); + int32_t exposure = context_.activeState.agc.exposure; + int32_t gain = camHelper_->gainCode(context_.activeState.agc.gain); ControlList ctrls(sensorCtrls_); ctrls.set(V4L2_CID_EXPOSURE, exposure); @@ -632,7 +633,7 @@ void IPAIPU3::setControls(unsigned int frame) ControlList lensCtrls(lensCtrls_); lensCtrls.set(V4L2_CID_FOCUS_ABSOLUTE, - static_cast(context_.frameContext.af.focus)); + static_cast(context_.activeState.af.focus)); setSensorControls.emit(frame, ctrls, lensCtrls); } From patchwork Wed May 18 13:10:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15948 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 44CE7C3256 for ; Wed, 18 May 2022 13:10:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 756FF65658; Wed, 18 May 2022 15:10:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652879444; bh=Qda6Wkdmk9B85sanqlXJcxj6JNor0Ye3TnC2+lv8Qhg=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=wUJFLxg8qJNIAHuCx1WP6AeQ6sKdCs7vlbktY4kNwSKFoLXxSHdmzHgOlwr5nh9Y0 CRVCFUSCTodJCSH7YjpMZ1Ly+37jnwHbs9A0WiBJHN9cMa7F0nmsfY0JJ4xY0XQc/4 nhvTYHTAwIbPKCxUmWOcyZ8T6Z0LBMDbkITdd76ywWqlWeTuWRqba9kKD6Qa293tb+ I87LBiV1d10Fe69FMAvAbzFKdUXSR0vhL8+h1smvJKmtrzmF/hlvBCbd7anedDDTH0 2GHKHZfEkJUgFKOGAKjoQK/BmGCCgB9MgzRgUI6dhFt4YgvKWBfg97XUUXtcpmYZHq gFrwKvdgx/buA== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B62B465656 for ; Wed, 18 May 2022 15:10:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ECKLyi75"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0FB96E50; Wed, 18 May 2022 15:10:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652879441; bh=Qda6Wkdmk9B85sanqlXJcxj6JNor0Ye3TnC2+lv8Qhg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ECKLyi75Ece89W7gCeoPYzCGgvkKYzfVYjlM01/PQardXPVebTBrCdVU78lh+Va5w +7RlIfONanWbkieWwSX5wa6ZlsnUX61mKExCvCNVX8n2tCFM1LxYdVrVUXucG62KaT s6y+pgLC3nbhvi4EpR1lPqGrBN9fJJ5Ud16uWKpA= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:10:29 +0200 Message-Id: <20220518131030.421225-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220518131030.421225-1-umang.jain@ideasonboard.com> References: <20220518131030.421225-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 2/3] ipa: libipa: Add frame context pointer in process() X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Currently we have a single structure of IPAFrameContext but subsequently, we shall have a ring buffer (or similar) container to keep IPAFrameContext structures for each frame. It would be a hassle to query out the frame context required for process() (since they will reside in a ring buffer) by the IPA for each process. Hence, prepare the process() libipa template to accept a particular IPAFrameContext early on. As for this patch, we shall pass in the pointer as nullptr, so that the changes compile and keep working as-is. Signed-off-by: Umang Jain Reviewed-by: Jacopo Mondi Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/af.cpp | 4 +++- src/ipa/ipu3/algorithms/af.h | 3 ++- src/ipa/ipu3/algorithms/agc.cpp | 4 +++- src/ipa/ipu3/algorithms/agc.h | 3 ++- src/ipa/ipu3/algorithms/algorithm.h | 4 +++- src/ipa/ipu3/algorithms/awb.cpp | 3 ++- src/ipa/ipu3/algorithms/awb.h | 3 ++- src/ipa/ipu3/algorithms/tone_mapping.cpp | 3 ++- src/ipa/ipu3/algorithms/tone_mapping.h | 3 ++- src/ipa/ipu3/ipu3.cpp | 2 +- src/ipa/libipa/algorithm.cpp | 1 + src/ipa/libipa/algorithm.h | 4 +++- src/ipa/rkisp1/algorithms/agc.cpp | 4 +++- src/ipa/rkisp1/algorithms/agc.h | 3 ++- src/ipa/rkisp1/algorithms/algorithm.h | 4 +++- src/ipa/rkisp1/algorithms/awb.cpp | 4 +++- src/ipa/rkisp1/algorithms/awb.h | 3 ++- src/ipa/rkisp1/rkisp1.cpp | 2 +- 18 files changed, 40 insertions(+), 17 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index 8a5a6b1a..d07521a0 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -406,6 +406,7 @@ bool Af::afIsOutOfFocus(IPAContext context) /** * \brief Determine the max contrast image and lens position. * \param[in] context The IPA context. + * \param[in] frameContext The current frame context * \param[in] stats The statistics buffer of IPU3. * * Ideally, a clear image also has a relatively higher contrast. So, every @@ -419,7 +420,8 @@ bool Af::afIsOutOfFocus(IPAContext context) * * [1] Hill Climbing Algorithm, https://en.wikipedia.org/wiki/Hill_climbing */ -void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +void Af::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) { /* Evaluate the AF buffer length */ uint32_t afRawBufferLen = context.configuration.af.afGrid.width * diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h index b85cf941..ccf015f3 100644 --- a/src/ipa/ipu3/algorithms/af.h +++ b/src/ipa/ipu3/algorithms/af.h @@ -32,7 +32,8 @@ public: void prepare(IPAContext &context, ipu3_uapi_params *params) override; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: void afCoarseScan(IPAContext &context); diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index fdeec09d..383a8deb 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -318,12 +318,14 @@ double Agc::estimateLuminance(IPAActiveState &activeState, /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context + * \param[in] frameContext The current frame context * \param[in] stats The IPU3 statistics and ISP results * * Identify the current image brightness, and use that to estimate the optimal * new exposure and gain for the scene. */ -void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +void Agc::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) { /* * Estimate the gain needed to have the proportion of pixels in a given diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 31420841..219a1a96 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -28,7 +28,8 @@ public: ~Agc() = default; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: double measureBrightness(const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/algorithms/algorithm.h b/src/ipa/ipu3/algorithms/algorithm.h index d2eecc78..234b2bd7 100644 --- a/src/ipa/ipu3/algorithms/algorithm.h +++ b/src/ipa/ipu3/algorithms/algorithm.h @@ -17,7 +17,9 @@ namespace libcamera { namespace ipa::ipu3 { -using Algorithm = libcamera::ipa::Algorithm; +using Algorithm = libcamera::ipa::Algorithm; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index ab6924eb..5c232d92 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -387,7 +387,8 @@ void Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats) /** * \copydoc libcamera::ipa::Algorithm::process */ -void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +void Awb::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) { calculateWBGains(stats); diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index ab4b0a33..9a50a985 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -40,7 +40,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, ipu3_uapi_params *params) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: /* \todo Make these structs available to all the ISPs ? */ diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 7c78d0d9..f86e79b2 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -72,12 +72,13 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, /** * \brief Calculate the tone mapping look up table * \param context The shared IPA context + * \param frameContext The current frame context * \param stats The IPU3 statistics and ISP results * * The tone mapping look up table is generated as an inverse power curve from * our gamma setting. */ -void ToneMapping::process(IPAContext &context, +void ToneMapping::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, [[maybe_unused]] const ipu3_uapi_stats_3a *stats) { /* diff --git a/src/ipa/ipu3/algorithms/tone_mapping.h b/src/ipa/ipu3/algorithms/tone_mapping.h index b727ab1e..d7d48006 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.h +++ b/src/ipa/ipu3/algorithms/tone_mapping.h @@ -20,7 +20,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, ipu3_uapi_params *params) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: double gamma_; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 3b4fc911..16e5028f 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -576,7 +576,7 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, ControlList ctrls(controls::controls); for (auto const &algo : algorithms_) - algo->process(context_, stats); + algo->process(context_, nullptr, stats); setControls(frame); diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp index 398d5372..cce2ed62 100644 --- a/src/ipa/libipa/algorithm.cpp +++ b/src/ipa/libipa/algorithm.cpp @@ -64,6 +64,7 @@ namespace ipa { * \fn Algorithm::process() * \brief Process ISP statistics, and run algorithm operations * \param[in] context The shared IPA context + * \param[in] frameContext The current frame's context * \param[in] stats The IPA statistics and ISP results * * This function is called while camera is running for every frame processed by diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h index 766aee5d..032a05b5 100644 --- a/src/ipa/libipa/algorithm.h +++ b/src/ipa/libipa/algorithm.h @@ -10,7 +10,8 @@ namespace libcamera { namespace ipa { -template +template class Algorithm { public: @@ -28,6 +29,7 @@ public: } virtual void process([[maybe_unused]] Context &context, + [[maybe_unused]] FrameContext *frameContext, [[maybe_unused]] const Stats *stats) { } diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 5f4c3f93..b5a184d9 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -280,7 +280,9 @@ double Agc::measureBrightness(const rkisp1_cif_isp_hist_stat *hist) const * Identify the current image brightness, and use that to estimate the optimal * new exposure and gain for the scene. */ -void Agc::process(IPAContext &context, const rkisp1_stat_buffer *stats) +void Agc::process(IPAContext &context, + [[maybe_unused]] IPAFrameContext *frameContext, + const rkisp1_stat_buffer *stats) { const rkisp1_cif_isp_stat *params = &stats->params; ASSERT(stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP); diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index ce1adf27..22c02779 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -29,7 +29,8 @@ public: int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; void prepare(IPAContext &context, rkisp1_params_cfg *params) override; - void process(IPAContext &context, const rkisp1_stat_buffer *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const rkisp1_stat_buffer *stats) override; private: void computeExposure(IPAContext &Context, double yGain, double iqMeanGain); diff --git a/src/ipa/rkisp1/algorithms/algorithm.h b/src/ipa/rkisp1/algorithms/algorithm.h index d46c3188..68e3a44e 100644 --- a/src/ipa/rkisp1/algorithms/algorithm.h +++ b/src/ipa/rkisp1/algorithms/algorithm.h @@ -19,7 +19,9 @@ namespace libcamera { namespace ipa::rkisp1 { -using Algorithm = libcamera::ipa::Algorithm; +using Algorithm = libcamera::ipa::Algorithm; } /* namespace ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index be4585c6..88441382 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -119,7 +119,9 @@ void Awb::prepare(IPAContext &context, rkisp1_params_cfg *params) /** * \copydoc libcamera::ipa::Algorithm::process */ -void Awb::process([[maybe_unused]] IPAContext &context, const rkisp1_stat_buffer *stats) +void Awb::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] IPAFrameContext *frameCtx, + const rkisp1_stat_buffer *stats) { const rkisp1_cif_isp_stat *params = &stats->params; const rkisp1_cif_isp_awb_stat *awb = ¶ms->awb; diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h index 11946643..7647842f 100644 --- a/src/ipa/rkisp1/algorithms/awb.h +++ b/src/ipa/rkisp1/algorithms/awb.h @@ -23,7 +23,8 @@ public: int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; void prepare(IPAContext &context, rkisp1_params_cfg *params) override; - void process(IPAContext &context, const rkisp1_stat_buffer *stats) override; + void process(IPAContext &context, IPAFrameContext *frameCtx, + const rkisp1_stat_buffer *stats) override; private: uint32_t estimateCCT(double red, double green, double blue); diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index ef1f0d56..c818a6d7 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -272,7 +272,7 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId unsigned int aeState = 0; for (auto const &algo : algorithms_) - algo->process(context_, stats); + algo->process(context_, nullptr, stats); setControls(frame); From patchwork Wed May 18 13:10:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15949 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 44479C3256 for ; Wed, 18 May 2022 13:10:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EC3E965656; Wed, 18 May 2022 15:10:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652879447; bh=NxHEyprEXwaTkMJhzvYAxM5BWeA7pKN9M4xNmoOnloo=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=NGIMlIK1EVnHmtAf0E+kofw5m15SAjDO1mJWfHNeqCW5O6UXetX7cMjxnLQ79Rpa8 s4G8aYyeNRkgC8VWYU1wv2HAxB3lOK/SL1dF4+Gf537UcGVXziByq8RtlB3lPzvpEC LYiiCh0XpPNB1fGaAcuXfoo5VVxyB9E3F8oWFJvrlHHSUMoLy7v0c19Koq/e7dAgDC 6isxmds79wfpzNZKnari77MnI7EBwJXEuihCzUdAYDCbRUW7X79Hos7iHeLIebcCoB uk+MNGdPkGvC5tx5UxI7MCucPu6LH0RpM1Fz16sQVV/LLGPvoIv/QbZ93X4ND6ToZm 2ksq84txqj1iQ== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ECC2B65661 for ; Wed, 18 May 2022 15:10:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="reST8H10"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 46DE4E50; Wed, 18 May 2022 15:10:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652879442; bh=NxHEyprEXwaTkMJhzvYAxM5BWeA7pKN9M4xNmoOnloo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=reST8H10pSJ8d+TIfP+mXRx/pu9MwxmPO7WrEXjdnjVH2D28RehAYeXWd2bsFKR3A Qg5rlVU+0ZM0MUPMNekbIQiMMGw3Y3iozMw8aByPBxcV/q1G0zyke4+4EFlh9/Wass oLzoiwaLl6ntY46NmsMmF50rWcQkngLDPb5NbRhs= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:10:30 +0200 Message-Id: <20220518131030.421225-4-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220518131030.421225-1-umang.jain@ideasonboard.com> References: <20220518131030.421225-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 3/3] ipa: ipu3: Put IPAFrameContext(s) in a ring buffer X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Instead of having one frame context constantly being updated, this patch aims to introduce per-frame IPAFrameContext which are stored in a ring buffer. Whenever a request is queued, a new IPAFrameContext is created and inserted into the ring buffer. The IPAFrameContext structure itself has been slightly extended to store a frame id and a ControlList for incoming frame controls (sent in by the application). The next step would be to read and set these controls whenever the request is actually queued to the hardware. Since now we are working in multiples of IPAFrameContext, the Algorithm::process() will actually take in a IPAFrameContext pointer (as opposed to a nullptr while preparing for this change). Signed-off-by: Umang Jain Reviewed-by: Jacopo Mondi Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 11 +++++------ src/ipa/ipu3/algorithms/agc.h | 4 ++-- src/ipa/ipu3/ipa_context.cpp | 26 ++++++++++++++++++++++---- src/ipa/ipu3/ipa_context.h | 14 +++++++++++++- src/ipa/ipu3/ipu3.cpp | 24 +++++++++++++++--------- 5 files changed, 57 insertions(+), 22 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 383a8deb..f16be534 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -183,14 +183,13 @@ utils::Duration Agc::filterExposure(utils::Duration exposureValue) * \param[in] yGain The gain calculated based on the relative luminance target * \param[in] iqMeanGain The gain calculated based on the relative luminance target */ -void Agc::computeExposure(IPAContext &context, double yGain, - double iqMeanGain) +void Agc::computeExposure(IPAContext &context, IPAFrameContext *frameContext, + double yGain, double iqMeanGain) { const IPASessionConfiguration &configuration = context.configuration; - IPAFrameContext &frameContext = context.frameContext; /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; + uint32_t exposure = frameContext->sensor.exposure; + double analogueGain = frameContext->sensor.gain; /* Use the highest of the two gain estimates. */ double evGain = std::max(yGain, iqMeanGain); @@ -360,7 +359,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameCo break; } - computeExposure(context, yGain, iqMeanGain); + computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 219a1a96..105ae0f2 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -35,8 +35,8 @@ private: double measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) const; utils::Duration filterExposure(utils::Duration currentExposure); - void computeExposure(IPAContext &context, double yGain, - double iqMeanGain); + void computeExposure(IPAContext &context, IPAFrameContext *frameContext, + double yGain, double iqMeanGain); double estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 383c2e37..13cdb835 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -58,13 +58,11 @@ namespace libcamera::ipa::ipu3 { * \var IPAContext::configuration * \brief The IPA session configuration, immutable during the session * - * \var IPAContext::frameContext - * \brief The frame context for the frame being processed + * \var IPAContext::frameContexts + * \brief Ring buffer of the IPAFrameContext(s) * * \var IPAContext::activeState * \brief The current state of IPA algorithms - * - * \todo The frame context needs to be turned into real per-frame data storage. */ /** @@ -183,6 +181,26 @@ namespace libcamera::ipa::ipu3 { */ /** + * \brief Default constructor for IPAFrameContext + */ +IPAFrameContext::IPAFrameContext() = default; + +/** + * \brief Construct a IPAFrameContext instance + */ +IPAFrameContext::IPAFrameContext(uint32_t id, const ControlList &reqControls) + : frame(id), frameControls(reqControls) +{ + sensor = {}; +} + +/** + * \var IPAFrameContext::frame + * \brief The frame number + * + * \var IPAFrameContext::frameControls + * \brief Controls sent in by the application while queuing the request + * * \var IPAFrameContext::sensor * \brief Effective sensor values that were applied for the frame * diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 8d681131..42e11141 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -8,16 +8,22 @@ #pragma once +#include + #include #include +#include #include namespace libcamera { namespace ipa::ipu3 { +/* Maximum number of frame contexts to be held */ +static constexpr uint32_t kMaxFrameContexts = 16; + struct IPASessionConfiguration { struct { ipu3_uapi_grid_config bdsGrid; @@ -71,17 +77,23 @@ struct IPAActiveState { }; struct IPAFrameContext { + IPAFrameContext(); + IPAFrameContext(uint32_t id, const ControlList &reqControls); + struct { uint32_t exposure; double gain; } sensor; + + uint32_t frame; + ControlList frameControls; }; struct IPAContext { IPASessionConfiguration configuration; IPAActiveState activeState; - IPAFrameContext frameContext; + std::array frameContexts; }; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 16e5028f..2f6bb672 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -313,7 +313,7 @@ int IPAIPU3::init(const IPASettings &settings, } /* Clean context */ - context_ = {}; + context_.configuration = {}; context_.configuration.sensor.lineDuration = sensorInfo.lineLength * 1.0s / sensorInfo.pixelRate; /* Construct our Algorithms */ @@ -456,7 +456,8 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, /* Clean IPAActiveState at each reconfiguration. */ context_.activeState = {}; - context_.frameContext = {}; + IPAFrameContext initFrameContext; + context_.frameContexts.fill(initFrameContext); if (!validateSensorControls()) { LOG(IPAIPU3, Error) << "Sensor control validation failed."; @@ -568,15 +569,20 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); - context_.frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); - context_.frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + IPAFrameContext &frameContext = context_.frameContexts[frame % kMaxFrameContexts]; + + if (frameContext.frame != frame) + LOG(IPAIPU3, Warning) << "Frame " << frame << " does not match its frame context"; + + frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); + frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); double lineDuration = context_.configuration.sensor.lineDuration.get(); int32_t vBlank = context_.configuration.sensor.defVBlank; ControlList ctrls(controls::controls); for (auto const &algo : algorithms_) - algo->process(context_, nullptr, stats); + algo->process(context_, &frameContext, stats); setControls(frame); @@ -584,11 +590,11 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, int64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration; ctrls.set(controls::FrameDuration, frameDuration); - ctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain); + ctrls.set(controls::AnalogueGain, frameContext.sensor.gain); ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); - ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration); + ctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); /* * \todo The Metadata provides a path to getting extended data @@ -609,10 +615,10 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, * Parse the request to handle any IPA-managed controls that were set from the * application such as manual sensor settings. */ -void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame, - [[maybe_unused]] const ControlList &controls) +void IPAIPU3::queueRequest(const uint32_t frame, const ControlList &controls) { /* \todo Start processing for 'frame' based on 'controls'. */ + context_.frameContexts[frame % kMaxFrameContexts] = { frame, controls }; } /**