From patchwork Tue May 17 19:18:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15940 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 35002C0F2A for ; Tue, 17 May 2022 19:18:44 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D59166565F; Tue, 17 May 2022 21:18:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652815123; bh=W//Lquwvzs9DXejBaD2D2wdGRni2vAA0x5vajaS3jcs=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=er47iHiS4fr7LfgjdVUUf29ODAo4IZ7UprINzfo97foB395ttjmSh0bi7wCXy0jVM bWxNQhbu1LJhlsgU0xCFpRh+P8hOuhtvsmnIPvFYT1T0Oo4cxRf3gnmiAbCD29yOmA nJP9pSXElWyunbh4rRUmuC9y7K2vV4AbQbGtBc+inGZAgZLBlHYWPCTCBS3hlsZolM t7Bfcj5pAHtGtL87uEQc8tFNWrUIfke6ml6iohJO3sGGDqLgf4crcvzTiDIyFAIegj XsVtisINlxyA49/NmAt2JhyV5tWOX7TyDqHHFauSu+/UgUgIDqwVSJ0657Yr+En4Gg DP+KBOKfhTZXg== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C0FCC65656 for ; Tue, 17 May 2022 21:18:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="i3A8UJGb"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AE9D16DC; Tue, 17 May 2022 21:18:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652815121; bh=W//Lquwvzs9DXejBaD2D2wdGRni2vAA0x5vajaS3jcs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i3A8UJGbHjmDDS43g7MJOFogVC4MBYt132wqRykZlhy/0Zcx3TZpsHpLhSAxpsIEV Z/dOy8zdXXHjwqA1bKZi33grcHR04y6UOE6Fj3O+6P1gHLzkC0Km2tzeEk5/jqARyj IAC+mCl1jQYpDx1JWbf2QRKbRDYZf678Z0WzqY/A= To: libcamera-devel@lists.libcamera.org Date: Tue, 17 May 2022 21:18:31 +0200 Message-Id: <20220517191833.333122-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220517191833.333122-1-umang.jain@ideasonboard.com> References: <20220517191833.333122-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 1/3] ipa: ipu3: Rework IPAFrameContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Currently, IPAFrameContext consolidates the values computed by the active state of the algorithms, along with the values applied on the sensor. Moving ahead, we want to have a frame context associated with each incoming request (or frame to be captured). This not necessarily should be tied to the "active state" of the algorithms hence: - Rename current IPAFrameContext -> IPAActiveState This will now reflect the latest active state of the algorithms and has nothing to do with any frame-related ops/values. - Re-instate IPAFrameContext with a sub-structure 'sensor' currently storing the exposure and gain value. Adapt the various access to the frame context to the new changes as described above. Subsequently, the re-instated IPAFrameContext will be extended to contain a frame number and ControlList to remember the incoming request controls provided by the application. A ring-buffer will be introduced to store these frame contexts for a certain number of frames. Signed-off-by: Umang Jain Reviewed-by: Jacopo Mondi Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/af.cpp | 42 ++++++------ src/ipa/ipu3/algorithms/agc.cpp | 21 +++--- src/ipa/ipu3/algorithms/agc.h | 2 +- src/ipa/ipu3/algorithms/awb.cpp | 16 ++--- src/ipa/ipu3/algorithms/tone_mapping.cpp | 10 +-- src/ipa/ipu3/ipa_context.cpp | 84 ++++++++++++++---------- src/ipa/ipu3/ipa_context.h | 16 +++-- src/ipa/ipu3/ipu3.cpp | 11 ++-- 8 files changed, 110 insertions(+), 92 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index f700b01f..8a5a6b1a 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -185,11 +185,11 @@ int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo) afIgnoreFrameReset(); /* Initial focus value */ - context.frameContext.af.focus = 0; + context.activeState.af.focus = 0; /* Maximum variance of the AF statistics */ - context.frameContext.af.maxVariance = 0; + context.activeState.af.maxVariance = 0; /* The stable AF value flag. if it is true, the AF should be in a stable state. */ - context.frameContext.af.stable = false; + context.activeState.af.stable = false; return 0; } @@ -211,10 +211,10 @@ void Af::afCoarseScan(IPAContext &context) if (afScan(context, kCoarseSearchStep)) { coarseCompleted_ = true; - context.frameContext.af.maxVariance = 0; - focus_ = context.frameContext.af.focus - - (context.frameContext.af.focus * kFineRange); - context.frameContext.af.focus = focus_; + context.activeState.af.maxVariance = 0; + focus_ = context.activeState.af.focus - + (context.activeState.af.focus * kFineRange); + context.activeState.af.focus = focus_; previousVariance_ = 0; maxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)), 0U, kMaxFocusSteps); @@ -237,7 +237,7 @@ void Af::afFineScan(IPAContext &context) return; if (afScan(context, kFineSearchStep)) { - context.frameContext.af.stable = true; + context.activeState.af.stable = true; fineCompleted_ = true; } } @@ -254,10 +254,10 @@ void Af::afReset(IPAContext &context) if (afNeedIgnoreFrame()) return; - context.frameContext.af.maxVariance = 0; - context.frameContext.af.focus = 0; + context.activeState.af.maxVariance = 0; + context.activeState.af.focus = 0; focus_ = 0; - context.frameContext.af.stable = false; + context.activeState.af.stable = false; ignoreCounter_ = kIgnoreFrame; previousVariance_ = 0.0; coarseCompleted_ = false; @@ -280,7 +280,7 @@ bool Af::afScan(IPAContext &context, int min_step) { if (focus_ > maxStep_) { /* If reach the max step, move lens to the position. */ - context.frameContext.af.focus = bestFocus_; + context.activeState.af.focus = bestFocus_; return true; } else { /* @@ -288,8 +288,8 @@ bool Af::afScan(IPAContext &context, int min_step) * derivative. If the direction changes, it means we have * passed a maximum one step before. */ - if ((currentVariance_ - context.frameContext.af.maxVariance) >= - -(context.frameContext.af.maxVariance * 0.1)) { + if ((currentVariance_ - context.activeState.af.maxVariance) >= + -(context.activeState.af.maxVariance * 0.1)) { /* * Positive and zero derivative: * The variance is still increasing. The focus could be @@ -298,8 +298,8 @@ bool Af::afScan(IPAContext &context, int min_step) */ bestFocus_ = focus_; focus_ += min_step; - context.frameContext.af.focus = focus_; - context.frameContext.af.maxVariance = currentVariance_; + context.activeState.af.focus = focus_; + context.activeState.af.maxVariance = currentVariance_; } else { /* * Negative derivative: @@ -307,7 +307,7 @@ bool Af::afScan(IPAContext &context, int min_step) * variance is found. Set focus step to previous good one * then return immediately. */ - context.frameContext.af.focus = bestFocus_; + context.activeState.af.focus = bestFocus_; return true; } } @@ -389,13 +389,13 @@ double Af::afEstimateVariance(Span y_items, bool isY1) bool Af::afIsOutOfFocus(IPAContext context) { const uint32_t diff_var = std::abs(currentVariance_ - - context.frameContext.af.maxVariance); - const double var_ratio = diff_var / context.frameContext.af.maxVariance; + context.activeState.af.maxVariance); + const double var_ratio = diff_var / context.activeState.af.maxVariance; LOG(IPU3Af, Debug) << "Variance change rate: " << var_ratio << " Current VCM step: " - << context.frameContext.af.focus; + << context.activeState.af.focus; if (var_ratio > kMaxChange) return true; @@ -437,7 +437,7 @@ void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) */ currentVariance_ = afEstimateVariance(y_items, !coarseCompleted_); - if (!context.frameContext.af.stable) { + if (!context.activeState.af.stable) { afCoarseScan(context); afFineScan(context); } else { diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 7d4b3503..fdeec09d 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -87,7 +87,7 @@ int Agc::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { const IPASessionConfiguration &configuration = context.configuration; - IPAFrameContext &frameContext = context.frameContext; + IPAActiveState &activeState = context.activeState; stride_ = configuration.grid.stride; @@ -99,8 +99,8 @@ int Agc::configure(IPAContext &context, maxAnalogueGain_ = std::min(configuration.agc.maxAnalogueGain, kMaxAnalogueGain); /* Configure the default exposure and gain. */ - frameContext.agc.gain = std::max(minAnalogueGain_, kMinAnalogueGain); - frameContext.agc.exposure = 10ms / configuration.sensor.lineDuration; + activeState.agc.gain = std::max(minAnalogueGain_, kMinAnalogueGain); + activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; frameCount_ = 0; return 0; @@ -249,9 +249,10 @@ void Agc::computeExposure(IPAContext &context, double yGain, << shutterTime << " and " << stepGain; + IPAActiveState &activeState = context.activeState; /* Update the estimated exposure and gain. */ - frameContext.agc.exposure = shutterTime / configuration.sensor.lineDuration; - frameContext.agc.gain = stepGain; + activeState.agc.exposure = shutterTime / configuration.sensor.lineDuration; + activeState.agc.gain = stepGain; } /** @@ -279,7 +280,7 @@ void Agc::computeExposure(IPAContext &context, double yGain, * More detailed information can be found in: * https://en.wikipedia.org/wiki/Relative_luminance */ -double Agc::estimateLuminance(IPAFrameContext &frameContext, +double Agc::estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain) @@ -307,9 +308,9 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, * Apply the AWB gains to approximate colours correctly, use the Rec. * 601 formula to calculate the relative luminance, and normalize it. */ - double ySum = redSum * frameContext.awb.gains.red * 0.299 - + greenSum * frameContext.awb.gains.green * 0.587 - + blueSum * frameContext.awb.gains.blue * 0.114; + double ySum = redSum * activeState.awb.gains.red * 0.299 + + greenSum * activeState.awb.gains.green * 0.587 + + blueSum * activeState.awb.gains.blue * 0.114; return ySum / (grid.height * grid.width) / 255; } @@ -344,7 +345,7 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) double yTarget = kRelativeLuminanceTarget; for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(context.frameContext, + double yValue = estimateLuminance(context.activeState, context.configuration.grid.bdsGrid, stats, yGain); double extraGain = std::min(10.0, yTarget / (yValue + .001)); diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index ad705605..31420841 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -36,7 +36,7 @@ private: utils::Duration filterExposure(utils::Duration currentExposure); void computeExposure(IPAContext &context, double yGain, double iqMeanGain); - double estimateLuminance(IPAFrameContext &frameContext, + double estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 87a6cc7a..ab6924eb 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -396,10 +396,10 @@ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) * The results are cached, so if no results were calculated, we set the * cached values from asyncResults_ here. */ - context.frameContext.awb.gains.blue = asyncResults_.blueGain; - context.frameContext.awb.gains.green = asyncResults_.greenGain; - context.frameContext.awb.gains.red = asyncResults_.redGain; - context.frameContext.awb.temperatureK = asyncResults_.temperatureK; + context.activeState.awb.gains.blue = asyncResults_.blueGain; + context.activeState.awb.gains.green = asyncResults_.greenGain; + context.activeState.awb.gains.red = asyncResults_.redGain; + context.activeState.awb.temperatureK = asyncResults_.temperatureK; } constexpr uint16_t Awb::threshold(float value) @@ -450,10 +450,10 @@ void Awb::prepare(IPAContext &context, ipu3_uapi_params *params) params->acc_param.bnr.opt_center_sqr.y_sqr_reset = params->acc_param.bnr.opt_center.y_reset * params->acc_param.bnr.opt_center.y_reset; /* Convert to u3.13 fixed point values */ - params->acc_param.bnr.wb_gains.gr = 8192 * context.frameContext.awb.gains.green; - params->acc_param.bnr.wb_gains.r = 8192 * context.frameContext.awb.gains.red; - params->acc_param.bnr.wb_gains.b = 8192 * context.frameContext.awb.gains.blue; - params->acc_param.bnr.wb_gains.gb = 8192 * context.frameContext.awb.gains.green; + params->acc_param.bnr.wb_gains.gr = 8192 * context.activeState.awb.gains.green; + params->acc_param.bnr.wb_gains.r = 8192 * context.activeState.awb.gains.red; + params->acc_param.bnr.wb_gains.b = 8192 * context.activeState.awb.gains.blue; + params->acc_param.bnr.wb_gains.gb = 8192 * context.activeState.awb.gains.green; LOG(IPU3Awb, Debug) << "Color temperature estimated: " << asyncResults_.temperatureK; diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 2040eda5..7c78d0d9 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -42,7 +42,7 @@ int ToneMapping::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { /* Initialise tone mapping gamma value. */ - context.frameContext.toneMapping.gamma = 0.0; + context.activeState.toneMapping.gamma = 0.0; return 0; } @@ -60,7 +60,7 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, { /* Copy the calculated LUT into the parameters buffer. */ memcpy(params->acc_param.gamma.gc_lut.lut, - context.frameContext.toneMapping.gammaCorrection.lut, + context.activeState.toneMapping.gammaCorrection.lut, IPU3_UAPI_GAMMA_CORR_LUT_ENTRIES * sizeof(params->acc_param.gamma.gc_lut.lut[0])); @@ -87,11 +87,11 @@ void ToneMapping::process(IPAContext &context, */ gamma_ = 1.1; - if (context.frameContext.toneMapping.gamma == gamma_) + if (context.activeState.toneMapping.gamma == gamma_) return; struct ipu3_uapi_gamma_corr_lut &lut = - context.frameContext.toneMapping.gammaCorrection; + context.activeState.toneMapping.gammaCorrection; for (uint32_t i = 0; i < std::size(lut.lut); i++) { double j = static_cast(i) / (std::size(lut.lut) - 1); @@ -101,7 +101,7 @@ void ToneMapping::process(IPAContext &context, lut.lut[i] = gamma * 8191; } - context.frameContext.toneMapping.gamma = gamma_; + context.activeState.toneMapping.gamma = gamma_; } } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index b1570dde..06eb2776 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -24,9 +24,20 @@ namespace libcamera::ipa::ipu3 { * may also be updated in the start() operation. */ +/** + * \struct IPAActiveState + * \brief The active state of the IPA algorithms + * + * The IPA is fed with the statistics generated from the latest frame captured + * by the hardware. The statistics are then processed by the IPA algorithms to + * compute ISP parameters required for the next frame capture. The current state + * of the algorithms is reflected through IPAActiveState. The latest computed + * values by the IPA algorithms are stored in this structure. + */ + /** * \struct IPAFrameContext - * \brief Per-frame context for algorithms + * \brief Context for a frame * * The frame context stores data specific to a single frame processed by the * IPA. Each frame processed by the IPA has a context associated with it, @@ -34,9 +45,10 @@ namespace libcamera::ipa::ipu3 { * * \todo Detail how to access contexts for a particular frame * - * Each of the fields in the frame context belongs to either a specific - * algorithm, or to the top-level IPA module. A field may be read by any - * algorithm, but should only be written by its owner. + * Fields in the frame context should reflect the values with which the frame + * was processed on the hardware ISP. A field may be read by any algorithm as + * and when required, or can be used to prepare the metadata container of the + * frame. */ /** @@ -49,10 +61,10 @@ namespace libcamera::ipa::ipu3 { * \var IPAContext::frameContext * \brief The frame context for the frame being processed * - * \todo While the frame context is supposed to be per-frame, this - * single frame context stores data related to both the current frame - * and the previous frames, with fields being updated as the algorithms - * are run. This needs to be turned into real per-frame data storage. + * \var IPAContext::activeState + * \brief The current state of IPA algorithms + * + * \todo The frame context needs to be turned into real per-frame data storage. */ /** @@ -78,17 +90,17 @@ namespace libcamera::ipa::ipu3 { */ /** - * \var IPAFrameContext::af + * \var IPAActiveState::af * \brief Context for the Automatic Focus algorithm * - * \struct IPAFrameContext::af - * \var IPAFrameContext::af.focus + * \struct IPAActiveState::af + * \var IPAActiveState::af.focus * \brief Current position of the lens * - * \var IPAFrameContext::af.maxVariance + * \var IPAActiveState::af.maxVariance * \brief The maximum variance of the current image. * - * \var IPAFrameContext::af.stable + * \var IPAActiveState::af.stable * \brief It is set to true, if the best focus is found. */ @@ -121,64 +133,64 @@ namespace libcamera::ipa::ipu3 { */ /** - * \var IPAFrameContext::agc + * \var IPAActiveState::agc * \brief Context for the Automatic Gain Control algorithm * * The exposure and gain determined are expected to be applied to the sensor * at the earliest opportunity. * - * \var IPAFrameContext::agc.exposure + * \var IPAActiveState::agc.exposure * \brief Exposure time expressed as a number of lines * - * \var IPAFrameContext::agc.gain + * \var IPAActiveState::agc.gain * \brief Analogue gain multiplier * * The gain should be adapted to the sensor specific gain code before applying. */ /** - * \var IPAFrameContext::awb + * \var IPAActiveState::awb * \brief Context for the Automatic White Balance algorithm * - * \struct IPAFrameContext::awb.gains + * \struct IPAActiveState::awb.gains * \brief White balance gains * - * \var IPAFrameContext::awb.gains.red + * \var IPAActiveState::awb.gains.red * \brief White balance gain for R channel * - * \var IPAFrameContext::awb.gains.green + * \var IPAActiveState::awb.gains.green * \brief White balance gain for G channel * - * \var IPAFrameContext::awb.gains.blue + * \var IPAActiveState::awb.gains.blue * \brief White balance gain for B channel * - * \var IPAFrameContext::awb.temperatureK + * \var IPAActiveState::awb.temperatureK * \brief Estimated color temperature */ /** - * \var IPAFrameContext::sensor - * \brief Effective sensor values - * - * \var IPAFrameContext::sensor.exposure - * \brief Exposure time expressed as a number of lines - * - * \var IPAFrameContext::sensor.gain - * \brief Analogue gain multiplier - */ - -/** - * \var IPAFrameContext::toneMapping + * \var IPAActiveState::toneMapping * \brief Context for ToneMapping and Gamma control * - * \var IPAFrameContext::toneMapping.gamma + * \var IPAActiveState::toneMapping.gamma * \brief Gamma value for the LUT * - * \var IPAFrameContext::toneMapping.gammaCorrection + * \var IPAActiveState::toneMapping.gammaCorrection * \brief Per-pixel tone mapping implemented as a LUT * * The LUT structure is defined by the IPU3 kernel interface. See * struct ipu3_uapi_gamma_corr_lut for further details. */ +/** + * \var IPAFrameContext::sensor + * \brief Effective sensor values that were applied for the frame + * + * \var IPAFrameContext::sensor.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::sensor.gain + * \brief Analogue gain multiplier + */ + } /* namespace libcamera::ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 103498ef..8d681131 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -42,7 +42,7 @@ struct IPASessionConfiguration { } sensor; }; -struct IPAFrameContext { +struct IPAActiveState { struct { uint32_t focus; double maxVariance; @@ -64,19 +64,23 @@ struct IPAFrameContext { double temperatureK; } awb; - struct { - uint32_t exposure; - double gain; - } sensor; - struct { double gamma; struct ipu3_uapi_gamma_corr_lut gammaCorrection; } toneMapping; }; +struct IPAFrameContext { + struct { + uint32_t exposure; + double gain; + } sensor; +}; + struct IPAContext { IPASessionConfiguration configuration; + IPAActiveState activeState; + IPAFrameContext frameContext; }; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index dd6cfd79..3b4fc911 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -454,7 +454,8 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, calculateBdsGrid(configInfo.bdsOutputSize); - /* Clean frameContext at each reconfiguration. */ + /* Clean IPAActiveState at each reconfiguration. */ + context_.activeState = {}; context_.frameContext = {}; if (!validateSensorControls()) { @@ -585,7 +586,7 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, ctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain); - ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration); @@ -623,8 +624,8 @@ void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame, */ void IPAIPU3::setControls(unsigned int frame) { - int32_t exposure = context_.frameContext.agc.exposure; - int32_t gain = camHelper_->gainCode(context_.frameContext.agc.gain); + int32_t exposure = context_.activeState.agc.exposure; + int32_t gain = camHelper_->gainCode(context_.activeState.agc.gain); ControlList ctrls(sensorCtrls_); ctrls.set(V4L2_CID_EXPOSURE, exposure); @@ -632,7 +633,7 @@ void IPAIPU3::setControls(unsigned int frame) ControlList lensCtrls(lensCtrls_); lensCtrls.set(V4L2_CID_FOCUS_ABSOLUTE, - static_cast(context_.frameContext.af.focus)); + static_cast(context_.activeState.af.focus)); setSensorControls.emit(frame, ctrls, lensCtrls); } From patchwork Tue May 17 19:18:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15941 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 26B46C0F2A for ; Tue, 17 May 2022 19:18:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AF7A965660; Tue, 17 May 2022 21:18:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652815125; bh=27si9dP945uWLD3Hwhd2DzgcUm0eVf4husfdps57WQQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=iYk+qCOwET3T9LTSjTB2gkC0gw6SkzGt6viEN8k/Hp4r1dE8WtDlB5Gkmc8Jv7AD/ dr2pCOfDr5tftXgxhJybDnLn7yCLcp6wcEbJP+T2MjALKfzZokTsBTAExRtlyz55Zs l64X45hEYTJ59Cs8xcsL5DtFP6gtmmfTO41nTUMGL4OFXcIceY7yNSf+VHNrLLFk67 zelwy9ZnjMf0dWl5kHH5YdHpTeRftwBZTC6anGndv0Irilmw7ALAjwNr2RTi+8i5T3 lXFRUJUhNwlkQ5wWaIcdqDWJ0r+Xn/vlP8h0yBww6Cwaax8Nus7VRFyCyoV0sHx5Yf oZBkwiHWVvTUA== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 850256041E for ; Tue, 17 May 2022 21:18:43 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NQeMyG50"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 552ED8E6; Tue, 17 May 2022 21:18:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652815123; bh=27si9dP945uWLD3Hwhd2DzgcUm0eVf4husfdps57WQQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NQeMyG50aXpxCJfNN5Tk8OuzxVBIGXwg9CAWnQckUeeHo7Xlikd1WeevSiOsEOqnC BxOmZjXoMvnRg2ZMi2S7KaVyBevMTwOnaH2nTtL/JvCAMGgQEQryQ1i/RWevw/iBXr /c07xScheAyYhZ3aS9IFSq2DXbSP3qKr4tujC8BA= To: libcamera-devel@lists.libcamera.org Date: Tue, 17 May 2022 21:18:32 +0200 Message-Id: <20220517191833.333122-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220517191833.333122-1-umang.jain@ideasonboard.com> References: <20220517191833.333122-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 2/3] ipa: libipa: Add frame context pointer in process() X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Currently we have a single structure of IPAFrameContext but subsequently, we shall have a ring buffer (or similar) container to keep IPAFrameContext structures for each frame. It would be a hassle to query out the frame context required for process() (since they will reside in a ring buffer) by the IPA for each process. Hence, prepare the process() libipa template to accept a particular IPAFrameContext early on. As for this patch, we shall pass in the pointer as nullptr, so that the changes compile and keep working as-is. Signed-off-by: Umang Jain Reviewed-by: Jacopo Mondi Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/af.cpp | 4 +++- src/ipa/ipu3/algorithms/af.h | 3 ++- src/ipa/ipu3/algorithms/agc.cpp | 4 +++- src/ipa/ipu3/algorithms/agc.h | 3 ++- src/ipa/ipu3/algorithms/algorithm.h | 4 +++- src/ipa/ipu3/algorithms/awb.cpp | 3 ++- src/ipa/ipu3/algorithms/awb.h | 3 ++- src/ipa/ipu3/algorithms/tone_mapping.cpp | 3 ++- src/ipa/ipu3/algorithms/tone_mapping.h | 3 ++- src/ipa/ipu3/ipu3.cpp | 2 +- src/ipa/libipa/algorithm.cpp | 1 + src/ipa/libipa/algorithm.h | 4 +++- src/ipa/rkisp1/algorithms/agc.cpp | 4 +++- src/ipa/rkisp1/algorithms/agc.h | 3 ++- src/ipa/rkisp1/algorithms/algorithm.h | 4 +++- src/ipa/rkisp1/algorithms/awb.cpp | 4 +++- src/ipa/rkisp1/algorithms/awb.h | 3 ++- src/ipa/rkisp1/rkisp1.cpp | 2 +- 18 files changed, 40 insertions(+), 17 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index 8a5a6b1a..d07521a0 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -406,6 +406,7 @@ bool Af::afIsOutOfFocus(IPAContext context) /** * \brief Determine the max contrast image and lens position. * \param[in] context The IPA context. + * \param[in] frameContext The current frame context * \param[in] stats The statistics buffer of IPU3. * * Ideally, a clear image also has a relatively higher contrast. So, every @@ -419,7 +420,8 @@ bool Af::afIsOutOfFocus(IPAContext context) * * [1] Hill Climbing Algorithm, https://en.wikipedia.org/wiki/Hill_climbing */ -void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +void Af::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) { /* Evaluate the AF buffer length */ uint32_t afRawBufferLen = context.configuration.af.afGrid.width * diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h index b85cf941..ccf015f3 100644 --- a/src/ipa/ipu3/algorithms/af.h +++ b/src/ipa/ipu3/algorithms/af.h @@ -32,7 +32,8 @@ public: void prepare(IPAContext &context, ipu3_uapi_params *params) override; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: void afCoarseScan(IPAContext &context); diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index fdeec09d..383a8deb 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -318,12 +318,14 @@ double Agc::estimateLuminance(IPAActiveState &activeState, /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context + * \param[in] frameContext The current frame context * \param[in] stats The IPU3 statistics and ISP results * * Identify the current image brightness, and use that to estimate the optimal * new exposure and gain for the scene. */ -void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +void Agc::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) { /* * Estimate the gain needed to have the proportion of pixels in a given diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 31420841..219a1a96 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -28,7 +28,8 @@ public: ~Agc() = default; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: double measureBrightness(const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/algorithms/algorithm.h b/src/ipa/ipu3/algorithms/algorithm.h index d2eecc78..234b2bd7 100644 --- a/src/ipa/ipu3/algorithms/algorithm.h +++ b/src/ipa/ipu3/algorithms/algorithm.h @@ -17,7 +17,9 @@ namespace libcamera { namespace ipa::ipu3 { -using Algorithm = libcamera::ipa::Algorithm; +using Algorithm = libcamera::ipa::Algorithm; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index ab6924eb..5c232d92 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -387,7 +387,8 @@ void Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats) /** * \copydoc libcamera::ipa::Algorithm::process */ -void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +void Awb::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) { calculateWBGains(stats); diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index ab4b0a33..9a50a985 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -40,7 +40,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, ipu3_uapi_params *params) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: /* \todo Make these structs available to all the ISPs ? */ diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 7c78d0d9..f86e79b2 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -72,12 +72,13 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, /** * \brief Calculate the tone mapping look up table * \param context The shared IPA context + * \param frameContext The current frame context * \param stats The IPU3 statistics and ISP results * * The tone mapping look up table is generated as an inverse power curve from * our gamma setting. */ -void ToneMapping::process(IPAContext &context, +void ToneMapping::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameContext, [[maybe_unused]] const ipu3_uapi_stats_3a *stats) { /* diff --git a/src/ipa/ipu3/algorithms/tone_mapping.h b/src/ipa/ipu3/algorithms/tone_mapping.h index b727ab1e..d7d48006 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.h +++ b/src/ipa/ipu3/algorithms/tone_mapping.h @@ -20,7 +20,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, ipu3_uapi_params *params) override; - void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const ipu3_uapi_stats_3a *stats) override; private: double gamma_; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 3b4fc911..16e5028f 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -576,7 +576,7 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, ControlList ctrls(controls::controls); for (auto const &algo : algorithms_) - algo->process(context_, stats); + algo->process(context_, nullptr, stats); setControls(frame); diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp index 398d5372..cce2ed62 100644 --- a/src/ipa/libipa/algorithm.cpp +++ b/src/ipa/libipa/algorithm.cpp @@ -64,6 +64,7 @@ namespace ipa { * \fn Algorithm::process() * \brief Process ISP statistics, and run algorithm operations * \param[in] context The shared IPA context + * \param[in] frameContext The current frame's context * \param[in] stats The IPA statistics and ISP results * * This function is called while camera is running for every frame processed by diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h index 766aee5d..032a05b5 100644 --- a/src/ipa/libipa/algorithm.h +++ b/src/ipa/libipa/algorithm.h @@ -10,7 +10,8 @@ namespace libcamera { namespace ipa { -template +template class Algorithm { public: @@ -28,6 +29,7 @@ public: } virtual void process([[maybe_unused]] Context &context, + [[maybe_unused]] FrameContext *frameContext, [[maybe_unused]] const Stats *stats) { } diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 5f4c3f93..b5a184d9 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -280,7 +280,9 @@ double Agc::measureBrightness(const rkisp1_cif_isp_hist_stat *hist) const * Identify the current image brightness, and use that to estimate the optimal * new exposure and gain for the scene. */ -void Agc::process(IPAContext &context, const rkisp1_stat_buffer *stats) +void Agc::process(IPAContext &context, + [[maybe_unused]] IPAFrameContext *frameContext, + const rkisp1_stat_buffer *stats) { const rkisp1_cif_isp_stat *params = &stats->params; ASSERT(stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP); diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index ce1adf27..22c02779 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -29,7 +29,8 @@ public: int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; void prepare(IPAContext &context, rkisp1_params_cfg *params) override; - void process(IPAContext &context, const rkisp1_stat_buffer *stats) override; + void process(IPAContext &context, IPAFrameContext *frameContext, + const rkisp1_stat_buffer *stats) override; private: void computeExposure(IPAContext &Context, double yGain, double iqMeanGain); diff --git a/src/ipa/rkisp1/algorithms/algorithm.h b/src/ipa/rkisp1/algorithms/algorithm.h index d46c3188..68e3a44e 100644 --- a/src/ipa/rkisp1/algorithms/algorithm.h +++ b/src/ipa/rkisp1/algorithms/algorithm.h @@ -19,7 +19,9 @@ namespace libcamera { namespace ipa::rkisp1 { -using Algorithm = libcamera::ipa::Algorithm; +using Algorithm = libcamera::ipa::Algorithm; } /* namespace ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index be4585c6..88441382 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -119,7 +119,9 @@ void Awb::prepare(IPAContext &context, rkisp1_params_cfg *params) /** * \copydoc libcamera::ipa::Algorithm::process */ -void Awb::process([[maybe_unused]] IPAContext &context, const rkisp1_stat_buffer *stats) +void Awb::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] IPAFrameContext *frameCtx, + const rkisp1_stat_buffer *stats) { const rkisp1_cif_isp_stat *params = &stats->params; const rkisp1_cif_isp_awb_stat *awb = ¶ms->awb; diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h index 11946643..7647842f 100644 --- a/src/ipa/rkisp1/algorithms/awb.h +++ b/src/ipa/rkisp1/algorithms/awb.h @@ -23,7 +23,8 @@ public: int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; void prepare(IPAContext &context, rkisp1_params_cfg *params) override; - void process(IPAContext &context, const rkisp1_stat_buffer *stats) override; + void process(IPAContext &context, IPAFrameContext *frameCtx, + const rkisp1_stat_buffer *stats) override; private: uint32_t estimateCCT(double red, double green, double blue); diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index ef1f0d56..c818a6d7 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -272,7 +272,7 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId unsigned int aeState = 0; for (auto const &algo : algorithms_) - algo->process(context_, stats); + algo->process(context_, nullptr, stats); setControls(frame); From patchwork Tue May 17 19:18:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15942 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C595CC326C for ; Tue, 17 May 2022 19:18:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 66BB865656; Tue, 17 May 2022 21:18:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652815127; bh=nu9bMmutfOGADBC/V+8yMMZu5RoW/gkXUZiQCMWjPdI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=3DA4Pp8CMAy8EeD4pj9PwD0k4LdoKyK690/+fTnAu0qf2uYn+ooO0AdcQcOSTs+1o tusMjyfORigvEtPUbV3+KOaSSEexcwn9Kf865WKzQ95gIKaTsH6H7EzXwcUO0ZpaL/ E19q2d0BkLshWeJU34Ah2g4+EXkQakwk7RgZfcmKCadUkdoA+hmIVIfmeueTWQUrOQ H1eMdq6iNMbas3RzJrJx0j65KF9+SsnujSOG7YvtlqqDRiS/Kh4p5LN9TIVpRlMnZT YuZmY2vZc6Fla6jbe6P/llW41qYWdRaqnZvHvmmyrm8SnfV3l8CWI/mIHeZ/JFYD7q cBktgEKSvt0+g== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 60BD06565C for ; Tue, 17 May 2022 21:18:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="SmT83e1h"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 41E806DC; Tue, 17 May 2022 21:18:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652815125; bh=nu9bMmutfOGADBC/V+8yMMZu5RoW/gkXUZiQCMWjPdI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SmT83e1hvtx0TTFyAwv5grcTCLm0ny+4g5hB1Gh3asVj/feqIarcz40snznKz+t7l j32xFTs/TBC8q6xu7yolqKwqO76SazxHgi5E7szBCsc34feW1mdir7Wx4bwHoZTmJq WXAE9WILLuxiqGLCOr9qnwj8i42U8oxclVM0AV/U= To: libcamera-devel@lists.libcamera.org Date: Tue, 17 May 2022 21:18:33 +0200 Message-Id: <20220517191833.333122-4-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220517191833.333122-1-umang.jain@ideasonboard.com> References: <20220517191833.333122-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 3/3] ipa: ipu3: Put IPAFrameContext(s) in a ring buffer X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Instead of having one frame context constantly being updated, this patch aims to introduce per-frame IPAFrameContext which are stored in a ring buffer. Whenever a request is queued, a new IPAFrameContext is created and inserted into the ring buffer. The IPAFrameContext structure itself has been slightly extended to store a frame id and a ControlList for incoming frame controls (sent in by the application). The next step would be to read and set these controls whenever the request is actually queued to the hardware. Since now we are working in multiples of IPAFrameContext, the Algorithm::process() will actually take in a IPAFrameContext pointer (as opposed to a nullptr while preparing for this change). Signed-off-by: Umang Jain Reviewed-by: Jacopo Mondi Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 11 +++++------ src/ipa/ipu3/algorithms/agc.h | 4 ++-- src/ipa/ipu3/ipa_context.cpp | 24 ++++++++++++++++++++++-- src/ipa/ipu3/ipa_context.h | 14 +++++++++++++- src/ipa/ipu3/ipu3.cpp | 22 +++++++++++++--------- 5 files changed, 55 insertions(+), 20 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 383a8deb..f16be534 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -183,14 +183,13 @@ utils::Duration Agc::filterExposure(utils::Duration exposureValue) * \param[in] yGain The gain calculated based on the relative luminance target * \param[in] iqMeanGain The gain calculated based on the relative luminance target */ -void Agc::computeExposure(IPAContext &context, double yGain, - double iqMeanGain) +void Agc::computeExposure(IPAContext &context, IPAFrameContext *frameContext, + double yGain, double iqMeanGain) { const IPASessionConfiguration &configuration = context.configuration; - IPAFrameContext &frameContext = context.frameContext; /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; + uint32_t exposure = frameContext->sensor.exposure; + double analogueGain = frameContext->sensor.gain; /* Use the highest of the two gain estimates. */ double evGain = std::max(yGain, iqMeanGain); @@ -360,7 +359,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameCo break; } - computeExposure(context, yGain, iqMeanGain); + computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 219a1a96..105ae0f2 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -35,8 +35,8 @@ private: double measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) const; utils::Duration filterExposure(utils::Duration currentExposure); - void computeExposure(IPAContext &context, double yGain, - double iqMeanGain); + void computeExposure(IPAContext &context, IPAFrameContext *frameContext, + double yGain, double iqMeanGain); double estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 06eb2776..81b30600 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -58,8 +58,8 @@ namespace libcamera::ipa::ipu3 { * \var IPAContext::configuration * \brief The IPA session configuration, immutable during the session * - * \var IPAContext::frameContext - * \brief The frame context for the frame being processed + * \var IPAContext::frameContexts + * \brief Ring buffer of the IPAFrameContext(s) * * \var IPAContext::activeState * \brief The current state of IPA algorithms @@ -183,6 +183,26 @@ namespace libcamera::ipa::ipu3 { */ /** + * \brief Default constructor for IPAFrameContext + */ +IPAFrameContext::IPAFrameContext() = default; + +/** + * \brief Construct a IPAFrameContext instance + */ +IPAFrameContext::IPAFrameContext(uint32_t id, const ControlList &reqControls) + : frame(id), frameControls(reqControls) +{ + sensor = {}; +} + +/** + * \var IPAFrameContext::frame + * \brief The frame number + * + * \var IPAFrameContext::frameControls + * \brief Controls sent in by the application while queuing the request + * * \var IPAFrameContext::sensor * \brief Effective sensor values that were applied for the frame * diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 8d681131..42e11141 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -8,16 +8,22 @@ #pragma once +#include + #include #include +#include #include namespace libcamera { namespace ipa::ipu3 { +/* Maximum number of frame contexts to be held */ +static constexpr uint32_t kMaxFrameContexts = 16; + struct IPASessionConfiguration { struct { ipu3_uapi_grid_config bdsGrid; @@ -71,17 +77,23 @@ struct IPAActiveState { }; struct IPAFrameContext { + IPAFrameContext(); + IPAFrameContext(uint32_t id, const ControlList &reqControls); + struct { uint32_t exposure; double gain; } sensor; + + uint32_t frame; + ControlList frameControls; }; struct IPAContext { IPASessionConfiguration configuration; IPAActiveState activeState; - IPAFrameContext frameContext; + std::array frameContexts; }; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 16e5028f..4322f96b 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -313,7 +313,7 @@ int IPAIPU3::init(const IPASettings &settings, } /* Clean context */ - context_ = {}; + context_.configuration = {}; context_.configuration.sensor.lineDuration = sensorInfo.lineLength * 1.0s / sensorInfo.pixelRate; /* Construct our Algorithms */ @@ -456,7 +456,8 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, /* Clean IPAActiveState at each reconfiguration. */ context_.activeState = {}; - context_.frameContext = {}; + IPAFrameContext initFrameContext; + context_.frameContexts.fill(initFrameContext); if (!validateSensorControls()) { LOG(IPAIPU3, Error) << "Sensor control validation failed."; @@ -568,15 +569,17 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); - context_.frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); - context_.frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + IPAFrameContext &frameContext = context_.frameContexts[frame % kMaxFrameContexts]; + + frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); + frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); double lineDuration = context_.configuration.sensor.lineDuration.get(); int32_t vBlank = context_.configuration.sensor.defVBlank; ControlList ctrls(controls::controls); for (auto const &algo : algorithms_) - algo->process(context_, nullptr, stats); + algo->process(context_, &frameContext, stats); setControls(frame); @@ -584,11 +587,11 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, int64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration; ctrls.set(controls::FrameDuration, frameDuration); - ctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain); + ctrls.set(controls::AnalogueGain, frameContext.sensor.gain); ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); - ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration); + ctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); /* * \todo The Metadata provides a path to getting extended data @@ -609,10 +612,11 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, * Parse the request to handle any IPA-managed controls that were set from the * application such as manual sensor settings. */ -void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame, - [[maybe_unused]] const ControlList &controls) +void IPAIPU3::queueRequest(const uint32_t frame, const ControlList &controls) { /* \todo Start processing for 'frame' based on 'controls'. */ + IPAFrameContext newFrameContext(frame, controls); + context_.frameContexts[frame % kMaxFrameContexts] = newFrameContext; } /**