From patchwork Wed May 18 13:10:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 15949 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 44479C3256 for ; Wed, 18 May 2022 13:10:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EC3E965656; Wed, 18 May 2022 15:10:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652879447; bh=NxHEyprEXwaTkMJhzvYAxM5BWeA7pKN9M4xNmoOnloo=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=NGIMlIK1EVnHmtAf0E+kofw5m15SAjDO1mJWfHNeqCW5O6UXetX7cMjxnLQ79Rpa8 s4G8aYyeNRkgC8VWYU1wv2HAxB3lOK/SL1dF4+Gf537UcGVXziByq8RtlB3lPzvpEC LYiiCh0XpPNB1fGaAcuXfoo5VVxyB9E3F8oWFJvrlHHSUMoLy7v0c19Koq/e7dAgDC 6isxmds79wfpzNZKnari77MnI7EBwJXEuihCzUdAYDCbRUW7X79Hos7iHeLIebcCoB uk+MNGdPkGvC5tx5UxI7MCucPu6LH0RpM1Fz16sQVV/LLGPvoIv/QbZ93X4ND6ToZm 2ksq84txqj1iQ== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ECC2B65661 for ; Wed, 18 May 2022 15:10:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="reST8H10"; dkim-atps=neutral Received: from perceval.ideasonboard.com (unknown [45.131.31.124]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 46DE4E50; Wed, 18 May 2022 15:10:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1652879442; bh=NxHEyprEXwaTkMJhzvYAxM5BWeA7pKN9M4xNmoOnloo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=reST8H10pSJ8d+TIfP+mXRx/pu9MwxmPO7WrEXjdnjVH2D28RehAYeXWd2bsFKR3A Qg5rlVU+0ZM0MUPMNekbIQiMMGw3Y3iozMw8aByPBxcV/q1G0zyke4+4EFlh9/Wass oLzoiwaLl6ntY46NmsMmF50rWcQkngLDPb5NbRhs= To: libcamera-devel@lists.libcamera.org Date: Wed, 18 May 2022 15:10:30 +0200 Message-Id: <20220518131030.421225-4-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220518131030.421225-1-umang.jain@ideasonboard.com> References: <20220518131030.421225-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 3/3] ipa: ipu3: Put IPAFrameContext(s) in a ring buffer X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Instead of having one frame context constantly being updated, this patch aims to introduce per-frame IPAFrameContext which are stored in a ring buffer. Whenever a request is queued, a new IPAFrameContext is created and inserted into the ring buffer. The IPAFrameContext structure itself has been slightly extended to store a frame id and a ControlList for incoming frame controls (sent in by the application). The next step would be to read and set these controls whenever the request is actually queued to the hardware. Since now we are working in multiples of IPAFrameContext, the Algorithm::process() will actually take in a IPAFrameContext pointer (as opposed to a nullptr while preparing for this change). Signed-off-by: Umang Jain Reviewed-by: Jacopo Mondi Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 11 +++++------ src/ipa/ipu3/algorithms/agc.h | 4 ++-- src/ipa/ipu3/ipa_context.cpp | 26 ++++++++++++++++++++++---- src/ipa/ipu3/ipa_context.h | 14 +++++++++++++- src/ipa/ipu3/ipu3.cpp | 24 +++++++++++++++--------- 5 files changed, 57 insertions(+), 22 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 383a8deb..f16be534 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -183,14 +183,13 @@ utils::Duration Agc::filterExposure(utils::Duration exposureValue) * \param[in] yGain The gain calculated based on the relative luminance target * \param[in] iqMeanGain The gain calculated based on the relative luminance target */ -void Agc::computeExposure(IPAContext &context, double yGain, - double iqMeanGain) +void Agc::computeExposure(IPAContext &context, IPAFrameContext *frameContext, + double yGain, double iqMeanGain) { const IPASessionConfiguration &configuration = context.configuration; - IPAFrameContext &frameContext = context.frameContext; /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; + uint32_t exposure = frameContext->sensor.exposure; + double analogueGain = frameContext->sensor.gain; /* Use the highest of the two gain estimates. */ double evGain = std::max(yGain, iqMeanGain); @@ -360,7 +359,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameCo break; } - computeExposure(context, yGain, iqMeanGain); + computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 219a1a96..105ae0f2 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -35,8 +35,8 @@ private: double measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) const; utils::Duration filterExposure(utils::Duration currentExposure); - void computeExposure(IPAContext &context, double yGain, - double iqMeanGain); + void computeExposure(IPAContext &context, IPAFrameContext *frameContext, + double yGain, double iqMeanGain); double estimateLuminance(IPAActiveState &activeState, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 383c2e37..13cdb835 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -58,13 +58,11 @@ namespace libcamera::ipa::ipu3 { * \var IPAContext::configuration * \brief The IPA session configuration, immutable during the session * - * \var IPAContext::frameContext - * \brief The frame context for the frame being processed + * \var IPAContext::frameContexts + * \brief Ring buffer of the IPAFrameContext(s) * * \var IPAContext::activeState * \brief The current state of IPA algorithms - * - * \todo The frame context needs to be turned into real per-frame data storage. */ /** @@ -183,6 +181,26 @@ namespace libcamera::ipa::ipu3 { */ /** + * \brief Default constructor for IPAFrameContext + */ +IPAFrameContext::IPAFrameContext() = default; + +/** + * \brief Construct a IPAFrameContext instance + */ +IPAFrameContext::IPAFrameContext(uint32_t id, const ControlList &reqControls) + : frame(id), frameControls(reqControls) +{ + sensor = {}; +} + +/** + * \var IPAFrameContext::frame + * \brief The frame number + * + * \var IPAFrameContext::frameControls + * \brief Controls sent in by the application while queuing the request + * * \var IPAFrameContext::sensor * \brief Effective sensor values that were applied for the frame * diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 8d681131..42e11141 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -8,16 +8,22 @@ #pragma once +#include + #include #include +#include #include namespace libcamera { namespace ipa::ipu3 { +/* Maximum number of frame contexts to be held */ +static constexpr uint32_t kMaxFrameContexts = 16; + struct IPASessionConfiguration { struct { ipu3_uapi_grid_config bdsGrid; @@ -71,17 +77,23 @@ struct IPAActiveState { }; struct IPAFrameContext { + IPAFrameContext(); + IPAFrameContext(uint32_t id, const ControlList &reqControls); + struct { uint32_t exposure; double gain; } sensor; + + uint32_t frame; + ControlList frameControls; }; struct IPAContext { IPASessionConfiguration configuration; IPAActiveState activeState; - IPAFrameContext frameContext; + std::array frameContexts; }; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 16e5028f..2f6bb672 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -313,7 +313,7 @@ int IPAIPU3::init(const IPASettings &settings, } /* Clean context */ - context_ = {}; + context_.configuration = {}; context_.configuration.sensor.lineDuration = sensorInfo.lineLength * 1.0s / sensorInfo.pixelRate; /* Construct our Algorithms */ @@ -456,7 +456,8 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, /* Clean IPAActiveState at each reconfiguration. */ context_.activeState = {}; - context_.frameContext = {}; + IPAFrameContext initFrameContext; + context_.frameContexts.fill(initFrameContext); if (!validateSensorControls()) { LOG(IPAIPU3, Error) << "Sensor control validation failed."; @@ -568,15 +569,20 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); - context_.frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); - context_.frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + IPAFrameContext &frameContext = context_.frameContexts[frame % kMaxFrameContexts]; + + if (frameContext.frame != frame) + LOG(IPAIPU3, Warning) << "Frame " << frame << " does not match its frame context"; + + frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); + frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); double lineDuration = context_.configuration.sensor.lineDuration.get(); int32_t vBlank = context_.configuration.sensor.defVBlank; ControlList ctrls(controls::controls); for (auto const &algo : algorithms_) - algo->process(context_, nullptr, stats); + algo->process(context_, &frameContext, stats); setControls(frame); @@ -584,11 +590,11 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, int64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration; ctrls.set(controls::FrameDuration, frameDuration); - ctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain); + ctrls.set(controls::AnalogueGain, frameContext.sensor.gain); ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); - ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration); + ctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); /* * \todo The Metadata provides a path to getting extended data @@ -609,10 +615,10 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, * Parse the request to handle any IPA-managed controls that were set from the * application such as manual sensor settings. */ -void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame, - [[maybe_unused]] const ControlList &controls) +void IPAIPU3::queueRequest(const uint32_t frame, const ControlList &controls) { /* \todo Start processing for 'frame' based on 'controls'. */ + context_.frameContexts[frame % kMaxFrameContexts] = { frame, controls }; } /**