{"id":15942,"url":"https://patchwork.libcamera.org/api/patches/15942/?format=json","web_url":"https://patchwork.libcamera.org/patch/15942/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220517191833.333122-4-umang.jain@ideasonboard.com>","date":"2022-05-17T19:18:33","name":"[libcamera-devel,v3,3/3] ipa: ipu3: Put IPAFrameContext(s) in a ring buffer","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"9d839598a31a39124a16e3f66b0f6ab2d5406426","submitter":{"id":86,"url":"https://patchwork.libcamera.org/api/people/86/?format=json","name":"Umang Jain","email":"umang.jain@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/15942/mbox/","series":[{"id":3118,"url":"https://patchwork.libcamera.org/api/series/3118/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3118","date":"2022-05-17T19:18:30","name":"IPAFrameContext Ring buffer","version":3,"mbox":"https://patchwork.libcamera.org/series/3118/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/15942/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/15942/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C595CC326C\n\tfor ;\n\tTue, 17 May 2022 19:18:47 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 66BB865656;\n\tTue, 17 May 2022 21:18:47 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 60BD06565C\n\tfor ;\n\tTue, 17 May 2022 21:18:45 +0200 (CEST)","from perceval.ideasonboard.com (unknown [45.131.31.124])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 41E806DC;\n\tTue, 17 May 2022 21:18:44 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1652815127;\n\tbh=nu9bMmutfOGADBC/V+8yMMZu5RoW/gkXUZiQCMWjPdI=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=3DA4Pp8CMAy8EeD4pj9PwD0k4LdoKyK690/+fTnAu0qf2uYn+ooO0AdcQcOSTs+1o\n\ttusMjyfORigvEtPUbV3+KOaSSEexcwn9Kf865WKzQ95gIKaTsH6H7EzXwcUO0ZpaL/\n\tE19q2d0BkLshWeJU34Ah2g4+EXkQakwk7RgZfcmKCadUkdoA+hmIVIfmeueTWQUrOQ\n\tH1eMdq6iNMbas3RzJrJx0j65KF9+SsnujSOG7YvtlqqDRiS/Kh4p5LN9TIVpRlMnZT\n\tYuZmY2vZc6Fla6jbe6P/llW41qYWdRaqnZvHvmmyrm8SnfV3l8CWI/mIHeZ/JFYD7q\n\tcBktgEKSvt0+g==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1652815125;\n\tbh=nu9bMmutfOGADBC/V+8yMMZu5RoW/gkXUZiQCMWjPdI=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=SmT83e1hvtx0TTFyAwv5grcTCLm0ny+4g5hB1Gh3asVj/feqIarcz40snznKz+t7l\n\tj32xFTs/TBC8q6xu7yolqKwqO76SazxHgi5E7szBCsc34feW1mdir7Wx4bwHoZTmJq\n\tWXAE9WILLuxiqGLCOr9qnwj8i42U8oxclVM0AV/U="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"SmT83e1h\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Tue, 17 May 2022 21:18:33 +0200","Message-Id":"<20220517191833.333122-4-umang.jain@ideasonboard.com>","X-Mailer":"git-send-email 2.31.1","In-Reply-To":"<20220517191833.333122-1-umang.jain@ideasonboard.com>","References":"<20220517191833.333122-1-umang.jain@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v3 3/3] ipa: ipu3: Put IPAFrameContext(s)\n\tin a ring buffer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Umang Jain via libcamera-devel ","Reply-To":"Umang Jain ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Instead of having one frame context constantly being updated,\nthis patch aims to introduce per-frame IPAFrameContext which\nare stored in a ring buffer. Whenever a request is queued, a new\nIPAFrameContext is created and inserted into the ring buffer.\n\nThe IPAFrameContext structure itself has been slightly extended\nto store a frame id and a ControlList for incoming frame\ncontrols (sent in by the application). The next step would be to\nread and set these controls whenever the request is actually queued\nto the hardware.\n\nSince now we are working in multiples of IPAFrameContext, the\nAlgorithm::process() will actually take in a IPAFrameContext pointer\n(as opposed to a nullptr while preparing for this change).\n\nSigned-off-by: Umang Jain \n---\n src/ipa/ipu3/algorithms/agc.cpp | 11 +++++------\n src/ipa/ipu3/algorithms/agc.h | 4 ++--\n src/ipa/ipu3/ipa_context.cpp | 24 ++++++++++++++++++++++--\n src/ipa/ipu3/ipa_context.h | 14 +++++++++++++-\n src/ipa/ipu3/ipu3.cpp | 22 +++++++++++++---------\n 5 files changed, 55 insertions(+), 20 deletions(-)","diff":"diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp\nindex 383a8deb..f16be534 100644\n--- a/src/ipa/ipu3/algorithms/agc.cpp\n+++ b/src/ipa/ipu3/algorithms/agc.cpp\n@@ -183,14 +183,13 @@ utils::Duration Agc::filterExposure(utils::Duration exposureValue)\n * \\param[in] yGain The gain calculated based on the relative luminance target\n * \\param[in] iqMeanGain The gain calculated based on the relative luminance target\n */\n-void Agc::computeExposure(IPAContext &context, double yGain,\n-\t\t\t double iqMeanGain)\n+void Agc::computeExposure(IPAContext &context, IPAFrameContext *frameContext,\n+\t\t\t double yGain, double iqMeanGain)\n {\n \tconst IPASessionConfiguration &configuration = context.configuration;\n-\tIPAFrameContext &frameContext = context.frameContext;\n \t/* Get the effective exposure and gain applied on the sensor. */\n-\tuint32_t exposure = frameContext.sensor.exposure;\n-\tdouble analogueGain = frameContext.sensor.gain;\n+\tuint32_t exposure = frameContext->sensor.exposure;\n+\tdouble analogueGain = frameContext->sensor.gain;\n \n \t/* Use the highest of the two gain estimates. */\n \tdouble evGain = std::max(yGain, iqMeanGain);\n@@ -360,7 +359,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameCo\n \t\t\tbreak;\n \t}\n \n-\tcomputeExposure(context, yGain, iqMeanGain);\n+\tcomputeExposure(context, frameContext, yGain, iqMeanGain);\n \tframeCount_++;\n }\n \ndiff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h\nindex 219a1a96..105ae0f2 100644\n--- a/src/ipa/ipu3/algorithms/agc.h\n+++ b/src/ipa/ipu3/algorithms/agc.h\n@@ -35,8 +35,8 @@ private:\n \tdouble measureBrightness(const ipu3_uapi_stats_3a *stats,\n \t\t\t\t const ipu3_uapi_grid_config &grid) const;\n \tutils::Duration filterExposure(utils::Duration currentExposure);\n-\tvoid computeExposure(IPAContext &context, double yGain,\n-\t\t\t double iqMeanGain);\n+\tvoid computeExposure(IPAContext &context, IPAFrameContext *frameContext,\n+\t\t\t double yGain, double iqMeanGain);\n \tdouble estimateLuminance(IPAActiveState &activeState,\n \t\t\t\t const ipu3_uapi_grid_config &grid,\n \t\t\t\t const ipu3_uapi_stats_3a *stats,\ndiff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\nindex 06eb2776..81b30600 100644\n--- a/src/ipa/ipu3/ipa_context.cpp\n+++ b/src/ipa/ipu3/ipa_context.cpp\n@@ -58,8 +58,8 @@ namespace libcamera::ipa::ipu3 {\n * \\var IPAContext::configuration\n * \\brief The IPA session configuration, immutable during the session\n *\n- * \\var IPAContext::frameContext\n- * \\brief The frame context for the frame being processed\n+ * \\var IPAContext::frameContexts\n+ * \\brief Ring buffer of the IPAFrameContext(s)\n *\n * \\var IPAContext::activeState\n * \\brief The current state of IPA algorithms\n@@ -183,6 +183,26 @@ namespace libcamera::ipa::ipu3 {\n */\n \n /**\n+ * \\brief Default constructor for IPAFrameContext\n+ */\n+IPAFrameContext::IPAFrameContext() = default;\n+\n+/**\n+ * \\brief Construct a IPAFrameContext instance\n+ */\n+IPAFrameContext::IPAFrameContext(uint32_t id, const ControlList &reqControls)\n+\t: frame(id), frameControls(reqControls)\n+{\n+\tsensor = {};\n+}\n+\n+/**\n+ * \\var IPAFrameContext::frame\n+ * \\brief The frame number\n+ *\n+ * \\var IPAFrameContext::frameControls\n+ * \\brief Controls sent in by the application while queuing the request\n+ *\n * \\var IPAFrameContext::sensor\n * \\brief Effective sensor values that were applied for the frame\n *\ndiff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\nindex 8d681131..42e11141 100644\n--- a/src/ipa/ipu3/ipa_context.h\n+++ b/src/ipa/ipu3/ipa_context.h\n@@ -8,16 +8,22 @@\n \n #pragma once\n \n+#include \n+\n #include \n \n #include \n \n+#include \n #include \n \n namespace libcamera {\n \n namespace ipa::ipu3 {\n \n+/* Maximum number of frame contexts to be held */\n+static constexpr uint32_t kMaxFrameContexts = 16;\n+\n struct IPASessionConfiguration {\n \tstruct {\n \t\tipu3_uapi_grid_config bdsGrid;\n@@ -71,17 +77,23 @@ struct IPAActiveState {\n };\n \n struct IPAFrameContext {\n+\tIPAFrameContext();\n+\tIPAFrameContext(uint32_t id, const ControlList &reqControls);\n+\n \tstruct {\n \t\tuint32_t exposure;\n \t\tdouble gain;\n \t} sensor;\n+\n+\tuint32_t frame;\n+\tControlList frameControls;\n };\n \n struct IPAContext {\n \tIPASessionConfiguration configuration;\n \tIPAActiveState activeState;\n \n-\tIPAFrameContext frameContext;\n+\tstd::array frameContexts;\n };\n \n } /* namespace ipa::ipu3 */\ndiff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\nindex 16e5028f..4322f96b 100644\n--- a/src/ipa/ipu3/ipu3.cpp\n+++ b/src/ipa/ipu3/ipu3.cpp\n@@ -313,7 +313,7 @@ int IPAIPU3::init(const IPASettings &settings,\n \t}\n \n \t/* Clean context */\n-\tcontext_ = {};\n+\tcontext_.configuration = {};\n \tcontext_.configuration.sensor.lineDuration = sensorInfo.lineLength * 1.0s / sensorInfo.pixelRate;\n \n \t/* Construct our Algorithms */\n@@ -456,7 +456,8 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo,\n \n \t/* Clean IPAActiveState at each reconfiguration. */\n \tcontext_.activeState = {};\n-\tcontext_.frameContext = {};\n+\tIPAFrameContext initFrameContext;\n+\tcontext_.frameContexts.fill(initFrameContext);\n \n \tif (!validateSensorControls()) {\n \t\tLOG(IPAIPU3, Error) << \"Sensor control validation failed.\";\n@@ -568,15 +569,17 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame,\n \tconst ipu3_uapi_stats_3a *stats =\n \t\treinterpret_cast(mem.data());\n \n-\tcontext_.frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get();\n-\tcontext_.frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get());\n+\tIPAFrameContext &frameContext = context_.frameContexts[frame % kMaxFrameContexts];\n+\n+\tframeContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get();\n+\tframeContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get());\n \n \tdouble lineDuration = context_.configuration.sensor.lineDuration.get();\n \tint32_t vBlank = context_.configuration.sensor.defVBlank;\n \tControlList ctrls(controls::controls);\n \n \tfor (auto const &algo : algorithms_)\n-\t\talgo->process(context_, nullptr, stats);\n+\t\talgo->process(context_, &frameContext, stats);\n \n \tsetControls(frame);\n \n@@ -584,11 +587,11 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame,\n \tint64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration;\n \tctrls.set(controls::FrameDuration, frameDuration);\n \n-\tctrls.set(controls::AnalogueGain, context_.frameContext.sensor.gain);\n+\tctrls.set(controls::AnalogueGain, frameContext.sensor.gain);\n \n \tctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK);\n \n-\tctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration);\n+\tctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration);\n \n \t/*\n \t * \\todo The Metadata provides a path to getting extended data\n@@ -609,10 +612,11 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame,\n * Parse the request to handle any IPA-managed controls that were set from the\n * application such as manual sensor settings.\n */\n-void IPAIPU3::queueRequest([[maybe_unused]] const uint32_t frame,\n-\t\t\t [[maybe_unused]] const ControlList &controls)\n+void IPAIPU3::queueRequest(const uint32_t frame, const ControlList &controls)\n {\n \t/* \\todo Start processing for 'frame' based on 'controls'. */\n+\tIPAFrameContext newFrameContext(frame, controls);\n+\tcontext_.frameContexts[frame % kMaxFrameContexts] = newFrameContext;\n }\n \n /**\n","prefixes":["libcamera-devel","v3","3/3"]}