{"id":16396,"url":"https://patchwork.libcamera.org/api/1.1/patches/16396/?format=json","web_url":"https://patchwork.libcamera.org/patch/16396/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220627162732.33160-16-jacopo@jmondi.org>","date":"2022-06-27T16:27:32","name":"[libcamera-devel,15/15] ipa: ipu3: Rework IPAIPU3::processStatsBuffer()","commit_ref":null,"pull_url":null,"state":"not-applicable","archived":true,"hash":"30514f00b70402224cda64652952b6c430cd5fbb","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/1.1/people/3/?format=json","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/16396/mbox/","series":[{"id":3224,"url":"https://patchwork.libcamera.org/api/1.1/series/3224/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3224","date":"2022-06-27T16:27:17","name":"Internal controls, sensor delays and IPA init/configure rework","version":1,"mbox":"https://patchwork.libcamera.org/series/3224/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/16396/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/16396/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9B996BD808\n\tfor ;\n\tMon, 27 Jun 2022 16:28:06 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0C23D65649;\n\tMon, 27 Jun 2022 18:28:06 +0200 (CEST)","from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net\n\t[217.70.183.200])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B4D0965650\n\tfor ;\n\tMon, 27 Jun 2022 18:27:55 +0200 (CEST)","(Authenticated sender: jacopo@jmondi.org)\n\tby mail.gandi.net (Postfix) with ESMTPSA id 0B57C20008;\n\tMon, 27 Jun 2022 16:27:54 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1656347286;\n\tbh=P9aa6O5zAz3cqNS1dxpJn9f54hgJ61tTUiMnkK+R99U=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=X9/OdMiA0wxKeZAWPs+JYM+ulTDATiBmqLtXeRPpInPMwB7QrPFN3Y3HZ4O3UZH82\n\tYJ1xv10uS9wakIWiyqF2lFHM1Fm5TxhOmB4nLv7dB0ZsRS7kbzIAjTaTUUGhEt55OZ\n\tla00oRSmzkzPibBbNP0OzQ4WNPp2CzmoXCCc6nf+rHYB7KEzfRHcKURPhnqfqMWPpu\n\tAyQ+Sbl83ryJJVbsVsFjmVcvnnM5z232ACrt0aK1pTNZGqIjTCAUYEVcQF093d/DI3\n\t+pFWkNwkzTKoE9byW0Ys2i0HYffBy3gB7vjOWOqwmxUXoNAse/8nQiQJYYI4ZlU79A\n\tdETL5pKLbK8kw==","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 27 Jun 2022 18:27:32 +0200","Message-Id":"<20220627162732.33160-16-jacopo@jmondi.org>","X-Mailer":"git-send-email 2.36.1","In-Reply-To":"<20220627162732.33160-1-jacopo@jmondi.org>","References":"<20220627162732.33160-1-jacopo@jmondi.org>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 15/15] ipa: ipu3: Rework\n\tIPAIPU3::processStatsBuffer()","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jacopo Mondi via libcamera-devel ","Reply-To":"Jacopo Mondi ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Before removing V4L2 controls from the IPAIPU3::processStatsBuffer()\ninterface, rationalize the existing code a bit.\n\n1) we have a context_.configuration which collects the per-capture\n session configuration data. At the same time the sensor and lens\n controls info are kept in class member variables.\n\n Move all per-capture session parameters to context_.configuration\n\n2) IPAIPU3::setControls() operates on the active state, which is\n implicitly retrieved from the global context. Pass it explicitely\n to make it clear.\n\n3) Now that we have the full IPACameraSensorInfo in the current context\n remove vblank from IPASessionConfiguration::sensor as it can be\n retrieved from IPASessionConfiguration::sensor.info.\n\n4) Rationalize the code of IPAIPU3::processStatsBuffer() to organize it\n in a logical flow of operations:\n - Update the per-frame context with the sensor's controls values\n - Map the stats buffer\n - Run algorithms with the the per-frame context and stats\n - setControls() using the active state as computed by algorithms\n - Populate the metadata buffer\n\nSigned-off-by: Jacopo Mondi \n---\n src/ipa/ipu3/ipa_context.cpp | 14 ++++++-\n src/ipa/ipu3/ipa_context.h | 9 ++++-\n src/ipa/ipu3/ipu3.cpp | 73 +++++++++++++++++++-----------------\n 3 files changed, 59 insertions(+), 37 deletions(-)","diff":"diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\nindex 06fdf2a1efc7..f09d061e2d19 100644\n--- a/src/ipa/ipu3/ipa_context.cpp\n+++ b/src/ipa/ipu3/ipa_context.cpp\n@@ -126,8 +126,18 @@ namespace libcamera::ipa::ipu3 {\n * \\var IPASessionConfiguration::sensor.lineDuration\n * \\brief Line duration in microseconds\n *\n- * \\var IPASessionConfiguration::sensor.vBlank\n- * \\brief The vertical blanking expressed in number of lines\n+ * \\var IPASessionConfiguration::sensor.info\n+ * \\brief The IPACameraSensorInfo valid for the session\n+ *\n+ * \\var IPASessionConfiguration::sensor.controls\n+ * \\brief The ControlInfoMap of camera sensor control limits valid for the\n+ * session\n+ *\n+ * \\var IPASessionConfiguration::lens\n+ * \\brief Lens-specific configuration of the IPA\n+ *\n+ * \\var IPASessionConfiguration::lens.controls\n+ * \\brief The ControlInfoMap of lens control limits valid for the session\n */\n \n /**\ndiff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\nindex a5b878ab7792..35baa17e8708 100644\n--- a/src/ipa/ipu3/ipa_context.h\n+++ b/src/ipa/ipu3/ipa_context.h\n@@ -17,6 +17,8 @@\n #include \n #include \n \n+#include \n+\n namespace libcamera {\n \n namespace ipa::ipu3 {\n@@ -43,9 +45,14 @@ struct IPASessionConfiguration {\n \t} agc;\n \n \tstruct {\n-\t\tint32_t vBlank;\n \t\tutils::Duration lineDuration;\n+\t\tControlInfoMap controls;\n+\t\tIPACameraSensorInfo info;\n \t} sensor;\n+\n+\tstruct {\n+\t\tControlInfoMap controls;\n+\t} lens;\n };\n \n struct IPAActiveState {\ndiff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\nindex b67ff1948bc2..791c03e5a2d3 100644\n--- a/src/ipa/ipu3/ipu3.cpp\n+++ b/src/ipa/ipu3/ipu3.cpp\n@@ -152,16 +152,11 @@ private:\n \n \tbool validateConfiguration(const IPAConfigInfo &config);\n \n-\tvoid setControls(unsigned int frame);\n+\tvoid setControls(unsigned int frame, const IPAActiveState &state);\n \tvoid calculateBdsGrid(const Size &bdsOutputSize);\n \n \tstd::map buffers_;\n \n-\tControlInfoMap sensorCtrls_;\n-\tControlInfoMap lensCtrls_;\n-\n-\tIPACameraSensorInfo sensorInfo_;\n-\n \t/* Interface to the Camera Helper */\n \tstd::unique_ptr camHelper_;\n \n@@ -179,7 +174,6 @@ private:\n void IPAIPU3::updateSessionConfiguration(const IPAConfigInfo &info)\n {\n \tconst IPACameraSensorInfo &sensorInfo = info.sensorInfo;\n-\tcontext_.configuration.sensor.vBlank = sensorInfo.vblank;\n \tcontext_.configuration.sensor.lineDuration = sensorInfo.lineLength * 1.0s\n \t\t\t\t\t\t / sensorInfo.pixelRate;\n \n@@ -202,6 +196,11 @@ void IPAIPU3::updateSessionConfiguration(const IPAConfigInfo &info)\n \n \tcontext_.configuration.agc.minAnalogueGain = camHelper_->gain(minGain);\n \tcontext_.configuration.agc.maxAnalogueGain = camHelper_->gain(maxGain);\n+\n+\t/* Store the sensor and lens configuration in the current session. */\n+\tcontext_.configuration.sensor.info = info.sensorInfo;\n+\tcontext_.configuration.sensor.controls = info.sensorControls;\n+\tcontext_.configuration.lens.controls = info.lensControls;\n }\n \n /**\n@@ -265,7 +264,7 @@ int IPAIPU3::start()\n \t * Set the sensors V4L2 controls before the first frame to ensure that\n \t * we have an expected and known configuration from the start.\n \t */\n-\tsetControls(0);\n+\tsetControls(0, context_.activeState);\n \n \treturn 0;\n }\n@@ -387,9 +386,9 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo)\n \t\t\treturn ret;\n \t}\n \n-\tsensorInfo_ = configInfo.sensorInfo;\n-\tlensCtrls_ = configInfo.lensControls;\n-\tsensorCtrls_ = configInfo.sensorControls;\n+\tcontext_.configuration.sensor.controls = configInfo.sensorControls;\n+\tcontext_.configuration.sensor.info = configInfo.sensorInfo;\n+\tcontext_.configuration.lens.controls = configInfo.lensControls;\n \n \treturn 0;\n }\n@@ -480,35 +479,39 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame,\n \t\treturn;\n \t}\n \n-\tSpan mem = it->second.planes()[0];\n-\tconst ipu3_uapi_stats_3a *stats =\n-\t\treinterpret_cast(mem.data());\n-\n+\t/*\n+\t * Update the per-frame context storing the sensor exposure and\n+\t * gain for later use by algorithms.\n+\t */\n \tIPAFrameContext &frameContext = context_.frameContexts[frame % kMaxFrameContexts];\n-\n \tif (frameContext.frame != frame)\n \t\tLOG(IPAIPU3, Warning) << \"Frame \" << frame << \" does not match its frame context\";\n \n \tframeContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get();\n \tframeContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get());\n \n-\tdouble lineDuration = context_.configuration.sensor.lineDuration.get();\n-\tint32_t vBlank = context_.configuration.sensor.vBlank;\n-\tControlList ctrls(controls::controls);\n-\n+\t/* Run algorithms on the statistics and per-frame context. */\n+\tSpan mem = it->second.planes()[0];\n+\tconst ipu3_uapi_stats_3a *stats =\n+\t\treinterpret_cast(mem.data());\n \tfor (auto const &algo : algorithms_)\n \t\talgo->process(context_, &frameContext, stats);\n \n-\tsetControls(frame);\n+\t/* Set sensor controls using the newly computed values. */\n+\tsetControls(frame, context_.activeState);\n \n-\tint64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration;\n-\tctrls.set(controls::FrameDuration, frameDuration);\n+\t/* Prepare metadata for the frame. */\n+\tconst IPASessionConfiguration &ipaConfig = context_.configuration;\n+\tconst IPACameraSensorInfo &sensorInfo = ipaConfig.sensor.info;\n+\tdouble lineDuration = ipaConfig.sensor.lineDuration.get();\n+\tint64_t frameDuration = (sensorInfo.vblank + sensorInfo.outputSize.height) * lineDuration;\n \n-\tctrls.set(controls::AnalogueGain, frameContext.sensor.gain);\n+\tControlList metadata(controls::controls);\n \n-\tctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK);\n-\n-\tctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration);\n+\tmetadata.set(controls::FrameDuration, frameDuration);\n+\tmetadata.set(controls::AnalogueGain, frameContext.sensor.gain);\n+\tmetadata.set(controls::ColourTemperature, context_.activeState.awb.temperatureK);\n+\tmetadata.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration);\n \n \t/*\n \t * \\todo The Metadata provides a path to getting extended data\n@@ -518,7 +521,7 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame,\n \t * likely want to avoid putting platform specific metadata in.\n \t */\n \n-\tmetadataReady.emit(frame, ctrls);\n+\tmetadataReady.emit(frame, metadata);\n }\n \n /**\n@@ -538,22 +541,24 @@ void IPAIPU3::queueRequest(const uint32_t frame, const ControlList &controls)\n /**\n * \\brief Handle sensor controls for a given \\a frame number\n * \\param[in] frame The frame on which the sensor controls should be set\n+ * \\param[in] state The IPA active state which contains the control values as\n+ * computed by the algorithms\n *\n * Send the desired sensor control values to the pipeline handler to request\n * that they are applied on the camera sensor.\n */\n-void IPAIPU3::setControls(unsigned int frame)\n+void IPAIPU3::setControls(unsigned int frame, const IPAActiveState &state)\n {\n-\tint32_t exposure = context_.activeState.agc.exposure;\n-\tint32_t gain = camHelper_->gainCode(context_.activeState.agc.gain);\n+\tint32_t exposure = state.agc.exposure;\n+\tint32_t gain = camHelper_->gainCode(state.agc.gain);\n \n-\tControlList ctrls(sensorCtrls_);\n+\tControlList ctrls(controls::controls);\n \tctrls.set(V4L2_CID_EXPOSURE, exposure);\n \tctrls.set(V4L2_CID_ANALOGUE_GAIN, gain);\n \n-\tControlList lensCtrls(lensCtrls_);\n+\tControlList lensCtrls(controls::controls);\n \tlensCtrls.set(V4L2_CID_FOCUS_ABSOLUTE,\n-\t\t static_cast(context_.activeState.af.focus));\n+\t\t static_cast(state.af.focus));\n \n \tsetSensorControls.emit(frame, ctrls, lensCtrls);\n }\n","prefixes":["libcamera-devel","15/15"]}