{"id":10275,"url":"https://patchwork.libcamera.org/api/1.1/patches/10275/?format=json","web_url":"https://patchwork.libcamera.org/patch/10275/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>","date":"2020-10-28T01:00:44","name":"[libcamera-devel,2/9] libcamera: delayed_controls: Add helper for controls that applies with a delay","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"38c26e343a83bc142dd45b38ac183b4a26155ab7","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/1.1/people/5/?format=json","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"delegate":{"id":16,"url":"https://patchwork.libcamera.org/api/1.1/users/16/?format=json","username":"neg","first_name":"Niklas","last_name":"Söderlund","email":"niklas.soderlund@ragnatech.se"},"mbox":"https://patchwork.libcamera.org/patch/10275/mbox/","series":[{"id":1423,"url":"https://patchwork.libcamera.org/api/1.1/series/1423/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1423","date":"2020-10-28T01:00:42","name":"libcamera: Add helper for controls that take effect with a delay","version":1,"mbox":"https://patchwork.libcamera.org/series/1423/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/10275/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/10275/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0B3F7C3B5C\n\tfor ;\n\tWed, 28 Oct 2020 01:01:16 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D91DE6225D;\n\tWed, 28 Oct 2020 02:01:15 +0100 (CET)","from bin-mail-out-06.binero.net (bin-mail-out-06.binero.net\n\t[195.74.38.229])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 444F06220D\n\tfor ;\n\tWed, 28 Oct 2020 02:01:12 +0100 (CET)","from bismarck.berto.se (p4fca2458.dip0.t-ipconnect.de\n\t[79.202.36.88])\n\tby bin-vsp-out-03.atm.binero.net (Halon) with ESMTPA\n\tid 1163a54f-18b9-11eb-954c-0050569116f7;\n\tWed, 28 Oct 2020 02:01:10 +0100 (CET)"],"X-Halon-ID":"1163a54f-18b9-11eb-954c-0050569116f7","Authorized-sender":"niklas.soderlund@fsdn.se","From":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","To":"libcamera-devel@lists.libcamera.org, david.plowman@raspberrypi.com,\n\tnaush@raspberrypi.com","Date":"Wed, 28 Oct 2020 02:00:44 +0100","Message-Id":"<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>","X-Mailer":"git-send-email 2.29.1","In-Reply-To":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Some sensor controls take effect with a delay as the sensor needs time\nto adjust, for example exposure. Add a optional helper DelayedControls\nto help pipelines deal with such controls.\n\nThe idea is to provide a queue of controls towards the V4L2 device and\napply individual controls with the specified delay with the aim to get\npredictable and retrievable control values for any given frame. To do\nthis the queue of controls needs to be at least as deep as the control\nwith the largest delay.\n\nThe DelayedControls needs to be informed of every start of exposure.\nThis can be emulated but the helper is designed to be used with this\nevent being provide by the kernel thru V4L2 events.\n\nThis helper is based on StaggeredCtrl from the Raspberry Pi pipeline\nhandler but expands on its API. This helpers aims to replace the\nRaspberry Pi implementations and mimics it behavior perfectly.\n\nSigned-off-by: Niklas Söderlund \n---\n include/libcamera/internal/delayed_controls.h | 87 ++++++\n src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n src/libcamera/meson.build | 1 +\n 3 files changed, 370 insertions(+)\n create mode 100644 include/libcamera/internal/delayed_controls.h\n create mode 100644 src/libcamera/delayed_controls.cpp","diff":"diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h\nnew file mode 100644\nindex 0000000000000000..df5520d240a54e4b\n--- /dev/null\n+++ b/include/libcamera/internal/delayed_controls.h\n@@ -0,0 +1,87 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2020, Google Inc.\n+ *\n+ * delayed_controls.h - Helper to deal with controls that are applied with a delay\n+ */\n+#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n+#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n+\n+#include \n+#include \n+#include \n+\n+#include \n+\n+namespace libcamera {\n+\n+class V4L2Device;\n+\n+class DelayedControls\n+{\n+public:\n+\tDelayedControls(V4L2Device *device,\n+\t\t\tconst std::unordered_map &delays);\n+\n+\tvoid reset(ControlList *controls = nullptr);\n+\n+\tbool push(const ControlList &controls);\n+\tControlList get(uint32_t sequence);\n+\n+\tvoid frameStart(uint32_t sequence);\n+\n+private:\n+\tclass ControlInfo\n+\t{\n+\tpublic:\n+\t\tControlInfo()\n+\t\t\t: updated(false)\n+\t\t{\n+\t\t}\n+\n+\t\tControlInfo(const ControlValue &v)\n+\t\t\t: value(v), updated(true)\n+\t\t{\n+\t\t}\n+\n+\t\tControlValue value;\n+\t\tbool updated;\n+\t};\n+\n+\tstatic constexpr int listSize = 16;\n+\tclass ControlArray : public std::array\n+\t{\n+\tpublic:\n+\t\tControlInfo &operator[](unsigned int index)\n+\t\t{\n+\t\t\treturn std::array::operator[](index % listSize);\n+\t\t}\n+\n+\t\tconst ControlInfo &operator[](unsigned int index) const\n+\t\t{\n+\t\t\treturn std::array::operator[](index % listSize);\n+\t\t}\n+\t};\n+\n+\tusing ControlsDelays = std::unordered_map;\n+\tusing ControlsValues = std::unordered_map;\n+\n+\tbool queue(const ControlList &controls);\n+\n+\tstd::mutex lock_;\n+\n+\tV4L2Device *device_;\n+\tControlsDelays delays_;\n+\tunsigned int maxDelay_;\n+\n+\tbool running_;\n+\tuint32_t fistSequence_;\n+\n+\tuint32_t queueCount_;\n+\tuint32_t writeCount_;\n+\tControlsValues ctrls_;\n+};\n+\n+} /* namespace libcamera */\n+\n+#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\ndiff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp\nnew file mode 100644\nindex 0000000000000000..0e32f417c5cc68b7\n--- /dev/null\n+++ b/src/libcamera/delayed_controls.cpp\n@@ -0,0 +1,282 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2020, Google Inc.\n+ *\n+ * delayed_controls.h - Helper to deal with controls that are applied with a delay\n+ */\n+\n+#include \"libcamera/internal/delayed_controls.h\"\n+#include \"libcamera/internal/v4l2_device.h\"\n+\n+#include \n+\n+#include \"libcamera/internal/log.h\"\n+\n+/**\n+ * \\file delayed_controls.h\n+ * \\brief Helper to deal with controls that are applied with a delay\n+ */\n+\n+namespace libcamera {\n+\n+LOG_DEFINE_CATEGORY(DelayedControls)\n+\n+/**\n+ * \\class DelayedControls\n+ * \\brief Helper to deal with controls that takes effect with a delay\n+ *\n+ * Some sensor controls take effect with a delay as the sensor needs time to\n+ * adjust, for example exposure and focus. This is an optional helper class to\n+ * deal with such controls and the intended user is pipeline handlers.\n+ *\n+ * The idea is to extend the concept of the pipeline depth the users needs to\n+ * maintain for buffers to controls. The depth is determined with by the control\n+ * with the grates delay. As long as the pipeline keeps the control queue above\n+ * this level the control values are guaranteed to be in effect at the specified\n+ * point in time.\n+ *\n+ * The downside is of course that the pipeline needs to know what controls to\n+ * set control depth frames in advance. But there really is no way around this\n+ * as the delay is a consequence of the physical world. Compare this with\n+ * parameter buffers where the buffer may be queued to the device while it's\n+ * still being written to as long as it's ready before it's consumed, this is\n+ * because the parameter buffer (usually) does not contain controls that\n+ * requires time to take effect.\n+ */\n+\n+\n+/**\n+ * \\brief Construct a DelayedControls\n+ * \\param[in] device The V4L2 device containing the delayed controls\n+ * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n+ * effect in frames\n+ *\n+ * Only controls specified in \\a delays are handled by the DelayedControls\n+ * instance. If it's desired to mix delayed controls and controls that takes\n+ * effect immediately the immediate controls must be listed in the \\a delays map\n+ * with a delay value of 0.\n+ */\n+DelayedControls::DelayedControls(V4L2Device *device,\n+\t\t\t\t const std::unordered_map &delays)\n+\t: device_(device), maxDelay_(0)\n+{\n+\tconst ControlInfoMap &controls = device_->controls();\n+\n+\t/*\n+\t * Sanity check that all controls where delays are requested are\n+\t * exposed byt the device.\n+\t */\n+\tfor (auto const &delay : delays) {\n+\t\tunsigned int id = delay.first;\n+\n+\t\tif (controls.find(id) == controls.end())\n+\t\t\tLOG(DelayedControls, Error)\n+\t\t\t\t<< \"Delay request for control id \"\n+\t\t\t\t<< utils::hex(id)\n+\t\t\t\t<< \" but control is not exposed by device \"\n+\t\t\t\t<< device_->deviceNode();\n+\t}\n+\n+\t/*\n+\t * Create a map of control to delay for all controls exposed by the\n+\t * device. If no delay is specified assume the control applies directly.\n+\t */\n+\tfor (auto const &control : controls) {\n+\t\tconst ControlId *id = control.first;\n+\n+\t\tauto it = delays.find(id->id());\n+\t\tif (it == delays.end())\n+\t\t\tcontinue;\n+\n+\t\tdelays_[id] = it->second;\n+\n+\t\tLOG(DelayedControls, Debug)\n+\t\t\t<< \"Set a delay of \" << delays_[id]\n+\t\t\t<< \" for \" << id->name();\n+\n+\t\tmaxDelay_ = std::max(maxDelay_, delays_[id]);\n+\t}\n+\n+\treset();\n+}\n+\n+/**\n+ * \\brief Reset the V4L2 device\n+ * \\param[in] controls List of controls to reset the device to or nullptr\n+ *\n+ * Resets the delayed controls state machine to its starting state. All controls\n+ * are fetched from the V4L2 device to provide a good starting point for the\n+ * first frames (length of control depth).\n+ *\n+ * Optionally \\a controls can be specified to set some or all of the handled\n+ * V4L2 controls prior to reading them back. If no controls needs to be set\n+ * nullptr may be used.\n+ */\n+void DelayedControls::reset(ControlList *controls)\n+{\n+\tstd::lock_guard lock(lock_);\n+\n+\trunning_ = false;\n+\tfistSequence_ = 0;\n+\tqueueCount_ = 0;\n+\twriteCount_ = 0;\n+\n+\t/* Set the controls on the device if requested. */\n+\tif (controls)\n+\t\tdevice_->setControls(controls);\n+\n+\t/* Retrieve current control values reported by the device. */\n+\tstd::vector ids;\n+\tfor (auto const &delay : delays_)\n+\t\tids.push_back(delay.first->id());\n+\n+\tControlList devCtrls = device_->getControls(ids);\n+\n+\t/* Seed the control queue with the controls reported by the device. */\n+\tctrls_.clear();\n+\tfor (const auto &ctrl : devCtrls) {\n+\t\tconst ControlId *id = devCtrls.infoMap()->idmap().at(ctrl.first);\n+\t\tctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n+\t}\n+\n+\tqueueCount_++;\n+}\n+\n+/**\n+ * \\brief Push a set of controls on the queue\n+ * \\param[in] controls List of controls to add the device queue\n+ *\n+ * Push a set of controls to the control queue. This increases the control queue\n+ * depth by one.\n+ *\n+ * \\returns true if \\a controls are accepted, or false otherwise\n+ */\n+bool DelayedControls::push(const ControlList &controls)\n+{\n+\tstd::lock_guard lock(lock_);\n+\n+\treturn queue(controls);\n+}\n+\n+bool DelayedControls::queue(const ControlList &controls)\n+{\n+\t/* Copy state from previous frame. */\n+\tfor (auto &ctrl : ctrls_) {\n+\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n+\t\tinfo.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n+\t\tinfo.updated = false;\n+\t}\n+\n+\t/* Update with new controls. */\n+\tfor (const auto &control : controls) {\n+\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n+\n+\t\tif (delays_.find(id) == delays_.end())\n+\t\t\treturn false;\n+\n+\t\tControlInfo &info = ctrls_[id][queueCount_];\n+\n+\t\tinfo.value = control.second;\n+\t\tinfo.updated = true;\n+\n+\t\tLOG(DelayedControls, Debug)\n+\t\t\t<< \"Queuing \" << id->name()\n+\t\t\t<< \" to \" << info.value.toString()\n+\t\t\t<< \" at index \" << queueCount_;\n+\t}\n+\n+\tqueueCount_++;\n+\n+\treturn true;\n+}\n+\n+/**\n+ * \\brief Read back controls in effect at a specific sequence number\n+ * \\param[in] sequence Sequence number to get read back controls for\n+ *\n+ * Read back what controls where in effect at a specific sequence number. The\n+ * history is a ring buffer of 16 entries where new and old values coexist. It's\n+ * the callers responsibility to not read to old sequence numbers that have been\n+ * pushed out of the history.\n+ *\n+ * Historic values are evicted by pushing new values onto the queue using\n+ * push(). The max history from the current sequence number that yields valid\n+ * values are thus 16 minus number of controls pushed.\n+ *\n+ * \\returns List of controls in effect at \\a sequence\n+ */\n+ControlList DelayedControls::get(uint32_t sequence)\n+{\n+\tstd::lock_guard lock(lock_);\n+\n+\tuint32_t adjustedSeq = sequence - fistSequence_ + 1;\n+\tunsigned int index = std::max(0, adjustedSeq - maxDelay_);\n+\n+\tControlList out(device_->controls());\n+\tfor (const auto &ctrl : ctrls_) {\n+\t\tconst ControlId *id = ctrl.first;\n+\t\tconst ControlInfo &info = ctrl.second[index];\n+\n+\t\tout.set(id->id(), info.value);\n+\n+\t\tLOG(DelayedControls, Debug)\n+\t\t\t<< \"Reading \" << id->name()\n+\t\t\t<< \" to \" << info.value.toString()\n+\t\t\t<< \" at index \" << index;\n+\t}\n+\n+\treturn out;\n+}\n+\n+/**\n+ * \\brief Inform DelayedControls of a start of a new frame\n+ * \\param[in] sequence Sequence number of the frame that started\n+ *\n+ * Inform the state machine that a new frame have started and it's sequence\n+ * number. It's user of this helpers responsibility to inform the helper\n+ * at the start of every frame. This can with ease be connected to the start\n+ * of exposure (SOE) V4L2 event.\n+ */\n+void DelayedControls::frameStart(uint32_t sequence)\n+{\n+\tLOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n+\n+\tstd::lock_guard lock(lock_);\n+\n+\tif (!running_) {\n+\t\tfistSequence_ = sequence;\n+\t\trunning_ = true;\n+\t}\n+\n+\t/*\n+\t * Create control list peaking ahead in the value queue to ensure\n+\t * values are set in time to satisfy the sensor delay.\n+\t */\n+\tControlList out(device_->controls());\n+\tfor (const auto &ctrl : ctrls_) {\n+\t\tconst ControlId *id = ctrl.first;\n+\t\tunsigned int delayDiff = maxDelay_ - delays_[id];\n+\t\tunsigned int index = std::max(0, writeCount_ - delayDiff);\n+\t\tconst ControlInfo &info = ctrl.second[index];\n+\n+\t\tif (info.updated) {\n+\t\t\tout.set(id->id(), info.value);\n+\t\t\tLOG(DelayedControls, Debug)\n+\t\t\t\t<< \"Setting \" << id->name()\n+\t\t\t\t<< \" to \" << info.value.toString()\n+\t\t\t\t<< \" at index \" << index;\n+\t\t}\n+\t}\n+\n+\twriteCount_++;\n+\n+\twhile (writeCount_ >= queueCount_) {\n+\t\tLOG(DelayedControls, Debug)\n+\t\t\t<< \"Queue is empty, auto queue no-op.\";\n+\t\tqueue({});\n+\t}\n+\n+\tdevice_->setControls(&out);\n+}\n+\n+} /* namespace libcamera */\ndiff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\nindex 07711b5f93bcc921..19f22f9c94e1d64d 100644\n--- a/src/libcamera/meson.build\n+++ b/src/libcamera/meson.build\n@@ -12,6 +12,7 @@ libcamera_sources = files([\n 'controls.cpp',\n 'control_serializer.cpp',\n 'control_validator.cpp',\n+ 'delayed_controls.cpp',\n 'device_enumerator.cpp',\n 'device_enumerator_sysfs.cpp',\n 'event_dispatcher.cpp',\n","prefixes":["libcamera-devel","2/9"]}