[{"id":13574,"web_url":"https://patchwork.libcamera.org/comment/13574/","msgid":"","date":"2020-11-03T10:27:06","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Niklas,\n\nThank you for your patch. Please see some initial comments below.\n\nBest regards,\nNaush\n\n\nOn Wed, 28 Oct 2020 at 01:01, Niklas Söderlund <\nniklas.soderlund@ragnatech.se> wrote:\n\n> Some sensor controls take effect with a delay as the sensor needs time\n> to adjust, for example exposure. Add a optional helper DelayedControls\n> to help pipelines deal with such controls.\n>\n> The idea is to provide a queue of controls towards the V4L2 device and\n> apply individual controls with the specified delay with the aim to get\n> predictable and retrievable control values for any given frame. To do\n> this the queue of controls needs to be at least as deep as the control\n> with the largest delay.\n>\n> The DelayedControls needs to be informed of every start of exposure.\n> This can be emulated but the helper is designed to be used with this\n> event being provide by the kernel thru V4L2 events.\n>\n> This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> handler but expands on its API. This helpers aims to replace the\n> Raspberry Pi implementations and mimics it behavior perfectly.\n>\n> Signed-off-by: Niklas Söderlund \n> ---\n> include/libcamera/internal/delayed_controls.h | 87 ++++++\n> src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> src/libcamera/meson.build | 1 +\n> 3 files changed, 370 insertions(+)\n> create mode 100644 include/libcamera/internal/delayed_controls.h\n> create mode 100644 src/libcamera/delayed_controls.cpp\n>\n> diff --git a/include/libcamera/internal/delayed_controls.h\n> b/include/libcamera/internal/delayed_controls.h\n> new file mode 100644\n> index 0000000000000000..df5520d240a54e4b\n> --- /dev/null\n> +++ b/include/libcamera/internal/delayed_controls.h\n> @@ -0,0 +1,87 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * delayed_controls.h - Helper to deal with controls that are applied\n> with a delay\n> + */\n>\n\nGiven this is heavily derived from the staggered_write work, would it be ok\nto keep the Raspberry Pi copyright in the header?\n\nCopyright (C) 2020, Raspberry Pi (Trading) Ltd.\n\n\n> +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +namespace libcamera {\n> +\n> +class V4L2Device;\n> +\n> +class DelayedControls\n> +{\n> +public:\n> + DelayedControls(V4L2Device *device,\n> + const std::unordered_map\n> &delays);\n> +\n> + void reset(ControlList *controls = nullptr);\n> +\n> + bool push(const ControlList &controls);\n> + ControlList get(uint32_t sequence);\n> +\n> + void frameStart(uint32_t sequence);\n> +\n> +private:\n> + class ControlInfo\n> + {\n> + public:\n> + ControlInfo()\n> + : updated(false)\n> + {\n> + }\n> +\n> + ControlInfo(const ControlValue &v)\n> + : value(v), updated(true)\n> + {\n> + }\n> +\n> + ControlValue value;\n> + bool updated;\n> + };\n> +\n> + static constexpr int listSize = 16;\n> + class ControlArray : public std::array\n> + {\n> + public:\n> + ControlInfo &operator[](unsigned int index)\n> + {\n> + return std::array listSize>::operator[](index % listSize);\n> + }\n> +\n> + const ControlInfo &operator[](unsigned int index) const\n> + {\n> + return std::array listSize>::operator[](index % listSize);\n> + }\n> + };\n> +\n> + using ControlsDelays = std::unordered_map unsigned int>;\n> + using ControlsValues = std::unordered_map ControlArray>;\n> +\n> + bool queue(const ControlList &controls);\n> +\n> + std::mutex lock_;\n> +\n> + V4L2Device *device_;\n> + ControlsDelays delays_;\n> + unsigned int maxDelay_;\n> +\n> + bool running_;\n> + uint32_t fistSequence_;\n> +\n> + uint32_t queueCount_;\n> + uint32_t writeCount_;\n> + ControlsValues ctrls_;\n> +};\n> +\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> diff --git a/src/libcamera/delayed_controls.cpp\n> b/src/libcamera/delayed_controls.cpp\n> new file mode 100644\n> index 0000000000000000..0e32f417c5cc68b7\n> --- /dev/null\n> +++ b/src/libcamera/delayed_controls.cpp\n> @@ -0,0 +1,282 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * delayed_controls.h - Helper to deal with controls that are applied\n> with a delay\n> + */\n>\n\nSimilar to above:\nCopyright (C) 2020, Raspberry Pi (Trading) Ltd.\n\n\n> +\n> +#include \"libcamera/internal/delayed_controls.h\"\n> +#include \"libcamera/internal/v4l2_device.h\"\n> +\n> +#include \n> +\n> +#include \"libcamera/internal/log.h\"\n> +\n> +/**\n> + * \\file delayed_controls.h\n> + * \\brief Helper to deal with controls that are applied with a delay\n> + */\n> +\n> +namespace libcamera {\n> +\n> +LOG_DEFINE_CATEGORY(DelayedControls)\n> +\n> +/**\n> + * \\class DelayedControls\n> + * \\brief Helper to deal with controls that takes effect with a delay\n> + *\n> + * Some sensor controls take effect with a delay as the sensor needs time\n> to\n> + * adjust, for example exposure and focus. This is an optional helper\n> class to\n> + * deal with such controls and the intended user is pipeline handlers.\n> + *\n> + * The idea is to extend the concept of the pipeline depth the users\n> needs to\n> + * maintain for buffers to controls. The depth is determined with by the\n> control\n> + * with the grates delay. As long as the pipeline keeps the control queue\n> above\n> + * this level the control values are guaranteed to be in effect at the\n> specified\n> + * point in time.\n> + *\n> + * The downside is of course that the pipeline needs to know what\n> controls to\n> + * set control depth frames in advance. But there really is no way around\n> this\n> + * as the delay is a consequence of the physical world. Compare this with\n> + * parameter buffers where the buffer may be queued to the device while\n> it's\n> + * still being written to as long as it's ready before it's consumed,\n> this is\n> + * because the parameter buffer (usually) does not contain controls that\n> + * requires time to take effect.\n> + */\n> +\n> +\n> +/**\n> + * \\brief Construct a DelayedControls\n> + * \\param[in] device The V4L2 device containing the delayed controls\n> + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> + * effect in frames\n> + *\n> + * Only controls specified in \\a delays are handled by the DelayedControls\n> + * instance. If it's desired to mix delayed controls and controls that\n> takes\n> + * effect immediately the immediate controls must be listed in the \\a\n> delays map\n> + * with a delay value of 0.\n> + */\n> +DelayedControls::DelayedControls(V4L2Device *device,\n> + const std::unordered_map unsigned int> &delays)\n> + : device_(device), maxDelay_(0)\n> +{\n> + const ControlInfoMap &controls = device_->controls();\n> +\n> + /*\n> + * Sanity check that all controls where delays are requested are\n> + * exposed byt the device.\n> + */\n> + for (auto const &delay : delays) {\n> + unsigned int id = delay.first;\n> +\n> + if (controls.find(id) == controls.end())\n> + LOG(DelayedControls, Error)\n> + << \"Delay request for control id \"\n> + << utils::hex(id)\n> + << \" but control is not exposed by device \"\n> + << device_->deviceNode();\n> + }\n> +\n> + /*\n> + * Create a map of control to delay for all controls exposed by the\n> + * device. If no delay is specified assume the control applies\n> directly.\n> + */\n> + for (auto const &control : controls) {\n> + const ControlId *id = control.first;\n> +\n> + auto it = delays.find(id->id());\n> + if (it == delays.end())\n> + continue;\n> +\n> + delays_[id] = it->second;\n> +\n> + LOG(DelayedControls, Debug)\n> + << \"Set a delay of \" << delays_[id]\n> + << \" for \" << id->name();\n> +\n> + maxDelay_ = std::max(maxDelay_, delays_[id]);\n> + }\n>\n\nCould you combine the two loops here into one here? If you iterate over\n\"delays\", you could do the error check as well as adding to the \"delays_\"\nmap.\n\n+\n> + reset();\n>\n\nDoes this need to be called here? But my understanding of reset() may be\nwrong... see below.\n\n\n> +}\n> +\n> +/**\n> + * \\brief Reset the V4L2 device\n> + * \\param[in] controls List of controls to reset the device to or nullptr\n> + *\n> + * Resets the delayed controls state machine to its starting state. All\n> controls\n> + * are fetched from the V4L2 device to provide a good starting point for\n> the\n> + * first frames (length of control depth).\n> + *\n> + * Optionally \\a controls can be specified to set some or all of the\n> handled\n> + * V4L2 controls prior to reading them back. If no controls needs to be\n> set\n> + * nullptr may be used.\n> + */\n> +void DelayedControls::reset(ControlList *controls)\n> +{\n> + std::lock_guard lock(lock_);\n> +\n> + running_ = false;\n> + fistSequence_ = 0;\n> + queueCount_ = 0;\n> + writeCount_ = 0;\n> +\n> + /* Set the controls on the device if requested. */\n> + if (controls)\n> + device_->setControls(controls);\n>\n\nShould this happen here in reset()? The reset method to me indicates that\nsome internal state of the class is meant to be reset, and it does nothing\nto send controls to the device. Writing to the device could happen from\nthe caller, if they desire.\n\n\n> +\n> + /* Retrieve current control values reported by the device. */\n> + std::vector ids;\n> + for (auto const &delay : delays_)\n> + ids.push_back(delay.first->id());\n> +\n> + ControlList devCtrls = device_->getControls(ids);\n> +\n> + /* Seed the control queue with the controls reported by the\n> device. */\n> + ctrls_.clear();\n> + for (const auto &ctrl : devCtrls) {\n> + const ControlId *id =\n> devCtrls.infoMap()->idmap().at(ctrl.first);\n> + ctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> + }\n> +\n> + queueCount_++;\n> +}\n>\n\nI'm a bit unclear what reset() does here. In staggered_writer, reset()\nwould essentially clear the queues and re-initialise with the last value\nthat was set. This seems to pull the value from the device, and put it\ninto the queue as a starting point. When we start streaming a sensor, the\nsettings will be applied without delay. So if we want to intialise the\nsensor with a new set of controls, it would not work here. Also, if we\nwere running a use case where we needed to do a mode switch (e.g. preview\nthen capture), then the last requested preview controls would not be used\nto seed the list, and would be lost.\n\n\n> +\n> +/**\n> + * \\brief Push a set of controls on the queue\n> + * \\param[in] controls List of controls to add the device queue\n> + *\n> + * Push a set of controls to the control queue. This increases the\n> control queue\n> + * depth by one.\n> + *\n> + * \\returns true if \\a controls are accepted, or false otherwise\n> + */\n> +bool DelayedControls::push(const ControlList &controls)\n> +{\n> + std::lock_guard lock(lock_);\n> +\n> + return queue(controls);\n> +}\n> +\n> +bool DelayedControls::queue(const ControlList &controls)\n> +{\n> + /* Copy state from previous frame. */\n> + for (auto &ctrl : ctrls_) {\n> + ControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> + info.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> + info.updated = false;\n> + }\n> +\n> + /* Update with new controls. */\n> + for (const auto &control : controls) {\n> + const ControlId *id =\n> device_->controls().idmap().at(control.first);\n> +\n> + if (delays_.find(id) == delays_.end())\n> + return false;\n>\n\nCould you add an error log message here? Do you think this should be a\ncontinue, my inclination is that return false is the right thing?\n\n\n> +\n> + ControlInfo &info = ctrls_[id][queueCount_];\n> +\n> + info.value = control.second;\n> + info.updated = true;\n> +\n> + LOG(DelayedControls, Debug)\n> + << \"Queuing \" << id->name()\n> + << \" to \" << info.value.toString()\n> + << \" at index \" << queueCount_;\n> + }\n> +\n> + queueCount_++;\n> +\n> + return true;\n> +}\n> +\n> +/**\n> + * \\brief Read back controls in effect at a specific sequence number\n> + * \\param[in] sequence Sequence number to get read back controls for\n> + *\n> + * Read back what controls where in effect at a specific sequence number.\n> The\n> + * history is a ring buffer of 16 entries where new and old values\n> coexist. It's\n> + * the callers responsibility to not read to old sequence numbers that\n> have been\n> + * pushed out of the history.\n> + *\n> + * Historic values are evicted by pushing new values onto the queue using\n> + * push(). The max history from the current sequence number that yields\n> valid\n> + * values are thus 16 minus number of controls pushed.\n> + *\n> + * \\returns List of controls in effect at \\a sequence\n> + */\n> +ControlList DelayedControls::get(uint32_t sequence)\n> +{\n> + std::lock_guard lock(lock_);\n> +\n> + uint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> + unsigned int index = std::max(0, adjustedSeq - maxDelay_);\n>\n\nProbably my misunderstanding, but will firstSequence ever be anything but 0?\n\n\n> +\n> + ControlList out(device_->controls());\n> + for (const auto &ctrl : ctrls_) {\n> + const ControlId *id = ctrl.first;\n> + const ControlInfo &info = ctrl.second[index];\n> +\n> + out.set(id->id(), info.value);\n> +\n> + LOG(DelayedControls, Debug)\n> + << \"Reading \" << id->name()\n> + << \" to \" << info.value.toString()\n> + << \" at index \" << index;\n> + }\n> +\n> + return out;\n> +}\n> +\n> +/**\n> + * \\brief Inform DelayedControls of a start of a new frame\n> + * \\param[in] sequence Sequence number of the frame that started\n> + *\n> + * Inform the state machine that a new frame have started and it's\n> sequence\n> + * number. It's user of this helpers responsibility to inform the helper\n> + * at the start of every frame. This can with ease be connected to the\n> start\n> + * of exposure (SOE) V4L2 event.\n> + */\n> +void DelayedControls::frameStart(uint32_t sequence)\n> +{\n> + LOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> +\n> + std::lock_guard lock(lock_);\n> +\n> + if (!running_) {\n> + fistSequence_ = sequence;\n> + running_ = true;\n> + }\n>\n\nAs above, can firstSequence_ be anything but 0?\n\n\n> +\n> + /*\n> + * Create control list peaking ahead in the value queue to ensure\n> + * values are set in time to satisfy the sensor delay.\n> + */\n> + ControlList out(device_->controls());\n> + for (const auto &ctrl : ctrls_) {\n> + const ControlId *id = ctrl.first;\n> + unsigned int delayDiff = maxDelay_ - delays_[id];\n> + unsigned int index = std::max(0, writeCount_ -\n> delayDiff);\n> + const ControlInfo &info = ctrl.second[index];\n> +\n> + if (info.updated) {\n> + out.set(id->id(), info.value);\n> + LOG(DelayedControls, Debug)\n> + << \"Setting \" << id->name()\n> + << \" to \" << info.value.toString()\n> + << \" at index \" << index;\n> + }\n> + }\n> +\n> + writeCount_++;\n> +\n> + while (writeCount_ >= queueCount_) {\n> + LOG(DelayedControls, Debug)\n> + << \"Queue is empty, auto queue no-op.\";\n> + queue({});\n> + }\n> +\n> + device_->setControls(&out);\n> +}\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> --- a/src/libcamera/meson.build\n> +++ b/src/libcamera/meson.build\n> @@ -12,6 +12,7 @@ libcamera_sources = files([\n> 'controls.cpp',\n> 'control_serializer.cpp',\n> 'control_validator.cpp',\n> + 'delayed_controls.cpp',\n> 'device_enumerator.cpp',\n> 'device_enumerator_sysfs.cpp',\n> 'event_dispatcher.cpp',\n> --\n> 2.29.1\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0E119BDB89\n\tfor ;\n\tTue, 3 Nov 2020 10:27:26 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6EC6362BF0;\n\tTue, 3 Nov 2020 11:27:25 +0100 (CET)","from mail-lj1-x241.google.com (mail-lj1-x241.google.com\n\t[IPv6:2a00:1450:4864:20::241])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1658262679\n\tfor ;\n\tTue, 3 Nov 2020 11:27:23 +0100 (CET)","by mail-lj1-x241.google.com with SMTP id t13so18377568ljk.12\n\tfor ;\n\tTue, 03 Nov 2020 02:27:23 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"JRhbVX7R\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=IQe8UnHfsFy9iI3eZndFXCLS+ebyfX5qNjcz1naMIok=;\n\tb=JRhbVX7Rd78AT8Ew2PfIrtmh/SiqJ71Q6d92kVn1hxhgGdkOG6qOPtgiRDupER/kz6\n\tnHUkjTfe5i3tASPxvaXp3fmTirxH6p819RJFu9QgDAdkrOpsws0C8F6khVJhqnfHzF9y\n\tZ8EMsmhlBfVaZhPfkjsr63ClnywNXfOFQClDdkRu0Z4lBtenKUEZ+aVhdFN1FWV31TlN\n\tnckzMQBhI1230Pbm4i3ljcAjKdj8kKDEeTXjNR7tfUB+LRa1aM9z6oVKSjLuGDMbXy4/\n\t/H/dsyn9yYoEXRaHlEyrsPJWFxAb1dPS/YTi7tW+MN6rhWuX9snbW/DTXc5PZCCsjRcC\n\tMrGg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=IQe8UnHfsFy9iI3eZndFXCLS+ebyfX5qNjcz1naMIok=;\n\tb=gVqy6KpncIAhnQHrZKfsn843hFI2ES3T+GKMpaGjZOz9bEQad8Eux8rnuYmmsar+yk\n\t50Cjv0MSHZB3jxiJwjvIRseJ/FjmPDSdR4ZBauALAzktyLhcfp5sHD6uInLYpM6Y7wkR\n\t1TzSM/EkPAhsXq+rk99S4pglsKSWlWoEPZ2ef/iI5vte6Zt8PlxbeDlAlVZYU/cZCZY6\n\t5B/5mQi1b8JOH6YLo0+TOzX5TSNUAnWgDD/DU1KztmyYsRSD/Vsu2Jqf+TWsH8xqRKnY\n\tsvSLQThDjA1G+2nS646lgIqrQCgVP07DD6xOYdlbZqd8Q5elXEYLiLmfNOIt/LXnYMIZ\n\t35kg==","X-Gm-Message-State":"AOAM532zhF7w0D9C2UG3r7a7e9Oz7LMrV5gQIdaLtbFLFqf0O7WGygng\n\t0XvwrGd5pJVRWOnzXN3gA8zn0doM3d7R58AgmDQsrg==","X-Google-Smtp-Source":"ABdhPJzgso0UerH41O/UmXxrILuvwN33e3+VwF/gu4fgy5V5U9JVSep5C4ouVKGDaxwzH2EDg2jsLEU17gIOx2QYaWg=","X-Received":"by 2002:a2e:a410:: with SMTP id\n\tp16mr8813023ljn.252.1604399243115; \n\tTue, 03 Nov 2020 02:27:23 -0800 (PST)","MIME-Version":"1.0","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>","In-Reply-To":"<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>","From":"Naushir Patuck ","Date":"Tue, 3 Nov 2020 10:27:06 +0000","Message-ID":"","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============0161724726297729379==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13581,"web_url":"https://patchwork.libcamera.org/comment/13581/","msgid":"<20201103174207.GD17496@pendragon.ideasonboard.com>","date":"2020-11-03T17:42:07","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Naush,\n\nOn Tue, Nov 03, 2020 at 10:27:06AM +0000, Naushir Patuck wrote:\n> Hi Niklas,\n> \n> Thank you for your patch. Please see some initial comments below.\n> \n> On Wed, 28 Oct 2020 at 01:01, Niklas Söderlund wrote:\n> \n> > Some sensor controls take effect with a delay as the sensor needs time\n> > to adjust, for example exposure. Add a optional helper DelayedControls\n> > to help pipelines deal with such controls.\n> >\n> > The idea is to provide a queue of controls towards the V4L2 device and\n> > apply individual controls with the specified delay with the aim to get\n> > predictable and retrievable control values for any given frame. To do\n> > this the queue of controls needs to be at least as deep as the control\n> > with the largest delay.\n> >\n> > The DelayedControls needs to be informed of every start of exposure.\n> > This can be emulated but the helper is designed to be used with this\n> > event being provide by the kernel thru V4L2 events.\n> >\n> > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > handler but expands on its API. This helpers aims to replace the\n> > Raspberry Pi implementations and mimics it behavior perfectly.\n> >\n> > Signed-off-by: Niklas Söderlund \n> > ---\n> > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > src/libcamera/meson.build | 1 +\n> > 3 files changed, 370 insertions(+)\n> > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > create mode 100644 src/libcamera/delayed_controls.cpp\n> >\n> > diff --git a/include/libcamera/internal/delayed_controls.h\n> > b/include/libcamera/internal/delayed_controls.h\n> > new file mode 100644\n> > index 0000000000000000..df5520d240a54e4b\n> > --- /dev/null\n> > +++ b/include/libcamera/internal/delayed_controls.h\n> > @@ -0,0 +1,87 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied\n> > with a delay\n> > + */\n> \n> Given this is heavily derived from the staggered_write work, would it be ok\n> to keep the Raspberry Pi copyright in the header?\n> \n> Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n\nI've discussed this with Niklas, it was an oversight caused by using a\ntemplate. The Raspberry Pi copyright definitely applies to the code,\nhere and in the other files.\n\n[snip]","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7327DBDB1E\n\tfor ;\n\tTue, 3 Nov 2020 17:42:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id F404062C09;\n\tTue, 3 Nov 2020 18:42:56 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AB77462B7E\n\tfor ;\n\tTue, 3 Nov 2020 18:42:55 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 232B0563;\n\tTue, 3 Nov 2020 18:42:55 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"wO4dnsP2\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1604425375;\n\tbh=8KQD862KDO6UcgWCJnJDwbC14q4GlshASKXjmF3h4DY=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=wO4dnsP2PaH1H854zYlX5hvjdzIznq38e97kLotZSEcMNbHpI1pV/5AFWX5Y3PMZt\n\t69sJguueZny4Lu9ZkuSoa6F54s/P2CDatHR6CVmABucvRjJCuVi3owTFVpRsjsywcX\n\tlbsxAoHM7pla5b/RCtGPJw1ONG0Y4Y2ROTIB5224=","Date":"Tue, 3 Nov 2020 19:42:07 +0200","From":"Laurent Pinchart ","To":"Naushir Patuck ","Message-ID":"<20201103174207.GD17496@pendragon.ideasonboard.com>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13604,"web_url":"https://patchwork.libcamera.org/comment/13604/","msgid":"<20201104160800.ethen7ty3al5cug2@uno.localdomain>","date":"2020-11-04T16:08:00","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi Niklas,\n\nOn Wed, Oct 28, 2020 at 02:00:44AM +0100, Niklas Söderlund wrote:\n> Some sensor controls take effect with a delay as the sensor needs time\n> to adjust, for example exposure. Add a optional helper DelayedControls\n> to help pipelines deal with such controls.\n>\n> The idea is to provide a queue of controls towards the V4L2 device and\n> apply individual controls with the specified delay with the aim to get\n> predictable and retrievable control values for any given frame. To do\n> this the queue of controls needs to be at least as deep as the control\n> with the largest delay.\n>\n> The DelayedControls needs to be informed of every start of exposure.\n> This can be emulated but the helper is designed to be used with this\n> event being provide by the kernel thru V4L2 events.\n\ns/thru/though ?\n>\n> This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> handler but expands on its API. This helpers aims to replace the\n> Raspberry Pi implementations and mimics it behavior perfectly.\n>\n> Signed-off-by: Niklas Söderlund \n> ---\n> include/libcamera/internal/delayed_controls.h | 87 ++++++\n> src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> src/libcamera/meson.build | 1 +\n> 3 files changed, 370 insertions(+)\n> create mode 100644 include/libcamera/internal/delayed_controls.h\n> create mode 100644 src/libcamera/delayed_controls.cpp\n>\n> diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h\n> new file mode 100644\n> index 0000000000000000..df5520d240a54e4b\n> --- /dev/null\n> +++ b/include/libcamera/internal/delayed_controls.h\n> @@ -0,0 +1,87 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> + */\n> +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +namespace libcamera {\n> +\n> +class V4L2Device;\n> +\n> +class DelayedControls\n> +{\n> +public:\n> +\tDelayedControls(V4L2Device *device,\n> +\t\t\tconst std::unordered_map &delays);\n> +\n> +\tvoid reset(ControlList *controls = nullptr);\n> +\n> +\tbool push(const ControlList &controls);\n> +\tControlList get(uint32_t sequence);\n> +\n> +\tvoid frameStart(uint32_t sequence);\n> +\n> +private:\n> +\tclass ControlInfo\n\nWith no other scope but public:, is it worth a class ?\n\n> +\t{\n> +\tpublic:\n> +\t\tControlInfo()\n> +\t\t\t: updated(false)\n> +\t\t{\n> +\t\t}\n> +\n> +\t\tControlInfo(const ControlValue &v)\n> +\t\t\t: value(v), updated(true)\n> +\t\t{\n> +\t\t}\n> +\n> +\t\tControlValue value;\n> +\t\tbool updated;\n> +\t};\n> +\n> +\tstatic constexpr int listSize = 16;\n> +\tclass ControlArray : public std::array\n> +\t{\n> +\tpublic:\n> +\t\tControlInfo &operator[](unsigned int index)\n> +\t\t{\n> +\t\t\treturn std::array::operator[](index % listSize);\n> +\t\t}\n> +\n> +\t\tconst ControlInfo &operator[](unsigned int index) const\n> +\t\t{\n> +\t\t\treturn std::array::operator[](index % listSize);\n> +\t\t}\n> +\t};\n> +\n> +\tusing ControlsDelays = std::unordered_map;\n> +\tusing ControlsValues = std::unordered_map;\n> +\n> +\tbool queue(const ControlList &controls);\n> +\n> +\tstd::mutex lock_;\n> +\n> +\tV4L2Device *device_;\n> +\tControlsDelays delays_;\n> +\tunsigned int maxDelay_;\n> +\n> +\tbool running_;\n> +\tuint32_t fistSequence_;\n\nShould this be 'first' ?\n\n> +\n> +\tuint32_t queueCount_;\n> +\tuint32_t writeCount_;\n> +\tControlsValues ctrls_;\n> +};\n> +\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp\n> new file mode 100644\n> index 0000000000000000..0e32f417c5cc68b7\n> --- /dev/null\n> +++ b/src/libcamera/delayed_controls.cpp\n> @@ -0,0 +1,282 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> + */\n> +\n> +#include \"libcamera/internal/delayed_controls.h\"\n> +#include \"libcamera/internal/v4l2_device.h\"\n> +\n> +#include \n> +\n> +#include \"libcamera/internal/log.h\"\n> +\n> +/**\n> + * \\file delayed_controls.h\n> + * \\brief Helper to deal with controls that are applied with a delay\n> + */\n> +\n> +namespace libcamera {\n> +\n> +LOG_DEFINE_CATEGORY(DelayedControls)\n> +\n> +/**\n> + * \\class DelayedControls\n> + * \\brief Helper to deal with controls that takes effect with a delay\n\ncontrols -> take\n\n> + *\n> + * Some sensor controls take effect with a delay as the sensor needs time to\n\ncontrols -> take\n\n> + * adjust, for example exposure and focus. This is an optional helper class to\n> + * deal with such controls and the intended user is pipeline handlers.\n\nusers are pipeline handlers\n\n> + *\n> + * The idea is to extend the concept of the pipeline depth the users needs to\n> + * maintain for buffers to controls. The depth is determined with by the control\n> + * with the grates delay. As long as the pipeline keeps the control queue above\n> + * this level the control values are guaranteed to be in effect at the specified\n> + * point in time.\n\nI would drop this as it mentions other concepts that might be more\nconfusing than helping.\n\n> + *\n> + * The downside is of course that the pipeline needs to know what controls to\n> + * set control depth frames in advance. But there really is no way around this\n> + * as the delay is a consequence of the physical world. Compare this with\n> + * parameter buffers where the buffer may be queued to the device while it's\n> + * still being written to as long as it's ready before it's consumed, this is\n> + * because the parameter buffer (usually) does not contain controls that\n> + * requires time to take effect.\n\nI find this confusing too, I'm sorry.\n\nI think the key concept is that controls have to be applied in\nadvance, the usage of \"Delayed\" took me a bit off-road as a\nDelayedControls class seems to be designed to apply controls at a\nlater (delayed) point in time while it's actually the other way\naround. One key point is that for each Control the pipeline handler\nhas to provide a known delay value, something I didn't realize until I\nlooked at the code. The class takes care of applying the control the\ncontrol in -advance- enough to make sure the control is fully applied\nat the time the Request it is associated to completes.\n\nIn the class description I would describe what the class offers and\nhow it has to be used, its API and how I should set\ncontrols specifying their 'delay'. Design documentation is good but\nonly to complement usage documentation.\n\n> + */\n> +\n> +\n> +/**\n> + * \\brief Construct a DelayedControls\n> + * \\param[in] device The V4L2 device containing the delayed controls\n\nThe V4L2 device the controls have to be applied to\n\n> + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> + * effect in frames\n\nisn't 'latency' better than delay ? Or any other term ? See how heavy\nis the description you had to make of this parameter...\n\n\\param[in] delays Map of V4L2 control ids and their associated latencies (in frames)\n\n?\n> + *\n> + * Only controls specified in \\a delays are handled by the DelayedControls\n> + * instance. If it's desired to mix delayed controls and controls that takes\n> + * effect immediately the immediate controls must be listed in the \\a delays map\n> + * with a delay value of 0.\n\ndelays is a map, you have to provide a delay value by design\n\n\"Controls with an associated 0 delay (latency) are applied immediately\"\n\n\n> + */\n> +DelayedControls::DelayedControls(V4L2Device *device,\n> +\t\t\t\t const std::unordered_map &delays)\n> +\t: device_(device), maxDelay_(0)\n> +{\n> +\tconst ControlInfoMap &controls = device_->controls();\n> +\n> +\t/*\n> +\t * Sanity check that all controls where delays are requested are\n\ns/where delays//\n\n> +\t * exposed byt the device.\n\nby\n\n> +\t */\n> +\tfor (auto const &delay : delays) {\n> +\t\tunsigned int id = delay.first;\n> +\n> +\t\tif (controls.find(id) == controls.end())\n> +\t\t\tLOG(DelayedControls, Error)\n> +\t\t\t\t<< \"Delay request for control id \"\n> +\t\t\t\t<< utils::hex(id)\n> +\t\t\t\t<< \" but control is not exposed by device \"\n> +\t\t\t\t<< device_->deviceNode();\n> +\t}\n> +\n> +\t/*\n> +\t * Create a map of control to delay for all controls exposed by the\n\ncontrols to delays\n\n> +\t * device. If no delay is specified assume the control applies directly.\n\nassume a 0 delay (latency)\n\n> +\t */\n> +\tfor (auto const &control : controls) {\n> +\t\tconst ControlId *id = control.first;\n> +\n> +\t\tauto it = delays.find(id->id());\n> +\t\tif (it == delays.end())\n> +\t\t\tcontinue;\n> +\n> +\t\tdelays_[id] = it->second;\n> +\n> +\t\tLOG(DelayedControls, Debug)\n> +\t\t\t<< \"Set a delay of \" << delays_[id]\n> +\t\t\t<< \" for \" << id->name();\n> +\n> +\t\tmaxDelay_ = std::max(maxDelay_, delays_[id]);\n> +\t}\n> +\n> +\treset();\n> +}\n> +\n> +/**\n> + * \\brief Reset the V4L2 device\n\nReset controls to their default values ?\n\n> + * \\param[in] controls List of controls to reset the device to or nullptr\n\nOptional list of controls to apply to the device ?\n\nThese is mostly on documentation, I'll look at implementation later!\n\nThanks\n j\n\n\n> + *\n> + * Resets the delayed controls state machine to its starting state. All controls\n> + * are fetched from the V4L2 device to provide a good starting point for the\n> + * first frames (length of control depth).\n> + *\n> + * Optionally \\a controls can be specified to set some or all of the handled\n> + * V4L2 controls prior to reading them back. If no controls needs to be set\n> + * nullptr may be used.\n> + */\n> +void DelayedControls::reset(ControlList *controls)\n> +{\n> +\tstd::lock_guard lock(lock_);\n> +\n> +\trunning_ = false;\n> +\tfistSequence_ = 0;\n> +\tqueueCount_ = 0;\n> +\twriteCount_ = 0;\n> +\n> +\t/* Set the controls on the device if requested. */\n> +\tif (controls)\n> +\t\tdevice_->setControls(controls);\n> +\n> +\t/* Retrieve current control values reported by the device. */\n> +\tstd::vector ids;\n> +\tfor (auto const &delay : delays_)\n> +\t\tids.push_back(delay.first->id());\n> +\n> +\tControlList devCtrls = device_->getControls(ids);\n> +\n> +\t/* Seed the control queue with the controls reported by the device. */\n> +\tctrls_.clear();\n> +\tfor (const auto &ctrl : devCtrls) {\n> +\t\tconst ControlId *id = devCtrls.infoMap()->idmap().at(ctrl.first);\n> +\t\tctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> +\t}\n> +\n> +\tqueueCount_++;\n> +}\n> +\n> +/**\n> + * \\brief Push a set of controls on the queue\n> + * \\param[in] controls List of controls to add the device queue\n> + *\n> + * Push a set of controls to the control queue. This increases the control queue\n> + * depth by one.\n> + *\n> + * \\returns true if \\a controls are accepted, or false otherwise\n> + */\n> +bool DelayedControls::push(const ControlList &controls)\n> +{\n> +\tstd::lock_guard lock(lock_);\n> +\n> +\treturn queue(controls);\n> +}\n> +\n> +bool DelayedControls::queue(const ControlList &controls)\n> +{\n> +\t/* Copy state from previous frame. */\n> +\tfor (auto &ctrl : ctrls_) {\n> +\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> +\t\tinfo.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> +\t\tinfo.updated = false;\n> +\t}\n> +\n> +\t/* Update with new controls. */\n> +\tfor (const auto &control : controls) {\n> +\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> +\n> +\t\tif (delays_.find(id) == delays_.end())\n> +\t\t\treturn false;\n> +\n> +\t\tControlInfo &info = ctrls_[id][queueCount_];\n> +\n> +\t\tinfo.value = control.second;\n> +\t\tinfo.updated = true;\n> +\n> +\t\tLOG(DelayedControls, Debug)\n> +\t\t\t<< \"Queuing \" << id->name()\n> +\t\t\t<< \" to \" << info.value.toString()\n> +\t\t\t<< \" at index \" << queueCount_;\n> +\t}\n> +\n> +\tqueueCount_++;\n> +\n> +\treturn true;\n> +}\n> +\n> +/**\n> + * \\brief Read back controls in effect at a specific sequence number\n> + * \\param[in] sequence Sequence number to get read back controls for\n> + *\n> + * Read back what controls where in effect at a specific sequence number. The\n> + * history is a ring buffer of 16 entries where new and old values coexist. It's\n> + * the callers responsibility to not read to old sequence numbers that have been\n> + * pushed out of the history.\n> + *\n> + * Historic values are evicted by pushing new values onto the queue using\n> + * push(). The max history from the current sequence number that yields valid\n> + * values are thus 16 minus number of controls pushed.\n> + *\n> + * \\returns List of controls in effect at \\a sequence\n> + */\n> +ControlList DelayedControls::get(uint32_t sequence)\n> +{\n> +\tstd::lock_guard lock(lock_);\n> +\n> +\tuint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> +\tunsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> +\n> +\tControlList out(device_->controls());\n> +\tfor (const auto &ctrl : ctrls_) {\n> +\t\tconst ControlId *id = ctrl.first;\n> +\t\tconst ControlInfo &info = ctrl.second[index];\n> +\n> +\t\tout.set(id->id(), info.value);\n> +\n> +\t\tLOG(DelayedControls, Debug)\n> +\t\t\t<< \"Reading \" << id->name()\n> +\t\t\t<< \" to \" << info.value.toString()\n> +\t\t\t<< \" at index \" << index;\n> +\t}\n> +\n> +\treturn out;\n> +}\n> +\n> +/**\n> + * \\brief Inform DelayedControls of a start of a new frame\n> + * \\param[in] sequence Sequence number of the frame that started\n> + *\n> + * Inform the state machine that a new frame have started and it's sequence\n> + * number. It's user of this helpers responsibility to inform the helper\n> + * at the start of every frame. This can with ease be connected to the start\n> + * of exposure (SOE) V4L2 event.\n> + */\n> +void DelayedControls::frameStart(uint32_t sequence)\n> +{\n> +\tLOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> +\n> +\tstd::lock_guard lock(lock_);\n> +\n> +\tif (!running_) {\n> +\t\tfistSequence_ = sequence;\n> +\t\trunning_ = true;\n> +\t}\n> +\n> +\t/*\n> +\t * Create control list peaking ahead in the value queue to ensure\n> +\t * values are set in time to satisfy the sensor delay.\n> +\t */\n> +\tControlList out(device_->controls());\n> +\tfor (const auto &ctrl : ctrls_) {\n> +\t\tconst ControlId *id = ctrl.first;\n> +\t\tunsigned int delayDiff = maxDelay_ - delays_[id];\n> +\t\tunsigned int index = std::max(0, writeCount_ - delayDiff);\n> +\t\tconst ControlInfo &info = ctrl.second[index];\n> +\n> +\t\tif (info.updated) {\n> +\t\t\tout.set(id->id(), info.value);\n> +\t\t\tLOG(DelayedControls, Debug)\n> +\t\t\t\t<< \"Setting \" << id->name()\n> +\t\t\t\t<< \" to \" << info.value.toString()\n> +\t\t\t\t<< \" at index \" << index;\n> +\t\t}\n> +\t}\n> +\n> +\twriteCount_++;\n> +\n> +\twhile (writeCount_ >= queueCount_) {\n> +\t\tLOG(DelayedControls, Debug)\n> +\t\t\t<< \"Queue is empty, auto queue no-op.\";\n> +\t\tqueue({});\n> +\t}\n> +\n> +\tdevice_->setControls(&out);\n> +}\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> --- a/src/libcamera/meson.build\n> +++ b/src/libcamera/meson.build\n> @@ -12,6 +12,7 @@ libcamera_sources = files([\n> 'controls.cpp',\n> 'control_serializer.cpp',\n> 'control_validator.cpp',\n> + 'delayed_controls.cpp',\n> 'device_enumerator.cpp',\n> 'device_enumerator_sysfs.cpp',\n> 'event_dispatcher.cpp',\n> --\n> 2.29.1\n>\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id F099ABE080\n\tfor ;\n\tWed, 4 Nov 2020 16:08:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9269262C85;\n\tWed, 4 Nov 2020 17:08:05 +0100 (CET)","from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net\n\t[217.70.183.193])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2CF8362B91\n\tfor ;\n\tWed, 4 Nov 2020 17:08:04 +0100 (CET)","from uno.localdomain (2-224-242-101.ip172.fastwebnet.it\n\t[2.224.242.101]) (Authenticated sender: jacopo@jmondi.org)\n\tby relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 12426240004;\n\tWed, 4 Nov 2020 16:08:02 +0000 (UTC)"],"X-Originating-IP":"2.224.242.101","Date":"Wed, 4 Nov 2020 17:08:00 +0100","From":"Jacopo Mondi ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Message-ID":"<20201104160800.ethen7ty3al5cug2@uno.localdomain>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13605,"web_url":"https://patchwork.libcamera.org/comment/13605/","msgid":"","date":"2020-11-04T16:18:00","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Jacopo and Niklas,\n\nOn Wed, 4 Nov 2020 at 16:08, Jacopo Mondi wrote:\n\n> Hi Niklas,\n>\n> On Wed, Oct 28, 2020 at 02:00:44AM +0100, Niklas Söderlund wrote:\n> > Some sensor controls take effect with a delay as the sensor needs time\n> > to adjust, for example exposure. Add a optional helper DelayedControls\n> > to help pipelines deal with such controls.\n> >\n> > The idea is to provide a queue of controls towards the V4L2 device and\n> > apply individual controls with the specified delay with the aim to get\n> > predictable and retrievable control values for any given frame. To do\n> > this the queue of controls needs to be at least as deep as the control\n> > with the largest delay.\n> >\n> > The DelayedControls needs to be informed of every start of exposure.\n> > This can be emulated but the helper is designed to be used with this\n> > event being provide by the kernel thru V4L2 events.\n>\n> s/thru/though ?\n> >\n> > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > handler but expands on its API. This helpers aims to replace the\n> > Raspberry Pi implementations and mimics it behavior perfectly.\n> >\n> > Signed-off-by: Niklas Söderlund \n> > ---\n> > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > src/libcamera/meson.build | 1 +\n> > 3 files changed, 370 insertions(+)\n> > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > create mode 100644 src/libcamera/delayed_controls.cpp\n> >\n> > diff --git a/include/libcamera/internal/delayed_controls.h\n> b/include/libcamera/internal/delayed_controls.h\n> > new file mode 100644\n> > index 0000000000000000..df5520d240a54e4b\n> > --- /dev/null\n> > +++ b/include/libcamera/internal/delayed_controls.h\n> > @@ -0,0 +1,87 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied\n> with a delay\n> > + */\n> > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +namespace libcamera {\n> > +\n> > +class V4L2Device;\n> > +\n> > +class DelayedControls\n> > +{\n> > +public:\n> > + DelayedControls(V4L2Device *device,\n> > + const std::unordered_map\n> &delays);\n> > +\n> > + void reset(ControlList *controls = nullptr);\n> > +\n> > + bool push(const ControlList &controls);\n> > + ControlList get(uint32_t sequence);\n> > +\n> > + void frameStart(uint32_t sequence);\n> > +\n> > +private:\n> > + class ControlInfo\n>\n> With no other scope but public:, is it worth a class ?\n>\n> > + {\n> > + public:\n> > + ControlInfo()\n> > + : updated(false)\n> > + {\n> > + }\n> > +\n> > + ControlInfo(const ControlValue &v)\n> > + : value(v), updated(true)\n> > + {\n> > + }\n> > +\n> > + ControlValue value;\n> > + bool updated;\n> > + };\n> > +\n> > + static constexpr int listSize = 16;\n> > + class ControlArray : public std::array\n> > + {\n> > + public:\n> > + ControlInfo &operator[](unsigned int index)\n> > + {\n> > + return std::array listSize>::operator[](index % listSize);\n> > + }\n> > +\n> > + const ControlInfo &operator[](unsigned int index) const\n> > + {\n> > + return std::array listSize>::operator[](index % listSize);\n> > + }\n> > + };\n> > +\n> > + using ControlsDelays = std::unordered_map unsigned int>;\n> > + using ControlsValues = std::unordered_map ControlArray>;\n> > +\n> > + bool queue(const ControlList &controls);\n> > +\n> > + std::mutex lock_;\n> > +\n> > + V4L2Device *device_;\n> > + ControlsDelays delays_;\n> > + unsigned int maxDelay_;\n> > +\n> > + bool running_;\n> > + uint32_t fistSequence_;\n>\n> Should this be 'first' ?\n>\n> > +\n> > + uint32_t queueCount_;\n> > + uint32_t writeCount_;\n> > + ControlsValues ctrls_;\n> > +};\n> > +\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > diff --git a/src/libcamera/delayed_controls.cpp\n> b/src/libcamera/delayed_controls.cpp\n> > new file mode 100644\n> > index 0000000000000000..0e32f417c5cc68b7\n> > --- /dev/null\n> > +++ b/src/libcamera/delayed_controls.cpp\n> > @@ -0,0 +1,282 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied\n> with a delay\n> > + */\n> > +\n> > +#include \"libcamera/internal/delayed_controls.h\"\n> > +#include \"libcamera/internal/v4l2_device.h\"\n> > +\n> > +#include \n> > +\n> > +#include \"libcamera/internal/log.h\"\n> > +\n> > +/**\n> > + * \\file delayed_controls.h\n> > + * \\brief Helper to deal with controls that are applied with a delay\n> > + */\n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > +\n> > +/**\n> > + * \\class DelayedControls\n> > + * \\brief Helper to deal with controls that takes effect with a delay\n>\n> controls -> take\n>\n> > + *\n> > + * Some sensor controls take effect with a delay as the sensor needs\n> time to\n>\n> controls -> take\n>\n> > + * adjust, for example exposure and focus. This is an optional helper\n> class to\n> > + * deal with such controls and the intended user is pipeline handlers.\n>\n> users are pipeline handlers\n>\n> > + *\n> > + * The idea is to extend the concept of the pipeline depth the users\n> needs to\n> > + * maintain for buffers to controls. The depth is determined with by\n> the control\n> > + * with the grates delay. As long as the pipeline keeps the control\n> queue above\n> > + * this level the control values are guaranteed to be in effect at the\n> specified\n> > + * point in time.\n>\n> I would drop this as it mentions other concepts that might be more\n> confusing than helping.\n>\n> > + *\n> > + * The downside is of course that the pipeline needs to know what\n> controls to\n> > + * set control depth frames in advance. But there really is no way\n> around this\n> > + * as the delay is a consequence of the physical world. Compare this\n> with\n> > + * parameter buffers where the buffer may be queued to the device while\n> it's\n> > + * still being written to as long as it's ready before it's consumed,\n> this is\n> > + * because the parameter buffer (usually) does not contain controls that\n> > + * requires time to take effect.\n>\n> I find this confusing too, I'm sorry.\n>\n> I think the key concept is that controls have to be applied in\n> advance, the usage of \"Delayed\" took me a bit off-road as a\n> DelayedControls class seems to be designed to apply controls at a\n> later (delayed) point in time while it's actually the other way\n> around. One key point is that for each Control the pipeline handler\n> has to provide a known delay value, something I didn't realize until I\n> looked at the code. The class takes care of applying the control the\n> control in -advance- enough to make sure the control is fully applied\n> at the time the Request it is associated to completes.\n>\n\nJust wanted to add a bit to this. The intention of this class (and the\nparent staggered write class) for the Raspberry Pi pipeline is to ensure\nthat gain/exposure/vblank/etc. get *applied* by the sensor on the same\nframe even though the relative delays between each control will be\ndifferent. This does not bear any relation to synchronising with a Request\n- although it could be adapted for that. If the use of \"Delayed\" may cause\nconfusion, perhaps we can use \"Staggered\" to indicate control writes are\nstaggered to account for the relative delay differences, but I am biased in\nthat choice ;-)\n\nRegards,\nNaush\n\n\n\n>\n> In the class description I would describe what the class offers and\n> how it has to be used, its API and how I should set\n> controls specifying their 'delay'. Design documentation is good but\n> only to complement usage documentation.\n>\n> > + */\n> > +\n> > +\n> > +/**\n> > + * \\brief Construct a DelayedControls\n> > + * \\param[in] device The V4L2 device containing the delayed controls\n>\n> The V4L2 device the controls have to be applied to\n>\n> > + * \\param[in] delays Map of numerical V4L2 control id to its delay to\n> take\n> > + * effect in frames\n>\n> isn't 'latency' better than delay ? Or any other term ? See how heavy\n> is the description you had to make of this parameter...\n>\n> \\param[in] delays Map of V4L2 control ids and their associated latencies\n> (in frames)\n>\n> ?\n> > + *\n> > + * Only controls specified in \\a delays are handled by the\n> DelayedControls\n> > + * instance. If it's desired to mix delayed controls and controls that\n> takes\n> > + * effect immediately the immediate controls must be listed in the \\a\n> delays map\n> > + * with a delay value of 0.\n>\n> delays is a map, you have to provide a delay value by design\n>\n> \"Controls with an associated 0 delay (latency) are applied immediately\"\n>\n>\n> > + */\n> > +DelayedControls::DelayedControls(V4L2Device *device,\n> > + const std::unordered_map unsigned int> &delays)\n> > + : device_(device), maxDelay_(0)\n> > +{\n> > + const ControlInfoMap &controls = device_->controls();\n> > +\n> > + /*\n> > + * Sanity check that all controls where delays are requested are\n>\n> s/where delays//\n>\n> > + * exposed byt the device.\n>\n> by\n>\n> > + */\n> > + for (auto const &delay : delays) {\n> > + unsigned int id = delay.first;\n> > +\n> > + if (controls.find(id) == controls.end())\n> > + LOG(DelayedControls, Error)\n> > + << \"Delay request for control id \"\n> > + << utils::hex(id)\n> > + << \" but control is not exposed by device \"\n> > + << device_->deviceNode();\n> > + }\n> > +\n> > + /*\n> > + * Create a map of control to delay for all controls exposed by the\n>\n> controls to delays\n>\n> > + * device. If no delay is specified assume the control applies\n> directly.\n>\n> assume a 0 delay (latency)\n>\n> > + */\n> > + for (auto const &control : controls) {\n> > + const ControlId *id = control.first;\n> > +\n> > + auto it = delays.find(id->id());\n> > + if (it == delays.end())\n> > + continue;\n> > +\n> > + delays_[id] = it->second;\n> > +\n> > + LOG(DelayedControls, Debug)\n> > + << \"Set a delay of \" << delays_[id]\n> > + << \" for \" << id->name();\n> > +\n> > + maxDelay_ = std::max(maxDelay_, delays_[id]);\n> > + }\n> > +\n> > + reset();\n> > +}\n> > +\n> > +/**\n> > + * \\brief Reset the V4L2 device\n>\n> Reset controls to their default values ?\n>\n> > + * \\param[in] controls List of controls to reset the device to or\n> nullptr\n>\n> Optional list of controls to apply to the device ?\n>\n> These is mostly on documentation, I'll look at implementation later!\n>\n> Thanks\n> j\n>\n>\n> > + *\n> > + * Resets the delayed controls state machine to its starting state. All\n> controls\n> > + * are fetched from the V4L2 device to provide a good starting point\n> for the\n> > + * first frames (length of control depth).\n> > + *\n> > + * Optionally \\a controls can be specified to set some or all of the\n> handled\n> > + * V4L2 controls prior to reading them back. If no controls needs to be\n> set\n> > + * nullptr may be used.\n> > + */\n> > +void DelayedControls::reset(ControlList *controls)\n> > +{\n> > + std::lock_guard lock(lock_);\n> > +\n> > + running_ = false;\n> > + fistSequence_ = 0;\n> > + queueCount_ = 0;\n> > + writeCount_ = 0;\n> > +\n> > + /* Set the controls on the device if requested. */\n> > + if (controls)\n> > + device_->setControls(controls);\n> > +\n> > + /* Retrieve current control values reported by the device. */\n> > + std::vector ids;\n> > + for (auto const &delay : delays_)\n> > + ids.push_back(delay.first->id());\n> > +\n> > + ControlList devCtrls = device_->getControls(ids);\n> > +\n> > + /* Seed the control queue with the controls reported by the\n> device. */\n> > + ctrls_.clear();\n> > + for (const auto &ctrl : devCtrls) {\n> > + const ControlId *id =\n> devCtrls.infoMap()->idmap().at(ctrl.first);\n> > + ctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > + }\n> > +\n> > + queueCount_++;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Push a set of controls on the queue\n> > + * \\param[in] controls List of controls to add the device queue\n> > + *\n> > + * Push a set of controls to the control queue. This increases the\n> control queue\n> > + * depth by one.\n> > + *\n> > + * \\returns true if \\a controls are accepted, or false otherwise\n> > + */\n> > +bool DelayedControls::push(const ControlList &controls)\n> > +{\n> > + std::lock_guard lock(lock_);\n> > +\n> > + return queue(controls);\n> > +}\n> > +\n> > +bool DelayedControls::queue(const ControlList &controls)\n> > +{\n> > + /* Copy state from previous frame. */\n> > + for (auto &ctrl : ctrls_) {\n> > + ControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> > + info.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > + info.updated = false;\n> > + }\n> > +\n> > + /* Update with new controls. */\n> > + for (const auto &control : controls) {\n> > + const ControlId *id =\n> device_->controls().idmap().at(control.first);\n> > +\n> > + if (delays_.find(id) == delays_.end())\n> > + return false;\n> > +\n> > + ControlInfo &info = ctrls_[id][queueCount_];\n> > +\n> > + info.value = control.second;\n> > + info.updated = true;\n> > +\n> > + LOG(DelayedControls, Debug)\n> > + << \"Queuing \" << id->name()\n> > + << \" to \" << info.value.toString()\n> > + << \" at index \" << queueCount_;\n> > + }\n> > +\n> > + queueCount_++;\n> > +\n> > + return true;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Read back controls in effect at a specific sequence number\n> > + * \\param[in] sequence Sequence number to get read back controls for\n> > + *\n> > + * Read back what controls where in effect at a specific sequence\n> number. The\n> > + * history is a ring buffer of 16 entries where new and old values\n> coexist. It's\n> > + * the callers responsibility to not read to old sequence numbers that\n> have been\n> > + * pushed out of the history.\n> > + *\n> > + * Historic values are evicted by pushing new values onto the queue\n> using\n> > + * push(). The max history from the current sequence number that yields\n> valid\n> > + * values are thus 16 minus number of controls pushed.\n> > + *\n> > + * \\returns List of controls in effect at \\a sequence\n> > + */\n> > +ControlList DelayedControls::get(uint32_t sequence)\n> > +{\n> > + std::lock_guard lock(lock_);\n> > +\n> > + uint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> > + unsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > +\n> > + ControlList out(device_->controls());\n> > + for (const auto &ctrl : ctrls_) {\n> > + const ControlId *id = ctrl.first;\n> > + const ControlInfo &info = ctrl.second[index];\n> > +\n> > + out.set(id->id(), info.value);\n> > +\n> > + LOG(DelayedControls, Debug)\n> > + << \"Reading \" << id->name()\n> > + << \" to \" << info.value.toString()\n> > + << \" at index \" << index;\n> > + }\n> > +\n> > + return out;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Inform DelayedControls of a start of a new frame\n> > + * \\param[in] sequence Sequence number of the frame that started\n> > + *\n> > + * Inform the state machine that a new frame have started and it's\n> sequence\n> > + * number. It's user of this helpers responsibility to inform the helper\n> > + * at the start of every frame. This can with ease be connected to the\n> start\n> > + * of exposure (SOE) V4L2 event.\n> > + */\n> > +void DelayedControls::frameStart(uint32_t sequence)\n> > +{\n> > + LOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> > +\n> > + std::lock_guard lock(lock_);\n> > +\n> > + if (!running_) {\n> > + fistSequence_ = sequence;\n> > + running_ = true;\n> > + }\n> > +\n> > + /*\n> > + * Create control list peaking ahead in the value queue to ensure\n> > + * values are set in time to satisfy the sensor delay.\n> > + */\n> > + ControlList out(device_->controls());\n> > + for (const auto &ctrl : ctrls_) {\n> > + const ControlId *id = ctrl.first;\n> > + unsigned int delayDiff = maxDelay_ - delays_[id];\n> > + unsigned int index = std::max(0, writeCount_ -\n> delayDiff);\n> > + const ControlInfo &info = ctrl.second[index];\n> > +\n> > + if (info.updated) {\n> > + out.set(id->id(), info.value);\n> > + LOG(DelayedControls, Debug)\n> > + << \"Setting \" << id->name()\n> > + << \" to \" << info.value.toString()\n> > + << \" at index \" << index;\n> > + }\n> > + }\n> > +\n> > + writeCount_++;\n> > +\n> > + while (writeCount_ >= queueCount_) {\n> > + LOG(DelayedControls, Debug)\n> > + << \"Queue is empty, auto queue no-op.\";\n> > + queue({});\n> > + }\n> > +\n> > + device_->setControls(&out);\n> > +}\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > --- a/src/libcamera/meson.build\n> > +++ b/src/libcamera/meson.build\n> > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > 'controls.cpp',\n> > 'control_serializer.cpp',\n> > 'control_validator.cpp',\n> > + 'delayed_controls.cpp',\n> > 'device_enumerator.cpp',\n> > 'device_enumerator_sysfs.cpp',\n> > 'event_dispatcher.cpp',\n> > --\n> > 2.29.1\n> >\n> > _______________________________________________\n> > libcamera-devel mailing list\n> > libcamera-devel@lists.libcamera.org\n> > https://lists.libcamera.org/listinfo/libcamera-devel\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5AE92BDB89\n\tfor ;\n\tWed, 4 Nov 2020 16:18:19 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E5E6362C94;\n\tWed, 4 Nov 2020 17:18:18 +0100 (CET)","from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com\n\t[IPv6:2a00:1450:4864:20::22c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1209562C1A\n\tfor ;\n\tWed, 4 Nov 2020 17:18:17 +0100 (CET)","by mail-lj1-x22c.google.com with SMTP id d24so23547067ljg.10\n\tfor ;\n\tWed, 04 Nov 2020 08:18:17 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"pZ4ey7EV\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=8JPieefRvdzisxdoefC6TGQlnY1wWbO2nQB2X8nq53s=;\n\tb=pZ4ey7EVS8Sf6RzBm9NqTsOntntPlbNJ2AQEW+W2NueCYOwcW2fFeYhsPIKSbLYfUC\n\tojhtB9voZu8i7N0RtY1GUJk3qL3nW/PB8Ov9ndkZoWNfKdkChbcxICmaLSjiuWFTafFj\n\tU5FE8HC07q06I57ql1axFCPNJ/+MG5xdrpDJemMtkG2eAtWMru6NfktPXJP2Lj+Wf5hc\n\t2k0zd545z7UHPASkhqMUeJ2EmJSPDYAmeKAoF0JMP3MwcO2BGlTX3l+1W8rLav+YPlJV\n\tHXOGQj93njPAEhn868C3trFPtYnizmGBpvHvjlRGJZOMa1T7fwuMVp6Q2akza0Z/8zmv\n\tEBag==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=8JPieefRvdzisxdoefC6TGQlnY1wWbO2nQB2X8nq53s=;\n\tb=TzMQMWN25tUxFxvnMf5uGTedJmgLMWO7gIdOai5nRaJwna/XVKlQbSGsUd3GTYQzVJ\n\tb2n9hFQQgDPTJrx4YEMyf/tQUpybrYlDd1qBItT1MHmPI9tL62RBK6vIt6CP2STHSOlV\n\tWlhGNZcFXRZDRG1T32WiJeqIdhX2QFrgaGiY5Yo50fQEsWTFfRFGzdHzmPn7jwtlplw3\n\tn2mphB2piVwDijt/jSMHWYj3Xnib6uciiWBUaNq8sJmtAi49cqAx/u9K3ZA3rNpZSnL3\n\tuaIvn8nJtOmoGTcq0gp/N9HKhtBA4VfrHNFGHjuEmGkhc+Q8n9ERGg4c59fHThbmrkka\n\ttGRw==","X-Gm-Message-State":"AOAM532F415+VyreSdzCyuptYWwBPN2CBqB44FevhE9h3a038leAdf3Y\n\tl/74hOjq6Ox6YhplgZTcfjvz1xi67OOL6w1FIoGt8g==","X-Google-Smtp-Source":"ABdhPJz8JbTo8qO/M19cWb4O4OibwSXhVmy1oYWqG6GNZykPJH9sutyyR80foxIgPXgx7uUtVFBP595wvZk4rhCZkSA=","X-Received":"by 2002:a2e:84c7:: with SMTP id\n\tq7mr10813482ljh.415.1604506696174; \n\tWed, 04 Nov 2020 08:18:16 -0800 (PST)","MIME-Version":"1.0","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t<20201104160800.ethen7ty3al5cug2@uno.localdomain>","In-Reply-To":"<20201104160800.ethen7ty3al5cug2@uno.localdomain>","From":"Naushir Patuck ","Date":"Wed, 4 Nov 2020 16:18:00 +0000","Message-ID":"","To":"Jacopo Mondi ","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============1251253516479233403==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13614,"web_url":"https://patchwork.libcamera.org/comment/13614/","msgid":"<20201105200848.655smknln4as62we@uno.localdomain>","date":"2020-11-05T20:08:48","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi again Niklas,\n\nOn Wed, Nov 04, 2020 at 05:08:00PM +0100, Jacopo Mondi wrote:\n> Hi Niklas,\n>\n> On Wed, Oct 28, 2020 at 02:00:44AM +0100, Niklas Söderlund wrote:\n> > Some sensor controls take effect with a delay as the sensor needs time\n> > to adjust, for example exposure. Add a optional helper DelayedControls\n> > to help pipelines deal with such controls.\n> >\n> > The idea is to provide a queue of controls towards the V4L2 device and\n> > apply individual controls with the specified delay with the aim to get\n> > predictable and retrievable control values for any given frame. To do\n> > this the queue of controls needs to be at least as deep as the control\n> > with the largest delay.\n> >\n> > The DelayedControls needs to be informed of every start of exposure.\n> > This can be emulated but the helper is designed to be used with this\n> > event being provide by the kernel thru V4L2 events.\n>\n> s/thru/though ?\n> >\n> > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > handler but expands on its API. This helpers aims to replace the\n> > Raspberry Pi implementations and mimics it behavior perfectly.\n> >\n> > Signed-off-by: Niklas Söderlund \n> > ---\n> > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > src/libcamera/meson.build | 1 +\n> > 3 files changed, 370 insertions(+)\n> > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > create mode 100644 src/libcamera/delayed_controls.cpp\n> >\n> > diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h\n> > new file mode 100644\n> > index 0000000000000000..df5520d240a54e4b\n> > --- /dev/null\n> > +++ b/include/libcamera/internal/delayed_controls.h\n> > @@ -0,0 +1,87 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > + */\n> > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +namespace libcamera {\n> > +\n> > +class V4L2Device;\n> > +\n> > +class DelayedControls\n> > +{\n> > +public:\n> > +\tDelayedControls(V4L2Device *device,\n> > +\t\t\tconst std::unordered_map &delays);\n> > +\n> > +\tvoid reset(ControlList *controls = nullptr);\n> > +\n> > +\tbool push(const ControlList &controls);\n> > +\tControlList get(uint32_t sequence);\n> > +\n> > +\tvoid frameStart(uint32_t sequence);\n> > +\n> > +private:\n> > +\tclass ControlInfo\n>\n> With no other scope but public:, is it worth a class ?\n>\n> > +\t{\n> > +\tpublic:\n> > +\t\tControlInfo()\n> > +\t\t\t: updated(false)\n> > +\t\t{\n> > +\t\t}\n> > +\n> > +\t\tControlInfo(const ControlValue &v)\n> > +\t\t\t: value(v), updated(true)\n> > +\t\t{\n> > +\t\t}\n> > +\n> > +\t\tControlValue value;\n> > +\t\tbool updated;\n> > +\t};\n> > +\n> > +\tstatic constexpr int listSize = 16;\n> > +\tclass ControlArray : public std::array\n> > +\t{\n> > +\tpublic:\n> > +\t\tControlInfo &operator[](unsigned int index)\n> > +\t\t{\n> > +\t\t\treturn std::array::operator[](index % listSize);\n> > +\t\t}\n> > +\n> > +\t\tconst ControlInfo &operator[](unsigned int index) const\n> > +\t\t{\n> > +\t\t\treturn std::array::operator[](index % listSize);\n> > +\t\t}\n> > +\t};\n> > +\n> > +\tusing ControlsDelays = std::unordered_map;\n> > +\tusing ControlsValues = std::unordered_map;\n\nDo you think indexing on ControlId is a good idea ? I see quite some\nchurn below to get from the integer id obtained by iterating a\nControlList to the ControlId used to index these maps:\n\n\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n\nWhile from ControlId we can always easily get the id back\n\n> > +\n> > +\tbool queue(const ControlList &controls);\n> > +\n> > +\tstd::mutex lock_;\n> > +\n> > +\tV4L2Device *device_;\n> > +\tControlsDelays delays_;\n> > +\tunsigned int maxDelay_;\n> > +\n> > +\tbool running_;\n> > +\tuint32_t fistSequence_;\n>\n> Should this be 'first' ?\n>\n> > +\n> > +\tuint32_t queueCount_;\n> > +\tuint32_t writeCount_;\n> > +\tControlsValues ctrls_;\n> > +};\n> > +\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp\n> > new file mode 100644\n> > index 0000000000000000..0e32f417c5cc68b7\n> > --- /dev/null\n> > +++ b/src/libcamera/delayed_controls.cpp\n> > @@ -0,0 +1,282 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > + */\n> > +\n> > +#include \"libcamera/internal/delayed_controls.h\"\n> > +#include \"libcamera/internal/v4l2_device.h\"\n> > +\n> > +#include \n> > +\n> > +#include \"libcamera/internal/log.h\"\n> > +\n> > +/**\n> > + * \\file delayed_controls.h\n> > + * \\brief Helper to deal with controls that are applied with a delay\n> > + */\n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > +\n> > +/**\n> > + * \\class DelayedControls\n> > + * \\brief Helper to deal with controls that takes effect with a delay\n>\n> controls -> take\n>\n> > + *\n> > + * Some sensor controls take effect with a delay as the sensor needs time to\n>\n> controls -> take\n>\n> > + * adjust, for example exposure and focus. This is an optional helper class to\n> > + * deal with such controls and the intended user is pipeline handlers.\n>\n> users are pipeline handlers\n>\n> > + *\n> > + * The idea is to extend the concept of the pipeline depth the users needs to\n> > + * maintain for buffers to controls. The depth is determined with by the control\n> > + * with the grates delay. As long as the pipeline keeps the control queue above\n> > + * this level the control values are guaranteed to be in effect at the specified\n> > + * point in time.\n>\n> I would drop this as it mentions other concepts that might be more\n> confusing than helping.\n>\n> > + *\n> > + * The downside is of course that the pipeline needs to know what controls to\n> > + * set control depth frames in advance. But there really is no way around this\n> > + * as the delay is a consequence of the physical world. Compare this with\n> > + * parameter buffers where the buffer may be queued to the device while it's\n> > + * still being written to as long as it's ready before it's consumed, this is\n> > + * because the parameter buffer (usually) does not contain controls that\n> > + * requires time to take effect.\n>\n> I find this confusing too, I'm sorry.\n>\n> I think the key concept is that controls have to be applied in\n> advance, the usage of \"Delayed\" took me a bit off-road as a\n> DelayedControls class seems to be designed to apply controls at a\n> later (delayed) point in time while it's actually the other way\n> around. One key point is that for each Control the pipeline handler\n> has to provide a known delay value, something I didn't realize until I\n> looked at the code. The class takes care of applying the control the\n> control in -advance- enough to make sure the control is fully applied\n> at the time the Request it is associated to completes.\n>\n> In the class description I would describe what the class offers and\n> how it has to be used, its API and how I should set\n> controls specifying their 'delay'. Design documentation is good but\n> only to complement usage documentation.\n>\n> > + */\n> > +\n\ndouble empy line\n\n> > +\n> > +/**\n> > + * \\brief Construct a DelayedControls\n> > + * \\param[in] device The V4L2 device containing the delayed controls\n>\n> The V4L2 device the controls have to be applied to\n>\n> > + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> > + * effect in frames\n>\n> isn't 'latency' better than delay ? Or any other term ? See how heavy\n> is the description you had to make of this parameter...\n>\n> \\param[in] delays Map of V4L2 control ids and their associated latencies (in frames)\n>\n> ?\n> > + *\n> > + * Only controls specified in \\a delays are handled by the DelayedControls\n> > + * instance. If it's desired to mix delayed controls and controls that takes\n> > + * effect immediately the immediate controls must be listed in the \\a delays map\n> > + * with a delay value of 0.\n>\n> delays is a map, you have to provide a delay value by design\n>\n> \"Controls with an associated 0 delay (latency) are applied immediately\"\n>\n>\n> > + */\n> > +DelayedControls::DelayedControls(V4L2Device *device,\n> > +\t\t\t\t const std::unordered_map &delays)\n> > +\t: device_(device), maxDelay_(0)\n> > +{\n> > +\tconst ControlInfoMap &controls = device_->controls();\n> > +\n> > +\t/*\n> > +\t * Sanity check that all controls where delays are requested are\n>\n> s/where delays//\n>\n\nActually I meant to make this\n Sanity check that all requested controls are exposed by the device\n\n> > +\t * exposed byt the device.\n>\n> by\n>\n> > +\t */\n> > +\tfor (auto const &delay : delays) {\n> > +\t\tunsigned int id = delay.first;\n> > +\n> > +\t\tif (controls.find(id) == controls.end())\n> > +\t\t\tLOG(DelayedControls, Error)\n> > +\t\t\t\t<< \"Delay request for control id \"\n> > +\t\t\t\t<< utils::hex(id)\n> > +\t\t\t\t<< \" but control is not exposed by device \"\n> > +\t\t\t\t<< device_->deviceNode();\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Create a map of control to delay for all controls exposed by the\n>\n> controls to delays\n>\n> > +\t * device. If no delay is specified assume the control applies directly.\n>\n> assume a 0 delay (latency)\n>\n> > +\t */\n> > +\tfor (auto const &control : controls) {\n> > +\t\tconst ControlId *id = control.first;\n> > +\n> > +\t\tauto it = delays.find(id->id());\n> > +\t\tif (it == delays.end())\n> > +\t\t\tcontinue;\n> > +\n> > +\t\tdelays_[id] = it->second;\n> > +\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Set a delay of \" << delays_[id]\n> > +\t\t\t<< \" for \" << id->name();\n> > +\n> > +\t\tmaxDelay_ = std::max(maxDelay_, delays_[id]);\n> > +\t}\n\nI wonder if these loops should not be unified. After all, if a\ndevice's control is not in the delay map it is skipped, so it is the\nsame as iterating the delay map as done in the previous loop ?\n\n> > +\n> > +\treset();\n> > +}\n> > +\n> > +/**\n> > + * \\brief Reset the V4L2 device\n>\n> Reset controls to their default values ?\n>\n> > + * \\param[in] controls List of controls to reset the device to or nullptr\n>\n> Optional list of controls to apply to the device ?\n>\n> These is mostly on documentation, I'll look at implementation later!\n>\n> Thanks\n> j\n>\n>\n> > + *\n> > + * Resets the delayed controls state machine to its starting state. All controls\n> > + * are fetched from the V4L2 device to provide a good starting point for the\n> > + * first frames (length of control depth).\n\nI don't get this last statement\n\n> > + *\n> > + * Optionally \\a controls can be specified to set some or all of the handled\n> > + * V4L2 controls prior to reading them back. If no controls needs to be set\n> > + * nullptr may be used.\n\nI would drop the last sentence\n\n> > + */\n> > +void DelayedControls::reset(ControlList *controls)\n> > +{\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\trunning_ = false;\n> > +\tfistSequence_ = 0;\n> > +\tqueueCount_ = 0;\n> > +\twriteCount_ = 0;\n> > +\n> > +\t/* Set the controls on the device if requested. */\n> > +\tif (controls)\n> > +\t\tdevice_->setControls(controls);\n> > +\n> > +\t/* Retrieve current control values reported by the device. */\n> > +\tstd::vector ids;\n> > +\tfor (auto const &delay : delays_)\n> > +\t\tids.push_back(delay.first->id());\n> > +\n> > +\tControlList devCtrls = device_->getControls(ids);\n> > +\n> > +\t/* Seed the control queue with the controls reported by the device. */\n> > +\tctrls_.clear();\n> > +\tfor (const auto &ctrl : devCtrls) {\n> > +\t\tconst ControlId *id = devCtrls.infoMap()->idmap().at(ctrl.first);\n> > +\t\tctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > +\t}\n> > +\n> > +\tqueueCount_++;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Push a set of controls on the queue\n> > + * \\param[in] controls List of controls to add the device queue\n> > + *\n> > + * Push a set of controls to the control queue. This increases the control queue\n> > + * depth by one.\n\ncontrol queue depth really confuses me.\n\nIsn't it better to just say:\n\"Push a new set of values for the controls\"\nor something similar ?\n\n> > + *\n> > + * \\returns true if \\a controls are accepted, or false otherwise\n> > + */\n> > +bool DelayedControls::push(const ControlList &controls)\n> > +{\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\treturn queue(controls);\n> > +}\n> > +\n> > +bool DelayedControls::queue(const ControlList &controls)\n> > +{\n> > +\t/* Copy state from previous frame. */\n> > +\tfor (auto &ctrl : ctrls_) {\n> > +\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n\nAm I wrong of\n\tfor (auto &ctrl : ctrls_) {\n\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n\nIs equivalent to\n\tfor (auto &ctrl : ctrls_) {\n\t\tControlInfo &info = ctrl.second[queueCount_];\n\n(the compiler agrees afaict)\n\n> > +\t\tinfo.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > +\t\tinfo.updated = false;\n> > +\t}\n\nHave you considered breaking this in 2 ? This would avoid the below\nqueue({}) which seems a workaround.\n\nCan't we:\n advance() {\n copy values from previous entry\n queueCount++;\n }\n\n update() {\n copy the new values in;\n }\n\nSo that below you only call advance() ?\n\n> > +\n> > +\t/* Update with new controls. */\n> > +\tfor (const auto &control : controls) {\n> > +\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> > +\n> > +\t\tif (delays_.find(id) == delays_.end())\n> > +\t\t\treturn false;\n> > +\n> > +\t\tControlInfo &info = ctrls_[id][queueCount_];\n\nThat's a double lookup that can be avoided. If I'm not mistaken all\ncontrols in delays_ are inserted in ctrls_, am I wrong ? If that's the\ncase you can just ctrls_.find() == ctls_.end() ?\n\n> > +\n> > +\t\tinfo.value = control.second;\n> > +\t\tinfo.updated = true;\n> > +\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Queuing \" << id->name()\n> > +\t\t\t<< \" to \" << info.value.toString()\n> > +\t\t\t<< \" at index \" << queueCount_;\n> > +\t}\n> > +\n> > +\tqueueCount_++;\n\nAh then you have to backtrack on queueCount if any of the supplied\nControl is not supported. That's anyway an error that should happen\nonly during development, if a distracted pipeline handler mis-uses\nthis class, applications have no way to get it wrong, right ?\n\nIn this case, wouldn't a LOG(Fatal) help to catch development errors\nearlier ? If you agree you can safely queueCount++, because if we\nfail, the whole library gets torn down.\n\n> > +\n> > +\treturn true;\n\nI've not looked ahead in the series, but is there value in this return\nvalue ? I mean, if a pipeline handler gets this wrong is an\nimplementation error, not a runtime one, right ?\n\n> > +}\n> > +\n> > +/**\n> > + * \\brief Read back controls in effect at a specific sequence number\n\nRead back the controls' values at a specific point in time (or\nsequence number as you've used) ?\n\n> > + * \\param[in] sequence Sequence number to get read back controls for\n\nThe sequence number used to read controls values\n\n> > + *\n> > + * Read back what controls where in effect at a specific sequence number. The\n\nMaybe it's me, but I cannot parse well \"controls where in effect\".\nNative speakers to the rescue ? I would anyway use \"controls' values\"\nhere and in other places.\n\n> > + * history is a ring buffer of 16 entries where new and old values coexist. It's\n> > + * the callers responsibility to not read to old sequence numbers that have been\n> > + * pushed out of the history.\n\nCan we invalidate them with a flag maybe ?\n\n> > + *\n> > + * Historic values are evicted by pushing new values onto the queue using\n> > + * push(). The max history from the current sequence number that yields valid\n> > + * values are thus 16 minus number of controls pushed.\n\nThis looks like class documentation I was asking for :)\n\n> > + *\n> > + * \\returns List of controls in effect at \\a sequence\n\nAs per above suggestion\n \\return The values of controls at \\a sequence time\n\n> > + */\n> > +ControlList DelayedControls::get(uint32_t sequence)\n> > +{\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\tuint32_t adjustedSeq = sequence - fistSequence_ + 1;\n\nCan it happen that sequence < firstSequence ?\n\n> > +\tunsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > +\n> > +\tControlList out(device_->controls());\n\nYou access device_->controls() quite often, can it be cached ?\n\n> > +\tfor (const auto &ctrl : ctrls_) {\n> > +\t\tconst ControlId *id = ctrl.first;\n> > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > +\n> > +\t\tout.set(id->id(), info.value);\n> > +\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Reading \" << id->name()\n> > +\t\t\t<< \" to \" << info.value.toString()\n> > +\t\t\t<< \" at index \" << index;\n> > +\t}\n> > +\n> > +\treturn out;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Inform DelayedControls of a start of a new frame\n\nMaybe I'm missing some piece that will come in the next patches, but\nwe have the video device, right ? Can't we connect to its frameStart\nsignal and handle this internally ?\n\n> > + * \\param[in] sequence Sequence number of the frame that started\n> > + *\n> > + * Inform the state machine that a new frame have started and it's sequence\n\ns/have/has\ns/it's/its\n\nSeems like a verb for \"sequence number is missing\"\n\nInform the state machine that a new frame has started and its sequence\nnumber is \\a sequence ?\n\n\n> > + * number. It's user of this helpers responsibility to inform the helper\n> > + * at the start of every frame. This can with ease be connected to the start\n> > + * of exposure (SOE) V4L2 event.\n\nIt's responsibility of the users of this class to notify the start of\nevery frame.\n(I would not mention V4L2, but I'm might be too concerned, it's fine\nif you like it)\n\n> > + */\n> > +void DelayedControls::frameStart(uint32_t sequence)\n> > +{\n> > +\tLOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> > +\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\tif (!running_) {\n> > +\t\tfistSequence_ = sequence;\n> > +\t\trunning_ = true;\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Create control list peaking ahead in the value queue to ensure\n> > +\t * values are set in time to satisfy the sensor delay.\n> > +\t */\n> > +\tControlList out(device_->controls());\n> > +\tfor (const auto &ctrl : ctrls_) {\n> > +\t\tconst ControlId *id = ctrl.first;\n> > +\t\tunsigned int delayDiff = maxDelay_ - delays_[id];\n> > +\t\tunsigned int index = std::max(0, writeCount_ - delayDiff);\n\nIt's me but I don't get this logic as I didn't get it in\nStaggeredControls. It was there already, so I'm sure it's correct, but\nI don't get it :)\n\nWhy is maxDelay_ relevant ?\n\n> > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > +\n> > +\t\tif (info.updated) {\n> > +\t\t\tout.set(id->id(), info.value);\n> > +\t\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t\t<< \"Setting \" << id->name()\n> > +\t\t\t\t<< \" to \" << info.value.toString()\n> > +\t\t\t\t<< \" at index \" << index;\n> > +\t\t}\n> > +\t}\n> > +\n> > +\twriteCount_++;\n> > +\n> > +\twhile (writeCount_ >= queueCount_) {\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Queue is empty, auto queue no-op.\";\n> > +\t\tqueue({});\n> > +\t}\n> > +\n> > +\tdevice_->setControls(&out);\n\nAny value in returning this ? This come from the device, something\nmight be happening there and maybe pipelines want to be informed ?\n\nThanks\n j\n\n> > +}\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > --- a/src/libcamera/meson.build\n> > +++ b/src/libcamera/meson.build\n> > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > 'controls.cpp',\n> > 'control_serializer.cpp',\n> > 'control_validator.cpp',\n> > + 'delayed_controls.cpp',\n> > 'device_enumerator.cpp',\n> > 'device_enumerator_sysfs.cpp',\n> > 'event_dispatcher.cpp',\n> > --\n> > 2.29.1\n> >\n> > _______________________________________________\n> > libcamera-devel mailing list\n> > libcamera-devel@lists.libcamera.org\n> > https://lists.libcamera.org/listinfo/libcamera-devel\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 77250BDB89\n\tfor ;\n\tThu, 5 Nov 2020 20:08:53 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 048C462C7D;\n\tThu, 5 Nov 2020 21:08:53 +0100 (CET)","from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net\n\t[217.70.183.199])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CEACD60353\n\tfor ;\n\tThu, 5 Nov 2020 21:08:50 +0100 (CET)","from uno.localdomain (2-224-242-101.ip172.fastwebnet.it\n\t[2.224.242.101]) (Authenticated sender: jacopo@jmondi.org)\n\tby relay9-d.mail.gandi.net (Postfix) with ESMTPSA id 398E7FF802;\n\tThu, 5 Nov 2020 20:08:49 +0000 (UTC)"],"X-Originating-IP":"2.224.242.101","Date":"Thu, 5 Nov 2020 21:08:48 +0100","From":"Jacopo Mondi ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Message-ID":"<20201105200848.655smknln4as62we@uno.localdomain>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t<20201104160800.ethen7ty3al5cug2@uno.localdomain>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201104160800.ethen7ty3al5cug2@uno.localdomain>","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13652,"web_url":"https://patchwork.libcamera.org/comment/13652/","msgid":"<20201109225424.GE88486@oden.dyn.berto.se>","date":"2020-11-09T22:54:24","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Naush,\n\nThanks for your feedback.\n\nOn 2020-11-03 10:27:06 +0000, Naushir Patuck wrote:\n> Hi Niklas,\n> \n> Thank you for your patch. Please see some initial comments below.\n> \n> Best regards,\n> Naush\n> \n> \n> On Wed, 28 Oct 2020 at 01:01, Niklas Söderlund <\n> niklas.soderlund@ragnatech.se> wrote:\n> \n> > Some sensor controls take effect with a delay as the sensor needs time\n> > to adjust, for example exposure. Add a optional helper DelayedControls\n> > to help pipelines deal with such controls.\n> >\n> > The idea is to provide a queue of controls towards the V4L2 device and\n> > apply individual controls with the specified delay with the aim to get\n> > predictable and retrievable control values for any given frame. To do\n> > this the queue of controls needs to be at least as deep as the control\n> > with the largest delay.\n> >\n> > The DelayedControls needs to be informed of every start of exposure.\n> > This can be emulated but the helper is designed to be used with this\n> > event being provide by the kernel thru V4L2 events.\n> >\n> > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > handler but expands on its API. This helpers aims to replace the\n> > Raspberry Pi implementations and mimics it behavior perfectly.\n> >\n> > Signed-off-by: Niklas Söderlund \n> > ---\n> > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > src/libcamera/meson.build | 1 +\n> > 3 files changed, 370 insertions(+)\n> > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > create mode 100644 src/libcamera/delayed_controls.cpp\n> >\n> > diff --git a/include/libcamera/internal/delayed_controls.h\n> > b/include/libcamera/internal/delayed_controls.h\n> > new file mode 100644\n> > index 0000000000000000..df5520d240a54e4b\n> > --- /dev/null\n> > +++ b/include/libcamera/internal/delayed_controls.h\n> > @@ -0,0 +1,87 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied\n> > with a delay\n> > + */\n> >\n> \n> Given this is heavily derived from the staggered_write work, would it be ok\n> to keep the Raspberry Pi copyright in the header?\n> \n> Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n\nAs discussed of-band this was always my intention, my templates got in \nthe way. Sorry about that and thanks for pointing it out.\n\n> \n> \n> > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +namespace libcamera {\n> > +\n> > +class V4L2Device;\n> > +\n> > +class DelayedControls\n> > +{\n> > +public:\n> > + DelayedControls(V4L2Device *device,\n> > + const std::unordered_map\n> > &delays);\n> > +\n> > + void reset(ControlList *controls = nullptr);\n> > +\n> > + bool push(const ControlList &controls);\n> > + ControlList get(uint32_t sequence);\n> > +\n> > + void frameStart(uint32_t sequence);\n> > +\n> > +private:\n> > + class ControlInfo\n> > + {\n> > + public:\n> > + ControlInfo()\n> > + : updated(false)\n> > + {\n> > + }\n> > +\n> > + ControlInfo(const ControlValue &v)\n> > + : value(v), updated(true)\n> > + {\n> > + }\n> > +\n> > + ControlValue value;\n> > + bool updated;\n> > + };\n> > +\n> > + static constexpr int listSize = 16;\n> > + class ControlArray : public std::array\n> > + {\n> > + public:\n> > + ControlInfo &operator[](unsigned int index)\n> > + {\n> > + return std::array > listSize>::operator[](index % listSize);\n> > + }\n> > +\n> > + const ControlInfo &operator[](unsigned int index) const\n> > + {\n> > + return std::array > listSize>::operator[](index % listSize);\n> > + }\n> > + };\n> > +\n> > + using ControlsDelays = std::unordered_map > unsigned int>;\n> > + using ControlsValues = std::unordered_map > ControlArray>;\n> > +\n> > + bool queue(const ControlList &controls);\n> > +\n> > + std::mutex lock_;\n> > +\n> > + V4L2Device *device_;\n> > + ControlsDelays delays_;\n> > + unsigned int maxDelay_;\n> > +\n> > + bool running_;\n> > + uint32_t fistSequence_;\n> > +\n> > + uint32_t queueCount_;\n> > + uint32_t writeCount_;\n> > + ControlsValues ctrls_;\n> > +};\n> > +\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > diff --git a/src/libcamera/delayed_controls.cpp\n> > b/src/libcamera/delayed_controls.cpp\n> > new file mode 100644\n> > index 0000000000000000..0e32f417c5cc68b7\n> > --- /dev/null\n> > +++ b/src/libcamera/delayed_controls.cpp\n> > @@ -0,0 +1,282 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied\n> > with a delay\n> > + */\n> >\n> \n> Similar to above:\n> Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n> \n> \n> > +\n> > +#include \"libcamera/internal/delayed_controls.h\"\n> > +#include \"libcamera/internal/v4l2_device.h\"\n> > +\n> > +#include \n> > +\n> > +#include \"libcamera/internal/log.h\"\n> > +\n> > +/**\n> > + * \\file delayed_controls.h\n> > + * \\brief Helper to deal with controls that are applied with a delay\n> > + */\n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > +\n> > +/**\n> > + * \\class DelayedControls\n> > + * \\brief Helper to deal with controls that takes effect with a delay\n> > + *\n> > + * Some sensor controls take effect with a delay as the sensor needs time\n> > to\n> > + * adjust, for example exposure and focus. This is an optional helper\n> > class to\n> > + * deal with such controls and the intended user is pipeline handlers.\n> > + *\n> > + * The idea is to extend the concept of the pipeline depth the users\n> > needs to\n> > + * maintain for buffers to controls. The depth is determined with by the\n> > control\n> > + * with the grates delay. As long as the pipeline keeps the control queue\n> > above\n> > + * this level the control values are guaranteed to be in effect at the\n> > specified\n> > + * point in time.\n> > + *\n> > + * The downside is of course that the pipeline needs to know what\n> > controls to\n> > + * set control depth frames in advance. But there really is no way around\n> > this\n> > + * as the delay is a consequence of the physical world. Compare this with\n> > + * parameter buffers where the buffer may be queued to the device while\n> > it's\n> > + * still being written to as long as it's ready before it's consumed,\n> > this is\n> > + * because the parameter buffer (usually) does not contain controls that\n> > + * requires time to take effect.\n> > + */\n> > +\n> > +\n> > +/**\n> > + * \\brief Construct a DelayedControls\n> > + * \\param[in] device The V4L2 device containing the delayed controls\n> > + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> > + * effect in frames\n> > + *\n> > + * Only controls specified in \\a delays are handled by the DelayedControls\n> > + * instance. If it's desired to mix delayed controls and controls that\n> > takes\n> > + * effect immediately the immediate controls must be listed in the \\a\n> > delays map\n> > + * with a delay value of 0.\n> > + */\n> > +DelayedControls::DelayedControls(V4L2Device *device,\n> > + const std::unordered_map > unsigned int> &delays)\n> > + : device_(device), maxDelay_(0)\n> > +{\n> > + const ControlInfoMap &controls = device_->controls();\n> > +\n> > + /*\n> > + * Sanity check that all controls where delays are requested are\n> > + * exposed byt the device.\n> > + */\n> > + for (auto const &delay : delays) {\n> > + unsigned int id = delay.first;\n> > +\n> > + if (controls.find(id) == controls.end())\n> > + LOG(DelayedControls, Error)\n> > + << \"Delay request for control id \"\n> > + << utils::hex(id)\n> > + << \" but control is not exposed by device \"\n> > + << device_->deviceNode();\n> > + }\n> > +\n> > + /*\n> > + * Create a map of control to delay for all controls exposed by the\n> > + * device. If no delay is specified assume the control applies\n> > directly.\n> > + */\n> > + for (auto const &control : controls) {\n> > + const ControlId *id = control.first;\n> > +\n> > + auto it = delays.find(id->id());\n> > + if (it == delays.end())\n> > + continue;\n> > +\n> > + delays_[id] = it->second;\n> > +\n> > + LOG(DelayedControls, Debug)\n> > + << \"Set a delay of \" << delays_[id]\n> > + << \" for \" << id->name();\n> > +\n> > + maxDelay_ = std::max(maxDelay_, delays_[id]);\n> > + }\n> >\n> \n> Could you combine the two loops here into one here? If you iterate over\n> \"delays\", you could do the error check as well as adding to the \"delays_\"\n> map.\n\nGood call!\n\n> \n> +\n> > + reset();\n> >\n> \n> Does this need to be called here? But my understanding of reset() may be\n> wrong... see below.\n\nIt does as reset() initialize the internal state machine.\n\n> \n> \n> > +}\n> > +\n> > +/**\n> > + * \\brief Reset the V4L2 device\n> > + * \\param[in] controls List of controls to reset the device to or nullptr\n> > + *\n> > + * Resets the delayed controls state machine to its starting state. All\n> > controls\n> > + * are fetched from the V4L2 device to provide a good starting point for\n> > the\n> > + * first frames (length of control depth).\n> > + *\n> > + * Optionally \\a controls can be specified to set some or all of the\n> > handled\n> > + * V4L2 controls prior to reading them back. If no controls needs to be\n> > set\n> > + * nullptr may be used.\n> > + */\n> > +void DelayedControls::reset(ControlList *controls)\n> > +{\n> > + std::lock_guard lock(lock_);\n> > +\n> > + running_ = false;\n> > + fistSequence_ = 0;\n> > + queueCount_ = 0;\n> > + writeCount_ = 0;\n> > +\n> > + /* Set the controls on the device if requested. */\n> > + if (controls)\n> > + device_->setControls(controls);\n> >\n> \n> Should this happen here in reset()? The reset method to me indicates that\n> some internal state of the class is meant to be reset, and it does nothing\n> to send controls to the device. Writing to the device could happen from\n> the caller, if they desire.\n\nMaybe this should be renamed to init() and the argument to \ninitialControls? As this function should only be called when we are not \nstreaming I see no harm in optionally allowing a init value. But maybe \nthis goes hand in had with your follow up comment below.\n\n> \n> \n> > +\n> > + /* Retrieve current control values reported by the device. */\n> > + std::vector ids;\n> > + for (auto const &delay : delays_)\n> > + ids.push_back(delay.first->id());\n> > +\n> > + ControlList devCtrls = device_->getControls(ids);\n> > +\n> > + /* Seed the control queue with the controls reported by the\n> > device. */\n> > + ctrls_.clear();\n> > + for (const auto &ctrl : devCtrls) {\n> > + const ControlId *id =\n> > devCtrls.infoMap()->idmap().at(ctrl.first);\n> > + ctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > + }\n> > +\n> > + queueCount_++;\n> > +}\n> >\n> \n> I'm a bit unclear what reset() does here. In staggered_writer, reset()\n> would essentially clear the queues and re-initialise with the last value\n> that was set. This seems to pull the value from the device, and put it\n> into the queue as a starting point. When we start streaming a sensor, the\n> settings will be applied without delay. So if we want to intialise the\n> sensor with a new set of controls, it would not work here. Also, if we\n> were running a use case where we needed to do a mode switch (e.g. preview\n> then capture), then the last requested preview controls would not be used\n> to seed the list, and would be lost.\n\nIs this not the same thing but without having to cache it? What ever \ncontrols are set on the device when we stop streaming are our starting \npoint. IIRC in StaggerdCtrls the initial value may not have been put to \nthe device but only been at the top of the queue when the streaming \nstopped. Another advantage of reading it back here is that we will catch \nany control changes done elsewhere in the pipeline perhaps as part of \nconfigure().\n\n> \n> \n> > +\n> > +/**\n> > + * \\brief Push a set of controls on the queue\n> > + * \\param[in] controls List of controls to add the device queue\n> > + *\n> > + * Push a set of controls to the control queue. This increases the\n> > control queue\n> > + * depth by one.\n> > + *\n> > + * \\returns true if \\a controls are accepted, or false otherwise\n> > + */\n> > +bool DelayedControls::push(const ControlList &controls)\n> > +{\n> > + std::lock_guard lock(lock_);\n> > +\n> > + return queue(controls);\n> > +}\n> > +\n> > +bool DelayedControls::queue(const ControlList &controls)\n> > +{\n> > + /* Copy state from previous frame. */\n> > + for (auto &ctrl : ctrls_) {\n> > + ControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> > + info.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > + info.updated = false;\n> > + }\n> > +\n> > + /* Update with new controls. */\n> > + for (const auto &control : controls) {\n> > + const ControlId *id =\n> > device_->controls().idmap().at(control.first);\n> > +\n> > + if (delays_.find(id) == delays_.end())\n> > + return false;\n> >\n> \n> Could you add an error log message here? Do you think this should be a\n> continue, my inclination is that return false is the right thing?\n\nI think this should fail as a control was queued which delay is unknown.\n\n> \n> \n> > +\n> > + ControlInfo &info = ctrls_[id][queueCount_];\n> > +\n> > + info.value = control.second;\n> > + info.updated = true;\n> > +\n> > + LOG(DelayedControls, Debug)\n> > + << \"Queuing \" << id->name()\n> > + << \" to \" << info.value.toString()\n> > + << \" at index \" << queueCount_;\n> > + }\n> > +\n> > + queueCount_++;\n> > +\n> > + return true;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Read back controls in effect at a specific sequence number\n> > + * \\param[in] sequence Sequence number to get read back controls for\n> > + *\n> > + * Read back what controls where in effect at a specific sequence number.\n> > The\n> > + * history is a ring buffer of 16 entries where new and old values\n> > coexist. It's\n> > + * the callers responsibility to not read to old sequence numbers that\n> > have been\n> > + * pushed out of the history.\n> > + *\n> > + * Historic values are evicted by pushing new values onto the queue using\n> > + * push(). The max history from the current sequence number that yields\n> > valid\n> > + * values are thus 16 minus number of controls pushed.\n> > + *\n> > + * \\returns List of controls in effect at \\a sequence\n> > + */\n> > +ControlList DelayedControls::get(uint32_t sequence)\n> > +{\n> > + std::lock_guard lock(lock_);\n> > +\n> > + uint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> > + unsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> >\n> \n> Probably my misunderstanding, but will firstSequence ever be anything but 0?\n\nUnfortunately yes. Some V4L2 drivers does not reset their sequence \nnumber to 0 when starting to stream. So reputing start/stop may have the \nfirst frame seq+1 from the previous stop.\n\n> \n> \n> > +\n> > + ControlList out(device_->controls());\n> > + for (const auto &ctrl : ctrls_) {\n> > + const ControlId *id = ctrl.first;\n> > + const ControlInfo &info = ctrl.second[index];\n> > +\n> > + out.set(id->id(), info.value);\n> > +\n> > + LOG(DelayedControls, Debug)\n> > + << \"Reading \" << id->name()\n> > + << \" to \" << info.value.toString()\n> > + << \" at index \" << index;\n> > + }\n> > +\n> > + return out;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Inform DelayedControls of a start of a new frame\n> > + * \\param[in] sequence Sequence number of the frame that started\n> > + *\n> > + * Inform the state machine that a new frame have started and it's\n> > sequence\n> > + * number. It's user of this helpers responsibility to inform the helper\n> > + * at the start of every frame. This can with ease be connected to the\n> > start\n> > + * of exposure (SOE) V4L2 event.\n> > + */\n> > +void DelayedControls::frameStart(uint32_t sequence)\n> > +{\n> > + LOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> > +\n> > + std::lock_guard lock(lock_);\n> > +\n> > + if (!running_) {\n> > + fistSequence_ = sequence;\n> > + running_ = true;\n> > + }\n> >\n> \n> As above, can firstSequence_ be anything but 0?\n> \n> \n> > +\n> > + /*\n> > + * Create control list peaking ahead in the value queue to ensure\n> > + * values are set in time to satisfy the sensor delay.\n> > + */\n> > + ControlList out(device_->controls());\n> > + for (const auto &ctrl : ctrls_) {\n> > + const ControlId *id = ctrl.first;\n> > + unsigned int delayDiff = maxDelay_ - delays_[id];\n> > + unsigned int index = std::max(0, writeCount_ -\n> > delayDiff);\n> > + const ControlInfo &info = ctrl.second[index];\n> > +\n> > + if (info.updated) {\n> > + out.set(id->id(), info.value);\n> > + LOG(DelayedControls, Debug)\n> > + << \"Setting \" << id->name()\n> > + << \" to \" << info.value.toString()\n> > + << \" at index \" << index;\n> > + }\n> > + }\n> > +\n> > + writeCount_++;\n> > +\n> > + while (writeCount_ >= queueCount_) {\n> > + LOG(DelayedControls, Debug)\n> > + << \"Queue is empty, auto queue no-op.\";\n> > + queue({});\n> > + }\n> > +\n> > + device_->setControls(&out);\n> > +}\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > --- a/src/libcamera/meson.build\n> > +++ b/src/libcamera/meson.build\n> > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > 'controls.cpp',\n> > 'control_serializer.cpp',\n> > 'control_validator.cpp',\n> > + 'delayed_controls.cpp',\n> > 'device_enumerator.cpp',\n> > 'device_enumerator_sysfs.cpp',\n> > 'event_dispatcher.cpp',\n> > --\n> > 2.29.1\n> >\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D9797BDB89\n\tfor ;\n\tMon, 9 Nov 2020 22:54:28 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4768063090;\n\tMon, 9 Nov 2020 23:54:28 +0100 (CET)","from mail-lj1-x236.google.com (mail-lj1-x236.google.com\n\t[IPv6:2a00:1450:4864:20::236])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1091962D3E\n\tfor ;\n\tMon, 9 Nov 2020 23:54:27 +0100 (CET)","by mail-lj1-x236.google.com with SMTP id y25so11523623lja.9\n\tfor ;\n\tMon, 09 Nov 2020 14:54:26 -0800 (PST)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tu4sm458673lfq.255.2020.11.09.14.54.24\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tMon, 09 Nov 2020 14:54:25 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"mZAlEsYp\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=N2v16RCriUzQDKhU6YBaLqZQdS38qaujtUCxB2XSpXs=;\n\tb=mZAlEsYpL2kd6EQ3brxGULNsqQJMbUHKN5chWZJhqAPLcAcaM3tGvLnlp61tcIVxmP\n\tlPe1DcgUZZrJkLE3occ29QilOFvBZGznVfEUyA7xci8TlRTTtjkdAvdnKGwMKapM4nOi\n\t4RkNNFWO4v5T6Ovy8qY2GHIDviQi+JtuRvpxjvJRWXB3skRJDqCDjzlia7GqQ2/hPnn5\n\t6CC9gvokZ/5hbTa5yrR7G5ADJ9RIpc+HPj8ZSOXgQI3Skvu4lbH9MkhroGkuLlfpS88p\n\t9UBj5jB4ny/oCimWX0xAE2Eh9LE9DgnVh81ztmnpx4hAbYmxSaDGxHw1dHp9HFmxW9u/\n\t4HtQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=N2v16RCriUzQDKhU6YBaLqZQdS38qaujtUCxB2XSpXs=;\n\tb=XItfWJiLsvyYb19C/VxiDllJ/H370TGei8RQW8MhkjUhIq7u/H0+Fg7nYNVm9aoXa/\n\tZ8mYawh361TTgbf/KbvQ0ofHfhbGdgW+dU3QEeaqT5RIw76FtX1hworY73hKiQ3eWa2q\n\t3UgRl0ZuawgIHPWmbeePZwhjKCNHvjQRNamgzdjAegz6RpGbSAmZgbioCsOWE5lhOUUd\n\tvnCUfJIo1pULqXGRcNGyZZqEvN5f5FknQBU2iMcvg4O13FyNHqBKAQZf4r6r70+UTK/4\n\t0lr8qCtPncHXhFRC0G4b6vyaku+dmARLFa+V3K9Chm0XLvfyOBH7MrcWi4urOsr5XPAg\n\tjYGw==","X-Gm-Message-State":"AOAM531bgnxMYB1WqU8ZF7psyOdxRkEIw7RIqUiNwmGQjD6s49sgVdeb\n\tiR3WdVBJDhn5dnVjK0p0z09JCGSZXOJ3sg==","X-Google-Smtp-Source":"ABdhPJyyGzyLhV1hh/OD/Eo/qZwVbV36I16sRQG7/b4DkO+HXviczHbSXZWnYSqAED1Ld5bvm0r5bg==","X-Received":"by 2002:a2e:920a:: with SMTP id\n\tk10mr6605121ljg.260.1604962466207; \n\tMon, 09 Nov 2020 14:54:26 -0800 (PST)","Date":"Mon, 9 Nov 2020 23:54:24 +0100","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Naushir Patuck ","Message-ID":"<20201109225424.GE88486@oden.dyn.berto.se>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13666,"web_url":"https://patchwork.libcamera.org/comment/13666/","msgid":"<20201110110253.GE484109@oden.dyn.berto.se>","date":"2020-11-10T11:02:53","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Jacopo,\n\nThanks for your feedback and sorry for messing up taking them all in \nwhen posting v2. I'm not sure what went wrong on my side.\n\nOn 2020-11-04 17:08:00 +0100, Jacopo Mondi wrote:\n> Hi Niklas,\n> \n> On Wed, Oct 28, 2020 at 02:00:44AM +0100, Niklas Söderlund wrote:\n> > Some sensor controls take effect with a delay as the sensor needs time\n> > to adjust, for example exposure. Add a optional helper DelayedControls\n> > to help pipelines deal with such controls.\n> >\n> > The idea is to provide a queue of controls towards the V4L2 device and\n> > apply individual controls with the specified delay with the aim to get\n> > predictable and retrievable control values for any given frame. To do\n> > this the queue of controls needs to be at least as deep as the control\n> > with the largest delay.\n> >\n> > The DelayedControls needs to be informed of every start of exposure.\n> > This can be emulated but the helper is designed to be used with this\n> > event being provide by the kernel thru V4L2 events.\n> \n> s/thru/though ?\n\nNo the V4L2 events are delivered thru the kernel.\n\n> >\n> > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > handler but expands on its API. This helpers aims to replace the\n> > Raspberry Pi implementations and mimics it behavior perfectly.\n> >\n> > Signed-off-by: Niklas Söderlund \n> > ---\n> > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > src/libcamera/meson.build | 1 +\n> > 3 files changed, 370 insertions(+)\n> > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > create mode 100644 src/libcamera/delayed_controls.cpp\n> >\n> > diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h\n> > new file mode 100644\n> > index 0000000000000000..df5520d240a54e4b\n> > --- /dev/null\n> > +++ b/include/libcamera/internal/delayed_controls.h\n> > @@ -0,0 +1,87 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > + */\n> > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +namespace libcamera {\n> > +\n> > +class V4L2Device;\n> > +\n> > +class DelayedControls\n> > +{\n> > +public:\n> > +\tDelayedControls(V4L2Device *device,\n> > +\t\t\tconst std::unordered_map &delays);\n> > +\n> > +\tvoid reset(ControlList *controls = nullptr);\n> > +\n> > +\tbool push(const ControlList &controls);\n> > +\tControlList get(uint32_t sequence);\n> > +\n> > +\tvoid frameStart(uint32_t sequence);\n> > +\n> > +private:\n> > +\tclass ControlInfo\n> \n> With no other scope but public:, is it worth a class ?\n\nI think this is a matter of taste, I get confused when a struct has \nmethods and not just data members.\n\n> \n> > +\t{\n> > +\tpublic:\n> > +\t\tControlInfo()\n> > +\t\t\t: updated(false)\n> > +\t\t{\n> > +\t\t}\n> > +\n> > +\t\tControlInfo(const ControlValue &v)\n> > +\t\t\t: value(v), updated(true)\n> > +\t\t{\n> > +\t\t}\n> > +\n> > +\t\tControlValue value;\n> > +\t\tbool updated;\n> > +\t};\n> > +\n> > +\tstatic constexpr int listSize = 16;\n> > +\tclass ControlArray : public std::array\n> > +\t{\n> > +\tpublic:\n> > +\t\tControlInfo &operator[](unsigned int index)\n> > +\t\t{\n> > +\t\t\treturn std::array::operator[](index % listSize);\n> > +\t\t}\n> > +\n> > +\t\tconst ControlInfo &operator[](unsigned int index) const\n> > +\t\t{\n> > +\t\t\treturn std::array::operator[](index % listSize);\n> > +\t\t}\n> > +\t};\n> > +\n> > +\tusing ControlsDelays = std::unordered_map;\n> > +\tusing ControlsValues = std::unordered_map;\n> > +\n> > +\tbool queue(const ControlList &controls);\n> > +\n> > +\tstd::mutex lock_;\n> > +\n> > +\tV4L2Device *device_;\n> > +\tControlsDelays delays_;\n> > +\tunsigned int maxDelay_;\n> > +\n> > +\tbool running_;\n> > +\tuint32_t fistSequence_;\n> \n> Should this be 'first' ?\n\nIt should, and I really missed to correct this.\n\n> \n> > +\n> > +\tuint32_t queueCount_;\n> > +\tuint32_t writeCount_;\n> > +\tControlsValues ctrls_;\n> > +};\n> > +\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp\n> > new file mode 100644\n> > index 0000000000000000..0e32f417c5cc68b7\n> > --- /dev/null\n> > +++ b/src/libcamera/delayed_controls.cpp\n> > @@ -0,0 +1,282 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > + */\n> > +\n> > +#include \"libcamera/internal/delayed_controls.h\"\n> > +#include \"libcamera/internal/v4l2_device.h\"\n> > +\n> > +#include \n> > +\n> > +#include \"libcamera/internal/log.h\"\n> > +\n> > +/**\n> > + * \\file delayed_controls.h\n> > + * \\brief Helper to deal with controls that are applied with a delay\n> > + */\n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > +\n> > +/**\n> > + * \\class DelayedControls\n> > + * \\brief Helper to deal with controls that takes effect with a delay\n> \n> controls -> take\n> \n> > + *\n> > + * Some sensor controls take effect with a delay as the sensor needs time to\n> \n> controls -> take\n\nSorry I don't understand this one.\n\n> \n> > + * adjust, for example exposure and focus. This is an optional helper class to\n> > + * deal with such controls and the intended user is pipeline handlers.\n> \n> users are pipeline handlers\n> \n> > + *\n> > + * The idea is to extend the concept of the pipeline depth the users needs to\n> > + * maintain for buffers to controls. The depth is determined with by the control\n> > + * with the grates delay. As long as the pipeline keeps the control queue above\n> > + * this level the control values are guaranteed to be in effect at the specified\n> > + * point in time.\n> \n> I would drop this as it mentions other concepts that might be more\n> confusing than helping.\n> \n> > + *\n> > + * The downside is of course that the pipeline needs to know what controls to\n> > + * set control depth frames in advance. But there really is no way around this\n> > + * as the delay is a consequence of the physical world. Compare this with\n> > + * parameter buffers where the buffer may be queued to the device while it's\n> > + * still being written to as long as it's ready before it's consumed, this is\n> > + * because the parameter buffer (usually) does not contain controls that\n> > + * requires time to take effect.\n> \n> I find this confusing too, I'm sorry.\n> \n> I think the key concept is that controls have to be applied in\n> advance, the usage of \"Delayed\" took me a bit off-road as a\n> DelayedControls class seems to be designed to apply controls at a\n> later (delayed) point in time while it's actually the other way\n> around. One key point is that for each Control the pipeline handler\n> has to provide a known delay value, something I didn't realize until I\n> looked at the code. The class takes care of applying the control the\n> control in -advance- enough to make sure the control is fully applied\n> at the time the Request it is associated to completes.\n> \n> In the class description I would describe what the class offers and\n> how it has to be used, its API and how I should set\n> controls specifying their 'delay'. Design documentation is good but\n> only to complement usage documentation.\n\nI agree, this is confusing and adds no value I have dropped it in v2.\n\n> \n> > + */\n> > +\n> > +\n> > +/**\n> > + * \\brief Construct a DelayedControls\n> > + * \\param[in] device The V4L2 device containing the delayed controls\n> \n> The V4L2 device the controls have to be applied to\n> \n> > + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> > + * effect in frames\n> \n> isn't 'latency' better than delay ? Or any other term ? See how heavy\n> is the description you had to make of this parameter...\n\nI'm not sure this is a matter of taste or that I'm not a native speaker \nbut delay makes much more sens then latency in this context for me.\n\n> \n> \\param[in] delays Map of V4L2 control ids and their associated latencies (in frames)\n> \n> ?\n> > + *\n> > + * Only controls specified in \\a delays are handled by the DelayedControls\n> > + * instance. If it's desired to mix delayed controls and controls that takes\n> > + * effect immediately the immediate controls must be listed in the \\a delays map\n> > + * with a delay value of 0.\n> \n> delays is a map, you have to provide a delay value by design\n> \n> \"Controls with an associated 0 delay (latency) are applied immediately\"\n> \n> \n> > + */\n> > +DelayedControls::DelayedControls(V4L2Device *device,\n> > +\t\t\t\t const std::unordered_map &delays)\n> > +\t: device_(device), maxDelay_(0)\n> > +{\n> > +\tconst ControlInfoMap &controls = device_->controls();\n> > +\n> > +\t/*\n> > +\t * Sanity check that all controls where delays are requested are\n> \n> s/where delays//\n> \n> > +\t * exposed byt the device.\n> \n> by\n> \n> > +\t */\n> > +\tfor (auto const &delay : delays) {\n> > +\t\tunsigned int id = delay.first;\n> > +\n> > +\t\tif (controls.find(id) == controls.end())\n> > +\t\t\tLOG(DelayedControls, Error)\n> > +\t\t\t\t<< \"Delay request for control id \"\n> > +\t\t\t\t<< utils::hex(id)\n> > +\t\t\t\t<< \" but control is not exposed by device \"\n> > +\t\t\t\t<< device_->deviceNode();\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Create a map of control to delay for all controls exposed by the\n> \n> controls to delays\n> \n> > +\t * device. If no delay is specified assume the control applies directly.\n> \n> assume a 0 delay (latency)\n> \n> > +\t */\n> > +\tfor (auto const &control : controls) {\n> > +\t\tconst ControlId *id = control.first;\n> > +\n> > +\t\tauto it = delays.find(id->id());\n> > +\t\tif (it == delays.end())\n> > +\t\t\tcontinue;\n> > +\n> > +\t\tdelays_[id] = it->second;\n> > +\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Set a delay of \" << delays_[id]\n> > +\t\t\t<< \" for \" << id->name();\n> > +\n> > +\t\tmaxDelay_ = std::max(maxDelay_, delays_[id]);\n> > +\t}\n> > +\n> > +\treset();\n> > +}\n> > +\n> > +/**\n> > + * \\brief Reset the V4L2 device\n> \n> Reset controls to their default values ?\n> \n> > + * \\param[in] controls List of controls to reset the device to or nullptr\n> \n> Optional list of controls to apply to the device ?\n> \n> These is mostly on documentation, I'll look at implementation later!\n> \n> Thanks\n> j\n> \n> \n> > + *\n> > + * Resets the delayed controls state machine to its starting state. All controls\n> > + * are fetched from the V4L2 device to provide a good starting point for the\n> > + * first frames (length of control depth).\n> > + *\n> > + * Optionally \\a controls can be specified to set some or all of the handled\n> > + * V4L2 controls prior to reading them back. If no controls needs to be set\n> > + * nullptr may be used.\n> > + */\n> > +void DelayedControls::reset(ControlList *controls)\n> > +{\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\trunning_ = false;\n> > +\tfistSequence_ = 0;\n> > +\tqueueCount_ = 0;\n> > +\twriteCount_ = 0;\n> > +\n> > +\t/* Set the controls on the device if requested. */\n> > +\tif (controls)\n> > +\t\tdevice_->setControls(controls);\n> > +\n> > +\t/* Retrieve current control values reported by the device. */\n> > +\tstd::vector ids;\n> > +\tfor (auto const &delay : delays_)\n> > +\t\tids.push_back(delay.first->id());\n> > +\n> > +\tControlList devCtrls = device_->getControls(ids);\n> > +\n> > +\t/* Seed the control queue with the controls reported by the device. */\n> > +\tctrls_.clear();\n> > +\tfor (const auto &ctrl : devCtrls) {\n> > +\t\tconst ControlId *id = devCtrls.infoMap()->idmap().at(ctrl.first);\n> > +\t\tctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > +\t}\n> > +\n> > +\tqueueCount_++;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Push a set of controls on the queue\n> > + * \\param[in] controls List of controls to add the device queue\n> > + *\n> > + * Push a set of controls to the control queue. This increases the control queue\n> > + * depth by one.\n> > + *\n> > + * \\returns true if \\a controls are accepted, or false otherwise\n> > + */\n> > +bool DelayedControls::push(const ControlList &controls)\n> > +{\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\treturn queue(controls);\n> > +}\n> > +\n> > +bool DelayedControls::queue(const ControlList &controls)\n> > +{\n> > +\t/* Copy state from previous frame. */\n> > +\tfor (auto &ctrl : ctrls_) {\n> > +\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> > +\t\tinfo.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > +\t\tinfo.updated = false;\n> > +\t}\n> > +\n> > +\t/* Update with new controls. */\n> > +\tfor (const auto &control : controls) {\n> > +\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> > +\n> > +\t\tif (delays_.find(id) == delays_.end())\n> > +\t\t\treturn false;\n> > +\n> > +\t\tControlInfo &info = ctrls_[id][queueCount_];\n> > +\n> > +\t\tinfo.value = control.second;\n> > +\t\tinfo.updated = true;\n> > +\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Queuing \" << id->name()\n> > +\t\t\t<< \" to \" << info.value.toString()\n> > +\t\t\t<< \" at index \" << queueCount_;\n> > +\t}\n> > +\n> > +\tqueueCount_++;\n> > +\n> > +\treturn true;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Read back controls in effect at a specific sequence number\n> > + * \\param[in] sequence Sequence number to get read back controls for\n> > + *\n> > + * Read back what controls where in effect at a specific sequence number. The\n> > + * history is a ring buffer of 16 entries where new and old values coexist. It's\n> > + * the callers responsibility to not read to old sequence numbers that have been\n> > + * pushed out of the history.\n> > + *\n> > + * Historic values are evicted by pushing new values onto the queue using\n> > + * push(). The max history from the current sequence number that yields valid\n> > + * values are thus 16 minus number of controls pushed.\n> > + *\n> > + * \\returns List of controls in effect at \\a sequence\n> > + */\n> > +ControlList DelayedControls::get(uint32_t sequence)\n> > +{\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\tuint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> > +\tunsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > +\n> > +\tControlList out(device_->controls());\n> > +\tfor (const auto &ctrl : ctrls_) {\n> > +\t\tconst ControlId *id = ctrl.first;\n> > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > +\n> > +\t\tout.set(id->id(), info.value);\n> > +\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Reading \" << id->name()\n> > +\t\t\t<< \" to \" << info.value.toString()\n> > +\t\t\t<< \" at index \" << index;\n> > +\t}\n> > +\n> > +\treturn out;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Inform DelayedControls of a start of a new frame\n> > + * \\param[in] sequence Sequence number of the frame that started\n> > + *\n> > + * Inform the state machine that a new frame have started and it's sequence\n> > + * number. It's user of this helpers responsibility to inform the helper\n> > + * at the start of every frame. This can with ease be connected to the start\n> > + * of exposure (SOE) V4L2 event.\n> > + */\n> > +void DelayedControls::frameStart(uint32_t sequence)\n> > +{\n> > +\tLOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> > +\n> > +\tstd::lock_guard lock(lock_);\n> > +\n> > +\tif (!running_) {\n> > +\t\tfistSequence_ = sequence;\n> > +\t\trunning_ = true;\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Create control list peaking ahead in the value queue to ensure\n> > +\t * values are set in time to satisfy the sensor delay.\n> > +\t */\n> > +\tControlList out(device_->controls());\n> > +\tfor (const auto &ctrl : ctrls_) {\n> > +\t\tconst ControlId *id = ctrl.first;\n> > +\t\tunsigned int delayDiff = maxDelay_ - delays_[id];\n> > +\t\tunsigned int index = std::max(0, writeCount_ - delayDiff);\n> > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > +\n> > +\t\tif (info.updated) {\n> > +\t\t\tout.set(id->id(), info.value);\n> > +\t\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t\t<< \"Setting \" << id->name()\n> > +\t\t\t\t<< \" to \" << info.value.toString()\n> > +\t\t\t\t<< \" at index \" << index;\n> > +\t\t}\n> > +\t}\n> > +\n> > +\twriteCount_++;\n> > +\n> > +\twhile (writeCount_ >= queueCount_) {\n> > +\t\tLOG(DelayedControls, Debug)\n> > +\t\t\t<< \"Queue is empty, auto queue no-op.\";\n> > +\t\tqueue({});\n> > +\t}\n> > +\n> > +\tdevice_->setControls(&out);\n> > +}\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > --- a/src/libcamera/meson.build\n> > +++ b/src/libcamera/meson.build\n> > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > 'controls.cpp',\n> > 'control_serializer.cpp',\n> > 'control_validator.cpp',\n> > + 'delayed_controls.cpp',\n> > 'device_enumerator.cpp',\n> > 'device_enumerator_sysfs.cpp',\n> > 'event_dispatcher.cpp',\n> > --\n> > 2.29.1\n> >\n> > _______________________________________________\n> > libcamera-devel mailing list\n> > libcamera-devel@lists.libcamera.org\n> > https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 81363BDB89\n\tfor ;\n\tTue, 10 Nov 2020 11:02:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E21DF630BF;\n\tTue, 10 Nov 2020 12:02:56 +0100 (CET)","from mail-lj1-x243.google.com (mail-lj1-x243.google.com\n\t[IPv6:2a00:1450:4864:20::243])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0592460342\n\tfor ;\n\tTue, 10 Nov 2020 12:02:56 +0100 (CET)","by mail-lj1-x243.google.com with SMTP id b17so1530390ljf.12\n\tfor ;\n\tTue, 10 Nov 2020 03:02:55 -0800 (PST)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tb18sm2060937lfp.89.2020.11.10.03.02.53\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tTue, 10 Nov 2020 03:02:54 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"b3Ne3GvT\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=uHAmWFNqM1hqrk56rYi9BvE7oLbjFjfecyAA7RTcbAA=;\n\tb=b3Ne3GvT/+LndCRIJuN+j8gL154a1IPp/S4rQiPS7ZSkASOVzrcA0MQdSaMddYU/Gk\n\tkRveqcQBZWEnas6Rja1HUkmLPE/5aYav8otPDx7OD+bP2bkV6CQh+FVKBABs9hYi495Y\n\tXNjPLuh4tUrcNT6ORVFt4aSivw11XgvCrmWQ/eMeaZuKCjGQGuCPO4/CqonqsXX0nZzd\n\t2G7+9cRLwSreNAA6/kWJhdMSv8LmkwmTKa/JYEoA3vbHBsUExyfMadHGHTG9aTV2rI7v\n\tEtK972TXI9J1rbobdXNpl0m+rtmB5R61MjBccZZqE/aboOsfE4CpIGkjDkY8HCwNstbc\n\ttxxQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=uHAmWFNqM1hqrk56rYi9BvE7oLbjFjfecyAA7RTcbAA=;\n\tb=Lq1ciDZs4L7FcEDNj0C7ZkRWZ0pbvK4Tn5SFoLQrO/MF9QfkGRQzdJGsCAvjWbAz6A\n\t8WYp8vZuCVrDi9uhWDqZmImFeEWTwf8EkcGA/2C2gOKB/DhcUseuewLmd+Ftfn+UwLeT\n\tVcLLY3Sm0ZsJngxaiTgBlh0P9FuChGvbOCNXGZuFpZkb4Mw5UoysvUulLEyl6qTeflkX\n\tedjR2P3NSlzEhsgLTBwRaKxDZBANMr2KJrqboMl7R0D6Cjga2Jr5+DYXr+HwwXabNMDm\n\ttCkqbhbHEWja93KMNbLvpbcBZpVLT64qXFUYg+xXYAwnVJrg9NNBH/Bar9lYnJo17HmH\n\tra9Q==","X-Gm-Message-State":"AOAM531fbAqcZkSw9ZMXLrRKChyye0cLzg42lWO9XVz32ZHoFe/buFbV\n\tu++pkd2Zt/F4BBmNdDA/gIID2A==","X-Google-Smtp-Source":"ABdhPJwBZ975XcLCzPe7MOOKAJIEb0OrlEjdM0Xkt9ukQzDJwzXezsxgBVP0BVpT3AUUeBWeGqJhwA==","X-Received":"by 2002:a2e:b70d:: with SMTP id j13mr496181ljo.240.1605006175112;\n\tTue, 10 Nov 2020 03:02:55 -0800 (PST)","Date":"Tue, 10 Nov 2020 12:02:53 +0100","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Jacopo Mondi ","Message-ID":"<20201110110253.GE484109@oden.dyn.berto.se>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t<20201104160800.ethen7ty3al5cug2@uno.localdomain>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201104160800.ethen7ty3al5cug2@uno.localdomain>","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13667,"web_url":"https://patchwork.libcamera.org/comment/13667/","msgid":"<20201110111355.6orf2n2mcvjxztof@uno.localdomain>","date":"2020-11-10T11:13:55","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi Niklas,\n\nOn Tue, Nov 10, 2020 at 12:02:53PM +0100, Niklas Söderlund wrote:\n> Hi Jacopo,\n>\n> Thanks for your feedback and sorry for messing up taking them all in\n> when posting v2. I'm not sure what went wrong on my side.\n>\n> On 2020-11-04 17:08:00 +0100, Jacopo Mondi wrote:\n> > Hi Niklas,\n> >\n> > On Wed, Oct 28, 2020 at 02:00:44AM +0100, Niklas Söderlund wrote:\n> > > Some sensor controls take effect with a delay as the sensor needs time\n> > > to adjust, for example exposure. Add a optional helper DelayedControls\n> > > to help pipelines deal with such controls.\n> > >\n> > > The idea is to provide a queue of controls towards the V4L2 device and\n> > > apply individual controls with the specified delay with the aim to get\n> > > predictable and retrievable control values for any given frame. To do\n> > > this the queue of controls needs to be at least as deep as the control\n> > > with the largest delay.\n> > >\n> > > The DelayedControls needs to be informed of every start of exposure.\n> > > This can be emulated but the helper is designed to be used with this\n> > > event being provide by the kernel thru V4L2 events.\n> >\n> > s/thru/though ?\n>\n> No the V4L2 events are delivered thru the kernel.\n\nIndeed, I meant 'through'\n'thru' doesn't show up as a word in my dictionary, so it might be\ncorrect as well\n\n>\n> > >\n> > > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > > handler but expands on its API. This helpers aims to replace the\n> > > Raspberry Pi implementations and mimics it behavior perfectly.\n> > >\n> > > Signed-off-by: Niklas Söderlund \n> > > ---\n> > > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > > src/libcamera/meson.build | 1 +\n> > > 3 files changed, 370 insertions(+)\n> > > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > > create mode 100644 src/libcamera/delayed_controls.cpp\n> > >\n> > > diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h\n> > > new file mode 100644\n> > > index 0000000000000000..df5520d240a54e4b\n> > > --- /dev/null\n> > > +++ b/include/libcamera/internal/delayed_controls.h\n> > > @@ -0,0 +1,87 @@\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Google Inc.\n> > > + *\n> > > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > > + */\n> > > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +class V4L2Device;\n> > > +\n> > > +class DelayedControls\n> > > +{\n> > > +public:\n> > > +\tDelayedControls(V4L2Device *device,\n> > > +\t\t\tconst std::unordered_map &delays);\n> > > +\n> > > +\tvoid reset(ControlList *controls = nullptr);\n> > > +\n> > > +\tbool push(const ControlList &controls);\n> > > +\tControlList get(uint32_t sequence);\n> > > +\n> > > +\tvoid frameStart(uint32_t sequence);\n> > > +\n> > > +private:\n> > > +\tclass ControlInfo\n> >\n> > With no other scope but public:, is it worth a class ?\n>\n> I think this is a matter of taste, I get confused when a struct has\n> methods and not just data members.\n>\n> >\n> > > +\t{\n> > > +\tpublic:\n> > > +\t\tControlInfo()\n> > > +\t\t\t: updated(false)\n> > > +\t\t{\n> > > +\t\t}\n> > > +\n> > > +\t\tControlInfo(const ControlValue &v)\n> > > +\t\t\t: value(v), updated(true)\n> > > +\t\t{\n> > > +\t\t}\n> > > +\n> > > +\t\tControlValue value;\n> > > +\t\tbool updated;\n> > > +\t};\n> > > +\n> > > +\tstatic constexpr int listSize = 16;\n> > > +\tclass ControlArray : public std::array\n> > > +\t{\n> > > +\tpublic:\n> > > +\t\tControlInfo &operator[](unsigned int index)\n> > > +\t\t{\n> > > +\t\t\treturn std::array::operator[](index % listSize);\n> > > +\t\t}\n> > > +\n> > > +\t\tconst ControlInfo &operator[](unsigned int index) const\n> > > +\t\t{\n> > > +\t\t\treturn std::array::operator[](index % listSize);\n> > > +\t\t}\n> > > +\t};\n> > > +\n> > > +\tusing ControlsDelays = std::unordered_map;\n> > > +\tusing ControlsValues = std::unordered_map;\n> > > +\n> > > +\tbool queue(const ControlList &controls);\n> > > +\n> > > +\tstd::mutex lock_;\n> > > +\n> > > +\tV4L2Device *device_;\n> > > +\tControlsDelays delays_;\n> > > +\tunsigned int maxDelay_;\n> > > +\n> > > +\tbool running_;\n> > > +\tuint32_t fistSequence_;\n> >\n> > Should this be 'first' ?\n>\n> It should, and I really missed to correct this.\n>\n> >\n> > > +\n> > > +\tuint32_t queueCount_;\n> > > +\tuint32_t writeCount_;\n> > > +\tControlsValues ctrls_;\n> > > +};\n> > > +\n> > > +} /* namespace libcamera */\n> > > +\n> > > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > > diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp\n> > > new file mode 100644\n> > > index 0000000000000000..0e32f417c5cc68b7\n> > > --- /dev/null\n> > > +++ b/src/libcamera/delayed_controls.cpp\n> > > @@ -0,0 +1,282 @@\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Google Inc.\n> > > + *\n> > > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > > + */\n> > > +\n> > > +#include \"libcamera/internal/delayed_controls.h\"\n> > > +#include \"libcamera/internal/v4l2_device.h\"\n> > > +\n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/log.h\"\n> > > +\n> > > +/**\n> > > + * \\file delayed_controls.h\n> > > + * \\brief Helper to deal with controls that are applied with a delay\n> > > + */\n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > > +\n> > > +/**\n> > > + * \\class DelayedControls\n> > > + * \\brief Helper to deal with controls that takes effect with a delay\n> >\n> > controls -> take\n> >\n> > > + *\n> > > + * Some sensor controls take effect with a delay as the sensor needs time to\n> >\n> > controls -> take\n>\n> Sorry I don't understand this one.\n>\n\nbad copy&paste ?\n\n> >\n> > > + * adjust, for example exposure and focus. This is an optional helper class to\n> > > + * deal with such controls and the intended user is pipeline handlers.\n> >\n> > users are pipeline handlers\n> >\n> > > + *\n> > > + * The idea is to extend the concept of the pipeline depth the users needs to\n> > > + * maintain for buffers to controls. The depth is determined with by the control\n> > > + * with the grates delay. As long as the pipeline keeps the control queue above\n> > > + * this level the control values are guaranteed to be in effect at the specified\n> > > + * point in time.\n> >\n> > I would drop this as it mentions other concepts that might be more\n> > confusing than helping.\n> >\n> > > + *\n> > > + * The downside is of course that the pipeline needs to know what controls to\n> > > + * set control depth frames in advance. But there really is no way around this\n> > > + * as the delay is a consequence of the physical world. Compare this with\n> > > + * parameter buffers where the buffer may be queued to the device while it's\n> > > + * still being written to as long as it's ready before it's consumed, this is\n> > > + * because the parameter buffer (usually) does not contain controls that\n> > > + * requires time to take effect.\n> >\n> > I find this confusing too, I'm sorry.\n> >\n> > I think the key concept is that controls have to be applied in\n> > advance, the usage of \"Delayed\" took me a bit off-road as a\n> > DelayedControls class seems to be designed to apply controls at a\n> > later (delayed) point in time while it's actually the other way\n> > around. One key point is that for each Control the pipeline handler\n> > has to provide a known delay value, something I didn't realize until I\n> > looked at the code. The class takes care of applying the control the\n> > control in -advance- enough to make sure the control is fully applied\n> > at the time the Request it is associated to completes.\n> >\n> > In the class description I would describe what the class offers and\n> > how it has to be used, its API and how I should set\n> > controls specifying their 'delay'. Design documentation is good but\n> > only to complement usage documentation.\n>\n> I agree, this is confusing and adds no value I have dropped it in v2.\n>\n\nThanks\n\n> >\n> > > + */\n> > > +\n> > > +\n> > > +/**\n> > > + * \\brief Construct a DelayedControls\n> > > + * \\param[in] device The V4L2 device containing the delayed controls\n> >\n> > The V4L2 device the controls have to be applied to\n> >\n> > > + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> > > + * effect in frames\n> >\n> > isn't 'latency' better than delay ? Or any other term ? See how heavy\n> > is the description you had to make of this parameter...\n>\n> I'm not sure this is a matter of taste or that I'm not a native speaker\n> but delay makes much more sens then latency in this context for me.\n>\n\nLet's defer this to native speakers. If it's just about tastes then\nyour code, your call.\n\n> >\n> > \\param[in] delays Map of V4L2 control ids and their associated latencies (in frames)\n> >\n> > ?\n> > > + *\n> > > + * Only controls specified in \\a delays are handled by the DelayedControls\n> > > + * instance. If it's desired to mix delayed controls and controls that takes\n> > > + * effect immediately the immediate controls must be listed in the \\a delays map\n> > > + * with a delay value of 0.\n> >\n> > delays is a map, you have to provide a delay value by design\n> >\n> > \"Controls with an associated 0 delay (latency) are applied immediately\"\n> >\n> >\n> > > + */\n> > > +DelayedControls::DelayedControls(V4L2Device *device,\n> > > +\t\t\t\t const std::unordered_map &delays)\n> > > +\t: device_(device), maxDelay_(0)\n> > > +{\n> > > +\tconst ControlInfoMap &controls = device_->controls();\n> > > +\n> > > +\t/*\n> > > +\t * Sanity check that all controls where delays are requested are\n> >\n> > s/where delays//\n> >\n> > > +\t * exposed byt the device.\n> >\n> > by\n> >\n> > > +\t */\n> > > +\tfor (auto const &delay : delays) {\n> > > +\t\tunsigned int id = delay.first;\n> > > +\n> > > +\t\tif (controls.find(id) == controls.end())\n> > > +\t\t\tLOG(DelayedControls, Error)\n> > > +\t\t\t\t<< \"Delay request for control id \"\n> > > +\t\t\t\t<< utils::hex(id)\n> > > +\t\t\t\t<< \" but control is not exposed by device \"\n> > > +\t\t\t\t<< device_->deviceNode();\n> > > +\t}\n> > > +\n> > > +\t/*\n> > > +\t * Create a map of control to delay for all controls exposed by the\n> >\n> > controls to delays\n> >\n> > > +\t * device. If no delay is specified assume the control applies directly.\n> >\n> > assume a 0 delay (latency)\n> >\n> > > +\t */\n> > > +\tfor (auto const &control : controls) {\n> > > +\t\tconst ControlId *id = control.first;\n> > > +\n> > > +\t\tauto it = delays.find(id->id());\n> > > +\t\tif (it == delays.end())\n> > > +\t\t\tcontinue;\n> > > +\n> > > +\t\tdelays_[id] = it->second;\n> > > +\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Set a delay of \" << delays_[id]\n> > > +\t\t\t<< \" for \" << id->name();\n> > > +\n> > > +\t\tmaxDelay_ = std::max(maxDelay_, delays_[id]);\n> > > +\t}\n> > > +\n> > > +\treset();\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Reset the V4L2 device\n> >\n> > Reset controls to their default values ?\n> >\n> > > + * \\param[in] controls List of controls to reset the device to or nullptr\n> >\n> > Optional list of controls to apply to the device ?\n> >\n> > These is mostly on documentation, I'll look at implementation later!\n> >\n> > Thanks\n> > j\n> >\n> >\n> > > + *\n> > > + * Resets the delayed controls state machine to its starting state. All controls\n> > > + * are fetched from the V4L2 device to provide a good starting point for the\n> > > + * first frames (length of control depth).\n> > > + *\n> > > + * Optionally \\a controls can be specified to set some or all of the handled\n> > > + * V4L2 controls prior to reading them back. If no controls needs to be set\n> > > + * nullptr may be used.\n> > > + */\n> > > +void DelayedControls::reset(ControlList *controls)\n> > > +{\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\trunning_ = false;\n> > > +\tfistSequence_ = 0;\n> > > +\tqueueCount_ = 0;\n> > > +\twriteCount_ = 0;\n> > > +\n> > > +\t/* Set the controls on the device if requested. */\n> > > +\tif (controls)\n> > > +\t\tdevice_->setControls(controls);\n> > > +\n> > > +\t/* Retrieve current control values reported by the device. */\n> > > +\tstd::vector ids;\n> > > +\tfor (auto const &delay : delays_)\n> > > +\t\tids.push_back(delay.first->id());\n> > > +\n> > > +\tControlList devCtrls = device_->getControls(ids);\n> > > +\n> > > +\t/* Seed the control queue with the controls reported by the device. */\n> > > +\tctrls_.clear();\n> > > +\tfor (const auto &ctrl : devCtrls) {\n> > > +\t\tconst ControlId *id = devCtrls.infoMap()->idmap().at(ctrl.first);\n> > > +\t\tctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > > +\t}\n> > > +\n> > > +\tqueueCount_++;\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Push a set of controls on the queue\n> > > + * \\param[in] controls List of controls to add the device queue\n> > > + *\n> > > + * Push a set of controls to the control queue. This increases the control queue\n> > > + * depth by one.\n> > > + *\n> > > + * \\returns true if \\a controls are accepted, or false otherwise\n> > > + */\n> > > +bool DelayedControls::push(const ControlList &controls)\n> > > +{\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\treturn queue(controls);\n> > > +}\n> > > +\n> > > +bool DelayedControls::queue(const ControlList &controls)\n> > > +{\n> > > +\t/* Copy state from previous frame. */\n> > > +\tfor (auto &ctrl : ctrls_) {\n> > > +\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> > > +\t\tinfo.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > > +\t\tinfo.updated = false;\n> > > +\t}\n> > > +\n> > > +\t/* Update with new controls. */\n> > > +\tfor (const auto &control : controls) {\n> > > +\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> > > +\n> > > +\t\tif (delays_.find(id) == delays_.end())\n> > > +\t\t\treturn false;\n> > > +\n> > > +\t\tControlInfo &info = ctrls_[id][queueCount_];\n> > > +\n> > > +\t\tinfo.value = control.second;\n> > > +\t\tinfo.updated = true;\n> > > +\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Queuing \" << id->name()\n> > > +\t\t\t<< \" to \" << info.value.toString()\n> > > +\t\t\t<< \" at index \" << queueCount_;\n> > > +\t}\n> > > +\n> > > +\tqueueCount_++;\n> > > +\n> > > +\treturn true;\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Read back controls in effect at a specific sequence number\n> > > + * \\param[in] sequence Sequence number to get read back controls for\n> > > + *\n> > > + * Read back what controls where in effect at a specific sequence number. The\n> > > + * history is a ring buffer of 16 entries where new and old values coexist. It's\n> > > + * the callers responsibility to not read to old sequence numbers that have been\n> > > + * pushed out of the history.\n> > > + *\n> > > + * Historic values are evicted by pushing new values onto the queue using\n> > > + * push(). The max history from the current sequence number that yields valid\n> > > + * values are thus 16 minus number of controls pushed.\n> > > + *\n> > > + * \\returns List of controls in effect at \\a sequence\n> > > + */\n> > > +ControlList DelayedControls::get(uint32_t sequence)\n> > > +{\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\tuint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> > > +\tunsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > > +\n> > > +\tControlList out(device_->controls());\n> > > +\tfor (const auto &ctrl : ctrls_) {\n> > > +\t\tconst ControlId *id = ctrl.first;\n> > > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > > +\n> > > +\t\tout.set(id->id(), info.value);\n> > > +\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Reading \" << id->name()\n> > > +\t\t\t<< \" to \" << info.value.toString()\n> > > +\t\t\t<< \" at index \" << index;\n> > > +\t}\n> > > +\n> > > +\treturn out;\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Inform DelayedControls of a start of a new frame\n> > > + * \\param[in] sequence Sequence number of the frame that started\n> > > + *\n> > > + * Inform the state machine that a new frame have started and it's sequence\n> > > + * number. It's user of this helpers responsibility to inform the helper\n> > > + * at the start of every frame. This can with ease be connected to the start\n> > > + * of exposure (SOE) V4L2 event.\n> > > + */\n> > > +void DelayedControls::frameStart(uint32_t sequence)\n> > > +{\n> > > +\tLOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> > > +\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\tif (!running_) {\n> > > +\t\tfistSequence_ = sequence;\n> > > +\t\trunning_ = true;\n> > > +\t}\n> > > +\n> > > +\t/*\n> > > +\t * Create control list peaking ahead in the value queue to ensure\n> > > +\t * values are set in time to satisfy the sensor delay.\n> > > +\t */\n> > > +\tControlList out(device_->controls());\n> > > +\tfor (const auto &ctrl : ctrls_) {\n> > > +\t\tconst ControlId *id = ctrl.first;\n> > > +\t\tunsigned int delayDiff = maxDelay_ - delays_[id];\n> > > +\t\tunsigned int index = std::max(0, writeCount_ - delayDiff);\n> > > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > > +\n> > > +\t\tif (info.updated) {\n> > > +\t\t\tout.set(id->id(), info.value);\n> > > +\t\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t\t<< \"Setting \" << id->name()\n> > > +\t\t\t\t<< \" to \" << info.value.toString()\n> > > +\t\t\t\t<< \" at index \" << index;\n> > > +\t\t}\n> > > +\t}\n> > > +\n> > > +\twriteCount_++;\n> > > +\n> > > +\twhile (writeCount_ >= queueCount_) {\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Queue is empty, auto queue no-op.\";\n> > > +\t\tqueue({});\n> > > +\t}\n> > > +\n> > > +\tdevice_->setControls(&out);\n> > > +}\n> > > +\n> > > +} /* namespace libcamera */\n> > > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > > --- a/src/libcamera/meson.build\n> > > +++ b/src/libcamera/meson.build\n> > > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > > 'controls.cpp',\n> > > 'control_serializer.cpp',\n> > > 'control_validator.cpp',\n> > > + 'delayed_controls.cpp',\n> > > 'device_enumerator.cpp',\n> > > 'device_enumerator_sysfs.cpp',\n> > > 'event_dispatcher.cpp',\n> > > --\n> > > 2.29.1\n> > >\n> > > _______________________________________________\n> > > libcamera-devel mailing list\n> > > libcamera-devel@lists.libcamera.org\n> > > https://lists.libcamera.org/listinfo/libcamera-devel\n>\n> --\n> Regards,\n> Niklas Söderlund","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9CA09BE082\n\tfor ;\n\tTue, 10 Nov 2020 11:13:56 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2D6A6630BF;\n\tTue, 10 Nov 2020 12:13:56 +0100 (CET)","from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net\n\t[217.70.183.198])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0093760342\n\tfor ;\n\tTue, 10 Nov 2020 12:13:54 +0100 (CET)","from uno.localdomain (93-34-118-233.ip49.fastwebnet.it\n\t[93.34.118.233]) (Authenticated sender: jacopo@jmondi.org)\n\tby relay6-d.mail.gandi.net (Postfix) with ESMTPSA id DF054C0002;\n\tTue, 10 Nov 2020 11:13:53 +0000 (UTC)"],"X-Originating-IP":"93.34.118.233","Date":"Tue, 10 Nov 2020 12:13:55 +0100","From":"Jacopo Mondi ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Message-ID":"<20201110111355.6orf2n2mcvjxztof@uno.localdomain>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t<20201104160800.ethen7ty3al5cug2@uno.localdomain>\n\t<20201110110253.GE484109@oden.dyn.berto.se>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201110110253.GE484109@oden.dyn.berto.se>","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13668,"web_url":"https://patchwork.libcamera.org/comment/13668/","msgid":"<20201110112647.GA660945@oden.dyn.berto.se>","date":"2020-11-10T11:26:47","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Jacopo,\n\nThanks for your feedback.\n\nOn 2020-11-05 21:08:48 +0100, Jacopo Mondi wrote:\n> Hi again Niklas,\n> \n> On Wed, Nov 04, 2020 at 05:08:00PM +0100, Jacopo Mondi wrote:\n> > Hi Niklas,\n> >\n> > On Wed, Oct 28, 2020 at 02:00:44AM +0100, Niklas Söderlund wrote:\n> > > Some sensor controls take effect with a delay as the sensor needs time\n> > > to adjust, for example exposure. Add a optional helper DelayedControls\n> > > to help pipelines deal with such controls.\n> > >\n> > > The idea is to provide a queue of controls towards the V4L2 device and\n> > > apply individual controls with the specified delay with the aim to get\n> > > predictable and retrievable control values for any given frame. To do\n> > > this the queue of controls needs to be at least as deep as the control\n> > > with the largest delay.\n> > >\n> > > The DelayedControls needs to be informed of every start of exposure.\n> > > This can be emulated but the helper is designed to be used with this\n> > > event being provide by the kernel thru V4L2 events.\n> >\n> > s/thru/though ?\n> > >\n> > > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > > handler but expands on its API. This helpers aims to replace the\n> > > Raspberry Pi implementations and mimics it behavior perfectly.\n> > >\n> > > Signed-off-by: Niklas Söderlund \n> > > ---\n> > > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > > src/libcamera/meson.build | 1 +\n> > > 3 files changed, 370 insertions(+)\n> > > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > > create mode 100644 src/libcamera/delayed_controls.cpp\n> > >\n> > > diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h\n> > > new file mode 100644\n> > > index 0000000000000000..df5520d240a54e4b\n> > > --- /dev/null\n> > > +++ b/include/libcamera/internal/delayed_controls.h\n> > > @@ -0,0 +1,87 @@\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Google Inc.\n> > > + *\n> > > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > > + */\n> > > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +class V4L2Device;\n> > > +\n> > > +class DelayedControls\n> > > +{\n> > > +public:\n> > > +\tDelayedControls(V4L2Device *device,\n> > > +\t\t\tconst std::unordered_map &delays);\n> > > +\n> > > +\tvoid reset(ControlList *controls = nullptr);\n> > > +\n> > > +\tbool push(const ControlList &controls);\n> > > +\tControlList get(uint32_t sequence);\n> > > +\n> > > +\tvoid frameStart(uint32_t sequence);\n> > > +\n> > > +private:\n> > > +\tclass ControlInfo\n> >\n> > With no other scope but public:, is it worth a class ?\n> >\n> > > +\t{\n> > > +\tpublic:\n> > > +\t\tControlInfo()\n> > > +\t\t\t: updated(false)\n> > > +\t\t{\n> > > +\t\t}\n> > > +\n> > > +\t\tControlInfo(const ControlValue &v)\n> > > +\t\t\t: value(v), updated(true)\n> > > +\t\t{\n> > > +\t\t}\n> > > +\n> > > +\t\tControlValue value;\n> > > +\t\tbool updated;\n> > > +\t};\n> > > +\n> > > +\tstatic constexpr int listSize = 16;\n> > > +\tclass ControlArray : public std::array\n> > > +\t{\n> > > +\tpublic:\n> > > +\t\tControlInfo &operator[](unsigned int index)\n> > > +\t\t{\n> > > +\t\t\treturn std::array::operator[](index % listSize);\n> > > +\t\t}\n> > > +\n> > > +\t\tconst ControlInfo &operator[](unsigned int index) const\n> > > +\t\t{\n> > > +\t\t\treturn std::array::operator[](index % listSize);\n> > > +\t\t}\n> > > +\t};\n> > > +\n> > > +\tusing ControlsDelays = std::unordered_map;\n> > > +\tusing ControlsValues = std::unordered_map;\n> \n> Do you think indexing on ControlId is a good idea ? I see quite some\n> churn below to get from the integer id obtained by iterating a\n> ControlList to the ControlId used to index these maps:\n> \n> \t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> \n> While from ControlId we can always easily get the id back\n\nI think it's worth it as we try to stay away form using V4L2 identifiers \nfor as long as possible. As a bonus we get access to the string name of \nthe control for the logging.\n\n> \n> > > +\n> > > +\tbool queue(const ControlList &controls);\n> > > +\n> > > +\tstd::mutex lock_;\n> > > +\n> > > +\tV4L2Device *device_;\n> > > +\tControlsDelays delays_;\n> > > +\tunsigned int maxDelay_;\n> > > +\n> > > +\tbool running_;\n> > > +\tuint32_t fistSequence_;\n> >\n> > Should this be 'first' ?\n> >\n> > > +\n> > > +\tuint32_t queueCount_;\n> > > +\tuint32_t writeCount_;\n> > > +\tControlsValues ctrls_;\n> > > +};\n> > > +\n> > > +} /* namespace libcamera */\n> > > +\n> > > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > > diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp\n> > > new file mode 100644\n> > > index 0000000000000000..0e32f417c5cc68b7\n> > > --- /dev/null\n> > > +++ b/src/libcamera/delayed_controls.cpp\n> > > @@ -0,0 +1,282 @@\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Google Inc.\n> > > + *\n> > > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > > + */\n> > > +\n> > > +#include \"libcamera/internal/delayed_controls.h\"\n> > > +#include \"libcamera/internal/v4l2_device.h\"\n> > > +\n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/log.h\"\n> > > +\n> > > +/**\n> > > + * \\file delayed_controls.h\n> > > + * \\brief Helper to deal with controls that are applied with a delay\n> > > + */\n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > > +\n> > > +/**\n> > > + * \\class DelayedControls\n> > > + * \\brief Helper to deal with controls that takes effect with a delay\n> >\n> > controls -> take\n> >\n> > > + *\n> > > + * Some sensor controls take effect with a delay as the sensor needs time to\n> >\n> > controls -> take\n> >\n> > > + * adjust, for example exposure and focus. This is an optional helper class to\n> > > + * deal with such controls and the intended user is pipeline handlers.\n> >\n> > users are pipeline handlers\n> >\n> > > + *\n> > > + * The idea is to extend the concept of the pipeline depth the users needs to\n> > > + * maintain for buffers to controls. The depth is determined with by the control\n> > > + * with the grates delay. As long as the pipeline keeps the control queue above\n> > > + * this level the control values are guaranteed to be in effect at the specified\n> > > + * point in time.\n> >\n> > I would drop this as it mentions other concepts that might be more\n> > confusing than helping.\n> >\n> > > + *\n> > > + * The downside is of course that the pipeline needs to know what controls to\n> > > + * set control depth frames in advance. But there really is no way around this\n> > > + * as the delay is a consequence of the physical world. Compare this with\n> > > + * parameter buffers where the buffer may be queued to the device while it's\n> > > + * still being written to as long as it's ready before it's consumed, this is\n> > > + * because the parameter buffer (usually) does not contain controls that\n> > > + * requires time to take effect.\n> >\n> > I find this confusing too, I'm sorry.\n> >\n> > I think the key concept is that controls have to be applied in\n> > advance, the usage of \"Delayed\" took me a bit off-road as a\n> > DelayedControls class seems to be designed to apply controls at a\n> > later (delayed) point in time while it's actually the other way\n> > around. One key point is that for each Control the pipeline handler\n> > has to provide a known delay value, something I didn't realize until I\n> > looked at the code. The class takes care of applying the control the\n> > control in -advance- enough to make sure the control is fully applied\n> > at the time the Request it is associated to completes.\n> >\n> > In the class description I would describe what the class offers and\n> > how it has to be used, its API and how I should set\n> > controls specifying their 'delay'. Design documentation is good but\n> > only to complement usage documentation.\n> >\n> > > + */\n> > > +\n> \n> double empy line\n> \n> > > +\n> > > +/**\n> > > + * \\brief Construct a DelayedControls\n> > > + * \\param[in] device The V4L2 device containing the delayed controls\n> >\n> > The V4L2 device the controls have to be applied to\n> >\n> > > + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> > > + * effect in frames\n> >\n> > isn't 'latency' better than delay ? Or any other term ? See how heavy\n> > is the description you had to make of this parameter...\n> >\n> > \\param[in] delays Map of V4L2 control ids and their associated latencies (in frames)\n> >\n> > ?\n> > > + *\n> > > + * Only controls specified in \\a delays are handled by the DelayedControls\n> > > + * instance. If it's desired to mix delayed controls and controls that takes\n> > > + * effect immediately the immediate controls must be listed in the \\a delays map\n> > > + * with a delay value of 0.\n> >\n> > delays is a map, you have to provide a delay value by design\n> >\n> > \"Controls with an associated 0 delay (latency) are applied immediately\"\n> >\n> >\n> > > + */\n> > > +DelayedControls::DelayedControls(V4L2Device *device,\n> > > +\t\t\t\t const std::unordered_map &delays)\n> > > +\t: device_(device), maxDelay_(0)\n> > > +{\n> > > +\tconst ControlInfoMap &controls = device_->controls();\n> > > +\n> > > +\t/*\n> > > +\t * Sanity check that all controls where delays are requested are\n> >\n> > s/where delays//\n> >\n> \n> Actually I meant to make this\n> Sanity check that all requested controls are exposed by the device\n> \n> > > +\t * exposed byt the device.\n> >\n> > by\n> >\n> > > +\t */\n> > > +\tfor (auto const &delay : delays) {\n> > > +\t\tunsigned int id = delay.first;\n> > > +\n> > > +\t\tif (controls.find(id) == controls.end())\n> > > +\t\t\tLOG(DelayedControls, Error)\n> > > +\t\t\t\t<< \"Delay request for control id \"\n> > > +\t\t\t\t<< utils::hex(id)\n> > > +\t\t\t\t<< \" but control is not exposed by device \"\n> > > +\t\t\t\t<< device_->deviceNode();\n> > > +\t}\n> > > +\n> > > +\t/*\n> > > +\t * Create a map of control to delay for all controls exposed by the\n> >\n> > controls to delays\n> >\n> > > +\t * device. If no delay is specified assume the control applies directly.\n> >\n> > assume a 0 delay (latency)\n> >\n> > > +\t */\n> > > +\tfor (auto const &control : controls) {\n> > > +\t\tconst ControlId *id = control.first;\n> > > +\n> > > +\t\tauto it = delays.find(id->id());\n> > > +\t\tif (it == delays.end())\n> > > +\t\t\tcontinue;\n> > > +\n> > > +\t\tdelays_[id] = it->second;\n> > > +\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Set a delay of \" << delays_[id]\n> > > +\t\t\t<< \" for \" << id->name();\n> > > +\n> > > +\t\tmaxDelay_ = std::max(maxDelay_, delays_[id]);\n> > > +\t}\n> \n> I wonder if these loops should not be unified. After all, if a\n> device's control is not in the delay map it is skipped, so it is the\n> same as iterating the delay map as done in the previous loop ?\n\nThey should and are in v2.\n\n> \n> > > +\n> > > +\treset();\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Reset the V4L2 device\n> >\n> > Reset controls to their default values ?\n> >\n> > > + * \\param[in] controls List of controls to reset the device to or nullptr\n> >\n> > Optional list of controls to apply to the device ?\n> >\n> > These is mostly on documentation, I'll look at implementation later!\n> >\n> > Thanks\n> > j\n> >\n> >\n> > > + *\n> > > + * Resets the delayed controls state machine to its starting state. All controls\n> > > + * are fetched from the V4L2 device to provide a good starting point for the\n> > > + * first frames (length of control depth).\n> \n> I don't get this last statement\n> \n> > > + *\n> > > + * Optionally \\a controls can be specified to set some or all of the handled\n> > > + * V4L2 controls prior to reading them back. If no controls needs to be set\n> > > + * nullptr may be used.\n> \n> I would drop the last sentence\n> \n> > > + */\n> > > +void DelayedControls::reset(ControlList *controls)\n> > > +{\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\trunning_ = false;\n> > > +\tfistSequence_ = 0;\n> > > +\tqueueCount_ = 0;\n> > > +\twriteCount_ = 0;\n> > > +\n> > > +\t/* Set the controls on the device if requested. */\n> > > +\tif (controls)\n> > > +\t\tdevice_->setControls(controls);\n> > > +\n> > > +\t/* Retrieve current control values reported by the device. */\n> > > +\tstd::vector ids;\n> > > +\tfor (auto const &delay : delays_)\n> > > +\t\tids.push_back(delay.first->id());\n> > > +\n> > > +\tControlList devCtrls = device_->getControls(ids);\n> > > +\n> > > +\t/* Seed the control queue with the controls reported by the device. */\n> > > +\tctrls_.clear();\n> > > +\tfor (const auto &ctrl : devCtrls) {\n> > > +\t\tconst ControlId *id = devCtrls.infoMap()->idmap().at(ctrl.first);\n> > > +\t\tctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > > +\t}\n> > > +\n> > > +\tqueueCount_++;\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Push a set of controls on the queue\n> > > + * \\param[in] controls List of controls to add the device queue\n> > > + *\n> > > + * Push a set of controls to the control queue. This increases the control queue\n> > > + * depth by one.\n> \n> control queue depth really confuses me.\n> \n> Isn't it better to just say:\n> \"Push a new set of values for the controls\"\n> or something similar ?\n> \n> > > + *\n> > > + * \\returns true if \\a controls are accepted, or false otherwise\n> > > + */\n> > > +bool DelayedControls::push(const ControlList &controls)\n> > > +{\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\treturn queue(controls);\n> > > +}\n> > > +\n> > > +bool DelayedControls::queue(const ControlList &controls)\n> > > +{\n> > > +\t/* Copy state from previous frame. */\n> > > +\tfor (auto &ctrl : ctrls_) {\n> > > +\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> \n> Am I wrong of\n> \tfor (auto &ctrl : ctrls_) {\n> \t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> \n> Is equivalent to\n> \tfor (auto &ctrl : ctrls_) {\n> \t\tControlInfo &info = ctrl.second[queueCount_];\n> \n> (the compiler agrees afaict)\n\nIt does, thanks for this so much nicer to read!\n\n> \n> > > +\t\tinfo.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > > +\t\tinfo.updated = false;\n> > > +\t}\n> \n> Have you considered breaking this in 2 ? This would avoid the below\n> queue({}) which seems a workaround.\n> \n> Can't we:\n> advance() {\n> copy values from previous entry\n> queueCount++;\n> }\n> \n> update() {\n> copy the new values in;\n> }\n> \n> So that below you only call advance() ?\n\nI like the notion of one having to queue an empty list as it fits the \nconcept of this helper. Then maybe we can optimize it further if an \nempty set is queue. Maybe this could be a future improvement as this \nwill only effect the internal API of this class anyhow. And we still \nneed to allow for pipelines to que empty controls list to keep in sync.\n\n> \n> > > +\n> > > +\t/* Update with new controls. */\n> > > +\tfor (const auto &control : controls) {\n> > > +\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> > > +\n> > > +\t\tif (delays_.find(id) == delays_.end())\n> > > +\t\t\treturn false;\n> > > +\n> > > +\t\tControlInfo &info = ctrls_[id][queueCount_];\n> \n> That's a double lookup that can be avoided. If I'm not mistaken all\n> controls in delays_ are inserted in ctrls_, am I wrong ? If that's the\n> case you can just ctrls_.find() == ctls_.end() ?\n\nI'm not sure I get this one.\n\n> \n> > > +\n> > > +\t\tinfo.value = control.second;\n> > > +\t\tinfo.updated = true;\n> > > +\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Queuing \" << id->name()\n> > > +\t\t\t<< \" to \" << info.value.toString()\n> > > +\t\t\t<< \" at index \" << queueCount_;\n> > > +\t}\n> > > +\n> > > +\tqueueCount_++;\n> \n> Ah then you have to backtrack on queueCount if any of the supplied\n> Control is not supported. That's anyway an error that should happen\n> only during development, if a distracted pipeline handler mis-uses\n> this class, applications have no way to get it wrong, right ?\n\nFor OSS IPA yes but we also need to think about closed source IPAs.\n\n> \n> In this case, wouldn't a LOG(Fatal) help to catch development errors\n> earlier ? If you agree you can safely queueCount++, because if we\n> fail, the whole library gets torn down.\n\nInteresting, I'm not sure which one I like. I'm slightly prefering this \napproach I think as then it will be a pipeline handler decision if to \ntry and recover or fatally fail.\n\n> \n> > > +\n> > > +\treturn true;\n> \n> I've not looked ahead in the series, but is there value in this return\n> value ? I mean, if a pipeline handler gets this wrong is an\n> implementation error, not a runtime one, right ?\n\nAs stated above I think it's better to let the pipeline handler judge \nhow to best handle this case, but I might be wrong.\n\n> \n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Read back controls in effect at a specific sequence number\n> \n> Read back the controls' values at a specific point in time (or\n> sequence number as you've used) ?\n> \n> > > + * \\param[in] sequence Sequence number to get read back controls for\n> \n> The sequence number used to read controls values\n> \n> > > + *\n> > > + * Read back what controls where in effect at a specific sequence number. The\n> \n> Maybe it's me, but I cannot parse well \"controls where in effect\".\n> Native speakers to the rescue ? I would anyway use \"controls' values\"\n> here and in other places.\n> \n> > > + * history is a ring buffer of 16 entries where new and old values coexist. It's\n> > > + * the callers responsibility to not read to old sequence numbers that have been\n> > > + * pushed out of the history.\n> \n> Can we invalidate them with a flag maybe ?\n\nWe can't as they are overwritten and reused as it's cyclic buffer.\n\n> \n> > > + *\n> > > + * Historic values are evicted by pushing new values onto the queue using\n> > > + * push(). The max history from the current sequence number that yields valid\n> > > + * values are thus 16 minus number of controls pushed.\n> \n> This looks like class documentation I was asking for :)\n> \n> > > + *\n> > > + * \\returns List of controls in effect at \\a sequence\n> \n> As per above suggestion\n> \\return The values of controls at \\a sequence time\n> \n> > > + */\n> > > +ControlList DelayedControls::get(uint32_t sequence)\n> > > +{\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\tuint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> \n> Can it happen that sequence < firstSequence ?\n\nYes, when sequence wraps around 32 bits. There is a specific test for \nthis in the next patch.\n\n> \n> > > +\tunsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > > +\n> > > +\tControlList out(device_->controls());\n> \n> You access device_->controls() quite often, can it be cached ?\n> \n> > > +\tfor (const auto &ctrl : ctrls_) {\n> > > +\t\tconst ControlId *id = ctrl.first;\n> > > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > > +\n> > > +\t\tout.set(id->id(), info.value);\n> > > +\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Reading \" << id->name()\n> > > +\t\t\t<< \" to \" << info.value.toString()\n> > > +\t\t\t<< \" at index \" << index;\n> > > +\t}\n> > > +\n> > > +\treturn out;\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Inform DelayedControls of a start of a new frame\n> \n> Maybe I'm missing some piece that will come in the next patches, but\n> we have the video device, right ? Can't we connect to its frameStart\n> signal and handle this internally ?\n\nYes and no :-)\n\nThis is intended to be connected to frameStart() but unfortunately it's \nnot the sensors V4L2Device that emits the SOE event but the CSI-2 \nrecivers. So the connection between the two needs to be done in the \npipeline handler.\n\n> \n> > > + * \\param[in] sequence Sequence number of the frame that started\n> > > + *\n> > > + * Inform the state machine that a new frame have started and it's sequence\n> \n> s/have/has\n> s/it's/its\n> \n> Seems like a verb for \"sequence number is missing\"\n> \n> Inform the state machine that a new frame has started and its sequence\n> number is \\a sequence ?\n> \n> \n> > > + * number. It's user of this helpers responsibility to inform the helper\n> > > + * at the start of every frame. This can with ease be connected to the start\n> > > + * of exposure (SOE) V4L2 event.\n> \n> It's responsibility of the users of this class to notify the start of\n> every frame.\n> (I would not mention V4L2, but I'm might be too concerned, it's fine\n> if you like it)\n> \n> > > + */\n> > > +void DelayedControls::frameStart(uint32_t sequence)\n> > > +{\n> > > +\tLOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> > > +\n> > > +\tstd::lock_guard lock(lock_);\n> > > +\n> > > +\tif (!running_) {\n> > > +\t\tfistSequence_ = sequence;\n> > > +\t\trunning_ = true;\n> > > +\t}\n> > > +\n> > > +\t/*\n> > > +\t * Create control list peaking ahead in the value queue to ensure\n> > > +\t * values are set in time to satisfy the sensor delay.\n> > > +\t */\n> > > +\tControlList out(device_->controls());\n> > > +\tfor (const auto &ctrl : ctrls_) {\n> > > +\t\tconst ControlId *id = ctrl.first;\n> > > +\t\tunsigned int delayDiff = maxDelay_ - delays_[id];\n> > > +\t\tunsigned int index = std::max(0, writeCount_ - delayDiff);\n> \n> It's me but I don't get this logic as I didn't get it in\n> StaggeredControls. It was there already, so I'm sure it's correct, but\n> I don't get it :)\n> \n> Why is maxDelay_ relevant ?\n\nIt's used to calculate how far in advanced a control needs to be set, \nit's basically the control depth. If Exposure needs 2 frames to take \neffect and Gain 1, we have a depth of 2. So in when frame 100 start we \nneed to set the Exposure for 102 and Gain for 101.\n\nFor exposure,\n delayDiff = maxDelay_ - delays_[Exposure] = 2 - 2 = 0\n\nFor gain\n delayDiff = maxDelay_ - delays_[Gain] = 2 - 1 = 1\n\nSo the controls we set are gain for the frame+1 and exposure for \nframe+2.\n\n> \n> > > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > > +\n> > > +\t\tif (info.updated) {\n> > > +\t\t\tout.set(id->id(), info.value);\n> > > +\t\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t\t<< \"Setting \" << id->name()\n> > > +\t\t\t\t<< \" to \" << info.value.toString()\n> > > +\t\t\t\t<< \" at index \" << index;\n> > > +\t\t}\n> > > +\t}\n> > > +\n> > > +\twriteCount_++;\n> > > +\n> > > +\twhile (writeCount_ >= queueCount_) {\n> > > +\t\tLOG(DelayedControls, Debug)\n> > > +\t\t\t<< \"Queue is empty, auto queue no-op.\";\n> > > +\t\tqueue({});\n> > > +\t}\n> > > +\n> > > +\tdevice_->setControls(&out);\n> \n> Any value in returning this ? This come from the device, something\n> might be happening there and maybe pipelines want to be informed ?\n\nWe can't return anything from this as this functinon is meant to be \nconnected to frameStart().\n\n> \n> Thanks\n> j\n> \n> > > +}\n> > > +\n> > > +} /* namespace libcamera */\n> > > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > > --- a/src/libcamera/meson.build\n> > > +++ b/src/libcamera/meson.build\n> > > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > > 'controls.cpp',\n> > > 'control_serializer.cpp',\n> > > 'control_validator.cpp',\n> > > + 'delayed_controls.cpp',\n> > > 'device_enumerator.cpp',\n> > > 'device_enumerator_sysfs.cpp',\n> > > 'event_dispatcher.cpp',\n> > > --\n> > > 2.29.1\n> > >\n> > > _______________________________________________\n> > > libcamera-devel mailing list\n> > > libcamera-devel@lists.libcamera.org\n> > > https://lists.libcamera.org/listinfo/libcamera-devel\n> > _______________________________________________\n> > libcamera-devel mailing list\n> > libcamera-devel@lists.libcamera.org\n> > https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id F21B3BDB89\n\tfor ;\n\tTue, 10 Nov 2020 11:26:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8D085630BD;\n\tTue, 10 Nov 2020 12:26:52 +0100 (CET)","from mail-lf1-x144.google.com (mail-lf1-x144.google.com\n\t[IPv6:2a00:1450:4864:20::144])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8B69760342\n\tfor ;\n\tTue, 10 Nov 2020 12:26:50 +0100 (CET)","by mail-lf1-x144.google.com with SMTP id j205so9611946lfj.6\n\tfor ;\n\tTue, 10 Nov 2020 03:26:50 -0800 (PST)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tj69sm1893169lfj.49.2020.11.10.03.26.48\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tTue, 10 Nov 2020 03:26:48 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"dQwyYe+9\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=VcqqTR7M5AuIrYfzZ3sGH6g1I+uatMhZ24Fwv1KMcUk=;\n\tb=dQwyYe+9vMBi87VOc6JpXB6bJtV70N5D5KWCiSNjx7aPWpYX7VOInH48hu5x5zF1l6\n\tCo62WKbMp+3SObUkTNOuP/KPDaM9NARTVGKyd34QTe4gixev0/IDWkcOQPKl44inRUSS\n\tB6fJw+IWVvUePemmXRDZOdD9iotBaL3KcX4NO/OYtmB+kPh3BEC0A1mnfoli1nsq/SCj\n\tPgGg+VgFtGPxL2QGMcfUtxs7mbzuqgdux2hdOdVA3WKbl+xqBKQVlOvMIVXs5SiNUw7t\n\thUSY3RC3IB6v0xLzhwrkd65FYUtkuOeAPgC7jH36YKp27U8HgIciMMdyhqMUKkG+MM7A\n\tTppg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=VcqqTR7M5AuIrYfzZ3sGH6g1I+uatMhZ24Fwv1KMcUk=;\n\tb=c+GF2VyTUrbPutrLIZCB9CvcbZ8jWnbaN9E7R9CNSm2SMNvsYJnhOZlu57XaweZpHA\n\tecJYU151NM0YF9PR6NeJBM2Tm8j735mOSzqIphZxrEP/FY/2DJWFSko+BvX71ps08MAG\n\tJR77oEISaGITsJcJdyj4NixjGCWzLk99PUTLEDAizN7fg1L02uMIf/LoNKBN2VMc0wji\n\tRKFHAqZtuIfiF1FcIF+RFooywqfeXR9nHgTA7G8HFHveApK8W46623lUjxQbvhjt2XgY\n\t2R/m9/V2h6SucWbs9fj5Z0OdsCZ+AfiRKwuZ8XQhGF1CmpoMd9cp3C1NLKgSsjIeDBEz\n\tYEFA==","X-Gm-Message-State":"AOAM533ROmGbUX5QsScH1wEwO/D8pdMX5NKJydwQohGkBOORM5/4CbqZ\n\tUstcHycdj8b3Qh2wBp/DI/BWvQ==","X-Google-Smtp-Source":"ABdhPJxUf6JDu/Q6/vkv4PqOGBJc0R9xD9muEa4fc0f59faBKxoAiYan4wHnc9lnLh+UyUrBaaV4EA==","X-Received":"by 2002:ac2:5e68:: with SMTP id a8mr6594361lfr.506.1605007609389;\n\tTue, 10 Nov 2020 03:26:49 -0800 (PST)","Date":"Tue, 10 Nov 2020 12:26:47 +0100","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Jacopo Mondi ","Message-ID":"<20201110112647.GA660945@oden.dyn.berto.se>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t<20201104160800.ethen7ty3al5cug2@uno.localdomain>\n\t<20201105200848.655smknln4as62we@uno.localdomain>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201105200848.655smknln4as62we@uno.localdomain>","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13669,"web_url":"https://patchwork.libcamera.org/comment/13669/","msgid":"<20201110115333.436a6rfqhtkzhtsc@uno.localdomain>","date":"2020-11-10T11:53:33","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi Niklas,\n\nOn Tue, Nov 10, 2020 at 12:26:47PM +0100, Niklas Söderlund wrote:\n> Hi Jacopo,\n>\n> Thanks for your feedback.\n>\n> On 2020-11-05 21:08:48 +0100, Jacopo Mondi wrote:\n> > Hi again Niklas,\n> >\n> > On Wed, Nov 04, 2020 at 05:08:00PM +0100, Jacopo Mondi wrote:\n> > > Hi Niklas,\n> > >\n> > > On Wed, Oct 28, 2020 at 02:00:44AM +0100, Niklas Söderlund wrote:\n> > > > Some sensor controls take effect with a delay as the sensor needs time\n> > > > to adjust, for example exposure. Add a optional helper DelayedControls\n> > > > to help pipelines deal with such controls.\n> > > >\n> > > > The idea is to provide a queue of controls towards the V4L2 device and\n> > > > apply individual controls with the specified delay with the aim to get\n> > > > predictable and retrievable control values for any given frame. To do\n> > > > this the queue of controls needs to be at least as deep as the control\n> > > > with the largest delay.\n> > > >\n> > > > The DelayedControls needs to be informed of every start of exposure.\n> > > > This can be emulated but the helper is designed to be used with this\n> > > > event being provide by the kernel thru V4L2 events.\n> > >\n> > > s/thru/though ?\n> > > >\n> > > > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > > > handler but expands on its API. This helpers aims to replace the\n> > > > Raspberry Pi implementations and mimics it behavior perfectly.\n> > > >\n> > > > Signed-off-by: Niklas Söderlund \n> > > > ---\n> > > > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > > > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > > > src/libcamera/meson.build | 1 +\n> > > > 3 files changed, 370 insertions(+)\n> > > > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > > > create mode 100644 src/libcamera/delayed_controls.cpp\n> > > >\n> > > > diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h\n> > > > new file mode 100644\n> > > > index 0000000000000000..df5520d240a54e4b\n> > > > --- /dev/null\n> > > > +++ b/include/libcamera/internal/delayed_controls.h\n> > > > @@ -0,0 +1,87 @@\n> > > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > > +/*\n> > > > + * Copyright (C) 2020, Google Inc.\n> > > > + *\n> > > > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > > > + */\n> > > > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > > +\n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +\n> > > > +#include \n> > > > +\n> > > > +namespace libcamera {\n> > > > +\n> > > > +class V4L2Device;\n> > > > +\n> > > > +class DelayedControls\n> > > > +{\n> > > > +public:\n> > > > +\tDelayedControls(V4L2Device *device,\n> > > > +\t\t\tconst std::unordered_map &delays);\n> > > > +\n> > > > +\tvoid reset(ControlList *controls = nullptr);\n> > > > +\n> > > > +\tbool push(const ControlList &controls);\n> > > > +\tControlList get(uint32_t sequence);\n> > > > +\n> > > > +\tvoid frameStart(uint32_t sequence);\n> > > > +\n> > > > +private:\n> > > > +\tclass ControlInfo\n> > >\n> > > With no other scope but public:, is it worth a class ?\n> > >\n> > > > +\t{\n> > > > +\tpublic:\n> > > > +\t\tControlInfo()\n> > > > +\t\t\t: updated(false)\n> > > > +\t\t{\n> > > > +\t\t}\n> > > > +\n> > > > +\t\tControlInfo(const ControlValue &v)\n> > > > +\t\t\t: value(v), updated(true)\n> > > > +\t\t{\n> > > > +\t\t}\n> > > > +\n> > > > +\t\tControlValue value;\n> > > > +\t\tbool updated;\n> > > > +\t};\n> > > > +\n> > > > +\tstatic constexpr int listSize = 16;\n> > > > +\tclass ControlArray : public std::array\n> > > > +\t{\n> > > > +\tpublic:\n> > > > +\t\tControlInfo &operator[](unsigned int index)\n> > > > +\t\t{\n> > > > +\t\t\treturn std::array::operator[](index % listSize);\n> > > > +\t\t}\n> > > > +\n> > > > +\t\tconst ControlInfo &operator[](unsigned int index) const\n> > > > +\t\t{\n> > > > +\t\t\treturn std::array::operator[](index % listSize);\n> > > > +\t\t}\n> > > > +\t};\n> > > > +\n> > > > +\tusing ControlsDelays = std::unordered_map;\n> > > > +\tusing ControlsValues = std::unordered_map;\n> >\n> > Do you think indexing on ControlId is a good idea ? I see quite some\n> > churn below to get from the integer id obtained by iterating a\n> > ControlList to the ControlId used to index these maps:\n> >\n> > \t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> >\n> > While from ControlId we can always easily get the id back\n>\n> I think it's worth it as we try to stay away form using V4L2 identifiers\n> for as long as possible. As a bonus we get access to the string name of\n> the control for the logging.\n>\n\nPlease note libcamera controls have a numeric id too\n\n> >\n> > > > +\n> > > > +\tbool queue(const ControlList &controls);\n> > > > +\n> > > > +\tstd::mutex lock_;\n> > > > +\n> > > > +\tV4L2Device *device_;\n> > > > +\tControlsDelays delays_;\n> > > > +\tunsigned int maxDelay_;\n> > > > +\n> > > > +\tbool running_;\n> > > > +\tuint32_t fistSequence_;\n> > >\n> > > Should this be 'first' ?\n> > >\n> > > > +\n> > > > +\tuint32_t queueCount_;\n> > > > +\tuint32_t writeCount_;\n> > > > +\tControlsValues ctrls_;\n> > > > +};\n> > > > +\n> > > > +} /* namespace libcamera */\n> > > > +\n> > > > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > > > diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp\n> > > > new file mode 100644\n> > > > index 0000000000000000..0e32f417c5cc68b7\n> > > > --- /dev/null\n> > > > +++ b/src/libcamera/delayed_controls.cpp\n> > > > @@ -0,0 +1,282 @@\n> > > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > > +/*\n> > > > + * Copyright (C) 2020, Google Inc.\n> > > > + *\n> > > > + * delayed_controls.h - Helper to deal with controls that are applied with a delay\n> > > > + */\n> > > > +\n> > > > +#include \"libcamera/internal/delayed_controls.h\"\n> > > > +#include \"libcamera/internal/v4l2_device.h\"\n> > > > +\n> > > > +#include \n> > > > +\n> > > > +#include \"libcamera/internal/log.h\"\n> > > > +\n> > > > +/**\n> > > > + * \\file delayed_controls.h\n> > > > + * \\brief Helper to deal with controls that are applied with a delay\n> > > > + */\n> > > > +\n> > > > +namespace libcamera {\n> > > > +\n> > > > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > > > +\n> > > > +/**\n> > > > + * \\class DelayedControls\n> > > > + * \\brief Helper to deal with controls that takes effect with a delay\n> > >\n> > > controls -> take\n> > >\n> > > > + *\n> > > > + * Some sensor controls take effect with a delay as the sensor needs time to\n> > >\n> > > controls -> take\n> > >\n> > > > + * adjust, for example exposure and focus. This is an optional helper class to\n> > > > + * deal with such controls and the intended user is pipeline handlers.\n> > >\n> > > users are pipeline handlers\n> > >\n> > > > + *\n> > > > + * The idea is to extend the concept of the pipeline depth the users needs to\n> > > > + * maintain for buffers to controls. The depth is determined with by the control\n> > > > + * with the grates delay. As long as the pipeline keeps the control queue above\n> > > > + * this level the control values are guaranteed to be in effect at the specified\n> > > > + * point in time.\n> > >\n> > > I would drop this as it mentions other concepts that might be more\n> > > confusing than helping.\n> > >\n> > > > + *\n> > > > + * The downside is of course that the pipeline needs to know what controls to\n> > > > + * set control depth frames in advance. But there really is no way around this\n> > > > + * as the delay is a consequence of the physical world. Compare this with\n> > > > + * parameter buffers where the buffer may be queued to the device while it's\n> > > > + * still being written to as long as it's ready before it's consumed, this is\n> > > > + * because the parameter buffer (usually) does not contain controls that\n> > > > + * requires time to take effect.\n> > >\n> > > I find this confusing too, I'm sorry.\n> > >\n> > > I think the key concept is that controls have to be applied in\n> > > advance, the usage of \"Delayed\" took me a bit off-road as a\n> > > DelayedControls class seems to be designed to apply controls at a\n> > > later (delayed) point in time while it's actually the other way\n> > > around. One key point is that for each Control the pipeline handler\n> > > has to provide a known delay value, something I didn't realize until I\n> > > looked at the code. The class takes care of applying the control the\n> > > control in -advance- enough to make sure the control is fully applied\n> > > at the time the Request it is associated to completes.\n> > >\n> > > In the class description I would describe what the class offers and\n> > > how it has to be used, its API and how I should set\n> > > controls specifying their 'delay'. Design documentation is good but\n> > > only to complement usage documentation.\n> > >\n> > > > + */\n> > > > +\n> >\n> > double empy line\n> >\n> > > > +\n> > > > +/**\n> > > > + * \\brief Construct a DelayedControls\n> > > > + * \\param[in] device The V4L2 device containing the delayed controls\n> > >\n> > > The V4L2 device the controls have to be applied to\n> > >\n> > > > + * \\param[in] delays Map of numerical V4L2 control id to its delay to take\n> > > > + * effect in frames\n> > >\n> > > isn't 'latency' better than delay ? Or any other term ? See how heavy\n> > > is the description you had to make of this parameter...\n> > >\n> > > \\param[in] delays Map of V4L2 control ids and their associated latencies (in frames)\n> > >\n> > > ?\n> > > > + *\n> > > > + * Only controls specified in \\a delays are handled by the DelayedControls\n> > > > + * instance. If it's desired to mix delayed controls and controls that takes\n> > > > + * effect immediately the immediate controls must be listed in the \\a delays map\n> > > > + * with a delay value of 0.\n> > >\n> > > delays is a map, you have to provide a delay value by design\n> > >\n> > > \"Controls with an associated 0 delay (latency) are applied immediately\"\n> > >\n> > >\n> > > > + */\n> > > > +DelayedControls::DelayedControls(V4L2Device *device,\n> > > > +\t\t\t\t const std::unordered_map &delays)\n> > > > +\t: device_(device), maxDelay_(0)\n> > > > +{\n> > > > +\tconst ControlInfoMap &controls = device_->controls();\n> > > > +\n> > > > +\t/*\n> > > > +\t * Sanity check that all controls where delays are requested are\n> > >\n> > > s/where delays//\n> > >\n> >\n> > Actually I meant to make this\n> > Sanity check that all requested controls are exposed by the device\n> >\n> > > > +\t * exposed byt the device.\n> > >\n> > > by\n> > >\n> > > > +\t */\n> > > > +\tfor (auto const &delay : delays) {\n> > > > +\t\tunsigned int id = delay.first;\n> > > > +\n> > > > +\t\tif (controls.find(id) == controls.end())\n> > > > +\t\t\tLOG(DelayedControls, Error)\n> > > > +\t\t\t\t<< \"Delay request for control id \"\n> > > > +\t\t\t\t<< utils::hex(id)\n> > > > +\t\t\t\t<< \" but control is not exposed by device \"\n> > > > +\t\t\t\t<< device_->deviceNode();\n> > > > +\t}\n> > > > +\n> > > > +\t/*\n> > > > +\t * Create a map of control to delay for all controls exposed by the\n> > >\n> > > controls to delays\n> > >\n> > > > +\t * device. If no delay is specified assume the control applies directly.\n> > >\n> > > assume a 0 delay (latency)\n> > >\n> > > > +\t */\n> > > > +\tfor (auto const &control : controls) {\n> > > > +\t\tconst ControlId *id = control.first;\n> > > > +\n> > > > +\t\tauto it = delays.find(id->id());\n> > > > +\t\tif (it == delays.end())\n> > > > +\t\t\tcontinue;\n> > > > +\n> > > > +\t\tdelays_[id] = it->second;\n> > > > +\n> > > > +\t\tLOG(DelayedControls, Debug)\n> > > > +\t\t\t<< \"Set a delay of \" << delays_[id]\n> > > > +\t\t\t<< \" for \" << id->name();\n> > > > +\n> > > > +\t\tmaxDelay_ = std::max(maxDelay_, delays_[id]);\n> > > > +\t}\n> >\n> > I wonder if these loops should not be unified. After all, if a\n> > device's control is not in the delay map it is skipped, so it is the\n> > same as iterating the delay map as done in the previous loop ?\n>\n> They should and are in v2.\n>\n> >\n> > > > +\n> > > > +\treset();\n> > > > +}\n> > > > +\n> > > > +/**\n> > > > + * \\brief Reset the V4L2 device\n> > >\n> > > Reset controls to their default values ?\n> > >\n> > > > + * \\param[in] controls List of controls to reset the device to or nullptr\n> > >\n> > > Optional list of controls to apply to the device ?\n> > >\n> > > These is mostly on documentation, I'll look at implementation later!\n> > >\n> > > Thanks\n> > > j\n> > >\n> > >\n> > > > + *\n> > > > + * Resets the delayed controls state machine to its starting state. All controls\n> > > > + * are fetched from the V4L2 device to provide a good starting point for the\n> > > > + * first frames (length of control depth).\n> >\n> > I don't get this last statement\n> >\n> > > > + *\n> > > > + * Optionally \\a controls can be specified to set some or all of the handled\n> > > > + * V4L2 controls prior to reading them back. If no controls needs to be set\n> > > > + * nullptr may be used.\n> >\n> > I would drop the last sentence\n> >\n> > > > + */\n> > > > +void DelayedControls::reset(ControlList *controls)\n> > > > +{\n> > > > +\tstd::lock_guard lock(lock_);\n> > > > +\n> > > > +\trunning_ = false;\n> > > > +\tfistSequence_ = 0;\n> > > > +\tqueueCount_ = 0;\n> > > > +\twriteCount_ = 0;\n> > > > +\n> > > > +\t/* Set the controls on the device if requested. */\n> > > > +\tif (controls)\n> > > > +\t\tdevice_->setControls(controls);\n> > > > +\n> > > > +\t/* Retrieve current control values reported by the device. */\n> > > > +\tstd::vector ids;\n> > > > +\tfor (auto const &delay : delays_)\n> > > > +\t\tids.push_back(delay.first->id());\n> > > > +\n> > > > +\tControlList devCtrls = device_->getControls(ids);\n> > > > +\n> > > > +\t/* Seed the control queue with the controls reported by the device. */\n> > > > +\tctrls_.clear();\n> > > > +\tfor (const auto &ctrl : devCtrls) {\n> > > > +\t\tconst ControlId *id = devCtrls.infoMap()->idmap().at(ctrl.first);\n> > > > +\t\tctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > > > +\t}\n> > > > +\n> > > > +\tqueueCount_++;\n> > > > +}\n> > > > +\n> > > > +/**\n> > > > + * \\brief Push a set of controls on the queue\n> > > > + * \\param[in] controls List of controls to add the device queue\n> > > > + *\n> > > > + * Push a set of controls to the control queue. This increases the control queue\n> > > > + * depth by one.\n> >\n> > control queue depth really confuses me.\n> >\n> > Isn't it better to just say:\n> > \"Push a new set of values for the controls\"\n> > or something similar ?\n> >\n> > > > + *\n> > > > + * \\returns true if \\a controls are accepted, or false otherwise\n> > > > + */\n> > > > +bool DelayedControls::push(const ControlList &controls)\n> > > > +{\n> > > > +\tstd::lock_guard lock(lock_);\n> > > > +\n> > > > +\treturn queue(controls);\n> > > > +}\n> > > > +\n> > > > +bool DelayedControls::queue(const ControlList &controls)\n> > > > +{\n> > > > +\t/* Copy state from previous frame. */\n> > > > +\tfor (auto &ctrl : ctrls_) {\n> > > > +\t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> >\n> > Am I wrong of\n> > \tfor (auto &ctrl : ctrls_) {\n> > \t\tControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> >\n> > Is equivalent to\n> > \tfor (auto &ctrl : ctrls_) {\n> > \t\tControlInfo &info = ctrl.second[queueCount_];\n> >\n> > (the compiler agrees afaict)\n>\n> It does, thanks for this so much nicer to read!\n>\n> >\n> > > > +\t\tinfo.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > > > +\t\tinfo.updated = false;\n> > > > +\t}\n> >\n> > Have you considered breaking this in 2 ? This would avoid the below\n> > queue({}) which seems a workaround.\n> >\n> > Can't we:\n> > advance() {\n> > copy values from previous entry\n> > queueCount++;\n> > }\n> >\n> > update() {\n> > copy the new values in;\n> > }\n> >\n> > So that below you only call advance() ?\n>\n> I like the notion of one having to queue an empty list as it fits the\n> concept of this helper. Then maybe we can optimize it further if an\n> empty set is queue. Maybe this could be a future improvement as this\n> will only effect the internal API of this class anyhow. And we still\n> need to allow for pipelines to que empty controls list to keep in sync.\n\nI was, at the contrary bothered by having to push an empy list, as it\nfeels like working against the API. But if pipeline handlers have to\ndo the same, then it's fine I guess\n\n>\n> >\n> > > > +\n> > > > +\t/* Update with new controls. */\n> > > > +\tfor (const auto &control : controls) {\n> > > > +\t\tconst ControlId *id = device_->controls().idmap().at(control.first);\n> > > > +\n> > > > +\t\tif (delays_.find(id) == delays_.end())\n> > > > +\t\t\treturn false;\n> > > > +\n> > > > +\t\tControlInfo &info = ctrls_[id][queueCount_];\n> >\n> > That's a double lookup that can be avoided. If I'm not mistaken all\n> > controls in delays_ are inserted in ctrls_, am I wrong ? If that's the\n> > case you can just ctrls_.find() == ctls_.end() ?\n>\n> I'm not sure I get this one.\n>\n\ndelays_.find(id) is equivalent to ctrls_[id] assuming delays_ and\nctrls_ contain the same set of controls\n\n> >\n> > > > +\n> > > > +\t\tinfo.value = control.second;\n> > > > +\t\tinfo.updated = true;\n> > > > +\n> > > > +\t\tLOG(DelayedControls, Debug)\n> > > > +\t\t\t<< \"Queuing \" << id->name()\n> > > > +\t\t\t<< \" to \" << info.value.toString()\n> > > > +\t\t\t<< \" at index \" << queueCount_;\n> > > > +\t}\n> > > > +\n> > > > +\tqueueCount_++;\n> >\n> > Ah then you have to backtrack on queueCount if any of the supplied\n> > Control is not supported. That's anyway an error that should happen\n> > only during development, if a distracted pipeline handler mis-uses\n> > this class, applications have no way to get it wrong, right ?\n>\n> For OSS IPA yes but we also need to think about closed source IPAs.\n>\n> >\n> > In this case, wouldn't a LOG(Fatal) help to catch development errors\n> > earlier ? If you agree you can safely queueCount++, because if we\n> > fail, the whole library gets torn down.\n>\n> Interesting, I'm not sure which one I like. I'm slightly prefering this\n> approach I think as then it will be a pipeline handler decision if to\n> try and recover or fatally fail.\n>\n\nIf it were a run-time error I could agree. But if this fail it means\nthe IPA (open or closed) has a bug.\n\nUp to you\n\n> >\n> > > > +\n> > > > +\treturn true;\n> >\n> > I've not looked ahead in the series, but is there value in this return\n> > value ? I mean, if a pipeline handler gets this wrong is an\n> > implementation error, not a runtime one, right ?\n>\n> As stated above I think it's better to let the pipeline handler judge\n> how to best handle this case, but I might be wrong.\n>\n> >\n> > > > +}\n> > > > +\n> > > > +/**\n> > > > + * \\brief Read back controls in effect at a specific sequence number\n> >\n> > Read back the controls' values at a specific point in time (or\n> > sequence number as you've used) ?\n> >\n> > > > + * \\param[in] sequence Sequence number to get read back controls for\n> >\n> > The sequence number used to read controls values\n> >\n> > > > + *\n> > > > + * Read back what controls where in effect at a specific sequence number. The\n> >\n> > Maybe it's me, but I cannot parse well \"controls where in effect\".\n> > Native speakers to the rescue ? I would anyway use \"controls' values\"\n> > here and in other places.\n> >\n> > > > + * history is a ring buffer of 16 entries where new and old values coexist. It's\n> > > > + * the callers responsibility to not read to old sequence numbers that have been\n> > > > + * pushed out of the history.\n> >\n> > Can we invalidate them with a flag maybe ?\n>\n> We can't as they are overwritten and reused as it's cyclic buffer.\n>\n\nSo this mean the caller must not go past [sequence - 16], right ? Is\nit enforced ?\n\n> >\n> > > > + *\n> > > > + * Historic values are evicted by pushing new values onto the queue using\n> > > > + * push(). The max history from the current sequence number that yields valid\n> > > > + * values are thus 16 minus number of controls pushed.\n> >\n> > This looks like class documentation I was asking for :)\n> >\n> > > > + *\n> > > > + * \\returns List of controls in effect at \\a sequence\n> >\n> > As per above suggestion\n> > \\return The values of controls at \\a sequence time\n> >\n> > > > + */\n> > > > +ControlList DelayedControls::get(uint32_t sequence)\n> > > > +{\n> > > > +\tstd::lock_guard lock(lock_);\n> > > > +\n> > > > +\tuint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> >\n> > Can it happen that sequence < firstSequence ?\n>\n> Yes, when sequence wraps around 32 bits. There is a specific test for\n> this in the next patch.\n>\n> >\n> > > > +\tunsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > > > +\n> > > > +\tControlList out(device_->controls());\n> >\n> > You access device_->controls() quite often, can it be cached ?\n> >\n> > > > +\tfor (const auto &ctrl : ctrls_) {\n> > > > +\t\tconst ControlId *id = ctrl.first;\n> > > > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > > > +\n> > > > +\t\tout.set(id->id(), info.value);\n> > > > +\n> > > > +\t\tLOG(DelayedControls, Debug)\n> > > > +\t\t\t<< \"Reading \" << id->name()\n> > > > +\t\t\t<< \" to \" << info.value.toString()\n> > > > +\t\t\t<< \" at index \" << index;\n> > > > +\t}\n> > > > +\n> > > > +\treturn out;\n> > > > +}\n> > > > +\n> > > > +/**\n> > > > + * \\brief Inform DelayedControls of a start of a new frame\n> >\n> > Maybe I'm missing some piece that will come in the next patches, but\n> > we have the video device, right ? Can't we connect to its frameStart\n> > signal and handle this internally ?\n>\n> Yes and no :-)\n>\n> This is intended to be connected to frameStart() but unfortunately it's\n> not the sensors V4L2Device that emits the SOE event but the CSI-2\n> recivers. So the connection between the two needs to be done in the\n> pipeline handler.\n>\n\nAh right :(\n\n> >\n> > > > + * \\param[in] sequence Sequence number of the frame that started\n> > > > + *\n> > > > + * Inform the state machine that a new frame have started and it's sequence\n> >\n> > s/have/has\n> > s/it's/its\n> >\n> > Seems like a verb for \"sequence number is missing\"\n> >\n> > Inform the state machine that a new frame has started and its sequence\n> > number is \\a sequence ?\n> >\n> >\n> > > > + * number. It's user of this helpers responsibility to inform the helper\n> > > > + * at the start of every frame. This can with ease be connected to the start\n> > > > + * of exposure (SOE) V4L2 event.\n> >\n> > It's responsibility of the users of this class to notify the start of\n> > every frame.\n> > (I would not mention V4L2, but I'm might be too concerned, it's fine\n> > if you like it)\n> >\n> > > > + */\n> > > > +void DelayedControls::frameStart(uint32_t sequence)\n> > > > +{\n> > > > +\tLOG(DelayedControls, Debug) << \"frame \" << sequence << \" started\";\n> > > > +\n> > > > +\tstd::lock_guard lock(lock_);\n> > > > +\n> > > > +\tif (!running_) {\n> > > > +\t\tfistSequence_ = sequence;\n> > > > +\t\trunning_ = true;\n> > > > +\t}\n> > > > +\n> > > > +\t/*\n> > > > +\t * Create control list peaking ahead in the value queue to ensure\n> > > > +\t * values are set in time to satisfy the sensor delay.\n> > > > +\t */\n> > > > +\tControlList out(device_->controls());\n> > > > +\tfor (const auto &ctrl : ctrls_) {\n> > > > +\t\tconst ControlId *id = ctrl.first;\n> > > > +\t\tunsigned int delayDiff = maxDelay_ - delays_[id];\n> > > > +\t\tunsigned int index = std::max(0, writeCount_ - delayDiff);\n> >\n> > It's me but I don't get this logic as I didn't get it in\n> > StaggeredControls. It was there already, so I'm sure it's correct, but\n> > I don't get it :)\n> >\n> > Why is maxDelay_ relevant ?\n>\n> It's used to calculate how far in advanced a control needs to be set,\n> it's basically the control depth. If Exposure needs 2 frames to take\n> effect and Gain 1, we have a depth of 2. So in when frame 100 start we\n> need to set the Exposure for 102 and Gain for 101.\n>\n> For exposure,\n> delayDiff = maxDelay_ - delays_[Exposure] = 2 - 2 = 0\n>\n> For gain\n> delayDiff = maxDelay_ - delays_[Gain] = 2 - 1 = 1\n>\n> So the controls we set are gain for the frame+1 and exposure for\n> frame+2.\n\nAh yes, index give us \"how far ahead\" we have to pick the control\nvalue, not which controls to set as I first thought!\n\nThanks\n\n\n>\n> >\n> > > > +\t\tconst ControlInfo &info = ctrl.second[index];\n> > > > +\n> > > > +\t\tif (info.updated) {\n> > > > +\t\t\tout.set(id->id(), info.value);\n> > > > +\t\t\tLOG(DelayedControls, Debug)\n> > > > +\t\t\t\t<< \"Setting \" << id->name()\n> > > > +\t\t\t\t<< \" to \" << info.value.toString()\n> > > > +\t\t\t\t<< \" at index \" << index;\n> > > > +\t\t}\n> > > > +\t}\n> > > > +\n> > > > +\twriteCount_++;\n> > > > +\n> > > > +\twhile (writeCount_ >= queueCount_) {\n> > > > +\t\tLOG(DelayedControls, Debug)\n> > > > +\t\t\t<< \"Queue is empty, auto queue no-op.\";\n> > > > +\t\tqueue({});\n> > > > +\t}\n> > > > +\n> > > > +\tdevice_->setControls(&out);\n> >\n> > Any value in returning this ? This come from the device, something\n> > might be happening there and maybe pipelines want to be informed ?\n>\n> We can't return anything from this as this functinon is meant to be\n> connected to frameStart().\n>\n\nAck!\n\nThanks for the explanation!\n\n> >\n> > Thanks\n> > j\n> >\n> > > > +}\n> > > > +\n> > > > +} /* namespace libcamera */\n> > > > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > > > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > > > --- a/src/libcamera/meson.build\n> > > > +++ b/src/libcamera/meson.build\n> > > > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > > > 'controls.cpp',\n> > > > 'control_serializer.cpp',\n> > > > 'control_validator.cpp',\n> > > > + 'delayed_controls.cpp',\n> > > > 'device_enumerator.cpp',\n> > > > 'device_enumerator_sysfs.cpp',\n> > > > 'event_dispatcher.cpp',\n> > > > --\n> > > > 2.29.1\n> > > >\n> > > > _______________________________________________\n> > > > libcamera-devel mailing list\n> > > > libcamera-devel@lists.libcamera.org\n> > > > https://lists.libcamera.org/listinfo/libcamera-devel\n> > > _______________________________________________\n> > > libcamera-devel mailing list\n> > > libcamera-devel@lists.libcamera.org\n> > > https://lists.libcamera.org/listinfo/libcamera-devel\n>\n> --\n> Regards,\n> Niklas Söderlund","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 309B2BE082\n\tfor ;\n\tTue, 10 Nov 2020 11:53:34 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id CE02F630C2;\n\tTue, 10 Nov 2020 12:53:33 +0100 (CET)","from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net\n\t[217.70.183.195])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C5EA1630BA\n\tfor ;\n\tTue, 10 Nov 2020 12:53:32 +0100 (CET)","from uno.localdomain (93-34-118-233.ip49.fastwebnet.it\n\t[93.34.118.233]) (Authenticated sender: jacopo@jmondi.org)\n\tby relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 44C8F60005;\n\tTue, 10 Nov 2020 11:53:31 +0000 (UTC)"],"X-Originating-IP":"93.34.118.233","Date":"Tue, 10 Nov 2020 12:53:33 +0100","From":"Jacopo Mondi ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Message-ID":"<20201110115333.436a6rfqhtkzhtsc@uno.localdomain>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t<20201104160800.ethen7ty3al5cug2@uno.localdomain>\n\t<20201105200848.655smknln4as62we@uno.localdomain>\n\t<20201110112647.GA660945@oden.dyn.berto.se>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201110112647.GA660945@oden.dyn.berto.se>","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13671,"web_url":"https://patchwork.libcamera.org/comment/13671/","msgid":"","date":"2020-11-10T15:45:45","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Niklas,\n\n\nOn Mon, 9 Nov 2020 at 22:54, Niklas Söderlund \nwrote:\n\n> Hi Naush,\n>\n> Thanks for your feedback.\n>\n> On 2020-11-03 10:27:06 +0000, Naushir Patuck wrote:\n> > Hi Niklas,\n> >\n> > Thank you for your patch. Please see some initial comments below.\n> >\n> > Best regards,\n> > Naush\n> >\n> >\n> > On Wed, 28 Oct 2020 at 01:01, Niklas Söderlund <\n> > niklas.soderlund@ragnatech.se> wrote:\n> >\n> > > Some sensor controls take effect with a delay as the sensor needs time\n> > > to adjust, for example exposure. Add a optional helper DelayedControls\n> > > to help pipelines deal with such controls.\n> > >\n> > > The idea is to provide a queue of controls towards the V4L2 device and\n> > > apply individual controls with the specified delay with the aim to get\n> > > predictable and retrievable control values for any given frame. To do\n> > > this the queue of controls needs to be at least as deep as the control\n> > > with the largest delay.\n> > >\n> > > The DelayedControls needs to be informed of every start of exposure.\n> > > This can be emulated but the helper is designed to be used with this\n> > > event being provide by the kernel thru V4L2 events.\n> > >\n> > > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > > handler but expands on its API. This helpers aims to replace the\n> > > Raspberry Pi implementations and mimics it behavior perfectly.\n> > >\n> > > Signed-off-by: Niklas Söderlund \n> > > ---\n> > > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > > src/libcamera/meson.build | 1 +\n> > > 3 files changed, 370 insertions(+)\n> > > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > > create mode 100644 src/libcamera/delayed_controls.cpp\n> > >\n> > > diff --git a/include/libcamera/internal/delayed_controls.h\n> > > b/include/libcamera/internal/delayed_controls.h\n> > > new file mode 100644\n> > > index 0000000000000000..df5520d240a54e4b\n> > > --- /dev/null\n> > > +++ b/include/libcamera/internal/delayed_controls.h\n> > > @@ -0,0 +1,87 @@\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Google Inc.\n> > > + *\n> > > + * delayed_controls.h - Helper to deal with controls that are applied\n> > > with a delay\n> > > + */\n> > >\n> >\n> > Given this is heavily derived from the staggered_write work, would it be\n> ok\n> > to keep the Raspberry Pi copyright in the header?\n> >\n> > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n>\n> As discussed of-band this was always my intention, my templates got in\n> the way. Sorry about that and thanks for pointing it out.\n>\n> >\n> >\n> > > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +class V4L2Device;\n> > > +\n> > > +class DelayedControls\n> > > +{\n> > > +public:\n> > > + DelayedControls(V4L2Device *device,\n> > > + const std::unordered_map int>\n> > > &delays);\n> > > +\n> > > + void reset(ControlList *controls = nullptr);\n> > > +\n> > > + bool push(const ControlList &controls);\n> > > + ControlList get(uint32_t sequence);\n> > > +\n> > > + void frameStart(uint32_t sequence);\n> > > +\n> > > +private:\n> > > + class ControlInfo\n> > > + {\n> > > + public:\n> > > + ControlInfo()\n> > > + : updated(false)\n> > > + {\n> > > + }\n> > > +\n> > > + ControlInfo(const ControlValue &v)\n> > > + : value(v), updated(true)\n> > > + {\n> > > + }\n> > > +\n> > > + ControlValue value;\n> > > + bool updated;\n> > > + };\n> > > +\n> > > + static constexpr int listSize = 16;\n> > > + class ControlArray : public std::array\n> > > + {\n> > > + public:\n> > > + ControlInfo &operator[](unsigned int index)\n> > > + {\n> > > + return std::array > > listSize>::operator[](index % listSize);\n> > > + }\n> > > +\n> > > + const ControlInfo &operator[](unsigned int index) const\n> > > + {\n> > > + return std::array > > listSize>::operator[](index % listSize);\n> > > + }\n> > > + };\n> > > +\n> > > + using ControlsDelays = std::unordered_map > > unsigned int>;\n> > > + using ControlsValues = std::unordered_map > > ControlArray>;\n> > > +\n> > > + bool queue(const ControlList &controls);\n> > > +\n> > > + std::mutex lock_;\n> > > +\n> > > + V4L2Device *device_;\n> > > + ControlsDelays delays_;\n> > > + unsigned int maxDelay_;\n> > > +\n> > > + bool running_;\n> > > + uint32_t fistSequence_;\n> > > +\n> > > + uint32_t queueCount_;\n> > > + uint32_t writeCount_;\n> > > + ControlsValues ctrls_;\n> > > +};\n> > > +\n> > > +} /* namespace libcamera */\n> > > +\n> > > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > > diff --git a/src/libcamera/delayed_controls.cpp\n> > > b/src/libcamera/delayed_controls.cpp\n> > > new file mode 100644\n> > > index 0000000000000000..0e32f417c5cc68b7\n> > > --- /dev/null\n> > > +++ b/src/libcamera/delayed_controls.cpp\n> > > @@ -0,0 +1,282 @@\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) 2020, Google Inc.\n> > > + *\n> > > + * delayed_controls.h - Helper to deal with controls that are applied\n> > > with a delay\n> > > + */\n> > >\n> >\n> > Similar to above:\n> > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n> >\n> >\n> > > +\n> > > +#include \"libcamera/internal/delayed_controls.h\"\n> > > +#include \"libcamera/internal/v4l2_device.h\"\n> > > +\n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/log.h\"\n> > > +\n> > > +/**\n> > > + * \\file delayed_controls.h\n> > > + * \\brief Helper to deal with controls that are applied with a delay\n> > > + */\n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > > +\n> > > +/**\n> > > + * \\class DelayedControls\n> > > + * \\brief Helper to deal with controls that takes effect with a delay\n> > > + *\n> > > + * Some sensor controls take effect with a delay as the sensor needs\n> time\n> > > to\n> > > + * adjust, for example exposure and focus. This is an optional helper\n> > > class to\n> > > + * deal with such controls and the intended user is pipeline handlers.\n> > > + *\n> > > + * The idea is to extend the concept of the pipeline depth the users\n> > > needs to\n> > > + * maintain for buffers to controls. The depth is determined with by\n> the\n> > > control\n> > > + * with the grates delay. As long as the pipeline keeps the control\n> queue\n> > > above\n> > > + * this level the control values are guaranteed to be in effect at the\n> > > specified\n> > > + * point in time.\n> > > + *\n> > > + * The downside is of course that the pipeline needs to know what\n> > > controls to\n> > > + * set control depth frames in advance. But there really is no way\n> around\n> > > this\n> > > + * as the delay is a consequence of the physical world. Compare this\n> with\n> > > + * parameter buffers where the buffer may be queued to the device\n> while\n> > > it's\n> > > + * still being written to as long as it's ready before it's consumed,\n> > > this is\n> > > + * because the parameter buffer (usually) does not contain controls\n> that\n> > > + * requires time to take effect.\n> > > + */\n> > > +\n> > > +\n> > > +/**\n> > > + * \\brief Construct a DelayedControls\n> > > + * \\param[in] device The V4L2 device containing the delayed controls\n> > > + * \\param[in] delays Map of numerical V4L2 control id to its delay to\n> take\n> > > + * effect in frames\n> > > + *\n> > > + * Only controls specified in \\a delays are handled by the\n> DelayedControls\n> > > + * instance. If it's desired to mix delayed controls and controls that\n> > > takes\n> > > + * effect immediately the immediate controls must be listed in the \\a\n> > > delays map\n> > > + * with a delay value of 0.\n> > > + */\n> > > +DelayedControls::DelayedControls(V4L2Device *device,\n> > > + const std::unordered_map > > unsigned int> &delays)\n> > > + : device_(device), maxDelay_(0)\n> > > +{\n> > > + const ControlInfoMap &controls = device_->controls();\n> > > +\n> > > + /*\n> > > + * Sanity check that all controls where delays are requested\n> are\n> > > + * exposed byt the device.\n> > > + */\n> > > + for (auto const &delay : delays) {\n> > > + unsigned int id = delay.first;\n> > > +\n> > > + if (controls.find(id) == controls.end())\n> > > + LOG(DelayedControls, Error)\n> > > + << \"Delay request for control id \"\n> > > + << utils::hex(id)\n> > > + << \" but control is not exposed by\n> device \"\n> > > + << device_->deviceNode();\n> > > + }\n> > > +\n> > > + /*\n> > > + * Create a map of control to delay for all controls exposed\n> by the\n> > > + * device. If no delay is specified assume the control applies\n> > > directly.\n> > > + */\n> > > + for (auto const &control : controls) {\n> > > + const ControlId *id = control.first;\n> > > +\n> > > + auto it = delays.find(id->id());\n> > > + if (it == delays.end())\n> > > + continue;\n> > > +\n> > > + delays_[id] = it->second;\n> > > +\n> > > + LOG(DelayedControls, Debug)\n> > > + << \"Set a delay of \" << delays_[id]\n> > > + << \" for \" << id->name();\n> > > +\n> > > + maxDelay_ = std::max(maxDelay_, delays_[id]);\n> > > + }\n> > >\n> >\n> > Could you combine the two loops here into one here? If you iterate over\n> > \"delays\", you could do the error check as well as adding to the \"delays_\"\n> > map.\n>\n> Good call!\n>\n> >\n> > +\n> > > + reset();\n> > >\n> >\n> > Does this need to be called here? But my understanding of reset() may be\n> > wrong... see below.\n>\n> It does as reset() initialize the internal state machine.\n>\n> >\n> >\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Reset the V4L2 device\n> > > + * \\param[in] controls List of controls to reset the device to or\n> nullptr\n> > > + *\n> > > + * Resets the delayed controls state machine to its starting state.\n> All\n> > > controls\n> > > + * are fetched from the V4L2 device to provide a good starting point\n> for\n> > > the\n> > > + * first frames (length of control depth).\n> > > + *\n> > > + * Optionally \\a controls can be specified to set some or all of the\n> > > handled\n> > > + * V4L2 controls prior to reading them back. If no controls needs to\n> be\n> > > set\n> > > + * nullptr may be used.\n> > > + */\n> > > +void DelayedControls::reset(ControlList *controls)\n> > > +{\n> > > + std::lock_guard lock(lock_);\n> > > +\n> > > + running_ = false;\n> > > + fistSequence_ = 0;\n> > > + queueCount_ = 0;\n> > > + writeCount_ = 0;\n> > > +\n> > > + /* Set the controls on the device if requested. */\n> > > + if (controls)\n> > > + device_->setControls(controls);\n> > >\n> >\n> > Should this happen here in reset()? The reset method to me indicates\n> that\n> > some internal state of the class is meant to be reset, and it does\n> nothing\n> > to send controls to the device. Writing to the device could happen from\n> > the caller, if they desire.\n>\n> Maybe this should be renamed to init() and the argument to\n> initialControls? As this function should only be called when we are not\n> streaming I see no harm in optionally allowing a init value. But maybe\n> this goes hand in had with your follow up comment below.\n>\n\n> >\n> >\n> > > +\n> > > + /* Retrieve current control values reported by the device. */\n> > > + std::vector ids;\n> > > + for (auto const &delay : delays_)\n> > > + ids.push_back(delay.first->id());\n> > > +\n> > > + ControlList devCtrls = device_->getControls(ids);\n> > > +\n> > > + /* Seed the control queue with the controls reported by the\n> > > device. */\n> > > + ctrls_.clear();\n> > > + for (const auto &ctrl : devCtrls) {\n> > > + const ControlId *id =\n> > > devCtrls.infoMap()->idmap().at(ctrl.first);\n> > > + ctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > > + }\n> > > +\n> > > + queueCount_++;\n> > > +}\n> > >\n> >\n> > I'm a bit unclear what reset() does here. In staggered_writer, reset()\n> > would essentially clear the queues and re-initialise with the last value\n> > that was set. This seems to pull the value from the device, and put it\n> > into the queue as a starting point. When we start streaming a sensor,\n> the\n> > settings will be applied without delay. So if we want to intialise the\n> > sensor with a new set of controls, it would not work here. Also, if we\n> > were running a use case where we needed to do a mode switch (e.g. preview\n> > then capture), then the last requested preview controls would not be used\n> > to seed the list, and would be lost.\n>\n> Is this not the same thing but without having to cache it? What ever\n> controls are set on the device when we stop streaming are our starting\n> point. IIRC in StaggerdCtrls the initial value may not have been put to\n> the device but only been at the top of the queue when the streaming\n> stopped.\n\n\nThat is indeed what the StaggeredCtrl does in the reset(). If the IPA had\nqueued up a ctrl to write, that would sit at the end of the delay queue.\nOn a reset() call (perhaps through a mode switch operation), I want to\nwrite that latest given values to the sensor (i.e. the end of the delayed\nqueue), not the last used sensor value. The StaggeredCtrl code would do\nthe former, and the new code above would do the latter correct?\n\n\n> Another advantage of reading it back here is that we will catch\n> any control changes done elsewhere in the pipeline perhaps as part of\n> configure().\n>\n\nI do see that it might be possible for other parts of the system to do\nthis. However, I would argue that it is incorrect. Only one module (i.e.\nthe IPA in our case) should be responsible for arbitrating and setting\nthese ctrl values. Any other modules that want to go and modify, e.g. gain\nand exposure, must go through that one module to avoid such conflicts.\n\nRegards,\nNaush\n\n\n\n>\n> >\n> >\n> > > +\n> > > +/**\n> > > + * \\brief Push a set of controls on the queue\n> > > + * \\param[in] controls List of controls to add the device queue\n> > > + *\n> > > + * Push a set of controls to the control queue. This increases the\n> > > control queue\n> > > + * depth by one.\n> > > + *\n> > > + * \\returns true if \\a controls are accepted, or false otherwise\n> > > + */\n> > > +bool DelayedControls::push(const ControlList &controls)\n> > > +{\n> > > + std::lock_guard lock(lock_);\n> > > +\n> > > + return queue(controls);\n> > > +}\n> > > +\n> > > +bool DelayedControls::queue(const ControlList &controls)\n> > > +{\n> > > + /* Copy state from previous frame. */\n> > > + for (auto &ctrl : ctrls_) {\n> > > + ControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> > > + info.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > > + info.updated = false;\n> > > + }\n> > > +\n> > > + /* Update with new controls. */\n> > > + for (const auto &control : controls) {\n> > > + const ControlId *id =\n> > > device_->controls().idmap().at(control.first);\n> > > +\n> > > + if (delays_.find(id) == delays_.end())\n> > > + return false;\n> > >\n> >\n> > Could you add an error log message here? Do you think this should be a\n> > continue, my inclination is that return false is the right thing?\n>\n> I think this should fail as a control was queued which delay is unknown.\n>\n> >\n> >\n> > > +\n> > > + ControlInfo &info = ctrls_[id][queueCount_];\n> > > +\n> > > + info.value = control.second;\n> > > + info.updated = true;\n> > > +\n> > > + LOG(DelayedControls, Debug)\n> > > + << \"Queuing \" << id->name()\n> > > + << \" to \" << info.value.toString()\n> > > + << \" at index \" << queueCount_;\n> > > + }\n> > > +\n> > > + queueCount_++;\n> > > +\n> > > + return true;\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Read back controls in effect at a specific sequence number\n> > > + * \\param[in] sequence Sequence number to get read back controls for\n> > > + *\n> > > + * Read back what controls where in effect at a specific sequence\n> number.\n> > > The\n> > > + * history is a ring buffer of 16 entries where new and old values\n> > > coexist. It's\n> > > + * the callers responsibility to not read to old sequence numbers that\n> > > have been\n> > > + * pushed out of the history.\n> > > + *\n> > > + * Historic values are evicted by pushing new values onto the queue\n> using\n> > > + * push(). The max history from the current sequence number that\n> yields\n> > > valid\n> > > + * values are thus 16 minus number of controls pushed.\n> > > + *\n> > > + * \\returns List of controls in effect at \\a sequence\n> > > + */\n> > > +ControlList DelayedControls::get(uint32_t sequence)\n> > > +{\n> > > + std::lock_guard lock(lock_);\n> > > +\n> > > + uint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> > > + unsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > >\n> >\n> > Probably my misunderstanding, but will firstSequence ever be anything\n> but 0?\n>\n> Unfortunately yes. Some V4L2 drivers does not reset their sequence\n> number to 0 when starting to stream. So reputing start/stop may have the\n> first frame seq+1 from the previous stop.\n>\n> >\n> >\n> > > +\n> > > + ControlList out(device_->controls());\n> > > + for (const auto &ctrl : ctrls_) {\n> > > + const ControlId *id = ctrl.first;\n> > > + const ControlInfo &info = ctrl.second[index];\n> > > +\n> > > + out.set(id->id(), info.value);\n> > > +\n> > > + LOG(DelayedControls, Debug)\n> > > + << \"Reading \" << id->name()\n> > > + << \" to \" << info.value.toString()\n> > > + << \" at index \" << index;\n> > > + }\n> > > +\n> > > + return out;\n> > > +}\n> > > +\n> > > +/**\n> > > + * \\brief Inform DelayedControls of a start of a new frame\n> > > + * \\param[in] sequence Sequence number of the frame that started\n> > > + *\n> > > + * Inform the state machine that a new frame have started and it's\n> > > sequence\n> > > + * number. It's user of this helpers responsibility to inform the\n> helper\n> > > + * at the start of every frame. This can with ease be connected to the\n> > > start\n> > > + * of exposure (SOE) V4L2 event.\n> > > + */\n> > > +void DelayedControls::frameStart(uint32_t sequence)\n> > > +{\n> > > + LOG(DelayedControls, Debug) << \"frame \" << sequence << \"\n> started\";\n> > > +\n> > > + std::lock_guard lock(lock_);\n> > > +\n> > > + if (!running_) {\n> > > + fistSequence_ = sequence;\n> > > + running_ = true;\n> > > + }\n> > >\n> >\n> > As above, can firstSequence_ be anything but 0?\n> >\n> >\n> > > +\n> > > + /*\n> > > + * Create control list peaking ahead in the value queue to\n> ensure\n> > > + * values are set in time to satisfy the sensor delay.\n> > > + */\n> > > + ControlList out(device_->controls());\n> > > + for (const auto &ctrl : ctrls_) {\n> > > + const ControlId *id = ctrl.first;\n> > > + unsigned int delayDiff = maxDelay_ - delays_[id];\n> > > + unsigned int index = std::max(0, writeCount_ -\n> > > delayDiff);\n> > > + const ControlInfo &info = ctrl.second[index];\n> > > +\n> > > + if (info.updated) {\n> > > + out.set(id->id(), info.value);\n> > > + LOG(DelayedControls, Debug)\n> > > + << \"Setting \" << id->name()\n> > > + << \" to \" << info.value.toString()\n> > > + << \" at index \" << index;\n> > > + }\n> > > + }\n> > > +\n> > > + writeCount_++;\n> > > +\n> > > + while (writeCount_ >= queueCount_) {\n> > > + LOG(DelayedControls, Debug)\n> > > + << \"Queue is empty, auto queue no-op.\";\n> > > + queue({});\n> > > + }\n> > > +\n> > > + device_->setControls(&out);\n> > > +}\n> > > +\n> > > +} /* namespace libcamera */\n> > > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > > --- a/src/libcamera/meson.build\n> > > +++ b/src/libcamera/meson.build\n> > > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > > 'controls.cpp',\n> > > 'control_serializer.cpp',\n> > > 'control_validator.cpp',\n> > > + 'delayed_controls.cpp',\n> > > 'device_enumerator.cpp',\n> > > 'device_enumerator_sysfs.cpp',\n> > > 'event_dispatcher.cpp',\n> > > --\n> > > 2.29.1\n> > >\n> > >\n>\n> --\n> Regards,\n> Niklas Söderlund\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3DE60BE082\n\tfor ;\n\tTue, 10 Nov 2020 15:46:06 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AE318630C0;\n\tTue, 10 Nov 2020 16:46:05 +0100 (CET)","from mail-lj1-x233.google.com (mail-lj1-x233.google.com\n\t[IPv6:2a00:1450:4864:20::233])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 16D0560342\n\tfor ;\n\tTue, 10 Nov 2020 16:46:04 +0100 (CET)","by mail-lj1-x233.google.com with SMTP id v20so6701325ljk.8\n\tfor ;\n\tTue, 10 Nov 2020 07:46:04 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"jGY6Q6ay\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=SuDCVxeyvH6GI3nKUE8x6VRC+iu+GU7OeiIH20IobxY=;\n\tb=jGY6Q6ay27chUKz6KO+thF3zru4r/5cXezv2UcbkN9LDXygrrcDitvPUvVG8zjo1Gc\n\t7kNGlEyyWfcrEHDE2ZF/AcYW4Jj+n11xKZof4Aj9lLJvOAJmy07cAqUVINwBN0Fyd25j\n\tGCsY+Kq3v90/9DYbiSOkgAg88SIfYbJW8+uJLkoaOVQuq3csZlNXsGayJLiVJphOVKEy\n\t7WqShRlXGg4SA7Bd+E/HWZDg/WwCHFuiXqMfiS0ww4a3gkvOCs/JMRdiUg0G+5gN6FQi\n\tReoAVx+I3nezKtrDFW1fdbd3QhIlzr2oZVg+uC9hBjmpzZYcrksRQTvgTDjLzRig9m0G\n\tSdOA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=SuDCVxeyvH6GI3nKUE8x6VRC+iu+GU7OeiIH20IobxY=;\n\tb=an7eIJQQ99uHxZyfu8CWeYJ7u2dU+5qBXIuwkadFPATIA1LAjwBGXpKtLc1E65z34D\n\tvZaztpc2kYa8jsej7uPOEQwmy2Kzl1nmXReFVL5yiBaBFgrDG712tm3lg3z25Jz+fMN2\n\thahy9WAevbb38Qd+ZAvMCKUffgwWQ+Xxf26+5UrCGVyERx3pKfvdu8Q2j3NMLIb2vlMy\n\tq3y1rFi2Mwwu6ejJdLoBVZpV2aHPUPy9Itd0763rbTIKvd899eOV5xvS3WeJyiQqLKV8\n\tL+1ZffB7avNOafQA803VZiC94RCGhgl6kar69M7oR7FQWlgygoRazJcKtCEmkojQjb6T\n\tKqZA==","X-Gm-Message-State":"AOAM530BX9kkBTa5lHeZbIlh9tVzuA9WhzZEf6RItCYsEUDLEeOunXmK\n\tqT+6txQRtyO+BRLCyaGRpL6ryWQUNww2dL8+2iI/vw==","X-Google-Smtp-Source":"ABdhPJw+nuZZfhI4l6t6brU+0VHZoLFAG2vJWtIzBID0jgAIheYncIEJv6HJzzEePFa2RlVgshJDsM2X8j2Y7UaItLs=","X-Received":"by 2002:a2e:b6c6:: with SMTP id m6mr1631742ljo.83.1605023161747; \n\tTue, 10 Nov 2020 07:46:01 -0800 (PST)","MIME-Version":"1.0","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t\n\t<20201109225424.GE88486@oden.dyn.berto.se>","In-Reply-To":"<20201109225424.GE88486@oden.dyn.berto.se>","From":"Naushir Patuck ","Date":"Tue, 10 Nov 2020 15:45:45 +0000","Message-ID":"","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/mixed;\n\tboundary=\"===============4803575882938569093==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13833,"web_url":"https://patchwork.libcamera.org/comment/13833/","msgid":"<20201123170638.GE1773213@oden.dyn.berto.se>","date":"2020-11-23T17:06:38","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Naush,\n\nOn 2020-11-10 15:45:45 +0000, Naushir Patuck wrote:\n> Hi Niklas,\n> \n> \n> On Mon, 9 Nov 2020 at 22:54, Niklas Söderlund \n> wrote:\n> \n> > Hi Naush,\n> >\n> > Thanks for your feedback.\n> >\n> > On 2020-11-03 10:27:06 +0000, Naushir Patuck wrote:\n> > > Hi Niklas,\n> > >\n> > > Thank you for your patch. Please see some initial comments below.\n> > >\n> > > Best regards,\n> > > Naush\n> > >\n> > >\n> > > On Wed, 28 Oct 2020 at 01:01, Niklas Söderlund <\n> > > niklas.soderlund@ragnatech.se> wrote:\n> > >\n> > > > Some sensor controls take effect with a delay as the sensor needs time\n> > > > to adjust, for example exposure. Add a optional helper DelayedControls\n> > > > to help pipelines deal with such controls.\n> > > >\n> > > > The idea is to provide a queue of controls towards the V4L2 device and\n> > > > apply individual controls with the specified delay with the aim to get\n> > > > predictable and retrievable control values for any given frame. To do\n> > > > this the queue of controls needs to be at least as deep as the control\n> > > > with the largest delay.\n> > > >\n> > > > The DelayedControls needs to be informed of every start of exposure.\n> > > > This can be emulated but the helper is designed to be used with this\n> > > > event being provide by the kernel thru V4L2 events.\n> > > >\n> > > > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline\n> > > > handler but expands on its API. This helpers aims to replace the\n> > > > Raspberry Pi implementations and mimics it behavior perfectly.\n> > > >\n> > > > Signed-off-by: Niklas Söderlund \n> > > > ---\n> > > > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > > > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++\n> > > > src/libcamera/meson.build | 1 +\n> > > > 3 files changed, 370 insertions(+)\n> > > > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > > > create mode 100644 src/libcamera/delayed_controls.cpp\n> > > >\n> > > > diff --git a/include/libcamera/internal/delayed_controls.h\n> > > > b/include/libcamera/internal/delayed_controls.h\n> > > > new file mode 100644\n> > > > index 0000000000000000..df5520d240a54e4b\n> > > > --- /dev/null\n> > > > +++ b/include/libcamera/internal/delayed_controls.h\n> > > > @@ -0,0 +1,87 @@\n> > > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > > +/*\n> > > > + * Copyright (C) 2020, Google Inc.\n> > > > + *\n> > > > + * delayed_controls.h - Helper to deal with controls that are applied\n> > > > with a delay\n> > > > + */\n> > > >\n> > >\n> > > Given this is heavily derived from the staggered_write work, would it be\n> > ok\n> > > to keep the Raspberry Pi copyright in the header?\n> > >\n> > > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n> >\n> > As discussed of-band this was always my intention, my templates got in\n> > the way. Sorry about that and thanks for pointing it out.\n> >\n> > >\n> > >\n> > > > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > > +\n> > > > +#include \n> > > > +#include \n> > > > +#include \n> > > > +\n> > > > +#include \n> > > > +\n> > > > +namespace libcamera {\n> > > > +\n> > > > +class V4L2Device;\n> > > > +\n> > > > +class DelayedControls\n> > > > +{\n> > > > +public:\n> > > > + DelayedControls(V4L2Device *device,\n> > > > + const std::unordered_map > int>\n> > > > &delays);\n> > > > +\n> > > > + void reset(ControlList *controls = nullptr);\n> > > > +\n> > > > + bool push(const ControlList &controls);\n> > > > + ControlList get(uint32_t sequence);\n> > > > +\n> > > > + void frameStart(uint32_t sequence);\n> > > > +\n> > > > +private:\n> > > > + class ControlInfo\n> > > > + {\n> > > > + public:\n> > > > + ControlInfo()\n> > > > + : updated(false)\n> > > > + {\n> > > > + }\n> > > > +\n> > > > + ControlInfo(const ControlValue &v)\n> > > > + : value(v), updated(true)\n> > > > + {\n> > > > + }\n> > > > +\n> > > > + ControlValue value;\n> > > > + bool updated;\n> > > > + };\n> > > > +\n> > > > + static constexpr int listSize = 16;\n> > > > + class ControlArray : public std::array\n> > > > + {\n> > > > + public:\n> > > > + ControlInfo &operator[](unsigned int index)\n> > > > + {\n> > > > + return std::array > > > listSize>::operator[](index % listSize);\n> > > > + }\n> > > > +\n> > > > + const ControlInfo &operator[](unsigned int index) const\n> > > > + {\n> > > > + return std::array > > > listSize>::operator[](index % listSize);\n> > > > + }\n> > > > + };\n> > > > +\n> > > > + using ControlsDelays = std::unordered_map > > > unsigned int>;\n> > > > + using ControlsValues = std::unordered_map > > > ControlArray>;\n> > > > +\n> > > > + bool queue(const ControlList &controls);\n> > > > +\n> > > > + std::mutex lock_;\n> > > > +\n> > > > + V4L2Device *device_;\n> > > > + ControlsDelays delays_;\n> > > > + unsigned int maxDelay_;\n> > > > +\n> > > > + bool running_;\n> > > > + uint32_t fistSequence_;\n> > > > +\n> > > > + uint32_t queueCount_;\n> > > > + uint32_t writeCount_;\n> > > > + ControlsValues ctrls_;\n> > > > +};\n> > > > +\n> > > > +} /* namespace libcamera */\n> > > > +\n> > > > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > > > diff --git a/src/libcamera/delayed_controls.cpp\n> > > > b/src/libcamera/delayed_controls.cpp\n> > > > new file mode 100644\n> > > > index 0000000000000000..0e32f417c5cc68b7\n> > > > --- /dev/null\n> > > > +++ b/src/libcamera/delayed_controls.cpp\n> > > > @@ -0,0 +1,282 @@\n> > > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > > +/*\n> > > > + * Copyright (C) 2020, Google Inc.\n> > > > + *\n> > > > + * delayed_controls.h - Helper to deal with controls that are applied\n> > > > with a delay\n> > > > + */\n> > > >\n> > >\n> > > Similar to above:\n> > > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n> > >\n> > >\n> > > > +\n> > > > +#include \"libcamera/internal/delayed_controls.h\"\n> > > > +#include \"libcamera/internal/v4l2_device.h\"\n> > > > +\n> > > > +#include \n> > > > +\n> > > > +#include \"libcamera/internal/log.h\"\n> > > > +\n> > > > +/**\n> > > > + * \\file delayed_controls.h\n> > > > + * \\brief Helper to deal with controls that are applied with a delay\n> > > > + */\n> > > > +\n> > > > +namespace libcamera {\n> > > > +\n> > > > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > > > +\n> > > > +/**\n> > > > + * \\class DelayedControls\n> > > > + * \\brief Helper to deal with controls that takes effect with a delay\n> > > > + *\n> > > > + * Some sensor controls take effect with a delay as the sensor needs\n> > time\n> > > > to\n> > > > + * adjust, for example exposure and focus. This is an optional helper\n> > > > class to\n> > > > + * deal with such controls and the intended user is pipeline handlers.\n> > > > + *\n> > > > + * The idea is to extend the concept of the pipeline depth the users\n> > > > needs to\n> > > > + * maintain for buffers to controls. The depth is determined with by\n> > the\n> > > > control\n> > > > + * with the grates delay. As long as the pipeline keeps the control\n> > queue\n> > > > above\n> > > > + * this level the control values are guaranteed to be in effect at the\n> > > > specified\n> > > > + * point in time.\n> > > > + *\n> > > > + * The downside is of course that the pipeline needs to know what\n> > > > controls to\n> > > > + * set control depth frames in advance. But there really is no way\n> > around\n> > > > this\n> > > > + * as the delay is a consequence of the physical world. Compare this\n> > with\n> > > > + * parameter buffers where the buffer may be queued to the device\n> > while\n> > > > it's\n> > > > + * still being written to as long as it's ready before it's consumed,\n> > > > this is\n> > > > + * because the parameter buffer (usually) does not contain controls\n> > that\n> > > > + * requires time to take effect.\n> > > > + */\n> > > > +\n> > > > +\n> > > > +/**\n> > > > + * \\brief Construct a DelayedControls\n> > > > + * \\param[in] device The V4L2 device containing the delayed controls\n> > > > + * \\param[in] delays Map of numerical V4L2 control id to its delay to\n> > take\n> > > > + * effect in frames\n> > > > + *\n> > > > + * Only controls specified in \\a delays are handled by the\n> > DelayedControls\n> > > > + * instance. If it's desired to mix delayed controls and controls that\n> > > > takes\n> > > > + * effect immediately the immediate controls must be listed in the \\a\n> > > > delays map\n> > > > + * with a delay value of 0.\n> > > > + */\n> > > > +DelayedControls::DelayedControls(V4L2Device *device,\n> > > > + const std::unordered_map > > > unsigned int> &delays)\n> > > > + : device_(device), maxDelay_(0)\n> > > > +{\n> > > > + const ControlInfoMap &controls = device_->controls();\n> > > > +\n> > > > + /*\n> > > > + * Sanity check that all controls where delays are requested\n> > are\n> > > > + * exposed byt the device.\n> > > > + */\n> > > > + for (auto const &delay : delays) {\n> > > > + unsigned int id = delay.first;\n> > > > +\n> > > > + if (controls.find(id) == controls.end())\n> > > > + LOG(DelayedControls, Error)\n> > > > + << \"Delay request for control id \"\n> > > > + << utils::hex(id)\n> > > > + << \" but control is not exposed by\n> > device \"\n> > > > + << device_->deviceNode();\n> > > > + }\n> > > > +\n> > > > + /*\n> > > > + * Create a map of control to delay for all controls exposed\n> > by the\n> > > > + * device. If no delay is specified assume the control applies\n> > > > directly.\n> > > > + */\n> > > > + for (auto const &control : controls) {\n> > > > + const ControlId *id = control.first;\n> > > > +\n> > > > + auto it = delays.find(id->id());\n> > > > + if (it == delays.end())\n> > > > + continue;\n> > > > +\n> > > > + delays_[id] = it->second;\n> > > > +\n> > > > + LOG(DelayedControls, Debug)\n> > > > + << \"Set a delay of \" << delays_[id]\n> > > > + << \" for \" << id->name();\n> > > > +\n> > > > + maxDelay_ = std::max(maxDelay_, delays_[id]);\n> > > > + }\n> > > >\n> > >\n> > > Could you combine the two loops here into one here? If you iterate over\n> > > \"delays\", you could do the error check as well as adding to the \"delays_\"\n> > > map.\n> >\n> > Good call!\n> >\n> > >\n> > > +\n> > > > + reset();\n> > > >\n> > >\n> > > Does this need to be called here? But my understanding of reset() may be\n> > > wrong... see below.\n> >\n> > It does as reset() initialize the internal state machine.\n> >\n> > >\n> > >\n> > > > +}\n> > > > +\n> > > > +/**\n> > > > + * \\brief Reset the V4L2 device\n> > > > + * \\param[in] controls List of controls to reset the device to or\n> > nullptr\n> > > > + *\n> > > > + * Resets the delayed controls state machine to its starting state.\n> > All\n> > > > controls\n> > > > + * are fetched from the V4L2 device to provide a good starting point\n> > for\n> > > > the\n> > > > + * first frames (length of control depth).\n> > > > + *\n> > > > + * Optionally \\a controls can be specified to set some or all of the\n> > > > handled\n> > > > + * V4L2 controls prior to reading them back. If no controls needs to\n> > be\n> > > > set\n> > > > + * nullptr may be used.\n> > > > + */\n> > > > +void DelayedControls::reset(ControlList *controls)\n> > > > +{\n> > > > + std::lock_guard lock(lock_);\n> > > > +\n> > > > + running_ = false;\n> > > > + fistSequence_ = 0;\n> > > > + queueCount_ = 0;\n> > > > + writeCount_ = 0;\n> > > > +\n> > > > + /* Set the controls on the device if requested. */\n> > > > + if (controls)\n> > > > + device_->setControls(controls);\n> > > >\n> > >\n> > > Should this happen here in reset()? The reset method to me indicates\n> > that\n> > > some internal state of the class is meant to be reset, and it does\n> > nothing\n> > > to send controls to the device. Writing to the device could happen from\n> > > the caller, if they desire.\n> >\n> > Maybe this should be renamed to init() and the argument to\n> > initialControls? As this function should only be called when we are not\n> > streaming I see no harm in optionally allowing a init value. But maybe\n> > this goes hand in had with your follow up comment below.\n> >\n> \n> > >\n> > >\n> > > > +\n> > > > + /* Retrieve current control values reported by the device. */\n> > > > + std::vector ids;\n> > > > + for (auto const &delay : delays_)\n> > > > + ids.push_back(delay.first->id());\n> > > > +\n> > > > + ControlList devCtrls = device_->getControls(ids);\n> > > > +\n> > > > + /* Seed the control queue with the controls reported by the\n> > > > device. */\n> > > > + ctrls_.clear();\n> > > > + for (const auto &ctrl : devCtrls) {\n> > > > + const ControlId *id =\n> > > > devCtrls.infoMap()->idmap().at(ctrl.first);\n> > > > + ctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > > > + }\n> > > > +\n> > > > + queueCount_++;\n> > > > +}\n> > > >\n> > >\n> > > I'm a bit unclear what reset() does here. In staggered_writer, reset()\n> > > would essentially clear the queues and re-initialise with the last value\n> > > that was set. This seems to pull the value from the device, and put it\n> > > into the queue as a starting point. When we start streaming a sensor,\n> > the\n> > > settings will be applied without delay. So if we want to intialise the\n> > > sensor with a new set of controls, it would not work here. Also, if we\n> > > were running a use case where we needed to do a mode switch (e.g. preview\n> > > then capture), then the last requested preview controls would not be used\n> > > to seed the list, and would be lost.\n> >\n> > Is this not the same thing but without having to cache it? What ever\n> > controls are set on the device when we stop streaming are our starting\n> > point. IIRC in StaggerdCtrls the initial value may not have been put to\n> > the device but only been at the top of the queue when the streaming\n> > stopped.\n> \n> \n> That is indeed what the StaggeredCtrl does in the reset(). If the IPA had\n> queued up a ctrl to write, that would sit at the end of the delay queue.\n> On a reset() call (perhaps through a mode switch operation), I want to\n> write that latest given values to the sensor (i.e. the end of the delayed\n> queue), not the last used sensor value. The StaggeredCtrl code would do\n> the former, and the new code above would do the latter correct?\n\nI understand your point, but is the behavior you outline best described \nas reset(). As we do not yet support mode change without a complete stop \nand start would it make sens to defer this to that point in time? I'm \nthinking depending on how that is done maybe DelayedControls needs to be \nextended with a flush() operation that perhaps closer matches the above? \n\n> \n> \n> > Another advantage of reading it back here is that we will catch\n> > any control changes done elsewhere in the pipeline perhaps as part of\n> > configure().\n> >\n> \n> I do see that it might be possible for other parts of the system to do\n> this. However, I would argue that it is incorrect. Only one module (i.e.\n> the IPA in our case) should be responsible for arbitrating and setting\n> these ctrl values. Any other modules that want to go and modify, e.g. gain\n> and exposure, must go through that one module to avoid such conflicts.\n\nEven if it's only a single component feeding the statemachine is it not \nreally useful to have a reset() operation that can fore it into a known \nstate? For this reason I opted to give the reset() an optimal argument \nthat sets the state on the device.\n\n> \n> Regards,\n> Naush\n> \n> \n> \n> >\n> > >\n> > >\n> > > > +\n> > > > +/**\n> > > > + * \\brief Push a set of controls on the queue\n> > > > + * \\param[in] controls List of controls to add the device queue\n> > > > + *\n> > > > + * Push a set of controls to the control queue. This increases the\n> > > > control queue\n> > > > + * depth by one.\n> > > > + *\n> > > > + * \\returns true if \\a controls are accepted, or false otherwise\n> > > > + */\n> > > > +bool DelayedControls::push(const ControlList &controls)\n> > > > +{\n> > > > + std::lock_guard lock(lock_);\n> > > > +\n> > > > + return queue(controls);\n> > > > +}\n> > > > +\n> > > > +bool DelayedControls::queue(const ControlList &controls)\n> > > > +{\n> > > > + /* Copy state from previous frame. */\n> > > > + for (auto &ctrl : ctrls_) {\n> > > > + ControlInfo &info = ctrls_[ctrl.first][queueCount_];\n> > > > + info.value = ctrls_[ctrl.first][queueCount_ - 1].value;\n> > > > + info.updated = false;\n> > > > + }\n> > > > +\n> > > > + /* Update with new controls. */\n> > > > + for (const auto &control : controls) {\n> > > > + const ControlId *id =\n> > > > device_->controls().idmap().at(control.first);\n> > > > +\n> > > > + if (delays_.find(id) == delays_.end())\n> > > > + return false;\n> > > >\n> > >\n> > > Could you add an error log message here? Do you think this should be a\n> > > continue, my inclination is that return false is the right thing?\n> >\n> > I think this should fail as a control was queued which delay is unknown.\n> >\n> > >\n> > >\n> > > > +\n> > > > + ControlInfo &info = ctrls_[id][queueCount_];\n> > > > +\n> > > > + info.value = control.second;\n> > > > + info.updated = true;\n> > > > +\n> > > > + LOG(DelayedControls, Debug)\n> > > > + << \"Queuing \" << id->name()\n> > > > + << \" to \" << info.value.toString()\n> > > > + << \" at index \" << queueCount_;\n> > > > + }\n> > > > +\n> > > > + queueCount_++;\n> > > > +\n> > > > + return true;\n> > > > +}\n> > > > +\n> > > > +/**\n> > > > + * \\brief Read back controls in effect at a specific sequence number\n> > > > + * \\param[in] sequence Sequence number to get read back controls for\n> > > > + *\n> > > > + * Read back what controls where in effect at a specific sequence\n> > number.\n> > > > The\n> > > > + * history is a ring buffer of 16 entries where new and old values\n> > > > coexist. It's\n> > > > + * the callers responsibility to not read to old sequence numbers that\n> > > > have been\n> > > > + * pushed out of the history.\n> > > > + *\n> > > > + * Historic values are evicted by pushing new values onto the queue\n> > using\n> > > > + * push(). The max history from the current sequence number that\n> > yields\n> > > > valid\n> > > > + * values are thus 16 minus number of controls pushed.\n> > > > + *\n> > > > + * \\returns List of controls in effect at \\a sequence\n> > > > + */\n> > > > +ControlList DelayedControls::get(uint32_t sequence)\n> > > > +{\n> > > > + std::lock_guard lock(lock_);\n> > > > +\n> > > > + uint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> > > > + unsigned int index = std::max(0, adjustedSeq - maxDelay_);\n> > > >\n> > >\n> > > Probably my misunderstanding, but will firstSequence ever be anything\n> > but 0?\n> >\n> > Unfortunately yes. Some V4L2 drivers does not reset their sequence\n> > number to 0 when starting to stream. So reputing start/stop may have the\n> > first frame seq+1 from the previous stop.\n> >\n> > >\n> > >\n> > > > +\n> > > > + ControlList out(device_->controls());\n> > > > + for (const auto &ctrl : ctrls_) {\n> > > > + const ControlId *id = ctrl.first;\n> > > > + const ControlInfo &info = ctrl.second[index];\n> > > > +\n> > > > + out.set(id->id(), info.value);\n> > > > +\n> > > > + LOG(DelayedControls, Debug)\n> > > > + << \"Reading \" << id->name()\n> > > > + << \" to \" << info.value.toString()\n> > > > + << \" at index \" << index;\n> > > > + }\n> > > > +\n> > > > + return out;\n> > > > +}\n> > > > +\n> > > > +/**\n> > > > + * \\brief Inform DelayedControls of a start of a new frame\n> > > > + * \\param[in] sequence Sequence number of the frame that started\n> > > > + *\n> > > > + * Inform the state machine that a new frame have started and it's\n> > > > sequence\n> > > > + * number. It's user of this helpers responsibility to inform the\n> > helper\n> > > > + * at the start of every frame. This can with ease be connected to the\n> > > > start\n> > > > + * of exposure (SOE) V4L2 event.\n> > > > + */\n> > > > +void DelayedControls::frameStart(uint32_t sequence)\n> > > > +{\n> > > > + LOG(DelayedControls, Debug) << \"frame \" << sequence << \"\n> > started\";\n> > > > +\n> > > > + std::lock_guard lock(lock_);\n> > > > +\n> > > > + if (!running_) {\n> > > > + fistSequence_ = sequence;\n> > > > + running_ = true;\n> > > > + }\n> > > >\n> > >\n> > > As above, can firstSequence_ be anything but 0?\n> > >\n> > >\n> > > > +\n> > > > + /*\n> > > > + * Create control list peaking ahead in the value queue to\n> > ensure\n> > > > + * values are set in time to satisfy the sensor delay.\n> > > > + */\n> > > > + ControlList out(device_->controls());\n> > > > + for (const auto &ctrl : ctrls_) {\n> > > > + const ControlId *id = ctrl.first;\n> > > > + unsigned int delayDiff = maxDelay_ - delays_[id];\n> > > > + unsigned int index = std::max(0, writeCount_ -\n> > > > delayDiff);\n> > > > + const ControlInfo &info = ctrl.second[index];\n> > > > +\n> > > > + if (info.updated) {\n> > > > + out.set(id->id(), info.value);\n> > > > + LOG(DelayedControls, Debug)\n> > > > + << \"Setting \" << id->name()\n> > > > + << \" to \" << info.value.toString()\n> > > > + << \" at index \" << index;\n> > > > + }\n> > > > + }\n> > > > +\n> > > > + writeCount_++;\n> > > > +\n> > > > + while (writeCount_ >= queueCount_) {\n> > > > + LOG(DelayedControls, Debug)\n> > > > + << \"Queue is empty, auto queue no-op.\";\n> > > > + queue({});\n> > > > + }\n> > > > +\n> > > > + device_->setControls(&out);\n> > > > +}\n> > > > +\n> > > > +} /* namespace libcamera */\n> > > > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > > > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > > > --- a/src/libcamera/meson.build\n> > > > +++ b/src/libcamera/meson.build\n> > > > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > > > 'controls.cpp',\n> > > > 'control_serializer.cpp',\n> > > > 'control_validator.cpp',\n> > > > + 'delayed_controls.cpp',\n> > > > 'device_enumerator.cpp',\n> > > > 'device_enumerator_sysfs.cpp',\n> > > > 'event_dispatcher.cpp',\n> > > > --\n> > > > 2.29.1\n> > > >\n> > > >\n> >\n> > --\n> > Regards,\n> > Niklas Söderlund\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4B8FEBE176\n\tfor ;\n\tMon, 23 Nov 2020 17:06:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AC190633F4;\n\tMon, 23 Nov 2020 18:06:41 +0100 (CET)","from mail-lj1-x233.google.com (mail-lj1-x233.google.com\n\t[IPv6:2a00:1450:4864:20::233])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C15BF631D0\n\tfor ;\n\tMon, 23 Nov 2020 18:06:40 +0100 (CET)","by mail-lj1-x233.google.com with SMTP id f18so831402ljg.9\n\tfor ;\n\tMon, 23 Nov 2020 09:06:40 -0800 (PST)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\te10sm1448206lfn.115.2020.11.23.09.06.38\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tMon, 23 Nov 2020 09:06:39 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"QaBGaIV/\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=ej0Xgw/xlB6LnHMkd6n0r8vi0Ap1G8u8/oYD7r8fBQA=;\n\tb=QaBGaIV//37woE/X/NgqnuIkumDCVc7P+jko63kevLD+FWPHKzMys9PvTR08i8FPg+\n\th0CtWWS5OGf1guW1OYSwX+H/RtesB/6JAde+2+dVE+nPJDvQgWS5xOVjT0ndRcougmzQ\n\tR7FcGcIBaT40z/ge6/e+1+sd4W27sJ12bfGER3h4RM+dD3iM/s4RqSWJmjLWSq97mk9l\n\tIc4ZkTMFkfG9wwXVx+8/snm5UH9W+IeT2YpakJqlRtRIiMOaj0lVhuamVW3lZpWT28gt\n\tA7wMGYN2tdtPd2ewlsVomsXA5YOOTyxishKYehBaY2COE3ArKgog0FuLxjwiFlOcmfAp\n\tbxtA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=ej0Xgw/xlB6LnHMkd6n0r8vi0Ap1G8u8/oYD7r8fBQA=;\n\tb=ket6aS2kuNvicXzVgocOZxo6F/7oHoLAQKJqdnf9lCggPMK49b8cOMucWYfJ/4l/ZS\n\tsSf7xkZgXSLNHxvUV+jmEBWyXYLoPJKu9L+qDWa+Q1aF5h9JXUvhMkcWHygBIEPl0OFs\n\tyrMWXsS33vinaAvgRRsxoNWeqGu5AobIsk+/YA/mzzZPMvcp5tolRQG52Aot0HqVTARb\n\teuX/DQImWuy/nMnlv1S1IPvW0CvZg6DSF3YsIPn51o4cNOfNoe7RN7pE4oe3yDk+nRv2\n\tDMvTWKvQU7w5RWZedWxaNcnfEfoW3wMTMlKENx0yOBLwWM6obBG8AbiLI9uhFwamifOI\n\tJmMQ==","X-Gm-Message-State":"AOAM531gKRzhfPkdKnUq3DXu1E0LxqcNEelY3fpBhAm4k44QvbjoVamE\n\txjsBHeqVI3TR0rVg3ManwVtaEQ==","X-Google-Smtp-Source":"ABdhPJyNWZxud4dxTfnGLRFswlNl/1RmKlHZAm1JPa5o/D9fa6+z2SRIhGZEtLZ2cdgnIlDEJ4P0Ag==","X-Received":"by 2002:a2e:b017:: with SMTP id y23mr129689ljk.361.1606151199848;\n\tMon, 23 Nov 2020 09:06:39 -0800 (PST)","Date":"Mon, 23 Nov 2020 18:06:38 +0100","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Naushir Patuck ","Message-ID":"<20201123170638.GE1773213@oden.dyn.berto.se>","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t\n\t<20201109225424.GE88486@oden.dyn.berto.se>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13841,"web_url":"https://patchwork.libcamera.org/comment/13841/","msgid":"","date":"2020-11-24T09:39:36","subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Niklas,\n\nOn Mon, 23 Nov 2020 at 17:06, Niklas Söderlund <\nniklas.soderlund@ragnatech.se> wrote:\n\n> Hi Naush,\n>\n> On 2020-11-10 15:45:45 +0000, Naushir Patuck wrote:\n> > Hi Niklas,\n> >\n> >\n> > On Mon, 9 Nov 2020 at 22:54, Niklas Söderlund <\n> niklas.soderlund@ragnatech.se>\n> > wrote:\n> >\n> > > Hi Naush,\n> > >\n> > > Thanks for your feedback.\n> > >\n> > > On 2020-11-03 10:27:06 +0000, Naushir Patuck wrote:\n> > > > Hi Niklas,\n> > > >\n> > > > Thank you for your patch. Please see some initial comments below.\n> > > >\n> > > > Best regards,\n> > > > Naush\n> > > >\n> > > >\n> > > > On Wed, 28 Oct 2020 at 01:01, Niklas Söderlund <\n> > > > niklas.soderlund@ragnatech.se> wrote:\n> > > >\n> > > > > Some sensor controls take effect with a delay as the sensor needs\n> time\n> > > > > to adjust, for example exposure. Add a optional helper\n> DelayedControls\n> > > > > to help pipelines deal with such controls.\n> > > > >\n> > > > > The idea is to provide a queue of controls towards the V4L2 device\n> and\n> > > > > apply individual controls with the specified delay with the aim to\n> get\n> > > > > predictable and retrievable control values for any given frame. To\n> do\n> > > > > this the queue of controls needs to be at least as deep as the\n> control\n> > > > > with the largest delay.\n> > > > >\n> > > > > The DelayedControls needs to be informed of every start of\n> exposure.\n> > > > > This can be emulated but the helper is designed to be used with\n> this\n> > > > > event being provide by the kernel thru V4L2 events.\n> > > > >\n> > > > > This helper is based on StaggeredCtrl from the Raspberry Pi\n> pipeline\n> > > > > handler but expands on its API. This helpers aims to replace the\n> > > > > Raspberry Pi implementations and mimics it behavior perfectly.\n> > > > >\n> > > > > Signed-off-by: Niklas Söderlund \n> > > > > ---\n> > > > > include/libcamera/internal/delayed_controls.h | 87 ++++++\n> > > > > src/libcamera/delayed_controls.cpp | 282\n> ++++++++++++++++++\n> > > > > src/libcamera/meson.build | 1 +\n> > > > > 3 files changed, 370 insertions(+)\n> > > > > create mode 100644 include/libcamera/internal/delayed_controls.h\n> > > > > create mode 100644 src/libcamera/delayed_controls.cpp\n> > > > >\n> > > > > diff --git a/include/libcamera/internal/delayed_controls.h\n> > > > > b/include/libcamera/internal/delayed_controls.h\n> > > > > new file mode 100644\n> > > > > index 0000000000000000..df5520d240a54e4b\n> > > > > --- /dev/null\n> > > > > +++ b/include/libcamera/internal/delayed_controls.h\n> > > > > @@ -0,0 +1,87 @@\n> > > > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > > > +/*\n> > > > > + * Copyright (C) 2020, Google Inc.\n> > > > > + *\n> > > > > + * delayed_controls.h - Helper to deal with controls that are\n> applied\n> > > > > with a delay\n> > > > > + */\n> > > > >\n> > > >\n> > > > Given this is heavily derived from the staggered_write work, would\n> it be\n> > > ok\n> > > > to keep the Raspberry Pi copyright in the header?\n> > > >\n> > > > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n> > >\n> > > As discussed of-band this was always my intention, my templates got in\n> > > the way. Sorry about that and thanks for pointing it out.\n> > >\n> > > >\n> > > >\n> > > > > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > > > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__\n> > > > > +\n> > > > > +#include \n> > > > > +#include \n> > > > > +#include \n> > > > > +\n> > > > > +#include \n> > > > > +\n> > > > > +namespace libcamera {\n> > > > > +\n> > > > > +class V4L2Device;\n> > > > > +\n> > > > > +class DelayedControls\n> > > > > +{\n> > > > > +public:\n> > > > > + DelayedControls(V4L2Device *device,\n> > > > > + const std::unordered_map > > int>\n> > > > > &delays);\n> > > > > +\n> > > > > + void reset(ControlList *controls = nullptr);\n> > > > > +\n> > > > > + bool push(const ControlList &controls);\n> > > > > + ControlList get(uint32_t sequence);\n> > > > > +\n> > > > > + void frameStart(uint32_t sequence);\n> > > > > +\n> > > > > +private:\n> > > > > + class ControlInfo\n> > > > > + {\n> > > > > + public:\n> > > > > + ControlInfo()\n> > > > > + : updated(false)\n> > > > > + {\n> > > > > + }\n> > > > > +\n> > > > > + ControlInfo(const ControlValue &v)\n> > > > > + : value(v), updated(true)\n> > > > > + {\n> > > > > + }\n> > > > > +\n> > > > > + ControlValue value;\n> > > > > + bool updated;\n> > > > > + };\n> > > > > +\n> > > > > + static constexpr int listSize = 16;\n> > > > > + class ControlArray : public std::array listSize>\n> > > > > + {\n> > > > > + public:\n> > > > > + ControlInfo &operator[](unsigned int index)\n> > > > > + {\n> > > > > + return std::array > > > > listSize>::operator[](index % listSize);\n> > > > > + }\n> > > > > +\n> > > > > + const ControlInfo &operator[](unsigned int index)\n> const\n> > > > > + {\n> > > > > + return std::array > > > > listSize>::operator[](index % listSize);\n> > > > > + }\n> > > > > + };\n> > > > > +\n> > > > > + using ControlsDelays = std::unordered_map *,\n> > > > > unsigned int>;\n> > > > > + using ControlsValues = std::unordered_map *,\n> > > > > ControlArray>;\n> > > > > +\n> > > > > + bool queue(const ControlList &controls);\n> > > > > +\n> > > > > + std::mutex lock_;\n> > > > > +\n> > > > > + V4L2Device *device_;\n> > > > > + ControlsDelays delays_;\n> > > > > + unsigned int maxDelay_;\n> > > > > +\n> > > > > + bool running_;\n> > > > > + uint32_t fistSequence_;\n> > > > > +\n> > > > > + uint32_t queueCount_;\n> > > > > + uint32_t writeCount_;\n> > > > > + ControlsValues ctrls_;\n> > > > > +};\n> > > > > +\n> > > > > +} /* namespace libcamera */\n> > > > > +\n> > > > > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */\n> > > > > diff --git a/src/libcamera/delayed_controls.cpp\n> > > > > b/src/libcamera/delayed_controls.cpp\n> > > > > new file mode 100644\n> > > > > index 0000000000000000..0e32f417c5cc68b7\n> > > > > --- /dev/null\n> > > > > +++ b/src/libcamera/delayed_controls.cpp\n> > > > > @@ -0,0 +1,282 @@\n> > > > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > > > +/*\n> > > > > + * Copyright (C) 2020, Google Inc.\n> > > > > + *\n> > > > > + * delayed_controls.h - Helper to deal with controls that are\n> applied\n> > > > > with a delay\n> > > > > + */\n> > > > >\n> > > >\n> > > > Similar to above:\n> > > > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.\n> > > >\n> > > >\n> > > > > +\n> > > > > +#include \"libcamera/internal/delayed_controls.h\"\n> > > > > +#include \"libcamera/internal/v4l2_device.h\"\n> > > > > +\n> > > > > +#include \n> > > > > +\n> > > > > +#include \"libcamera/internal/log.h\"\n> > > > > +\n> > > > > +/**\n> > > > > + * \\file delayed_controls.h\n> > > > > + * \\brief Helper to deal with controls that are applied with a\n> delay\n> > > > > + */\n> > > > > +\n> > > > > +namespace libcamera {\n> > > > > +\n> > > > > +LOG_DEFINE_CATEGORY(DelayedControls)\n> > > > > +\n> > > > > +/**\n> > > > > + * \\class DelayedControls\n> > > > > + * \\brief Helper to deal with controls that takes effect with a\n> delay\n> > > > > + *\n> > > > > + * Some sensor controls take effect with a delay as the sensor\n> needs\n> > > time\n> > > > > to\n> > > > > + * adjust, for example exposure and focus. This is an optional\n> helper\n> > > > > class to\n> > > > > + * deal with such controls and the intended user is pipeline\n> handlers.\n> > > > > + *\n> > > > > + * The idea is to extend the concept of the pipeline depth the\n> users\n> > > > > needs to\n> > > > > + * maintain for buffers to controls. The depth is determined with\n> by\n> > > the\n> > > > > control\n> > > > > + * with the grates delay. As long as the pipeline keeps the\n> control\n> > > queue\n> > > > > above\n> > > > > + * this level the control values are guaranteed to be in effect\n> at the\n> > > > > specified\n> > > > > + * point in time.\n> > > > > + *\n> > > > > + * The downside is of course that the pipeline needs to know what\n> > > > > controls to\n> > > > > + * set control depth frames in advance. But there really is no way\n> > > around\n> > > > > this\n> > > > > + * as the delay is a consequence of the physical world. Compare\n> this\n> > > with\n> > > > > + * parameter buffers where the buffer may be queued to the device\n> > > while\n> > > > > it's\n> > > > > + * still being written to as long as it's ready before it's\n> consumed,\n> > > > > this is\n> > > > > + * because the parameter buffer (usually) does not contain\n> controls\n> > > that\n> > > > > + * requires time to take effect.\n> > > > > + */\n> > > > > +\n> > > > > +\n> > > > > +/**\n> > > > > + * \\brief Construct a DelayedControls\n> > > > > + * \\param[in] device The V4L2 device containing the delayed\n> controls\n> > > > > + * \\param[in] delays Map of numerical V4L2 control id to its\n> delay to\n> > > take\n> > > > > + * effect in frames\n> > > > > + *\n> > > > > + * Only controls specified in \\a delays are handled by the\n> > > DelayedControls\n> > > > > + * instance. If it's desired to mix delayed controls and controls\n> that\n> > > > > takes\n> > > > > + * effect immediately the immediate controls must be listed in\n> the \\a\n> > > > > delays map\n> > > > > + * with a delay value of 0.\n> > > > > + */\n> > > > > +DelayedControls::DelayedControls(V4L2Device *device,\n> > > > > + const std::unordered_map > > > > unsigned int> &delays)\n> > > > > + : device_(device), maxDelay_(0)\n> > > > > +{\n> > > > > + const ControlInfoMap &controls = device_->controls();\n> > > > > +\n> > > > > + /*\n> > > > > + * Sanity check that all controls where delays are\n> requested\n> > > are\n> > > > > + * exposed byt the device.\n> > > > > + */\n> > > > > + for (auto const &delay : delays) {\n> > > > > + unsigned int id = delay.first;\n> > > > > +\n> > > > > + if (controls.find(id) == controls.end())\n> > > > > + LOG(DelayedControls, Error)\n> > > > > + << \"Delay request for control id \"\n> > > > > + << utils::hex(id)\n> > > > > + << \" but control is not exposed by\n> > > device \"\n> > > > > + << device_->deviceNode();\n> > > > > + }\n> > > > > +\n> > > > > + /*\n> > > > > + * Create a map of control to delay for all controls\n> exposed\n> > > by the\n> > > > > + * device. If no delay is specified assume the control\n> applies\n> > > > > directly.\n> > > > > + */\n> > > > > + for (auto const &control : controls) {\n> > > > > + const ControlId *id = control.first;\n> > > > > +\n> > > > > + auto it = delays.find(id->id());\n> > > > > + if (it == delays.end())\n> > > > > + continue;\n> > > > > +\n> > > > > + delays_[id] = it->second;\n> > > > > +\n> > > > > + LOG(DelayedControls, Debug)\n> > > > > + << \"Set a delay of \" << delays_[id]\n> > > > > + << \" for \" << id->name();\n> > > > > +\n> > > > > + maxDelay_ = std::max(maxDelay_, delays_[id]);\n> > > > > + }\n> > > > >\n> > > >\n> > > > Could you combine the two loops here into one here? If you iterate\n> over\n> > > > \"delays\", you could do the error check as well as adding to the\n> \"delays_\"\n> > > > map.\n> > >\n> > > Good call!\n> > >\n> > > >\n> > > > +\n> > > > > + reset();\n> > > > >\n> > > >\n> > > > Does this need to be called here? But my understanding of reset()\n> may be\n> > > > wrong... see below.\n> > >\n> > > It does as reset() initialize the internal state machine.\n> > >\n> > > >\n> > > >\n> > > > > +}\n> > > > > +\n> > > > > +/**\n> > > > > + * \\brief Reset the V4L2 device\n> > > > > + * \\param[in] controls List of controls to reset the device to or\n> > > nullptr\n> > > > > + *\n> > > > > + * Resets the delayed controls state machine to its starting\n> state.\n> > > All\n> > > > > controls\n> > > > > + * are fetched from the V4L2 device to provide a good starting\n> point\n> > > for\n> > > > > the\n> > > > > + * first frames (length of control depth).\n> > > > > + *\n> > > > > + * Optionally \\a controls can be specified to set some or all of\n> the\n> > > > > handled\n> > > > > + * V4L2 controls prior to reading them back. If no controls needs\n> to\n> > > be\n> > > > > set\n> > > > > + * nullptr may be used.\n> > > > > + */\n> > > > > +void DelayedControls::reset(ControlList *controls)\n> > > > > +{\n> > > > > + std::lock_guard lock(lock_);\n> > > > > +\n> > > > > + running_ = false;\n> > > > > + fistSequence_ = 0;\n> > > > > + queueCount_ = 0;\n> > > > > + writeCount_ = 0;\n> > > > > +\n> > > > > + /* Set the controls on the device if requested. */\n> > > > > + if (controls)\n> > > > > + device_->setControls(controls);\n> > > > >\n> > > >\n> > > > Should this happen here in reset()? The reset method to me indicates\n> > > that\n> > > > some internal state of the class is meant to be reset, and it does\n> > > nothing\n> > > > to send controls to the device. Writing to the device could happen\n> from\n> > > > the caller, if they desire.\n> > >\n> > > Maybe this should be renamed to init() and the argument to\n> > > initialControls? As this function should only be called when we are not\n> > > streaming I see no harm in optionally allowing a init value. But maybe\n> > > this goes hand in had with your follow up comment below.\n> > >\n> >\n> > > >\n> > > >\n> > > > > +\n> > > > > + /* Retrieve current control values reported by the device.\n> */\n> > > > > + std::vector ids;\n> > > > > + for (auto const &delay : delays_)\n> > > > > + ids.push_back(delay.first->id());\n> > > > > +\n> > > > > + ControlList devCtrls = device_->getControls(ids);\n> > > > > +\n> > > > > + /* Seed the control queue with the controls reported by the\n> > > > > device. */\n> > > > > + ctrls_.clear();\n> > > > > + for (const auto &ctrl : devCtrls) {\n> > > > > + const ControlId *id =\n> > > > > devCtrls.infoMap()->idmap().at(ctrl.first);\n> > > > > + ctrls_[id][queueCount_] = ControlInfo(ctrl.second);\n> > > > > + }\n> > > > > +\n> > > > > + queueCount_++;\n> > > > > +}\n> > > > >\n> > > >\n> > > > I'm a bit unclear what reset() does here. In staggered_writer,\n> reset()\n> > > > would essentially clear the queues and re-initialise with the last\n> value\n> > > > that was set. This seems to pull the value from the device, and\n> put it\n> > > > into the queue as a starting point. When we start streaming a\n> sensor,\n> > > the\n> > > > settings will be applied without delay. So if we want to intialise\n> the\n> > > > sensor with a new set of controls, it would not work here. Also, if\n> we\n> > > > were running a use case where we needed to do a mode switch (e.g.\n> preview\n> > > > then capture), then the last requested preview controls would not be\n> used\n> > > > to seed the list, and would be lost.\n> > >\n> > > Is this not the same thing but without having to cache it? What ever\n> > > controls are set on the device when we stop streaming are our starting\n> > > point. IIRC in StaggerdCtrls the initial value may not have been put to\n> > > the device but only been at the top of the queue when the streaming\n> > > stopped.\n> >\n> >\n> > That is indeed what the StaggeredCtrl does in the reset(). If the IPA\n> had\n> > queued up a ctrl to write, that would sit at the end of the delay queue.\n> > On a reset() call (perhaps through a mode switch operation), I want to\n> > write that latest given values to the sensor (i.e. the end of the delayed\n> > queue), not the last used sensor value. The StaggeredCtrl code would do\n> > the former, and the new code above would do the latter correct?\n>\n> I understand your point, but is the behavior you outline best described\n> as reset(). As we do not yet support mode change without a complete stop\n> and start would it make sens to defer this to that point in time? I'm\n> thinking depending on how that is done maybe DelayedControls needs to be\n> extended with a flush() operation that perhaps closer matches the above?\n>\n>\nUnfortunately, I do not think we can defer this behavior. Our internal\nlibcamera app does use a mode switch by doing a manual stop() for a stills\nimage capture. However, your suggestion of using a new flush() operation\nmight be what we need. So would flush() simply program the sensor with the\nlast provided values in the DelayedCtrl? If so then the pipeline handler\ncan do a flush() followed by a reset() to behave in the same way as the\nSaggeredCtrl::reset() correct?\n\nRegards,\nNaush\n\n>\n> >\n> > > Another advantage of reading it back here is that we will catch\n> > > any control changes done elsewhere in the pipeline perhaps as part of\n> > > configure().\n> > >\n> >\n> > I do see that it might be possible for other parts of the system to do\n> > this. However, I would argue that it is incorrect. Only one module\n> (i.e.\n> > the IPA in our case) should be responsible for arbitrating and setting\n> > these ctrl values. Any other modules that want to go and modify, e.g.\n> gain\n> > and exposure, must go through that one module to avoid such conflicts.\n>\n> Even if it's only a single component feeding the statemachine is it not\n> really useful to have a reset() operation that can fore it into a known\n> state? For this reason I opted to give the reset() an optimal argument\n> that sets the state on the device.\n>\n> >\n> > Regards,\n> > Naush\n> >\n> >\n> >\n> > >\n> > > >\n> > > >\n> > > > > +\n> > > > > +/**\n> > > > > + * \\brief Push a set of controls on the queue\n> > > > > + * \\param[in] controls List of controls to add the device queue\n> > > > > + *\n> > > > > + * Push a set of controls to the control queue. This increases the\n> > > > > control queue\n> > > > > + * depth by one.\n> > > > > + *\n> > > > > + * \\returns true if \\a controls are accepted, or false otherwise\n> > > > > + */\n> > > > > +bool DelayedControls::push(const ControlList &controls)\n> > > > > +{\n> > > > > + std::lock_guard lock(lock_);\n> > > > > +\n> > > > > + return queue(controls);\n> > > > > +}\n> > > > > +\n> > > > > +bool DelayedControls::queue(const ControlList &controls)\n> > > > > +{\n> > > > > + /* Copy state from previous frame. */\n> > > > > + for (auto &ctrl : ctrls_) {\n> > > > > + ControlInfo &info =\n> ctrls_[ctrl.first][queueCount_];\n> > > > > + info.value = ctrls_[ctrl.first][queueCount_ -\n> 1].value;\n> > > > > + info.updated = false;\n> > > > > + }\n> > > > > +\n> > > > > + /* Update with new controls. */\n> > > > > + for (const auto &control : controls) {\n> > > > > + const ControlId *id =\n> > > > > device_->controls().idmap().at(control.first);\n> > > > > +\n> > > > > + if (delays_.find(id) == delays_.end())\n> > > > > + return false;\n> > > > >\n> > > >\n> > > > Could you add an error log message here? Do you think this should\n> be a\n> > > > continue, my inclination is that return false is the right thing?\n> > >\n> > > I think this should fail as a control was queued which delay is\n> unknown.\n> > >\n> > > >\n> > > >\n> > > > > +\n> > > > > + ControlInfo &info = ctrls_[id][queueCount_];\n> > > > > +\n> > > > > + info.value = control.second;\n> > > > > + info.updated = true;\n> > > > > +\n> > > > > + LOG(DelayedControls, Debug)\n> > > > > + << \"Queuing \" << id->name()\n> > > > > + << \" to \" << info.value.toString()\n> > > > > + << \" at index \" << queueCount_;\n> > > > > + }\n> > > > > +\n> > > > > + queueCount_++;\n> > > > > +\n> > > > > + return true;\n> > > > > +}\n> > > > > +\n> > > > > +/**\n> > > > > + * \\brief Read back controls in effect at a specific sequence\n> number\n> > > > > + * \\param[in] sequence Sequence number to get read back controls\n> for\n> > > > > + *\n> > > > > + * Read back what controls where in effect at a specific sequence\n> > > number.\n> > > > > The\n> > > > > + * history is a ring buffer of 16 entries where new and old values\n> > > > > coexist. It's\n> > > > > + * the callers responsibility to not read to old sequence numbers\n> that\n> > > > > have been\n> > > > > + * pushed out of the history.\n> > > > > + *\n> > > > > + * Historic values are evicted by pushing new values onto the\n> queue\n> > > using\n> > > > > + * push(). The max history from the current sequence number that\n> > > yields\n> > > > > valid\n> > > > > + * values are thus 16 minus number of controls pushed.\n> > > > > + *\n> > > > > + * \\returns List of controls in effect at \\a sequence\n> > > > > + */\n> > > > > +ControlList DelayedControls::get(uint32_t sequence)\n> > > > > +{\n> > > > > + std::lock_guard lock(lock_);\n> > > > > +\n> > > > > + uint32_t adjustedSeq = sequence - fistSequence_ + 1;\n> > > > > + unsigned int index = std::max(0, adjustedSeq -\n> maxDelay_);\n> > > > >\n> > > >\n> > > > Probably my misunderstanding, but will firstSequence ever be anything\n> > > but 0?\n> > >\n> > > Unfortunately yes. Some V4L2 drivers does not reset their sequence\n> > > number to 0 when starting to stream. So reputing start/stop may have\n> the\n> > > first frame seq+1 from the previous stop.\n> > >\n> > > >\n> > > >\n> > > > > +\n> > > > > + ControlList out(device_->controls());\n> > > > > + for (const auto &ctrl : ctrls_) {\n> > > > > + const ControlId *id = ctrl.first;\n> > > > > + const ControlInfo &info = ctrl.second[index];\n> > > > > +\n> > > > > + out.set(id->id(), info.value);\n> > > > > +\n> > > > > + LOG(DelayedControls, Debug)\n> > > > > + << \"Reading \" << id->name()\n> > > > > + << \" to \" << info.value.toString()\n> > > > > + << \" at index \" << index;\n> > > > > + }\n> > > > > +\n> > > > > + return out;\n> > > > > +}\n> > > > > +\n> > > > > +/**\n> > > > > + * \\brief Inform DelayedControls of a start of a new frame\n> > > > > + * \\param[in] sequence Sequence number of the frame that started\n> > > > > + *\n> > > > > + * Inform the state machine that a new frame have started and it's\n> > > > > sequence\n> > > > > + * number. It's user of this helpers responsibility to inform the\n> > > helper\n> > > > > + * at the start of every frame. This can with ease be connected\n> to the\n> > > > > start\n> > > > > + * of exposure (SOE) V4L2 event.\n> > > > > + */\n> > > > > +void DelayedControls::frameStart(uint32_t sequence)\n> > > > > +{\n> > > > > + LOG(DelayedControls, Debug) << \"frame \" << sequence << \"\n> > > started\";\n> > > > > +\n> > > > > + std::lock_guard lock(lock_);\n> > > > > +\n> > > > > + if (!running_) {\n> > > > > + fistSequence_ = sequence;\n> > > > > + running_ = true;\n> > > > > + }\n> > > > >\n> > > >\n> > > > As above, can firstSequence_ be anything but 0?\n> > > >\n> > > >\n> > > > > +\n> > > > > + /*\n> > > > > + * Create control list peaking ahead in the value queue to\n> > > ensure\n> > > > > + * values are set in time to satisfy the sensor delay.\n> > > > > + */\n> > > > > + ControlList out(device_->controls());\n> > > > > + for (const auto &ctrl : ctrls_) {\n> > > > > + const ControlId *id = ctrl.first;\n> > > > > + unsigned int delayDiff = maxDelay_ - delays_[id];\n> > > > > + unsigned int index = std::max(0, writeCount_ -\n> > > > > delayDiff);\n> > > > > + const ControlInfo &info = ctrl.second[index];\n> > > > > +\n> > > > > + if (info.updated) {\n> > > > > + out.set(id->id(), info.value);\n> > > > > + LOG(DelayedControls, Debug)\n> > > > > + << \"Setting \" << id->name()\n> > > > > + << \" to \" << info.value.toString()\n> > > > > + << \" at index \" << index;\n> > > > > + }\n> > > > > + }\n> > > > > +\n> > > > > + writeCount_++;\n> > > > > +\n> > > > > + while (writeCount_ >= queueCount_) {\n> > > > > + LOG(DelayedControls, Debug)\n> > > > > + << \"Queue is empty, auto queue no-op.\";\n> > > > > + queue({});\n> > > > > + }\n> > > > > +\n> > > > > + device_->setControls(&out);\n> > > > > +}\n> > > > > +\n> > > > > +} /* namespace libcamera */\n> > > > > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > > > > index 07711b5f93bcc921..19f22f9c94e1d64d 100644\n> > > > > --- a/src/libcamera/meson.build\n> > > > > +++ b/src/libcamera/meson.build\n> > > > > @@ -12,6 +12,7 @@ libcamera_sources = files([\n> > > > > 'controls.cpp',\n> > > > > 'control_serializer.cpp',\n> > > > > 'control_validator.cpp',\n> > > > > + 'delayed_controls.cpp',\n> > > > > 'device_enumerator.cpp',\n> > > > > 'device_enumerator_sysfs.cpp',\n> > > > > 'event_dispatcher.cpp',\n> > > > > --\n> > > > > 2.29.1\n> > > > >\n> > > > >\n> > >\n> > > --\n> > > Regards,\n> > > Niklas Söderlund\n> > >\n>\n> --\n> Regards,\n> Niklas Söderlund\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B7C79BE176\n\tfor ;\n\tTue, 24 Nov 2020 09:39:56 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1F75B633F6;\n\tTue, 24 Nov 2020 10:39:56 +0100 (CET)","from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com\n\t[IPv6:2a00:1450:4864:20::22a])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7705860333\n\tfor ;\n\tTue, 24 Nov 2020 10:39:54 +0100 (CET)","by mail-lj1-x22a.google.com with SMTP id y16so21270177ljk.1\n\tfor ;\n\tTue, 24 Nov 2020 01:39:54 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"pHmWZFo8\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=bN7aLIHRHKmluJ8ihgiFJfN5nw3rRygTBxrC2Z/mWMw=;\n\tb=pHmWZFo8fybWAiLhJPxQkEAmXzmq1E+RI15/MMXsup6WYRYsjjZnqf5FAP+wRF8pvw\n\tRk2JRsVb44zI2b0Qkig7A5olpN/NjA/tKV9yeWiILjMSRS9V7PRWU93kwLkejG86wm3l\n\tTUz1IsscDhJgRlv4ZKiDCvc62DuuqbufyJlkJ0HsDPEwC/+XzBmRyfiFh25ysYYEVVdn\n\tgpLNiSnpdWbbLQZVLl13LH7rSQj+x8DifoLgGILtR33sqWh4MTWA2yud5gtdn1k6WZ/Z\n\tpP13/rj+ARrLjlb+hjYzInYWX4oSahRgFX5zwkPW353YHtn39sSIdltnUCYsW2bc0cvA\n\th/rw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=bN7aLIHRHKmluJ8ihgiFJfN5nw3rRygTBxrC2Z/mWMw=;\n\tb=e7SqpSpUOTvTN5epwOFHdydXnXf0tzHIvKYtYv49mTivFPXyoynHpoyYn8jJL2DCDM\n\t1AX40NJMPmIb2/ZeyRlCRYXTPVNJW1UOdjGa9ofOE6QQSFaBsTu5GaHD9ruJZ271UOfg\n\tFj9e9SEIiXX6ckyuxt9K886YdUVg08FjwsiWMEEQpoxFQeoxgS7W9bZRmdFkCEuKm7C4\n\t0knYoV1h0ebIoGzNKVYeqWRVsEcOJy7m4+ke4wrtiaDqKqtEFBqZsz683+b8Uw71PLD8\n\tSw5iB6LPbeAFy1yQrddkZ1jqkqQr3kb6dnF/4QlMwUK+LL+1zeubArv0jXu6gQEzKB4e\n\t8h0A==","X-Gm-Message-State":"AOAM532/tcJIFed2ZKZ6SLoZavMQi4Ro3oKOJlrPJl8nhvfhUa4cupA+\n\tLO94cfzJmPFuMW5fQBINDRzFEFtzC5L7hBHfuYP1UA==","X-Google-Smtp-Source":"ABdhPJwtHvvs4H/iOwyHKGdrz/ofroZx/lsrPu62kQN4U3IW2cuPJ8PLuTpSIEQWLOoGex0VVZKOzxV1Ud2wuwqT82A=","X-Received":"by 2002:a2e:3310:: with SMTP id\n\td16mr1532884ljc.103.1606210793522; \n\tTue, 24 Nov 2020 01:39:53 -0800 (PST)","MIME-Version":"1.0","References":"<20201028010051.3830668-1-niklas.soderlund@ragnatech.se>\n\t<20201028010051.3830668-3-niklas.soderlund@ragnatech.se>\n\t\n\t<20201109225424.GE88486@oden.dyn.berto.se>\n\t\n\t<20201123170638.GE1773213@oden.dyn.berto.se>","In-Reply-To":"<20201123170638.GE1773213@oden.dyn.berto.se>","From":"Naushir Patuck ","Date":"Tue, 24 Nov 2020 09:39:36 +0000","Message-ID":"","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","Subject":"Re: [libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add\n\thelper for controls that applies with a delay","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Content-Type":"multipart/mixed;\n\tboundary=\"===============4468595297732616920==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]