[{"id":23692,"web_url":"https://patchwork.libcamera.org/comment/23692/","msgid":"","date":"2022-06-30T20:35:42","subject":"Re: [libcamera-devel] [PATCH v2.1 12/12] gstreamer: Fix race\n\tconditions in task pause/resume","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Thu, Jun 30, 2022 at 11:32:21PM +0300, Laurent Pinchart via libcamera-devel wrote:\n> The task run function races with two other threads that want to resume\n> the task: the requestCompleted() handler and the buffer-notify signal\n> handler. If the former queues completed requests or the latter queues\n> back buffers to the pool, and then resume the task, after the task run\n> handler checks the queues but before it attemps to pause the task, then\n> the task may be paused without noticing that more work is available.\n> \n> The most immediate way to fix this is to take the stream_lock in the\n> requestCompleted() and buffer-notify signal handlers, or cover the whole\n> task run handler with the GstLibcameraSrcState lock. This could cause\n> long delays in the requestCompleted() handler, so that's not a good\n> option.\n> \n> Instead, add a wakeup flag, preotected by the GstLibcameraSrcState lock,\n> that allows detection of a lost race, and retry the task run.\n\nOf course I forgot to update this. I'll replace this paragraph with\n\nInstead, pause the task unconditionally at the beginning of its run\nfunction, and track while processing buffers and requests if the task\nneeds to be resumed. It may also get resumed externally by the\nbuffer-notify signal handler or the request completion handler, which\nare guaranteed not to race due to the lock taken by the gst_task_pause()\nand gst_task_resume() functions.\n\n> Signed-off-by: Laurent Pinchart \n> ---\n> Changes since v2:\n> \n> - Invert the pause/resume logic\n> \n> Changes since v1:\n> \n> - Fix incorrect wakeup and pause logic\n> ---\n> src/gstreamer/gstlibcamerasrc.cpp | 71 ++++++++++++++++++++++---------\n> 1 file changed, 50 insertions(+), 21 deletions(-)\n> \n> diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp\n> index 9ea59631a9f2..a7a0c4403e28 100644\n> --- a/src/gstreamer/gstlibcamerasrc.cpp\n> +++ b/src/gstreamer/gstlibcamerasrc.cpp\n> @@ -259,6 +259,7 @@ GstLibcameraSrcState::requestCompleted(Request *request)\n> int GstLibcameraSrcState::processRequest()\n> {\n> \tstd::unique_ptr wrap;\n> +\tint err = 0;\n> \n> \t{\n> \t\tMutexLocker locker(lock_);\n> @@ -267,10 +268,13 @@ int GstLibcameraSrcState::processRequest()\n> \t\t\twrap = std::move(completedRequests_.front());\n> \t\t\tcompletedRequests_.pop();\n> \t\t}\n> +\n> +\t\tif (completedRequests_.empty())\n> +\t\t\terr = -ENOBUFS;\n> \t}\n> \n> \tif (!wrap)\n> -\t\treturn -ENODATA;\n> +\t\treturn -ENOBUFS;\n> \n> \tGstFlowReturn ret = GST_FLOW_OK;\n> \tgst_flow_combiner_reset(src_->flow_combiner);\n> @@ -310,7 +314,7 @@ int GstLibcameraSrcState::processRequest()\n> \t\treturn -EPIPE;\n> \t}\n> \n> -\treturn 0;\n> +\treturn err;\n> }\n> \n> static bool\n> @@ -380,47 +384,72 @@ gst_libcamera_src_task_run(gpointer user_data)\n> \tGstLibcameraSrc *self = GST_LIBCAMERA_SRC(user_data);\n> \tGstLibcameraSrcState *state = self->state;\n> \n> +\t/*\n> +\t * Start by pausing the task. The task may also get resumed by the\n> +\t * buffer-notify signal when new buffers are queued back to the pool,\n> +\t * or by the request completion handler when a new request has\n> +\t * completed. Both will resume the task after adding the buffers or\n> +\t * request to their respective lists, which are checked below to decide\n> +\t * if the task needs to be resumed for another iteration. This is thus\n> +\t * guaranteed to be race-free, the lock taken by gst_task_pause() and\n> +\t * gst_task_resume() serves as a memory barrier.\n> +\t */\n> +\tgst_task_pause(self->task);\n> +\n> +\tbool doResume = false;\n> +\n> \t/*\n> \t * Create and queue one request. If no buffers are available the\n> \t * function returns -ENOBUFS, which we ignore here as that's not a\n> \t * fatal error.\n> \t */\n> \tint ret = state->queueRequest();\n> -\tif (ret == -ENOMEM) {\n> +\tswitch (ret) {\n> +\tcase 0:\n> +\t\t/*\n> +\t\t * The request was successfully queued, there may be enough\n> +\t\t * buffers to create a new one. Don't pause the task to give it\n> +\t\t * another try.\n> +\t\t */\n> +\t\tdoResume = true;\n> +\t\tbreak;\n> +\n> +\tcase -ENOMEM:\n> \t\tGST_ELEMENT_ERROR(self, RESOURCE, NO_SPACE_LEFT,\n> \t\t\t\t (\"Failed to allocate request for camera '%s'.\",\n> \t\t\t\t state->cam_->id().c_str()),\n> \t\t\t\t (\"libcamera::Camera::createRequest() failed\"));\n> \t\tgst_task_stop(self->task);\n> \t\treturn;\n> +\n> +\tcase -ENOBUFS:\n> +\tdefault:\n> +\t\tbreak;\n> \t}\n> \n> -\t/* Process one completed request, if available. */\n> +\t/*\n> +\t * Process one completed request, if available, and record if further\n> +\t * requests are ready for processing.\n> +\t */\n> \tret = state->processRequest();\n> \tswitch (ret) {\n> +\tcase 0:\n> +\t\t/* Another completed request is available, resume the task. */\n> +\t\tdoResume = true;\n> +\t\tbreak;\n> +\n> \tcase -EPIPE:\n> \t\tgst_task_stop(self->task);\n> \t\treturn;\n> \n> -\tcase -ENODATA:\n> -\t\tgst_task_pause(self->task);\n> -\t\treturn;\n> +\tcase -ENOBUFS:\n> +\tdefault:\n> +\t\tbreak;\n> \t}\n> \n> -\t/*\n> -\t * Here we need to decide if we want to pause. This needs to\n> -\t * happen in lock step with the callback thread which may want\n> -\t * to resume the task and might push pending buffers.\n> -\t */\n> -\tbool do_pause;\n> -\n> -\t{\n> -\t\tMutexLocker locker(state->lock_);\n> -\t\tdo_pause = state->completedRequests_.empty();\n> -\t}\n> -\n> -\tif (do_pause)\n> -\t\tgst_task_pause(self->task);\n> +\t/* Resume the task for another iteration if needed. */\n> +\tif (doResume)\n> +\t\tgst_task_resume(self->task);\n> }\n> \n> static void","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1C291BE173\n\tfor ;\n\tThu, 30 Jun 2022 20:36:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4CC276564F;\n\tThu, 30 Jun 2022 22:36:04 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B5B1B65633\n\tfor ;\n\tThu, 30 Jun 2022 22:36:02 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 2E5ED45F;\n\tThu, 30 Jun 2022 22:36:02 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1656621364;\n\tbh=PCbhxIj3RzG+ZhkMNoapwp5GIyuOM1IGZCNf0cJGcbw=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=X137OjFPjNHCdDd/vESLdpjwGxy5frLPz0YzZsHZ3H3iqsxzguJRpO0pcmQYTxWqN\n\tCcBaceGy0lCP6NSEjpHWiKedUozDKLSP1DAffJ9GaAa/HQMywf7VPQo3iCNu/x/7yf\n\tOxCiiEnN8Kpasx9ko5V6crpIWxdrHNQQA27mkWQtLvM2hznNm3J2ixhy04GnlZ1NPM\n\tqkI/0EEOu7JwMD/ezi80fyM3+QIjgFIIm7Yi0mXVAjM5zhIn0nzm9xP0A6IuGR7lFY\n\tuTPdVUF1WMtvGv9Eusps4YFx+W3NNrNvOB7WLHRm0FeOPUEBJsiHoOmhtnXpMUAEr/\n\teQzBlqoeu0AQQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1656621362;\n\tbh=PCbhxIj3RzG+ZhkMNoapwp5GIyuOM1IGZCNf0cJGcbw=;\n\th=Date:From:To:Subject:References:In-Reply-To:From;\n\tb=GDEYoFfhop/y+VE2bLu5P0KpnAZqQJDxUyFwE/mE0X+4AR9FwdIhuhi6e/KIDkGxh\n\tP05tBd/lWevy3H9Vm87UZVqmezO4NBIWXoqo4D7U2ELbr8wIA97SxocYn9kHh8Byvy\n\tD2XqBvNHmriIyQCWy4TCtfz42ZoAToOzT66W7NDo="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"GDEYoFfh\"; dkim-atps=neutral","Date":"Thu, 30 Jun 2022 23:35:42 +0300","To":"libcamera-devel@lists.libcamera.org,\n\tNicolas Dufresne ","Message-ID":"","References":"<20220630000251.31295-13-laurent.pinchart@ideasonboard.com>\n\t<20220630203221.12440-1-laurent.pinchart@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20220630203221.12440-1-laurent.pinchart@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v2.1 12/12] gstreamer: Fix race\n\tconditions in task pause/resume","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":23727,"web_url":"https://patchwork.libcamera.org/comment/23727/","msgid":"","date":"2022-07-04T19:29:22","subject":"Re: [libcamera-devel] [PATCH v2.1 12/12] gstreamer: Fix race\n\tconditions in task pause/resume","submitter":{"id":31,"url":"https://patchwork.libcamera.org/api/people/31/","name":"Nicolas Dufresne","email":"nicolas.dufresne@collabora.com"},"content":"Le jeudi 30 juin 2022 à 23:35 +0300, Laurent Pinchart via libcamera-devel a\nécrit :\n> On Thu, Jun 30, 2022 at 11:32:21PM +0300, Laurent Pinchart via libcamera-devel wrote:\n> > The task run function races with two other threads that want to resume\n> > the task: the requestCompleted() handler and the buffer-notify signal\n> > handler. If the former queues completed requests or the latter queues\n> > back buffers to the pool, and then resume the task, after the task run\n> > handler checks the queues but before it attemps to pause the task, then\n> > the task may be paused without noticing that more work is available.\n> > \n> > The most immediate way to fix this is to take the stream_lock in the\n> > requestCompleted() and buffer-notify signal handlers, or cover the whole\n> > task run handler with the GstLibcameraSrcState lock. This could cause\n> > long delays in the requestCompleted() handler, so that's not a good\n> > option.\n> > \n> > Instead, add a wakeup flag, preotected by the GstLibcameraSrcState lock,\n> > that allows detection of a lost race, and retry the task run.\n> \n> Of course I forgot to update this. I'll replace this paragraph with\n> \n> Instead, pause the task unconditionally at the beginning of its run\n> function, and track while processing buffers and requests if the task\n> needs to be resumed. It may also get resumed externally by the\n> buffer-notify signal handler or the request completion handler, which\n> are guaranteed not to race due to the lock taken by the gst_task_pause()\n> and gst_task_resume() functions.\n> \n> > Signed-off-by: Laurent Pinchart \n\nThanks Laurent for your hard work on.\n\nReviewed-by: Nicolas Dufresne \n\n> > ---\n> > Changes since v2:\n> > \n> > - Invert the pause/resume logic\n> > \n> > Changes since v1:\n> > \n> > - Fix incorrect wakeup and pause logic\n> > ---\n> > src/gstreamer/gstlibcamerasrc.cpp | 71 ++++++++++++++++++++++---------\n> > 1 file changed, 50 insertions(+), 21 deletions(-)\n> > \n> > diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp\n> > index 9ea59631a9f2..a7a0c4403e28 100644\n> > --- a/src/gstreamer/gstlibcamerasrc.cpp\n> > +++ b/src/gstreamer/gstlibcamerasrc.cpp\n> > @@ -259,6 +259,7 @@ GstLibcameraSrcState::requestCompleted(Request *request)\n> > int GstLibcameraSrcState::processRequest()\n> > {\n> > \tstd::unique_ptr wrap;\n> > +\tint err = 0;\n> > \n> > \t{\n> > \t\tMutexLocker locker(lock_);\n> > @@ -267,10 +268,13 @@ int GstLibcameraSrcState::processRequest()\n> > \t\t\twrap = std::move(completedRequests_.front());\n> > \t\t\tcompletedRequests_.pop();\n> > \t\t}\n> > +\n> > +\t\tif (completedRequests_.empty())\n> > +\t\t\terr = -ENOBUFS;\n> > \t}\n> > \n> > \tif (!wrap)\n> > -\t\treturn -ENODATA;\n> > +\t\treturn -ENOBUFS;\n> > \n> > \tGstFlowReturn ret = GST_FLOW_OK;\n> > \tgst_flow_combiner_reset(src_->flow_combiner);\n> > @@ -310,7 +314,7 @@ int GstLibcameraSrcState::processRequest()\n> > \t\treturn -EPIPE;\n> > \t}\n> > \n> > -\treturn 0;\n> > +\treturn err;\n> > }\n> > \n> > static bool\n> > @@ -380,47 +384,72 @@ gst_libcamera_src_task_run(gpointer user_data)\n> > \tGstLibcameraSrc *self = GST_LIBCAMERA_SRC(user_data);\n> > \tGstLibcameraSrcState *state = self->state;\n> > \n> > +\t/*\n> > +\t * Start by pausing the task. The task may also get resumed by the\n> > +\t * buffer-notify signal when new buffers are queued back to the pool,\n> > +\t * or by the request completion handler when a new request has\n> > +\t * completed. Both will resume the task after adding the buffers or\n> > +\t * request to their respective lists, which are checked below to decide\n> > +\t * if the task needs to be resumed for another iteration. This is thus\n> > +\t * guaranteed to be race-free, the lock taken by gst_task_pause() and\n> > +\t * gst_task_resume() serves as a memory barrier.\n> > +\t */\n> > +\tgst_task_pause(self->task);\n> > +\n> > +\tbool doResume = false;\n> > +\n> > \t/*\n> > \t * Create and queue one request. If no buffers are available the\n> > \t * function returns -ENOBUFS, which we ignore here as that's not a\n> > \t * fatal error.\n> > \t */\n> > \tint ret = state->queueRequest();\n> > -\tif (ret == -ENOMEM) {\n> > +\tswitch (ret) {\n> > +\tcase 0:\n> > +\t\t/*\n> > +\t\t * The request was successfully queued, there may be enough\n> > +\t\t * buffers to create a new one. Don't pause the task to give it\n> > +\t\t * another try.\n> > +\t\t */\n> > +\t\tdoResume = true;\n> > +\t\tbreak;\n> > +\n> > +\tcase -ENOMEM:\n> > \t\tGST_ELEMENT_ERROR(self, RESOURCE, NO_SPACE_LEFT,\n> > \t\t\t\t (\"Failed to allocate request for camera '%s'.\",\n> > \t\t\t\t state->cam_->id().c_str()),\n> > \t\t\t\t (\"libcamera::Camera::createRequest() failed\"));\n> > \t\tgst_task_stop(self->task);\n> > \t\treturn;\n> > +\n> > +\tcase -ENOBUFS:\n> > +\tdefault:\n> > +\t\tbreak;\n> > \t}\n> > \n> > -\t/* Process one completed request, if available. */\n> > +\t/*\n> > +\t * Process one completed request, if available, and record if further\n> > +\t * requests are ready for processing.\n> > +\t */\n> > \tret = state->processRequest();\n> > \tswitch (ret) {\n> > +\tcase 0:\n> > +\t\t/* Another completed request is available, resume the task. */\n> > +\t\tdoResume = true;\n> > +\t\tbreak;\n> > +\n> > \tcase -EPIPE:\n> > \t\tgst_task_stop(self->task);\n> > \t\treturn;\n> > \n> > -\tcase -ENODATA:\n> > -\t\tgst_task_pause(self->task);\n> > -\t\treturn;\n> > +\tcase -ENOBUFS:\n> > +\tdefault:\n> > +\t\tbreak;\n> > \t}\n> > \n> > -\t/*\n> > -\t * Here we need to decide if we want to pause. This needs to\n> > -\t * happen in lock step with the callback thread which may want\n> > -\t * to resume the task and might push pending buffers.\n> > -\t */\n> > -\tbool do_pause;\n> > -\n> > -\t{\n> > -\t\tMutexLocker locker(state->lock_);\n> > -\t\tdo_pause = state->completedRequests_.empty();\n> > -\t}\n> > -\n> > -\tif (do_pause)\n> > -\t\tgst_task_pause(self->task);\n> > +\t/* Resume the task for another iteration if needed. */\n> > +\tif (doResume)\n> > +\t\tgst_task_resume(self->task);\n> > }\n> > \n> > static void\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8C833BD808\n\tfor ;\n\tMon, 4 Jul 2022 19:29:34 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E2EBA60C1A;\n\tMon, 4 Jul 2022 21:29:33 +0200 (CEST)","from madras.collabora.co.uk (madras.collabora.co.uk\n\t[46.235.227.172])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C470F60BCE\n\tfor ;\n\tMon, 4 Jul 2022 21:29:31 +0200 (CEST)","from nicolas-tpx395.localdomain (mtl.collabora.ca [66.171.169.34])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (4096 bits)\n\tserver-digest SHA256)\n\t(No client certificate requested) (Authenticated sender: nicolas)\n\tby madras.collabora.co.uk (Postfix) with ESMTPSA id 7D98266018E6;\n\tMon, 4 Jul 2022 20:29:30 +0100 (BST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1656962974;\n\tbh=//achsjTQm/1OzMVyG9h3pnem680M3iYXwZbMc1Ly9k=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=MlDbqNFfvuE5ACV6ztVBQuNC7IoBuAny8BDTUOs5P0rPdIqR6HITOrqKjt2UKA69g\n\tNNQAvVULBWplE4Qzk+YKh2VwOFuSE0TJ6cVn2ybqVkYzVdXHUEY18eyvwkkB5Q91XV\n\tXSubydGz+cQtat/FOOqkiYIpo1rafmK3ZYQ2Dx2XlvtiMuAFnP+5kWWaMYddpAdShk\n\thphIuPyphSS0keaCv0Moj5Qwv87/L3UMQPeKlj5YKdyEf/smxA/S8NAdpsVKCp2cIB\n\t+TpSbPPS0IuX/e7+WIX6YYQJDxK4WAJmPsvgTC8JKh2K0ku3AXDBaPH156fZDrlJDF\n\t0HlyXQwZD+bvA==","v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com;\n\ts=mail; t=1656962970;\n\tbh=//achsjTQm/1OzMVyG9h3pnem680M3iYXwZbMc1Ly9k=;\n\th=Subject:From:To:Date:In-Reply-To:References:From;\n\tb=OlhcPuWhbOUXQpWJd8rlIVBZnwbufUW8FDynQbJJ8yWBta1EQM8JVO51Wt//ArOe7\n\tt7Z+1cDPDidFtWV3BmrCOCjYnVFOQwrzVRhPF8nSl2dO51FeNDZPS57BDAPjewWWZK\n\tzA2pXx2ZdBqDj9uxj2c/eQbs54mXJ/5QOA1WUkieNm7wQiuTtogfgeD+52fWWwnOBG\n\t6LDXmqz9vj0vCSSOyL3aZ5iDnEaUsYkeh5Te0QCMfYnGIpf7CsEJwqvgRJLOezNY7+\n\t/3kT2gXxjkr7qiTPzkaa6SMVDWfSMvc6eSl7DSChUMqQv5/MEbn4W9cTl1V6GL7Ifj\n\t/SvGvxoiA9Jng=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=collabora.com\n\theader.i=@collabora.com\n\theader.b=\"OlhcPuWh\"; dkim-atps=neutral","Message-ID":"","To":"Laurent Pinchart , \n\tlibcamera-devel@lists.libcamera.org","Date":"Mon, 04 Jul 2022 15:29:22 -0400","In-Reply-To":"","References":"<20220630000251.31295-13-laurent.pinchart@ideasonboard.com>\n\t<20220630203221.12440-1-laurent.pinchart@ideasonboard.com>\n\t","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","User-Agent":"Evolution 3.44.2 (3.44.2-1.fc36) ","MIME-Version":"1.0","Subject":"Re: [libcamera-devel] [PATCH v2.1 12/12] gstreamer: Fix race\n\tconditions in task pause/resume","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Nicolas Dufresne via libcamera-devel\n\t","Reply-To":"Nicolas Dufresne ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]