[{"id":27189,"web_url":"https://patchwork.libcamera.org/comment/27189/","msgid":"","date":"2023-05-31T12:57:16","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi David,\n\nThank you for this patch. Indeed removing the drop frame logic the pipeline\nhandler would simplify things!\n\nOn Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel\n wrote:\n>\n> This control is passed back in a frame as metadata to indicate whether\n> the camera system is still in a startup phase, and the application is\n> advised to avoid using the frame.\n>\n> Signed-off-by: David Plowman \n> ---\n> src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> 1 file changed, 15 insertions(+)\n>\n> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> index adea5f90..4742d907 100644\n> --- a/src/libcamera/control_ids.yaml\n> +++ b/src/libcamera/control_ids.yaml\n> @@ -694,6 +694,21 @@ controls:\n> Continuous AF is paused. No further state changes or lens movements\n> will occur until the AfPauseResume control is sent.\n>\n> + - StartupFrame:\n> + type: bool\n> + description: |\n> + The value true indicates that the camera system is still in a startup\n> + phase where the output images may not be reliable, or that certain of\n> + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> + still be changing quite rapidly.\n> +\n> + Applications are advised to avoid using these frames. Mostly, they will\n> + occur when the camera system starts for the first time, although,\n> + depending on the sensor and the implementation, they could occur at\n> + other times.\n> +\n> + The value false indicates that this is a normal frame.\n\nJust throwing it out there, but would it be useful if this control was an\ninteger with the count of startup frames left to handle? A value of 0, or the\nabsence of the control would indicate this is a \"valid\" frame.\n\nRegards,\nNaush\n\n> +\n> # ----------------------------------------------------------------------------\n> # Draft controls section\n>\n> --\n> 2.30.2\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id ECF34C3213\n\tfor ;\n\tWed, 31 May 2023 12:57:33 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 57E8A62728;\n\tWed, 31 May 2023 14:57:33 +0200 (CEST)","from mail-yw1-x1131.google.com (mail-yw1-x1131.google.com\n\t[IPv6:2607:f8b0:4864:20::1131])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4F851626F8\n\tfor ;\n\tWed, 31 May 2023 14:57:31 +0200 (CEST)","by mail-yw1-x1131.google.com with SMTP id\n\t00721157ae682-565ba2c7554so45485107b3.3\n\tfor ;\n\tWed, 31 May 2023 05:57:31 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1685537853;\n\tbh=On3qfM8pZuZQl0cTNbfwGHpndumc1Z+3XLrw78uFCvg=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=zeVfbOUXQCcZkR6AJ2oDbXTrWwSceaO5peYOZTNJn9In8DTOWTU3s4DmlnNqXk1GQ\n\tPOLFahMzpIwY496RbvJzzOWhfqmTRxTCVCCf4jnCsJe/oEQBDOesMV+jUCtMZVrZGL\n\tuLwLE9y5gJe6kX9RwpRsSZG1ElE3wNo9BpdPJcyz/PFtt8xIjU7wGQfnTp1VCZCbcu\n\tlqQ0KQ1KOMgr6IDJQP8S7SY/zAliwvmCVlpSEYg0+MlskFLb8JgHMUlPeIw8SJ6EWO\n\tm6MlbXIoGeObvs3mMsTgJ2eZK7TPMxi31JN7/N6mcYv2TQduHy6wsi7taXxaPDwde6\n\t9r3J0/2JwydjQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1685537850; x=1688129850;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=KLYhraWBJXKT8c2DHKOXOMCBi9pZRxHoqp1tRiwQ8WQ=;\n\tb=hNq/7168ciUAAt4Ctbjt09dJSG3jGaNrG9ijvawNIr0JkL0tVj9dQkoSZF2KPFqbuX\n\t4dfBkPZpKE0dgQSnUU4Yh+ZToHjfWeV8CTGzeJs1k6fnIdHw6j++XMTg340dOJrzmkbj\n\tRTBO4eEpu+EqcBxloMOdhLvNcgiwi7I7KzV72iDz2kFVyc/VAzN/a9CtCn92YNGNIisW\n\t/uJY2xIeQlyM2kyaOJqXM+fyEgEMX87LY3A0OuLoAhy0IHCopJB+k5frmdMudbmIEr1+\n\tvW/yu2mULs4X2KFAWwWvAFxNUVYgKU12Wt/LCrOD9xBn/dNReXyuIFE4KnCfcUhQ3UWq\n\t5eag=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"hNq/7168\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1685537850; x=1688129850;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=KLYhraWBJXKT8c2DHKOXOMCBi9pZRxHoqp1tRiwQ8WQ=;\n\tb=cFWeSpWZdZBbmRuUYaRhVAmJOL0j8jNJHZ3dsAfoJxUxLziHUeY6IsvwDobFO1x22E\n\tfxr7R+QhY91HHetwkFMMvuebZn7cvzfZnkHHyJmxyS9z1w6eWX3VREJ8fP+rMeYt0UnM\n\tNGdW2e64sid1kIrLtCIwWvWuoQvyN1AarMeTDzKbrYRxRMHmAB6duO7LzinDYMCAgIbR\n\tDL7TBzzpq7YL2UMtpfCc2hO2fBLkr86T/VSD7bXig+DScX4v63Gl9z/PxJCNiq62QTK6\n\tZ5z02najnnGaN2TnQpRzHnDoujbZx5n8yWmp73utmHKJVqj0i0R8+PCcho1fLS3xdu+g\n\trOhQ==","X-Gm-Message-State":"AC+VfDyGLLxYKnS9NobTwPUX1tsDeR39MHL9ddgRcNOUXZH2AhFg49Dc\n\tbieYjYumxIgel2+HA80gtO5wnidwY+oNmcwh0NtlBQ==","X-Google-Smtp-Source":"ACHHUZ5/0WZbo3vvFV5vc9LxQpFqeaWm677ec4NMqO3nh1PlvJ6FdvBhGNhAqivs7vVxjIdAYMJrqKYpEzZ5kcIC0TM=","X-Received":"by 2002:a25:4e54:0:b0:b9e:45e1:3e3 with SMTP id\n\tc81-20020a254e54000000b00b9e45e103e3mr5208477ybb.2.1685537849952;\n\tWed, 31 May 2023 05:57:29 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>","In-Reply-To":"<20230531125016.5540-2-david.plowman@raspberrypi.com>","Date":"Wed, 31 May 2023 13:57:16 +0100","Message-ID":"","To":"David Plowman ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27249,"web_url":"https://patchwork.libcamera.org/comment/27249/","msgid":"<20230605081739.GC30841@pendragon.ideasonboard.com>","date":"2023-06-05T08:17:39","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hello,\n\nOn Wed, May 31, 2023 at 01:57:16PM +0100, Naushir Patuck via libcamera-devel wrote:\n> Hi David,\n> \n> Thank you for this patch. Indeed removing the drop frame logic the pipeline\n> handler would simplify things!\n\nThat's a change I would welcome :-)\n\n> On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> >\n> > This control is passed back in a frame as metadata to indicate whether\n> > the camera system is still in a startup phase, and the application is\n> > advised to avoid using the frame.\n> >\n> > Signed-off-by: David Plowman \n> > ---\n> > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > 1 file changed, 15 insertions(+)\n> >\n> > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > index adea5f90..4742d907 100644\n> > --- a/src/libcamera/control_ids.yaml\n> > +++ b/src/libcamera/control_ids.yaml\n> > @@ -694,6 +694,21 @@ controls:\n> > Continuous AF is paused. No further state changes or lens movements\n> > will occur until the AfPauseResume control is sent.\n> >\n> > + - StartupFrame:\n> > + type: bool\n> > + description: |\n> > + The value true indicates that the camera system is still in a startup\n> > + phase where the output images may not be reliable, or that certain of\n> > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > + still be changing quite rapidly.\n\nI think we need to decide with a bit more details what constitute a\n\"startup frame\" and what doesn't, or we will have all kinds of\ninconsistent behaviour.\n\nI read it that we have multiple criteria on which we base the decision:\n\n- output images not being \"reliable\"\n- 3A algorithms convergence\n- \"possibly others\"\n\nThe second criteria is fairly clear, but I'm thinking we should possibly\nexclude it from the startup frames and report convergence of the 3A\nalgorithms instead.\n\nThe first and third criteria are quite vague. If I recall correctly, the\nfirst one includes bad frames from the sensor (as in completely\ncorrupted frames, such as frames that are all black or made of random\ndata). Those are completely unusable for applications, is there a value\nin still making them available instead of dropping them correctly ?\n\nWhat are the other reasons a frame would be a \"startup frame\" ?\n\n> > +\n> > + Applications are advised to avoid using these frames. Mostly, they will\n> > + occur when the camera system starts for the first time, although,\n> > + depending on the sensor and the implementation, they could occur at\n> > + other times.\n> > +\n> > + The value false indicates that this is a normal frame.\n> \n> Just throwing it out there, but would it be useful if this control was an\n> integer with the count of startup frames left to handle? A value of 0, or the\n> absence of the control would indicate this is a \"valid\" frame.\n> \n> > +\n> > # ----------------------------------------------------------------------------\n> > # Draft controls section\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id AE49BC3200\n\tfor ;\n\tMon, 5 Jun 2023 08:17:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E5D9E6287D;\n\tMon, 5 Jun 2023 10:17:42 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9036161EA2\n\tfor ;\n\tMon, 5 Jun 2023 10:17:41 +0200 (CEST)","from pendragon.ideasonboard.com (om126156242094.26.openmobile.ne.jp\n\t[126.156.242.94])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id C79961BA;\n\tMon, 5 Jun 2023 10:17:15 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1685953062;\n\tbh=N7GiPvCg4XUhwVrRNWphnCCrjsgo7/jIuWGL34WTnv0=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=GC7ESBEosDq4uzcZ88EjgA/uw7XB8zZ+xliucTm+7zyv91OhBsYlOTHU3uz1DBLZt\n\tujC/3LcO2SqyNT3B1rzQ0PI7l3Bd1OKf5QcVciSRV2poF6poKlfhPUZCyjHnwwr4hm\n\tvmqrnSQU7dYN6tXU253Klq04a/fi5mwjAu0wpplrqA8YyF9RtQZRR7EN6kmF1Gd7v9\n\tAfLnQomoFVord5S9Pc1KoBfBA0T33dGynhmglDpoYOpQ6XsROwqiCGNcJ1TrTNqq6j\n\toXiNaNxeUTFCLmWYVlnWv1FIiE8YlO7/b87DqiPv0zgfX+HM5rbPbXcJ7KqALWTB3j\n\tDsyUDL0RykdXg==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1685953036;\n\tbh=N7GiPvCg4XUhwVrRNWphnCCrjsgo7/jIuWGL34WTnv0=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=qgE3coIRec8qqiOnteYtQaIGKbSylmNfW/uDsYPuqMQWY2aBotTgJYhuZw7EUz7mX\n\tU6WyYz8uFBbEj1DljvBu9RCC9KUEyNpwzU4fCM/mGphGCBt3xdleTrp8i0SgVyy1r6\n\tqfHggF5Biy/lC+MSbfCowhlXGAokFHb39rMheYQo="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"qgE3coIR\"; dkim-atps=neutral","Date":"Mon, 5 Jun 2023 11:17:39 +0300","To":"Naushir Patuck ","Message-ID":"<20230605081739.GC30841@pendragon.ideasonboard.com>","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27251,"web_url":"https://patchwork.libcamera.org/comment/27251/","msgid":"","date":"2023-06-05T09:32:25","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Laurent,\n\nDavid is away this week so his reply will be delayed.\n\nOn Mon, 5 Jun 2023 at 09:17, Laurent Pinchart\n wrote:\n>\n> Hello,\n>\n> On Wed, May 31, 2023 at 01:57:16PM +0100, Naushir Patuck via libcamera-devel wrote:\n> > Hi David,\n> >\n> > Thank you for this patch. Indeed removing the drop frame logic the pipeline\n> > handler would simplify things!\n>\n> That's a change I would welcome :-)\n>\n> > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > >\n> > > This control is passed back in a frame as metadata to indicate whether\n> > > the camera system is still in a startup phase, and the application is\n> > > advised to avoid using the frame.\n> > >\n> > > Signed-off-by: David Plowman \n> > > ---\n> > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > 1 file changed, 15 insertions(+)\n> > >\n> > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > index adea5f90..4742d907 100644\n> > > --- a/src/libcamera/control_ids.yaml\n> > > +++ b/src/libcamera/control_ids.yaml\n> > > @@ -694,6 +694,21 @@ controls:\n> > > Continuous AF is paused. No further state changes or lens movements\n> > > will occur until the AfPauseResume control is sent.\n> > >\n> > > + - StartupFrame:\n> > > + type: bool\n> > > + description: |\n> > > + The value true indicates that the camera system is still in a startup\n> > > + phase where the output images may not be reliable, or that certain of\n> > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > + still be changing quite rapidly.\n>\n> I think we need to decide with a bit more details what constitute a\n> \"startup frame\" and what doesn't, or we will have all kinds of\n> inconsistent behaviour.\n>\n> I read it that we have multiple criteria on which we base the decision:\n>\n> - output images not being \"reliable\"\n> - 3A algorithms convergence\n> - \"possibly others\"\n>\n> The second criteria is fairly clear, but I'm thinking we should possibly\n> exclude it from the startup frames and report convergence of the 3A\n> algorithms instead.\n\nThe 3A \"startup\" convergence is included as a criteria here because during\nstartup, we drive the AE/AWB/ALSC as aggressively (i.e. no filtering) as\npossible to achieve convergence as fast as we possibly can. This means the\nimage output can oscillate quite badly - hence applications should avoid\ndisplaying or consuming them.\n\nI feel that this startup phase needs to be treated differently compared to a\nnormal \"converging\" phase where the algorithms are filtering the outputs and the\ntransitions our smooth, and conversely the application can (and probably should)\ndisplay/consume these frames.\n\n>\n> The first and third criteria are quite vague. If I recall correctly, the\n> first one includes bad frames from the sensor (as in completely\n> corrupted frames, such as frames that are all black or made of random\n> data). Those are completely unusable for applications, is there a value\n> in still making them available instead of dropping them correctly ?\n\nThis is what we do now (plus dropping the startup frames that may be good from\nthe sensor but still in the 3A startup phase), but I think a key goals of this\nchange is be consistent (i.e. let the application handle all frames always), and\nalso help with avoiding some framebuffer allocations to help with startup\nconvergence performance.\n\nRegards,\nNaush\n\n>\n> What are the other reasons a frame would be a \"startup frame\" ?\n>\n> > > +\n> > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > + occur when the camera system starts for the first time, although,\n> > > + depending on the sensor and the implementation, they could occur at\n> > > + other times.\n> > > +\n> > > + The value false indicates that this is a normal frame.\n> >\n> > Just throwing it out there, but would it be useful if this control was an\n> > integer with the count of startup frames left to handle? A value of 0, or the\n> > absence of the control would indicate this is a \"valid\" frame.\n> >\n> > > +\n> > > # ----------------------------------------------------------------------------\n> > > # Draft controls section\n> > >\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 418F3C3200\n\tfor ;\n\tMon, 5 Jun 2023 09:32:45 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A713C61EA3;\n\tMon, 5 Jun 2023 11:32:44 +0200 (CEST)","from mail-yb1-xb36.google.com (mail-yb1-xb36.google.com\n\t[IPv6:2607:f8b0:4864:20::b36])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 02BF761EA2\n\tfor ;\n\tMon, 5 Jun 2023 11:32:42 +0200 (CEST)","by mail-yb1-xb36.google.com with SMTP id\n\t3f1490d57ef6-ba1815e12efso3948437276.3\n\tfor ;\n\tMon, 05 Jun 2023 02:32:42 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1685957564;\n\tbh=U3i8Zn1KqT/8H6aIiV0qdWzhR/ZkTBkmjpVmv9voZss=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=B2+9/8urymKd6lrVUxxrO08RxLFpOra0mBW4g23NEbeGbUBCeUplbvmvc4M7J68fB\n\tyXe5gANaOlg2W6y2iRCkqzKjarOp1I44fuZPX6FVFpFzlMnPJuQI0UhMiAvahephjy\n\ttZfdWDU5EPbnDHjYMFOo/jzq3F2btAzE9QZxDX3aZnm0dzv34/1Vp0MBTDJIHXZNyR\n\tmnWcABOA83rxMsf1TzimCAPTr6FcZD1Pt0oXrPfsAop+SarPTs9n1KjFEDPiE5OAG+\n\tZav5416qL/quDmYMC/w0WsQ6zSMqfz9y7J8e+CLL1UvXFBwS7iXEH+tP+2YGR/0W9G\n\tzHUaNBYeCYH0g==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1685957562; x=1688549562;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=apzW3mo507Cpk2eZ3CEpTMhO0plN0biKpqHSFGJ4+sI=;\n\tb=P9sDQ5rqDnTyuGQ/w6I226nMRDtEkPWnjDf1VHyyjuKnxR6TgjkMvnyAub+BWJ46M+\n\trGR6wzCI4KLDvERFZVIlS4fGTe9wtwbPquSWsxcX7DCxB6mWV7jK2PlGgacyOGWj7k+Q\n\tY0E+2PNRmkQrBcTueMdgSxTq0KckwErYNzUkd38oPsPCMusVYyrFmqpm7vHEFA6DLuEI\n\tW1HDAZ1sN0pCQ7TzBQXEQnx3Lbp/YbRa6mNnxAlhMvGcIxtvKgwcvW3I43tsItsbSFoq\n\t12yRSNVUW+8EBF8OX0rpnMKPa5f7xWjsAzavXUc9GdR3bBLgfaq/ZD+duG/vBcQle8oj\n\t0lDA=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"P9sDQ5rq\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1685957562; x=1688549562;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=apzW3mo507Cpk2eZ3CEpTMhO0plN0biKpqHSFGJ4+sI=;\n\tb=geIJCxcXwI80pv/TfmObWTs4X2LtnF5tesnhgH/RhjJozr+j80eXwIWxf0U5wy90Q3\n\tYiNY3a7fxnC6Ci+DlmvqgkoXE9KTDPH7n1zR/dhXzqw/yms7z6hIF4LY/Mi8eXsWm6iY\n\tudHXvvmJfwAtBQd1ZUlq2FTskWmPkSLPBkqByIrGu85qwwZDxXKjv8swhnCVEHG+BUtq\n\t56O1LEeZ1VJ8LW2H45e82osMVWQMt2zKtAYqNRg9iKghVAeUcQdMr28rtzDdYW9AjtBk\n\t9hWR0FpLGgUsQBMlLVDUKJY+Zlfil85EOVrB9xkaXQCBba63XIhU2w1kyd4i5GJoy7df\n\tdpMA==","X-Gm-Message-State":"AC+VfDxpA8RHUH1CfTlT9YWQ0+9c5TaMIson1Z1l5ozVYaoGOKblXDwI\n\ty038XvsPKjfjubIU1may7/HuMATlMnN98NkEkPKvXA==","X-Google-Smtp-Source":"ACHHUZ48fJo/HM/5dBPmyh1cjxRNpfO7ygDGsfCS3brjwp3nSoRvXFRzGkv2ILQ79GzQqdr912pZ5m0q/MNht88fZ3c=","X-Received":"by 2002:a0d:cb92:0:b0:565:c96b:f526 with SMTP id\n\tn140-20020a0dcb92000000b00565c96bf526mr7787797ywd.19.1685957560362;\n\tMon, 05 Jun 2023 02:32:40 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<20230605081739.GC30841@pendragon.ideasonboard.com>","In-Reply-To":"<20230605081739.GC30841@pendragon.ideasonboard.com>","Date":"Mon, 5 Jun 2023 10:32:25 +0100","Message-ID":"","To":"Laurent Pinchart ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27318,"web_url":"https://patchwork.libcamera.org/comment/27318/","msgid":"","date":"2023-06-12T09:43:52","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi again\n\nThanks for the discussion on this! Mostly I probably want to reiterate\nwhat Naush has said.\n\nOn Mon, 5 Jun 2023 at 10:32, Naushir Patuck wrote:\n>\n> Hi Laurent,\n>\n> David is away this week so his reply will be delayed.\n>\n> On Mon, 5 Jun 2023 at 09:17, Laurent Pinchart\n> wrote:\n> >\n> > Hello,\n> >\n> > On Wed, May 31, 2023 at 01:57:16PM +0100, Naushir Patuck via libcamera-devel wrote:\n> > > Hi David,\n> > >\n> > > Thank you for this patch. Indeed removing the drop frame logic the pipeline\n> > > handler would simplify things!\n> >\n> > That's a change I would welcome :-)\n> >\n> > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > > >\n> > > > This control is passed back in a frame as metadata to indicate whether\n> > > > the camera system is still in a startup phase, and the application is\n> > > > advised to avoid using the frame.\n> > > >\n> > > > Signed-off-by: David Plowman \n> > > > ---\n> > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > 1 file changed, 15 insertions(+)\n> > > >\n> > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > index adea5f90..4742d907 100644\n> > > > --- a/src/libcamera/control_ids.yaml\n> > > > +++ b/src/libcamera/control_ids.yaml\n> > > > @@ -694,6 +694,21 @@ controls:\n> > > > Continuous AF is paused. No further state changes or lens movements\n> > > > will occur until the AfPauseResume control is sent.\n> > > >\n> > > > + - StartupFrame:\n> > > > + type: bool\n> > > > + description: |\n> > > > + The value true indicates that the camera system is still in a startup\n> > > > + phase where the output images may not be reliable, or that certain of\n> > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > + still be changing quite rapidly.\n> >\n> > I think we need to decide with a bit more details what constitute a\n> > \"startup frame\" and what doesn't, or we will have all kinds of\n> > inconsistent behaviour.\n> >\n> > I read it that we have multiple criteria on which we base the decision:\n> >\n> > - output images not being \"reliable\"\n> > - 3A algorithms convergence\n> > - \"possibly others\"\n> >\n> > The second criteria is fairly clear, but I'm thinking we should possibly\n> > exclude it from the startup frames and report convergence of the 3A\n> > algorithms instead.\n>\n> The 3A \"startup\" convergence is included as a criteria here because during\n> startup, we drive the AE/AWB/ALSC as aggressively (i.e. no filtering) as\n> possible to achieve convergence as fast as we possibly can. This means the\n> image output can oscillate quite badly - hence applications should avoid\n> displaying or consuming them.\n>\n> I feel that this startup phase needs to be treated differently compared to a\n> normal \"converging\" phase where the algorithms are filtering the outputs and the\n> transitions our smooth, and conversely the application can (and probably should)\n> display/consume these frames.\n\nBasically, yes. The startup phase is different. For some algorithms we\nindicate \"converged\" already, but that doesn't mean you shouldn't\ndisplay \"unconverged\" frames - of course we do. But during startup we\ncan expect algorithms possibly to overshoot and the recommendation is\nsimply not to use them. I suppose it doesn't mean an application\ncouldn't do something different - but we want to make the recommended\nbehaviour easy.\n\n>\n> >\n> > The first and third criteria are quite vague. If I recall correctly, the\n> > first one includes bad frames from the sensor (as in completely\n> > corrupted frames, such as frames that are all black or made of random\n> > data). Those are completely unusable for applications, is there a value\n> > in still making them available instead of dropping them correctly ?\n>\n> This is what we do now (plus dropping the startup frames that may be good from\n> the sensor but still in the 3A startup phase), but I think a key goals of this\n> change is be consistent (i.e. let the application handle all frames always), and\n> also help with avoiding some framebuffer allocations to help with startup\n> convergence performance.\n>\n> Regards,\n> Naush\n>\n> >\n> > What are the other reasons a frame would be a \"startup frame\" ?\n\nThere is a genuine discussion to have here, I think. There are some\nvery clear reasons for telling an application not to use a frame - if\nit's complete garbage, for example. And there are some more \"advisory\"\nones, which is what we can have for several frames. Categorising the\n\"usability\" of a frame is certainly an idea, though I don't know if we\nwant to do that without a clear reason.\n\nBut it's also very hard to predict the behaviour of all pipeline\nhandlers in this respect. Perhaps someone has a pipeline handler where\nthe first frame after every mode switch comes out badly, perhaps it\nneeds to consume a frame first for its IPAs to sort themselves out.\nBut maybe another PH doesn't. So the idea is to have a portable way to\nindicate this kind of thing so that applications don't have to start\nguessing the behaviour underneath.\n\nNote that these \"behaviours\" can be quite complex. For us, it's\ncompletely different if you request fixed exposure and gain, and\ncolour gains. But only slightly different, IIRC, if you don't give the\ncolour gains. Forcing this kind of stuff into applications,\nparticularly ones that expect to use different PHs, feels quite\nundesirable.\n\nDon't know if I've really added anything, but I hope it makes a bit of sense!\n\nThanks\nDavid\n\n> >\n> > > > +\n> > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > + occur when the camera system starts for the first time, although,\n> > > > + depending on the sensor and the implementation, they could occur at\n> > > > + other times.\n> > > > +\n> > > > + The value false indicates that this is a normal frame.\n> > >\n> > > Just throwing it out there, but would it be useful if this control was an\n> > > integer with the count of startup frames left to handle? A value of 0, or the\n> > > absence of the control would indicate this is a \"valid\" frame.\n> > >\n> > > > +\n> > > > # ----------------------------------------------------------------------------\n> > > > # Draft controls section\n> > > >\n> >\n> > --\n> > Regards,\n> >\n> > Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5BA6DC3220\n\tfor ;\n\tMon, 12 Jun 2023 09:44:07 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9D6C561E4B;\n\tMon, 12 Jun 2023 11:44:06 +0200 (CEST)","from mail-oa1-x2a.google.com (mail-oa1-x2a.google.com\n\t[IPv6:2001:4860:4864:20::2a])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AEA6361E48\n\tfor ;\n\tMon, 12 Jun 2023 11:44:04 +0200 (CEST)","by mail-oa1-x2a.google.com with SMTP id\n\t586e51a60fabf-1a67ea77c3bso956915fac.3\n\tfor ;\n\tMon, 12 Jun 2023 02:44:04 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1686563046;\n\tbh=NRFSexICuFGd47pxtL+WZ4AFb+n8VslaOgJoQD/ph8U=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=Pqxd96EkVXf0yL50gISs0KbKzNCqyIH3GGrBDy043JlGP0rGBGpCcMaXhQk/tvWw6\n\tUunMUKQMNLUpEsYOLNCuf7FNTo9mddWfDM4gs/Nba6hZbuxczmI/COLeOSxO+FpjNf\n\tWGh3ELVm+zA48qnzaFDo/XCCiUm4fMw3jA5qN9IoqqcKC2CwOp1/6K4GqCnm7dpXzr\n\tpWn0LGOlfer2GcqkM5Rv+5CvTFu6Ho5S39ewwDsOSNdqXPOjUxvCuKjrAtm5EX+Iow\n\tPvqrRr9E2Yu6TdQtmeN9IVxq76tlOU88Z4VNmCydRZWriCF6hPNhTcyvDEL1EZ8Noi\n\tjrXKpIYeOkY4A==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1686563043; x=1689155043;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=P/y305QAQtF+EGxkGlTr0CQzMQGGlWwo4oi5y4r5BhY=;\n\tb=IciDiZ06aFsAXP/QgMIgeSaDqt4PBb8hFMhmFhoocXs9K290I5llOJWYiGp3F9iru0\n\tu6nU46t+2sqDyD/V8tmBTsiZvTTIN7njSQLkG9n7kn1IFj4D7QXI6wCXagBE5MPwAK/X\n\t36W2GQylwQFD2vGrhn48qbzUn//iD+TxGDaBRh3NUy8FClC3A7kDkI4lmRYExygrYdX7\n\tDqUKe12tY6jhDD4tbryjNl/AW0LvFQ6N071TRYPWEc3KIQLnmUgt/KWq3p+YCwXKhml+\n\tWGYN9GG1gQP1oINSCwGS7dxbujo3BozR9RGA9eVVLZd0f3jKCr9694LfdocRz3XBReUV\n\thB1Q=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"IciDiZ06\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1686563043; x=1689155043;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=P/y305QAQtF+EGxkGlTr0CQzMQGGlWwo4oi5y4r5BhY=;\n\tb=NuztRZlXRwysNu5yTPldJERG9gBno2ECFKpX4iyVLf7Urh5HZEi0xGdvD4nuqxHyGY\n\tNy12iEg2V2y/vouh+Kpq1CPWGyTZIBvBX9E+IABoiMYu+RNWzfHgP2TqMEC2kA0D3eTS\n\tzVKKihY5wSXWbBAfCtoyDZ/GTtT5YVC9IMWjICM1cIxAxBoerO/d/5wMRCnwT5n7TWb3\n\twslii0RemkGOtW+qOgQnikxbnyMcUGIPdsMYV1nErJ1NNQbZ30Vtkv0CZAii082PFqaW\n\tspsWxLHC2xBCMcuJjaTfeYNfFWwklBiJO3Peu9Ak9IYJ1o45vsKk8vl2m73I7mN/aktJ\n\tnsrA==","X-Gm-Message-State":"AC+VfDwYcod3xeHg2tusDP/mMsYOod+jF2kCv4Lc2mml01jP/wGluQ/d\n\tG7LewNqCemeQd41eV7Xv3mzmL0o/x+OGDFKX71YsOmqWDQns2gOkUlA=","X-Google-Smtp-Source":"ACHHUZ4INLnMGYON6fP0sCsnoew8AJ0GL+HPq55Q3DxyDeXGU3wSUOOVzINwxupMEgdWtRhe+t4/nPaddHQ3goVdzI0=","X-Received":"by 2002:a05:6870:3a27:b0:1a6:6e31:1c36 with SMTP id\n\tdu39-20020a0568703a2700b001a66e311c36mr1939718oab.7.1686563043413;\n\tMon, 12 Jun 2023 02:44:03 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<20230605081739.GC30841@pendragon.ideasonboard.com>\n\t","In-Reply-To":"","Date":"Mon, 12 Jun 2023 10:43:52 +0100","Message-ID":"","To":"Naushir Patuck ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27324,"web_url":"https://patchwork.libcamera.org/comment/27324/","msgid":"<168660919148.2127140.3112212521021418025@Monstersaurus>","date":"2023-06-12T22:33:11","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Quoting David Plowman via libcamera-devel (2023-06-12 10:43:52)\n> Hi again\n> \n> Thanks for the discussion on this! Mostly I probably want to reiterate\n> what Naush has said.\n> \n> On Mon, 5 Jun 2023 at 10:32, Naushir Patuck wrote:\n> >\n> > Hi Laurent,\n> >\n> > David is away this week so his reply will be delayed.\n> >\n> > On Mon, 5 Jun 2023 at 09:17, Laurent Pinchart\n> > wrote:\n> > >\n> > > Hello,\n> > >\n> > > On Wed, May 31, 2023 at 01:57:16PM +0100, Naushir Patuck via libcamera-devel wrote:\n> > > > Hi David,\n> > > >\n> > > > Thank you for this patch. Indeed removing the drop frame logic the pipeline\n> > > > handler would simplify things!\n> > >\n> > > That's a change I would welcome :-)\n> > >\n> > > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > > > >\n> > > > > This control is passed back in a frame as metadata to indicate whether\n> > > > > the camera system is still in a startup phase, and the application is\n> > > > > advised to avoid using the frame.\n> > > > >\n> > > > > Signed-off-by: David Plowman \n> > > > > ---\n> > > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > > 1 file changed, 15 insertions(+)\n> > > > >\n> > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > index adea5f90..4742d907 100644\n> > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > @@ -694,6 +694,21 @@ controls:\n> > > > > Continuous AF is paused. No further state changes or lens movements\n> > > > > will occur until the AfPauseResume control is sent.\n> > > > >\n> > > > > + - StartupFrame:\n> > > > > + type: bool\n> > > > > + description: |\n> > > > > + The value true indicates that the camera system is still in a startup\n> > > > > + phase where the output images may not be reliable, or that certain of\n> > > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > > + still be changing quite rapidly.\n> > >\n> > > I think we need to decide with a bit more details what constitute a\n> > > \"startup frame\" and what doesn't, or we will have all kinds of\n> > > inconsistent behaviour.\n> > >\n> > > I read it that we have multiple criteria on which we base the decision:\n> > >\n> > > - output images not being \"reliable\"\n> > > - 3A algorithms convergence\n> > > - \"possibly others\"\n> > >\n> > > The second criteria is fairly clear, but I'm thinking we should possibly\n> > > exclude it from the startup frames and report convergence of the 3A\n> > > algorithms instead.\n> >\n> > The 3A \"startup\" convergence is included as a criteria here because during\n> > startup, we drive the AE/AWB/ALSC as aggressively (i.e. no filtering) as\n> > possible to achieve convergence as fast as we possibly can. This means the\n> > image output can oscillate quite badly - hence applications should avoid\n> > displaying or consuming them.\n> >\n> > I feel that this startup phase needs to be treated differently compared to a\n> > normal \"converging\" phase where the algorithms are filtering the outputs and the\n> > transitions our smooth, and conversely the application can (and probably should)\n> > display/consume these frames.\n> \n> Basically, yes. The startup phase is different. For some algorithms we\n> indicate \"converged\" already, but that doesn't mean you shouldn't\n> display \"unconverged\" frames - of course we do. But during startup we\n> can expect algorithms possibly to overshoot and the recommendation is\n> simply not to use them. I suppose it doesn't mean an application\n> couldn't do something different - but we want to make the recommended\n> behaviour easy.\n\nI'm pleased to see this metadata introduction. I already see visible\n'flashes' on the RkISP pipeline handler in my tests as the frames aren't\ndropped there, nor are they reported to the applications as being\nnot-yet-ready for consumption.\n\nI know in the RPi pipeline handler it's well supported to capture a\nsingle frame with pre-set manual controls. I guess we would expect\neverything to be reported as converged in that instance, so it wouldn't\nget marked as a StartupFrame? (Just trying to see if I can find odd\ncorner cases, but I expect this would already be fine).\n\n\n\n> \n> >\n> > >\n> > > The first and third criteria are quite vague. If I recall correctly, the\n> > > first one includes bad frames from the sensor (as in completely\n> > > corrupted frames, such as frames that are all black or made of random\n> > > data). Those are completely unusable for applications, is there a value\n> > > in still making them available instead of dropping them correctly ?\n> >\n> > This is what we do now (plus dropping the startup frames that may be good from\n> > the sensor but still in the 3A startup phase), but I think a key goals of this\n> > change is be consistent (i.e. let the application handle all frames always), and\n> > also help with avoiding some framebuffer allocations to help with startup\n> > convergence performance.\n> >\n> > Regards,\n> > Naush\n> >\n> > >\n> > > What are the other reasons a frame would be a \"startup frame\" ?\n> \n> There is a genuine discussion to have here, I think. There are some\n> very clear reasons for telling an application not to use a frame - if\n> it's complete garbage, for example. And there are some more \"advisory\"\n> ones, which is what we can have for several frames. Categorising the\n> \"usability\" of a frame is certainly an idea, though I don't know if we\n> want to do that without a clear reason.\n\nI expect minimising latency to getting the first 'usable' frame to be a\npriority on occasions here - so if applications can know some quantative\nlevel of detail about 'how' usable the frame is, that could potetnially\nreduce the number of frames an appliction might discard in some use\ncases.\n\nI'm not sure how easy 'quantifying' the usability of the frame will be\nthough.\n\n\n\n> \n> But it's also very hard to predict the behaviour of all pipeline\n> handlers in this respect. Perhaps someone has a pipeline handler where\n> the first frame after every mode switch comes out badly, perhaps it\n> needs to consume a frame first for its IPAs to sort themselves out.\n> But maybe another PH doesn't. So the idea is to have a portable way to\n> indicate this kind of thing so that applications don't have to start\n> guessing the behaviour underneath.\n> \n> Note that these \"behaviours\" can be quite complex. For us, it's\n> completely different if you request fixed exposure and gain, and\n> colour gains. But only slightly different, IIRC, if you don't give the\n> colour gains. Forcing this kind of stuff into applications,\n> particularly ones that expect to use different PHs, feels quite\n> undesirable.\n> \n> Don't know if I've really added anything, but I hope it makes a bit of sense!\n> \n> Thanks\n> David\n> \n> > >\n> > > > > +\n> > > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > > + occur when the camera system starts for the first time, although,\n> > > > > + depending on the sensor and the implementation, they could occur at\n> > > > > + other times.\n> > > > > +\n> > > > > + The value false indicates that this is a normal frame.\n\nPresumably an application can also 'assume' that a lack of this metadata\nwould also indicate 'false' ...\n\n> > > >\n> > > > Just throwing it out there, but would it be useful if this control was an\n> > > > integer with the count of startup frames left to handle? A value of 0, or the\n> > > > absence of the control would indicate this is a \"valid\" frame.\n\nAh yes - that's what I mean ;-)\n\n\n\n> > > >\n> > > > > +\n> > > > > # ----------------------------------------------------------------------------\n> > > > > # Draft controls section\n> > > > >\n> > >\n> > > --\n> > > Regards,\n> > >\n> > > Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0797CC31E9\n\tfor ;\n\tMon, 12 Jun 2023 22:33:18 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1D50E61E50;\n\tTue, 13 Jun 2023 00:33:17 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 801B96020C\n\tfor ;\n\tTue, 13 Jun 2023 00:33:14 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(aztw-30-b2-v4wan-166917-cust845.vm26.cable.virginm.net\n\t[82.37.23.78])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 9C517C7F;\n\tTue, 13 Jun 2023 00:32:44 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1686609197;\n\tbh=jhm74uB55onXklBo/hYwBzTMcRvCEZnH2vtG3qz16jE=;\n\th=In-Reply-To:References:To:Date:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=brY8ZHrEsdeNVXrEhlvm4PMtaVE0sUCE+YWG5BPlHlsMfwIW58GrnPBpdH1ywTdXs\n\tv2HF3iIdiutU1/ZxDyFuUC0/Bb2ZvY+BsNPBCm9Qa4jnAFPG/r1FDYs8fChN3tfc0R\n\tubWsFevYxOS9NLClIFkxY6EOPzIrd4ehAzkB0u52kc/tCNE2V09/SWD/8KkKsifPMw\n\txD0JA9Us8G8HbMndHzJxdyyCbDqC4FGEVKMIhrMnj5zUR3K+ZvqrUaVGTX499+yCNH\n\tkfGadPIqelPdRAZIIuJwqZr8B6jAt90avQpIFPfHK+YXPqJBpyaLpadNe+rigSgaDu\n\t13Nd5B5OodKag==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1686609164;\n\tbh=jhm74uB55onXklBo/hYwBzTMcRvCEZnH2vtG3qz16jE=;\n\th=In-Reply-To:References:Subject:From:Cc:To:Date:From;\n\tb=K6By6QFdKylSzjrP8MIIg3Knz55kSMaHJD7jWVUqC+geVQFYfcNvGWt5+M+vTS/VV\n\t3q2F/dVXkKQay7hA/xTyeqseTvol5Jpx2oEB9o/nGrHIW3quit+tsQUITDnFbxQ/va\n\tVNdXq/8tCY5ritPUoPQyS/WKcP9ia9F2vub2MuyI="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"K6By6QFd\"; dkim-atps=neutral","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<20230605081739.GC30841@pendragon.ideasonboard.com>\n\t\n\t","To":"David Plowman ,\n\tDavid Plowman via libcamera-devel , \n\tNaushir Patuck ","Date":"Mon, 12 Jun 2023 23:33:11 +0100","Message-ID":"<168660919148.2127140.3112212521021418025@Monstersaurus>","User-Agent":"alot/0.10","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Kieran Bingham via libcamera-devel\n\t","Reply-To":"Kieran Bingham ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27328,"web_url":"https://patchwork.libcamera.org/comment/27328/","msgid":"","date":"2023-06-13T08:19:25","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Kieran,\n\nOn Mon, 12 Jun 2023 at 23:33, Kieran Bingham\n wrote:\n>\n> Quoting David Plowman via libcamera-devel (2023-06-12 10:43:52)\n> > Hi again\n> >\n> > Thanks for the discussion on this! Mostly I probably want to reiterate\n> > what Naush has said.\n> >\n> > On Mon, 5 Jun 2023 at 10:32, Naushir Patuck wrote:\n> > >\n> > > Hi Laurent,\n> > >\n> > > David is away this week so his reply will be delayed.\n> > >\n> > > On Mon, 5 Jun 2023 at 09:17, Laurent Pinchart\n> > > wrote:\n> > > >\n> > > > Hello,\n> > > >\n> > > > On Wed, May 31, 2023 at 01:57:16PM +0100, Naushir Patuck via libcamera-devel wrote:\n> > > > > Hi David,\n> > > > >\n> > > > > Thank you for this patch. Indeed removing the drop frame logic the pipeline\n> > > > > handler would simplify things!\n> > > >\n> > > > That's a change I would welcome :-)\n> > > >\n> > > > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > > > > >\n> > > > > > This control is passed back in a frame as metadata to indicate whether\n> > > > > > the camera system is still in a startup phase, and the application is\n> > > > > > advised to avoid using the frame.\n> > > > > >\n> > > > > > Signed-off-by: David Plowman \n> > > > > > ---\n> > > > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > > > 1 file changed, 15 insertions(+)\n> > > > > >\n> > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > > index adea5f90..4742d907 100644\n> > > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > > @@ -694,6 +694,21 @@ controls:\n> > > > > > Continuous AF is paused. No further state changes or lens movements\n> > > > > > will occur until the AfPauseResume control is sent.\n> > > > > >\n> > > > > > + - StartupFrame:\n> > > > > > + type: bool\n> > > > > > + description: |\n> > > > > > + The value true indicates that the camera system is still in a startup\n> > > > > > + phase where the output images may not be reliable, or that certain of\n> > > > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > > > + still be changing quite rapidly.\n> > > >\n> > > > I think we need to decide with a bit more details what constitute a\n> > > > \"startup frame\" and what doesn't, or we will have all kinds of\n> > > > inconsistent behaviour.\n> > > >\n> > > > I read it that we have multiple criteria on which we base the decision:\n> > > >\n> > > > - output images not being \"reliable\"\n> > > > - 3A algorithms convergence\n> > > > - \"possibly others\"\n> > > >\n> > > > The second criteria is fairly clear, but I'm thinking we should possibly\n> > > > exclude it from the startup frames and report convergence of the 3A\n> > > > algorithms instead.\n> > >\n> > > The 3A \"startup\" convergence is included as a criteria here because during\n> > > startup, we drive the AE/AWB/ALSC as aggressively (i.e. no filtering) as\n> > > possible to achieve convergence as fast as we possibly can. This means the\n> > > image output can oscillate quite badly - hence applications should avoid\n> > > displaying or consuming them.\n> > >\n> > > I feel that this startup phase needs to be treated differently compared to a\n> > > normal \"converging\" phase where the algorithms are filtering the outputs and the\n> > > transitions our smooth, and conversely the application can (and probably should)\n> > > display/consume these frames.\n> >\n> > Basically, yes. The startup phase is different. For some algorithms we\n> > indicate \"converged\" already, but that doesn't mean you shouldn't\n> > display \"unconverged\" frames - of course we do. But during startup we\n> > can expect algorithms possibly to overshoot and the recommendation is\n> > simply not to use them. I suppose it doesn't mean an application\n> > couldn't do something different - but we want to make the recommended\n> > behaviour easy.\n>\n> I'm pleased to see this metadata introduction. I already see visible\n> 'flashes' on the RkISP pipeline handler in my tests as the frames aren't\n> dropped there, nor are they reported to the applications as being\n> not-yet-ready for consumption.\n>\n> I know in the RPi pipeline handler it's well supported to capture a\n> single frame with pre-set manual controls. I guess we would expect\n> everything to be reported as converged in that instance, so it wouldn't\n> get marked as a StartupFrame? (Just trying to see if I can find odd\n> corner cases, but I expect this would already be fine).\n\nIf the user requests fully manual controls (i.e. shutter speed, analogue gain,\nmanual red/blue gains), then we don't signal any drop frames for algorithm\nconvergence. However, we would still signal drop frames if the sensor is\nknown to produce N garbage frames on startup.\n\nRegards,\nNaush\n\n\n>\n>\n>\n> >\n> > >\n> > > >\n> > > > The first and third criteria are quite vague. If I recall correctly, the\n> > > > first one includes bad frames from the sensor (as in completely\n> > > > corrupted frames, such as frames that are all black or made of random\n> > > > data). Those are completely unusable for applications, is there a value\n> > > > in still making them available instead of dropping them correctly ?\n> > >\n> > > This is what we do now (plus dropping the startup frames that may be good from\n> > > the sensor but still in the 3A startup phase), but I think a key goals of this\n> > > change is be consistent (i.e. let the application handle all frames always), and\n> > > also help with avoiding some framebuffer allocations to help with startup\n> > > convergence performance.\n> > >\n> > > Regards,\n> > > Naush\n> > >\n> > > >\n> > > > What are the other reasons a frame would be a \"startup frame\" ?\n> >\n> > There is a genuine discussion to have here, I think. There are some\n> > very clear reasons for telling an application not to use a frame - if\n> > it's complete garbage, for example. And there are some more \"advisory\"\n> > ones, which is what we can have for several frames. Categorising the\n> > \"usability\" of a frame is certainly an idea, though I don't know if we\n> > want to do that without a clear reason.\n>\n> I expect minimising latency to getting the first 'usable' frame to be a\n> priority on occasions here - so if applications can know some quantative\n> level of detail about 'how' usable the frame is, that could potetnially\n> reduce the number of frames an appliction might discard in some use\n> cases.\n>\n> I'm not sure how easy 'quantifying' the usability of the frame will be\n> though.\n>\n>\n>\n> >\n> > But it's also very hard to predict the behaviour of all pipeline\n> > handlers in this respect. Perhaps someone has a pipeline handler where\n> > the first frame after every mode switch comes out badly, perhaps it\n> > needs to consume a frame first for its IPAs to sort themselves out.\n> > But maybe another PH doesn't. So the idea is to have a portable way to\n> > indicate this kind of thing so that applications don't have to start\n> > guessing the behaviour underneath.\n> >\n> > Note that these \"behaviours\" can be quite complex. For us, it's\n> > completely different if you request fixed exposure and gain, and\n> > colour gains. But only slightly different, IIRC, if you don't give the\n> > colour gains. Forcing this kind of stuff into applications,\n> > particularly ones that expect to use different PHs, feels quite\n> > undesirable.\n> >\n> > Don't know if I've really added anything, but I hope it makes a bit of sense!\n> >\n> > Thanks\n> > David\n> >\n> > > >\n> > > > > > +\n> > > > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > > > + occur when the camera system starts for the first time, although,\n> > > > > > + depending on the sensor and the implementation, they could occur at\n> > > > > > + other times.\n> > > > > > +\n> > > > > > + The value false indicates that this is a normal frame.\n>\n> Presumably an application can also 'assume' that a lack of this metadata\n> would also indicate 'false' ...\n>\n> > > > >\n> > > > > Just throwing it out there, but would it be useful if this control was an\n> > > > > integer with the count of startup frames left to handle? A value of 0, or the\n> > > > > absence of the control would indicate this is a \"valid\" frame.\n>\n> Ah yes - that's what I mean ;-)\n>\n>\n>\n> > > > >\n> > > > > > +\n> > > > > > # ----------------------------------------------------------------------------\n> > > > > > # Draft controls section\n> > > > > >\n> > > >\n> > > > --\n> > > > Regards,\n> > > >\n> > > > Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D216AC31E9\n\tfor ;\n\tTue, 13 Jun 2023 08:19:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 43B5E61E4B;\n\tTue, 13 Jun 2023 10:19:42 +0200 (CEST)","from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com\n\t[IPv6:2607:f8b0:4864:20::112b])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8DB7261E47\n\tfor ;\n\tTue, 13 Jun 2023 10:19:40 +0200 (CEST)","by mail-yw1-x112b.google.com with SMTP id\n\t00721157ae682-56d06711fccso21573917b3.2\n\tfor ;\n\tTue, 13 Jun 2023 01:19:40 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1686644382;\n\tbh=H7QUktCjKGR9CafZfrz3vrDPAF1lPqQ7YFj++RjuEFc=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=GbpBCBsxfDp2Ush4xqYgaDfnhoca1DrfEWc17FdMJpucIWSV/SfPnQ4+TKxZk1CLq\n\tjM8R/APMumCOR0lnmdTZMuGzPBrL9KNkgaxejt4Dk4Dg64k/6j7iI3cA545g1M0Ngl\n\tZ+ycvL0BsCSDz3e/pCK4OBOecPuuV6WOMBobT8NICzMkXQgM9lNgv4fmJmmAQDHiaz\n\tWy0Zj2Btr+qctElIKDVNlHLaHV/gsWopo/QBb7Xf4H6Bq2vigqDhwbDPIlxh169YMF\n\tNdBbW/o5hgPrVWZBwHK6b8JpG2+jfhS8COcq6m+lRufi+L7Cl1guEJSzQ8QkTLlTmy\n\tdPF/FeXCBh92g==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1686644379; x=1689236379;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=IJwg0ikpmylpYcHHAJaXYrjfY+t/wjBzcOIJiru58oM=;\n\tb=RukMhZ0b41ntiWO1pB1fNgVpEiA596x07iD2KFZCcjvca5h2MvN9Vn+/SzMP3aQMTH\n\tlzb7e6yq3iYuAJPRLbSyVXuSK/uBIgEzJaIVFZEoxyudoGkGq59zCeUfb/5Zk/ZUc2S8\n\tce1B+2Mc54ojI8mCPius7e0U8y4pwbD/yrFerSfmXJaiIFt3edU699fOOrn0zoXlmIbl\n\t4JlMB1cCDmpVcceCZhN6r8K5DUSOHkWNhFFYnDTeVRKkP16SA/iTeCXCdXsPFiXX5arp\n\tBj7XmpgCv1jBtHs/9shcXUXfffYvBU/ojBLKveVCJtDaBoosB8BrZaaqbHcdAC0nplt1\n\tf1fw=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"RukMhZ0b\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1686644379; x=1689236379;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=IJwg0ikpmylpYcHHAJaXYrjfY+t/wjBzcOIJiru58oM=;\n\tb=JVfYq2C15Gd/bD5ZARveBZBRiH9SFrNfYrknD9VjzvDGhfQtH0fJZgq3ExE9K2BLy5\n\t/Mur6TkUAEhBDc1lMDIjImLOAWFJI5NE5I1SHuvV5xpdRF/H9hxFnUfUtF1zAzVTJTuj\n\tRaDKmKxj9U3/sg40da+gQlVxpmoDPa4/+bvk4p4ys7d9fVH19PAoTFuFALwc+FNbkJUI\n\tUHTgUaJcIHD4sgjRSlbLO42Eke/yodLdfQKFjCVQ+xkC7n/0ACSweHZWtHMvW9aGr0xk\n\tUZWUFfDdCFMw+KlLC0LHzle/MVciXXeKYvDCKLZtmkYuWNyaEgnoL7lwb0V3+MorhrkN\n\t1FRw==","X-Gm-Message-State":"AC+VfDw2hVXoOLAp9rlHqrtBuytQwXfPsxaPMg5o4nE0ZXwp6OlltljQ\n\t6yTM4XtUbkKQRi9UOv/adi+2RS/H7Bd+bOp79GL6Yg==","X-Google-Smtp-Source":"ACHHUZ4MS3Kqpoa0Sy0KBxPro0IR5AwuL2mqUEkOXhbqZPThGfZm8cM0D4Flj596wafZkXEeV1TCSntAWsTbHqGhlY4=","X-Received":"by 2002:a0d:eaca:0:b0:56d:6c9:d82d with SMTP id\n\tt193-20020a0deaca000000b0056d06c9d82dmr1069389ywe.21.1686644379263;\n\tTue, 13 Jun 2023 01:19:39 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<20230605081739.GC30841@pendragon.ideasonboard.com>\n\t\n\t\n\t<168660919148.2127140.3112212521021418025@Monstersaurus>","In-Reply-To":"<168660919148.2127140.3112212521021418025@Monstersaurus>","Date":"Tue, 13 Jun 2023 09:19:25 +0100","Message-ID":"","To":"Kieran Bingham ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"David Plowman via libcamera-devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27450,"web_url":"https://patchwork.libcamera.org/comment/27450/","msgid":"<20230703134134.GA21480@pendragon.ideasonboard.com>","date":"2023-07-03T13:41:34","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hello,\n\nOn Tue, Jun 13, 2023 at 09:19:25AM +0100, Naushir Patuck via libcamera-devel wrote:\n> On Mon, 12 Jun 2023 at 23:33, Kieran Bingham wrote:\n> > Quoting David Plowman via libcamera-devel (2023-06-12 10:43:52)\n> > > On Mon, 5 Jun 2023 at 10:32, Naushir Patuck wrote:\n> > > > On Mon, 5 Jun 2023 at 09:17, Laurent Pinchart wrote:\n> > > > > On Wed, May 31, 2023 at 01:57:16PM +0100, Naushir Patuck via libcamera-devel wrote:\n> > > > > > Hi David,\n> > > > > >\n> > > > > > Thank you for this patch. Indeed removing the drop frame logic the pipeline\n> > > > > > handler would simplify things!\n> > > > >\n> > > > > That's a change I would welcome :-)\n> > > > >\n> > > > > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > > > > > >\n> > > > > > > This control is passed back in a frame as metadata to indicate whether\n> > > > > > > the camera system is still in a startup phase, and the application is\n> > > > > > > advised to avoid using the frame.\n> > > > > > >\n> > > > > > > Signed-off-by: David Plowman \n> > > > > > > ---\n> > > > > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > > > > 1 file changed, 15 insertions(+)\n> > > > > > >\n> > > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > > > index adea5f90..4742d907 100644\n> > > > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > > > @@ -694,6 +694,21 @@ controls:\n> > > > > > > Continuous AF is paused. No further state changes or lens movements\n> > > > > > > will occur until the AfPauseResume control is sent.\n> > > > > > >\n> > > > > > > + - StartupFrame:\n> > > > > > > + type: bool\n> > > > > > > + description: |\n> > > > > > > + The value true indicates that the camera system is still in a startup\n> > > > > > > + phase where the output images may not be reliable, or that certain of\n> > > > > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > > > > + still be changing quite rapidly.\n> > > > >\n> > > > > I think we need to decide with a bit more details what constitute a\n> > > > > \"startup frame\" and what doesn't, or we will have all kinds of\n> > > > > inconsistent behaviour.\n> > > > >\n> > > > > I read it that we have multiple criteria on which we base the decision:\n> > > > >\n> > > > > - output images not being \"reliable\"\n> > > > > - 3A algorithms convergence\n> > > > > - \"possibly others\"\n> > > > >\n> > > > > The second criteria is fairly clear, but I'm thinking we should possibly\n> > > > > exclude it from the startup frames and report convergence of the 3A\n> > > > > algorithms instead.\n> > > >\n> > > > The 3A \"startup\" convergence is included as a criteria here because during\n> > > > startup, we drive the AE/AWB/ALSC as aggressively (i.e. no filtering) as\n> > > > possible to achieve convergence as fast as we possibly can. This means the\n> > > > image output can oscillate quite badly - hence applications should avoid\n> > > > displaying or consuming them.\n> > > >\n> > > > I feel that this startup phase needs to be treated differently compared to a\n> > > > normal \"converging\" phase where the algorithms are filtering the outputs and the\n> > > > transitions our smooth, and conversely the application can (and probably should)\n> > > > display/consume these frames.\n> > >\n> > > Basically, yes. The startup phase is different. For some algorithms we\n> > > indicate \"converged\" already, but that doesn't mean you shouldn't\n> > > display \"unconverged\" frames - of course we do.\n\nI don't get this.\n\n> > > But during startup we\n> > > can expect algorithms possibly to overshoot and the recommendation is\n> > > simply not to use them.\n\nThat I agree with.\n\n> > > I suppose it doesn't mean an application\n> > > couldn't do something different - but we want to make the recommended\n> > > behaviour easy.\n\nApplications can easily differentiate the startup phase from the rest of\nthe camera operation, as startup occurs, by definition, at startup :-)\nWe could thus tell applications that they should ignore unconverged\nframes immediately after starting the camera, but not later.\n\nWhat I can't tell at the moment is whether the algorithm convergence is\nthe right criteria at start time. I can imagine that the convergence\nphase could be split into a short initial part with large oscillations,\nand a second part with smoother transitions until full convergence is\nreached. Now that I've written this I of course expect someone to tell\nthat they absolutely need to differenciate between the two, but is it\nneeded for real ?\n\n> > I'm pleased to see this metadata introduction. I already see visible\n> > 'flashes' on the RkISP pipeline handler in my tests as the frames aren't\n> > dropped there, nor are they reported to the applications as being\n> > not-yet-ready for consumption.\n> >\n> > I know in the RPi pipeline handler it's well supported to capture a\n> > single frame with pre-set manual controls. I guess we would expect\n> > everything to be reported as converged in that instance, so it wouldn't\n> > get marked as a StartupFrame? (Just trying to see if I can find odd\n> > corner cases, but I expect this would already be fine).\n> \n> If the user requests fully manual controls (i.e. shutter speed, analogue gain,\n> manual red/blue gains), then we don't signal any drop frames for algorithm\n> convergence. However, we would still signal drop frames if the sensor is\n> known to produce N garbage frames on startup.\n> \n> > > > > The first and third criteria are quite vague. If I recall correctly, the\n> > > > > first one includes bad frames from the sensor (as in completely\n> > > > > corrupted frames, such as frames that are all black or made of random\n> > > > > data). Those are completely unusable for applications, is there a value\n> > > > > in still making them available instead of dropping them correctly ?\n> > > >\n> > > > This is what we do now (plus dropping the startup frames that may be good from\n> > > > the sensor but still in the 3A startup phase), but I think a key goals of this\n> > > > change is be consistent (i.e. let the application handle all frames always),\n\nIf the sensor produces frames that can't be used in any circumstance, I\nreally see no value in exposing them to applications. From an\napplication point of view this wouldn't be an inconsistent behaviour,\nthose frames would simply not exist.\n\n> > > > and\n> > > > also help with avoiding some framebuffer allocations to help with startup\n> > > > convergence performance.\n\nI'm not sure to get this.\n\n> > > > > What are the other reasons a frame would be a \"startup frame\" ?\n> > >\n> > > There is a genuine discussion to have here, I think. There are some\n> > > very clear reasons for telling an application not to use a frame - if\n> > > it's complete garbage, for example. And there are some more \"advisory\"\n> > > ones, which is what we can have for several frames. Categorising the\n> > > \"usability\" of a frame is certainly an idea, though I don't know if we\n> > > want to do that without a clear reason.\n> >\n> > I expect minimising latency to getting the first 'usable' frame to be a\n> > priority on occasions here - so if applications can know some quantative\n> > level of detail about 'how' usable the frame is, that could potetnially\n> > reduce the number of frames an appliction might discard in some use\n> > cases.\n> >\n> > I'm not sure how easy 'quantifying' the usability of the frame will be\n> > though.\n\nI can't imagine a way to do so at the moment, but I'd be happy to hear\nproposals :-)\n\n> > > But it's also very hard to predict the behaviour of all pipeline\n> > > handlers in this respect. Perhaps someone has a pipeline handler where\n> > > the first frame after every mode switch comes out badly,\n\nWe need to discuss how \"mode switch\" and \"startup\" interact. From the\npoint of view of the application, the libcamera API doesn't expose a\n\"mode switch\" concept. We can stop, reconfigure and restart the camera,\nand the documented behaviour doesn't differ from the first start. I know\nhow IPA modules can take advantage of information from the previous\ncapture session to accelerate algorithm convergence, and they should do\nso as an internal optimization, but I don't think I want to expose this\nconcept explicitly. In particular, I don't like the \"when the camera\nsystem starts for the first time\" in the documentation.\n\n> > > perhaps it\n> > > needs to consume a frame first for its IPAs to sort themselves out.\n> > > But maybe another PH doesn't. So the idea is to have a portable way to\n> > > indicate this kind of thing so that applications don't have to start\n> > > guessing the behaviour underneath.\n> > >\n> > > Note that these \"behaviours\" can be quite complex. For us, it's\n> > > completely different if you request fixed exposure and gain, and\n> > > colour gains. But only slightly different, IIRC, if you don't give the\n> > > colour gains. Forcing this kind of stuff into applications,\n> > > particularly ones that expect to use different PHs, feels quite\n> > > undesirable.\n\nI'm not following here either, I don't see what you're forcing onto\napplications here.\n\n> > > Don't know if I've really added anything, but I hope it makes a bit of sense!\n> > >\n> > > > > > > +\n> > > > > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > > > > + occur when the camera system starts for the first time, although,\n> > > > > > > + depending on the sensor and the implementation, they could occur at\n> > > > > > > + other times.\n> > > > > > > +\n> > > > > > > + The value false indicates that this is a normal frame.\n> >\n> > Presumably an application can also 'assume' that a lack of this metadata\n> > would also indicate 'false' ...\n> >\n> > > > > >\n> > > > > > Just throwing it out there, but would it be useful if this control was an\n> > > > > > integer with the count of startup frames left to handle? A value of 0, or the\n> > > > > > absence of the control would indicate this is a \"valid\" frame.\n> >\n> > Ah yes - that's what I mean ;-)\n> >\n> > > > > >\n> > > > > > > +\n> > > > > > > # ----------------------------------------------------------------------------\n> > > > > > > # Draft controls section\n> > > > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 66959BE175\n\tfor ;\n\tMon, 3 Jul 2023 13:41:36 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B60C7628BC;\n\tMon, 3 Jul 2023 15:41:35 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 07FAF628BB\n\tfor ;\n\tMon, 3 Jul 2023 15:41:34 +0200 (CEST)","from pendragon.ideasonboard.com (85-160-45-219.reb.o2.cz\n\t[85.160.45.219])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 207EA6C8;\n\tMon, 3 Jul 2023 15:40:50 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1688391695;\n\tbh=qDI1tIhCBBz+qYR9uQP1qW4nEQ1ZNSlYgoHznPJg8Vg=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=nuPp4TNrAxVIAaEVnpAEhJUNavXhm+TE8UVsVjcRzCcH9sm+ZYhDDzjQD1Hs5cySm\n\tF/qtzJSZYq/3YpM4NJ0EMoh5vWB3kcuTn/vSUftapnrcLx9R8XdqoXSKOXA6Z3xyMs\n\tPIjcvTBc0X8U9nwQOgql6AIfL4JwD3UG8JPwzBsYQzWIxKyyiMishcDYFS9VjdYsvp\n\tnDOrnJnoICaxXUEUjZkGekCPzbMyJMIz5nVEqvkhTV4SiLex4Ed2YVOn9dn13mY+jU\n\tcZRy+vn5RD5H4ZgjweVEEgI30jp/6nzNBsmXm5VLct7Mil4tsaGIING0CvlnhxZbHP\n\tbz+jKRuIDtSKg==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1688391650;\n\tbh=qDI1tIhCBBz+qYR9uQP1qW4nEQ1ZNSlYgoHznPJg8Vg=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=tmFDaP6r1djM6n1BdFrWXp3Mk7qSEiB5g8nRfTVtJiubwuAEz7wA0r/8OJRzTVFWc\n\tBVo515idzm2oP9ev/7KnSpAbbvqoO0TUEVvjUfQ47F1bwpzdU/FhdVQKyHxIysMQ/N\n\tajihezQUcRT/GKt0EOm3gyAHLH3OJeoi72z/ryeA="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"tmFDaP6r\"; dkim-atps=neutral","Date":"Mon, 3 Jul 2023 16:41:34 +0300","To":"Naushir Patuck ","Message-ID":"<20230703134134.GA21480@pendragon.ideasonboard.com>","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<20230605081739.GC30841@pendragon.ideasonboard.com>\n\t\n\t\n\t<168660919148.2127140.3112212521021418025@Monstersaurus>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Cc":"David Plowman via libcamera-devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27530,"web_url":"https://patchwork.libcamera.org/comment/27530/","msgid":"","date":"2023-07-11T10:23:32","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi all,\n\nOn a semi-related topic, we talked offline about improving the drop frame\nsupport by queuing a request buffer multiple times to avoid the need for\nallocating internal buffers. I've tried this out and here are my findings.\n\nFirstly, to handle queuing a single buffer multiple times, I need to increase\nthe number of cache slots in V4L2BufferCache(). Perhaps\nV4L2VideoDevice::importBuffers()\nshould be updated to not take in a count parameter and we just allocate slots\nfor the maximum buffer count possible in V4L2 (32 I think)? There has been a\nlong-standing \\todo in the RPi code to choose an appropriate value, and the\nmaximum number is really the only appropriate value I can think of.\n\nOnce I got this working, unfortunately I realised this method would never\nactually work correctly in the common scenario where the application configures\nand uses a RAW stream. In such cases we would queue the RAW buffer into Unicam\nN times for N dropped frames. However this buffer is also imported into the ISP\nfor processing and stats generation, all while it is also being filled by Unicam\nfor the next sensor frame. This makes the stats entirely unusable.\n\nSo in the end we either have to allocate additional buffers for drop frames\n(like we do right now), or we implement something like this series where the\napplication is responsible for dropping/ignoring these frames.\n\nRegards,\nNaush\n\n\n\n\n\nOn Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel\n wrote:\n>\n> This control is passed back in a frame as metadata to indicate whether\n> the camera system is still in a startup phase, and the application is\n> advised to avoid using the frame.\n>\n> Signed-off-by: David Plowman \n> ---\n> src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> 1 file changed, 15 insertions(+)\n>\n> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> index adea5f90..4742d907 100644\n> --- a/src/libcamera/control_ids.yaml\n> +++ b/src/libcamera/control_ids.yaml\n> @@ -694,6 +694,21 @@ controls:\n> Continuous AF is paused. No further state changes or lens movements\n> will occur until the AfPauseResume control is sent.\n>\n> + - StartupFrame:\n> + type: bool\n> + description: |\n> + The value true indicates that the camera system is still in a startup\n> + phase where the output images may not be reliable, or that certain of\n> + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> + still be changing quite rapidly.\n> +\n> + Applications are advised to avoid using these frames. Mostly, they will\n> + occur when the camera system starts for the first time, although,\n> + depending on the sensor and the implementation, they could occur at\n> + other times.\n> +\n> + The value false indicates that this is a normal frame.\n> +\n> # ----------------------------------------------------------------------------\n> # Draft controls section\n>\n> --\n> 2.30.2\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5E7E0BEFBE\n\tfor ;\n\tTue, 11 Jul 2023 10:23:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7F4EC628C0;\n\tTue, 11 Jul 2023 12:23:51 +0200 (CEST)","from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com\n\t[IPv6:2607:f8b0:4864:20::1134])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 46C3660570\n\tfor ;\n\tTue, 11 Jul 2023 12:23:50 +0200 (CEST)","by mail-yw1-x1134.google.com with SMTP id\n\t00721157ae682-5728df0a7d9so68767447b3.1\n\tfor ;\n\tTue, 11 Jul 2023 03:23:50 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1689071031;\n\tbh=R3wlbLlO513wLgE7G2HQ4IGNc4TYdb7LRBHby8/Yr08=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=Sm/FHamTZlMg2JmLYqIUispTvq9DlmyOX9s6sLG3asLCXz/3Iml5iz8XloIAK19vN\n\tlvLhyhZ/UkeX67MAlYttw6m7hPcFBwWtfPtMnvYE8d8qaAxisnaKfFnSHBG2HN+M8l\n\twIKYkcEqT78zSU+JWkblM35JXURteU6MQNpq3NZsEXqmU8rm3iMRpYw+Qqxz/Ay4du\n\tV+xpYr0a34CTslGG4JthSirkkWYYlolyTo0snD+DQrJkGGDzo0iSg6olQu8O2tE/zr\n\t13EhAM55lTkENdreHNildQgY2JDrwJeKOoZBZaNpoCTwynsKTYpTuaUu0chpOtBjo5\n\tuwXTqf3ushZMg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1689071029; x=1691663029;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=PsDAB5GvnQMfNlEy7eOLc0UHkUgURvrVfgk7xIzN6lM=;\n\tb=csLMqpZZm1EoyMRVTokE6dT7atTpZVcYqbcpGCagsgAa5hRIis0VhMgiAUqc3HYagJ\n\twENgdvk7SOqt+C0gxXghs2YEjf6F0cECzUafE8q0De6lpkbB9XnmkYxLvcX5vHtRBIry\n\tCbPS0BaT8j7MQhWsio8DRUWfNAMvx6CKIpiepAxJPcYNOLy0NwkyG9FNA5fwptWe88Aq\n\trdXzp3X4XNQOrnDwz3Fa2RrM5DHsA/oJMa23cT1eww9EevHpAzs9l6NLaWYh28MzDpfb\n\t2J34ZH2QxAcDZ2XMnVYwJ/AgjOt5/2ilNEaDWXYoZPaEWDUq/PhCsDNGSe4AyWgKtS69\n\tpV1g=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"csLMqpZZ\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1689071029; x=1691663029;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=PsDAB5GvnQMfNlEy7eOLc0UHkUgURvrVfgk7xIzN6lM=;\n\tb=dL6SSpzZbjnnyYA0ODt/3HhrnDAvoUiSMDccvqQr7/OS/4K8MG/d1Alss5827C8bic\n\tJ3wA/fCar/f/ZpsNF7P6rtbE7QBrz3rqTJ7M1517j68Joa3BuOq9Eg5Iy0REdGxLDtRY\n\tK3VRIn4073goNvye4wR+slcFRb/P1HeWHC/h6pe2UhUDvXuEQ8HbMFHWZ4l/Ua9kPJEA\n\tOt6NN6VD9SGvpQztRzcUP5Fz0BqXknC3EV7DJPtsat3nQeCBLPb8EecfplWAuit0iutm\n\twXH1IX+N6LPvKU47sbAT0CYNLLhe1M7pSy3DIKzgLEhtJDCZxXPR4XjkIckEnJqNt/Vp\n\teqig==","X-Gm-Message-State":"ABy/qLapn8IGlalQkEChakUKRRA1FZnhWd+PT9epJPNjp1V+fCgfhvM/\n\tpcfy9rdplLfOz6HsL5/wFf8ka9T8MF2YXTkmBSaqJl/wHVdKgMXFBi9Wjg==","X-Google-Smtp-Source":"APBJJlHkuNhVaYdFqzswx8C1rdaUytt2KKj0kZWlySfmHq5IbdnWmdQN0xike1gaoCYF8jw/WdpdNcVeOUUw/33ex90=","X-Received":"by 2002:a81:6c4b:0:b0:57a:295c:9103 with SMTP id\n\th72-20020a816c4b000000b0057a295c9103mr14164882ywc.33.1689071029143;\n\tTue, 11 Jul 2023 03:23:49 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>","In-Reply-To":"<20230531125016.5540-2-david.plowman@raspberrypi.com>","Date":"Tue, 11 Jul 2023 11:23:32 +0100","Message-ID":"","To":"David Plowman ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27532,"web_url":"https://patchwork.libcamera.org/comment/27532/","msgid":"<168907150298.540247.16063484386946286910@Monstersaurus>","date":"2023-07-11T10:31:42","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)\n> Hi all,\n> \n> On a semi-related topic, we talked offline about improving the drop frame\n> support by queuing a request buffer multiple times to avoid the need for\n> allocating internal buffers. I've tried this out and here are my findings.\n> \n> Firstly, to handle queuing a single buffer multiple times, I need to increase\n> the number of cache slots in V4L2BufferCache(). Perhaps\n> V4L2VideoDevice::importBuffers()\n> should be updated to not take in a count parameter and we just allocate slots\n> for the maximum buffer count possible in V4L2 (32 I think)? There has been a\n> long-standing \\todo in the RPi code to choose an appropriate value, and the\n> maximum number is really the only appropriate value I can think of.\n\nI still think allocating the maximum here in the v4l2 components is\nappropriate as they are 'cheap' ...\n\n\n\n> Once I got this working, unfortunately I realised this method would never\n> actually work correctly in the common scenario where the application configures\n> and uses a RAW stream. In such cases we would queue the RAW buffer into Unicam\n> N times for N dropped frames. However this buffer is also imported into the ISP\n> for processing and stats generation, all while it is also being filled by Unicam\n> for the next sensor frame. This makes the stats entirely unusable.\n\n\nAha that's a shame. I thought the restrictions were going to be in the\nkernel side, so at least it's interesting to know that we /can/ queue\nthe same buffer multiple times (with a distinct v4l2_buffer id) and get\nsomewhat of the expected behaviour....\n\n> \n> So in the end we either have to allocate additional buffers for drop frames\n> (like we do right now), or we implement something like this series where the\n> application is responsible for dropping/ignoring these frames.\n\nOf course if the expected behaviour doesn't suit the use case ... then\n...\n\nThis may all be different for pipeline's with an inline ISP though ...\nso the research is still useful.\n\n--\nKieran\n\n> \n> Regards,\n> Naush\n> \n> \n> \n> \n> \n> On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel\n> wrote:\n> >\n> > This control is passed back in a frame as metadata to indicate whether\n> > the camera system is still in a startup phase, and the application is\n> > advised to avoid using the frame.\n> >\n> > Signed-off-by: David Plowman \n> > ---\n> > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > 1 file changed, 15 insertions(+)\n> >\n> > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > index adea5f90..4742d907 100644\n> > --- a/src/libcamera/control_ids.yaml\n> > +++ b/src/libcamera/control_ids.yaml\n> > @@ -694,6 +694,21 @@ controls:\n> > Continuous AF is paused. No further state changes or lens movements\n> > will occur until the AfPauseResume control is sent.\n> >\n> > + - StartupFrame:\n> > + type: bool\n> > + description: |\n> > + The value true indicates that the camera system is still in a startup\n> > + phase where the output images may not be reliable, or that certain of\n> > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > + still be changing quite rapidly.\n> > +\n> > + Applications are advised to avoid using these frames. Mostly, they will\n> > + occur when the camera system starts for the first time, although,\n> > + depending on the sensor and the implementation, they could occur at\n> > + other times.\n> > +\n> > + The value false indicates that this is a normal frame.\n> > +\n> > # ----------------------------------------------------------------------------\n> > # Draft controls section\n> >\n> > --\n> > 2.30.2\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 031E8BDC71\n\tfor ;\n\tTue, 11 Jul 2023 10:31:47 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5A299628C1;\n\tTue, 11 Jul 2023 12:31:46 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 728B261E32\n\tfor ;\n\tTue, 11 Jul 2023 12:31:45 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(aztw-30-b2-v4wan-166917-cust845.vm26.cable.virginm.net\n\t[82.37.23.78])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 9D65949E;\n\tTue, 11 Jul 2023 12:30:56 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1689071506;\n\tbh=yjWGe7Rkz3BV/ouM+vx82YvP4mpbRLDoca7998oZTkY=;\n\th=In-Reply-To:References:To:Date:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=Jwt9DpHuj7XK+Iafo5BNiyyo8P+wlViDLdYHn9iWiIE5PnVDQHwXjWkMsAaaa6+eg\n\tgMpQQaeSX4lr4ZfOe9hHSMrNEVuibX2Xrxhzgn28REZUypLr1x5IjE6bJeD6BFkde/\n\t0CfT1g3cgyFUcVrJs6SUI8ac9fFdjA/uLqrO4jtHZbsAPKElKYaGS76NLcuRi+Ez6M\n\tsnx8/dq6GSwZUamhuVn2KJMKkTwGE3Jef1fFH0ovzvCxkq2TTupNPxE5Xzl+w7Sr/H\n\tcppsWNyzUqQLrprrUQmbCYxwiE9BHgAbbgENgBhSUIYXcmkAJFNjv0On+ftsPSSwyo\n\t3HIQhfqd+aoaQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1689071456;\n\tbh=yjWGe7Rkz3BV/ouM+vx82YvP4mpbRLDoca7998oZTkY=;\n\th=In-Reply-To:References:Subject:From:Cc:To:Date:From;\n\tb=HZHNfn+lLcphdw3zP3ynWSxoMj88LhWcRHjrezEOGdhVzyfUHKya5F3fFSJdOO2MY\n\tNcHGRdAIKfSF18Or3BpIukz+KkOj1MR0wK2vLH7LrPltNcWRvZ+l98kGSHhc+zwu1Y\n\tiZTREnYpVjzCrSS79VSTPWdtnoHN/a0ENhCo0658="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"HZHNfn+l\"; dkim-atps=neutral","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t","To":"David Plowman ,\n\tNaushir Patuck ,\n\tNaushir Patuck via libcamera-devel ","Date":"Tue, 11 Jul 2023 11:31:42 +0100","Message-ID":"<168907150298.540247.16063484386946286910@Monstersaurus>","User-Agent":"alot/0.10","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Kieran Bingham via libcamera-devel\n\t","Reply-To":"Kieran Bingham ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27541,"web_url":"https://patchwork.libcamera.org/comment/27541/","msgid":"","date":"2023-07-11T14:58:55","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"On Tue, 11 Jul 2023 at 11:31, Kieran Bingham\n wrote:\n>\n> Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)\n> > Hi all,\n> >\n> > On a semi-related topic, we talked offline about improving the drop frame\n> > support by queuing a request buffer multiple times to avoid the need for\n> > allocating internal buffers. I've tried this out and here are my findings.\n> >\n> > Firstly, to handle queuing a single buffer multiple times, I need to increase\n> > the number of cache slots in V4L2BufferCache(). Perhaps\n> > V4L2VideoDevice::importBuffers()\n> > should be updated to not take in a count parameter and we just allocate slots\n> > for the maximum buffer count possible in V4L2 (32 I think)? There has been a\n> > long-standing \\todo in the RPi code to choose an appropriate value, and the\n> > maximum number is really the only appropriate value I can think of.\n>\n> I still think allocating the maximum here in the v4l2 components is\n> appropriate as they are 'cheap' ...\n\nAgree, I think 32 is the limit according to V4L2.\n\n>\n>\n>\n> > Once I got this working, unfortunately I realised this method would never\n> > actually work correctly in the common scenario where the application configures\n> > and uses a RAW stream. In such cases we would queue the RAW buffer into Unicam\n> > N times for N dropped frames. However this buffer is also imported into the ISP\n> > for processing and stats generation, all while it is also being filled by Unicam\n> > for the next sensor frame. This makes the stats entirely unusable.\n>\n>\n> Aha that's a shame. I thought the restrictions were going to be in the\n> kernel side, so at least it's interesting to know that we /can/ queue\n> the same buffer multiple times (with a distinct v4l2_buffer id) and get\n> somewhat of the expected behaviour....\n>\n> >\n> > So in the end we either have to allocate additional buffers for drop frames\n> > (like we do right now), or we implement something like this series where the\n> > application is responsible for dropping/ignoring these frames.\n>\n> Of course if the expected behaviour doesn't suit the use case ... then\n> ...\n>\n> This may all be different for pipeline's with an inline ISP though ...\n> so the research is still useful.\n\nSo the question is... should we continue with this series as a possible\nimprovement if the pipeline handler wants to support this control?\n\n>\n> --\n> Kieran\n>\n> >\n> > Regards,\n> > Naush\n> >\n> >\n> >\n> >\n> >\n> > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel\n> > wrote:\n> > >\n> > > This control is passed back in a frame as metadata to indicate whether\n> > > the camera system is still in a startup phase, and the application is\n> > > advised to avoid using the frame.\n> > >\n> > > Signed-off-by: David Plowman \n> > > ---\n> > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > 1 file changed, 15 insertions(+)\n> > >\n> > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > index adea5f90..4742d907 100644\n> > > --- a/src/libcamera/control_ids.yaml\n> > > +++ b/src/libcamera/control_ids.yaml\n> > > @@ -694,6 +694,21 @@ controls:\n> > > Continuous AF is paused. No further state changes or lens movements\n> > > will occur until the AfPauseResume control is sent.\n> > >\n> > > + - StartupFrame:\n> > > + type: bool\n> > > + description: |\n> > > + The value true indicates that the camera system is still in a startup\n> > > + phase where the output images may not be reliable, or that certain of\n> > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > + still be changing quite rapidly.\n> > > +\n> > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > + occur when the camera system starts for the first time, although,\n> > > + depending on the sensor and the implementation, they could occur at\n> > > + other times.\n> > > +\n> > > + The value false indicates that this is a normal frame.\n> > > +\n> > > # ----------------------------------------------------------------------------\n> > > # Draft controls section\n> > >\n> > > --\n> > > 2.30.2\n> > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7D1B7BDC71\n\tfor ;\n\tTue, 11 Jul 2023 14:59:14 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E6701628C1;\n\tTue, 11 Jul 2023 16:59:13 +0200 (CEST)","from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com\n\t[IPv6:2607:f8b0:4864:20::112b])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9909A61E32\n\tfor ;\n\tTue, 11 Jul 2023 16:59:12 +0200 (CEST)","by mail-yw1-x112b.google.com with SMTP id\n\t00721157ae682-579dd20b1c8so66397427b3.1\n\tfor ;\n\tTue, 11 Jul 2023 07:59:12 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1689087553;\n\tbh=sHlBmoQB4MDPFUajq3U5JFa5tfCDVLfLKnt2l+T8Z0s=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=SO0cUEgQkD1+nKhw9DUQTBhjup0fjJhO+yupE0AvM3O+VcMCgljz1NTNwAVkT+CYz\n\tYIA9o0wQXQxBoblmXBqEkQpzbd65cwX7aZCBdQ3DE0CJz7ptlfZnjEi8g/YUeKwGKS\n\t1il/yJV+Tws81BI2algKU6eHW5e574z/DvIDUgpkFjwLRyfYzuZIJegzvCpGsfGRxG\n\t9Y7TbTidr924/fP5siiQ5XcAys4yvK3x98HYgaQoxncZ4S/+rrzlBePYL68MVtoYpV\n\ta2lg08iycMhv0s5GaZKwI53b5CBF0G+XotGzfp2BzussCQcTMB9XWnLhwfb56RtWvz\n\tzWVaQGwzIfnPQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1689087551; x=1691679551;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=LgUsuxgwdZV6TpBq+nfZtiydSOvvCtPXauw2MgkFcVA=;\n\tb=RF5+3iDm/7bVxOfxCX511nlNm7fkjhAfLEjNblfUJmFKle5M8F4fgEp2YsiK8pLTez\n\tPEYpXL3HPpUylSD4tl0ZG3NtGbI/82L1BNuVFt8JYMQSNwPRA7AT0NgXnHU8TRQWEwfn\n\tN6OCz09c+7WOEasM/WiAoy7FxC2UHN0nIq1iJ4t/EPxx5eRE8gk1u+zWgcSxdoxcjQ30\n\tqYYvaxZhw+niM0a7vnzLwj0ScLnkKea0gJthsll1++/OmJjsQmvaazQevp8qfaxKmfDE\n\tQO5Hji7fsnovi2Hqa5yUudoNO8We8tkqIu4vqSnYxjGJ1kcP3Dr6TiyHJOuygJHS++/Y\n\t11PA=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"RF5+3iDm\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1689087551; x=1691679551;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=LgUsuxgwdZV6TpBq+nfZtiydSOvvCtPXauw2MgkFcVA=;\n\tb=iAe7ehJjrQLs/exg/0M8LdB7QPuMD0jCG9Op8zFwCOETnyKn/S8SIJp74ST//SwNae\n\tYpCMFEEEAznzwzPAUxRNrCuZvp0VNl/QHhe++wEnKJGHzOgK5lGt3iKkh9A5aa3qvo59\n\tdEI2b069h1dyzipmHAlHLEpLQJaUNTAArZTrJabCl0i7av3jrxymTjD6Bw+rFj9w3z79\n\tuC3nb0JbiF/iuCNn46pki2KKTfXamwlMUEGQ1rDwnBMIFgzbgK6IrJb0y2/R5tMTQX3L\n\tc7tfStrCCkT2W1g524YAbvHKXwUIOBEtmdnTVNimmIcsACwh64Nwix4lxMtxfBxHhhG2\n\t7fdA==","X-Gm-Message-State":"ABy/qLZN9e2wx7THX/OMtogq6P96vv1xtu6O2Znr3nCQZDFwXH4p4Pas\n\to0VB2e3i4ldMGMjGST6j0CpbTrJZ6o8SETpfYuf5ZA==","X-Google-Smtp-Source":"APBJJlHnm2JNC21DyebQLJWyy8V9Cpx+yUOGdMSrVQE1TxO3+3y/Pfo57hDURRDsWG3HdiooTzv0jASumc3BnCskJPY=","X-Received":"by 2002:a0d:e587:0:b0:57c:9e81:723 with SMTP id\n\to129-20020a0de587000000b0057c9e810723mr1813227ywe.5.1689087551252;\n\tTue, 11 Jul 2023 07:59:11 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<168907150298.540247.16063484386946286910@Monstersaurus>","In-Reply-To":"<168907150298.540247.16063484386946286910@Monstersaurus>","Date":"Tue, 11 Jul 2023 15:58:55 +0100","Message-ID":"","To":"Kieran Bingham ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"Naushir Patuck via libcamera-devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27639,"web_url":"https://patchwork.libcamera.org/comment/27639/","msgid":"","date":"2023-07-31T14:35:36","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi again everyone!\n\nOn Tue, 11 Jul 2023 at 15:59, Naushir Patuck wrote:\n>\n> On Tue, 11 Jul 2023 at 11:31, Kieran Bingham\n> wrote:\n> >\n> > Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)\n> > > Hi all,\n> > >\n> > > On a semi-related topic, we talked offline about improving the drop frame\n> > > support by queuing a request buffer multiple times to avoid the need for\n> > > allocating internal buffers. I've tried this out and here are my findings.\n> > >\n> > > Firstly, to handle queuing a single buffer multiple times, I need to increase\n> > > the number of cache slots in V4L2BufferCache(). Perhaps\n> > > V4L2VideoDevice::importBuffers()\n> > > should be updated to not take in a count parameter and we just allocate slots\n> > > for the maximum buffer count possible in V4L2 (32 I think)? There has been a\n> > > long-standing \\todo in the RPi code to choose an appropriate value, and the\n> > > maximum number is really the only appropriate value I can think of.\n> >\n> > I still think allocating the maximum here in the v4l2 components is\n> > appropriate as they are 'cheap' ...\n>\n> Agree, I think 32 is the limit according to V4L2.\n>\n> >\n> >\n> >\n> > > Once I got this working, unfortunately I realised this method would never\n> > > actually work correctly in the common scenario where the application configures\n> > > and uses a RAW stream. In such cases we would queue the RAW buffer into Unicam\n> > > N times for N dropped frames. However this buffer is also imported into the ISP\n> > > for processing and stats generation, all while it is also being filled by Unicam\n> > > for the next sensor frame. This makes the stats entirely unusable.\n> >\n> >\n> > Aha that's a shame. I thought the restrictions were going to be in the\n> > kernel side, so at least it's interesting to know that we /can/ queue\n> > the same buffer multiple times (with a distinct v4l2_buffer id) and get\n> > somewhat of the expected behaviour....\n> >\n> > >\n> > > So in the end we either have to allocate additional buffers for drop frames\n> > > (like we do right now), or we implement something like this series where the\n> > > application is responsible for dropping/ignoring these frames.\n> >\n> > Of course if the expected behaviour doesn't suit the use case ... then\n> > ...\n> >\n> > This may all be different for pipeline's with an inline ISP though ...\n> > so the research is still useful.\n>\n> So the question is... should we continue with this series as a possible\n> improvement if the pipeline handler wants to support this control?\n\nSo yes, I would indeed like to revisit this question. I still think\nit's useful for the original reasons, and the new use-case I'd like to\nbring into the mix now is HDR.\n\nOne HDR method combines a number of different exposures to create the\noutput. Now this isn't so relevant for the Pi seeing as we have no\nhardware for combining images, but you could imagine it being\nimportant to platforms more widely. The problem comes when this HDR\nmode is engaged... what do we do while we wait for all those different\nexposures to come through? Because until we have one of each, we can't\nreally produce the image that the application is expecting.\n\nThe idea would be that requests cycle round as normal, but come with\nmetadata saying \"I'm not ready yet\" while HDR is \"starting up\". Note\nthat this could of course happen at any time now, not just when the\ncamera starts (or mode switches).\n\nI still like the idea of generic \"I'm not ready\" metadata because\napplications won't want to understand all the different ways in which\nthings might not be ready. Though supplementary information that\ndetails what we're still waiting for might be helpful. Thoughts...?\n\nThanks!\nDavid\n\n>\n> >\n> > --\n> > Kieran\n> >\n> > >\n> > > Regards,\n> > > Naush\n> > >\n> > >\n> > >\n> > >\n> > >\n> > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel\n> > > wrote:\n> > > >\n> > > > This control is passed back in a frame as metadata to indicate whether\n> > > > the camera system is still in a startup phase, and the application is\n> > > > advised to avoid using the frame.\n> > > >\n> > > > Signed-off-by: David Plowman \n> > > > ---\n> > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > 1 file changed, 15 insertions(+)\n> > > >\n> > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > index adea5f90..4742d907 100644\n> > > > --- a/src/libcamera/control_ids.yaml\n> > > > +++ b/src/libcamera/control_ids.yaml\n> > > > @@ -694,6 +694,21 @@ controls:\n> > > > Continuous AF is paused. No further state changes or lens movements\n> > > > will occur until the AfPauseResume control is sent.\n> > > >\n> > > > + - StartupFrame:\n> > > > + type: bool\n> > > > + description: |\n> > > > + The value true indicates that the camera system is still in a startup\n> > > > + phase where the output images may not be reliable, or that certain of\n> > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > + still be changing quite rapidly.\n> > > > +\n> > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > + occur when the camera system starts for the first time, although,\n> > > > + depending on the sensor and the implementation, they could occur at\n> > > > + other times.\n> > > > +\n> > > > + The value false indicates that this is a normal frame.\n> > > > +\n> > > > # ----------------------------------------------------------------------------\n> > > > # Draft controls section\n> > > >\n> > > > --\n> > > > 2.30.2\n> > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B900ABDB13\n\tfor ;\n\tMon, 31 Jul 2023 14:35:49 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3039E627E9;\n\tMon, 31 Jul 2023 16:35:49 +0200 (CEST)","from mail-ua1-x936.google.com (mail-ua1-x936.google.com\n\t[IPv6:2607:f8b0:4864:20::936])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2E3716037D\n\tfor ;\n\tMon, 31 Jul 2023 16:35:48 +0200 (CEST)","by mail-ua1-x936.google.com with SMTP id\n\ta1e0cc1a2514c-7948c329363so1436352241.0\n\tfor ;\n\tMon, 31 Jul 2023 07:35:48 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1690814149;\n\tbh=4XARINes/NKqQfRjfmmxuTQ5uznWvyjOFQheUifh1do=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=V//cCJxWLz/9D1jzP9q3mfZVMWvtYqTf4Dn7c2pJlDeqiZxOywLTOeXa5LTk7NyMM\n\tfyu79BrMYwsBHcPbSpOGHZn9cYgrh83tiX79Kheb7zlHgPlIiFs5aXcZ+yMkGXsGkf\n\tkPhtvWvcUIX9w/SelisMdRfKI2LmR4zFxlDv8boS4WswjvhYGEnuKBJ70OIgra9uR5\n\t0vJaAlz9B/cwpsbVZAvqZ8PwzdJCjlJRSoxHKJLpu0kvSc6FPxniJuKBjvsJYQGxJm\n\tg7bKNFafqAAXULp506+EgErA72rnitdDZ/j85cT42PgVGcoNMv5gJUmKG27hbojG+u\n\teuTJu2dzYYpxg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1690814147; x=1691418947;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=1QhOPMFHQjG/TfnufkKAWF+TDBIh5eHgGQbJB1LqIUQ=;\n\tb=sP7t9LiTTfDtrnKh6iBNQ2gKQNU/390y8f3Cj08Z7Qbpdm5wqb1LES/UUQiALqsRpO\n\t7oWC2BeR9Pa+1Hv/5kbaTusW78YEuh65tI5lh6z1+L26ROLLK/ZJu+PIl0vz1sewmDry\n\tstZ0h8HJD8hxu+xbtVI50AHuAa+35H2yFkd/4YagShc9agsICYgGdaS47SmXjONBB2Tx\n\t6NkU6+ngDw8Vid0wqUuofGY8NrU/0sg60GT3bbVSWGLqgQQR8iHByCE0PW219yBBdRq7\n\tW65pr1KXe0JDI/oz2dMzQ9YCt2Ny5x0DuaNdiheKvaTXLfMfpc5ZMRZclaZALCzpZoHc\n\tOkAg=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"sP7t9LiT\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1690814147; x=1691418947;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=1QhOPMFHQjG/TfnufkKAWF+TDBIh5eHgGQbJB1LqIUQ=;\n\tb=ku4YY9v90jveh/kjI3qi/zE8zi/xjM9IQYl8Is4EZZgrgf9ZyQgH/chxtuBhdnZOOH\n\tGGAjfsQz902aPyiVm2XPBzzEYMukLVVS8iQMubip8l/7o/iAV+/zJ+tRG/vsSeEAQNqe\n\tRECk/SRMuDrS36bboCLyjscSWicWLpfd7RHyIIP+iG1eSgido1pGM13wF601q8CgGDsp\n\tK11I5ZgpaR18RU6O5Ym1Bi2817cj31grrO8Hpz7ktUkVo5+5iOpo2F/k59ZNrxCmZlp+\n\tlif7i75YlH4vsj8i8ckPzg1iXmaMNgmV67SBxqZ3YKvaS5tnkkzXJcCPFqxaci84OfK6\n\tHCZw==","X-Gm-Message-State":"ABy/qLaZPmAWlQhRtgBK1pw8kK6djQS35wViK1v7rVkqoukCq3utp5il\n\tP9NS0xYfPtA5iix4H2zN6rM6X4qCSG9zXX0BbA3Oxf9p8N7OCwRt","X-Google-Smtp-Source":"APBJJlGoK+JQBiWduHQKsweYqeB7D4nPYAwuTnOC3vE8H2GZwT1LPZOeRyANPr2pPTJJzP5iXy74e2qtFgZv+5euS1o=","X-Received":"by 2002:a1f:5f90:0:b0:46e:85b8:c019 with SMTP id\n\tt138-20020a1f5f90000000b0046e85b8c019mr175871vkb.1.1690814146939;\n\tMon, 31 Jul 2023 07:35:46 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<168907150298.540247.16063484386946286910@Monstersaurus>\n\t","In-Reply-To":"","Date":"Mon, 31 Jul 2023 15:35:36 +0100","Message-ID":"","To":"Naushir Patuck ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27658,"web_url":"https://patchwork.libcamera.org/comment/27658/","msgid":"<20230808081449.GA17073@pendragon.ideasonboard.com>","date":"2023-08-08T08:14:49","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hello,\n\nOn Mon, Jul 31, 2023 at 03:35:36PM +0100, David Plowman via libcamera-devel wrote:\n> On Tue, 11 Jul 2023 at 15:59, Naushir Patuck wrote:\n> > On Tue, 11 Jul 2023 at 11:31, Kieran Bingham wrote:\n> > > Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)\n> > > > Hi all,\n> > > >\n> > > > On a semi-related topic, we talked offline about improving the drop frame\n> > > > support by queuing a request buffer multiple times to avoid the need for\n> > > > allocating internal buffers. I've tried this out and here are my findings.\n> > > >\n> > > > Firstly, to handle queuing a single buffer multiple times, I need to increase\n> > > > the number of cache slots in V4L2BufferCache(). Perhaps\n> > > > V4L2VideoDevice::importBuffers()\n> > > > should be updated to not take in a count parameter and we just allocate slots\n> > > > for the maximum buffer count possible in V4L2 (32 I think)? There has been a\n> > > > long-standing \\todo in the RPi code to choose an appropriate value, and the\n> > > > maximum number is really the only appropriate value I can think of.\n> > >\n> > > I still think allocating the maximum here in the v4l2 components is\n> > > appropriate as they are 'cheap' ...\n> >\n> > Agree, I think 32 is the limit according to V4L2.\n\nThere's one \"small\" drawback though. Allocating buffers is indeed cheap\n(for DMABUF), but once a V4L2 buffer has been queued with dmabuf\nobjects, those objects will stay referenced until the V4L2 buffer is\nfreed. On systems where the user keeps allocating and freeing buffers,\nthis means that we will hold on to 32 buffers until the Camera is\nreconfigured or released.\n\nThis is an issue with V4L2, and we're already affected by it. Increasing\nthe number of buffers will make it worse in some use cases, but doesn't\nsignificantly change the nature of the problem. The proposed new V4L2\nDELETE_BUFS ioctl may help solving this. In the meantime, I think we can\nincrease the number of buffers despite the issue, but I would limit the\nincrease to a smaller value.\n\n> > > > Once I got this working, unfortunately I realised this method would never\n> > > > actually work correctly in the common scenario where the application configures\n\nShould I quote Henri from An American Tail ? :-)\n\n> > > > and uses a RAW stream. In such cases we would queue the RAW buffer into Unicam\n> > > > N times for N dropped frames. However this buffer is also imported into the ISP\n> > > > for processing and stats generation, all while it is also being filled by Unicam\n> > > > for the next sensor frame. This makes the stats entirely unusable.\n> > >\n> > > Aha that's a shame. I thought the restrictions were going to be in the\n> > > kernel side, so at least it's interesting to know that we /can/ queue\n> > > the same buffer multiple times (with a distinct v4l2_buffer id) and get\n> > > somewhat of the expected behaviour....\n> > >\n> > > > So in the end we either have to allocate additional buffers for drop frames\n> > > > (like we do right now), or we implement something like this series where the\n> > > > application is responsible for dropping/ignoring these frames.\n\nThere may be something that escapes me, but I think you're trying to\nburry the idea too fast.\n\nLooking at the problem from an API point of view, ignoring the internal\nimplementation in the pipeline handler for a moment, your proposal is to\nadd a mechanism to tell applications that they should ignore the\ncontents of a request and resubmit it. If this is possible to do for\napplications without any negative side effects such as the one you've\ndescribed above, then I don't see why it would be impossible for\nthe pipeline handler to do the same before the request reaches the\napplication.\n\nAddressing the exact issue you're facing, it seems that the problem is\ncaused by using the raw buffer from the first request only. Under normal\noperation conditions, the pipeline handler will need at least two raw\nbuffers, otherwise frames will be dropped. It's the application's\nresponsibility to queue enough requests for proper operation if it wants\nto avoid frame drops. I think you can thus use raw buffers supplied in\nthe first two requests.\n\n> > > Of course if the expected behaviour doesn't suit the use case ... then\n> > > ...\n> > >\n> > > This may all be different for pipeline's with an inline ISP though ...\n> > > so the research is still useful.\n> >\n> > So the question is... should we continue with this series as a possible\n> > improvement if the pipeline handler wants to support this control?\n> \n> So yes, I would indeed like to revisit this question. I still think\n> it's useful for the original reasons, and the new use-case I'd like to\n> bring into the mix now is HDR.\n> \n> One HDR method combines a number of different exposures to create the\n> output. Now this isn't so relevant for the Pi seeing as we have no\n> hardware for combining images, but you could imagine it being\n> important to platforms more widely. The problem comes when this HDR\n> mode is engaged... what do we do while we wait for all those different\n> exposures to come through? Because until we have one of each, we can't\n> really produce the image that the application is expecting.\n> \n> The idea would be that requests cycle round as normal, but come with\n> metadata saying \"I'm not ready yet\" while HDR is \"starting up\". Note\n> that this could of course happen at any time now, not just when the\n> camera starts (or mode switches).\n> \n> I still like the idea of generic \"I'm not ready\" metadata because\n> applications won't want to understand all the different ways in which\n> things might not be ready. Though supplementary information that\n> details what we're still waiting for might be helpful. Thoughts...?\n\nFor the HDR case, would this supplementary information be conveyed in\nthe form of an HDR-specific control in the request metadata ?\n\nIf an application decides to enable HDR in the middle of a capture\nsession, is it unreasonable to expect the application to understand this\nparticular reason why frames are not \"ready\" ?\n\n> > > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > > > >\n> > > > > This control is passed back in a frame as metadata to indicate whether\n> > > > > the camera system is still in a startup phase, and the application is\n> > > > > advised to avoid using the frame.\n> > > > >\n> > > > > Signed-off-by: David Plowman \n> > > > > ---\n> > > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > > 1 file changed, 15 insertions(+)\n> > > > >\n> > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > index adea5f90..4742d907 100644\n> > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > @@ -694,6 +694,21 @@ controls:\n> > > > > Continuous AF is paused. No further state changes or lens movements\n> > > > > will occur until the AfPauseResume control is sent.\n> > > > >\n> > > > > + - StartupFrame:\n> > > > > + type: bool\n> > > > > + description: |\n> > > > > + The value true indicates that the camera system is still in a startup\n> > > > > + phase where the output images may not be reliable, or that certain of\n> > > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > > + still be changing quite rapidly.\n> > > > > +\n> > > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > > + occur when the camera system starts for the first time, although,\n> > > > > + depending on the sensor and the implementation, they could occur at\n> > > > > + other times.\n> > > > > +\n> > > > > + The value false indicates that this is a normal frame.\n> > > > > +\n> > > > > # ----------------------------------------------------------------------------\n> > > > > # Draft controls section\n> > > > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DD09BBDCBF\n\tfor ;\n\tTue, 8 Aug 2023 08:14:47 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 28B13627EB;\n\tTue, 8 Aug 2023 10:14:47 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4F99B61E13\n\tfor ;\n\tTue, 8 Aug 2023 10:14:44 +0200 (CEST)","from pendragon.ideasonboard.com (213-243-189-158.bb.dnainternet.fi\n\t[213.243.189.158])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 76A5336B0;\n\tTue, 8 Aug 2023 10:13:36 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1691482487;\n\tbh=oLOLuxZhKit+91S7eEIVxTjInk6JyArFpiacXm9nBEo=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=QbQ7lVo83eOkXe7zhBiN/U/fZGyK99+zWmdwEZUxK+Hu1grAYuDjfuosQOQqcZyHt\n\tHl/QMjJJUn7JHv/zMjryh68hYZtCB83PjsRq+KmBzrskb2oL+CNfKfnym5ztP55DSh\n\tAdT6YWzu9i8K8Lq41AveTML9M/v09UINjE4+g2bZIJp2hKjE6MznlXMjeQsM06JOmO\n\ttM7ESnEDr+LL6kmt6iHXLmnmmf76TnpJ2i61IflP0y6YmVtytXuIRW/BElCPWDx5kh\n\tSO1ibaRfkY/ln3JMhAtS1fzm3+Jz3OirPxpgScIElRoOs4TIX47zHKzWil5q9YtaFu\n\tqDO2rpVe1ztnQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1691482416;\n\tbh=oLOLuxZhKit+91S7eEIVxTjInk6JyArFpiacXm9nBEo=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=ebcuV0A4ZdxleeeNGhDzELeKcqOzaSxoJ3sWPB4RvycQp5SYFrSBR/suYgsd2eaOc\n\tl6/40hSFL9Wu9P2O9skqRpOv55Ia7fEZ4fUySA847X/JjuJd8XZfTre8R+IzGkNeYr\n\tYvebiF5WcVgpEqjxAKkiZAahFlN/p3Qrlnb8rGpQ="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"ebcuV0A4\"; dkim-atps=neutral","Date":"Tue, 8 Aug 2023 11:14:49 +0300","To":"David Plowman ","Message-ID":"<20230808081449.GA17073@pendragon.ideasonboard.com>","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<168907150298.540247.16063484386946286910@Monstersaurus>\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27659,"web_url":"https://patchwork.libcamera.org/comment/27659/","msgid":"","date":"2023-08-09T10:53:30","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Laurent\n\nThanks for the comments!\n\nOn Tue, 8 Aug 2023 at 09:14, Laurent Pinchart\n wrote:\n>\n> Hello,\n>\n> On Mon, Jul 31, 2023 at 03:35:36PM +0100, David Plowman via libcamera-devel wrote:\n> > On Tue, 11 Jul 2023 at 15:59, Naushir Patuck wrote:\n> > > On Tue, 11 Jul 2023 at 11:31, Kieran Bingham wrote:\n> > > > Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)\n> > > > > Hi all,\n> > > > >\n> > > > > On a semi-related topic, we talked offline about improving the drop frame\n> > > > > support by queuing a request buffer multiple times to avoid the need for\n> > > > > allocating internal buffers. I've tried this out and here are my findings.\n> > > > >\n> > > > > Firstly, to handle queuing a single buffer multiple times, I need to increase\n> > > > > the number of cache slots in V4L2BufferCache(). Perhaps\n> > > > > V4L2VideoDevice::importBuffers()\n> > > > > should be updated to not take in a count parameter and we just allocate slots\n> > > > > for the maximum buffer count possible in V4L2 (32 I think)? There has been a\n> > > > > long-standing \\todo in the RPi code to choose an appropriate value, and the\n> > > > > maximum number is really the only appropriate value I can think of.\n> > > >\n> > > > I still think allocating the maximum here in the v4l2 components is\n> > > > appropriate as they are 'cheap' ...\n> > >\n> > > Agree, I think 32 is the limit according to V4L2.\n>\n> There's one \"small\" drawback though. Allocating buffers is indeed cheap\n> (for DMABUF), but once a V4L2 buffer has been queued with dmabuf\n> objects, those objects will stay referenced until the V4L2 buffer is\n> freed. On systems where the user keeps allocating and freeing buffers,\n> this means that we will hold on to 32 buffers until the Camera is\n> reconfigured or released.\n>\n> This is an issue with V4L2, and we're already affected by it. Increasing\n> the number of buffers will make it worse in some use cases, but doesn't\n> significantly change the nature of the problem. The proposed new V4L2\n> DELETE_BUFS ioctl may help solving this. In the meantime, I think we can\n> increase the number of buffers despite the issue, but I would limit the\n> increase to a smaller value.\n>\n> > > > > Once I got this working, unfortunately I realised this method would never\n> > > > > actually work correctly in the common scenario where the application configures\n>\n> Should I quote Henri from An American Tail ? :-)\n>\n> > > > > and uses a RAW stream. In such cases we would queue the RAW buffer into Unicam\n> > > > > N times for N dropped frames. However this buffer is also imported into the ISP\n> > > > > for processing and stats generation, all while it is also being filled by Unicam\n> > > > > for the next sensor frame. This makes the stats entirely unusable.\n> > > >\n> > > > Aha that's a shame. I thought the restrictions were going to be in the\n> > > > kernel side, so at least it's interesting to know that we /can/ queue\n> > > > the same buffer multiple times (with a distinct v4l2_buffer id) and get\n> > > > somewhat of the expected behaviour....\n> > > >\n> > > > > So in the end we either have to allocate additional buffers for drop frames\n> > > > > (like we do right now), or we implement something like this series where the\n> > > > > application is responsible for dropping/ignoring these frames.\n>\n> There may be something that escapes me, but I think you're trying to\n> burry the idea too fast.\n>\n> Looking at the problem from an API point of view, ignoring the internal\n> implementation in the pipeline handler for a moment, your proposal is to\n> add a mechanism to tell applications that they should ignore the\n> contents of a request and resubmit it. If this is possible to do for\n> applications without any negative side effects such as the one you've\n> described above, then I don't see why it would be impossible for\n> the pipeline handler to do the same before the request reaches the\n> application.\n>\n> Addressing the exact issue you're facing, it seems that the problem is\n> caused by using the raw buffer from the first request only. Under normal\n> operation conditions, the pipeline handler will need at least two raw\n> buffers, otherwise frames will be dropped. It's the application's\n> responsibility to queue enough requests for proper operation if it wants\n> to avoid frame drops. I think you can thus use raw buffers supplied in\n> the first two requests.\n>\n> > > > Of course if the expected behaviour doesn't suit the use case ... then\n> > > > ...\n> > > >\n> > > > This may all be different for pipeline's with an inline ISP though ...\n> > > > so the research is still useful.\n> > >\n> > > So the question is... should we continue with this series as a possible\n> > > improvement if the pipeline handler wants to support this control?\n> >\n> > So yes, I would indeed like to revisit this question. I still think\n> > it's useful for the original reasons, and the new use-case I'd like to\n> > bring into the mix now is HDR.\n> >\n> > One HDR method combines a number of different exposures to create the\n> > output. Now this isn't so relevant for the Pi seeing as we have no\n> > hardware for combining images, but you could imagine it being\n> > important to platforms more widely. The problem comes when this HDR\n> > mode is engaged... what do we do while we wait for all those different\n> > exposures to come through? Because until we have one of each, we can't\n> > really produce the image that the application is expecting.\n> >\n> > The idea would be that requests cycle round as normal, but come with\n> > metadata saying \"I'm not ready yet\" while HDR is \"starting up\". Note\n> > that this could of course happen at any time now, not just when the\n> > camera starts (or mode switches).\n> >\n> > I still like the idea of generic \"I'm not ready\" metadata because\n> > applications won't want to understand all the different ways in which\n> > things might not be ready. Though supplementary information that\n> > details what we're still waiting for might be helpful. Thoughts...?\n>\n> For the HDR case, would this supplementary information be conveyed in\n> the form of an HDR-specific control in the request metadata ?\n>\n> If an application decides to enable HDR in the middle of a capture\n> session, is it unreasonable to expect the application to understand this\n> particular reason why frames are not \"ready\" ?\n\nJust to answer this last question, what I really want is to be\nnice to our application developers. For instance:\n\n1. There will be more features like this in future. Frame\nstacking, night modes, various things. Do we want to force\napplications to have to check all this stuff explicitly?\n(A bit like \"validating\" a configuration - you get a\nhandy \"something got adjusted\" indication.)\n\n2. As new features get added, it would be good to minimse the\namount of pain we cause to existing applications. A simple \"not\nready\" means application developers will have fewer code paths\nthat need to be found and edited. Vendors might be able to add\nnew features to their processing without breaking existing\napplications.\n\nA couple of notes to this:\n\n* I guess we could handle this by \"swallowing\" frames and not\n letting them out into the world. This obviously refers back to\n the discussion further up which needs to be resolved.\n\n* I certainly think it's good if further information is available\n for those who are interested, this is mostly about making\n libcamera easier to use when folks aren't.\n\nDavid\n\n>\n> > > > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > > > > >\n> > > > > > This control is passed back in a frame as metadata to indicate whether\n> > > > > > the camera system is still in a startup phase, and the application is\n> > > > > > advised to avoid using the frame.\n> > > > > >\n> > > > > > Signed-off-by: David Plowman \n> > > > > > ---\n> > > > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > > > 1 file changed, 15 insertions(+)\n> > > > > >\n> > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > > index adea5f90..4742d907 100644\n> > > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > > @@ -694,6 +694,21 @@ controls:\n> > > > > > Continuous AF is paused. No further state changes or lens movements\n> > > > > > will occur until the AfPauseResume control is sent.\n> > > > > >\n> > > > > > + - StartupFrame:\n> > > > > > + type: bool\n> > > > > > + description: |\n> > > > > > + The value true indicates that the camera system is still in a startup\n> > > > > > + phase where the output images may not be reliable, or that certain of\n> > > > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > > > + still be changing quite rapidly.\n> > > > > > +\n> > > > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > > > + occur when the camera system starts for the first time, although,\n> > > > > > + depending on the sensor and the implementation, they could occur at\n> > > > > > + other times.\n> > > > > > +\n> > > > > > + The value false indicates that this is a normal frame.\n> > > > > > +\n> > > > > > # ----------------------------------------------------------------------------\n> > > > > > # Draft controls section\n> > > > > >\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4598EBDCBF\n\tfor ;\n\tWed, 9 Aug 2023 10:53:45 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 64AF5628C0;\n\tWed, 9 Aug 2023 12:53:44 +0200 (CEST)","from mail-qt1-x830.google.com (mail-qt1-x830.google.com\n\t[IPv6:2607:f8b0:4864:20::830])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AFAEB600F7\n\tfor ;\n\tWed, 9 Aug 2023 12:53:42 +0200 (CEST)","by mail-qt1-x830.google.com with SMTP id\n\td75a77b69052e-407ff54164dso43819981cf.2\n\tfor ;\n\tWed, 09 Aug 2023 03:53:42 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1691578424;\n\tbh=OghOv5gKts6MOF6k66YTy9ZztfQ1PxHEsL283avm3HM=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=pCfN2DxiNi63DRzjDSIhaehGViB4Bti6FHectH6nexPBFr0CbVmM+D7+ZCiCkJnSC\n\tI54Neam71ucyVvs/vg9rRSZJD4EnB7xeE++VfvLifSV8Eon0KQzdivlw5H+Rw/7ruG\n\tW647kqBfma4r6vXERrH0cVv8HnjNK39w+w2kdr2E3M2jFeJDlZBlY6RiHA9zy9MSri\n\tZQihFVRfIpjIat+RtclysZR8dpSaNMrOGnDgOkCEBPoAEAUQVMe3A0NhoFox4Tht8W\n\txDJfDG5OAljqxgvZ2RrWddhkNlAr21bbUiSpYxJQqQUvnt2hSm6VLRbIw4r3NoPBPU\n\tcrvMRRudDzlEg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1691578421; x=1692183221;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=nT3p0be0ktDw01ROx39Agk82QJtGMvNrhJ2yWcuNZ/g=;\n\tb=iCQNUP7H0wueFdNnXijfcxkVlfHIfbbsT8M0+LT83q9DrRo74Wb1K0ExWUpcB1a6L2\n\tdN6dMPb8224ThIO1Zhy7RcwYQoBn8Lf9z98KeUscCzExdfXXPSmHg/3ouGUXHHHzXogj\n\tOSwogvZ6CyUVc57xGHSvh/rKNqMtKApx8K/ih1IY0IN3ouRvQnQ0MGuDfiFl13KsfcBK\n\tL9yyVesI2pCv624rJ7UvFu99oAw/+dgGRCTdDGqq/3Qw8maqlcYQz1Ex8Hg8aRL3/y5a\n\tRuDw+t4k9bmqc97pKulZVAnIJ2v5dYkJ1+Mhr+sF8gCVSJm7F4BYhLMHtPOYM+hRfniw\n\tyWsA=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"iCQNUP7H\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1691578421; x=1692183221;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=nT3p0be0ktDw01ROx39Agk82QJtGMvNrhJ2yWcuNZ/g=;\n\tb=fjsnCwoAp3WZsj6OFIC0mlEUfaIcvHQL0Nv9nXEwItrO4FtFqAFLZh70p5ZErXiz2O\n\tbdXb9zz+w71Iz4zA6IGt+ktKUIhE7GRFfITwin10J+WVYUOTjCts/9mst9fxU26MnymB\n\tJ6FpRrKK++chIZ4OGnxu2/YEl6Q/dBufGXM76yJ8dMkpdQSB+sHwHGx4cOXbYdSFtweD\n\tAZJVhDK0IeoxhTBv3mHRQ8gS3In4xYqD6bTZxSGYVT04/YRZUCly/GxsiDNCZUYItPMy\n\tFeBp3b+duS162ZJh0vZ2ptuH9JGgwABVyJEq0eleC9VrVS0FV+RGikmw1iLBAHWFrEBl\n\tnY1A==","X-Gm-Message-State":"AOJu0YxNLgdV7TO3XiQkwwVNchTloEI5/9OYkxli2jpUH/1ek5ajIu6B\n\tylNCqRdP9dzZsT8bJtZDkCCH5X0R1EYpfhrMZWyKAw==","X-Google-Smtp-Source":"AGHT+IHM6HlZnCOl3K955S2OzovZiZEVEhnAFL+e4gVpxPRgp85G7lyyBY+HOnbz7GEc5tx2T4c/+4TWpTJx+ttmjzo=","X-Received":"by 2002:ac8:580e:0:b0:3ff:c677:a033 with SMTP id\n\tg14-20020ac8580e000000b003ffc677a033mr2911109qtg.29.1691578421301;\n\tWed, 09 Aug 2023 03:53:41 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<168907150298.540247.16063484386946286910@Monstersaurus>\n\t\n\t\n\t<20230808081449.GA17073@pendragon.ideasonboard.com>","In-Reply-To":"<20230808081449.GA17073@pendragon.ideasonboard.com>","Date":"Wed, 9 Aug 2023 11:53:30 +0100","Message-ID":"","To":"Laurent Pinchart ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":27660,"web_url":"https://patchwork.libcamera.org/comment/27660/","msgid":"","date":"2023-08-14T12:49:37","subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Laurent,\n\nOn Tue, 8 Aug 2023 at 09:14, Laurent Pinchart\n wrote:\n>\n> Hello,\n>\n> On Mon, Jul 31, 2023 at 03:35:36PM +0100, David Plowman via libcamera-devel wrote:\n> > On Tue, 11 Jul 2023 at 15:59, Naushir Patuck wrote:\n> > > On Tue, 11 Jul 2023 at 11:31, Kieran Bingham wrote:\n> > > > Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)\n> > > > > Hi all,\n> > > > >\n> > > > > On a semi-related topic, we talked offline about improving the drop frame\n> > > > > support by queuing a request buffer multiple times to avoid the need for\n> > > > > allocating internal buffers. I've tried this out and here are my findings.\n> > > > >\n> > > > > Firstly, to handle queuing a single buffer multiple times, I need to increase\n> > > > > the number of cache slots in V4L2BufferCache(). Perhaps\n> > > > > V4L2VideoDevice::importBuffers()\n> > > > > should be updated to not take in a count parameter and we just allocate slots\n> > > > > for the maximum buffer count possible in V4L2 (32 I think)? There has been a\n> > > > > long-standing \\todo in the RPi code to choose an appropriate value, and the\n> > > > > maximum number is really the only appropriate value I can think of.\n> > > >\n> > > > I still think allocating the maximum here in the v4l2 components is\n> > > > appropriate as they are 'cheap' ...\n> > >\n> > > Agree, I think 32 is the limit according to V4L2.\n>\n> There's one \"small\" drawback though. Allocating buffers is indeed cheap\n> (for DMABUF), but once a V4L2 buffer has been queued with dmabuf\n> objects, those objects will stay referenced until the V4L2 buffer is\n> freed. On systems where the user keeps allocating and freeing buffers,\n> this means that we will hold on to 32 buffers until the Camera is\n> reconfigured or released.\n>\n> This is an issue with V4L2, and we're already affected by it. Increasing\n> the number of buffers will make it worse in some use cases, but doesn't\n> significantly change the nature of the problem. The proposed new V4L2\n> DELETE_BUFS ioctl may help solving this. In the meantime, I think we can\n> increase the number of buffers despite the issue, but I would limit the\n> increase to a smaller value.\n>\n> > > > > Once I got this working, unfortunately I realised this method would never\n> > > > > actually work correctly in the common scenario where the application configures\n>\n> Should I quote Henri from An American Tail ? :-)\n>\n> > > > > and uses a RAW stream. In such cases we would queue the RAW buffer into Unicam\n> > > > > N times for N dropped frames. However this buffer is also imported into the ISP\n> > > > > for processing and stats generation, all while it is also being filled by Unicam\n> > > > > for the next sensor frame. This makes the stats entirely unusable.\n> > > >\n> > > > Aha that's a shame. I thought the restrictions were going to be in the\n> > > > kernel side, so at least it's interesting to know that we /can/ queue\n> > > > the same buffer multiple times (with a distinct v4l2_buffer id) and get\n> > > > somewhat of the expected behaviour....\n> > > >\n> > > > > So in the end we either have to allocate additional buffers for drop frames\n> > > > > (like we do right now), or we implement something like this series where the\n> > > > > application is responsible for dropping/ignoring these frames.\n>\n> There may be something that escapes me, but I think you're trying to\n> burry the idea too fast.\n>\n> Looking at the problem from an API point of view, ignoring the internal\n> implementation in the pipeline handler for a moment, your proposal is to\n> add a mechanism to tell applications that they should ignore the\n> contents of a request and resubmit it. If this is possible to do for\n> applications without any negative side effects such as the one you've\n> described above, then I don't see why it would be impossible for\n> the pipeline handler to do the same before the request reaches the\n> application.\n>\n> Addressing the exact issue you're facing, it seems that the problem is\n> caused by using the raw buffer from the first request only. Under normal\n> operation conditions, the pipeline handler will need at least two raw\n> buffers, otherwise frames will be dropped. It's the application's\n> responsibility to queue enough requests for proper operation if it wants\n> to avoid frame drops. I think you can thus use raw buffers supplied in\n> the first two requests.\n>\n\nIt's certainly possible for the pipeline handler to do this kind of thing\nassuming the application has queued an appropriate number of buffers. But it\nfeels quite cumbersome because I'm \"borrowing\" buffers associated with requests,\nand this needs careful management of buffer ordering. If/when this association\nis removed, this can become quite a bit simpler.\n\n> > > > Of course if the expected behaviour doesn't suit the use case ... then\n> > > > ...\n> > > >\n> > > > This may all be different for pipeline's with an inline ISP though ...\n> > > > so the research is still useful.\n> > >\n> > > So the question is... should we continue with this series as a possible\n> > > improvement if the pipeline handler wants to support this control?\n> >\n> > So yes, I would indeed like to revisit this question. I still think\n> > it's useful for the original reasons, and the new use-case I'd like to\n> > bring into the mix now is HDR.\n> >\n> > One HDR method combines a number of different exposures to create the\n> > output. Now this isn't so relevant for the Pi seeing as we have no\n> > hardware for combining images, but you could imagine it being\n> > important to platforms more widely. The problem comes when this HDR\n> > mode is engaged... what do we do while we wait for all those different\n> > exposures to come through? Because until we have one of each, we can't\n> > really produce the image that the application is expecting.\n> >\n> > The idea would be that requests cycle round as normal, but come with\n> > metadata saying \"I'm not ready yet\" while HDR is \"starting up\". Note\n> > that this could of course happen at any time now, not just when the\n> > camera starts (or mode switches).\n> >\n> > I still like the idea of generic \"I'm not ready\" metadata because\n> > applications won't want to understand all the different ways in which\n> > things might not be ready. Though supplementary information that\n> > details what we're still waiting for might be helpful. Thoughts...?\n>\n> For the HDR case, would this supplementary information be conveyed in\n> the form of an HDR-specific control in the request metadata ?\n>\n> If an application decides to enable HDR in the middle of a capture\n> session, is it unreasonable to expect the application to understand this\n> particular reason why frames are not \"ready\" ?\n>\n> > > > > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel wrote:\n> > > > > >\n> > > > > > This control is passed back in a frame as metadata to indicate whether\n> > > > > > the camera system is still in a startup phase, and the application is\n> > > > > > advised to avoid using the frame.\n> > > > > >\n> > > > > > Signed-off-by: David Plowman \n> > > > > > ---\n> > > > > > src/libcamera/control_ids.yaml | 15 +++++++++++++++\n> > > > > > 1 file changed, 15 insertions(+)\n> > > > > >\n> > > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > > index adea5f90..4742d907 100644\n> > > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > > @@ -694,6 +694,21 @@ controls:\n> > > > > > Continuous AF is paused. No further state changes or lens movements\n> > > > > > will occur until the AfPauseResume control is sent.\n> > > > > >\n> > > > > > + - StartupFrame:\n> > > > > > + type: bool\n> > > > > > + description: |\n> > > > > > + The value true indicates that the camera system is still in a startup\n> > > > > > + phase where the output images may not be reliable, or that certain of\n> > > > > > + the control algorithms (such as AEC/AGC, AWB, and possibly others) may\n> > > > > > + still be changing quite rapidly.\n> > > > > > +\n> > > > > > + Applications are advised to avoid using these frames. Mostly, they will\n> > > > > > + occur when the camera system starts for the first time, although,\n> > > > > > + depending on the sensor and the implementation, they could occur at\n> > > > > > + other times.\n> > > > > > +\n> > > > > > + The value false indicates that this is a normal frame.\n> > > > > > +\n> > > > > > # ----------------------------------------------------------------------------\n> > > > > > # Draft controls section\n> > > > > >\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7594EBF415\n\tfor ;\n\tMon, 14 Aug 2023 12:49:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B9F29628D3;\n\tMon, 14 Aug 2023 14:49:56 +0200 (CEST)","from mail-yw1-x112f.google.com (mail-yw1-x112f.google.com\n\t[IPv6:2607:f8b0:4864:20::112f])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 745B361E0B\n\tfor ;\n\tMon, 14 Aug 2023 14:49:54 +0200 (CEST)","by mail-yw1-x112f.google.com with SMTP id\n\t00721157ae682-586bacac98aso38622447b3.2\n\tfor ;\n\tMon, 14 Aug 2023 05:49:54 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1692017396;\n\tbh=E88+6Z563xSWu4ylXZNYyHY3WycWwerfX4HCMPhOEdQ=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=FEQROklNdXAVtKbCp4nMVn58/u5rod6ookemjb/oEVC2ci15nbyDLlK9BWmUWwmbL\n\thQZGMfq0refGVHJ1oF7opvzI5CrZ6N1BPfr2HntBizeQ4fXPhrOhSv7vjwP4IAJGLp\n\t50dvN3MBA6XgS/BVgbopcH8MX/nfQFNEIB/l6zrQeaf+03SqftwJJYjLXAv5RCTQGi\n\tUZscTkKOHtoJqfnwoxKFHFf+RxZObi+iFA6uIHoJMNRo+MeITtOLkJo8BVEWgmyzNb\n\tgrxTmNyp75aFTGMV0aDBDjRQTKXRni1P4VvL5XkeA9UC51lNXMsssbT6eZLlKjmbPI\n\ti4kSlh2bhL3Fw==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1692017393; x=1692622193;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=yy1z8pbsXwGBQLkoKi7XFTF71/VpCcXDqTFCiZiitvM=;\n\tb=t8cgBDhGsUyMhSTtoNkQAhMQfylDsScbOsU98+z5/2JWbXpT5w5WjJOxcW5r2QXhHQ\n\tmt6MHce21mCMOzrzrQZh+rDrEcVkDGEbtJv5K+DNKGrBICWV7530QnFJeAoijQDpmnPI\n\t4dQJ9Eo7hQLP70HNxbukshqrynBkvsqj5l10/xDKhB47yUApub0ptL9Sc2bw1pSqt/kO\n\tBBjhcVLZEsqg0ajEErQMwJ7OpdRCxcoscuBVYEAFnXyOvcArKMEZuH7LGZgQuewvhpaE\n\tlIkLMC5p+sirh8HC+VF9EaXOAdBDZnOsFYp/8Vyrn8tDbQssaukwXCvzWOAsI8UArPqP\n\tUP+Q=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"t8cgBDhG\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1692017393; x=1692622193;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=yy1z8pbsXwGBQLkoKi7XFTF71/VpCcXDqTFCiZiitvM=;\n\tb=ewGU2/hc13VOPJUejt4X6gTrQk551bqY4Z3dVJjm25lbiKjENkzZ/XOpXBKwlI6xYW\n\tCY9yoRsttMUwiLEzT0tEyWVmlNTIBwzs82W+DKDV2yBVXYoa4FpWlZCc4SLohLG7DbgU\n\t4t41ajcBecJ9y2BL2HbZHBRQculbAYESYEYK1+xuJPPpLfEZj4WTL81OwKoGJTLxTXJN\n\ttoa3XYK+5ugZvXOZvGixdeww/vG5B1OtKS77Haeq8J9thOak6JRboRF5IsN08F+h8grQ\n\teHt6e0KNGrS5TJwEKFwdN2cK+BRbMeFRNzvxGIB0XMXw29ej7pwblQt6+wRTO8pbbslh\n\t8vhw==","X-Gm-Message-State":"AOJu0YyNyTpsJrA1tSvWmdGkmobjpckJ4MVzDjVq51wAvKrBDAyMqhtg\n\tPE8nfX7ys6n+Hrdv0J1u98SIsEZ/JoxNiPYxFwC7nA==","X-Google-Smtp-Source":"AGHT+IHybMrIR1U8DDquMrrhKElPAFURFPLEA6p0gSeFfRFQwI1f7SEl/OcUw/O4w85vR2Ug75wrrMk1Gb5xwQ+Jo/k=","X-Received":"by 2002:a81:7b08:0:b0:586:a249:d396 with SMTP id\n\tw8-20020a817b08000000b00586a249d396mr7643895ywc.12.1692017391648;\n\tMon, 14 Aug 2023 05:49:51 -0700 (PDT)","MIME-Version":"1.0","References":"<20230531125016.5540-1-david.plowman@raspberrypi.com>\n\t<20230531125016.5540-2-david.plowman@raspberrypi.com>\n\t\n\t<168907150298.540247.16063484386946286910@Monstersaurus>\n\t\n\t\n\t<20230808081449.GA17073@pendragon.ideasonboard.com>","In-Reply-To":"<20230808081449.GA17073@pendragon.ideasonboard.com>","Date":"Mon, 14 Aug 2023 13:49:37 +0100","Message-ID":"","To":"Laurent Pinchart ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH 1/1] libcamera: controls: Add\n\tStartupFrame control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]