[{"id":26032,"web_url":"https://patchwork.libcamera.org/comment/26032/","msgid":"","date":"2022-12-09T08:47:30","subject":"Re: [libcamera-devel] [PATCH v3 08/13] pipeline: raspberrypi:\n\tDisable StreamOn for ISP Output0/1 when dropping frames","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi,\n\nUnfortunately this patch has introduced a tiny regression. It turns out\nthat\ncalling stream on for an ISP node takes some hundreds of milliseconds to\ncomplete.\nMost of this time is communicating and setup within the firmware layer. In\nvideo encode use cases, this will cause a frame drop if we do not have\nenough\nbuffers to smooth out the glitch.\n\nNot wanting to spend too much time and effort optimising this in the\nfirmware\n(particularly because I will likely break something else!), I will remove\nthis\npatch from the series, and we accept that we have to allocate a single\nbuffer if\nwe have to drop frames for AE convergence. This is not disastrous, as\ntypically\nthis will be a viewfinder size buffer.\n\nI will post an updated series shortly.\n\nThanks,\nNaush\n\n\nOn Tue, 6 Dec 2022 at 13:55, Naushir Patuck wrote:\n\n> If the pipeline handler is required to drop startup frames by the IPA, do\n> not\n> call StreamOn on the ISP Output0 and Output1 device nodes from\n> PipelineHandlerRPi::start(). This stops the ISP from generating output on\n> those\n> channels giving improved latency, and more crucially does not require\n> internal\n> buffers to be allocated to deal with this condition.\n>\n> Signed-off-by: Naushir Patuck \n> Reviewed-by: David Plowman \n> ---\n> .../pipeline/raspberrypi/raspberrypi.cpp | 36 ++++++++++++++++---\n> 1 file changed, 32 insertions(+), 4 deletions(-)\n>\n> diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> index 9a316eda6fea..302279ac5e01 100644\n> --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> @@ -1088,8 +1088,18 @@ int PipelineHandlerRPi::start(Camera *camera, const\n> ControlList *controls)\n>\n> data->state_ = RPiCameraData::State::Idle;\n>\n> - /* Start all streams. */\n> + /*\n> + * Start all streams with the exception of ISP Output0 and Output1\n> if\n> + * we are dropping some start-up frames. This saves a tiny bit of\n> latency\n> + * and avoids the need for allocating an Output0 buffer only to\n> handle\n> + * startup drop frame conditions.\n> + */\n> for (auto const stream : data->streams_) {\n> + if (data->dropFrameCount_ &&\n> + (stream == &data->isp_[Isp::Output0] ||\n> + stream == &data->isp_[Isp::Output1]))\n> + continue;\n> +\n> ret = stream->dev()->streamOn();\n> if (ret) {\n> stop(camera);\n> @@ -1425,6 +1435,14 @@ int PipelineHandlerRPi::queueAllBuffers(Camera\n> *camera)\n> if (ret < 0)\n> return ret;\n> } else {\n> + /*\n> + * We haven't enabled streaming yet for Output0\n> and Output1\n> + * for startup frame drops, so don't queue any\n> buffers.\n> + */\n> + if (stream == &data->isp_[Isp::Output0] ||\n> + stream == &data->isp_[Isp::Output1])\n> + continue;\n> +\n> /*\n> * For external streams, we must queue up a set of\n> internal\n> * buffers to handle the number of drop frames\n> requested by\n> @@ -2158,16 +2176,26 @@ void RPiCameraData::checkRequestCompleted()\n> }\n>\n> /*\n> - * Make sure we have three outputs completed in the case of a\n> dropped\n> - * frame.\n> + * Only the ISP statistics output is generated when we are dropping\n> + * frames on startup.\n> */\n> if (state_ == State::IpaComplete &&\n> - ((ispOutputCount_ == 3 && dropFrameCount_) ||\n> requestCompleted)) {\n> + ((ispOutputCount_ == 1 && dropFrameCount_) ||\n> requestCompleted)) {\n> state_ = State::Idle;\n> if (dropFrameCount_) {\n> dropFrameCount_--;\n> LOG(RPI, Debug) << \"Dropping frame at the request\n> of the IPA (\"\n> << dropFrameCount_ << \" left)\";\n> +\n> + /*\n> + * If we have finished dropping startup frames,\n> start\n> + * streaming on the ISP Output0 and Output1 nodes\n> for\n> + * normal operation.\n> + */\n> + if (!dropFrameCount_) {\n> + isp_[Isp::Output0].dev()->streamOn();\n> + isp_[Isp::Output1].dev()->streamOn();\n> + }\n> }\n> }\n> }\n> --\n> 2.25.1\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 359CBBE08B\n\tfor ;\n\tFri, 9 Dec 2022 08:47:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 90D0A63351;\n\tFri, 9 Dec 2022 09:47:42 +0100 (CET)","from mail-il1-x134.google.com (mail-il1-x134.google.com\n\t[IPv6:2607:f8b0:4864:20::134])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8FEA661F1E\n\tfor ;\n\tFri, 9 Dec 2022 09:47:41 +0100 (CET)","by mail-il1-x134.google.com with SMTP id g7so2373590ile.0\n\tfor ;\n\tFri, 09 Dec 2022 00:47:41 -0800 (PST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1670575662;\n\tbh=m1in2QpRCQJ0PvjMjd1IJK2/glWqvYelWyQutdRnt2k=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=r86ASCdkPNI0zI4c30r7vFtjyB4oAir/XeH3XuaEW1oGEO9/7Hc+7gBb8rEzJ1qyH\n\tsDEp8F0lFlBPu9U16FyPpD8hlSA0LZCkX0xLjm7ct/8gJwEuSAqLEBdWuXNuML3mBK\n\tcOdaHnyV5IRYcHZBDaCwX/j60Zy7ttPYaGOCwnh3evFkYMl/VwgCR45wkSw8J0TEwn\n\t/q4RaOGyaNkaW7jEfJ4rr0exrzCJ3mA25EsY2qPfzfQ0lIL0uV4wU2Erm5bAdoWf91\n\tFMarSAtF0TVky+AUfP0ahj07qaZE1+jAWSYvVnpe6x6K6lsVxjwctEMvqUGSwM3LbM\n\tRQfkQYalXfJIw==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=opfRNeLh+JvCAyt9iGhtbo0bPU8BVsVldNv1wCj/1Io=;\n\tb=j1hBXLLTu4JWjc6PUn5nFvk3mMBKS7eoOFU1QerOoFG6d+Yi0q6E6EdE957mci43y1\n\tUVBXvOYKUSJQ3YGHrcCcwvWR4BBMuUjpFZYhvoheEekgOKJmVSeQ1FWM1tA9rGW8nbkR\n\tn1P2sq3FblwcicblkRlQ3QbmH8+FUeBhAEiMeJT5MzO8tZOwIoGOm4HXOoD5cxTpkNAh\n\tJj7zv5nGZNAP2wXUP+TfQQRe0pPtoqpILumyJjfiLi+oTCs9XzngJPhqJNGOZF41jFyx\n\taz/yoBV1p2SqunCYWIL2o8gc7U4wXyYSMJkt6Y4DAHMVwLVLzKlnXkAsUwTw4LlJX4Du\n\t46rg=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"j1hBXLLT\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=opfRNeLh+JvCAyt9iGhtbo0bPU8BVsVldNv1wCj/1Io=;\n\tb=NmE0KPynXwkr6/x0vetHGSZvEs/fsEuqdBT0MylTWyMyMcqqTMOXsyyRJqe3Cvehj0\n\t5K6Kb0e+5+CxlRSiwr3l7wuBzWlTn+USYxDFsjjWz40CFp7V/JGIkgyHncozDe+Gl6kz\n\tXssd6QpTphvJdHitnAcm/GCCsyDneaJOQmhYfAaUuA7PCYCKBM39iHxl0rQGZSISQDqX\n\tq/otec77Ah2DgHKXwEiBgztv9tVzm3jJl0W6ZMosMaFQhCWu3ARZovLV/NOM36T5wchU\n\tXh1GVjQIM3g2g4+qzNM/+Urync2JCJZC6LLtxhUKxWuTEbWdpEIjxQoQ0ZI0bGQ1UnHr\n\tic2g==","X-Gm-Message-State":"ANoB5pmipzLUkHK+fJ/BL0RumMZHmogbmRtlBVbcfbw0ND2tyPtHBd6j\n\tG/LrpUGrvHA5dHp8okikNTidJxLMBIT9whh3D5akRBGAty12bJtO","X-Google-Smtp-Source":"AA0mqf7VPIraBC66ytzIK51TLNiqCtKy+LXFQvpx6AnIL8b7DYYU06PyMMNAdUg2lIHecyBBk9jvFDuumtEdBZ8BymA=","X-Received":"by 2002:a92:d3cf:0:b0:303:3c91:5f55 with SMTP id\n\tc15-20020a92d3cf000000b003033c915f55mr13565340ilh.151.1670575660062;\n\tFri, 09 Dec 2022 00:47:40 -0800 (PST)","MIME-Version":"1.0","References":"<20221206135459.25521-1-naush@raspberrypi.com>\n\t<20221206135459.25521-9-naush@raspberrypi.com>","In-Reply-To":"<20221206135459.25521-9-naush@raspberrypi.com>","Date":"Fri, 9 Dec 2022 08:47:30 +0000","Message-ID":"","To":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/alternative; boundary=\"000000000000c9f82005ef613449\"","Subject":"Re: [libcamera-devel] [PATCH v3 08/13] pipeline: raspberrypi:\n\tDisable StreamOn for ISP Output0/1 when dropping frames","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]