[{"id":25421,"web_url":"https://patchwork.libcamera.org/comment/25421/","msgid":"","date":"2022-10-14T14:55:46","subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"On Fri, 14 Oct 2022 at 14:18, Naushir Patuck wrote:\n\n> Hi,\n>\n> One big issue that is plaguing libcamera users on Raspberry Pi is memory\n> usage.\n> All our frame buffers have to be allocated in CMA space because of hardware\n> restrictions. Unfortunately CMA space is very limited (particularly on the\n> Raspberry Pi Zero), and is prone to frequent fragmentation as\n> display/rendering\n> buffers also use this pool. Not to mention that the kernel is also allowed\n> to\n> allocate in the CMA pool for its own needs. As a result, some of our\n> libcamera\n> applications intermittently fail to run because of buffer allocation\n> failure,\n> and some use-cases (e.g. > 20MPix captures) will simply not be able to run\n> on\n> some platforms at all.\n>\n> This patch series attempts to try and fix this by easing the memory usage\n> of\n> the Raspberry Pi pipeline handler. It does this by using a platform\n> configuration\n> file which (amongst other things) instructs the pipeline handler on how\n> many\n> internal buffers it should allocate. There may be a performance penality\n> (e.g\n> dropped frames, or longer time taken for initial AE/AWB convergence) that\n> have\n> been documented. But if the application can guarantee certain behavior,\n> we can\n> ensure the pipeline can still process requests with the same performance\n> as we\n> have today. The Raspberry Pi libcamera-apps and picamera2 framework\n> already use\n> this behavior, so the change should be transparent to applications built on\n> these, apart from using significantly less memory.\n>\n> Patches 1 to 4 and the framework for using a configuration file to setup\n> the\n> buffer allocation parameters\n>\n> Patch 5 to 7 improve the buffer handling dealing with dropped frames,\n> allowing\n> us to use fewer internal buffers.\n>\n\nI should add that this particular change requires a firmware update to work\ncorrectly.\nIf you want to test this out, it's best to run \"rpi-update\" to get the\nlatest firmware.\n\nNaush\n\n\n>\n> Patch 8 adds a parameter to modify drop frame behavior - not related to\n> this\n> problem, but a nice to have.\n>\n> Patches 9 and 10 add configuration files for minimal memory configuration,\n> and\n> another for optimal memory usage for libcamera-apps and picamera2.\n>\n> I've considered how these various performance tuning parameters could be\n> made\n> part of the core StreamConfiguration structure or even as a Control.\n> However,\n> they are entirely platform specific, and I am struggling to come up with a\n> clean\n> way to put this into libcamera core with polluting it with various\n> platform specific\n> parameters. If others have any ideas or suggestions on using alternate\n> means\n> of passing platform specific parameters into the pipeline handler, please\n> do\n> comment on this review.\n>\n> Thanks,\n> Naush\n>\n> Naushir Patuck (10):\n> libcamera: pipeline: Add a platform configuration file helper\n> pipeline: raspberrypi: Add a pipeline config structure\n> pipeline: raspberrypi: Split out ISP Output0 buffer allocation\n> pipeline: raspberrypi: Read config parameters from a file\n> pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when\n> dropping frames\n> pipeline: raspberrypi: Reorder startup drop frame initialisation\n> pipeline: raspberrypi: Add a parameter to disable startup drop frames\n> pipeline: raspberrypi: Allow pipeline handler to always use the newest\n> frame\n> pipeline: raspberrypi: Add minimal memory usage config file\n> pipeline: raspberrypi: Add libcamera-apps and picamera2 config file\n>\n> include/libcamera/internal/pipeline_handler.h | 2 +\n> .../pipeline/raspberrypi/data/default.json | 28 +++\n> .../pipeline/raspberrypi/data/meson.build | 10 +\n> .../raspberrypi/data/minimal_mem.json | 28 +++\n> .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++\n> .../pipeline/raspberrypi/meson.build | 2 +\n> .../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++---\n> src/libcamera/pipeline_handler.cpp | 57 +++++\n> 8 files changed, 339 insertions(+), 31 deletions(-)\n> create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json\n> create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build\n> create mode 100644\n> src/libcamera/pipeline/raspberrypi/data/minimal_mem.json\n> create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json\n>\n> --\n> 2.25.1\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BE629BD16B\n\tfor ;\n\tFri, 14 Oct 2022 14:56:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0B80762DC0;\n\tFri, 14 Oct 2022 16:56:05 +0200 (CEST)","from mail-lf1-x131.google.com (mail-lf1-x131.google.com\n\t[IPv6:2a00:1450:4864:20::131])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B845D62D8E\n\tfor ;\n\tFri, 14 Oct 2022 16:56:03 +0200 (CEST)","by mail-lf1-x131.google.com with SMTP id y5so7622686lfl.4\n\tfor ;\n\tFri, 14 Oct 2022 07:56:03 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1665759365;\n\tbh=Gg9c9OXhZJX2oo8tBiZR4/3Ipt0wlivN8I83enImcLw=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=R+SO5npm+unhAbUNKfXa47pKwIK6rokt3OT39cpgZINJ5F3sm8neJNYqXuK/Bj+69\n\t8FgUjr0zjKllCHCsd1VkxxtJ+fvSmzB5RJ5gfDsXHzEy4Nv1h5hRaKH3gNynokcaBp\n\tidK9MUoT5Z4Ds3y6iRPKU9mOrcue17gJXNbhQeiqmypgUCzbszZKbnoMRZItQAGotw\n\tIbGUVTSVeFPSK5DkvueKcY+ZBeNXyhXHwoxioDdB4hdGX/WwQYyiKVbeNeThmNKiP9\n\txOkD5F0prjOBL1rZTQUYRNJM/+J0Lv2VSmsxgEUz7m9Wy3ku494cK+2MjvjPMcj7h/\n\tgmEph88G3Q7yw==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=to:subject:message-id:date:from:in-reply-to:references:mime-version\n\t:from:to:cc:subject:date:message-id:reply-to;\n\tbh=81ki+YvNNHWoVK61fTE2CWvyWqhrD0QT+E0xqJ4HmlY=;\n\tb=EGxG4Y/v4YlmbLMb522rKGm3Trpc3FdKwdMCFdPvq9Rz7evqZz8zY6oBgnXzMeC2VP\n\t8MNS9VOZ7agGAC4ScL5Fw/QRF8wHiPIVSuEa1Ez2u9FJcC7NXzjGyExA22rDvguPUqTi\n\t4aeAqxY4KF3xkqd2f1HUwREj8CyaY6uUsBMoxoljma6B1r6CC7VK7T2vhWVrj6WxzoJo\n\temG1xPui7THFaCuvs9t5n6alJkLAF+imbC9f1vwoStAt2wvd1IRqVQoLz4RnjN0lOOrc\n\t82bSV3GM9CBst37akR5GiQLre7hFrjxXEbnloIOpbeBFi8ZfDNzIBudymO3GdJTcOquO\n\tA0Rw=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"EGxG4Y/v\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=to:subject:message-id:date:from:in-reply-to:references:mime-version\n\t:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n\tbh=81ki+YvNNHWoVK61fTE2CWvyWqhrD0QT+E0xqJ4HmlY=;\n\tb=21hx98aGWItZZjy7ZZcwvzu4V+S7keqiBWE2omFCkHiyM4UbYoATpe/gM1LmeOS24L\n\ttC23EFVHmB+U9L0eV3UGg156DZ70Ac43UFlrw8cLp5iRhKaZV6PgIDZjVyPG+d2WhRvl\n\tf1pSA0iUc6mmS6wjOalGa7yYXyQBxtr9j3ozEjIACRR7QCqypD0pRQ01UmIjwV2k3jRK\n\trP+EECpWZ5Rz5BuP015IhtjJNZW9OXcI0u2vjj/hCRX8y3rbEzavy0O+X7EFxzo/Wxpm\n\tsDhDXzcoAKbGoRpdGQXk4cXzb78BUv0WNBaqa5VCOBwEBTY+ohDZTE2aq123fm6LUYvP\n\tlwVg==","X-Gm-Message-State":"ACrzQf01N8BSIFAIkO9ojLWYBvPfcVrlTyK4KRcDTdKPQq6OwFl/rugz\n\tK4rvzkRW5X3StHvo7pKSqsyIMAIJ+TVvktRrU7irliEbCZU=","X-Google-Smtp-Source":"AMsMyM6rUKMyx20DZ/ubG+BxBy6KrjYII/xp/8NLYQ5h3Hpr/QkfpPfO4qOHH2nTvBylASTzLBl9zxvwodV5UsBU5pY=","X-Received":"by 2002:a05:6512:2616:b0:4a2:1723:cf40 with SMTP id\n\tbt22-20020a056512261600b004a21723cf40mr2034057lfb.354.1665759361978;\n\tFri, 14 Oct 2022 07:56:01 -0700 (PDT)","MIME-Version":"1.0","References":"<20221014131846.27169-1-naush@raspberrypi.com>","In-Reply-To":"<20221014131846.27169-1-naush@raspberrypi.com>","Date":"Fri, 14 Oct 2022 15:55:46 +0100","Message-ID":"","To":"libcamera-devel@lists.libcamera.org","Content-Type":"multipart/alternative; boundary=\"0000000000000d704905eaffd3ba\"","Subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":25431,"web_url":"https://patchwork.libcamera.org/comment/25431/","msgid":"<166599721556.3760285.1007409215962478729@Monstersaurus>","date":"2022-10-17T09:00:15","subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)\n> On Fri, 14 Oct 2022 at 14:18, Naushir Patuck wrote:\n> \n> > Hi,\n> >\n> > One big issue that is plaguing libcamera users on Raspberry Pi is memory\n> > usage.\n> > All our frame buffers have to be allocated in CMA space because of hardware\n> > restrictions. Unfortunately CMA space is very limited (particularly on the\n> > Raspberry Pi Zero), and is prone to frequent fragmentation as\n> > display/rendering\n> > buffers also use this pool. Not to mention that the kernel is also allowed\n> > to\n> > allocate in the CMA pool for its own needs. As a result, some of our\n> > libcamera\n> > applications intermittently fail to run because of buffer allocation\n> > failure,\n> > and some use-cases (e.g. > 20MPix captures) will simply not be able to run\n> > on\n> > some platforms at all.\n> >\n> > This patch series attempts to try and fix this by easing the memory usage\n> > of\n> > the Raspberry Pi pipeline handler. It does this by using a platform\n> > configuration\n> > file which (amongst other things) instructs the pipeline handler on how\n> > many\n> > internal buffers it should allocate. There may be a performance penality\n> > (e.g\n> > dropped frames, or longer time taken for initial AE/AWB convergence) that\n> > have\n> > been documented. But if the application can guarantee certain behavior,\n> > we can\n> > ensure the pipeline can still process requests with the same performance\n> > as we\n> > have today. The Raspberry Pi libcamera-apps and picamera2 framework\n> > already use\n> > this behavior, so the change should be transparent to applications built on\n> > these, apart from using significantly less memory.\n> >\n> > Patches 1 to 4 and the framework for using a configuration file to setup\n> > the\n> > buffer allocation parameters\n> >\n> > Patch 5 to 7 improve the buffer handling dealing with dropped frames,\n> > allowing\n> > us to use fewer internal buffers.\n> >\n> \n> I should add that this particular change requires a firmware update to work\n> correctly.\n> If you want to test this out, it's best to run \"rpi-update\" to get the\n> latest firmware.\n\nMostly from curiosity - is there a 'standard' way for the Raspberry Pi\npipeline handler to interogate or query the rpi firmware to do a check\nin case it is not the correct version?\n\n--\nKieran\n \n> Naush\n> \n> \n> >\n> > Patch 8 adds a parameter to modify drop frame behavior - not related to\n> > this\n> > problem, but a nice to have.\n> >\n> > Patches 9 and 10 add configuration files for minimal memory configuration,\n> > and\n> > another for optimal memory usage for libcamera-apps and picamera2.\n> >\n> > I've considered how these various performance tuning parameters could be\n> > made\n> > part of the core StreamConfiguration structure or even as a Control.\n> > However,\n> > they are entirely platform specific, and I am struggling to come up with a\n> > clean\n> > way to put this into libcamera core with polluting it with various\n> > platform specific\n> > parameters. If others have any ideas or suggestions on using alternate\n> > means\n> > of passing platform specific parameters into the pipeline handler, please\n> > do\n> > comment on this review.\n> >\n> > Thanks,\n> > Naush\n> >\n> > Naushir Patuck (10):\n> > libcamera: pipeline: Add a platform configuration file helper\n> > pipeline: raspberrypi: Add a pipeline config structure\n> > pipeline: raspberrypi: Split out ISP Output0 buffer allocation\n> > pipeline: raspberrypi: Read config parameters from a file\n> > pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when\n> > dropping frames\n> > pipeline: raspberrypi: Reorder startup drop frame initialisation\n> > pipeline: raspberrypi: Add a parameter to disable startup drop frames\n> > pipeline: raspberrypi: Allow pipeline handler to always use the newest\n> > frame\n> > pipeline: raspberrypi: Add minimal memory usage config file\n> > pipeline: raspberrypi: Add libcamera-apps and picamera2 config file\n> >\n> > include/libcamera/internal/pipeline_handler.h | 2 +\n> > .../pipeline/raspberrypi/data/default.json | 28 +++\n> > .../pipeline/raspberrypi/data/meson.build | 10 +\n> > .../raspberrypi/data/minimal_mem.json | 28 +++\n> > .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++\n> > .../pipeline/raspberrypi/meson.build | 2 +\n> > .../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++---\n> > src/libcamera/pipeline_handler.cpp | 57 +++++\n> > 8 files changed, 339 insertions(+), 31 deletions(-)\n> > create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json\n> > create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build\n> > create mode 100644\n> > src/libcamera/pipeline/raspberrypi/data/minimal_mem.json\n> > create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json\n> >\n> > --\n> > 2.25.1\n> >\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D54E2BD16B\n\tfor ;\n\tMon, 17 Oct 2022 09:00:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9A4EB62DF0;\n\tMon, 17 Oct 2022 11:00:20 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E60AC603D1\n\tfor ;\n\tMon, 17 Oct 2022 11:00:18 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 8D987E70;\n\tMon, 17 Oct 2022 11:00:18 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1665997220;\n\tbh=8qG1zRfbgE9zYkm2+P7v9IJc2JzT4sav+yMkB0FF5zM=;\n\th=In-Reply-To:References:To:Date:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=kfm7oJx6GtuponaBQGjjYSPpzM3FZJFYz2iSP0pl3oSE4Xx+nU6a2/AwUbxG/MLc+\n\tExJI3HiGeb23q7GIlcjAPo1mMe/xbEPRQWDoe86CyCDvubIClz9AuSvWUFjJt04RE+\n\ti7trS+4OB9dhrpSCUkv8nD0blwlyqxs2PUNnGIlGCd3z6PIjRgEOcqfmBq71sNt44g\n\t8/jnWZD241sv3dyI1nFRkB/7ff9Oeum2edqjoUKoSUkvENEh16bUekziqFj7AyGZNm\n\tkVCr5XKX4OXEJMrGJ0AOPI1UDWkZxYK/hp70tdsPK2LiC/VJT8uoAClPy5kdSUEkiI\n\t5wFwCbyTy8ZyQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1665997218;\n\tbh=8qG1zRfbgE9zYkm2+P7v9IJc2JzT4sav+yMkB0FF5zM=;\n\th=In-Reply-To:References:Subject:From:To:Date:From;\n\tb=FtBqxoAt04E3NIXym7N4dZ7QIUDrXUurQ001VJw8fuEv5+nMB6zUhZZ25GYIJbVh5\n\tVJcj/ZKVx7qm9+eO7Y9G0pRQNz+IPh0LHMl5p68+DfG2wTFTBj357Wh07HwMUs7Tyx\n\tB//pAzPNxYlGxOZOxBGp+4dxtxGR6R3dx4oivVzk="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"FtBqxoAt\"; dkim-atps=neutral","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"","References":"<20221014131846.27169-1-naush@raspberrypi.com>\n\t","To":"Naushir Patuck ,\n\tlibcamera-devel@lists.libcamera.org","Date":"Mon, 17 Oct 2022 10:00:15 +0100","Message-ID":"<166599721556.3760285.1007409215962478729@Monstersaurus>","User-Agent":"alot/0.10","Subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Kieran Bingham via libcamera-devel\n\t","Reply-To":"Kieran Bingham ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":25432,"web_url":"https://patchwork.libcamera.org/comment/25432/","msgid":"","date":"2022-10-17T09:10:43","subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"On Mon, 17 Oct 2022 at 10:00, Kieran Bingham <\nkieran.bingham@ideasonboard.com> wrote:\n\n> Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)\n> > On Fri, 14 Oct 2022 at 14:18, Naushir Patuck \n> wrote:\n> >\n> > > Hi,\n> > >\n> > > One big issue that is plaguing libcamera users on Raspberry Pi is\n> memory\n> > > usage.\n> > > All our frame buffers have to be allocated in CMA space because of\n> hardware\n> > > restrictions. Unfortunately CMA space is very limited (particularly\n> on the\n> > > Raspberry Pi Zero), and is prone to frequent fragmentation as\n> > > display/rendering\n> > > buffers also use this pool. Not to mention that the kernel is also\n> allowed\n> > > to\n> > > allocate in the CMA pool for its own needs. As a result, some of our\n> > > libcamera\n> > > applications intermittently fail to run because of buffer allocation\n> > > failure,\n> > > and some use-cases (e.g. > 20MPix captures) will simply not be able to\n> run\n> > > on\n> > > some platforms at all.\n> > >\n> > > This patch series attempts to try and fix this by easing the memory\n> usage\n> > > of\n> > > the Raspberry Pi pipeline handler. It does this by using a platform\n> > > configuration\n> > > file which (amongst other things) instructs the pipeline handler on how\n> > > many\n> > > internal buffers it should allocate. There may be a performance\n> penality\n> > > (e.g\n> > > dropped frames, or longer time taken for initial AE/AWB convergence)\n> that\n> > > have\n> > > been documented. But if the application can guarantee certain\n> behavior,\n> > > we can\n> > > ensure the pipeline can still process requests with the same\n> performance\n> > > as we\n> > > have today. The Raspberry Pi libcamera-apps and picamera2 framework\n> > > already use\n> > > this behavior, so the change should be transparent to applications\n> built on\n> > > these, apart from using significantly less memory.\n> > >\n> > > Patches 1 to 4 and the framework for using a configuration file to\n> setup\n> > > the\n> > > buffer allocation parameters\n> > >\n> > > Patch 5 to 7 improve the buffer handling dealing with dropped frames,\n> > > allowing\n> > > us to use fewer internal buffers.\n> > >\n> >\n> > I should add that this particular change requires a firmware update to\n> work\n> > correctly.\n> > If you want to test this out, it's best to run \"rpi-update\" to get the\n> > latest firmware.\n>\n> Mostly from curiosity - is there a 'standard' way for the Raspberry Pi\n> pipeline handler to interogate or query the rpi firmware to do a check\n> in case it is not the correct version?\n>\n\nUnfortunately not in an easy way.\nWe could run a shell command \"vcgencmd version\" to get a sha\nhash, but I would find that really ugly to do in the pipeline hanalder.\n\n\n\n\n>\n> --\n> Kieran\n>\n> > Naush\n> >\n> >\n> > >\n> > > Patch 8 adds a parameter to modify drop frame behavior - not related to\n> > > this\n> > > problem, but a nice to have.\n> > >\n> > > Patches 9 and 10 add configuration files for minimal memory\n> configuration,\n> > > and\n> > > another for optimal memory usage for libcamera-apps and picamera2.\n> > >\n> > > I've considered how these various performance tuning parameters could\n> be\n> > > made\n> > > part of the core StreamConfiguration structure or even as a Control.\n> > > However,\n> > > they are entirely platform specific, and I am struggling to come up\n> with a\n> > > clean\n> > > way to put this into libcamera core with polluting it with various\n> > > platform specific\n> > > parameters. If others have any ideas or suggestions on using alternate\n> > > means\n> > > of passing platform specific parameters into the pipeline handler,\n> please\n> > > do\n> > > comment on this review.\n> > >\n> > > Thanks,\n> > > Naush\n> > >\n> > > Naushir Patuck (10):\n> > > libcamera: pipeline: Add a platform configuration file helper\n> > > pipeline: raspberrypi: Add a pipeline config structure\n> > > pipeline: raspberrypi: Split out ISP Output0 buffer allocation\n> > > pipeline: raspberrypi: Read config parameters from a file\n> > > pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when\n> > > dropping frames\n> > > pipeline: raspberrypi: Reorder startup drop frame initialisation\n> > > pipeline: raspberrypi: Add a parameter to disable startup drop frames\n> > > pipeline: raspberrypi: Allow pipeline handler to always use the\n> newest\n> > > frame\n> > > pipeline: raspberrypi: Add minimal memory usage config file\n> > > pipeline: raspberrypi: Add libcamera-apps and picamera2 config file\n> > >\n> > > include/libcamera/internal/pipeline_handler.h | 2 +\n> > > .../pipeline/raspberrypi/data/default.json | 28 +++\n> > > .../pipeline/raspberrypi/data/meson.build | 10 +\n> > > .../raspberrypi/data/minimal_mem.json | 28 +++\n> > > .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++\n> > > .../pipeline/raspberrypi/meson.build | 2 +\n> > > .../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++---\n> > > src/libcamera/pipeline_handler.cpp | 57 +++++\n> > > 8 files changed, 339 insertions(+), 31 deletions(-)\n> > > create mode 100644\n> src/libcamera/pipeline/raspberrypi/data/default.json\n> > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build\n> > > create mode 100644\n> > > src/libcamera/pipeline/raspberrypi/data/minimal_mem.json\n> > > create mode 100644\n> src/libcamera/pipeline/raspberrypi/data/rpi_apps.json\n> > >\n> > > --\n> > > 2.25.1\n> > >\n> > >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8DD71C0DA4\n\tfor ;\n\tMon, 17 Oct 2022 09:11:01 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id CED7F62DF1;\n\tMon, 17 Oct 2022 11:11:00 +0200 (CEST)","from mail-lf1-x130.google.com (mail-lf1-x130.google.com\n\t[IPv6:2a00:1450:4864:20::130])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D53A7603D1\n\tfor ;\n\tMon, 17 Oct 2022 11:10:59 +0200 (CEST)","by mail-lf1-x130.google.com with SMTP id f37so16480877lfv.8\n\tfor ;\n\tMon, 17 Oct 2022 02:10:59 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1665997860;\n\tbh=fRAfyQ04FBpTcaVSPhMmGIYkP6b7Et3W2o0TQLNf+fk=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=f+l0K9Smu6H9yfzlCtoFiioHUq8C0OIVKCehcoFGqUfTYFKEoDDlQQwp7BMVyVQ2R\n\t4E8T7QbL+6EJ9dIbFrSUtc4PG12DNN1N784BdAVxS0ZmPe7xalQ5EZt6DrJo4rV59t\n\tTain7oBHDbZdg1BfvG8qjwYgoOkb4yjRNBdIVS8ttdAGF+OreVZ0PN5FJmE+neO6Oe\n\tj4x1zme90/t0pHPDGD4B7WCmFFi8gNb18oZiunSeoGQGoKT6vGJ4D6icMI2n9R8jiH\n\twdFl0mbBFZwcGTdcwxK2pf9nW9y27Zb+B1BgPDBNfhPh9uax0B3WvHpB1RSlylxj1n\n\tPlpkAfOGEEQcA==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=0zEA7k17eRBn5jWiIZsIUT7aTpHHYx5JtKNetm8aGgU=;\n\tb=DX+CNa+HEhQ7aBwhyuB8tx1xTXi/gtlWT8d9FLbS3TOmx5PLtIGakv0+7IcjCt+d9u\n\t2u5AjoQRO2UQOTUR/ikJrfcd1q9eXudE6ii979wqo9Y1h8ZGStWhaz5wJFm2wYPGkLbQ\n\tUWEV/zQbKMXeF2dhqH8u5ZAYgOXM88/cvMnEZh9o+tlc+wjhoCrE0f270QC8PrtTEMIG\n\tAVHw7mWk+GwInU0ySWZjhHKVluq4DCvZ1WrRcpocW4vbvYCicsM0kBL9XpcTncF4gV75\n\tZLRFS4S0kv1749qae/9OibOlDOw3kFVwPfiOmkUigqJE4sWsOKEtrp7Rp3rSutWMMTn3\n\tOVMA=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"DX+CNa+H\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=0zEA7k17eRBn5jWiIZsIUT7aTpHHYx5JtKNetm8aGgU=;\n\tb=l8y5H8aKdoRgt7NJ1VkipAmcs9NYUtydXFbwWR+9pJ80uMISO/7KxiRSIJRlauseAr\n\tQWPJ1P6Ibkz2RW+kUEgPh70Q509AOEh5/CFVvdGq3Cm/kjQtBfJ9R737Repg2MPc1rZ2\n\tlsThmkq2Ntzh5jAq0CM/E6egHYOZSSfRFcbIOegONHw+ymWaey/rfUSxdbFkkb1s+sp+\n\tKveUv/VZ1G8WiURXxRf2EhzgVXRcTbhCrL2/0R19p6YV/8h5DzPowl7sui2S6Or2hkCR\n\tWq9L/uoAMGcFVCHW4Q8UCW2jd7SkRdN+mGKwdfrUH8Vl+fL9hPul/CMIr8DnxaaJmgYv\n\tsYcA==","X-Gm-Message-State":"ACrzQf3nmQhUHszYdVaMVHYtKzRx+o5hDBLvvqDvdaDk4ZNKO3yI2VVf\n\tnIQKSmV9tXgkddMzRJkid2udVGl7sTAu89L9dFt6iQ==","X-Google-Smtp-Source":"AMsMyM4VGEzXmqFZRrEXm69MbREN87i/BZ4qk+LSsfMLvZ/6ifQo7fNIUP5QHR2jpePTtXIADdUEPaAKA4q/M5+X+Rk=","X-Received":"by 2002:a05:6512:3d1d:b0:4a2:48c8:8a29 with SMTP id\n\td29-20020a0565123d1d00b004a248c88a29mr4014960lfv.552.1665997858934;\n\tMon, 17 Oct 2022 02:10:58 -0700 (PDT)","MIME-Version":"1.0","References":"<20221014131846.27169-1-naush@raspberrypi.com>\n\t\n\t<166599721556.3760285.1007409215962478729@Monstersaurus>","In-Reply-To":"<166599721556.3760285.1007409215962478729@Monstersaurus>","Date":"Mon, 17 Oct 2022 10:10:43 +0100","Message-ID":"","To":"Kieran Bingham ","Content-Type":"multipart/alternative; boundary=\"0000000000009428a805eb375a26\"","Subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":25434,"web_url":"https://patchwork.libcamera.org/comment/25434/","msgid":"","date":"2022-10-17T09:30:33","subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Naush,\n\nOn Mon, Oct 17, 2022 at 10:10:43AM +0100, Naushir Patuck via libcamera-devel wrote:\n> On Mon, 17 Oct 2022 at 10:00, Kieran Bingham wrote:\n> > Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)\n> > > On Fri, 14 Oct 2022 at 14:18, Naushir Patuck wrote:\n> > > > Hi,\n> > > >\n> > > > One big issue that is plaguing libcamera users on Raspberry Pi is memory usage.\n> > > > All our frame buffers have to be allocated in CMA space because of hardware\n> > > > restrictions. Unfortunately CMA space is very limited (particularly on the\n> > > > Raspberry Pi Zero), and is prone to frequent fragmentation as display/rendering\n> > > > buffers also use this pool. Not to mention that the kernel is also allowed to\n> > > > allocate in the CMA pool for its own needs. As a result, some of our libcamera\n> > > > applications intermittently fail to run because of buffer allocation failure,\n> > > > and some use-cases (e.g. > 20MPix captures) will simply not be able to run on\n> > > > some platforms at all.\n> > > >\n> > > > This patch series attempts to try and fix this by easing the memory usage of\n> > > > the Raspberry Pi pipeline handler. It does this by using a platform configuration\n> > > > file which (amongst other things) instructs the pipeline handler on how many\n> > > > internal buffers it should allocate. There may be a performance penality (e.g\n> > > > dropped frames, or longer time taken for initial AE/AWB convergence) that have\n> > > > been documented. But if the application can guarantee certain behavior, we can\n> > > > ensure the pipeline can still process requests with the same performance as we\n> > > > have today. The Raspberry Pi libcamera-apps and picamera2 framework already use\n> > > > this behavior, so the change should be transparent to applications built on\n> > > > these, apart from using significantly less memory.\n> > > >\n> > > > Patches 1 to 4 and the framework for using a configuration file to setup the\n> > > > buffer allocation parameters\n> > > >\n> > > > Patch 5 to 7 improve the buffer handling dealing with dropped frames, allowing\n> > > > us to use fewer internal buffers.\n> > >\n> > > I should add that this particular change requires a firmware update to work correctly.\n> > > If you want to test this out, it's best to run \"rpi-update\" to get the\n> > > latest firmware.\n> >\n> > Mostly from curiosity - is there a 'standard' way for the Raspberry Pi\n> > pipeline handler to interogate or query the rpi firmware to do a check\n> > in case it is not the correct version?\n> \n> Unfortunately not in an easy way.\n> We could run a shell command \"vcgencmd version\" to get a sha\n> hash, but I would find that really ugly to do in the pipeline hanalder.\n\nNot great indeed.\n\nIs this something we can improve going forward ? I'm thinking of a\nmessage to query the vc4 firmware version at runtime, that the ISP\ndriver would use to retrieve the version. We could then expose it to\nuserspace through the ISP video nodes.\n\nIs there a \"traditional\" version number for the firmware, or just a git\nsha1 ?\n\n> > > > Patch 8 adds a parameter to modify drop frame behavior - not related to this\n> > > > problem, but a nice to have.\n> > > >\n> > > > Patches 9 and 10 add configuration files for minimal memory configuration, and\n> > > > another for optimal memory usage for libcamera-apps and picamera2.\n> > > >\n> > > > I've considered how these various performance tuning parameters could be made\n> > > > part of the core StreamConfiguration structure or even as a Control. However,\n> > > > they are entirely platform specific, and I am struggling to come up with a clean\n> > > > way to put this into libcamera core with polluting it with various platform specific\n> > > > parameters. If others have any ideas or suggestions on using alternate means\n> > > > of passing platform specific parameters into the pipeline handler, please do\n> > > > comment on this review.\n> > > >\n> > > > Thanks,\n> > > > Naush\n> > > >\n> > > > Naushir Patuck (10):\n> > > > libcamera: pipeline: Add a platform configuration file helper\n> > > > pipeline: raspberrypi: Add a pipeline config structure\n> > > > pipeline: raspberrypi: Split out ISP Output0 buffer allocation\n> > > > pipeline: raspberrypi: Read config parameters from a file\n> > > > pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when\n> > > > dropping frames\n> > > > pipeline: raspberrypi: Reorder startup drop frame initialisation\n> > > > pipeline: raspberrypi: Add a parameter to disable startup drop frames\n> > > > pipeline: raspberrypi: Allow pipeline handler to always use the newest\n> > > > frame\n> > > > pipeline: raspberrypi: Add minimal memory usage config file\n> > > > pipeline: raspberrypi: Add libcamera-apps and picamera2 config file\n> > > >\n> > > > include/libcamera/internal/pipeline_handler.h | 2 +\n> > > > .../pipeline/raspberrypi/data/default.json | 28 +++\n> > > > .../pipeline/raspberrypi/data/meson.build | 10 +\n> > > > .../raspberrypi/data/minimal_mem.json | 28 +++\n> > > > .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++\n> > > > .../pipeline/raspberrypi/meson.build | 2 +\n> > > > .../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++---\n> > > > src/libcamera/pipeline_handler.cpp | 57 +++++\n> > > > 8 files changed, 339 insertions(+), 31 deletions(-)\n> > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json\n> > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build\n> > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/minimal_mem.json\n> > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1CB3BBD16B\n\tfor ;\n\tMon, 17 Oct 2022 09:31:00 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 77A7A62DF4;\n\tMon, 17 Oct 2022 11:30:59 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 67AC1603D1\n\tfor ;\n\tMon, 17 Oct 2022 11:30:58 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id AA7DDCCF;\n\tMon, 17 Oct 2022 11:30:57 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1665999059;\n\tbh=Fa512WjA1jpuN0urd+BuYXmj3ob0G6Q8TqgK5YtqDkI=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=eMOZqxdoM74vDfweVtIX+x4nDrqU3HiPaViBG4SqGLbboXha8T7+hrrkZwnqLTjy8\n\tHHJxHqm8ZtF0d7ydBdSOlLm0zQyrTrMcODxvpG0gFnBCsduxhjiyOWJT+HzaLAAD1w\n\tm4ZFUQ/JNr4RELi/QaFie3GjXos6Uam0G+BunXJgC2DVSt8HqwF+dEpuNyz5CFxfU1\n\tY9HZVpPEtsh1HCbfHZxEeJ1+lzDug/6WshEeWP8fXCLXladDSBw+ZgEZVlCD/d3+pc\n\t4QqcNZh1dIpwnMTOmk8L9IxyCzkR37qAadi/k0+ldDKVew+F/iWIspSKG0RNwX3ZpU\n\tTIbEP6AUBV7Yg==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1665999057;\n\tbh=Fa512WjA1jpuN0urd+BuYXmj3ob0G6Q8TqgK5YtqDkI=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=ENTyhrvGPPuKvL+YsomV1v36szB2atSUPTRjhuPL//1QDxdyqLloupK9+aeG22IFN\n\tNHycGZtLj2Gn8QVcjdR9JWNK0EFcoTPgVJmDOIckkFjRtMCVWiu/9/z+zYbyFZsfq3\n\tVBemNggRaaQvHqAEahVMv/STnkAM5KVA5X7d2SbU="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"ENTyhrvG\"; dkim-atps=neutral","Date":"Mon, 17 Oct 2022 12:30:33 +0300","To":"Naushir Patuck ","Message-ID":"","References":"<20221014131846.27169-1-naush@raspberrypi.com>\n\t\n\t<166599721556.3760285.1007409215962478729@Monstersaurus>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":25435,"web_url":"https://patchwork.libcamera.org/comment/25435/","msgid":"","date":"2022-10-17T09:38:42","subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"On Mon, 17 Oct 2022 at 10:30, Laurent Pinchart <\nlaurent.pinchart@ideasonboard.com> wrote:\n\n> Hi Naush,\n>\n> On Mon, Oct 17, 2022 at 10:10:43AM +0100, Naushir Patuck via\n> libcamera-devel wrote:\n> > On Mon, 17 Oct 2022 at 10:00, Kieran Bingham wrote:\n> > > Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)\n> > > > On Fri, 14 Oct 2022 at 14:18, Naushir Patuck wrote:\n> > > > > Hi,\n> > > > >\n> > > > > One big issue that is plaguing libcamera users on Raspberry Pi is\n> memory usage.\n> > > > > All our frame buffers have to be allocated in CMA space because of\n> hardware\n> > > > > restrictions. Unfortunately CMA space is very limited\n> (particularly on the\n> > > > > Raspberry Pi Zero), and is prone to frequent fragmentation as\n> display/rendering\n> > > > > buffers also use this pool. Not to mention that the kernel is also\n> allowed to\n> > > > > allocate in the CMA pool for its own needs. As a result, some of\n> our libcamera\n> > > > > applications intermittently fail to run because of buffer\n> allocation failure,\n> > > > > and some use-cases (e.g. > 20MPix captures) will simply not be\n> able to run on\n> > > > > some platforms at all.\n> > > > >\n> > > > > This patch series attempts to try and fix this by easing the\n> memory usage of\n> > > > > the Raspberry Pi pipeline handler. It does this by using a\n> platform configuration\n> > > > > file which (amongst other things) instructs the pipeline handler\n> on how many\n> > > > > internal buffers it should allocate. There may be a performance\n> penality (e.g\n> > > > > dropped frames, or longer time taken for initial AE/AWB\n> convergence) that have\n> > > > > been documented. But if the application can guarantee certain\n> behavior, we can\n> > > > > ensure the pipeline can still process requests with the same\n> performance as we\n> > > > > have today. The Raspberry Pi libcamera-apps and picamera2\n> framework already use\n> > > > > this behavior, so the change should be transparent to applications\n> built on\n> > > > > these, apart from using significantly less memory.\n> > > > >\n> > > > > Patches 1 to 4 and the framework for using a configuration file to\n> setup the\n> > > > > buffer allocation parameters\n> > > > >\n> > > > > Patch 5 to 7 improve the buffer handling dealing with dropped\n> frames, allowing\n> > > > > us to use fewer internal buffers.\n> > > >\n> > > > I should add that this particular change requires a firmware update\n> to work correctly.\n> > > > If you want to test this out, it's best to run \"rpi-update\" to get\n> the\n> > > > latest firmware.\n> > >\n> > > Mostly from curiosity - is there a 'standard' way for the Raspberry Pi\n> > > pipeline handler to interogate or query the rpi firmware to do a check\n> > > in case it is not the correct version?\n> >\n> > Unfortunately not in an easy way.\n> > We could run a shell command \"vcgencmd version\" to get a sha\n> > hash, but I would find that really ugly to do in the pipeline hanalder.\n>\n> Not great indeed.\n>\n> Is this something we can improve going forward ? I'm thinking of a\n> message to query the vc4 firmware version at runtime, that the ISP\n> driver would use to retrieve the version. We could then expose it to\n> userspace through the ISP video nodes.\n>\n> Is there a \"traditional\" version number for the firmware, or just a git\n> sha1 ?\n>\n\nWe don't have a traditional version number available, only the sha1 hash.\nI doubt this is something that folks over here will have an appetite to\nchange.\n\nHowever, things are not really that bad for the regular user. Our apt\ndependency\nsetup will ensure that if people get the latest libcamera which requires a\nfirmware\nchange, the firmware blob will also be updated. This only really hits\nfolks who\ndo manual builds of libcamera.\n\nNaush\n\n\n\n>\n> > > > > Patch 8 adds a parameter to modify drop frame behavior - not\n> related to this\n> > > > > problem, but a nice to have.\n> > > > >\n> > > > > Patches 9 and 10 add configuration files for minimal memory\n> configuration, and\n> > > > > another for optimal memory usage for libcamera-apps and picamera2.\n> > > > >\n> > > > > I've considered how these various performance tuning parameters\n> could be made\n> > > > > part of the core StreamConfiguration structure or even as a\n> Control. However,\n> > > > > they are entirely platform specific, and I am struggling to come\n> up with a clean\n> > > > > way to put this into libcamera core with polluting it with various\n> platform specific\n> > > > > parameters. If others have any ideas or suggestions on using\n> alternate means\n> > > > > of passing platform specific parameters into the pipeline handler,\n> please do\n> > > > > comment on this review.\n> > > > >\n> > > > > Thanks,\n> > > > > Naush\n> > > > >\n> > > > > Naushir Patuck (10):\n> > > > > libcamera: pipeline: Add a platform configuration file helper\n> > > > > pipeline: raspberrypi: Add a pipeline config structure\n> > > > > pipeline: raspberrypi: Split out ISP Output0 buffer allocation\n> > > > > pipeline: raspberrypi: Read config parameters from a file\n> > > > > pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when\n> > > > > dropping frames\n> > > > > pipeline: raspberrypi: Reorder startup drop frame initialisation\n> > > > > pipeline: raspberrypi: Add a parameter to disable startup drop\n> frames\n> > > > > pipeline: raspberrypi: Allow pipeline handler to always use the\n> newest\n> > > > > frame\n> > > > > pipeline: raspberrypi: Add minimal memory usage config file\n> > > > > pipeline: raspberrypi: Add libcamera-apps and picamera2 config\n> file\n> > > > >\n> > > > > include/libcamera/internal/pipeline_handler.h | 2 +\n> > > > > .../pipeline/raspberrypi/data/default.json | 28 +++\n> > > > > .../pipeline/raspberrypi/data/meson.build | 10 +\n> > > > > .../raspberrypi/data/minimal_mem.json | 28 +++\n> > > > > .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++\n> > > > > .../pipeline/raspberrypi/meson.build | 2 +\n> > > > > .../pipeline/raspberrypi/raspberrypi.cpp | 215\n> +++++++++++++++---\n> > > > > src/libcamera/pipeline_handler.cpp | 57 +++++\n> > > > > 8 files changed, 339 insertions(+), 31 deletions(-)\n> > > > > create mode 100644\n> src/libcamera/pipeline/raspberrypi/data/default.json\n> > > > > create mode 100644\n> src/libcamera/pipeline/raspberrypi/data/meson.build\n> > > > > create mode 100644\n> src/libcamera/pipeline/raspberrypi/data/minimal_mem.json\n> > > > > create mode 100644\n> src/libcamera/pipeline/raspberrypi/data/rpi_apps.json\n>\n> --\n> Regards,\n>\n> Laurent Pinchart\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 85A95C0DA4\n\tfor ;\n\tMon, 17 Oct 2022 09:39:02 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id CE9BD62DF4;\n\tMon, 17 Oct 2022 11:39:01 +0200 (CEST)","from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com\n\t[IPv6:2a00:1450:4864:20::12a])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CF1B0603D1\n\tfor ;\n\tMon, 17 Oct 2022 11:38:59 +0200 (CEST)","by mail-lf1-x12a.google.com with SMTP id be13so978218lfb.4\n\tfor ;\n\tMon, 17 Oct 2022 02:38:59 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1665999541;\n\tbh=vTVYzB15VR+d2O3KCONzCSfID8We+MYnIFn2VK/4cbo=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=C+j74Kvyxe9o4T7iMgJfFhI5mU/GUT25eJpHCw1T74mhP69dg6/MeD/qrjqNG4mfL\n\txLtKLEx11JBe8WrwzarwIPOxcVzoZzGgbJkV1VCngtBzy6mnZJsk2SEDO84bwJWq7b\n\tcH18Zj55awMwEylgJ1sjitCtBcfAbkWqHmOViFKAYUZxYCLciK2KzZe08fArNQaQNp\n\tkrU+luNs7L1DFQbbxTsACDhoxQBgvwySRcCxolDUDC+Qo+Sm/vllSzp7MmfoBfjjnc\n\toCqjlYrMNZ65klF93YyTEhdPosIdb9UbmAEf20O8T910gjWTqftPbZxnr4Qw167opk\n\tNh6Q3ydOed2wQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=lmibPjvHVxLJTAxHNWYmoNwxx7zdr/moTYwJp88FEaM=;\n\tb=kqc8o0cbCf+CthxWoVXiyqLw3bnGEmmErOWOvneYefsMuqTKUfHw3l7u+FagdbODFN\n\txUs4bGS0gJNTpl1LiZDZeKPQE1QsVBCZoi/0Rv7Upy2JGkUKTeYcWjKsrgpJUhcNi3Fk\n\tytFwy005WwlEP8697c8txOvKHAhKlZlKE6815PYaVJymolBB0O8YlKKhyCs5bF8FgUw3\n\tIltjEPY6YxEqdvF5zDfPPfi/Vzyl9522FUK9OWwKiUOPQ02EW9YAL/FsmzJdsQ8wm0ar\n\thRiiUpWiGnuj9LIb/OoBvGbiSoXlsJY+hfeZk3wp50ZrtZOqojoquhkyOt6ud9QjpZLp\n\tWApQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"kqc8o0cb\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=lmibPjvHVxLJTAxHNWYmoNwxx7zdr/moTYwJp88FEaM=;\n\tb=3WbxM0rOCvIXxozTdU+oHPCa5ISqpgk+Ter9MTl9j4pXQRXDf6lSuoro117Tuwoyi3\n\tRFhiWBUIo3XFd5GE6YpKTKAl96e1lhWpDb48tcTVPPl5EQQN1N0UW81nqGjsacMZsqvE\n\tevLaTBlVIHHS2vMKnpoYdNEUFVtMcepC9RKEKUVl/z3pZT8BmvJ8UDQpeXWVfT/MZCn2\n\tgF1hzOZWHIXH7EeacXc5bFPCV5ikv26jiypL7HKGXLCf9MCg/gAlrQI9Js9bIOeJp24x\n\ty0y37TlydH86i92bFTfKSMNtqIraEop8r0OpCgiGoxEt7prPLXx2sCdYsMJWsIJCWBaX\n\tNPUQ==","X-Gm-Message-State":"ACrzQf163Xs2fR26W5tyWDQ8hhDeLWcptSL3UKk5C2nfrqHdyt+zMb1K\n\t/G72AOfTpjAxVHpyI2Hu5xioJItlUHlnNXTVof5fp9dsq9w=","X-Google-Smtp-Source":"AMsMyM6Cz/9LaKV5fH49rzpjtB1t6LW+ADWID72hlqZxadRqZIrC5eSjX6b3Cj4I3Ba8ZjIOSTw2kGxF0a+Z1upcVXI=","X-Received":"by 2002:ac2:4c03:0:b0:4a2:2273:89c6 with SMTP id\n\tt3-20020ac24c03000000b004a2227389c6mr3304330lfq.245.1665999539169;\n\tMon, 17 Oct 2022 02:38:59 -0700 (PDT)","MIME-Version":"1.0","References":"<20221014131846.27169-1-naush@raspberrypi.com>\n\t\n\t<166599721556.3760285.1007409215962478729@Monstersaurus>\n\t\n\t","In-Reply-To":"","Date":"Mon, 17 Oct 2022 10:38:42 +0100","Message-ID":"","To":"Laurent Pinchart ","Content-Type":"multipart/alternative; boundary=\"000000000000ba816705eb37be11\"","Subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Naushir Patuck via libcamera-devel\n\t","Reply-To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":25437,"web_url":"https://patchwork.libcamera.org/comment/25437/","msgid":"","date":"2022-10-17T10:42:39","subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","submitter":{"id":27,"url":"https://patchwork.libcamera.org/api/people/27/","name":"Dave Stevenson","email":"dave.stevenson@raspberrypi.com"},"content":"On Mon, 17 Oct 2022 at 10:39, Naushir Patuck via libcamera-devel\n wrote:\n>\n>\n>\n> On Mon, 17 Oct 2022 at 10:30, Laurent Pinchart wrote:\n>>\n>> Hi Naush,\n>>\n>> On Mon, Oct 17, 2022 at 10:10:43AM +0100, Naushir Patuck via libcamera-devel wrote:\n>> > On Mon, 17 Oct 2022 at 10:00, Kieran Bingham wrote:\n>> > > Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)\n>> > > > On Fri, 14 Oct 2022 at 14:18, Naushir Patuck wrote:\n>> > > > > Hi,\n>> > > > >\n>> > > > > One big issue that is plaguing libcamera users on Raspberry Pi is memory usage.\n>> > > > > All our frame buffers have to be allocated in CMA space because of hardware\n>> > > > > restrictions. Unfortunately CMA space is very limited (particularly on the\n>> > > > > Raspberry Pi Zero), and is prone to frequent fragmentation as display/rendering\n>> > > > > buffers also use this pool. Not to mention that the kernel is also allowed to\n>> > > > > allocate in the CMA pool for its own needs. As a result, some of our libcamera\n>> > > > > applications intermittently fail to run because of buffer allocation failure,\n>> > > > > and some use-cases (e.g. > 20MPix captures) will simply not be able to run on\n>> > > > > some platforms at all.\n>> > > > >\n>> > > > > This patch series attempts to try and fix this by easing the memory usage of\n>> > > > > the Raspberry Pi pipeline handler. It does this by using a platform configuration\n>> > > > > file which (amongst other things) instructs the pipeline handler on how many\n>> > > > > internal buffers it should allocate. There may be a performance penality (e.g\n>> > > > > dropped frames, or longer time taken for initial AE/AWB convergence) that have\n>> > > > > been documented. But if the application can guarantee certain behavior, we can\n>> > > > > ensure the pipeline can still process requests with the same performance as we\n>> > > > > have today. The Raspberry Pi libcamera-apps and picamera2 framework already use\n>> > > > > this behavior, so the change should be transparent to applications built on\n>> > > > > these, apart from using significantly less memory.\n>> > > > >\n>> > > > > Patches 1 to 4 and the framework for using a configuration file to setup the\n>> > > > > buffer allocation parameters\n>> > > > >\n>> > > > > Patch 5 to 7 improve the buffer handling dealing with dropped frames, allowing\n>> > > > > us to use fewer internal buffers.\n>> > > >\n>> > > > I should add that this particular change requires a firmware update to work correctly.\n>> > > > If you want to test this out, it's best to run \"rpi-update\" to get the\n>> > > > latest firmware.\n>> > >\n>> > > Mostly from curiosity - is there a 'standard' way for the Raspberry Pi\n>> > > pipeline handler to interogate or query the rpi firmware to do a check\n>> > > in case it is not the correct version?\n>> >\n>> > Unfortunately not in an easy way.\n>> > We could run a shell command \"vcgencmd version\" to get a sha\n>> > hash, but I would find that really ugly to do in the pipeline hanalder.\n>>\n>> Not great indeed.\n>>\n>> Is this something we can improve going forward ? I'm thinking of a\n>> message to query the vc4 firmware version at runtime, that the ISP\n>> driver would use to retrieve the version. We could then expose it to\n>> userspace through the ISP video nodes.\n>>\n>> Is there a \"traditional\" version number for the firmware, or just a git\n>> sha1 ?\n>\n>\n> We don't have a traditional version number available, only the sha1 hash.\n> I doubt this is something that folks over here will have an appetite to change.\n\nWe do have both the hash and build date available through the firmware\ninterface. The firmware driver logs it on boot\n[ 0.070048] raspberrypi-firmware soc:firmware: Attached to firmware\nfrom 2022-08-26T14:03:16, variant start\nCode at [1].\n\nThe ISP driver could read that and add it to either a custom control,\nor via part of the device description.\n\n Dave\n\n[1] https://github.com/raspberrypi/linux/blob/rpi-5.15.y/drivers/firmware/raspberrypi.c#L253\n\n> However, things are not really that bad for the regular user. Our apt dependency\n> setup will ensure that if people get the latest libcamera which requires a firmware\n> change, the firmware blob will also be updated. This only really hits folks who\n> do manual builds of libcamera.\n>\n> Naush\n>\n>\n>>\n>>\n>> > > > > Patch 8 adds a parameter to modify drop frame behavior - not related to this\n>> > > > > problem, but a nice to have.\n>> > > > >\n>> > > > > Patches 9 and 10 add configuration files for minimal memory configuration, and\n>> > > > > another for optimal memory usage for libcamera-apps and picamera2.\n>> > > > >\n>> > > > > I've considered how these various performance tuning parameters could be made\n>> > > > > part of the core StreamConfiguration structure or even as a Control. However,\n>> > > > > they are entirely platform specific, and I am struggling to come up with a clean\n>> > > > > way to put this into libcamera core with polluting it with various platform specific\n>> > > > > parameters. If others have any ideas or suggestions on using alternate means\n>> > > > > of passing platform specific parameters into the pipeline handler, please do\n>> > > > > comment on this review.\n>> > > > >\n>> > > > > Thanks,\n>> > > > > Naush\n>> > > > >\n>> > > > > Naushir Patuck (10):\n>> > > > > libcamera: pipeline: Add a platform configuration file helper\n>> > > > > pipeline: raspberrypi: Add a pipeline config structure\n>> > > > > pipeline: raspberrypi: Split out ISP Output0 buffer allocation\n>> > > > > pipeline: raspberrypi: Read config parameters from a file\n>> > > > > pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when\n>> > > > > dropping frames\n>> > > > > pipeline: raspberrypi: Reorder startup drop frame initialisation\n>> > > > > pipeline: raspberrypi: Add a parameter to disable startup drop frames\n>> > > > > pipeline: raspberrypi: Allow pipeline handler to always use the newest\n>> > > > > frame\n>> > > > > pipeline: raspberrypi: Add minimal memory usage config file\n>> > > > > pipeline: raspberrypi: Add libcamera-apps and picamera2 config file\n>> > > > >\n>> > > > > include/libcamera/internal/pipeline_handler.h | 2 +\n>> > > > > .../pipeline/raspberrypi/data/default.json | 28 +++\n>> > > > > .../pipeline/raspberrypi/data/meson.build | 10 +\n>> > > > > .../raspberrypi/data/minimal_mem.json | 28 +++\n>> > > > > .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++\n>> > > > > .../pipeline/raspberrypi/meson.build | 2 +\n>> > > > > .../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++---\n>> > > > > src/libcamera/pipeline_handler.cpp | 57 +++++\n>> > > > > 8 files changed, 339 insertions(+), 31 deletions(-)\n>> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json\n>> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build\n>> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/minimal_mem.json\n>> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json\n>>\n>> --\n>> Regards,\n>>\n>> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id AD53AC0DA4\n\tfor ;\n\tMon, 17 Oct 2022 10:42:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E6FDE62DF8;\n\tMon, 17 Oct 2022 12:42:56 +0200 (CEST)","from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com\n\t[IPv6:2a00:1450:4864:20::62c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 73E7861F5D\n\tfor ;\n\tMon, 17 Oct 2022 12:42:55 +0200 (CEST)","by mail-ej1-x62c.google.com with SMTP id k2so23928504ejr.2\n\tfor ;\n\tMon, 17 Oct 2022 03:42:55 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1666003377;\n\tbh=vLPaPUaFZX2hdITz2E/cQ7E+6nVi2wp0W1EIYdHrRew=;\n\th=References:In-Reply-To:Date:To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=V/GMR8jGd96zlDhAsHLNNczagOppJdcAsjYBPT2GAGj79TONF7xktNXQXbYTVMrrO\n\tx+Hrjap6ehDJkN8Tx78dZeUjNb67K/UBRJeQILe12/Ro9dU2efeN5yrZ1wVNgq8Uc+\n\tKRZITPIkwXNkZUb0AggLpg9k3ctB0veVn3KeFR8ytusTECvn34NpQRw/Ra5oHsLfVB\n\ttMIozc6KcohLjhGCQ1je+CuiRgJZmpfTaPOAwcvD2lRKExtiU30k0s166Pm2ySUtwm\n\t6HLWHcqh7+hL/bO/uYnHcLd91OIh1lrYtMmPLJfJhLtglV9Q+9xeMuqITyxPjiX97b\n\tedopLOFLzKnGw==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=KRoVXajHdq+uzu9Hq8bp514SS+M6TB+8ozfi7+dYNws=;\n\tb=USzP4Fy1lnDaFeY3TvBNwwXCC3j3zoXzs6JTA5fgMElrgW+wevOmuLWmisNP/zlrES\n\tWOsuXg4rzH/04oaaDnXbHmUUpKPIxaKnhBtEKQ/k9pNNb7ftik5La/ZbLWGVeJvT0mRz\n\thBuYDBV/jq4V2CBjNBdCytEUN2fSLr9gPi7+lR+YHIK+0RWLvo3UCU5R2q3JjjWFedJz\n\tMu9X5o2Bx97R4dNDWUAjJuQZVG3E0eJ/te6qkhFEP/ehTiLMuww9nRCWDhxgomrxxFR0\n\t3f2SU22PeMzB2KPIXNNCZQJ7RZGd+JmpKi1GxE+sVgmAZ2SMf8QaX37EZfqlAGl9qwPu\n\tlvsQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"USzP4Fy1\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=KRoVXajHdq+uzu9Hq8bp514SS+M6TB+8ozfi7+dYNws=;\n\tb=rivH7gRv79SWx7kCti9jChy8kvU9Yn3t6ayh5IopJdJs7i4Afu/gfa9KTexBHClLCt\n\t//hgvPqvhP13kL5HwQn8tKXGEuN4H28qBWeaJ2Uf/KJHcrykOQwrPg/vVNYHPY+bEuQ4\n\t61uM1W5QZHmfaYNTdU+dodbg5NVtmpXN6f0Kq4SapWqeh4sOC8Vq9m3FARRB1ZpK7FhZ\n\t17UW2W9t81YhFE4Oo4bvH/aMKYovLGUBTqsQJUSFOkEyWg29+JYG40Fx+JjC16PD9mZe\n\tQZejAMPlkXaL1n6pkoJqIlycAQNcYw7Tld4VYE4i05DYz4Ga7jQMUB7QuTPPMNq591vV\n\tqk8w==","X-Gm-Message-State":"ACrzQf3LeL9fnB6iCZTNBZW8Ry3NXipKZNlZLyJmX067jFg65gCkreH8\n\tPaNB4L2j322BJpg6SrauUyDsV5s4QYwOqploP+pQrw==","X-Google-Smtp-Source":"AMsMyM40uSowXlSj2pejNKSlWsMgSWflQIbL5+x041LsPirkCXMSO98nwHJWbA7tpX06yeqPL7kpZyA3QKbV0byBfvw=","X-Received":"by 2002:a17:907:80b:b0:77a:86a1:db52 with SMTP id\n\twv11-20020a170907080b00b0077a86a1db52mr8463576ejb.294.1666003374878;\n\tMon, 17 Oct 2022 03:42:54 -0700 (PDT)","MIME-Version":"1.0","References":"<20221014131846.27169-1-naush@raspberrypi.com>\n\t\n\t<166599721556.3760285.1007409215962478729@Monstersaurus>\n\t\n\t\n\t","In-Reply-To":"","Date":"Mon, 17 Oct 2022 11:42:39 +0100","Message-ID":"","To":"Naushir Patuck ","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Dave Stevenson via libcamera-devel\n\t","Reply-To":"Dave Stevenson ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":25482,"web_url":"https://patchwork.libcamera.org/comment/25482/","msgid":"","date":"2022-10-19T21:19:49","subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Dave,\n\nOn Mon, Oct 17, 2022 at 11:42:39AM +0100, Dave Stevenson wrote:\n> On Mon, 17 Oct 2022 at 10:39, Naushir Patuck wrote:\n> > On Mon, 17 Oct 2022 at 10:30, Laurent Pinchart wrote:\n> >> On Mon, Oct 17, 2022 at 10:10:43AM +0100, Naushir Patuck via libcamera-devel wrote:\n> >> > On Mon, 17 Oct 2022 at 10:00, Kieran Bingham wrote:\n> >> > > Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)\n> >> > > > On Fri, 14 Oct 2022 at 14:18, Naushir Patuck wrote:\n> >> > > > > Hi,\n> >> > > > >\n> >> > > > > One big issue that is plaguing libcamera users on Raspberry Pi is memory usage.\n> >> > > > > All our frame buffers have to be allocated in CMA space because of hardware\n> >> > > > > restrictions. Unfortunately CMA space is very limited (particularly on the\n> >> > > > > Raspberry Pi Zero), and is prone to frequent fragmentation as display/rendering\n> >> > > > > buffers also use this pool. Not to mention that the kernel is also allowed to\n> >> > > > > allocate in the CMA pool for its own needs. As a result, some of our libcamera\n> >> > > > > applications intermittently fail to run because of buffer allocation failure,\n> >> > > > > and some use-cases (e.g. > 20MPix captures) will simply not be able to run on\n> >> > > > > some platforms at all.\n> >> > > > >\n> >> > > > > This patch series attempts to try and fix this by easing the memory usage of\n> >> > > > > the Raspberry Pi pipeline handler. It does this by using a platform configuration\n> >> > > > > file which (amongst other things) instructs the pipeline handler on how many\n> >> > > > > internal buffers it should allocate. There may be a performance penality (e.g\n> >> > > > > dropped frames, or longer time taken for initial AE/AWB convergence) that have\n> >> > > > > been documented. But if the application can guarantee certain behavior, we can\n> >> > > > > ensure the pipeline can still process requests with the same performance as we\n> >> > > > > have today. The Raspberry Pi libcamera-apps and picamera2 framework already use\n> >> > > > > this behavior, so the change should be transparent to applications built on\n> >> > > > > these, apart from using significantly less memory.\n> >> > > > >\n> >> > > > > Patches 1 to 4 and the framework for using a configuration file to setup the\n> >> > > > > buffer allocation parameters\n> >> > > > >\n> >> > > > > Patch 5 to 7 improve the buffer handling dealing with dropped frames, allowing\n> >> > > > > us to use fewer internal buffers.\n> >> > > >\n> >> > > > I should add that this particular change requires a firmware update to work correctly.\n> >> > > > If you want to test this out, it's best to run \"rpi-update\" to get the\n> >> > > > latest firmware.\n> >> > >\n> >> > > Mostly from curiosity - is there a 'standard' way for the Raspberry Pi\n> >> > > pipeline handler to interogate or query the rpi firmware to do a check\n> >> > > in case it is not the correct version?\n> >> >\n> >> > Unfortunately not in an easy way.\n> >> > We could run a shell command \"vcgencmd version\" to get a sha\n> >> > hash, but I would find that really ugly to do in the pipeline hanalder.\n> >>\n> >> Not great indeed.\n> >>\n> >> Is this something we can improve going forward ? I'm thinking of a\n> >> message to query the vc4 firmware version at runtime, that the ISP\n> >> driver would use to retrieve the version. We could then expose it to\n> >> userspace through the ISP video nodes.\n> >>\n> >> Is there a \"traditional\" version number for the firmware, or just a git\n> >> sha1 ?\n> >\n> >\n> > We don't have a traditional version number available, only the sha1 hash.\n> > I doubt this is something that folks over here will have an appetite to change.\n> \n> We do have both the hash and build date available through the firmware\n> interface. The firmware driver logs it on boot\n> [ 0.070048] raspberrypi-firmware soc:firmware: Attached to firmware\n> from 2022-08-26T14:03:16, variant start\n> Code at [1].\n> \n> The ISP driver could read that and add it to either a custom control,\n> or via part of the device description.\n\nNo urgency, but I think that wouldbe nice to have. As you're fully in\ncontrol of the firmware, I assume that a more recent release date will\nalways mean a more recent version, and that could be correlated with a\nfeature set ?\n\n> [1] https://github.com/raspberrypi/linux/blob/rpi-5.15.y/drivers/firmware/raspberrypi.c#L253\n> \n> > However, things are not really that bad for the regular user. Our apt dependency\n> > setup will ensure that if people get the latest libcamera which requires a firmware\n> > change, the firmware blob will also be updated. This only really hits folks who\n> > do manual builds of libcamera.\n> >\n> >> > > > > Patch 8 adds a parameter to modify drop frame behavior - not related to this\n> >> > > > > problem, but a nice to have.\n> >> > > > >\n> >> > > > > Patches 9 and 10 add configuration files for minimal memory configuration, and\n> >> > > > > another for optimal memory usage for libcamera-apps and picamera2.\n> >> > > > >\n> >> > > > > I've considered how these various performance tuning parameters could be made\n> >> > > > > part of the core StreamConfiguration structure or even as a Control. However,\n> >> > > > > they are entirely platform specific, and I am struggling to come up with a clean\n> >> > > > > way to put this into libcamera core with polluting it with various platform specific\n> >> > > > > parameters. If others have any ideas or suggestions on using alternate means\n> >> > > > > of passing platform specific parameters into the pipeline handler, please do\n> >> > > > > comment on this review.\n> >> > > > >\n> >> > > > > Thanks,\n> >> > > > > Naush\n> >> > > > >\n> >> > > > > Naushir Patuck (10):\n> >> > > > > libcamera: pipeline: Add a platform configuration file helper\n> >> > > > > pipeline: raspberrypi: Add a pipeline config structure\n> >> > > > > pipeline: raspberrypi: Split out ISP Output0 buffer allocation\n> >> > > > > pipeline: raspberrypi: Read config parameters from a file\n> >> > > > > pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when\n> >> > > > > dropping frames\n> >> > > > > pipeline: raspberrypi: Reorder startup drop frame initialisation\n> >> > > > > pipeline: raspberrypi: Add a parameter to disable startup drop frames\n> >> > > > > pipeline: raspberrypi: Allow pipeline handler to always use the newest\n> >> > > > > frame\n> >> > > > > pipeline: raspberrypi: Add minimal memory usage config file\n> >> > > > > pipeline: raspberrypi: Add libcamera-apps and picamera2 config file\n> >> > > > >\n> >> > > > > include/libcamera/internal/pipeline_handler.h | 2 +\n> >> > > > > .../pipeline/raspberrypi/data/default.json | 28 +++\n> >> > > > > .../pipeline/raspberrypi/data/meson.build | 10 +\n> >> > > > > .../raspberrypi/data/minimal_mem.json | 28 +++\n> >> > > > > .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++\n> >> > > > > .../pipeline/raspberrypi/meson.build | 2 +\n> >> > > > > .../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++---\n> >> > > > > src/libcamera/pipeline_handler.cpp | 57 +++++\n> >> > > > > 8 files changed, 339 insertions(+), 31 deletions(-)\n> >> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json\n> >> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build\n> >> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/minimal_mem.json\n> >> > > > > create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 07BB3C0DA4\n\tfor ;\n\tWed, 19 Oct 2022 21:20:16 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B94FB62E6F;\n\tWed, 19 Oct 2022 23:20:15 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 25D7562DD1\n\tfor ;\n\tWed, 19 Oct 2022 23:20:14 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A6C24570;\n\tWed, 19 Oct 2022 23:20:13 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1666214415;\n\tbh=ot5JrRohgiA1ur6yBiwmJ2lZhyS6ny70pdpoXOSHbCU=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=SMIL5N7G5Qunc65eUy9MQf0O5lW0emPq3+ciHXmmSklMwZroGfEvN1BRYc3pSHWY+\n\tWtNBMybvSSb/5LnuStlBk2Yn6GKmgW2sQAKnPmBSry2g4LMBGUTiy+TgHjb7LaG29V\n\tetAocUXAbICly9dewRnkEoom1Y1mPGq9ZpQ3xiQKC/o3E2S1seKZ8b0QqgYshhCnga\n\tGDXibjd43ejFNedC6aY/mlE8O7LheelFjrOPbIBUA2yn4QnDrsnsHmmiItpLugf9QP\n\t5tQNEV3cJgQQ7u63aJr1rV8A0zBDPjTqL7u2FyGhp/3fW4WcLrMnBIcUSu7mvf9ktY\n\t/11vV5Mwlu51g==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1666214413;\n\tbh=ot5JrRohgiA1ur6yBiwmJ2lZhyS6ny70pdpoXOSHbCU=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=IsBRWam77fY29sYOP+TNGbyIowj0njxCqMO7in2n9W9kC+ER8Vw4qwKykOia4ka/U\n\trlF1kd8AKxe/rmRfxGeYX2XfEUJswp/IluyvkhAN+hutyhiq4y5ohEYO2YdUcbkb6s\n\tkHaF+/wd7z5SXfVYAAuFAv2BorK8g8CgiaPuQvD4="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"IsBRWam7\"; dkim-atps=neutral","Date":"Thu, 20 Oct 2022 00:19:49 +0300","To":"Dave Stevenson ","Message-ID":"","References":"<20221014131846.27169-1-naush@raspberrypi.com>\n\t\n\t<166599721556.3760285.1007409215962478729@Monstersaurus>\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform\n\tconfiguration and buffer allocation improvements","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]