[{"id":35806,"web_url":"https://patchwork.libcamera.org/comment/35806/","msgid":"<20250912140020.GG31682@pendragon.ideasonboard.com>","date":"2025-09-12T14:00:20","subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Fri, Sep 12, 2025 at 02:55:27PM +0200, Barnabás Pőcze wrote:\n> Other code generation scripts do that already and let pyyaml deal with\n> decoding utf-8, etc. So do the same here as well.\n\nHow does pyyaml determine the encoding ? Does it just hardcode utf-8 ?\n\n> Signed-off-by: Barnabás Pőcze \n> ---\n> src/py/libcamera/gen-py-formats.py | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/src/py/libcamera/gen-py-formats.py b/src/py/libcamera/gen-py-formats.py\n> index 0ff1d12ac..6323e237f 100755\n> --- a/src/py/libcamera/gen-py-formats.py\n> +++ b/src/py/libcamera/gen-py-formats.py\n> @@ -37,7 +37,7 @@ def main(argv):\n> help='Template file name.')\n> args = parser.parse_args(argv[1:])\n> \n> - with open(args.input, encoding='utf-8') as f:\n> + with open(args.input, 'rb') as f:\n> formats = yaml.safe_load(f)['formats']\n> \n> data = generate(formats)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 132D7BDB13\n\tfor ;\n\tFri, 12 Sep 2025 14:00:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0EC076936A;\n\tFri, 12 Sep 2025 16:00:49 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E0C7669367\n\tfor ;\n\tFri, 12 Sep 2025 16:00:46 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 67617596;\n\tFri, 12 Sep 2025 15:59:31 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"VQwY84xL\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1757685571;\n\tbh=DXqzls1rgH+Zh4t+ezfIpUp/XrqoZiXyBHBRCksKylA=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=VQwY84xL7y/Kio2/huTVsoSr/aXKsRbTsz57D7oUgPevKBCemnP2dFSFLQgEd/RLu\n\tfbPRGeMpWl9lK8d/DBVp6C3ZGZM6cqRFI49gxVUanlDtU211SpO2NPA2k35M+wqeG3\n\tvGt4FYIPtgEUe6r8erVR86XVdLw25ek9KR+Hx3qg=","Date":"Fri, 12 Sep 2025 17:00:20 +0300","From":"Laurent Pinchart ","To":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","Message-ID":"<20250912140020.GG31682@pendragon.ideasonboard.com>","References":"<20250912125528.1963619-1-barnabas.pocze@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20250912125528.1963619-1-barnabas.pocze@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":35807,"web_url":"https://patchwork.libcamera.org/comment/35807/","msgid":"<66173321-c368-4660-83bb-a56e2e247837@ideasonboard.com>","date":"2025-09-12T14:08:50","subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","submitter":{"id":216,"url":"https://patchwork.libcamera.org/api/people/216/","name":"Barnabás Pőcze","email":"barnabas.pocze@ideasonboard.com"},"content":"2025. 09. 12. 16:00 keltezéssel, Laurent Pinchart írta:\n> On Fri, Sep 12, 2025 at 02:55:27PM +0200, Barnabás Pőcze wrote:\n>> Other code generation scripts do that already and let pyyaml deal with\n>> decoding utf-8, etc. So do the same here as well.\n> \n> How does pyyaml determine the encoding ? Does it just hardcode utf-8 ?\n\nhttps://yaml.org/spec/1.2.2/#52-character-encodings says that if there is no\nBOM, then it is utf-8. And additionally:\n\n If a character stream begins with a byte order mark, the character encoding will be\n taken to be as indicated by the byte order mark. Otherwise, the stream must begin\n with an ASCII character. This allows the encoding to be deduced by the pattern of\n null (x00) characters.\n\nSo for our purposes it will deduce utf-8 since no yaml file that is used here starts\nwith a BOM or a \"long ascii character\" as far as I can tell.\n\nDue to this special behaviour, I'd say opening it in binary mode is the correct choice.\n\n\nRegards,\nBarnabás Pőcze\n\n\n> \n>> Signed-off-by: Barnabás Pőcze \n>> ---\n>> src/py/libcamera/gen-py-formats.py | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/src/py/libcamera/gen-py-formats.py b/src/py/libcamera/gen-py-formats.py\n>> index 0ff1d12ac..6323e237f 100755\n>> --- a/src/py/libcamera/gen-py-formats.py\n>> +++ b/src/py/libcamera/gen-py-formats.py\n>> @@ -37,7 +37,7 @@ def main(argv):\n>> help='Template file name.')\n>> args = parser.parse_args(argv[1:])\n>> \n>> - with open(args.input, encoding='utf-8') as f:\n>> + with open(args.input, 'rb') as f:\n>> formats = yaml.safe_load(f)['formats']\n>> \n>> data = generate(formats)\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 412F0C324E\n\tfor ;\n\tFri, 12 Sep 2025 14:08:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 45BAF6936F;\n\tFri, 12 Sep 2025 16:08:58 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EB4DC69367\n\tfor ;\n\tFri, 12 Sep 2025 16:08:56 +0200 (CEST)","from [192.168.33.2] (185.221.142.115.nat.pool.zt.hu\n\t[185.221.142.115])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A1440C7B;\n\tFri, 12 Sep 2025 16:07:39 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"iMal5Y1y\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1757686059;\n\tbh=fJ+veVMQF3EDc8QyX0daiK9G9C+9GPpFN5rfhP7ifmg=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=iMal5Y1yLEZsYju9jiic4ORjyqRDYWdEdEKx+EVxk+pGaRqPyg6J1J0YMe83bnXp9\n\tJcQMMwZrK99LLaNdnMiHWp6iOc5fVVlLHInGq4SPlmg6EEbaT6hVcXZFqARLMZgXkP\n\tglIQZTl4sDQGbUr7xVazVdTVliL7bNTv7i5C1BDM=","Message-ID":"<66173321-c368-4660-83bb-a56e2e247837@ideasonboard.com>","Date":"Fri, 12 Sep 2025 16:08:50 +0200","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","References":"<20250912125528.1963619-1-barnabas.pocze@ideasonboard.com>\n\t<20250912140020.GG31682@pendragon.ideasonboard.com>","From":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ","Content-Language":"en-US, hu-HU","In-Reply-To":"<20250912140020.GG31682@pendragon.ideasonboard.com>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":35809,"web_url":"https://patchwork.libcamera.org/comment/35809/","msgid":"<20250912142322.GH31682@pendragon.ideasonboard.com>","date":"2025-09-12T14:23:22","subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Fri, Sep 12, 2025 at 04:08:50PM +0200, Barnabás Pőcze wrote:\n> 2025. 09. 12. 16:00 keltezéssel, Laurent Pinchart írta:\n> > On Fri, Sep 12, 2025 at 02:55:27PM +0200, Barnabás Pőcze wrote:\n> >> Other code generation scripts do that already and let pyyaml deal with\n> >> decoding utf-8, etc. So do the same here as well.\n> > \n> > How does pyyaml determine the encoding ? Does it just hardcode utf-8 ?\n> \n> https://yaml.org/spec/1.2.2/#52-character-encodings says that if there is no\n> BOM, then it is utf-8. And additionally:\n> \n> If a character stream begins with a byte order mark, the character encoding will be\n> taken to be as indicated by the byte order mark. Otherwise, the stream must begin\n> with an ASCII character. This allows the encoding to be deduced by the pattern of\n> null (x00) characters.\n> \n> So for our purposes it will deduce utf-8 since no yaml file that is used here starts\n> with a BOM or a \"long ascii character\" as far as I can tell.\n> \n> Due to this special behaviour, I'd say opening it in binary mode is the correct choice.\n\nReviewed-by: Laurent Pinchart \n\n> >> Signed-off-by: Barnabás Pőcze \n> >> ---\n> >> src/py/libcamera/gen-py-formats.py | 2 +-\n> >> 1 file changed, 1 insertion(+), 1 deletion(-)\n> >>\n> >> diff --git a/src/py/libcamera/gen-py-formats.py b/src/py/libcamera/gen-py-formats.py\n> >> index 0ff1d12ac..6323e237f 100755\n> >> --- a/src/py/libcamera/gen-py-formats.py\n> >> +++ b/src/py/libcamera/gen-py-formats.py\n> >> @@ -37,7 +37,7 @@ def main(argv):\n> >> help='Template file name.')\n> >> args = parser.parse_args(argv[1:])\n> >> \n> >> - with open(args.input, encoding='utf-8') as f:\n> >> + with open(args.input, 'rb') as f:\n> >> formats = yaml.safe_load(f)['formats']\n> >> \n> >> data = generate(formats)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C0BA5C324E\n\tfor ;\n\tFri, 12 Sep 2025 14:23:51 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B7EB06936A;\n\tFri, 12 Sep 2025 16:23:50 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 06AC969367\n\tfor ;\n\tFri, 12 Sep 2025 16:23:49 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 67638ED;\n\tFri, 12 Sep 2025 16:22:33 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"kSN70d0z\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1757686953;\n\tbh=tJgIejLdanYlJSJncqE2lJVHwR/GBRo8REsjuGRS49o=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=kSN70d0z+nDo4FN9JXjHr6ZPpRCB1FwcNTurVO8NclrnaiuGx+cflcGVNrrrE4kO1\n\tWW/rIAmn2mlq7SaPuf2sO048Oslxa2N1NLIbanzyBq4YkSVHiD5iL+SOalEVkouokN\n\t+K+LdPym09HJKr/OmZj0jkMSXcPVdC3RPZ5eI4b8=","Date":"Fri, 12 Sep 2025 17:23:22 +0300","From":"Laurent Pinchart ","To":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","Message-ID":"<20250912142322.GH31682@pendragon.ideasonboard.com>","References":"<20250912125528.1963619-1-barnabas.pocze@ideasonboard.com>\n\t<20250912140020.GG31682@pendragon.ideasonboard.com>\n\t<66173321-c368-4660-83bb-a56e2e247837@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<66173321-c368-4660-83bb-a56e2e247837@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36590,"web_url":"https://patchwork.libcamera.org/comment/36590/","msgid":"","date":"2025-11-01T16:21:47","subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Barnabás\n\nOn Fri, Sep 12, 2025 at 02:55:27PM +0200, Barnabás Pőcze wrote:\n> Other code generation scripts do that already and let pyyaml deal with\n> decoding utf-8, etc. So do the same here as well.\n>\n> Signed-off-by: Barnabás Pőcze \n\nReviewed-by: Jacopo Mondi \n\nThanks\n j\n\n> ---\n> src/py/libcamera/gen-py-formats.py | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/src/py/libcamera/gen-py-formats.py b/src/py/libcamera/gen-py-formats.py\n> index 0ff1d12ac..6323e237f 100755\n> --- a/src/py/libcamera/gen-py-formats.py\n> +++ b/src/py/libcamera/gen-py-formats.py\n> @@ -37,7 +37,7 @@ def main(argv):\n> help='Template file name.')\n> args = parser.parse_args(argv[1:])\n>\n> - with open(args.input, encoding='utf-8') as f:\n> + with open(args.input, 'rb') as f:\n> formats = yaml.safe_load(f)['formats']\n>\n> data = generate(formats)\n> --\n> 2.51.0\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C821AC3241\n\tfor ;\n\tSat, 1 Nov 2025 16:21:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 31CC86086F;\n\tSat, 1 Nov 2025 17:21:52 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DF8A9606D5\n\tfor ;\n\tSat, 1 Nov 2025 17:21:50 +0100 (CET)","from ideasonboard.com (unknown\n\t[IPv6:2001:b07:6462:5de2:153:f9b8:5024:faa2])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 05548666;\n\tSat, 1 Nov 2025 17:19:58 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Tc6exlE7\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1762013999;\n\tbh=vFxAdZSScrIctrc9YyYh+o0Et59UbnEuXXl4GhxlGeU=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=Tc6exlE7mkB1DoE/LWYg2bXKQC28HLQ006f6JGIfJyTALyV+xt2TUZNxXDW2lRjC4\n\tl4Y7y6tXmjk+qDpHvqpc2qIm8bkAf65NMTn9C+PTtdz82WPk8qRALMzS8UcIAdGd6R\n\txYZ9Z5TYj86L2uz2akH80Q0pE07l2Q9PkHsFlgRk=","Date":"Sat, 1 Nov 2025 17:21:47 +0100","From":"Jacopo Mondi ","To":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ","Cc":"libcamera-devel@lists.libcamera.org, \n\tLaurent Pinchart ","Subject":"Re: [RFC PATCH v2 1/2] py: gen-py-formats.py: Open input file in\n\tbinary mode","Message-ID":"","References":"<20250912125528.1963619-1-barnabas.pocze@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20250912125528.1963619-1-barnabas.pocze@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]