[{"id":2151,"web_url":"https://patchwork.libcamera.org/comment/2151/","msgid":"<9e0e8baf-8986-94d3-0714-a6236048bcdf@ideasonboard.com>","date":"2019-07-04T09:50:07","subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"On 04/07/2019 10:35, Kieran Bingham wrote:\n> Process python additions with pep8 and report any errors that are added.\n> \n> Signed-off-by: Kieran Bingham \n> ---\n> utils/checkstyle.py | 29 +++++++++++++++++++++++++++++\n> 1 file changed, 29 insertions(+)\n> \n> This checker allows the checkstyle script to self-identify any issue.\n> Isn't that recursivly fun?\n> \n> \n> Todo:\n> - This should not fail if pep8 is not present, and should instead report an\n> issue that pep8 is not installed.\n\nFor reference, this was actually easy to add.\n\nThe following is now in v2, but I'll await any further comments before\nreposting.\n\n> - ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> - input=self.__data, stdout=subprocess.PIPE)\n> + try:\n> + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> + input=self.__data, stdout=subprocess.PIPE)\n> + except FileNotFoundError:\n> + issues.append(StyleIssue(0, \"\", \"Please install pep8 to validate python additions\"))\n> + return issues\n\n\n\n\n\n> - The line_no_regex should ideally be compiled once per class, or globally?\n> But how then do you access a class object from the class instance...\n> - All of the pep8 failures in checkstyle.py should be fixed up ...\n> \n> diff --git a/utils/checkstyle.py b/utils/checkstyle.py\n> index fab4b116d2ff..a960affc71a9 100755\n> --- a/utils/checkstyle.py\n> +++ b/utils/checkstyle.py\n> @@ -276,6 +276,35 @@ class MesonChecker(StyleChecker):\n> return issues\n> \n> \n> +class Pep8Checker(StyleChecker):\n> + patterns = ('*.py',)\n> +\n> + def __init__(self, content):\n> + super().__init__()\n> + self.__content = content\n> + self.__data = \"\".join(content).encode('utf-8')\n> +\n> + def check(self, line_numbers):\n> + issues = []\n> + line_no_regex = re.compile('stdin:([0-9]+):([0-9]+)(.*)')\n> +\n> + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> + input=self.__data, stdout=subprocess.PIPE)\n> +\n> + results = ret.stdout.decode('utf-8').splitlines()\n> + for item in results:\n> + search = re.search(line_no_regex, item)\n> + line_number = int(search.group(1))\n> + position = int(search.group(2))\n> + msg = search.group(3)\n> +\n> + if line_number in line_numbers:\n> + line = self.__content[line_number - 1]\n> + issues.append(StyleIssue(line_number, line, msg))\n> +\n> + return issues\n> +\n> +\n> # ------------------------------------------------------------------------------\n> # Formatters\n> #\n>","headers":{"Return-Path":"","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1346860C23\n\tfor ;\n\tThu, 4 Jul 2019 11:50:11 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 74A0F24B;\n\tThu, 4 Jul 2019 11:50:10 +0200 (CEST)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1562233810;\n\tbh=UWoTUvCz4H+rdlCZ7zhsPBmTG9Jv+u9gWfj0tz2kf7M=;\n\th=Reply-To:Subject:To:References:Cc:From:Date:In-Reply-To:From;\n\tb=godoGFPANS2OsNbW4/UtU0LjsswHt4uRX1gwMwJnYvDmN0azX3L3RMDuJ7F43KePh\n\tvYlFm2oh9RRnVE9soDEYb/V362b9NfXNAINFmUX+A8CNX4LcGiCr6MRyop1KcAQgmU\n\tjhxHORFoQsCWHasQAM8uZyeD1L8a5LJq3Up8h6Uo=","Reply-To":"kieran.bingham@ideasonboard.com","To":"Kieran Bingham ","References":"<20190704093502.27448-1-kieran.bingham@ideasonboard.com>","Cc":"LibCamera Devel ","From":"Kieran Bingham ","Openpgp":"preference=signencrypt","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAkAEEwEKACoCGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEFAlnDk/gFCQeA/YsACgkQoR5GchCkYf3X5w/9EaZ7\n\tcnUcT6dxjxrcmmMnfFPoQA1iQXr/MXQJBjFWfxRUWYzjvUJb2D/FpA8FY7y+vksoJP7pWDL7\n\tQTbksdwzagUEk7CU45iLWL/CZ/knYhj1I/+5LSLFmvZ/5Gf5xn2ZCsmg7C0MdW/GbJ8IjWA8\n\t/LKJSEYH8tefoiG6+9xSNp1p0Gesu3vhje/GdGX4wDsfAxx1rIYDYVoX4bDM+uBUQh7sQox/\n\tR1bS0AaVJzPNcjeC14MS226mQRUaUPc9250aj44WmDfcg44/kMsoLFEmQo2II9aOlxUDJ+x1\n\txohGbh9mgBoVawMO3RMBihcEjo/8ytW6v7xSF+xP4Oc+HOn7qebAkxhSWcRxQVaQYw3S9iZz\n\t2iA09AXAkbvPKuMSXi4uau5daXStfBnmOfalG0j+9Y6hOFjz5j0XzaoF6Pln0jisDtWltYhP\n\tX9LjFVhhLkTzPZB/xOeWGmsG4gv2V2ExbU3uAmb7t1VSD9+IO3Km4FtnYOKBWlxwEd8qOFpS\n\tjEqMXURKOiJvnw3OXe9MqG19XdeENA1KyhK5rqjpwdvPGfSn2V+SlsdJA0DFsobUScD9qXQw\n\tOvhapHe3XboK2+Rd7L+g/9Ud7ZKLQHAsMBXOVJbufA1AT+IaOt0ugMcFkAR5UbBg5+dZUYJj\n\t1QbPQcGmM3wfvuaWV5+SlJ+WeKIb8ta5Ag0EVgT9ZgEQAM4o5G/kmruIQJ3K9SYzmPishRHV\n\tDcUcvoakyXSX2mIoccmo9BHtD9MxIt+QmxOpYFNFM7YofX4lG0ld8H7FqoNVLd/+a0yru5Cx\n\tadeZBe3qr1eLns10Q90LuMo7/6zJhCW2w+HE7xgmCHejAwuNe3+7yt4QmwlSGUqdxl8cgtS1\n\tPlEK93xXDsgsJj/bw1EfSVdAUqhx8UQ3aVFxNug5OpoX9FdWJLKROUrfNeBE16RLrNrq2ROc\n\tiSFETpVjyC/oZtzRFnwD9Or7EFMi76/xrWzk+/b15RJ9WrpXGMrttHUUcYZEOoiC2lEXMSAF\n\tSSSj4vHbKDJ0vKQdEFtdgB1roqzxdIOg4rlHz5qwOTynueiBpaZI3PHDudZSMR5Fk6QjFooE\n\tXTw3sSl/km/lvUFiv9CYyHOLdygWohvDuMkV/Jpdkfq8XwFSjOle+vT/4VqERnYFDIGBxaRx\n\tkoBLfNDiiuR3lD8tnJ4A1F88K6ojOUs+jndKsOaQpDZV6iNFv8IaNIklTPvPkZsmNDhJMRHH\n\tIu60S7BpzNeQeT4yyY4dX9lC2JL/LOEpw8DGf5BNOP1KgjCvyp1/KcFxDAo89IeqljaRsCdP\n\t7WCIECWYem6pLwaw6IAL7oX+tEqIMPph/G/jwZcdS6Hkyt/esHPuHNwX4guqTbVEuRqbDzDI\n\t2DJO5FbxABEBAAGJAiUEGAEKAA8CGwwFAlnDlGsFCQeA/gIACgkQoR5GchCkYf1yYRAAq+Yo\n\tnbf9DGdK1kTAm2RTFg+w9oOp2Xjqfhds2PAhFFvrHQg1XfQR/UF/SjeUmaOmLSczM0s6XMeO\n\tVcE77UFtJ/+hLo4PRFKm5X1Pcar6g5m4xGqa+Xfzi9tRkwC29KMCoQOag1BhHChgqYaUH3yo\n\tUzaPwT/fY75iVI+yD0ih/e6j8qYvP8pvGwMQfrmN9YB0zB39YzCSdaUaNrWGD3iCBxg6lwSO\n\tLKeRhxxfiXCIYEf3vwOsP3YMx2JkD5doseXmWBGW1U0T/oJF+DVfKB6mv5UfsTzpVhJRgee7\n\t4jkjqFq4qsUGxcvF2xtRkfHFpZDbRgRlVmiWkqDkT4qMA+4q1y/dWwshSKi/uwVZNycuLsz+\n\t+OD8xPNCsMTqeUkAKfbD8xW4LCay3r/dD2ckoxRxtMD9eOAyu5wYzo/ydIPTh1QEj9SYyvp8\n\tO0g6CpxEwyHUQtF5oh15O018z3ZLztFJKR3RD42VKVsrnNDKnoY0f4U0z7eJv2NeF8xHMuiU\n\tRCIzqxX1GVYaNkKTnb/Qja8hnYnkUzY1Lc+OtwiGmXTwYsPZjjAaDX35J/RSKAoy5wGo/YFA\n\tJxB1gWThL4kOTbsqqXj9GLcyOImkW0lJGGR3o/fV91Zh63S5TKnf2YGGGzxki+ADdxVQAm+Q\n\tsbsRB8KNNvVXBOVNwko86rQqF9drZuw=","Organization":"Ideas on Board","Message-ID":"<9e0e8baf-8986-94d3-0714-a6236048bcdf@ideasonboard.com>","Date":"Thu, 4 Jul 2019 10:50:07 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101\n\tThunderbird/60.7.1","MIME-Version":"1.0","In-Reply-To":"<20190704093502.27448-1-kieran.bingham@ideasonboard.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-GB","Content-Transfer-Encoding":"7bit","Subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.23","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","X-List-Received-Date":"Thu, 04 Jul 2019 09:50:11 -0000"}},{"id":2152,"web_url":"https://patchwork.libcamera.org/comment/2152/","msgid":"<20190704115640.GB6569@pendragon.ideasonboard.com>","date":"2019-07-04T11:56:40","subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Kieran,\n\nThank you for the patch.\n\nOn Thu, Jul 04, 2019 at 10:50:07AM +0100, Kieran Bingham wrote:\n> On 04/07/2019 10:35, Kieran Bingham wrote:\n> > Process python additions with pep8 and report any errors that are added.\n> > \n> > Signed-off-by: Kieran Bingham \n> > ---\n> > utils/checkstyle.py | 29 +++++++++++++++++++++++++++++\n> > 1 file changed, 29 insertions(+)\n> > \n> > This checker allows the checkstyle script to self-identify any issue.\n> > Isn't that recursivly fun?\n> > \n> > \n> > Todo:\n> > - This should not fail if pep8 is not present, and should instead report an\n> > issue that pep8 is not installed.\n> \n> For reference, this was actually easy to add.\n> \n> The following is now in v2, but I'll await any further comments before\n> reposting.\n> \n> > - ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> > - input=self.__data, stdout=subprocess.PIPE)\n> > + try:\n> > + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> > + input=self.__data, stdout=subprocess.PIPE)\n> > + except FileNotFoundError:\n> > + issues.append(StyleIssue(0, \"\", \"Please install pep8 to validate python additions\"))\n> > + return issues\n\nHow about doing it the same way we handle the astyle and clang-format\ndependencies, in the main function ?\n\n> > - The line_no_regex should ideally be compiled once per class, or globally?\n> > But how then do you access a class object from the class instance...\n> > - All of the pep8 failures in checkstyle.py should be fixed up ...\n> > \n> > diff --git a/utils/checkstyle.py b/utils/checkstyle.py\n> > index fab4b116d2ff..a960affc71a9 100755\n> > --- a/utils/checkstyle.py\n> > +++ b/utils/checkstyle.py\n> > @@ -276,6 +276,35 @@ class MesonChecker(StyleChecker):\n> > return issues\n> > \n> > \n> > +class Pep8Checker(StyleChecker):\n> > + patterns = ('*.py',)\n> > +\n> > + def __init__(self, content):\n> > + super().__init__()\n> > + self.__content = content\n> > + self.__data = \"\".join(content).encode('utf-8')\n\ns/\"\"/''/\n\nIt makes not difference in practice, but the script tends to use single\nquotes, so let's be consistent.\n\nThere's also no need to store __data in the class, you can make it a\nlocal variable where used.\n\n> > +\n> > + def check(self, line_numbers):\n> > + issues = []\n> > + line_no_regex = re.compile('stdin:([0-9]+):([0-9]+)(.*)')\n\nYou can move this to the class level, below patterns, and use it with\nPep8Checker.line_no_regex.\n\n> > +\n> > + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> > + input=self.__data, stdout=subprocess.PIPE)\n> > +\n> > + results = ret.stdout.decode('utf-8').splitlines()\n> > + for item in results:\n> > + search = re.search(line_no_regex, item)\n> > + line_number = int(search.group(1))\n> > + position = int(search.group(2))\n> > + msg = search.group(3)\n> > +\n> > + if line_number in line_numbers:\n> > + line = self.__content[line_number - 1]\n> > + issues.append(StyleIssue(line_number, line, msg))\n> > +\n> > + return issues\n> > +\n> > +\n> > # ------------------------------------------------------------------------------\n> > # Formatters\n> > #","headers":{"Return-Path":"","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C748A6156A\n\tfor ;\n\tThu, 4 Jul 2019 13:57:01 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(dfj612yhrgyx302h3jwwy-3.rev.dnainternet.fi\n\t[IPv6:2001:14ba:21f5:5b00:ce28:277f:58d7:3ca4])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 1C42824B;\n\tThu, 4 Jul 2019 13:57:01 +0200 (CEST)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1562241421;\n\tbh=+hrJ07v9y204J0GWMbBtT0Q7vQzMvBd5gr7noO0C3Sc=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=hp3IPrQK2PPEUkjd/3f7QJIHH0SWW6zgXxfrPgpWma5JP4tT6/2KDa28zj14qeEIq\n\tF/G2c/ywghlLJMKU8JrENFDST2PSWCYyEDUVCtNwubAIid3XRrv+AYAqF6mSmqoBNv\n\ta7m31NlFJmpLh8lYO7trSGU2q5HH5KF0A4Xtzkfo=","Date":"Thu, 4 Jul 2019 14:56:40 +0300","From":"Laurent Pinchart ","To":"Kieran Bingham ","Cc":"LibCamera Devel ","Message-ID":"<20190704115640.GB6569@pendragon.ideasonboard.com>","References":"<20190704093502.27448-1-kieran.bingham@ideasonboard.com>\n\t<9e0e8baf-8986-94d3-0714-a6236048bcdf@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<9e0e8baf-8986-94d3-0714-a6236048bcdf@ideasonboard.com>","User-Agent":"Mutt/1.10.1 (2018-07-13)","Subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.23","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","X-List-Received-Date":"Thu, 04 Jul 2019 11:57:02 -0000"}},{"id":2155,"web_url":"https://patchwork.libcamera.org/comment/2155/","msgid":"<575e3049-3d59-7e95-8c66-9e8385fb1170@ideasonboard.com>","date":"2019-07-04T13:29:58","subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Laurent,\n\nOn 04/07/2019 12:56, Laurent Pinchart wrote:\n> Hi Kieran,\n> \n> Thank you for the patch.\n> \n> On Thu, Jul 04, 2019 at 10:50:07AM +0100, Kieran Bingham wrote:\n>> On 04/07/2019 10:35, Kieran Bingham wrote:\n>>> Process python additions with pep8 and report any errors that are added.\n>>>\n>>> Signed-off-by: Kieran Bingham \n>>> ---\n>>> utils/checkstyle.py | 29 +++++++++++++++++++++++++++++\n>>> 1 file changed, 29 insertions(+)\n>>>\n>>> This checker allows the checkstyle script to self-identify any issue.\n>>> Isn't that recursivly fun?\n>>>\n>>>\n>>> Todo:\n>>> - This should not fail if pep8 is not present, and should instead report an\n>>> issue that pep8 is not installed.\n>>\n>> For reference, this was actually easy to add.\n>>\n>> The following is now in v2, but I'll await any further comments before\n>> reposting.\n>>\n>>> - ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n>>> - input=self.__data, stdout=subprocess.PIPE)\n>>> + try:\n>>> + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n>>> + input=self.__data, stdout=subprocess.PIPE)\n>>> + except FileNotFoundError:\n>>> + issues.append(StyleIssue(0, \"\", \"Please install pep8 to validate python additions\"))\n>>> + return issues\n> \n> How about doing it the same way we handle the astyle and clang-format\n> dependencies, in the main function ?\n\nHrm, that only gives me a way to declare pep8 as required or not required.\n\nI think pep8 should be required if someone edits any .py file in the\nproject, but not otherwise ...\n\nany suggestions?\n\n\n>>> - The line_no_regex should ideally be compiled once per class, or globally?\n>>> But how then do you access a class object from the class instance...\n>>> - All of the pep8 failures in checkstyle.py should be fixed up ...\n>>>\n>>> diff --git a/utils/checkstyle.py b/utils/checkstyle.py\n>>> index fab4b116d2ff..a960affc71a9 100755\n>>> --- a/utils/checkstyle.py\n>>> +++ b/utils/checkstyle.py\n>>> @@ -276,6 +276,35 @@ class MesonChecker(StyleChecker):\n>>> return issues\n>>> \n>>> \n>>> +class Pep8Checker(StyleChecker):\n>>> + patterns = ('*.py',)\n>>> +\n>>> + def __init__(self, content):\n>>> + super().__init__()\n>>> + self.__content = content\n>>> + self.__data = \"\".join(content).encode('utf-8')\n> \n> s/\"\"/''/\n> \n> It makes not difference in practice, but the script tends to use single\n> quotes, so let's be consistent.\n\nSure.\n\n> \n> There's also no need to store __data in the class, you can make it a\n> local variable where used.\n\nMoved.\n\n> \n>>> +\n>>> + def check(self, line_numbers):\n>>> + issues = []\n>>> + line_no_regex = re.compile('stdin:([0-9]+):([0-9]+)(.*)')\n> \n> You can move this to the class level, below patterns, and use it with\n> Pep8Checker.line_no_regex.\n\nDone, thanks - That was simpler than I realised :D\n\n> \n>>> +\n>>> + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n>>> + input=self.__data, stdout=subprocess.PIPE)\n>>> +\n>>> + results = ret.stdout.decode('utf-8').splitlines()\n>>> + for item in results:\n>>> + search = re.search(line_no_regex, item)\n>>> + line_number = int(search.group(1))\n>>> + position = int(search.group(2))\n>>> + msg = search.group(3)\n>>> +\n>>> + if line_number in line_numbers:\n>>> + line = self.__content[line_number - 1]\n>>> + issues.append(StyleIssue(line_number, line, msg))\n>>> +\n>>> + return issues\n>>> +\n>>> +\n>>> # ------------------------------------------------------------------------------\n>>> # Formatters\n>>> #\n>","headers":{"Return-Path":"","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5FC6261569\n\tfor ;\n\tThu, 4 Jul 2019 15:30:01 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 9965124B;\n\tThu, 4 Jul 2019 15:30:00 +0200 (CEST)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1562247000;\n\tbh=Ihlap9ETOb7aZHE0rAXE3yoatT2vT3fvHcQkSlPKtis=;\n\th=Reply-To:Subject:To:Cc:References:From:Date:In-Reply-To:From;\n\tb=iMQONp+WHbgTdpEWxdfztfF5uNqjhNXwAalWVdaaln6CYwvjUpupMcRH+QZYIiFKC\n\tg2GzofR99QFD51mclb1JCASK8Atw/rX3acGQ3bJwME1EamMM7H5WVFa5PVsNILgMsG\n\thCdk0oxRSV+g+YoptC8wG7/85NuJ448sShaAgkT4=","Reply-To":"kieran.bingham@ideasonboard.com","To":"Laurent Pinchart ","Cc":"LibCamera Devel ","References":"<20190704093502.27448-1-kieran.bingham@ideasonboard.com>\n\t<9e0e8baf-8986-94d3-0714-a6236048bcdf@ideasonboard.com>\n\t<20190704115640.GB6569@pendragon.ideasonboard.com>","From":"Kieran Bingham ","Openpgp":"preference=signencrypt","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAkAEEwEKACoCGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEFAlnDk/gFCQeA/YsACgkQoR5GchCkYf3X5w/9EaZ7\n\tcnUcT6dxjxrcmmMnfFPoQA1iQXr/MXQJBjFWfxRUWYzjvUJb2D/FpA8FY7y+vksoJP7pWDL7\n\tQTbksdwzagUEk7CU45iLWL/CZ/knYhj1I/+5LSLFmvZ/5Gf5xn2ZCsmg7C0MdW/GbJ8IjWA8\n\t/LKJSEYH8tefoiG6+9xSNp1p0Gesu3vhje/GdGX4wDsfAxx1rIYDYVoX4bDM+uBUQh7sQox/\n\tR1bS0AaVJzPNcjeC14MS226mQRUaUPc9250aj44WmDfcg44/kMsoLFEmQo2II9aOlxUDJ+x1\n\txohGbh9mgBoVawMO3RMBihcEjo/8ytW6v7xSF+xP4Oc+HOn7qebAkxhSWcRxQVaQYw3S9iZz\n\t2iA09AXAkbvPKuMSXi4uau5daXStfBnmOfalG0j+9Y6hOFjz5j0XzaoF6Pln0jisDtWltYhP\n\tX9LjFVhhLkTzPZB/xOeWGmsG4gv2V2ExbU3uAmb7t1VSD9+IO3Km4FtnYOKBWlxwEd8qOFpS\n\tjEqMXURKOiJvnw3OXe9MqG19XdeENA1KyhK5rqjpwdvPGfSn2V+SlsdJA0DFsobUScD9qXQw\n\tOvhapHe3XboK2+Rd7L+g/9Ud7ZKLQHAsMBXOVJbufA1AT+IaOt0ugMcFkAR5UbBg5+dZUYJj\n\t1QbPQcGmM3wfvuaWV5+SlJ+WeKIb8ta5Ag0EVgT9ZgEQAM4o5G/kmruIQJ3K9SYzmPishRHV\n\tDcUcvoakyXSX2mIoccmo9BHtD9MxIt+QmxOpYFNFM7YofX4lG0ld8H7FqoNVLd/+a0yru5Cx\n\tadeZBe3qr1eLns10Q90LuMo7/6zJhCW2w+HE7xgmCHejAwuNe3+7yt4QmwlSGUqdxl8cgtS1\n\tPlEK93xXDsgsJj/bw1EfSVdAUqhx8UQ3aVFxNug5OpoX9FdWJLKROUrfNeBE16RLrNrq2ROc\n\tiSFETpVjyC/oZtzRFnwD9Or7EFMi76/xrWzk+/b15RJ9WrpXGMrttHUUcYZEOoiC2lEXMSAF\n\tSSSj4vHbKDJ0vKQdEFtdgB1roqzxdIOg4rlHz5qwOTynueiBpaZI3PHDudZSMR5Fk6QjFooE\n\tXTw3sSl/km/lvUFiv9CYyHOLdygWohvDuMkV/Jpdkfq8XwFSjOle+vT/4VqERnYFDIGBxaRx\n\tkoBLfNDiiuR3lD8tnJ4A1F88K6ojOUs+jndKsOaQpDZV6iNFv8IaNIklTPvPkZsmNDhJMRHH\n\tIu60S7BpzNeQeT4yyY4dX9lC2JL/LOEpw8DGf5BNOP1KgjCvyp1/KcFxDAo89IeqljaRsCdP\n\t7WCIECWYem6pLwaw6IAL7oX+tEqIMPph/G/jwZcdS6Hkyt/esHPuHNwX4guqTbVEuRqbDzDI\n\t2DJO5FbxABEBAAGJAiUEGAEKAA8CGwwFAlnDlGsFCQeA/gIACgkQoR5GchCkYf1yYRAAq+Yo\n\tnbf9DGdK1kTAm2RTFg+w9oOp2Xjqfhds2PAhFFvrHQg1XfQR/UF/SjeUmaOmLSczM0s6XMeO\n\tVcE77UFtJ/+hLo4PRFKm5X1Pcar6g5m4xGqa+Xfzi9tRkwC29KMCoQOag1BhHChgqYaUH3yo\n\tUzaPwT/fY75iVI+yD0ih/e6j8qYvP8pvGwMQfrmN9YB0zB39YzCSdaUaNrWGD3iCBxg6lwSO\n\tLKeRhxxfiXCIYEf3vwOsP3YMx2JkD5doseXmWBGW1U0T/oJF+DVfKB6mv5UfsTzpVhJRgee7\n\t4jkjqFq4qsUGxcvF2xtRkfHFpZDbRgRlVmiWkqDkT4qMA+4q1y/dWwshSKi/uwVZNycuLsz+\n\t+OD8xPNCsMTqeUkAKfbD8xW4LCay3r/dD2ckoxRxtMD9eOAyu5wYzo/ydIPTh1QEj9SYyvp8\n\tO0g6CpxEwyHUQtF5oh15O018z3ZLztFJKR3RD42VKVsrnNDKnoY0f4U0z7eJv2NeF8xHMuiU\n\tRCIzqxX1GVYaNkKTnb/Qja8hnYnkUzY1Lc+OtwiGmXTwYsPZjjAaDX35J/RSKAoy5wGo/YFA\n\tJxB1gWThL4kOTbsqqXj9GLcyOImkW0lJGGR3o/fV91Zh63S5TKnf2YGGGzxki+ADdxVQAm+Q\n\tsbsRB8KNNvVXBOVNwko86rQqF9drZuw=","Organization":"Ideas on Board","Message-ID":"<575e3049-3d59-7e95-8c66-9e8385fb1170@ideasonboard.com>","Date":"Thu, 4 Jul 2019 14:29:58 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101\n\tThunderbird/60.7.1","MIME-Version":"1.0","In-Reply-To":"<20190704115640.GB6569@pendragon.ideasonboard.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-GB","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.23","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","X-List-Received-Date":"Thu, 04 Jul 2019 13:30:01 -0000"}},{"id":2158,"web_url":"https://patchwork.libcamera.org/comment/2158/","msgid":"<20190704134625.GL6569@pendragon.ideasonboard.com>","date":"2019-07-04T13:46:25","subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Kieran,\n\nOn Thu, Jul 04, 2019 at 02:29:58PM +0100, Kieran Bingham wrote:\n> On 04/07/2019 12:56, Laurent Pinchart wrote:\n> > On Thu, Jul 04, 2019 at 10:50:07AM +0100, Kieran Bingham wrote:\n> >> On 04/07/2019 10:35, Kieran Bingham wrote:\n> >>> Process python additions with pep8 and report any errors that are added.\n> >>>\n> >>> Signed-off-by: Kieran Bingham \n> >>> ---\n> >>> utils/checkstyle.py | 29 +++++++++++++++++++++++++++++\n> >>> 1 file changed, 29 insertions(+)\n> >>>\n> >>> This checker allows the checkstyle script to self-identify any issue.\n> >>> Isn't that recursivly fun?\n> >>>\n> >>>\n> >>> Todo:\n> >>> - This should not fail if pep8 is not present, and should instead report an\n> >>> issue that pep8 is not installed.\n> >>\n> >> For reference, this was actually easy to add.\n> >>\n> >> The following is now in v2, but I'll await any further comments before\n> >> reposting.\n> >>\n> >>> - ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> >>> - input=self.__data, stdout=subprocess.PIPE)\n> >>> + try:\n> >>> + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> >>> + input=self.__data, stdout=subprocess.PIPE)\n> >>> + except FileNotFoundError:\n> >>> + issues.append(StyleIssue(0, \"\", \"Please install pep8 to validate python additions\"))\n> >>> + return issues\n> > \n> > How about doing it the same way we handle the astyle and clang-format\n> > dependencies, in the main function ?\n> \n> Hrm, that only gives me a way to declare pep8 as required or not required.\n> \n> I think pep8 should be required if someone edits any .py file in the\n> project, but not otherwise ...\n> \n> any suggestions?\n\nThat's a good point. Let's keep the above code then, and refactor it\nlater if more checkers or formatters require dependencies. Ideally we\nshould extract a list of all the files that need to be touched, and\nprint a warning for unmet dependencies for those files only.\n\n> >>> - The line_no_regex should ideally be compiled once per class, or globally?\n> >>> But how then do you access a class object from the class instance...\n> >>> - All of the pep8 failures in checkstyle.py should be fixed up ...\n> >>>\n> >>> diff --git a/utils/checkstyle.py b/utils/checkstyle.py\n> >>> index fab4b116d2ff..a960affc71a9 100755\n> >>> --- a/utils/checkstyle.py\n> >>> +++ b/utils/checkstyle.py\n> >>> @@ -276,6 +276,35 @@ class MesonChecker(StyleChecker):\n> >>> return issues\n> >>> \n> >>> \n> >>> +class Pep8Checker(StyleChecker):\n> >>> + patterns = ('*.py',)\n> >>> +\n> >>> + def __init__(self, content):\n> >>> + super().__init__()\n> >>> + self.__content = content\n> >>> + self.__data = \"\".join(content).encode('utf-8')\n> > \n> > s/\"\"/''/\n> > \n> > It makes not difference in practice, but the script tends to use single\n> > quotes, so let's be consistent.\n> \n> Sure.\n> \n> > There's also no need to store __data in the class, you can make it a\n> > local variable where used.\n> \n> Moved.\n> \n> >>> +\n> >>> + def check(self, line_numbers):\n> >>> + issues = []\n> >>> + line_no_regex = re.compile('stdin:([0-9]+):([0-9]+)(.*)')\n> > \n> > You can move this to the class level, below patterns, and use it with\n> > Pep8Checker.line_no_regex.\n> \n> Done, thanks - That was simpler than I realised :D\n> \n> >>> +\n> >>> + ret = subprocess.run(['pep8', '--ignore=E501', '-'],\n> >>> + input=self.__data, stdout=subprocess.PIPE)\n> >>> +\n> >>> + results = ret.stdout.decode('utf-8').splitlines()\n> >>> + for item in results:\n> >>> + search = re.search(line_no_regex, item)\n> >>> + line_number = int(search.group(1))\n> >>> + position = int(search.group(2))\n> >>> + msg = search.group(3)\n> >>> +\n> >>> + if line_number in line_numbers:\n> >>> + line = self.__content[line_number - 1]\n> >>> + issues.append(StyleIssue(line_number, line, msg))\n> >>> +\n> >>> + return issues\n> >>> +\n> >>> +\n> >>> # ------------------------------------------------------------------------------\n> >>> # Formatters\n> >>> #","headers":{"Return-Path":"","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CB45F6156B\n\tfor ;\n\tThu, 4 Jul 2019 15:46:46 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(dfj612yhrgyx302h3jwwy-3.rev.dnainternet.fi\n\t[IPv6:2001:14ba:21f5:5b00:ce28:277f:58d7:3ca4])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 415F624B;\n\tThu, 4 Jul 2019 15:46:46 +0200 (CEST)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1562248006;\n\tbh=WTYu8SJKJqlw9PZ/dwgQaqxk5keOjTTacZExCH2KB3c=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=XBW3QISYSzrpSAQ8r/fbwW5lhjkeWopKpaXTns3Kgmk+NOoqdxl0SJ4ff1//l0lc8\n\ttx0t/B/WkMjfwPzXSP/krEihLsXsQJw5yh6CI2TumWyh09xUp7nbDy11WyUuoJOBUt\n\t+srUFU98IMCNLhKKj7fsyxq5IezNJAZfv/dm6v9Y=","Date":"Thu, 4 Jul 2019 16:46:25 +0300","From":"Laurent Pinchart ","To":"Kieran Bingham ","Cc":"LibCamera Devel ","Message-ID":"<20190704134625.GL6569@pendragon.ideasonboard.com>","References":"<20190704093502.27448-1-kieran.bingham@ideasonboard.com>\n\t<9e0e8baf-8986-94d3-0714-a6236048bcdf@ideasonboard.com>\n\t<20190704115640.GB6569@pendragon.ideasonboard.com>\n\t<575e3049-3d59-7e95-8c66-9e8385fb1170@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<575e3049-3d59-7e95-8c66-9e8385fb1170@ideasonboard.com>","User-Agent":"Mutt/1.10.1 (2018-07-13)","Subject":"Re: [libcamera-devel] [PATCH] utils: checkstyle.py: Add pep8 checker","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.23","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","X-List-Received-Date":"Thu, 04 Jul 2019 13:46:47 -0000"}}]