[{"id":24333,"web_url":"https://patchwork.libcamera.org/comment/24333/","msgid":"","date":"2022-08-03T11:30:37","subject":"Re: [libcamera-devel] [PATCH v3] utils: raspberrypi: ctt:\n\tdng_load_image: Work with DNG files from Picamera2","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi William,\n\nThank you for the patch.\n\nOn Wed, Aug 03, 2022 at 09:25:39AM +0100, William Vinnicombe via libcamera-devel wrote:\n> From: William Vinnicombe \n> \n> The DNG specification is based on the TIFF file format and recommends\n> storing the raw image data in a SubIFD and the Exif tags in an Exif IFD.\n> Other options are allowed, even if not recommended, such as storing both\n> the raw image data and the Exif data in IFD0, as done by the TIFF/EP\n> specification.\n> \n> libcamera-apps use pyexiv2 to produce DNG files, following the DNG\n> recommendation, while applications based on picamera2 use PiDNG, which\n> adopts the TIFF/EP structure. Why it does so is not currently clear (see\n> https://github.com/schoolpost/PiDNG/issues/65 for discussions on this\n> topic), but as files based on the DNG and TIFF/EP variants exist in the\n> wild, both need to be supported by ctt.\n> \n> Add code to identify which tags are being used, and then load the\n> metadata from the correct tags.\n> \n> Signed-off-by: William Vinnicombe \n> Reviewed-by: David Plowman \n> Reviewed-by: Naushir Patuck \n\nReviewed-by: Laurent Pinchart \n\n> ---\n> utils/raspberrypi/ctt/ctt_image_load.py | 32 +++++++++++++++++++------\n> 1 file changed, 25 insertions(+), 7 deletions(-)\n> \n> diff --git a/utils/raspberrypi/ctt/ctt_image_load.py b/utils/raspberrypi/ctt/ctt_image_load.py\n> index 934db123..7f8ba72a 100644\n> --- a/utils/raspberrypi/ctt/ctt_image_load.py\n> +++ b/utils/raspberrypi/ctt/ctt_image_load.py\n> @@ -301,17 +301,35 @@ def dng_load_image(Cam, im_str):\n> metadata.read()\n> \n> Img.ver = 100 # random value\n> - Img.w = metadata['Exif.SubImage1.ImageWidth'].value\n> + \"\"\"\n> + The DNG and TIFF/EP specifications use different IFDs to store the raw\n> + image data and the Exif tags. DNG stores them in a SubIFD and in an Exif\n> + IFD respectively (named \"SubImage1\" and \"Photo\" by pyexiv2), while\n> + TIFF/EP stores them both in IFD0 (name \"Image\"). Both are used in \"DNG\"\n> + files, with libcamera-apps following the DNG recommendation and\n> + applications based on picamera2 following TIFF/EP.\n> +\n> + This code detects which tags are being used, and therefore extracts the\n> + correct values.\n> + \"\"\"\n> + try:\n> + Img.w = metadata['Exif.SubImage1.ImageWidth'].value\n> + subimage = \"SubImage1\"\n> + photo = \"Photo\"\n> + except KeyError:\n> + Img.w = metadata['Exif.Image.ImageWidth'].value\n> + subimage = \"Image\"\n> + photo = \"Image\"\n> Img.pad = 0\n> - Img.h = metadata['Exif.SubImage1.ImageLength'].value\n> - white = metadata['Exif.SubImage1.WhiteLevel'].value\n> + Img.h = metadata[f'Exif.{subimage}.ImageLength'].value\n> + white = metadata[f'Exif.{subimage}.WhiteLevel'].value\n> Img.sigbits = int(white).bit_length()\n> Img.fmt = (Img.sigbits - 4) // 2\n> - Img.exposure = int(metadata['Exif.Photo.ExposureTime'].value*1000000)\n> - Img.againQ8 = metadata['Exif.Photo.ISOSpeedRatings'].value*256/100\n> + Img.exposure = int(metadata[f'Exif.{photo}.ExposureTime'].value*1000000)\n> + Img.againQ8 = metadata[f'Exif.{photo}.ISOSpeedRatings'].value*256/100\n> Img.againQ8_norm = Img.againQ8 / 256\n> Img.camName = metadata['Exif.Image.Model'].value\n> - Img.blacklevel = int(metadata['Exif.SubImage1.BlackLevel'].value[0])\n> + Img.blacklevel = int(metadata[f'Exif.{subimage}.BlackLevel'].value[0])\n> Img.blacklevel_16 = Img.blacklevel << (16 - Img.sigbits)\n> bayer_case = {\n> '0 1 1 2': (0, (0, 1, 2, 3)),\n> @@ -319,7 +337,7 @@ def dng_load_image(Cam, im_str):\n> '2 1 1 0': (2, (3, 2, 1, 0)),\n> '1 0 2 1': (3, (1, 0, 3, 2))\n> }\n> - cfa_pattern = metadata['Exif.SubImage1.CFAPattern'].value\n> + cfa_pattern = metadata[f'Exif.{subimage}.CFAPattern'].value\n> Img.pattern = bayer_case[cfa_pattern][0]\n> Img.order = bayer_case[cfa_pattern][1]\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id CB2C9C3272\n\tfor ;\n\tWed, 3 Aug 2022 11:30:44 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8415561FAE;\n\tWed, 3 Aug 2022 13:30:44 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 34283603E6\n\tfor ;\n\tWed, 3 Aug 2022 13:30:43 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id BCD9F8B;\n\tWed, 3 Aug 2022 13:30:42 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1659526244;\n\tbh=0sTHGdY9g6sGAVcsRGkp6qcVQXhnjZg3AWL35qP+FHI=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=tRk2oyzM3NJJhc3Z1lToCWOO8QcCKJdwlXITFxHsuhn1H7pm4PKVCFT8PJAU/BodL\n\tDacalpYPVDKYPW6DUuSgOSA/iWcFjO1M/95c4fOCmUZ5+EUMrpRc6FmcqJfGryLyC3\n\tZNbyjig8Nhv0m8epEDR2GgqjZEwYUlsWAZGrGlKGo/wGOGY9vcXss+3dheoSkDk444\n\tqyjYAcwl9lxH2CfqxAejrn7VFIIPdGVvpInoHBGiUTUd1FRofp9Lq15NQEkAQlOKkQ\n\t8eke7pBEJNX6C6XZRaDSlPmRyECZdpGhLllVQxl5gbeJ+xZYcjertVFQXJoPWyNseP\n\t03MwMZf6Dvi7A==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1659526242;\n\tbh=0sTHGdY9g6sGAVcsRGkp6qcVQXhnjZg3AWL35qP+FHI=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=LUkkV8vdf0AhLXhnCoD9/KC3nWiBPZKqu8mOJw14+/d7aUPSjgr65plUNR2FuctKe\n\tQcnHhgj7RrFUsodzr5RWgJYwzZ/NjltU22+UJSNRn1sjOqXHusjCPWHVi93ZvdVZZ2\n\tjKVTorgcwbqsj/nnVX6D4MQDc+0Ew4PWNjR0PEoI="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"LUkkV8vd\"; dkim-atps=neutral","Date":"Wed, 3 Aug 2022 14:30:37 +0300","To":"william.vinnicombe@raspberrypi.com","Message-ID":"","References":"<20220803082539.11172-1-william.vinnicombe@raspberrypi.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20220803082539.11172-1-william.vinnicombe@raspberrypi.com>","Subject":"Re: [libcamera-devel] [PATCH v3] utils: raspberrypi: ctt:\n\tdng_load_image: Work with DNG files from Picamera2","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":24334,"web_url":"https://patchwork.libcamera.org/comment/24334/","msgid":"","date":"2022-08-03T11:33:34","subject":"Re: [libcamera-devel] [PATCH v3] utils: raspberrypi: ctt:\n\tdng_load_image: Work with DNG files from Picamera2","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Wed, Aug 03, 2022 at 02:30:38PM +0300, Laurent Pinchart wrote:\n> Hi William,\n> \n> Thank you for the patch.\n> \n> On Wed, Aug 03, 2022 at 09:25:39AM +0100, William Vinnicombe via libcamera-devel wrote:\n> > From: William Vinnicombe \n> > \n> > The DNG specification is based on the TIFF file format and recommends\n> > storing the raw image data in a SubIFD and the Exif tags in an Exif IFD.\n> > Other options are allowed, even if not recommended, such as storing both\n> > the raw image data and the Exif data in IFD0, as done by the TIFF/EP\n> > specification.\n> > \n> > libcamera-apps use pyexiv2 to produce DNG files, following the DNG\n> > recommendation, while applications based on picamera2 use PiDNG, which\n> > adopts the TIFF/EP structure. Why it does so is not currently clear (see\n> > https://github.com/schoolpost/PiDNG/issues/65 for discussions on this\n> > topic), but as files based on the DNG and TIFF/EP variants exist in the\n> > wild, both need to be supported by ctt.\n> > \n> > Add code to identify which tags are being used, and then load the\n> > metadata from the correct tags.\n> > \n> > Signed-off-by: William Vinnicombe \n> > Reviewed-by: David Plowman \n> > Reviewed-by: Naushir Patuck \n> \n> Reviewed-by: Laurent Pinchart \n> \n> > ---\n> > utils/raspberrypi/ctt/ctt_image_load.py | 32 +++++++++++++++++++------\n> > 1 file changed, 25 insertions(+), 7 deletions(-)\n> > \n> > diff --git a/utils/raspberrypi/ctt/ctt_image_load.py b/utils/raspberrypi/ctt/ctt_image_load.py\n> > index 934db123..7f8ba72a 100644\n> > --- a/utils/raspberrypi/ctt/ctt_image_load.py\n> > +++ b/utils/raspberrypi/ctt/ctt_image_load.py\n> > @@ -301,17 +301,35 @@ def dng_load_image(Cam, im_str):\n> > metadata.read()\n> > \n> > Img.ver = 100 # random value\n> > - Img.w = metadata['Exif.SubImage1.ImageWidth'].value\n> > + \"\"\"\n> > + The DNG and TIFF/EP specifications use different IFDs to store the raw\n> > + image data and the Exif tags. DNG stores them in a SubIFD and in an Exif\n> > + IFD respectively (named \"SubImage1\" and \"Photo\" by pyexiv2), while\n> > + TIFF/EP stores them both in IFD0 (name \"Image\"). Both are used in \"DNG\"\n> > + files, with libcamera-apps following the DNG recommendation and\n> > + applications based on picamera2 following TIFF/EP.\n> > +\n> > + This code detects which tags are being used, and therefore extracts the\n> > + correct values.\n> > + \"\"\"\n> > + try:\n> > + Img.w = metadata['Exif.SubImage1.ImageWidth'].value\n> > + subimage = \"SubImage1\"\n> > + photo = \"Photo\"\n> > + except KeyError:\n> > + Img.w = metadata['Exif.Image.ImageWidth'].value\n> > + subimage = \"Image\"\n> > + photo = \"Image\"\n> > Img.pad = 0\n> > - Img.h = metadata['Exif.SubImage1.ImageLength'].value\n> > - white = metadata['Exif.SubImage1.WhiteLevel'].value\n> > + Img.h = metadata[f'Exif.{subimage}.ImageLength'].value\n> > + white = metadata[f'Exif.{subimage}.WhiteLevel'].value\n> > Img.sigbits = int(white).bit_length()\n> > Img.fmt = (Img.sigbits - 4) // 2\n> > - Img.exposure = int(metadata['Exif.Photo.ExposureTime'].value*1000000)\n> > - Img.againQ8 = metadata['Exif.Photo.ISOSpeedRatings'].value*256/100\n> > + Img.exposure = int(metadata[f'Exif.{photo}.ExposureTime'].value*1000000)\n> > + Img.againQ8 = metadata[f'Exif.{photo}.ISOSpeedRatings'].value*256/100\n\nActually, checkstyle.py reports\n\n--------------------------------------------------------------------------------------------------------------------\n31444bed873fbafa1bfa66d76f34044ace09f7d5 utils: raspberrypi: ctt: dng_load_image: Work with DNG files from Picamera2\n--------------------------------------------------------------------------------------------------------------------\n--- utils/raspberrypi/ctt/ctt_image_load.py\n+++ utils/raspberrypi/ctt/ctt_image_load.py\n#328: : E226 missing whitespace around arithmetic operator\n+ Img.exposure = int(metadata[f'Exif.{photo}.ExposureTime'].value*1000000)\n#329: : E226 missing whitespace around arithmetic operator\n+ Img.againQ8 = metadata[f'Exif.{photo}.ISOSpeedRatings'].value*256/100\n#329: : E226 missing whitespace around arithmetic operator\n+ Img.againQ8 = metadata[f'Exif.{photo}.ISOSpeedRatings'].value*256/100\n---\n3 potential issues detected, please review\n\nI'll fix this by adding the missing spaces.\n\nCould you install the libcamera git post-commit hook that runs\ncheckstyle.py automatically ?\n\n$ cp utils/hooks/post-commit .git/hooks/\n\n> > Img.againQ8_norm = Img.againQ8 / 256\n> > Img.camName = metadata['Exif.Image.Model'].value\n> > - Img.blacklevel = int(metadata['Exif.SubImage1.BlackLevel'].value[0])\n> > + Img.blacklevel = int(metadata[f'Exif.{subimage}.BlackLevel'].value[0])\n> > Img.blacklevel_16 = Img.blacklevel << (16 - Img.sigbits)\n> > bayer_case = {\n> > '0 1 1 2': (0, (0, 1, 2, 3)),\n> > @@ -319,7 +337,7 @@ def dng_load_image(Cam, im_str):\n> > '2 1 1 0': (2, (3, 2, 1, 0)),\n> > '1 0 2 1': (3, (1, 0, 3, 2))\n> > }\n> > - cfa_pattern = metadata['Exif.SubImage1.CFAPattern'].value\n> > + cfa_pattern = metadata[f'Exif.{subimage}.CFAPattern'].value\n> > Img.pattern = bayer_case[cfa_pattern][0]\n> > Img.order = bayer_case[cfa_pattern][1]\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A7201BE173\n\tfor ;\n\tWed, 3 Aug 2022 11:33:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 21D5161FAE;\n\tWed, 3 Aug 2022 13:33:42 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 771DC603E6\n\tfor ;\n\tWed, 3 Aug 2022 13:33:40 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id F28458B;\n\tWed, 3 Aug 2022 13:33:39 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1659526422;\n\tbh=UccnndmQpIUS507Hxdp9A18LnsbH5NQq1mdpiBFE6Go=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=xuWsYTKxmpz80LHd2s9c+oErGlqWGqJjNsUlGemC67KxjyvBjhsZiFpJkraRQ5a0h\n\tslVGHTOS0GbkHyfu5QeADc7FluZtFoUbnQaAyj7ralmrg+aWwJ5IgtJdGIP2em3cBJ\n\tRfA2Fp7lxuQsPsNMok9zbCj9NsWVHLmbec3t6NMypYpevLcaLqKctBcTBHK5+VBsQl\n\tAvxHPWVdF/Zj7+feoqnkrHiUexBmFnh9zPDIV6YH25/lQscyfirMTU/SHUR+hDJvtb\n\tHNozr6aXiP8F9wDf0iqo7o/woArEQl02ajp7jmpKIvty/xGBqde0aZP26/84CdmgKS\n\tGajsF9SC8OVqw==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1659526420;\n\tbh=UccnndmQpIUS507Hxdp9A18LnsbH5NQq1mdpiBFE6Go=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=gKs8kWdYcOiloJyOMG4voujVetAJNNBJigVspo2O3annOvcilid+MMkLOMciE+d6B\n\tTmDGEF/lu34e+QVWe844L4FwTNeRRCvPu6Zyr8QfMia+g7ZS0KSBIocynkTqnBOe1m\n\tvHg+FRAjGpQoJHPRvqsAa8p3ovHJxYGPEHQBv9uE="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"gKs8kWdY\"; dkim-atps=neutral","Date":"Wed, 3 Aug 2022 14:33:34 +0300","To":"william.vinnicombe@raspberrypi.com","Message-ID":"","References":"<20220803082539.11172-1-william.vinnicombe@raspberrypi.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH v3] utils: raspberrypi: ctt:\n\tdng_load_image: Work with DNG files from Picamera2","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]