[{"id":31704,"web_url":"https://patchwork.libcamera.org/comment/31704/","msgid":"<20241010194509.GI32107@pendragon.ideasonboard.com>","date":"2024-10-10T19:45:09","subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Hans,\n\nOn Wed, Oct 09, 2024 at 10:01:06PM +0200, Hans de Goede wrote:\n> Hi All,\n> \n> Here is a patch series adding a processFrame() method to gather\n> statistics on an entire frame in one go, rather then using the\n> line by line approach which is used by the somewhat tightly coupled\n> DebayerCpu class.\n> \n> ATM there are no users for processFrame() yet, so the code is\n> compile tested only which is why it is marked as RFC.\n> \n> I see at least 3 possible use-cases for this:\n> \n> 1. Gathering sw statistics for 3A on raw bayer buffers before\n> passing them along for a to-be-added software CPU ISP raw mode.\n> \n> 2. Gathering sw statistics for 3A on raw bayer buffers before\n> doing further processing on the GPU for the software GPU ISP\n> (possibly as an intermediate step until the stats are gathered\n> on the GPU too and/or as a way to compare CPU vs GPU stats\n> for verification purposes).\n> \n> 3. I'm working on a pipeline handler for the atomISP2, where\n> there is a working hw ISP but no usable statistics.\n\nWhy are the stats not usable ? Is it a hardware issue, a firmware issue,\nor \"just\" that we don't have information about the statistics format ?\n\n> The plan is to gather sw statistics on the output buffers\n> (so on e.g. YUV images) and then re-use the 3A code from\n> the software ISP (I realize gathering statistics on the\n> pipeline output buffers is not ideal, but the atomISP driver\n> has no way to access the raw bayer data).\n\nIt's more than non-ideal, it will make algorithms development much more\npainful :-(\n\n> Usage would look something like this:\n> \n> once:\n> \n> \tif (stats_->configure(inputCfg) != 0)\n> \t\treturn -EINVAL;\n> \n> \tstats_->setWindow(Rectangle(window_.size()));\n> \n> per frame:\n> \n> \tstats_->startFrame();\n> \tstats_->processFrame(inputFrameBuffer);\n> \tstats_->finishFrame(frameNo, bufferID);\n> \n> Regards,\n> \n> Hans\n> \n> \n> Hans de Goede (4):\n> libcamera: swstats_cpu: Update statsProcessFn() / processLine0()\n> documentation\n> libcamera: swstats_cpu: Drop patternSize_ documentation\n> libcamera: software_isp: Move benchmark code to its own class\n> libcamera: swstats_cpu: Add processFrame() method\n> \n> src/libcamera/software_isp/benchmark.cpp | 78 +++++++++++++++\n> src/libcamera/software_isp/benchmark.h | 36 +++++++\n> src/libcamera/software_isp/debayer_cpu.cpp | 32 +-----\n> src/libcamera/software_isp/debayer_cpu.h | 7 +-\n> src/libcamera/software_isp/meson.build | 1 +\n> src/libcamera/software_isp/swstats_cpu.cpp | 108 +++++++++++++++++----\n> src/libcamera/software_isp/swstats_cpu.h | 10 ++\n> 7 files changed, 216 insertions(+), 56 deletions(-)\n> create mode 100644 src/libcamera/software_isp/benchmark.cpp\n> create mode 100644 src/libcamera/software_isp/benchmark.h","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C02C5C32E0\n\tfor ;\n\tThu, 10 Oct 2024 19:45:17 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EF65065365;\n\tThu, 10 Oct 2024 21:45:16 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EAF376353B\n\tfor ;\n\tThu, 10 Oct 2024 21:45:14 +0200 (CEST)","from pendragon.ideasonboard.com (unknown [132.205.230.15])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 27CC59CA;\n\tThu, 10 Oct 2024 21:43:36 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"REp54oZG\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1728589416;\n\tbh=EJkp7AnL7nHjIEqLGsLI2DqoAfBmwFvuWkQYXamDloA=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=REp54oZGRM5ULs9dhJHk9/OOiyRO0rf0b6PzkxmTrHjYHDUXgAfzTqAp4RpvyyfV+\n\tvI2jSE1AcwC+WwE9fsQeN9o5skanI+GhmUqfTR8XP50oFIJBhVVOAPtJP0b7VN3n3G\n\tSLyCzqxwpy7LUApAA/u3uE92/Db7hDZIrBIiYE7A=","Date":"Thu, 10 Oct 2024 22:45:09 +0300","From":"Laurent Pinchart ","To":"Hans de Goede ","Cc":"libcamera-devel@lists.libcamera.org, Milan Zamazal ,\n\tMaxime Ripard ","Subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","Message-ID":"<20241010194509.GI32107@pendragon.ideasonboard.com>","References":"<20241009200110.275544-1-hdegoede@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20241009200110.275544-1-hdegoede@redhat.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":31994,"web_url":"https://patchwork.libcamera.org/comment/31994/","msgid":"<40f2d2a3-e8cd-4578-9588-12cac2b7c4d3@redhat.com>","date":"2024-11-03T12:05:29","subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","submitter":{"id":102,"url":"https://patchwork.libcamera.org/api/people/102/","name":"Hans de Goede","email":"hdegoede@redhat.com"},"content":"Hi Laurent,\n\nOn 10-Oct-24 9:45 PM, Laurent Pinchart wrote:\n> Hi Hans,\n> \n> On Wed, Oct 09, 2024 at 10:01:06PM +0200, Hans de Goede wrote:\n>> Hi All,\n>>\n>> Here is a patch series adding a processFrame() method to gather\n>> statistics on an entire frame in one go, rather then using the\n>> line by line approach which is used by the somewhat tightly coupled\n>> DebayerCpu class.\n>>\n>> ATM there are no users for processFrame() yet, so the code is\n>> compile tested only which is why it is marked as RFC.\n>>\n>> I see at least 3 possible use-cases for this:\n>>\n>> 1. Gathering sw statistics for 3A on raw bayer buffers before\n>> passing them along for a to-be-added software CPU ISP raw mode.\n>>\n>> 2. Gathering sw statistics for 3A on raw bayer buffers before\n>> doing further processing on the GPU for the software GPU ISP\n>> (possibly as an intermediate step until the stats are gathered\n>> on the GPU too and/or as a way to compare CPU vs GPU stats\n>> for verification purposes).\n>>\n>> 3. I'm working on a pipeline handler for the atomISP2, where\n>> there is a working hw ISP but no usable statistics.\n> \n> Why are the stats not usable ? Is it a hardware issue, a firmware issue,\n> or \"just\" that we don't have information about the statistics format ?\n\nThere are 2 problems:\n\n1. As you say '\"just\" that we don't have information about\nthe statistics format ?'\n\nI'll email Sakari to see if he can help with at least a .h files\nwith a C struct definition for the buffers.\n\nOne thing to keep in mind here is that the hw using the atomisp\nis getting pretty old. So while I'm very interested in getting\nthis going at the same time I also want to not spend too much\ntime on this.\n\nUsing the swstats_cpu code + reusing the ipa_soft_simple.so IPA\nis a relatively easy / low-risk way of doing this.\n\nI actually am about to post a first version of an atomisp\npipeline handler using this approach to do aec/agc.\n\nAWB will follow later. First I need to find out where\nthe atomisp kernel driver sets the ISP R/G/B gains and export\nthose as standard V4L2 controls, rather then whatever custom\nioctl it is currently using ...\n\n2. The kernel driver does receive what it calls \"3a stat\" buffers\nbut atm it simply discards these. I think there is or used to be\na custom ioctl to retrieve these, but I may have ripped that out\nalready. Regardless we need to design a proper userspace API for\nthis. Maybe a separate /dev/video# node using videobuf2 to pass\nthe buffers? Anyways this is something to look at if we can\nget help with figuring out the statistics format. So for now I\nhave not really looked at this yet.\n\n>> The plan is to gather sw statistics on the output buffers\n>> (so on e.g. YUV images) and then re-use the 3A code from\n>> the software ISP (I realize gathering statistics on the\n>> pipeline output buffers is not ideal, but the atomISP driver\n>> has no way to access the raw bayer data).\n> \n> It's more than non-ideal, it will make algorithms development much more\n> painful :-(\n\nYes this is less then ideal. But at least for basic agc + awb\nI think we can make this work.\n\nActually for aec/awb it works pretty well since the ISP outputs\nYUV data getting a Y histogram is easy and even in\nthe postprocessed data it is easy to see if the image is too\nlight / dark.\n\nRegards,\n\nHans","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 43490BD87C\n\tfor ;\n\tSun, 3 Nov 2024 12:05:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id F390E653BC;\n\tSun, 3 Nov 2024 13:05:41 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0E3A1653A1\n\tfor ;\n\tSun, 3 Nov 2024 13:05:39 +0100 (CET)","from mail-ej1-f69.google.com (mail-ej1-f69.google.com\n\t[209.85.218.69]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n\tus-mta-595-yzgh_nAiO-OscOUkYW97Sg-1; Sun, 03 Nov 2024 07:05:32 -0500","by mail-ej1-f69.google.com with SMTP id\n\ta640c23a62f3a-a99fa9f0c25so242897066b.3\n\tfor ;\n\tSun, 03 Nov 2024 04:05:31 -0800 (PST)","from ?IPV6:2001:1c00:c32:7800:5bfa:a036:83f0:f9ec?\n\t(2001-1c00-0c32-7800-5bfa-a036-83f0-f9ec.cable.dynamic.v6.ziggo.nl.\n\t[2001:1c00:c32:7800:5bfa:a036:83f0:f9ec])\n\tby smtp.gmail.com with ESMTPSA id\n\ta640c23a62f3a-a9e565e0883sm412580066b.109.2024.11.03.04.05.29\n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tSun, 03 Nov 2024 04:05:29 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"YVcDeN+s\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1730635538;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=s4aADtlFH3Tr0wk0hw+b9HWA7pNRUdBGarCI0umb/LU=;\n\tb=YVcDeN+s3+ZDJGDyiY6/psd3l5qfplmeDgAY5HNtd6RwgFejl2P1lOcyT3n/u6yaG4GnbS\n\tJXP/2SSYYF+2YpfH797e0ciOTuRfGhaklUSN1Ww4MZNOorFw+tc/5fX6SI1vamQAc5kfxc\n\t3ruHi6F4sJgCFPgmABdkhkjWPtAeIpU=","X-MC-Unique":"yzgh_nAiO-OscOUkYW97Sg-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1730635531; x=1731240331;\n\th=content-transfer-encoding:in-reply-to:from:content-language\n\t:references:cc:to:subject:user-agent:mime-version:date:message-id\n\t:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n\tbh=s4aADtlFH3Tr0wk0hw+b9HWA7pNRUdBGarCI0umb/LU=;\n\tb=MzPR4a4AFAB/BYZvm0lUNODMmX8kcNG4xawL+evcWNdgV22AEn2qZTUzZZaPLTWAWg\n\tTfd6GPdY/mBsvIWuUN4D9UgGGGiZxxR6gAu1wyg92zaPg0T8RqMc3ltFlLWbro6+elTY\n\ttlaHzXukS8eG1rFk4lQBP9wEYi6QXaF4uPID5h1KknQd1uPmG9v+nWCLB8FP5T5XSGwE\n\t3XcZ2ID0F94Pkn0mMBP2ykFGW00YhjBL5l34F7dS3GEQfS1FDcsVIJ4W8+TC+3OEuaED\n\tGN5Z1VEfFcH2DicbT8mIBHO/EfbGPSOz2GQm9TTAET0M9QxAgHRXsgTgr1o0D5rj+fwf\n\tkeZg==","X-Gm-Message-State":"AOJu0YzmB559Km62o9LlcB+/erbDuzEbqYGDct21mlaFQApnjI8JM+eJ\n\tegzdawcO8oc4xU3M1Pxnnm6tNnUG8QGCVbKFe4oaIbg64IsIR9FN5ZTP+92L5aFRDF9yuX78wHr\n\tgC/KcYANGypbNTVqSGYxdYuGAfnFINTgw6EK+StgKwRFJmEhc98PJHPDaPHc5sFrjMPy+2gc=","X-Received":["by 2002:a17:907:9413:b0:a9a:1f8:6c9b with SMTP id\n\ta640c23a62f3a-a9de5fe7455mr2875924566b.37.1730635530765; \n\tSun, 03 Nov 2024 04:05:30 -0800 (PST)","by 2002:a17:907:9413:b0:a9a:1f8:6c9b with SMTP id\n\ta640c23a62f3a-a9de5fe7455mr2875921966b.37.1730635530221; \n\tSun, 03 Nov 2024 04:05:30 -0800 (PST)"],"X-Google-Smtp-Source":"AGHT+IHkyg4zl6SMg0SuMQTusODw+h5mi2JGfvQAGWDK7GPsd8/gtseEgDQz8Mhb5SJPdebE2HiRPg==","Message-ID":"<40f2d2a3-e8cd-4578-9588-12cac2b7c4d3@redhat.com>","Date":"Sun, 3 Nov 2024 13:05:29 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org, Milan Zamazal ,\n\tMaxime Ripard ","References":"<20241009200110.275544-1-hdegoede@redhat.com>\n\t<20241010194509.GI32107@pendragon.ideasonboard.com>","From":"Hans de Goede ","In-Reply-To":"<20241010194509.GI32107@pendragon.ideasonboard.com>","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Language":"en-US, nl","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"7bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32043,"web_url":"https://patchwork.libcamera.org/comment/32043/","msgid":"<20241106112223.GF17733@pendragon.ideasonboard.com>","date":"2024-11-06T11:22:23","subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Sun, Nov 03, 2024 at 01:05:29PM +0100, Hans de Goede wrote:\n> On 10-Oct-24 9:45 PM, Laurent Pinchart wrote:\n> > On Wed, Oct 09, 2024 at 10:01:06PM +0200, Hans de Goede wrote:\n> >> Hi All,\n> >>\n> >> Here is a patch series adding a processFrame() method to gather\n> >> statistics on an entire frame in one go, rather then using the\n> >> line by line approach which is used by the somewhat tightly coupled\n> >> DebayerCpu class.\n> >>\n> >> ATM there are no users for processFrame() yet, so the code is\n> >> compile tested only which is why it is marked as RFC.\n> >>\n> >> I see at least 3 possible use-cases for this:\n> >>\n> >> 1. Gathering sw statistics for 3A on raw bayer buffers before\n> >> passing them along for a to-be-added software CPU ISP raw mode.\n> >>\n> >> 2. Gathering sw statistics for 3A on raw bayer buffers before\n> >> doing further processing on the GPU for the software GPU ISP\n> >> (possibly as an intermediate step until the stats are gathered\n> >> on the GPU too and/or as a way to compare CPU vs GPU stats\n> >> for verification purposes).\n> >>\n> >> 3. I'm working on a pipeline handler for the atomISP2, where\n> >> there is a working hw ISP but no usable statistics.\n> > \n> > Why are the stats not usable ? Is it a hardware issue, a firmware issue,\n> > or \"just\" that we don't have information about the statistics format ?\n> \n> There are 2 problems:\n> \n> 1. As you say '\"just\" that we don't have information about\n> the statistics format ?'\n> \n> I'll email Sakari to see if he can help with at least a .h files\n> with a C struct definition for the buffers.\n> \n> One thing to keep in mind here is that the hw using the atomisp\n> is getting pretty old. So while I'm very interested in getting\n> this going at the same time I also want to not spend too much\n> time on this.\n\nI understand that, but let's not forget that the driver is in staging,\nand what it implies. There should be continuous effort to clean up the\ndriver to get it out of staging, and I believe that should include\nthings like stats support. It doesn't have to be done right away, but we\ndon't want drivers to bitrot in staging, we have too many of those\nalready (I'm thinking about the IPU3 ImgU driver in particular that will\nneed work, or should be removed from the kernel if it becomes clear\nnobody will fix its issues).\n\n> Using the swstats_cpu code + reusing the ipa_soft_simple.so IPA\n> is a relatively easy / low-risk way of doing this.\n\nAs a first step I have no issue about that, we don't have to make\neverything perfect overnight :-)\n\n> I actually am about to post a first version of an atomisp\n> pipeline handler using this approach to do aec/agc.\n> \n> AWB will follow later. First I need to find out where\n> the atomisp kernel driver sets the ISP R/G/B gains and export\n> those as standard V4L2 controls, rather then whatever custom\n> ioctl it is currently using ...\n> \n> 2. The kernel driver does receive what it calls \"3a stat\" buffers\n> but atm it simply discards these. I think there is or used to be\n> a custom ioctl to retrieve these, but I may have ripped that out\n> already. Regardless we need to design a proper userspace API for\n> this. Maybe a separate /dev/video# node using videobuf2 to pass\n> the buffers?\n\nYes, that's what ISP drivers do. It also implies using the MC API.\n\n> Anyways this is something to look at if we can\n> get help with figuring out the statistics format. So for now I\n> have not really looked at this yet.\n\nBeing able to get the stats buffers out would also mean someone could\ngive a go at reverse engineering the format, if we need to go that\nroute. I don't know how big the atomisp community is though, and if\nthere could be a volunteer for this task, or if you're alone :-)\n\n> >> The plan is to gather sw statistics on the output buffers\n> >> (so on e.g. YUV images) and then re-use the 3A code from\n> >> the software ISP (I realize gathering statistics on the\n> >> pipeline output buffers is not ideal, but the atomISP driver\n> >> has no way to access the raw bayer data).\n> > \n> > It's more than non-ideal, it will make algorithms development much more\n> > painful :-(\n> \n> Yes this is less then ideal. But at least for basic agc + awb\n> I think we can make this work.\n> \n> Actually for aec/awb it works pretty well since the ISP outputs\n> YUV data getting a Y histogram is easy and even in\n> the postprocessed data it is easy to see if the image is too\n> light / dark.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 02F36BDB13\n\tfor ;\n\tWed, 6 Nov 2024 11:22:32 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E677B6541D;\n\tWed, 6 Nov 2024 12:22:31 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DBFD2653C5\n\tfor ;\n\tWed, 6 Nov 2024 12:22:29 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 85139475;\n\tWed, 6 Nov 2024 12:22:21 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"QBnhmc7+\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1730892141;\n\tbh=saH7V/Itn9hn/HeDPigXDXiA+nSZdzUIguAKojuWM0k=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=QBnhmc7+rY9Uk3Ph5gnrrYKNNZwanXTwbcx4T1IHw9gYhNA61r/2i0KHXD8sV9Mou\n\t98YNmT5ph/vnjy9UajlGGFXRHr6VVMSqaZEwx6sNM/Pn+rzmz86kiJsL+VS+yeRYHg\n\tJhmeMdhZyTDS8BaZFgNbpMdl8v8ogJJTELHVOaF8=","Date":"Wed, 6 Nov 2024 13:22:23 +0200","From":"Laurent Pinchart ","To":"Hans de Goede ","Cc":"libcamera-devel@lists.libcamera.org, Milan Zamazal ,\n\tMaxime Ripard ","Subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","Message-ID":"<20241106112223.GF17733@pendragon.ideasonboard.com>","References":"<20241009200110.275544-1-hdegoede@redhat.com>\n\t<20241010194509.GI32107@pendragon.ideasonboard.com>\n\t<40f2d2a3-e8cd-4578-9588-12cac2b7c4d3@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<40f2d2a3-e8cd-4578-9588-12cac2b7c4d3@redhat.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32052,"web_url":"https://patchwork.libcamera.org/comment/32052/","msgid":"<02226681-6b9a-4236-ac94-26b259adf1dd@redhat.com>","date":"2024-11-06T13:42:08","subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","submitter":{"id":102,"url":"https://patchwork.libcamera.org/api/people/102/","name":"Hans de Goede","email":"hdegoede@redhat.com"},"content":"Hi Laurent,\n\nOn 6-Nov-24 12:22 PM, Laurent Pinchart wrote:\n> On Sun, Nov 03, 2024 at 01:05:29PM +0100, Hans de Goede wrote:\n>> On 10-Oct-24 9:45 PM, Laurent Pinchart wrote:\n>>> On Wed, Oct 09, 2024 at 10:01:06PM +0200, Hans de Goede wrote:\n>>>> Hi All,\n>>>>\n>>>> Here is a patch series adding a processFrame() method to gather\n>>>> statistics on an entire frame in one go, rather then using the\n>>>> line by line approach which is used by the somewhat tightly coupled\n>>>> DebayerCpu class.\n>>>>\n>>>> ATM there are no users for processFrame() yet, so the code is\n>>>> compile tested only which is why it is marked as RFC.\n>>>>\n>>>> I see at least 3 possible use-cases for this:\n>>>>\n>>>> 1. Gathering sw statistics for 3A on raw bayer buffers before\n>>>> passing them along for a to-be-added software CPU ISP raw mode.\n>>>>\n>>>> 2. Gathering sw statistics for 3A on raw bayer buffers before\n>>>> doing further processing on the GPU for the software GPU ISP\n>>>> (possibly as an intermediate step until the stats are gathered\n>>>> on the GPU too and/or as a way to compare CPU vs GPU stats\n>>>> for verification purposes).\n>>>>\n>>>> 3. I'm working on a pipeline handler for the atomISP2, where\n>>>> there is a working hw ISP but no usable statistics.\n>>>\n>>> Why are the stats not usable ? Is it a hardware issue, a firmware issue,\n>>> or \"just\" that we don't have information about the statistics format ?\n>>\n>> There are 2 problems:\n>>\n>> 1. As you say '\"just\" that we don't have information about\n>> the statistics format ?'\n>>\n>> I'll email Sakari to see if he can help with at least a .h files\n>> with a C struct definition for the buffers.\n>>\n>> One thing to keep in mind here is that the hw using the atomisp\n>> is getting pretty old. So while I'm very interested in getting\n>> this going at the same time I also want to not spend too much\n>> time on this.\n> \n> I understand that, but let's not forget that the driver is in staging,\n> and what it implies. There should be continuous effort to clean up the\n> driver to get it out of staging, and I believe that should include\n> things like stats support. It doesn't have to be done right away, but we\n> don't want drivers to bitrot in staging, we have too many of those\n> already (I'm thinking about the IPU3 ImgU driver in particular that will\n> need work, or should be removed from the kernel if it becomes clear\n> nobody will fix its issues).\n\nI do plan to keep supporting and improving the atomisp code. We first\ngot it working (as in displaying any image at all) in 2021 and since\nthen a lot has been improved / changed already.\n\nBut it has been very slow going since this is a spare time side\nproject for me.\n\n> \n>> Using the swstats_cpu code + reusing the ipa_soft_simple.so IPA\n>> is a relatively easy / low-risk way of doing this.\n> \n> As a first step I have no issue about that, we don't have to make\n> everything perfect overnight :-)\n> \n>> I actually am about to post a first version of an atomisp\n>> pipeline handler using this approach to do aec/agc.\n>>\n>> AWB will follow later. First I need to find out where\n>> the atomisp kernel driver sets the ISP R/G/B gains and export\n>> those as standard V4L2 controls, rather then whatever custom\n>> ioctl it is currently using ...\n>>\n>> 2. The kernel driver does receive what it calls \"3a stat\" buffers\n>> but atm it simply discards these. I think there is or used to be\n>> a custom ioctl to retrieve these, but I may have ripped that out\n>> already. Regardless we need to design a proper userspace API for\n>> this. Maybe a separate /dev/video# node using videobuf2 to pass\n>> the buffers?\n> \n> Yes, that's what ISP drivers do. It also implies using the MC API.\n\nRight, note atomisp already used the MC API. So this would mean\nhaving a second /dev/video# MC node with the ISP as the parent /\nsource I presume.\n\nAnd then use videobuf2 \n\n>> Anyways this is something to look at if we can\n>> get help with figuring out the statistics format. So for now I\n>> have not really looked at this yet.\n> \n> Being able to get the stats buffers out would also mean someone could\n> give a go at reverse engineering the format, if we need to go that\n> route. I don't know how big the atomisp community is though, and if\n> there could be a volunteer for this task, or if you're alone :-)\n\nATM I'm mostly alone. It seems there is some interest in getting\nthings to work but no one has really stepped up to help.\n\nI hope that getting a minimal POC with Firefox / snapshot\nshowing video through pipewire on these devices ready will help\nto gather some attention and hopefully more help.\n\nSo I'm happy to read that you write \"As a first step I have no\nissue about that, we don't have to make everything perfect overnight\"\n\nwhich I hope means that we can merge a swstats based pipeline\nhandler as a first step and hopefully that will help with\nbootstrapping some sort of community of atomisp users and\ncontributors.\n\nRegards,\n\nHans","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 57CB3BE173\n\tfor ;\n\tWed, 6 Nov 2024 13:42:16 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 62CA16542B;\n\tWed, 6 Nov 2024 14:42:15 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 97EC3653C5\n\tfor ;\n\tWed, 6 Nov 2024 14:42:14 +0100 (CET)","from mail-ed1-f70.google.com (mail-ed1-f70.google.com\n\t[209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n\tus-mta-487-kXxikdGwNUa2lFcVt8o2gA-1; Wed, 06 Nov 2024 08:42:11 -0500","by mail-ed1-f70.google.com with SMTP id\n\t4fb4d7f45d1cf-5c938ae4152so958177a12.1\n\tfor ;\n\tWed, 06 Nov 2024 05:42:11 -0800 (PST)","from ?IPV6:2001:1c00:c32:7800:5bfa:a036:83f0:f9ec?\n\t(2001-1c00-0c32-7800-5bfa-a036-83f0-f9ec.cable.dynamic.v6.ziggo.nl.\n\t[2001:1c00:c32:7800:5bfa:a036:83f0:f9ec])\n\tby smtp.gmail.com with ESMTPSA id\n\t4fb4d7f45d1cf-5ceecfcde5asm1780832a12.70.2024.11.06.05.42.09\n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tWed, 06 Nov 2024 05:42:09 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"aZerPfY6\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1730900533;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=ENfHveV7ZtkbmGQsFfjown5UZcdt/wKO2taBXRCbOqY=;\n\tb=aZerPfY6Mg4BtTGzGtc+9FWe6FCS0TIwNRi3K4tckHlyUJ2sx5Wid/nuc83mP/NRpluAFU\n\tcZC9WuOVTozzFZZkGAFQzz4IEvnHlhWqtKGOAEutgseU8TFBIzPqX/r99nbnb4zm7k99KW\n\tut/GZsvsjoVI+2EQy/nmIrpNMyJ1YzA=","X-MC-Unique":"kXxikdGwNUa2lFcVt8o2gA-1","X-Mimecast-MFC-AGG-ID":"kXxikdGwNUa2lFcVt8o2gA","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1730900530; x=1731505330;\n\th=content-transfer-encoding:in-reply-to:from:content-language\n\t:references:cc:to:subject:user-agent:mime-version:date:message-id\n\t:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n\tbh=ENfHveV7ZtkbmGQsFfjown5UZcdt/wKO2taBXRCbOqY=;\n\tb=jzbiKj7YSqemfHKySr4yjTRW95f8yJCFA5QFrhW2aBtY2hUv2291oj5GWE5Of/yuGT\n\tL1U1vzJVZKAdZFWs3hNC8H+bhC39DU+MoVVFYUzGr8psiD9IDbByPH7yGR3PV/T3/drO\n\trvYBdrQjg6imrGG+klZ2YIjF7C0wiHTlA9R6Mz3tMWL1F+stAm1M5Vhrg0TLenIYlf7k\n\t8OzOcs69rl6TZlvysmPpbHhDWzwIj8q5KZPxzP82910vnHPS8AXpYcVEz0zDFPgImNEX\n\t29dd2hcGsvEdqjEto+tgTivLOmSgIHj4cndlh63LuhL5ONIlrjmWQnV9WXIGwrKLPOIR\n\tk/WA==","X-Gm-Message-State":"AOJu0YzX1jyz756wz7BvrZL8RhHwcDLGEANnZpCC6Q0SsMW1ivorEkrk\n\tl4TtVM8GOex6KEj3fzpYyKtjP9IwCQCTbr4X4bdMWY2yOjFQqm9tBCf757oUhxOoTMGHdW2dR8x\n\t/trOb2gcxGhqXqt49/XzED1prk+O/bBRHgNHFXUgWqh5I1JRM66k5dZ7CBaH8SUhNWpvli08=","X-Received":["by 2002:a05:6402:2709:b0:5ce:c7b1:e052 with SMTP id\n\t4fb4d7f45d1cf-5cef55266f9mr2750090a12.9.1730900530404; \n\tWed, 06 Nov 2024 05:42:10 -0800 (PST)","by 2002:a05:6402:2709:b0:5ce:c7b1:e052 with SMTP id\n\t4fb4d7f45d1cf-5cef55266f9mr2750063a12.9.1730900529920; \n\tWed, 06 Nov 2024 05:42:09 -0800 (PST)"],"X-Google-Smtp-Source":"AGHT+IEPoThH+AzLNoQdlsMfMAPppGWlhrd3+tCeZmD2T1hlwWTdkU1kTyluqdWVM9UF4+LJ4FY+hg==","Message-ID":"<02226681-6b9a-4236-ac94-26b259adf1dd@redhat.com>","Date":"Wed, 6 Nov 2024 14:42:08 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org, Milan Zamazal ,\n\tMaxime Ripard ","References":"<20241009200110.275544-1-hdegoede@redhat.com>\n\t<20241010194509.GI32107@pendragon.ideasonboard.com>\n\t<40f2d2a3-e8cd-4578-9588-12cac2b7c4d3@redhat.com>\n\t<20241106112223.GF17733@pendragon.ideasonboard.com>","From":"Hans de Goede ","In-Reply-To":"<20241106112223.GF17733@pendragon.ideasonboard.com>","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"RclUcbHJtyv0pteeiGd4tZHSZLT4YwrhCmdHCs5nrZw_1730900531","X-Mimecast-Originator":"redhat.com","Content-Language":"en-US, nl","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"7bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32055,"web_url":"https://patchwork.libcamera.org/comment/32055/","msgid":"<20241106140247.GJ9369@pendragon.ideasonboard.com>","date":"2024-11-06T14:02:47","subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Wed, Nov 06, 2024 at 02:42:08PM +0100, Hans de Goede wrote:\n> On 6-Nov-24 12:22 PM, Laurent Pinchart wrote:\n> > On Sun, Nov 03, 2024 at 01:05:29PM +0100, Hans de Goede wrote:\n> >> On 10-Oct-24 9:45 PM, Laurent Pinchart wrote:\n> >>> On Wed, Oct 09, 2024 at 10:01:06PM +0200, Hans de Goede wrote:\n> >>>> Hi All,\n> >>>>\n> >>>> Here is a patch series adding a processFrame() method to gather\n> >>>> statistics on an entire frame in one go, rather then using the\n> >>>> line by line approach which is used by the somewhat tightly coupled\n> >>>> DebayerCpu class.\n> >>>>\n> >>>> ATM there are no users for processFrame() yet, so the code is\n> >>>> compile tested only which is why it is marked as RFC.\n> >>>>\n> >>>> I see at least 3 possible use-cases for this:\n> >>>>\n> >>>> 1. Gathering sw statistics for 3A on raw bayer buffers before\n> >>>> passing them along for a to-be-added software CPU ISP raw mode.\n> >>>>\n> >>>> 2. Gathering sw statistics for 3A on raw bayer buffers before\n> >>>> doing further processing on the GPU for the software GPU ISP\n> >>>> (possibly as an intermediate step until the stats are gathered\n> >>>> on the GPU too and/or as a way to compare CPU vs GPU stats\n> >>>> for verification purposes).\n> >>>>\n> >>>> 3. I'm working on a pipeline handler for the atomISP2, where\n> >>>> there is a working hw ISP but no usable statistics.\n> >>>\n> >>> Why are the stats not usable ? Is it a hardware issue, a firmware issue,\n> >>> or \"just\" that we don't have information about the statistics format ?\n> >>\n> >> There are 2 problems:\n> >>\n> >> 1. As you say '\"just\" that we don't have information about\n> >> the statistics format ?'\n> >>\n> >> I'll email Sakari to see if he can help with at least a .h files\n> >> with a C struct definition for the buffers.\n> >>\n> >> One thing to keep in mind here is that the hw using the atomisp\n> >> is getting pretty old. So while I'm very interested in getting\n> >> this going at the same time I also want to not spend too much\n> >> time on this.\n> > \n> > I understand that, but let's not forget that the driver is in staging,\n> > and what it implies. There should be continuous effort to clean up the\n> > driver to get it out of staging, and I believe that should include\n> > things like stats support. It doesn't have to be done right away, but we\n> > don't want drivers to bitrot in staging, we have too many of those\n> > already (I'm thinking about the IPU3 ImgU driver in particular that will\n> > need work, or should be removed from the kernel if it becomes clear\n> > nobody will fix its issues).\n> \n> I do plan to keep supporting and improving the atomisp code. We first\n> got it working (as in displaying any image at all) in 2021 and since\n> then a lot has been improved / changed already.\n> \n> But it has been very slow going since this is a spare time side\n> project for me.\n\nThanks for the confirmation, and for the work you're doing there. Slow\nis absolutely fine.\n\n> >> Using the swstats_cpu code + reusing the ipa_soft_simple.so IPA\n> >> is a relatively easy / low-risk way of doing this.\n> > \n> > As a first step I have no issue about that, we don't have to make\n> > everything perfect overnight :-)\n> > \n> >> I actually am about to post a first version of an atomisp\n> >> pipeline handler using this approach to do aec/agc.\n> >>\n> >> AWB will follow later. First I need to find out where\n> >> the atomisp kernel driver sets the ISP R/G/B gains and export\n> >> those as standard V4L2 controls, rather then whatever custom\n> >> ioctl it is currently using ...\n> >>\n> >> 2. The kernel driver does receive what it calls \"3a stat\" buffers\n> >> but atm it simply discards these. I think there is or used to be\n> >> a custom ioctl to retrieve these, but I may have ripped that out\n> >> already. Regardless we need to design a proper userspace API for\n> >> this. Maybe a separate /dev/video# node using videobuf2 to pass\n> >> the buffers?\n> > \n> > Yes, that's what ISP drivers do. It also implies using the MC API.\n> \n> Right, note atomisp already used the MC API. So this would mean\n> having a second /dev/video# MC node with the ISP as the parent /\n> source I presume.\n\nYes that's correct. And eventually there should be a third video node,\nto pass parameters to the ISP.\n\n> And then use videobuf2 \n> \n> >> Anyways this is something to look at if we can\n> >> get help with figuring out the statistics format. So for now I\n> >> have not really looked at this yet.\n> > \n> > Being able to get the stats buffers out would also mean someone could\n> > give a go at reverse engineering the format, if we need to go that\n> > route. I don't know how big the atomisp community is though, and if\n> > there could be a volunteer for this task, or if you're alone :-)\n> \n> ATM I'm mostly alone. It seems there is some interest in getting\n> things to work but no one has really stepped up to help.\n> \n> I hope that getting a minimal POC with Firefox / snapshot\n> showing video through pipewire on these devices ready will help\n> to gather some attention and hopefully more help.\n> \n> So I'm happy to read that you write \"As a first step I have no\n> issue about that, we don't have to make everything perfect overnight\"\n> \n> which I hope means that we can merge a swstats based pipeline\n> handler as a first step and hopefully that will help with\n> bootstrapping some sort of community of atomisp users and\n> contributors.\n\nYes, I'm fine with that. I trust that you will work with the other swisp\ndevelopers to make sure you won't step on each other's toes, now that\nthere will be multiple users of the code.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8499BBE173\n\tfor ;\n\tWed, 6 Nov 2024 14:02:55 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3518E65430;\n\tWed, 6 Nov 2024 15:02:55 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 281FD65429\n\tfor ;\n\tWed, 6 Nov 2024 15:02:54 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id D196259D;\n\tWed, 6 Nov 2024 15:02:45 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"mKkWzWZD\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1730901766;\n\tbh=4wjVXzVJjVgFv8M2dVcvUL8EYzIoW6Wpx6zz9CiBP00=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=mKkWzWZDF/NjNwLB+nKyTL7/s984pfr1bLP7YVi1I7pajatm5pKcmHnb4m/Nhzhbj\n\tbQzrkWecwTsRb+Sg7QfVWZCBwQzXNYnrChqt+JoPLwmVxU1+/6PRfIP/abFDff9PwG\n\tIiyBudqZ6vmN36eK1aQA8G9oomDqflxHSKpB6oK0=","Date":"Wed, 6 Nov 2024 16:02:47 +0200","From":"Laurent Pinchart ","To":"Hans de Goede ","Cc":"libcamera-devel@lists.libcamera.org, Milan Zamazal ,\n\tMaxime Ripard ","Subject":"Re: [RFC 0/4] libcamera: swstats_cpu: Add processFrame() method","Message-ID":"<20241106140247.GJ9369@pendragon.ideasonboard.com>","References":"<20241009200110.275544-1-hdegoede@redhat.com>\n\t<20241010194509.GI32107@pendragon.ideasonboard.com>\n\t<40f2d2a3-e8cd-4578-9588-12cac2b7c4d3@redhat.com>\n\t<20241106112223.GF17733@pendragon.ideasonboard.com>\n\t<02226681-6b9a-4236-ac94-26b259adf1dd@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<02226681-6b9a-4236-ac94-26b259adf1dd@redhat.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]