[{"id":10887,"web_url":"https://patchwork.libcamera.org/comment/10887/","msgid":"","date":"2020-06-26T12:28:56","subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Naush\n\nThanks for this. Just the one nit-picky little typo (sorry!),\notherwise looks good to me!\n\nBest regards\nDavid\n\nReviewed-by: David Plowman \n\n\nOn Fri, 26 Jun 2020 at 11:25, Naushir Patuck wrote:\n>\n> Provide a control to allow the IPA to return a FoM to indicate how\n> in-focus a scene is. Note, this is not to be used as a means to\n> implement a focus algorithm by the application, rather an indication of\n> how in-focus a scene is.\n>\n> Signed-off-by: Naushir Patuck \n> ---\n> src/libcamera/control_ids.yaml | 11 +++++++++++\n> 1 file changed, 11 insertions(+)\n>\n> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> index 8c3e4c72..b46693ec 100644\n> --- a/src/libcamera/control_ids.yaml\n> +++ b/src/libcamera/control_ids.yaml\n> @@ -251,4 +251,15 @@ controls:\n> higher than anyone could reasonably want. Negative values are\n> not allowed. Note also that sharpening is not applied to raw\n> streams.\n> +\n> + - FocusFoM:\n> + type: int32_t\n> + description: |\n> + Reports a Figure of Merit (FoM) to indicate how in-focus the frame is.\n> + A larger FocusFoM value indicates a more in-focus frame. This control\n> + depends on the IPA to gather ISP statistics from the defined focus\n> + region, and combine them in a suitable way to generate a FocusFoM value.\n> + In this respect, it is not necessarily aimed a providing a way to\n\n\"aimed at providing\"\n\n> + implement a focus algorithm by the application, rather an indication of\n> + how in-focus a frame is.\n> ...\n> --\n> 2.25.1\n>\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 05787C2E66\n\tfor ;\n\tFri, 26 Jun 2020 12:29:10 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 583BC609C7;\n\tFri, 26 Jun 2020 14:29:09 +0200 (CEST)","from mail-oi1-x244.google.com (mail-oi1-x244.google.com\n\t[IPv6:2607:f8b0:4864:20::244])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7A0B1603BB\n\tfor ;\n\tFri, 26 Jun 2020 14:29:08 +0200 (CEST)","by mail-oi1-x244.google.com with SMTP id t25so7900788oij.7\n\tfor ;\n\tFri, 26 Jun 2020 05:29:08 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"Y82AkItB\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=khZnAO/jtov1Z2+l6C+InZKZNDHI7LeGmwQWISx7fug=;\n\tb=Y82AkItB3C1X3aY6IV7qeo7WB4wLpIqDuR6xsNZQ/9El/jou3NV1N28/tdp+g+HYdW\n\tCekcUV+kfjMK95vuOkzEWl9avqWFxu/8b9wFhOYkYCvehQmcKI90l//LuUi5eq90dRQJ\n\tjCE++UIdYZOMlGeywk/Um9c9o2HDF7oqLVr/kDsHZofBPLqUUoktZvL+JHETtwREjGGe\n\twOdBzoUbo4jRrV2lIRQuSa9m+2TNTyTC0qDVIVmPn5mEFo6Y4Ka4iZ9lYMHG5+CpfbPW\n\tsufBcV5uSsF4Xok3QqheWnC48mQG5UndKRAGGP7LqnLfO5b71QEAN56c5eEVWQX154Ew\n\tyJZA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=khZnAO/jtov1Z2+l6C+InZKZNDHI7LeGmwQWISx7fug=;\n\tb=ha73K+f1m53HqH2h0k2zujnBv4HlE429CGJuo0zT6PiheL/dZ8APjY6Mwwk88Hm6wx\n\thGRWnS3oW6W4nRBahZEYBjrYr1/iQixAzHg+tARsEo44/wMEC5hdujNvK6HOnS5pxsW4\n\tg1spufeZOhJzGDVnAF7yFJYrIJZAonlxahjvBMcBwLzGmSqMeRcOcuak5Sr0KFov97FB\n\tYngSAcgU/W+1ODjI24K0t6aB9IDhirvNl7e30pIQqSsli+K2gPFIPVuKoI3PQRQhPJRC\n\tU6jwWBun6gchGS7OPSFzd7p/r/Bjf90Hlp08Kow2A1wFNNCiG41UPcgm6AUYiCy9rWEA\n\tu0Zw==","X-Gm-Message-State":"AOAM531FmPXlWkQRHsWaeFM/hhwGys9ToeF5/kyaycFb2hnDk1QISPkB\n\td0ORlTx2QPOzlls82zosoG9MotloKeLS5qR0T8HAaQ==","X-Google-Smtp-Source":"ABdhPJx6QMRJYYG00i23ONE31r8PQ8tuwV/hfCH34AkXBG42QzDe/U+dYUKKdaQJnLUvtzf+MoQnlObtS5GW2ufnGcM=","X-Received":"by 2002:aca:783:: with SMTP id 125mr2080043oih.55.1593174547077; \n\tFri, 26 Jun 2020 05:29:07 -0700 (PDT)","MIME-Version":"1.0","References":"<20200626102531.1187650-1-naush@raspberrypi.com>\n\t<20200626102531.1187650-2-naush@raspberrypi.com>","In-Reply-To":"<20200626102531.1187650-2-naush@raspberrypi.com>","From":"David Plowman ","Date":"Fri, 26 Jun 2020 13:28:56 +0100","Message-ID":"","To":"Naushir Patuck ","Subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":10948,"web_url":"https://patchwork.libcamera.org/comment/10948/","msgid":"<20200629001544.GH6954@pendragon.ideasonboard.com>","date":"2020-06-29T00:15:44","subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Naush,\n\nThank you for the patch.\n\nOn Fri, Jun 26, 2020 at 11:25:28AM +0100, Naushir Patuck wrote:\n> Provide a control to allow the IPA to return a FoM to indicate how\n> in-focus a scene is. Note, this is not to be used as a means to\n> implement a focus algorithm by the application, rather an indication of\n> how in-focus a scene is.\n> \n> Signed-off-by: Naushir Patuck \n> ---\n> src/libcamera/control_ids.yaml | 11 +++++++++++\n> 1 file changed, 11 insertions(+)\n> \n> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> index 8c3e4c72..b46693ec 100644\n> --- a/src/libcamera/control_ids.yaml\n> +++ b/src/libcamera/control_ids.yaml\n> @@ -251,4 +251,15 @@ controls:\n> higher than anyone could reasonably want. Negative values are\n> not allowed. Note also that sharpening is not applied to raw\n> streams.\n> +\n> + - FocusFoM:\n> + type: int32_t\n> + description: |\n> + Reports a Figure of Merit (FoM) to indicate how in-focus the frame is.\n> + A larger FocusFoM value indicates a more in-focus frame. This control\n> + depends on the IPA to gather ISP statistics from the defined focus\n> + region, and combine them in a suitable way to generate a FocusFoM value.\n> + In this respect, it is not necessarily aimed a providing a way to\n> + implement a focus algorithm by the application, rather an indication of\n> + how in-focus a frame is.\n\nDo you think this would be sufficient for applications, or would it be\nbetter to use an array control that would return an array of values ?\nLooking a 4/4 your ISP returns a grid of 4x3 values, so that could fit\nin a control without a large performance impact.\n\nThis leads to the next question of whether we should then report all\nstatistics in a buffer instead, and possibly even standardize the\nstatistics format. That could open pandora's box though, it's a rabbit\nhole that could end up being pretty deep. I tried to look around for\nother implementations, and there's support for histogram and focus\n(sharpness) data in the Android camera HAL API, but marked as\n\"future\". It seems to have been work in progress pretty much from the\nstart, without much interest in finalizing it. Looking at OpenKCam could\nbe interesting too, but there's no public spec I could find. Are you\naware of any other attempt at standardizing something in this area ?\n\n> ...","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0F959C2E66\n\tfor ;\n\tMon, 29 Jun 2020 00:15:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 74001609C9;\n\tMon, 29 Jun 2020 02:15:49 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DFCF9603B6\n\tfor ;\n\tMon, 29 Jun 2020 02:15:47 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi\n\t[81.175.216.236])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5AEBA4FB;\n\tMon, 29 Jun 2020 02:15:47 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"dCRZUphA\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1593389747;\n\tbh=sI/8QMGgfXs4U8fpHD1ltoqZuCZW6sNWW02ZOjNCyZ0=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=dCRZUphADQKoWivoxvOL5qKShV1mP5Qq68m1GgGRxoUTdzZBZ5zP2j6ZmePuxqGCO\n\tN/GvWHsUmD3s0ifQ/+xJbKMplzuCB02PQXC7L5wMDW1QUlK2762oLn0NeaQTVvjQVT\n\tIFgsLL43syh/gis4dQSiJArOhfa0h+wpJjygLBQ0=","Date":"Mon, 29 Jun 2020 03:15:44 +0300","From":"Laurent Pinchart ","To":"Naushir Patuck ","Message-ID":"<20200629001544.GH6954@pendragon.ideasonboard.com>","References":"<20200626102531.1187650-1-naush@raspberrypi.com>\n\t<20200626102531.1187650-2-naush@raspberrypi.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20200626102531.1187650-2-naush@raspberrypi.com>","Subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":10960,"web_url":"https://patchwork.libcamera.org/comment/10960/","msgid":"","date":"2020-06-29T10:41:45","subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Laurent,\n\nThank you for the comments.\n\nOn Mon, 29 Jun 2020 at 01:15, Laurent Pinchart\n wrote:\n>\n> Hi Naush,\n>\n> Thank you for the patch.\n>\n> On Fri, Jun 26, 2020 at 11:25:28AM +0100, Naushir Patuck wrote:\n> > Provide a control to allow the IPA to return a FoM to indicate how\n> > in-focus a scene is. Note, this is not to be used as a means to\n> > implement a focus algorithm by the application, rather an indication of\n> > how in-focus a scene is.\n> >\n> > Signed-off-by: Naushir Patuck \n> > ---\n> > src/libcamera/control_ids.yaml | 11 +++++++++++\n> > 1 file changed, 11 insertions(+)\n> >\n> > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > index 8c3e4c72..b46693ec 100644\n> > --- a/src/libcamera/control_ids.yaml\n> > +++ b/src/libcamera/control_ids.yaml\n> > @@ -251,4 +251,15 @@ controls:\n> > higher than anyone could reasonably want. Negative values are\n> > not allowed. Note also that sharpening is not applied to raw\n> > streams.\n> > +\n> > + - FocusFoM:\n> > + type: int32_t\n> > + description: |\n> > + Reports a Figure of Merit (FoM) to indicate how in-focus the frame is.\n> > + A larger FocusFoM value indicates a more in-focus frame. This control\n> > + depends on the IPA to gather ISP statistics from the defined focus\n> > + region, and combine them in a suitable way to generate a FocusFoM value.\n> > + In this respect, it is not necessarily aimed a providing a way to\n> > + implement a focus algorithm by the application, rather an indication of\n> > + how in-focus a frame is.\n>\n> Do you think this would be sufficient for applications, or would it be\n> better to use an array control that would return an array of values ?\n> Looking a 4/4 your ISP returns a grid of 4x3 values, so that could fit\n> in a control without a large performance impact.\n\nDavid and I did discuss this at length. We concluded that a single\nvalue here was good because it simplified what the application wants\nto know from this control, i.e. how good is the focus in this scene.\nAnything more complicated, like a real focus algorithm would need\naccess to all of the grid, and that can live happily in the IPAs. I\nam ok to extend this to an array to provide all 12 values, but if we\ndo that, arguably we should add another control like FocusFomRegions\nthat list the region coordinates for each FoM. This will allow the\ncontrol to be more generic for other HW vendors.\n\n>\n> This leads to the next question of whether we should then report all\n> statistics in a buffer instead, and possibly even standardize the\n> statistics format. That could open pandora's box though, it's a rabbit\n> hole that could end up being pretty deep. I tried to look around for\n> other implementations, and there's support for histogram and focus\n> (sharpness) data in the Android camera HAL API, but marked as\n> \"future\". It seems to have been work in progress pretty much from the\n> start, without much interest in finalizing it. Looking at OpenKCam could\n> be interesting too, but there's no public spec I could find. Are you\n> aware of any other attempt at standardizing something in this area ?\n>\n\nI recall we talked about standard formats for statistics a year (or\nso) ago and got very nervous about doing such a thing :) I was\nbriefly involved in the OpenKCam effort, and we did talk about exactly\nthis. If I recall, the consensus between hardware vendors was the\nsame - we might spend too much effort (and add lots of complexity) to\ndefine a generic statistics format, which might never get used by\nusers, and vendors never implement.\n\nPart of me feels that is still the case with libcamera. Vendors like\nRaspberry Pi do have the statistics buffer documented. In the rare\ncase a user does want statistics to do some magic stuff in the\napplication, they can request the stats buffer (when the code is\navailable), and resign to the fact that how they handle the statistics\ndata will not be portable with other vendors.\n\nRegards,\nNaush\n\n\n> > ...\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id F1D12BF415\n\tfor ;\n\tMon, 29 Jun 2020 10:42:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6BA53609C9;\n\tMon, 29 Jun 2020 12:42:05 +0200 (CEST)","from mail-lj1-x243.google.com (mail-lj1-x243.google.com\n\t[IPv6:2a00:1450:4864:20::243])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 584CD603B4\n\tfor ;\n\tMon, 29 Jun 2020 12:42:03 +0200 (CEST)","by mail-lj1-x243.google.com with SMTP id d17so2771324ljl.3\n\tfor ;\n\tMon, 29 Jun 2020 03:42:03 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"RWLp2CTN\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=QWdaoJpEukxsR23rm/0dJLicDGIXzv7PJ8kF3gxM1B0=;\n\tb=RWLp2CTNwVtA5NAYmxexCdoWjTo4UnhPuDNGhgxt8VWseaTBRL8q6cXmx4Q2sb9iIG\n\tV/JEVy+X0YtXy5WkmgdlD9UwGJwxYEYucRQiR85YeePp1AI2oh5wT94VL+56FtQa+rp1\n\tBUsgDsHrT4yGrcfNenpsFF5/7G/Qr0UIOa1sK5bpda8Wq8pbCQHQgiwdso3NRw+LgN7g\n\tQ+HapXMqwtIZP5FQFMxj13mkrJ3Dznbkp1bf5aqQZnOl/fYXtDw13GXbOmvkHmgvbXh0\n\t5smzUM4m3yaivaoy7SzGxtH64B465zrKscxyPZ8vgX0tOuuGWyVTQyQkybyhyYK1kp1a\n\tSY/A==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=QWdaoJpEukxsR23rm/0dJLicDGIXzv7PJ8kF3gxM1B0=;\n\tb=DSMhGdXR3lKXxW8JknEiV7hUJMycP5eEZUIOfNBU84GO/0rZv/6axw2mj6hpYZ80DI\n\tfnUz8OAVpFnePS+qYEuS/ZPwtxQv8qbOfkGlnYx5W1LW1HZkOdKiPnOo38J9XCpwU/Wg\n\tLTTAaS9ZiP/kCmd2BEyitLKHM7Y7xOAZYzb+AAQmmcHcKnZo7NaCHZDop+WyHtwrPQde\n\t1cGNlzXTqDoPV/+VSRwcsBoWreZUuGYllQjT+gLH6Dee7lSyizVOYRlgLXtkJ0EIjDgz\n\tZm5OBKHW5vy0nPOutjXlux5OEUb9RmHp7Lm6MeZ/GcVUiY2KaIegMBqmASxlMZ7Id1Py\n\tP5Qw==","X-Gm-Message-State":"AOAM531iQS5XkJ19I/AqYHxgVJDkQVOBOE9JHzrdF3FzroVbOhiSbWNM\n\tLeWiZtPUd1V5TkJxazRB3m6xiH2iOmTRXMDsAGNt03TU","X-Google-Smtp-Source":"ABdhPJyumNtwVf6anxZdBz75CbvDw4roCYot6q0iC+vGruKgABphGKKv/XUTClIzixzEM8KCwNt/k3kNjiJYuzDkT0I=","X-Received":"by 2002:a2e:3a04:: with SMTP id h4mr8001224lja.103.1593427322471;\n\tMon, 29 Jun 2020 03:42:02 -0700 (PDT)","MIME-Version":"1.0","References":"<20200626102531.1187650-1-naush@raspberrypi.com>\n\t<20200626102531.1187650-2-naush@raspberrypi.com>\n\t<20200629001544.GH6954@pendragon.ideasonboard.com>","In-Reply-To":"<20200629001544.GH6954@pendragon.ideasonboard.com>","From":"Naushir Patuck ","Date":"Mon, 29 Jun 2020 11:41:45 +0100","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11111,"web_url":"https://patchwork.libcamera.org/comment/11111/","msgid":"<20200703012106.GM12562@pendragon.ideasonboard.com>","date":"2020-07-03T01:21:06","subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Naush,\n\nOn Mon, Jun 29, 2020 at 11:41:45AM +0100, Naushir Patuck wrote:\n> On Mon, 29 Jun 2020 at 01:15, Laurent Pinchart wrote:\n> > On Fri, Jun 26, 2020 at 11:25:28AM +0100, Naushir Patuck wrote:\n> > > Provide a control to allow the IPA to return a FoM to indicate how\n> > > in-focus a scene is. Note, this is not to be used as a means to\n> > > implement a focus algorithm by the application, rather an indication of\n> > > how in-focus a scene is.\n> > >\n> > > Signed-off-by: Naushir Patuck \n> > > ---\n> > > src/libcamera/control_ids.yaml | 11 +++++++++++\n> > > 1 file changed, 11 insertions(+)\n> > >\n> > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > index 8c3e4c72..b46693ec 100644\n> > > --- a/src/libcamera/control_ids.yaml\n> > > +++ b/src/libcamera/control_ids.yaml\n> > > @@ -251,4 +251,15 @@ controls:\n> > > higher than anyone could reasonably want. Negative values are\n> > > not allowed. Note also that sharpening is not applied to raw\n> > > streams.\n> > > +\n> > > + - FocusFoM:\n> > > + type: int32_t\n> > > + description: |\n> > > + Reports a Figure of Merit (FoM) to indicate how in-focus the frame is.\n> > > + A larger FocusFoM value indicates a more in-focus frame. This control\n> > > + depends on the IPA to gather ISP statistics from the defined focus\n> > > + region, and combine them in a suitable way to generate a FocusFoM value.\n> > > + In this respect, it is not necessarily aimed a providing a way to\n> > > + implement a focus algorithm by the application, rather an indication of\n> > > + how in-focus a frame is.\n> >\n> > Do you think this would be sufficient for applications, or would it be\n> > better to use an array control that would return an array of values ?\n> > Looking a 4/4 your ISP returns a grid of 4x3 values, so that could fit\n> > in a control without a large performance impact.\n> \n> David and I did discuss this at length. We concluded that a single\n> value here was good because it simplified what the application wants\n> to know from this control, i.e. how good is the focus in this scene.\n> Anything more complicated, like a real focus algorithm would need\n> access to all of the grid, and that can live happily in the IPAs.\n\nAgreed, auto-focus should live in the IPA.\n\nI'm curious, what are the use cases for applications, how would they\nconsume this value ?\n\nAlso, do you see any issue with the FocusFoM control as defined here\nwhen used with sensors that produce PDAF data ? I have no particular\nconcern, but you have more experience than I do in this area.\n\nAnother question, I assume that in most (all ?) case the focus\nstatistics are a measure of sharpness, and that two images with the same\nfocus quality but with a very different level of details in the scene\nwould have different focus FoM values, right ? I'd be tempted to name\nthe control to refer to sharpness instead of focus then, but we already\nhave a sharpness control that is completely unrelated. We can of course\nalways change names later, but I'd like your opinion here. Also, is my\nunderstanding applicable to PDAF too, or would there be differences\nthere ?\n\n> I am ok to extend this to an array to provide all 12 values, but if we\n> do that, arguably we should add another control like FocusFomRegions\n> that list the region coordinates for each FoM. This will allow the\n> control to be more generic for other HW vendors.\n\nYes, I agree that would make sense.\n\n> > This leads to the next question of whether we should then report all\n> > statistics in a buffer instead, and possibly even standardize the\n> > statistics format. That could open pandora's box though, it's a rabbit\n> > hole that could end up being pretty deep. I tried to look around for\n> > other implementations, and there's support for histogram and focus\n> > (sharpness) data in the Android camera HAL API, but marked as\n> > \"future\". It seems to have been work in progress pretty much from the\n> > start, without much interest in finalizing it. Looking at OpenKCam could\n> > be interesting too, but there's no public spec I could find. Are you\n> > aware of any other attempt at standardizing something in this area ?\n> \n> I recall we talked about standard formats for statistics a year (or\n> so) ago and got very nervous about doing such a thing :) I was\n> briefly involved in the OpenKCam effort, and we did talk about exactly\n> this. If I recall, the consensus between hardware vendors was the\n> same - we might spend too much effort (and add lots of complexity) to\n> define a generic statistics format, which might never get used by\n> users, and vendors never implement.\n\nI share the same concern. It may quickly become too complicated to\nreally be useful.\n\n> Part of me feels that is still the case with libcamera. Vendors like\n> Raspberry Pi do have the statistics buffer documented. In the rare\n> case a user does want statistics to do some magic stuff in the\n> application, they can request the stats buffer (when the code is\n> available), and resign to the fact that how they handle the statistics\n> data will not be portable with other vendors.\n\nI think this makes sense. If we decide to standardize statistics, it\nwould probably be better implemented as a library that parses vendor\nstatistics formats, instead of implemented in the IPA.\n\nI however don't rule out, for some statistics, the possibility of having\nthe IPA reporting metadata about the statistics format, in order to ease\nusage in applications, or even to allow coding sharing of algorithms\nbetween different vendors. That's a topic for later though.\n\n> > > ...","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id CEE44BFFE2\n\tfor ;\n\tFri, 3 Jul 2020 01:21:12 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 647C960C56;\n\tFri, 3 Jul 2020 03:21:12 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 38C8B603B3\n\tfor ;\n\tFri, 3 Jul 2020 03:21:11 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi\n\t[81.175.216.236])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A329531F;\n\tFri, 3 Jul 2020 03:21:10 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"t19fKk8x\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1593739270;\n\tbh=EJf5BrJMC+okjTFnTGBXlAgK0aibRNA4zDTFifz1hIg=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=t19fKk8xoV3BKIUEwC4Y7P9fAgXlhR4VDdiTn6KmJwj4eE8N8p3MDybxmwPc2Gy1A\n\t2n2HGj8yEp37nM1eOcAPe3dNfkmEE357Vrov5bJ0JzBWiqsfoeiC45rws5i+MWqRnQ\n\tn2ebds0qtwiUDYZJZcIi5Um7aHgmrcs9nVGzWUNw=","Date":"Fri, 3 Jul 2020 04:21:06 +0300","From":"Laurent Pinchart ","To":"Naushir Patuck ","Message-ID":"<20200703012106.GM12562@pendragon.ideasonboard.com>","References":"<20200626102531.1187650-1-naush@raspberrypi.com>\n\t<20200626102531.1187650-2-naush@raspberrypi.com>\n\t<20200629001544.GH6954@pendragon.ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11113,"web_url":"https://patchwork.libcamera.org/comment/11113/","msgid":"","date":"2020-07-03T07:29:49","subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Laurent,\n\nOn Fri, 3 Jul 2020 at 02:21, Laurent Pinchart\n wrote:\n>\n> Hi Naush,\n>\n> On Mon, Jun 29, 2020 at 11:41:45AM +0100, Naushir Patuck wrote:\n> > On Mon, 29 Jun 2020 at 01:15, Laurent Pinchart wrote:\n> > > On Fri, Jun 26, 2020 at 11:25:28AM +0100, Naushir Patuck wrote:\n> > > > Provide a control to allow the IPA to return a FoM to indicate how\n> > > > in-focus a scene is. Note, this is not to be used as a means to\n> > > > implement a focus algorithm by the application, rather an indication of\n> > > > how in-focus a scene is.\n> > > >\n> > > > Signed-off-by: Naushir Patuck \n> > > > ---\n> > > > src/libcamera/control_ids.yaml | 11 +++++++++++\n> > > > 1 file changed, 11 insertions(+)\n> > > >\n> > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > index 8c3e4c72..b46693ec 100644\n> > > > --- a/src/libcamera/control_ids.yaml\n> > > > +++ b/src/libcamera/control_ids.yaml\n> > > > @@ -251,4 +251,15 @@ controls:\n> > > > higher than anyone could reasonably want. Negative values are\n> > > > not allowed. Note also that sharpening is not applied to raw\n> > > > streams.\n> > > > +\n> > > > + - FocusFoM:\n> > > > + type: int32_t\n> > > > + description: |\n> > > > + Reports a Figure of Merit (FoM) to indicate how in-focus the frame is.\n> > > > + A larger FocusFoM value indicates a more in-focus frame. This control\n> > > > + depends on the IPA to gather ISP statistics from the defined focus\n> > > > + region, and combine them in a suitable way to generate a FocusFoM value.\n> > > > + In this respect, it is not necessarily aimed a providing a way to\n> > > > + implement a focus algorithm by the application, rather an indication of\n> > > > + how in-focus a frame is.\n> > >\n> > > Do you think this would be sufficient for applications, or would it be\n> > > better to use an array control that would return an array of values ?\n> > > Looking a 4/4 your ISP returns a grid of 4x3 values, so that could fit\n> > > in a control without a large performance impact.\n> >\n> > David and I did discuss this at length. We concluded that a single\n> > value here was good because it simplified what the application wants\n> > to know from this control, i.e. how good is the focus in this scene.\n> > Anything more complicated, like a real focus algorithm would need\n> > access to all of the grid, and that can live happily in the IPAs.\n>\n> Agreed, auto-focus should live in the IPA.\n>\n> I'm curious, what are the use cases for applications, how would they\n> consume this value ?\n\nThe immediate use would be for the HQ camera with manual focus lenses.\nRelying on observing the viewfinder image is generally not good enough\nto determine the best focus position. Instead, users can observe the\nFoM to get a numeric measure of the best focus position when manually\nfocussing. Of course, this is only really applicable if the scene is\nstatic and the camera is sat at a fixed position.\n\n>\n> Also, do you see any issue with the FocusFoM control as defined here\n> when used with sensors that produce PDAF data ? I have no particular\n> concern, but you have more experience than I do in this area.\n\nFigure of Merit is a generic enough term that I think could encompass\nPDAF metrics as well. Again, this is just a measure of \"how good is\nthe focus on this scene\", and it is up to the IPAs to determine how to\nprovide a numeric value to that.\n\n>\n> Another question, I assume that in most (all ?) case the focus\n> statistics are a measure of sharpness, and that two images with the same\n> focus quality but with a very different level of details in the scene\n> would have different focus FoM values, right ? I'd be tempted to name\n> the control to refer to sharpness instead of focus then, but we already\n> have a sharpness control that is completely unrelated. We can of course\n> always change names later, but I'd like your opinion here. Also, is my\n> understanding applicable to PDAF too, or would there be differences\n> there ?\n\nTwo scenes with the same focus quality but with a very different level\nof details would indeed produce different FoM values. I did dither\nabout calling it a sharpness or contrast measure, but that might be\nconfusing as we have sharpness and contrast controls to set for the\nimage. And as you pointed out, PDAF won't necessarily provide a\ncontrast/sharpness value. Happy to change the control name if we can\nagree on something suitable :)\n\n>\n> > I am ok to extend this to an array to provide all 12 values, but if we\n> > do that, arguably we should add another control like FocusFomRegions\n> > that list the region coordinates for each FoM. This will allow the\n> > control to be more generic for other HW vendors.\n>\n> Yes, I agree that would make sense.\n\nIf the consensus is to provide all 12 regions I can easily make the\nchanges, and include the new FocusFomRegions control.\n\n>\n> > > This leads to the next question of whether we should then report all\n> > > statistics in a buffer instead, and possibly even standardize the\n> > > statistics format. That could open pandora's box though, it's a rabbit\n> > > hole that could end up being pretty deep. I tried to look around for\n> > > other implementations, and there's support for histogram and focus\n> > > (sharpness) data in the Android camera HAL API, but marked as\n> > > \"future\". It seems to have been work in progress pretty much from the\n> > > start, without much interest in finalizing it. Looking at OpenKCam could\n> > > be interesting too, but there's no public spec I could find. Are you\n> > > aware of any other attempt at standardizing something in this area ?\n> >\n> > I recall we talked about standard formats for statistics a year (or\n> > so) ago and got very nervous about doing such a thing :) I was\n> > briefly involved in the OpenKCam effort, and we did talk about exactly\n> > this. If I recall, the consensus between hardware vendors was the\n> > same - we might spend too much effort (and add lots of complexity) to\n> > define a generic statistics format, which might never get used by\n> > users, and vendors never implement.\n>\n> I share the same concern. It may quickly become too complicated to\n> really be useful.\n>\n> > Part of me feels that is still the case with libcamera. Vendors like\n> > Raspberry Pi do have the statistics buffer documented. In the rare\n> > case a user does want statistics to do some magic stuff in the\n> > application, they can request the stats buffer (when the code is\n> > available), and resign to the fact that how they handle the statistics\n> > data will not be portable with other vendors.\n>\n> I think this makes sense. If we decide to standardize statistics, it\n> would probably be better implemented as a library that parses vendor\n> statistics formats, instead of implemented in the IPA.\n>\n> I however don't rule out, for some statistics, the possibility of having\n> the IPA reporting metadata about the statistics format, in order to ease\n> usage in applications, or even to allow coding sharing of algorithms\n> between different vendors. That's a topic for later though.\n>\n> > > > ...\n>\n> --\n> Regards,\n>\n> Laurent Pinchart\n\nRegards,\nNaush","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C2D88BFFE2\n\tfor ;\n\tFri, 3 Jul 2020 07:30:08 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5C47C60C50;\n\tFri, 3 Jul 2020 09:30:08 +0200 (CEST)","from mail-lj1-x244.google.com (mail-lj1-x244.google.com\n\t[IPv6:2a00:1450:4864:20::244])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2FBA5603AD\n\tfor ;\n\tFri, 3 Jul 2020 09:30:07 +0200 (CEST)","by mail-lj1-x244.google.com with SMTP id z24so10902230ljn.8\n\tfor ;\n\tFri, 03 Jul 2020 00:30:07 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"MPMp3RlC\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=zzhn74U7ocqrw2EooByHJtC17NSeN2nxUTEpIPZgCc4=;\n\tb=MPMp3RlC0jgmlSwRZ7UqhPUe9EM6MO5f1g2oZRIR3BSfSFbuLjhgWn1jjaNu8+qYmR\n\tdvMXGLPmJV+BVfN3WcwloJ5GL5UJefdbY1IpRv677V7W7oB6cNqRT4CjTa2ht4EI2qpl\n\t5i9tw4jpe3XScF+zmFoco6MIfHEd+7OoC86w835xky+OcVdS4JoOUnm2b++lh+JmGBBl\n\tDhmXPT4PYAPxEWJeYvAX0yKQr0LWNNTwrkWhEX8eVApM7nWbAOkifSXNvhExycoelo6l\n\tvGTfESRd5bKbOB/4OVV8JUEnuwHFDo+kvmaEgrLM8LyArNmMuP1amcs7Pq/8D2zVSQv8\n\tBVJg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=zzhn74U7ocqrw2EooByHJtC17NSeN2nxUTEpIPZgCc4=;\n\tb=L6Ax1Sg/WH9TlGlXHzeXvHsxbL1BuHSPX/NhgNoqUFqKpZKKz5MM9rt1CcZhKuDkaK\n\t/GWOqRc6mMQhxEy9Lo3C5bfcrmc32SqTj3uGQR4C7JAg4F5l1A5dyfFY4G1L4iIZidNX\n\tU9AAae4W5fnKXyyFKIqkZRIoLAX804pigbQLrZ7SBe4QxL5hmuoErEmDKtnQXFm7+BPU\n\tVcgFeOTKi9Nx2NHF66zei0ax61f/ABZNbGoFvMh1mEJFgsBKt1BxjER1r/fN3mp8cfKi\n\t0BcIVkur/oCdzDGEFkFGNK05UZnRX/j6UhzE63sJy77eKWdADAi314RVEW7pO7p1Op7s\n\tuE0w==","X-Gm-Message-State":"AOAM530lRDdHjDGsiD8feOiBYLrs2Cc/TZYtnGVyhQWRuIzWlob+ZDhk\n\t6UmG23jb9I3htZHBKC7PDbP2NYWyPTppW0ZCuroVlw==","X-Google-Smtp-Source":"ABdhPJwmAVPH+9nPxzlF6Hoba1deaohCbCwfN0eouac/8sRCNfUi20t8oq7YKlFK7Dh6R4LbW96VHBjY/YUXZfzoDF0=","X-Received":"by 2002:a2e:3a15:: with SMTP id\n\th21mr6890732lja.112.1593761406108; \n\tFri, 03 Jul 2020 00:30:06 -0700 (PDT)","MIME-Version":"1.0","References":"<20200626102531.1187650-1-naush@raspberrypi.com>\n\t<20200626102531.1187650-2-naush@raspberrypi.com>\n\t<20200629001544.GH6954@pendragon.ideasonboard.com>\n\t\n\t<20200703012106.GM12562@pendragon.ideasonboard.com>","In-Reply-To":"<20200703012106.GM12562@pendragon.ideasonboard.com>","From":"Naushir Patuck ","Date":"Fri, 3 Jul 2020 08:29:49 +0100","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":11140,"web_url":"https://patchwork.libcamera.org/comment/11140/","msgid":"<20200703103923.GC5963@pendragon.ideasonboard.com>","date":"2020-07-03T10:39:23","subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Naush,\n\nOn Fri, Jul 03, 2020 at 08:29:49AM +0100, Naushir Patuck wrote:\n> On Fri, 3 Jul 2020 at 02:21, Laurent Pinchart wrote:\n> > On Mon, Jun 29, 2020 at 11:41:45AM +0100, Naushir Patuck wrote:\n> > > On Mon, 29 Jun 2020 at 01:15, Laurent Pinchart wrote:\n> > > > On Fri, Jun 26, 2020 at 11:25:28AM +0100, Naushir Patuck wrote:\n> > > > > Provide a control to allow the IPA to return a FoM to indicate how\n> > > > > in-focus a scene is. Note, this is not to be used as a means to\n> > > > > implement a focus algorithm by the application, rather an indication of\n> > > > > how in-focus a scene is.\n> > > > >\n> > > > > Signed-off-by: Naushir Patuck \n> > > > > ---\n> > > > > src/libcamera/control_ids.yaml | 11 +++++++++++\n> > > > > 1 file changed, 11 insertions(+)\n> > > > >\n> > > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml\n> > > > > index 8c3e4c72..b46693ec 100644\n> > > > > --- a/src/libcamera/control_ids.yaml\n> > > > > +++ b/src/libcamera/control_ids.yaml\n> > > > > @@ -251,4 +251,15 @@ controls:\n> > > > > higher than anyone could reasonably want. Negative values are\n> > > > > not allowed. Note also that sharpening is not applied to raw\n> > > > > streams.\n> > > > > +\n> > > > > + - FocusFoM:\n> > > > > + type: int32_t\n> > > > > + description: |\n> > > > > + Reports a Figure of Merit (FoM) to indicate how in-focus the frame is.\n> > > > > + A larger FocusFoM value indicates a more in-focus frame. This control\n> > > > > + depends on the IPA to gather ISP statistics from the defined focus\n> > > > > + region, and combine them in a suitable way to generate a FocusFoM value.\n> > > > > + In this respect, it is not necessarily aimed a providing a way to\n> > > > > + implement a focus algorithm by the application, rather an indication of\n> > > > > + how in-focus a frame is.\n> > > >\n> > > > Do you think this would be sufficient for applications, or would it be\n> > > > better to use an array control that would return an array of values ?\n> > > > Looking a 4/4 your ISP returns a grid of 4x3 values, so that could fit\n> > > > in a control without a large performance impact.\n> > >\n> > > David and I did discuss this at length. We concluded that a single\n> > > value here was good because it simplified what the application wants\n> > > to know from this control, i.e. how good is the focus in this scene.\n> > > Anything more complicated, like a real focus algorithm would need\n> > > access to all of the grid, and that can live happily in the IPAs.\n> >\n> > Agreed, auto-focus should live in the IPA.\n> >\n> > I'm curious, what are the use cases for applications, how would they\n> > consume this value ?\n> \n> The immediate use would be for the HQ camera with manual focus lenses.\n> Relying on observing the viewfinder image is generally not good enough\n> to determine the best focus position. Instead, users can observe the\n> FoM to get a numeric measure of the best focus position when manually\n> focussing. Of course, this is only really applicable if the scene is\n> static and the camera is sat at a fixed position.\n\nDo we need to display the focus value in qcam then ? :-)\n\n> > Also, do you see any issue with the FocusFoM control as defined here\n> > when used with sensors that produce PDAF data ? I have no particular\n> > concern, but you have more experience than I do in this area.\n> \n> Figure of Merit is a generic enough term that I think could encompass\n> PDAF metrics as well. Again, this is just a measure of \"how good is\n> the focus on this scene\", and it is up to the IPAs to determine how to\n> provide a numeric value to that.\n> \n> > Another question, I assume that in most (all ?) case the focus\n> > statistics are a measure of sharpness, and that two images with the same\n> > focus quality but with a very different level of details in the scene\n> > would have different focus FoM values, right ? I'd be tempted to name\n> > the control to refer to sharpness instead of focus then, but we already\n> > have a sharpness control that is completely unrelated. We can of course\n> > always change names later, but I'd like your opinion here. Also, is my\n> > understanding applicable to PDAF too, or would there be differences\n> > there ?\n> \n> Two scenes with the same focus quality but with a very different level\n> of details would indeed produce different FoM values. I did dither\n> about calling it a sharpness or contrast measure, but that might be\n> confusing as we have sharpness and contrast controls to set for the\n> image. And as you pointed out, PDAF won't necessarily provide a\n> contrast/sharpness value. Happy to change the control name if we can\n> agree on something suitable :)\n\nFor what it's worth, Android calls this sharpness, and uses edge\nenhancement for what we call sharpness. It doesn't seem bad, but we\ndon't have to follow that naming scheme.\n\nAs I just told David in another e-mail, I'm sure there will be a large\ncleanup of controls before we stabilize the API, so we'll have an\noccasion to rename controls to make everything consistent. Of course,\nalready picking a name that we think will stay is best.\n\n> > > I am ok to extend this to an array to provide all 12 values, but if we\n> > > do that, arguably we should add another control like FocusFomRegions\n> > > that list the region coordinates for each FoM. This will allow the\n> > > control to be more generic for other HW vendors.\n> >\n> > Yes, I agree that would make sense.\n> \n> If the consensus is to provide all 12 regions I can easily make the\n> changes, and include the new FocusFomRegions control.\n\nI'm fine going for a single value for now, we can reconsider this later\nin the context of providing full statistics to the application, with\nhelpers to interpret them.\n\n> > > > This leads to the next question of whether we should then report all\n> > > > statistics in a buffer instead, and possibly even standardize the\n> > > > statistics format. That could open pandora's box though, it's a rabbit\n> > > > hole that could end up being pretty deep. I tried to look around for\n> > > > other implementations, and there's support for histogram and focus\n> > > > (sharpness) data in the Android camera HAL API, but marked as\n> > > > \"future\". It seems to have been work in progress pretty much from the\n> > > > start, without much interest in finalizing it. Looking at OpenKCam could\n> > > > be interesting too, but there's no public spec I could find. Are you\n> > > > aware of any other attempt at standardizing something in this area ?\n> > >\n> > > I recall we talked about standard formats for statistics a year (or\n> > > so) ago and got very nervous about doing such a thing :) I was\n> > > briefly involved in the OpenKCam effort, and we did talk about exactly\n> > > this. If I recall, the consensus between hardware vendors was the\n> > > same - we might spend too much effort (and add lots of complexity) to\n> > > define a generic statistics format, which might never get used by\n> > > users, and vendors never implement.\n> >\n> > I share the same concern. It may quickly become too complicated to\n> > really be useful.\n> >\n> > > Part of me feels that is still the case with libcamera. Vendors like\n> > > Raspberry Pi do have the statistics buffer documented. In the rare\n> > > case a user does want statistics to do some magic stuff in the\n> > > application, they can request the stats buffer (when the code is\n> > > available), and resign to the fact that how they handle the statistics\n> > > data will not be portable with other vendors.\n> >\n> > I think this makes sense. If we decide to standardize statistics, it\n> > would probably be better implemented as a library that parses vendor\n> > statistics formats, instead of implemented in the IPA.\n> >\n> > I however don't rule out, for some statistics, the possibility of having\n> > the IPA reporting metadata about the statistics format, in order to ease\n> > usage in applications, or even to allow coding sharing of algorithms\n> > between different vendors. That's a topic for later though.\n> >\n> > > > > ...","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 01129BFFE2\n\tfor ;\n\tFri, 3 Jul 2020 10:39:29 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 896CE60C53;\n\tFri, 3 Jul 2020 12:39:29 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 24187603AE\n\tfor ;\n\tFri, 3 Jul 2020 12:39:28 +0200 (CEST)","from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi\n\t[81.175.216.236])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A09B351B;\n\tFri, 3 Jul 2020 12:39:27 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"E0+v0sSv\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1593772767;\n\tbh=Tmf7vodr4nNy6gzwYX3vUXUj5+t78DV52ioudIPGdbM=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=E0+v0sSv6a3nVj+zz9VaLePIqbSmPxAYZa67otNZ/EH7qCA5f4+92rXDOL4g4mqK6\n\tf0ZBNIdK6a0O15J7nbdP3K3FA6+hdOZqDAls0vgMDWXboKTGrRg5Znfa6XVu550rqd\n\tpZuASLQMYeRpyMp1hP+xyBJu2I56P3dCW/VDDXKI=","Date":"Fri, 3 Jul 2020 13:39:23 +0300","From":"Laurent Pinchart ","To":"Naushir Patuck ","Message-ID":"<20200703103923.GC5963@pendragon.ideasonboard.com>","References":"<20200626102531.1187650-1-naush@raspberrypi.com>\n\t<20200626102531.1187650-2-naush@raspberrypi.com>\n\t<20200629001544.GH6954@pendragon.ideasonboard.com>\n\t\n\t<20200703012106.GM12562@pendragon.ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH 1/4] libcamera: controls: Add focus\n\tFigure of Merit (FoM) control","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]