{"id":16210,"url":"https://patchwork.libcamera.org/api/patches/16210/?format=json","web_url":"https://patchwork.libcamera.org/patch/16210/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220613142853.98484-2-jeanmichel.hautbois@ideasonboard.com>","date":"2022-06-13T14:28:44","name":"[libcamera-devel,01/10] ipa: raspberrypi: Introduce an autofocus algorithm","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"b17e4c12f64c0007abbbb76b260b0d4f3d465573","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/?format=json","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/16210/mbox/","series":[{"id":3174,"url":"https://patchwork.libcamera.org/api/series/3174/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3174","date":"2022-06-13T14:28:43","name":"ipa: raspberrypi: Introduce an autofocus algorithm","version":1,"mbox":"https://patchwork.libcamera.org/series/3174/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/16210/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/16210/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B96F7C3273\n\tfor ;\n\tMon, 13 Jun 2022 14:29:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3C83065646;\n\tMon, 13 Jun 2022 16:29:02 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 32D8665631\n\tfor ;\n\tMon, 13 Jun 2022 16:29:00 +0200 (CEST)","from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:a1c7:c28c:7720:9b30])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id C7606440;\n\tMon, 13 Jun 2022 16:28:59 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1655130542;\n\tbh=pCUBdvfmR8EpviWT+F/VZq32WxVsPa1B4jht1VZ92xA=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=iV7M2VX4TddV3XEUJw4NjopPaxyTxIJZuuCuq/i2CI3sSEdF2srLuz4DkbLNc9VvE\n\tnzc1Jvai+e9JJJQqHoZQMu5f6+uwxD0+itxvocXly+fDIzxVRmCbDyiD6GMXZ/g1J1\n\t5yIoLM+1ENr3GImH5LTg1shnMkLoWmh9OpR2LwMi8YEN2mmzyP4NJWwRcNRexT3pD3\n\tmG9bBt+pX8fdCxRQz7AcG9/VmtzK5tZQJyGxuIkIdd9afJk8wfXGnL+5Bh9lefQPKJ\n\tMNw+GpV5lPEw2gC9J9cr2vgWJKxVgIwnhlY21ClZEbjA47OYq1CCLzc695VLnzHGNZ\n\txs+3BsVPwoPAQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1655130539;\n\tbh=pCUBdvfmR8EpviWT+F/VZq32WxVsPa1B4jht1VZ92xA=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=f1hm0GwqlXfk7cUZKM6OIVOY36kS5lbAFDu0zkmOok97uMVxVgTLUqj4pGDkKMc4O\n\tUZr8campxZSoYyIbrNctOZajy3YllVD5YqhNNVcmFbylndC6qHFpPX198oEUCXwYNH\n\t9egicTpspIuocgmrdJO8tySbKsZx9m525wsOtlno="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"f1hm0Gwq\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 13 Jun 2022 16:28:44 +0200","Message-Id":"<20220613142853.98484-2-jeanmichel.hautbois@ideasonboard.com>","X-Mailer":"git-send-email 2.34.1","In-Reply-To":"<20220613142853.98484-1-jeanmichel.hautbois@ideasonboard.com>","References":"<20220613142853.98484-1-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 01/10] ipa: raspberrypi: Introduce an\n\tautofocus algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jean-Michel Hautbois via libcamera-devel\n\t","Reply-To":"Jean-Michel Hautbois ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Now that the ancillary links are plumbed and we can set the lens\nposition, implement a contrast-based algorithm for RPi. This algorithm\nis adapted from the one proposed for the IPU3 IPA.\n\nIt is currently taking all the regions and tries to make the focus on\nthe global scene in a first attempt.\n\nSigned-off-by: Jean-Michel Hautbois \n---\n .../raspberrypi/controller/af_algorithm.hpp | 33 +++\n src/ipa/raspberrypi/controller/af_status.h | 21 ++\n src/ipa/raspberrypi/controller/iob/af.cpp | 212 ++++++++++++++++++\n src/ipa/raspberrypi/controller/iob/af.h | 62 +++++\n src/ipa/raspberrypi/meson.build | 1 +\n 5 files changed, 329 insertions(+)\n create mode 100644 src/ipa/raspberrypi/controller/af_algorithm.hpp\n create mode 100644 src/ipa/raspberrypi/controller/af_status.h\n create mode 100644 src/ipa/raspberrypi/controller/iob/af.cpp\n create mode 100644 src/ipa/raspberrypi/controller/iob/af.h","diff":"diff --git a/src/ipa/raspberrypi/controller/af_algorithm.hpp b/src/ipa/raspberrypi/controller/af_algorithm.hpp\nnew file mode 100644\nindex 00000000..1e7aaa45\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/af_algorithm.hpp\n@@ -0,0 +1,33 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2022, Raspberry Pi (Trading) Limited\n+ *\n+ * af_algorithm.hpp - autofocus control algorithm interface\n+ */\n+#pragma once\n+\n+#include \n+\n+#include \"algorithm.hpp\"\n+\n+namespace RPiController {\n+\n+class AfAlgorithm : public Algorithm\n+{\n+public:\n+\tAfAlgorithm(Controller *controller) : Algorithm(controller) {}\n+\t// An af algorithm must provide the following:\n+\tvirtual void SetMode(const uint32_t &mode) = 0;\n+\t// start a cycle (in auto mode)\n+\tvirtual void Trigger() = 0;\n+\t// cancel a cycle (in auto mode)\n+\tvirtual void Cancel() = 0;\n+\t// set AF windows\n+\tvirtual void SetWindows(const libcamera::Rectangle &afWindows) = 0;\n+\t// set AF range\n+\tvirtual void SetRange(const uint32_t &range) = 0;\n+\t// set AF speed\n+\tvirtual void setSpeed(const uint32_t &speed) = 0;\n+};\n+\n+} // namespace RPiController\ndiff --git a/src/ipa/raspberrypi/controller/af_status.h b/src/ipa/raspberrypi/controller/af_status.h\nnew file mode 100644\nindex 00000000..f8cb1301\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/af_status.h\n@@ -0,0 +1,21 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2022, Raspberry Pi (Trading) Limited\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af_status.h - autofocus measurement status\n+ */\n+#pragma once\n+\n+#include \n+\n+/*\n+ * The focus algorithm should post the following structure into the image's\n+ * \"af.status\" metadata.\n+ */\n+\n+struct AfStatus {\n+\tuint32_t lensPosition;\n+\tuint32_t state;\n+\tlibcamera::Rectangle windows;\n+};\ndiff --git a/src/ipa/raspberrypi/controller/iob/af.cpp b/src/ipa/raspberrypi/controller/iob/af.cpp\nnew file mode 100644\nindex 00000000..12d41c81\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/iob/af.cpp\n@@ -0,0 +1,212 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Red Hat\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af.cpp - automatic contrast-based focus algorithm\n+ */\n+#include \n+\n+#include \n+\n+#include \n+\n+#include \"af.h\"\n+\n+using namespace RPiController;\n+using namespace libcamera;\n+\n+LOG_DEFINE_CATEGORY(IoBAf)\n+\n+#define NAME \"iob.af\"\n+\n+/*\n+ * Maximum focus steps of the VCM control\n+ * \\todo should be obtained from the VCM driver\n+ */\n+static constexpr uint32_t kMaxFocusSteps = 1023;\n+\n+/* Minimum focus step for searching appropriate focus */\n+static constexpr uint32_t kCoarseSearchStep = 30;\n+static constexpr uint32_t kFineSearchStep = 1;\n+\n+/* Max ratio of variance change, 0.0 < kMaxChange < 1.0 */\n+static constexpr double kMaxChange = 0.5;\n+\n+/* Fine scan range 0 < kFineRange < 1 */\n+static constexpr double kFineRange = 0.05;\n+\n+Af::Af(Controller *controller)\n+\t: AfAlgorithm(controller), focus_(0), bestFocus_(0),\n+\t currentContrast_(0.0), previousContrast_(0.0), maxContrast_(0.0),\n+\t maxStep_(0), coarseCompleted_(false), fineCompleted_(false),\n+\t mode_(0)\n+{\n+}\n+\n+char const *Af::Name() const\n+{\n+\treturn NAME;\n+}\n+\n+void Af::SetMode([[maybe_unused]] const uint32_t &mode)\n+{\n+\tmode_ = mode;\n+}\n+\n+void Af::Trigger()\n+{\n+}\n+\n+void Af::Cancel()\n+{\n+}\n+\n+void Af::SetWindows([[maybe_unused]] const libcamera::Rectangle &afWindows)\n+{\n+}\n+\n+void Af::SetRange([[maybe_unused]] const uint32_t &range)\n+{\n+}\n+\n+void Af::setSpeed([[maybe_unused]] const uint32_t &speed)\n+{\n+}\n+\n+void Af::Initialise()\n+{\n+\tstatus_.lensPosition = 0.0;\n+\tmaxContrast_ = 0.0;\n+\tstatus_.state = 1;\n+}\n+\n+void Af::Prepare(Metadata *image_metadata)\n+{\n+\timage_metadata->Set(\"af.status\", status_);\n+}\n+\n+void Af::afCoarseScan()\n+{\n+\tif (coarseCompleted_)\n+\t\treturn;\n+\n+\tif (afScan(kCoarseSearchStep)) {\n+\t\tcoarseCompleted_ = true;\n+\t\tmaxContrast_ = 0;\n+\t\tfocus_ = status_.lensPosition - (status_.lensPosition * kFineRange);\n+\t\tstatus_.lensPosition = focus_;\n+\t\tpreviousContrast_ = 0;\n+\t\tmaxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)),\n+\t\t\t\t 0U, kMaxFocusSteps);\n+\t}\n+}\n+\n+void Af::afFineScan()\n+{\n+\tif (!coarseCompleted_)\n+\t\treturn;\n+\n+\tif (afScan(kFineSearchStep)) {\n+\t\tLOG(IoBAf, Debug) << \"AF found the best focus position !\";\n+\t\tstatus_.state = 2;\n+\t\tfineCompleted_ = true;\n+\t}\n+}\n+\n+bool Af::afScan(uint32_t minSteps)\n+{\n+\tif (focus_ > maxStep_) {\n+\t\t/* If the max step is reached, move lens to the position. */\n+\t\tstatus_.lensPosition = bestFocus_;\n+\t\treturn true;\n+\t} else {\n+\t\t/*\n+\t\t * Find the maximum of the variance by estimating its\n+\t\t * derivative. If the direction changes, it means we have passed\n+\t\t * a maximum one step before.\n+\t\t */\n+\t\tif ((currentContrast_ - maxContrast_) >= -(maxContrast_ * 0.1)) {\n+\t\t\t/*\n+\t\t\t * Positive and zero derivative:\n+\t\t\t * The variance is still increasing. The focus could be\n+\t\t\t * increased for the next comparison. Also, the max\n+\t\t\t * variance and previous focus value are updated.\n+\t\t\t */\n+\t\t\tbestFocus_ = focus_;\n+\t\t\tfocus_ += minSteps;\n+\t\t\tmaxContrast_ = currentContrast_;\n+\t\t\tstatus_.lensPosition = focus_;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * Negative derivative:\n+\t\t\t * The variance starts to decrease which means the maximum\n+\t\t\t * variance is found. Set focus step to previous good one\n+\t\t\t * then return immediately.\n+\t\t\t */\n+\t\t\tstatus_.lensPosition = bestFocus_;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\tpreviousContrast_ = currentContrast_;\n+\tLOG(IoBAf, Debug) << \" Previous step is \"\n+\t\t\t << bestFocus_\n+\t\t\t << \" Current step is \"\n+\t\t\t << focus_;\n+\treturn false;\n+}\n+\n+void Af::afReset()\n+{\n+\tLOG(IoBAf, Debug) << \"Reset AF parameters\";\n+\tstatus_.lensPosition = 0;\n+\tfocus_ = 0;\n+\tstatus_.state = 0;\n+\tpreviousContrast_ = 0.0;\n+\tcoarseCompleted_ = false;\n+\tfineCompleted_ = false;\n+\tmaxStep_ = kMaxFocusSteps;\n+\tmaxContrast_ = 0.0;\n+}\n+\n+bool Af::afIsOutOfFocus()\n+{\n+\tconst uint32_t diff_var = std::abs(currentContrast_ -\n+\t\t\t\t\t maxContrast_);\n+\tconst double var_ratio = diff_var / maxContrast_;\n+\tLOG(IoBAf, Debug) << \"Variance change rate: \"\n+\t\t\t << var_ratio\n+\t\t\t << \" Current VCM step: \"\n+\t\t\t << status_.lensPosition;\n+\tif (var_ratio > kMaxChange)\n+\t\treturn true;\n+\telse\n+\t\treturn false;\n+}\n+\n+void Af::Process(StatisticsPtr &stats, [[maybe_unused]] Metadata *image_metadata)\n+{\n+\tunsigned int i;\n+\tcurrentContrast_ = 0.0;\n+\n+\t/* Use the second filter results only, and cache those. */\n+\tfor (i = 0; i < FOCUS_REGIONS; i++)\n+\t\tcurrentContrast_ += stats->focus_stats[i].contrast_val[1][1]\n+\t\t\t\t / stats->focus_stats[i].contrast_val_num[1][1];\n+\n+\tif (status_.state != 2) {\n+\t\tafCoarseScan();\n+\t\tafFineScan();\n+\t} else {\n+\t\tif (afIsOutOfFocus())\n+\t\t\tafReset();\n+\t}\n+}\n+\n+/* Register algorithm with the system. */\n+static Algorithm *Create(Controller *controller)\n+{\n+\treturn new Af(controller);\n+}\n+static RegisterAlgorithm reg(NAME, &Create);\ndiff --git a/src/ipa/raspberrypi/controller/iob/af.h b/src/ipa/raspberrypi/controller/iob/af.h\nnew file mode 100644\nindex 00000000..4ed50cfb\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/iob/af.h\n@@ -0,0 +1,62 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Red Hat\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af.h - automatic contrast-based focus algorithm\n+ */\n+#pragma once\n+\n+#include \n+\n+#include \"../af_algorithm.hpp\"\n+#include \"../af_status.h\"\n+#include \"../metadata.hpp\"\n+\n+namespace RPiController {\n+\n+class Af : public AfAlgorithm\n+{\n+public:\n+\tAf(Controller *controller);\n+\tchar const *Name() const override;\n+\tvoid Initialise() override;\n+\tvoid Prepare(Metadata *image_metadata) override;\n+\tvoid Process(StatisticsPtr &stats, Metadata *image_metadata) override;\n+\tvoid SetMode(const uint32_t &mode) override;\n+\tvoid Trigger() override;\n+\tvoid Cancel() override;\n+\tvoid SetWindows(const libcamera::Rectangle &afWindows) override;\n+\tvoid SetRange(const uint32_t &range) override;\n+\tvoid setSpeed(const uint32_t &speed) override;\n+private:\n+\tbool afNeedIgnoreFrame();\n+\tvoid afCoarseScan();\n+\tvoid afFineScan();\n+\tbool afScan(uint32_t minSteps);\n+\tvoid afReset();\n+\tbool afIsOutOfFocus();\n+\n+\tAfStatus status_;\n+\n+\t/* VCM step configuration. It is the current setting of the VCM step. */\n+\tuint32_t focus_;\n+\t/* The best VCM step. It is a local optimum VCM step during scanning. */\n+\tuint32_t bestFocus_;\n+\n+\t/* Current AF statistic contrast. */\n+\tdouble currentContrast_;\n+\t/* It is used to determine the derivative during scanning */\n+\tdouble previousContrast_;\n+\tdouble maxContrast_;\n+\t/* The designated maximum range of focus scanning. */\n+\tuint32_t maxStep_;\n+\t/* If the coarse scan completes, it is set to true. */\n+\tbool coarseCompleted_;\n+\t/* If the fine scan completes, it is set to true. */\n+\tbool fineCompleted_;\n+\n+\tuint32_t mode_;\n+};\n+\n+} /* namespace RPiController */\ndiff --git a/src/ipa/raspberrypi/meson.build b/src/ipa/raspberrypi/meson.build\nindex 32897e07..37068ecc 100644\n--- a/src/ipa/raspberrypi/meson.build\n+++ b/src/ipa/raspberrypi/meson.build\n@@ -28,6 +28,7 @@ rpi_ipa_sources = files([\n 'controller/controller.cpp',\n 'controller/histogram.cpp',\n 'controller/algorithm.cpp',\n+ 'controller/iob/af.cpp',\n 'controller/rpi/alsc.cpp',\n 'controller/rpi/awb.cpp',\n 'controller/rpi/sharpen.cpp',\n","prefixes":["libcamera-devel","01/10"]}