{"id":17930,"url":"https://patchwork.libcamera.org/api/1.1/patches/17930/?format=json","web_url":"https://patchwork.libcamera.org/patch/17930/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20221201145133.479794-2-umang.jain@ideasonboard.com>","date":"2022-12-01T14:51:29","name":"[libcamera-devel,1/5] ipa: raspberrypi: Introduce an autofocus algorithm","commit_ref":null,"pull_url":null,"state":"not-applicable","archived":false,"hash":"b9b686bd86ba85cb657a4ae1205c2527a5e2dc7f","submitter":{"id":86,"url":"https://patchwork.libcamera.org/api/1.1/people/86/?format=json","name":"Umang Jain","email":"umang.jain@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/17930/mbox/","series":[{"id":3653,"url":"https://patchwork.libcamera.org/api/1.1/series/3653/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3653","date":"2022-12-01T14:51:28","name":"ipa: rapsberrypi: Introduce auto-focus (auto-mode)","version":1,"mbox":"https://patchwork.libcamera.org/series/3653/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/17930/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/17930/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8344EBE08B\n\tfor ;\n\tThu, 1 Dec 2022 14:52:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 33BE963342;\n\tThu, 1 Dec 2022 15:52:43 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E44EA63335\n\tfor ;\n\tThu, 1 Dec 2022 15:52:41 +0100 (CET)","from umang.jainideasonboard.com (unknown [210.186.188.117])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id B2A252D9;\n\tThu, 1 Dec 2022 15:52:40 +0100 (CET)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1669906363;\n\tbh=9QHTIxtxNjyrescww7dX9bQpgbhBsBAAA0Admp9949M=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=kxKrFHrNEosLxakuVHjRMH2UIpZc8oZOwbYGIB/KAuUXvrXUYUDp6TdXg2Uz9aDpG\n\tDJ2pzH41LW/cIEl/whfQdeS7DdB2lB577IkSkMug5AhLRzMUWTMpRjfz0E/l93XnIj\n\tkbycrJqDpPepuYrG1oSQLXSn91YlAh+cWEQc1sAoW3cjsvq/XHyPCkYX9m9/sHaOiA\n\tMKExYjSmhDENYBDL+bJuSTxwtc+PNow3hh43kuCD9J3bkulGYECQnFVxR54icaFXbo\n\tcI4gmtupNcnGy4YHJkCNsKLe7TX0AbcR9bi+roIB3c/hF49nQPzkv/3wnVqi9Pe/jw\n\tliZiPhLQSCE1Q==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1669906361;\n\tbh=9QHTIxtxNjyrescww7dX9bQpgbhBsBAAA0Admp9949M=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=XF1rQDeoX3d3rWMtINiaFWWm6vBnQKvwElBGMAMzr5l1eyGv4Kf6Lt4oF5rotNFoY\n\txKZZVHXCeTsA1g1P6axSlwm/Co5FPz873kzp9YoazVtea2x8HTCEdUsQbfI3M361iU\n\tpmedrJicpvQ9NcgFsCeaSySbSDlF/GMzFqEoAqo0="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"XF1rQDeo\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 1 Dec 2022 22:51:29 +0800","Message-Id":"<20221201145133.479794-2-umang.jain@ideasonboard.com>","X-Mailer":"git-send-email 2.38.1","In-Reply-To":"<20221201145133.479794-1-umang.jain@ideasonboard.com>","References":"<20221201145133.479794-1-umang.jain@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 1/5] ipa: raspberrypi: Introduce an\n\tautofocus algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Umang Jain via libcamera-devel ","Reply-To":"Umang Jain ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"From: Jean-Michel Hautbois \n\nNow that the ancillary links are plumbed and we can set the lens\nposition, implement a contrast-based algorithm for RPi. This algorithm\nis adapted from the one proposed for the IPU3 IPA.\n\nIt is currently taking all the regions and tries to make the focus on\nthe global scene in a first attempt.\n\nSigned-off-by: Jean-Michel Hautbois \n---\n src/ipa/raspberrypi/controller/af_algorithm.h | 33 +++\n src/ipa/raspberrypi/controller/af_status.h | 21 ++\n src/ipa/raspberrypi/controller/iob/af.cpp | 212 ++++++++++++++++++\n src/ipa/raspberrypi/controller/iob/af.h | 62 +++++\n src/ipa/raspberrypi/meson.build | 1 +\n 5 files changed, 329 insertions(+)\n create mode 100644 src/ipa/raspberrypi/controller/af_algorithm.h\n create mode 100644 src/ipa/raspberrypi/controller/af_status.h\n create mode 100644 src/ipa/raspberrypi/controller/iob/af.cpp\n create mode 100644 src/ipa/raspberrypi/controller/iob/af.h","diff":"diff --git a/src/ipa/raspberrypi/controller/af_algorithm.h b/src/ipa/raspberrypi/controller/af_algorithm.h\nnew file mode 100644\nindex 00000000..e88683a0\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/af_algorithm.h\n@@ -0,0 +1,33 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2022, Raspberry Pi (Trading) Limited\n+ *\n+ * af_algorithm.h - autofocus control algorithm interface\n+ */\n+#pragma once\n+\n+#include \n+\n+#include \"algorithm.h\"\n+\n+namespace RPiController {\n+\n+class AfAlgorithm : public Algorithm\n+{\n+public:\n+\tAfAlgorithm(Controller *controller) : Algorithm(controller) {}\n+\t/* An af algorithm must provide the following: */\n+\tvirtual void setMode(const uint32_t &mode) = 0;\n+\t/* start a cycle (in auto mode) */\n+\tvirtual void trigger() = 0;\n+\t/* cancel a cycle (in auto mode) */\n+\tvirtual void cancel() = 0;\n+\t/* set AF windows */\n+\tvirtual void setWindows(const libcamera::Rectangle &afWindows) = 0;\n+\t/* set AF range */\n+\tvirtual void setRange(const uint32_t &range) = 0;\n+\t/* set AF speed */\n+\tvirtual void setSpeed(const uint32_t &speed) = 0;\n+};\n+\n+} /* namespace RPiController */\ndiff --git a/src/ipa/raspberrypi/controller/af_status.h b/src/ipa/raspberrypi/controller/af_status.h\nnew file mode 100644\nindex 00000000..f8cb1301\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/af_status.h\n@@ -0,0 +1,21 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2022, Raspberry Pi (Trading) Limited\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af_status.h - autofocus measurement status\n+ */\n+#pragma once\n+\n+#include \n+\n+/*\n+ * The focus algorithm should post the following structure into the image's\n+ * \"af.status\" metadata.\n+ */\n+\n+struct AfStatus {\n+\tuint32_t lensPosition;\n+\tuint32_t state;\n+\tlibcamera::Rectangle windows;\n+};\ndiff --git a/src/ipa/raspberrypi/controller/iob/af.cpp b/src/ipa/raspberrypi/controller/iob/af.cpp\nnew file mode 100644\nindex 00000000..0100bbaf\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/iob/af.cpp\n@@ -0,0 +1,212 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Red Hat\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af.cpp - automatic contrast-based focus algorithm\n+ */\n+\n+#include \"af.h\"\n+\n+#include \n+#include \n+\n+#include \n+\n+using namespace RPiController;\n+using namespace libcamera;\n+\n+LOG_DEFINE_CATEGORY(IoBAf)\n+\n+#define NAME \"iob.af\"\n+\n+/*\n+ * Maximum focus steps of the VCM control\n+ * \\todo should be obtained from the VCM driver\n+ */\n+static constexpr uint32_t kMaxFocusSteps = 1023;\n+\n+/* Minimum focus step for searching appropriate focus */\n+static constexpr uint32_t kCoarseSearchStep = 30;\n+static constexpr uint32_t kFineSearchStep = 1;\n+\n+/* Max ratio of variance change, 0.0 < kMaxChange < 1.0 */\n+static constexpr double kMaxChange = 0.5;\n+\n+/* Fine scan range 0 < kFineRange < 1 */\n+static constexpr double kFineRange = 0.05;\n+\n+Af::Af(Controller *controller)\n+\t: AfAlgorithm(controller), focus_(0), bestFocus_(0),\n+\t currentContrast_(0.0), previousContrast_(0.0), maxContrast_(0.0),\n+\t maxStep_(0), coarseCompleted_(false), fineCompleted_(false),\n+\t mode_(0)\n+{\n+}\n+\n+char const *Af::name() const\n+{\n+\treturn NAME;\n+}\n+\n+void Af::setMode([[maybe_unused]] const uint32_t &mode)\n+{\n+\tmode_ = mode;\n+}\n+\n+void Af::trigger()\n+{\n+}\n+\n+void Af::cancel()\n+{\n+}\n+\n+void Af::setWindows([[maybe_unused]] const libcamera::Rectangle &afWindows)\n+{\n+}\n+\n+void Af::setRange([[maybe_unused]] const uint32_t &range)\n+{\n+}\n+\n+void Af::setSpeed([[maybe_unused]] const uint32_t &speed)\n+{\n+}\n+\n+void Af::initialise()\n+{\n+\tstatus_.lensPosition = 0.0;\n+\tmaxContrast_ = 0.0;\n+\tstatus_.state = 1;\n+}\n+\n+void Af::prepare(Metadata *imageMetadata)\n+{\n+\timageMetadata->set(\"af.status\", status_);\n+}\n+\n+void Af::afCoarseScan()\n+{\n+\tif (coarseCompleted_)\n+\t\treturn;\n+\n+\tif (afScan(kCoarseSearchStep)) {\n+\t\tcoarseCompleted_ = true;\n+\t\tmaxContrast_ = 0;\n+\t\tfocus_ = status_.lensPosition - (status_.lensPosition * kFineRange);\n+\t\tstatus_.lensPosition = focus_;\n+\t\tpreviousContrast_ = 0;\n+\t\tmaxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)),\n+\t\t\t\t 0U, kMaxFocusSteps);\n+\t}\n+}\n+\n+void Af::afFineScan()\n+{\n+\tif (!coarseCompleted_)\n+\t\treturn;\n+\n+\tif (afScan(kFineSearchStep)) {\n+\t\tLOG(IoBAf, Debug) << \"AF found the best focus position !\";\n+\t\tstatus_.state = 2;\n+\t\tfineCompleted_ = true;\n+\t}\n+}\n+\n+bool Af::afScan(uint32_t minSteps)\n+{\n+\tif (focus_ > maxStep_) {\n+\t\t/* If the max step is reached, move lens to the position. */\n+\t\tstatus_.lensPosition = bestFocus_;\n+\t\treturn true;\n+\t} else {\n+\t\t/*\n+\t\t * Find the maximum of the variance by estimating its\n+\t\t * derivative. If the direction changes, it means we have passed\n+\t\t * a maximum one step before.\n+\t\t */\n+\t\tif ((currentContrast_ - maxContrast_) >= -(maxContrast_ * 0.1)) {\n+\t\t\t/*\n+\t\t\t * Positive and zero derivative:\n+\t\t\t * The variance is still increasing. The focus could be\n+\t\t\t * increased for the next comparison. Also, the max\n+\t\t\t * variance and previous focus value are updated.\n+\t\t\t */\n+\t\t\tbestFocus_ = focus_;\n+\t\t\tfocus_ += minSteps;\n+\t\t\tmaxContrast_ = currentContrast_;\n+\t\t\tstatus_.lensPosition = focus_;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * Negative derivative:\n+\t\t\t * The variance starts to decrease which means the maximum\n+\t\t\t * variance is found. Set focus step to previous good one\n+\t\t\t * then return immediately.\n+\t\t\t */\n+\t\t\tstatus_.lensPosition = bestFocus_;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\tpreviousContrast_ = currentContrast_;\n+\tLOG(IoBAf, Debug) << \" Previous step is \"\n+\t\t\t << bestFocus_\n+\t\t\t << \" Current step is \"\n+\t\t\t << focus_;\n+\treturn false;\n+}\n+\n+void Af::afReset()\n+{\n+\tLOG(IoBAf, Debug) << \"Reset AF parameters\";\n+\tstatus_.lensPosition = 0;\n+\tfocus_ = 0;\n+\tstatus_.state = 0;\n+\tpreviousContrast_ = 0.0;\n+\tcoarseCompleted_ = false;\n+\tfineCompleted_ = false;\n+\tmaxStep_ = kMaxFocusSteps;\n+\tmaxContrast_ = 0.0;\n+}\n+\n+bool Af::afIsOutOfFocus()\n+{\n+\tconst uint32_t diff_var = std::abs(currentContrast_ -\n+\t\t\t\t\t maxContrast_);\n+\tconst double var_ratio = diff_var / maxContrast_;\n+\tLOG(IoBAf, Debug) << \"Variance change rate: \"\n+\t\t\t << var_ratio\n+\t\t\t << \" Current VCM step: \"\n+\t\t\t << status_.lensPosition;\n+\tif (var_ratio > kMaxChange)\n+\t\treturn true;\n+\telse\n+\t\treturn false;\n+}\n+\n+void Af::process(StatisticsPtr &stats, [[maybe_unused]] Metadata *imageMetadata)\n+{\n+\tunsigned int i;\n+\tcurrentContrast_ = 0.0;\n+\n+\t/* Use the second filter results only, and cache those. */\n+\tfor (i = 0; i < FOCUS_REGIONS; i++)\n+\t\tcurrentContrast_ += stats->focus_stats[i].contrast_val[1][1]\n+\t\t\t\t / stats->focus_stats[i].contrast_val_num[1][1];\n+\n+\tif (status_.state != 2) {\n+\t\tafCoarseScan();\n+\t\tafFineScan();\n+\t} else {\n+\t\tif (afIsOutOfFocus())\n+\t\t\tafReset();\n+\t}\n+}\n+\n+/* Register algorithm with the system. */\n+static Algorithm *Create(Controller *controller)\n+{\n+\treturn new Af(controller);\n+}\n+static RegisterAlgorithm reg(NAME, &Create);\ndiff --git a/src/ipa/raspberrypi/controller/iob/af.h b/src/ipa/raspberrypi/controller/iob/af.h\nnew file mode 100644\nindex 00000000..c7ed6109\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/iob/af.h\n@@ -0,0 +1,62 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Red Hat\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af.h - automatic contrast-based focus algorithm\n+ */\n+#pragma once\n+\n+#include \n+\n+#include \"../af_algorithm.h\"\n+#include \"../af_status.h\"\n+#include \"../metadata.h\"\n+\n+namespace RPiController {\n+\n+class Af : public AfAlgorithm\n+{\n+public:\n+\tAf(Controller *controller);\n+\tchar const *name() const override;\n+\tvoid initialise() override;\n+\tvoid prepare(Metadata *image_metadata) override;\n+\tvoid process(StatisticsPtr &stats, Metadata *image_metadata) override;\n+\tvoid setMode(const uint32_t &mode) override;\n+\tvoid trigger() override;\n+\tvoid cancel() override;\n+\tvoid setWindows(const libcamera::Rectangle &afWindows) override;\n+\tvoid setRange(const uint32_t &range) override;\n+\tvoid setSpeed(const uint32_t &speed) override;\n+private:\n+\tbool afNeedIgnoreFrame();\n+\tvoid afCoarseScan();\n+\tvoid afFineScan();\n+\tbool afScan(uint32_t minSteps);\n+\tvoid afReset();\n+\tbool afIsOutOfFocus();\n+\n+\tAfStatus status_;\n+\n+\t/* VCM step configuration. It is the current setting of the VCM step. */\n+\tuint32_t focus_;\n+\t/* The best VCM step. It is a local optimum VCM step during scanning. */\n+\tuint32_t bestFocus_;\n+\n+\t/* Current AF statistic contrast. */\n+\tdouble currentContrast_;\n+\t/* It is used to determine the derivative during scanning */\n+\tdouble previousContrast_;\n+\tdouble maxContrast_;\n+\t/* The designated maximum range of focus scanning. */\n+\tuint32_t maxStep_;\n+\t/* If the coarse scan completes, it is set to true. */\n+\tbool coarseCompleted_;\n+\t/* If the fine scan completes, it is set to true. */\n+\tbool fineCompleted_;\n+\n+\tuint32_t mode_;\n+};\n+\n+} /* namespace RPiController */\ndiff --git a/src/ipa/raspberrypi/meson.build b/src/ipa/raspberrypi/meson.build\nindex 517d815b..10ba9608 100644\n--- a/src/ipa/raspberrypi/meson.build\n+++ b/src/ipa/raspberrypi/meson.build\n@@ -27,6 +27,7 @@ rpi_ipa_sources = files([\n 'controller/controller.cpp',\n 'controller/histogram.cpp',\n 'controller/algorithm.cpp',\n+ 'controller/iob/af.cpp',\n 'controller/rpi/alsc.cpp',\n 'controller/rpi/awb.cpp',\n 'controller/rpi/sharpen.cpp',\n","prefixes":["libcamera-devel","1/5"]}