{"id":15515,"url":"https://patchwork.libcamera.org/api/1.1/patches/15515/?format=json","web_url":"https://patchwork.libcamera.org/patch/15515/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220323133622.61593-2-jeanmichel.hautbois@ideasonboard.com>","date":"2022-03-23T13:36:20","name":"[libcamera-devel,RFC,1/3] ipa: raspberrypi: Introduce an autofocus algorithm","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"91ba272108467c6a56ec99369cbc434f1e3e798a","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/1.1/people/75/?format=json","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/15515/mbox/","series":[{"id":2988,"url":"https://patchwork.libcamera.org/api/1.1/series/2988/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2988","date":"2022-03-23T13:36:19","name":"RPi: Introduce AF algorithm","version":1,"mbox":"https://patchwork.libcamera.org/series/2988/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/15515/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/15515/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8E3B9C0F1B\n\tfor ;\n\tWed, 23 Mar 2022 13:36:30 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6A01C604DC;\n\tWed, 23 Mar 2022 14:36:29 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E784F604C6\n\tfor ;\n\tWed, 23 Mar 2022 14:36:26 +0100 (CET)","from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:2e05:b10:1542:d51b])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 84F56130D;\n\tWed, 23 Mar 2022 14:36:26 +0100 (CET)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1648042589;\n\tbh=hAYcoqx5KqhNup/OmO2+ilIeuLssAinCiaas26Uw+ww=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=p4sfgS2hW9rv2OHN4un4X3jTgm8PF8l/l4izuOxvIeSEdhD79UQGvxeq9DLZ1jCJG\n\tX6Bb+m+OSYJRYTom6CYm7saoInGkD3yqVBX0TjOlGSq5lcZ+rtMnPlZjPZ5C0x0d3U\n\tHZ0b5ckl083WujR+MFQ4ftAwPI5lwQcirs3rRTkQ4mUl3YUUNnl0HJdeIBR9/EjTYD\n\tjyrOSwKWbOAhVLZJKJ/H5I61AIch5PMkL2F3N9pDluCWaR2LivOFboj7egv6Gsmu3J\n\t1sos7nMMmT6VLe0JFi3UnhK45yqZlaGS0FViuMKg3hq//phvKQYRC3HvMgivoR1bhU\n\teTKivHOCsXwvQ==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1648042586;\n\tbh=hAYcoqx5KqhNup/OmO2+ilIeuLssAinCiaas26Uw+ww=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=L1JIrIKJJXYaEEsxylwawozgzpSbJlwUM+AamG3X3Rc40eRDhCJ5gzWW7imjl+Mqf\n\tFGluAjjc+nGqBUwq4EmqJK0/AAH983uk/Bq5VV0Ut/1THxaGEsumKUeYi8JvBa4eFs\n\t6ZSgj5YhSnZH8LvgZMtyxxkK6iX/Czigi/HdJ+4M="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"L1JIrIKJ\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Wed, 23 Mar 2022 14:36:20 +0100","Message-Id":"<20220323133622.61593-2-jeanmichel.hautbois@ideasonboard.com>","X-Mailer":"git-send-email 2.32.0","In-Reply-To":"<20220323133622.61593-1-jeanmichel.hautbois@ideasonboard.com>","References":"<20220323133622.61593-1-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [RFC PATCH 1/3] ipa: raspberrypi: Introduce an\n\tautofocus algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jean-Michel Hautbois via libcamera-devel\n\t","Reply-To":"Jean-Michel Hautbois ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Now that the ancillary links are plumbed and we can set the lens\nposition, implement a contrast-based algorithm for RPi. This algorithm\nis adapted from the one proposed for the IPU3 IPA.\n\nIt is currently taking all the regions and tries to make the focus on\nthe global scene in a first attempt.\n\nSigned-off-by: Jean-Michel Hautbois \n---\n .../raspberrypi/controller/af_algorithm.hpp | 20 ++\n src/ipa/raspberrypi/controller/af_status.h | 31 +++\n src/ipa/raspberrypi/controller/focus_status.h | 3 +\n src/ipa/raspberrypi/controller/iob/af.cpp | 231 ++++++++++++++++++\n src/ipa/raspberrypi/controller/iob/af.h | 55 +++++\n src/ipa/raspberrypi/meson.build | 1 +\n 6 files changed, 341 insertions(+)\n create mode 100644 src/ipa/raspberrypi/controller/af_algorithm.hpp\n create mode 100644 src/ipa/raspberrypi/controller/af_status.h\n create mode 100644 src/ipa/raspberrypi/controller/iob/af.cpp\n create mode 100644 src/ipa/raspberrypi/controller/iob/af.h","diff":"diff --git a/src/ipa/raspberrypi/controller/af_algorithm.hpp b/src/ipa/raspberrypi/controller/af_algorithm.hpp\nnew file mode 100644\nindex 00000000..553a37e1\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/af_algorithm.hpp\n@@ -0,0 +1,20 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2019, Raspberry Pi (Trading) Limited\n+ *\n+ * af_algorithm.hpp - autofocus control algorithm interface\n+ */\n+#pragma once\n+\n+#include \"algorithm.hpp\"\n+\n+namespace RPiController {\n+\n+class AfAlgorithm : public Algorithm\n+{\n+public:\n+\tAfAlgorithm(Controller *controller) : Algorithm(controller) {}\n+\t// An af algorithm must provide the following:\n+};\n+\n+} // namespace RPiController\ndiff --git a/src/ipa/raspberrypi/controller/af_status.h b/src/ipa/raspberrypi/controller/af_status.h\nnew file mode 100644\nindex 00000000..835e1e2f\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/af_status.h\n@@ -0,0 +1,31 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2020, Raspberry Pi (Trading) Limited\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af_status.h - autofocus measurement status\n+ */\n+#pragma once\n+\n+#include \n+\n+/*\n+ * The focus algorithm should post the following structure into the image's\n+ * \"af.status\" metadata.\n+ */\n+\n+#ifdef __cplusplus\n+extern \"C\" {\n+#endif\n+\n+struct AfStatus {\n+\tunsigned int num;\n+\tuint32_t focus_measures[FOCUS_REGIONS];\n+\tbool stable;\n+\tuint32_t focus;\n+\tdouble maxVariance;\n+};\n+\n+#ifdef __cplusplus\n+}\n+#endif\ndiff --git a/src/ipa/raspberrypi/controller/focus_status.h b/src/ipa/raspberrypi/controller/focus_status.h\nindex ace2fe2c..8122df4b 100644\n--- a/src/ipa/raspberrypi/controller/focus_status.h\n+++ b/src/ipa/raspberrypi/controller/focus_status.h\n@@ -19,6 +19,9 @@ extern \"C\" {\n struct FocusStatus {\n \tunsigned int num;\n \tuint32_t focus_measures[FOCUS_REGIONS];\n+\tbool stable;\n+\tuint32_t focus;\n+\tdouble maxVariance;\n };\n \n #ifdef __cplusplus\ndiff --git a/src/ipa/raspberrypi/controller/iob/af.cpp b/src/ipa/raspberrypi/controller/iob/af.cpp\nnew file mode 100644\nindex 00000000..dc5258ba\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/iob/af.cpp\n@@ -0,0 +1,231 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Red Hat\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af.cpp - automatic contrast-based focus algorithm\n+ */\n+#include \n+\n+#include \n+\n+#include \n+\n+#include \"af.h\"\n+\n+using namespace RPiController;\n+using namespace libcamera;\n+\n+LOG_DEFINE_CATEGORY(IoBAf)\n+\n+#define NAME \"iob.af\"\n+\n+/*\n+ * Maximum focus steps of the VCM control\n+ * \\todo should be obtained from the VCM driver\n+ */\n+static constexpr uint32_t kMaxFocusSteps = 1023;\n+\n+/* Minimum focus step for searching appropriate focus */\n+static constexpr uint32_t kCoarseSearchStep = 30;\n+static constexpr uint32_t kFineSearchStep = 1;\n+\n+/* Max ratio of variance change, 0.0 < kMaxChange < 1.0 */\n+static constexpr double kMaxChange = 0.5;\n+\n+/* The numbers of frame to be ignored, before performing focus scan. */\n+static constexpr uint32_t kIgnoreFrame = 10;\n+\n+/* Fine scan range 0 < kFineRange < 1 */\n+static constexpr double kFineRange = 0.05;\n+\n+Af::Af(Controller *controller)\n+\t: AfAlgorithm(controller), focus_(0), bestFocus_(0), ignoreCounter_(0),\n+\t currentVariance_(0.0), previousVariance_(0.0), maxStep_(0),\n+\t coarseCompleted_(false), fineCompleted_(false)\n+{\n+}\n+\n+char const *Af::Name() const\n+{\n+\treturn NAME;\n+}\n+\n+void Af::Initialise()\n+{\n+\tstatus_.focus = 0.0;\n+\tstatus_.maxVariance = 0.0;\n+\tstatus_.stable = false;\n+}\n+\n+void Af::Prepare(Metadata *image_metadata)\n+{\n+\timage_metadata->Set(\"af.status\", status_);\n+}\n+\n+double Af::estimateVariance()\n+{\n+\tunsigned int i;\n+\tdouble mean;\n+\tuint64_t total = 0;\n+\tdouble var_sum = 0.0;\n+\n+\t/* Compute the mean value. */\n+\tfor (i = 0; i < FOCUS_REGIONS; i++)\n+\t\ttotal += status_.focus_measures[i];\n+\tmean = total / FOCUS_REGIONS;\n+\n+\t/* Compute the sum of the squared variance. */\n+\tfor (i = 0; i < FOCUS_REGIONS; i++)\n+\t\tvar_sum += std::pow(status_.focus_measures[i] - mean, 2);\n+\n+\treturn var_sum / FOCUS_REGIONS;\n+}\n+\n+bool Af::afNeedIgnoreFrame()\n+{\n+\tif (ignoreCounter_ == 0)\n+\t\treturn false;\n+\telse\n+\t\tignoreCounter_--;\n+\treturn true;\n+}\n+\n+void Af::afCoarseScan()\n+{\n+\tif (coarseCompleted_)\n+\t\treturn;\n+\n+\tif (afNeedIgnoreFrame())\n+\t\treturn;\n+\n+\tif (afScan(kCoarseSearchStep)) {\n+\t\tcoarseCompleted_ = true;\n+\t\tstatus_.maxVariance = 0;\n+\t\tfocus_ = status_.focus - (status_.focus * kFineRange);\n+\t\tstatus_.focus = focus_;\n+\t\tpreviousVariance_ = 0;\n+\t\tmaxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)),\n+\t\t\t\t 0U, kMaxFocusSteps);\n+\t}\n+}\n+\n+void Af::afFineScan()\n+{\n+\tif (!coarseCompleted_)\n+\t\treturn;\n+\n+\tif (afNeedIgnoreFrame())\n+\t\treturn;\n+\n+\tif (afScan(kFineSearchStep)) {\n+\t\tstatus_.stable = true;\n+\t\tfineCompleted_ = true;\n+\t}\n+}\n+\n+bool Af::afScan(uint32_t minSteps)\n+{\n+\tif (focus_ > maxStep_) {\n+\t\t/* If the max step is reached, move lens to the position. */\n+\t\tstatus_.focus = bestFocus_;\n+\t\treturn true;\n+\t} else {\n+\t\t/*\n+\t\t * Find the maximum of the variance by estimating its\n+\t\t * derivative. If the direction changes, it means we have passed\n+\t\t * a maximum one step before.\n+\t\t */\n+\t\tif ((currentVariance_ - status_.maxVariance) >=\n+\t\t -(status_.maxVariance * 0.1)) {\n+\t\t\t/*\n+\t\t\t * Positive and zero derivative:\n+\t\t\t * The variance is still increasing. The focus could be\n+\t\t\t * increased for the next comparison. Also, the max\n+\t\t\t * variance and previous focus value are updated.\n+\t\t\t */\n+\t\t\tbestFocus_ = focus_;\n+\t\t\tfocus_ += minSteps;\n+\t\t\tstatus_.focus = focus_;\n+\t\t\tstatus_.maxVariance = currentVariance_;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * Negative derivative:\n+\t\t\t * The variance starts to decrease which means the maximum\n+\t\t\t * variance is found. Set focus step to previous good one\n+\t\t\t * then return immediately.\n+\t\t\t */\n+\t\t\tstatus_.focus = bestFocus_;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\tpreviousVariance_ = currentVariance_;\n+\tLOG(IoBAf, Debug) << \" Previous step is \"\n+\t\t\t << bestFocus_\n+\t\t\t << \" Current step is \"\n+\t\t\t << focus_;\n+\treturn false;\n+}\n+\n+void Af::afReset()\n+{\n+\tif (afNeedIgnoreFrame())\n+\t\treturn;\n+\n+\tstatus_.maxVariance = 0;\n+\tstatus_.focus = 0;\n+\tfocus_ = 0;\n+\tstatus_.stable = false;\n+\tignoreCounter_ = kIgnoreFrame;\n+\tpreviousVariance_ = 0.0;\n+\tcoarseCompleted_ = false;\n+\tfineCompleted_ = false;\n+\tmaxStep_ = kMaxFocusSteps;\n+}\n+\n+bool Af::afIsOutOfFocus()\n+{\n+\tconst uint32_t diff_var = std::abs(currentVariance_ -\n+\t\t\t\t\t status_.maxVariance);\n+\tconst double var_ratio = diff_var / status_.maxVariance;\n+\tLOG(IoBAf, Debug) << \"Variance change rate: \"\n+\t\t\t << var_ratio\n+\t\t\t << \" Current VCM step: \"\n+\t\t\t << status_.focus;\n+\tif (var_ratio > kMaxChange)\n+\t\treturn true;\n+\telse\n+\t\treturn false;\n+}\n+\n+void Af::Process(StatisticsPtr &stats, Metadata *image_metadata)\n+{\n+\tunsigned int i;\n+\timage_metadata->Get(\"af.status\", status_);\n+\n+\t/* Use the second filter results only, and cache those. */\n+\tfor (i = 0; i < FOCUS_REGIONS; i++)\n+\t\tstatus_.focus_measures[i] = stats->focus_stats[i].contrast_val[1][1]\n+\t\t\t\t\t / stats->focus_stats[i].contrast_val_num[1][1];\n+\tstatus_.num = i;\n+\n+\tcurrentVariance_ = estimateVariance();\n+\n+\tif (!status_.stable) {\n+\t\tafCoarseScan();\n+\t\tafFineScan();\n+\t} else {\n+\t\tif (afIsOutOfFocus())\n+\t\t\tafReset();\n+\t\telse\n+\t\t\tignoreCounter_ = kIgnoreFrame;\n+\t}\n+}\n+\n+/* Register algorithm with the system. */\n+static Algorithm *Create(Controller *controller)\n+{\n+\treturn new Af(controller);\n+}\n+static RegisterAlgorithm reg(NAME, &Create);\ndiff --git a/src/ipa/raspberrypi/controller/iob/af.h b/src/ipa/raspberrypi/controller/iob/af.h\nnew file mode 100644\nindex 00000000..45c9711f\n--- /dev/null\n+++ b/src/ipa/raspberrypi/controller/iob/af.h\n@@ -0,0 +1,55 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Red Hat\n+ * Copyright (C) 2022, Ideas On Board\n+ *\n+ * af.h - automatic contrast-based focus algorithm\n+ */\n+#pragma once\n+\n+#include \"../af_algorithm.hpp\"\n+#include \"../af_status.h\"\n+#include \"../metadata.hpp\"\n+\n+namespace RPiController {\n+\n+class Af : public AfAlgorithm\n+{\n+public:\n+\tAf(Controller *controller);\n+\tchar const *Name() const override;\n+\tvoid Initialise() override;\n+\tvoid Prepare(Metadata *image_metadata) override;\n+\tvoid Process(StatisticsPtr &stats, Metadata *image_metadata) override;\n+private:\n+\tdouble estimateVariance();\n+\tbool afNeedIgnoreFrame();\n+\tvoid afCoarseScan();\n+\tvoid afFineScan();\n+\tbool afScan(uint32_t minSteps);\n+\tvoid afReset();\n+\tbool afIsOutOfFocus();\n+\n+\tAfStatus status_;\n+\n+\t/* VCM step configuration. It is the current setting of the VCM step. */\n+\tuint32_t focus_;\n+\t/* The best VCM step. It is a local optimum VCM step during scanning. */\n+\tuint32_t bestFocus_;\n+\n+\t/* The frames ignored before starting measuring. */\n+\tuint32_t ignoreCounter_;\n+\n+\t/* Current AF statistic variance. */\n+\tdouble currentVariance_;\n+\t/* It is used to determine the derivative during scanning */\n+\tdouble previousVariance_;\n+\t/* The designated maximum range of focus scanning. */\n+\tuint32_t maxStep_;\n+\t/* If the coarse scan completes, it is set to true. */\n+\tbool coarseCompleted_;\n+\t/* If the fine scan completes, it is set to true. */\n+\tbool fineCompleted_;\n+};\n+\n+} /* namespace RPiController */\ndiff --git a/src/ipa/raspberrypi/meson.build b/src/ipa/raspberrypi/meson.build\nindex 32897e07..37068ecc 100644\n--- a/src/ipa/raspberrypi/meson.build\n+++ b/src/ipa/raspberrypi/meson.build\n@@ -28,6 +28,7 @@ rpi_ipa_sources = files([\n 'controller/controller.cpp',\n 'controller/histogram.cpp',\n 'controller/algorithm.cpp',\n+ 'controller/iob/af.cpp',\n 'controller/rpi/alsc.cpp',\n 'controller/rpi/awb.cpp',\n 'controller/rpi/sharpen.cpp',\n","prefixes":["libcamera-devel","RFC","1/3"]}