[{"id":22171,"web_url":"https://patchwork.libcamera.org/comment/22171/","msgid":"","date":"2022-02-15T06:41:12","subject":"Re: [libcamera-devel] [PATCH v8] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Kieran,\n\n\n\nOn Mon, Feb 14, 2022 at 5:51 PM Kate Hsuan wrote:\n>\n> Since VCM for surface Go 2 (dw9719) had been successfully\n> driven, this Af module can be used to control the VCM and\n> determine the focus value based on the IPU3 AF state.\n>\n> Based on the values from the IPU3 AF buffer, the variance\n> of each focus step is determined and a greedy approach is\n> used to find the maximum variance of the AF state and an\n> appropriate focus value.\n>\n> The grid configuration is implemented as a context. Also,\n> the grid parameter- AF_MIN_BLOCK_WIDTH is set to 4 (default\n> is 3) since if the default value is used, x_start\n> (x_start > 640) will be at an incorrect location of the\n> image (rightmost of the sensor).\n>\n> Signed-off-by: Kate Hsuan \n> Tested-by: Jean-Michel Hautbois \n> ---\n> Changes in v8:\n> 1. Revised and improved the comments.\n> 2. Simplified the algorithm.\n> ---\n> src/ipa/ipu3/algorithms/af.cpp | 435 ++++++++++++++++++++++++++++\n> src/ipa/ipu3/algorithms/af.h | 79 +++++\n> src/ipa/ipu3/algorithms/meson.build | 1 +\n> src/ipa/ipu3/ipa_context.cpp | 23 ++\n> src/ipa/ipu3/ipa_context.h | 10 +\n> src/ipa/ipu3/ipu3.cpp | 2 +\n> 6 files changed, 550 insertions(+)\n> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> create mode 100644 src/ipa/ipu3/algorithms/af.h\n>\n> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> new file mode 100644\n> index 00000000..17055c04\n> --- /dev/null\n> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> @@ -0,0 +1,435 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2021, Red Hat\n> + *\n> + * af.cpp - IPU3 auto focus algorithm\n> + */\n> +\n> +#include \"af.h\"\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"libipa/histogram.h\"\n> +\n> +/**\n> + * \\file af.h\n> + */\n> +\n> +/**\n> + * \\var kAfMinGridWidth\n> + * \\brief the minimum width of AF grid.\n> + * The minimum grid horizontal dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMinGridHeight\n> + * \\brief the minimum height of AF grid.\n> + * The minimum grid vertical dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMaxGridWidth\n> + * \\brief the maximum width of AF grid.\n> + * The maximum grid horizontal dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMaxGridHeight\n> + * \\brief The maximum height of AF grid.\n> + * The maximum grid vertical dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMinGridBlockWidth\n> + * \\brief The minimum block size of the width.\n> + * The minimum value of Log2 of the width of the grid cell.\n> + */\n> +\n> +/**\n> + * \\var kAfMinGridBlockHeight\n> + * \\brief The minimum block size of the height.\n> + * The minimum value of Log2 of the height of the grid cell.\n> + */\n> +\n> +/**\n> + * \\def kAfMaxGridBlockWidth\n> + * \\brief The maximum block size of the width.\n> + * The maximum value of Log2 of the width of the grid cell.\n> + */\n> +\n> +/**\n> + * \\var kAfMaxGridBlockHeight\n> + * \\brief The maximum block size of the height.\n> + * The maximum value of Log2 of the height of the grid cell.\n> + */\n> +\n> +/**\n> + * \\var kAfDefaultHeightPerSlice\n> + * \\brief The default number of blocks in vertical axis per slice.\n> + * The number of blocks in vertical axis per slice.\n> + */\n> +\n> +namespace libcamera {\n> +\n> +using namespace std::literals::chrono_literals;\n> +\n> +namespace ipa::ipu3::algorithms {\n> +\n> +LOG_DEFINE_CATEGORY(IPU3Af)\n> +\n> +/**\n> + * Maximum focus steps of the VCM control\n> + * \\todo should be obtained from the VCM driver\n> + */\n> +static constexpr uint32_t kMaxFocusSteps = 1023;\n> +\n> +/* Minimum focus step for searching appropriate focus */\n> +static constexpr uint32_t kCoarseSearchStep = 30;\n> +static constexpr uint32_t kFineSearchStep = 1;\n> +\n> +/* Max ratio of variance change, 0.0 < kMaxChange < 1.0 */\n> +static constexpr double kMaxChange = 0.5;\n> +\n> +/* The numbers of frame to be ignored, before performing focus scan. */\n> +static constexpr uint32_t kIgnoreFrame = 10;\n> +\n> +/* Fine scan range 0 < kFineRange < 1 */\n> +static constexpr double kFineRange = 0.05;\n> +\n> +/* Settings for IPU3 AF filter */\n> +static struct ipu3_uapi_af_filter_config afFilterConfigDefault = {\n> + .y1_coeff_0 = { 0, 1, 3, 7 },\n> + .y1_coeff_1 = { 11, 13, 1, 2 },\n> + .y1_coeff_2 = { 8, 19, 34, 242 },\n> + .y1_sign_vec = 0x7fdffbfe,\n> + .y2_coeff_0 = { 0, 1, 6, 6 },\n> + .y2_coeff_1 = { 13, 25, 3, 0 },\n> + .y2_coeff_2 = { 25, 3, 177, 254 },\n> + .y2_sign_vec = 0x4e53ca72,\n> + .y_calc = { 8, 8, 8, 8 },\n> + .nf = { 0, 9, 0, 9, 0 },\n> +};\n> +\n> +/**\n> + * \\class Af\n> + * \\brief An auto-focus algorithm based on IPU3 statistics\n> + * This algorithm is used to determine the position of the lens to make a\n> + * focused image. The IPU3 AF processing block computes the statistics that\n> + * are composed by two types of filtered value and stores in a AF buffer.\n> + * Typically, for a clear image, it has a relatively higher contrast than a\n> + * blurred one. Therefore, if an image with the highest contrast can be\n> + * found through the scan, the position of the len indicates to a clearest\n> + * image.\n> + */\n> +Af::Af()\n> + : focus_(0), bestFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> + coarseCompleted_(false), fineCompleted_(false)\n> +{\n> +}\n> +\n> +/**\n> + * \\copydoc libcamera::ipa::Algorithm::prepare\n> + */\n> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n> +{\n> + const struct ipu3_uapi_grid_config &grid = context.configuration.af.afGrid;\n> + params->acc_param.af.grid_cfg = grid;\n> + params->acc_param.af.filter_config = afFilterConfigDefault;\n> +\n> + /* Enable AF processing block */\n> + params->use.acc_af = 1;\n> +}\n> +\n> +/**\n> + * \\brief Configure the Af given a configInfo\n> + * \\param[in] context The shared IPA context\n> + * \\param[in] configInfo The IPA configuration data\n> + * \\return 0\n> + */\n> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> +{\n> + struct ipu3_uapi_grid_config &grid = context.configuration.af.afGrid;\n> + grid.width = kAfMinGridWidth;\n> + grid.height = kAfMinGridHeight;\n> + grid.block_width_log2 = kAfMinGridBlockWidth;\n> + grid.block_height_log2 = kAfMinGridBlockHeight;\n> + grid.height_per_slice = kAfDefaultHeightPerSlice;\n> +\n> + /* x_start and y start are default to BDS center */\n> + grid.x_start = (configInfo.bdsOutputSize.width / 2) -\n> + (((grid.width << grid.block_width_log2) / 2));\n> + grid.y_start = (configInfo.bdsOutputSize.height / 2) -\n> + (((grid.height << grid.block_height_log2) / 2));\n> +\n> + /* x_start and y_start should be even */\n> + grid.x_start = (grid.x_start / 2) * 2;\n> + grid.y_start = (grid.y_start / 2) * 2;\n> + grid.y_start = grid.y_start | IPU3_UAPI_GRID_Y_START_EN;\n> +\n> + /* Initial max focus step */\n> + maxStep_ = kMaxFocusSteps;\n> +\n> + /* Initial focus value */\n> + context.frameContext.af.focus = 0;\n> + /* Maximum variance of the AF statistics */\n> + context.frameContext.af.maxVariance = 0;\n> + /* The stable AF value flag. if it is true, the AF should be in a stable state. */\n> + context.frameContext.af.stable = false;\n> +\n> + return 0;\n> +}\n> +\n> +/**\n> + * \\brief AF coarse scan\n> + * Find a near focused image using a coarse step. The step is determined by coarseSearchStep.\n> + * \\param[in] context The shared IPA context\n> + */\n> +void Af::afCoarseScan(IPAContext &context)\n> +{\n> + if (coarseCompleted_)\n> + return;\n> +\n> + if (afNeedIgnoreFrame())\n> + return;\n> +\n> + if (afScan(context, kCoarseSearchStep)) {\n> + coarseCompleted_ = true;\n> + context.frameContext.af.maxVariance = 0;\n> + focus_ = context.frameContext.af.focus -\n> + (context.frameContext.af.focus * kFineRange);\n> + context.frameContext.af.focus = focus_;\n> + previousVariance_ = 0;\n> + maxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)),\n> + 0U, kMaxFocusSteps);\n> + }\n> +}\n> +\n> +/**\n> + * \\brief AF fine scan\n> + * Find an optimum lens position with moving 1 step for each search.\n> + * \\param[in] context The shared IPA context\n> + */\n> +void Af::afFineScan(IPAContext &context)\n> +{\n> + if (!coarseCompleted_)\n> + return;\n> +\n> + if (afNeedIgnoreFrame())\n> + return;\n> +\n> + if (afScan(context, kFineSearchStep)) {\n> + context.frameContext.af.stable = true;\n> + fineCompleted_ = true;\n> + }\n> +}\n> +\n> +/**\n> + * \\brief AF reset\n> + * Reset all the parameters to start over the AF process.\n> + * \\param[in] context The shared IPA context\n> + */\n> +void Af::afReset(IPAContext &context)\n> +{\n> + if (afNeedIgnoreFrame())\n> + return;\n> +\n> + context.frameContext.af.maxVariance = 0;\n> + context.frameContext.af.focus = 0;\n> + focus_ = 0;\n> + context.frameContext.af.stable = false;\n> + ignoreCounter_ = kIgnoreFrame;\n> + previousVariance_ = 0.0;\n> + coarseCompleted_ = false;\n> + fineCompleted_ = false;\n> + maxStep_ = kMaxFocusSteps;\n> +}\n> +\n> +/**\n> + * \\brief AF variance comparison.\n> + * It always picks the largest variance to replace the previous one. The image\n> + * with a larger variance also indicates it is a clearer image than previous\n> + * one. If it finds the negative sign of derivative, it returns immediately.\n> + * \\param[in] context The IPA context\n> + * \\param min_step The VCM movement step.\n> + * \\return True, if it finds a AF value.\n> + */\n> +bool Af::afScan(IPAContext &context, int min_step)\n> +{\n> + if (focus_ > maxStep_) {\n> + /* If reach the max step, move lens to the position. */\n> + context.frameContext.af.focus = bestFocus_;\n> + return true;\n> + } else {\n> + /*\n> + * Find the maximum of the variance by estimating its\n> + * derivative. If the direction changes, it means we have\n> + * passed a maximum one step before.\n> + */\n> + if ((currentVariance_ - context.frameContext.af.maxVariance) >=\n> + -(context.frameContext.af.maxVariance * 0.1)) {\n> + /*\n> + * Positive and zero derivative:\n> + * The variance is still increasing. The focus could be\n> + * increased for the next comparison. Also, the max variance\n> + * and previous focus value are updated.\n> + */\n> + bestFocus_ = focus_;\n> + focus_ += min_step;\n> + context.frameContext.af.focus = focus_;\n> + context.frameContext.af.maxVariance = currentVariance_;\n> + } else {\n> + /*\n> + * Negative derivative:\n> + * The variance starts to decrease which means the maximum\n> + * variance is found. Set focus step to previous good one\n> + * then return immediately.\n> + */\n> + context.frameContext.af.focus = bestFocus_;\n> + return true;\n> + }\n> + }\n> +\n> + previousVariance_ = currentVariance_;\n> + LOG(IPU3Af, Debug) << \" Previous step is \"\n> + << bestFocus_\n> + << \" Current step is \"\n> + << focus_;\n> + return false;\n> +}\n> +\n> +/**\n> + * \\brief Determine the frame to be ignored.\n> + * \\return Return true the frame is ignored.\n> + * \\return Return false the frame should be processed.\n> + */\n> +bool Af::afNeedIgnoreFrame()\n> +{\n> + if (ignoreCounter_ == 0)\n> + return false;\n> + else\n> + ignoreCounter_--;\n> + return true;\n> +}\n> +\n> +/**\n> + * \\brief Reset frame ignore counter.\n> + */\n> +void Af::afIgnoreFrameReset()\n> +{\n> + ignoreCounter_ = kIgnoreFrame;\n> +}\n> +\n> +/**\n> + * \\brief Estemate variance\n> + */\n> +double Af::afEstemateVariance(y_table_item_t *y_item, uint32_t len,\n> + bool isY1)\n> +{\n> + uint32_t z = 0;\n> + uint32_t total = 0;\n> + double mean;\n> + double var_sum = 0;\n> +\n> + for (z = 0; z < len; z++) {\n> + if (isY1)\n> + total += y_item[z].y1_avg;\n> + else\n> + total += y_item[z].y2_avg;\n> + }\n> + mean = total / len;\n> + for (z = 0; z < len; z++) {\n> + if (isY1)\n> + var_sum += pow((y_item[z].y1_avg - mean), 2);\n> + else\n> + var_sum += pow((y_item[z].y2_avg - mean), 2);\n> + }\n> +\n> + return var_sum / static_cast(len);\n> +}\n> +\n> +/**\n> + * \\brief Determine out-of-focus situation.\n> + * Out-of-focus means that the variance change rate for a focused and a new\n> + * variance is greater than a threshold.\n> + * \\param context The IPA context.\n> + * \\return If it is out-of-focus, return true.\n> + * \\return If is is focused, return false.\n> + */\n> +bool Af::afIsOutOfFocus(IPAContext context)\n> +{\n> + const uint32_t diff_var = std::abs(currentVariance_ -\n> + context.frameContext.af.maxVariance);\n> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> + LOG(IPU3Af, Debug) << \"Variance change rate: \"\n> + << var_ratio\n> + << \" Current VCM step: \"\n> + << context.frameContext.af.focus;\n> + if (var_ratio > kMaxChange)\n> + return true;\n> + else\n> + return false;\n> +}\n> +\n> +/**\n> + * \\brief Determine the max contrast image and lens position.\n> + * Ideally, a clear image also has a raletively higher contrast. So, every\n> + * images for each focus step should be tested to find a optimal focus step.\n> + * The Hill Climbing Algorithm[1] is used to find the maximum variance of the\n> + * AF statistic which is the AF output of IPU3. The focus step is increased\n> + * then the variance of the AF statistic is estimated. If it finds the negative\n> + * derivative which means we just passed the peak, the best focus is found.\n> + *\n> + * [1] Hill Climbing Algorithm, https://en.wikipedia.org/wiki/Hill_climbing\n> + * \\param[in] context The IPA context.\n> + * \\param[in] stats The statistic buffer of IPU3.\n> + */\n> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> +{\n> + y_table_item_t y_item[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE / sizeof(y_table_item_t)];\n> + uint32_t afRawBufferLen;\n> +\n> + /* Evaluate the AF buffer length */\n> + afRawBufferLen = context.configuration.af.afGrid.width *\n> + context.configuration.af.afGrid.height;\n> +\n> + memcpy(y_item, stats->af_raw_buffer.y_table,\n> + afRawBufferLen * sizeof(y_table_item_t));\n> +\n> + /*\n> + * Calculate the mean and the variance of AF statistics for a given grid.\n> + * For coarse: y1 are used.\n> + * For fine: y2 results are used.\n> + */\n> + if (coarseCompleted_)\n> + currentVariance_ = afEstemateVariance(y_item, afRawBufferLen, false);\n> + else\n> + currentVariance_ = afEstemateVariance(y_item, afRawBufferLen, true);\n> +\n> + if (!context.frameContext.af.stable) {\n> + afCoarseScan(context);\n> + afFineScan(context);\n> + } else {\n> + if (afIsOutOfFocus(context))\n> + afReset(context);\n> + else\n> + afIgnoreFrameReset();\n> + }\n> +}\n> +\n> +} /* namespace ipa::ipu3::algorithms */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> new file mode 100644\n> index 00000000..2ca78c84\n> --- /dev/null\n> +++ b/src/ipa/ipu3/algorithms/af.h\n> @@ -0,0 +1,79 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2021, Red Hat\n> + *\n> + * af.h - IPU3 Af algorithm\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"algorithm.h\"\n> +\n> +/* Static variables from repo of chromium */\n> +static constexpr uint8_t kAfMinGridWidth = 16;\n> +static constexpr uint8_t kAfMinGridHeight = 16;\n> +static constexpr uint8_t kAfMaxGridWidth = 32;\n> +static constexpr uint8_t kAfMaxGridHeight = 24;\n> +static constexpr uint16_t kAfMinGridBlockWidth = 4;\n> +static constexpr uint16_t kAfMinGridBlockHeight = 3;\n> +static constexpr uint16_t kAfMaxGridBlockWidth = 6;\n> +static constexpr uint16_t kAfMaxGridBlockHeight = 6;\n> +static constexpr uint16_t kAfDefaultHeightPerSlice = 2;\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa::ipu3::algorithms {\n> +\n> +class Af : public Algorithm\n> +{\n> + /* The format of y_table. From ipu3-ipa repo */\n> + typedef struct __attribute__((packed)) y_table_item {\n> + uint16_t y1_avg;\n> + uint16_t y2_avg;\n> + } y_table_item_t;\n> +public:\n> + Af();\n> + ~Af() = default;\n> +\n> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> +\n> +private:\n> + void afCoarseScan(IPAContext &context);\n> + void afFineScan(IPAContext &context);\n> + bool afScan(IPAContext &context, int min_step);\n> + void afReset(IPAContext &context);\n> + bool afNeedIgnoreFrame();\n> + void afIgnoreFrameReset();\n> + double afEstemateVariance(y_table_item_t *y_item, uint32_t len,\n> + bool isY1);\n> + bool afIsOutOfFocus(IPAContext context);\n> +\n> + /* VCM step configuration. It is the current setting of the VCM step. */\n> + uint32_t focus_;\n> + /* The best VCM step. It is a local optimum VCM step during scanning. */\n> + uint32_t bestFocus_;\n> + /* Current AF statistic variance. */\n> + double currentVariance_;\n> + /* The frames are ignore before starting measuring. */\n> + uint32_t ignoreCounter_;\n> + /* It is used to determine the derivative during scanning */\n> + double previousVariance_;\n> + /* The designated maximum range of focus scanning. */\n> + uint32_t maxStep_;\n> + /* If the coarse scan completes, it is set to true. */\n> + bool coarseCompleted_;\n> + /* If the fine scan completes, it is set to true. */\n> + bool fineCompleted_;\n> +};\n> +\n> +} /* namespace ipa::ipu3::algorithms */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> index 4db6ae1d..b70a551c 100644\n> --- a/src/ipa/ipu3/algorithms/meson.build\n> +++ b/src/ipa/ipu3/algorithms/meson.build\n> @@ -1,6 +1,7 @@\n> # SPDX-License-Identifier: CC0-1.0\n>\n> ipu3_ipa_algorithms = files([\n> + 'af.cpp',\n> 'agc.cpp',\n> 'awb.cpp',\n> 'blc.cpp',\n> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> index 86794ac1..e8f7367c 100644\n> --- a/src/ipa/ipu3/ipa_context.cpp\n> +++ b/src/ipa/ipu3/ipa_context.cpp\n> @@ -69,6 +69,29 @@ namespace libcamera::ipa::ipu3 {\n> * \\brief Number of cells on one line including the ImgU padding\n> */\n>\n> +/**\n> + * \\var IPASessionConfiguration::af\n> + * \\brief AF grid configuration of the IPA\n> + *\n> + * \\var IPASessionConfiguration::af.afGrid\n> + * \\brief AF scene grid configuration.\n> + */\n> +\n> +/**\n> + * \\var IPAFrameContext::af\n> + * \\brief Context for the Automatic Focus algorithm\n> + *\n> + * \\struct IPAFrameContext::af\n> + * \\var IPAFrameContext::af.focus\n> + * \\brief Current position of the lens\n> + *\n> + * \\var IPAFrameContext::af.maxVariance\n> + * \\brief The maximum variance of the current image.\n> + *\n> + * \\var IPAFrameContext::af.stable\n> + * \\brief It is set to true, if the best focus is found.\n> + */\n> +\n> /**\n> * \\var IPASessionConfiguration::agc\n> * \\brief AGC parameters configuration of the IPA\n> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> index c6dc0814..60ad3194 100644\n> --- a/src/ipa/ipu3/ipa_context.h\n> +++ b/src/ipa/ipu3/ipa_context.h\n> @@ -25,6 +25,10 @@ struct IPASessionConfiguration {\n> uint32_t stride;\n> } grid;\n>\n> + struct {\n> + ipu3_uapi_grid_config afGrid;\n> + } af;\n> +\n> struct {\n> utils::Duration minShutterSpeed;\n> utils::Duration maxShutterSpeed;\n> @@ -34,6 +38,12 @@ struct IPASessionConfiguration {\n> };\n>\n> struct IPAFrameContext {\n> + struct {\n> + uint32_t focus;\n> + double maxVariance;\n> + bool stable;\n> + } af;\n> +\n> struct {\n> uint32_t exposure;\n> double gain;\n> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> index e44a31bb..417e0562 100644\n> --- a/src/ipa/ipu3/ipu3.cpp\n> +++ b/src/ipa/ipu3/ipu3.cpp\n> @@ -30,6 +30,7 @@\n>\n> #include \"libcamera/internal/mapped_framebuffer.h\"\n>\n> +#include \"algorithms/af.h\"\n> #include \"algorithms/agc.h\"\n> #include \"algorithms/algorithm.h\"\n> #include \"algorithms/awb.h\"\n> @@ -295,6 +296,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> }\n>\n> /* Construct our Algorithms */\n> + algorithms_.push_back(std::make_unique());\n> algorithms_.push_back(std::make_unique());\n> algorithms_.push_back(std::make_unique());\n> algorithms_.push_back(std::make_unique());\n> --\n> 2.33.1\n>\n\nSorry about that. I forget to loop you in.\nI resend the mail again.\n\nThank you.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id ABD7BBF415\n\tfor ;\n\tTue, 15 Feb 2022 06:41:33 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0FF0A61123;\n\tTue, 15 Feb 2022 07:41:33 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6ED2F604EE\n\tfor ;\n\tTue, 15 Feb 2022 07:41:31 +0100 (CET)","from mail-lj1-f199.google.com (mail-lj1-f199.google.com\n\t[209.85.208.199]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-611-vNFMJ-jfMneaWgImGNVyLQ-1; Tue, 15 Feb 2022 01:41:27 -0500","by mail-lj1-f199.google.com with SMTP id\n\tc31-20020a2ebf1f000000b0022d87a28911so6743942ljr.1\n\tfor ;\n\tMon, 14 Feb 2022 22:41:27 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"ZvmK/qA0\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1644907290;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=gAR2/c2nzVJxkL5yJAwpMfthLYy0jxLgLMAZN2Ja/6g=;\n\tb=ZvmK/qA0s5snOL8ilTY+QODQJH1+f7/4mTTJ0tdVn//U8nz+oNeNl5J7GeuyhPrhF/wYsa\n\t1cz3E3oUwKTAtZyS/F53Lv53uzLsyeoHgyJit6XuaOYfFIZ+/Yg0D67ZUnkA0c6sQPLQon\n\tWGVuMgg8/8MlsS1eG3C/AwXjAlnt/Dk=","X-MC-Unique":"vNFMJ-jfMneaWgImGNVyLQ-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=gAR2/c2nzVJxkL5yJAwpMfthLYy0jxLgLMAZN2Ja/6g=;\n\tb=XaUkeWlMj2WKnTrPy3/VD43405wM1ET+DjRFyhbcMsvOQhww98G72tcitg21h0lxU/\n\t37vdl6Eg/b7WrWZx7HAhq/ZCFeRpX4GOROJcD+5fuJoTLOz4VRniuCy3BNGedmUAtNLq\n\tJNi4AUGgvO77a6ztFccfBS/u2HtMJfovo+J5RQfxqgXYspzkwA7PN03X77/scMD9X7s5\n\tqsymP0kH+cCF5fhtk0YMUfws5u69PK8jykKNOjo6K47YRR5B/f2O5YbT0aELhn058WtC\n\t/JVhacdGfQhRqlTX8uvnDHwJE7VK3O9eIz0RJaNyt/ZRA7egDrhPaDz8a3jpserw44cf\n\ti9TQ==","X-Gm-Message-State":"AOAM532XMyQPykn4P0LLuthJvv7vmzxyja6Jj0HQ6N63U6D1S7pfxxJN\n\t3hqiY2QoJxNqAemrF6mdPBwlDB+9vuTYDCzx1aMV/WiOPF/kbODv7vq+H0KcPUzxBCp8lYQyibu\n\tx4xKJ51Ef1HcIN36e141IH2cznO1RqmLEgzEmLYEcXt0gaJS8MA==","X-Received":["by 2002:a05:6512:3f03:: with SMTP id\n\ty3mr2042054lfa.515.1644907285585; \n\tMon, 14 Feb 2022 22:41:25 -0800 (PST)","by 2002:a05:6512:3f03:: with SMTP id\n\ty3mr2042028lfa.515.1644907284923; \n\tMon, 14 Feb 2022 22:41:24 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJxUAhBFK3mefL60hVuF2ePCLF+jHB1ybz+7yEm9SSlahkgApckRQo6nnaBGWxWqAUqS6z8tRrGqGd+zlsGtw4o=","MIME-Version":"1.0","References":"<20220214095136.25425-1-hpa@redhat.com>","In-Reply-To":"<20220214095136.25425-1-hpa@redhat.com>","From":"Kate Hsuan ","Date":"Tue, 15 Feb 2022 14:41:12 +0800","Message-ID":"","To":"libcamera devel ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [PATCH v8] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22186,"web_url":"https://patchwork.libcamera.org/comment/22186/","msgid":"<516133b3-995e-7642-2922-aadce32256ef@ideasonboard.com>","date":"2022-02-22T13:38:00","subject":"Re: [libcamera-devel] [PATCH v8] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate,\n\nThanks for the patch !\n\nOn 14/02/2022 10:51, Kate Hsuan wrote:\n> Since VCM for surface Go 2 (dw9719) had been successfully\n> driven, this Af module can be used to control the VCM and\n> determine the focus value based on the IPU3 AF state.\n> \n> Based on the values from the IPU3 AF buffer, the variance\n> of each focus step is determined and a greedy approach is\n> used to find the maximum variance of the AF state and an\n> appropriate focus value.\n> \n> The grid configuration is implemented as a context. Also,\n> the grid parameter- AF_MIN_BLOCK_WIDTH is set to 4 (default\n> is 3) since if the default value is used, x_start\n> (x_start > 640) will be at an incorrect location of the\n> image (rightmost of the sensor).\n> \n> Signed-off-by: Kate Hsuan \n> Tested-by: Jean-Michel Hautbois \n> ---\n> Changes in v8:\n> 1. Revised and improved the comments.\n> 2. Simplified the algorithm.\n> ---\n> src/ipa/ipu3/algorithms/af.cpp | 435 ++++++++++++++++++++++++++++\n> src/ipa/ipu3/algorithms/af.h | 79 +++++\n> src/ipa/ipu3/algorithms/meson.build | 1 +\n> src/ipa/ipu3/ipa_context.cpp | 23 ++\n> src/ipa/ipu3/ipa_context.h | 10 +\n> src/ipa/ipu3/ipu3.cpp | 2 +\n> 6 files changed, 550 insertions(+)\n> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> create mode 100644 src/ipa/ipu3/algorithms/af.h\n> \n> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> new file mode 100644\n> index 00000000..17055c04\n> --- /dev/null\n> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> @@ -0,0 +1,435 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2021, Red Hat\n> + *\n> + * af.cpp - IPU3 auto focus algorithm\n> + */\n> +\n> +#include \"af.h\"\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"libipa/histogram.h\"\n> +\n> +/**\n> + * \\file af.h\n> + */\n> +\n> +/**\n> + * \\var kAfMinGridWidth\n> + * \\brief the minimum width of AF grid.\n> + * The minimum grid horizontal dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMinGridHeight\n> + * \\brief the minimum height of AF grid.\n> + * The minimum grid vertical dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMaxGridWidth\n> + * \\brief the maximum width of AF grid.\n> + * The maximum grid horizontal dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMaxGridHeight\n> + * \\brief The maximum height of AF grid.\n> + * The maximum grid vertical dimensions.\n> +*/\n> +\n> +/**\n> + * \\var kAfMinGridBlockWidth\n> + * \\brief The minimum block size of the width.\n> + * The minimum value of Log2 of the width of the grid cell.\n> + */\n> +\n> +/**\n> + * \\var kAfMinGridBlockHeight\n> + * \\brief The minimum block size of the height.\n> + * The minimum value of Log2 of the height of the grid cell.\n> + */\n> +\n> +/**\n> + * \\def kAfMaxGridBlockWidth\n> + * \\brief The maximum block size of the width.\n> + * The maximum value of Log2 of the width of the grid cell.\n> + */\n> +\n> +/**\n> + * \\var kAfMaxGridBlockHeight\n> + * \\brief The maximum block size of the height.\n> + * The maximum value of Log2 of the height of the grid cell.\n> + */\n> +\n> +/**\n> + * \\var kAfDefaultHeightPerSlice\n> + * \\brief The default number of blocks in vertical axis per slice.\n> + * The number of blocks in vertical axis per slice.\n> + */\n> +\n> +namespace libcamera {\n> +\n> +using namespace std::literals::chrono_literals;\n> +\n> +namespace ipa::ipu3::algorithms {\n> +\n> +LOG_DEFINE_CATEGORY(IPU3Af)\n> +\n> +/**\n> + * Maximum focus steps of the VCM control\n> + * \\todo should be obtained from the VCM driver\n> + */\n> +static constexpr uint32_t kMaxFocusSteps = 1023;\n> +\n> +/* Minimum focus step for searching appropriate focus */\n> +static constexpr uint32_t kCoarseSearchStep = 30;\n> +static constexpr uint32_t kFineSearchStep = 1;\n> +\n> +/* Max ratio of variance change, 0.0 < kMaxChange < 1.0 */\n> +static constexpr double kMaxChange = 0.5;\n> +\n> +/* The numbers of frame to be ignored, before performing focus scan. */\n> +static constexpr uint32_t kIgnoreFrame = 10;\n> +\n> +/* Fine scan range 0 < kFineRange < 1 */\n> +static constexpr double kFineRange = 0.05;\n> +\n> +/* Settings for IPU3 AF filter */\n> +static struct ipu3_uapi_af_filter_config afFilterConfigDefault = {\n> +\t.y1_coeff_0 = { 0, 1, 3, 7 },\n> +\t.y1_coeff_1 = { 11, 13, 1, 2 },\n> +\t.y1_coeff_2 = { 8, 19, 34, 242 },\n> +\t.y1_sign_vec = 0x7fdffbfe,\n> +\t.y2_coeff_0 = { 0, 1, 6, 6 },\n> +\t.y2_coeff_1 = { 13, 25, 3, 0 },\n> +\t.y2_coeff_2 = { 25, 3, 177, 254 },\n> +\t.y2_sign_vec = 0x4e53ca72,\n> +\t.y_calc = { 8, 8, 8, 8 },\n> +\t.nf = { 0, 9, 0, 9, 0 },\n> +};\n> +\n> +/**\n> + * \\class Af\n> + * \\brief An auto-focus algorithm based on IPU3 statistics\n> + * This algorithm is used to determine the position of the lens to make a\n> + * focused image. The IPU3 AF processing block computes the statistics that\n> + * are composed by two types of filtered value and stores in a AF buffer.\n> + * Typically, for a clear image, it has a relatively higher contrast than a\n> + * blurred one. Therefore, if an image with the highest contrast can be\n> + * found through the scan, the position of the len indicates to a clearest\n> + * image.\n> + */\n> +Af::Af()\n> +\t: focus_(0), bestFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> +\t coarseCompleted_(false), fineCompleted_(false)\n> +{\n> +}\n> +\n> +/**\n> + * \\copydoc libcamera::ipa::Algorithm::prepare\n> + */\n> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n> +{\n> +\tconst struct ipu3_uapi_grid_config &grid = context.configuration.af.afGrid;\n> +\tparams->acc_param.af.grid_cfg = grid;\n> +\tparams->acc_param.af.filter_config = afFilterConfigDefault;\n> +\n> +\t/* Enable AF processing block */\n> +\tparams->use.acc_af = 1;\n> +}\n> +\n> +/**\n> + * \\brief Configure the Af given a configInfo\n> + * \\param[in] context The shared IPA context\n> + * \\param[in] configInfo The IPA configuration data\n> + * \\return 0\n> + */\n> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> +{\n> +\tstruct ipu3_uapi_grid_config &grid = context.configuration.af.afGrid;\n> +\tgrid.width = kAfMinGridWidth;\n> +\tgrid.height = kAfMinGridHeight;\n> +\tgrid.block_width_log2 = kAfMinGridBlockWidth;\n> +\tgrid.block_height_log2 = kAfMinGridBlockHeight;\n> +\tgrid.height_per_slice = kAfDefaultHeightPerSlice;\n> +\n> +\t/* x_start and y start are default to BDS center */\n> +\tgrid.x_start = (configInfo.bdsOutputSize.width / 2) -\n> +\t\t (((grid.width << grid.block_width_log2) / 2));\n> +\tgrid.y_start = (configInfo.bdsOutputSize.height / 2) -\n> +\t\t (((grid.height << grid.block_height_log2) / 2));\n> +\n> +\t/* x_start and y_start should be even */\n> +\tgrid.x_start = (grid.x_start / 2) * 2;\n> +\tgrid.y_start = (grid.y_start / 2) * 2;\n> +\tgrid.y_start = grid.y_start | IPU3_UAPI_GRID_Y_START_EN;\n> +\n> +\t/* Initial max focus step */\n> +\tmaxStep_ = kMaxFocusSteps;\n> +\n> +\t/* Initial focus value */\n> +\tcontext.frameContext.af.focus = 0;\n> +\t/* Maximum variance of the AF statistics */\n> +\tcontext.frameContext.af.maxVariance = 0;\n> +\t/* The stable AF value flag. if it is true, the AF should be in a stable state. */\n> +\tcontext.frameContext.af.stable = false;\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief AF coarse scan\n> + * Find a near focused image using a coarse step. The step is determined by coarseSearchStep.\n> + * \\param[in] context The shared IPA context\n> + */\n> +void Af::afCoarseScan(IPAContext &context)\n> +{\n> +\tif (coarseCompleted_)\n> +\t\treturn;\n> +\n> +\tif (afNeedIgnoreFrame())\n> +\t\treturn;\n> +\n> +\tif (afScan(context, kCoarseSearchStep)) {\n> +\t\tcoarseCompleted_ = true;\n> +\t\tcontext.frameContext.af.maxVariance = 0;\n> +\t\tfocus_ = context.frameContext.af.focus -\n> +\t\t\t (context.frameContext.af.focus * kFineRange);\n> +\t\tcontext.frameContext.af.focus = focus_;\n> +\t\tpreviousVariance_ = 0;\n> +\t\tmaxStep_ = std::clamp(focus_ + static_cast((focus_ * kFineRange)),\n> +\t\t\t\t 0U, kMaxFocusSteps);\n> +\t}\n> +}\n> +\n> +/**\n> + * \\brief AF fine scan\n> + * Find an optimum lens position with moving 1 step for each search.\n> + * \\param[in] context The shared IPA context\n> + */\n> +void Af::afFineScan(IPAContext &context)\n> +{\n> +\tif (!coarseCompleted_)\n> +\t\treturn;\n> +\n> +\tif (afNeedIgnoreFrame())\n> +\t\treturn;\n> +\n> +\tif (afScan(context, kFineSearchStep)) {\n> +\t\tcontext.frameContext.af.stable = true;\n> +\t\tfineCompleted_ = true;\n> +\t}\n> +}\n> +\n> +/**\n> + * \\brief AF reset\n> + * Reset all the parameters to start over the AF process.\n> + * \\param[in] context The shared IPA context\n> + */\n> +void Af::afReset(IPAContext &context)\n> +{\n> +\tif (afNeedIgnoreFrame())\n> +\t\treturn;\n> +\n> +\tcontext.frameContext.af.maxVariance = 0;\n> +\tcontext.frameContext.af.focus = 0;\n> +\tfocus_ = 0;\n> +\tcontext.frameContext.af.stable = false;\n> +\tignoreCounter_ = kIgnoreFrame;\n> +\tpreviousVariance_ = 0.0;\n> +\tcoarseCompleted_ = false;\n> +\tfineCompleted_ = false;\n> +\tmaxStep_ = kMaxFocusSteps;\n> +}\n> +\n> +/**\n> + * \\brief AF variance comparison.\n> + * It always picks the largest variance to replace the previous one. The image\n> + * with a larger variance also indicates it is a clearer image than previous\n> + * one. If it finds the negative sign of derivative, it returns immediately.\n> + * \\param[in] context The IPA context\n> + * \\param min_step The VCM movement step.\n> + * \\return True, if it finds a AF value.\n> + */\n> +bool Af::afScan(IPAContext &context, int min_step)\n> +{\n> +\tif (focus_ > maxStep_) {\n> +\t\t/* If reach the max step, move lens to the position. */\n> +\t\tcontext.frameContext.af.focus = bestFocus_;\n> +\t\treturn true;\n> +\t} else {\n> +\t\t/*\n> +\t\t * Find the maximum of the variance by estimating its\n> +\t\t * derivative. If the direction changes, it means we have\n> +\t\t * passed a maximum one step before.\n> +\t\t*/\n> +\t\tif ((currentVariance_ - context.frameContext.af.maxVariance) >=\n> +\t\t -(context.frameContext.af.maxVariance * 0.1)) {\n> +\t\t\t/*\n> +\t\t\t * Positive and zero derivative:\n> +\t\t\t * The variance is still increasing. The focus could be\n> +\t\t\t * increased for the next comparison. Also, the max variance\n> +\t\t\t * and previous focus value are updated.\n> +\t\t\t */\n> +\t\t\tbestFocus_ = focus_;\n> +\t\t\tfocus_ += min_step;\n> +\t\t\tcontext.frameContext.af.focus = focus_;\n> +\t\t\tcontext.frameContext.af.maxVariance = currentVariance_;\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * Negative derivative:\n> +\t\t\t * The variance starts to decrease which means the maximum\n> +\t\t\t * variance is found. Set focus step to previous good one\n> +\t\t\t * then return immediately.\n> +\t\t\t */\n> +\t\t\tcontext.frameContext.af.focus = bestFocus_;\n> +\t\t\treturn true;\n> +\t\t}\n> +\t}\n> +\n> +\tpreviousVariance_ = currentVariance_;\n> +\tLOG(IPU3Af, Debug) << \" Previous step is \"\n> +\t\t\t << bestFocus_\n> +\t\t\t << \" Current step is \"\n> +\t\t\t << focus_;\n> +\treturn false;\n> +}\n> +\n> +/**\n> + * \\brief Determine the frame to be ignored.\n> + * \\return Return true the frame is ignored.\n> + * \\return Return false the frame should be processed.\n> + */\n> +bool Af::afNeedIgnoreFrame()\n> +{\n> +\tif (ignoreCounter_ == 0)\n> +\t\treturn false;\n> +\telse\n> +\t\tignoreCounter_--;\n> +\treturn true;\n> +}\n> +\n> +/**\n> + * \\brief Reset frame ignore counter.\n> + */\n> +void Af::afIgnoreFrameReset()\n> +{\n> +\tignoreCounter_ = kIgnoreFrame;\n> +}\n> +\n> +/**\n> + * \\brief Estemate variance\n> + */\n> +double Af::afEstemateVariance(y_table_item_t *y_item, uint32_t len,\n> +\t\t\t bool isY1)\n> +{\n> +\tuint32_t z = 0;\n> +\tuint32_t total = 0;\n> +\tdouble mean;\n> +\tdouble var_sum = 0;\n> +\n> +\tfor (z = 0; z < len; z++) {\n> +\t\tif (isY1)\n> +\t\t\ttotal += y_item[z].y1_avg;\n> +\t\telse\n> +\t\t\ttotal += y_item[z].y2_avg;\n> +\t}\n> +\tmean = total / len;\n> +\tfor (z = 0; z < len; z++) {\n> +\t\tif (isY1)\n> +\t\t\tvar_sum += pow((y_item[z].y1_avg - mean), 2);\n> +\t\telse\n> +\t\t\tvar_sum += pow((y_item[z].y2_avg - mean), 2);\n> +\t}\n> +\n> +\treturn var_sum / static_cast(len);\n> +}\n> +\n> +/**\n> + * \\brief Determine out-of-focus situation.\n> + * Out-of-focus means that the variance change rate for a focused and a new\n> + * variance is greater than a threshold.\n> + * \\param context The IPA context.\n> + * \\return If it is out-of-focus, return true.\n> + * \\return If is is focused, return false.\n> + */\n> +bool Af::afIsOutOfFocus(IPAContext context)\n> +{\n> +\tconst uint32_t diff_var = std::abs(currentVariance_ -\n> +\t\t\t\t\t context.frameContext.af.maxVariance);\n> +\tconst double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> +\tLOG(IPU3Af, Debug) << \"Variance change rate: \"\n> +\t\t\t << var_ratio\n> +\t\t\t << \" Current VCM step: \"\n> +\t\t\t << context.frameContext.af.focus;\n> +\tif (var_ratio > kMaxChange)\n> +\t\treturn true;\n> +\telse\n> +\t\treturn false;\n> +}\n> +\n> +/**\n> + * \\brief Determine the max contrast image and lens position.\n> + * Ideally, a clear image also has a raletively higher contrast. So, every\n> + * images for each focus step should be tested to find a optimal focus step.\n> + * The Hill Climbing Algorithm[1] is used to find the maximum variance of the\n> + * AF statistic which is the AF output of IPU3. The focus step is increased\n> + * then the variance of the AF statistic is estimated. If it finds the negative\n> + * derivative which means we just passed the peak, the best focus is found.\n> + *\n> + * [1] Hill Climbing Algorithm, https://en.wikipedia.org/wiki/Hill_climbing\n\nThanks for this reference ;-).\nReviewed-by: Jean-Michel Hautbois \n\n> + * \\param[in] context The IPA context.\n> + * \\param[in] stats The statistic buffer of IPU3.\n> + */\n> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> +{\n> +\ty_table_item_t y_item[IPU3_UAPI_AF_Y_TABLE_MAX_SIZE / sizeof(y_table_item_t)];\n> +\tuint32_t afRawBufferLen;\n> +\n> +\t/* Evaluate the AF buffer length */\n> +\tafRawBufferLen = context.configuration.af.afGrid.width *\n> +\t\t\t context.configuration.af.afGrid.height;\n> +\n> +\tmemcpy(y_item, stats->af_raw_buffer.y_table,\n> +\t afRawBufferLen * sizeof(y_table_item_t));\n> +\n> +\t/*\n> +\t * Calculate the mean and the variance of AF statistics for a given grid.\n> +\t * For coarse: y1 are used.\n> +\t * For fine: y2 results are used.\n> +\t */\n> +\tif (coarseCompleted_)\n> +\t\tcurrentVariance_ = afEstemateVariance(y_item, afRawBufferLen, false);\n> +\telse\n> +\t\tcurrentVariance_ = afEstemateVariance(y_item, afRawBufferLen, true);\n> +\n> +\tif (!context.frameContext.af.stable) {\n> +\t\tafCoarseScan(context);\n> +\t\tafFineScan(context);\n> +\t} else {\n> +\t\tif (afIsOutOfFocus(context))\n> +\t\t\tafReset(context);\n> +\t\telse\n> +\t\t\tafIgnoreFrameReset();\n> +\t}\n> +}\n> +\n> +} /* namespace ipa::ipu3::algorithms */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> new file mode 100644\n> index 00000000..2ca78c84\n> --- /dev/null\n> +++ b/src/ipa/ipu3/algorithms/af.h\n> @@ -0,0 +1,79 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2021, Red Hat\n> + *\n> + * af.h - IPU3 Af algorithm\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"algorithm.h\"\n> +\n> +/* Static variables from repo of chromium */\n> +static constexpr uint8_t kAfMinGridWidth = 16;\n> +static constexpr uint8_t kAfMinGridHeight = 16;\n> +static constexpr uint8_t kAfMaxGridWidth = 32;\n> +static constexpr uint8_t kAfMaxGridHeight = 24;\n> +static constexpr uint16_t kAfMinGridBlockWidth = 4;\n> +static constexpr uint16_t kAfMinGridBlockHeight = 3;\n> +static constexpr uint16_t kAfMaxGridBlockWidth = 6;\n> +static constexpr uint16_t kAfMaxGridBlockHeight = 6;\n> +static constexpr uint16_t kAfDefaultHeightPerSlice = 2;\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa::ipu3::algorithms {\n> +\n> +class Af : public Algorithm\n> +{\n> +\t/* The format of y_table. From ipu3-ipa repo */\n> +\ttypedef struct __attribute__((packed)) y_table_item {\n> +\t\tuint16_t y1_avg;\n> +\t\tuint16_t y2_avg;\n> +\t} y_table_item_t;\n> +public:\n> +\tAf();\n> +\t~Af() = default;\n> +\n> +\tvoid prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> +\tint configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> +\tvoid process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> +\n> +private:\n> +\tvoid afCoarseScan(IPAContext &context);\n> +\tvoid afFineScan(IPAContext &context);\n> +\tbool afScan(IPAContext &context, int min_step);\n> +\tvoid afReset(IPAContext &context);\n> +\tbool afNeedIgnoreFrame();\n> +\tvoid afIgnoreFrameReset();\n> +\tdouble afEstemateVariance(y_table_item_t *y_item, uint32_t len,\n> +\t\t\t\t bool isY1);\n> +\tbool afIsOutOfFocus(IPAContext context);\n> +\n> +\t/* VCM step configuration. It is the current setting of the VCM step. */\n> +\tuint32_t focus_;\n> +\t/* The best VCM step. It is a local optimum VCM step during scanning. */\n> +\tuint32_t bestFocus_;\n> +\t/* Current AF statistic variance. */\n> +\tdouble currentVariance_;\n> +\t/* The frames are ignore before starting measuring. */\n> +\tuint32_t ignoreCounter_;\n> +\t/* It is used to determine the derivative during scanning */\n> +\tdouble previousVariance_;\n> +\t/* The designated maximum range of focus scanning. */\n> +\tuint32_t maxStep_;\n> +\t/* If the coarse scan completes, it is set to true. */\n> +\tbool coarseCompleted_;\n> +\t/* If the fine scan completes, it is set to true. */\n> +\tbool fineCompleted_;\n> +};\n> +\n> +} /* namespace ipa::ipu3::algorithms */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> index 4db6ae1d..b70a551c 100644\n> --- a/src/ipa/ipu3/algorithms/meson.build\n> +++ b/src/ipa/ipu3/algorithms/meson.build\n> @@ -1,6 +1,7 @@\n> # SPDX-License-Identifier: CC0-1.0\n> \n> ipu3_ipa_algorithms = files([\n> + 'af.cpp',\n> 'agc.cpp',\n> 'awb.cpp',\n> 'blc.cpp',\n> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> index 86794ac1..e8f7367c 100644\n> --- a/src/ipa/ipu3/ipa_context.cpp\n> +++ b/src/ipa/ipu3/ipa_context.cpp\n> @@ -69,6 +69,29 @@ namespace libcamera::ipa::ipu3 {\n> * \\brief Number of cells on one line including the ImgU padding\n> */\n> \n> +/**\n> + * \\var IPASessionConfiguration::af\n> + * \\brief AF grid configuration of the IPA\n> + *\n> + * \\var IPASessionConfiguration::af.afGrid\n> + * \\brief AF scene grid configuration.\n> + */\n> +\n> +/**\n> + * \\var IPAFrameContext::af\n> + * \\brief Context for the Automatic Focus algorithm\n> + *\n> + * \\struct IPAFrameContext::af\n> + * \\var IPAFrameContext::af.focus\n> + * \\brief Current position of the lens\n> + *\n> + * \\var IPAFrameContext::af.maxVariance\n> + * \\brief The maximum variance of the current image.\n> + *\n> + * \\var IPAFrameContext::af.stable\n> + * \\brief It is set to true, if the best focus is found.\n> + */\n> +\n> /**\n> * \\var IPASessionConfiguration::agc\n> * \\brief AGC parameters configuration of the IPA\n> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> index c6dc0814..60ad3194 100644\n> --- a/src/ipa/ipu3/ipa_context.h\n> +++ b/src/ipa/ipu3/ipa_context.h\n> @@ -25,6 +25,10 @@ struct IPASessionConfiguration {\n> \t\tuint32_t stride;\n> \t} grid;\n> \n> +\tstruct {\n> +\t\tipu3_uapi_grid_config afGrid;\n> +\t} af;\n> +\n> \tstruct {\n> \t\tutils::Duration minShutterSpeed;\n> \t\tutils::Duration maxShutterSpeed;\n> @@ -34,6 +38,12 @@ struct IPASessionConfiguration {\n> };\n> \n> struct IPAFrameContext {\n> +\tstruct {\n> +\t\tuint32_t focus;\n> +\t\tdouble maxVariance;\n> +\t\tbool stable;\n> +\t} af;\n> +\n> \tstruct {\n> \t\tuint32_t exposure;\n> \t\tdouble gain;\n> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> index e44a31bb..417e0562 100644\n> --- a/src/ipa/ipu3/ipu3.cpp\n> +++ b/src/ipa/ipu3/ipu3.cpp\n> @@ -30,6 +30,7 @@\n> \n> #include \"libcamera/internal/mapped_framebuffer.h\"\n> \n> +#include \"algorithms/af.h\"\n> #include \"algorithms/agc.h\"\n> #include \"algorithms/algorithm.h\"\n> #include \"algorithms/awb.h\"\n> @@ -295,6 +296,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> \t}\n> \n> \t/* Construct our Algorithms */\n> +\talgorithms_.push_back(std::make_unique());\n> \talgorithms_.push_back(std::make_unique());\n> \talgorithms_.push_back(std::make_unique());\n> \talgorithms_.push_back(std::make_unique());","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 40253BE08A\n\tfor ;\n\tTue, 22 Feb 2022 13:38:06 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 83F7961145;\n\tTue, 22 Feb 2022 14:38:05 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 445BE604EE\n\tfor ;\n\tTue, 22 Feb 2022 14:38:03 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:4b22:285f:3029:cd1c] (unknown\n\t[IPv6:2a01:e0a:169:7140:4b22:285f:3029:cd1c])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id C785B47F;\n\tTue, 22 Feb 2022 14:38:02 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Ownb5gfJ\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1645537082;\n\tbh=nKdv5P8blXLj0PUhzK3fVJcEK7K0i+ZvoK3Ecu2OvOI=;\n\th=Date:Subject:To:References:From:In-Reply-To:From;\n\tb=Ownb5gfJpgZAUS6J+BIYoH6QyBNjswz+d+pueZre6voJF6Bp2ZVziHk1pOx6+h2Cw\n\tE5n1NI+RE8rFx8Xkimp3qI+lHcFNIoSl/63T9LxT3PIz1GAGHnu2mLYoDl9vjFP8d0\n\t5VfDB6yrXLGBxTwUrlxgiDlMF1IaNFXknxGkqEWw=","Message-ID":"<516133b3-995e-7642-2922-aadce32256ef@ideasonboard.com>","Date":"Tue, 22 Feb 2022 14:38:00 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.5.0","Content-Language":"en-US","To":"Kate Hsuan ,\n\tlibcamera devel ","References":"<20220214095136.25425-1-hpa@redhat.com>","From":"Jean-Michel Hautbois ","In-Reply-To":"<20220214095136.25425-1-hpa@redhat.com>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Subject":"Re: [libcamera-devel] [PATCH v8] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]