[{"id":21809,"web_url":"https://patchwork.libcamera.org/comment/21809/","msgid":"<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>","date":"2021-12-16T15:32:04","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate,\n\nThanks for the patch :-) !\n\nI have a few comments below, and I have to tell you I couldn't make it \nwork here... I don't know how you are testing it ? I can see the lens \nposition changing but it passes over the correct focus and never locks \non it.\n\nOn 13/12/2021 13:28, Kate Hsuan wrote:\n> Since VCM for surface Go 2 (dw9719) had been successfully\n> driven, this Af module can be used to control the VCM and\n> determine the focus value based on the IPU3 AF state.\n> \n> The variance of each focus step is determined and a greedy\n> approah is used to find the maximum variance of the AF\n> state and a appropriate focus value.\n> \n> Changes in v2:\n> 1. Add grid configuration interface.\n> 2. Add AF accelerator configuration.\n> 3. Move default focus area to the center of the image.\n> \n> Changes in v3:\n> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n> algorithm.\n> 2. Fix greedy AF algorithm since V4l2 interface had been\n> removed.\n> 3. Simplify default grid config for AF.\n> 4. AF start area is determined by bdsOutputSize.\n> \n> Changes in v4:\n> In v4, it significant improves the AF scan time using a two pass\n> scaning method and the AF scan will be terminated when it finds\n> a negative gradient.\n> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n> the AF success rate.\n> 2. The low pass filter convolution results are used to determine\n> a coarse AF result.\n> 3. The high pass convolution result is used to find a good lens\n> step in a given AF range determined in pass 1.\n> \n\nThe lines above describe the changes between the versions, which is \ngreat ! But those should go after the \"---\" as we don't want those in \nthe final commit message.\n\n> Signed-off-by: Kate Hsuan \n> ---\n\nChanges in ...\n\n---\n> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n> src/ipa/ipu3/algorithms/meson.build | 3 +-\n> src/ipa/ipu3/ipa_context.cpp | 27 +++\n> src/ipa/ipu3/ipa_context.h | 11 +\n> src/ipa/ipu3/ipu3.cpp | 2 +\n> 6 files changed, 442 insertions(+), 1 deletion(-)\n> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> create mode 100644 src/ipa/ipu3/algorithms/af.h\n> \n\nI applied your patch, and I have compilation issues:\n../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for \nmember ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n 89 | };\n | ^\n../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for \nmember ‘ipu3_uapi_grid_config::height_per_slice’ \n[-Werror=missing-field-initializers]\n../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for \nmember ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for \nmember ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void \nlibcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&, \nconst ipu3_uapi_stats_3a*)’:\n../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of \npacked member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer \nvalue [-Werror=address-of-packed-member]\n 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n\nI solved them locally (with a quick and dirty patch) how did you not \nhave those ?\n\n> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> new file mode 100644\n> index 00000000..620be9fa\n> --- /dev/null\n> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> @@ -0,0 +1,334 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2021, Red Hat\n> + *\n> + * af.cpp - IPU3 auto focus control\n> + */\n> +\n> +#include \"af.h\"\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"libipa/histogram.h\"\n> +\n> +/**\n> + * \\file af.h\n> + */\n> +\n> +namespace libcamera {\n> +\n> +using namespace std::literals::chrono_literals;\n> +\n> +namespace ipa::ipu3::algorithms {\n> +\n> +/**\n> + * \\class Af\n> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n> + *\n> + * This algorithm is used to determine the position of the lens and get a\n> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n> + * by high pass and low pass filtered value and stores in a AF buffer.\n> + * Typically, for a focused image, it has relative high contrast than a\n> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n> + * highest contrast can be found from the AF scan, the lens' position is the\n> + * best step of the focus.\n> + *\n> + */\n> +\n> +LOG_DEFINE_CATEGORY(IPU3Af)\n> +\n> +/**\n> + * Maximum focus value of the VCM control\n> + * \\todo should be obtained from the VCM driver\n> + */\n> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n> +\n> +/* minimum focus step for searching appropriate focus*/\n> +static constexpr uint32_t coarseSearchStep_ = 10;\n> +static constexpr uint32_t fineSearchStep_ = 1;\n> +\n> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n> +static constexpr double MaxChange_ = 0.8;\n\nWhy 0.8 ? Any explanation on how to set it ?\n\n> +\n> +/* settings for Auto Focus from the kernel */\n> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> +\t.filter_config = {\n> +\t\t{ 0, 0, 0, 0 },\n> +\t\t{ 0, 0, 0, 0 },\n> +\t\t{ 0, 0, 0, 128 },\n> +\t\t0,\n> +\t\t{ 0, 0, 0, 0 },\n> +\t\t{ 0, 0, 0, 0 },\n> +\t\t{ 0, 0, 0, 128 },\n> +\t\t0,\n> +\t\t.y_calc = { 8, 8, 8, 8 },\n> +\t\t.nf = { 0, 7, 0, 7, 0 },\n> +\t},\n> +\t.grid_cfg = {\n> +\t\t.width = 16,\n> +\t\t.height = 16,\n> +\t\t.block_width_log2 = 3,\n> +\t\t.block_height_log2 = 3,\n> +\t\t.x_start = 10,\n> +\t\t.y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> +\t},\n> +};\n\nAs mentionned, you missed some fields:\n\n'''\n@@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s \nimgu_css_af_defaults = {\n .y_calc = { 8, 8, 8, 8 },\n .nf = { 0, 7, 0, 7, 0 },\n },\n+ .padding = { 0, 0, 0, 0 },\n .grid_cfg = {\n .width = 16,\n .height = 16,\n .block_width_log2 = 3,\n .block_height_log2 = 3,\n+ .height_per_slice = 1,\n .x_start = 10,\n .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n+ .x_end = 138,\n+ .y_end = 130,\n },\n };\n'''\n\n> +\n> +Af::Af()\n> +\t: focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> +\t pass1Done_(false), pass2Done_(false)\n> +{\n> +\tmaxStep_ = MaxFocusSteps_;\n> +}\n> +\n> +Af::~Af()\n> +{\n> +}\n> +\n> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n\nMissing documentation for prepare() => you should build the doc ;-)\n\n> +{\n> +\tparams->use.acc_af = 1;\n> +\tparams->acc_param.af = imgu_css_af_defaults;\n> +\tparams->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n> +\tparams->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n\nYou are never changing the grid size, is it intended ?\n\n> +}\n> +\n> +/**\n> + * \\brief Configure the Af given a configInfo\n> + * \\param[in] context The shared IPA context\n> + * \\param[in] configInfo The IPA configuration data\n> + *\n> + * \\return 0\n> + */\n> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> +{\n> +\t/* determined focus value i.e. current focus value */\n> +\tcontext.frameContext.af.focus = 0;\n> +\t/* maximum variance of the AF statistics */\n> +\tcontext.frameContext.af.maxVariance = 0;\ndouble type, maybe s/= 0/= 0.0/ ?\n\n> +\t/* is focused? if it is true, the AF should be in a stable state. */\n> +\tcontext.frameContext.af.stable = false;\n> +\t/* frame to be ignored before start to estimate AF variance. */\n> +\tignoreFrame_ = 10;\n\nCould be a constexpr ?\n\n> +\n> +\t/*\n> +\t * AF default area configuration\n> +\t * Move AF area to the center of the image.\n> +\t */\n> +\t/* Default AF width is 16x8 = 128 */\n> +\tcontext.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n\nWhy are you using a fixed value for the grid ? You could use the value \nstored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n\n> +\tcontext.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n> +\n> +\tLOG(IPU3Af, Debug) << \"BDS X: \"\n> +\t\t\t << configInfo.bdsOutputSize.width\n> +\t\t\t << \" Y: \"\n> +\t\t\t << configInfo.bdsOutputSize.height;\n> +\tLOG(IPU3Af, Debug) << \"AF start from X: \"\n> +\t\t\t << context.configuration.af.start_x\n> +\t\t\t << \" Y: \"\n> +\t\t\t << context.configuration.af.start_y;\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief AF coarse scan\n> + * \\param[in] context The shared IPA context\n> + *\n> + */\n> +void Af::af_coarse_scan(IPAContext &context)\n\nBad naming (camelCase only). Same below.\n\n> +{\n> +\tif (pass1Done_ == true)\n> +\t\treturn;\n> +\n> +\tif (af_scan(context, coarseSearchStep_)) {\n> +\t\tpass1Done_ = true;\n> +\t\tcontext.frameContext.af.maxVariance = 0;\n> +\t\tfocus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n> +\t\tcontext.frameContext.af.focus = focus_;\n> +\t\tpreviousVariance_ = 0;\n> +\t\tmaxStep_ = focus_ + (focus_ * 0.2);\n> +\t}\n> +}\n> +\n> +/**\n> + * \\brief AF fine scan\n> + * \\param[in] context The shared IPA context\n> + *\n> + */\n> +void Af::af_fine_scan(IPAContext &context)\n> +{\n> +\tif (pass1Done_ != true)\n> +\t\treturn;\n> +\n> +\tif (af_scan(context, fineSearchStep_)) {\n> +\t\tcontext.frameContext.af.stable = true;\n> +\t\tpass2Done_ = true;\n> +\t}\n> +}\nI am not a big fan of these pass1Done_ and pass2Done_ variables, let's \nseee where it is used...\n\n> +\n> +/**\n> + * \\brief AF reset\n> + * \\param[in] context The shared IPA context\n> + *\n> + */\n> +void Af::af_reset(IPAContext &context)\n> +{\n> +\tcontext.frameContext.af.maxVariance = 0;\n> +\tcontext.frameContext.af.focus = 0;\n> +\tfocus_ = 0;\n> +\tcontext.frameContext.af.stable = false;\n> +\tignoreFrame_ = 60;\n\nWhy 60 after reset and 10 at configure ? It seems to be a lot ?\n\n> +\tpreviousVariance_ = 0.0;\n> +\tpass1Done_ = false;\n> +\tpass2Done_ = false;\n> +\tmaxStep_ = MaxFocusSteps_;\n> +}\n> +\n> +/**\n> + * \\brief AF scan\n> + * \\param[in] context The shared IPA context\n> + *\n> + * \\return True, if it finds a AF value.\n> + */\n> +bool Af::af_scan(IPAContext &context, int min_step)\n> +{\n> +\t/* find the maximum variance during the AF scan using a greedy strategy */\n> +\tif (currentVariance_ > context.frameContext.af.maxVariance) {\n> +\t\tcontext.frameContext.af.maxVariance = currentVariance_;\n> +\t\tgoodFocus_ = focus_;\n> +\t}\n> +\n> +\tif (focus_ > maxStep_) {\n> +\t\t/* if reach the max step, move lens to the position and set \"focus stable\". */\n> +\t\tcontext.frameContext.af.focus = goodFocus_;\n> +\t\treturn true;\n> +\t} else {\n> +\t\t/* check negative gradient */\n> +\t\tif ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n\nWhere is the 0.15 coming from ?\n\n> +\t\t\tfocus_ += min_step;\n> +\t\t\tcontext.frameContext.af.focus = focus_;\n> +\t\t} else {\n> +\t\t\tcontext.frameContext.af.focus = goodFocus_;\n> +\t\t\tpreviousVariance_ = currentVariance_;\n> +\t\t\treturn true;\n> +\t\t}\n> +\t}\n> +\tLOG(IPU3Af, Debug) << \"Variance prevrious: \"\ns/prevrious/previous/\n\n> +\t\t\t << previousVariance_\n> +\t\t\t << \" current: \"\n> +\t\t\t << currentVariance_\n> +\t\t\t << \" Diff: \"\n> +\t\t\t << (currentVariance_ - context.frameContext.af.maxVariance);\n\nInterestingly, it seems there is an issue here, according to the log I get ?\n\n[0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0 \ncurrent: 20012.4 Diff: 0\n\nShouldn't diff be 20012.4 ?\n\nI am not sure the algorithm is very strong though... You may never \nconverge and stay in a local minimum without knowing ?\n\n> +\tpreviousVariance_ = currentVariance_;\n> +\tLOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n> +\t\t\t << context.frameContext.af.maxVariance\n> +\t\t\t << \" Focus step is \"\n> +\t\t\t << goodFocus_\n> +\t\t\t << \" Current scan is \"\n> +\t\t\t << focus_;\n> +\treturn false;\n> +}\n\nI am not sure to understand how this function should behave... Is it a \nminimum searching ? Are you looking for argmin(currentVariance - \nmaxVariance) ?\n\n> +\n> +/**\n> + * \\brief Determine the max contrast image and lens position. y_table is the\n> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n> + * value. High pass filtered value also represents the sharpness of the image.\n> + * Based on this, if the image with highest variance of the high pass filtered\n> + * value (contrast) during the AF scan, the position of the len should be the\n> + * best focus.\n> + * \\param[in] context The shared IPA context.\n> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n> + */\n> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> +{\n> +\tuint32_t total = 0;\n> +\tdouble mean;\n> +\tuint64_t var_sum = 0;\n> +\ty_table_item_t *y_item;\n> +\tint z = 0;\n> +\n> +\ty_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> +\n> +\t/**\n> +\t * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n> +\t * for a given grid.\n> +\t * For pass1: low pass results are used.\n> +\t * For pass2: high pass results are used.\n> +\t */\n> +\tif (pass1Done_) {\n> +\t\tfor (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> +\t\t\ttotal = total + y_item[z].y2_avg;\n> +\t\t\tif (y_item[z].y2_avg == 0)\n> +\t\t\t\tbreak;\n> +\t\t}\n> +\t\tmean = total / z;\n> +\n> +\t\tfor (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n> +\t\t\tvar_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n> +\t\t\tif (y_item[z].y2_avg == 0)\n> +\t\t\t\tbreak;\n> +\t\t}\n> +\t} else {\n\n/* Calculate the mean of the low pass filter values. */\n\n> +\t\tfor (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> +\t\t\ttotal = total + y_item[z].y1_avg;\n> +\t\t\tif (y_item[z].y1_avg == 0)\n> +\t\t\t\tbreak;\nAre you sure of that ? Shouldn't you use the grid you set to parse the \nright number of data ? If you have a local dark area you may stop even \nif it is not yet the end of the grid ?\n\n> +\t\t}\n> +\t\tmean = total / z;\n> +\n\n/* Calulate the deviation from the mean. */\n\n> +\t\tfor (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n> +\t\t\tvar_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n> +\t\t\tif (y_item[z].y1_avg == 0)\n> +\t\t\t\tbreak;\n> +\t\t}\n> +\t}\n\nThe two if() blocks are very similar, the only exception beeing the item \nof the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg \nin the second one). This is not easy to read (according to me).\n\n> +\t/* Determine the average variance of the frame. */\n> +\tcurrentVariance_ = static_cast(var_sum) / static_cast(z);\n\nSide note:\nIf my stats are still correct, I think it should be divided by (z-1) and \nnot z, as we have a sample and not a full population ?\n\n> +\tLOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n> +\n> +\tif (context.frameContext.af.stable == true) {\n> +\t\tconst uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n> +\t\tconst double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> +\t\tLOG(IPU3Af, Debug) << \"Change ratio: \"\n> +\t\t\t\t << var_ratio\n> +\t\t\t\t << \" current focus: \"\n> +\t\t\t\t << context.frameContext.af.focus;\n> +\t\t/**\n> +\t\t * If the change ratio of contrast is over Maxchange_ (out of focus),\n> +\t\t * trigger AF again.\n> +\t\t */\n> +\t\tif (var_ratio > MaxChange_) {\n> +\t\t\tif (ignoreFrame_ == 0) {\n> +\t\t\t\taf_reset(context);\n> +\t\t\t} else\n> +\t\t\t\tignoreFrame_--;\n> +\t\t} else\n> +\t\t\tignoreFrame_ = 10;\n> +\t} else {\n> +\t\tif (ignoreFrame_ != 0)\n> +\t\t\tignoreFrame_--;\n> +\t\telse {\n> +\t\t\taf_coarse_scan(context);\n> +\t\t\taf_fine_scan(context);\n> +\t\t}\n> +\t}\n> +}\n> +\n> +} /* namespace ipa::ipu3::algorithms */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> new file mode 100644\n> index 00000000..b9295b19\n> --- /dev/null\n> +++ b/src/ipa/ipu3/algorithms/af.h\n> @@ -0,0 +1,66 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2021, Red Hat\n> + *\n> + * af.h - IPU3 Af control\n> + */\n> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"algorithm.h\"\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa::ipu3::algorithms {\n> +\n> +class Af : public Algorithm\n> +{\n> +\t/* The format of y_table. From ipu3-ipa repo */\n> +\ttypedef struct y_table_item {\n> +\t\tuint16_t y1_avg;\n> +\t\tuint16_t y2_avg;\n> +\t} y_table_item_t;\n> +\n> +public:\n> +\tAf();\n> +\t~Af();\n> +\n> +\tvoid prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> +\tint configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> +\tvoid process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> +\n> +private:\n> +\tvoid af_coarse_scan(IPAContext &context);\n> +\tvoid af_fine_scan(IPAContext &context);\n> +\tbool af_scan(IPAContext &context, int min_step);\n> +\tvoid af_reset(IPAContext &context);\n> +\n> +\t/* Used for focus scan. */\n> +\tuint32_t focus_;\n> +\t/* Focus good */\n> +\tuint32_t goodFocus_;\n> +\t/* Recent AF statistic variance. */\n> +\tdouble currentVariance_;\n> +\t/* The frames to be ignore before starting measuring. */\n> +\tuint32_t ignoreFrame_;\n> +\t/* previous variance. it is used to determine the gradient */\n> +\tdouble previousVariance_;\n> +\t/* Max scan steps of each pass of AF scaning */\n> +\tuint32_t maxStep_;\n> +\t/* Pass 1 stable. Complete low pass search (coarse) scan) */\n> +\tbool pass1Done_;\n> +\t/* Pass 2 stable. Complete high pass scan (fine scan) */\n> +\tbool pass2Done_;\n> +};\n> +\n> +} /* namespace ipa::ipu3::algorithms */\n> +\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> index 4db6ae1d..e1099169 100644\n> --- a/src/ipa/ipu3/algorithms/meson.build\n> +++ b/src/ipa/ipu3/algorithms/meson.build\n> @@ -1,8 +1,9 @@\n> # SPDX-License-Identifier: CC0-1.0\n> \n> ipu3_ipa_algorithms = files([\n> + 'af.cpp',\n> 'agc.cpp',\n> 'awb.cpp',\n> 'blc.cpp',\n> - 'tone_mapping.cpp',\n> + 'tone_mapping.cpp'\n> ])\n> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> index 86794ac1..ee644d3c 100644\n> --- a/src/ipa/ipu3/ipa_context.cpp\n> +++ b/src/ipa/ipu3/ipa_context.cpp\n> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n> *\n> * \\var IPASessionConfiguration::grid.stride\n> * \\brief Number of cells on one line including the ImgU padding\n> + *\n> + */\n> +\n> +/**\n> + * \\var IPASessionConfiguration::af\n> + * \\brief AF parameters configuration of the IPA\n> + *\n> + * \\var IPASessionConfiguration::af.start_x\n> + * \\brief The start X position of the AF area\n> + *\n> + * \\var IPASessionConfiguration::af.start_y\n> + * \\brief The start Y position of the AF area\n> + */\n> +\n> +/**\n> + * \\var IPAFrameContext::af\n> + * \\brief Context for the Automatic Focus algorithm\n> + *\n> + * \\struct IPAFrameContext::af\n> + * \\var IPAFrameContext::af.focus\n> + * \\brief Current position of the lens\n> + *\n> + * \\var IPAFrameContext::af.maxVariance\n> + * \\brief The maximum variance of the current image.\n> + *\n> + * \\var IPAFrameContext::af.stable\n> + * \\brief is the image focused?\n> */\n> \n> /**\n> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> index c6dc0814..aa5bf97f 100644\n> --- a/src/ipa/ipu3/ipa_context.h\n> +++ b/src/ipa/ipu3/ipa_context.h\n> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n> \t\tdouble minAnalogueGain;\n> \t\tdouble maxAnalogueGain;\n> \t} agc;\n> +\n> +\tstruct {\n> +\t\tuint16_t start_x;\n> +\t\tuint16_t start_y;\n> +\t} af;\n> };\n> \n> struct IPAFrameContext {\n> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n> \t\tdouble temperatureK;\n> \t} awb;\n> \n> +\tstruct {\n> +\t\tuint32_t focus;\n> +\t\tdouble maxVariance;\n> +\t\tbool stable;\n> +\t} af;\n> +\n> \tstruct {\n> \t\tuint32_t exposure;\n> \t\tdouble gain;\n> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> index 3d307708..93966c6f 100644\n> --- a/src/ipa/ipu3/ipu3.cpp\n> +++ b/src/ipa/ipu3/ipu3.cpp\n> @@ -30,6 +30,7 @@\n> \n> #include \"libcamera/internal/mapped_framebuffer.h\"\n> \n> +#include \"algorithms/af.h\"\n> #include \"algorithms/agc.h\"\n> #include \"algorithms/algorithm.h\"\n> #include \"algorithms/awb.h\"\n> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> \t}\n> \n> \t/* Construct our Algorithms */\n> +\talgorithms_.push_back(std::make_unique());\n> \talgorithms_.push_back(std::make_unique());\n> \talgorithms_.push_back(std::make_unique());\n> \talgorithms_.push_back(std::make_unique());\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 13F1DBDB13\n\tfor ;\n\tThu, 16 Dec 2021 15:32:09 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 22DBF60868;\n\tThu, 16 Dec 2021 16:32:08 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 01B8260868\n\tfor ;\n\tThu, 16 Dec 2021 16:32:06 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:bb3:626f:be1d:4ad7] (unknown\n\t[IPv6:2a01:e0a:169:7140:bb3:626f:be1d:4ad7])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 96DE03F6;\n\tThu, 16 Dec 2021 16:32:06 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"jfRZ52Nk\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1639668726;\n\tbh=z6/cxvg06GD65GjXuiv0DlGI+xg1G5bR6IhU1Rali8g=;\n\th=Date:Subject:To:References:From:In-Reply-To:From;\n\tb=jfRZ52NkN7sWPqyk6rQfg8fV3rN/HGrRTbyL831S3BlPELBFHSqxhuD8+Y5a8lBGL\n\tfPf74RLuq/9pzxymvPXNYbuOPZinMCjqgm8D4cf7YxF3Wb4FCwuStuPjNtd2an7+nw\n\tjMotznlrUosYApGF2boMmJ0nsQpV6R9cm5kKvpHA=","Message-ID":"<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>","Date":"Thu, 16 Dec 2021 16:32:04 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.3.1","Content-Language":"en-US","To":"Kate Hsuan ,\n\tlibcamera devel ","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>","From":"Jean-Michel Hautbois ","In-Reply-To":"<20211213122847.26260-2-hpa@redhat.com>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21816,"web_url":"https://patchwork.libcamera.org/comment/21816/","msgid":"","date":"2021-12-17T08:57:12","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"\"Hi Jean-Michel,\n\nThank you for your review.\n\nOn Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n wrote:\n>\n> Hi Kate,\n>\n> Thanks for the patch :-) !\n>\n> I have a few comments below, and I have to tell you I couldn't make it\n> work here... I don't know how you are testing it ? I can see the lens\n> position changing but it passes over the correct focus and never locks\n> on it.\n>\n> On 13/12/2021 13:28, Kate Hsuan wrote:\n> > Since VCM for surface Go 2 (dw9719) had been successfully\n> > driven, this Af module can be used to control the VCM and\n> > determine the focus value based on the IPU3 AF state.\n> >\n> > The variance of each focus step is determined and a greedy\n> > approah is used to find the maximum variance of the AF\n> > state and a appropriate focus value.\n> >\n> > Changes in v2:\n> > 1. Add grid configuration interface.\n> > 2. Add AF accelerator configuration.\n> > 3. Move default focus area to the center of the image.\n> >\n> > Changes in v3:\n> > 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n> > algorithm.\n> > 2. Fix greedy AF algorithm since V4l2 interface had been\n> > removed.\n> > 3. Simplify default grid config for AF.\n> > 4. AF start area is determined by bdsOutputSize.\n> >\n> > Changes in v4:\n> > In v4, it significant improves the AF scan time using a two pass\n> > scaning method and the AF scan will be terminated when it finds\n> > a negative gradient.\n> > 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n> > the AF success rate.\n> > 2. The low pass filter convolution results are used to determine\n> > a coarse AF result.\n> > 3. The high pass convolution result is used to find a good lens\n> > step in a given AF range determined in pass 1.\n> >\n>\n> The lines above describe the changes between the versions, which is\n> great ! But those should go after the \"---\" as we don't want those in\n> the final commit message.\n\nOk. Thank you I'll move the changes after \"---\".\n\n>\n> > Signed-off-by: Kate Hsuan \n> > ---\n>\n> Changes in ...\n>\n> ---\n> > src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n> > src/ipa/ipu3/algorithms/af.h | 66 ++++++\n> > src/ipa/ipu3/algorithms/meson.build | 3 +-\n> > src/ipa/ipu3/ipa_context.cpp | 27 +++\n> > src/ipa/ipu3/ipa_context.h | 11 +\n> > src/ipa/ipu3/ipu3.cpp | 2 +\n> > 6 files changed, 442 insertions(+), 1 deletion(-)\n> > create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> > create mode 100644 src/ipa/ipu3/algorithms/af.h\n> >\n>\n> I applied your patch, and I have compilation issues:\n> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n> 89 | };\n> | ^\n> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> member ‘ipu3_uapi_grid_config::height_per_slice’\n> [-Werror=missing-field-initializers]\n> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n> const ipu3_uapi_stats_3a*)’:\n> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n> value [-Werror=address-of-packed-member]\n> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n>\n> I solved them locally (with a quick and dirty patch) how did you not\n> have those ?\n\nIt was my fault. I can build them successfully before. But, after\nre-cloning the git repo and patching my v4 patch, it happened. I'll\nfix those issues.\nSorry about the inconvenience. T_T\n\n>\n> > diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> > new file mode 100644\n> > index 00000000..620be9fa\n> > --- /dev/null\n> > +++ b/src/ipa/ipu3/algorithms/af.cpp\n> > @@ -0,0 +1,334 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2021, Red Hat\n> > + *\n> > + * af.cpp - IPU3 auto focus control\n> > + */\n> > +\n> > +#include \"af.h\"\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \"libipa/histogram.h\"\n> > +\n> > +/**\n> > + * \\file af.h\n> > + */\n> > +\n> > +namespace libcamera {\n> > +\n> > +using namespace std::literals::chrono_literals;\n> > +\n> > +namespace ipa::ipu3::algorithms {\n> > +\n> > +/**\n> > + * \\class Af\n> > + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n> > + *\n> > + * This algorithm is used to determine the position of the lens and get a\n> > + * focused image. The IPU3 AF accelerator computes the statistics, composed\n> > + * by high pass and low pass filtered value and stores in a AF buffer.\n> > + * Typically, for a focused image, it has relative high contrast than a\n> > + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n> > + * highest contrast can be found from the AF scan, the lens' position is the\n> > + * best step of the focus.\n> > + *\n> > + */\n> > +\n> > +LOG_DEFINE_CATEGORY(IPU3Af)\n> > +\n> > +/**\n> > + * Maximum focus value of the VCM control\n> > + * \\todo should be obtained from the VCM driver\n> > + */\n> > +static constexpr uint32_t MaxFocusSteps_ = 1023;\n> > +\n> > +/* minimum focus step for searching appropriate focus*/\n> > +static constexpr uint32_t coarseSearchStep_ = 10;\n> > +static constexpr uint32_t fineSearchStep_ = 1;\n> > +\n> > +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n> > +static constexpr double MaxChange_ = 0.8;\n>\n> Why 0.8 ? Any explanation on how to set it ?\n\nOK. I'll put my explanation here. Basically, it is used to represent\nthe boundary of a focused and blurred image.\n\n\n>\n> > +\n> > +/* settings for Auto Focus from the kernel */\n> > +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> > + .filter_config = {\n> > + { 0, 0, 0, 0 },\n> > + { 0, 0, 0, 0 },\n> > + { 0, 0, 0, 128 },\n> > + 0,\n> > + { 0, 0, 0, 0 },\n> > + { 0, 0, 0, 0 },\n> > + { 0, 0, 0, 128 },\n> > + 0,\n> > + .y_calc = { 8, 8, 8, 8 },\n> > + .nf = { 0, 7, 0, 7, 0 },\n> > + },\n> > + .grid_cfg = {\n> > + .width = 16,\n> > + .height = 16,\n> > + .block_width_log2 = 3,\n> > + .block_height_log2 = 3,\n> > + .x_start = 10,\n> > + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> > + },\n> > +};\n>\n> As mentionned, you missed some fields:\n\nI'll fix this. Thank you.\n\n>\n> '''\n> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n> imgu_css_af_defaults = {\n> .y_calc = { 8, 8, 8, 8 },\n> .nf = { 0, 7, 0, 7, 0 },\n> },\n> + .padding = { 0, 0, 0, 0 },\n> .grid_cfg = {\n> .width = 16,\n> .height = 16,\n> .block_width_log2 = 3,\n> .block_height_log2 = 3,\n> + .height_per_slice = 1,\n> .x_start = 10,\n> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> + .x_end = 138,\n> + .y_end = 130,\n> },\n> };\n> '''\n>\n> > +\n> > +Af::Af()\n> > + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> > + pass1Done_(false), pass2Done_(false)\n> > +{\n> > + maxStep_ = MaxFocusSteps_;\n> > +}\n> > +\n> > +Af::~Af()\n> > +{\n> > +}\n> > +\n> > +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n>\n> Missing documentation for prepare() => you should build the doc ;-)\n\nOk. I'll add the prepare() document here.\n\n>\n> > +{\n> > + params->use.acc_af = 1;\n> > + params->acc_param.af = imgu_css_af_defaults;\n> > + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n> > + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n>\n> You are never changing the grid size, is it intended ?\n\nDo you mean \"ipu3_uapi_grid_config\"? Could you please explain this\nmore detail? I can modify this according to your opinions.\n\n>\n> > +}\n> > +\n> > +/**\n> > + * \\brief Configure the Af given a configInfo\n> > + * \\param[in] context The shared IPA context\n> > + * \\param[in] configInfo The IPA configuration data\n> > + *\n> > + * \\return 0\n> > + */\n> > +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> > +{\n> > + /* determined focus value i.e. current focus value */\n> > + context.frameContext.af.focus = 0;\n> > + /* maximum variance of the AF statistics */\n> > + context.frameContext.af.maxVariance = 0;\n> double type, maybe s/= 0/= 0.0/ ?\n\n0.0 is correct. I'll correct this thank you.\n\n>\n> > + /* is focused? if it is true, the AF should be in a stable state. */\n> > + context.frameContext.af.stable = false;\n> > + /* frame to be ignored before start to estimate AF variance. */\n> > + ignoreFrame_ = 10;\n>\n> Could be a constexpr ?\n\nIt is a counter to ignore frames since we have to ignore some frames\nto wait for the value from the accelerator to be stable.\nIf this should be a constant variable, I can modify this. :)\n\n\n>\n> > +\n> > + /*\n> > + * AF default area configuration\n> > + * Move AF area to the center of the image.\n> > + */\n> > + /* Default AF width is 16x8 = 128 */\n> > + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n>\n> Why are you using a fixed value for the grid ? You could use the value\n> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n>\n> > + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n> > +\n> > + LOG(IPU3Af, Debug) << \"BDS X: \"\n> > + << configInfo.bdsOutputSize.width\n> > + << \" Y: \"\n> > + << configInfo.bdsOutputSize.height;\n> > + LOG(IPU3Af, Debug) << \"AF start from X: \"\n> > + << context.configuration.af.start_x\n> > + << \" Y: \"\n> > + << context.configuration.af.start_y;\n> > +\n> > + return 0;\n> > +}\n> > +\n> > +/**\n> > + * \\brief AF coarse scan\n> > + * \\param[in] context The shared IPA context\n> > + *\n> > + */\n> > +void Af::af_coarse_scan(IPAContext &context)\n>\n> Bad naming (camelCase only). Same below.\n\nOps, sure, I'll modify this.\n\n>\n> > +{\n> > + if (pass1Done_ == true)\n> > + return;\n> > +\n> > + if (af_scan(context, coarseSearchStep_)) {\n> > + pass1Done_ = true;\n> > + context.frameContext.af.maxVariance = 0;\n> > + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n> > + context.frameContext.af.focus = focus_;\n> > + previousVariance_ = 0;\n> > + maxStep_ = focus_ + (focus_ * 0.2);\n> > + }\n> > +}\n> > +\n> > +/**\n> > + * \\brief AF fine scan\n> > + * \\param[in] context The shared IPA context\n> > + *\n> > + */\n> > +void Af::af_fine_scan(IPAContext &context)\n> > +{\n> > + if (pass1Done_ != true)\n> > + return;\n> > +\n> > + if (af_scan(context, fineSearchStep_)) {\n> > + context.frameContext.af.stable = true;\n> > + pass2Done_ = true;\n> > + }\n> > +}\n> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n> seee where it is used...\n\nSince we used two stage AF searching strategy I simply say this is a\ntwo pass algorithm :)\nI'll renaming this.\n\n>\n> > +\n> > +/**\n> > + * \\brief AF reset\n> > + * \\param[in] context The shared IPA context\n> > + *\n> > + */\n> > +void Af::af_reset(IPAContext &context)\n> > +{\n> > + context.frameContext.af.maxVariance = 0;\n> > + context.frameContext.af.focus = 0;\n> > + focus_ = 0;\n> > + context.frameContext.af.stable = false;\n> > + ignoreFrame_ = 60;\n>\n> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n\nWe can tweak this value.\n\n>\n> > + previousVariance_ = 0.0;\n> > + pass1Done_ = false;\n> > + pass2Done_ = false;\n> > + maxStep_ = MaxFocusSteps_;\n> > +}\n> > +\n> > +/**\n> > + * \\brief AF scan\n> > + * \\param[in] context The shared IPA context\n> > + *\n> > + * \\return True, if it finds a AF value.\n> > + */\n> > +bool Af::af_scan(IPAContext &context, int min_step)\n> > +{\n> > + /* find the maximum variance during the AF scan using a greedy strategy */\n> > + if (currentVariance_ > context.frameContext.af.maxVariance) {\n> > + context.frameContext.af.maxVariance = currentVariance_;\n> > + goodFocus_ = focus_;\n> > + }\n> > +\n> > + if (focus_ > maxStep_) {\n> > + /* if reach the max step, move lens to the position and set \"focus stable\". */\n> > + context.frameContext.af.focus = goodFocus_;\n> > + return true;\n> > + } else {\n> > + /* check negative gradient */\n> > + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n>\n> Where is the 0.15 coming from ?\n\nAfter some testing, I think that If the negative gradient ismore than\n15% maxVaraiance, the focused image and lens position we get!\nI could put this as a constant variable in my v5 patch.\n\n>\n> > + focus_ += min_step;\n> > + context.frameContext.af.focus = focus_;\n> > + } else {\n> > + context.frameContext.af.focus = goodFocus_;\n> > + previousVariance_ = currentVariance_;\n> > + return true;\n> > + }\n> > + }\n> > + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n> s/prevrious/previous/\n\nOps. sorry about the typo. T_T\n\n>\n> > + << previousVariance_\n> > + << \" current: \"\n> > + << currentVariance_\n> > + << \" Diff: \"\n> > + << (currentVariance_ - context.frameContext.af.maxVariance);\n>\n> Interestingly, it seems there is an issue here, according to the log I get ?\n>\n> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n> current: 20012.4 Diff: 0\n>\n> Shouldn't diff be 20012.4 ?\n>\n> I am not sure the algorithm is very strong though... You may never\n> converge and stay in a local minimum without knowing ?\n>\n> > + previousVariance_ = currentVariance_;\n> > + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n> > + << context.frameContext.af.maxVariance\n> > + << \" Focus step is \"\n> > + << goodFocus_\n> > + << \" Current scan is \"\n> > + << focus_;\n> > + return false;\n> > +}\n>\n> I am not sure to understand how this function should behave... Is it a\n> minimum searching ? Are you looking for argmin(currentVariance -\n> maxVariance) ?\n\nActually. it is a maximum search. I try to search for the maximum\nvariance of the image.\nIf the difference is positive, it means the variance still increases.\nWe can continue search for the maximum value and update maxVariance_.\nUntil, we found a negative difference which means the variance starts\nto go down and image will become blurred.\n\n\n>\n> > +\n> > +/**\n> > + * \\brief Determine the max contrast image and lens position. y_table is the\n> > + * statictic data from IPU3 and is composed of low pass and high pass filtered\n> > + * value. High pass filtered value also represents the sharpness of the image.\n> > + * Based on this, if the image with highest variance of the high pass filtered\n> > + * value (contrast) during the AF scan, the position of the len should be the\n> > + * best focus.\n> > + * \\param[in] context The shared IPA context.\n> > + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n> > + */\n> > +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> > +{\n> > + uint32_t total = 0;\n> > + double mean;\n> > + uint64_t var_sum = 0;\n> > + y_table_item_t *y_item;\n> > + int z = 0;\n> > +\n> > + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> > +\n> > + /**\n> > + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n> > + * for a given grid.\n> > + * For pass1: low pass results are used.\n> > + * For pass2: high pass results are used.\n> > + */\n> > + if (pass1Done_) {\n> > + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> > + total = total + y_item[z].y2_avg;\n> > + if (y_item[z].y2_avg == 0)\n> > + break;\n> > + }\n> > + mean = total / z;\n> > +\n> > + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n> > + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n> > + if (y_item[z].y2_avg == 0)\n> > + break;\n> > + }\n> > + } else {\n>\n> /* Calculate the mean of the low pass filter values. */\n>\n> > + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> > + total = total + y_item[z].y1_avg;\n> > + if (y_item[z].y1_avg == 0)\n> > + break;\n> Are you sure of that ? Shouldn't you use the grid you set to parse the\n> right number of data ? If you have a local dark area you may stop even\n> if it is not yet the end of the grid ?\n\nI'll find another way to determine the range of the results. May be it\ncan based on the grids setting.\n\n>\n> > + }\n> > + mean = total / z;\n> > +\n>\n> /* Calulate the deviation from the mean. */\n>\n> > + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n> > + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n> > + if (y_item[z].y1_avg == 0)\n> > + break;\n> > + }\n> > + }\n>\n> The two if() blocks are very similar, the only exception beeing the item\n> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n> in the second one). This is not easy to read (according to me).\n\nThey are high pass (y2) and low pass (y1) filtered convolution\nresults. This structure referees from ipu-ipa repo and I would like to\nkeep their original naming.\n\n\n>\n> > + /* Determine the average variance of the frame. */\n> > + currentVariance_ = static_cast(var_sum) / static_cast(z);\n>\n> Side note:\n> If my stats are still correct, I think it should be divided by (z-1) and\n> not z, as we have a sample and not a full population ?\n\nit is z-1 and my fault. I'll correct this. :)\n\n>\n> > + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n> > +\n> > + if (context.frameContext.af.stable == true) {\n> > + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n> > + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> > + LOG(IPU3Af, Debug) << \"Change ratio: \"\n> > + << var_ratio\n> > + << \" current focus: \"\n> > + << context.frameContext.af.focus;\n> > + /**\n> > + * If the change ratio of contrast is over Maxchange_ (out of focus),\n> > + * trigger AF again.\n> > + */\n> > + if (var_ratio > MaxChange_) {\n> > + if (ignoreFrame_ == 0) {\n> > + af_reset(context);\n> > + } else\n> > + ignoreFrame_--;\n> > + } else\n> > + ignoreFrame_ = 10;\n> > + } else {\n> > + if (ignoreFrame_ != 0)\n> > + ignoreFrame_--;\n> > + else {\n> > + af_coarse_scan(context);\n> > + af_fine_scan(context);\n> > + }\n> > + }\n> > +}\n> > +\n> > +} /* namespace ipa::ipu3::algorithms */\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> > new file mode 100644\n> > index 00000000..b9295b19\n> > --- /dev/null\n> > +++ b/src/ipa/ipu3/algorithms/af.h\n> > @@ -0,0 +1,66 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2021, Red Hat\n> > + *\n> > + * af.h - IPU3 Af control\n> > + */\n> > +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> > +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> > +\n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \"algorithm.h\"\n> > +\n> > +namespace libcamera {\n> > +\n> > +namespace ipa::ipu3::algorithms {\n> > +\n> > +class Af : public Algorithm\n> > +{\n> > + /* The format of y_table. From ipu3-ipa repo */\n> > + typedef struct y_table_item {\n> > + uint16_t y1_avg;\n> > + uint16_t y2_avg;\n> > + } y_table_item_t;\n> > +\n> > +public:\n> > + Af();\n> > + ~Af();\n> > +\n> > + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> > + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> > + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> > +\n> > +private:\n> > + void af_coarse_scan(IPAContext &context);\n> > + void af_fine_scan(IPAContext &context);\n> > + bool af_scan(IPAContext &context, int min_step);\n> > + void af_reset(IPAContext &context);\n> > +\n> > + /* Used for focus scan. */\n> > + uint32_t focus_;\n> > + /* Focus good */\n> > + uint32_t goodFocus_;\n> > + /* Recent AF statistic variance. */\n> > + double currentVariance_;\n> > + /* The frames to be ignore before starting measuring. */\n> > + uint32_t ignoreFrame_;\n> > + /* previous variance. it is used to determine the gradient */\n> > + double previousVariance_;\n> > + /* Max scan steps of each pass of AF scaning */\n> > + uint32_t maxStep_;\n> > + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n> > + bool pass1Done_;\n> > + /* Pass 2 stable. Complete high pass scan (fine scan) */\n> > + bool pass2Done_;\n> > +};\n> > +\n> > +} /* namespace ipa::ipu3::algorithms */\n> > +\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n> > diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> > index 4db6ae1d..e1099169 100644\n> > --- a/src/ipa/ipu3/algorithms/meson.build\n> > +++ b/src/ipa/ipu3/algorithms/meson.build\n> > @@ -1,8 +1,9 @@\n> > # SPDX-License-Identifier: CC0-1.0\n> >\n> > ipu3_ipa_algorithms = files([\n> > + 'af.cpp',\n> > 'agc.cpp',\n> > 'awb.cpp',\n> > 'blc.cpp',\n> > - 'tone_mapping.cpp',\n> > + 'tone_mapping.cpp'\n> > ])\n> > diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> > index 86794ac1..ee644d3c 100644\n> > --- a/src/ipa/ipu3/ipa_context.cpp\n> > +++ b/src/ipa/ipu3/ipa_context.cpp\n> > @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n> > *\n> > * \\var IPASessionConfiguration::grid.stride\n> > * \\brief Number of cells on one line including the ImgU padding\n> > + *\n> > + */\n> > +\n> > +/**\n> > + * \\var IPASessionConfiguration::af\n> > + * \\brief AF parameters configuration of the IPA\n> > + *\n> > + * \\var IPASessionConfiguration::af.start_x\n> > + * \\brief The start X position of the AF area\n> > + *\n> > + * \\var IPASessionConfiguration::af.start_y\n> > + * \\brief The start Y position of the AF area\n> > + */\n> > +\n> > +/**\n> > + * \\var IPAFrameContext::af\n> > + * \\brief Context for the Automatic Focus algorithm\n> > + *\n> > + * \\struct IPAFrameContext::af\n> > + * \\var IPAFrameContext::af.focus\n> > + * \\brief Current position of the lens\n> > + *\n> > + * \\var IPAFrameContext::af.maxVariance\n> > + * \\brief The maximum variance of the current image.\n> > + *\n> > + * \\var IPAFrameContext::af.stable\n> > + * \\brief is the image focused?\n> > */\n> >\n> > /**\n> > diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> > index c6dc0814..aa5bf97f 100644\n> > --- a/src/ipa/ipu3/ipa_context.h\n> > +++ b/src/ipa/ipu3/ipa_context.h\n> > @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n> > double minAnalogueGain;\n> > double maxAnalogueGain;\n> > } agc;\n> > +\n> > + struct {\n> > + uint16_t start_x;\n> > + uint16_t start_y;\n> > + } af;\n> > };\n> >\n> > struct IPAFrameContext {\n> > @@ -49,6 +54,12 @@ struct IPAFrameContext {\n> > double temperatureK;\n> > } awb;\n> >\n> > + struct {\n> > + uint32_t focus;\n> > + double maxVariance;\n> > + bool stable;\n> > + } af;\n> > +\n> > struct {\n> > uint32_t exposure;\n> > double gain;\n> > diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> > index 3d307708..93966c6f 100644\n> > --- a/src/ipa/ipu3/ipu3.cpp\n> > +++ b/src/ipa/ipu3/ipu3.cpp\n> > @@ -30,6 +30,7 @@\n> >\n> > #include \"libcamera/internal/mapped_framebuffer.h\"\n> >\n> > +#include \"algorithms/af.h\"\n> > #include \"algorithms/agc.h\"\n> > #include \"algorithms/algorithm.h\"\n> > #include \"algorithms/awb.h\"\n> > @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> > }\n> >\n> > /* Construct our Algorithms */\n> > + algorithms_.push_back(std::make_unique());\n> > algorithms_.push_back(std::make_unique());\n> > algorithms_.push_back(std::make_unique());\n> > algorithms_.push_back(std::make_unique());\n> >\n>\n\nThank you :)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 31BD2BDB13\n\tfor ;\n\tFri, 17 Dec 2021 08:57:32 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8D85C608A5;\n\tFri, 17 Dec 2021 09:57:31 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6F55360223\n\tfor ;\n\tFri, 17 Dec 2021 09:57:29 +0100 (CET)","from mail-lf1-f71.google.com (mail-lf1-f71.google.com\n\t[209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-311-b5ychApdMSevg0nb7bJ0VQ-1; Fri, 17 Dec 2021 03:57:26 -0500","by mail-lf1-f71.google.com with SMTP id\n\tj9-20020a05651231c900b004037efe9fddso736112lfe.18\n\tfor ;\n\tFri, 17 Dec 2021 00:57:26 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"YzDMiq4y\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1639731448;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=8ryxGIfq7EVfsFYi+z1k38BSadTd/2O7talm8bHhyY4=;\n\tb=YzDMiq4yy33v+2MzipiziH0KjWlBot4iYDbAEIoP4bpNGRYnyZ8fv0po1ztiPuIEo4g64x\n\tb2W22bkfMhrnTueto7jLrWWYRahU4veaTAczXVHeATHeA8Fzs5mIQol24pgW2np5KCtF83\n\t3lhfzbb8mFMNsKcmR+U+clAjZozgxRM=","X-MC-Unique":"b5ychApdMSevg0nb7bJ0VQ-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=8ryxGIfq7EVfsFYi+z1k38BSadTd/2O7talm8bHhyY4=;\n\tb=AKv4ptu7DMwA4skad6oS218qoIbhIEzBYd/j7LwsliakgeEuePaRGcIij6IXRQo8vi\n\tlsACy5MnAUawELvvXpo10eg6QOk3A+FAsQfYk49ly7J/HC5l5iSKCOZjKvYebw3nw2JG\n\tPtouHj8lIb3HRd0ObmOQTa+OY+A0MoHhNW9AwPkUlFuGdAwl65GvnIqFtQEHaNMbtS/3\n\tZAGrg2zvXV4ypZnp3hD/hXFSy15x9Nnr8d5ffs9hJgcGZqhqEwmw6BgR3ZoEZKc6R8+5\n\twcn7d7g9dYJNoVisFWEZrgC7xIuZNpm1baW40RSXsjRPw/AFSt20zXXk6pgLXQ1W4+1g\n\twJiQ==","X-Gm-Message-State":"AOAM533IbHMOJd1mBSF4OZzev5KqTPAVwYMdEKS3xnimp/BXDSN47D/a\n\tNqTL9EQDxPByfuseriy1LAt1XbW/Bmnql1kRRVT5its95bsKJ4GDl5kVP0RvRtjfORU6k3zS9Um\n\tQj7ZnXjdPrybTHetcZWKE4hKz20kehWZs7KjWnQaGW3LQqLTrpg==","X-Received":["by 2002:ac2:4bc1:: with SMTP id o1mr1612382lfq.384.1639731444718;\n\tFri, 17 Dec 2021 00:57:24 -0800 (PST)","by 2002:ac2:4bc1:: with SMTP id o1mr1612338lfq.384.1639731443846;\n\tFri, 17 Dec 2021 00:57:23 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJwMzjZmCDMMubH1R5R4d3GIBw+c3Biz/DUFVIAJKS3kzCFDK/8VHpM+990WMHBllYcWEbUxldKGz9OKi+s3P/M=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>","In-Reply-To":"<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>","From":"Kate Hsuan ","Date":"Fri, 17 Dec 2021 16:57:12 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21822,"web_url":"https://patchwork.libcamera.org/comment/21822/","msgid":"<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>","date":"2021-12-17T16:18:47","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate,\n\nOn 17/12/2021 09:57, Kate Hsuan wrote:\n> \"Hi Jean-Michel,\n> \n> Thank you for your review.\n> \n> On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n> wrote:\n>>\n>> Hi Kate,\n>>\n>> Thanks for the patch :-) !\n>>\n>> I have a few comments below, and I have to tell you I couldn't make it\n>> work here... I don't know how you are testing it ? I can see the lens\n>> position changing but it passes over the correct focus and never locks\n>> on it.\n>>\n>> On 13/12/2021 13:28, Kate Hsuan wrote:\n>>> Since VCM for surface Go 2 (dw9719) had been successfully\n>>> driven, this Af module can be used to control the VCM and\n>>> determine the focus value based on the IPU3 AF state.\n>>>\n>>> The variance of each focus step is determined and a greedy\n>>> approah is used to find the maximum variance of the AF\n>>> state and a appropriate focus value.\n>>>\n>>> Changes in v2:\n>>> 1. Add grid configuration interface.\n>>> 2. Add AF accelerator configuration.\n>>> 3. Move default focus area to the center of the image.\n>>>\n>>> Changes in v3:\n>>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n>>> algorithm.\n>>> 2. Fix greedy AF algorithm since V4l2 interface had been\n>>> removed.\n>>> 3. Simplify default grid config for AF.\n>>> 4. AF start area is determined by bdsOutputSize.\n>>>\n>>> Changes in v4:\n>>> In v4, it significant improves the AF scan time using a two pass\n>>> scaning method and the AF scan will be terminated when it finds\n>>> a negative gradient.\n>>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n>>> the AF success rate.\n>>> 2. The low pass filter convolution results are used to determine\n>>> a coarse AF result.\n>>> 3. The high pass convolution result is used to find a good lens\n>>> step in a given AF range determined in pass 1.\n>>>\n>>\n>> The lines above describe the changes between the versions, which is\n>> great ! But those should go after the \"---\" as we don't want those in\n>> the final commit message.\n> \n> Ok. Thank you I'll move the changes after \"---\".\n> \n>>\n>>> Signed-off-by: Kate Hsuan \n>>> ---\n>>\n>> Changes in ...\n>>\n>> ---\n>>> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n>>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n>>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n>>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n>>> src/ipa/ipu3/ipa_context.h | 11 +\n>>> src/ipa/ipu3/ipu3.cpp | 2 +\n>>> 6 files changed, 442 insertions(+), 1 deletion(-)\n>>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n>>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n>>>\n>>\n>> I applied your patch, and I have compilation issues:\n>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n>> 89 | };\n>> | ^\n>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>> member ‘ipu3_uapi_grid_config::height_per_slice’\n>> [-Werror=missing-field-initializers]\n>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n>> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n>> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n>> const ipu3_uapi_stats_3a*)’:\n>> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n>> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n>> value [-Werror=address-of-packed-member]\n>> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n>>\n>> I solved them locally (with a quick and dirty patch) how did you not\n>> have those ?\n> \n> It was my fault. I can build them successfully before. But, after\n> re-cloning the git repo and patching my v4 patch, it happened. I'll\n> fix those issues.\n> Sorry about the inconvenience. T_T\n\nNo problem, but it was surprising :-).\n\n> \n>>\n>>> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n>>> new file mode 100644\n>>> index 00000000..620be9fa\n>>> --- /dev/null\n>>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n>>> @@ -0,0 +1,334 @@\n>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>> +/*\n>>> + * Copyright (C) 2021, Red Hat\n>>> + *\n>>> + * af.cpp - IPU3 auto focus control\n>>> + */\n>>> +\n>>> +#include \"af.h\"\n>>> +\n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +\n>>> +#include \n>>> +\n>>> +#include \n>>> +\n>>> +#include \n>>> +\n>>> +#include \"libipa/histogram.h\"\n>>> +\n>>> +/**\n>>> + * \\file af.h\n>>> + */\n>>> +\n>>> +namespace libcamera {\n>>> +\n>>> +using namespace std::literals::chrono_literals;\n>>> +\n>>> +namespace ipa::ipu3::algorithms {\n>>> +\n>>> +/**\n>>> + * \\class Af\n>>> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n>>> + *\n>>> + * This algorithm is used to determine the position of the lens and get a\n>>> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n>>> + * by high pass and low pass filtered value and stores in a AF buffer.\n>>> + * Typically, for a focused image, it has relative high contrast than a\n>>> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n>>> + * highest contrast can be found from the AF scan, the lens' position is the\n>>> + * best step of the focus.\n>>> + *\n>>> + */\n>>> +\n>>> +LOG_DEFINE_CATEGORY(IPU3Af)\n>>> +\n>>> +/**\n>>> + * Maximum focus value of the VCM control\n>>> + * \\todo should be obtained from the VCM driver\n>>> + */\n>>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n>>> +\n>>> +/* minimum focus step for searching appropriate focus*/\n>>> +static constexpr uint32_t coarseSearchStep_ = 10;\n>>> +static constexpr uint32_t fineSearchStep_ = 1;\n>>> +\n>>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n>>> +static constexpr double MaxChange_ = 0.8;\n>>\n>> Why 0.8 ? Any explanation on how to set it ?\n> \n> OK. I'll put my explanation here. Basically, it is used to represent\n> the boundary of a focused and blurred image.\n> \n> \n>>\n>>> +\n>>> +/* settings for Auto Focus from the kernel */\n>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n>>> + .filter_config = {\n>>> + { 0, 0, 0, 0 },\n>>> + { 0, 0, 0, 0 },\n>>> + { 0, 0, 0, 128 },\n>>> + 0,\n>>> + { 0, 0, 0, 0 },\n>>> + { 0, 0, 0, 0 },\n>>> + { 0, 0, 0, 128 },\n>>> + 0,\n>>> + .y_calc = { 8, 8, 8, 8 },\n>>> + .nf = { 0, 7, 0, 7, 0 },\n>>> + },\n\nI would like your thinking here (and perhaps someone else). It looks \nlike the ImgU implements a Gabor filter, but I can't see all the \nparameters I would expect from it. Putting it down, I can't see with \nthis default filter how y1 and y2 could represent low-pass and high \npass, as the filters are the same. But, it could be a horizontal and a \nvertical filter instead, with y1 the horizontal and y2 the vertical \nresulting filtered values.\n\nIt would be very interesting to display the y1_avg and y2_avg values as \nan image, I just don't have lots of time for it. We have 16 bits values, \nso we could dump those into a binary file and use python to display the \nvalues as a raw gray image or something similar.\n\nThe other things which I find annoying is the non-reliable values I get \nright now for the variance. Same scene, exposure and gain fixed to a \nvalue, controled luminosity, and between two executions I might not have \nthe same result...\n\n>>> + .grid_cfg = {\n>>> + .width = 16,\n>>> + .height = 16,\n>>> + .block_width_log2 = 3,\n>>> + .block_height_log2 = 3,\n>>> + .x_start = 10,\n>>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n>>> + },\n>>> +};\n>>\n>> As mentionned, you missed some fields:\n> \n> I'll fix this. Thank you.\n> \n>>\n>> '''\n>> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n>> imgu_css_af_defaults = {\n>> .y_calc = { 8, 8, 8, 8 },\n>> .nf = { 0, 7, 0, 7, 0 },\n>> },\n>> + .padding = { 0, 0, 0, 0 },\n>> .grid_cfg = {\n>> .width = 16,\n>> .height = 16,\n>> .block_width_log2 = 3,\n>> .block_height_log2 = 3,\n>> + .height_per_slice = 1,\n>> .x_start = 10,\n>> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n>> + .x_end = 138,\n>> + .y_end = 130,\n>> },\n>> };\n>> '''\n>>\n>>> +\n>>> +Af::Af()\n>>> + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n>>> + pass1Done_(false), pass2Done_(false)\n>>> +{\n>>> + maxStep_ = MaxFocusSteps_;\n>>> +}\n>>> +\n>>> +Af::~Af()\n>>> +{\n>>> +}\n>>> +\n>>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n>>\n>> Missing documentation for prepare() => you should build the doc ;-)\n> \n> Ok. I'll add the prepare() document here.\n> \n>>\n>>> +{\n>>> + params->use.acc_af = 1;\n>>> + params->acc_param.af = imgu_css_af_defaults;\n>>> + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n>>> + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n>>\n>> You are never changing the grid size, is it intended ?\n> \n> Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n> more detail? I can modify this according to your opinions.\n\nWell, yes, you are always using the default 128x128 grid size. But you \ncan go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in \nheight. Thus, it means we could evaluate the scene on a small part or \nalmost all of it (copying the bds grid is not doable as it might have \nlarger block width, up to 80x60).\n\nSee:\nhttps://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n\nNote:\nhttps://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n\n=> It makes me wonder if the filtering is fast, or if it may take some \ntime, meaning that focus evaluation in one frame could be done only at \nframe+2 or something like that.\n\nCould you verify that the focus estimated is in the center of the scene \n right now, as it is what you have configured ?\n\n> \n>>\n>>> +}\n>>> +\n>>> +/**\n>>> + * \\brief Configure the Af given a configInfo\n>>> + * \\param[in] context The shared IPA context\n>>> + * \\param[in] configInfo The IPA configuration data\n>>> + *\n>>> + * \\return 0\n>>> + */\n>>> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n>>> +{\n>>> + /* determined focus value i.e. current focus value */\n>>> + context.frameContext.af.focus = 0;\n>>> + /* maximum variance of the AF statistics */\n>>> + context.frameContext.af.maxVariance = 0;\n>> double type, maybe s/= 0/= 0.0/ ?\n> \n> 0.0 is correct. I'll correct this thank you.\n> \n>>\n>>> + /* is focused? if it is true, the AF should be in a stable state. */\n>>> + context.frameContext.af.stable = false;\n>>> + /* frame to be ignored before start to estimate AF variance. */\n>>> + ignoreFrame_ = 10;\n>>\n>> Could be a constexpr ?\n> \n> It is a counter to ignore frames since we have to ignore some frames\n> to wait for the value from the accelerator to be stable.\n> If this should be a constant variable, I can modify this. :)\n> \n> \n>>\n>>> +\n>>> + /*\n>>> + * AF default area configuration\n>>> + * Move AF area to the center of the image.\n>>> + */\n>>> + /* Default AF width is 16x8 = 128 */\n>>> + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n>>\n>> Why are you using a fixed value for the grid ? You could use the value\n>> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n>>\n>>> + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n>>> +\n>>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n>>> + << configInfo.bdsOutputSize.width\n>>> + << \" Y: \"\n>>> + << configInfo.bdsOutputSize.height;\n>>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n>>> + << context.configuration.af.start_x\n>>> + << \" Y: \"\n>>> + << context.configuration.af.start_y;\n>>> +\n>>> + return 0;\n>>> +}\n>>> +\n>>> +/**\n>>> + * \\brief AF coarse scan\n>>> + * \\param[in] context The shared IPA context\n>>> + *\n>>> + */\n>>> +void Af::af_coarse_scan(IPAContext &context)\n>>\n>> Bad naming (camelCase only). Same below.\n> \n> Ops, sure, I'll modify this.\n> \n>>\n>>> +{\n>>> + if (pass1Done_ == true)\n>>> + return;\n>>> +\n>>> + if (af_scan(context, coarseSearchStep_)) {\n>>> + pass1Done_ = true;\n>>> + context.frameContext.af.maxVariance = 0;\n>>> + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n>>> + context.frameContext.af.focus = focus_;\n>>> + previousVariance_ = 0;\n>>> + maxStep_ = focus_ + (focus_ * 0.2);\n>>> + }\n>>> +}\n>>> +\n>>> +/**\n>>> + * \\brief AF fine scan\n>>> + * \\param[in] context The shared IPA context\n>>> + *\n>>> + */\n>>> +void Af::af_fine_scan(IPAContext &context)\n>>> +{\n>>> + if (pass1Done_ != true)\n>>> + return;\n>>> +\n>>> + if (af_scan(context, fineSearchStep_)) {\n>>> + context.frameContext.af.stable = true;\n>>> + pass2Done_ = true;\n>>> + }\n>>> +}\n>> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n>> seee where it is used...\n> \n> Since we used two stage AF searching strategy I simply say this is a\n> two pass algorithm :)\n> I'll renaming this.\n> \n>>\n>>> +\n>>> +/**\n>>> + * \\brief AF reset\n>>> + * \\param[in] context The shared IPA context\n>>> + *\n>>> + */\n>>> +void Af::af_reset(IPAContext &context)\n>>> +{\n>>> + context.frameContext.af.maxVariance = 0;\n>>> + context.frameContext.af.focus = 0;\n>>> + focus_ = 0;\n>>> + context.frameContext.af.stable = false;\n>>> + ignoreFrame_ = 60;\n>>\n>> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n> \n> We can tweak this value.\n> \n>>\n>>> + previousVariance_ = 0.0;\n>>> + pass1Done_ = false;\n>>> + pass2Done_ = false;\n>>> + maxStep_ = MaxFocusSteps_;\n>>> +}\n>>> +\n>>> +/**\n>>> + * \\brief AF scan\n>>> + * \\param[in] context The shared IPA context\n>>> + *\n>>> + * \\return True, if it finds a AF value.\n>>> + */\n>>> +bool Af::af_scan(IPAContext &context, int min_step)\n>>> +{\n>>> + /* find the maximum variance during the AF scan using a greedy strategy */\n>>> + if (currentVariance_ > context.frameContext.af.maxVariance) {\n>>> + context.frameContext.af.maxVariance = currentVariance_;\n>>> + goodFocus_ = focus_;\n>>> + }\n>>> +\n>>> + if (focus_ > maxStep_) {\n>>> + /* if reach the max step, move lens to the position and set \"focus stable\". */\n>>> + context.frameContext.af.focus = goodFocus_;\n>>> + return true;\n>>> + } else {\n>>> + /* check negative gradient */\n>>> + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n>>\n>> Where is the 0.15 coming from ?\n> \n> After some testing, I think that If the negative gradient ismore than\n> 15% maxVaraiance, the focused image and lens position we get!\n> I could put this as a constant variable in my v5 patch.\n> \n>>\n>>> + focus_ += min_step;\n>>> + context.frameContext.af.focus = focus_;\n>>> + } else {\n>>> + context.frameContext.af.focus = goodFocus_;\n>>> + previousVariance_ = currentVariance_;\n>>> + return true;\n>>> + }\n>>> + }\n>>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n>> s/prevrious/previous/\n> \n> Ops. sorry about the typo. T_T\n> \n>>\n>>> + << previousVariance_\n>>> + << \" current: \"\n>>> + << currentVariance_\n>>> + << \" Diff: \"\n>>> + << (currentVariance_ - context.frameContext.af.maxVariance);\n>>\n>> Interestingly, it seems there is an issue here, according to the log I get ?\n>>\n>> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n>> current: 20012.4 Diff: 0\n>>\n>> Shouldn't diff be 20012.4 ?\n>>\n>> I am not sure the algorithm is very strong though... You may never\n>> converge and stay in a local minimum without knowing ?\n>>\n>>> + previousVariance_ = currentVariance_;\n>>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n>>> + << context.frameContext.af.maxVariance\n>>> + << \" Focus step is \"\n>>> + << goodFocus_\n>>> + << \" Current scan is \"\n>>> + << focus_;\n>>> + return false;\n>>> +}\n>>\n>> I am not sure to understand how this function should behave... Is it a\n>> minimum searching ? Are you looking for argmin(currentVariance -\n>> maxVariance) ?\n> \n> Actually. it is a maximum search. I try to search for the maximum\n> variance of the image.\n> If the difference is positive, it means the variance still increases.\n> We can continue search for the maximum value and update maxVariance_.\n> Until, we found a negative difference which means the variance starts\n> to go down and image will become blurred.\n> \n> \n>>\n>>> +\n>>> +/**\n>>> + * \\brief Determine the max contrast image and lens position. y_table is the\n>>> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n>>> + * value. High pass filtered value also represents the sharpness of the image.\n>>> + * Based on this, if the image with highest variance of the high pass filtered\n>>> + * value (contrast) during the AF scan, the position of the len should be the\n>>> + * best focus.\n>>> + * \\param[in] context The shared IPA context.\n>>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n>>> + */\n>>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n>>> +{\n>>> + uint32_t total = 0;\n>>> + double mean;\n>>> + uint64_t var_sum = 0;\n>>> + y_table_item_t *y_item;\n>>> + int z = 0;\n>>> +\n>>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n>>> +\n>>> + /**\n>>> + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n>>> + * for a given grid.\n>>> + * For pass1: low pass results are used.\n>>> + * For pass2: high pass results are used.\n>>> + */\n>>> + if (pass1Done_) {\n>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n>>> + total = total + y_item[z].y2_avg;\n>>> + if (y_item[z].y2_avg == 0)\n>>> + break;\n>>> + }\n>>> + mean = total / z;\n>>> +\n>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n>>> + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n>>> + if (y_item[z].y2_avg == 0)\n>>> + break;\n>>> + }\n>>> + } else {\n>>\n>> /* Calculate the mean of the low pass filter values. */\n>>\n>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n>>> + total = total + y_item[z].y1_avg;\n>>> + if (y_item[z].y1_avg == 0)\n>>> + break;\n>> Are you sure of that ? Shouldn't you use the grid you set to parse the\n>> right number of data ? If you have a local dark area you may stop even\n>> if it is not yet the end of the grid ?\n> \n> I'll find another way to determine the range of the results. May be it\n> can based on the grids setting.\n> \n>>\n>>> + }\n>>> + mean = total / z;\n>>> +\n>>\n>> /* Calulate the deviation from the mean. */\n>>\n>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n>>> + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n>>> + if (y_item[z].y1_avg == 0)\n>>> + break;\n>>> + }\n>>> + }\n>>\n>> The two if() blocks are very similar, the only exception beeing the item\n>> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n>> in the second one). This is not easy to read (according to me).\n> \n> They are high pass (y2) and low pass (y1) filtered convolution\n> results. This structure referees from ipu-ipa repo and I would like to\n> keep their original naming.\n> \n> \n>>\n>>> + /* Determine the average variance of the frame. */\n>>> + currentVariance_ = static_cast(var_sum) / static_cast(z);\n>>\n>> Side note:\n>> If my stats are still correct, I think it should be divided by (z-1) and\n>> not z, as we have a sample and not a full population ?\n> \n> it is z-1 and my fault. I'll correct this. :)\n> \n>>\n>>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n>>> +\n>>> + if (context.frameContext.af.stable == true) {\n>>> + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n>>> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n>>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n>>> + << var_ratio\n>>> + << \" current focus: \"\n>>> + << context.frameContext.af.focus;\n>>> + /**\n>>> + * If the change ratio of contrast is over Maxchange_ (out of focus),\n>>> + * trigger AF again.\n>>> + */\n>>> + if (var_ratio > MaxChange_) {\n>>> + if (ignoreFrame_ == 0) {\n>>> + af_reset(context);\n>>> + } else\n>>> + ignoreFrame_--;\n>>> + } else\n>>> + ignoreFrame_ = 10;\n>>> + } else {\n>>> + if (ignoreFrame_ != 0)\n>>> + ignoreFrame_--;\n>>> + else {\n>>> + af_coarse_scan(context);\n>>> + af_fine_scan(context);\n>>> + }\n>>> + }\n>>> +}\n>>> +\n>>> +} /* namespace ipa::ipu3::algorithms */\n>>> +\n>>> +} /* namespace libcamera */\n>>> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n>>> new file mode 100644\n>>> index 00000000..b9295b19\n>>> --- /dev/null\n>>> +++ b/src/ipa/ipu3/algorithms/af.h\n>>> @@ -0,0 +1,66 @@\n>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>> +/*\n>>> + * Copyright (C) 2021, Red Hat\n>>> + *\n>>> + * af.h - IPU3 Af control\n>>> + */\n>>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n>>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n>>> +\n>>> +#include \n>>> +\n>>> +#include \n>>> +\n>>> +#include \n>>> +\n>>> +#include \"algorithm.h\"\n>>> +\n>>> +namespace libcamera {\n>>> +\n>>> +namespace ipa::ipu3::algorithms {\n>>> +\n>>> +class Af : public Algorithm\n>>> +{\n>>> + /* The format of y_table. From ipu3-ipa repo */\n>>> + typedef struct y_table_item {\n>>> + uint16_t y1_avg;\n>>> + uint16_t y2_avg;\n>>> + } y_table_item_t;\n>>> +\n>>> +public:\n>>> + Af();\n>>> + ~Af();\n>>> +\n>>> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n>>> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n>>> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n>>> +\n>>> +private:\n>>> + void af_coarse_scan(IPAContext &context);\n>>> + void af_fine_scan(IPAContext &context);\n>>> + bool af_scan(IPAContext &context, int min_step);\n>>> + void af_reset(IPAContext &context);\n>>> +\n>>> + /* Used for focus scan. */\n>>> + uint32_t focus_;\n>>> + /* Focus good */\n>>> + uint32_t goodFocus_;\n>>> + /* Recent AF statistic variance. */\n>>> + double currentVariance_;\n>>> + /* The frames to be ignore before starting measuring. */\n>>> + uint32_t ignoreFrame_;\n>>> + /* previous variance. it is used to determine the gradient */\n>>> + double previousVariance_;\n>>> + /* Max scan steps of each pass of AF scaning */\n>>> + uint32_t maxStep_;\n>>> + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n>>> + bool pass1Done_;\n>>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n>>> + bool pass2Done_;\n>>> +};\n>>> +\n>>> +} /* namespace ipa::ipu3::algorithms */\n>>> +\n>>> +} /* namespace libcamera */\n>>> +\n>>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n>>> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n>>> index 4db6ae1d..e1099169 100644\n>>> --- a/src/ipa/ipu3/algorithms/meson.build\n>>> +++ b/src/ipa/ipu3/algorithms/meson.build\n>>> @@ -1,8 +1,9 @@\n>>> # SPDX-License-Identifier: CC0-1.0\n>>>\n>>> ipu3_ipa_algorithms = files([\n>>> + 'af.cpp',\n>>> 'agc.cpp',\n>>> 'awb.cpp',\n>>> 'blc.cpp',\n>>> - 'tone_mapping.cpp',\n>>> + 'tone_mapping.cpp'\n>>> ])\n>>> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n>>> index 86794ac1..ee644d3c 100644\n>>> --- a/src/ipa/ipu3/ipa_context.cpp\n>>> +++ b/src/ipa/ipu3/ipa_context.cpp\n>>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n>>> *\n>>> * \\var IPASessionConfiguration::grid.stride\n>>> * \\brief Number of cells on one line including the ImgU padding\n>>> + *\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\var IPASessionConfiguration::af\n>>> + * \\brief AF parameters configuration of the IPA\n>>> + *\n>>> + * \\var IPASessionConfiguration::af.start_x\n>>> + * \\brief The start X position of the AF area\n>>> + *\n>>> + * \\var IPASessionConfiguration::af.start_y\n>>> + * \\brief The start Y position of the AF area\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\var IPAFrameContext::af\n>>> + * \\brief Context for the Automatic Focus algorithm\n>>> + *\n>>> + * \\struct IPAFrameContext::af\n>>> + * \\var IPAFrameContext::af.focus\n>>> + * \\brief Current position of the lens\n>>> + *\n>>> + * \\var IPAFrameContext::af.maxVariance\n>>> + * \\brief The maximum variance of the current image.\n>>> + *\n>>> + * \\var IPAFrameContext::af.stable\n>>> + * \\brief is the image focused?\n>>> */\n>>>\n>>> /**\n>>> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n>>> index c6dc0814..aa5bf97f 100644\n>>> --- a/src/ipa/ipu3/ipa_context.h\n>>> +++ b/src/ipa/ipu3/ipa_context.h\n>>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n>>> double minAnalogueGain;\n>>> double maxAnalogueGain;\n>>> } agc;\n>>> +\n>>> + struct {\n>>> + uint16_t start_x;\n>>> + uint16_t start_y;\n>>> + } af;\n>>> };\n>>>\n>>> struct IPAFrameContext {\n>>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n>>> double temperatureK;\n>>> } awb;\n>>>\n>>> + struct {\n>>> + uint32_t focus;\n>>> + double maxVariance;\n>>> + bool stable;\n>>> + } af;\n>>> +\n>>> struct {\n>>> uint32_t exposure;\n>>> double gain;\n>>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n>>> index 3d307708..93966c6f 100644\n>>> --- a/src/ipa/ipu3/ipu3.cpp\n>>> +++ b/src/ipa/ipu3/ipu3.cpp\n>>> @@ -30,6 +30,7 @@\n>>>\n>>> #include \"libcamera/internal/mapped_framebuffer.h\"\n>>>\n>>> +#include \"algorithms/af.h\"\n>>> #include \"algorithms/agc.h\"\n>>> #include \"algorithms/algorithm.h\"\n>>> #include \"algorithms/awb.h\"\n>>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n>>> }\n>>>\n>>> /* Construct our Algorithms */\n>>> + algorithms_.push_back(std::make_unique());\n>>> algorithms_.push_back(std::make_unique());\n>>> algorithms_.push_back(std::make_unique());\n>>> algorithms_.push_back(std::make_unique());\n>>>\n>>\n> \n> Thank you :)\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1B42EBF415\n\tfor ;\n\tFri, 17 Dec 2021 16:18:53 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 41013608A5;\n\tFri, 17 Dec 2021 17:18:52 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 953C560114\n\tfor ;\n\tFri, 17 Dec 2021 17:18:50 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:7f55:44a0:7b3f:1443] (unknown\n\t[IPv6:2a01:e0a:169:7140:7f55:44a0:7b3f:1443])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 34BA393;\n\tFri, 17 Dec 2021 17:18:50 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"is4U6mTW\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1639757930;\n\tbh=HnrVU4rPjtX1InkM5WJnMsKlcJPEV/5OnuLkTvrGrGc=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=is4U6mTW4/Uepc/age5L82PDLybSWWlSdmJnka2xjthxORJgsyadGFRqgafReSMh5\n\t9/pqc1NRSeUVy5HwuL7GHXLNZfJ1OLB+beYUzawzLwi9fjRfHJm/XVAc8XyhLoTJUx\n\tWaH3ChiW4xJjMq//MAeIay6j17pUC0xr6IF+pkjk=","Message-ID":"<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>","Date":"Fri, 17 Dec 2021 17:18:47 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.3.1","Content-Language":"en-US","To":"Kate Hsuan ","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t","From":"Jean-Michel Hautbois ","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21845,"web_url":"https://patchwork.libcamera.org/comment/21845/","msgid":"","date":"2021-12-21T10:42:36","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel,\n\nThank you for your response.\n\n\nOn Sat, Dec 18, 2021 at 12:18 AM Jean-Michel Hautbois\n wrote:\n>\n> Hi Kate,\n>\n> On 17/12/2021 09:57, Kate Hsuan wrote:\n> > \"Hi Jean-Michel,\n> >\n> > Thank you for your review.\n> >\n> > On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n> > wrote:\n> >>\n> >> Hi Kate,\n> >>\n> >> Thanks for the patch :-) !\n> >>\n> >> I have a few comments below, and I have to tell you I couldn't make it\n> >> work here... I don't know how you are testing it ? I can see the lens\n> >> position changing but it passes over the correct focus and never locks\n> >> on it.\n> >>\n> >> On 13/12/2021 13:28, Kate Hsuan wrote:\n> >>> Since VCM for surface Go 2 (dw9719) had been successfully\n> >>> driven, this Af module can be used to control the VCM and\n> >>> determine the focus value based on the IPU3 AF state.\n> >>>\n> >>> The variance of each focus step is determined and a greedy\n> >>> approah is used to find the maximum variance of the AF\n> >>> state and a appropriate focus value.\n> >>>\n> >>> Changes in v2:\n> >>> 1. Add grid configuration interface.\n> >>> 2. Add AF accelerator configuration.\n> >>> 3. Move default focus area to the center of the image.\n> >>>\n> >>> Changes in v3:\n> >>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n> >>> algorithm.\n> >>> 2. Fix greedy AF algorithm since V4l2 interface had been\n> >>> removed.\n> >>> 3. Simplify default grid config for AF.\n> >>> 4. AF start area is determined by bdsOutputSize.\n> >>>\n> >>> Changes in v4:\n> >>> In v4, it significant improves the AF scan time using a two pass\n> >>> scaning method and the AF scan will be terminated when it finds\n> >>> a negative gradient.\n> >>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n> >>> the AF success rate.\n> >>> 2. The low pass filter convolution results are used to determine\n> >>> a coarse AF result.\n> >>> 3. The high pass convolution result is used to find a good lens\n> >>> step in a given AF range determined in pass 1.\n> >>>\n> >>\n> >> The lines above describe the changes between the versions, which is\n> >> great ! But those should go after the \"---\" as we don't want those in\n> >> the final commit message.\n> >\n> > Ok. Thank you I'll move the changes after \"---\".\n> >\n> >>\n> >>> Signed-off-by: Kate Hsuan \n> >>> ---\n> >>\n> >> Changes in ...\n> >>\n> >> ---\n> >>> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n> >>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n> >>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n> >>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n> >>> src/ipa/ipu3/ipa_context.h | 11 +\n> >>> src/ipa/ipu3/ipu3.cpp | 2 +\n> >>> 6 files changed, 442 insertions(+), 1 deletion(-)\n> >>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> >>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n> >>>\n> >>\n> >> I applied your patch, and I have compilation issues:\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n> >> 89 | };\n> >> | ^\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_grid_config::height_per_slice’\n> >> [-Werror=missing-field-initializers]\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n> >> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n> >> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n> >> const ipu3_uapi_stats_3a*)’:\n> >> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n> >> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n> >> value [-Werror=address-of-packed-member]\n> >> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>\n> >> I solved them locally (with a quick and dirty patch) how did you not\n> >> have those ?\n> >\n> > It was my fault. I can build them successfully before. But, after\n> > re-cloning the git repo and patching my v4 patch, it happened. I'll\n> > fix those issues.\n> > Sorry about the inconvenience. T_T\n>\n> No problem, but it was surprising :-).\n>\n\nI had fixed it. My fault :)\n\n> >\n> >>\n> >>> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> >>> new file mode 100644\n> >>> index 00000000..620be9fa\n> >>> --- /dev/null\n> >>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> >>> @@ -0,0 +1,334 @@\n> >>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>> +/*\n> >>> + * Copyright (C) 2021, Red Hat\n> >>> + *\n> >>> + * af.cpp - IPU3 auto focus control\n> >>> + */\n> >>> +\n> >>> +#include \"af.h\"\n> >>> +\n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \"libipa/histogram.h\"\n> >>> +\n> >>> +/**\n> >>> + * \\file af.h\n> >>> + */\n> >>> +\n> >>> +namespace libcamera {\n> >>> +\n> >>> +using namespace std::literals::chrono_literals;\n> >>> +\n> >>> +namespace ipa::ipu3::algorithms {\n> >>> +\n> >>> +/**\n> >>> + * \\class Af\n> >>> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n> >>> + *\n> >>> + * This algorithm is used to determine the position of the lens and get a\n> >>> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n> >>> + * by high pass and low pass filtered value and stores in a AF buffer.\n> >>> + * Typically, for a focused image, it has relative high contrast than a\n> >>> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n> >>> + * highest contrast can be found from the AF scan, the lens' position is the\n> >>> + * best step of the focus.\n> >>> + *\n> >>> + */\n> >>> +\n> >>> +LOG_DEFINE_CATEGORY(IPU3Af)\n> >>> +\n> >>> +/**\n> >>> + * Maximum focus value of the VCM control\n> >>> + * \\todo should be obtained from the VCM driver\n> >>> + */\n> >>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n> >>> +\n> >>> +/* minimum focus step for searching appropriate focus*/\n> >>> +static constexpr uint32_t coarseSearchStep_ = 10;\n> >>> +static constexpr uint32_t fineSearchStep_ = 1;\n> >>> +\n> >>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n> >>> +static constexpr double MaxChange_ = 0.8;\n> >>\n> >> Why 0.8 ? Any explanation on how to set it ?\n> >\n> > OK. I'll put my explanation here. Basically, it is used to represent\n> > the boundary of a focused and blurred image.\n> >\n> >\n> >>\n> >>> +\n> >>> +/* settings for Auto Focus from the kernel */\n> >>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> >>> + .filter_config = {\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 128 },\n> >>> + 0,\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 128 },\n> >>> + 0,\n> >>> + .y_calc = { 8, 8, 8, 8 },\n> >>> + .nf = { 0, 7, 0, 7, 0 },\n> >>> + },\n>\n> I would like your thinking here (and perhaps someone else). It looks\n> like the ImgU implements a Gabor filter, but I can't see all the\n> parameters I would expect from it. Putting it down, I can't see with\n> this default filter how y1 and y2 could represent low-pass and high\n> pass, as the filters are the same. But, it could be a horizontal and a\n> vertical filter instead, with y1 the horizontal and y2 the vertical\n> resulting filtered values.\n>\n> It would be very interesting to display the y1_avg and y2_avg values as\n> an image, I just don't have lots of time for it. We have 16 bits values,\n> so we could dump those into a binary file and use python to display the\n> values as a raw gray image or something similar.\n>\n> The other things which I find annoying is the non-reliable values I get\n> right now for the variance. Same scene, exposure and gain fixed to a\n> value, controled luminosity, and between two executions I might not have\n> the same result...\n\nMe too. So I set 80% rage to determine the stability. If the ratio of\nvariance change is larger than 80% and continue for 10 frames, the AF\nscan will be triggered.\n\n>\n> >>> + .grid_cfg = {\n> >>> + .width = 16,\n> >>> + .height = 16,\n> >>> + .block_width_log2 = 3,\n> >>> + .block_height_log2 = 3,\n> >>> + .x_start = 10,\n> >>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >>> + },\n> >>> +};\n> >>\n> >> As mentionned, you missed some fields:\n> >\n> > I'll fix this. Thank you.\n> >\n> >>\n> >> '''\n> >> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n> >> imgu_css_af_defaults = {\n> >> .y_calc = { 8, 8, 8, 8 },\n> >> .nf = { 0, 7, 0, 7, 0 },\n> >> },\n> >> + .padding = { 0, 0, 0, 0 },\n> >> .grid_cfg = {\n> >> .width = 16,\n> >> .height = 16,\n> >> .block_width_log2 = 3,\n> >> .block_height_log2 = 3,\n> >> + .height_per_slice = 1,\n> >> .x_start = 10,\n> >> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >> + .x_end = 138,\n> >> + .y_end = 130,\n> >> },\n> >> };\n> >> '''\n> >>\n> >>> +\n> >>> +Af::Af()\n> >>> + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> >>> + pass1Done_(false), pass2Done_(false)\n> >>> +{\n> >>> + maxStep_ = MaxFocusSteps_;\n> >>> +}\n> >>> +\n> >>> +Af::~Af()\n> >>> +{\n> >>> +}\n> >>> +\n> >>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n> >>\n> >> Missing documentation for prepare() => you should build the doc ;-)\n> >\n> > Ok. I'll add the prepare() document here.\n> >\n> >>\n> >>> +{\n> >>> + params->use.acc_af = 1;\n> >>> + params->acc_param.af = imgu_css_af_defaults;\n> >>> + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n> >>> + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n> >>\n> >> You are never changing the grid size, is it intended ?\n> >\n> > Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n> > more detail? I can modify this according to your opinions.\n>\n> Well, yes, you are always using the default 128x128 grid size. But you\n> can go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in\n> height. Thus, it means we could evaluate the scene on a small part or\n> almost all of it (copying the bds grid is not doable as it might have\n> larger block width, up to 80x60).\n>\n> See:\n> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n>\n> Note:\n> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n>\n> => It makes me wonder if the filtering is fast, or if it may take some\n> time, meaning that focus evaluation in one frame could be done only at\n> frame+2 or something like that.\n>\n> Could you verify that the focus estimated is in the center of the scene\n> right now, as it is what you have configured ?\n\nI can quickly answer this.\n\nThe default is (10, 2) where is the top left corner of the sensor for\nsure if we don't modify anything.\n\nIn the previous patch, the output format (1280x720 NV12) is used to\ndetermine the centre of the scene.\nAn approximate center area can get through the following equation.\n(tested by my gray card, not so accurate but works)\nx = (1280/2) - ((16 x 2^3) / 2) = 576\ny = (720/2) - ((16 x 2^3) / 2) = 296\n\nIf bsdGrid is used, the width and height would be (Here is a little\nbit weird. this configuration moves the area very close to the edge of\nright of the sensor)\nx = (1440/2) - ((16 x 2^3) / 2) = 656\ny = (832/2) - ((16 x 2^3) / 2) = 352\n\nSo the AF should be based on output size to get the current centre of the scene.\n\n\n\n>\n> >\n> >>\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief Configure the Af given a configInfo\n> >>> + * \\param[in] context The shared IPA context\n> >>> + * \\param[in] configInfo The IPA configuration data\n> >>> + *\n> >>> + * \\return 0\n> >>> + */\n> >>> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> >>> +{\n> >>> + /* determined focus value i.e. current focus value */\n> >>> + context.frameContext.af.focus = 0;\n> >>> + /* maximum variance of the AF statistics */\n> >>> + context.frameContext.af.maxVariance = 0;\n> >> double type, maybe s/= 0/= 0.0/ ?\n> >\n> > 0.0 is correct. I'll correct this thank you.\n> >\n> >>\n> >>> + /* is focused? if it is true, the AF should be in a stable state. */\n> >>> + context.frameContext.af.stable = false;\n> >>> + /* frame to be ignored before start to estimate AF variance. */\n> >>> + ignoreFrame_ = 10;\n> >>\n> >> Could be a constexpr ?\n> >\n> > It is a counter to ignore frames since we have to ignore some frames\n> > to wait for the value from the accelerator to be stable.\n> > If this should be a constant variable, I can modify this. :)\n> >\n> >\n> >>\n> >>> +\n> >>> + /*\n> >>> + * AF default area configuration\n> >>> + * Move AF area to the center of the image.\n> >>> + */\n> >>> + /* Default AF width is 16x8 = 128 */\n> >>> + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n> >>\n> >> Why are you using a fixed value for the grid ? You could use the value\n> >> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n> >>\n> >>> + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n> >>> +\n> >>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n> >>> + << configInfo.bdsOutputSize.width\n> >>> + << \" Y: \"\n> >>> + << configInfo.bdsOutputSize.height;\n> >>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n> >>> + << context.configuration.af.start_x\n> >>> + << \" Y: \"\n> >>> + << context.configuration.af.start_y;\n> >>> +\n> >>> + return 0;\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF coarse scan\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + */\n> >>> +void Af::af_coarse_scan(IPAContext &context)\n> >>\n> >> Bad naming (camelCase only). Same below.\n> >\n> > Ops, sure, I'll modify this.\n> >\n> >>\n> >>> +{\n> >>> + if (pass1Done_ == true)\n> >>> + return;\n> >>> +\n> >>> + if (af_scan(context, coarseSearchStep_)) {\n> >>> + pass1Done_ = true;\n> >>> + context.frameContext.af.maxVariance = 0;\n> >>> + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n> >>> + context.frameContext.af.focus = focus_;\n> >>> + previousVariance_ = 0;\n> >>> + maxStep_ = focus_ + (focus_ * 0.2);\n> >>> + }\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF fine scan\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + */\n> >>> +void Af::af_fine_scan(IPAContext &context)\n> >>> +{\n> >>> + if (pass1Done_ != true)\n> >>> + return;\n> >>> +\n> >>> + if (af_scan(context, fineSearchStep_)) {\n> >>> + context.frameContext.af.stable = true;\n> >>> + pass2Done_ = true;\n> >>> + }\n> >>> +}\n> >> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n> >> seee where it is used...\n> >\n> > Since we used two stage AF searching strategy I simply say this is a\n> > two pass algorithm :)\n> > I'll renaming this.\n> >\n> >>\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF reset\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + */\n> >>> +void Af::af_reset(IPAContext &context)\n> >>> +{\n> >>> + context.frameContext.af.maxVariance = 0;\n> >>> + context.frameContext.af.focus = 0;\n> >>> + focus_ = 0;\n> >>> + context.frameContext.af.stable = false;\n> >>> + ignoreFrame_ = 60;\n> >>\n> >> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n> >\n> > We can tweak this value.\n> >\n> >>\n> >>> + previousVariance_ = 0.0;\n> >>> + pass1Done_ = false;\n> >>> + pass2Done_ = false;\n> >>> + maxStep_ = MaxFocusSteps_;\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF scan\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + * \\return True, if it finds a AF value.\n> >>> + */\n> >>> +bool Af::af_scan(IPAContext &context, int min_step)\n> >>> +{\n> >>> + /* find the maximum variance during the AF scan using a greedy strategy */\n> >>> + if (currentVariance_ > context.frameContext.af.maxVariance) {\n> >>> + context.frameContext.af.maxVariance = currentVariance_;\n> >>> + goodFocus_ = focus_;\n> >>> + }\n> >>> +\n> >>> + if (focus_ > maxStep_) {\n> >>> + /* if reach the max step, move lens to the position and set \"focus stable\". */\n> >>> + context.frameContext.af.focus = goodFocus_;\n> >>> + return true;\n> >>> + } else {\n> >>> + /* check negative gradient */\n> >>> + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n> >>\n> >> Where is the 0.15 coming from ?\n> >\n> > After some testing, I think that If the negative gradient ismore than\n> > 15% maxVaraiance, the focused image and lens position we get!\n> > I could put this as a constant variable in my v5 patch.\n> >\n> >>\n> >>> + focus_ += min_step;\n> >>> + context.frameContext.af.focus = focus_;\n> >>> + } else {\n> >>> + context.frameContext.af.focus = goodFocus_;\n> >>> + previousVariance_ = currentVariance_;\n> >>> + return true;\n> >>> + }\n> >>> + }\n> >>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n> >> s/prevrious/previous/\n> >\n> > Ops. sorry about the typo. T_T\n> >\n> >>\n> >>> + << previousVariance_\n> >>> + << \" current: \"\n> >>> + << currentVariance_\n> >>> + << \" Diff: \"\n> >>> + << (currentVariance_ - context.frameContext.af.maxVariance);\n> >>\n> >> Interestingly, it seems there is an issue here, according to the log I get ?\n> >>\n> >> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n> >> current: 20012.4 Diff: 0\n> >>\n> >> Shouldn't diff be 20012.4 ?\n> >>\n> >> I am not sure the algorithm is very strong though... You may never\n> >> converge and stay in a local minimum without knowing ?\n> >>\n> >>> + previousVariance_ = currentVariance_;\n> >>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n> >>> + << context.frameContext.af.maxVariance\n> >>> + << \" Focus step is \"\n> >>> + << goodFocus_\n> >>> + << \" Current scan is \"\n> >>> + << focus_;\n> >>> + return false;\n> >>> +}\n> >>\n> >> I am not sure to understand how this function should behave... Is it a\n> >> minimum searching ? Are you looking for argmin(currentVariance -\n> >> maxVariance) ?\n> >\n> > Actually. it is a maximum search. I try to search for the maximum\n> > variance of the image.\n> > If the difference is positive, it means the variance still increases.\n> > We can continue search for the maximum value and update maxVariance_.\n> > Until, we found a negative difference which means the variance starts\n> > to go down and image will become blurred.\n> >\n> >\n> >>\n> >>> +\n> >>> +/**\n> >>> + * \\brief Determine the max contrast image and lens position. y_table is the\n> >>> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n> >>> + * value. High pass filtered value also represents the sharpness of the image.\n> >>> + * Based on this, if the image with highest variance of the high pass filtered\n> >>> + * value (contrast) during the AF scan, the position of the len should be the\n> >>> + * best focus.\n> >>> + * \\param[in] context The shared IPA context.\n> >>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n> >>> + */\n> >>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> >>> +{\n> >>> + uint32_t total = 0;\n> >>> + double mean;\n> >>> + uint64_t var_sum = 0;\n> >>> + y_table_item_t *y_item;\n> >>> + int z = 0;\n> >>> +\n> >>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>> +\n> >>> + /**\n> >>> + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n> >>> + * for a given grid.\n> >>> + * For pass1: low pass results are used.\n> >>> + * For pass2: high pass results are used.\n> >>> + */\n> >>> + if (pass1Done_) {\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>> + total = total + y_item[z].y2_avg;\n> >>> + if (y_item[z].y2_avg == 0)\n> >>> + break;\n> >>> + }\n> >>> + mean = total / z;\n> >>> +\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n> >>> + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n> >>> + if (y_item[z].y2_avg == 0)\n> >>> + break;\n> >>> + }\n> >>> + } else {\n> >>\n> >> /* Calculate the mean of the low pass filter values. */\n> >>\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>> + total = total + y_item[z].y1_avg;\n> >>> + if (y_item[z].y1_avg == 0)\n> >>> + break;\n> >> Are you sure of that ? Shouldn't you use the grid you set to parse the\n> >> right number of data ? If you have a local dark area you may stop even\n> >> if it is not yet the end of the grid ?\n> >\n> > I'll find another way to determine the range of the results. May be it\n> > can based on the grids setting.\n> >\n> >>\n> >>> + }\n> >>> + mean = total / z;\n> >>> +\n> >>\n> >> /* Calulate the deviation from the mean. */\n> >>\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n> >>> + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n> >>> + if (y_item[z].y1_avg == 0)\n> >>> + break;\n> >>> + }\n> >>> + }\n> >>\n> >> The two if() blocks are very similar, the only exception beeing the item\n> >> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n> >> in the second one). This is not easy to read (according to me).\n> >\n> > They are high pass (y2) and low pass (y1) filtered convolution\n> > results. This structure referees from ipu-ipa repo and I would like to\n> > keep their original naming.\n> >\n> >\n> >>\n> >>> + /* Determine the average variance of the frame. */\n> >>> + currentVariance_ = static_cast(var_sum) / static_cast(z);\n> >>\n> >> Side note:\n> >> If my stats are still correct, I think it should be divided by (z-1) and\n> >> not z, as we have a sample and not a full population ?\n> >\n> > it is z-1 and my fault. I'll correct this. :)\n> >\n> >>\n> >>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n> >>> +\n> >>> + if (context.frameContext.af.stable == true) {\n> >>> + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n> >>> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> >>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n> >>> + << var_ratio\n> >>> + << \" current focus: \"\n> >>> + << context.frameContext.af.focus;\n> >>> + /**\n> >>> + * If the change ratio of contrast is over Maxchange_ (out of focus),\n> >>> + * trigger AF again.\n> >>> + */\n> >>> + if (var_ratio > MaxChange_) {\n> >>> + if (ignoreFrame_ == 0) {\n> >>> + af_reset(context);\n> >>> + } else\n> >>> + ignoreFrame_--;\n> >>> + } else\n> >>> + ignoreFrame_ = 10;\n> >>> + } else {\n> >>> + if (ignoreFrame_ != 0)\n> >>> + ignoreFrame_--;\n> >>> + else {\n> >>> + af_coarse_scan(context);\n> >>> + af_fine_scan(context);\n> >>> + }\n> >>> + }\n> >>> +}\n> >>> +\n> >>> +} /* namespace ipa::ipu3::algorithms */\n> >>> +\n> >>> +} /* namespace libcamera */\n> >>> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> >>> new file mode 100644\n> >>> index 00000000..b9295b19\n> >>> --- /dev/null\n> >>> +++ b/src/ipa/ipu3/algorithms/af.h\n> >>> @@ -0,0 +1,66 @@\n> >>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>> +/*\n> >>> + * Copyright (C) 2021, Red Hat\n> >>> + *\n> >>> + * af.h - IPU3 Af control\n> >>> + */\n> >>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \"algorithm.h\"\n> >>> +\n> >>> +namespace libcamera {\n> >>> +\n> >>> +namespace ipa::ipu3::algorithms {\n> >>> +\n> >>> +class Af : public Algorithm\n> >>> +{\n> >>> + /* The format of y_table. From ipu3-ipa repo */\n> >>> + typedef struct y_table_item {\n> >>> + uint16_t y1_avg;\n> >>> + uint16_t y2_avg;\n> >>> + } y_table_item_t;\n> >>> +\n> >>> +public:\n> >>> + Af();\n> >>> + ~Af();\n> >>> +\n> >>> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> >>> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> >>> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> >>> +\n> >>> +private:\n> >>> + void af_coarse_scan(IPAContext &context);\n> >>> + void af_fine_scan(IPAContext &context);\n> >>> + bool af_scan(IPAContext &context, int min_step);\n> >>> + void af_reset(IPAContext &context);\n> >>> +\n> >>> + /* Used for focus scan. */\n> >>> + uint32_t focus_;\n> >>> + /* Focus good */\n> >>> + uint32_t goodFocus_;\n> >>> + /* Recent AF statistic variance. */\n> >>> + double currentVariance_;\n> >>> + /* The frames to be ignore before starting measuring. */\n> >>> + uint32_t ignoreFrame_;\n> >>> + /* previous variance. it is used to determine the gradient */\n> >>> + double previousVariance_;\n> >>> + /* Max scan steps of each pass of AF scaning */\n> >>> + uint32_t maxStep_;\n> >>> + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n> >>> + bool pass1Done_;\n> >>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n> >>> + bool pass2Done_;\n> >>> +};\n> >>> +\n> >>> +} /* namespace ipa::ipu3::algorithms */\n> >>> +\n> >>> +} /* namespace libcamera */\n> >>> +\n> >>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n> >>> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> >>> index 4db6ae1d..e1099169 100644\n> >>> --- a/src/ipa/ipu3/algorithms/meson.build\n> >>> +++ b/src/ipa/ipu3/algorithms/meson.build\n> >>> @@ -1,8 +1,9 @@\n> >>> # SPDX-License-Identifier: CC0-1.0\n> >>>\n> >>> ipu3_ipa_algorithms = files([\n> >>> + 'af.cpp',\n> >>> 'agc.cpp',\n> >>> 'awb.cpp',\n> >>> 'blc.cpp',\n> >>> - 'tone_mapping.cpp',\n> >>> + 'tone_mapping.cpp'\n> >>> ])\n> >>> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> >>> index 86794ac1..ee644d3c 100644\n> >>> --- a/src/ipa/ipu3/ipa_context.cpp\n> >>> +++ b/src/ipa/ipu3/ipa_context.cpp\n> >>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n> >>> *\n> >>> * \\var IPASessionConfiguration::grid.stride\n> >>> * \\brief Number of cells on one line including the ImgU padding\n> >>> + *\n> >>> + */\n> >>> +\n> >>> +/**\n> >>> + * \\var IPASessionConfiguration::af\n> >>> + * \\brief AF parameters configuration of the IPA\n> >>> + *\n> >>> + * \\var IPASessionConfiguration::af.start_x\n> >>> + * \\brief The start X position of the AF area\n> >>> + *\n> >>> + * \\var IPASessionConfiguration::af.start_y\n> >>> + * \\brief The start Y position of the AF area\n> >>> + */\n> >>> +\n> >>> +/**\n> >>> + * \\var IPAFrameContext::af\n> >>> + * \\brief Context for the Automatic Focus algorithm\n> >>> + *\n> >>> + * \\struct IPAFrameContext::af\n> >>> + * \\var IPAFrameContext::af.focus\n> >>> + * \\brief Current position of the lens\n> >>> + *\n> >>> + * \\var IPAFrameContext::af.maxVariance\n> >>> + * \\brief The maximum variance of the current image.\n> >>> + *\n> >>> + * \\var IPAFrameContext::af.stable\n> >>> + * \\brief is the image focused?\n> >>> */\n> >>>\n> >>> /**\n> >>> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> >>> index c6dc0814..aa5bf97f 100644\n> >>> --- a/src/ipa/ipu3/ipa_context.h\n> >>> +++ b/src/ipa/ipu3/ipa_context.h\n> >>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n> >>> double minAnalogueGain;\n> >>> double maxAnalogueGain;\n> >>> } agc;\n> >>> +\n> >>> + struct {\n> >>> + uint16_t start_x;\n> >>> + uint16_t start_y;\n> >>> + } af;\n> >>> };\n> >>>\n> >>> struct IPAFrameContext {\n> >>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n> >>> double temperatureK;\n> >>> } awb;\n> >>>\n> >>> + struct {\n> >>> + uint32_t focus;\n> >>> + double maxVariance;\n> >>> + bool stable;\n> >>> + } af;\n> >>> +\n> >>> struct {\n> >>> uint32_t exposure;\n> >>> double gain;\n> >>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> >>> index 3d307708..93966c6f 100644\n> >>> --- a/src/ipa/ipu3/ipu3.cpp\n> >>> +++ b/src/ipa/ipu3/ipu3.cpp\n> >>> @@ -30,6 +30,7 @@\n> >>>\n> >>> #include \"libcamera/internal/mapped_framebuffer.h\"\n> >>>\n> >>> +#include \"algorithms/af.h\"\n> >>> #include \"algorithms/agc.h\"\n> >>> #include \"algorithms/algorithm.h\"\n> >>> #include \"algorithms/awb.h\"\n> >>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> >>> }\n> >>>\n> >>> /* Construct our Algorithms */\n> >>> + algorithms_.push_back(std::make_unique());\n> >>> algorithms_.push_back(std::make_unique());\n> >>> algorithms_.push_back(std::make_unique());\n> >>> algorithms_.push_back(std::make_unique());\n> >>>\n> >>\n> >\n> > Thank you :)\n> >\n>\n\n\n--\nBR,\nKate","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4D03CBF415\n\tfor ;\n\tTue, 21 Dec 2021 10:42:56 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 07F6C60868;\n\tTue, 21 Dec 2021 11:42:56 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3DE7060113\n\tfor ;\n\tTue, 21 Dec 2021 11:42:54 +0100 (CET)","from mail-lj1-f198.google.com (mail-lj1-f198.google.com\n\t[209.85.208.198]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-252-vdAzSHh6PYKTnRXX7LXazg-1; Tue, 21 Dec 2021 05:42:50 -0500","by mail-lj1-f198.google.com with SMTP id\n\tu8-20020a05651c130800b0022d6dad0418so232275lja.11\n\tfor ;\n\tTue, 21 Dec 2021 02:42:49 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"gyk9Wd2Y\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1640083373;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=JHxzu6Vd8cDMOfWus+6FuSLsfK4DaU156fsDHFv9Re8=;\n\tb=gyk9Wd2YMsXXK8kJrZTb0lzERjtliI2PTHm6ejYkeXaC6eS1G2tPtZ+CvJuznSdEBZ3nSR\n\tiJqVM/mf0HJCD2h7xgjyGUp6N3HkIYU8xj+DMVBm6QduvXnuTpylKyvtG4TYQ4Grdnnot9\n\thIgzL7V42Vlge6QJ6UbUmcNTZ32u5qk=","X-MC-Unique":"vdAzSHh6PYKTnRXX7LXazg-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=JHxzu6Vd8cDMOfWus+6FuSLsfK4DaU156fsDHFv9Re8=;\n\tb=aoX2OVq/Dc8OC7i566oU1/tC1UMTWyuWw/F7PbY5EZLkuO06J4rDGuo1l55QqTYWxM\n\t0oBDSldb+a2Hlw3cYiA7rERaqBya7DKinMgFCspr5/Gi8QAe8ASveYMu5LfF1oRsGk0Q\n\tfAGqW/mFp1AZW3i7+oB3Tz3KiORCt+IbzbZQd8chG/hEBqxTCOykCMjf9A57BQwjmBfF\n\tNrzx/AXBpGYL4KG4PGzLxVXb3DD0KwfM9Q8s42VHQFrwM58kqWMZoSMK44hrLPI8Cj/m\n\t9j3PEZl8xk/SE//JP53xMYVNF8L2NlhW9LiD3SPEEDwQHnGeo85gfeOyZDNQcedTLLyZ\n\tAEUg==","X-Gm-Message-State":"AOAM533G2C/OPZKBKE/ZdeCkZxkHqMyWmO1rrPIqrcwIqRAT8sXb0nX1\n\tnG1FqimB8mabTmag8Hym1VKM8NTz7lLeoqhOicziLaDy2h9ltxsMuNz9bhNbpgQjuhTj+vifxds\n\tfB+Nhxo7oMwI448UDwR25y25O8J7K/fAyYZvJ958aWsA1IvXoQA==","X-Received":["by 2002:a2e:a78e:: with SMTP id\n\tc14mr2159255ljf.162.1640083367971; \n\tTue, 21 Dec 2021 02:42:47 -0800 (PST)","by 2002:a2e:a78e:: with SMTP id\n\tc14mr2159229ljf.162.1640083367398; \n\tTue, 21 Dec 2021 02:42:47 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJwWAEJAoM6JocLBBo0wxhD3QJH3zvxMwYW2mWMrbXP59h1Qk46uYzt/uH3n8sJpx4/alDMwL3iqIQm+JnAzg3k=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>","In-Reply-To":"<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>","From":"Kate Hsuan ","Date":"Tue, 21 Dec 2021 18:42:36 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21885,"web_url":"https://patchwork.libcamera.org/comment/21885/","msgid":"","date":"2021-12-27T09:54:54","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel,\n\nOn Sat, Dec 18, 2021 at 12:18 AM Jean-Michel Hautbois\n wrote:\n>\n> Hi Kate,\n>\n> On 17/12/2021 09:57, Kate Hsuan wrote:\n> > \"Hi Jean-Michel,\n> >\n> > Thank you for your review.\n> >\n> > On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n> > wrote:\n> >>\n> >> Hi Kate,\n> >>\n> >> Thanks for the patch :-) !\n> >>\n> >> I have a few comments below, and I have to tell you I couldn't make it\n> >> work here... I don't know how you are testing it ? I can see the lens\n> >> position changing but it passes over the correct focus and never locks\n> >> on it.\n> >>\n> >> On 13/12/2021 13:28, Kate Hsuan wrote:\n> >>> Since VCM for surface Go 2 (dw9719) had been successfully\n> >>> driven, this Af module can be used to control the VCM and\n> >>> determine the focus value based on the IPU3 AF state.\n> >>>\n> >>> The variance of each focus step is determined and a greedy\n> >>> approah is used to find the maximum variance of the AF\n> >>> state and a appropriate focus value.\n> >>>\n> >>> Changes in v2:\n> >>> 1. Add grid configuration interface.\n> >>> 2. Add AF accelerator configuration.\n> >>> 3. Move default focus area to the center of the image.\n> >>>\n> >>> Changes in v3:\n> >>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n> >>> algorithm.\n> >>> 2. Fix greedy AF algorithm since V4l2 interface had been\n> >>> removed.\n> >>> 3. Simplify default grid config for AF.\n> >>> 4. AF start area is determined by bdsOutputSize.\n> >>>\n> >>> Changes in v4:\n> >>> In v4, it significant improves the AF scan time using a two pass\n> >>> scaning method and the AF scan will be terminated when it finds\n> >>> a negative gradient.\n> >>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n> >>> the AF success rate.\n> >>> 2. The low pass filter convolution results are used to determine\n> >>> a coarse AF result.\n> >>> 3. The high pass convolution result is used to find a good lens\n> >>> step in a given AF range determined in pass 1.\n> >>>\n> >>\n> >> The lines above describe the changes between the versions, which is\n> >> great ! But those should go after the \"---\" as we don't want those in\n> >> the final commit message.\n> >\n> > Ok. Thank you I'll move the changes after \"---\".\n> >\n> >>\n> >>> Signed-off-by: Kate Hsuan \n> >>> ---\n> >>\n> >> Changes in ...\n> >>\n> >> ---\n> >>> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n> >>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n> >>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n> >>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n> >>> src/ipa/ipu3/ipa_context.h | 11 +\n> >>> src/ipa/ipu3/ipu3.cpp | 2 +\n> >>> 6 files changed, 442 insertions(+), 1 deletion(-)\n> >>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> >>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n> >>>\n> >>\n> >> I applied your patch, and I have compilation issues:\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n> >> 89 | };\n> >> | ^\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_grid_config::height_per_slice’\n> >> [-Werror=missing-field-initializers]\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n> >> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n> >> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n> >> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n> >> const ipu3_uapi_stats_3a*)’:\n> >> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n> >> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n> >> value [-Werror=address-of-packed-member]\n> >> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>\n> >> I solved them locally (with a quick and dirty patch) how did you not\n> >> have those ?\n> >\n> > It was my fault. I can build them successfully before. But, after\n> > re-cloning the git repo and patching my v4 patch, it happened. I'll\n> > fix those issues.\n> > Sorry about the inconvenience. T_T\n>\n> No problem, but it was surprising :-).\n>\n> >\n> >>\n> >>> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> >>> new file mode 100644\n> >>> index 00000000..620be9fa\n> >>> --- /dev/null\n> >>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> >>> @@ -0,0 +1,334 @@\n> >>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>> +/*\n> >>> + * Copyright (C) 2021, Red Hat\n> >>> + *\n> >>> + * af.cpp - IPU3 auto focus control\n> >>> + */\n> >>> +\n> >>> +#include \"af.h\"\n> >>> +\n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \"libipa/histogram.h\"\n> >>> +\n> >>> +/**\n> >>> + * \\file af.h\n> >>> + */\n> >>> +\n> >>> +namespace libcamera {\n> >>> +\n> >>> +using namespace std::literals::chrono_literals;\n> >>> +\n> >>> +namespace ipa::ipu3::algorithms {\n> >>> +\n> >>> +/**\n> >>> + * \\class Af\n> >>> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n> >>> + *\n> >>> + * This algorithm is used to determine the position of the lens and get a\n> >>> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n> >>> + * by high pass and low pass filtered value and stores in a AF buffer.\n> >>> + * Typically, for a focused image, it has relative high contrast than a\n> >>> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n> >>> + * highest contrast can be found from the AF scan, the lens' position is the\n> >>> + * best step of the focus.\n> >>> + *\n> >>> + */\n> >>> +\n> >>> +LOG_DEFINE_CATEGORY(IPU3Af)\n> >>> +\n> >>> +/**\n> >>> + * Maximum focus value of the VCM control\n> >>> + * \\todo should be obtained from the VCM driver\n> >>> + */\n> >>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n> >>> +\n> >>> +/* minimum focus step for searching appropriate focus*/\n> >>> +static constexpr uint32_t coarseSearchStep_ = 10;\n> >>> +static constexpr uint32_t fineSearchStep_ = 1;\n> >>> +\n> >>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n> >>> +static constexpr double MaxChange_ = 0.8;\n> >>\n> >> Why 0.8 ? Any explanation on how to set it ?\n> >\n> > OK. I'll put my explanation here. Basically, it is used to represent\n> > the boundary of a focused and blurred image.\n> >\n> >\n> >>\n> >>> +\n> >>> +/* settings for Auto Focus from the kernel */\n> >>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> >>> + .filter_config = {\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 128 },\n> >>> + 0,\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 0 },\n> >>> + { 0, 0, 0, 128 },\n> >>> + 0,\n> >>> + .y_calc = { 8, 8, 8, 8 },\n> >>> + .nf = { 0, 7, 0, 7, 0 },\n> >>> + },\n>\n> I would like your thinking here (and perhaps someone else). It looks\n> like the ImgU implements a Gabor filter, but I can't see all the\n> parameters I would expect from it. Putting it down, I can't see with\n> this default filter how y1 and y2 could represent low-pass and high\n> pass, as the filters are the same. But, it could be a horizontal and a\n> vertical filter instead, with y1 the horizontal and y2 the vertical\n> resulting filtered values.\n>\n> It would be very interesting to display the y1_avg and y2_avg values as\n> an image, I just don't have lots of time for it. We have 16 bits values,\n> so we could dump those into a binary file and use python to display the\n> values as a raw gray image or something similar.\n\nYou could check out the URL below to see the grey scale image of the AF buffer.\nhttps://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n\nSince the result is a tiny 16x16 image, all the detail of the original\nimage was compressed into a black block.\n\n>\n> The other things which I find annoying is the non-reliable values I get\n> right now for the variance. Same scene, exposure and gain fixed to a\n> value, controled luminosity, and between two executions I might not have\n> the same result...\n>\n> >>> + .grid_cfg = {\n> >>> + .width = 16,\n> >>> + .height = 16,\n> >>> + .block_width_log2 = 3,\n> >>> + .block_height_log2 = 3,\n> >>> + .x_start = 10,\n> >>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >>> + },\n> >>> +};\n> >>\n> >> As mentionned, you missed some fields:\n> >\n> > I'll fix this. Thank you.\n> >\n> >>\n> >> '''\n> >> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n> >> imgu_css_af_defaults = {\n> >> .y_calc = { 8, 8, 8, 8 },\n> >> .nf = { 0, 7, 0, 7, 0 },\n> >> },\n> >> + .padding = { 0, 0, 0, 0 },\n> >> .grid_cfg = {\n> >> .width = 16,\n> >> .height = 16,\n> >> .block_width_log2 = 3,\n> >> .block_height_log2 = 3,\n> >> + .height_per_slice = 1,\n> >> .x_start = 10,\n> >> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >> + .x_end = 138,\n> >> + .y_end = 130,\n> >> },\n> >> };\n> >> '''\n> >>\n> >>> +\n> >>> +Af::Af()\n> >>> + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> >>> + pass1Done_(false), pass2Done_(false)\n> >>> +{\n> >>> + maxStep_ = MaxFocusSteps_;\n> >>> +}\n> >>> +\n> >>> +Af::~Af()\n> >>> +{\n> >>> +}\n> >>> +\n> >>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n> >>\n> >> Missing documentation for prepare() => you should build the doc ;-)\n> >\n> > Ok. I'll add the prepare() document here.\n> >\n> >>\n> >>> +{\n> >>> + params->use.acc_af = 1;\n> >>> + params->acc_param.af = imgu_css_af_defaults;\n> >>> + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n> >>> + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n> >>\n> >> You are never changing the grid size, is it intended ?\n> >\n> > Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n> > more detail? I can modify this according to your opinions.\n>\n> Well, yes, you are always using the default 128x128 grid size. But you\n> can go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in\n> height. Thus, it means we could evaluate the scene on a small part or\n> almost all of it (copying the bds grid is not doable as it might have\n> larger block width, up to 80x60).\n>\n> See:\n> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n>\n> Note:\n> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n>\n> => It makes me wonder if the filtering is fast, or if it may take some\n> time, meaning that focus evaluation in one frame could be done only at\n> frame+2 or something like that.\n>\n> Could you verify that the focus estimated is in the center of the scene\n> right now, as it is what you have configured ?\n>\n> >\n> >>\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief Configure the Af given a configInfo\n> >>> + * \\param[in] context The shared IPA context\n> >>> + * \\param[in] configInfo The IPA configuration data\n> >>> + *\n> >>> + * \\return 0\n> >>> + */\n> >>> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> >>> +{\n> >>> + /* determined focus value i.e. current focus value */\n> >>> + context.frameContext.af.focus = 0;\n> >>> + /* maximum variance of the AF statistics */\n> >>> + context.frameContext.af.maxVariance = 0;\n> >> double type, maybe s/= 0/= 0.0/ ?\n> >\n> > 0.0 is correct. I'll correct this thank you.\n> >\n> >>\n> >>> + /* is focused? if it is true, the AF should be in a stable state. */\n> >>> + context.frameContext.af.stable = false;\n> >>> + /* frame to be ignored before start to estimate AF variance. */\n> >>> + ignoreFrame_ = 10;\n> >>\n> >> Could be a constexpr ?\n> >\n> > It is a counter to ignore frames since we have to ignore some frames\n> > to wait for the value from the accelerator to be stable.\n> > If this should be a constant variable, I can modify this. :)\n> >\n> >\n> >>\n> >>> +\n> >>> + /*\n> >>> + * AF default area configuration\n> >>> + * Move AF area to the center of the image.\n> >>> + */\n> >>> + /* Default AF width is 16x8 = 128 */\n> >>> + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n> >>\n> >> Why are you using a fixed value for the grid ? You could use the value\n> >> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n> >>\n> >>> + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n> >>> +\n> >>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n> >>> + << configInfo.bdsOutputSize.width\n> >>> + << \" Y: \"\n> >>> + << configInfo.bdsOutputSize.height;\n> >>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n> >>> + << context.configuration.af.start_x\n> >>> + << \" Y: \"\n> >>> + << context.configuration.af.start_y;\n> >>> +\n> >>> + return 0;\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF coarse scan\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + */\n> >>> +void Af::af_coarse_scan(IPAContext &context)\n> >>\n> >> Bad naming (camelCase only). Same below.\n> >\n> > Ops, sure, I'll modify this.\n> >\n> >>\n> >>> +{\n> >>> + if (pass1Done_ == true)\n> >>> + return;\n> >>> +\n> >>> + if (af_scan(context, coarseSearchStep_)) {\n> >>> + pass1Done_ = true;\n> >>> + context.frameContext.af.maxVariance = 0;\n> >>> + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n> >>> + context.frameContext.af.focus = focus_;\n> >>> + previousVariance_ = 0;\n> >>> + maxStep_ = focus_ + (focus_ * 0.2);\n> >>> + }\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF fine scan\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + */\n> >>> +void Af::af_fine_scan(IPAContext &context)\n> >>> +{\n> >>> + if (pass1Done_ != true)\n> >>> + return;\n> >>> +\n> >>> + if (af_scan(context, fineSearchStep_)) {\n> >>> + context.frameContext.af.stable = true;\n> >>> + pass2Done_ = true;\n> >>> + }\n> >>> +}\n> >> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n> >> seee where it is used...\n> >\n> > Since we used two stage AF searching strategy I simply say this is a\n> > two pass algorithm :)\n> > I'll renaming this.\n> >\n> >>\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF reset\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + */\n> >>> +void Af::af_reset(IPAContext &context)\n> >>> +{\n> >>> + context.frameContext.af.maxVariance = 0;\n> >>> + context.frameContext.af.focus = 0;\n> >>> + focus_ = 0;\n> >>> + context.frameContext.af.stable = false;\n> >>> + ignoreFrame_ = 60;\n> >>\n> >> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n> >\n> > We can tweak this value.\n> >\n> >>\n> >>> + previousVariance_ = 0.0;\n> >>> + pass1Done_ = false;\n> >>> + pass2Done_ = false;\n> >>> + maxStep_ = MaxFocusSteps_;\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief AF scan\n> >>> + * \\param[in] context The shared IPA context\n> >>> + *\n> >>> + * \\return True, if it finds a AF value.\n> >>> + */\n> >>> +bool Af::af_scan(IPAContext &context, int min_step)\n> >>> +{\n> >>> + /* find the maximum variance during the AF scan using a greedy strategy */\n> >>> + if (currentVariance_ > context.frameContext.af.maxVariance) {\n> >>> + context.frameContext.af.maxVariance = currentVariance_;\n> >>> + goodFocus_ = focus_;\n> >>> + }\n> >>> +\n> >>> + if (focus_ > maxStep_) {\n> >>> + /* if reach the max step, move lens to the position and set \"focus stable\". */\n> >>> + context.frameContext.af.focus = goodFocus_;\n> >>> + return true;\n> >>> + } else {\n> >>> + /* check negative gradient */\n> >>> + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n> >>\n> >> Where is the 0.15 coming from ?\n> >\n> > After some testing, I think that If the negative gradient ismore than\n> > 15% maxVaraiance, the focused image and lens position we get!\n> > I could put this as a constant variable in my v5 patch.\n> >\n> >>\n> >>> + focus_ += min_step;\n> >>> + context.frameContext.af.focus = focus_;\n> >>> + } else {\n> >>> + context.frameContext.af.focus = goodFocus_;\n> >>> + previousVariance_ = currentVariance_;\n> >>> + return true;\n> >>> + }\n> >>> + }\n> >>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n> >> s/prevrious/previous/\n> >\n> > Ops. sorry about the typo. T_T\n> >\n> >>\n> >>> + << previousVariance_\n> >>> + << \" current: \"\n> >>> + << currentVariance_\n> >>> + << \" Diff: \"\n> >>> + << (currentVariance_ - context.frameContext.af.maxVariance);\n> >>\n> >> Interestingly, it seems there is an issue here, according to the log I get ?\n> >>\n> >> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n> >> current: 20012.4 Diff: 0\n> >>\n> >> Shouldn't diff be 20012.4 ?\n> >>\n> >> I am not sure the algorithm is very strong though... You may never\n> >> converge and stay in a local minimum without knowing ?\n> >>\n> >>> + previousVariance_ = currentVariance_;\n> >>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n> >>> + << context.frameContext.af.maxVariance\n> >>> + << \" Focus step is \"\n> >>> + << goodFocus_\n> >>> + << \" Current scan is \"\n> >>> + << focus_;\n> >>> + return false;\n> >>> +}\n> >>\n> >> I am not sure to understand how this function should behave... Is it a\n> >> minimum searching ? Are you looking for argmin(currentVariance -\n> >> maxVariance) ?\n> >\n> > Actually. it is a maximum search. I try to search for the maximum\n> > variance of the image.\n> > If the difference is positive, it means the variance still increases.\n> > We can continue search for the maximum value and update maxVariance_.\n> > Until, we found a negative difference which means the variance starts\n> > to go down and image will become blurred.\n> >\n> >\n> >>\n> >>> +\n> >>> +/**\n> >>> + * \\brief Determine the max contrast image and lens position. y_table is the\n> >>> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n> >>> + * value. High pass filtered value also represents the sharpness of the image.\n> >>> + * Based on this, if the image with highest variance of the high pass filtered\n> >>> + * value (contrast) during the AF scan, the position of the len should be the\n> >>> + * best focus.\n> >>> + * \\param[in] context The shared IPA context.\n> >>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n> >>> + */\n> >>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> >>> +{\n> >>> + uint32_t total = 0;\n> >>> + double mean;\n> >>> + uint64_t var_sum = 0;\n> >>> + y_table_item_t *y_item;\n> >>> + int z = 0;\n> >>> +\n> >>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>> +\n> >>> + /**\n> >>> + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n> >>> + * for a given grid.\n> >>> + * For pass1: low pass results are used.\n> >>> + * For pass2: high pass results are used.\n> >>> + */\n> >>> + if (pass1Done_) {\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>> + total = total + y_item[z].y2_avg;\n> >>> + if (y_item[z].y2_avg == 0)\n> >>> + break;\n> >>> + }\n> >>> + mean = total / z;\n> >>> +\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n> >>> + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n> >>> + if (y_item[z].y2_avg == 0)\n> >>> + break;\n> >>> + }\n> >>> + } else {\n> >>\n> >> /* Calculate the mean of the low pass filter values. */\n> >>\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>> + total = total + y_item[z].y1_avg;\n> >>> + if (y_item[z].y1_avg == 0)\n> >>> + break;\n> >> Are you sure of that ? Shouldn't you use the grid you set to parse the\n> >> right number of data ? If you have a local dark area you may stop even\n> >> if it is not yet the end of the grid ?\n> >\n> > I'll find another way to determine the range of the results. May be it\n> > can based on the grids setting.\n> >\n> >>\n> >>> + }\n> >>> + mean = total / z;\n> >>> +\n> >>\n> >> /* Calulate the deviation from the mean. */\n> >>\n> >>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n> >>> + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n> >>> + if (y_item[z].y1_avg == 0)\n> >>> + break;\n> >>> + }\n> >>> + }\n> >>\n> >> The two if() blocks are very similar, the only exception beeing the item\n> >> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n> >> in the second one). This is not easy to read (according to me).\n> >\n> > They are high pass (y2) and low pass (y1) filtered convolution\n> > results. This structure referees from ipu-ipa repo and I would like to\n> > keep their original naming.\n> >\n> >\n> >>\n> >>> + /* Determine the average variance of the frame. */\n> >>> + currentVariance_ = static_cast(var_sum) / static_cast(z);\n> >>\n> >> Side note:\n> >> If my stats are still correct, I think it should be divided by (z-1) and\n> >> not z, as we have a sample and not a full population ?\n> >\n> > it is z-1 and my fault. I'll correct this. :)\n> >\n> >>\n> >>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n> >>> +\n> >>> + if (context.frameContext.af.stable == true) {\n> >>> + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n> >>> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> >>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n> >>> + << var_ratio\n> >>> + << \" current focus: \"\n> >>> + << context.frameContext.af.focus;\n> >>> + /**\n> >>> + * If the change ratio of contrast is over Maxchange_ (out of focus),\n> >>> + * trigger AF again.\n> >>> + */\n> >>> + if (var_ratio > MaxChange_) {\n> >>> + if (ignoreFrame_ == 0) {\n> >>> + af_reset(context);\n> >>> + } else\n> >>> + ignoreFrame_--;\n> >>> + } else\n> >>> + ignoreFrame_ = 10;\n> >>> + } else {\n> >>> + if (ignoreFrame_ != 0)\n> >>> + ignoreFrame_--;\n> >>> + else {\n> >>> + af_coarse_scan(context);\n> >>> + af_fine_scan(context);\n> >>> + }\n> >>> + }\n> >>> +}\n> >>> +\n> >>> +} /* namespace ipa::ipu3::algorithms */\n> >>> +\n> >>> +} /* namespace libcamera */\n> >>> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> >>> new file mode 100644\n> >>> index 00000000..b9295b19\n> >>> --- /dev/null\n> >>> +++ b/src/ipa/ipu3/algorithms/af.h\n> >>> @@ -0,0 +1,66 @@\n> >>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>> +/*\n> >>> + * Copyright (C) 2021, Red Hat\n> >>> + *\n> >>> + * af.h - IPU3 Af control\n> >>> + */\n> >>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \n> >>> +\n> >>> +#include \"algorithm.h\"\n> >>> +\n> >>> +namespace libcamera {\n> >>> +\n> >>> +namespace ipa::ipu3::algorithms {\n> >>> +\n> >>> +class Af : public Algorithm\n> >>> +{\n> >>> + /* The format of y_table. From ipu3-ipa repo */\n> >>> + typedef struct y_table_item {\n> >>> + uint16_t y1_avg;\n> >>> + uint16_t y2_avg;\n> >>> + } y_table_item_t;\n> >>> +\n> >>> +public:\n> >>> + Af();\n> >>> + ~Af();\n> >>> +\n> >>> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> >>> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> >>> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> >>> +\n> >>> +private:\n> >>> + void af_coarse_scan(IPAContext &context);\n> >>> + void af_fine_scan(IPAContext &context);\n> >>> + bool af_scan(IPAContext &context, int min_step);\n> >>> + void af_reset(IPAContext &context);\n> >>> +\n> >>> + /* Used for focus scan. */\n> >>> + uint32_t focus_;\n> >>> + /* Focus good */\n> >>> + uint32_t goodFocus_;\n> >>> + /* Recent AF statistic variance. */\n> >>> + double currentVariance_;\n> >>> + /* The frames to be ignore before starting measuring. */\n> >>> + uint32_t ignoreFrame_;\n> >>> + /* previous variance. it is used to determine the gradient */\n> >>> + double previousVariance_;\n> >>> + /* Max scan steps of each pass of AF scaning */\n> >>> + uint32_t maxStep_;\n> >>> + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n> >>> + bool pass1Done_;\n> >>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n> >>> + bool pass2Done_;\n> >>> +};\n> >>> +\n> >>> +} /* namespace ipa::ipu3::algorithms */\n> >>> +\n> >>> +} /* namespace libcamera */\n> >>> +\n> >>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n> >>> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> >>> index 4db6ae1d..e1099169 100644\n> >>> --- a/src/ipa/ipu3/algorithms/meson.build\n> >>> +++ b/src/ipa/ipu3/algorithms/meson.build\n> >>> @@ -1,8 +1,9 @@\n> >>> # SPDX-License-Identifier: CC0-1.0\n> >>>\n> >>> ipu3_ipa_algorithms = files([\n> >>> + 'af.cpp',\n> >>> 'agc.cpp',\n> >>> 'awb.cpp',\n> >>> 'blc.cpp',\n> >>> - 'tone_mapping.cpp',\n> >>> + 'tone_mapping.cpp'\n> >>> ])\n> >>> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> >>> index 86794ac1..ee644d3c 100644\n> >>> --- a/src/ipa/ipu3/ipa_context.cpp\n> >>> +++ b/src/ipa/ipu3/ipa_context.cpp\n> >>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n> >>> *\n> >>> * \\var IPASessionConfiguration::grid.stride\n> >>> * \\brief Number of cells on one line including the ImgU padding\n> >>> + *\n> >>> + */\n> >>> +\n> >>> +/**\n> >>> + * \\var IPASessionConfiguration::af\n> >>> + * \\brief AF parameters configuration of the IPA\n> >>> + *\n> >>> + * \\var IPASessionConfiguration::af.start_x\n> >>> + * \\brief The start X position of the AF area\n> >>> + *\n> >>> + * \\var IPASessionConfiguration::af.start_y\n> >>> + * \\brief The start Y position of the AF area\n> >>> + */\n> >>> +\n> >>> +/**\n> >>> + * \\var IPAFrameContext::af\n> >>> + * \\brief Context for the Automatic Focus algorithm\n> >>> + *\n> >>> + * \\struct IPAFrameContext::af\n> >>> + * \\var IPAFrameContext::af.focus\n> >>> + * \\brief Current position of the lens\n> >>> + *\n> >>> + * \\var IPAFrameContext::af.maxVariance\n> >>> + * \\brief The maximum variance of the current image.\n> >>> + *\n> >>> + * \\var IPAFrameContext::af.stable\n> >>> + * \\brief is the image focused?\n> >>> */\n> >>>\n> >>> /**\n> >>> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> >>> index c6dc0814..aa5bf97f 100644\n> >>> --- a/src/ipa/ipu3/ipa_context.h\n> >>> +++ b/src/ipa/ipu3/ipa_context.h\n> >>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n> >>> double minAnalogueGain;\n> >>> double maxAnalogueGain;\n> >>> } agc;\n> >>> +\n> >>> + struct {\n> >>> + uint16_t start_x;\n> >>> + uint16_t start_y;\n> >>> + } af;\n> >>> };\n> >>>\n> >>> struct IPAFrameContext {\n> >>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n> >>> double temperatureK;\n> >>> } awb;\n> >>>\n> >>> + struct {\n> >>> + uint32_t focus;\n> >>> + double maxVariance;\n> >>> + bool stable;\n> >>> + } af;\n> >>> +\n> >>> struct {\n> >>> uint32_t exposure;\n> >>> double gain;\n> >>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> >>> index 3d307708..93966c6f 100644\n> >>> --- a/src/ipa/ipu3/ipu3.cpp\n> >>> +++ b/src/ipa/ipu3/ipu3.cpp\n> >>> @@ -30,6 +30,7 @@\n> >>>\n> >>> #include \"libcamera/internal/mapped_framebuffer.h\"\n> >>>\n> >>> +#include \"algorithms/af.h\"\n> >>> #include \"algorithms/agc.h\"\n> >>> #include \"algorithms/algorithm.h\"\n> >>> #include \"algorithms/awb.h\"\n> >>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> >>> }\n> >>>\n> >>> /* Construct our Algorithms */\n> >>> + algorithms_.push_back(std::make_unique());\n> >>> algorithms_.push_back(std::make_unique());\n> >>> algorithms_.push_back(std::make_unique());\n> >>> algorithms_.push_back(std::make_unique());\n> >>>\n> >>\n> >\n> > Thank you :)\n> >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7EE34BE080\n\tfor ;\n\tMon, 27 Dec 2021 09:55:19 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 99FDA60912;\n\tMon, 27 Dec 2021 10:55:18 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6DC0B605A8\n\tfor ;\n\tMon, 27 Dec 2021 10:55:16 +0100 (CET)","from mail-lj1-f200.google.com (mail-lj1-f200.google.com\n\t[209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-448-t93-xtQUOwaf9kjt8LRYyg-1; Mon, 27 Dec 2021 04:55:08 -0500","by mail-lj1-f200.google.com with SMTP id\n\tc20-20020a2e9d94000000b0021cf7c089d0so705422ljj.21\n\tfor ;\n\tMon, 27 Dec 2021 01:55:08 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"DHQxBh5B\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1640598915;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=lJZ/XkrycJ+xyVObfy9af+k6dhbunSCAMA+1G3/Knmk=;\n\tb=DHQxBh5B/DMKHlg2H4KnMq4GgXTPo6DwTXqP/fg70SSr35fAs24Qe2s0OV4N8FFGVVVNGk\n\tjr2mNexQF6pcb7rxSPsOjUU0dVhB3WRWx19Pe5oGhMWlPcHnZg+I4CS1sC30XbUhA3qEea\n\tQrlQoMQQTPOxH5fReqHq6aOLLN+RCPg=","X-MC-Unique":"t93-xtQUOwaf9kjt8LRYyg-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=lJZ/XkrycJ+xyVObfy9af+k6dhbunSCAMA+1G3/Knmk=;\n\tb=EWR9WQjJ48GVJ/eZQdeMo1QEj9wE0WkXKAMg8HdxOppDZdu6GIRltOxFUhB3+r7URw\n\tG+jiCXLW0HROx+wbHIgDPXZtmGVE/wvqg37DETL1D+Q+y28KrqBELy/Udr5caeJaiVej\n\tLmwxmP4XafaaJs8J5LMTWT17azgrc6CGTVlvh/EWdLObQWJOXRzXdTnTrnwAoIB1pPdN\n\t6rfm3TH+poguL4ElzkOUOx6JqkLrjbrqu9kG7Dgx1myA9sj2lIRJEw5k1EOZBruNdkjh\n\t0mHmOgDUmy0vOF0hRdVg7WjUmNJF61pHv21eHQqhn26bDrIR+nt2fGofsLdGrh8tHlS9\n\tXzeQ==","X-Gm-Message-State":"AOAM5322Yom4KuCB5i5Bu66Bz1U5AT6MsNaUeNAW46bhkYFT0chGp/fW\n\t2z+5blt8SEwqBKgqpX0DkCfIU6kQbM5o0XX+h3za09CN9Ngl3Pp5IyQZpiaJcfy3L2sIrMtWx2F\n\tF6icfwNZt8TTX/Hz9nb3NuNJaAS49YODEf7cXGMOaeY8Udo+u+w==","X-Received":["by 2002:a05:6512:3588:: with SMTP id\n\tm8mr14330188lfr.665.1640598906359; \n\tMon, 27 Dec 2021 01:55:06 -0800 (PST)","by 2002:a05:6512:3588:: with SMTP id\n\tm8mr14330150lfr.665.1640598905514; \n\tMon, 27 Dec 2021 01:55:05 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJzyqs78nJWEZ84PPAcZVdO8q372Bfd0r2XwvwMKJARemj94U4eGMY8WT0GZ8TSIkSlE2NENoUbSHls3DYTwBXo=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>","In-Reply-To":"<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>","From":"Kate Hsuan ","Date":"Mon, 27 Dec 2021 17:54:54 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21886,"web_url":"https://patchwork.libcamera.org/comment/21886/","msgid":"<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>","date":"2021-12-27T12:54:59","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate !\n\nOn 27/12/2021 10:54, Kate Hsuan wrote:\n> Hi Jean-Michel,\n> \n> On Sat, Dec 18, 2021 at 12:18 AM Jean-Michel Hautbois\n> wrote:\n>>\n>> Hi Kate,\n>>\n>> On 17/12/2021 09:57, Kate Hsuan wrote:\n>>> \"Hi Jean-Michel,\n>>>\n>>> Thank you for your review.\n>>>\n>>> On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n>>> wrote:\n>>>>\n>>>> Hi Kate,\n>>>>\n>>>> Thanks for the patch :-) !\n>>>>\n>>>> I have a few comments below, and I have to tell you I couldn't make it\n>>>> work here... I don't know how you are testing it ? I can see the lens\n>>>> position changing but it passes over the correct focus and never locks\n>>>> on it.\n>>>>\n>>>> On 13/12/2021 13:28, Kate Hsuan wrote:\n>>>>> Since VCM for surface Go 2 (dw9719) had been successfully\n>>>>> driven, this Af module can be used to control the VCM and\n>>>>> determine the focus value based on the IPU3 AF state.\n>>>>>\n>>>>> The variance of each focus step is determined and a greedy\n>>>>> approah is used to find the maximum variance of the AF\n>>>>> state and a appropriate focus value.\n>>>>>\n>>>>> Changes in v2:\n>>>>> 1. Add grid configuration interface.\n>>>>> 2. Add AF accelerator configuration.\n>>>>> 3. Move default focus area to the center of the image.\n>>>>>\n>>>>> Changes in v3:\n>>>>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n>>>>> algorithm.\n>>>>> 2. Fix greedy AF algorithm since V4l2 interface had been\n>>>>> removed.\n>>>>> 3. Simplify default grid config for AF.\n>>>>> 4. AF start area is determined by bdsOutputSize.\n>>>>>\n>>>>> Changes in v4:\n>>>>> In v4, it significant improves the AF scan time using a two pass\n>>>>> scaning method and the AF scan will be terminated when it finds\n>>>>> a negative gradient.\n>>>>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n>>>>> the AF success rate.\n>>>>> 2. The low pass filter convolution results are used to determine\n>>>>> a coarse AF result.\n>>>>> 3. The high pass convolution result is used to find a good lens\n>>>>> step in a given AF range determined in pass 1.\n>>>>>\n>>>>\n>>>> The lines above describe the changes between the versions, which is\n>>>> great ! But those should go after the \"---\" as we don't want those in\n>>>> the final commit message.\n>>>\n>>> Ok. Thank you I'll move the changes after \"---\".\n>>>\n>>>>\n>>>>> Signed-off-by: Kate Hsuan \n>>>>> ---\n>>>>\n>>>> Changes in ...\n>>>>\n>>>> ---\n>>>>> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n>>>>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n>>>>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n>>>>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n>>>>> src/ipa/ipu3/ipa_context.h | 11 +\n>>>>> src/ipa/ipu3/ipu3.cpp | 2 +\n>>>>> 6 files changed, 442 insertions(+), 1 deletion(-)\n>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n>>>>>\n>>>>\n>>>> I applied your patch, and I have compilation issues:\n>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n>>>> 89 | };\n>>>> | ^\n>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>> member ‘ipu3_uapi_grid_config::height_per_slice’\n>>>> [-Werror=missing-field-initializers]\n>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n>>>> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n>>>> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n>>>> const ipu3_uapi_stats_3a*)’:\n>>>> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n>>>> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n>>>> value [-Werror=address-of-packed-member]\n>>>> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n>>>>\n>>>> I solved them locally (with a quick and dirty patch) how did you not\n>>>> have those ?\n>>>\n>>> It was my fault. I can build them successfully before. But, after\n>>> re-cloning the git repo and patching my v4 patch, it happened. I'll\n>>> fix those issues.\n>>> Sorry about the inconvenience. T_T\n>>\n>> No problem, but it was surprising :-).\n>>\n>>>\n>>>>\n>>>>> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n>>>>> new file mode 100644\n>>>>> index 00000000..620be9fa\n>>>>> --- /dev/null\n>>>>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n>>>>> @@ -0,0 +1,334 @@\n>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>>>> +/*\n>>>>> + * Copyright (C) 2021, Red Hat\n>>>>> + *\n>>>>> + * af.cpp - IPU3 auto focus control\n>>>>> + */\n>>>>> +\n>>>>> +#include \"af.h\"\n>>>>> +\n>>>>> +#include \n>>>>> +#include \n>>>>> +#include \n>>>>> +#include \n>>>>> +#include \n>>>>> +#include \n>>>>> +#include \n>>>>> +#include \n>>>>> +#include \n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \"libipa/histogram.h\"\n>>>>> +\n>>>>> +/**\n>>>>> + * \\file af.h\n>>>>> + */\n>>>>> +\n>>>>> +namespace libcamera {\n>>>>> +\n>>>>> +using namespace std::literals::chrono_literals;\n>>>>> +\n>>>>> +namespace ipa::ipu3::algorithms {\n>>>>> +\n>>>>> +/**\n>>>>> + * \\class Af\n>>>>> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n>>>>> + *\n>>>>> + * This algorithm is used to determine the position of the lens and get a\n>>>>> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n>>>>> + * by high pass and low pass filtered value and stores in a AF buffer.\n>>>>> + * Typically, for a focused image, it has relative high contrast than a\n>>>>> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n>>>>> + * highest contrast can be found from the AF scan, the lens' position is the\n>>>>> + * best step of the focus.\n>>>>> + *\n>>>>> + */\n>>>>> +\n>>>>> +LOG_DEFINE_CATEGORY(IPU3Af)\n>>>>> +\n>>>>> +/**\n>>>>> + * Maximum focus value of the VCM control\n>>>>> + * \\todo should be obtained from the VCM driver\n>>>>> + */\n>>>>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n>>>>> +\n>>>>> +/* minimum focus step for searching appropriate focus*/\n>>>>> +static constexpr uint32_t coarseSearchStep_ = 10;\n>>>>> +static constexpr uint32_t fineSearchStep_ = 1;\n>>>>> +\n>>>>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n>>>>> +static constexpr double MaxChange_ = 0.8;\n>>>>\n>>>> Why 0.8 ? Any explanation on how to set it ?\n>>>\n>>> OK. I'll put my explanation here. Basically, it is used to represent\n>>> the boundary of a focused and blurred image.\n>>>\n>>>\n>>>>\n>>>>> +\n>>>>> +/* settings for Auto Focus from the kernel */\n>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n>>>>> + .filter_config = {\n>>>>> + { 0, 0, 0, 0 },\n>>>>> + { 0, 0, 0, 0 },\n>>>>> + { 0, 0, 0, 128 },\n>>>>> + 0,\n>>>>> + { 0, 0, 0, 0 },\n>>>>> + { 0, 0, 0, 0 },\n>>>>> + { 0, 0, 0, 128 },\n>>>>> + 0,\n>>>>> + .y_calc = { 8, 8, 8, 8 },\n>>>>> + .nf = { 0, 7, 0, 7, 0 },\n>>>>> + },\n>>\n>> I would like your thinking here (and perhaps someone else). It looks\n>> like the ImgU implements a Gabor filter, but I can't see all the\n>> parameters I would expect from it. Putting it down, I can't see with\n>> this default filter how y1 and y2 could represent low-pass and high\n>> pass, as the filters are the same. But, it could be a horizontal and a\n>> vertical filter instead, with y1 the horizontal and y2 the vertical\n>> resulting filtered values.\n>>\n>> It would be very interesting to display the y1_avg and y2_avg values as\n>> an image, I just don't have lots of time for it. We have 16 bits values,\n>> so we could dump those into a binary file and use python to display the\n>> values as a raw gray image or something similar.\n> \n> You could check out the URL below to see the grey scale image of the AF buffer.\n> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> \n> Since the result is a tiny 16x16 image, all the detail of the original\n> image was compressed into a black block.\n> \n\nNice :-).\nAs expected, as y1 and y2 are the same filter, images are the same.\nCould you test those values ?\n\ny1 =>\n(0, 1, 3, 7\n 11, 13, 1, 2\n 8, 19, 34, 242)\n vector: 7fdffbfe - normalization factor: 9\ny2 =>\n(0, 1, 6, 6\n 13, 25, 3, 0\n 25, 3, 117, 254)\n vector: 4e53ca72 - normalization factor: 9\nChannels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n\n>>\n>> The other things which I find annoying is the non-reliable values I get\n>> right now for the variance. Same scene, exposure and gain fixed to a\n>> value, controled luminosity, and between two executions I might not have\n>> the same result...\n>>\n>>>>> + .grid_cfg = {\n>>>>> + .width = 16,\n>>>>> + .height = 16,\n>>>>> + .block_width_log2 = 3,\n>>>>> + .block_height_log2 = 3,\n>>>>> + .x_start = 10,\n>>>>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n>>>>> + },\n>>>>> +};\n>>>>\n>>>> As mentionned, you missed some fields:\n>>>\n>>> I'll fix this. Thank you.\n>>>\n>>>>\n>>>> '''\n>>>> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n>>>> imgu_css_af_defaults = {\n>>>> .y_calc = { 8, 8, 8, 8 },\n>>>> .nf = { 0, 7, 0, 7, 0 },\n>>>> },\n>>>> + .padding = { 0, 0, 0, 0 },\n>>>> .grid_cfg = {\n>>>> .width = 16,\n>>>> .height = 16,\n>>>> .block_width_log2 = 3,\n>>>> .block_height_log2 = 3,\n>>>> + .height_per_slice = 1,\n>>>> .x_start = 10,\n>>>> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n>>>> + .x_end = 138,\n>>>> + .y_end = 130,\n>>>> },\n>>>> };\n>>>> '''\n>>>>\n>>>>> +\n>>>>> +Af::Af()\n>>>>> + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n>>>>> + pass1Done_(false), pass2Done_(false)\n>>>>> +{\n>>>>> + maxStep_ = MaxFocusSteps_;\n>>>>> +}\n>>>>> +\n>>>>> +Af::~Af()\n>>>>> +{\n>>>>> +}\n>>>>> +\n>>>>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n>>>>\n>>>> Missing documentation for prepare() => you should build the doc ;-)\n>>>\n>>> Ok. I'll add the prepare() document here.\n>>>\n>>>>\n>>>>> +{\n>>>>> + params->use.acc_af = 1;\n>>>>> + params->acc_param.af = imgu_css_af_defaults;\n>>>>> + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n>>>>> + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n>>>>\n>>>> You are never changing the grid size, is it intended ?\n>>>\n>>> Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n>>> more detail? I can modify this according to your opinions.\n>>\n>> Well, yes, you are always using the default 128x128 grid size. But you\n>> can go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in\n>> height. Thus, it means we could evaluate the scene on a small part or\n>> almost all of it (copying the bds grid is not doable as it might have\n>> larger block width, up to 80x60).\n>>\n>> See:\n>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n>>\n>> Note:\n>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n>>\n>> => It makes me wonder if the filtering is fast, or if it may take some\n>> time, meaning that focus evaluation in one frame could be done only at\n>> frame+2 or something like that.\n>>\n>> Could you verify that the focus estimated is in the center of the scene\n>> right now, as it is what you have configured ?\n>>\n>>>\n>>>>\n>>>>> +}\n>>>>> +\n>>>>> +/**\n>>>>> + * \\brief Configure the Af given a configInfo\n>>>>> + * \\param[in] context The shared IPA context\n>>>>> + * \\param[in] configInfo The IPA configuration data\n>>>>> + *\n>>>>> + * \\return 0\n>>>>> + */\n>>>>> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n>>>>> +{\n>>>>> + /* determined focus value i.e. current focus value */\n>>>>> + context.frameContext.af.focus = 0;\n>>>>> + /* maximum variance of the AF statistics */\n>>>>> + context.frameContext.af.maxVariance = 0;\n>>>> double type, maybe s/= 0/= 0.0/ ?\n>>>\n>>> 0.0 is correct. I'll correct this thank you.\n>>>\n>>>>\n>>>>> + /* is focused? if it is true, the AF should be in a stable state. */\n>>>>> + context.frameContext.af.stable = false;\n>>>>> + /* frame to be ignored before start to estimate AF variance. */\n>>>>> + ignoreFrame_ = 10;\n>>>>\n>>>> Could be a constexpr ?\n>>>\n>>> It is a counter to ignore frames since we have to ignore some frames\n>>> to wait for the value from the accelerator to be stable.\n>>> If this should be a constant variable, I can modify this. :)\n>>>\n>>>\n>>>>\n>>>>> +\n>>>>> + /*\n>>>>> + * AF default area configuration\n>>>>> + * Move AF area to the center of the image.\n>>>>> + */\n>>>>> + /* Default AF width is 16x8 = 128 */\n>>>>> + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n>>>>\n>>>> Why are you using a fixed value for the grid ? You could use the value\n>>>> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n>>>>\n>>>>> + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n>>>>> +\n>>>>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n>>>>> + << configInfo.bdsOutputSize.width\n>>>>> + << \" Y: \"\n>>>>> + << configInfo.bdsOutputSize.height;\n>>>>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n>>>>> + << context.configuration.af.start_x\n>>>>> + << \" Y: \"\n>>>>> + << context.configuration.af.start_y;\n>>>>> +\n>>>>> + return 0;\n>>>>> +}\n>>>>> +\n>>>>> +/**\n>>>>> + * \\brief AF coarse scan\n>>>>> + * \\param[in] context The shared IPA context\n>>>>> + *\n>>>>> + */\n>>>>> +void Af::af_coarse_scan(IPAContext &context)\n>>>>\n>>>> Bad naming (camelCase only). Same below.\n>>>\n>>> Ops, sure, I'll modify this.\n>>>\n>>>>\n>>>>> +{\n>>>>> + if (pass1Done_ == true)\n>>>>> + return;\n>>>>> +\n>>>>> + if (af_scan(context, coarseSearchStep_)) {\n>>>>> + pass1Done_ = true;\n>>>>> + context.frameContext.af.maxVariance = 0;\n>>>>> + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n>>>>> + context.frameContext.af.focus = focus_;\n>>>>> + previousVariance_ = 0;\n>>>>> + maxStep_ = focus_ + (focus_ * 0.2);\n>>>>> + }\n>>>>> +}\n>>>>> +\n>>>>> +/**\n>>>>> + * \\brief AF fine scan\n>>>>> + * \\param[in] context The shared IPA context\n>>>>> + *\n>>>>> + */\n>>>>> +void Af::af_fine_scan(IPAContext &context)\n>>>>> +{\n>>>>> + if (pass1Done_ != true)\n>>>>> + return;\n>>>>> +\n>>>>> + if (af_scan(context, fineSearchStep_)) {\n>>>>> + context.frameContext.af.stable = true;\n>>>>> + pass2Done_ = true;\n>>>>> + }\n>>>>> +}\n>>>> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n>>>> seee where it is used...\n>>>\n>>> Since we used two stage AF searching strategy I simply say this is a\n>>> two pass algorithm :)\n>>> I'll renaming this.\n>>>\n>>>>\n>>>>> +\n>>>>> +/**\n>>>>> + * \\brief AF reset\n>>>>> + * \\param[in] context The shared IPA context\n>>>>> + *\n>>>>> + */\n>>>>> +void Af::af_reset(IPAContext &context)\n>>>>> +{\n>>>>> + context.frameContext.af.maxVariance = 0;\n>>>>> + context.frameContext.af.focus = 0;\n>>>>> + focus_ = 0;\n>>>>> + context.frameContext.af.stable = false;\n>>>>> + ignoreFrame_ = 60;\n>>>>\n>>>> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n>>>\n>>> We can tweak this value.\n>>>\n>>>>\n>>>>> + previousVariance_ = 0.0;\n>>>>> + pass1Done_ = false;\n>>>>> + pass2Done_ = false;\n>>>>> + maxStep_ = MaxFocusSteps_;\n>>>>> +}\n>>>>> +\n>>>>> +/**\n>>>>> + * \\brief AF scan\n>>>>> + * \\param[in] context The shared IPA context\n>>>>> + *\n>>>>> + * \\return True, if it finds a AF value.\n>>>>> + */\n>>>>> +bool Af::af_scan(IPAContext &context, int min_step)\n>>>>> +{\n>>>>> + /* find the maximum variance during the AF scan using a greedy strategy */\n>>>>> + if (currentVariance_ > context.frameContext.af.maxVariance) {\n>>>>> + context.frameContext.af.maxVariance = currentVariance_;\n>>>>> + goodFocus_ = focus_;\n>>>>> + }\n>>>>> +\n>>>>> + if (focus_ > maxStep_) {\n>>>>> + /* if reach the max step, move lens to the position and set \"focus stable\". */\n>>>>> + context.frameContext.af.focus = goodFocus_;\n>>>>> + return true;\n>>>>> + } else {\n>>>>> + /* check negative gradient */\n>>>>> + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n>>>>\n>>>> Where is the 0.15 coming from ?\n>>>\n>>> After some testing, I think that If the negative gradient ismore than\n>>> 15% maxVaraiance, the focused image and lens position we get!\n>>> I could put this as a constant variable in my v5 patch.\n>>>\n>>>>\n>>>>> + focus_ += min_step;\n>>>>> + context.frameContext.af.focus = focus_;\n>>>>> + } else {\n>>>>> + context.frameContext.af.focus = goodFocus_;\n>>>>> + previousVariance_ = currentVariance_;\n>>>>> + return true;\n>>>>> + }\n>>>>> + }\n>>>>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n>>>> s/prevrious/previous/\n>>>\n>>> Ops. sorry about the typo. T_T\n>>>\n>>>>\n>>>>> + << previousVariance_\n>>>>> + << \" current: \"\n>>>>> + << currentVariance_\n>>>>> + << \" Diff: \"\n>>>>> + << (currentVariance_ - context.frameContext.af.maxVariance);\n>>>>\n>>>> Interestingly, it seems there is an issue here, according to the log I get ?\n>>>>\n>>>> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n>>>> current: 20012.4 Diff: 0\n>>>>\n>>>> Shouldn't diff be 20012.4 ?\n>>>>\n>>>> I am not sure the algorithm is very strong though... You may never\n>>>> converge and stay in a local minimum without knowing ?\n>>>>\n>>>>> + previousVariance_ = currentVariance_;\n>>>>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n>>>>> + << context.frameContext.af.maxVariance\n>>>>> + << \" Focus step is \"\n>>>>> + << goodFocus_\n>>>>> + << \" Current scan is \"\n>>>>> + << focus_;\n>>>>> + return false;\n>>>>> +}\n>>>>\n>>>> I am not sure to understand how this function should behave... Is it a\n>>>> minimum searching ? Are you looking for argmin(currentVariance -\n>>>> maxVariance) ?\n>>>\n>>> Actually. it is a maximum search. I try to search for the maximum\n>>> variance of the image.\n>>> If the difference is positive, it means the variance still increases.\n>>> We can continue search for the maximum value and update maxVariance_.\n>>> Until, we found a negative difference which means the variance starts\n>>> to go down and image will become blurred.\n>>>\n>>>\n>>>>\n>>>>> +\n>>>>> +/**\n>>>>> + * \\brief Determine the max contrast image and lens position. y_table is the\n>>>>> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n>>>>> + * value. High pass filtered value also represents the sharpness of the image.\n>>>>> + * Based on this, if the image with highest variance of the high pass filtered\n>>>>> + * value (contrast) during the AF scan, the position of the len should be the\n>>>>> + * best focus.\n>>>>> + * \\param[in] context The shared IPA context.\n>>>>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n>>>>> + */\n>>>>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n>>>>> +{\n>>>>> + uint32_t total = 0;\n>>>>> + double mean;\n>>>>> + uint64_t var_sum = 0;\n>>>>> + y_table_item_t *y_item;\n>>>>> + int z = 0;\n>>>>> +\n>>>>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n>>>>> +\n>>>>> + /**\n>>>>> + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n>>>>> + * for a given grid.\n>>>>> + * For pass1: low pass results are used.\n>>>>> + * For pass2: high pass results are used.\n>>>>> + */\n>>>>> + if (pass1Done_) {\n>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n>>>>> + total = total + y_item[z].y2_avg;\n>>>>> + if (y_item[z].y2_avg == 0)\n>>>>> + break;\n>>>>> + }\n>>>>> + mean = total / z;\n>>>>> +\n>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n>>>>> + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n>>>>> + if (y_item[z].y2_avg == 0)\n>>>>> + break;\n>>>>> + }\n>>>>> + } else {\n>>>>\n>>>> /* Calculate the mean of the low pass filter values. */\n>>>>\n>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n>>>>> + total = total + y_item[z].y1_avg;\n>>>>> + if (y_item[z].y1_avg == 0)\n>>>>> + break;\n>>>> Are you sure of that ? Shouldn't you use the grid you set to parse the\n>>>> right number of data ? If you have a local dark area you may stop even\n>>>> if it is not yet the end of the grid ?\n>>>\n>>> I'll find another way to determine the range of the results. May be it\n>>> can based on the grids setting.\n>>>\n>>>>\n>>>>> + }\n>>>>> + mean = total / z;\n>>>>> +\n>>>>\n>>>> /* Calulate the deviation from the mean. */\n>>>>\n>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n>>>>> + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n>>>>> + if (y_item[z].y1_avg == 0)\n>>>>> + break;\n>>>>> + }\n>>>>> + }\n>>>>\n>>>> The two if() blocks are very similar, the only exception beeing the item\n>>>> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n>>>> in the second one). This is not easy to read (according to me).\n>>>\n>>> They are high pass (y2) and low pass (y1) filtered convolution\n>>> results. This structure referees from ipu-ipa repo and I would like to\n>>> keep their original naming.\n>>>\n>>>\n>>>>\n>>>>> + /* Determine the average variance of the frame. */\n>>>>> + currentVariance_ = static_cast(var_sum) / static_cast(z);\n>>>>\n>>>> Side note:\n>>>> If my stats are still correct, I think it should be divided by (z-1) and\n>>>> not z, as we have a sample and not a full population ?\n>>>\n>>> it is z-1 and my fault. I'll correct this. :)\n>>>\n>>>>\n>>>>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n>>>>> +\n>>>>> + if (context.frameContext.af.stable == true) {\n>>>>> + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n>>>>> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n>>>>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n>>>>> + << var_ratio\n>>>>> + << \" current focus: \"\n>>>>> + << context.frameContext.af.focus;\n>>>>> + /**\n>>>>> + * If the change ratio of contrast is over Maxchange_ (out of focus),\n>>>>> + * trigger AF again.\n>>>>> + */\n>>>>> + if (var_ratio > MaxChange_) {\n>>>>> + if (ignoreFrame_ == 0) {\n>>>>> + af_reset(context);\n>>>>> + } else\n>>>>> + ignoreFrame_--;\n>>>>> + } else\n>>>>> + ignoreFrame_ = 10;\n>>>>> + } else {\n>>>>> + if (ignoreFrame_ != 0)\n>>>>> + ignoreFrame_--;\n>>>>> + else {\n>>>>> + af_coarse_scan(context);\n>>>>> + af_fine_scan(context);\n>>>>> + }\n>>>>> + }\n>>>>> +}\n>>>>> +\n>>>>> +} /* namespace ipa::ipu3::algorithms */\n>>>>> +\n>>>>> +} /* namespace libcamera */\n>>>>> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n>>>>> new file mode 100644\n>>>>> index 00000000..b9295b19\n>>>>> --- /dev/null\n>>>>> +++ b/src/ipa/ipu3/algorithms/af.h\n>>>>> @@ -0,0 +1,66 @@\n>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>>>> +/*\n>>>>> + * Copyright (C) 2021, Red Hat\n>>>>> + *\n>>>>> + * af.h - IPU3 Af control\n>>>>> + */\n>>>>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n>>>>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \"algorithm.h\"\n>>>>> +\n>>>>> +namespace libcamera {\n>>>>> +\n>>>>> +namespace ipa::ipu3::algorithms {\n>>>>> +\n>>>>> +class Af : public Algorithm\n>>>>> +{\n>>>>> + /* The format of y_table. From ipu3-ipa repo */\n>>>>> + typedef struct y_table_item {\n>>>>> + uint16_t y1_avg;\n>>>>> + uint16_t y2_avg;\n>>>>> + } y_table_item_t;\n>>>>> +\n>>>>> +public:\n>>>>> + Af();\n>>>>> + ~Af();\n>>>>> +\n>>>>> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n>>>>> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n>>>>> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n>>>>> +\n>>>>> +private:\n>>>>> + void af_coarse_scan(IPAContext &context);\n>>>>> + void af_fine_scan(IPAContext &context);\n>>>>> + bool af_scan(IPAContext &context, int min_step);\n>>>>> + void af_reset(IPAContext &context);\n>>>>> +\n>>>>> + /* Used for focus scan. */\n>>>>> + uint32_t focus_;\n>>>>> + /* Focus good */\n>>>>> + uint32_t goodFocus_;\n>>>>> + /* Recent AF statistic variance. */\n>>>>> + double currentVariance_;\n>>>>> + /* The frames to be ignore before starting measuring. */\n>>>>> + uint32_t ignoreFrame_;\n>>>>> + /* previous variance. it is used to determine the gradient */\n>>>>> + double previousVariance_;\n>>>>> + /* Max scan steps of each pass of AF scaning */\n>>>>> + uint32_t maxStep_;\n>>>>> + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n>>>>> + bool pass1Done_;\n>>>>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n>>>>> + bool pass2Done_;\n>>>>> +};\n>>>>> +\n>>>>> +} /* namespace ipa::ipu3::algorithms */\n>>>>> +\n>>>>> +} /* namespace libcamera */\n>>>>> +\n>>>>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n>>>>> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n>>>>> index 4db6ae1d..e1099169 100644\n>>>>> --- a/src/ipa/ipu3/algorithms/meson.build\n>>>>> +++ b/src/ipa/ipu3/algorithms/meson.build\n>>>>> @@ -1,8 +1,9 @@\n>>>>> # SPDX-License-Identifier: CC0-1.0\n>>>>>\n>>>>> ipu3_ipa_algorithms = files([\n>>>>> + 'af.cpp',\n>>>>> 'agc.cpp',\n>>>>> 'awb.cpp',\n>>>>> 'blc.cpp',\n>>>>> - 'tone_mapping.cpp',\n>>>>> + 'tone_mapping.cpp'\n>>>>> ])\n>>>>> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n>>>>> index 86794ac1..ee644d3c 100644\n>>>>> --- a/src/ipa/ipu3/ipa_context.cpp\n>>>>> +++ b/src/ipa/ipu3/ipa_context.cpp\n>>>>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n>>>>> *\n>>>>> * \\var IPASessionConfiguration::grid.stride\n>>>>> * \\brief Number of cells on one line including the ImgU padding\n>>>>> + *\n>>>>> + */\n>>>>> +\n>>>>> +/**\n>>>>> + * \\var IPASessionConfiguration::af\n>>>>> + * \\brief AF parameters configuration of the IPA\n>>>>> + *\n>>>>> + * \\var IPASessionConfiguration::af.start_x\n>>>>> + * \\brief The start X position of the AF area\n>>>>> + *\n>>>>> + * \\var IPASessionConfiguration::af.start_y\n>>>>> + * \\brief The start Y position of the AF area\n>>>>> + */\n>>>>> +\n>>>>> +/**\n>>>>> + * \\var IPAFrameContext::af\n>>>>> + * \\brief Context for the Automatic Focus algorithm\n>>>>> + *\n>>>>> + * \\struct IPAFrameContext::af\n>>>>> + * \\var IPAFrameContext::af.focus\n>>>>> + * \\brief Current position of the lens\n>>>>> + *\n>>>>> + * \\var IPAFrameContext::af.maxVariance\n>>>>> + * \\brief The maximum variance of the current image.\n>>>>> + *\n>>>>> + * \\var IPAFrameContext::af.stable\n>>>>> + * \\brief is the image focused?\n>>>>> */\n>>>>>\n>>>>> /**\n>>>>> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n>>>>> index c6dc0814..aa5bf97f 100644\n>>>>> --- a/src/ipa/ipu3/ipa_context.h\n>>>>> +++ b/src/ipa/ipu3/ipa_context.h\n>>>>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n>>>>> double minAnalogueGain;\n>>>>> double maxAnalogueGain;\n>>>>> } agc;\n>>>>> +\n>>>>> + struct {\n>>>>> + uint16_t start_x;\n>>>>> + uint16_t start_y;\n>>>>> + } af;\n>>>>> };\n>>>>>\n>>>>> struct IPAFrameContext {\n>>>>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n>>>>> double temperatureK;\n>>>>> } awb;\n>>>>>\n>>>>> + struct {\n>>>>> + uint32_t focus;\n>>>>> + double maxVariance;\n>>>>> + bool stable;\n>>>>> + } af;\n>>>>> +\n>>>>> struct {\n>>>>> uint32_t exposure;\n>>>>> double gain;\n>>>>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n>>>>> index 3d307708..93966c6f 100644\n>>>>> --- a/src/ipa/ipu3/ipu3.cpp\n>>>>> +++ b/src/ipa/ipu3/ipu3.cpp\n>>>>> @@ -30,6 +30,7 @@\n>>>>>\n>>>>> #include \"libcamera/internal/mapped_framebuffer.h\"\n>>>>>\n>>>>> +#include \"algorithms/af.h\"\n>>>>> #include \"algorithms/agc.h\"\n>>>>> #include \"algorithms/algorithm.h\"\n>>>>> #include \"algorithms/awb.h\"\n>>>>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n>>>>> }\n>>>>>\n>>>>> /* Construct our Algorithms */\n>>>>> + algorithms_.push_back(std::make_unique());\n>>>>> algorithms_.push_back(std::make_unique());\n>>>>> algorithms_.push_back(std::make_unique());\n>>>>> algorithms_.push_back(std::make_unique());\n>>>>>\n>>>>\n>>>\n>>> Thank you :)\n>>>\n>>\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DE4A3BF415\n\tfor ;\n\tMon, 27 Dec 2021 12:55:04 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0390960913;\n\tMon, 27 Dec 2021 13:55:04 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2EA2760115\n\tfor ;\n\tMon, 27 Dec 2021 13:55:02 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:9af5:240d:be73:de7f] (unknown\n\t[IPv6:2a01:e0a:169:7140:9af5:240d:be73:de7f])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id BEC398EB;\n\tMon, 27 Dec 2021 13:55:01 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"lVvFvxp3\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1640609701;\n\tbh=rZwySDpzS57YNG/8JTu3nsECJrrL/ppmeYk6+b3CP1I=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=lVvFvxp3HtGJ4dtoOoqBhsTLV3ugjrCFl0lpfppdCYHHcQuTXPB2F8bRd/BfENIm3\n\t4xV/PHrYPB0Rp0Z6eyaUM0Q2iAl3DlLf/pwQNcr/x24W9/Kmcy1KbKjyxYooIV7pIw\n\tBGy2R1u+TbhjuXkcpz6j0HNnNVZEYLNyv1BgQsFQ=","Message-ID":"<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>","Date":"Mon, 27 Dec 2021 13:54:59 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.3.1","Content-Language":"en-US","To":"Kate Hsuan ","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t","From":"Jean-Michel Hautbois ","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21918,"web_url":"https://patchwork.libcamera.org/comment/21918/","msgid":"","date":"2021-12-31T11:02:14","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel,\n\nOn Mon, Dec 27, 2021 at 8:55 PM Jean-Michel Hautbois\n wrote:\n>\n> Hi Kate !\n>\n> On 27/12/2021 10:54, Kate Hsuan wrote:\n> > Hi Jean-Michel,\n> >\n> > On Sat, Dec 18, 2021 at 12:18 AM Jean-Michel Hautbois\n> > wrote:\n> >>\n> >> Hi Kate,\n> >>\n> >> On 17/12/2021 09:57, Kate Hsuan wrote:\n> >>> \"Hi Jean-Michel,\n> >>>\n> >>> Thank you for your review.\n> >>>\n> >>> On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n> >>> wrote:\n> >>>>\n> >>>> Hi Kate,\n> >>>>\n> >>>> Thanks for the patch :-) !\n> >>>>\n> >>>> I have a few comments below, and I have to tell you I couldn't make it\n> >>>> work here... I don't know how you are testing it ? I can see the lens\n> >>>> position changing but it passes over the correct focus and never locks\n> >>>> on it.\n> >>>>\n> >>>> On 13/12/2021 13:28, Kate Hsuan wrote:\n> >>>>> Since VCM for surface Go 2 (dw9719) had been successfully\n> >>>>> driven, this Af module can be used to control the VCM and\n> >>>>> determine the focus value based on the IPU3 AF state.\n> >>>>>\n> >>>>> The variance of each focus step is determined and a greedy\n> >>>>> approah is used to find the maximum variance of the AF\n> >>>>> state and a appropriate focus value.\n> >>>>>\n> >>>>> Changes in v2:\n> >>>>> 1. Add grid configuration interface.\n> >>>>> 2. Add AF accelerator configuration.\n> >>>>> 3. Move default focus area to the center of the image.\n> >>>>>\n> >>>>> Changes in v3:\n> >>>>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n> >>>>> algorithm.\n> >>>>> 2. Fix greedy AF algorithm since V4l2 interface had been\n> >>>>> removed.\n> >>>>> 3. Simplify default grid config for AF.\n> >>>>> 4. AF start area is determined by bdsOutputSize.\n> >>>>>\n> >>>>> Changes in v4:\n> >>>>> In v4, it significant improves the AF scan time using a two pass\n> >>>>> scaning method and the AF scan will be terminated when it finds\n> >>>>> a negative gradient.\n> >>>>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n> >>>>> the AF success rate.\n> >>>>> 2. The low pass filter convolution results are used to determine\n> >>>>> a coarse AF result.\n> >>>>> 3. The high pass convolution result is used to find a good lens\n> >>>>> step in a given AF range determined in pass 1.\n> >>>>>\n> >>>>\n> >>>> The lines above describe the changes between the versions, which is\n> >>>> great ! But those should go after the \"---\" as we don't want those in\n> >>>> the final commit message.\n> >>>\n> >>> Ok. Thank you I'll move the changes after \"---\".\n> >>>\n> >>>>\n> >>>>> Signed-off-by: Kate Hsuan \n> >>>>> ---\n> >>>>\n> >>>> Changes in ...\n> >>>>\n> >>>> ---\n> >>>>> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n> >>>>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n> >>>>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n> >>>>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n> >>>>> src/ipa/ipu3/ipa_context.h | 11 +\n> >>>>> src/ipa/ipu3/ipu3.cpp | 2 +\n> >>>>> 6 files changed, 442 insertions(+), 1 deletion(-)\n> >>>>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> >>>>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n> >>>>>\n> >>>>\n> >>>> I applied your patch, and I have compilation issues:\n> >>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n> >>>> 89 | };\n> >>>> | ^\n> >>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>> member ‘ipu3_uapi_grid_config::height_per_slice’\n> >>>> [-Werror=missing-field-initializers]\n> >>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n> >>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n> >>>> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n> >>>> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n> >>>> const ipu3_uapi_stats_3a*)’:\n> >>>> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n> >>>> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n> >>>> value [-Werror=address-of-packed-member]\n> >>>> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>>>\n> >>>> I solved them locally (with a quick and dirty patch) how did you not\n> >>>> have those ?\n> >>>\n> >>> It was my fault. I can build them successfully before. But, after\n> >>> re-cloning the git repo and patching my v4 patch, it happened. I'll\n> >>> fix those issues.\n> >>> Sorry about the inconvenience. T_T\n> >>\n> >> No problem, but it was surprising :-).\n> >>\n> >>>\n> >>>>\n> >>>>> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> >>>>> new file mode 100644\n> >>>>> index 00000000..620be9fa\n> >>>>> --- /dev/null\n> >>>>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> >>>>> @@ -0,0 +1,334 @@\n> >>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>>>> +/*\n> >>>>> + * Copyright (C) 2021, Red Hat\n> >>>>> + *\n> >>>>> + * af.cpp - IPU3 auto focus control\n> >>>>> + */\n> >>>>> +\n> >>>>> +#include \"af.h\"\n> >>>>> +\n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \"libipa/histogram.h\"\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\file af.h\n> >>>>> + */\n> >>>>> +\n> >>>>> +namespace libcamera {\n> >>>>> +\n> >>>>> +using namespace std::literals::chrono_literals;\n> >>>>> +\n> >>>>> +namespace ipa::ipu3::algorithms {\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\class Af\n> >>>>> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n> >>>>> + *\n> >>>>> + * This algorithm is used to determine the position of the lens and get a\n> >>>>> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n> >>>>> + * by high pass and low pass filtered value and stores in a AF buffer.\n> >>>>> + * Typically, for a focused image, it has relative high contrast than a\n> >>>>> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n> >>>>> + * highest contrast can be found from the AF scan, the lens' position is the\n> >>>>> + * best step of the focus.\n> >>>>> + *\n> >>>>> + */\n> >>>>> +\n> >>>>> +LOG_DEFINE_CATEGORY(IPU3Af)\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * Maximum focus value of the VCM control\n> >>>>> + * \\todo should be obtained from the VCM driver\n> >>>>> + */\n> >>>>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n> >>>>> +\n> >>>>> +/* minimum focus step for searching appropriate focus*/\n> >>>>> +static constexpr uint32_t coarseSearchStep_ = 10;\n> >>>>> +static constexpr uint32_t fineSearchStep_ = 1;\n> >>>>> +\n> >>>>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n> >>>>> +static constexpr double MaxChange_ = 0.8;\n> >>>>\n> >>>> Why 0.8 ? Any explanation on how to set it ?\n> >>>\n> >>> OK. I'll put my explanation here. Basically, it is used to represent\n> >>> the boundary of a focused and blurred image.\n> >>>\n> >>>\n> >>>>\n> >>>>> +\n> >>>>> +/* settings for Auto Focus from the kernel */\n> >>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> >>>>> + .filter_config = {\n> >>>>> + { 0, 0, 0, 0 },\n> >>>>> + { 0, 0, 0, 0 },\n> >>>>> + { 0, 0, 0, 128 },\n> >>>>> + 0,\n> >>>>> + { 0, 0, 0, 0 },\n> >>>>> + { 0, 0, 0, 0 },\n> >>>>> + { 0, 0, 0, 128 },\n> >>>>> + 0,\n> >>>>> + .y_calc = { 8, 8, 8, 8 },\n> >>>>> + .nf = { 0, 7, 0, 7, 0 },\n> >>>>> + },\n> >>\n> >> I would like your thinking here (and perhaps someone else). It looks\n> >> like the ImgU implements a Gabor filter, but I can't see all the\n> >> parameters I would expect from it. Putting it down, I can't see with\n> >> this default filter how y1 and y2 could represent low-pass and high\n> >> pass, as the filters are the same. But, it could be a horizontal and a\n> >> vertical filter instead, with y1 the horizontal and y2 the vertical\n> >> resulting filtered values.\n> >>\n> >> It would be very interesting to display the y1_avg and y2_avg values as\n> >> an image, I just don't have lots of time for it. We have 16 bits values,\n> >> so we could dump those into a binary file and use python to display the\n> >> values as a raw gray image or something similar.\n> >\n> > You could check out the URL below to see the grey scale image of the AF buffer.\n> > https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> >\n> > Since the result is a tiny 16x16 image, all the detail of the original\n> > image was compressed into a black block.\n> >\n>\n> Nice :-).\n> As expected, as y1 and y2 are the same filter, images are the same.\n> Could you test those values ?\n>\n> y1 =>\n> (0, 1, 3, 7\n> 11, 13, 1, 2\n> 8, 19, 34, 242)\n> vector: 7fdffbfe - normalization factor: 9\n> y2 =>\n> (0, 1, 6, 6\n> 13, 25, 3, 0\n> 25, 3, 117, 254)\n> vector: 4e53ca72 - normalization factor: 9\n> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n>\n\nPlease check out the results from the URL below.\n\nhttps://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n\nY1 became a black image and was difficult to determine the coarse focus value.\n\nHappy new year :)\n\n> >>\n> >> The other things which I find annoying is the non-reliable values I get\n> >> right now for the variance. Same scene, exposure and gain fixed to a\n> >> value, controled luminosity, and between two executions I might not have\n> >> the same result...\n> >>\n> >>>>> + .grid_cfg = {\n> >>>>> + .width = 16,\n> >>>>> + .height = 16,\n> >>>>> + .block_width_log2 = 3,\n> >>>>> + .block_height_log2 = 3,\n> >>>>> + .x_start = 10,\n> >>>>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >>>>> + },\n> >>>>> +};\n> >>>>\n> >>>> As mentionned, you missed some fields:\n> >>>\n> >>> I'll fix this. Thank you.\n> >>>\n> >>>>\n> >>>> '''\n> >>>> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n> >>>> imgu_css_af_defaults = {\n> >>>> .y_calc = { 8, 8, 8, 8 },\n> >>>> .nf = { 0, 7, 0, 7, 0 },\n> >>>> },\n> >>>> + .padding = { 0, 0, 0, 0 },\n> >>>> .grid_cfg = {\n> >>>> .width = 16,\n> >>>> .height = 16,\n> >>>> .block_width_log2 = 3,\n> >>>> .block_height_log2 = 3,\n> >>>> + .height_per_slice = 1,\n> >>>> .x_start = 10,\n> >>>> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >>>> + .x_end = 138,\n> >>>> + .y_end = 130,\n> >>>> },\n> >>>> };\n> >>>> '''\n> >>>>\n> >>>>> +\n> >>>>> +Af::Af()\n> >>>>> + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> >>>>> + pass1Done_(false), pass2Done_(false)\n> >>>>> +{\n> >>>>> + maxStep_ = MaxFocusSteps_;\n> >>>>> +}\n> >>>>> +\n> >>>>> +Af::~Af()\n> >>>>> +{\n> >>>>> +}\n> >>>>> +\n> >>>>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n> >>>>\n> >>>> Missing documentation for prepare() => you should build the doc ;-)\n> >>>\n> >>> Ok. I'll add the prepare() document here.\n> >>>\n> >>>>\n> >>>>> +{\n> >>>>> + params->use.acc_af = 1;\n> >>>>> + params->acc_param.af = imgu_css_af_defaults;\n> >>>>> + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n> >>>>> + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n> >>>>\n> >>>> You are never changing the grid size, is it intended ?\n> >>>\n> >>> Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n> >>> more detail? I can modify this according to your opinions.\n> >>\n> >> Well, yes, you are always using the default 128x128 grid size. But you\n> >> can go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in\n> >> height. Thus, it means we could evaluate the scene on a small part or\n> >> almost all of it (copying the bds grid is not doable as it might have\n> >> larger block width, up to 80x60).\n> >>\n> >> See:\n> >> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n> >>\n> >> Note:\n> >> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n> >>\n> >> => It makes me wonder if the filtering is fast, or if it may take some\n> >> time, meaning that focus evaluation in one frame could be done only at\n> >> frame+2 or something like that.\n> >>\n> >> Could you verify that the focus estimated is in the center of the scene\n> >> right now, as it is what you have configured ?\n> >>\n> >>>\n> >>>>\n> >>>>> +}\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\brief Configure the Af given a configInfo\n> >>>>> + * \\param[in] context The shared IPA context\n> >>>>> + * \\param[in] configInfo The IPA configuration data\n> >>>>> + *\n> >>>>> + * \\return 0\n> >>>>> + */\n> >>>>> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> >>>>> +{\n> >>>>> + /* determined focus value i.e. current focus value */\n> >>>>> + context.frameContext.af.focus = 0;\n> >>>>> + /* maximum variance of the AF statistics */\n> >>>>> + context.frameContext.af.maxVariance = 0;\n> >>>> double type, maybe s/= 0/= 0.0/ ?\n> >>>\n> >>> 0.0 is correct. I'll correct this thank you.\n> >>>\n> >>>>\n> >>>>> + /* is focused? if it is true, the AF should be in a stable state. */\n> >>>>> + context.frameContext.af.stable = false;\n> >>>>> + /* frame to be ignored before start to estimate AF variance. */\n> >>>>> + ignoreFrame_ = 10;\n> >>>>\n> >>>> Could be a constexpr ?\n> >>>\n> >>> It is a counter to ignore frames since we have to ignore some frames\n> >>> to wait for the value from the accelerator to be stable.\n> >>> If this should be a constant variable, I can modify this. :)\n> >>>\n> >>>\n> >>>>\n> >>>>> +\n> >>>>> + /*\n> >>>>> + * AF default area configuration\n> >>>>> + * Move AF area to the center of the image.\n> >>>>> + */\n> >>>>> + /* Default AF width is 16x8 = 128 */\n> >>>>> + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n> >>>>\n> >>>> Why are you using a fixed value for the grid ? You could use the value\n> >>>> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n> >>>>\n> >>>>> + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n> >>>>> +\n> >>>>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n> >>>>> + << configInfo.bdsOutputSize.width\n> >>>>> + << \" Y: \"\n> >>>>> + << configInfo.bdsOutputSize.height;\n> >>>>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n> >>>>> + << context.configuration.af.start_x\n> >>>>> + << \" Y: \"\n> >>>>> + << context.configuration.af.start_y;\n> >>>>> +\n> >>>>> + return 0;\n> >>>>> +}\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\brief AF coarse scan\n> >>>>> + * \\param[in] context The shared IPA context\n> >>>>> + *\n> >>>>> + */\n> >>>>> +void Af::af_coarse_scan(IPAContext &context)\n> >>>>\n> >>>> Bad naming (camelCase only). Same below.\n> >>>\n> >>> Ops, sure, I'll modify this.\n> >>>\n> >>>>\n> >>>>> +{\n> >>>>> + if (pass1Done_ == true)\n> >>>>> + return;\n> >>>>> +\n> >>>>> + if (af_scan(context, coarseSearchStep_)) {\n> >>>>> + pass1Done_ = true;\n> >>>>> + context.frameContext.af.maxVariance = 0;\n> >>>>> + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n> >>>>> + context.frameContext.af.focus = focus_;\n> >>>>> + previousVariance_ = 0;\n> >>>>> + maxStep_ = focus_ + (focus_ * 0.2);\n> >>>>> + }\n> >>>>> +}\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\brief AF fine scan\n> >>>>> + * \\param[in] context The shared IPA context\n> >>>>> + *\n> >>>>> + */\n> >>>>> +void Af::af_fine_scan(IPAContext &context)\n> >>>>> +{\n> >>>>> + if (pass1Done_ != true)\n> >>>>> + return;\n> >>>>> +\n> >>>>> + if (af_scan(context, fineSearchStep_)) {\n> >>>>> + context.frameContext.af.stable = true;\n> >>>>> + pass2Done_ = true;\n> >>>>> + }\n> >>>>> +}\n> >>>> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n> >>>> seee where it is used...\n> >>>\n> >>> Since we used two stage AF searching strategy I simply say this is a\n> >>> two pass algorithm :)\n> >>> I'll renaming this.\n> >>>\n> >>>>\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\brief AF reset\n> >>>>> + * \\param[in] context The shared IPA context\n> >>>>> + *\n> >>>>> + */\n> >>>>> +void Af::af_reset(IPAContext &context)\n> >>>>> +{\n> >>>>> + context.frameContext.af.maxVariance = 0;\n> >>>>> + context.frameContext.af.focus = 0;\n> >>>>> + focus_ = 0;\n> >>>>> + context.frameContext.af.stable = false;\n> >>>>> + ignoreFrame_ = 60;\n> >>>>\n> >>>> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n> >>>\n> >>> We can tweak this value.\n> >>>\n> >>>>\n> >>>>> + previousVariance_ = 0.0;\n> >>>>> + pass1Done_ = false;\n> >>>>> + pass2Done_ = false;\n> >>>>> + maxStep_ = MaxFocusSteps_;\n> >>>>> +}\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\brief AF scan\n> >>>>> + * \\param[in] context The shared IPA context\n> >>>>> + *\n> >>>>> + * \\return True, if it finds a AF value.\n> >>>>> + */\n> >>>>> +bool Af::af_scan(IPAContext &context, int min_step)\n> >>>>> +{\n> >>>>> + /* find the maximum variance during the AF scan using a greedy strategy */\n> >>>>> + if (currentVariance_ > context.frameContext.af.maxVariance) {\n> >>>>> + context.frameContext.af.maxVariance = currentVariance_;\n> >>>>> + goodFocus_ = focus_;\n> >>>>> + }\n> >>>>> +\n> >>>>> + if (focus_ > maxStep_) {\n> >>>>> + /* if reach the max step, move lens to the position and set \"focus stable\". */\n> >>>>> + context.frameContext.af.focus = goodFocus_;\n> >>>>> + return true;\n> >>>>> + } else {\n> >>>>> + /* check negative gradient */\n> >>>>> + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n> >>>>\n> >>>> Where is the 0.15 coming from ?\n> >>>\n> >>> After some testing, I think that If the negative gradient ismore than\n> >>> 15% maxVaraiance, the focused image and lens position we get!\n> >>> I could put this as a constant variable in my v5 patch.\n> >>>\n> >>>>\n> >>>>> + focus_ += min_step;\n> >>>>> + context.frameContext.af.focus = focus_;\n> >>>>> + } else {\n> >>>>> + context.frameContext.af.focus = goodFocus_;\n> >>>>> + previousVariance_ = currentVariance_;\n> >>>>> + return true;\n> >>>>> + }\n> >>>>> + }\n> >>>>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n> >>>> s/prevrious/previous/\n> >>>\n> >>> Ops. sorry about the typo. T_T\n> >>>\n> >>>>\n> >>>>> + << previousVariance_\n> >>>>> + << \" current: \"\n> >>>>> + << currentVariance_\n> >>>>> + << \" Diff: \"\n> >>>>> + << (currentVariance_ - context.frameContext.af.maxVariance);\n> >>>>\n> >>>> Interestingly, it seems there is an issue here, according to the log I get ?\n> >>>>\n> >>>> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n> >>>> current: 20012.4 Diff: 0\n> >>>>\n> >>>> Shouldn't diff be 20012.4 ?\n> >>>>\n> >>>> I am not sure the algorithm is very strong though... You may never\n> >>>> converge and stay in a local minimum without knowing ?\n> >>>>\n> >>>>> + previousVariance_ = currentVariance_;\n> >>>>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n> >>>>> + << context.frameContext.af.maxVariance\n> >>>>> + << \" Focus step is \"\n> >>>>> + << goodFocus_\n> >>>>> + << \" Current scan is \"\n> >>>>> + << focus_;\n> >>>>> + return false;\n> >>>>> +}\n> >>>>\n> >>>> I am not sure to understand how this function should behave... Is it a\n> >>>> minimum searching ? Are you looking for argmin(currentVariance -\n> >>>> maxVariance) ?\n> >>>\n> >>> Actually. it is a maximum search. I try to search for the maximum\n> >>> variance of the image.\n> >>> If the difference is positive, it means the variance still increases.\n> >>> We can continue search for the maximum value and update maxVariance_.\n> >>> Until, we found a negative difference which means the variance starts\n> >>> to go down and image will become blurred.\n> >>>\n> >>>\n> >>>>\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\brief Determine the max contrast image and lens position. y_table is the\n> >>>>> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n> >>>>> + * value. High pass filtered value also represents the sharpness of the image.\n> >>>>> + * Based on this, if the image with highest variance of the high pass filtered\n> >>>>> + * value (contrast) during the AF scan, the position of the len should be the\n> >>>>> + * best focus.\n> >>>>> + * \\param[in] context The shared IPA context.\n> >>>>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n> >>>>> + */\n> >>>>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> >>>>> +{\n> >>>>> + uint32_t total = 0;\n> >>>>> + double mean;\n> >>>>> + uint64_t var_sum = 0;\n> >>>>> + y_table_item_t *y_item;\n> >>>>> + int z = 0;\n> >>>>> +\n> >>>>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>>>> +\n> >>>>> + /**\n> >>>>> + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n> >>>>> + * for a given grid.\n> >>>>> + * For pass1: low pass results are used.\n> >>>>> + * For pass2: high pass results are used.\n> >>>>> + */\n> >>>>> + if (pass1Done_) {\n> >>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>>>> + total = total + y_item[z].y2_avg;\n> >>>>> + if (y_item[z].y2_avg == 0)\n> >>>>> + break;\n> >>>>> + }\n> >>>>> + mean = total / z;\n> >>>>> +\n> >>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n> >>>>> + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n> >>>>> + if (y_item[z].y2_avg == 0)\n> >>>>> + break;\n> >>>>> + }\n> >>>>> + } else {\n> >>>>\n> >>>> /* Calculate the mean of the low pass filter values. */\n> >>>>\n> >>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>>>> + total = total + y_item[z].y1_avg;\n> >>>>> + if (y_item[z].y1_avg == 0)\n> >>>>> + break;\n> >>>> Are you sure of that ? Shouldn't you use the grid you set to parse the\n> >>>> right number of data ? If you have a local dark area you may stop even\n> >>>> if it is not yet the end of the grid ?\n> >>>\n> >>> I'll find another way to determine the range of the results. May be it\n> >>> can based on the grids setting.\n> >>>\n> >>>>\n> >>>>> + }\n> >>>>> + mean = total / z;\n> >>>>> +\n> >>>>\n> >>>> /* Calulate the deviation from the mean. */\n> >>>>\n> >>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n> >>>>> + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n> >>>>> + if (y_item[z].y1_avg == 0)\n> >>>>> + break;\n> >>>>> + }\n> >>>>> + }\n> >>>>\n> >>>> The two if() blocks are very similar, the only exception beeing the item\n> >>>> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n> >>>> in the second one). This is not easy to read (according to me).\n> >>>\n> >>> They are high pass (y2) and low pass (y1) filtered convolution\n> >>> results. This structure referees from ipu-ipa repo and I would like to\n> >>> keep their original naming.\n> >>>\n> >>>\n> >>>>\n> >>>>> + /* Determine the average variance of the frame. */\n> >>>>> + currentVariance_ = static_cast(var_sum) / static_cast(z);\n> >>>>\n> >>>> Side note:\n> >>>> If my stats are still correct, I think it should be divided by (z-1) and\n> >>>> not z, as we have a sample and not a full population ?\n> >>>\n> >>> it is z-1 and my fault. I'll correct this. :)\n> >>>\n> >>>>\n> >>>>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n> >>>>> +\n> >>>>> + if (context.frameContext.af.stable == true) {\n> >>>>> + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n> >>>>> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> >>>>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n> >>>>> + << var_ratio\n> >>>>> + << \" current focus: \"\n> >>>>> + << context.frameContext.af.focus;\n> >>>>> + /**\n> >>>>> + * If the change ratio of contrast is over Maxchange_ (out of focus),\n> >>>>> + * trigger AF again.\n> >>>>> + */\n> >>>>> + if (var_ratio > MaxChange_) {\n> >>>>> + if (ignoreFrame_ == 0) {\n> >>>>> + af_reset(context);\n> >>>>> + } else\n> >>>>> + ignoreFrame_--;\n> >>>>> + } else\n> >>>>> + ignoreFrame_ = 10;\n> >>>>> + } else {\n> >>>>> + if (ignoreFrame_ != 0)\n> >>>>> + ignoreFrame_--;\n> >>>>> + else {\n> >>>>> + af_coarse_scan(context);\n> >>>>> + af_fine_scan(context);\n> >>>>> + }\n> >>>>> + }\n> >>>>> +}\n> >>>>> +\n> >>>>> +} /* namespace ipa::ipu3::algorithms */\n> >>>>> +\n> >>>>> +} /* namespace libcamera */\n> >>>>> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> >>>>> new file mode 100644\n> >>>>> index 00000000..b9295b19\n> >>>>> --- /dev/null\n> >>>>> +++ b/src/ipa/ipu3/algorithms/af.h\n> >>>>> @@ -0,0 +1,66 @@\n> >>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>>>> +/*\n> >>>>> + * Copyright (C) 2021, Red Hat\n> >>>>> + *\n> >>>>> + * af.h - IPU3 Af control\n> >>>>> + */\n> >>>>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>>>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \"algorithm.h\"\n> >>>>> +\n> >>>>> +namespace libcamera {\n> >>>>> +\n> >>>>> +namespace ipa::ipu3::algorithms {\n> >>>>> +\n> >>>>> +class Af : public Algorithm\n> >>>>> +{\n> >>>>> + /* The format of y_table. From ipu3-ipa repo */\n> >>>>> + typedef struct y_table_item {\n> >>>>> + uint16_t y1_avg;\n> >>>>> + uint16_t y2_avg;\n> >>>>> + } y_table_item_t;\n> >>>>> +\n> >>>>> +public:\n> >>>>> + Af();\n> >>>>> + ~Af();\n> >>>>> +\n> >>>>> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> >>>>> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> >>>>> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> >>>>> +\n> >>>>> +private:\n> >>>>> + void af_coarse_scan(IPAContext &context);\n> >>>>> + void af_fine_scan(IPAContext &context);\n> >>>>> + bool af_scan(IPAContext &context, int min_step);\n> >>>>> + void af_reset(IPAContext &context);\n> >>>>> +\n> >>>>> + /* Used for focus scan. */\n> >>>>> + uint32_t focus_;\n> >>>>> + /* Focus good */\n> >>>>> + uint32_t goodFocus_;\n> >>>>> + /* Recent AF statistic variance. */\n> >>>>> + double currentVariance_;\n> >>>>> + /* The frames to be ignore before starting measuring. */\n> >>>>> + uint32_t ignoreFrame_;\n> >>>>> + /* previous variance. it is used to determine the gradient */\n> >>>>> + double previousVariance_;\n> >>>>> + /* Max scan steps of each pass of AF scaning */\n> >>>>> + uint32_t maxStep_;\n> >>>>> + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n> >>>>> + bool pass1Done_;\n> >>>>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n> >>>>> + bool pass2Done_;\n> >>>>> +};\n> >>>>> +\n> >>>>> +} /* namespace ipa::ipu3::algorithms */\n> >>>>> +\n> >>>>> +} /* namespace libcamera */\n> >>>>> +\n> >>>>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n> >>>>> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> >>>>> index 4db6ae1d..e1099169 100644\n> >>>>> --- a/src/ipa/ipu3/algorithms/meson.build\n> >>>>> +++ b/src/ipa/ipu3/algorithms/meson.build\n> >>>>> @@ -1,8 +1,9 @@\n> >>>>> # SPDX-License-Identifier: CC0-1.0\n> >>>>>\n> >>>>> ipu3_ipa_algorithms = files([\n> >>>>> + 'af.cpp',\n> >>>>> 'agc.cpp',\n> >>>>> 'awb.cpp',\n> >>>>> 'blc.cpp',\n> >>>>> - 'tone_mapping.cpp',\n> >>>>> + 'tone_mapping.cpp'\n> >>>>> ])\n> >>>>> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> >>>>> index 86794ac1..ee644d3c 100644\n> >>>>> --- a/src/ipa/ipu3/ipa_context.cpp\n> >>>>> +++ b/src/ipa/ipu3/ipa_context.cpp\n> >>>>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n> >>>>> *\n> >>>>> * \\var IPASessionConfiguration::grid.stride\n> >>>>> * \\brief Number of cells on one line including the ImgU padding\n> >>>>> + *\n> >>>>> + */\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\var IPASessionConfiguration::af\n> >>>>> + * \\brief AF parameters configuration of the IPA\n> >>>>> + *\n> >>>>> + * \\var IPASessionConfiguration::af.start_x\n> >>>>> + * \\brief The start X position of the AF area\n> >>>>> + *\n> >>>>> + * \\var IPASessionConfiguration::af.start_y\n> >>>>> + * \\brief The start Y position of the AF area\n> >>>>> + */\n> >>>>> +\n> >>>>> +/**\n> >>>>> + * \\var IPAFrameContext::af\n> >>>>> + * \\brief Context for the Automatic Focus algorithm\n> >>>>> + *\n> >>>>> + * \\struct IPAFrameContext::af\n> >>>>> + * \\var IPAFrameContext::af.focus\n> >>>>> + * \\brief Current position of the lens\n> >>>>> + *\n> >>>>> + * \\var IPAFrameContext::af.maxVariance\n> >>>>> + * \\brief The maximum variance of the current image.\n> >>>>> + *\n> >>>>> + * \\var IPAFrameContext::af.stable\n> >>>>> + * \\brief is the image focused?\n> >>>>> */\n> >>>>>\n> >>>>> /**\n> >>>>> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> >>>>> index c6dc0814..aa5bf97f 100644\n> >>>>> --- a/src/ipa/ipu3/ipa_context.h\n> >>>>> +++ b/src/ipa/ipu3/ipa_context.h\n> >>>>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n> >>>>> double minAnalogueGain;\n> >>>>> double maxAnalogueGain;\n> >>>>> } agc;\n> >>>>> +\n> >>>>> + struct {\n> >>>>> + uint16_t start_x;\n> >>>>> + uint16_t start_y;\n> >>>>> + } af;\n> >>>>> };\n> >>>>>\n> >>>>> struct IPAFrameContext {\n> >>>>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n> >>>>> double temperatureK;\n> >>>>> } awb;\n> >>>>>\n> >>>>> + struct {\n> >>>>> + uint32_t focus;\n> >>>>> + double maxVariance;\n> >>>>> + bool stable;\n> >>>>> + } af;\n> >>>>> +\n> >>>>> struct {\n> >>>>> uint32_t exposure;\n> >>>>> double gain;\n> >>>>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> >>>>> index 3d307708..93966c6f 100644\n> >>>>> --- a/src/ipa/ipu3/ipu3.cpp\n> >>>>> +++ b/src/ipa/ipu3/ipu3.cpp\n> >>>>> @@ -30,6 +30,7 @@\n> >>>>>\n> >>>>> #include \"libcamera/internal/mapped_framebuffer.h\"\n> >>>>>\n> >>>>> +#include \"algorithms/af.h\"\n> >>>>> #include \"algorithms/agc.h\"\n> >>>>> #include \"algorithms/algorithm.h\"\n> >>>>> #include \"algorithms/awb.h\"\n> >>>>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> >>>>> }\n> >>>>>\n> >>>>> /* Construct our Algorithms */\n> >>>>> + algorithms_.push_back(std::make_unique());\n> >>>>> algorithms_.push_back(std::make_unique());\n> >>>>> algorithms_.push_back(std::make_unique());\n> >>>>> algorithms_.push_back(std::make_unique());\n> >>>>>\n> >>>>\n> >>>\n> >>> Thank you :)\n> >>>\n> >>\n> >\n> >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 91FD4BF415\n\tfor ;\n\tFri, 31 Dec 2021 11:02:36 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B86F660914;\n\tFri, 31 Dec 2021 12:02:35 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E0C9A604F8\n\tfor ;\n\tFri, 31 Dec 2021 12:02:33 +0100 (CET)","from mail-lj1-f200.google.com (mail-lj1-f200.google.com\n\t[209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-202-EymLgA_ZMqi20Pb_2OfTpQ-1; Fri, 31 Dec 2021 06:02:29 -0500","by mail-lj1-f200.google.com with SMTP id\n\ts11-20020a05651c048b00b0022d8722e7b5so3641002ljc.23\n\tfor ;\n\tFri, 31 Dec 2021 03:02:28 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"A422XVVQ\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1640948552;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=tsMjyvdp8uop19Wgk9alq2vHm6a+fbb/Zbevmm8/4AY=;\n\tb=A422XVVQq1qKYQLk+p5IJbR5MW2XLv6bgykX0GM55NtmhR8wS0I4CXLOW8kBmyTld6wiNc\n\tL1o1otBOr4B9AbmftPfCZtcOqVy6rLb4ed6AEMzqOf4S91uD+MgUMtNqS4PPdtYyQ+klyk\n\tUX7IsGiaN7oi5LKr61K2ZdXQczO4wXI=","X-MC-Unique":"EymLgA_ZMqi20Pb_2OfTpQ-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=tsMjyvdp8uop19Wgk9alq2vHm6a+fbb/Zbevmm8/4AY=;\n\tb=l/3qrDN/5PWXpMJIu1XKHSbSAr3wwRCNxBdHj8XwuQUMonah/KYKjG51qMt1Tm12aI\n\t8cxaWTs27KD+XVwfMth0CxUhFsLFmmFg458hlvN5EtOH021GvyH0KGCoAL5r2aEOFfik\n\ttz4kWOvT5OtrrJx/hDt9+4g8u1aCV5wUz4hVsorMnGAqa4VLergiCmNTnXe+Hgx2nMB5\n\tTcLcwG0j12cA+AeuXCGs6TY6ieHe6InPjb8RtRYSiDDw6QA6gYEgMA9+1gSY1nXHboia\n\tYyT4c+QJ6i0kAHIYAH8+qoTRGh51D0Mhbl2q4ZuK4RMiqLo7c/OYAgh1GAws0llL7gtZ\n\tSCEw==","X-Gm-Message-State":"AOAM533BRJnh+MEZtybNXKlevuAxBw98DAu/5TDTTmizXwb5kWY+HiTq\n\tgGa5ReKUcpa0Zp/vAObrKnpmHy40h1ww0pKDVYcJyQ5MC/jOH11RPkStDrBQCoiaZ36EMkhP4dV\n\trN0sSOuyTWH7533y7sZLK8F9Kcvqo3kU5PZXUtX88BkdavbBX7g==","X-Received":["by 2002:a05:651c:12c3:: with SMTP id\n\t3mr29105332lje.311.1640948546823; \n\tFri, 31 Dec 2021 03:02:26 -0800 (PST)","by 2002:a05:651c:12c3:: with SMTP id\n\t3mr29105303lje.311.1640948545867; \n\tFri, 31 Dec 2021 03:02:25 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJy5oEWzvhXb/zaZSZW03Pf/nN2B5jECKgG3Sgg0OVhe4VVYK2g+KRAHIGo596HqxGHbhekLCwqjLyToz0PaUy8=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>","In-Reply-To":"<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>","From":"Kate Hsuan ","Date":"Fri, 31 Dec 2021 19:02:14 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21919,"web_url":"https://patchwork.libcamera.org/comment/21919/","msgid":"","date":"2021-12-31T13:13:29","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate,\n\nOn 31/12/2021 12:02, Kate Hsuan wrote:\n> Hi Jean-Michel,\n> \n> On Mon, Dec 27, 2021 at 8:55 PM Jean-Michel Hautbois\n> wrote:\n>>\n>> Hi Kate !\n>>\n>> On 27/12/2021 10:54, Kate Hsuan wrote:\n>>> Hi Jean-Michel,\n>>>\n>>> On Sat, Dec 18, 2021 at 12:18 AM Jean-Michel Hautbois\n>>> wrote:\n>>>>\n>>>> Hi Kate,\n>>>>\n>>>> On 17/12/2021 09:57, Kate Hsuan wrote:\n>>>>> \"Hi Jean-Michel,\n>>>>>\n>>>>> Thank you for your review.\n>>>>>\n>>>>> On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n>>>>> wrote:\n>>>>>>\n>>>>>> Hi Kate,\n>>>>>>\n>>>>>> Thanks for the patch :-) !\n>>>>>>\n>>>>>> I have a few comments below, and I have to tell you I couldn't make it\n>>>>>> work here... I don't know how you are testing it ? I can see the lens\n>>>>>> position changing but it passes over the correct focus and never locks\n>>>>>> on it.\n>>>>>>\n>>>>>> On 13/12/2021 13:28, Kate Hsuan wrote:\n>>>>>>> Since VCM for surface Go 2 (dw9719) had been successfully\n>>>>>>> driven, this Af module can be used to control the VCM and\n>>>>>>> determine the focus value based on the IPU3 AF state.\n>>>>>>>\n>>>>>>> The variance of each focus step is determined and a greedy\n>>>>>>> approah is used to find the maximum variance of the AF\n>>>>>>> state and a appropriate focus value.\n>>>>>>>\n>>>>>>> Changes in v2:\n>>>>>>> 1. Add grid configuration interface.\n>>>>>>> 2. Add AF accelerator configuration.\n>>>>>>> 3. Move default focus area to the center of the image.\n>>>>>>>\n>>>>>>> Changes in v3:\n>>>>>>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n>>>>>>> algorithm.\n>>>>>>> 2. Fix greedy AF algorithm since V4l2 interface had been\n>>>>>>> removed.\n>>>>>>> 3. Simplify default grid config for AF.\n>>>>>>> 4. AF start area is determined by bdsOutputSize.\n>>>>>>>\n>>>>>>> Changes in v4:\n>>>>>>> In v4, it significant improves the AF scan time using a two pass\n>>>>>>> scaning method and the AF scan will be terminated when it finds\n>>>>>>> a negative gradient.\n>>>>>>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n>>>>>>> the AF success rate.\n>>>>>>> 2. The low pass filter convolution results are used to determine\n>>>>>>> a coarse AF result.\n>>>>>>> 3. The high pass convolution result is used to find a good lens\n>>>>>>> step in a given AF range determined in pass 1.\n>>>>>>>\n>>>>>>\n>>>>>> The lines above describe the changes between the versions, which is\n>>>>>> great ! But those should go after the \"---\" as we don't want those in\n>>>>>> the final commit message.\n>>>>>\n>>>>> Ok. Thank you I'll move the changes after \"---\".\n>>>>>\n>>>>>>\n>>>>>>> Signed-off-by: Kate Hsuan \n>>>>>>> ---\n>>>>>>\n>>>>>> Changes in ...\n>>>>>>\n>>>>>> ---\n>>>>>>> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n>>>>>>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n>>>>>>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n>>>>>>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n>>>>>>> src/ipa/ipu3/ipa_context.h | 11 +\n>>>>>>> src/ipa/ipu3/ipu3.cpp | 2 +\n>>>>>>> 6 files changed, 442 insertions(+), 1 deletion(-)\n>>>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n>>>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n>>>>>>>\n>>>>>>\n>>>>>> I applied your patch, and I have compilation issues:\n>>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>>>> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n>>>>>> 89 | };\n>>>>>> | ^\n>>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>>>> member ‘ipu3_uapi_grid_config::height_per_slice’\n>>>>>> [-Werror=missing-field-initializers]\n>>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>>>> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n>>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n>>>>>> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n>>>>>> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n>>>>>> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n>>>>>> const ipu3_uapi_stats_3a*)’:\n>>>>>> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n>>>>>> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n>>>>>> value [-Werror=address-of-packed-member]\n>>>>>> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n>>>>>>\n>>>>>> I solved them locally (with a quick and dirty patch) how did you not\n>>>>>> have those ?\n>>>>>\n>>>>> It was my fault. I can build them successfully before. But, after\n>>>>> re-cloning the git repo and patching my v4 patch, it happened. I'll\n>>>>> fix those issues.\n>>>>> Sorry about the inconvenience. T_T\n>>>>\n>>>> No problem, but it was surprising :-).\n>>>>\n>>>>>\n>>>>>>\n>>>>>>> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n>>>>>>> new file mode 100644\n>>>>>>> index 00000000..620be9fa\n>>>>>>> --- /dev/null\n>>>>>>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n>>>>>>> @@ -0,0 +1,334 @@\n>>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>>>>>> +/*\n>>>>>>> + * Copyright (C) 2021, Red Hat\n>>>>>>> + *\n>>>>>>> + * af.cpp - IPU3 auto focus control\n>>>>>>> + */\n>>>>>>> +\n>>>>>>> +#include \"af.h\"\n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \"libipa/histogram.h\"\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\file af.h\n>>>>>>> + */\n>>>>>>> +\n>>>>>>> +namespace libcamera {\n>>>>>>> +\n>>>>>>> +using namespace std::literals::chrono_literals;\n>>>>>>> +\n>>>>>>> +namespace ipa::ipu3::algorithms {\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\class Af\n>>>>>>> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n>>>>>>> + *\n>>>>>>> + * This algorithm is used to determine the position of the lens and get a\n>>>>>>> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n>>>>>>> + * by high pass and low pass filtered value and stores in a AF buffer.\n>>>>>>> + * Typically, for a focused image, it has relative high contrast than a\n>>>>>>> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n>>>>>>> + * highest contrast can be found from the AF scan, the lens' position is the\n>>>>>>> + * best step of the focus.\n>>>>>>> + *\n>>>>>>> + */\n>>>>>>> +\n>>>>>>> +LOG_DEFINE_CATEGORY(IPU3Af)\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * Maximum focus value of the VCM control\n>>>>>>> + * \\todo should be obtained from the VCM driver\n>>>>>>> + */\n>>>>>>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n>>>>>>> +\n>>>>>>> +/* minimum focus step for searching appropriate focus*/\n>>>>>>> +static constexpr uint32_t coarseSearchStep_ = 10;\n>>>>>>> +static constexpr uint32_t fineSearchStep_ = 1;\n>>>>>>> +\n>>>>>>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n>>>>>>> +static constexpr double MaxChange_ = 0.8;\n>>>>>>\n>>>>>> Why 0.8 ? Any explanation on how to set it ?\n>>>>>\n>>>>> OK. I'll put my explanation here. Basically, it is used to represent\n>>>>> the boundary of a focused and blurred image.\n>>>>>\n>>>>>\n>>>>>>\n>>>>>>> +\n>>>>>>> +/* settings for Auto Focus from the kernel */\n>>>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n>>>>>>> + .filter_config = {\n>>>>>>> + { 0, 0, 0, 0 },\n>>>>>>> + { 0, 0, 0, 0 },\n>>>>>>> + { 0, 0, 0, 128 },\n>>>>>>> + 0,\n>>>>>>> + { 0, 0, 0, 0 },\n>>>>>>> + { 0, 0, 0, 0 },\n>>>>>>> + { 0, 0, 0, 128 },\n>>>>>>> + 0,\n>>>>>>> + .y_calc = { 8, 8, 8, 8 },\n>>>>>>> + .nf = { 0, 7, 0, 7, 0 },\n>>>>>>> + },\n>>>>\n>>>> I would like your thinking here (and perhaps someone else). It looks\n>>>> like the ImgU implements a Gabor filter, but I can't see all the\n>>>> parameters I would expect from it. Putting it down, I can't see with\n>>>> this default filter how y1 and y2 could represent low-pass and high\n>>>> pass, as the filters are the same. But, it could be a horizontal and a\n>>>> vertical filter instead, with y1 the horizontal and y2 the vertical\n>>>> resulting filtered values.\n>>>>\n>>>> It would be very interesting to display the y1_avg and y2_avg values as\n>>>> an image, I just don't have lots of time for it. We have 16 bits values,\n>>>> so we could dump those into a binary file and use python to display the\n>>>> values as a raw gray image or something similar.\n>>>\n>>> You could check out the URL below to see the grey scale image of the AF buffer.\n>>> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n>>>\n>>> Since the result is a tiny 16x16 image, all the detail of the original\n>>> image was compressed into a black block.\n>>>\n>>\n>> Nice :-).\n>> As expected, as y1 and y2 are the same filter, images are the same.\n>> Could you test those values ?\n>>\n>> y1 =>\n>> (0, 1, 3, 7\n>> 11, 13, 1, 2\n>> 8, 19, 34, 242)\n>> vector: 7fdffbfe - normalization factor: 9\n>> y2 =>\n>> (0, 1, 6, 6\n>> 13, 25, 3, 0\n>> 25, 3, 117, 254)\n>> vector: 4e53ca72 - normalization factor: 9\n>> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n>>\n> \n> Please check out the results from the URL below.\n> \n> https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n> \n> Y1 became a black image and was difficult to determine the coarse focus value.\n> \n\nSounds surprising... Could you share the way you are generating the \nimages of Y1 and Y2 BTW ?\n\n> Happy new year :)\n\nBest wishes, hoping for a far away Covid next year !\n\n> \n>>>>\n>>>> The other things which I find annoying is the non-reliable values I get\n>>>> right now for the variance. Same scene, exposure and gain fixed to a\n>>>> value, controled luminosity, and between two executions I might not have\n>>>> the same result...\n>>>>\n>>>>>>> + .grid_cfg = {\n>>>>>>> + .width = 16,\n>>>>>>> + .height = 16,\n>>>>>>> + .block_width_log2 = 3,\n>>>>>>> + .block_height_log2 = 3,\n>>>>>>> + .x_start = 10,\n>>>>>>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n>>>>>>> + },\n>>>>>>> +};\n>>>>>>\n>>>>>> As mentionned, you missed some fields:\n>>>>>\n>>>>> I'll fix this. Thank you.\n>>>>>\n>>>>>>\n>>>>>> '''\n>>>>>> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n>>>>>> imgu_css_af_defaults = {\n>>>>>> .y_calc = { 8, 8, 8, 8 },\n>>>>>> .nf = { 0, 7, 0, 7, 0 },\n>>>>>> },\n>>>>>> + .padding = { 0, 0, 0, 0 },\n>>>>>> .grid_cfg = {\n>>>>>> .width = 16,\n>>>>>> .height = 16,\n>>>>>> .block_width_log2 = 3,\n>>>>>> .block_height_log2 = 3,\n>>>>>> + .height_per_slice = 1,\n>>>>>> .x_start = 10,\n>>>>>> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n>>>>>> + .x_end = 138,\n>>>>>> + .y_end = 130,\n>>>>>> },\n>>>>>> };\n>>>>>> '''\n>>>>>>\n>>>>>>> +\n>>>>>>> +Af::Af()\n>>>>>>> + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n>>>>>>> + pass1Done_(false), pass2Done_(false)\n>>>>>>> +{\n>>>>>>> + maxStep_ = MaxFocusSteps_;\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +Af::~Af()\n>>>>>>> +{\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n>>>>>>\n>>>>>> Missing documentation for prepare() => you should build the doc ;-)\n>>>>>\n>>>>> Ok. I'll add the prepare() document here.\n>>>>>\n>>>>>>\n>>>>>>> +{\n>>>>>>> + params->use.acc_af = 1;\n>>>>>>> + params->acc_param.af = imgu_css_af_defaults;\n>>>>>>> + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n>>>>>>> + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n>>>>>>\n>>>>>> You are never changing the grid size, is it intended ?\n>>>>>\n>>>>> Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n>>>>> more detail? I can modify this according to your opinions.\n>>>>\n>>>> Well, yes, you are always using the default 128x128 grid size. But you\n>>>> can go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in\n>>>> height. Thus, it means we could evaluate the scene on a small part or\n>>>> almost all of it (copying the bds grid is not doable as it might have\n>>>> larger block width, up to 80x60).\n>>>>\n>>>> See:\n>>>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n>>>>\n>>>> Note:\n>>>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n>>>>\n>>>> => It makes me wonder if the filtering is fast, or if it may take some\n>>>> time, meaning that focus evaluation in one frame could be done only at\n>>>> frame+2 or something like that.\n>>>>\n>>>> Could you verify that the focus estimated is in the center of the scene\n>>>> right now, as it is what you have configured ?\n>>>>\n>>>>>\n>>>>>>\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\brief Configure the Af given a configInfo\n>>>>>>> + * \\param[in] context The shared IPA context\n>>>>>>> + * \\param[in] configInfo The IPA configuration data\n>>>>>>> + *\n>>>>>>> + * \\return 0\n>>>>>>> + */\n>>>>>>> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n>>>>>>> +{\n>>>>>>> + /* determined focus value i.e. current focus value */\n>>>>>>> + context.frameContext.af.focus = 0;\n>>>>>>> + /* maximum variance of the AF statistics */\n>>>>>>> + context.frameContext.af.maxVariance = 0;\n>>>>>> double type, maybe s/= 0/= 0.0/ ?\n>>>>>\n>>>>> 0.0 is correct. I'll correct this thank you.\n>>>>>\n>>>>>>\n>>>>>>> + /* is focused? if it is true, the AF should be in a stable state. */\n>>>>>>> + context.frameContext.af.stable = false;\n>>>>>>> + /* frame to be ignored before start to estimate AF variance. */\n>>>>>>> + ignoreFrame_ = 10;\n>>>>>>\n>>>>>> Could be a constexpr ?\n>>>>>\n>>>>> It is a counter to ignore frames since we have to ignore some frames\n>>>>> to wait for the value from the accelerator to be stable.\n>>>>> If this should be a constant variable, I can modify this. :)\n>>>>>\n>>>>>\n>>>>>>\n>>>>>>> +\n>>>>>>> + /*\n>>>>>>> + * AF default area configuration\n>>>>>>> + * Move AF area to the center of the image.\n>>>>>>> + */\n>>>>>>> + /* Default AF width is 16x8 = 128 */\n>>>>>>> + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n>>>>>>\n>>>>>> Why are you using a fixed value for the grid ? You could use the value\n>>>>>> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n>>>>>>\n>>>>>>> + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n>>>>>>> +\n>>>>>>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n>>>>>>> + << configInfo.bdsOutputSize.width\n>>>>>>> + << \" Y: \"\n>>>>>>> + << configInfo.bdsOutputSize.height;\n>>>>>>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n>>>>>>> + << context.configuration.af.start_x\n>>>>>>> + << \" Y: \"\n>>>>>>> + << context.configuration.af.start_y;\n>>>>>>> +\n>>>>>>> + return 0;\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\brief AF coarse scan\n>>>>>>> + * \\param[in] context The shared IPA context\n>>>>>>> + *\n>>>>>>> + */\n>>>>>>> +void Af::af_coarse_scan(IPAContext &context)\n>>>>>>\n>>>>>> Bad naming (camelCase only). Same below.\n>>>>>\n>>>>> Ops, sure, I'll modify this.\n>>>>>\n>>>>>>\n>>>>>>> +{\n>>>>>>> + if (pass1Done_ == true)\n>>>>>>> + return;\n>>>>>>> +\n>>>>>>> + if (af_scan(context, coarseSearchStep_)) {\n>>>>>>> + pass1Done_ = true;\n>>>>>>> + context.frameContext.af.maxVariance = 0;\n>>>>>>> + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n>>>>>>> + context.frameContext.af.focus = focus_;\n>>>>>>> + previousVariance_ = 0;\n>>>>>>> + maxStep_ = focus_ + (focus_ * 0.2);\n>>>>>>> + }\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\brief AF fine scan\n>>>>>>> + * \\param[in] context The shared IPA context\n>>>>>>> + *\n>>>>>>> + */\n>>>>>>> +void Af::af_fine_scan(IPAContext &context)\n>>>>>>> +{\n>>>>>>> + if (pass1Done_ != true)\n>>>>>>> + return;\n>>>>>>> +\n>>>>>>> + if (af_scan(context, fineSearchStep_)) {\n>>>>>>> + context.frameContext.af.stable = true;\n>>>>>>> + pass2Done_ = true;\n>>>>>>> + }\n>>>>>>> +}\n>>>>>> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n>>>>>> seee where it is used...\n>>>>>\n>>>>> Since we used two stage AF searching strategy I simply say this is a\n>>>>> two pass algorithm :)\n>>>>> I'll renaming this.\n>>>>>\n>>>>>>\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\brief AF reset\n>>>>>>> + * \\param[in] context The shared IPA context\n>>>>>>> + *\n>>>>>>> + */\n>>>>>>> +void Af::af_reset(IPAContext &context)\n>>>>>>> +{\n>>>>>>> + context.frameContext.af.maxVariance = 0;\n>>>>>>> + context.frameContext.af.focus = 0;\n>>>>>>> + focus_ = 0;\n>>>>>>> + context.frameContext.af.stable = false;\n>>>>>>> + ignoreFrame_ = 60;\n>>>>>>\n>>>>>> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n>>>>>\n>>>>> We can tweak this value.\n>>>>>\n>>>>>>\n>>>>>>> + previousVariance_ = 0.0;\n>>>>>>> + pass1Done_ = false;\n>>>>>>> + pass2Done_ = false;\n>>>>>>> + maxStep_ = MaxFocusSteps_;\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\brief AF scan\n>>>>>>> + * \\param[in] context The shared IPA context\n>>>>>>> + *\n>>>>>>> + * \\return True, if it finds a AF value.\n>>>>>>> + */\n>>>>>>> +bool Af::af_scan(IPAContext &context, int min_step)\n>>>>>>> +{\n>>>>>>> + /* find the maximum variance during the AF scan using a greedy strategy */\n>>>>>>> + if (currentVariance_ > context.frameContext.af.maxVariance) {\n>>>>>>> + context.frameContext.af.maxVariance = currentVariance_;\n>>>>>>> + goodFocus_ = focus_;\n>>>>>>> + }\n>>>>>>> +\n>>>>>>> + if (focus_ > maxStep_) {\n>>>>>>> + /* if reach the max step, move lens to the position and set \"focus stable\". */\n>>>>>>> + context.frameContext.af.focus = goodFocus_;\n>>>>>>> + return true;\n>>>>>>> + } else {\n>>>>>>> + /* check negative gradient */\n>>>>>>> + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n>>>>>>\n>>>>>> Where is the 0.15 coming from ?\n>>>>>\n>>>>> After some testing, I think that If the negative gradient ismore than\n>>>>> 15% maxVaraiance, the focused image and lens position we get!\n>>>>> I could put this as a constant variable in my v5 patch.\n>>>>>\n>>>>>>\n>>>>>>> + focus_ += min_step;\n>>>>>>> + context.frameContext.af.focus = focus_;\n>>>>>>> + } else {\n>>>>>>> + context.frameContext.af.focus = goodFocus_;\n>>>>>>> + previousVariance_ = currentVariance_;\n>>>>>>> + return true;\n>>>>>>> + }\n>>>>>>> + }\n>>>>>>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n>>>>>> s/prevrious/previous/\n>>>>>\n>>>>> Ops. sorry about the typo. T_T\n>>>>>\n>>>>>>\n>>>>>>> + << previousVariance_\n>>>>>>> + << \" current: \"\n>>>>>>> + << currentVariance_\n>>>>>>> + << \" Diff: \"\n>>>>>>> + << (currentVariance_ - context.frameContext.af.maxVariance);\n>>>>>>\n>>>>>> Interestingly, it seems there is an issue here, according to the log I get ?\n>>>>>>\n>>>>>> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n>>>>>> current: 20012.4 Diff: 0\n>>>>>>\n>>>>>> Shouldn't diff be 20012.4 ?\n>>>>>>\n>>>>>> I am not sure the algorithm is very strong though... You may never\n>>>>>> converge and stay in a local minimum without knowing ?\n>>>>>>\n>>>>>>> + previousVariance_ = currentVariance_;\n>>>>>>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n>>>>>>> + << context.frameContext.af.maxVariance\n>>>>>>> + << \" Focus step is \"\n>>>>>>> + << goodFocus_\n>>>>>>> + << \" Current scan is \"\n>>>>>>> + << focus_;\n>>>>>>> + return false;\n>>>>>>> +}\n>>>>>>\n>>>>>> I am not sure to understand how this function should behave... Is it a\n>>>>>> minimum searching ? Are you looking for argmin(currentVariance -\n>>>>>> maxVariance) ?\n>>>>>\n>>>>> Actually. it is a maximum search. I try to search for the maximum\n>>>>> variance of the image.\n>>>>> If the difference is positive, it means the variance still increases.\n>>>>> We can continue search for the maximum value and update maxVariance_.\n>>>>> Until, we found a negative difference which means the variance starts\n>>>>> to go down and image will become blurred.\n>>>>>\n>>>>>\n>>>>>>\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\brief Determine the max contrast image and lens position. y_table is the\n>>>>>>> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n>>>>>>> + * value. High pass filtered value also represents the sharpness of the image.\n>>>>>>> + * Based on this, if the image with highest variance of the high pass filtered\n>>>>>>> + * value (contrast) during the AF scan, the position of the len should be the\n>>>>>>> + * best focus.\n>>>>>>> + * \\param[in] context The shared IPA context.\n>>>>>>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n>>>>>>> + */\n>>>>>>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n>>>>>>> +{\n>>>>>>> + uint32_t total = 0;\n>>>>>>> + double mean;\n>>>>>>> + uint64_t var_sum = 0;\n>>>>>>> + y_table_item_t *y_item;\n>>>>>>> + int z = 0;\n>>>>>>> +\n>>>>>>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n>>>>>>> +\n>>>>>>> + /**\n>>>>>>> + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n>>>>>>> + * for a given grid.\n>>>>>>> + * For pass1: low pass results are used.\n>>>>>>> + * For pass2: high pass results are used.\n>>>>>>> + */\n>>>>>>> + if (pass1Done_) {\n>>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n>>>>>>> + total = total + y_item[z].y2_avg;\n>>>>>>> + if (y_item[z].y2_avg == 0)\n>>>>>>> + break;\n>>>>>>> + }\n>>>>>>> + mean = total / z;\n>>>>>>> +\n>>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n>>>>>>> + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n>>>>>>> + if (y_item[z].y2_avg == 0)\n>>>>>>> + break;\n>>>>>>> + }\n>>>>>>> + } else {\n>>>>>>\n>>>>>> /* Calculate the mean of the low pass filter values. */\n>>>>>>\n>>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n>>>>>>> + total = total + y_item[z].y1_avg;\n>>>>>>> + if (y_item[z].y1_avg == 0)\n>>>>>>> + break;\n>>>>>> Are you sure of that ? Shouldn't you use the grid you set to parse the\n>>>>>> right number of data ? If you have a local dark area you may stop even\n>>>>>> if it is not yet the end of the grid ?\n>>>>>\n>>>>> I'll find another way to determine the range of the results. May be it\n>>>>> can based on the grids setting.\n>>>>>\n>>>>>>\n>>>>>>> + }\n>>>>>>> + mean = total / z;\n>>>>>>> +\n>>>>>>\n>>>>>> /* Calulate the deviation from the mean. */\n>>>>>>\n>>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n>>>>>>> + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n>>>>>>> + if (y_item[z].y1_avg == 0)\n>>>>>>> + break;\n>>>>>>> + }\n>>>>>>> + }\n>>>>>>\n>>>>>> The two if() blocks are very similar, the only exception beeing the item\n>>>>>> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n>>>>>> in the second one). This is not easy to read (according to me).\n>>>>>\n>>>>> They are high pass (y2) and low pass (y1) filtered convolution\n>>>>> results. This structure referees from ipu-ipa repo and I would like to\n>>>>> keep their original naming.\n>>>>>\n>>>>>\n>>>>>>\n>>>>>>> + /* Determine the average variance of the frame. */\n>>>>>>> + currentVariance_ = static_cast(var_sum) / static_cast(z);\n>>>>>>\n>>>>>> Side note:\n>>>>>> If my stats are still correct, I think it should be divided by (z-1) and\n>>>>>> not z, as we have a sample and not a full population ?\n>>>>>\n>>>>> it is z-1 and my fault. I'll correct this. :)\n>>>>>\n>>>>>>\n>>>>>>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n>>>>>>> +\n>>>>>>> + if (context.frameContext.af.stable == true) {\n>>>>>>> + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n>>>>>>> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n>>>>>>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n>>>>>>> + << var_ratio\n>>>>>>> + << \" current focus: \"\n>>>>>>> + << context.frameContext.af.focus;\n>>>>>>> + /**\n>>>>>>> + * If the change ratio of contrast is over Maxchange_ (out of focus),\n>>>>>>> + * trigger AF again.\n>>>>>>> + */\n>>>>>>> + if (var_ratio > MaxChange_) {\n>>>>>>> + if (ignoreFrame_ == 0) {\n>>>>>>> + af_reset(context);\n>>>>>>> + } else\n>>>>>>> + ignoreFrame_--;\n>>>>>>> + } else\n>>>>>>> + ignoreFrame_ = 10;\n>>>>>>> + } else {\n>>>>>>> + if (ignoreFrame_ != 0)\n>>>>>>> + ignoreFrame_--;\n>>>>>>> + else {\n>>>>>>> + af_coarse_scan(context);\n>>>>>>> + af_fine_scan(context);\n>>>>>>> + }\n>>>>>>> + }\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +} /* namespace ipa::ipu3::algorithms */\n>>>>>>> +\n>>>>>>> +} /* namespace libcamera */\n>>>>>>> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n>>>>>>> new file mode 100644\n>>>>>>> index 00000000..b9295b19\n>>>>>>> --- /dev/null\n>>>>>>> +++ b/src/ipa/ipu3/algorithms/af.h\n>>>>>>> @@ -0,0 +1,66 @@\n>>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>>>>>> +/*\n>>>>>>> + * Copyright (C) 2021, Red Hat\n>>>>>>> + *\n>>>>>>> + * af.h - IPU3 Af control\n>>>>>>> + */\n>>>>>>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n>>>>>>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \"algorithm.h\"\n>>>>>>> +\n>>>>>>> +namespace libcamera {\n>>>>>>> +\n>>>>>>> +namespace ipa::ipu3::algorithms {\n>>>>>>> +\n>>>>>>> +class Af : public Algorithm\n>>>>>>> +{\n>>>>>>> + /* The format of y_table. From ipu3-ipa repo */\n>>>>>>> + typedef struct y_table_item {\n>>>>>>> + uint16_t y1_avg;\n>>>>>>> + uint16_t y2_avg;\n>>>>>>> + } y_table_item_t;\n>>>>>>> +\n>>>>>>> +public:\n>>>>>>> + Af();\n>>>>>>> + ~Af();\n>>>>>>> +\n>>>>>>> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n>>>>>>> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n>>>>>>> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n>>>>>>> +\n>>>>>>> +private:\n>>>>>>> + void af_coarse_scan(IPAContext &context);\n>>>>>>> + void af_fine_scan(IPAContext &context);\n>>>>>>> + bool af_scan(IPAContext &context, int min_step);\n>>>>>>> + void af_reset(IPAContext &context);\n>>>>>>> +\n>>>>>>> + /* Used for focus scan. */\n>>>>>>> + uint32_t focus_;\n>>>>>>> + /* Focus good */\n>>>>>>> + uint32_t goodFocus_;\n>>>>>>> + /* Recent AF statistic variance. */\n>>>>>>> + double currentVariance_;\n>>>>>>> + /* The frames to be ignore before starting measuring. */\n>>>>>>> + uint32_t ignoreFrame_;\n>>>>>>> + /* previous variance. it is used to determine the gradient */\n>>>>>>> + double previousVariance_;\n>>>>>>> + /* Max scan steps of each pass of AF scaning */\n>>>>>>> + uint32_t maxStep_;\n>>>>>>> + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n>>>>>>> + bool pass1Done_;\n>>>>>>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n>>>>>>> + bool pass2Done_;\n>>>>>>> +};\n>>>>>>> +\n>>>>>>> +} /* namespace ipa::ipu3::algorithms */\n>>>>>>> +\n>>>>>>> +} /* namespace libcamera */\n>>>>>>> +\n>>>>>>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n>>>>>>> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n>>>>>>> index 4db6ae1d..e1099169 100644\n>>>>>>> --- a/src/ipa/ipu3/algorithms/meson.build\n>>>>>>> +++ b/src/ipa/ipu3/algorithms/meson.build\n>>>>>>> @@ -1,8 +1,9 @@\n>>>>>>> # SPDX-License-Identifier: CC0-1.0\n>>>>>>>\n>>>>>>> ipu3_ipa_algorithms = files([\n>>>>>>> + 'af.cpp',\n>>>>>>> 'agc.cpp',\n>>>>>>> 'awb.cpp',\n>>>>>>> 'blc.cpp',\n>>>>>>> - 'tone_mapping.cpp',\n>>>>>>> + 'tone_mapping.cpp'\n>>>>>>> ])\n>>>>>>> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n>>>>>>> index 86794ac1..ee644d3c 100644\n>>>>>>> --- a/src/ipa/ipu3/ipa_context.cpp\n>>>>>>> +++ b/src/ipa/ipu3/ipa_context.cpp\n>>>>>>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n>>>>>>> *\n>>>>>>> * \\var IPASessionConfiguration::grid.stride\n>>>>>>> * \\brief Number of cells on one line including the ImgU padding\n>>>>>>> + *\n>>>>>>> + */\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\var IPASessionConfiguration::af\n>>>>>>> + * \\brief AF parameters configuration of the IPA\n>>>>>>> + *\n>>>>>>> + * \\var IPASessionConfiguration::af.start_x\n>>>>>>> + * \\brief The start X position of the AF area\n>>>>>>> + *\n>>>>>>> + * \\var IPASessionConfiguration::af.start_y\n>>>>>>> + * \\brief The start Y position of the AF area\n>>>>>>> + */\n>>>>>>> +\n>>>>>>> +/**\n>>>>>>> + * \\var IPAFrameContext::af\n>>>>>>> + * \\brief Context for the Automatic Focus algorithm\n>>>>>>> + *\n>>>>>>> + * \\struct IPAFrameContext::af\n>>>>>>> + * \\var IPAFrameContext::af.focus\n>>>>>>> + * \\brief Current position of the lens\n>>>>>>> + *\n>>>>>>> + * \\var IPAFrameContext::af.maxVariance\n>>>>>>> + * \\brief The maximum variance of the current image.\n>>>>>>> + *\n>>>>>>> + * \\var IPAFrameContext::af.stable\n>>>>>>> + * \\brief is the image focused?\n>>>>>>> */\n>>>>>>>\n>>>>>>> /**\n>>>>>>> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n>>>>>>> index c6dc0814..aa5bf97f 100644\n>>>>>>> --- a/src/ipa/ipu3/ipa_context.h\n>>>>>>> +++ b/src/ipa/ipu3/ipa_context.h\n>>>>>>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n>>>>>>> double minAnalogueGain;\n>>>>>>> double maxAnalogueGain;\n>>>>>>> } agc;\n>>>>>>> +\n>>>>>>> + struct {\n>>>>>>> + uint16_t start_x;\n>>>>>>> + uint16_t start_y;\n>>>>>>> + } af;\n>>>>>>> };\n>>>>>>>\n>>>>>>> struct IPAFrameContext {\n>>>>>>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n>>>>>>> double temperatureK;\n>>>>>>> } awb;\n>>>>>>>\n>>>>>>> + struct {\n>>>>>>> + uint32_t focus;\n>>>>>>> + double maxVariance;\n>>>>>>> + bool stable;\n>>>>>>> + } af;\n>>>>>>> +\n>>>>>>> struct {\n>>>>>>> uint32_t exposure;\n>>>>>>> double gain;\n>>>>>>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n>>>>>>> index 3d307708..93966c6f 100644\n>>>>>>> --- a/src/ipa/ipu3/ipu3.cpp\n>>>>>>> +++ b/src/ipa/ipu3/ipu3.cpp\n>>>>>>> @@ -30,6 +30,7 @@\n>>>>>>>\n>>>>>>> #include \"libcamera/internal/mapped_framebuffer.h\"\n>>>>>>>\n>>>>>>> +#include \"algorithms/af.h\"\n>>>>>>> #include \"algorithms/agc.h\"\n>>>>>>> #include \"algorithms/algorithm.h\"\n>>>>>>> #include \"algorithms/awb.h\"\n>>>>>>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n>>>>>>> }\n>>>>>>>\n>>>>>>> /* Construct our Algorithms */\n>>>>>>> + algorithms_.push_back(std::make_unique());\n>>>>>>> algorithms_.push_back(std::make_unique());\n>>>>>>> algorithms_.push_back(std::make_unique());\n>>>>>>> algorithms_.push_back(std::make_unique());\n>>>>>>>\n>>>>>>\n>>>>>\n>>>>> Thank you :)\n>>>>>\n>>>>\n>>>\n>>>\n>>\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 24039BE080\n\tfor ;\n\tFri, 31 Dec 2021 13:13:35 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0349E60914;\n\tFri, 31 Dec 2021 14:13:34 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 14E4F604F8\n\tfor ;\n\tFri, 31 Dec 2021 14:13:33 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:2eab:95bb:5c1:3a27] (unknown\n\t[IPv6:2a01:e0a:169:7140:2eab:95bb:5c1:3a27])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 9CD045B0;\n\tFri, 31 Dec 2021 14:13:32 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"GcDGLXGT\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1640956412;\n\tbh=E1RvDsOnt+fHW7SNiBnny7WSqx8SxZeyLMdCob+GvdE=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=GcDGLXGTrhVb9d6KCJWv/m3EdzgTtTCKF58bhr56XG937ztUyM9tkeB367S7O2z0c\n\t/S5DcfCjDDL3HtwZh6+EZmY2ai90rjIBu/91KuLMUkHi91/oDCE5YI4y3iCOQ1WSMB\n\tWHPqNjbhr31RCCF0BMHBs43GLD4VVAS5uSOn6AEE=","Message-ID":"","Date":"Fri, 31 Dec 2021 14:13:29 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.3.1","Content-Language":"en-US","To":"Kate Hsuan ","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t","From":"Jean-Michel Hautbois ","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21948,"web_url":"https://patchwork.libcamera.org/comment/21948/","msgid":"","date":"2022-01-05T07:13:01","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel\n\n\nOn Fri, Dec 31, 2021 at 9:13 PM Jean-Michel Hautbois\n wrote:\n>\n> Hi Kate,\n>\n> On 31/12/2021 12:02, Kate Hsuan wrote:\n> > Hi Jean-Michel,\n> >\n> > On Mon, Dec 27, 2021 at 8:55 PM Jean-Michel Hautbois\n> > wrote:\n> >>\n> >> Hi Kate !\n> >>\n> >> On 27/12/2021 10:54, Kate Hsuan wrote:\n> >>> Hi Jean-Michel,\n> >>>\n> >>> On Sat, Dec 18, 2021 at 12:18 AM Jean-Michel Hautbois\n> >>> wrote:\n> >>>>\n> >>>> Hi Kate,\n> >>>>\n> >>>> On 17/12/2021 09:57, Kate Hsuan wrote:\n> >>>>> \"Hi Jean-Michel,\n> >>>>>\n> >>>>> Thank you for your review.\n> >>>>>\n> >>>>> On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n> >>>>> wrote:\n> >>>>>>\n> >>>>>> Hi Kate,\n> >>>>>>\n> >>>>>> Thanks for the patch :-) !\n> >>>>>>\n> >>>>>> I have a few comments below, and I have to tell you I couldn't make it\n> >>>>>> work here... I don't know how you are testing it ? I can see the lens\n> >>>>>> position changing but it passes over the correct focus and never locks\n> >>>>>> on it.\n> >>>>>>\n> >>>>>> On 13/12/2021 13:28, Kate Hsuan wrote:\n> >>>>>>> Since VCM for surface Go 2 (dw9719) had been successfully\n> >>>>>>> driven, this Af module can be used to control the VCM and\n> >>>>>>> determine the focus value based on the IPU3 AF state.\n> >>>>>>>\n> >>>>>>> The variance of each focus step is determined and a greedy\n> >>>>>>> approah is used to find the maximum variance of the AF\n> >>>>>>> state and a appropriate focus value.\n> >>>>>>>\n> >>>>>>> Changes in v2:\n> >>>>>>> 1. Add grid configuration interface.\n> >>>>>>> 2. Add AF accelerator configuration.\n> >>>>>>> 3. Move default focus area to the center of the image.\n> >>>>>>>\n> >>>>>>> Changes in v3:\n> >>>>>>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n> >>>>>>> algorithm.\n> >>>>>>> 2. Fix greedy AF algorithm since V4l2 interface had been\n> >>>>>>> removed.\n> >>>>>>> 3. Simplify default grid config for AF.\n> >>>>>>> 4. AF start area is determined by bdsOutputSize.\n> >>>>>>>\n> >>>>>>> Changes in v4:\n> >>>>>>> In v4, it significant improves the AF scan time using a two pass\n> >>>>>>> scaning method and the AF scan will be terminated when it finds\n> >>>>>>> a negative gradient.\n> >>>>>>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n> >>>>>>> the AF success rate.\n> >>>>>>> 2. The low pass filter convolution results are used to determine\n> >>>>>>> a coarse AF result.\n> >>>>>>> 3. The high pass convolution result is used to find a good lens\n> >>>>>>> step in a given AF range determined in pass 1.\n> >>>>>>>\n> >>>>>>\n> >>>>>> The lines above describe the changes between the versions, which is\n> >>>>>> great ! But those should go after the \"---\" as we don't want those in\n> >>>>>> the final commit message.\n> >>>>>\n> >>>>> Ok. Thank you I'll move the changes after \"---\".\n> >>>>>\n> >>>>>>\n> >>>>>>> Signed-off-by: Kate Hsuan \n> >>>>>>> ---\n> >>>>>>\n> >>>>>> Changes in ...\n> >>>>>>\n> >>>>>> ---\n> >>>>>>> src/ipa/ipu3/algorithms/af.cpp | 334 ++++++++++++++++++++++++++++\n> >>>>>>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n> >>>>>>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n> >>>>>>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n> >>>>>>> src/ipa/ipu3/ipa_context.h | 11 +\n> >>>>>>> src/ipa/ipu3/ipu3.cpp | 2 +\n> >>>>>>> 6 files changed, 442 insertions(+), 1 deletion(-)\n> >>>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> >>>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n> >>>>>>>\n> >>>>>>\n> >>>>>> I applied your patch, and I have compilation issues:\n> >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>>>> member ‘ipu3_uapi_af_config_s::padding’ [-Werror=missing-field-initializers]\n> >>>>>> 89 | };\n> >>>>>> | ^\n> >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>>>> member ‘ipu3_uapi_grid_config::height_per_slice’\n> >>>>>> [-Werror=missing-field-initializers]\n> >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>>>> member ‘ipu3_uapi_grid_config::x_end’ [-Werror=missing-field-initializers]\n> >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing initializer for\n> >>>>>> member ‘ipu3_uapi_grid_config::y_end’ [-Werror=missing-field-initializers]\n> >>>>>> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual void\n> >>>>>> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n> >>>>>> const ipu3_uapi_stats_3a*)’:\n> >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n> >>>>>> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned pointer\n> >>>>>> value [-Werror=address-of-packed-member]\n> >>>>>> 265 | y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>>>>>\n> >>>>>> I solved them locally (with a quick and dirty patch) how did you not\n> >>>>>> have those ?\n> >>>>>\n> >>>>> It was my fault. I can build them successfully before. But, after\n> >>>>> re-cloning the git repo and patching my v4 patch, it happened. I'll\n> >>>>> fix those issues.\n> >>>>> Sorry about the inconvenience. T_T\n> >>>>\n> >>>> No problem, but it was surprising :-).\n> >>>>\n> >>>>>\n> >>>>>>\n> >>>>>>> diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp\n> >>>>>>> new file mode 100644\n> >>>>>>> index 00000000..620be9fa\n> >>>>>>> --- /dev/null\n> >>>>>>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> >>>>>>> @@ -0,0 +1,334 @@\n> >>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>>>>>> +/*\n> >>>>>>> + * Copyright (C) 2021, Red Hat\n> >>>>>>> + *\n> >>>>>>> + * af.cpp - IPU3 auto focus control\n> >>>>>>> + */\n> >>>>>>> +\n> >>>>>>> +#include \"af.h\"\n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \"libipa/histogram.h\"\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\file af.h\n> >>>>>>> + */\n> >>>>>>> +\n> >>>>>>> +namespace libcamera {\n> >>>>>>> +\n> >>>>>>> +using namespace std::literals::chrono_literals;\n> >>>>>>> +\n> >>>>>>> +namespace ipa::ipu3::algorithms {\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\class Af\n> >>>>>>> + * \\brief A IPU3 auto-focus accelerator based auto focus algorthim\n> >>>>>>> + *\n> >>>>>>> + * This algorithm is used to determine the position of the lens and get a\n> >>>>>>> + * focused image. The IPU3 AF accelerator computes the statistics, composed\n> >>>>>>> + * by high pass and low pass filtered value and stores in a AF buffer.\n> >>>>>>> + * Typically, for a focused image, it has relative high contrast than a\n> >>>>>>> + * blurred image, i.e. an out of focus image. Therefore, if an image with the\n> >>>>>>> + * highest contrast can be found from the AF scan, the lens' position is the\n> >>>>>>> + * best step of the focus.\n> >>>>>>> + *\n> >>>>>>> + */\n> >>>>>>> +\n> >>>>>>> +LOG_DEFINE_CATEGORY(IPU3Af)\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * Maximum focus value of the VCM control\n> >>>>>>> + * \\todo should be obtained from the VCM driver\n> >>>>>>> + */\n> >>>>>>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n> >>>>>>> +\n> >>>>>>> +/* minimum focus step for searching appropriate focus*/\n> >>>>>>> +static constexpr uint32_t coarseSearchStep_ = 10;\n> >>>>>>> +static constexpr uint32_t fineSearchStep_ = 1;\n> >>>>>>> +\n> >>>>>>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n> >>>>>>> +static constexpr double MaxChange_ = 0.8;\n> >>>>>>\n> >>>>>> Why 0.8 ? Any explanation on how to set it ?\n> >>>>>\n> >>>>> OK. I'll put my explanation here. Basically, it is used to represent\n> >>>>> the boundary of a focused and blurred image.\n> >>>>>\n> >>>>>\n> >>>>>>\n> >>>>>>> +\n> >>>>>>> +/* settings for Auto Focus from the kernel */\n> >>>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> >>>>>>> + .filter_config = {\n> >>>>>>> + { 0, 0, 0, 0 },\n> >>>>>>> + { 0, 0, 0, 0 },\n> >>>>>>> + { 0, 0, 0, 128 },\n> >>>>>>> + 0,\n> >>>>>>> + { 0, 0, 0, 0 },\n> >>>>>>> + { 0, 0, 0, 0 },\n> >>>>>>> + { 0, 0, 0, 128 },\n> >>>>>>> + 0,\n> >>>>>>> + .y_calc = { 8, 8, 8, 8 },\n> >>>>>>> + .nf = { 0, 7, 0, 7, 0 },\n> >>>>>>> + },\n> >>>>\n> >>>> I would like your thinking here (and perhaps someone else). It looks\n> >>>> like the ImgU implements a Gabor filter, but I can't see all the\n> >>>> parameters I would expect from it. Putting it down, I can't see with\n> >>>> this default filter how y1 and y2 could represent low-pass and high\n> >>>> pass, as the filters are the same. But, it could be a horizontal and a\n> >>>> vertical filter instead, with y1 the horizontal and y2 the vertical\n> >>>> resulting filtered values.\n> >>>>\n> >>>> It would be very interesting to display the y1_avg and y2_avg values as\n> >>>> an image, I just don't have lots of time for it. We have 16 bits values,\n> >>>> so we could dump those into a binary file and use python to display the\n> >>>> values as a raw gray image or something similar.\n> >>>\n> >>> You could check out the URL below to see the grey scale image of the AF buffer.\n> >>> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> >>>\n> >>> Since the result is a tiny 16x16 image, all the detail of the original\n> >>> image was compressed into a black block.\n> >>>\n> >>\n> >> Nice :-).\n> >> As expected, as y1 and y2 are the same filter, images are the same.\n> >> Could you test those values ?\n> >>\n> >> y1 =>\n> >> (0, 1, 3, 7\n> >> 11, 13, 1, 2\n> >> 8, 19, 34, 242)\n> >> vector: 7fdffbfe - normalization factor: 9\n> >> y2 =>\n> >> (0, 1, 6, 6\n> >> 13, 25, 3, 0\n> >> 25, 3, 117, 254)\n> >> vector: 4e53ca72 - normalization factor: 9\n> >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n> >>\n> >\n> > Please check out the results from the URL below.\n> >\n> > https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n> >\n> > Y1 became a black image and was difficult to determine the coarse focus value.\n> >\n>\n\n> Sounds surprising... Could you share the way you are generating the\n> images of Y1 and Y2 BTW ?\n>\n> > Happy new year :)\n>\n> Best wishes, hoping for a far away Covid next year !\nI hope too :)\n\nAfter some checks, I made some mistakes to convert the image to png file.\n\nCheck out the following URL for the new results.\nhttps://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing\n\nBut, for the Y2, the result becomes very bright and sometimes the\nimage will be all white. Are the results right?\n\nMy python code for converting the pixels to tiff is shown below.\n\n=========\nimport numpy as np\nfrom scipy import ndimage, misc\nimport imageio\nfrom PIL import Image\nimport sys\n\nrawfile = np.fromfile(sys.argv[1], \"uint16\")\nrawfile.shape = (16,16)\nnew_image = Image.fromarray(rawfile, mode='I;16')\nnew_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n==========\n\n\n>\n> >\n> >>>>\n> >>>> The other things which I find annoying is the non-reliable values I get\n> >>>> right now for the variance. Same scene, exposure and gain fixed to a\n> >>>> value, controled luminosity, and between two executions I might not have\n> >>>> the same result...\n> >>>>\n> >>>>>>> + .grid_cfg = {\n> >>>>>>> + .width = 16,\n> >>>>>>> + .height = 16,\n> >>>>>>> + .block_width_log2 = 3,\n> >>>>>>> + .block_height_log2 = 3,\n> >>>>>>> + .x_start = 10,\n> >>>>>>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >>>>>>> + },\n> >>>>>>> +};\n> >>>>>>\n> >>>>>> As mentionned, you missed some fields:\n> >>>>>\n> >>>>> I'll fix this. Thank you.\n> >>>>>\n> >>>>>>\n> >>>>>> '''\n> >>>>>> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n> >>>>>> imgu_css_af_defaults = {\n> >>>>>> .y_calc = { 8, 8, 8, 8 },\n> >>>>>> .nf = { 0, 7, 0, 7, 0 },\n> >>>>>> },\n> >>>>>> + .padding = { 0, 0, 0, 0 },\n> >>>>>> .grid_cfg = {\n> >>>>>> .width = 16,\n> >>>>>> .height = 16,\n> >>>>>> .block_width_log2 = 3,\n> >>>>>> .block_height_log2 = 3,\n> >>>>>> + .height_per_slice = 1,\n> >>>>>> .x_start = 10,\n> >>>>>> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> >>>>>> + .x_end = 138,\n> >>>>>> + .y_end = 130,\n> >>>>>> },\n> >>>>>> };\n> >>>>>> '''\n> >>>>>>\n> >>>>>>> +\n> >>>>>>> +Af::Af()\n> >>>>>>> + : focus_(0), goodFocus_(0), currentVariance_(0.0), previousVariance_(0.0),\n> >>>>>>> + pass1Done_(false), pass2Done_(false)\n> >>>>>>> +{\n> >>>>>>> + maxStep_ = MaxFocusSteps_;\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +Af::~Af()\n> >>>>>>> +{\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n> >>>>>>\n> >>>>>> Missing documentation for prepare() => you should build the doc ;-)\n> >>>>>\n> >>>>> Ok. I'll add the prepare() document here.\n> >>>>>\n> >>>>>>\n> >>>>>>> +{\n> >>>>>>> + params->use.acc_af = 1;\n> >>>>>>> + params->acc_param.af = imgu_css_af_defaults;\n> >>>>>>> + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;\n> >>>>>>> + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n> >>>>>>\n> >>>>>> You are never changing the grid size, is it intended ?\n> >>>>>\n> >>>>> Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n> >>>>> more detail? I can modify this according to your opinions.\n> >>>>\n> >>>> Well, yes, you are always using the default 128x128 grid size. But you\n> >>>> can go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in\n> >>>> height. Thus, it means we could evaluate the scene on a small part or\n> >>>> almost all of it (copying the bds grid is not doable as it might have\n> >>>> larger block width, up to 80x60).\n> >>>>\n> >>>> See:\n> >>>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n> >>>>\n> >>>> Note:\n> >>>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n> >>>>\n> >>>> => It makes me wonder if the filtering is fast, or if it may take some\n> >>>> time, meaning that focus evaluation in one frame could be done only at\n> >>>> frame+2 or something like that.\n> >>>>\n> >>>> Could you verify that the focus estimated is in the center of the scene\n> >>>> right now, as it is what you have configured ?\n> >>>>\n> >>>>>\n> >>>>>>\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\brief Configure the Af given a configInfo\n> >>>>>>> + * \\param[in] context The shared IPA context\n> >>>>>>> + * \\param[in] configInfo The IPA configuration data\n> >>>>>>> + *\n> >>>>>>> + * \\return 0\n> >>>>>>> + */\n> >>>>>>> +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n> >>>>>>> +{\n> >>>>>>> + /* determined focus value i.e. current focus value */\n> >>>>>>> + context.frameContext.af.focus = 0;\n> >>>>>>> + /* maximum variance of the AF statistics */\n> >>>>>>> + context.frameContext.af.maxVariance = 0;\n> >>>>>> double type, maybe s/= 0/= 0.0/ ?\n> >>>>>\n> >>>>> 0.0 is correct. I'll correct this thank you.\n> >>>>>\n> >>>>>>\n> >>>>>>> + /* is focused? if it is true, the AF should be in a stable state. */\n> >>>>>>> + context.frameContext.af.stable = false;\n> >>>>>>> + /* frame to be ignored before start to estimate AF variance. */\n> >>>>>>> + ignoreFrame_ = 10;\n> >>>>>>\n> >>>>>> Could be a constexpr ?\n> >>>>>\n> >>>>> It is a counter to ignore frames since we have to ignore some frames\n> >>>>> to wait for the value from the accelerator to be stable.\n> >>>>> If this should be a constant variable, I can modify this. :)\n> >>>>>\n> >>>>>\n> >>>>>>\n> >>>>>>> +\n> >>>>>>> + /*\n> >>>>>>> + * AF default area configuration\n> >>>>>>> + * Move AF area to the center of the image.\n> >>>>>>> + */\n> >>>>>>> + /* Default AF width is 16x8 = 128 */\n> >>>>>>> + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64;\n> >>>>>>\n> >>>>>> Why are you using a fixed value for the grid ? You could use the value\n> >>>>>> stored in imgu_css_af_defaults directly instead of applying 128/2 manualy ?\n> >>>>>>\n> >>>>>>> + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64;\n> >>>>>>> +\n> >>>>>>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n> >>>>>>> + << configInfo.bdsOutputSize.width\n> >>>>>>> + << \" Y: \"\n> >>>>>>> + << configInfo.bdsOutputSize.height;\n> >>>>>>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n> >>>>>>> + << context.configuration.af.start_x\n> >>>>>>> + << \" Y: \"\n> >>>>>>> + << context.configuration.af.start_y;\n> >>>>>>> +\n> >>>>>>> + return 0;\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\brief AF coarse scan\n> >>>>>>> + * \\param[in] context The shared IPA context\n> >>>>>>> + *\n> >>>>>>> + */\n> >>>>>>> +void Af::af_coarse_scan(IPAContext &context)\n> >>>>>>\n> >>>>>> Bad naming (camelCase only). Same below.\n> >>>>>\n> >>>>> Ops, sure, I'll modify this.\n> >>>>>\n> >>>>>>\n> >>>>>>> +{\n> >>>>>>> + if (pass1Done_ == true)\n> >>>>>>> + return;\n> >>>>>>> +\n> >>>>>>> + if (af_scan(context, coarseSearchStep_)) {\n> >>>>>>> + pass1Done_ = true;\n> >>>>>>> + context.frameContext.af.maxVariance = 0;\n> >>>>>>> + focus_ = context.frameContext.af.focus - (context.frameContext.af.focus * 0.1);\n> >>>>>>> + context.frameContext.af.focus = focus_;\n> >>>>>>> + previousVariance_ = 0;\n> >>>>>>> + maxStep_ = focus_ + (focus_ * 0.2);\n> >>>>>>> + }\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\brief AF fine scan\n> >>>>>>> + * \\param[in] context The shared IPA context\n> >>>>>>> + *\n> >>>>>>> + */\n> >>>>>>> +void Af::af_fine_scan(IPAContext &context)\n> >>>>>>> +{\n> >>>>>>> + if (pass1Done_ != true)\n> >>>>>>> + return;\n> >>>>>>> +\n> >>>>>>> + if (af_scan(context, fineSearchStep_)) {\n> >>>>>>> + context.frameContext.af.stable = true;\n> >>>>>>> + pass2Done_ = true;\n> >>>>>>> + }\n> >>>>>>> +}\n> >>>>>> I am not a big fan of these pass1Done_ and pass2Done_ variables, let's\n> >>>>>> seee where it is used...\n> >>>>>\n> >>>>> Since we used two stage AF searching strategy I simply say this is a\n> >>>>> two pass algorithm :)\n> >>>>> I'll renaming this.\n> >>>>>\n> >>>>>>\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\brief AF reset\n> >>>>>>> + * \\param[in] context The shared IPA context\n> >>>>>>> + *\n> >>>>>>> + */\n> >>>>>>> +void Af::af_reset(IPAContext &context)\n> >>>>>>> +{\n> >>>>>>> + context.frameContext.af.maxVariance = 0;\n> >>>>>>> + context.frameContext.af.focus = 0;\n> >>>>>>> + focus_ = 0;\n> >>>>>>> + context.frameContext.af.stable = false;\n> >>>>>>> + ignoreFrame_ = 60;\n> >>>>>>\n> >>>>>> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n> >>>>>\n> >>>>> We can tweak this value.\n> >>>>>\n> >>>>>>\n> >>>>>>> + previousVariance_ = 0.0;\n> >>>>>>> + pass1Done_ = false;\n> >>>>>>> + pass2Done_ = false;\n> >>>>>>> + maxStep_ = MaxFocusSteps_;\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\brief AF scan\n> >>>>>>> + * \\param[in] context The shared IPA context\n> >>>>>>> + *\n> >>>>>>> + * \\return True, if it finds a AF value.\n> >>>>>>> + */\n> >>>>>>> +bool Af::af_scan(IPAContext &context, int min_step)\n> >>>>>>> +{\n> >>>>>>> + /* find the maximum variance during the AF scan using a greedy strategy */\n> >>>>>>> + if (currentVariance_ > context.frameContext.af.maxVariance) {\n> >>>>>>> + context.frameContext.af.maxVariance = currentVariance_;\n> >>>>>>> + goodFocus_ = focus_;\n> >>>>>>> + }\n> >>>>>>> +\n> >>>>>>> + if (focus_ > maxStep_) {\n> >>>>>>> + /* if reach the max step, move lens to the position and set \"focus stable\". */\n> >>>>>>> + context.frameContext.af.focus = goodFocus_;\n> >>>>>>> + return true;\n> >>>>>>> + } else {\n> >>>>>>> + /* check negative gradient */\n> >>>>>>> + if ((currentVariance_ - context.frameContext.af.maxVariance) > -(context.frameContext.af.maxVariance * 0.15)) {\n> >>>>>>\n> >>>>>> Where is the 0.15 coming from ?\n> >>>>>\n> >>>>> After some testing, I think that If the negative gradient ismore than\n> >>>>> 15% maxVaraiance, the focused image and lens position we get!\n> >>>>> I could put this as a constant variable in my v5 patch.\n> >>>>>\n> >>>>>>\n> >>>>>>> + focus_ += min_step;\n> >>>>>>> + context.frameContext.af.focus = focus_;\n> >>>>>>> + } else {\n> >>>>>>> + context.frameContext.af.focus = goodFocus_;\n> >>>>>>> + previousVariance_ = currentVariance_;\n> >>>>>>> + return true;\n> >>>>>>> + }\n> >>>>>>> + }\n> >>>>>>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n> >>>>>> s/prevrious/previous/\n> >>>>>\n> >>>>> Ops. sorry about the typo. T_T\n> >>>>>\n> >>>>>>\n> >>>>>>> + << previousVariance_\n> >>>>>>> + << \" current: \"\n> >>>>>>> + << currentVariance_\n> >>>>>>> + << \" Diff: \"\n> >>>>>>> + << (currentVariance_ - context.frameContext.af.maxVariance);\n> >>>>>>\n> >>>>>> Interestingly, it seems there is an issue here, according to the log I get ?\n> >>>>>>\n> >>>>>> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance prevrious: 0\n> >>>>>> current: 20012.4 Diff: 0\n> >>>>>>\n> >>>>>> Shouldn't diff be 20012.4 ?\n> >>>>>>\n> >>>>>> I am not sure the algorithm is very strong though... You may never\n> >>>>>> converge and stay in a local minimum without knowing ?\n> >>>>>>\n> >>>>>>> + previousVariance_ = currentVariance_;\n> >>>>>>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n> >>>>>>> + << context.frameContext.af.maxVariance\n> >>>>>>> + << \" Focus step is \"\n> >>>>>>> + << goodFocus_\n> >>>>>>> + << \" Current scan is \"\n> >>>>>>> + << focus_;\n> >>>>>>> + return false;\n> >>>>>>> +}\n> >>>>>>\n> >>>>>> I am not sure to understand how this function should behave... Is it a\n> >>>>>> minimum searching ? Are you looking for argmin(currentVariance -\n> >>>>>> maxVariance) ?\n> >>>>>\n> >>>>> Actually. it is a maximum search. I try to search for the maximum\n> >>>>> variance of the image.\n> >>>>> If the difference is positive, it means the variance still increases.\n> >>>>> We can continue search for the maximum value and update maxVariance_.\n> >>>>> Until, we found a negative difference which means the variance starts\n> >>>>> to go down and image will become blurred.\n> >>>>>\n> >>>>>\n> >>>>>>\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\brief Determine the max contrast image and lens position. y_table is the\n> >>>>>>> + * statictic data from IPU3 and is composed of low pass and high pass filtered\n> >>>>>>> + * value. High pass filtered value also represents the sharpness of the image.\n> >>>>>>> + * Based on this, if the image with highest variance of the high pass filtered\n> >>>>>>> + * value (contrast) during the AF scan, the position of the len should be the\n> >>>>>>> + * best focus.\n> >>>>>>> + * \\param[in] context The shared IPA context.\n> >>>>>>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n> >>>>>>> + */\n> >>>>>>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n> >>>>>>> +{\n> >>>>>>> + uint32_t total = 0;\n> >>>>>>> + double mean;\n> >>>>>>> + uint64_t var_sum = 0;\n> >>>>>>> + y_table_item_t *y_item;\n> >>>>>>> + int z = 0;\n> >>>>>>> +\n> >>>>>>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> >>>>>>> +\n> >>>>>>> + /**\n> >>>>>>> + * Calculate the mean and the varience of each non-zero AF statistics, since IPU3 only determine the AF value\n> >>>>>>> + * for a given grid.\n> >>>>>>> + * For pass1: low pass results are used.\n> >>>>>>> + * For pass2: high pass results are used.\n> >>>>>>> + */\n> >>>>>>> + if (pass1Done_) {\n> >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>>>>>> + total = total + y_item[z].y2_avg;\n> >>>>>>> + if (y_item[z].y2_avg == 0)\n> >>>>>>> + break;\n> >>>>>>> + }\n> >>>>>>> + mean = total / z;\n> >>>>>>> +\n> >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {\n> >>>>>>> + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));\n> >>>>>>> + if (y_item[z].y2_avg == 0)\n> >>>>>>> + break;\n> >>>>>>> + }\n> >>>>>>> + } else {\n> >>>>>>\n> >>>>>> /* Calculate the mean of the low pass filter values. */\n> >>>>>>\n> >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {\n> >>>>>>> + total = total + y_item[z].y1_avg;\n> >>>>>>> + if (y_item[z].y1_avg == 0)\n> >>>>>>> + break;\n> >>>>>> Are you sure of that ? Shouldn't you use the grid you set to parse the\n> >>>>>> right number of data ? If you have a local dark area you may stop even\n> >>>>>> if it is not yet the end of the grid ?\n> >>>>>\n> >>>>> I'll find another way to determine the range of the results. May be it\n> >>>>> can based on the grids setting.\n> >>>>>\n> >>>>>>\n> >>>>>>> + }\n> >>>>>>> + mean = total / z;\n> >>>>>>> +\n> >>>>>>\n> >>>>>> /* Calulate the deviation from the mean. */\n> >>>>>>\n> >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y1_avg != 0; z++) {\n> >>>>>>> + var_sum = var_sum + ((y_item[z].y1_avg - mean) * (y_item[z].y1_avg - mean));\n> >>>>>>> + if (y_item[z].y1_avg == 0)\n> >>>>>>> + break;\n> >>>>>>> + }\n> >>>>>>> + }\n> >>>>>>\n> >>>>>> The two if() blocks are very similar, the only exception beeing the item\n> >>>>>> of the table used (y_item[z].y1_avg in the first pass, y_item[z].y2_avg\n> >>>>>> in the second one). This is not easy to read (according to me).\n> >>>>>\n> >>>>> They are high pass (y2) and low pass (y1) filtered convolution\n> >>>>> results. This structure referees from ipu-ipa repo and I would like to\n> >>>>> keep their original naming.\n> >>>>>\n> >>>>>\n> >>>>>>\n> >>>>>>> + /* Determine the average variance of the frame. */\n> >>>>>>> + currentVariance_ = static_cast(var_sum) / static_cast(z);\n> >>>>>>\n> >>>>>> Side note:\n> >>>>>> If my stats are still correct, I think it should be divided by (z-1) and\n> >>>>>> not z, as we have a sample and not a full population ?\n> >>>>>\n> >>>>> it is z-1 and my fault. I'll correct this. :)\n> >>>>>\n> >>>>>>\n> >>>>>>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n> >>>>>>> +\n> >>>>>>> + if (context.frameContext.af.stable == true) {\n> >>>>>>> + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n> >>>>>>> + const double var_ratio = diff_var / context.frameContext.af.maxVariance;\n> >>>>>>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n> >>>>>>> + << var_ratio\n> >>>>>>> + << \" current focus: \"\n> >>>>>>> + << context.frameContext.af.focus;\n> >>>>>>> + /**\n> >>>>>>> + * If the change ratio of contrast is over Maxchange_ (out of focus),\n> >>>>>>> + * trigger AF again.\n> >>>>>>> + */\n> >>>>>>> + if (var_ratio > MaxChange_) {\n> >>>>>>> + if (ignoreFrame_ == 0) {\n> >>>>>>> + af_reset(context);\n> >>>>>>> + } else\n> >>>>>>> + ignoreFrame_--;\n> >>>>>>> + } else\n> >>>>>>> + ignoreFrame_ = 10;\n> >>>>>>> + } else {\n> >>>>>>> + if (ignoreFrame_ != 0)\n> >>>>>>> + ignoreFrame_--;\n> >>>>>>> + else {\n> >>>>>>> + af_coarse_scan(context);\n> >>>>>>> + af_fine_scan(context);\n> >>>>>>> + }\n> >>>>>>> + }\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +} /* namespace ipa::ipu3::algorithms */\n> >>>>>>> +\n> >>>>>>> +} /* namespace libcamera */\n> >>>>>>> diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h\n> >>>>>>> new file mode 100644\n> >>>>>>> index 00000000..b9295b19\n> >>>>>>> --- /dev/null\n> >>>>>>> +++ b/src/ipa/ipu3/algorithms/af.h\n> >>>>>>> @@ -0,0 +1,66 @@\n> >>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >>>>>>> +/*\n> >>>>>>> + * Copyright (C) 2021, Red Hat\n> >>>>>>> + *\n> >>>>>>> + * af.h - IPU3 Af control\n> >>>>>>> + */\n> >>>>>>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>>>>>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \"algorithm.h\"\n> >>>>>>> +\n> >>>>>>> +namespace libcamera {\n> >>>>>>> +\n> >>>>>>> +namespace ipa::ipu3::algorithms {\n> >>>>>>> +\n> >>>>>>> +class Af : public Algorithm\n> >>>>>>> +{\n> >>>>>>> + /* The format of y_table. From ipu3-ipa repo */\n> >>>>>>> + typedef struct y_table_item {\n> >>>>>>> + uint16_t y1_avg;\n> >>>>>>> + uint16_t y2_avg;\n> >>>>>>> + } y_table_item_t;\n> >>>>>>> +\n> >>>>>>> +public:\n> >>>>>>> + Af();\n> >>>>>>> + ~Af();\n> >>>>>>> +\n> >>>>>>> + void prepare(IPAContext &context, ipu3_uapi_params *params) override;\n> >>>>>>> + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;\n> >>>>>>> + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;\n> >>>>>>> +\n> >>>>>>> +private:\n> >>>>>>> + void af_coarse_scan(IPAContext &context);\n> >>>>>>> + void af_fine_scan(IPAContext &context);\n> >>>>>>> + bool af_scan(IPAContext &context, int min_step);\n> >>>>>>> + void af_reset(IPAContext &context);\n> >>>>>>> +\n> >>>>>>> + /* Used for focus scan. */\n> >>>>>>> + uint32_t focus_;\n> >>>>>>> + /* Focus good */\n> >>>>>>> + uint32_t goodFocus_;\n> >>>>>>> + /* Recent AF statistic variance. */\n> >>>>>>> + double currentVariance_;\n> >>>>>>> + /* The frames to be ignore before starting measuring. */\n> >>>>>>> + uint32_t ignoreFrame_;\n> >>>>>>> + /* previous variance. it is used to determine the gradient */\n> >>>>>>> + double previousVariance_;\n> >>>>>>> + /* Max scan steps of each pass of AF scaning */\n> >>>>>>> + uint32_t maxStep_;\n> >>>>>>> + /* Pass 1 stable. Complete low pass search (coarse) scan) */\n> >>>>>>> + bool pass1Done_;\n> >>>>>>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n> >>>>>>> + bool pass2Done_;\n> >>>>>>> +};\n> >>>>>>> +\n> >>>>>>> +} /* namespace ipa::ipu3::algorithms */\n> >>>>>>> +\n> >>>>>>> +} /* namespace libcamera */\n> >>>>>>> +\n> >>>>>>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n> >>>>>>> diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build\n> >>>>>>> index 4db6ae1d..e1099169 100644\n> >>>>>>> --- a/src/ipa/ipu3/algorithms/meson.build\n> >>>>>>> +++ b/src/ipa/ipu3/algorithms/meson.build\n> >>>>>>> @@ -1,8 +1,9 @@\n> >>>>>>> # SPDX-License-Identifier: CC0-1.0\n> >>>>>>>\n> >>>>>>> ipu3_ipa_algorithms = files([\n> >>>>>>> + 'af.cpp',\n> >>>>>>> 'agc.cpp',\n> >>>>>>> 'awb.cpp',\n> >>>>>>> 'blc.cpp',\n> >>>>>>> - 'tone_mapping.cpp',\n> >>>>>>> + 'tone_mapping.cpp'\n> >>>>>>> ])\n> >>>>>>> diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp\n> >>>>>>> index 86794ac1..ee644d3c 100644\n> >>>>>>> --- a/src/ipa/ipu3/ipa_context.cpp\n> >>>>>>> +++ b/src/ipa/ipu3/ipa_context.cpp\n> >>>>>>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n> >>>>>>> *\n> >>>>>>> * \\var IPASessionConfiguration::grid.stride\n> >>>>>>> * \\brief Number of cells on one line including the ImgU padding\n> >>>>>>> + *\n> >>>>>>> + */\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\var IPASessionConfiguration::af\n> >>>>>>> + * \\brief AF parameters configuration of the IPA\n> >>>>>>> + *\n> >>>>>>> + * \\var IPASessionConfiguration::af.start_x\n> >>>>>>> + * \\brief The start X position of the AF area\n> >>>>>>> + *\n> >>>>>>> + * \\var IPASessionConfiguration::af.start_y\n> >>>>>>> + * \\brief The start Y position of the AF area\n> >>>>>>> + */\n> >>>>>>> +\n> >>>>>>> +/**\n> >>>>>>> + * \\var IPAFrameContext::af\n> >>>>>>> + * \\brief Context for the Automatic Focus algorithm\n> >>>>>>> + *\n> >>>>>>> + * \\struct IPAFrameContext::af\n> >>>>>>> + * \\var IPAFrameContext::af.focus\n> >>>>>>> + * \\brief Current position of the lens\n> >>>>>>> + *\n> >>>>>>> + * \\var IPAFrameContext::af.maxVariance\n> >>>>>>> + * \\brief The maximum variance of the current image.\n> >>>>>>> + *\n> >>>>>>> + * \\var IPAFrameContext::af.stable\n> >>>>>>> + * \\brief is the image focused?\n> >>>>>>> */\n> >>>>>>>\n> >>>>>>> /**\n> >>>>>>> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> >>>>>>> index c6dc0814..aa5bf97f 100644\n> >>>>>>> --- a/src/ipa/ipu3/ipa_context.h\n> >>>>>>> +++ b/src/ipa/ipu3/ipa_context.h\n> >>>>>>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n> >>>>>>> double minAnalogueGain;\n> >>>>>>> double maxAnalogueGain;\n> >>>>>>> } agc;\n> >>>>>>> +\n> >>>>>>> + struct {\n> >>>>>>> + uint16_t start_x;\n> >>>>>>> + uint16_t start_y;\n> >>>>>>> + } af;\n> >>>>>>> };\n> >>>>>>>\n> >>>>>>> struct IPAFrameContext {\n> >>>>>>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n> >>>>>>> double temperatureK;\n> >>>>>>> } awb;\n> >>>>>>>\n> >>>>>>> + struct {\n> >>>>>>> + uint32_t focus;\n> >>>>>>> + double maxVariance;\n> >>>>>>> + bool stable;\n> >>>>>>> + } af;\n> >>>>>>> +\n> >>>>>>> struct {\n> >>>>>>> uint32_t exposure;\n> >>>>>>> double gain;\n> >>>>>>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> >>>>>>> index 3d307708..93966c6f 100644\n> >>>>>>> --- a/src/ipa/ipu3/ipu3.cpp\n> >>>>>>> +++ b/src/ipa/ipu3/ipu3.cpp\n> >>>>>>> @@ -30,6 +30,7 @@\n> >>>>>>>\n> >>>>>>> #include \"libcamera/internal/mapped_framebuffer.h\"\n> >>>>>>>\n> >>>>>>> +#include \"algorithms/af.h\"\n> >>>>>>> #include \"algorithms/agc.h\"\n> >>>>>>> #include \"algorithms/algorithm.h\"\n> >>>>>>> #include \"algorithms/awb.h\"\n> >>>>>>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings,\n> >>>>>>> }\n> >>>>>>>\n> >>>>>>> /* Construct our Algorithms */\n> >>>>>>> + algorithms_.push_back(std::make_unique());\n> >>>>>>> algorithms_.push_back(std::make_unique());\n> >>>>>>> algorithms_.push_back(std::make_unique());\n> >>>>>>> algorithms_.push_back(std::make_unique());\n> >>>>>>>\n> >>>>>>\n> >>>>>\n> >>>>> Thank you :)\n> >>>>>\n> >>>>\n> >>>\n> >>>\n> >>\n> >\n> >\n>\n\n\n--\nBR,\nKate","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2D80FBF415\n\tfor ;\n\tWed, 5 Jan 2022 07:13:29 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 26B6D60926;\n\tWed, 5 Jan 2022 08:13:28 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EC9D460217\n\tfor ;\n\tWed, 5 Jan 2022 08:13:22 +0100 (CET)","from mail-lf1-f69.google.com (mail-lf1-f69.google.com\n\t[209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-467-iWN8EjyFNfGenwaRu-Fjzg-1; Wed, 05 Jan 2022 02:13:15 -0500","by mail-lf1-f69.google.com with SMTP id\n\tcf14-20020a056512280e00b00426460c9af9so8649363lfb.4\n\tfor ;\n\tTue, 04 Jan 2022 23:13:15 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"DDv/MAzG\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1641366801;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=XGgBWuMKZkgPiC/DToRu7nr5pq1HESghZWYllU2wnmU=;\n\tb=DDv/MAzGzudsXa/esL1bf7MIOu67ZLjHJeucfbcA+FLrEP+gHeRGPp4O7oitvNZflBxnlH\n\tFeZcN8YPlljzHBVcc39bv1NPf6RUJuOoayS1+kWvJd31VPNG3QXaUF2UinXHI1JdZwMl/v\n\tjUYC3H1fayI0M/0FY1z/l9xCnaFc3KY=","X-MC-Unique":"iWN8EjyFNfGenwaRu-Fjzg-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=XGgBWuMKZkgPiC/DToRu7nr5pq1HESghZWYllU2wnmU=;\n\tb=5eDXLv7FaHJr+n7zhooGmyYCo2XTqZUBV2cj/9+W1dfAaQKIxGuWr8PX0x/1WrDQ/O\n\tJugBhJi12tSidwCAmAL1H+sXI9Eb35BiBLk/DHM8339ueNaCVhVNp7eC/QWYXXGfjHEk\n\tOh2Z77cn8XtBxc7olxi9TDeB8eYkgl4q57qsWFt0EuvgYWmOg7Ed+uhjCZ77qOdZDIdO\n\tjQMeYuX4sExxaT6QQq0diQvdJE5s0Ap79rBA/0lCVowdAggXx7Oomm26redb9s3NKr4P\n\tq4jGifzpwKTcJOIuMQEdiAqF3dAk9xcWGlyQ8NIs9z4C7qEKXRkY1VWQemaX47oOw9Di\n\t+r2w==","X-Gm-Message-State":"AOAM531VUjJg4fihoo/ESvzpws8qF8oqvntu4TQ3IrJLxaxh1qTvJmfm\n\tj8bm/5nm2xCLE51E6Hhed9gsKLRppCGTqx71KSA8oD47x06XmHBRhTrF18QpxYuWknyOW2msch4\n\toSf2uLoCeAhESkv6sK9/FA/eXct9vbsi4YWbgZS8+8PiN4+nOjQ==","X-Received":["by 2002:a05:651c:882:: with SMTP id\n\td2mr8937636ljq.311.1641366793800; \n\tTue, 04 Jan 2022 23:13:13 -0800 (PST)","by 2002:a05:651c:882:: with SMTP id\n\td2mr8937614ljq.311.1641366793163; \n\tTue, 04 Jan 2022 23:13:13 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJxQhGGNIdVBFcd118c55IJ57O7Hm3RBs2qR3wty8uDL6eb9lxJNaiUw2yxDbOk7dmqHl4QBWgfyHxy+ugy3GNs=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t","In-Reply-To":"","From":"Kate Hsuan ","Date":"Wed, 5 Jan 2022 15:13:01 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22025,"web_url":"https://patchwork.libcamera.org/comment/22025/","msgid":"","date":"2022-01-13T11:35:52","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel,\n\n\n\nOn Wed, Jan 5, 2022 at 3:13 PM Kate Hsuan wrote:\n\n> Hi Jean-Michel\n>\n>\n> On Fri, Dec 31, 2021 at 9:13 PM Jean-Michel Hautbois\n> wrote:\n> >\n> > Hi Kate,\n> >\n> > On 31/12/2021 12:02, Kate Hsuan wrote:\n> > > Hi Jean-Michel,\n> > >\n> > > On Mon, Dec 27, 2021 at 8:55 PM Jean-Michel Hautbois\n> > > wrote:\n> > >>\n> > >> Hi Kate !\n> > >>\n> > >> On 27/12/2021 10:54, Kate Hsuan wrote:\n> > >>> Hi Jean-Michel,\n> > >>>\n> > >>> On Sat, Dec 18, 2021 at 12:18 AM Jean-Michel Hautbois\n> > >>> wrote:\n> > >>>>\n> > >>>> Hi Kate,\n> > >>>>\n> > >>>> On 17/12/2021 09:57, Kate Hsuan wrote:\n> > >>>>> \"Hi Jean-Michel,\n> > >>>>>\n> > >>>>> Thank you for your review.\n> > >>>>>\n> > >>>>> On Thu, Dec 16, 2021 at 11:32 PM Jean-Michel Hautbois\n> > >>>>> wrote:\n> > >>>>>>\n> > >>>>>> Hi Kate,\n> > >>>>>>\n> > >>>>>> Thanks for the patch :-) !\n> > >>>>>>\n> > >>>>>> I have a few comments below, and I have to tell you I couldn't\n> make it\n> > >>>>>> work here... I don't know how you are testing it ? I can see the\n> lens\n> > >>>>>> position changing but it passes over the correct focus and never\n> locks\n> > >>>>>> on it.\n> > >>>>>>\n> > >>>>>> On 13/12/2021 13:28, Kate Hsuan wrote:\n> > >>>>>>> Since VCM for surface Go 2 (dw9719) had been successfully\n> > >>>>>>> driven, this Af module can be used to control the VCM and\n> > >>>>>>> determine the focus value based on the IPU3 AF state.\n> > >>>>>>>\n> > >>>>>>> The variance of each focus step is determined and a greedy\n> > >>>>>>> approah is used to find the maximum variance of the AF\n> > >>>>>>> state and a appropriate focus value.\n> > >>>>>>>\n> > >>>>>>> Changes in v2:\n> > >>>>>>> 1. Add grid configuration interface.\n> > >>>>>>> 2. Add AF accelerator configuration.\n> > >>>>>>> 3. Move default focus area to the center of the image.\n> > >>>>>>>\n> > >>>>>>> Changes in v3:\n> > >>>>>>> 1. Squash Daniel's commit- Remove v4l2 interaction from AF\n> > >>>>>>> algorithm.\n> > >>>>>>> 2. Fix greedy AF algorithm since V4l2 interface had been\n> > >>>>>>> removed.\n> > >>>>>>> 3. Simplify default grid config for AF.\n> > >>>>>>> 4. AF start area is determined by bdsOutputSize.\n> > >>>>>>>\n> > >>>>>>> Changes in v4:\n> > >>>>>>> In v4, it significant improves the AF scan time using a two pass\n> > >>>>>>> scaning method and the AF scan will be terminated when it finds\n> > >>>>>>> a negative gradient.\n> > >>>>>>> 1. Introduce 2 pass AF scan (coarse and fine scan) to increase\n> > >>>>>>> the AF success rate.\n> > >>>>>>> 2. The low pass filter convolution results are used to determine\n> > >>>>>>> a coarse AF result.\n> > >>>>>>> 3. The high pass convolution result is used to find a good lens\n> > >>>>>>> step in a given AF range determined in pass 1.\n> > >>>>>>>\n> > >>>>>>\n> > >>>>>> The lines above describe the changes between the versions, which\n> is\n> > >>>>>> great ! But those should go after the \"---\" as we don't want\n> those in\n> > >>>>>> the final commit message.\n> > >>>>>\n> > >>>>> Ok. Thank you I'll move the changes after \"---\".\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> Signed-off-by: Kate Hsuan \n> > >>>>>>> ---\n> > >>>>>>\n> > >>>>>> Changes in ...\n> > >>>>>>\n> > >>>>>> ---\n> > >>>>>>> src/ipa/ipu3/algorithms/af.cpp | 334\n> ++++++++++++++++++++++++++++\n> > >>>>>>> src/ipa/ipu3/algorithms/af.h | 66 ++++++\n> > >>>>>>> src/ipa/ipu3/algorithms/meson.build | 3 +-\n> > >>>>>>> src/ipa/ipu3/ipa_context.cpp | 27 +++\n> > >>>>>>> src/ipa/ipu3/ipa_context.h | 11 +\n> > >>>>>>> src/ipa/ipu3/ipu3.cpp | 2 +\n> > >>>>>>> 6 files changed, 442 insertions(+), 1 deletion(-)\n> > >>>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.cpp\n> > >>>>>>> create mode 100644 src/ipa/ipu3/algorithms/af.h\n> > >>>>>>>\n> > >>>>>>\n> > >>>>>> I applied your patch, and I have compilation issues:\n> > >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing\n> initializer for\n> > >>>>>> member ‘ipu3_uapi_af_config_s::padding’\n> [-Werror=missing-field-initializers]\n> > >>>>>> 89 | };\n> > >>>>>> | ^\n> > >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing\n> initializer for\n> > >>>>>> member ‘ipu3_uapi_grid_config::height_per_slice’\n> > >>>>>> [-Werror=missing-field-initializers]\n> > >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing\n> initializer for\n> > >>>>>> member ‘ipu3_uapi_grid_config::x_end’\n> [-Werror=missing-field-initializers]\n> > >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:89:1: error: missing\n> initializer for\n> > >>>>>> member ‘ipu3_uapi_grid_config::y_end’\n> [-Werror=missing-field-initializers]\n> > >>>>>> ../src/ipa/ipu3/algorithms/af.cpp: In member function ‘virtual\n> void\n> > >>>>>>\n> libcamera::ipa::ipu3::algorithms::Af::process(libcamera::ipa::ipu3::IPAContext&,\n> > >>>>>> const ipu3_uapi_stats_3a*)’:\n> > >>>>>> ../src/ipa/ipu3/algorithms/af.cpp:265:50: error: taking address of\n> > >>>>>> packed member of ‘ipu3_uapi_stats_3a’ may result in an unaligned\n> pointer\n> > >>>>>> value [-Werror=address-of-packed-member]\n> > >>>>>> 265 | y_item = (y_table_item_t\n> *)stats->af_raw_buffer.y_table;\n> > >>>>>>\n> > >>>>>> I solved them locally (with a quick and dirty patch) how did you\n> not\n> > >>>>>> have those ?\n> > >>>>>\n> > >>>>> It was my fault. I can build them successfully before. But, after\n> > >>>>> re-cloning the git repo and patching my v4 patch, it happened. I'll\n> > >>>>> fix those issues.\n> > >>>>> Sorry about the inconvenience. T_T\n> > >>>>\n> > >>>> No problem, but it was surprising :-).\n> > >>>>\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> diff --git a/src/ipa/ipu3/algorithms/af.cpp\n> b/src/ipa/ipu3/algorithms/af.cpp\n> > >>>>>>> new file mode 100644\n> > >>>>>>> index 00000000..620be9fa\n> > >>>>>>> --- /dev/null\n> > >>>>>>> +++ b/src/ipa/ipu3/algorithms/af.cpp\n> > >>>>>>> @@ -0,0 +1,334 @@\n> > >>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > >>>>>>> +/*\n> > >>>>>>> + * Copyright (C) 2021, Red Hat\n> > >>>>>>> + *\n> > >>>>>>> + * af.cpp - IPU3 auto focus control\n> > >>>>>>> + */\n> > >>>>>>> +\n> > >>>>>>> +#include \"af.h\"\n> > >>>>>>> +\n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +#include \n> > >>>>>>> +\n> > >>>>>>> +#include \n> > >>>>>>> +\n> > >>>>>>> +#include \n> > >>>>>>> +\n> > >>>>>>> +#include \n> > >>>>>>> +\n> > >>>>>>> +#include \"libipa/histogram.h\"\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\file af.h\n> > >>>>>>> + */\n> > >>>>>>> +\n> > >>>>>>> +namespace libcamera {\n> > >>>>>>> +\n> > >>>>>>> +using namespace std::literals::chrono_literals;\n> > >>>>>>> +\n> > >>>>>>> +namespace ipa::ipu3::algorithms {\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\class Af\n> > >>>>>>> + * \\brief A IPU3 auto-focus accelerator based auto focus\n> algorthim\n> > >>>>>>> + *\n> > >>>>>>> + * This algorithm is used to determine the position of the lens\n> and get a\n> > >>>>>>> + * focused image. The IPU3 AF accelerator computes the\n> statistics, composed\n> > >>>>>>> + * by high pass and low pass filtered value and stores in a AF\n> buffer.\n> > >>>>>>> + * Typically, for a focused image, it has relative high\n> contrast than a\n> > >>>>>>> + * blurred image, i.e. an out of focus image. Therefore, if an\n> image with the\n> > >>>>>>> + * highest contrast can be found from the AF scan, the lens'\n> position is the\n> > >>>>>>> + * best step of the focus.\n> > >>>>>>> + *\n> > >>>>>>> + */\n> > >>>>>>> +\n> > >>>>>>> +LOG_DEFINE_CATEGORY(IPU3Af)\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * Maximum focus value of the VCM control\n> > >>>>>>> + * \\todo should be obtained from the VCM driver\n> > >>>>>>> + */\n> > >>>>>>> +static constexpr uint32_t MaxFocusSteps_ = 1023;\n> > >>>>>>> +\n> > >>>>>>> +/* minimum focus step for searching appropriate focus*/\n> > >>>>>>> +static constexpr uint32_t coarseSearchStep_ = 10;\n> > >>>>>>> +static constexpr uint32_t fineSearchStep_ = 1;\n> > >>>>>>> +\n> > >>>>>>> +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/\n> > >>>>>>> +static constexpr double MaxChange_ = 0.8;\n> > >>>>>>\n> > >>>>>> Why 0.8 ? Any explanation on how to set it ?\n> > >>>>>\n> > >>>>> OK. I'll put my explanation here. Basically, it is used to\n> represent\n> > >>>>> the boundary of a focused and blurred image.\n> > >>>>>\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +\n> > >>>>>>> +/* settings for Auto Focus from the kernel */\n> > >>>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> > >>>>>>> + .filter_config = {\n> > >>>>>>> + { 0, 0, 0, 0 },\n> > >>>>>>> + { 0, 0, 0, 0 },\n> > >>>>>>> + { 0, 0, 0, 128 },\n> > >>>>>>> + 0,\n> > >>>>>>> + { 0, 0, 0, 0 },\n> > >>>>>>> + { 0, 0, 0, 0 },\n> > >>>>>>> + { 0, 0, 0, 128 },\n> > >>>>>>> + 0,\n> > >>>>>>> + .y_calc = { 8, 8, 8, 8 },\n> > >>>>>>> + .nf = { 0, 7, 0, 7, 0 },\n> > >>>>>>> + },\n> > >>>>\n> > >>>> I would like your thinking here (and perhaps someone else). It looks\n> > >>>> like the ImgU implements a Gabor filter, but I can't see all the\n> > >>>> parameters I would expect from it. Putting it down, I can't see with\n> > >>>> this default filter how y1 and y2 could represent low-pass and high\n> > >>>> pass, as the filters are the same. But, it could be a horizontal\n> and a\n> > >>>> vertical filter instead, with y1 the horizontal and y2 the vertical\n> > >>>> resulting filtered values.\n> > >>>>\n> > >>>> It would be very interesting to display the y1_avg and y2_avg\n> values as\n> > >>>> an image, I just don't have lots of time for it. We have 16 bits\n> values,\n> > >>>> so we could dump those into a binary file and use python to display\n> the\n> > >>>> values as a raw gray image or something similar.\n> > >>>\n> > >>> You could check out the URL below to see the grey scale image of the\n> AF buffer.\n> > >>>\n> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> > >>>\n> > >>> Since the result is a tiny 16x16 image, all the detail of the\n> original\n> > >>> image was compressed into a black block.\n> > >>>\n> > >>\n> > >> Nice :-).\n> > >> As expected, as y1 and y2 are the same filter, images are the same.\n> > >> Could you test those values ?\n> > >>\n> > >> y1 =>\n> > >> (0, 1, 3, 7\n> > >> 11, 13, 1, 2\n> > >> 8, 19, 34, 242)\n> > >> vector: 7fdffbfe - normalization factor: 9\n> > >> y2 =>\n> > >> (0, 1, 6, 6\n> > >> 13, 25, 3, 0\n> > >> 25, 3, 117, 254)\n> > >> vector: 4e53ca72 - normalization factor: 9\n> > >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n> > >>\n> > >\n> > > Please check out the results from the URL below.\n> > >\n> > >\n> https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n> > >\n> > > Y1 became a black image and was difficult to determine the coarse\n> focus value.\n> > >\n> >\n>\n> > Sounds surprising... Could you share the way you are generating the\n> > images of Y1 and Y2 BTW ?\n> >\n> > > Happy new year :)\n> >\n> > Best wishes, hoping for a far away Covid next year !\n> I hope too :)\n>\n> After some checks, I made some mistakes to convert the image to png file.\n>\n> Check out the following URL for the new results.\n>\n> https://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing\n>\n> But, for the Y2, the result becomes very bright and sometimes the\n> image will be all white. Are the results right?\n>\n> My python code for converting the pixels to tiff is shown below.\n>\n> =========\n> import numpy as np\n> from scipy import ndimage, misc\n> import imageio\n> from PIL import Image\n> import sys\n>\n> rawfile = np.fromfile(sys.argv[1], \"uint16\")\n> rawfile.shape = (16,16)\n> new_image = Image.fromarray(rawfile, mode='I;16')\n> new_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n> ==========\n>\n>\n> >\n> > >\n> > >>>>\n> > >>>> The other things which I find annoying is the non-reliable values I\n> get\n> > >>>> right now for the variance. Same scene, exposure and gain fixed to a\n> > >>>> value, controled luminosity, and between two executions I might not\n> have\n> > >>>> the same result...\n> > >>>>\n> > >>>>>>> + .grid_cfg = {\n> > >>>>>>> + .width = 16,\n> > >>>>>>> + .height = 16,\n> > >>>>>>> + .block_width_log2 = 3,\n> > >>>>>>> + .block_height_log2 = 3,\n> > >>>>>>> + .x_start = 10,\n> > >>>>>>> + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> > >>>>>>> + },\n> > >>>>>>> +};\n> > >>>>>>\n> > >>>>>> As mentionned, you missed some fields:\n> > >>>>>\n> > >>>>> I'll fix this. Thank you.\n> > >>>>>\n> > >>>>>>\n> > >>>>>> '''\n> > >>>>>> @@ -78,13 +78,17 @@ static struct ipu3_uapi_af_config_s\n> > >>>>>> imgu_css_af_defaults = {\n> > >>>>>> .y_calc = { 8, 8, 8, 8 },\n> > >>>>>> .nf = { 0, 7, 0, 7, 0 },\n> > >>>>>> },\n> > >>>>>> + .padding = { 0, 0, 0, 0 },\n> > >>>>>> .grid_cfg = {\n> > >>>>>> .width = 16,\n> > >>>>>> .height = 16,\n> > >>>>>> .block_width_log2 = 3,\n> > >>>>>> .block_height_log2 = 3,\n> > >>>>>> + .height_per_slice = 1,\n> > >>>>>> .x_start = 10,\n> > >>>>>> .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN,\n> > >>>>>> + .x_end = 138,\n> > >>>>>> + .y_end = 130,\n> > >>>>>> },\n> > >>>>>> };\n> > >>>>>> '''\n> > >>>>>>\n> > >>>>>>> +\n> > >>>>>>> +Af::Af()\n> > >>>>>>> + : focus_(0), goodFocus_(0), currentVariance_(0.0),\n> previousVariance_(0.0),\n> > >>>>>>> + pass1Done_(false), pass2Done_(false)\n> > >>>>>>> +{\n> > >>>>>>> + maxStep_ = MaxFocusSteps_;\n> > >>>>>>> +}\n> > >>>>>>> +\n> > >>>>>>> +Af::~Af()\n> > >>>>>>> +{\n> > >>>>>>> +}\n> > >>>>>>> +\n> > >>>>>>> +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)\n> > >>>>>>\n> > >>>>>> Missing documentation for prepare() => you should build the doc\n> ;-)\n> > >>>>>\n> > >>>>> Ok. I'll add the prepare() document here.\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +{\n> > >>>>>>> + params->use.acc_af = 1;\n> > >>>>>>> + params->acc_param.af = imgu_css_af_defaults;\n> > >>>>>>> + params->acc_param.af.grid_cfg.x_start =\n> context.configuration.af.start_x;\n> > >>>>>>> + params->acc_param.af.grid_cfg.y_start =\n> context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;\n> > >>>>>>\n> > >>>>>> You are never changing the grid size, is it intended ?\n> > >>>>>\n> > >>>>> Do you mean \"ipu3_uapi_grid_config\"? Could you please explain this\n> > >>>>> more detail? I can modify this according to your opinions.\n> > >>>>\n> > >>>> Well, yes, you are always using the default 128x128 grid size. But\n> you\n> > >>>> can go from 16*2^3 to 32*2^6 in width, and from 16*2^3 to 24*2^6 in\n> > >>>> height. Thus, it means we could evaluate the scene on a small part\n> or\n> > >>>> almost all of it (copying the bds grid is not doable as it might\n> have\n> > >>>> larger block width, up to 80x60).\n> > >>>>\n> > >>>> See:\n> > >>>>\n> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#32\n> > >>>>\n> > >>>> Note:\n> > >>>>\n> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n> > >>>>\n> > >>>> => It makes me wonder if the filtering is fast, or if it may take\n> some\n> > >>>> time, meaning that focus evaluation in one frame could be done only\n> at\n> > >>>> frame+2 or something like that.\n> > >>>>\n> > >>>> Could you verify that the focus estimated is in the center of the\n> scene\n> > >>>> right now, as it is what you have configured ?\n> > >>>>\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +}\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\brief Configure the Af given a configInfo\n> > >>>>>>> + * \\param[in] context The shared IPA context\n> > >>>>>>> + * \\param[in] configInfo The IPA configuration data\n> > >>>>>>> + *\n> > >>>>>>> + * \\return 0\n> > >>>>>>> + */\n> > >>>>>>> +int Af::configure(IPAContext &context, const IPAConfigInfo\n> &configInfo)\n> > >>>>>>> +{\n> > >>>>>>> + /* determined focus value i.e. current focus value */\n> > >>>>>>> + context.frameContext.af.focus = 0;\n> > >>>>>>> + /* maximum variance of the AF statistics */\n> > >>>>>>> + context.frameContext.af.maxVariance = 0;\n> > >>>>>> double type, maybe s/= 0/= 0.0/ ?\n> > >>>>>\n> > >>>>> 0.0 is correct. I'll correct this thank you.\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> + /* is focused? if it is true, the AF should be in a stable\n> state. */\n> > >>>>>>> + context.frameContext.af.stable = false;\n> > >>>>>>> + /* frame to be ignored before start to estimate AF\n> variance. */\n> > >>>>>>> + ignoreFrame_ = 10;\n> > >>>>>>\n> > >>>>>> Could be a constexpr ?\n> > >>>>>\n> > >>>>> It is a counter to ignore frames since we have to ignore some\n> frames\n> > >>>>> to wait for the value from the accelerator to be stable.\n> > >>>>> If this should be a constant variable, I can modify this. :)\n> > >>>>>\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +\n> > >>>>>>> + /*\n> > >>>>>>> + * AF default area configuration\n> > >>>>>>> + * Move AF area to the center of the image.\n> > >>>>>>> + */\n> > >>>>>>> + /* Default AF width is 16x8 = 128 */\n> > >>>>>>> + context.configuration.af.start_x =\n> (configInfo.bdsOutputSize.width / 2) - 64;\n> > >>>>>>\n> > >>>>>> Why are you using a fixed value for the grid ? You could use the\n> value\n> > >>>>>> stored in imgu_css_af_defaults directly instead of applying 128/2\n> manualy ?\n> > >>>>>>\n> > >>>>>>> + context.configuration.af.start_y =\n> (configInfo.bdsOutputSize.height / 2) - 64;\n> > >>>>>>> +\n> > >>>>>>> + LOG(IPU3Af, Debug) << \"BDS X: \"\n> > >>>>>>> + << configInfo.bdsOutputSize.width\n> > >>>>>>> + << \" Y: \"\n> > >>>>>>> + << configInfo.bdsOutputSize.height;\n> > >>>>>>> + LOG(IPU3Af, Debug) << \"AF start from X: \"\n> > >>>>>>> + << context.configuration.af.start_x\n> > >>>>>>> + << \" Y: \"\n> > >>>>>>> + << context.configuration.af.start_y;\n> > >>>>>>> +\n> > >>>>>>> + return 0;\n> > >>>>>>> +}\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\brief AF coarse scan\n> > >>>>>>> + * \\param[in] context The shared IPA context\n> > >>>>>>> + *\n> > >>>>>>> + */\n> > >>>>>>> +void Af::af_coarse_scan(IPAContext &context)\n> > >>>>>>\n> > >>>>>> Bad naming (camelCase only). Same below.\n> > >>>>>\n> > >>>>> Ops, sure, I'll modify this.\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +{\n> > >>>>>>> + if (pass1Done_ == true)\n> > >>>>>>> + return;\n> > >>>>>>> +\n> > >>>>>>> + if (af_scan(context, coarseSearchStep_)) {\n> > >>>>>>> + pass1Done_ = true;\n> > >>>>>>> + context.frameContext.af.maxVariance = 0;\n> > >>>>>>> + focus_ = context.frameContext.af.focus -\n> (context.frameContext.af.focus * 0.1);\n> > >>>>>>> + context.frameContext.af.focus = focus_;\n> > >>>>>>> + previousVariance_ = 0;\n> > >>>>>>> + maxStep_ = focus_ + (focus_ * 0.2);\n> > >>>>>>> + }\n> > >>>>>>> +}\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\brief AF fine scan\n> > >>>>>>> + * \\param[in] context The shared IPA context\n> > >>>>>>> + *\n> > >>>>>>> + */\n> > >>>>>>> +void Af::af_fine_scan(IPAContext &context)\n> > >>>>>>> +{\n> > >>>>>>> + if (pass1Done_ != true)\n> > >>>>>>> + return;\n> > >>>>>>> +\n> > >>>>>>> + if (af_scan(context, fineSearchStep_)) {\n> > >>>>>>> + context.frameContext.af.stable = true;\n> > >>>>>>> + pass2Done_ = true;\n> > >>>>>>> + }\n> > >>>>>>> +}\n> > >>>>>> I am not a big fan of these pass1Done_ and pass2Done_ variables,\n> let's\n> > >>>>>> seee where it is used...\n> > >>>>>\n> > >>>>> Since we used two stage AF searching strategy I simply say this is\n> a\n> > >>>>> two pass algorithm :)\n> > >>>>> I'll renaming this.\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\brief AF reset\n> > >>>>>>> + * \\param[in] context The shared IPA context\n> > >>>>>>> + *\n> > >>>>>>> + */\n> > >>>>>>> +void Af::af_reset(IPAContext &context)\n> > >>>>>>> +{\n> > >>>>>>> + context.frameContext.af.maxVariance = 0;\n> > >>>>>>> + context.frameContext.af.focus = 0;\n> > >>>>>>> + focus_ = 0;\n> > >>>>>>> + context.frameContext.af.stable = false;\n> > >>>>>>> + ignoreFrame_ = 60;\n> > >>>>>>\n> > >>>>>> Why 60 after reset and 10 at configure ? It seems to be a lot ?\n> > >>>>>\n> > >>>>> We can tweak this value.\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> + previousVariance_ = 0.0;\n> > >>>>>>> + pass1Done_ = false;\n> > >>>>>>> + pass2Done_ = false;\n> > >>>>>>> + maxStep_ = MaxFocusSteps_;\n> > >>>>>>> +}\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\brief AF scan\n> > >>>>>>> + * \\param[in] context The shared IPA context\n> > >>>>>>> + *\n> > >>>>>>> + * \\return True, if it finds a AF value.\n> > >>>>>>> + */\n> > >>>>>>> +bool Af::af_scan(IPAContext &context, int min_step)\n> > >>>>>>> +{\n> > >>>>>>> + /* find the maximum variance during the AF scan using a\n> greedy strategy */\n> > >>>>>>> + if (currentVariance_ > context.frameContext.af.maxVariance)\n> {\n> > >>>>>>> + context.frameContext.af.maxVariance =\n> currentVariance_;\n> > >>>>>>> + goodFocus_ = focus_;\n> > >>>>>>> + }\n> > >>>>>>> +\n> > >>>>>>> + if (focus_ > maxStep_) {\n> > >>>>>>> + /* if reach the max step, move lens to the\n> position and set \"focus stable\". */\n> > >>>>>>> + context.frameContext.af.focus = goodFocus_;\n> > >>>>>>> + return true;\n> > >>>>>>> + } else {\n> > >>>>>>> + /* check negative gradient */\n> > >>>>>>> + if ((currentVariance_ - context.frameContext.af.maxVariance)\n> > -(context.frameContext.af.maxVariance * 0.15)) {\n> > >>>>>>\n> > >>>>>> Where is the 0.15 coming from ?\n> > >>>>>\n> > >>>>> After some testing, I think that If the negative gradient ismore\n> than\n> > >>>>> 15% maxVaraiance, the focused image and lens position we get!\n> > >>>>> I could put this as a constant variable in my v5 patch.\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> + focus_ += min_step;\n> > >>>>>>> + context.frameContext.af.focus = focus_;\n> > >>>>>>> + } else {\n> > >>>>>>> + context.frameContext.af.focus = goodFocus_;\n> > >>>>>>> + previousVariance_ = currentVariance_;\n> > >>>>>>> + return true;\n> > >>>>>>> + }\n> > >>>>>>> + }\n> > >>>>>>> + LOG(IPU3Af, Debug) << \"Variance prevrious: \"\n> > >>>>>> s/prevrious/previous/\n> > >>>>>\n> > >>>>> Ops. sorry about the typo. T_T\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> + << previousVariance_\n> > >>>>>>> + << \" current: \"\n> > >>>>>>> + << currentVariance_\n> > >>>>>>> + << \" Diff: \"\n> > >>>>>>> + << (currentVariance_ -\n> context.frameContext.af.maxVariance);\n> > >>>>>>\n> > >>>>>> Interestingly, it seems there is an issue here, according to the\n> log I get ?\n> > >>>>>>\n> > >>>>>> [0:58:23.867136346] [6811] DEBUG IPU3Af af.cpp:238 Variance\n> prevrious: 0\n> > >>>>>> current: 20012.4 Diff: 0\n> > >>>>>>\n> > >>>>>> Shouldn't diff be 20012.4 ?\n> > >>>>>>\n> > >>>>>> I am not sure the algorithm is very strong though... You may never\n> > >>>>>> converge and stay in a local minimum without knowing ?\n> > >>>>>>\n> > >>>>>>> + previousVariance_ = currentVariance_;\n> > >>>>>>> + LOG(IPU3Af, Debug) << \"Focus searching max variance is: \"\n> > >>>>>>> + << context.frameContext.af.maxVariance\n> > >>>>>>> + << \" Focus step is \"\n> > >>>>>>> + << goodFocus_\n> > >>>>>>> + << \" Current scan is \"\n> > >>>>>>> + << focus_;\n> > >>>>>>> + return false;\n> > >>>>>>> +}\n> > >>>>>>\n> > >>>>>> I am not sure to understand how this function should behave... Is\n> it a\n> > >>>>>> minimum searching ? Are you looking for argmin(currentVariance -\n> > >>>>>> maxVariance) ?\n> > >>>>>\n> > >>>>> Actually. it is a maximum search. I try to search for the maximum\n> > >>>>> variance of the image.\n> > >>>>> If the difference is positive, it means the variance still\n> increases.\n> > >>>>> We can continue search for the maximum value and update\n> maxVariance_.\n> > >>>>> Until, we found a negative difference which means the variance\n> starts\n> > >>>>> to go down and image will become blurred.\n> > >>>>>\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\brief Determine the max contrast image and lens position.\n> y_table is the\n> > >>>>>>> + * statictic data from IPU3 and is composed of low pass and\n> high pass filtered\n> > >>>>>>> + * value. High pass filtered value also represents the\n> sharpness of the image.\n> > >>>>>>> + * Based on this, if the image with highest variance of the\n> high pass filtered\n> > >>>>>>> + * value (contrast) during the AF scan, the position of the len\n> should be the\n> > >>>>>>> + * best focus.\n> > >>>>>>> + * \\param[in] context The shared IPA context.\n> > >>>>>>> + * \\param[in] stats The statistic buffer of 3A from the IPU3.\n> > >>>>>>> + */\n> > >>>>>>> +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a\n> *stats)\n> > >>>>>>> +{\n> > >>>>>>> + uint32_t total = 0;\n> > >>>>>>> + double mean;\n> > >>>>>>> + uint64_t var_sum = 0;\n> > >>>>>>> + y_table_item_t *y_item;\n> > >>>>>>> + int z = 0;\n> > >>>>>>> +\n> > >>>>>>> + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;\n> > >>>>>>> +\n> > >>>>>>> + /**\n> > >>>>>>> + * Calculate the mean and the varience of each non-zero AF\n> statistics, since IPU3 only determine the AF value\n> > >>>>>>> + * for a given grid.\n> > >>>>>>> + * For pass1: low pass results are used.\n> > >>>>>>> + * For pass2: high pass results are used.\n> > >>>>>>> + */\n> > >>>>>>> + if (pass1Done_) {\n> > >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) /\n> 4; z++) {\n> > >>>>>>> + total = total + y_item[z].y2_avg;\n> > >>>>>>> + if (y_item[z].y2_avg == 0)\n> > >>>>>>> + break;\n> > >>>>>>> + }\n> > >>>>>>> + mean = total / z;\n> > >>>>>>> +\n> > >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) /\n> 4 && y_item[z].y2_avg != 0; z++) {\n> > >>>>>>> + var_sum = var_sum + ((y_item[z].y2_avg -\n> mean) * (y_item[z].y2_avg - mean));\n> > >>>>>>> + if (y_item[z].y2_avg == 0)\n> > >>>>>>> + break;\n> > >>>>>>> + }\n> > >>>>>>> + } else {\n> > >>>>>>\n> > >>>>>> /* Calculate the mean of the low pass filter values. */\n> > >>>>>>\n> > >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) /\n> 4; z++) {\n> > >>>>>>> + total = total + y_item[z].y1_avg;\n> > >>>>>>> + if (y_item[z].y1_avg == 0)\n> > >>>>>>> + break;\n> > >>>>>> Are you sure of that ? Shouldn't you use the grid you set to\n> parse the\n> > >>>>>> right number of data ? If you have a local dark area you may stop\n> even\n> > >>>>>> if it is not yet the end of the grid ?\n> > >>>>>\n> > >>>>> I'll find another way to determine the range of the results. May\n> be it\n> > >>>>> can based on the grids setting.\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> + }\n> > >>>>>>> + mean = total / z;\n> > >>>>>>> +\n> > >>>>>>\n> > >>>>>> /* Calulate the deviation from the mean. */\n> > >>>>>>\n> > >>>>>>> + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) /\n> 4 && y_item[z].y1_avg != 0; z++) {\n> > >>>>>>> + var_sum = var_sum + ((y_item[z].y1_avg -\n> mean) * (y_item[z].y1_avg - mean));\n> > >>>>>>> + if (y_item[z].y1_avg == 0)\n> > >>>>>>> + break;\n> > >>>>>>> + }\n> > >>>>>>> + }\n> > >>>>>>\n> > >>>>>> The two if() blocks are very similar, the only exception beeing\n> the item\n> > >>>>>> of the table used (y_item[z].y1_avg in the first pass,\n> y_item[z].y2_avg\n> > >>>>>> in the second one). This is not easy to read (according to me).\n> > >>>>>\n> > >>>>> They are high pass (y2) and low pass (y1) filtered convolution\n> > >>>>> results. This structure referees from ipu-ipa repo and I would\n> like to\n> > >>>>> keep their original naming.\n> > >>>>>\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> + /* Determine the average variance of the frame. */\n> > >>>>>>> + currentVariance_ = static_cast(var_sum) /\n> static_cast(z);\n> > >>>>>>\n> > >>>>>> Side note:\n> > >>>>>> If my stats are still correct, I think it should be divided by\n> (z-1) and\n> > >>>>>> not z, as we have a sample and not a full population ?\n> > >>>>>\n> > >>>>> it is z-1 and my fault. I'll correct this. :)\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> + LOG(IPU3Af, Debug) << \"variance: \" << currentVariance_;\n> > >>>>>>> +\n> > >>>>>>> + if (context.frameContext.af.stable == true) {\n> > >>>>>>> + const uint32_t diff_var =\n> std::abs(currentVariance_ - context.frameContext.af.maxVariance);\n> > >>>>>>> + const double var_ratio = diff_var /\n> context.frameContext.af.maxVariance;\n> > >>>>>>> + LOG(IPU3Af, Debug) << \"Change ratio: \"\n> > >>>>>>> + << var_ratio\n> > >>>>>>> + << \" current focus: \"\n> > >>>>>>> + <<\n> context.frameContext.af.focus;\n> > >>>>>>> + /**\n> > >>>>>>> + * If the change ratio of contrast is over\n> Maxchange_ (out of focus),\n> > >>>>>>> + * trigger AF again.\n> > >>>>>>> + */\n> > >>>>>>> + if (var_ratio > MaxChange_) {\n> > >>>>>>> + if (ignoreFrame_ == 0) {\n> > >>>>>>> + af_reset(context);\n> > >>>>>>> + } else\n> > >>>>>>> + ignoreFrame_--;\n> > >>>>>>> + } else\n> > >>>>>>> + ignoreFrame_ = 10;\n> > >>>>>>> + } else {\n> > >>>>>>> + if (ignoreFrame_ != 0)\n> > >>>>>>> + ignoreFrame_--;\n> > >>>>>>> + else {\n> > >>>>>>> + af_coarse_scan(context);\n> > >>>>>>> + af_fine_scan(context);\n> > >>>>>>> + }\n> > >>>>>>> + }\n> > >>>>>>> +}\n> > >>>>>>> +\n> > >>>>>>> +} /* namespace ipa::ipu3::algorithms */\n> > >>>>>>> +\n> > >>>>>>> +} /* namespace libcamera */\n> > >>>>>>> diff --git a/src/ipa/ipu3/algorithms/af.h\n> b/src/ipa/ipu3/algorithms/af.h\n> > >>>>>>> new file mode 100644\n> > >>>>>>> index 00000000..b9295b19\n> > >>>>>>> --- /dev/null\n> > >>>>>>> +++ b/src/ipa/ipu3/algorithms/af.h\n> > >>>>>>> @@ -0,0 +1,66 @@\n> > >>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > >>>>>>> +/*\n> > >>>>>>> + * Copyright (C) 2021, Red Hat\n> > >>>>>>> + *\n> > >>>>>>> + * af.h - IPU3 Af control\n> > >>>>>>> + */\n> > >>>>>>> +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> > >>>>>>> +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__\n> > >>>>>>> +\n> > >>>>>>> +#include \n> > >>>>>>> +\n> > >>>>>>> +#include \n> > >>>>>>> +\n> > >>>>>>> +#include \n> > >>>>>>> +\n> > >>>>>>> +#include \"algorithm.h\"\n> > >>>>>>> +\n> > >>>>>>> +namespace libcamera {\n> > >>>>>>> +\n> > >>>>>>> +namespace ipa::ipu3::algorithms {\n> > >>>>>>> +\n> > >>>>>>> +class Af : public Algorithm\n> > >>>>>>> +{\n> > >>>>>>> + /* The format of y_table. From ipu3-ipa repo */\n> > >>>>>>> + typedef struct y_table_item {\n> > >>>>>>> + uint16_t y1_avg;\n> > >>>>>>> + uint16_t y2_avg;\n> > >>>>>>> + } y_table_item_t;\n> > >>>>>>> +\n> > >>>>>>> +public:\n> > >>>>>>> + Af();\n> > >>>>>>> + ~Af();\n> > >>>>>>> +\n> > >>>>>>> + void prepare(IPAContext &context, ipu3_uapi_params\n> *params) override;\n> > >>>>>>> + int configure(IPAContext &context, const IPAConfigInfo\n> &configInfo) override;\n> > >>>>>>> + void process(IPAContext &context, const ipu3_uapi_stats_3a\n> *stats) override;\n> > >>>>>>> +\n> > >>>>>>> +private:\n> > >>>>>>> + void af_coarse_scan(IPAContext &context);\n> > >>>>>>> + void af_fine_scan(IPAContext &context);\n> > >>>>>>> + bool af_scan(IPAContext &context, int min_step);\n> > >>>>>>> + void af_reset(IPAContext &context);\n> > >>>>>>> +\n> > >>>>>>> + /* Used for focus scan. */\n> > >>>>>>> + uint32_t focus_;\n> > >>>>>>> + /* Focus good */\n> > >>>>>>> + uint32_t goodFocus_;\n> > >>>>>>> + /* Recent AF statistic variance. */\n> > >>>>>>> + double currentVariance_;\n> > >>>>>>> + /* The frames to be ignore before starting measuring. */\n> > >>>>>>> + uint32_t ignoreFrame_;\n> > >>>>>>> + /* previous variance. it is used to determine the gradient\n> */\n> > >>>>>>> + double previousVariance_;\n> > >>>>>>> + /* Max scan steps of each pass of AF scaning */\n> > >>>>>>> + uint32_t maxStep_;\n> > >>>>>>> + /* Pass 1 stable. Complete low pass search (coarse) scan)\n> */\n> > >>>>>>> + bool pass1Done_;\n> > >>>>>>> + /* Pass 2 stable. Complete high pass scan (fine scan) */\n> > >>>>>>> + bool pass2Done_;\n> > >>>>>>> +};\n> > >>>>>>> +\n> > >>>>>>> +} /* namespace ipa::ipu3::algorithms */\n> > >>>>>>> +\n> > >>>>>>> +} /* namespace libcamera */\n> > >>>>>>> +\n> > >>>>>>> +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */\n> > >>>>>>> diff --git a/src/ipa/ipu3/algorithms/meson.build\n> b/src/ipa/ipu3/algorithms/meson.build\n> > >>>>>>> index 4db6ae1d..e1099169 100644\n> > >>>>>>> --- a/src/ipa/ipu3/algorithms/meson.build\n> > >>>>>>> +++ b/src/ipa/ipu3/algorithms/meson.build\n> > >>>>>>> @@ -1,8 +1,9 @@\n> > >>>>>>> # SPDX-License-Identifier: CC0-1.0\n> > >>>>>>>\n> > >>>>>>> ipu3_ipa_algorithms = files([\n> > >>>>>>> + 'af.cpp',\n> > >>>>>>> 'agc.cpp',\n> > >>>>>>> 'awb.cpp',\n> > >>>>>>> 'blc.cpp',\n> > >>>>>>> - 'tone_mapping.cpp',\n> > >>>>>>> + 'tone_mapping.cpp'\n> > >>>>>>> ])\n> > >>>>>>> diff --git a/src/ipa/ipu3/ipa_context.cpp\n> b/src/ipa/ipu3/ipa_context.cpp\n> > >>>>>>> index 86794ac1..ee644d3c 100644\n> > >>>>>>> --- a/src/ipa/ipu3/ipa_context.cpp\n> > >>>>>>> +++ b/src/ipa/ipu3/ipa_context.cpp\n> > >>>>>>> @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 {\n> > >>>>>>> *\n> > >>>>>>> * \\var IPASessionConfiguration::grid.stride\n> > >>>>>>> * \\brief Number of cells on one line including the ImgU\n> padding\n> > >>>>>>> + *\n> > >>>>>>> + */\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\var IPASessionConfiguration::af\n> > >>>>>>> + * \\brief AF parameters configuration of the IPA\n> > >>>>>>> + *\n> > >>>>>>> + * \\var IPASessionConfiguration::af.start_x\n> > >>>>>>> + * \\brief The start X position of the AF area\n> > >>>>>>> + *\n> > >>>>>>> + * \\var IPASessionConfiguration::af.start_y\n> > >>>>>>> + * \\brief The start Y position of the AF area\n> > >>>>>>> + */\n> > >>>>>>> +\n> > >>>>>>> +/**\n> > >>>>>>> + * \\var IPAFrameContext::af\n> > >>>>>>> + * \\brief Context for the Automatic Focus algorithm\n> > >>>>>>> + *\n> > >>>>>>> + * \\struct IPAFrameContext::af\n> > >>>>>>> + * \\var IPAFrameContext::af.focus\n> > >>>>>>> + * \\brief Current position of the lens\n> > >>>>>>> + *\n> > >>>>>>> + * \\var IPAFrameContext::af.maxVariance\n> > >>>>>>> + * \\brief The maximum variance of the current image.\n> > >>>>>>> + *\n> > >>>>>>> + * \\var IPAFrameContext::af.stable\n> > >>>>>>> + * \\brief is the image focused?\n> > >>>>>>> */\n> > >>>>>>>\n> > >>>>>>> /**\n> > >>>>>>> diff --git a/src/ipa/ipu3/ipa_context.h\n> b/src/ipa/ipu3/ipa_context.h\n> > >>>>>>> index c6dc0814..aa5bf97f 100644\n> > >>>>>>> --- a/src/ipa/ipu3/ipa_context.h\n> > >>>>>>> +++ b/src/ipa/ipu3/ipa_context.h\n> > >>>>>>> @@ -31,6 +31,11 @@ struct IPASessionConfiguration {\n> > >>>>>>> double minAnalogueGain;\n> > >>>>>>> double maxAnalogueGain;\n> > >>>>>>> } agc;\n> > >>>>>>> +\n> > >>>>>>> + struct {\n> > >>>>>>> + uint16_t start_x;\n> > >>>>>>> + uint16_t start_y;\n> > >>>>>>> + } af;\n> > >>>>>>> };\n> > >>>>>>>\n> > >>>>>>> struct IPAFrameContext {\n> > >>>>>>> @@ -49,6 +54,12 @@ struct IPAFrameContext {\n> > >>>>>>> double temperatureK;\n> > >>>>>>> } awb;\n> > >>>>>>>\n> > >>>>>>> + struct {\n> > >>>>>>> + uint32_t focus;\n> > >>>>>>> + double maxVariance;\n> > >>>>>>> + bool stable;\n> > >>>>>>> + } af;\n> > >>>>>>> +\n> > >>>>>>> struct {\n> > >>>>>>> uint32_t exposure;\n> > >>>>>>> double gain;\n> > >>>>>>> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> > >>>>>>> index 3d307708..93966c6f 100644\n> > >>>>>>> --- a/src/ipa/ipu3/ipu3.cpp\n> > >>>>>>> +++ b/src/ipa/ipu3/ipu3.cpp\n> > >>>>>>> @@ -30,6 +30,7 @@\n> > >>>>>>>\n> > >>>>>>> #include \"libcamera/internal/mapped_framebuffer.h\"\n> > >>>>>>>\n> > >>>>>>> +#include \"algorithms/af.h\"\n> > >>>>>>> #include \"algorithms/agc.h\"\n> > >>>>>>> #include \"algorithms/algorithm.h\"\n> > >>>>>>> #include \"algorithms/awb.h\"\n> > >>>>>>> @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings\n> &settings,\n> > >>>>>>> }\n> > >>>>>>>\n> > >>>>>>> /* Construct our Algorithms */\n> > >>>>>>> + algorithms_.push_back(std::make_unique());\n> > >>>>>>>\n> algorithms_.push_back(std::make_unique());\n> > >>>>>>>\n> algorithms_.push_back(std::make_unique());\n> > >>>>>>>\n> algorithms_.push_back(std::make_unique());\n> > >>>>>>>\n> > >>>>>>\n> > >>>>>\n> > >>>>> Thank you :)\n> > >>>>>\n> > >>>>\n> > >>>\n> > >>>\n> > >>\n> > >\n> > >\n> >\n>\n>\n> --\n> BR,\n> Kate\n>\n\nI also did some experiments on grid configuration of AF scene.\n\nYou can see the picture below.\n\nhttps://drive.google.com/file/d/1UQc6emy4Qx-TAMqN3Bk-BO7lOOzoPRyq/view?usp=sharing\nhttps://drive.google.com/file/d/1RC5JmCrLFetw5PEo_hlGnjbJXnplsc6F/view?usp=sharing\n\nThe first image showed the configuration of the AF grid. The green box was\nthe range of the AF scene.\nThe image of Y1 was the same as the AF scene but Y2 was completely white :(.\n\nThe grid configuration as shown below\n\n{\n.width = 16,\n.height = 16,\n.block_width_log2 = 3,\n.block_height_log2 = 3,\n.height_per_slice = 2,\n.x_start = 640,\n.y_start = 360 | IPU3_UAPI_GRID_Y_START_EN,\n.x_end = 0,\n.y_end = 0\n},\n\nMoreover, the x_start seems can't be set to greater than 650 (image\nwidth/2+10). If it is >650 the AF scene will be (10, 2) the default\nconfiguration.\nWhat do you think about this?\n\nThank you.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 035ABBE086\n\tfor ;\n\tThu, 13 Jan 2022 11:36:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3A321604F8;\n\tThu, 13 Jan 2022 12:36:20 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4ADA4604F4\n\tfor ;\n\tThu, 13 Jan 2022 12:36:17 +0100 (CET)","from mail-lf1-f69.google.com (mail-lf1-f69.google.com\n\t[209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-403-o0gc29kEO0mv6Jy0u91V8A-1; Thu, 13 Jan 2022 06:36:07 -0500","by mail-lf1-f69.google.com with SMTP id\n\ts16-20020a056512215000b0042bd76cb189so3808311lfr.6\n\tfor ;\n\tThu, 13 Jan 2022 03:36:07 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"MwgBXI/P\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1642073776;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=GOTvV//2AdS3C9yJlrid28CmV/QqF8/kBr9wDjrPkwg=;\n\tb=MwgBXI/PiVJ5DV2Jq8e3RWVfig/VC7upgGEM+FLlyC+bniNSPwkeFsnrJWszGU4If6guRn\n\t9wVBVJQHfhdt0vI65tdu5TuEcWVMw3wLj4naCcFHFXphubCGMsUq25wcM3vCkAEm2m5CKO\n\t/9VUu4Es41Kt8RqZL1sGgf0fytU2EnQ=","X-MC-Unique":"o0gc29kEO0mv6Jy0u91V8A-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=2H2wCJGE2BGdnWRmfBE3/AUW5Ls8exmZSPmczuN63ZY=;\n\tb=WDTEo5DVJlGChX7djxiyjPkYDyM5JNWniMUvOHd17wEDGt8Fqx6vsOEQ3E4PObawtv\n\t4ifxE4Dr4PELElSd/1EuQAxDzpIK3y2xH7NbUTrRD1qxVowUP6J7z3f3eKHPM7Ic+Ac4\n\txlt1U6dKqQkn7cOKXQ2WZ2jwxCUE1DeTArWmiGPDsCbfz1Yk3Xi1lDpqSZPmr3prYzoG\n\tnhNHWQdmrhP5PRmT+qLbe/32tbkIgUuCNdyS1O6a91+fC3nYGMWlDOv0WiZrgKwCc/9T\n\tR6G8WjKK20pwnPOn7cRlckzwFZVLpWzOQe6874vK0E8WfrOpuFMlOoe+BznpIRfl2WKc\n\tFplQ==","X-Gm-Message-State":"AOAM532y3V19zdiW0C2LsfeE/nP77MrJW2EzC17JsnjA/AHIHX3B4N0G\n\t4CxxGkZ2j4uitByw2YtaxqKs8lHwF43VlFlTlpwamLWi/ydehmF5go7iOneGVWiG74thust7yyx\n\tZCE9sv9T330BemM5BO6rubc1IT4K82xmOWrez/gmw1hdyp3aG4g==","X-Received":["by 2002:a05:6512:3084:: with SMTP id\n\tz4mr2836556lfd.515.1642073765413; \n\tThu, 13 Jan 2022 03:36:05 -0800 (PST)","by 2002:a05:6512:3084:: with SMTP id\n\tz4mr2836532lfd.515.1642073764818; \n\tThu, 13 Jan 2022 03:36:04 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJxEl0B1NWpj7+gNM1o5ODbj49fj/DD5PH0taQlMdLOjxGyxJB2DbhVRRaCRCV+CAErHmkGXNq53cwaNIvl/gOY=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t\n\t","In-Reply-To":"","From":"Kate Hsuan ","Date":"Thu, 13 Jan 2022 19:35:52 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"multipart/alternative; boundary=\"0000000000007288e505d5751755\"","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22026,"web_url":"https://patchwork.libcamera.org/comment/22026/","msgid":"<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>","date":"2022-01-13T12:58:27","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate,\n\nBefore I forget it again :-) could you please answer in plain-text and \nnot HTML ;-) ?\n\nOn 13/01/2022 12:35, Kate Hsuan wrote:\n> Hi Jean-Michel,\n> \n> \n\n\n\n> > >>>>>\n> > >>>>>>\n> > >>>>>>> +\n> > >>>>>>> +/* settings for Auto Focus from the kernel */\n> > >>>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> > >>>>>>> +     .filter_config = {\n> > >>>>>>> +             { 0, 0, 0, 0 },\n> > >>>>>>> +             { 0, 0, 0, 0 },\n> > >>>>>>> +             { 0, 0, 0, 128 },\n> > >>>>>>> +             0,\n> > >>>>>>> +             { 0, 0, 0, 0 },\n> > >>>>>>> +             { 0, 0, 0, 0 },\n> > >>>>>>> +             { 0, 0, 0, 128 },\n> > >>>>>>> +             0,\n> > >>>>>>> +             .y_calc = { 8, 8, 8, 8 },\n> > >>>>>>> +             .nf = { 0, 7, 0, 7, 0 },\n> > >>>>>>> +     },\n> > >>>>\n> > >>>> I would like your thinking here (and perhaps someone else).\n> It looks\n> > >>>> like the ImgU implements a Gabor filter, but I can't see all the\n> > >>>> parameters I would expect from it. Putting it down, I can't\n> see with\n> > >>>> this default filter how y1 and y2 could represent low-pass\n> and high\n> > >>>> pass, as the filters are the same. But, it could be a\n> horizontal and a\n> > >>>> vertical filter instead, with y1 the horizontal and y2 the\n> vertical\n> > >>>> resulting filtered values.\n> > >>>>\n> > >>>> It would be very interesting to display the y1_avg and\n> y2_avg values as\n> > >>>> an image, I just don't have lots of time for it. We have 16\n> bits values,\n> > >>>> so we could dump those into a binary file and use python to\n> display the\n> > >>>> values as a raw gray image or something similar.\n> > >>>\n> > >>> You could check out the URL below to see the grey scale image\n> of the AF buffer.\n> > >>>\n> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> \n> > >>>\n> > >>> Since the result is a tiny 16x16 image, all the detail of the\n> original\n> > >>> image was compressed into a black block.\n> > >>>\n> > >>\n> > >> Nice :-).\n> > >> As expected, as y1 and y2 are the same filter, images are the\n> same.\n> > >> Could you test those values ?\n> > >>\n> > >> y1 =>\n> > >> (0, 1, 3, 7\n> > >>    11, 13, 1, 2\n> > >>    8, 19, 34, 242)\n> > >>    vector: 7fdffbfe - normalization factor: 9\n> > >> y2 =>\n> > >> (0, 1, 6, 6\n> > >>    13, 25, 3, 0\n> > >>    25, 3, 117, 254)\n> > >>    vector: 4e53ca72 - normalization factor: 9\n> > >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n> > >>\n> > >\n> > > Please check out the results from the URL below.\n> > >\n> > >\n> https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n> \n> > >\n> > > Y1 became a black image and was difficult to determine the\n> coarse focus value.\n> > >\n> >\n> \n> > Sounds surprising... Could you share the way you are generating the\n> > images of Y1 and Y2 BTW ?\n> >\n> > > Happy new year :)\n> >\n> > Best wishes, hoping for a far away Covid next year !\n> I hope too :)\n> \n> After some checks, I made some mistakes to convert the image to png\n> file.\n> \n> Check out the following URL for the new results.\n> https://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing\n> \n> \n> But, for the Y2, the result becomes very bright and sometimes the\n> image will be all white. Are the results right?\n> \n> My python code for converting the pixels to tiff is shown below.\n> \n> =========\n> import numpy as np\n> from scipy import ndimage, misc\n> import imageio\n> from PIL import Image\n> import sys\n> \n> rawfile = np.fromfile(sys.argv[1], \"uint16\")\n> rawfile.shape = (16,16)\n> new_image = Image.fromarray(rawfile, mode='I;16')\n> new_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n> ==========\n> \n\n\n\n> \n> I also did some experiments on grid configuration of AF scene.\n> \n> You can see the picture below.\n> \n> https://drive.google.com/file/d/1UQc6emy4Qx-TAMqN3Bk-BO7lOOzoPRyq/view?usp=sharing \n> \n> https://drive.google.com/file/d/1RC5JmCrLFetw5PEo_hlGnjbJXnplsc6F/view?usp=sharing \n> \n> \n> The first image showed the configuration of the AF grid. The green box \n> was the range of the AF scene.\n> The image of Y1 was the same as the AF scene but Y2 was completely white :(.\n> \n\nThe image used for AF calculations is the BDS output one, so you should \nconfigure your grid based on this.\n\nSee:\nhttps://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2541\n\n> The grid configuration as shown below\n> \n> {\n> .width = 16,\n> .height = 16,\n> .block_width_log2 = 3,\n> .block_height_log2 = 3,\n> .height_per_slice = 2,\n> .x_start = 640,\n> .y_start = 360 | IPU3_UAPI_GRID_Y_START_EN,\n> .x_end = 0,\n> .y_end = 0\n> },\n\nThis grid configuration is probably not what you really want :-).\nYou are configuring a 128x128 grid, and your BDS output is (fill in the \nblank please :-)). So you want:\nx_start = (BdsOutput.width / 2 - 64),\ny_start = (BdsOutput.height / 2 - 64) | IPU3_UAPI_GRID_Y_START_EN,\n\n> Moreover, the x_start seems can't be set to greater than 650 (image \n> width/2+10). If it is >650 the AF scene will be (10, 2) the default \n> configuration.\n\nFrom:\nhttps://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n\n\"margins of 10 per horizontal direction and 2 per vertical direction\" => \nIt means the real AF grid is using 10 pixels before your x_start, so it \ncan't be less than 10, and 2 pixels before your y_start, so it can't be \nless than 2.\n\nYou can obviously take it into account if you really want to, but you \nare trying to center the grid, so it should not be an issue.\n\nYou should set height_per_slice to 1, as it is calculated by the kernel, \nand I would be interested by this: could it be a limitation related to:\nhttps://patchwork.linuxtv.org/project/linux-media/patch/20210916172504.677919-1-jeanmichel.hautbois@ideasonboard.com/\n\nCan you try building the kernel with this patch by any chance ?\n\nThanks for your feedback !\nJM","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DD701BE086\n\tfor ;\n\tThu, 13 Jan 2022 12:58:32 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E7194605A8;\n\tThu, 13 Jan 2022 13:58:31 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 98B62604F4\n\tfor ;\n\tThu, 13 Jan 2022 13:58:30 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:2ab9:bf59:c61a:cf11] (unknown\n\t[IPv6:2a01:e0a:169:7140:2ab9:bf59:c61a:cf11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 361D497;\n\tThu, 13 Jan 2022 13:58:30 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"WzpfHijk\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1642078710;\n\tbh=tUUxETygoLGFqGECGXKx8kJms2h2BvbfgeoWYsKYCF4=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=WzpfHijkYZt3HYl8W9YBtGk95frUJv8fuq9Vz+U9Dvp4PvlX3pXXxEOrhqX5+SQ9f\n\tEDw5DYGOmskXexc95zEyojlJ60K3lAsEzRNlvXHuKuxJoAQkZQ6gPLM/z/kSnU0ZnK\n\tZ1On8i9C/RrrLSInwiAqzbZLKaiP7KDWdgDAoq2g=","Message-ID":"<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>","Date":"Thu, 13 Jan 2022 13:58:27 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.3.1","Content-Language":"en-US","To":"Kate Hsuan ","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t\n\t\n\t","From":"Jean-Michel Hautbois ","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22031,"web_url":"https://patchwork.libcamera.org/comment/22031/","msgid":"","date":"2022-01-14T10:46:17","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel,\n\nOn Thu, Jan 13, 2022 at 8:58 PM Jean-Michel Hautbois\n wrote:\n>\n> Hi Kate,\n>\n> Before I forget it again :-) could you please answer in plain-text and\n> not HTML ;-) ?\n\nOps, my fault. I'll double-check my Gmail setting. Sorry for the trouble.\n\n\n>\n> On 13/01/2022 12:35, Kate Hsuan wrote:\n> > Hi Jean-Michel,\n> >\n> >\n>\n> \n>\n> > > >>>>>\n> > > >>>>>>\n> > > >>>>>>> +\n> > > >>>>>>> +/* settings for Auto Focus from the kernel */\n> > > >>>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> > > >>>>>>> + .filter_config = {\n> > > >>>>>>> + { 0, 0, 0, 0 },\n> > > >>>>>>> + { 0, 0, 0, 0 },\n> > > >>>>>>> + { 0, 0, 0, 128 },\n> > > >>>>>>> + 0,\n> > > >>>>>>> + { 0, 0, 0, 0 },\n> > > >>>>>>> + { 0, 0, 0, 0 },\n> > > >>>>>>> + { 0, 0, 0, 128 },\n> > > >>>>>>> + 0,\n> > > >>>>>>> + .y_calc = { 8, 8, 8, 8 },\n> > > >>>>>>> + .nf = { 0, 7, 0, 7, 0 },\n> > > >>>>>>> + },\n> > > >>>>\n> > > >>>> I would like your thinking here (and perhaps someone else).\n> > It looks\n> > > >>>> like the ImgU implements a Gabor filter, but I can't see all the\n> > > >>>> parameters I would expect from it. Putting it down, I can't\n> > see with\n> > > >>>> this default filter how y1 and y2 could represent low-pass\n> > and high\n> > > >>>> pass, as the filters are the same. But, it could be a\n> > horizontal and a\n> > > >>>> vertical filter instead, with y1 the horizontal and y2 the\n> > vertical\n> > > >>>> resulting filtered values.\n> > > >>>>\n> > > >>>> It would be very interesting to display the y1_avg and\n> > y2_avg values as\n> > > >>>> an image, I just don't have lots of time for it. We have 16\n> > bits values,\n> > > >>>> so we could dump those into a binary file and use python to\n> > display the\n> > > >>>> values as a raw gray image or something similar.\n> > > >>>\n> > > >>> You could check out the URL below to see the grey scale image\n> > of the AF buffer.\n> > > >>>\n> > https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> > \n> > > >>>\n> > > >>> Since the result is a tiny 16x16 image, all the detail of the\n> > original\n> > > >>> image was compressed into a black block.\n> > > >>>\n> > > >>\n> > > >> Nice :-).\n> > > >> As expected, as y1 and y2 are the same filter, images are the\n> > same.\n> > > >> Could you test those values ?\n> > > >>\n> > > >> y1 =>\n> > > >> (0, 1, 3, 7\n> > > >> 11, 13, 1, 2\n> > > >> 8, 19, 34, 242)\n> > > >> vector: 7fdffbfe - normalization factor: 9\n> > > >> y2 =>\n> > > >> (0, 1, 6, 6\n> > > >> 13, 25, 3, 0\n> > > >> 25, 3, 117, 254)\n> > > >> vector: 4e53ca72 - normalization factor: 9\n> > > >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n> > > >>\n> > > >\n> > > > Please check out the results from the URL below.\n> > > >\n> > > >\n> > https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n> > \n> > > >\n> > > > Y1 became a black image and was difficult to determine the\n> > coarse focus value.\n> > > >\n> > >\n> >\n> > > Sounds surprising... Could you share the way you are generating the\n> > > images of Y1 and Y2 BTW ?\n> > >\n> > > > Happy new year :)\n> > >\n> > > Best wishes, hoping for a far away Covid next year !\n> > I hope too :)\n> >\n> > After some checks, I made some mistakes to convert the image to png\n> > file.\n> >\n> > Check out the following URL for the new results.\n> > https://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing\n> > \n> >\n> > But, for the Y2, the result becomes very bright and sometimes the\n> > image will be all white. Are the results right?\n> >\n> > My python code for converting the pixels to tiff is shown below.\n> >\n> > =========\n> > import numpy as np\n> > from scipy import ndimage, misc\n> > import imageio\n> > from PIL import Image\n> > import sys\n> >\n> > rawfile = np.fromfile(sys.argv[1], \"uint16\")\n> > rawfile.shape = (16,16)\n> > new_image = Image.fromarray(rawfile, mode='I;16')\n> > new_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n> > ==========\n> >\n>\n> \n>\n> >\n> > I also did some experiments on grid configuration of AF scene.\n> >\n> > You can see the picture below.\n> >\n> > https://drive.google.com/file/d/1UQc6emy4Qx-TAMqN3Bk-BO7lOOzoPRyq/view?usp=sharing\n> > \n> > https://drive.google.com/file/d/1RC5JmCrLFetw5PEo_hlGnjbJXnplsc6F/view?usp=sharing\n> > \n> >\n> > The first image showed the configuration of the AF grid. The green box\n> > was the range of the AF scene.\n> > The image of Y1 was the same as the AF scene but Y2 was completely white :(.\n> >\n>\n> The image used for AF calculations is the BDS output one, so you should\n> configure your grid based on this.\n>\n> See:\n> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2541\n>\n> > The grid configuration as shown below\n> >\n> > {\n> > .width = 16,\n> > .height = 16,\n> > .block_width_log2 = 3,\n> > .block_height_log2 = 3,\n> > .height_per_slice = 2,\n> > .x_start = 640,\n> > .y_start = 360 | IPU3_UAPI_GRID_Y_START_EN,\n> > .x_end = 0,\n> > .y_end = 0\n> > },\n>\n> This grid configuration is probably not what you really want :-).\n> You are configuring a 128x128 grid, and your BDS output is (fill in the\n> blank please :-)). So you want:\n> x_start = (BdsOutput.width / 2 - 64),\n> y_start = (BdsOutput.height / 2 - 64) | IPU3_UAPI_GRID_Y_START_EN,\n>\n> > Moreover, the x_start seems can't be set to greater than 650 (image\n> > width/2+10). If it is >650 the AF scene will be (10, 2) the default\n> > configuration.\n>\n> From:\n> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n>\n> \"margins of 10 per horizontal direction and 2 per vertical direction\" =>\n> It means the real AF grid is using 10 pixels before your x_start, so it\n> can't be less than 10, and 2 pixels before your y_start, so it can't be\n> less than 2.\n>\n> You can obviously take it into account if you really want to, but you\n> are trying to center the grid, so it should not be an issue.\n>\n> You should set height_per_slice to 1, as it is calculated by the kernel,\n> and I would be interested by this: could it be a limitation related to:\n> https://patchwork.linuxtv.org/project/linux-media/patch/20210916172504.677919-1-jeanmichel.hautbois@ideasonboard.com/\n>\n> Can you try building the kernel with this patch by any chance ?\n\nThanks for the reply. I'll test all of them then share the results\nwith you. Please allow me some time to test them.\n\nOne thing I discovered, the grid coordinate seems start from the\ncenter of the AF grid if x_end and y_end are set to 0. Once I make\nsure the behaviour, I'll let you know.\n\n>\n> Thanks for your feedback !\n> JM\n>\n\n\n--\nBR,\nKate","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DFA1DBE086\n\tfor ;\n\tFri, 14 Jan 2022 10:46:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0A667604F8;\n\tFri, 14 Jan 2022 11:46:42 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AA96E6017D\n\tfor ;\n\tFri, 14 Jan 2022 11:46:39 +0100 (CET)","from mail-lf1-f71.google.com (mail-lf1-f71.google.com\n\t[209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-214-RCVv4BGjNOipe510bWSezA-1; Fri, 14 Jan 2022 05:46:30 -0500","by mail-lf1-f71.google.com with SMTP id\n\tl27-20020a056512333b00b0042fcd89952bso1846590lfe.15\n\tfor ;\n\tFri, 14 Jan 2022 02:46:30 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"CKRte7Uw\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1642157198;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=DSQd/KtV7HJkmYWXcCwUmj0KWOR/efoX2SL2lNRuMjU=;\n\tb=CKRte7UwGXVvxJuvWeivLrMShko9ngSPMFVGzVeq9u/BMdSCwfEES5mzGbHR7+89t/OUC1\n\tIhGip7aCmqGrUTl+E69FFBumJTSigFC/ODe7menKJ9O02zlrf6+wETUSNYZN29Oo5XZSPX\n\t1SPWNRROc76h4v1OG23iYzgselm8JNc=","X-MC-Unique":"RCVv4BGjNOipe510bWSezA-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=DSQd/KtV7HJkmYWXcCwUmj0KWOR/efoX2SL2lNRuMjU=;\n\tb=pNLcfkBCRHNiEPwQ0BR8l28ZBW+PtbPicjQdJ7Iy62hqS/SfHx0nhY3lxrY18+AXvz\n\tRKgeBNRcpu1vnumhe6ntBP3lTKwQbEGuY/xhgNOZyhhbj3kTinwUw0kSSjOOo3lAumsm\n\tdZt1SLhAbMJpLwlve8VNRCf9UYG7VbRny3pv4b3v3VG77dIfeMFQ7icJ3SeiNXkGmlS4\n\tYFtRPIibi84lza1SvO971IHZsA+uyz1u9xtRYL9ZTjBgRQMb33WQYseZfikycwKBjty0\n\tah81r/f77Kj3L+S4pYT/6YPAUVEaiCt3WoeCG3KWa+8o+FHkc6kqG8j8TbaGyGBJHOB4\n\tS3qg==","X-Gm-Message-State":"AOAM533tT+x9OVGEwUXtL2NXpSaXsHdRClaE/MbCV3SnkMOLPbQS6tl1\n\t0K71ECdugoiU9RJR4t9q5hxeCdDyrVE7m01I0eLSukWwLwRQp80CxFCl4nAhQFC9gpdaivF1v+p\n\tucCPojtR+Ztr24qZ+QgC/TQAVRtxYARvQMVgNv0eWlX1j4lYi7w==","X-Received":["by 2002:a05:6512:280a:: with SMTP id\n\tcf10mr3314377lfb.539.1642157188683; \n\tFri, 14 Jan 2022 02:46:28 -0800 (PST)","by 2002:a05:6512:280a:: with SMTP id\n\tcf10mr3314350lfb.539.1642157188066; \n\tFri, 14 Jan 2022 02:46:28 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJyPGc6LR85F5ULBbarZhmGrKXpKwSEx4rV2P5azI4LkGdQZ2UGvUNDhaXB/74MSMfhDOBj6cDwJR+kJDfJ6CxE=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>","In-Reply-To":"<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>","From":"Kate Hsuan ","Date":"Fri, 14 Jan 2022 18:46:17 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22055,"web_url":"https://patchwork.libcamera.org/comment/22055/","msgid":"","date":"2022-01-20T11:01:40","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel,\n\nOn Fri, Jan 14, 2022 at 6:46 PM Kate Hsuan wrote:\n>\n> Hi Jean-Michel,\n>\n> On Thu, Jan 13, 2022 at 8:58 PM Jean-Michel Hautbois\n> wrote:\n> >\n> > Hi Kate,\n> >\n> > Before I forget it again :-) could you please answer in plain-text and\n> > not HTML ;-) ?\n>\n> Ops, my fault. I'll double-check my Gmail setting. Sorry for the trouble.\n>\n>\n> >\n> > On 13/01/2022 12:35, Kate Hsuan wrote:\n> > > Hi Jean-Michel,\n> > >\n> > >\n> >\n> > \n> >\n> > > > >>>>>\n> > > > >>>>>>\n> > > > >>>>>>> +\n> > > > >>>>>>> +/* settings for Auto Focus from the kernel */\n> > > > >>>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n> > > > >>>>>>> + .filter_config = {\n> > > > >>>>>>> + { 0, 0, 0, 0 },\n> > > > >>>>>>> + { 0, 0, 0, 0 },\n> > > > >>>>>>> + { 0, 0, 0, 128 },\n> > > > >>>>>>> + 0,\n> > > > >>>>>>> + { 0, 0, 0, 0 },\n> > > > >>>>>>> + { 0, 0, 0, 0 },\n> > > > >>>>>>> + { 0, 0, 0, 128 },\n> > > > >>>>>>> + 0,\n> > > > >>>>>>> + .y_calc = { 8, 8, 8, 8 },\n> > > > >>>>>>> + .nf = { 0, 7, 0, 7, 0 },\n> > > > >>>>>>> + },\n> > > > >>>>\n> > > > >>>> I would like your thinking here (and perhaps someone else).\n> > > It looks\n> > > > >>>> like the ImgU implements a Gabor filter, but I can't see all the\n> > > > >>>> parameters I would expect from it. Putting it down, I can't\n> > > see with\n> > > > >>>> this default filter how y1 and y2 could represent low-pass\n> > > and high\n> > > > >>>> pass, as the filters are the same. But, it could be a\n> > > horizontal and a\n> > > > >>>> vertical filter instead, with y1 the horizontal and y2 the\n> > > vertical\n> > > > >>>> resulting filtered values.\n> > > > >>>>\n> > > > >>>> It would be very interesting to display the y1_avg and\n> > > y2_avg values as\n> > > > >>>> an image, I just don't have lots of time for it. We have 16\n> > > bits values,\n> > > > >>>> so we could dump those into a binary file and use python to\n> > > display the\n> > > > >>>> values as a raw gray image or something similar.\n> > > > >>>\n> > > > >>> You could check out the URL below to see the grey scale image\n> > > of the AF buffer.\n> > > > >>>\n> > > https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> > > \n> > > > >>>\n> > > > >>> Since the result is a tiny 16x16 image, all the detail of the\n> > > original\n> > > > >>> image was compressed into a black block.\n> > > > >>>\n> > > > >>\n> > > > >> Nice :-).\n> > > > >> As expected, as y1 and y2 are the same filter, images are the\n> > > same.\n> > > > >> Could you test those values ?\n> > > > >>\n> > > > >> y1 =>\n> > > > >> (0, 1, 3, 7\n> > > > >> 11, 13, 1, 2\n> > > > >> 8, 19, 34, 242)\n> > > > >> vector: 7fdffbfe - normalization factor: 9\n> > > > >> y2 =>\n> > > > >> (0, 1, 6, 6\n> > > > >> 13, 25, 3, 0\n> > > > >> 25, 3, 117, 254)\n> > > > >> vector: 4e53ca72 - normalization factor: 9\n> > > > >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n> > > > >>\n> > > > >\n> > > > > Please check out the results from the URL below.\n> > > > >\n> > > > >\n> > > https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n> > > \n> > > > >\n> > > > > Y1 became a black image and was difficult to determine the\n> > > coarse focus value.\n> > > > >\n> > > >\n> > >\n> > > > Sounds surprising... Could you share the way you are generating the\n> > > > images of Y1 and Y2 BTW ?\n> > > >\n> > > > > Happy new year :)\n> > > >\n> > > > Best wishes, hoping for a far away Covid next year !\n> > > I hope too :)\n> > >\n> > > After some checks, I made some mistakes to convert the image to png\n> > > file.\n> > >\n> > > Check out the following URL for the new results.\n> > > https://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing\n> > > \n> > >\n> > > But, for the Y2, the result becomes very bright and sometimes the\n> > > image will be all white. Are the results right?\n> > >\n> > > My python code for converting the pixels to tiff is shown below.\n> > >\n> > > =========\n> > > import numpy as np\n> > > from scipy import ndimage, misc\n> > > import imageio\n> > > from PIL import Image\n> > > import sys\n> > >\n> > > rawfile = np.fromfile(sys.argv[1], \"uint16\")\n> > > rawfile.shape = (16,16)\n> > > new_image = Image.fromarray(rawfile, mode='I;16')\n> > > new_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n> > > ==========\n> > >\n> >\n> > \n> >\n> > >\n> > > I also did some experiments on grid configuration of AF scene.\n> > >\n> > > You can see the picture below.\n> > >\n> > > https://drive.google.com/file/d/1UQc6emy4Qx-TAMqN3Bk-BO7lOOzoPRyq/view?usp=sharing\n> > > \n> > > https://drive.google.com/file/d/1RC5JmCrLFetw5PEo_hlGnjbJXnplsc6F/view?usp=sharing\n> > > \n> > >\n> > > The first image showed the configuration of the AF grid. The green box\n> > > was the range of the AF scene.\n> > > The image of Y1 was the same as the AF scene but Y2 was completely white :(.\n> > >\n> >\n> > The image used for AF calculations is the BDS output one, so you should\n> > configure your grid based on this.\n> >\n> > See:\n> > https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2541\n> >\n> > > The grid configuration as shown below\n> > >\n> > > {\n> > > .width = 16,\n> > > .height = 16,\n> > > .block_width_log2 = 3,\n> > > .block_height_log2 = 3,\n> > > .height_per_slice = 2,\n> > > .x_start = 640,\n> > > .y_start = 360 | IPU3_UAPI_GRID_Y_START_EN,\n> > > .x_end = 0,\n> > > .y_end = 0\n> > > },\n> >\n> > This grid configuration is probably not what you really want :-).\n> > You are configuring a 128x128 grid, and your BDS output is (fill in the\n> > blank please :-)). So you want:\n> > x_start = (BdsOutput.width / 2 - 64),\n> > y_start = (BdsOutput.height / 2 - 64) | IPU3_UAPI_GRID_Y_START_EN,\n> >\n> > > Moreover, the x_start seems can't be set to greater than 650 (image\n> > > width/2+10). If it is >650 the AF scene will be (10, 2) the default\n> > > configuration.\n> >\n> > From:\n> > https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n> >\n> > \"margins of 10 per horizontal direction and 2 per vertical direction\" =>\n> > It means the real AF grid is using 10 pixels before your x_start, so it\n> > can't be less than 10, and 2 pixels before your y_start, so it can't be\n> > less than 2.\n> >\n> > You can obviously take it into account if you really want to, but you\n> > are trying to center the grid, so it should not be an issue.\n> >\n> > You should set height_per_slice to 1, as it is calculated by the kernel,\n> > and I would be interested by this: could it be a limitation related to:\n> > https://patchwork.linuxtv.org/project/linux-media/patch/20210916172504.677919-1-jeanmichel.hautbois@ideasonboard.com/\n> >\n> > Can you try building the kernel with this patch by any chance ?\n\nI tried the patch. It did not work.\nx_start still addressed to the incorrect location of the image\n(rightmost of the sensor) if the block size is 3.\nAfter some testing, if the width block size was set to 4, it seems to\nwork but it made the AF scene wider.\n\n>\n> Thanks for the reply. I'll test all of them then share the results\n> with you. Please allow me some time to test them.\n>\n> One thing I discovered, the grid coordinate seems start from the\n> center of the AF grid if x_end and y_end are set to 0. Once I make\n> sure the behaviour, I'll let you know.\n\n>\n> >\n> > Thanks for your feedback !\n> > JM\n> >\n>\n>\n> --\n> BR,\n> Kate","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 298E9BF415\n\tfor ;\n\tThu, 20 Jan 2022 11:02:21 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C532860986;\n\tThu, 20 Jan 2022 12:02:19 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 88F6860215\n\tfor ;\n\tThu, 20 Jan 2022 12:02:18 +0100 (CET)","from mail-lf1-f71.google.com (mail-lf1-f71.google.com\n\t[209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-653-BwiaM-84MZOOPsudMKFxQQ-1; Thu, 20 Jan 2022 06:02:09 -0500","by mail-lf1-f71.google.com with SMTP id\n\tw42-20020a0565120b2a00b00432f6a227e0so3496450lfu.3\n\tfor ;\n\tThu, 20 Jan 2022 03:01:56 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"XcpwULK5\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1642676534;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=8K04d6DtHur653i3SQvLCMDoQRmBKIHpeYJgDZVpPGs=;\n\tb=XcpwULK5xdcLQqphh9sqc7Aiu9l7LHcC5cFpfFjToT80vZlh7ZpfaYRbhT29xl+K2buLQI\n\tvuYJvRi4T8gal1qmc2sqmarLmGJiV8nqslaNISCJ9tMyh90JQyo0IzY3E/laUIBNyyHCUj\n\tRYfgy5D0gnJTP/+NGZd+Ct2zOsTOVTk=","X-MC-Unique":"BwiaM-84MZOOPsudMKFxQQ-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=8K04d6DtHur653i3SQvLCMDoQRmBKIHpeYJgDZVpPGs=;\n\tb=HBOxNAZySJWvgO1trp0/KVOCHF7OL2TFyPQxmUIu3PyuSwAs1i7uTwZ30mcXQuwqUP\n\tKzYuaH92QM8NWTWCQnCpO6sH3oanIL/U18FLznfzPckT+QgEH4gb14ewjqROpqaRqMcb\n\tQUQoIfJuPxueC3NqKewhopJ9xwmzeAyKAai3KchAqLdf41Kdh5aUg2uI2FbDUifEiD4C\n\tFv9uudydIkGxUpMILBp4Tiv9bmkGzzXDOy2ndrcDbAv6hm3URAC7W6doJC237mGwWIif\n\tA64wP7wTVVuHkqGE3D+Fh6eBYSW7SVNE5zM6vzWtLexHG+5QJ7Hut+KRAprf7uiqLcuX\n\tqFYQ==","X-Gm-Message-State":"AOAM533mpXCgM08Zi3LUBJyym6aEUSR5zSoEjurYkL/26k1eMVpsqxv4\n\t1gUQEZFUIUHUJDCuH+rui4nAqk4xtpreg6psQMTQtwifQjxFJPmDKyOFaogT2bgVUKUqe2wKCFq\n\t757L4sO389aEhR9GoYT03m6vKXu3LZXB7WjjVkerhcO1HmSMivA==","X-Received":["by 2002:a05:6512:280a:: with SMTP id\n\tcf10mr27790279lfb.539.1642676512572; \n\tThu, 20 Jan 2022 03:01:52 -0800 (PST)","by 2002:a05:6512:280a:: with SMTP id\n\tcf10mr27790231lfb.539.1642676511865; \n\tThu, 20 Jan 2022 03:01:51 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJxKSZGUsPD6Fu1a1I8Wd7mZMrnBMd5JwVSpHtwKTbkwf9eEKPA/sA/zfSTppy6myyQjavYvd0+XJRuWaBTbCAI=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>\n\t","In-Reply-To":"","From":"Kate Hsuan ","Date":"Thu, 20 Jan 2022 19:01:40 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22056,"web_url":"https://patchwork.libcamera.org/comment/22056/","msgid":"<9304fe2f-d615-3273-4fbb-635a91aba712@ideasonboard.com>","date":"2022-01-20T11:15:33","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kate,\n\nOn 20/01/2022 12:01, Kate Hsuan wrote:\n> Hi Jean-Michel,\n> \n> On Fri, Jan 14, 2022 at 6:46 PM Kate Hsuan wrote:\n>>\n>> Hi Jean-Michel,\n>>\n>> On Thu, Jan 13, 2022 at 8:58 PM Jean-Michel Hautbois\n>> wrote:\n>>>\n>>> Hi Kate,\n>>>\n>>> Before I forget it again :-) could you please answer in plain-text and\n>>> not HTML ;-) ?\n>>\n>> Ops, my fault. I'll double-check my Gmail setting. Sorry for the trouble.\n>>\n>>\n>>>\n>>> On 13/01/2022 12:35, Kate Hsuan wrote:\n>>>> Hi Jean-Michel,\n>>>>\n>>>>\n>>>\n>>> \n>>>\n>>>> > >>>>>\n>>>> > >>>>>>\n>>>> > >>>>>>> +\n>>>> > >>>>>>> +/* settings for Auto Focus from the kernel */\n>>>> > >>>>>>> +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = {\n>>>> > >>>>>>> + .filter_config = {\n>>>> > >>>>>>> + { 0, 0, 0, 0 },\n>>>> > >>>>>>> + { 0, 0, 0, 0 },\n>>>> > >>>>>>> + { 0, 0, 0, 128 },\n>>>> > >>>>>>> + 0,\n>>>> > >>>>>>> + { 0, 0, 0, 0 },\n>>>> > >>>>>>> + { 0, 0, 0, 0 },\n>>>> > >>>>>>> + { 0, 0, 0, 128 },\n>>>> > >>>>>>> + 0,\n>>>> > >>>>>>> + .y_calc = { 8, 8, 8, 8 },\n>>>> > >>>>>>> + .nf = { 0, 7, 0, 7, 0 },\n>>>> > >>>>>>> + },\n>>>> > >>>>\n>>>> > >>>> I would like your thinking here (and perhaps someone else).\n>>>> It looks\n>>>> > >>>> like the ImgU implements a Gabor filter, but I can't see all the\n>>>> > >>>> parameters I would expect from it. Putting it down, I can't\n>>>> see with\n>>>> > >>>> this default filter how y1 and y2 could represent low-pass\n>>>> and high\n>>>> > >>>> pass, as the filters are the same. But, it could be a\n>>>> horizontal and a\n>>>> > >>>> vertical filter instead, with y1 the horizontal and y2 the\n>>>> vertical\n>>>> > >>>> resulting filtered values.\n>>>> > >>>>\n>>>> > >>>> It would be very interesting to display the y1_avg and\n>>>> y2_avg values as\n>>>> > >>>> an image, I just don't have lots of time for it. We have 16\n>>>> bits values,\n>>>> > >>>> so we could dump those into a binary file and use python to\n>>>> display the\n>>>> > >>>> values as a raw gray image or something similar.\n>>>> > >>>\n>>>> > >>> You could check out the URL below to see the grey scale image\n>>>> of the AF buffer.\n>>>> > >>>\n>>>> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n>>>> \n>>>> > >>>\n>>>> > >>> Since the result is a tiny 16x16 image, all the detail of the\n>>>> original\n>>>> > >>> image was compressed into a black block.\n>>>> > >>>\n>>>> > >>\n>>>> > >> Nice :-).\n>>>> > >> As expected, as y1 and y2 are the same filter, images are the\n>>>> same.\n>>>> > >> Could you test those values ?\n>>>> > >>\n>>>> > >> y1 =>\n>>>> > >> (0, 1, 3, 7\n>>>> > >> 11, 13, 1, 2\n>>>> > >> 8, 19, 34, 242)\n>>>> > >> vector: 7fdffbfe - normalization factor: 9\n>>>> > >> y2 =>\n>>>> > >> (0, 1, 6, 6\n>>>> > >> 13, 25, 3, 0\n>>>> > >> 25, 3, 117, 254)\n>>>> > >> vector: 4e53ca72 - normalization factor: 9\n>>>> > >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n>>>> > >>\n>>>> > >\n>>>> > > Please check out the results from the URL below.\n>>>> > >\n>>>> > >\n>>>> https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n>>>> \n>>>> > >\n>>>> > > Y1 became a black image and was difficult to determine the\n>>>> coarse focus value.\n>>>> > >\n>>>> >\n>>>>\n>>>> > Sounds surprising... Could you share the way you are generating the\n>>>> > images of Y1 and Y2 BTW ?\n>>>> >\n>>>> > > Happy new year :)\n>>>> >\n>>>> > Best wishes, hoping for a far away Covid next year !\n>>>> I hope too :)\n>>>>\n>>>> After some checks, I made some mistakes to convert the image to png\n>>>> file.\n>>>>\n>>>> Check out the following URL for the new results.\n>>>> https://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing\n>>>> \n>>>>\n>>>> But, for the Y2, the result becomes very bright and sometimes the\n>>>> image will be all white. Are the results right?\n>>>>\n>>>> My python code for converting the pixels to tiff is shown below.\n>>>>\n>>>> =========\n>>>> import numpy as np\n>>>> from scipy import ndimage, misc\n>>>> import imageio\n>>>> from PIL import Image\n>>>> import sys\n>>>>\n>>>> rawfile = np.fromfile(sys.argv[1], \"uint16\")\n>>>> rawfile.shape = (16,16)\n>>>> new_image = Image.fromarray(rawfile, mode='I;16')\n>>>> new_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n>>>> ==========\n>>>>\n>>>\n>>> \n>>>\n>>>>\n>>>> I also did some experiments on grid configuration of AF scene.\n>>>>\n>>>> You can see the picture below.\n>>>>\n>>>> https://drive.google.com/file/d/1UQc6emy4Qx-TAMqN3Bk-BO7lOOzoPRyq/view?usp=sharing\n>>>> \n>>>> https://drive.google.com/file/d/1RC5JmCrLFetw5PEo_hlGnjbJXnplsc6F/view?usp=sharing\n>>>> \n>>>>\n>>>> The first image showed the configuration of the AF grid. The green box\n>>>> was the range of the AF scene.\n>>>> The image of Y1 was the same as the AF scene but Y2 was completely white :(.\n>>>>\n>>>\n>>> The image used for AF calculations is the BDS output one, so you should\n>>> configure your grid based on this.\n>>>\n>>> See:\n>>> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2541\n>>>\n>>>> The grid configuration as shown below\n>>>>\n>>>> {\n>>>> .width = 16,\n>>>> .height = 16,\n>>>> .block_width_log2 = 3,\n>>>> .block_height_log2 = 3,\n>>>> .height_per_slice = 2,\n>>>> .x_start = 640,\n>>>> .y_start = 360 | IPU3_UAPI_GRID_Y_START_EN,\n>>>> .x_end = 0,\n>>>> .y_end = 0\n>>>> },\n>>>\n>>> This grid configuration is probably not what you really want :-).\n>>> You are configuring a 128x128 grid, and your BDS output is (fill in the\n>>> blank please :-)). So you want:\n>>> x_start = (BdsOutput.width / 2 - 64),\n>>> y_start = (BdsOutput.height / 2 - 64) | IPU3_UAPI_GRID_Y_START_EN,\n>>>\n>>>> Moreover, the x_start seems can't be set to greater than 650 (image\n>>>> width/2+10). If it is >650 the AF scene will be (10, 2) the default\n>>>> configuration.\n>>>\n>>> From:\n>>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n>>>\n>>> \"margins of 10 per horizontal direction and 2 per vertical direction\" =>\n>>> It means the real AF grid is using 10 pixels before your x_start, so it\n>>> can't be less than 10, and 2 pixels before your y_start, so it can't be\n>>> less than 2.\n>>>\n>>> You can obviously take it into account if you really want to, but you\n>>> are trying to center the grid, so it should not be an issue.\n>>>\n>>> You should set height_per_slice to 1, as it is calculated by the kernel,\n>>> and I would be interested by this: could it be a limitation related to:\n>>> https://patchwork.linuxtv.org/project/linux-media/patch/20210916172504.677919-1-jeanmichel.hautbois@ideasonboard.com/\n>>>\n>>> Can you try building the kernel with this patch by any chance ?\n> \n> I tried the patch. It did not work.\n> x_start still addressed to the incorrect location of the image\n> (rightmost of the sensor) if the block size is 3.\n> After some testing, if the width block size was set to 4, it seems to\n> work but it made the AF scene wider.\n\nThanks for testing !\nI think you are in this case:\nhttps://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2394\n\nadding some printk() could help understand where you are exactly, and \nfind out the values really used by the ISP.\nSetting block width to 4 means you now have a 256 pixels window in \nwidth, so, your x_start is below (bdsOutput.width/2+10). Then you go in \nthe \"both stripes\" case I suppose. Or did you change the .width field to 8 ?\n\n> \n>>\n>> Thanks for the reply. I'll test all of them then share the results\n>> with you. Please allow me some time to test them.\n>>\n>> One thing I discovered, the grid coordinate seems start from the\n>> center of the AF grid if x_end and y_end are set to 0. Once I make\n>> sure the behaviour, I'll let you know.\n> \n>>\n>>>\n>>> Thanks for your feedback !\n>>> JM\n>>>\n>>\n>>\n>> --\n>> BR,\n>> Kate\n> \n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4DC5FBE086\n\tfor ;\n\tThu, 20 Jan 2022 11:15:39 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A52A66095E;\n\tThu, 20 Jan 2022 12:15:38 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C18F460215\n\tfor ;\n\tThu, 20 Jan 2022 12:15:36 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:5abc:ed67:26d0:8614] (unknown\n\t[IPv6:2a01:e0a:169:7140:5abc:ed67:26d0:8614])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 6804F4FB;\n\tThu, 20 Jan 2022 12:15:36 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"VY1NY3/J\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1642677336;\n\tbh=QrWa3g6GBFxMBvNc85FSvTJZVIq3jx5eTH07rU6V3/0=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=VY1NY3/JChjrV6A0HbikOSkCjf2zjuhkcCQoOlbbu22lZIxjWMTFBqa/4YpM5TV3Z\n\tsIv54Iv7u8Tq/CLaU8hYexSZVBac/AnDZwr4uZkvlqmpg60cjFsPDOgRDr/kqNz1GV\n\tfkbuKxi+Mc0H310Nu/2iAx2YFHOj5sKRonqeQRbA=","Message-ID":"<9304fe2f-d615-3273-4fbb-635a91aba712@ideasonboard.com>","Date":"Thu, 20 Jan 2022 12:15:33 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.3.1","Content-Language":"en-US","To":"Kate Hsuan ","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>\n\t\n\t","From":"Jean-Michel Hautbois ","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22057,"web_url":"https://patchwork.libcamera.org/comment/22057/","msgid":"<7da4ac05-aa8f-34cf-0177-c060cec55dbd@ideasonboard.com>","date":"2022-01-20T11:20:36","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"On 20/01/2022 12:15, Jean-Michel Hautbois wrote:\n> Hi Kate,\n> \n> On 20/01/2022 12:01, Kate Hsuan wrote:\n>> Hi Jean-Michel,\n>>\n>> On Fri, Jan 14, 2022 at 6:46 PM Kate Hsuan wrote:\n>>>\n>>> Hi Jean-Michel,\n>>>\n>>> On Thu, Jan 13, 2022 at 8:58 PM Jean-Michel Hautbois\n>>> wrote:\n>>>>\n>>>> Hi Kate,\n>>>>\n>>>> Before I forget it again :-) could you please answer in plain-text and\n>>>> not HTML ;-) ?\n>>>\n>>> Ops, my fault. I'll double-check my Gmail setting. Sorry for the \n>>> trouble.\n>>>\n>>>\n>>>>\n>>>> On 13/01/2022 12:35, Kate Hsuan wrote:\n>>>>> Hi Jean-Michel,\n>>>>>\n>>>>>\n>>>>\n>>>> \n>>>>\n>>>>>       > >>>>>\n>>>>>       > >>>>>>\n>>>>>       > >>>>>>> +\n>>>>>       > >>>>>>> +/* settings for Auto Focus from the kernel */\n>>>>>       > >>>>>>> +static struct ipu3_uapi_af_config_s \n>>>>> imgu_css_af_defaults = {\n>>>>>       > >>>>>>> +     .filter_config = {\n>>>>>       > >>>>>>> +             { 0, 0, 0, 0 },\n>>>>>       > >>>>>>> +             { 0, 0, 0, 0 },\n>>>>>       > >>>>>>> +             { 0, 0, 0, 128 },\n>>>>>       > >>>>>>> +             0,\n>>>>>       > >>>>>>> +             { 0, 0, 0, 0 },\n>>>>>       > >>>>>>> +             { 0, 0, 0, 0 },\n>>>>>       > >>>>>>> +             { 0, 0, 0, 128 },\n>>>>>       > >>>>>>> +             0,\n>>>>>       > >>>>>>> +             .y_calc = { 8, 8, 8, 8 },\n>>>>>       > >>>>>>> +             .nf = { 0, 7, 0, 7, 0 },\n>>>>>       > >>>>>>> +     },\n>>>>>       > >>>>\n>>>>>       > >>>> I would like your thinking here (and perhaps someone \n>>>>> else).\n>>>>>      It looks\n>>>>>       > >>>> like the ImgU implements a Gabor filter, but I can't \n>>>>> see all the\n>>>>>       > >>>> parameters I would expect from it. Putting it down, I \n>>>>> can't\n>>>>>      see with\n>>>>>       > >>>> this default filter how y1 and y2 could represent \n>>>>> low-pass\n>>>>>      and high\n>>>>>       > >>>> pass, as the filters are the same. But, it could be a\n>>>>>      horizontal and a\n>>>>>       > >>>> vertical filter instead, with y1 the horizontal and y2 \n>>>>> the\n>>>>>      vertical\n>>>>>       > >>>> resulting filtered values.\n>>>>>       > >>>>\n>>>>>       > >>>> It would be very interesting to display the y1_avg and\n>>>>>      y2_avg values as\n>>>>>       > >>>> an image, I just don't have lots of time for it. We \n>>>>> have 16\n>>>>>      bits values,\n>>>>>       > >>>> so we could dump those into a binary file and use \n>>>>> python to\n>>>>>      display the\n>>>>>       > >>>> values as a raw gray image or something similar.\n>>>>>       > >>>\n>>>>>       > >>> You could check out the URL below to see the grey scale \n>>>>> image\n>>>>>      of the AF buffer.\n>>>>>       > >>>\n>>>>> \n>>>>> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing \n>>>>>\n>>>>> \n>>>>> \n>>>>>\n>>>>>       > >>>\n>>>>>       > >>> Since the result is a tiny 16x16 image, all the detail \n>>>>> of the\n>>>>>      original\n>>>>>       > >>> image was compressed into a black block.\n>>>>>       > >>>\n>>>>>       > >>\n>>>>>       > >> Nice :-).\n>>>>>       > >> As expected, as y1 and y2 are the same filter, images \n>>>>> are the\n>>>>>      same.\n>>>>>       > >> Could you test those values ?\n>>>>>       > >>\n>>>>>       > >> y1 =>\n>>>>>       > >> (0, 1, 3, 7\n>>>>>       > >>    11, 13, 1, 2\n>>>>>       > >>    8, 19, 34, 242)\n>>>>>       > >>    vector: 7fdffbfe - normalization factor: 9\n>>>>>       > >> y2 =>\n>>>>>       > >> (0, 1, 6, 6\n>>>>>       > >>    13, 25, 3, 0\n>>>>>       > >>    25, 3, 117, 254)\n>>>>>       > >>    vector: 4e53ca72 - normalization factor: 9\n>>>>>       > >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n>>>>>       > >>\n>>>>>       > >\n>>>>>       > > Please check out the results from the URL below.\n>>>>>       > >\n>>>>>       > >\n>>>>> \n>>>>> https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing \n>>>>>\n>>>>> \n>>>>> \n>>>>>\n>>>>>       > >\n>>>>>       > > Y1 became a black image and was difficult to determine the\n>>>>>      coarse focus value.\n>>>>>       > >\n>>>>>       >\n>>>>>\n>>>>>       > Sounds surprising... Could you share the way you are \n>>>>> generating the\n>>>>>       > images of Y1 and Y2 BTW ?\n>>>>>       >\n>>>>>       > > Happy new year :)\n>>>>>       >\n>>>>>       > Best wishes, hoping for a far away Covid next year !\n>>>>>      I hope too :)\n>>>>>\n>>>>>      After some checks, I made some mistakes to convert the image \n>>>>> to png\n>>>>>      file.\n>>>>>\n>>>>>      Check out the following URL for the new results.\n>>>>> \n>>>>> https://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing \n>>>>>\n>>>>> \n>>>>> \n>>>>>\n>>>>>\n>>>>>      But, for the Y2, the result becomes very bright and sometimes the\n>>>>>      image will be all white. Are the results right?\n>>>>>\n>>>>>      My python code for converting the pixels to tiff is shown below.\n>>>>>\n>>>>>      =========\n>>>>>      import numpy as np\n>>>>>      from scipy import ndimage, misc\n>>>>>      import imageio\n>>>>>      from PIL import Image\n>>>>>      import sys\n>>>>>\n>>>>>      rawfile = np.fromfile(sys.argv[1], \"uint16\")\n>>>>>      rawfile.shape = (16,16)\n>>>>>      new_image = Image.fromarray(rawfile, mode='I;16')\n>>>>>      new_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n>>>>>      ==========\n>>>>>\n>>>>\n>>>> \n>>>>\n>>>>>\n>>>>> I also did some experiments on grid configuration of AF scene.\n>>>>>\n>>>>> You can see the picture below.\n>>>>>\n>>>>> https://drive.google.com/file/d/1UQc6emy4Qx-TAMqN3Bk-BO7lOOzoPRyq/view?usp=sharing \n>>>>>\n>>>>> \n>>>>>\n>>>>> https://drive.google.com/file/d/1RC5JmCrLFetw5PEo_hlGnjbJXnplsc6F/view?usp=sharing \n>>>>>\n>>>>> \n>>>>>\n>>>>>\n>>>>> The first image showed the configuration of the AF grid. The green box\n>>>>> was the range of the AF scene.\n>>>>> The image of Y1 was the same as the AF scene but Y2 was completely \n>>>>> white :(.\n>>>>>\n>>>>\n>>>> The image used for AF calculations is the BDS output one, so you should\n>>>> configure your grid based on this.\n>>>>\n>>>> See:\n>>>> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2541 \n>>>>\n>>>>\n>>>>> The grid configuration as shown below\n>>>>>\n>>>>> {\n>>>>> .width = 16,\n>>>>> .height = 16,\n>>>>> .block_width_log2 = 3,\n>>>>> .block_height_log2 = 3,\n>>>>> .height_per_slice = 2,\n>>>>> .x_start = 640,\n>>>>> .y_start = 360 | IPU3_UAPI_GRID_Y_START_EN,\n>>>>> .x_end = 0,\n>>>>> .y_end = 0\n>>>>> },\n>>>>\n>>>> This grid configuration is probably not what you really want :-).\n>>>> You are configuring a 128x128 grid, and your BDS output is (fill in the\n>>>> blank please :-)). So you want:\n>>>> x_start = (BdsOutput.width / 2 - 64),\n>>>> y_start = (BdsOutput.height / 2 - 64) | IPU3_UAPI_GRID_Y_START_EN,\n>>>>\n>>>>> Moreover, the x_start seems can't be set to greater than 650 (image\n>>>>> width/2+10). If it is >650 the AF scene will be (10, 2) the default\n>>>>> configuration.\n>>>>\n>>>> From:\n>>>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74 \n>>>>\n>>>>\n>>>> \"margins of 10 per horizontal direction and 2 per vertical \n>>>> direction\" =>\n>>>> It means the real AF grid is using 10 pixels before your x_start, so it\n>>>> can't be less than 10, and 2 pixels before your y_start, so it can't be\n>>>> less than 2.\n>>>>\n>>>> You can obviously take it into account if you really want to, but you\n>>>> are trying to center the grid, so it should not be an issue.\n>>>>\n>>>> You should set height_per_slice to 1, as it is calculated by the \n>>>> kernel,\n>>>> and I would be interested by this: could it be a limitation related to:\n>>>> https://patchwork.linuxtv.org/project/linux-media/patch/20210916172504.677919-1-jeanmichel.hautbois@ideasonboard.com/ \n>>>>\n>>>>\n>>>> Can you try building the kernel with this patch by any chance ?\n>>\n>> I tried the patch. It did not work.\n>> x_start still addressed to the incorrect location of the image\n>> (rightmost of the sensor) if the block size is 3.\n>> After some testing, if the width block size was set to 4, it seems to\n>> work but it made the AF scene wider.\n> \n> Thanks for testing !\n> I think you are in this case:\n> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2394 \n\nNot the right path, sorry :-( :\nhttps://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2557\n\n> \n> \n> adding some printk() could help understand where you are exactly, and \n> find out the values really used by the ISP.\n> Setting block width to 4 means you now have a 256 pixels window in \n> width, so, your x_start is below (bdsOutput.width/2+10). Then you go in \n> the \"both stripes\" case I suppose. Or did you change the .width field to \n> 8 ?\n> \n>>\n>>>\n>>> Thanks for the reply. I'll test all of them then share the results\n>>> with you. Please allow me some time to test them.\n>>>\n>>> One thing I discovered, the grid coordinate seems start from the\n>>> center of the AF grid if x_end and y_end are set to 0. Once I make\n>>> sure the behaviour, I'll let you know.\n>>\n>>>\n>>>>\n>>>> Thanks for your feedback !\n>>>> JM\n>>>>\n>>>\n>>>\n>>> -- \n>>> BR,\n>>> Kate\n>>\n>>\n>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 29C47BF415\n\tfor ;\n\tThu, 20 Jan 2022 11:20:40 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6C8A160997;\n\tThu, 20 Jan 2022 12:20:39 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 47A6F60215\n\tfor ;\n\tThu, 20 Jan 2022 12:20:38 +0100 (CET)","from [IPV6:2a01:e0a:169:7140:5abc:ed67:26d0:8614] (unknown\n\t[IPv6:2a01:e0a:169:7140:5abc:ed67:26d0:8614])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DD6654FB;\n\tThu, 20 Jan 2022 12:20:37 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Mpqyw7cE\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1642677637;\n\tbh=JtysbgvdbqzNV5eU69Rv95xs5eUVrQrPt5WIgiz87XU=;\n\th=Date:Subject:From:To:Cc:References:In-Reply-To:From;\n\tb=Mpqyw7cE0OD509z+oP8WCB+hxzJ4N3yAl3jZghhekBuXLlz38YqMq6rETPeNTpO2M\n\thn2MsdVUVsuAGrWEllP1WQtCdaOwhmPUPYI1SVC7qNjgOgoi913MT6uhEjYXKcK+j3\n\tSU+R9D35nCGxlJwVye5xg/P6jJHg7Xw62yadUA1U=","Message-ID":"<7da4ac05-aa8f-34cf-0177-c060cec55dbd@ideasonboard.com>","Date":"Thu, 20 Jan 2022 12:20:36 +0100","MIME-Version":"1.0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101\n\tThunderbird/91.3.1","Content-Language":"en-US","From":"Jean-Michel Hautbois ","To":"Kate Hsuan ","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>\n\t\n\t\n\t<9304fe2f-d615-3273-4fbb-635a91aba712@ideasonboard.com>","In-Reply-To":"<9304fe2f-d615-3273-4fbb-635a91aba712@ideasonboard.com>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":22058,"web_url":"https://patchwork.libcamera.org/comment/22058/","msgid":"","date":"2022-01-20T11:38:49","subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","submitter":{"id":105,"url":"https://patchwork.libcamera.org/api/people/105/","name":"Kate Hsuan","email":"hpa@redhat.com"},"content":"Hi Jean-Michel,\n\nOn Thu, Jan 20, 2022 at 7:20 PM Jean-Michel Hautbois\n wrote:\n>\n> On 20/01/2022 12:15, Jean-Michel Hautbois wrote:\n> > Hi Kate,\n> >\n> > On 20/01/2022 12:01, Kate Hsuan wrote:\n> >> Hi Jean-Michel,\n> >>\n> >> On Fri, Jan 14, 2022 at 6:46 PM Kate Hsuan wrote:\n> >>>\n> >>> Hi Jean-Michel,\n> >>>\n> >>> On Thu, Jan 13, 2022 at 8:58 PM Jean-Michel Hautbois\n> >>> wrote:\n> >>>>\n> >>>> Hi Kate,\n> >>>>\n> >>>> Before I forget it again :-) could you please answer in plain-text and\n> >>>> not HTML ;-) ?\n> >>>\n> >>> Ops, my fault. I'll double-check my Gmail setting. Sorry for the\n> >>> trouble.\n> >>>\n> >>>\n> >>>>\n> >>>> On 13/01/2022 12:35, Kate Hsuan wrote:\n> >>>>> Hi Jean-Michel,\n> >>>>>\n> >>>>>\n> >>>>\n> >>>> \n> >>>>\n> >>>>> > >>>>>\n> >>>>> > >>>>>>\n> >>>>> > >>>>>>> +\n> >>>>> > >>>>>>> +/* settings for Auto Focus from the kernel */\n> >>>>> > >>>>>>> +static struct ipu3_uapi_af_config_s\n> >>>>> imgu_css_af_defaults = {\n> >>>>> > >>>>>>> + .filter_config = {\n> >>>>> > >>>>>>> + { 0, 0, 0, 0 },\n> >>>>> > >>>>>>> + { 0, 0, 0, 0 },\n> >>>>> > >>>>>>> + { 0, 0, 0, 128 },\n> >>>>> > >>>>>>> + 0,\n> >>>>> > >>>>>>> + { 0, 0, 0, 0 },\n> >>>>> > >>>>>>> + { 0, 0, 0, 0 },\n> >>>>> > >>>>>>> + { 0, 0, 0, 128 },\n> >>>>> > >>>>>>> + 0,\n> >>>>> > >>>>>>> + .y_calc = { 8, 8, 8, 8 },\n> >>>>> > >>>>>>> + .nf = { 0, 7, 0, 7, 0 },\n> >>>>> > >>>>>>> + },\n> >>>>> > >>>>\n> >>>>> > >>>> I would like your thinking here (and perhaps someone\n> >>>>> else).\n> >>>>> It looks\n> >>>>> > >>>> like the ImgU implements a Gabor filter, but I can't\n> >>>>> see all the\n> >>>>> > >>>> parameters I would expect from it. Putting it down, I\n> >>>>> can't\n> >>>>> see with\n> >>>>> > >>>> this default filter how y1 and y2 could represent\n> >>>>> low-pass\n> >>>>> and high\n> >>>>> > >>>> pass, as the filters are the same. But, it could be a\n> >>>>> horizontal and a\n> >>>>> > >>>> vertical filter instead, with y1 the horizontal and y2\n> >>>>> the\n> >>>>> vertical\n> >>>>> > >>>> resulting filtered values.\n> >>>>> > >>>>\n> >>>>> > >>>> It would be very interesting to display the y1_avg and\n> >>>>> y2_avg values as\n> >>>>> > >>>> an image, I just don't have lots of time for it. We\n> >>>>> have 16\n> >>>>> bits values,\n> >>>>> > >>>> so we could dump those into a binary file and use\n> >>>>> python to\n> >>>>> display the\n> >>>>> > >>>> values as a raw gray image or something similar.\n> >>>>> > >>>\n> >>>>> > >>> You could check out the URL below to see the grey scale\n> >>>>> image\n> >>>>> of the AF buffer.\n> >>>>> > >>>\n> >>>>>\n> >>>>> https://drive.google.com/file/d/15rrfeRKA1hgGngzRRk-a9Uzhf8tYEU1T/view?usp=sharing\n> >>>>>\n> >>>>>\n> >>>>> \n> >>>>>\n> >>>>> > >>>\n> >>>>> > >>> Since the result is a tiny 16x16 image, all the detail\n> >>>>> of the\n> >>>>> original\n> >>>>> > >>> image was compressed into a black block.\n> >>>>> > >>>\n> >>>>> > >>\n> >>>>> > >> Nice :-).\n> >>>>> > >> As expected, as y1 and y2 are the same filter, images\n> >>>>> are the\n> >>>>> same.\n> >>>>> > >> Could you test those values ?\n> >>>>> > >>\n> >>>>> > >> y1 =>\n> >>>>> > >> (0, 1, 3, 7\n> >>>>> > >> 11, 13, 1, 2\n> >>>>> > >> 8, 19, 34, 242)\n> >>>>> > >> vector: 7fdffbfe - normalization factor: 9\n> >>>>> > >> y2 =>\n> >>>>> > >> (0, 1, 6, 6\n> >>>>> > >> 13, 25, 3, 0\n> >>>>> > >> 25, 3, 117, 254)\n> >>>>> > >> vector: 4e53ca72 - normalization factor: 9\n> >>>>> > >> Channels coefficients: (gr: 8, r: 8, gb: 8, b: 8)\n> >>>>> > >>\n> >>>>> > >\n> >>>>> > > Please check out the results from the URL below.\n> >>>>> > >\n> >>>>> > >\n> >>>>>\n> >>>>> https://drive.google.com/file/d/1jePzgsw0zwO0EtDnLBrOaCKucx4QWLIg/view?usp=sharing\n> >>>>>\n> >>>>>\n> >>>>> \n> >>>>>\n> >>>>> > >\n> >>>>> > > Y1 became a black image and was difficult to determine the\n> >>>>> coarse focus value.\n> >>>>> > >\n> >>>>> >\n> >>>>>\n> >>>>> > Sounds surprising... Could you share the way you are\n> >>>>> generating the\n> >>>>> > images of Y1 and Y2 BTW ?\n> >>>>> >\n> >>>>> > > Happy new year :)\n> >>>>> >\n> >>>>> > Best wishes, hoping for a far away Covid next year !\n> >>>>> I hope too :)\n> >>>>>\n> >>>>> After some checks, I made some mistakes to convert the image\n> >>>>> to png\n> >>>>> file.\n> >>>>>\n> >>>>> Check out the following URL for the new results.\n> >>>>>\n> >>>>> https://drive.google.com/file/d/1o4EcY9EsYrBYu5YTZ-_tlxP8sevDd3QY/view?usp=sharing\n> >>>>>\n> >>>>>\n> >>>>> \n> >>>>>\n> >>>>>\n> >>>>> But, for the Y2, the result becomes very bright and sometimes the\n> >>>>> image will be all white. Are the results right?\n> >>>>>\n> >>>>> My python code for converting the pixels to tiff is shown below.\n> >>>>>\n> >>>>> =========\n> >>>>> import numpy as np\n> >>>>> from scipy import ndimage, misc\n> >>>>> import imageio\n> >>>>> from PIL import Image\n> >>>>> import sys\n> >>>>>\n> >>>>> rawfile = np.fromfile(sys.argv[1], \"uint16\")\n> >>>>> rawfile.shape = (16,16)\n> >>>>> new_image = Image.fromarray(rawfile, mode='I;16')\n> >>>>> new_image.convert('L').save('{}.tiff'.format(sys.argv[1][:-4]))\n> >>>>> ==========\n> >>>>>\n> >>>>\n> >>>> \n> >>>>\n> >>>>>\n> >>>>> I also did some experiments on grid configuration of AF scene.\n> >>>>>\n> >>>>> You can see the picture below.\n> >>>>>\n> >>>>> https://drive.google.com/file/d/1UQc6emy4Qx-TAMqN3Bk-BO7lOOzoPRyq/view?usp=sharing\n> >>>>>\n> >>>>> \n> >>>>>\n> >>>>> https://drive.google.com/file/d/1RC5JmCrLFetw5PEo_hlGnjbJXnplsc6F/view?usp=sharing\n> >>>>>\n> >>>>> \n> >>>>>\n> >>>>>\n> >>>>> The first image showed the configuration of the AF grid. The green box\n> >>>>> was the range of the AF scene.\n> >>>>> The image of Y1 was the same as the AF scene but Y2 was completely\n> >>>>> white :(.\n> >>>>>\n> >>>>\n> >>>> The image used for AF calculations is the BDS output one, so you should\n> >>>> configure your grid based on this.\n> >>>>\n> >>>> See:\n> >>>> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2541\n> >>>>\n> >>>>\n> >>>>> The grid configuration as shown below\n> >>>>>\n> >>>>> {\n> >>>>> .width = 16,\n> >>>>> .height = 16,\n> >>>>> .block_width_log2 = 3,\n> >>>>> .block_height_log2 = 3,\n> >>>>> .height_per_slice = 2,\n> >>>>> .x_start = 640,\n> >>>>> .y_start = 360 | IPU3_UAPI_GRID_Y_START_EN,\n> >>>>> .x_end = 0,\n> >>>>> .y_end = 0\n> >>>>> },\n> >>>>\n> >>>> This grid configuration is probably not what you really want :-).\n> >>>> You are configuring a 128x128 grid, and your BDS output is (fill in the\n> >>>> blank please :-)). So you want:\n> >>>> x_start = (BdsOutput.width / 2 - 64),\n> >>>> y_start = (BdsOutput.height / 2 - 64) | IPU3_UAPI_GRID_Y_START_EN,\n> >>>>\n> >>>>> Moreover, the x_start seems can't be set to greater than 650 (image\n> >>>>> width/2+10). If it is >650 the AF scene will be (10, 2) the default\n> >>>>> configuration.\n> >>>>\n> >>>> From:\n> >>>> https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/refs/heads/master/hal/intel/include/ia_imaging/af_public.h#74\n> >>>>\n> >>>>\n> >>>> \"margins of 10 per horizontal direction and 2 per vertical\n> >>>> direction\" =>\n> >>>> It means the real AF grid is using 10 pixels before your x_start, so it\n> >>>> can't be less than 10, and 2 pixels before your y_start, so it can't be\n> >>>> less than 2.\n> >>>>\n> >>>> You can obviously take it into account if you really want to, but you\n> >>>> are trying to center the grid, so it should not be an issue.\n> >>>>\n> >>>> You should set height_per_slice to 1, as it is calculated by the\n> >>>> kernel,\n> >>>> and I would be interested by this: could it be a limitation related to:\n> >>>> https://patchwork.linuxtv.org/project/linux-media/patch/20210916172504.677919-1-jeanmichel.hautbois@ideasonboard.com/\n> >>>>\n> >>>>\n> >>>> Can you try building the kernel with this patch by any chance ?\n> >>\n> >> I tried the patch. It did not work.\n> >> x_start still addressed to the incorrect location of the image\n> >> (rightmost of the sensor) if the block size is 3.\n> >> After some testing, if the width block size was set to 4, it seems to\n> >> work but it made the AF scene wider.\n> >\n> > Thanks for testing !\n> > I think you are in this case:\n> > https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2394\n>\n> Not the right path, sorry :-( :\n> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-css-params.c#L2557\n>\nIt's ok :).\nActually, I was tracing this section of codes and I also comment the\ncodes from lines 2557 to 2667 to force enable both stripes but it\ndidn't work. :(\nLater, I'll double-check my modifications and my testing results.\n\n> >\n> >\n> > adding some printk() could help understand where you are exactly, and\n> > find out the values really used by the ISP.\n> > Setting block width to 4 means you now have a 256 pixels window in\n> > width, so, your x_start is below (bdsOutput.width/2+10). Then you go in\n> > the \"both stripes\" case I suppose. Or did you change the .width field to\n> > 8 ?\n\nAccording to the code from chromium,\n#define AF_MIN_GRID_WIDTH 16. <-- the min with is 16.\n#define AF_MIN_GRID_HEIGHT 16\n#define AF_MAX_GRID_WIDTH 32\n#define AF_MAX_GRID_HEIGHT 24\n\nCould we set the .width to 8? If it could be, I'll try it later.\n\nBy the way, I'm preparing my v5 patch which moves the grid\nconfiguration to the context and includes the grid configuration (I\nhope it works). :~\n\nThank you :)\n> >\n> >>\n> >>>\n> >>> Thanks for the reply. I'll test all of them then share the results\n> >>> with you. Please allow me some time to test them.\n> >>>\n> >>> One thing I discovered, the grid coordinate seems start from the\n> >>> center of the AF grid if x_end and y_end are set to 0. Once I make\n> >>> sure the behaviour, I'll let you know.\n> >>\n> >>>\n> >>>>\n> >>>> Thanks for your feedback !\n> >>>> JM\n> >>>>\n> >>>\n> >>>\n> >>> --\n> >>> BR,\n> >>> Kate\n> >>\n> >>\n> >>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4B5FCBE086\n\tfor ;\n\tThu, 20 Jan 2022 11:39:13 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8A75360986;\n\tThu, 20 Jan 2022 12:39:12 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9573C60215\n\tfor ;\n\tThu, 20 Jan 2022 12:39:11 +0100 (CET)","from mail-lf1-f70.google.com (mail-lf1-f70.google.com\n\t[209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\tus-mta-300-7hJX_oW_OqeDaOhLQerT8Q-1; Thu, 20 Jan 2022 06:39:04 -0500","by mail-lf1-f70.google.com with SMTP id\n\tu1-20020ac258c1000000b004304234f41aso3558798lfo.8\n\tfor ;\n\tThu, 20 Jan 2022 03:39:03 -0800 (PST)"],"Authentication-Results":["lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"ZAKgriyb\"; dkim-atps=neutral","relay.mimecast.com;\n\tauth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1642678750;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=eKfKp1gkW0p1XV4HpwE2YX8JnoUa/3Is+IX4bz78xaM=;\n\tb=ZAKgriybn8erMJPpDcz6TNbq/e6So/O1wLr3ropW9WFecZ/P2kIXtL2jeK2pVtrqfy0q+Y\n\tkViJEmHIFRpq1c/8mJOQOS3gdcl2CqZW62Mx/Yxa0eOIcae+zSqyXGj1Jo5gU/NQ7t/A9H\n\t9ZOgf57MURzaZ1nA3ia7nN/+bAGPkVU=","X-MC-Unique":"7hJX_oW_OqeDaOhLQerT8Q-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=eKfKp1gkW0p1XV4HpwE2YX8JnoUa/3Is+IX4bz78xaM=;\n\tb=ookFTirS8oxdAMYU3Chx76gUoZq9Bom/ajjH2BipofIuxBqN0bJX5laUSpHl0X4fcL\n\tpScsACA32gMqKQBKYQxBOPwV+pxJmzsFhUQI5i5bJcdInrq9+DZAT+btz1uqVeJ2qQJ8\n\t+FmD2mfBwpx9B4k7Dgjf8NAc5jOk7te5RhyibxoznE14u1aq3nNWkmGqz/rcWc1wMVC2\n\tyYOvT+WnfmhS2CpR4zMWEHeoOhI41oVBozVBFhFCavybFClxPWG9FPnn2+MHnd58FPKT\n\tOL6fQuIDD7JHAND2Qb5QQw8J8WHiWEpYIQ+qzxIDLHrTQh9MtpFCkpOhEJrAT8IuEJZD\n\tG6IA==","X-Gm-Message-State":"AOAM531P/5qwFtrZnYzizzYYgqgdrI/G5se9ZbHDNd78DRzy8jTAr97u\n\tqsRE+CVxMSIPm8Yf+fgyolXdLPXbjzx8m+2gZf9e0ScCJBKCzaljesYBu4TdC3mh7iJTh09p2jT\n\tIfYF0EbIfUmUNujqS7dr5izT+Xb3QwRKur5vaRI6z5wfdswZUbQ==","X-Received":["by 2002:a2e:9813:: with SMTP id\n\ta19mr29651043ljj.23.1642678741883; \n\tThu, 20 Jan 2022 03:39:01 -0800 (PST)","by 2002:a2e:9813:: with SMTP id\n\ta19mr29651010ljj.23.1642678741266; \n\tThu, 20 Jan 2022 03:39:01 -0800 (PST)"],"X-Google-Smtp-Source":"ABdhPJx+Gkj/JkJwFrHmV/iLSdV96d28gapf3cNb5noDsz/mq920w0Tz279RJAZQD2/ysxiL3OYOQNZi8FdHXQMOb9M=","MIME-Version":"1.0","References":"<20211213122847.26260-1-hpa@redhat.com>\n\t<20211213122847.26260-2-hpa@redhat.com>\n\t<797c06d5-ff6d-fae9-b5ee-b56cbe2ef5e4@ideasonboard.com>\n\t\n\t<28d3f3d0-521d-e14b-72bb-43f5390c2092@ideasonboard.com>\n\t\n\t<61c43229-46e2-9b1d-6bbe-83e655cb2725@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t<726f2f42-65bf-3a92-a4e7-f85dfd36d07d@ideasonboard.com>\n\t\n\t\n\t<9304fe2f-d615-3273-4fbb-635a91aba712@ideasonboard.com>\n\t<7da4ac05-aa8f-34cf-0177-c060cec55dbd@ideasonboard.com>","In-Reply-To":"<7da4ac05-aa8f-34cf-0177-c060cec55dbd@ideasonboard.com>","From":"Kate Hsuan ","Date":"Thu, 20 Jan 2022 19:38:49 +0800","Message-ID":"","To":"Jean-Michel Hautbois ","X-Mimecast-Spam-Score":"0","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain; charset=\"UTF-8\"","Subject":"Re: [libcamera-devel] [RFC v4 1/1] ipa: ipu3: af: Auto focus for\n\tdw9719 Surface Go2 VCM","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]