From patchwork Fri Dec 3 12:06:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kate Hsuan X-Patchwork-Id: 15011 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id F25A4BDB13 for ; Fri, 3 Dec 2021 12:06:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A95EF60839; Fri, 3 Dec 2021 13:06:55 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="HFQOczTS"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0F33360725 for ; Fri, 3 Dec 2021 13:06:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638533213; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EJlz5jG2HVWHeXbwXh34GAfhkrXgJ9qfzqKL59Rv3EE=; b=HFQOczTSzvrcmLCs4+STQ2Ny5pc9vBDiOEPfH+F+eSpMYR17IU7mmuD7p1oxqjEBxhGDR6 bcR+Cx9GZQPDGRjLjB4KdDWMPTLn2ogEIbb7WIOYvZvEmohSeQoiubiYorARuupBdUUUuU q5x2PdChP2vLvd2cg+H3UZaW4SHerIo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-166-lB00HCKNMmSl_WE-m4bKeQ-1; Fri, 03 Dec 2021 07:06:49 -0500 X-MC-Unique: lB00HCKNMmSl_WE-m4bKeQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E4D8518C8C00; Fri, 3 Dec 2021 12:06:47 +0000 (UTC) Received: from fedora.rhtw.internal (unknown [10.39.193.159]) by smtp.corp.redhat.com (Postfix) with ESMTP id E58995C1D5; Fri, 3 Dec 2021 12:06:44 +0000 (UTC) From: Kate Hsuan To: libcamera devel Date: Fri, 3 Dec 2021 20:06:24 +0800 Message-Id: <20211203120624.36657-2-hpa@redhat.com> In-Reply-To: <20211203120624.36657-1-hpa@redhat.com> References: <20211203120624.36657-1-hpa@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=hpa@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [RFC v3 1/1] ipa: ipu3: af: Auto focus for dw9719 Surface Go2 VCM X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Since VCM for surface Go 2 (dw9719) had been successfully driven, this Af module can be used to control the VCM and determine the focus value based on the IPU3 AF state. The variance of each focus step is determined and a greedy approah is used to find the maximum variance of the AF state and a appropriate focus value. Changes in v2: 1. Add grid configuration interface. 2. Add AF accelerator configuration. 3. Move default focus area to the center of the image. Changes in v3: 1. Squash Daniel's commit- Remove v4l2 interaction from AF algorithm. 2. Fix greedy AF algorithm since V4l2 interface had been removed. 3. Simplify default grid config for AF. 4. AF start area is determined by bdsOutputSize. Signed-off-by: Kate Hsuan Signed-off-by: Daniel Scally --- src/ipa/ipu3/algorithms/af.cpp | 239 ++++++++++++++++++++++++++++ src/ipa/ipu3/algorithms/af.h | 53 ++++++ src/ipa/ipu3/algorithms/meson.build | 3 +- src/ipa/ipu3/ipa_context.cpp | 27 ++++ src/ipa/ipu3/ipa_context.h | 11 ++ src/ipa/ipu3/ipu3.cpp | 2 + 6 files changed, 334 insertions(+), 1 deletion(-) create mode 100644 src/ipa/ipu3/algorithms/af.cpp create mode 100644 src/ipa/ipu3/algorithms/af.h diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp new file mode 100644 index 00000000..e7c69706 --- /dev/null +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -0,0 +1,239 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Red Hat + * + * af.cpp - IPU3 auto focus control + */ + +#include "af.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +#include + +#include "libipa/histogram.h" + +/** + * \file af.h + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +namespace ipa::ipu3::algorithms { + +/** + * \class Af + * \brief A IPU3 auto-focus accelerator based auto focus algorthim + * + * This algorithm is used to determine the position of the lens and get a + * focused image. The IPU3 AF accelerator computes the statistics, composed + * by high pass and low pass filtered value and stores in a AF buffer. + * Typically, for a focused image, it has relative high contrast than a + * blurred image, i.e. an out of focus image. Therefore, if an image with the + * highest contrast can be found from the AF scan, the lens' position is the + * best step of the focus. + * + */ + +LOG_DEFINE_CATEGORY(IPU3Af) + +/** + * Maximum focus value of the VCM control + * \todo should be obtained from the VCM driver + */ +static constexpr uint32_t MaxFocusSteps_ = 1023; + +/* minimum focus step for searching appropriate focus*/ +static constexpr uint32_t MinSearchStep_ = 5; + +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/ +static constexpr double MaxChange_ = 0.8; + +/* settings for Auto Focus from the kernel */ +static struct ipu3_uapi_af_config_s imgu_css_af_defaults = { + .filter_config = { + { 0, 0, 0, 0 }, + { 0, 0, 0, 0 }, + { 0, 0, 0, 128 }, + 0, + { 0, 0, 0, 0 }, + { 0, 0, 0, 0 }, + { 0, 0, 0, 128 }, + 0, + .y_calc = { 8, 8, 8, 8 }, + .nf = { 0, 7, 0, 7, 0 }, + }, + .grid_cfg = { + .width = 16, + .height = 16, + .block_width_log2 = 3, + .block_height_log2 = 3, + .x_start = 10, + .y_start = 2 | IPU3_UAPI_GRID_Y_START_EN, + }, +}; + +Af::Af() + : focus_(0), goodFocus_(0), currentVariance_(0.0) +{ +} + +Af::~Af() +{ +} + +void Af::prepare(IPAContext &context, ipu3_uapi_params *params) +{ + params->use.acc_af = 1; + params->acc_param.af = imgu_css_af_defaults; + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x; + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN; +} + +/** + * \brief Configure the Af given a configInfo + * \param[in] context The shared IPA context + * \param[in] configInfo The IPA configuration data + * + * \return 0 + */ +int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo) +{ + /* Determined focus value i.e. current focus value */ + context.frameContext.af.focus = 0; + /* Maximum variance of the AF statistics */ + context.frameContext.af.maxVariance = 0; + /* is focused? if it is true, the AF should be in a stable state. */ + context.frameContext.af.stable = false; + /* Frame to be ignored before start to estimate AF variance. */ + ignoreFrame_ = 10; + + /* + * AF default area configuration + * Move AF area to the center of the image. + */ + /* Default AF width is 16x8 = 128 */ + context.configuration.af.start_x = (configInfo.bdsOutputSize.width / 2) - 64; + context.configuration.af.start_y = (configInfo.bdsOutputSize.height / 2) - 64; + + LOG(IPU3Af, Debug) << "BDS X: " + << configInfo.bdsOutputSize.width + << " Y: " + << configInfo.bdsOutputSize.height; + LOG(IPU3Af, Debug) << "AF start from X: " + << context.configuration.af.start_x + << " Y: " + << context.configuration.af.start_y; + + return 0; +} + +/** + * \brief Determine the max contrast image and lens position. y_table is the + * statictic data from IPU3 and is composed of low pass and high pass filtered + * value. High pass filtered value also represents the sharpness of the image. + * Based on this, if the image with highest variance of the high pass filtered + * value (contrast) during the AF scan, the position of the len should be the + * best focus. + * \param[in] context The shared IPA context. + * \param[in] stats The statistic buffer of 3A from the IPU3. + */ +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +{ + uint32_t total = 0; + double mean; + uint64_t var_sum = 0; + y_table_item_t *y_item; + int z = 0; + + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table; + + /** + * Calculate the mean of each non-zero AF statistics, since IPU3 only determine the AF value + * for a given grid. + */ + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) { + total = total + y_item[z].y2_avg; + if (y_item[z].y2_avg == 0) + break; + } + mean = total / z; + + /* Calculate the variance of every AF statistic value. */ + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) { + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean)); + if (y_item[z].y2_avg == 0) + break; + } + + /* Determine the average variance of the frame. */ + currentVariance_ = static_cast(var_sum) / static_cast(z); + LOG(IPU3Af, Debug) << "variance: " << currentVariance_; + + if (context.frameContext.af.stable == true) { + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance); + const double var_ratio = diff_var / context.frameContext.af.maxVariance; + LOG(IPU3Af, Debug) << "Change ratio: " + << var_ratio + << " current focus: " + << context.frameContext.af.focus; + /** + * If the change ratio of contrast is over Maxchange_ (out of focus), + * trigger AF again. + */ + if (var_ratio > MaxChange_) { + if (ignoreFrame_ == 0) { + context.frameContext.af.maxVariance = 0; + context.frameContext.af.focus = 0; + focus_ = 0; + context.frameContext.af.stable = false; + ignoreFrame_ = 60; + } else + ignoreFrame_--; + } else + ignoreFrame_ = 10; + } else { + if (ignoreFrame_ != 0) + ignoreFrame_--; + else { + /* Find the maximum variance during the AF scan using a greedy strategy */ + if (currentVariance_ > context.frameContext.af.maxVariance) { + context.frameContext.af.maxVariance = currentVariance_; + goodFocus_ = focus_; + } + + if (focus_ > MaxFocusSteps_) { + /* If reach the max step, move lens to the position and set "focus stable". */ + context.frameContext.af.stable = true; + context.frameContext.af.focus = goodFocus_; + } else { + focus_ += MinSearchStep_; + context.frameContext.af.focus = focus_; + } + LOG(IPU3Af, Debug) << "Focus searching max variance is: " + << context.frameContext.af.maxVariance + << " Focus step is " + << goodFocus_ + << " Current scan is " + << focus_; + } + } +} + +} /* namespace ipa::ipu3::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h new file mode 100644 index 00000000..fb36de2a --- /dev/null +++ b/src/ipa/ipu3/algorithms/af.h @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Red Hat + * + * af.h - IPU3 Af control + */ +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ + +#include + +#include + +#include + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::ipu3::algorithms { + +class Af : public Algorithm +{ + /* The format of y_table. From ipu3-ipa repo */ + typedef struct y_table_item { + uint16_t y1_avg; + uint16_t y2_avg; + } y_table_item_t; + +public: + Af(); + ~Af(); + + void prepare(IPAContext &context, ipu3_uapi_params *params) override; + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + +private: + /* Used for focus scan. */ + uint32_t focus_; + /* Focus good */ + uint32_t goodFocus_; + /* Recent AF statistic variance. */ + double currentVariance_; + /* The frames to be ignore before starting measuring. */ + uint32_t ignoreFrame_; +}; + +} /* namespace ipa::ipu3::algorithms */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */ diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build index 4db6ae1d..e1099169 100644 --- a/src/ipa/ipu3/algorithms/meson.build +++ b/src/ipa/ipu3/algorithms/meson.build @@ -1,8 +1,9 @@ # SPDX-License-Identifier: CC0-1.0 ipu3_ipa_algorithms = files([ + 'af.cpp', 'agc.cpp', 'awb.cpp', 'blc.cpp', - 'tone_mapping.cpp', + 'tone_mapping.cpp' ]) diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 86794ac1..ee644d3c 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -67,6 +67,33 @@ namespace libcamera::ipa::ipu3 { * * \var IPASessionConfiguration::grid.stride * \brief Number of cells on one line including the ImgU padding + * + */ + +/** + * \var IPASessionConfiguration::af + * \brief AF parameters configuration of the IPA + * + * \var IPASessionConfiguration::af.start_x + * \brief The start X position of the AF area + * + * \var IPASessionConfiguration::af.start_y + * \brief The start Y position of the AF area + */ + +/** + * \var IPAFrameContext::af + * \brief Context for the Automatic Focus algorithm + * + * \struct IPAFrameContext::af + * \var IPAFrameContext::af.focus + * \brief Current position of the lens + * + * \var IPAFrameContext::af.maxVariance + * \brief The maximum variance of the current image. + * + * \var IPAFrameContext::af.stable + * \brief is the image focused? */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index c6dc0814..aa5bf97f 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -31,6 +31,11 @@ struct IPASessionConfiguration { double minAnalogueGain; double maxAnalogueGain; } agc; + + struct { + uint16_t start_x; + uint16_t start_y; + } af; }; struct IPAFrameContext { @@ -49,6 +54,12 @@ struct IPAFrameContext { double temperatureK; } awb; + struct { + uint32_t focus; + double maxVariance; + bool stable; + } af; + struct { uint32_t exposure; double gain; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index b0c75541..8e1091bf 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -30,6 +30,7 @@ #include "libcamera/internal/mapped_framebuffer.h" +#include "algorithms/af.h" #include "algorithms/agc.h" #include "algorithms/algorithm.h" #include "algorithms/awb.h" @@ -294,6 +295,7 @@ int IPAIPU3::init(const IPASettings &settings, } /* Construct our Algorithms */ + algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique());