From patchwork Fri Mar 24 14:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Semkowicz X-Patchwork-Id: 18447 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 500F3C329C for ; Fri, 24 Mar 2023 14:29:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E58736273D; Fri, 24 Mar 2023 15:29:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1679668174; bh=71gherLPKH0KVRevos7fn6CtT5f5q1Lq0vDofBzbnSs=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=dIQx955IRStyrIA3fp1kdV91iNaRUw/ekdPNaBC2S0KTbe4Q/3wCajDFz5cIK60DU yrd6LxcMm/Oi5CbnSH7fswyKyWCq+v3AZR4HecWYfVbRoTpxn7hICAvJX9DR6IvTUz 4mKlpgnvAfXcD/fipZ9oVZLs2rXM5ZOlpNq2TBJ8p2J2yJRO6kcCdsxKluOHhmaBcl OLzXXbvfl5Aw+BUDCkqeNl85n//B0OCWZvctL400mWE1/E+wgedadgaOhLX+juBLbP e2r5YUtHWYHwIZuAEogTCX8TUsfX1KSbNjOmGKpFhRX1iMBhd0cTHuRoFlw6ygGRsq 8WFGgkM8nSZCQ== Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2733A62715 for ; Fri, 24 Mar 2023 15:29:27 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=thaumatec-com.20210112.gappssmtp.com header.i=@thaumatec-com.20210112.gappssmtp.com header.b="UPOVy/zS"; dkim-atps=neutral Received: by mail-lf1-x12b.google.com with SMTP id y20so2497437lfj.2 for ; Fri, 24 Mar 2023 07:29:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thaumatec-com.20210112.gappssmtp.com; s=20210112; t=1679668166; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aAPkIGXrDpXgztSRfv8F1aKgcAkKp15YtqP6MCql3D8=; b=UPOVy/zScj27iG6qExLy0bMfsvMoEIa/i2WfcSYYh40Srl9ahuMruHHdZ3KYfWB51g d408uincP0JrEjW9MbnpGqPRxgs4SH/gGISqHBzHq32Cemda9CfmH5xKg705q7kjibbM oxCobMApswA5mRjTNZOUPBeTqB8lrf0omidrrYmfrM0EscgzFCrqQ0GXtpaislnWzX+m C8yoZ0hMOG4ueafBNOCw2OJ5w/bXnmyoUmJvg9ziAghyCuxDZpuLIEO2G82Q2Pxi83SU jxmjP3n9YmdLFeXNEj0HtA6Esdf3NB9jb5rYjCQi5Ng4eSlTvpiASCm0aJnvB3UVpyMs YSkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679668166; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aAPkIGXrDpXgztSRfv8F1aKgcAkKp15YtqP6MCql3D8=; b=EZZPl8IREVgty4dWV+VU22NopTYJRlsp/MHFC9Te613PgsIvuXpdQyp32SDXuoj/58 yFGzDcTBy+9y0UH98iO7sqoXLtCzAx/NIZB4mKgQvI0mJbgv3jzO4VqRHIQaT4VReLnr j3ICXfTCvIxm8mxNatIgD5tSRQkP3TiiZQxH8aMOInTJoIFPXZ20/dkyR+3HiuaoFPiz P+YkXsbTbbP7YAM7T4fudLLaFrJtwtuEtelItz7KhZRNUep1aEvL9Mkl2oSQxXNxiPi1 H8Spp6dYxpMyLTyGWyK/Uh73vQvHr7SyiyciTX1sV9bynOymMEUJbkBhribD0f6iiOzm c7MQ== X-Gm-Message-State: AAQBX9dzTtcr1bwWqQM/5vzRLUjXVUUOp7rrofU57HOElQS4Ej7Q4xFv 1AJpQqf7V12ykCu7Vg8tCCPYLqR3A30o7s2THWw= X-Google-Smtp-Source: AKy350b0Nc8GFl/2hLxzUKFPMlhu0/zAefTQPyw9K5IpOpIRGXQIyJ9k17uxfxjcx4FETCRl1njmng== X-Received: by 2002:ac2:442a:0:b0:4ea:e68c:91bc with SMTP id w10-20020ac2442a000000b004eae68c91bcmr820049lfl.18.1679668166323; Fri, 24 Mar 2023 07:29:26 -0700 (PDT) Received: from localhost.localdomain ([91.90.166.178]) by smtp.gmail.com with ESMTPSA id j21-20020a2e8515000000b002a10b2ea089sm981714lji.16.2023.03.24.07.29.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Mar 2023 07:29:26 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 24 Mar 2023 15:29:02 +0100 Message-Id: <20230324142908.64224-5-dse@thaumatec.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230324142908.64224-1-dse@thaumatec.com> References: <20230324142908.64224-1-dse@thaumatec.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 04/10] ipa: Add common contrast based AF implementation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Daniel Semkowicz via libcamera-devel From: Daniel Semkowicz Reply-To: Daniel Semkowicz Cc: jacopo.mondi@ideasonboard.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Create a new class with contrast based Auto Focus implementation using hill climbing algorithm. This common implementation is independent of platform specific code. This way, each platform can just implement contrast calculation and run the AF control loop basing on this class. This implementation is based on the code that was common for IPU3 and RPi AF algorithms. Signed-off-by: Daniel Semkowicz --- .../libipa/algorithms/af_hill_climbing.cpp | 379 ++++++++++++++++++ src/ipa/libipa/algorithms/af_hill_climbing.h | 91 +++++ src/ipa/libipa/algorithms/meson.build | 2 + 3 files changed, 472 insertions(+) create mode 100644 src/ipa/libipa/algorithms/af_hill_climbing.cpp create mode 100644 src/ipa/libipa/algorithms/af_hill_climbing.h diff --git a/src/ipa/libipa/algorithms/af_hill_climbing.cpp b/src/ipa/libipa/algorithms/af_hill_climbing.cpp new file mode 100644 index 00000000..6ee090eb --- /dev/null +++ b/src/ipa/libipa/algorithms/af_hill_climbing.cpp @@ -0,0 +1,379 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Red Hat + * Copyright (C) 2022, Ideas On Board + * Copyright (C) 2023, Theobroma Systems + * + * af_hill_climbing.cpp - AF contrast based hill climbing common algorithm + */ + +#include "af_hill_climbing.h" + +#include "libcamera/internal/yaml_parser.h" + +/** + * \file af_hill_climbing.h + * \brief AF contrast based hill climbing common algorithm + */ + +namespace libcamera { + +namespace ipa::algorithms { + +LOG_DEFINE_CATEGORY(Af) + +/** + * \class AfHillClimbing + * \brief Contrast based hill climbing auto focus control algorithm + * implementation + * + * Control part of the auto focus algorithm. It calculates the lens position + * based on the contrast measure supplied by the platform-specific + * implementation. This way it is independent from the platform. + * + * Platform layer that use this algorithm should call process() function + * for each each frame and set the lens to the calculated position. + * + * Focus search is divided into two phases: + * 1. coarse search, + * 2. fine search. + * + * In the first phase, the lens is moved with bigger steps to quickly find + * a rough position for the best focus. Then, based on the outcome of coarse + * search, the second phase is executed. Lens is moved with smaller steps + * in a limited range within the rough position to find the exact position + * for best focus. + * + * Tuning file parameters: + * - **coarse-search-step:** The value by which the lens position will change + * in one step in the *coarse search* phase. Unit is lens specific. + * - **fine-search-step:** The value by which the lens position will change + * in one step in the *fine search* phase. Unit is lens specific. + * - **fine-search-range:** Search range in the *fine search* phase, expressed + * as a percentage of the coarse search result. Valid values are + * in the [0, 100] interval. Value 5 means 5%. If coarse search stopped + * at value 200, the fine search range will be [190, 210]. + * - **max-variance-change:** ratio of contrast variance change in the + * *continuous mode* needed for triggering the focus change. When the variance + * change exceeds this value, focus search will be triggered. Valid values are + * in the [0.0, 1.0] interval. + * . + * + * \todo Search range in the *fine search* phase should depend on the lens + * movement range rather than coarse search result. + * \todo Implement setRange. + * \todo Implement setSpeed. + * \todo Implement setMeteringMode. + * \todo Implement setWindows. + * \todo Implement the AfPauseDeferred mode. + */ + +/** + * \brief Initialize the AfHillClimbing with tuning data + * \param[in] tuningData The tuning data for the algorithm + * + * This method should be called in the libcamera::ipa::Algorithm::init() + * method of the platform layer. + * + * \return 0 if successful, an error code otherwise + */ +int AfHillClimbing::init(const YamlObject &tuningData) +{ + coarseSearchStep_ = tuningData["coarse-search-step"].get(30); + fineSearchStep_ = tuningData["fine-search-step"].get(1); + fineRange_ = tuningData["fine-search-range"].get(5); + fineRange_ /= 100; + maxChange_ = tuningData["max-variance-change"].get(0.5); + + LOG(Af, Debug) << "coarseSearchStep_: " << coarseSearchStep_ + << ", fineSearchStep_: " << fineSearchStep_ + << ", fineRange_: " << fineRange_ + << ", maxChange_: " << maxChange_; + + return 0; +} + +/** + * \brief Configure the AfHillClimbing with sensor and lens information + * \param[in] minFocusPosition Minimum position supported by camera lens + * \param[in] maxFocusPosition Maximum position supported by camera lens + * + * This method should be called in the libcamera::ipa::Algorithm::configure() + * method of the platform layer. + */ +void AfHillClimbing::configure(int32_t minFocusPosition, + int32_t maxFocusPosition) +{ + minLensPosition_ = minFocusPosition; + maxLensPosition_ = maxFocusPosition; +} + +/** + * \brief Run the auto focus algorithm loop + * \param[in] currentContrast New value of contrast measured for current frame + * + * This method should be called in the libcamera::ipa::Algorithm::process() + * method of the platform layer for each frame. + * + * Contrast value supplied in the \p currentContrast parameter can be platform + * specific. The only requirement is the contrast value must increase with + * the increasing image focus. Contrast value must be highest when image is in + * focus. + * + * \return New lens position calculated by the AF algorithm + */ +int32_t AfHillClimbing::process(double currentContrast) +{ + currentContrast_ = currentContrast; + + if (shouldSkipFrame()) + return lensPosition_; + + switch (mode_) { + case controls::AfModeManual: + /* Nothing to process. */ + break; + case controls::AfModeAuto: + processAutoMode(); + break; + case controls::AfModeContinuous: + processContinuousMode(); + break; + } + + return lensPosition_; +} + +void AfHillClimbing::processAutoMode() +{ + if (state_ == controls::AfStateScanning) { + afCoarseScan(); + afFineScan(); + } +} + +void AfHillClimbing::processContinuousMode() +{ + /* If we are in a paused state, we won't process the stats. */ + if (pauseState_ == controls::AfPauseStatePaused) + return; + + if (state_ == controls::AfStateScanning) { + afCoarseScan(); + afFineScan(); + return; + } + + /* + * AF scan can be started at any moment in AfModeContinuous, + * except when the state is already AfStateScanning. + */ + if (afIsOutOfFocus()) + afReset(); +} + +/** + * \brief Request AF to skip n frames + * \param[in] n Number of frames to be skipped + * + * For the requested number of frames, the AF calculation will be skipped + * and lens position will not change. The platform layer still needs to + * call process() function for each frame during this time. + * This function can be used by the platform layer if the hardware needs more + * time for some operations. + * + * The number of the requested frames (\p n) will be applied only if \p n has + * higher value than the number of frames already requested to be skipped. + * For example, if *setFramesToSkip(5)* was already called for the current + * frame, then calling *setFramesToSkip(3)* will not override the previous + * request and 5 frames will be skipped. + */ +void AfHillClimbing::setFramesToSkip(uint32_t n) +{ + if (n > framesToSkip_) + framesToSkip_ = n; +} + +void AfHillClimbing::setMode(controls::AfModeEnum mode) +{ + if (mode == mode_) + return; + + LOG(Af, Debug) << "Switched AF mode from " << mode_ << " to " << mode; + mode_ = mode; + + state_ = controls::AfStateIdle; + pauseState_ = controls::AfPauseStateRunning; + + if (mode_ == controls::AfModeContinuous) + afReset(); +} + +void AfHillClimbing::setRange([[maybe_unused]] controls::AfRangeEnum range) +{ + LOG(Af, Error) << __FUNCTION__ << " not implemented!"; +} + +void AfHillClimbing::setSpeed([[maybe_unused]] controls::AfSpeedEnum speed) +{ + LOG(Af, Error) << __FUNCTION__ << " not implemented!"; +} + +void AfHillClimbing::setMeteringMode([[maybe_unused]] controls::AfMeteringEnum metering) +{ + LOG(Af, Error) << __FUNCTION__ << " not implemented!"; +} + +void AfHillClimbing::setWindows([[maybe_unused]] Span windows) +{ + LOG(Af, Error) << __FUNCTION__ << " not implemented!"; +} + +void AfHillClimbing::setTrigger(controls::AfTriggerEnum trigger) +{ + if (mode_ != controls::AfModeAuto) { + LOG(Af, Warning) + << __FUNCTION__ << " not valid in mode " << mode_; + return; + } + + LOG(Af, Debug) << "Trigger called with " << trigger; + + if (trigger == controls::AfTriggerStart) + afReset(); + else + state_ = controls::AfStateIdle; +} + +void AfHillClimbing::setPause(controls::AfPauseEnum pause) +{ + if (mode_ != controls::AfModeContinuous) { + LOG(Af, Warning) + << __FUNCTION__ << " not valid in mode " << mode_; + return; + } + + switch (pause) { + case controls::AfPauseImmediate: + pauseState_ = controls::AfPauseStatePaused; + break; + case controls::AfPauseDeferred: + LOG(Af, Warning) << "AfPauseDeferred is not supported!"; + break; + case controls::AfPauseResume: + pauseState_ = controls::AfPauseStateRunning; + break; + } +} + +void AfHillClimbing::setLensPosition(float lensPosition) +{ + if (mode_ != controls::AfModeManual) { + LOG(Af, Warning) + << __FUNCTION__ << " not valid in mode " << mode_; + return; + } + + lensPosition_ = static_cast(lensPosition); + + LOG(Af, Debug) << "Requesting lens position " << lensPosition_; +} + +void AfHillClimbing::afCoarseScan() +{ + if (coarseCompleted_) + return; + + if (afScan(coarseSearchStep_)) { + coarseCompleted_ = true; + maxContrast_ = 0; + const auto diff = static_cast( + std::abs(lensPosition_) * fineRange_); + lensPosition_ = std::max(lensPosition_ - diff, minLensPosition_); + maxStep_ = std::min(lensPosition_ + diff, maxLensPosition_); + } +} + +void AfHillClimbing::afFineScan() +{ + if (!coarseCompleted_) + return; + + if (afScan(fineSearchStep_)) { + LOG(Af, Debug) << "AF found the best focus position!"; + state_ = controls::AfStateFocused; + } +} + +bool AfHillClimbing::afScan(uint32_t steps) +{ + if (lensPosition_ + static_cast(steps) > maxStep_) { + /* If the max step is reached, move lens to the position. */ + lensPosition_ = bestPosition_; + return true; + } else { + /* + * Find the maximum of the variance by estimating its + * derivative. If the direction changes, it means we have passed + * a maximum one step before. + */ + if ((currentContrast_ - maxContrast_) >= -(maxContrast_ * 0.1)) { + /* + * Positive and zero derivative: + * The variance is still increasing. The focus could be + * increased for the next comparison. Also, the max + * variance and previous focus value are updated. + */ + bestPosition_ = lensPosition_; + lensPosition_ += static_cast(steps); + maxContrast_ = currentContrast_; + } else { + /* + * Negative derivative: + * The variance starts to decrease which means the maximum + * variance is found. Set focus step to previous good one + * then return immediately. + */ + lensPosition_ = bestPosition_; + return true; + } + } + + LOG(Af, Debug) << "Previous step is " << bestPosition_ + << ", Current step is " << lensPosition_; + return false; +} + +void AfHillClimbing::afReset() +{ + LOG(Af, Debug) << "Reset AF parameters"; + lensPosition_ = minLensPosition_; + maxStep_ = maxLensPosition_; + state_ = controls::AfStateScanning; + coarseCompleted_ = false; + maxContrast_ = 0.0; + setFramesToSkip(1); +} + +bool AfHillClimbing::afIsOutOfFocus() const +{ + const double diff_var = std::abs(currentContrast_ - maxContrast_); + const double var_ratio = diff_var / maxContrast_; + LOG(Af, Debug) << "Variance change rate: " << var_ratio + << ", Current lens step: " << lensPosition_; + return var_ratio > maxChange_; +} + +bool AfHillClimbing::shouldSkipFrame() +{ + if (framesToSkip_ > 0) { + framesToSkip_--; + return true; + } + + return false; +} + +} /* namespace ipa::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/algorithms/af_hill_climbing.h b/src/ipa/libipa/algorithms/af_hill_climbing.h new file mode 100644 index 00000000..47d2bbec --- /dev/null +++ b/src/ipa/libipa/algorithms/af_hill_climbing.h @@ -0,0 +1,91 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Red Hat + * Copyright (C) 2022, Ideas On Board + * Copyright (C) 2023, Theobroma Systems + * + * af_hill_climbing.h - AF contrast based hill climbing common algorithm + */ + +#pragma once + +#include + +#include "af.h" + +namespace libcamera { + +class YamlObject; + +namespace ipa::algorithms { + +LOG_DECLARE_CATEGORY(Af) + +class AfHillClimbing : public Af +{ +public: + int init(const YamlObject &tuningData); + void configure(int32_t minFocusPosition, int32_t maxFocusPosition); + int32_t process(double currentContrast); + void setFramesToSkip(uint32_t n); + + controls::AfStateEnum state() final { return state_; } + controls::AfPauseStateEnum pauseState() final { return pauseState_; } + +private: + void setMode(controls::AfModeEnum mode) final; + void setRange(controls::AfRangeEnum range) final; + void setSpeed(controls::AfSpeedEnum speed) final; + void setMeteringMode(controls::AfMeteringEnum metering) final; + void setWindows(Span windows) final; + void setTrigger(controls::AfTriggerEnum trigger) final; + void setPause(controls::AfPauseEnum pause) final; + void setLensPosition(float lensPosition) final; + + void processAutoMode(); + void processContinuousMode(); + void afCoarseScan(); + void afFineScan(); + bool afScan(uint32_t steps); + void afReset(); + [[nodiscard]] bool afIsOutOfFocus() const; + bool shouldSkipFrame(); + + controls::AfModeEnum mode_ = controls::AfModeManual; + controls::AfStateEnum state_ = controls::AfStateIdle; + controls::AfPauseStateEnum pauseState_ = controls::AfPauseStateRunning; + + /* Current focus lens position. */ + int32_t lensPosition_ = 0; + /* Local optimum focus lens position during scanning. */ + int32_t bestPosition_ = 0; + + /* Current AF statistic contrast. */ + double currentContrast_ = 0; + /* It is used to determine the derivative during scanning */ + double maxContrast_ = 0; + /* The designated maximum range of focus scanning. */ + int32_t maxStep_ = 0; + /* If the coarse scan completes, it is set to true. */ + bool coarseCompleted_ = false; + + uint32_t framesToSkip_ = 0; + + /* Position limits of the focus lens. */ + int32_t minLensPosition_; + int32_t maxLensPosition_; + + /* Minimum position step for searching appropriate focus. */ + uint32_t coarseSearchStep_; + uint32_t fineSearchStep_; + + /* Fine scan range 0 < fineRange_ < 1. */ + double fineRange_; + + /* Max ratio of variance change, 0.0 < maxChange_ < 1.0. */ + double maxChange_; +}; + +} /* namespace ipa::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/algorithms/meson.build b/src/ipa/libipa/algorithms/meson.build index 3df4798f..20e437fc 100644 --- a/src/ipa/libipa/algorithms/meson.build +++ b/src/ipa/libipa/algorithms/meson.build @@ -2,8 +2,10 @@ libipa_algorithms_headers = files([ 'af.h', + 'af_hill_climbing.h', ]) libipa_algorithms_sources = files([ 'af.cpp', + 'af_hill_climbing.cpp', ])