From patchwork Fri Mar 31 08:19:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Semkowicz X-Patchwork-Id: 18513 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2C096C32A0 for ; Fri, 31 Mar 2023 08:19:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B942462750; Fri, 31 Mar 2023 10:19:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1680250798; bh=cvm0p2xzbPrL3WWPGL+Zj+DNBa64YRXSKp2GZkL4TUQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=zGmoralrNUqXJFr7hZ4h0Xasbc2cGUH5Xj/30qu5AZtldwcOU1JtN+nPyOfPfrJF7 hNqh/lCL0rlweKXtRe3ockl4F7ca1wX4zsNdg+TvdOl6Tven7sv901A46VasjEYoAe K1EQwITosVKPzuWSRxj7NHbRyUzCkTNXMUbI+S59jWRM1CHRGIAcV08Jf9bC2rJXOH 2WLhQ1zljdMr1o3gbWQVlCGd2jRumW34VKUNiqExX+zRO8B6LHFk1RTVIqebsMjLxn lAAO84mmHiAX6aF9kgzPOTMZ9ZcuYhdxkHz62mnPNH8i/2f931WhJYfNTdwWYmzx3a h7/sNEGIU1aAw== Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DC9BA62756 for ; Fri, 31 Mar 2023 10:19:50 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=thaumatec-com.20210112.gappssmtp.com header.i=@thaumatec-com.20210112.gappssmtp.com header.b="vmRApNIr"; dkim-atps=neutral Received: by mail-lf1-x129.google.com with SMTP id g17so27898114lfv.4 for ; Fri, 31 Mar 2023 01:19:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thaumatec-com.20210112.gappssmtp.com; s=20210112; t=1680250790; x=1682842790; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KuEVT9yr3uileFamXcqJ52XI9cacPgm5UgjSltD6O0I=; b=vmRApNIrNh7EjBdIfPm3URfxDuWXBNkpn/4xjM34Maa7FNeJVU9//mzTX8IvaEbssc taSchExq/1Z6DR9QjnV8Aw/klwdAeuJCugdqjx5tFYVB5sCStU5ZOvLIG8+HcKLe4RuT /U29xYtACWRYPPq+Rj2rCWorTfhIwNNHS5reseEcQfuuGlscfVLDVixIqnW9c7qrTx6v NLD+O2Ssnrhkt/eHBNFRxBPq07nf99vTC5htu9lkEDUbC1oppsoOq0+ERueDgdJx9fxq hOWpP10ffKkBsq/AFaqUBgR+BCuS4VrKqwsrJEYTVITS/6Skw1K56mIDSgi78RVi9RZ/ HFGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680250790; x=1682842790; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KuEVT9yr3uileFamXcqJ52XI9cacPgm5UgjSltD6O0I=; b=7Qe6cHOP66fp0B2NbsLemf0vl9fmz0liZyKjwrWoBrouNnOarizBqYvXQ6vxVk70Vh 1CJ7OIRWvZhb2eIktUQD+9yp5s6z/y9st+wctP8wxvCkyf2zPj/3m3Q/dlACffALl682 /DRw8NfTzAPxFisD9/5iQj+/uRchQl2nOPFEFN1G+3DmorGZiDQGoVRSxZqAKrdWPK0H uZLmuSYZ/udBxpadbRwsCciB3h1iURA++uubkj8j3SBFslMFSnjTmTPvDLikXpETtWKJ FO2Sq7UA8leIGHokhErYwk2v7+COdTitnqauJTgA7NWSfCCuntbcLYdC/UXXQ7k+XAdH ckFA== X-Gm-Message-State: AAQBX9c5gg47tzzkf/ZUwwMPxyo6perGJy6d2sF1d2aFDVE0/HnLhWda emhGxGGKem37ln0Zc9R/rkhsgvwLEpWtYcyipa5nNQ== X-Google-Smtp-Source: AKy350Zl/hVLFcg7uKKOC7EiEHmz2EfBLTeaJVNhAge0Jqgf54mhaU0urN1dtW/4XNvOR0P4POrLIw== X-Received: by 2002:a19:ad47:0:b0:4ea:c730:aac3 with SMTP id s7-20020a19ad47000000b004eac730aac3mr2335520lfd.3.1680250790194; Fri, 31 Mar 2023 01:19:50 -0700 (PDT) Received: from localhost.localdomain ([91.90.166.178]) by smtp.gmail.com with ESMTPSA id t1-20020ac24c01000000b004e84a8c3d86sm281459lfq.42.2023.03.31.01.19.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Mar 2023 01:19:49 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 31 Mar 2023 10:19:26 +0200 Message-Id: <20230331081930.19289-7-dse@thaumatec.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230331081930.19289-1-dse@thaumatec.com> References: <20230331081930.19289-1-dse@thaumatec.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v6 06/10] ipa: rkisp1: Add AF algorithm based on AfHillClimbing X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Daniel Semkowicz via libcamera-devel From: Daniel Semkowicz Reply-To: Daniel Semkowicz Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add contrast based auto focus implementation for rkisp1 platform using the common AfHillClimbing algorithm. Rockchip ISP AF block allows calculation of sharpness and luminance in up to three user defined windows. If no windows are set, there are some default settings applied for the first window and exposed through the driver. For each frame, use the sharpness value calculated for this default window and feed the hill climbing algorithm with them. Then set the lens position to value calculated by the algorithm. Signed-off-by: Daniel Semkowicz Reviewed-by: Jacopo Mondi --- src/ipa/rkisp1/algorithms/af.cpp | 121 ++++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/af.h | 43 +++++++++ src/ipa/rkisp1/algorithms/meson.build | 1 + src/ipa/rkisp1/ipa_context.cpp | 13 +++ src/ipa/rkisp1/ipa_context.h | 5 ++ 5 files changed, 183 insertions(+) create mode 100644 src/ipa/rkisp1/algorithms/af.cpp create mode 100644 src/ipa/rkisp1/algorithms/af.h diff --git a/src/ipa/rkisp1/algorithms/af.cpp b/src/ipa/rkisp1/algorithms/af.cpp new file mode 100644 index 00000000..fde924d4 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/af.cpp @@ -0,0 +1,121 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Theobroma Systems + * + * af.cpp - RkISP1 AF hill climbing based control algorithm + */ + +#include "af.h" + +/** + * \file af.h + */ + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +/** + * \class Af + * \brief AF contrast based hill climbing control algorithm for RkISP platforms + * + * Auto focus algorithm for RkISP platforms, based on the common hill climbing + * auto focus implementation (libcamera::ipa::algorithms::AfHillClimbing). + * + * This is the platform layer of the algorithm. + * + * Tuning file parameters: + * - **wait-frames-lens:** Number of frames that should be skipped when lens + * position is changed. Lens movement takes some time and statistics measured + * during the lens movement are unstable. Currently there is no way to know + * when lens movement finished and this is a workaround for this. Wait a fixed + * amount of time on each movement. This parameter should be set according + * to the worst case - the number of frames it takes to move lens between + * limit positions. + * . + * \sa libcamera::ipa::algorithms::AfHillClimbing for additional tuning + * parameters. + * + * \todo Model the lens delay as number of frames required for the lens position + * to stabilize in the CameraLens class. + */ + +LOG_DEFINE_CATEGORY(RkISP1Af) + +/** + * \copydoc libcamera::ipa::Algorithm::init + */ +int Af::init(IPAContext &context, const YamlObject &tuningData) +{ + const auto &lensConfig = context.configuration.lens; + if (!lensConfig) { + LOG(RkISP1Af, Error) << "Lens not found"; + return 1; + } + + waitFramesLens_ = tuningData["wait-frames-lens"].get(1); + + LOG(RkISP1Af, Debug) << "waitFramesLens_: " << waitFramesLens_; + + return af.init(lensConfig->minFocusPosition, + lensConfig->maxFocusPosition, tuningData); +} + +/** + * \copydoc libcamera::ipa::Algorithm::configure + */ +int Af::configure(IPAContext &context, + const IPACameraSensorInfo &configInfo) +{ + /* + * Lens position is unknown at the startup, so initialize + * the current position to something out of range. + */ + context.activeState.af.lensPosition = + context.configuration.lens->maxFocusPosition + 1; + + af.configure(configInfo.outputSize); + return 0; +} + +/** + * \copydoc libcamera::ipa::Algorithm::queueRequest + */ +void Af::queueRequest([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const ControlList &controls) +{ + af.queueRequest(controls); +} + +/** + * \copydoc libcamera::ipa::Algorithm::process + */ +void Af::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + [[maybe_unused]] ControlList &metadata) +{ + uint32_t sharpness = stats->params.af.window[0].sum; + uint32_t luminance = stats->params.af.window[0].lum; + + LOG(RkISP1Af, Debug) + << "lensPosition: " << context.activeState.af.lensPosition + << ", Sharpness: " << sharpness + << ", Luminance: " << luminance; + + int32_t newLensPosition = af.process(sharpness); + + if (newLensPosition != context.activeState.af.lensPosition) { + context.activeState.af.lensPosition = newLensPosition; + context.activeState.af.applyLensCtrls = true; + af.skipFrames(waitFramesLens_); + } +} + +REGISTER_IPA_ALGORITHM(Af, "Af") + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/af.h b/src/ipa/rkisp1/algorithms/af.h new file mode 100644 index 00000000..3ba66d38 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/af.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Theobroma Systems + * + * af.h - RkISP1 AF hill climbing based control algorithm + */ + +#pragma once + +#include + +#include "libipa/algorithms/af_hill_climbing.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +class Af : public Algorithm +{ +public: + int init(IPAContext &context, const YamlObject &tuningData) override; + int configure(IPAContext &context, + const IPACameraSensorInfo &configInfo) override; + void queueRequest(IPAContext &context, uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) override; + void process(IPAContext &context, uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) override; + +private: + ipa::algorithms::AfHillClimbing af; + + /* Wait number of frames after changing lens position */ + uint32_t waitFramesLens_ = 0; +}; + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build index 93a48329..ab7e44f3 100644 --- a/src/ipa/rkisp1/algorithms/meson.build +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 rkisp1_ipa_algorithms = files([ + 'af.cpp', 'agc.cpp', 'awb.cpp', 'blc.cpp', diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp index aea99299..d80a7b1b 100644 --- a/src/ipa/rkisp1/ipa_context.cpp +++ b/src/ipa/rkisp1/ipa_context.cpp @@ -139,6 +139,19 @@ namespace libcamera::ipa::rkisp1 { * member may be read by any algorithm, but shall only be written by its owner. */ +/** + * \var IPAActiveState::af + * \brief State for the Automatic Focus Control algorithm + * + * \var IPAActiveState::af.lensPosition + * \brief Lens position calculated by the AF algorithm + * + * \var IPAActiveState::af.applyLensCtrls + * \brief Whether the lens position should be applied + * + * If true, IPA should send new controls to the PH to set new lens position. + */ + /** * \var IPAActiveState::agc * \brief State for the Automatic Gain Control algorithm diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 65b3fbfe..3dcc5aa0 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -62,6 +62,11 @@ struct IPASessionConfiguration { }; struct IPAActiveState { + struct { + int32_t lensPosition; + bool applyLensCtrls; + } af; + struct { struct { uint32_t exposure;