From patchwork Mon Dec 4 00:10:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 19266 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1ED62C322E for ; Mon, 4 Dec 2023 00:10:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BDC14629DA; Mon, 4 Dec 2023 01:10:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1701648648; bh=TH/uA5CaxgtfBx74YKHme0Y3mbcInfDdz+cHB5vzdpI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=JObBKND85HLbsUyVOQLmb9IFijvOmpzfxARHuN0euHUDIXhZBVXZQZ9ltr+FVr920 0+D79OWb/w7NEQSqlJWQBcB6YOq1Gdk2zzhAOyTOqarQRIPJVnHaYR4jKkEVF7cN8J jlQUj2AKG7Z5CwpSzvR/ZOGkx2YduVQKEbC0t69ZiwtqSv9ie9QT8G03aoDEaeFX8p oCdGzp1WMPZx0wy1zFByCJLmkeAEuCa2ROrSAxGO/ZwddfbYqbmjlmxAYuW3ccSHPN h0n8wFP63id9klejMYQh0MzXi9g9M3ADDZyqUexIttVw0S8YFva6PZUZyxSNNmZ/So ssVHBtQOJ/tQg== Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C51B7629CE for ; Mon, 4 Dec 2023 01:10:46 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="VtyaEUw4"; dkim-atps=neutral Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-3332ad5b3e3so2595480f8f.2 for ; Sun, 03 Dec 2023 16:10:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1701648646; x=1702253446; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I05whO6yD/81OplpogdCagQbY76H+d64aUOxjzpYx3I=; b=VtyaEUw41yJnGnWBKKOn52AB46pUVpsL053y+XtSh2aiyIc+YfJ3XOnTXnjkrK7sLA /wzapK22w5KMbecK5OTzBC/jtyP6os37VCCII7N2hdpc2mh1JBMVmZwim7ncK5Dj992X Pc21/fHoy+qkjvOUTDNxf7ojwrtLpefKNmzytO2u4o2wfc7oKDucAYd8uV+FGz4rm9b3 e2yrUaP6Xd9VtCzDiMgw49x80/zqoialNVBEeEtsnFQ6uBlBMLQlhAoSyKfxboISCERp v85rNp1l1UsFcTPmRz83Et29GLntbX7zHmy4rU77snljmMpPaQz/SHbP/FcnvL+UXMEX iBUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701648646; x=1702253446; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I05whO6yD/81OplpogdCagQbY76H+d64aUOxjzpYx3I=; b=CweaKPGj4+8vD3DcV+OP5iRlwVzdbtRBbqfmJRvpN6hdDzFXejRO1Fa1oee2rOuBJT RyzRSlO15rimmsh1xjQ3YgK35YzvAKeU56liktD/PXy4Zj6vwfjvdWaM4myLBauebpXC qWix1JU41+fzqoKpVpr6VneNdupLgG+2RCN7WwJ2u7RKCNGoFea0ltFeNzH4D/+SzC7c 9Z9yinyPqYx35ioYELbdhQ4aw51LWJKaQDFAevyTOq06Xi4BDPvV0jWbauWreGt1uUsZ 04A640y76iFKem+u6t1M656WbFsp34/vKg7jeMqMkuIre4La9uxh5ueQ6LCqTCdjKuM0 8yFw== X-Gm-Message-State: AOJu0Ywyw7kg16TE5kWOZvEJ6dwuenN/gadMa/FKFOhy2DRn7sgwz2Qy /PRg1FEzJ1QXG4rDF48HotMqqNA9lsT3/TdbaRo= X-Google-Smtp-Source: AGHT+IFKSb1MpVQS82/j3nc49GV7icqpt4l8esiYXmiIliACWxjhOX7YKKDxg+fU855wM+DMeM13iQ== X-Received: by 2002:a05:6000:c43:b0:333:4f67:774a with SMTP id do3-20020a0560000c4300b003334f67774amr21377wrb.65.1701648646548; Sun, 03 Dec 2023 16:10:46 -0800 (PST) Received: from Lat-5310.. ([87.116.165.212]) by smtp.gmail.com with ESMTPSA id u29-20020adfa19d000000b003332db7d91dsm8835997wru.39.2023.12.03.16.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Dec 2023 16:10:46 -0800 (PST) To: libcamera-devel@lists.libcamera.org Date: Mon, 4 Dec 2023 03:10:10 +0300 Message-Id: <20231204001013.404720-5-andrey.konovalov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231204001013.404720-1-andrey.konovalov@linaro.org> References: <20231204001013.404720-1-andrey.konovalov@linaro.org> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 4/7] libcamera: ipa: add Soft IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Andrey Konovalov via libcamera-devel From: Andrey Konovalov Reply-To: Andrey Konovalov Cc: mripard@redhat.com, g.martti@gmail.com, t.langendam@gmail.com, srinivas.kandagatla@linaro.org, bryan.odonoghue@linaro.org, admin@dennisbonke.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Signed-off-by: Andrey Konovalov --- include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 27 ++++ meson_options.txt | 3 +- src/ipa/simple/common/meson.build | 17 ++ src/ipa/simple/common/soft_base.cpp | 66 ++++++++ src/ipa/simple/common/soft_base.h | 47 ++++++ src/ipa/simple/linaro/data/meson.build | 8 + src/ipa/simple/linaro/data/soft.conf | 3 + src/ipa/simple/linaro/meson.build | 26 +++ src/ipa/simple/linaro/soft_linaro.cpp | 210 +++++++++++++++++++++++++ src/ipa/simple/meson.build | 12 ++ 11 files changed, 419 insertions(+), 1 deletion(-) create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/common/meson.build create mode 100644 src/ipa/simple/common/soft_base.cpp create mode 100644 src/ipa/simple/common/soft_base.h create mode 100644 src/ipa/simple/linaro/data/meson.build create mode 100644 src/ipa/simple/linaro/data/soft.conf create mode 100644 src/ipa/simple/linaro/meson.build create mode 100644 src/ipa/simple/linaro/soft_linaro.cpp create mode 100644 src/ipa/simple/meson.build diff --git a/include/libcamera/ipa/meson.build b/include/libcamera/ipa/meson.build index f3b4881c..aaee5cbf 100644 --- a/include/libcamera/ipa/meson.build +++ b/include/libcamera/ipa/meson.build @@ -65,6 +65,7 @@ pipeline_ipa_mojom_mapping = { 'ipu3': 'ipu3.mojom', 'rkisp1': 'rkisp1.mojom', 'rpi/vc4': 'raspberrypi.mojom', + 'simple/linaro': 'soft.mojom', 'vimc': 'vimc.mojom', } diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom new file mode 100644 index 00000000..c3449188 --- /dev/null +++ b/include/libcamera/ipa/soft.mojom @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +/* + * \todo Document the interface and remove the related EXCLUDE_PATTERNS entry. + * \todo Add a way to tell SoftIPA the list of params SoftISP accepts? + */ + +module ipa.soft; + +import "include/libcamera/ipa/core.mojom"; + +interface IPASoftInterface { + init(libcamera.IPASettings settings, + libcamera.SharedFD fdStats, + libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + start() => (int32 ret); + stop(); + configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + + [async] processStats(libcamera.ControlList sensorControls); +}; + +interface IPASoftEventInterface { + setSensorControls(libcamera.ControlList sensorControls); +}; diff --git a/meson_options.txt b/meson_options.txt index fad928af..2fb51ff4 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -27,7 +27,7 @@ option('gstreamer', option('ipas', type : 'array', - choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'vimc'], + choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'simple/linaro', 'vimc'], description : 'Select which IPA modules to build') option('lc-compliance', @@ -46,6 +46,7 @@ option('pipelines', 'rkisp1', 'rpi/vc4', 'simple', + 'simple/linaro', 'uvcvideo', 'vimc' ], diff --git a/src/ipa/simple/common/meson.build b/src/ipa/simple/common/meson.build new file mode 100644 index 00000000..023e617b --- /dev/null +++ b/src/ipa/simple/common/meson.build @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: CC0-1.0 + +soft_ipa_common_sources = files([ + 'soft_base.cpp', +]) + +soft_ipa_common_includes = [ + include_directories('..'), +] + +soft_ipa_common_deps = [ + libcamera_private, +] + +soft_ipa_common_lib = static_library('soft_ipa_common', soft_ipa_common_sources, + include_directories : soft_ipa_common_includes, + dependencies : soft_ipa_common_deps) diff --git a/src/ipa/simple/common/soft_base.cpp b/src/ipa/simple/common/soft_base.cpp new file mode 100644 index 00000000..7bd9b8de --- /dev/null +++ b/src/ipa/simple/common/soft_base.cpp @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * soft-base.cpp - Software IPA base class + */ + +#include "soft_base.h" + +#include + +#include +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoft) + +namespace ipa::soft { + +IPASoftBase::IPASoftBase() +{ +} + +IPASoftBase::~IPASoftBase() +{ +} + +int IPASoftBase::init([[maybe_unused]] const IPASettings &settings, + const SharedFD &fdStats, + const ControlInfoMap &sensorInfoMap) +{ + fdStats_ = std::move(fdStats); + if (!fdStats_.isValid()) { + LOG(IPASoft, Error) << "Invalid Statistics handle"; + return -ENODEV; + } + + return platformInit(sensorInfoMap); +} + +int IPASoftBase::configure(const ControlInfoMap &sensorInfoMap) +{ + return platformConfigure(sensorInfoMap); +} + +int IPASoftBase::start() +{ + return platformStart(); +} + +void IPASoftBase::stop() +{ + return platformStop(); +} + +void IPASoftBase::processStats(const ControlList &sensorControls) +{ + return platformProcessStats(sensorControls); +} + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/common/soft_base.h b/src/ipa/simple/common/soft_base.h new file mode 100644 index 00000000..bff53713 --- /dev/null +++ b/src/ipa/simple/common/soft_base.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * soft-base.h - Software IPA base class + */ +#pragma once + +#include +#include + +#include + +namespace libcamera { + +namespace ipa::soft { + +class IPASoftBase : public ipa::soft::IPASoftInterface +{ +public: + IPASoftBase(); + ~IPASoftBase(); + + int init(const IPASettings &settings, + const SharedFD &fdStats, + const ControlInfoMap &sensorInfoMap) override; + int configure(const ControlInfoMap &sensorInfoMap) override; + + int start() override; + void stop() override; + + void processStats(const ControlList &sensorControls) override; + +protected: + SharedFD fdStats_; + +private: + virtual int platformInit(const ControlInfoMap &sensorInfoMap) = 0; + virtual int platformConfigure(const ControlInfoMap &sensorInfoMap) = 0; + virtual int platformStart() = 0; + virtual void platformStop() = 0; + virtual void platformProcessStats(const ControlList &sensorControls) = 0; +}; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/linaro/data/meson.build b/src/ipa/simple/linaro/data/meson.build new file mode 100644 index 00000000..f3464375 --- /dev/null +++ b/src/ipa/simple/linaro/data/meson.build @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: CC0-1.0 + +conf_files = files([ + 'soft.conf', +]) + +install_data(conf_files, + install_dir : ipa_data_dir / 'soft') diff --git a/src/ipa/simple/linaro/data/soft.conf b/src/ipa/simple/linaro/data/soft.conf new file mode 100644 index 00000000..0c70e7c0 --- /dev/null +++ b/src/ipa/simple/linaro/data/soft.conf @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later +# +# Dummy configuration file for the soft IPA. diff --git a/src/ipa/simple/linaro/meson.build b/src/ipa/simple/linaro/meson.build new file mode 100644 index 00000000..97bf5d6f --- /dev/null +++ b/src/ipa/simple/linaro/meson.build @@ -0,0 +1,26 @@ +# SPDX-License-Identifier: CC0-1.0 + +ipa_name = 'ipa_soft_linaro' + +mod = shared_module(ipa_name, + ['soft_linaro.cpp', libcamera_generated_ipa_headers], + name_prefix : '', + include_directories : [ipa_includes, libipa_includes, '..'], + dependencies : libcamera_private, + link_with : libipa, + link_whole : soft_ipa_common_lib, + install : true, + install_dir : ipa_install_dir) + +if ipa_sign_module + custom_target(ipa_name + '.so.sign', + input : mod, + output : ipa_name + '.so.sign', + command : [ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) +endif + +subdir('data') + +ipa_names += ipa_name diff --git a/src/ipa/simple/linaro/soft_linaro.cpp b/src/ipa/simple/linaro/soft_linaro.cpp new file mode 100644 index 00000000..0b2e83bf --- /dev/null +++ b/src/ipa/simple/linaro/soft_linaro.cpp @@ -0,0 +1,210 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * soft.cpp - Software Image Processing Algorithm module + */ + +#include + +#include +#include + +#include + +#include +#include + +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/software_isp/statistics-linaro.h" + +#include "common/soft_base.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(IPASoft) + +namespace ipa::soft { + +class IPASoftLinaro final : public IPASoftBase +{ +public: + IPASoftLinaro() + : IPASoftBase(), ignore_updates_(0) + { + } + + ~IPASoftLinaro() + { + if (stats_) + munmap(stats_, sizeof(SwIspStats)); + } + + int platformInit(const ControlInfoMap &sensorInfoMap) override; + int platformConfigure(const ControlInfoMap &sensorInfoMap) override; + int platformStart() override; + void platformStop() override; + void platformProcessStats(const ControlList &sensorControls) override; + +private: + void update_exposure(double ev_adjustment); + + SwIspStats *stats_; + int exposure_min_, exposure_max_; + int again_min_, again_max_; + int again_, exposure_; + int ignore_updates_; +}; + +int IPASoftLinaro::platformInit(const ControlInfoMap &sensorInfoMap) +{ + stats_ = static_cast(mmap(nullptr, sizeof(SwIspStats), + PROT_READ | PROT_WRITE, MAP_SHARED, + fdStats_.get(), 0)); + if (!stats_) { + LOG(IPASoft, Error) << "Unable to map Statistics"; + return -ENODEV; + } + + if (sensorInfoMap.find(V4L2_CID_EXPOSURE) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have exposure control"; + return -EINVAL; + } + + if (sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have gain control"; + return -EINVAL; + } + + return 0; +} + +int IPASoftLinaro::platformConfigure(const ControlInfoMap &sensorInfoMap) +{ + const ControlInfo &exposure_info = sensorInfoMap.find(V4L2_CID_EXPOSURE)->second; + const ControlInfo &gain_info = sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN)->second; + + exposure_min_ = exposure_info.min().get(); + if (!exposure_min_) { + LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; + exposure_min_ = 1; + } + exposure_max_ = exposure_info.max().get(); + again_min_ = gain_info.min().get(); + if (!again_min_) { + LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; + again_min_ = 100; + } + again_max_ = gain_info.max().get(); + + LOG(IPASoft, Info) << "Exposure " << exposure_min_ << "-" << exposure_max_ + << ", gain " << again_min_ << "-" << again_max_; + + return 0; +} + +int IPASoftLinaro::platformStart() +{ + return 0; +} + +void IPASoftLinaro::platformStop() +{ +} + +void IPASoftLinaro::platformProcessStats(const ControlList &sensorControls) +{ + double ev_adjustment = 0.0; + ControlList ctrls(sensorControls); + + /* + * Use 2 frames delay to make sure that the exposure and the gain set + * have applied to the camera sensor + */ + if (ignore_updates_ > 0) { + LOG(IPASoft, Debug) << "Skipping exposure update: " + << ignore_updates_; + --ignore_updates_; + return; + } + + if (stats_->bright_ratio < 0.01) + ev_adjustment = 1.1; + if (stats_->too_bright_ratio > 0.04) + ev_adjustment = 0.9; + + if (ev_adjustment != 0.0) { + /* sanity check */ + if (!sensorControls.contains(V4L2_CID_EXPOSURE) || + !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { + LOG(IPASoft, Error) << "Control(s) missing"; + return; + } + + exposure_ = ctrls.get(V4L2_CID_EXPOSURE).get(); + again_ = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + + update_exposure(ev_adjustment); + + ctrls.set(V4L2_CID_EXPOSURE, exposure_); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, again_); + + ignore_updates_ = 2; + + setSensorControls.emit(ctrls); + } +} + +void IPASoftLinaro::update_exposure(double ev_adjustment) +{ + double exp = (double)exposure_; + double gain = (double)again_; + double ev = ev_adjustment * exp * gain; + + /* + * Try to use the minimal possible analogue gain. + * The exposure can be any value from exposure_min_ to exposure_max_, + * and normally this should keep the frame rate intact. + */ + + exp = ev / again_min_; + if (exp > exposure_max_) + exposure_ = exposure_max_; + else if (exp < exposure_min_) + exposure_ = exposure_min_; + else + exposure_ = (int)exp; + + gain = ev / exposure_; + if (gain > again_max_) + again_ = again_max_; + else if (gain < again_min_) + again_ = again_min_; + else + again_ = (int)gain; + + LOG(IPASoft, Debug) << "Desired EV = " << ev + << ", real EV = " << (double)again_ * exposure_; +} + +} /* namespace ipa::soft */ + +/* + * External IPA module interface + */ +extern "C" { +const struct IPAModuleInfo ipaModuleInfo = { + IPA_MODULE_API_VERSION, + 0, + "SimplePipelineHandler", + "soft/linaro", +}; + +IPAInterface *ipaCreate() +{ + return new ipa::soft::IPASoftLinaro(); +} + +} /* extern "C" */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build new file mode 100644 index 00000000..14be5dc2 --- /dev/null +++ b/src/ipa/simple/meson.build @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: CC0-1.0 + +subdir('common') + +foreach pipeline : pipelines + pipeline = pipeline.split('/') + if pipeline.length() < 2 or pipeline[0] != 'simple' + continue + endif + + subdir(pipeline[1]) +endforeach