From patchwork Wed Aug 23 14:49:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 18957 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5EF05C3262 for ; Wed, 23 Aug 2023 14:49:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3F2F7627E4; Wed, 23 Aug 2023 16:49:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692802172; bh=/6OAm32O/8/h8ATkL6ZBAhkFN4CvYDrUnHEB/K6oh6I=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=3VlCje7OKEar+rMgTwANHo+CaczKfJgF8EL4GiTUu26X8l7UXPTqXPF9w63KwManu NKvYMo/51Q5ZiMgny9cMTj2ypE8c0yEC8pwGTobBmJDz71bJ3nfSkB+VlyO8G9UA71 re95//AmkfsMhDq3EGY7Ro+6byOopZf5rKyQYk8Msi18Bld0mKe9JtHuvpJugoW3GG 3vyrPUCpeYG9rzFt3V+a2erRpxcbO5rxzqN4ef1lBVWBL5nRZ8g32SeygVR3c7PRNE pXLrNGHIrybk+2Yz+JDZonvTxaf+BlYvpUq5nXjmEnx1jfOWLJ4HLLeVs4p+s4UzKi Nigl836emjZgw== Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AF5E461F19 for ; Wed, 23 Aug 2023 16:49:29 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="XY9wgkoB"; dkim-atps=neutral Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-3fee8b78097so36443725e9.0 for ; Wed, 23 Aug 2023 07:49:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1692802169; x=1693406969; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e+nFoh2hDbHr55JapTHYhFfFI6sILsJ6CGAcRKCu4OU=; b=XY9wgkoB8whsAxwhka1Rom9MF5QLb53UoK+8QUHuP50tbrMKaJe2CbEHmI7SB6cG85 zMRNT4cn6HYhyz+4ZyG7D65y6uJsEfr4MQtV3vhu63172HEfaBHf2VUY94TuMbr1a8b+ TTr9/X8igC3fDBLBxCM7iPOQx9B4V8xI9S2mOj+Mb4uN16o5oni1Zpl9DX7OTiwlH2hY Z0JTDtT7N1wiYkfHhkUFiwnzbDvrUFX05WsrQlyUXDhZdIaUdfNGBrHXYf2YVWkAiBKs 4akfEbxFTTKWO37mBiRDpugo9WWOEUzl1gGXS/sQUSW7SFV07BOLl1mSGl4MGU29I1Kh xUtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692802169; x=1693406969; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e+nFoh2hDbHr55JapTHYhFfFI6sILsJ6CGAcRKCu4OU=; b=QViICP+lv0v338d3oBF3p25a72HKfMtndkuZPm3S9do43A5ssqKXyA5s7jhA4s2qh3 RQMs+joblwsiLmguNVLksJJ0kFmmsvjrtMbbZdTIin6Kh92/ljdGkkEWklSFribaHJea sf60uKhNn1tckoeLKsbf3c82BrmeGo/ptu78C4TCWwZwGUF4CpAxVc8mN+qWCcXP1mTu 0l4+TZTuEijj7bQtjQAitLSsCcPeATGtJlP2v8IZHRvr0IF6NMF5vQh6fRqJG0o079mN ZWUdy4s2ClRvTi5XY16SiCXezid8v2PRN8ezjvK3ZeipYkVMH7Qp3OhWli6kGJZ7qyta vLug== X-Gm-Message-State: AOJu0YykCsZgJMH0irCn4zUukw7TO9cz0/aH+62uimvkAm9Y8x/cCWmk h9QLXDKBcRcc7LkWCAWmGTXfjXosoxltObdstVM= X-Google-Smtp-Source: AGHT+IFe4uky670Fz9LQURU2WqnTharTeSTkM/agwxSxgw+0FlrtzJq9QJbjbNSkcThyDn4xxSaEPQ== X-Received: by 2002:a05:600c:2296:b0:3fd:2e6b:10c8 with SMTP id 22-20020a05600c229600b003fd2e6b10c8mr10140323wmf.23.1692802168779; Wed, 23 Aug 2023 07:49:28 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:2bce:64d6:1a5c:49a2]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003feef9a6cb2sm10983535wmi.28.2023.08.23.07.49.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 07:49:28 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 23 Aug 2023 15:49:24 +0100 Message-Id: <20230823144925.2542-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230823144925.2542-1-david.plowman@raspberrypi.com> References: <20230823144925.2542-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/2] ipa: rpi: Add an HDR algorithm to drive multi-channel AGC X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: David Plowman via libcamera-devel From: David Plowman Reply-To: David Plowman Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This HDR algorithm doesn't combine images in any way, but simply allows an application to drive the AGC in a multi-channel HDR type of mode, such as to produce short and long exposure images. Sufficient plumbing is added to enable the libcamera HDR controls and metadata to work. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck --- src/ipa/rpi/common/ipa_base.cpp | 49 ++++++++++ src/ipa/rpi/controller/hdr_algorithm.h | 23 +++++ src/ipa/rpi/controller/hdr_status.h | 25 +++++ src/ipa/rpi/controller/meson.build | 1 + src/ipa/rpi/controller/rpi/hdr.cpp | 127 +++++++++++++++++++++++++ src/ipa/rpi/controller/rpi/hdr.h | 42 ++++++++ 6 files changed, 267 insertions(+) create mode 100644 src/ipa/rpi/controller/hdr_algorithm.h create mode 100644 src/ipa/rpi/controller/hdr_status.h create mode 100644 src/ipa/rpi/controller/rpi/hdr.cpp create mode 100644 src/ipa/rpi/controller/rpi/hdr.h diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index f7e7ad5e..231e8f96 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -24,6 +24,8 @@ #include "controller/ccm_status.h" #include "controller/contrast_algorithm.h" #include "controller/denoise_algorithm.h" +#include "controller/hdr_algorithm.h" +#include "controller/hdr_status.h" #include "controller/lux_status.h" #include "controller/sharpen_algorithm.h" #include "controller/statistics.h" @@ -67,6 +69,7 @@ const ControlInfoMap::Map ipaControls{ { &controls::AeFlickerPeriod, ControlInfo(100, 1000000) }, { &controls::Brightness, ControlInfo(-1.0f, 1.0f, 0.0f) }, { &controls::Contrast, ControlInfo(0.0f, 32.0f, 1.0f) }, + { &controls::HdrMode, ControlInfo(controls::HdrModeValues) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, { &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) }, { &controls::FrameDurationLimits, ControlInfo(INT64_C(33333), INT64_C(120000)) }, @@ -658,9 +661,17 @@ static const std::map AfPauseTable { controls::AfPauseResume, RPiController::AfAlgorithm::AfPauseResume }, }; +static const std::map HdrModeTable = { + { controls::HdrModeOff, "Off" }, + { controls::HdrModeMultiExposure, "MultiExposure" }, + { controls::HdrModeSingleExposure, "SingleExposure" }, +}; + void IpaBase::applyControls(const ControlList &controls) { + using RPiController::AgcAlgorithm; using RPiController::AfAlgorithm; + using RPiController::HdrAlgorithm; /* Clear the return metadata buffer. */ libcameraMetadata_.clear(); @@ -1157,6 +1168,34 @@ void IpaBase::applyControls(const ControlList &controls) break; } + case controls::HDR_MODE: { + HdrAlgorithm *hdr = dynamic_cast(controller_.getAlgorithm("hdr")); + if (!hdr) { + LOG(IPARPI, Warning) << "No HDR algorithm available"; + break; + } + + auto mode = HdrModeTable.find(ctrl.second.get()); + if (mode == HdrModeTable.end()) { + LOG(IPARPI, Warning) << "Unrecognised HDR mode"; + break; + } + + AgcAlgorithm *agc = dynamic_cast(controller_.getAlgorithm("agc")); + if (!agc) { + LOG(IPARPI, Warning) << "HDR requires an AGC algorithm"; + break; + } + + if (hdr->setMode(mode->second) == 0) + agc->setActiveChannels(hdr->getChannels()); + else + LOG(IPARPI, Warning) + << "HDR mode " << mode->second << " not supported"; + + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() @@ -1304,6 +1343,16 @@ void IpaBase::reportMetadata(unsigned int ipaContext) libcameraMetadata_.set(controls::AfPauseState, p); } + const HdrStatus *hdrStatus = rpiMetadata.getLocked("hdr.status"); + if (hdrStatus) { + if (hdrStatus->channel == "short") + libcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelShort); + else if (hdrStatus->channel == "long") + libcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelLong); + else + libcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelNone); + } + metadataReady.emit(libcameraMetadata_); } diff --git a/src/ipa/rpi/controller/hdr_algorithm.h b/src/ipa/rpi/controller/hdr_algorithm.h new file mode 100644 index 00000000..5164d50e --- /dev/null +++ b/src/ipa/rpi/controller/hdr_algorithm.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * hdr_algorithm.h - HDR control algorithm interface + */ +#pragma once + +#include "algorithm.h" + +namespace RPiController { + +class HdrAlgorithm : public Algorithm +{ +public: + HdrAlgorithm(Controller *controller) + : Algorithm(controller) {} + /* An HDR algorithm must provide the following: */ + virtual int setMode(std::string const &modeName) = 0; + virtual std::vector getChannels() const = 0; +}; + +} /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/hdr_status.h b/src/ipa/rpi/controller/hdr_status.h new file mode 100644 index 00000000..120dcb1a --- /dev/null +++ b/src/ipa/rpi/controller/hdr_status.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * hdr_status.h - HDR control algorithm status + */ +#pragma once + +#include + +/* + * The HDR algorithm process method should post an HdrStatus into the image + * metadata under the tag "hdr.status". + */ + +struct HdrStatus { + std::string mode; + std::string channel; + + /* Parameters for programming the stitch block (where available). */ + bool stitch_enable; + uint16_t thresholdLo; + uint8_t diffPower; + double motionThreshold; +}; diff --git a/src/ipa/rpi/controller/meson.build b/src/ipa/rpi/controller/meson.build index 20b9cda9..c9a3e850 100644 --- a/src/ipa/rpi/controller/meson.build +++ b/src/ipa/rpi/controller/meson.build @@ -16,6 +16,7 @@ rpi_ipa_controller_sources = files([ 'rpi/contrast.cpp', 'rpi/dpc.cpp', 'rpi/geq.cpp', + 'rpi/hdr.cpp', 'rpi/lux.cpp', 'rpi/noise.cpp', 'rpi/sdn.cpp', diff --git a/src/ipa/rpi/controller/rpi/hdr.cpp b/src/ipa/rpi/controller/rpi/hdr.cpp new file mode 100644 index 00000000..2700d35e --- /dev/null +++ b/src/ipa/rpi/controller/rpi/hdr.cpp @@ -0,0 +1,127 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022 Raspberry Pi Ltd + * + * hdr.cpp - HDR control algorithm + */ + +#include "hdr.h" + +#include + +#include "../agc_status.h" + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiHdr) + +#define NAME "rpi.hdr" + +void HdrConfig::read(const libcamera::YamlObject ¶ms, const std::string &modeName) +{ + name = modeName; + + if (!params.contains("cadence")) + LOG(RPiHdr, Fatal) << "No cadence for HDR mode " << name; + cadence = params["cadence"].getList().value(); + if (cadence.empty()) + LOG(RPiHdr, Fatal) << "Empty cadence in HDR mode " << name; + + /* + * In the JSON file it's easier to use the channel name as the key, but + * for us it's convenient to swap them over. + */ + for (const auto &[k, v] : params["channel_map"].asDict()) + channelMap[v.get().value()] = k; +} + +Hdr::Hdr(Controller *controller) + : HdrAlgorithm(controller), + currentConfig_(nullptr) +{ +} + +char const *Hdr::name() const +{ + return NAME; +} + +int Hdr::read(const libcamera::YamlObject ¶ms) +{ + /* Make an "HDR off" mode by default so that tuning files don't have to. */ + HdrConfig &offMode = config_["Off"]; + offMode.name = "Off"; + offMode.cadence = { 0 }; + currentConfig_ = &offMode; + + for (const auto &[key, value] : params.asDict()) + config_[key].read(value, key); + + return 0; +} + +int Hdr::setMode(std::string const &mode) +{ + auto it = config_.find(mode); + if (it == config_.end()) { + LOG(RPiHdr, Warning) << "No such HDR mode " << mode; + return -1; + } + + currentConfig_ = &it->second; + + return 0; +} + +std::vector Hdr::getChannels() const +{ + return currentConfig_->cadence; +} + +void Hdr::switchMode([[maybe_unused]] CameraMode const &cameraMode, + [[maybe_unused]] Metadata *metadata) +{ +} + +void Hdr::process([[maybe_unused]] StatisticsPtr &stats, Metadata *imageMetadata) +{ + if (currentConfig_->name == "Off") + return; + + /* First find out what AGC channel this is, which comes from the delayed_status. */ + bool channelKnown = false; + unsigned int agcChannel = 0; + { + std::scoped_lock lock(*imageMetadata); + AgcStatus *agcStatus = imageMetadata->getLocked("agc.delayed_status"); + if (agcStatus) { + channelKnown = true; + agcChannel = agcStatus->channel; + } + } + + /* Fill out the HdrStatus information. */ + HdrStatus hdrStatus; + hdrStatus.mode = currentConfig_->name; + if (channelKnown) { + /* + * Channels that aren't required for HDR processing might come through for + * various reasons, such as when HDR starts up, or even because the cadence + * demands some frames that you need for other purposes (e.g. for preview). + * We'll leave the channel name empty in these cases. + */ + auto it = currentConfig_->channelMap.find(agcChannel); + if (it != currentConfig_->channelMap.end()) + hdrStatus.channel = it->second; + } + + imageMetadata->set("hdr.status", hdrStatus); +} + +/* Register algorithm with the system. */ +static Algorithm *create(Controller *controller) +{ + return (Algorithm *)new Hdr(controller); +} +static RegisterAlgorithm reg(NAME, &create); diff --git a/src/ipa/rpi/controller/rpi/hdr.h b/src/ipa/rpi/controller/rpi/hdr.h new file mode 100644 index 00000000..8f6457f2 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/hdr.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022, Raspberry Pi Ltd + * + * hdr.h - HDR control algorithm + */ +#pragma once + +#include + +#include "../hdr_algorithm.h" +#include "../hdr_status.h" + +/* This is our implementation of an HDR algorithm. */ + +namespace RPiController { + +struct HdrConfig { + std::string name; + std::vector cadence; + std::map channelMap; + + void read(const libcamera::YamlObject ¶ms, const std::string &name); +}; + +class Hdr : public HdrAlgorithm +{ +public: + Hdr(Controller *controller); + char const *name() const override; + void switchMode(CameraMode const &cameraMode, Metadata *metadata) override; + int read(const libcamera::YamlObject ¶ms) override; + void process(StatisticsPtr &stats, Metadata *imageMetadata) override; + int setMode(std::string const &mode) override; + std::vector getChannels() const override; + +private: + std::map config_; + const HdrConfig *currentConfig_; +}; + +} /* namespace RPiController */