From patchwork Mon Sep 18 10:50:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 19057 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 38E7FC326C for ; Mon, 18 Sep 2023 10:50:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F069462942; Mon, 18 Sep 2023 12:50:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1695034234; bh=5o6snAwmMyZrdUu5jablI7TO1M+9BxAx8bHMSauCrgY=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=kO67BHKuBbYF9ciRGtM7JsHossM9UdLABOjVIQdmB6QUI7vVTBQ6In4PTyd8AAj4o AAI02JZxO95lXcfCIpEU5hJ9n9Tdouj5HLIH+oWnxJvSiAWPECRCZ+TUCFFPp3awJr i3sV3FBcPwuVOVThaYuD1DW4rcogrySTpeeNDM6EpMHnkPCDPt4imGYWnT36SWA5/S 1CxT4t+DeUQkPxfXGCuxSmlg0HxqjkSKWr0zeGck66EPjtA47yC5kvtYIzrK+8yI1F ioBXYOBhlZUSoTvOF9E5jVb0j7tYHTo+9I98pM68ZYswcIecG/Daxx+MKtvTsC4r5g 8oDwNshIENbzA== Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8C2306293D for ; Mon, 18 Sep 2023 12:50:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="dsorKgOw"; dkim-atps=neutral Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-31dca134c83so4375657f8f.3 for ; Mon, 18 Sep 2023 03:50:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1695034231; x=1695639031; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KRDkhyuWeq3rCXoVU2No3kNPzaL5NXFZPlh8z0oNz/Q=; b=dsorKgOwx7+fkzpZtI/9BKBaHfUuSmFX+LOjnFG1q0O3PeD917W7Vs1j2x6A6oGdFX swOvOo/E5QDuKh6argB9afTXd9hfRbJWuaJmVETLN7spLzkx+tixk1PW0aVYJj0N2nu/ bAzSlAN5I5fDv3MVfX8e+lN2BXOSOhMN5mHlKw5zjhwP+eaEhCgd9RHrb1tc+pQr7+7y z8H+B5bbspEkteSVn/KxZdFg/V+we96bNNDUuqkoFKzzrld6YtGpLmqr8W5fQND4Ihfr gxaTeeUeoCPo2CHODArMRPweFf2rLdBrYVwphU9jNO3TADNa1AI+NNnNrQYAlkFGddsE DCFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695034231; x=1695639031; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KRDkhyuWeq3rCXoVU2No3kNPzaL5NXFZPlh8z0oNz/Q=; b=rAl58bsd28FrvrtEpNxotLAJFaCnb04fADthlatJJpxSO4NAH18IEGGhTcIgyn2WDD WACZ/TDbm9WTtlhIYZ7raEgPAMpH262bHS94fUbr5umj7gUjTVRj3Dg2rAx9CxRBMcrF EsvdFqt2oolvWedzOOzKrUE/izNvvNujOE3ikrhg8o0jp6NJ4hcymoFc9Z1t/4IGTZVM Oz6ASNPJq/SSjelZx3MEfe1bIOdvTi1Gu6HXCRQi2qsZPSR3I+7YqZGxu6nwMvQ1ACXw uXsIxyW9RbgO4gRRlP/ZWe1qfcgHtViS4KIAGZFwEAs5TWFZijygY4CUn6i6uwECOjp8 dyAw== X-Gm-Message-State: AOJu0YzeMbbOIb3Quov1VQkOB6Ss0nPbGChzmLXTyI6fTKKb1uZkEY6e l3xPgEmlPa8c2GBLTlXTx0Qedps7l5CA7VAZOgk= X-Google-Smtp-Source: AGHT+IFrnkiMtekEgYqui6bEb9XIDBvYRw56kWAw3n08VwUXWtJHKWsaRK4oHep/FqqTwTkgpSyS5A== X-Received: by 2002:a5d:4706:0:b0:319:7c14:b06b with SMTP id y6-20020a5d4706000000b003197c14b06bmr6747568wrq.47.1695034231628; Mon, 18 Sep 2023 03:50:31 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:2bce:64d6:1a5c:49a2]) by smtp.gmail.com with ESMTPSA id o12-20020adfeacc000000b003176c6e87b1sm3415249wrn.81.2023.09.18.03.50.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 03:50:30 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Mon, 18 Sep 2023 11:50:26 +0100 Message-Id: <20230918105027.4170-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230918105027.4170-1-david.plowman@raspberrypi.com> References: <20230918105027.4170-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 1/2] ipa: rpi: Add an HDR algorithm to drive multi-channel AGC X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: David Plowman via libcamera-devel From: David Plowman Reply-To: David Plowman Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This HDR algorithm doesn't combine images in any way, but simply allows an application to drive the AGC in a multi-channel HDR type of mode, such as to produce short and long exposure images. Sufficient plumbing is added to enable the libcamera HDR controls and metadata to work. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck --- src/ipa/rpi/common/ipa_base.cpp | 49 +++++++++ src/ipa/rpi/controller/hdr_algorithm.h | 25 +++++ src/ipa/rpi/controller/hdr_status.h | 19 ++++ src/ipa/rpi/controller/meson.build | 1 + src/ipa/rpi/controller/rpi/hdr.cpp | 132 +++++++++++++++++++++++++ src/ipa/rpi/controller/rpi/hdr.h | 44 +++++++++ 6 files changed, 270 insertions(+) create mode 100644 src/ipa/rpi/controller/hdr_algorithm.h create mode 100644 src/ipa/rpi/controller/hdr_status.h create mode 100644 src/ipa/rpi/controller/rpi/hdr.cpp create mode 100644 src/ipa/rpi/controller/rpi/hdr.h diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index f7e7ad5e..231e8f96 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -24,6 +24,8 @@ #include "controller/ccm_status.h" #include "controller/contrast_algorithm.h" #include "controller/denoise_algorithm.h" +#include "controller/hdr_algorithm.h" +#include "controller/hdr_status.h" #include "controller/lux_status.h" #include "controller/sharpen_algorithm.h" #include "controller/statistics.h" @@ -67,6 +69,7 @@ const ControlInfoMap::Map ipaControls{ { &controls::AeFlickerPeriod, ControlInfo(100, 1000000) }, { &controls::Brightness, ControlInfo(-1.0f, 1.0f, 0.0f) }, { &controls::Contrast, ControlInfo(0.0f, 32.0f, 1.0f) }, + { &controls::HdrMode, ControlInfo(controls::HdrModeValues) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, { &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) }, { &controls::FrameDurationLimits, ControlInfo(INT64_C(33333), INT64_C(120000)) }, @@ -658,9 +661,17 @@ static const std::map AfPauseTable { controls::AfPauseResume, RPiController::AfAlgorithm::AfPauseResume }, }; +static const std::map HdrModeTable = { + { controls::HdrModeOff, "Off" }, + { controls::HdrModeMultiExposure, "MultiExposure" }, + { controls::HdrModeSingleExposure, "SingleExposure" }, +}; + void IpaBase::applyControls(const ControlList &controls) { + using RPiController::AgcAlgorithm; using RPiController::AfAlgorithm; + using RPiController::HdrAlgorithm; /* Clear the return metadata buffer. */ libcameraMetadata_.clear(); @@ -1157,6 +1168,34 @@ void IpaBase::applyControls(const ControlList &controls) break; } + case controls::HDR_MODE: { + HdrAlgorithm *hdr = dynamic_cast(controller_.getAlgorithm("hdr")); + if (!hdr) { + LOG(IPARPI, Warning) << "No HDR algorithm available"; + break; + } + + auto mode = HdrModeTable.find(ctrl.second.get()); + if (mode == HdrModeTable.end()) { + LOG(IPARPI, Warning) << "Unrecognised HDR mode"; + break; + } + + AgcAlgorithm *agc = dynamic_cast(controller_.getAlgorithm("agc")); + if (!agc) { + LOG(IPARPI, Warning) << "HDR requires an AGC algorithm"; + break; + } + + if (hdr->setMode(mode->second) == 0) + agc->setActiveChannels(hdr->getChannels()); + else + LOG(IPARPI, Warning) + << "HDR mode " << mode->second << " not supported"; + + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() @@ -1304,6 +1343,16 @@ void IpaBase::reportMetadata(unsigned int ipaContext) libcameraMetadata_.set(controls::AfPauseState, p); } + const HdrStatus *hdrStatus = rpiMetadata.getLocked("hdr.status"); + if (hdrStatus) { + if (hdrStatus->channel == "short") + libcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelShort); + else if (hdrStatus->channel == "long") + libcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelLong); + else + libcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelNone); + } + metadataReady.emit(libcameraMetadata_); } diff --git a/src/ipa/rpi/controller/hdr_algorithm.h b/src/ipa/rpi/controller/hdr_algorithm.h new file mode 100644 index 00000000..f622e099 --- /dev/null +++ b/src/ipa/rpi/controller/hdr_algorithm.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * hdr_algorithm.h - HDR control algorithm interface + */ +#pragma once + +#include + +#include "algorithm.h" + +namespace RPiController { + +class HdrAlgorithm : public Algorithm +{ +public: + HdrAlgorithm(Controller *controller) + : Algorithm(controller) {} + /* An HDR algorithm must provide the following: */ + virtual int setMode(std::string const &modeName) = 0; + virtual std::vector getChannels() const = 0; +}; + +} /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/hdr_status.h b/src/ipa/rpi/controller/hdr_status.h new file mode 100644 index 00000000..24b1a935 --- /dev/null +++ b/src/ipa/rpi/controller/hdr_status.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * hdr_status.h - HDR control algorithm status + */ +#pragma once + +#include + +/* + * The HDR algorithm process method should post an HdrStatus into the image + * metadata under the tag "hdr.status". + */ + +struct HdrStatus { + std::string mode; + std::string channel; +}; diff --git a/src/ipa/rpi/controller/meson.build b/src/ipa/rpi/controller/meson.build index 20b9cda9..c9a3e850 100644 --- a/src/ipa/rpi/controller/meson.build +++ b/src/ipa/rpi/controller/meson.build @@ -16,6 +16,7 @@ rpi_ipa_controller_sources = files([ 'rpi/contrast.cpp', 'rpi/dpc.cpp', 'rpi/geq.cpp', + 'rpi/hdr.cpp', 'rpi/lux.cpp', 'rpi/noise.cpp', 'rpi/sdn.cpp', diff --git a/src/ipa/rpi/controller/rpi/hdr.cpp b/src/ipa/rpi/controller/rpi/hdr.cpp new file mode 100644 index 00000000..97c7e949 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/hdr.cpp @@ -0,0 +1,132 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * hdr.cpp - HDR control algorithm + */ + +#include "hdr.h" + +#include + +#include "../agc_status.h" + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiHdr) + +#define NAME "rpi.hdr" + +void HdrConfig::read(const libcamera::YamlObject ¶ms, const std::string &modeName) +{ + name = modeName; + + if (!params.contains("cadence")) + LOG(RPiHdr, Fatal) << "No cadence for HDR mode " << name; + cadence = params["cadence"].getList().value(); + if (cadence.empty()) + LOG(RPiHdr, Fatal) << "Empty cadence in HDR mode " << name; + + /* + * In the JSON file it's easier to use the channel name as the key, but + * for us it's convenient to swap them over. + */ + for (const auto &[k, v] : params["channel_map"].asDict()) + channelMap[v.get().value()] = k; +} + +Hdr::Hdr(Controller *controller) + : HdrAlgorithm(controller), + currentConfig_(nullptr) +{ +} + +char const *Hdr::name() const +{ + return NAME; +} + +int Hdr::read(const libcamera::YamlObject ¶ms) +{ + /* Make an "HDR off" mode by default so that tuning files don't have to. */ + HdrConfig &offMode = config_["Off"]; + offMode.name = "Off"; + offMode.cadence = { 0 }; + currentConfig_ = &offMode; + + /* + * But we still allow the tuning file to override the "Off" mode if it wants. + * For example, maybe an application will make channel 0 be the "short" + * channel, in order to apply other AGC controls to it. + */ + for (const auto &[key, value] : params.asDict()) + config_[key].read(value, key); + + return 0; +} + +int Hdr::setMode(std::string const &mode) +{ + auto it = config_.find(mode); + if (it == config_.end()) { + LOG(RPiHdr, Warning) << "No such HDR mode " << mode; + return -1; + } + + currentConfig_ = &it->second; + + return 0; +} + +std::vector Hdr::getChannels() const +{ + return currentConfig_->cadence; +} + +void Hdr::switchMode([[maybe_unused]] CameraMode const &cameraMode, + [[maybe_unused]] Metadata *metadata) +{ +} + +void Hdr::process([[maybe_unused]] StatisticsPtr &stats, Metadata *imageMetadata) +{ + if (currentConfig_->name == "Off") + return; + + /* First find out what AGC channel this is, which comes from the delayed_status. */ + bool channelKnown = false; + unsigned int agcChannel = 0; + { + std::scoped_lock lock(*imageMetadata); + AgcStatus *agcStatus = imageMetadata->getLocked("agc.delayed_status"); + if (agcStatus) { + channelKnown = true; + agcChannel = agcStatus->channel; + } + } + + /* Fill out the HdrStatus information. */ + HdrStatus hdrStatus; + hdrStatus.mode = currentConfig_->name; + if (channelKnown) { + /* + * Channels that aren't required for HDR processing might come through for + * various reasons, such as when HDR starts up, or even because the cadence + * demands some frames that you need for other purposes (e.g. for preview). + * We'll leave the channel name empty in these cases. + */ + auto it = currentConfig_->channelMap.find(agcChannel); + if (it != currentConfig_->channelMap.end()) + hdrStatus.channel = it->second; + } + + imageMetadata->set("hdr.status", hdrStatus); +} + +/* Register algorithm with the system. */ +static Algorithm *create(Controller *controller) +{ + return (Algorithm *)new Hdr(controller); +} +static RegisterAlgorithm reg(NAME, &create); diff --git a/src/ipa/rpi/controller/rpi/hdr.h b/src/ipa/rpi/controller/rpi/hdr.h new file mode 100644 index 00000000..7419ad64 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/hdr.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * hdr.h - HDR control algorithm + */ +#pragma once + +#include +#include +#include + +#include "../hdr_algorithm.h" +#include "../hdr_status.h" + +/* This is our implementation of an HDR algorithm. */ + +namespace RPiController { + +struct HdrConfig { + std::string name; + std::vector cadence; + std::map channelMap; + + void read(const libcamera::YamlObject ¶ms, const std::string &name); +}; + +class Hdr : public HdrAlgorithm +{ +public: + Hdr(Controller *controller); + char const *name() const override; + void switchMode(CameraMode const &cameraMode, Metadata *metadata) override; + int read(const libcamera::YamlObject ¶ms) override; + void process(StatisticsPtr &stats, Metadata *imageMetadata) override; + int setMode(std::string const &mode) override; + std::vector getChannels() const override; + +private: + std::map config_; + const HdrConfig *currentConfig_; +}; + +} /* namespace RPiController */