From patchwork Fri Oct 13 07:48:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 19136 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 84763C32BE for ; Fri, 13 Oct 2023 07:49:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0FC4C62986; Fri, 13 Oct 2023 09:49:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1697183344; bh=InJk5nE/LHRBOTGRWjRFoKav2SdmaIyuGdMs8XQP2Ek=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=AuFTsBtJf7UG715DjW7aBOCQ+bqPYsNooJrskZYnPjbp90VbKbvfjVvPEdb6xf+ic l6g2QwNxgD8ynnlsLJV9Id86NvOX9KHMBc/UttfCdotWN8PL/0RrCH3yRwOQADYmOP armPGtS5BR42Xfv1BV1s4Y8Y9/T/OxSM2jwSjw9uWcSkv32SeR0pPViVTBMGZdeGqX xq4Aapjx8QS4dKZ9uWRZPOSDCw5pzxdF/POVaCP3fECY1BkLq1RNpvb+Q+D1u0tTHG byU6BOG9IX3sC6Hlt5lW30CojRZoqIPWUY59YggI0gigNadA5JpXjcI7DEFGuAyNVY DS+3TbCeh6tzA== Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7BBE262985 for ; Fri, 13 Oct 2023 09:48:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="P0GdK46u"; dkim-atps=neutral Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-9adca291f99so269716266b.2 for ; Fri, 13 Oct 2023 00:48:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1697183335; x=1697788135; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VR+P5mQIrApbu0Xnw/5rpn4ma5izTxpcrx2OEmGcuNM=; b=P0GdK46u89r+6WZ19KATX+/h8tCl/e241FQ2t4wvmSS9Lw4fDMI9xWGngpsbTGwegx BQI/ghglm14/4Oj11OhfdTdNRp7kQ5EdnAoj2oB0vjIAtZKCoHZ+XBFHX32Ga4i1RGjZ JIPB0CYwRSIs6WHfB+tm3wQqsSKFt+aNV7F6bkWktS3ATW5SUT+iUcJZYXDEe0cnZLYE NwIv2yRq0BBBYRs+4aDapqyBoSbg2avrN8uNxlt8g2oLW+H6ax8Pb7Bl+DJT9+LL4HkT cynNuLej2A7Wia3MC13D7nNaBqYdGsiNJBl9hkXzJzPrFPZz/PRqc6Q3THRE3Bpobvyk Bc/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697183335; x=1697788135; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VR+P5mQIrApbu0Xnw/5rpn4ma5izTxpcrx2OEmGcuNM=; b=DzdqUT5pzxNRE1l1pSqI0IFk8oRpsfGckAy5k6Va42c3TPCLHzFZYULpZ52HQx/+/K b1+zfnPt/D08g2fT3S1kNK3RdRM0eM5xfCQ4v/PmG1k0K4p3i9zK55zc8RW279n2HJNB dgqWEFeC8L8rLmnXI3lfeFrukHeZ3LXBHpTG3uKJ2BRXl2tg3xfYkFnqp7S88b4QWOWL u3ESYLgzf1JvcWu+ZIFAjEqkeHg2Y5xto0qhtMJu5CAJA4Nf+MHSdJrbTLso8YGFtPRs uVJrIJsaPRvnWDKpTEzgQMqWWzGccf7uuRZT5KilLI/Jtj3ObOiEeai6yAHmaXTx4ZTq enJQ== X-Gm-Message-State: AOJu0YzF2oxM0p4WLnvVJLqZWwE2lNxKp3rz3qm5Z1QkRaZbEAcYx9Sj lM69JpmKpS117uvNWV4e19BCxhXta0cSIi1mhwUUSg== X-Google-Smtp-Source: AGHT+IHFq8vmCifoWhvGboCptFa00ahf602vhXu3HRe1bXrfJGbvv/5WCyYbdTFD3TQryFkjgpVUoA== X-Received: by 2002:a17:906:1d:b0:9b2:9a0e:9972 with SMTP id 29-20020a170906001d00b009b29a0e9972mr24309921eja.13.1697183334359; Fri, 13 Oct 2023 00:48:54 -0700 (PDT) Received: from localhost.localdomain ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id m16-20020a7bca50000000b003fee6e170f9sm1791890wml.45.2023.10.13.00.48.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Oct 2023 00:48:53 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 13 Oct 2023 08:48:32 +0100 Message-Id: <20231013074841.16972-12-naush@raspberrypi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231013074841.16972-1-naush@raspberrypi.com> References: <20231013074841.16972-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 11/20] ipa: rpi: Add new algorithms for PiSP X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Naushir Patuck via libcamera-devel From: Naushir Patuck Reply-To: Naushir Patuck Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add new CAC, HDR, Saturation and Tonemapping algorithms. Add a new Denoise algorithm that handles spatial/temporal/colour denoise through one interface. With this change, the old SDN algorithm is now considered deprecated and a warning message will be displayed if it is enabled. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman --- src/ipa/rpi/controller/agc_status.h | 3 + src/ipa/rpi/controller/cac_status.h | 16 ++ src/ipa/rpi/controller/denoise_status.h | 19 ++ src/ipa/rpi/controller/hdr_algorithm.h | 25 ++ src/ipa/rpi/controller/hdr_status.h | 19 ++ src/ipa/rpi/controller/meson.build | 5 + src/ipa/rpi/controller/rpi/cac.cpp | 81 +++++++ src/ipa/rpi/controller/rpi/cac.h | 38 +++ src/ipa/rpi/controller/rpi/denoise.cpp | 156 ++++++++++++ src/ipa/rpi/controller/rpi/denoise.h | 49 ++++ src/ipa/rpi/controller/rpi/hdr.cpp | 270 +++++++++++++++++++++ src/ipa/rpi/controller/rpi/hdr.h | 72 ++++++ src/ipa/rpi/controller/rpi/saturation.cpp | 57 +++++ src/ipa/rpi/controller/rpi/saturation.h | 32 +++ src/ipa/rpi/controller/rpi/sdn.cpp | 2 + src/ipa/rpi/controller/rpi/tonemap.cpp | 61 +++++ src/ipa/rpi/controller/rpi/tonemap.h | 35 +++ src/ipa/rpi/controller/saturation_status.h | 13 + src/ipa/rpi/controller/stitch_status.h | 17 ++ src/ipa/rpi/controller/tonemap_status.h | 17 ++ 20 files changed, 987 insertions(+) create mode 100644 src/ipa/rpi/controller/cac_status.h create mode 100644 src/ipa/rpi/controller/hdr_algorithm.h create mode 100644 src/ipa/rpi/controller/hdr_status.h create mode 100644 src/ipa/rpi/controller/rpi/cac.cpp create mode 100644 src/ipa/rpi/controller/rpi/cac.h create mode 100644 src/ipa/rpi/controller/rpi/denoise.cpp create mode 100644 src/ipa/rpi/controller/rpi/denoise.h create mode 100644 src/ipa/rpi/controller/rpi/hdr.cpp create mode 100644 src/ipa/rpi/controller/rpi/hdr.h create mode 100644 src/ipa/rpi/controller/rpi/saturation.cpp create mode 100644 src/ipa/rpi/controller/rpi/saturation.h create mode 100644 src/ipa/rpi/controller/rpi/tonemap.cpp create mode 100644 src/ipa/rpi/controller/rpi/tonemap.h create mode 100644 src/ipa/rpi/controller/saturation_status.h create mode 100644 src/ipa/rpi/controller/stitch_status.h create mode 100644 src/ipa/rpi/controller/tonemap_status.h diff --git a/src/ipa/rpi/controller/agc_status.h b/src/ipa/rpi/controller/agc_status.h index e5c4ee2239d9..68f899585740 100644 --- a/src/ipa/rpi/controller/agc_status.h +++ b/src/ipa/rpi/controller/agc_status.h @@ -10,6 +10,8 @@ #include +#include "hdr_status.h" + /* * The AGC algorithm process method should post an AgcStatus into the image * metadata under the tag "agc.status". @@ -37,6 +39,7 @@ struct AgcStatus { libcamera::utils::Duration fixedShutter; double fixedAnalogueGain; unsigned int channel; + HdrStatus hdr; }; struct AgcPrepareStatus { diff --git a/src/ipa/rpi/controller/cac_status.h b/src/ipa/rpi/controller/cac_status.h new file mode 100644 index 000000000000..475d4c5cc734 --- /dev/null +++ b/src/ipa/rpi/controller/cac_status.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * CAC (Chromatic Abberation Correction) algorithm status + */ +#pragma once + +#include "pwl.h" + +struct CacStatus { + std::vector lutRx; + std::vector lutRy; + std::vector lutBx; + std::vector lutBy; +}; diff --git a/src/ipa/rpi/controller/denoise_status.h b/src/ipa/rpi/controller/denoise_status.h index f6b9ee29dad6..4d2bd291f2f1 100644 --- a/src/ipa/rpi/controller/denoise_status.h +++ b/src/ipa/rpi/controller/denoise_status.h @@ -14,3 +14,22 @@ struct DenoiseStatus { double strength; unsigned int mode; }; + +struct SdnStatus { + double noiseConstant; + double noiseSlope; + double noiseConstant2; + double noiseSlope2; + double strength; +}; + +struct CdnStatus { + double strength; + double threshold; +}; + +struct TdnStatus { + double noiseConstant; + double noiseSlope; + double threshold; +}; diff --git a/src/ipa/rpi/controller/hdr_algorithm.h b/src/ipa/rpi/controller/hdr_algorithm.h new file mode 100644 index 000000000000..f622e099b6f5 --- /dev/null +++ b/src/ipa/rpi/controller/hdr_algorithm.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * hdr_algorithm.h - HDR control algorithm interface + */ +#pragma once + +#include + +#include "algorithm.h" + +namespace RPiController { + +class HdrAlgorithm : public Algorithm +{ +public: + HdrAlgorithm(Controller *controller) + : Algorithm(controller) {} + /* An HDR algorithm must provide the following: */ + virtual int setMode(std::string const &modeName) = 0; + virtual std::vector getChannels() const = 0; +}; + +} /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/hdr_status.h b/src/ipa/rpi/controller/hdr_status.h new file mode 100644 index 000000000000..24b1a9358871 --- /dev/null +++ b/src/ipa/rpi/controller/hdr_status.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * hdr_status.h - HDR control algorithm status + */ +#pragma once + +#include + +/* + * The HDR algorithm process method should post an HdrStatus into the image + * metadata under the tag "hdr.status". + */ + +struct HdrStatus { + std::string mode; + std::string channel; +}; diff --git a/src/ipa/rpi/controller/meson.build b/src/ipa/rpi/controller/meson.build index 20b9cda93661..32a4d31cfada 100644 --- a/src/ipa/rpi/controller/meson.build +++ b/src/ipa/rpi/controller/meson.build @@ -12,14 +12,19 @@ rpi_ipa_controller_sources = files([ 'rpi/alsc.cpp', 'rpi/awb.cpp', 'rpi/black_level.cpp', + 'rpi/cac.cpp', 'rpi/ccm.cpp', 'rpi/contrast.cpp', + 'rpi/denoise.cpp', 'rpi/dpc.cpp', 'rpi/geq.cpp', + 'rpi/hdr.cpp', 'rpi/lux.cpp', 'rpi/noise.cpp', + 'rpi/saturation.cpp', 'rpi/sdn.cpp', 'rpi/sharpen.cpp', + 'rpi/tonemap.cpp', ]) rpi_ipa_controller_deps = [ diff --git a/src/ipa/rpi/controller/rpi/cac.cpp b/src/ipa/rpi/controller/rpi/cac.cpp new file mode 100644 index 000000000000..7c123da1530a --- /dev/null +++ b/src/ipa/rpi/controller/rpi/cac.cpp @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * cac.cpp - Chromatic Aberration Correction algorithm + */ +#include "cac.h" + +#include + +#include "cac_status.h" + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiCac) + +#define NAME "rpi.cac" + +Cac::Cac(Controller *controller) + : Algorithm(controller) +{ +} + +char const *Cac::name() const +{ + return NAME; +} + +int Cac::read(const libcamera::YamlObject ¶ms) +{ + arrayToSet(params["lut_rx"], config_.lutRx); + arrayToSet(params["lut_ry"], config_.lutRy); + arrayToSet(params["lut_bx"], config_.lutBx); + arrayToSet(params["lut_by"], config_.lutBy); + cacStatus_.lutRx = config_.lutRx; + cacStatus_.lutRy = config_.lutRy; + cacStatus_.lutBx = config_.lutBx; + cacStatus_.lutBy = config_.lutBy; + double strength = params["strength"].get(1); + setStrength(config_.lutRx, cacStatus_.lutRx, strength); + setStrength(config_.lutBx, cacStatus_.lutBx, strength); + setStrength(config_.lutRy, cacStatus_.lutRy, strength); + setStrength(config_.lutBy, cacStatus_.lutBy, strength); + return 0; +} + +void Cac::initialise() +{ +} + +void Cac::arrayToSet(const libcamera::YamlObject ¶ms, std::vector &inputArray) +{ + int num = 0; + const Size &size = getHardwareConfig().cacRegions; + inputArray.resize((size.width + 1) * (size.height + 1)); + for (const auto &p : params.asList()) { + inputArray[num++] = p.get(0); + } +} + +void Cac::setStrength(std::vector &inputArray, std::vector &outputArray, + double strengthFactor) +{ + int num = 0; + for (const auto &p : inputArray) { + outputArray[num++] = p * strengthFactor; + } +} + +void Cac::prepare(Metadata *imageMetadata) +{ + imageMetadata->set("cac.status", cacStatus_); +} + +// Register algorithm with the system. +static Algorithm *Create(Controller *controller) +{ + return (Algorithm *)new Cac(controller); +} +static RegisterAlgorithm reg(NAME, &Create); diff --git a/src/ipa/rpi/controller/rpi/cac.h b/src/ipa/rpi/controller/rpi/cac.h new file mode 100644 index 000000000000..419180ab7d29 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/cac.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * cac.hpp - CAC control algorithm + */ +#pragma once + +#include "algorithm.h" +#include "cac_status.h" + +namespace RPiController { + +struct CacConfig { + std::vector lutRx; + std::vector lutRy; + std::vector lutBx; + std::vector lutBy; +}; + +class Cac : public Algorithm +{ +public: + Cac(Controller *controller = NULL); + char const *name() const override; + int read(const libcamera::YamlObject ¶ms) override; + void initialise() override; + void prepare(Metadata *imageMetadata) override; + void setStrength(std::vector &inputArray, std::vector &outputArray, + double strengthFactor); + +private: + CacConfig config_; + CacStatus cacStatus_; + void arrayToSet(const libcamera::YamlObject ¶ms, std::vector &inputArray); +}; + +} // namespace RPiController diff --git a/src/ipa/rpi/controller/rpi/denoise.cpp b/src/ipa/rpi/controller/rpi/denoise.cpp new file mode 100644 index 000000000000..440ee4425534 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/denoise.cpp @@ -0,0 +1,156 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022 Raspberry Pi Ltd + * + * Denoise.cpp - Denoise (spatial, colour, temporal) control algorithm + */ +#include "denoise.h" + +#include + +#include "denoise_status.h" +#include "noise_status.h" + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiDenoise) + +// Calculate settings for the denoise blocks using the noise profile in +// the image metadata. + +#define NAME "rpi.denoise" + +Denoise::Denoise(Controller *controller) + : DenoiseAlgorithm(controller), mode_(DenoiseMode::ColourHighQuality) +{ +} + +char const *Denoise::name() const +{ + return NAME; +} + +int Denoise::read(const libcamera::YamlObject ¶ms) +{ + sdnEnable_ = params.contains("sdn"); + if (sdnEnable_) { + auto &sdnParams = params["sdn"]; + sdnDeviation_ = sdnParams["deviation"].get(3.2); + sdnStrength_ = sdnParams["strength"].get(0.25); + sdnDeviation2_ = sdnParams["deviation2"].get(sdnDeviation_); + sdnDeviationNoTdn_ = sdnParams["deviation_no_tdn"].get(sdnDeviation_); + sdnStrengthNoTdn_ = sdnParams["strength_no_tdn"].get(sdnStrength_); + sdnTdnBackoff_ = sdnParams["backoff"].get(0.75); + } + + cdnEnable_ = params.contains("cdn"); + if (cdnEnable_) { + auto &cdnParams = params["cdn"]; + cdnDeviation_ = cdnParams["deviation"].get(120); + cdnStrength_ = cdnParams["strength"].get(0.2); + } + + tdnEnable_ = params.contains("tdn"); + if (tdnEnable_) { + auto &tdnParams = params["tdn"]; + tdnDeviation_ = tdnParams["deviation"].get(0.5); + tdnThreshold_ = tdnParams["threshold"].get(0.75); + } else if (sdnEnable_) { + /* + * If SDN is enabled but TDN isn't, overwrite all the SDN settings + * with the "no TDN" versions. This makes it easier to enable or + * disable TDN in the tuning file without editing all the other + * parameters. + */ + sdnDeviation_ = sdnDeviation2_ = sdnDeviationNoTdn_; + sdnStrength_ = sdnStrengthNoTdn_; + } + + return 0; +} + +void Denoise::initialise() +{ +} + +void Denoise::switchMode([[maybe_unused]] CameraMode const &cameraMode, + [[maybe_unused]] Metadata *metadata) +{ + /* A mode switch effectively resets temporal denoise and it has to start over. */ + currentSdnDeviation_ = sdnDeviationNoTdn_; + currentSdnStrength_ = sdnStrengthNoTdn_; + currentSdnDeviation2_ = sdnDeviationNoTdn_; +} + +void Denoise::prepare(Metadata *imageMetadata) +{ + struct NoiseStatus noiseStatus = {}; + noiseStatus.noiseSlope = 3.0; // in case no metadata + if (imageMetadata->get("noise.status", noiseStatus) != 0) + LOG(RPiDenoise, Warning) << "no noise profile found"; + + LOG(RPiDenoise, Debug) + << "Noise profile: constant " << noiseStatus.noiseConstant + << " slope " << noiseStatus.noiseSlope; + + if (mode_ == DenoiseMode::Off) + return; + + if (sdnEnable_) { + struct SdnStatus sdn; + sdn.noiseConstant = noiseStatus.noiseConstant * currentSdnDeviation_; + sdn.noiseSlope = noiseStatus.noiseSlope * currentSdnDeviation_; + sdn.noiseConstant2 = noiseStatus.noiseConstant * sdnDeviation2_; + sdn.noiseSlope2 = noiseStatus.noiseSlope * currentSdnDeviation2_; + sdn.strength = currentSdnStrength_; + imageMetadata->set("sdn.status", sdn); + LOG(RPiDenoise, Debug) + << "const " << sdn.noiseConstant + << " slope " << sdn.noiseSlope + << " str " << sdn.strength + << " const2 " << sdn.noiseConstant2 + << " slope2 " << sdn.noiseSlope2; + + /* For the next frame, we back off the SDN parameters as TDN ramps up. */ + double f = sdnTdnBackoff_; + currentSdnDeviation_ = f * currentSdnDeviation_ + (1 - f) * sdnDeviation_; + currentSdnStrength_ = f * currentSdnStrength_ + (1 - f) * sdnStrength_; + currentSdnDeviation2_ = f * currentSdnDeviation2_ + (1 - f) * sdnDeviation2_; + } + + if (tdnEnable_) { + struct TdnStatus tdn; + tdn.noiseConstant = noiseStatus.noiseConstant * tdnDeviation_; + tdn.noiseSlope = noiseStatus.noiseSlope * tdnDeviation_; + tdn.threshold = tdnThreshold_; + imageMetadata->set("tdn.status", tdn); + LOG(RPiDenoise, Debug) + << "programmed tdn threshold " << tdn.threshold + << " constant " << tdn.noiseConstant + << " slope " << tdn.noiseSlope; + } + + if (cdnEnable_ && mode_ != DenoiseMode::ColourOff) { + struct CdnStatus cdn; + cdn.threshold = cdnDeviation_ * noiseStatus.noiseSlope + noiseStatus.noiseConstant; + cdn.strength = cdnStrength_; + imageMetadata->set("cdn.status", cdn); + LOG(RPiDenoise, Debug) + << "programmed cdn threshold " << cdn.threshold + << " strength " << cdn.strength; + } +} + +void Denoise::setMode(DenoiseMode mode) +{ + // We only distinguish between off and all other modes. + mode_ = mode; +} + +// Register algorithm with the system. +static Algorithm *Create(Controller *controller) +{ + return (Algorithm *)new Denoise(controller); +} +static RegisterAlgorithm reg(NAME, &Create); diff --git a/src/ipa/rpi/controller/rpi/denoise.h b/src/ipa/rpi/controller/rpi/denoise.h new file mode 100644 index 000000000000..88b37663e569 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/denoise.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022, Raspberry Pi Ltd + * + * denoise.hpp - Denoise (spatial, colour, temporal) control algorithm + */ +#pragma once + +#include "algorithm.h" +#include "denoise_algorithm.h" + +namespace RPiController { + +// Algorithm to calculate correct denoise settings. + +class Denoise : public DenoiseAlgorithm +{ +public: + Denoise(Controller *controller); + char const *name() const override; + int read(const libcamera::YamlObject ¶ms) override; + void initialise() override; + void switchMode(CameraMode const &cameraMode, Metadata *metadata) override; + void prepare(Metadata *imageMetadata) override; + void setMode(DenoiseMode mode) override; + +private: + double sdnDeviation_; + double sdnStrength_; + double sdnDeviation2_; + double sdnDeviationNoTdn_; + double sdnStrengthNoTdn_; + double sdnTdnBackoff_; + double cdnDeviation_; + double cdnStrength_; + double tdnDeviation_; + double tdnThreshold_; + DenoiseMode mode_; + bool tdnEnable_; + bool sdnEnable_; + bool cdnEnable_; + + /* SDN parameters attenuate over time if TDN is running. */ + double currentSdnDeviation_; + double currentSdnStrength_; + double currentSdnDeviation2_; +}; + +} // namespace RPiController diff --git a/src/ipa/rpi/controller/rpi/hdr.cpp b/src/ipa/rpi/controller/rpi/hdr.cpp new file mode 100644 index 000000000000..295e4c5f1c0a --- /dev/null +++ b/src/ipa/rpi/controller/rpi/hdr.cpp @@ -0,0 +1,270 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * hdr.cpp - HDR control algorithm + */ + +#include "hdr.h" + +#include + +#include "../agc_status.h" +#include "../stitch_status.h" +#include "../tonemap_status.h" + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiHdr) + +#define NAME "rpi.hdr" + +void HdrConfig::read(const libcamera::YamlObject ¶ms, const std::string &modeName) +{ + name = modeName; + + if (!params.contains("cadence")) + LOG(RPiHdr, Fatal) << "No cadence for HDR mode " << name; + cadence = params["cadence"].getList().value(); + if (cadence.empty()) + LOG(RPiHdr, Fatal) << "Empty cadence in HDR mode " << name; + + /* + * In the JSON file it's easier to use the channel name as the key, but + * for us it's convenient to swap them over. + */ + for (const auto &[k, v] : params["channel_map"].asDict()) + channelMap[v.get().value()] = k; + + /* Read any tonemap parameters. */ + tonemapEnable = params["tonemap_enable"].get(0); + detailConstant = params["detail_constant"].get(50); + detailSlope = params["detail_slope"].get(8.0); + iirStrength = params["iir_strength"].get(8.0); + strength = params["strength"].get(1.5); + + if (tonemapEnable) { + /* We need either an explicit tonemap, or the information to build them dynamically. */ + if (params.contains("tonemap")) { + if (tonemap.read(params["tonemap"])) + LOG(RPiHdr, Fatal) << "Failed to read tonemap in HDR mode " << name; + } else { + if (target.read(params["target"])) + LOG(RPiHdr, Fatal) << "Failed to read target in HDR mode " << name; + if (maxSlope.read(params["max_slope"])) + LOG(RPiHdr, Fatal) << "Failed to read max_slope in HDR mode " << name; + minSlope = params["min_slope"].get(1.0); + maxGain = params["max_gain"].get(64.0); + step = params["step"].get(0.05); + speed = params["speed"].get(0.5); + } + } + + /* Read any stitch parameters. */ + stitchEnable = params["stitch_enable"].get(0); + thresholdLo = params["threshold_lo"].get(50000); + motionThreshold = params["motion_threshold"].get(0.005); + diffPower = params["diff_power"].get(13); + if (diffPower > 15) + LOG(RPiHdr, Fatal) << "Bad diff_power value in HDR mode " << name; +} + +Hdr::Hdr(Controller *controller) + : HdrAlgorithm(controller) +{ +} + +char const *Hdr::name() const +{ + return NAME; +} + +int Hdr::read(const libcamera::YamlObject ¶ms) +{ + /* Make an "HDR off" mode by default so that tuning files don't have to. */ + HdrConfig &offMode = config_["Off"]; + offMode.name = "Off"; + offMode.cadence = { 0 }; + offMode.channelMap[0] = "None"; + status_.mode = offMode.name; + delayedStatus_.mode = offMode.name; + + /* + * But we still allow the tuning file to override the "Off" mode if it wants. + * For example, maybe an application will make channel 0 be the "short" + * channel, in order to apply other AGC controls to it. + */ + for (const auto &[key, value] : params.asDict()) + config_[key].read(value, key); + + return 0; +} + +int Hdr::setMode(std::string const &mode) +{ + /* Always validate the mode, so it can be used later without checking. */ + auto it = config_.find(mode); + if (it == config_.end()) { + LOG(RPiHdr, Warning) << "No such HDR mode " << mode; + return -1; + } + + status_.mode = it->second.name; + + return 0; +} + +std::vector Hdr::getChannels() const +{ + return config_.at(status_.mode).cadence; +} + +void Hdr::updateAgcStatus(Metadata *metadata) +{ + std::scoped_lock lock(*metadata); + AgcStatus *agcStatus = metadata->getLocked("agc.status"); + if (agcStatus) { + HdrConfig &hdrConfig = config_[status_.mode]; + auto it = hdrConfig.channelMap.find(agcStatus->channel); + if (it != hdrConfig.channelMap.end()) { + status_.channel = it->second; + agcStatus->hdr = status_; + } else + LOG(RPiHdr, Warning) << "Channel " << agcStatus->channel + << " not found in mode " << status_.mode; + } else + LOG(RPiHdr, Warning) << "No agc.status found"; +} + +void Hdr::switchMode([[maybe_unused]] CameraMode const &cameraMode, Metadata *metadata) +{ + updateAgcStatus(metadata); + delayedStatus_ = status_; +} + +bool Hdr::updateTonemap(StatisticsPtr &stats, HdrConfig &config) +{ + /* When there's a change of HDR mode we start over with a new tonemap curve. */ + if (delayedStatus_.mode != previousMode_) { + previousMode_ = delayedStatus_.mode; + tonemap_ = Pwl(); + } + + /* No tonemapping. No need to output a tonemap.status. */ + if (!config.tonemapEnable) + return false; + + /* If an explicit tonemap was given, use it. */ + if (!config.tonemap.empty()) { + tonemap_ = config.tonemap; + return true; + } + + /* + * We only update the tonemap on short frames when in multi-exposure mode. But + * we still need to output the most recent tonemap. Possibly we should make the + * config indicate the channels for which we should update the tonemap? + */ + if (delayedStatus_.mode == "MultiExposure" && delayedStatus_.channel != "short") + return true; + + /* Build the tonemap dynamically using the image histogram. */ + Pwl tonemap; + tonemap.append(0, 0); + + double prev_input_val = 0; + double prev_output_val = 0; + const double step2 = config.step / 2; + for (double q = config.step; q < 1.0 - step2; q += config.step) { + double q_lo = std::max(0.0, q - step2); + double q_hi = std::min(1.0, q + step2); + double iqm = stats->yHist.interQuantileMean(q_lo, q_hi); + double input_val = std::min(iqm * 64, 65535.0); + + if (input_val > prev_input_val + 1) { + /* We're going to calcualte a Pwl to map input_val to this output_val. */ + double want_output_val = config.target.eval(q) * 65535; + /* But we must ensure we aren't applying too small or too great a local gain. */ + double want_slope = (want_output_val - prev_output_val) / (input_val - prev_input_val); + double slope = std::clamp(want_slope, config.minSlope, + config.maxSlope.eval(q)); + double output_val = prev_output_val + slope * (input_val - prev_input_val); + output_val = std::min(output_val, config.maxGain * input_val); + output_val = std::clamp(output_val, 0.0, 65535.0); + /* Let the tonemap adapte slightly more gently from frame to frame. */ + if (!tonemap_.empty()) { + double old_output_val = tonemap_.eval(input_val); + output_val = config.speed * output_val + + (1 - config.speed) * old_output_val; + } + LOG(RPiHdr, Debug) << "q " << q << " input " << input_val + << " output " << want_output_val << " slope " << want_slope + << " slope " << slope << " output " << output_val; + tonemap.append(input_val, output_val); + prev_input_val = input_val; + prev_output_val = output_val; + } + } + + tonemap.append(65535, 65535); + /* tonemap.debug(); */ + tonemap_ = tonemap; + + return true; +} + +void Hdr::process(StatisticsPtr &stats, Metadata *imageMetadata) +{ + /* Note what HDR channel this frame will be once it comes back to us. */ + updateAgcStatus(imageMetadata); + + /* + * Now figure out what HDR channel this frame is. It should be available in the + * agc.delayed_status, unless this is an early frame after a mode switch, in which + * case delayedStatus_ should be right. + */ + AgcStatus agcStatus; + if (!imageMetadata->get("agc.delayed_status", agcStatus)) + delayedStatus_ = agcStatus.hdr; + + auto it = config_.find(delayedStatus_.mode); + if (it == config_.end()) { + /* Shouldn't be possible. There would be nothing we could do. */ + LOG(RPiHdr, Warning) << "Unexpected HDR mode " << delayedStatus_.mode; + return; + } + + HdrConfig &config = it->second; + + if (updateTonemap(stats, config)) { + /* Add tonemap.status metadata. */ + TonemapStatus tonemapStatus; + + tonemapStatus.detailConstant = config.detailConstant; + tonemapStatus.detailSlope = config.detailSlope; + tonemapStatus.iirStrength = config.iirStrength; + tonemapStatus.strength = config.strength; + tonemapStatus.tonemap = tonemap_; + + imageMetadata->set("tonemap.status", tonemapStatus); + } + + if (config.stitchEnable) { + /* Add stitch.status metadata. */ + StitchStatus stitchStatus; + + stitchStatus.diffPower = config.diffPower; + stitchStatus.motionThreshold = config.motionThreshold; + stitchStatus.thresholdLo = config.thresholdLo; + + imageMetadata->set("stitch.status", stitchStatus); + } +} + +/* Register algorithm with the system. */ +static Algorithm *create(Controller *controller) +{ + return (Algorithm *)new Hdr(controller); +} +static RegisterAlgorithm reg(NAME, &create); diff --git a/src/ipa/rpi/controller/rpi/hdr.h b/src/ipa/rpi/controller/rpi/hdr.h new file mode 100644 index 000000000000..01ba45f1d3dc --- /dev/null +++ b/src/ipa/rpi/controller/rpi/hdr.h @@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * hdr.h - HDR control algorithm + */ +#pragma once + +#include +#include +#include + +#include "../hdr_algorithm.h" +#include "../hdr_status.h" +#include "../pwl.h" + +/* This is our implementation of an HDR algorithm. */ + +namespace RPiController { + +struct HdrConfig { + std::string name; + std::vector cadence; + std::map channelMap; + + /* Tonemap related parameters. */ + bool tonemapEnable; + uint16_t detailConstant; + double detailSlope; + double iirStrength; + double strength; + /* We must have either an explicit tonemap curve, or the other parameters. */ + Pwl tonemap; + Pwl target; /* maps histogram quatile to desired target output value */ + Pwl maxSlope; /* the maximum slope allowed at each point in the mapping */ + double minSlope; /* the minimum allowed slope */ + double maxGain; /* limit to the max absolute gain */ + double step; /* the histogram granularity for building the mapping */ + double speed; /* rate at which tonemap is updated */ + + /* Stitch related parameters. */ + bool stitchEnable; + uint16_t thresholdLo; + uint8_t diffPower; + double motionThreshold; + + void read(const libcamera::YamlObject ¶ms, const std::string &name); +}; + +class Hdr : public HdrAlgorithm +{ +public: + Hdr(Controller *controller); + char const *name() const override; + void switchMode(CameraMode const &cameraMode, Metadata *metadata) override; + int read(const libcamera::YamlObject ¶ms) override; + void process(StatisticsPtr &stats, Metadata *imageMetadata) override; + int setMode(std::string const &mode) override; + std::vector getChannels() const override; + +private: + void updateAgcStatus(Metadata *metadata); + bool updateTonemap(StatisticsPtr &stats, HdrConfig &config); + + std::map config_; + HdrStatus status_; /* track the current HDR mode and channel */ + HdrStatus delayedStatus_; /* track the delayed HDR mode and channel */ + std::string previousMode_; + Pwl tonemap_; +}; + +} /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/rpi/saturation.cpp b/src/ipa/rpi/controller/rpi/saturation.cpp new file mode 100644 index 000000000000..813540e5154d --- /dev/null +++ b/src/ipa/rpi/controller/rpi/saturation.cpp @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022 Raspberry Pi Ltd + * + * saturation.cpp - Saturation control algorithm + */ +#include "saturation.h" + +#include + +#include "saturation_status.h" + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiSaturation) + +#define NAME "rpi.saturation" + +Saturation::Saturation(Controller *controller) + : Algorithm(controller) +{ +} + +char const *Saturation::name() const +{ + return NAME; +} + +int Saturation::read(const libcamera::YamlObject ¶ms) +{ + config_.shiftR = params["shift_r"].get(0); + config_.shiftG = params["shift_g"].get(0); + config_.shiftB = params["shift_b"].get(0); + return 0; +} + +void Saturation::initialise() +{ +} + +void Saturation::prepare(Metadata *imageMetadata) +{ + SaturationStatus saturation; + + saturation.shiftR = config_.shiftR; + saturation.shiftG = config_.shiftG; + saturation.shiftB = config_.shiftB; + imageMetadata->set("saturation.status", saturation); +} + +// Register algorithm with the system. +static Algorithm *Create(Controller *controller) +{ + return (Algorithm *)new Saturation(controller); +} +static RegisterAlgorithm reg(NAME, &Create); diff --git a/src/ipa/rpi/controller/rpi/saturation.h b/src/ipa/rpi/controller/rpi/saturation.h new file mode 100644 index 000000000000..97da412ad59a --- /dev/null +++ b/src/ipa/rpi/controller/rpi/saturation.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022, Raspberry Pi Ltd + * + * saturation.hpp - Saturation control algorithm + */ +#pragma once + +#include "algorithm.h" + +namespace RPiController { + +struct SaturationConfig { + uint8_t shiftR; + uint8_t shiftG; + uint8_t shiftB; +}; + +class Saturation : public Algorithm +{ +public: + Saturation(Controller *controller = NULL); + char const *name() const override; + int read(const libcamera::YamlObject ¶ms) override; + void initialise() override; + void prepare(Metadata *imageMetadata) override; + +private: + SaturationConfig config_; +}; + +} // namespace RPiController diff --git a/src/ipa/rpi/controller/rpi/sdn.cpp b/src/ipa/rpi/controller/rpi/sdn.cpp index b6b662518f2c..6743919e6b36 100644 --- a/src/ipa/rpi/controller/rpi/sdn.cpp +++ b/src/ipa/rpi/controller/rpi/sdn.cpp @@ -36,6 +36,8 @@ char const *Sdn::name() const int Sdn::read(const libcamera::YamlObject ¶ms) { + LOG(RPiSdn, Warning) + << "Using legacy SDN tuning - please consider moving SDN inside rpi.denoise"; deviation_ = params["deviation"].get(3.2); strength_ = params["strength"].get(0.75); return 0; diff --git a/src/ipa/rpi/controller/rpi/tonemap.cpp b/src/ipa/rpi/controller/rpi/tonemap.cpp new file mode 100644 index 000000000000..5f8b2bf25aeb --- /dev/null +++ b/src/ipa/rpi/controller/rpi/tonemap.cpp @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022 Raspberry Pi Ltd + * + * tonemap.cpp - Tonemap control algorithm + */ +#include "tonemap.h" + +#include + +#include "tonemap_status.h" + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiTonemap) + +#define NAME "rpi.tonemap" + +Tonemap::Tonemap(Controller *controller) + : Algorithm(controller) +{ +} + +char const *Tonemap::name() const +{ + return NAME; +} + +int Tonemap::read(const libcamera::YamlObject ¶ms) +{ + config_.detailConstant = params["detail_constant"].get(0); + config_.detailSlope = params["detail_slope"].get(0.1); + config_.iirStrength = params["iir_strength"].get(1.0); + config_.strength = params["strength"].get(1.0); + config_.tonemap.read(params["tone_curve"]); + return 0; +} + +void Tonemap::initialise() +{ +} + +void Tonemap::prepare(Metadata *imageMetadata) +{ + TonemapStatus tonemapStatus; + + tonemapStatus.detailConstant = config_.detailConstant; + tonemapStatus.detailSlope = config_.detailSlope; + tonemapStatus.iirStrength = config_.iirStrength; + tonemapStatus.strength = config_.strength; + tonemapStatus.tonemap = config_.tonemap; + imageMetadata->set("tonemap.status", tonemapStatus); +} + +// Register algorithm with the system. +static Algorithm *Create(Controller *controller) +{ + return (Algorithm *)new Tonemap(controller); +} +static RegisterAlgorithm reg(NAME, &Create); diff --git a/src/ipa/rpi/controller/rpi/tonemap.h b/src/ipa/rpi/controller/rpi/tonemap.h new file mode 100644 index 000000000000..f25aa47f86c2 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/tonemap.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022, Raspberry Pi Ltd + * + * tonemap.hpp - Tonemap control algorithm + */ +#pragma once + +#include "algorithm.h" +#include "pwl.h" + +namespace RPiController { + +struct TonemapConfig { + uint16_t detailConstant; + double detailSlope; + double iirStrength; + double strength; + Pwl tonemap; +}; + +class Tonemap : public Algorithm +{ +public: + Tonemap(Controller *controller = NULL); + char const *name() const override; + int read(const libcamera::YamlObject ¶ms) override; + void initialise() override; + void prepare(Metadata *imageMetadata) override; + +private: + TonemapConfig config_; +}; + +} // namespace RPiController diff --git a/src/ipa/rpi/controller/saturation_status.h b/src/ipa/rpi/controller/saturation_status.h new file mode 100644 index 000000000000..337b66a3e91e --- /dev/null +++ b/src/ipa/rpi/controller/saturation_status.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022 Raspberry Pi Ltd + * + * saturation_status.h - Saturation control algorithm status + */ +#pragma once + +struct SaturationStatus { + uint8_t shiftR; + uint8_t shiftG; + uint8_t shiftB; +}; diff --git a/src/ipa/rpi/controller/stitch_status.h b/src/ipa/rpi/controller/stitch_status.h new file mode 100644 index 000000000000..b17800ed6697 --- /dev/null +++ b/src/ipa/rpi/controller/stitch_status.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2023 Raspberry Pi Ltd + * + * stitch_status.h - stitch control algorithm status + */ +#pragma once + +/* + * Parameters for the stitch block. + */ + +struct StitchStatus { + uint16_t thresholdLo; + uint8_t diffPower; + double motionThreshold; +}; diff --git a/src/ipa/rpi/controller/tonemap_status.h b/src/ipa/rpi/controller/tonemap_status.h new file mode 100644 index 000000000000..0e6399467869 --- /dev/null +++ b/src/ipa/rpi/controller/tonemap_status.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2022 Raspberry Pi Ltd + * + * hdr.h - Tonemap control algorithm status + */ +#pragma once + +#include "pwl.h" + +struct TonemapStatus { + uint16_t detailConstant; + double detailSlope; + double iirStrength; + double strength; + RPiController::Pwl tonemap; +};