{"id":18916,"url":"https://patchwork.libcamera.org/api/patches/18916/?format=json","web_url":"https://patchwork.libcamera.org/patch/18916/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20230731134833.79687-2-david.plowman@raspberrypi.com>","date":"2023-07-31T13:48:32","name":"[libcamera-devel,1/2] ipa: rpi: Add an HDR algorithm to drive multi-channel AGC","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"71921cb5f7bbae70f05d33f135e3c8b455da6af8","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/?format=json","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/18916/mbox/","series":[{"id":3998,"url":"https://patchwork.libcamera.org/api/series/3998/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3998","date":"2023-07-31T13:48:31","name":"HDR for Raspberry Pi","version":1,"mbox":"https://patchwork.libcamera.org/series/3998/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/18916/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/18916/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 238D7C32A9\n\tfor ;\n\tMon, 31 Jul 2023 13:48:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6B502627F1;\n\tMon, 31 Jul 2023 15:48:42 +0200 (CEST)","from mail-wm1-x334.google.com (mail-wm1-x334.google.com\n\t[IPv6:2a00:1450:4864:20::334])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6ABF46037D\n\tfor ;\n\tMon, 31 Jul 2023 15:48:39 +0200 (CEST)","by mail-wm1-x334.google.com with SMTP id\n\t5b1f17b1804b1-3fbef8ad9bbso50846685e9.0\n\tfor ;\n\tMon, 31 Jul 2023 06:48:39 -0700 (PDT)","from pi4-davidp.pitowers.org\n\t([2a00:1098:3142:14:2bce:64d6:1a5c:49a2])\n\tby smtp.gmail.com with ESMTPSA id\n\tt20-20020a7bc3d4000000b003fe146fae45sm7046097wmj.46.2023.07.31.06.48.37\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tMon, 31 Jul 2023 06:48:37 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1690811322;\n\tbh=fidlZBZzrdssih8uSJdaw+xIadWMl0Ge0EaZN9Fq/nY=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=h+erlEmoY+rjZbEKEPd3FRjz9OI2ro//eMQTRrnqXWh4xOC8D5KNa3yU99TG4Vg70\n\tY+SejmjTf8LBmbOVsU7tLIWLHA1G81FIBWVOeCWMdu8s12xY0VwjvnTTJgnBaqXkeo\n\tlYHamuGp1w+FBwLwaqfAOpFpS8AVDaTklrBUrrFZRw4KKkO35/gIb5ffdPr5eoXPoZ\n\tYKq0+91NKQm6WlGqX+Cff6F7ZrAYUvjUUxhcLjYQ1G/ybVgnaDnlXbYkpNVJX/25oh\n\tufo19k/dMtkR9XPUthuIReG0bCZmTtInUq7vkDEdkv1pd0GhxylBGffXFkBOc7fV3V\n\tdhVghVTYyCkYA==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1690811319; x=1691416119;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=yD/SYwztMbkjVnlK0pC5AQQ6juvAZ96DKgFzs85l3Vs=;\n\tb=Y/yLTbn3xYrOsKOajucY23di8fX1btdWFSkYqCeR10GhjUdqaWUgO+33BoGmkshpEp\n\tQqYLM7l+3GAqWycB3qx1nHdMTT/+hTVBRU0RUP55BkchtPOy5x2EgSu0etbBzkgC07sP\n\tPuE7hdXuml+eEAq8Wn3cKQlJGwf3amNfdySefE6aBoc5/eiOp5SFvJftTZe9RJVwY893\n\thHDoTiTAgosHxNBOxaB8TlOTAFd18t9epFQlPQ1taB4fZcaqZRuskxaZScCd/Q2su491\n\tj28ymghGpNvqeJ6U1SAF9qirliD+8G0LgW/ZjWLaxoiRlwPN7VtinRbwjh++gzCqm+Qg\n\tkkHg=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=raspberrypi.com\n\theader.i=@raspberrypi.com\n\theader.b=\"Y/yLTbn3\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1690811319; x=1691416119;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=yD/SYwztMbkjVnlK0pC5AQQ6juvAZ96DKgFzs85l3Vs=;\n\tb=PsxZxKln3IIRZZwe+6WhWEJKEcqWrtk4I2b8MVoqIf6deTAGhtahaYlrMsz8vbtR0s\n\tXKaKMUd8bh5Zhqk+NqbbUv9QZhgVK0w9yzHQ+VJlhmOSw2qbbAuSUuhZvdEqEkVoQAE6\n\tEAZP38B+6H3WXycXPAngnsZXKItljFJUQpMi3sxmZsJsbnbk+fqZKWLG5KITfWoUtTFw\n\tspmnRR2olcJL3jmeIKk4k/BcfgPhaBqtqjQLZVHmZS7OVatm21d9SwXmXNz5ql2GYQoK\n\tj5jddP6RRiHOuKQ/rp6eQyrByVQOOR0Hg8Q9Vobp30mB6oQPBa6IMyDZmGedyo0hSVVq\n\txYFg==","X-Gm-Message-State":"ABy/qLYwwHt1QjrdoBdfEAbxVwFoZ17vni/lnwMLSgozhf9pQm749i1L\n\t0UXwjCXklwSsqFoon/de6BmxgShlRC+T3Nrk2UY=","X-Google-Smtp-Source":"APBJJlH1v1jJsroE/w9h/tvC8rYnjYh1n0xe75Rf18IktLihfRD+or5hGQMHyEkgR5isSbbNlqDu+w==","X-Received":"by 2002:a7b:c3d7:0:b0:3fc:3f31:4233 with SMTP id\n\tt23-20020a7bc3d7000000b003fc3f314233mr8306680wmj.38.1690811318217; \n\tMon, 31 Jul 2023 06:48:38 -0700 (PDT)","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 31 Jul 2023 14:48:32 +0100","Message-Id":"<20230731134833.79687-2-david.plowman@raspberrypi.com>","X-Mailer":"git-send-email 2.30.2","In-Reply-To":"<20230731134833.79687-1-david.plowman@raspberrypi.com>","References":"<20230731134833.79687-1-david.plowman@raspberrypi.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 1/2] ipa: rpi: Add an HDR algorithm to\n\tdrive multi-channel AGC","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"David Plowman via libcamera-devel ","Reply-To":"David Plowman ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"This HDR algorithm doesn't combine images in any way, but simply\nallows an application to drive the AGC in a multi-channel HDR type of\nmode, such as to produce short and long exposure images.\n\nSufficient plumbing is added to enable the libcamera HDR controls and\nmetadata to work.\n\nSigned-off-by: David Plowman \n---\n src/ipa/rpi/common/ipa_base.cpp | 49 ++++++++++\n src/ipa/rpi/controller/hdr_algorithm.h | 23 +++++\n src/ipa/rpi/controller/hdr_status.h | 25 +++++\n src/ipa/rpi/controller/meson.build | 1 +\n src/ipa/rpi/controller/rpi/hdr.cpp | 129 +++++++++++++++++++++++++\n src/ipa/rpi/controller/rpi/hdr.h | 42 ++++++++\n 6 files changed, 269 insertions(+)\n create mode 100644 src/ipa/rpi/controller/hdr_algorithm.h\n create mode 100644 src/ipa/rpi/controller/hdr_status.h\n create mode 100644 src/ipa/rpi/controller/rpi/hdr.cpp\n create mode 100644 src/ipa/rpi/controller/rpi/hdr.h","diff":"diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp\nindex f29c32fd..956b1ca0 100644\n--- a/src/ipa/rpi/common/ipa_base.cpp\n+++ b/src/ipa/rpi/common/ipa_base.cpp\n@@ -24,6 +24,8 @@\n #include \"controller/ccm_status.h\"\n #include \"controller/contrast_algorithm.h\"\n #include \"controller/denoise_algorithm.h\"\n+#include \"controller/hdr_algorithm.h\"\n+#include \"controller/hdr_status.h\"\n #include \"controller/lux_status.h\"\n #include \"controller/sharpen_algorithm.h\"\n #include \"controller/statistics.h\"\n@@ -63,6 +65,7 @@ const ControlInfoMap::Map ipaControls{\n \t{ &controls::ExposureValue, ControlInfo(-8.0f, 8.0f, 0.0f) },\n \t{ &controls::Brightness, ControlInfo(-1.0f, 1.0f, 0.0f) },\n \t{ &controls::Contrast, ControlInfo(0.0f, 32.0f, 1.0f) },\n+\t{ &controls::HdrMode, ControlInfo(controls::HdrModeValues) },\n \t{ &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) },\n \t{ &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) },\n \t{ &controls::FrameDurationLimits, ControlInfo(INT64_C(33333), INT64_C(120000)) },\n@@ -651,9 +654,17 @@ static const std::map AfPauseTable\n \t{ controls::AfPauseResume, RPiController::AfAlgorithm::AfPauseResume },\n };\n \n+static const std::map HdrModeTable = {\n+\t{ controls::HdrModeOff, \"Off\" },\n+\t{ controls::HdrModeMultiExposure, \"MultiExposure\" },\n+\t{ controls::HdrModeSingleExposure, \"SingleExposure\" },\n+};\n+\n void IpaBase::applyControls(const ControlList &controls)\n {\n+\tusing RPiController::AgcAlgorithm;\n \tusing RPiController::AfAlgorithm;\n+\tusing RPiController::HdrAlgorithm;\n \n \t/* Clear the return metadata buffer. */\n \tlibcameraMetadata_.clear();\n@@ -1092,6 +1103,34 @@ void IpaBase::applyControls(const ControlList &controls)\n \t\t\tbreak;\n \t\t}\n \n+\t\tcase controls::HDR_MODE: {\n+\t\t\tHdrAlgorithm *hdr = dynamic_cast(controller_.getAlgorithm(\"hdr\"));\n+\t\t\tif (!hdr) {\n+\t\t\t\tLOG(IPARPI, Warning) << \"No HDR algorithm available\";\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tauto mode = HdrModeTable.find(ctrl.second.get());\n+\t\t\tif (mode == HdrModeTable.end()) {\n+\t\t\t\tLOG(IPARPI, Warning) << \"Unrecognised HDR mode\";\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tAgcAlgorithm *agc = dynamic_cast(controller_.getAlgorithm(\"agc\"));\n+\t\t\tif (!agc) {\n+\t\t\t\tLOG(IPARPI, Warning) << \"HDR requires an AGC algorithm\";\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (hdr->setMode(mode->second) == 0)\n+\t\t\t\tagc->setActiveChannels(hdr->getChannels());\n+\t\t\telse\n+\t\t\t\tLOG(IPARPI, Warning)\n+\t\t\t\t\t<< \"HDR mode \" << mode->second << \" not supported\";\n+\n+\t\t\tbreak;\n+\t\t}\n+\n \t\tdefault:\n \t\t\tLOG(IPARPI, Warning)\n \t\t\t\t<< \"Ctrl \" << controls::controls.at(ctrl.first)->name()\n@@ -1239,6 +1278,16 @@ void IpaBase::reportMetadata(unsigned int ipaContext)\n \t\tlibcameraMetadata_.set(controls::AfPauseState, p);\n \t}\n \n+\tconst HdrStatus *hdrStatus = rpiMetadata.getLocked(\"hdr.status\");\n+\tif (hdrStatus) {\n+\t\tif (hdrStatus->channel == \"short\")\n+\t\t\tlibcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelShort);\n+\t\telse if (hdrStatus->channel == \"long\")\n+\t\t\tlibcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelLong);\n+\t\telse\n+\t\t\tlibcameraMetadata_.set(controls::HdrChannel, controls::HdrChannelNone);\n+\t}\n+\n \tmetadataReady.emit(libcameraMetadata_);\n }\n \ndiff --git a/src/ipa/rpi/controller/hdr_algorithm.h b/src/ipa/rpi/controller/hdr_algorithm.h\nnew file mode 100644\nindex 00000000..5164d50e\n--- /dev/null\n+++ b/src/ipa/rpi/controller/hdr_algorithm.h\n@@ -0,0 +1,23 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2023, Raspberry Pi Ltd\n+ *\n+ * hdr_algorithm.h - HDR control algorithm interface\n+ */\n+#pragma once\n+\n+#include \"algorithm.h\"\n+\n+namespace RPiController {\n+\n+class HdrAlgorithm : public Algorithm\n+{\n+public:\n+\tHdrAlgorithm(Controller *controller)\n+\t\t: Algorithm(controller) {}\n+\t/* An HDR algorithm must provide the following: */\n+\tvirtual int setMode(std::string const &modeName) = 0;\n+\tvirtual std::vector getChannels() const = 0;\n+};\n+\n+} /* namespace RPiController */\ndiff --git a/src/ipa/rpi/controller/hdr_status.h b/src/ipa/rpi/controller/hdr_status.h\nnew file mode 100644\nindex 00000000..26b538a0\n--- /dev/null\n+++ b/src/ipa/rpi/controller/hdr_status.h\n@@ -0,0 +1,25 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2022 Raspberry Pi Ltd\n+ *\n+ * hdr_status.h - HDR control algorithm status\n+ */\n+#pragma once\n+\n+#include \n+\n+/*\n+ * The HDR algorithm process method should post an HdrStatus into the image\n+ * metadata under the tag \"hdr.status\".\n+ */\n+\n+struct HdrStatus {\n+\tstd::string mode;\n+\tstd::string channel;\n+\n+\t/* Parameters for programming the stitch block (where available). */\n+\tbool stitch_enable;\n+\tuint16_t thresholdLo;\n+\tuint8_t diffPower;\n+\tdouble motionThreshold;\n+};\ndiff --git a/src/ipa/rpi/controller/meson.build b/src/ipa/rpi/controller/meson.build\nindex 20b9cda9..c9a3e850 100644\n--- a/src/ipa/rpi/controller/meson.build\n+++ b/src/ipa/rpi/controller/meson.build\n@@ -16,6 +16,7 @@ rpi_ipa_controller_sources = files([\n 'rpi/contrast.cpp',\n 'rpi/dpc.cpp',\n 'rpi/geq.cpp',\n+ 'rpi/hdr.cpp',\n 'rpi/lux.cpp',\n 'rpi/noise.cpp',\n 'rpi/sdn.cpp',\ndiff --git a/src/ipa/rpi/controller/rpi/hdr.cpp b/src/ipa/rpi/controller/rpi/hdr.cpp\nnew file mode 100644\nindex 00000000..2587f622\n--- /dev/null\n+++ b/src/ipa/rpi/controller/rpi/hdr.cpp\n@@ -0,0 +1,129 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2022 Raspberry Pi Ltd\n+ *\n+ * hdr.cpp - HDR control algorithm\n+ */\n+\n+#include \"hdr.h\"\n+\n+#include \n+\n+#include \"../agc_status.h\"\n+\n+using namespace RPiController;\n+using namespace libcamera;\n+\n+LOG_DEFINE_CATEGORY(RPiHdr)\n+\n+#define NAME \"rpi.hdr\"\n+\n+void HdrConfig::read(const libcamera::YamlObject ¶ms, const std::string &modeName)\n+{\n+\tname = modeName;\n+\n+\tif (!params.contains(\"cadence\"))\n+\t\tLOG(RPiHdr, Fatal) << \"No cadence for HDR mode \" << name;\n+\tcadence = params[\"cadence\"].getList().value();\n+\tif (cadence.empty())\n+\t\tLOG(RPiHdr, Fatal) << \"Empty cadence in HDR mode \" << name;\n+\n+\t/*\n+\t * In the JSON file it's easier to use the channel name as the key, but\n+\t * for us it's convenient to swap them over.\n+\t */\n+\tfor (const auto &[k, v] : params[\"channel_map\"].asDict())\n+\t\tchannelMap[v.get().value()] = k;\n+}\n+\n+Hdr::Hdr(Controller *controller)\n+\t: HdrAlgorithm(controller),\n+\t currentConfig_(nullptr)\n+{\n+}\n+\n+char const *Hdr::name() const\n+{\n+\treturn NAME;\n+}\n+\n+int Hdr::read(const libcamera::YamlObject ¶ms)\n+{\n+\tfor (const auto &[key, value] : params.asDict()) {\n+\t\tHdrConfig hdrConfig;\n+\t\thdrConfig.read(value, key);\n+\t\tconfig_[key] = std::move(hdrConfig);\n+\t\tif (!currentConfig_)\n+\t\t\tcurrentConfig_ = &config_[key];\n+\t}\n+\n+\tif (!currentConfig_)\n+\t\tLOG(RPiHdr, Fatal) << \"No HDR modes read\";\n+\n+\treturn 0;\n+}\n+\n+int Hdr::setMode(std::string const &mode)\n+{\n+\tauto it = config_.find(mode);\n+\tif (it == config_.end()) {\n+\t\tLOG(RPiHdr, Warning) << \"No such HDR mode \" << mode;\n+\t\treturn -1;\n+\t}\n+\n+\tcurrentConfig_ = &it->second;\n+\n+\treturn 0;\n+}\n+\n+std::vector Hdr::getChannels() const\n+{\n+\treturn currentConfig_->cadence;\n+}\n+\n+void Hdr::switchMode([[maybe_unused]] CameraMode const &cameraMode,\n+\t\t [[maybe_unused]] Metadata *metadata)\n+{\n+}\n+\n+void Hdr::process([[maybe_unused]] StatisticsPtr &stats, Metadata *imageMetadata)\n+{\n+\tif (currentConfig_->name == \"Off\")\n+\t\treturn;\n+\n+\t/* First find out what AGC channel this is, which comes from the delayed_status. */\n+\tbool channelKnown = false;\n+\tunsigned int agcChannel = 0;\n+\t{\n+\t\tstd::scoped_lock lock(*imageMetadata);\n+\t\tAgcStatus *agcStatus = imageMetadata->getLocked(\"agc.delayed_status\");\n+\t\tif (agcStatus) {\n+\t\t\tchannelKnown = true;\n+\t\t\tagcChannel = agcStatus->channel;\n+\t\t}\n+\t}\n+\n+\t/* Fill out the HdrStatus information. */\n+\tHdrStatus hdrStatus;\n+\thdrStatus.mode = currentConfig_->name;\n+\tif (channelKnown) {\n+\t\t/*\n+\t\t * Channels that aren't required for HDR processing might come through for\n+\t\t * various reasons, such as when HDR starts up, or even because the cadence\n+\t\t * demands some frames that you need for other purposes (e.g. for preview).\n+\t\t * We'll leave the channel name empty in these cases.\n+\t\t */\n+\t\tauto it = currentConfig_->channelMap.find(agcChannel);\n+\t\tif (it != currentConfig_->channelMap.end())\n+\t\t\thdrStatus.channel = it->second;\n+\t}\n+\n+\timageMetadata->set(\"hdr.status\", hdrStatus);\n+}\n+\n+/* Register algorithm with the system. */\n+static Algorithm *create(Controller *controller)\n+{\n+\treturn (Algorithm *)new Hdr(controller);\n+}\n+static RegisterAlgorithm reg(NAME, &create);\ndiff --git a/src/ipa/rpi/controller/rpi/hdr.h b/src/ipa/rpi/controller/rpi/hdr.h\nnew file mode 100644\nindex 00000000..8f6457f2\n--- /dev/null\n+++ b/src/ipa/rpi/controller/rpi/hdr.h\n@@ -0,0 +1,42 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2022, Raspberry Pi Ltd\n+ *\n+ * hdr.h - HDR control algorithm\n+ */\n+#pragma once\n+\n+#include \n+\n+#include \"../hdr_algorithm.h\"\n+#include \"../hdr_status.h\"\n+\n+/* This is our implementation of an HDR algorithm. */\n+\n+namespace RPiController {\n+\n+struct HdrConfig {\n+\tstd::string name;\n+\tstd::vector cadence;\n+\tstd::map channelMap;\n+\n+\tvoid read(const libcamera::YamlObject ¶ms, const std::string &name);\n+};\n+\n+class Hdr : public HdrAlgorithm\n+{\n+public:\n+\tHdr(Controller *controller);\n+\tchar const *name() const override;\n+\tvoid switchMode(CameraMode const &cameraMode, Metadata *metadata) override;\n+\tint read(const libcamera::YamlObject ¶ms) override;\n+\tvoid process(StatisticsPtr &stats, Metadata *imageMetadata) override;\n+\tint setMode(std::string const &mode) override;\n+\tstd::vector getChannels() const override;\n+\n+private:\n+\tstd::map config_;\n+\tconst HdrConfig *currentConfig_;\n+};\n+\n+} /* namespace RPiController */\n","prefixes":["libcamera-devel","1/2"]}