{"id":21549,"url":"https://patchwork.libcamera.org/api/patches/21549/?format=json","web_url":"https://patchwork.libcamera.org/patch/21549/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20241008231314.744556-4-kieran.bingham@ideasonboard.com>","date":"2024-10-08T23:13:13","name":"[v3,3/4] libcamera: internal: Add MediaPipeline helper","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"5fc69aaff5a253c31437d55150cd89d2c8f060f1","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/?format=json","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/21549/mbox/","series":[{"id":4668,"url":"https://patchwork.libcamera.org/api/series/4668/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4668","date":"2024-10-08T23:13:10","name":"MediaPipeline: Complex input device support","version":3,"mbox":"https://patchwork.libcamera.org/series/4668/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/21549/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/21549/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 35B9CC32DF\n\tfor ;\n\tTue, 8 Oct 2024 23:13:41 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B0FDD6536F;\n\tWed, 9 Oct 2024 01:13:39 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7ADC46536B\n\tfor ;\n\tWed, 9 Oct 2024 01:13:23 +0200 (CEST)","from Monstersaurus.tail69b4.ts.net\n\t(cpc89244-aztw30-2-0-cust6594.18-1.cable.virginm.net [86.31.185.195])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 559C24D4;\n\tWed, 9 Oct 2024 01:11:46 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"B6nD/cwA\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1728429106;\n\tbh=ws/WBB+UwyJShsgbTSLSIYDZv2rTxDZ+cd2b+o9J0P0=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=B6nD/cwAYMjvzmQ9N2NXNTDRwFpZ/GHQ002URnZYwmYFANzx05NdHAf1QgfM+0Cvk\n\tEljmecaI44mNCS4QJGKrcipHA1XXdC6N/1Dq3kjjlun9tZkgQV5bwDY2FDNQBAaOe7\n\tnoJ4+Qs/ttksZsJeVE1kXsmwv6vr6MJsHXs57Np4=","From":"Kieran Bingham ","To":"libcamera devel ","Cc":"Kieran Bingham ,\n\tUmang Jain ","Subject":"[PATCH v3 3/4] libcamera: internal: Add MediaPipeline helper","Date":"Wed, 9 Oct 2024 00:13:13 +0100","Message-Id":"<20241008231314.744556-4-kieran.bingham@ideasonboard.com>","X-Mailer":"git-send-email 2.34.1","In-Reply-To":"<20241008231314.744556-1-kieran.bingham@ideasonboard.com>","References":"<20241008231314.744556-1-kieran.bingham@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Provide a MediaPipeline class to help identifing and managing pipelines across\na MediaDevice graph.\n\nReviewed-by: Umang Jain \nSigned-off-by: Kieran Bingham \n\n---\nv2:\n\n- use srcPads to clearly identify which pads are managed\n- Only report enabling links when a change is made\n- fix header includes\n- Fix includes\n- Remove period at end of briefs\n- Document function parameters\n- expand documentation throughout\n- Fix debug log capitalisation\n- reduce scope of single use 'ret'\n\nv3:\n- Add doxygen documentation for the format parameter of configure()\n- Add 'format' identifier to configure parameter\n\n include/libcamera/internal/media_pipeline.h | 59 ++++\n include/libcamera/internal/meson.build | 1 +\n src/libcamera/media_pipeline.cpp | 311 ++++++++++++++++++++\n src/libcamera/meson.build | 1 +\n 4 files changed, 372 insertions(+)\n create mode 100644 include/libcamera/internal/media_pipeline.h\n create mode 100644 src/libcamera/media_pipeline.cpp","diff":"diff --git a/include/libcamera/internal/media_pipeline.h b/include/libcamera/internal/media_pipeline.h\nnew file mode 100644\nindex 000000000000..ee773b892719\n--- /dev/null\n+++ b/include/libcamera/internal/media_pipeline.h\n@@ -0,0 +1,59 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2024, Ideas on Board Oy\n+ *\n+ * Media pipeline handler\n+ */\n+\n+#pragma once\n+\n+#include \n+#include \n+#include \n+\n+namespace libcamera {\n+\n+class CameraSensor;\n+class MediaEntity;\n+class MediaLink;\n+class MediaPad;\n+struct V4L2SubdeviceFormat;\n+\n+class MediaPipeline\n+{\n+public:\n+\tint init(MediaEntity *source, std::string sink);\n+\tint initLinks();\n+\tint configure(CameraSensor *sensor, V4L2SubdeviceFormat *format);\n+\n+private:\n+\tstruct Entity {\n+\t\t/* The media entity, always valid. */\n+\t\tMediaEntity *entity;\n+\t\t/*\n+\t\t * Whether or not the entity is a subdev that supports the\n+\t\t * routing API.\n+\t\t */\n+\t\tbool supportsRouting;\n+\t\t/*\n+\t\t * The local sink pad connected to the upstream entity, null for\n+\t\t * the camera sensor at the beginning of the pipeline.\n+\t\t */\n+\t\tconst MediaPad *sink;\n+\t\t/*\n+\t\t * The local source pad connected to the downstream entity, null\n+\t\t * for the video node at the end of the pipeline.\n+\t\t */\n+\t\tconst MediaPad *source;\n+\t\t/*\n+\t\t * The link on the source pad, to the downstream entity, null\n+\t\t * for the video node at the end of the pipeline.\n+\t\t */\n+\t\tMediaLink *sourceLink;\n+\t};\n+\n+\tstd::vector routedSourcePads(MediaPad *sink);\n+\tstd::list entities_;\n+};\n+\n+} /* namespace libcamera */\ndiff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build\nindex 1c5eef9cab80..60a35d3e0357 100644\n--- a/include/libcamera/internal/meson.build\n+++ b/include/libcamera/internal/meson.build\n@@ -30,6 +30,7 @@ libcamera_internal_headers = files([\n 'mapped_framebuffer.h',\n 'media_device.h',\n 'media_object.h',\n+ 'media_pipeline.h',\n 'pipeline_handler.h',\n 'process.h',\n 'pub_key.h',\ndiff --git a/src/libcamera/media_pipeline.cpp b/src/libcamera/media_pipeline.cpp\nnew file mode 100644\nindex 000000000000..ec78b78e2f75\n--- /dev/null\n+++ b/src/libcamera/media_pipeline.cpp\n@@ -0,0 +1,311 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2024, Ideas on Board Oy\n+ *\n+ * Media pipeline support\n+ */\n+\n+#include \"libcamera/internal/media_pipeline.h\"\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \n+\n+#include \n+\n+#include \"libcamera/internal/camera_sensor.h\"\n+#include \"libcamera/internal/media_device.h\"\n+#include \"libcamera/internal/media_object.h\"\n+#include \"libcamera/internal/v4l2_subdevice.h\"\n+\n+/**\n+ * \\file media_pipeline.h\n+ * \\brief Provide a representation of a pipeline of devices using the Media\n+ * Controller\n+ */\n+\n+namespace libcamera {\n+\n+LOG_DEFINE_CATEGORY(MediaPipeline)\n+\n+/**\n+ * \\class MediaPipeline\n+ * \\brief The MediaPipeline represents a set of entities that together form a\n+ * data path for stream data\n+ *\n+ * A MediaPipeline instance is constructed from a sink and a source between\n+ * two entities in a media graph.\n+ */\n+\n+/**\n+ * \\brief Retrieve all source pads connected to a sink pad through active routes\n+ * \\param[in] sink The sink pad to examine\n+ *\n+ * Examine the entity using the V4L2 Subdevice Routing API to collect all the\n+ * source pads which are connected with an active route to the sink pad.\n+ *\n+ * \\return A vector of source MediaPads\n+ */\n+std::vector MediaPipeline::routedSourcePads(MediaPad *sink)\n+{\n+\tMediaEntity *entity = sink->entity();\n+\tstd::unique_ptr subdev =\n+\t\tstd::make_unique(entity);\n+\n+\tint ret = subdev->open();\n+\tif (ret < 0)\n+\t\treturn {};\n+\n+\tV4L2Subdevice::Routing routing = {};\n+\tret = subdev->getRouting(&routing, V4L2Subdevice::ActiveFormat);\n+\tif (ret < 0)\n+\t\treturn {};\n+\n+\tstd::vector pads;\n+\n+\tfor (const V4L2Subdevice::Route &route : routing) {\n+\t\tif (sink->index() != route.sink.pad ||\n+\t\t !(route.flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))\n+\t\t\tcontinue;\n+\n+\t\tconst MediaPad *pad = entity->getPadByIndex(route.source.pad);\n+\t\tif (!pad) {\n+\t\t\tLOG(MediaPipeline, Warning)\n+\t\t\t\t<< \"Entity \" << entity->name()\n+\t\t\t\t<< \" has invalid route source pad \"\n+\t\t\t\t<< route.source.pad;\n+\t\t}\n+\n+\t\tpads.push_back(pad);\n+\t}\n+\n+\treturn pads;\n+}\n+\n+/**\n+ * \\brief Find the path from source to sink\n+ * \\param[in] source The source entity to start from\n+ * \\param[in] sink The sink entity name to search for\n+ *\n+ * Starting from a source entity, determine the shortest path to the target\n+ * described by sink.\n+ *\n+ * If sink can not be found, or a route from source to sink can not be achieved,\n+ * an error of -ENOLINK will be returned.\n+ *\n+ * When successful, the MediaPipeline will internally store the representation\n+ * of entities and links to describe the path between the two entities.\n+ *\n+ * It is expected that the Source entity is a sensor represented by the\n+ * CameraSensor class.\n+ *\n+ * \\return 0 on success, a negative errno otherwise\n+ */\n+int MediaPipeline::init(MediaEntity *source, std::string sink)\n+{\n+\t/*\n+\t * Find the shortest path between from the source and the\n+\t * target entity.\n+\t */\n+\tstd::unordered_set visited;\n+\tstd::queue> queue;\n+\n+\t/* Remember at each entity where we came from. */\n+\tstd::unordered_map parents;\n+\tMediaEntity *entity = nullptr;\n+\tMediaEntity *target = nullptr;\n+\tMediaPad *sinkPad;\n+\n+\tqueue.push({ source, nullptr });\n+\n+\twhile (!queue.empty()) {\n+\t\tstd::tie(entity, sinkPad) = queue.front();\n+\t\tqueue.pop();\n+\n+\t\t/* Found the target device. */\n+\t\tif (entity->name() == sink) {\n+\t\t\tLOG(MediaPipeline, Debug)\n+\t\t\t\t<< \"Found pipeline target \" << entity->name();\n+\t\t\ttarget = entity;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tvisited.insert(entity);\n+\n+\t\t/*\n+\t\t * Add direct downstream entities to the search queue. If the\n+\t\t * current entity supports the subdev internal routing API,\n+\t\t * restrict the search to downstream entities reachable through\n+\t\t * active routes.\n+\t\t */\n+\n+\t\tstd::vector srcPads;\n+\t\tbool supportsRouting = false;\n+\n+\t\tif (sinkPad) {\n+\t\t\tsrcPads = routedSourcePads(sinkPad);\n+\t\t\tif (!srcPads.empty())\n+\t\t\t\tsupportsRouting = true;\n+\t\t}\n+\n+\t\tif (srcPads.empty()) {\n+\t\t\tfor (const MediaPad *pad : entity->pads()) {\n+\t\t\t\tif (!(pad->flags() & MEDIA_PAD_FL_SOURCE))\n+\t\t\t\t\tcontinue;\n+\t\t\t\tsrcPads.push_back(pad);\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (const MediaPad *srcPad : srcPads) {\n+\t\t\tfor (MediaLink *link : srcPad->links()) {\n+\t\t\t\tMediaEntity *next = link->sink()->entity();\n+\t\t\t\tif (visited.find(next) == visited.end()) {\n+\t\t\t\t\tqueue.push({ next, link->sink() });\n+\n+\t\t\t\t\tEntity e{ entity, supportsRouting, sinkPad,\n+\t\t\t\t\t\t srcPad, link };\n+\t\t\t\t\tparents.insert({ next, e });\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (!target) {\n+\t\tLOG(MediaPipeline, Error) << \"Failed to connect \" << source->name();\n+\t\treturn -ENOLINK;\n+\t}\n+\n+\t/*\n+\t * With the parents, we can follow back our way from the capture device\n+\t * to the sensor. Store all the entities in the pipeline, from the\n+\t * camera sensor to the video node, in entities_.\n+\t */\n+\tentities_.push_front({ entity, false, sinkPad, nullptr, nullptr });\n+\n+\tfor (auto it = parents.find(entity); it != parents.end();\n+\t it = parents.find(entity)) {\n+\t\tconst Entity &e = it->second;\n+\t\tentities_.push_front(e);\n+\t\tentity = e.entity;\n+\t}\n+\n+\tLOG(MediaPipeline, Info)\n+\t\t<< \"Found pipeline: \"\n+\t\t<< utils::join(entities_, \" -> \",\n+\t\t\t [](const Entity &e) {\n+\t\t\t\t std::string s = \"[\";\n+\t\t\t\t if (e.sink)\n+\t\t\t\t\t s += std::to_string(e.sink->index()) + \"|\";\n+\t\t\t\t s += e.entity->name();\n+\t\t\t\t if (e.source)\n+\t\t\t\t\t s += \"|\" + std::to_string(e.source->index());\n+\t\t\t\t s += \"]\";\n+\t\t\t\t return s;\n+\t\t\t });\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * \\brief Initialise and enable all links through the MediaPipeline\n+ * \\return 0 on success, or a negative errno otherwise\n+ */\n+int MediaPipeline::initLinks()\n+{\n+\tMediaLink *sinkLink = nullptr;\n+\tfor (Entity &e : entities_) {\n+\t\t/* Sensor entities have no connected sink. */\n+\t\tif (!sinkLink) {\n+\t\t\tsinkLink = e.sourceLink;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!(sinkLink->flags() & MEDIA_LNK_FL_ENABLED)) {\n+\t\t\tLOG(MediaPipeline, Debug) << \"Enabling : \" << *sinkLink;\n+\n+\t\t\tint ret = sinkLink->setEnabled(true);\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\t\t}\n+\n+\t\tsinkLink = e.sourceLink;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * \\brief Configure the entities of this MediaPipeline\n+ * \\param[in] sensor The configured CameraSensor to propogate\n+ * \\param[inout] format The format to propogate through the pipeline\n+ *\n+ * Propagate formats through each of the entities of the Pipeline, validating\n+ * that each one was not adjusted by the driver from the desired format.\n+ *\n+ * The format is updated with the final accepted format of the last entity of\n+ * the pipeline.\n+ *\n+ * \\return 0 on success or a negative errno otherwise\n+ */\n+int MediaPipeline::configure(CameraSensor *sensor, V4L2SubdeviceFormat *format)\n+{\n+\tint ret;\n+\n+\tfor (const Entity &e : entities_) {\n+\t\t/* The sensor is configured through the CameraSensor */\n+\t\tif (!e.sourceLink)\n+\t\t\tbreak;\n+\n+\t\tMediaLink *link = e.sourceLink;\n+\t\tMediaPad *source = link->source();\n+\t\tMediaPad *sink = link->sink();\n+\n+\t\t/* 'format' already contains the sensor configuration */\n+\t\tif (source->entity() != sensor->entity()) {\n+\t\t\t/* todo: Add MediaDevice cache to reduce FD pressure */\n+\t\t\tV4L2Subdevice subdev(source->entity());\n+\t\t\tret = subdev.open();\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n+\n+\t\t\tret = subdev.getFormat(source->index(), format);\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\t\t}\n+\n+\t\tV4L2SubdeviceFormat sourceFormat = *format;\n+\t\t/* todo: Add MediaDevice cache to reduce FD pressure */\n+\t\tV4L2Subdevice subdev(sink->entity());\n+\t\tret = subdev.open();\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tret = subdev.setFormat(sink->index(), format);\n+\t\tif (ret < 0)\n+\t\t\treturn ret;\n+\n+\t\tif (format->code != sourceFormat.code ||\n+\t\t format->size != sourceFormat.size) {\n+\t\t\tLOG(MediaPipeline, Debug)\n+\t\t\t\t<< \"Source '\" << *source\n+\t\t\t\t<< \" produces \" << sourceFormat\n+\t\t\t\t<< \", sink '\" << *sink\n+\t\t\t\t<< \" requires \" << format;\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\n+\t\tLOG(MediaPipeline, Debug)\n+\t\t\t<< \"Link \" << *link << \" configured with format \"\n+\t\t\t<< format;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+} /* namespace libcamera */\ndiff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\nindex aa9ab0291854..2c0f8603b231 100644\n--- a/src/libcamera/meson.build\n+++ b/src/libcamera/meson.build\n@@ -41,6 +41,7 @@ libcamera_internal_sources = files([\n 'mapped_framebuffer.cpp',\n 'media_device.cpp',\n 'media_object.cpp',\n+ 'media_pipeline.cpp',\n 'pipeline_handler.cpp',\n 'process.cpp',\n 'pub_key.cpp',\n","prefixes":["v3","3/4"]}