From patchwork Fri Nov 8 09:21:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Qi X-Patchwork-Id: 21863 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 084B3BE173 for ; Fri, 8 Nov 2024 09:21:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1124965495; Fri, 8 Nov 2024 10:21:33 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=nxp.com header.i=@nxp.com header.b="DqiTFb3X"; dkim-atps=neutral Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on20629.outbound.protection.outlook.com [IPv6:2a01:111:f403:2612::629]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 25B1D60589 for ; Fri, 8 Nov 2024 10:21:31 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gA9Vu4VAK1LJ+SbIfRwgZ9EM+33PBFHrEk0yz3Mq4ha+MKM4y9LlKI8mPbo7+7JcIydHEL/gJPV5G3gmODKvDY/C0XXUvcyvEFYTTQwgxmdeSePpPqs6FfGIOrJogd7Dj2Upp7/+4F2mL+JA96EnzoDb30hwWmaMcgWN3OUp2TYVEw0Oq9FfjsrHvTUnBqI8zhvtcZ171o0W6voP7KL0T8vgtdoDFjCTQeO470XrBnbJbj46IFCLFAZq4ooTOgbMlP9IqXN5j9YxDQmAR//zGzg8FewUSkrkN3skMsm4cQgQGJ0eFtY1OzwB4KvDQnvhmPPHG40duT3NEpS/JzMKqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kaIU6ts2O45I8fJK8umwmc6jwO1wDNvY0khcSPjfjlI=; b=r/LxjqHFJny8TYQwC8q73mX4pLsfu+jlxZAc34MLZmrl+LgacvTMUpWoEjr7cmgnN/UK0Jbmaab7j2HU047pvOw4A1jbDdOUV+zD1uzomyQOZPEcQvtgx4RSxTIBRfrgWI4f7PqGBcRGIka4roRjHLSnS5wR8dm1Q6iTOApNJ3pd0h/XhXEN3XDIO06LyaFUxWqTpFLhlQU2gac2eWUA9rDvCpKJx+1FAzC0DjpAnQDnfPBbPvSRE1pOYSqeowx1ahFIGdZrSN/0tXoTCbWlnXAjoOvKhoHiFs865d4Mtm1CuSmeOPDJFNVOXjFedWlw95tHTWItH1sybO1C/299Ag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kaIU6ts2O45I8fJK8umwmc6jwO1wDNvY0khcSPjfjlI=; b=DqiTFb3XQq7dK0MYbFIGh6wOj6M0fdAOMFDTMXSGehm12MiRTayEW+/MplOmnaWSwKc3ovHLlV4d8nF2v2saeXwfhqwQh1d9c5lqMbn0/uGAnPsCRIA2/M9/7F7I+GpYDORWBQfee8FMh7GCDg7ojeTVbX1XmkmuFWFKdHrZO9V8XAiVRukpmjSEANMc93AILLKDLlLjnpz1M9hjyjEIab1jc4qhZx7umEh9OGrn3o7LpjjxqB/zawm4wUgKiU/JSvNvMNoy14tIPYAds8f9hGoZUd7YLWyI2HMOKnZPO/Q5qoesD47CHwlUGy/0JjkgRnoJcwm3vOVsLdnk0Z9q5g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB8285.eurprd04.prod.outlook.com (2603:10a6:102:1ca::15) by GVXPR04MB9851.eurprd04.prod.outlook.com (2603:10a6:150:118::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8137.22; Fri, 8 Nov 2024 09:21:29 +0000 Received: from PAXPR04MB8285.eurprd04.prod.outlook.com ([fe80::e003:8fb:64ea:acfd]) by PAXPR04MB8285.eurprd04.prod.outlook.com ([fe80::e003:8fb:64ea:acfd%3]) with mapi id 15.20.8137.018; Fri, 8 Nov 2024 09:21:29 +0000 From: Hou Qi To: libcamera-devel@lists.libcamera.org Cc: jared.hu@nxp.com, qi.hou@nxp.com, julien.vuillaumier@nxp.com Subject: [PATCH v2] gstreamer: Add GstVideoMeta support Date: Fri, 8 Nov 2024 18:21:13 +0900 Message-Id: <20241108092113.3769844-1-qi.hou@nxp.com> X-Mailer: git-send-email 2.34.1 X-ClientProxiedBy: SI1PR02CA0002.apcprd02.prod.outlook.com (2603:1096:4:1f7::16) To PAXPR04MB8285.eurprd04.prod.outlook.com (2603:10a6:102:1ca::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR04MB8285:EE_|GVXPR04MB9851:EE_ X-MS-Office365-Filtering-Correlation-Id: 2e618583-279c-46a4-02e9-08dcffd6b913 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|366016|376014|52116014|38350700014; X-Microsoft-Antispam-Message-Info: LonvvLH2qpGizXtYEjCHd9VrIwWq3GpGkjrnm7maxr0pM6zWntZJPzacf781cnx70MTzw6Qp1pCZZvG4dQhoxl58LpSYTgiyzMk5dtSfO4Vhfki8gKxs/rFH/Wyyrg/FpOI3bZBZT2yYbwW/VCJPKWmYXDYqQ6mOdggMPxY+2tCcy5NFCFEnymaWuy476pmMaHia+kGD9XZ2e13n//QiQTUP2Zfo4Hq4CakLqIDXl8cQtDmMN3w9xmaAG5OcO27FyAYd9mbzIF3Zv0/OgKsQ5UTgLZG3Dwq45FSa0AboJr/PpJPlbdkRbf27dOJlTxtQwGIurZP/N0AE0foESdhDPb9hOw0ldyj14WplM2UENdwhW9lk0bfNDLRQ7rog+y8bB6rN8Qzko9fIdUy8Y/iuTQMu3LMmkbFsXuFL633bJt1zWM5yReViecAXYaXX9J3yfulC09M5x3HqRlvOkgElA9JZ9+61Sh/mwJ6bP/rTbstOGXnklZXssk2Y0lNv3Dt7T/esVLEy1jboXz/0erdUYNZoyNRrkaqEOyi/umRBjG0m0ddNqmfMMtNemWjN+nApbFd1AVCin9H/3sGgqYeHll0YrpnpYCXQGD7IuOIveFPudNAEJCsyrJlSi+aXvQK0cg29AbQD+4ZRU3xY+5H38WYmuS8SlEB2fjKAt2iEt1RMnskRIlIQDFuxZrGMnyh5Dq+x5PjBj1geyE3MhJnePGPbCO6smecOlZVfpomMX4GU7tfcDG0C7Ea1UKyXoeG2Pqy84xvVgCElzWBccne6pFZCSeCF+7cVr9WK4xRG1FqV7yaaQbqBmyJSByA3NWyjK11qyQU8krrJ2CHER+cWd1/YYs7vZoC1WeJeYdKU6yz08uVUQpx49B2U0/pDnLf8au1i29/sYNLc5YRtCBU/VcIdFXUyIHmkTXf9ROvoBZoUN9U/ha8E1LazAd168LLc1SmQCZyZueCom3FwCOdchYCU0bGg5icOzmpXobJeDKK+kwnCJikzmtY7HTzfhIyRS+xqaqwb4xOIbOcK10O6BUvfaAdii+DKeCon13je7sjxwjvFsxyGicX+cM5PMwlh6h7bu1+ACtqpZirHPJTNI8Z/zEJfx8tO+Vumq+0D2yv3VEKh43Ogavv1w5J/ZVbcrbXOe6BuUjyHRkCuYY8alRsRV/5baLJwpyMza23Xrvwj9QB6kthYLH55qoQyjN1penK3jQ38gEFtOU70+bT0zcisebY1fkU7fvBWw/csKRygc3Jt+OD717i2wZ7c8KH13fnqbbZk9RHG5ZHMbGB0pXxTHFIBA5ebfVXsYkSW7fhMD1Sny6ybiEORrs/8zK51rv8a6toBkaS/KORItJR8cCjlEx7Le7aqKR7pTyVDyuQ= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR04MB8285.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(52116014)(38350700014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2qX5IAkV4d8YoW5aiFlRGCBGhtTa1eYJ9PHEh0amqbbZNPqu46y+37DXtsai9w6gOSRWy7bf5a3LlgYp+0DTon03MjCy+QxyFhdL53TlfSU49vb3iJTeGesVKwQC87I0Y1vxLZF6UQmUd3Len98r0n7zCEl5F+dvnpGKWf61i5OBw03HJEpfGGICijEMfpcDXqUlIkzV8327fms3lNmVN3NhV6krxww67PH//EOI1ox1Y/zE8Q6O0zYHFd6i6636Os4atjBo3yYgzk6Sva60h4MNrpkv6Hh4VmUZg9yvFHUzQrIbcX3z5OqEHzM8aMkKC5z51Wfdq9hsEUMSkJArewmlVUR4/tcA9C4JKjNaZtkXzwxZOHiPZC/xO3yzNBTd4WODWO44x0OmNtr4ZPzsOI623vDOacDxELEkeej1N4O/uPMSzFNebAASeRSsDf47iwEHO4Vn9LNGYKR3rOkzE5Xpk+vQk87mDICAzeUSXbY+99c3h8w8b78Driv5ieK0DIyYSWTPhmuHlBlUcC+Q4tMKwX0a0/TlX9N0CDR57gu+MnD8RHueOGnEJP2H1JSSmwlq1ULh+mw6XR+K0WZBMQHl4qTzqAHVqz2+DJ/YRCoSx387tShkcpH8K2OXbU/NDNEPK/0K/NdRP9/VL6FZC3Jdv9dR3Dmw6BuIOVhJ2W6Rj6QUxtXHOUbhk3DSkWgIFVVTw+uMSTYBtnQ3A77mEUpwbNXU7sphmnJNawjTteEvboXnuBSDMcnFiWWWRf6d8+138aqVAaBCqKEgY8zphNWWD1s9MMyyCyiK7gDw9i+SXUiQdVEcIbQbJyGct7PNoJjM/S0B9EAfnsuQQbo/O7F+f6frB+6Ll6m5tmAeUtBQcsKFijSdfMyg2qZgBm7BYeMg/wTJIfjQf/mNq1GE8zRXhcu6ylHRFSgErVwFiP/rsNml1Po1ZBBt3RG3GMU8xbcpBNiYGpsB3YSgyadJDHCEA8pmu2ZhMvg4yPeQXSL4Nbmt9MmImnlUZqV9ERu6wcYgvkJd553YtWYQauaNbaz8/O3Ojk56edZzshrR3SgJ0teXkQbC+YXd5ewzIUecO0htuyUHchwxD5FbpVts74Ipu4/mmFDRv3QfOixhK+/G0klUJ4VF1Fy3/A698Vaf0rBwNkkqjn7rMjv3gNJSx3B+Z7OvlDVbx57V+/7/BvShlAmueBIm5Q9bpIxPbUssE/Tyezpesl2xCW52lPU4iSKY+yWsvy9RoGWDaKsasqzXQHLKMQXs1mtqga+hLBOGVJHkVF2yJumAtdAv4MVFBlrhATDJteG1tS5tLO9KBJ9IexKk+QuhqEkzPyWX/zTnbyAiDZLzJ1TB2Bmn94gY0VICOeS60fTBwUgCStgAS07P6XHU2eAj34d64M3A2tBbbllJUo4c0YdLDtn3FyhCg04K+/G51dOF83BADYuIR2m6yiUbMmUSBb2cegByYMxwZ6LRYCps4HBkO70FqurIw10xhbBr7q2V7BMnCgdI+bGf5/enQgKT2nj1dly2jdu5AKllWR/7kF/t6DyMIcpAPcE+ate35NAX693rz8qOYLneRfscz+6cmWXUUlelxtUP X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2e618583-279c-46a4-02e9-08dcffd6b913 X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB8285.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2024 09:21:29.0233 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: a1G87TjsGPzkhOFr0mB00z/TWXnSpecDXgP8z+TD8TvYR1RNc/+P1HGE5fkBNm9d X-MS-Exchange-Transport-CrossTenantHeadersStamped: GVXPR04MB9851 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" GStreamer video-info calculated stride and offset may differ from those used by the camera. This patch enhances downstream plugin's support for videometa by adding videometa support for libcamerasrc. It ensures that when downstream plugin supports videometa by allocation query, libcamerasrc also attaches videometa to buffer, preventing discrepancies between the stride and offset calculated by video-info and camera. Signed-off-by: Hou Qi --- src/gstreamer/gstlibcamerapool.cpp | 28 ++++++++++++++++++++++++---- src/gstreamer/gstlibcamerapool.h | 2 +- src/gstreamer/gstlibcamerasrc.cpp | 14 +++++++++++++- 3 files changed, 38 insertions(+), 6 deletions(-) diff --git a/src/gstreamer/gstlibcamerapool.cpp b/src/gstreamer/gstlibcamerapool.cpp index 9cd7eccb..6593b3ca 100644 --- a/src/gstreamer/gstlibcamerapool.cpp +++ b/src/gstreamer/gstlibcamerapool.cpp @@ -135,16 +135,36 @@ gst_libcamera_pool_class_init(GstLibcameraPoolClass *klass) } GstLibcameraPool * -gst_libcamera_pool_new(GstLibcameraAllocator *allocator, Stream *stream) +gst_libcamera_pool_new(GstLibcameraAllocator *allocator, const StreamConfiguration &stream_cfg, + GstCaps *caps, gboolean add_video_meta) { auto *pool = GST_LIBCAMERA_POOL(g_object_new(GST_TYPE_LIBCAMERA_POOL, nullptr)); + GstVideoInfo info; pool->allocator = GST_LIBCAMERA_ALLOCATOR(g_object_ref(allocator)); - pool->stream = stream; - - gsize pool_size = gst_libcamera_allocator_get_pool_size(allocator, stream); + pool->stream = stream_cfg.stream(); + + if (caps && gst_video_info_from_caps(&info, caps)) { + guint k, stride; + gsize offset = 0; + for (k = 0; k < GST_VIDEO_INFO_N_PLANES(&info); k++) { + stride = gst_video_format_info_extrapolate_stride(info.finfo, k, stream_cfg.stride); + info.stride[k] = stride; + info.offset[k] = offset; + offset += stride * GST_VIDEO_FORMAT_INFO_SCALE_HEIGHT(info.finfo, k, GST_VIDEO_INFO_HEIGHT(&info)); + } + } else + add_video_meta = false; + + gsize pool_size = gst_libcamera_allocator_get_pool_size(allocator, stream_cfg.stream()); for (gsize i = 0; i < pool_size; i++) { GstBuffer *buffer = gst_buffer_new(); + if (add_video_meta) { + gst_buffer_add_video_meta_full(buffer, GST_VIDEO_FRAME_FLAG_NONE, + GST_VIDEO_INFO_FORMAT(&info), GST_VIDEO_INFO_WIDTH(&info), + GST_VIDEO_INFO_HEIGHT(&info), GST_VIDEO_INFO_N_PLANES(&info), + info.offset, info.stride); + } pool->queue->push_back(buffer); } diff --git a/src/gstreamer/gstlibcamerapool.h b/src/gstreamer/gstlibcamerapool.h index 2a7a9c77..8ad87cab 100644 --- a/src/gstreamer/gstlibcamerapool.h +++ b/src/gstreamer/gstlibcamerapool.h @@ -21,7 +21,7 @@ G_DECLARE_FINAL_TYPE(GstLibcameraPool, gst_libcamera_pool, GST_LIBCAMERA, POOL, GstBufferPool) GstLibcameraPool *gst_libcamera_pool_new(GstLibcameraAllocator *allocator, - libcamera::Stream *stream); + const libcamera::StreamConfiguration &stream_cfg, GstCaps *caps, gboolean add_video_meta); libcamera::Stream *gst_libcamera_pool_get_stream(GstLibcameraPool *self); diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 8efa25f4..c05a31e7 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -497,9 +497,21 @@ gst_libcamera_src_negotiate(GstLibcameraSrc *self) for (gsize i = 0; i < state->srcpads_.size(); i++) { GstPad *srcpad = state->srcpads_[i]; const StreamConfiguration &stream_cfg = state->config_->at(i); + GstQuery *query = NULL; + gboolean add_video_meta = false; + + g_autoptr(GstCaps) caps = gst_libcamera_stream_configuration_to_caps(stream_cfg); + gst_libcamera_framerate_to_caps(caps, element_caps); + + query = gst_query_new_allocation(caps, false); + if (!gst_pad_peer_query(srcpad, query)) + GST_DEBUG_OBJECT(self, "didn't get downstream ALLOCATION hints"); + else + add_video_meta = gst_query_find_allocation_meta(query, GST_VIDEO_META_API_TYPE, NULL); + gst_query_unref(query); GstLibcameraPool *pool = gst_libcamera_pool_new(self->allocator, - stream_cfg.stream()); + stream_cfg, caps, add_video_meta); g_signal_connect_swapped(pool, "buffer-notify", G_CALLBACK(gst_task_resume), self->task);