From patchwork Wed Nov 13 07:29:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Hou X-Patchwork-Id: 21881 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 63F1BBE173 for ; Wed, 13 Nov 2024 07:30:38 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id ACF4D65800; Wed, 13 Nov 2024 08:30:37 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=nxp.com header.i=@nxp.com header.b="MhNSE9uF"; dkim-atps=neutral Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on20614.outbound.protection.outlook.com [IPv6:2a01:111:f403:2614::614]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AE2B6657FA for ; Wed, 13 Nov 2024 08:30:35 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nmOx98fJsH+k9fyNUdOvoA6aDgHD24uMvjFbZcN/w/4jrt5JyJHqnyqM6jNd6J831zkhMHwDC7EEB7RL467TWU6E2ILuJ6BgEyMopX3waaIGl6RJtXljKN4yDh5lSrMMcgqdjsYCVTMlyJ4MFnxqAqsuyCsA2wcHbf1eqHzQoByWSeLsn/5WhiCf9z+uz+7KYBz2ykcVm0snukOyBGvZlVh8fZ3Ef7jXUiHESQ9P/FDhMQiIdPl7RXylPnLj3hfw8jT0+pCl1yt+0EYGBQDA8OfJl/LYp5cEhMrtrX+R9QYUqLf1LHapT8Uw23DKxb+atWxfvOJ9ql8CGyIDAA6WfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dqF6enlbE4G0XW88dLHz3yvqwAsx5ZL8egpRlgK/jXQ=; b=oCRVF5KyUFxCpyqJGbP9zWUOhOgfy7t6DDWBLlP5w2esl0lIV7HcAqF0oBz2QXLjOl9hlFvVF8X+R7NMJCCbnS9aM6nqRPVgNb/MdZmHkb81J2YDy5F8A9g2ARReQ5d+zQhOMkRjgef75whZjWxAf09kMPmGw1N1NuYeYr5VwWsHUDKTrFWh5p6KTORBx+SJshLQZ1EYU3ac5jkQ81VQ3QYtWYW7hUgdS1Lsy/ZnfiauulDiB1z+NZyhjFI7+VbfXeaxlQAwSCdCOSKR04Pt/9NOFz1sIAkxR4vzJCSY/QZgFCoPJCy2trF53/EBUcoPc2eXvP1oTeUuZFb93KZULg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dqF6enlbE4G0XW88dLHz3yvqwAsx5ZL8egpRlgK/jXQ=; b=MhNSE9uFaXiwx/MFZazd+XrMQqkzFp3Kmu5FO7jqGVNR7QiRr2nGXcId6ssfepqRPSKuuUY063YahpnVKoMyCbyBnh3pHY+VwOiqoDlp65U+Z/VSsj17jX2gCyPBqEn3XtaBBw5bvVRPuypylIaMZsx1z6elQFbEBsB04sf0bHaN9TCj6SQlA0Y0yvSLAukADh3D+dnIIFJOQGtxtDYb9jHo1yjrWUPk2fHwLZqtXSS6zBKy6ggZfY1+zzyRE6RwMRonoQj256at4Jzsepg0vsSlQv3dWxHCrjXsW65hzk6j/vgAbllBpkkuz9L5Bt8K9fvX7ZfNGDfXxD8cNb6Qcg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB8285.eurprd04.prod.outlook.com (2603:10a6:102:1ca::15) by DB8PR04MB7132.eurprd04.prod.outlook.com (2603:10a6:10:12e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8137.29; Wed, 13 Nov 2024 07:30:33 +0000 Received: from PAXPR04MB8285.eurprd04.prod.outlook.com ([fe80::e003:8fb:64ea:acfd]) by PAXPR04MB8285.eurprd04.prod.outlook.com ([fe80::e003:8fb:64ea:acfd%3]) with mapi id 15.20.8158.013; Wed, 13 Nov 2024 07:30:32 +0000 From: Hou Qi To: libcamera-devel@lists.libcamera.org Cc: jared.hu@nxp.com, qi.hou@nxp.com, julien.vuillaumier@nxp.com Subject: [PATCH v3] gstreamer: Add GstVideoMeta support Date: Wed, 13 Nov 2024 16:29:47 +0900 Message-Id: <20241113072947.1579075-1-qi.hou@nxp.com> X-Mailer: git-send-email 2.34.1 X-ClientProxiedBy: SG2PR02CA0041.apcprd02.prod.outlook.com (2603:1096:3:18::29) To PAXPR04MB8285.eurprd04.prod.outlook.com (2603:10a6:102:1ca::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR04MB8285:EE_|DB8PR04MB7132:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e06de68-6ebc-462b-67b8-08dd03b50e99 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|52116014|376014|366016|38350700014; X-Microsoft-Antispam-Message-Info: eF5kKVudTYHsaRo26+3ccIZZImenkWrQgF4cpJU4L1OkKWumN60pNyJzZZJZPTpZ70XbnGI2YkXQ/5sv9MVGM9FiRJ1sBQY+yP/66XWAzMyfDTuRDsFRNcxhAAwcXIpGbMZ7wHoWLe1JeQowc8oTrB9+rsX3bU/vW5QfpRD6/LJvs5xNs7NFk0au7ZaxhVIWkdqlE7q18Z4yLPNWM4yQr6uI9JUpYbeAm/7/bF4DhIRjHcrAkPXV8CxIjFQ9Wvd8c6kWR7Bdm5fGDraMpDvDY4Ep6km7g6KLLXee306oRojJ7hfj3ixemTX8E7Vil1Fhze0wEHjGayMlwSBh1osdlnUiiwTP1gPdFYcjaelgC5oFnk4o2nFyEn4LxcRm6ar/sPdhp3EwrhEIyStIvLjnr1wsqgQjNIyIztHwcEHzOQJs0SIqlhG1mnGmPCpgih2XHiwFFVQldFNnErUBHPBNh06EuyojWKYp1HcxMgReBgJwiIj4VHauk4WzLoRuihqsC7+jc6hsYxGfBSl1cWzxbByWfVqVPxvvJQOhdpjIurMtvqAFfn+bZunElWhaChOfWh9WKpy66PiFG3ygmBc1JdP7OdbiDPQ6zP6yRUr24b5K/eoLSiFLLI1kO12KasmQXAEyyJHR7V7BxYSKgJ3broDj2IDrOiwRB3O9Co5nZzNeHbbtFye3DKxsnoDKaOl+lKviuMuhN5X0yqdPdyn9tcuWgxq4j0qXkw1XFJDUxEz+1L2MMUFgb721ejUPxm/Wvm8ioLnFSr+o6zgUVdopjH2RqPTbnrNVZ3NfkRanOVHYgoCJI4huYSXumVW6PXdASudniVVD/emvIK8Rhm8LzuNQJIHaghK5b2t+1c9Kcmb4axEvriExqsl8snvkjAcRHODJU4LONRQjHLxNo0o/z+y//r4TDqFSu3l3zNW5Zym0NHhacM5Q8J4DRzC8L903vHj1CyGgZGae2tLGa0caG274mg056MbojMwceWgDuIeGfve5vaS8yqkN0yn1uIGESzobKUJfPNqSn0uC7rbjKx9prjnW/yNkh9q98FFmDOCoAolWjzdwDggPD6POFAujfMBb7sYabYDI9e9AhY8IpwmfcPi9uvuMxr5ld6ICKunti7tmlz2KxkVJilghR83SXmsbF+WHLf4d4jlW0Tz/I60yTO8YTfLc1dDPKTujG4P2wCnW/UYA169GMag4wQUCc8H4Fq2g9BHk0iHJQfZ+RYFLEIyttDZ/lgZjrmqdLatKv3kRbJyUFpzD5dFhSt+dP0YTjeorbGZ6cnYiZtD/RzIBJk0mFrxeEhI7FHn6xyeDAiZez5W5EXNOKAu1ZMRDoF7CPbosXGFn3Z1EhzGSd1ivVuhn6JkmXV1oW7S0HwM= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR04MB8285.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(52116014)(376014)(366016)(38350700014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: hcyfs6n7zSH0lax+5uQ+vs1phnbLpdHhqhjQ7Tg0xkgwSTjkkTKP1agxxdhrFHRZQP2irpFzHbFVr4IpYGcEBvNf3Fp/SDXCNAzVPUbnEdon62BxqKks50iyindBgpv1pXjUT/PUuRvqcRriTOEw0Qr4wOaMH3tWix5U6LaMw+o19Ozzr2G6r4WEFq00zyKuC6+86HkpwgXJKJpvGFeWLfFcJcRnvTNmZoKbEVPEQfhJR5y5Wsin0gE+l8uLRecI12CXdlfOYXNArA5RhwRaiWWMaApsMzbvg4Cb/6qYR+05YzPJbiKKHOdjfrcx9Dn9BuCf6fYt+kD9gdGbVHr32xKo6VPh+7Zbglo/NnyU1Co6Or1K7+QXcp18ySMKJdAiAUUBqToAEPnBBAzpn9AuD0+G0P5oPb/EdQnmXY/lUlbO1DXOvOvpPkIfjkSK3PVNZP3kNxnq5ZBAIqHMG/KU5RlP5My+URBi3I1Ine7jnXbb2j5qQ3/5Q1m90GxxnxPOXX9wTvC6wiv6yZ+IIfj8F1aQlmvjQ3tiA4eVOXTn4J1d8wGc9iXHkCFs59PhJmbRkx9qi47cO77i7lVeG2BnrWxyXzBgyfKlvI4Dcv3azzV2/fUI3EkMt6kLmH5V9pkNYG5dCANtzHrLdzKtAUOq7ygTwg+xnJ7vQzTNtf0hEZednStLB7EIDgyVFZ3lXyQ9p0YI2TZSrA/yC8vYY/Pp7q7SeumCd5L86Xu3XIkq+fontYhkNr4LUZpLZPHpWc9J6IwGJjqIWB4zyYhZvJSSXc6SJH8FhNzTLukwH39prHv1+gQefVaePLoeTtuXS/hlUj42hVS4kvSv9VyYsw4I2X6soBr31vDPb6oYU3OQ+LFq0GPLCk98KIa9LkSvVxnAg2JfynoWJ9E3sfmTJ6D5LyD/UDulC0xznA3Sv2EuckEKdMM4Rhb6cLSj232UKxjKGtEQnH3uDFxCg9R/8xKtbPzdmQ146Po16gtrbTBPOFCZvagPdEtvSPeBjgb2jCiYyvHfo5Mw5LZWpE2VY9iSZHXCIOZ9yMLCODm1MvrMPTFP9R+TQrUoJugA6jecm+HFPeSzIT7vCujLBfok4QqXFLGR0L8I8r03UMVwA6gr1JcfNYtyYYwXDA770Qrm/6VI3Q/3yVHfAxe/lp84CDFmcJl93E3adae8LOCM1b/ZRgTq5gzvy5rM2Hd0n3+P/3yNB3p+jj+o5rJ5fD+hhzuhawxQkt6UBm6lX2dPNoTfcbN3zTWV9GW/ZXOrDu9alKSZUJf/5HOidRgo0G1tNhUx2T0delaI65A8gQTOTlOLlucx80O1LCC2UtuvAjNbvbHmZML/AzvDCjmlAUkbM03OctAb9A2ofMdN6QXmT56Bwty3tV3CW988RXJ/z4sJyqaL3XY1EuCKIgkeCsz2JMHknTZmSC2vaxGQgRg4Uc//GWT+A6N5OWAzDZ3lbXphQKHydpWF3dYeM4vHUzX7RNnkgxFzR0oern6vd/f6mqhRhTLfbsPTNxhK5l9C3vElYHHA56X+2RWcagVFRToD2OyY5FlwTVoHNnusmMh8rYr46eY= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1e06de68-6ebc-462b-67b8-08dd03b50e99 X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB8285.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Nov 2024 07:30:32.8442 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1OXlM6l36HjhbqbP+QpyYqcnZV5EcVMCuBtxAaDKXoZfBxg8SIQdJECpSgh2S7LG X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR04MB7132 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" GStreamer video-info calculated stride and offset may differ from those used by the camera. This patch enhances downstream plugin's support for videometa by adding videometa support for libcamerasrc. It ensures that when downstream plugin supports videometa by allocation query, libcamerasrc also attaches videometa to buffer, preventing discrepancies between the stride and offset calculated by video-info and camera. Signed-off-by: Hou Qi --- src/gstreamer/gstlibcamera-utils.cpp | 28 ++++++++++++++++++++++++ src/gstreamer/gstlibcamera-utils.h | 7 ++++++ src/gstreamer/gstlibcamerapool.cpp | 32 ++++++++++++++++++++++++---- src/gstreamer/gstlibcamerapool.h | 2 +- src/gstreamer/gstlibcamerasrc.cpp | 14 +++++++++++- 5 files changed, 77 insertions(+), 6 deletions(-) diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp index 732987ef..91de1d44 100644 --- a/src/gstreamer/gstlibcamera-utils.cpp +++ b/src/gstreamer/gstlibcamera-utils.cpp @@ -2,6 +2,9 @@ /* * Copyright (C) 2020, Collabora Ltd. * Author: Nicolas Dufresne + * Copyright (C) <1999> Erik Walthinsen + * Library <2002> Ronald Bultje + * Copyright (C) <2007> David A. Schleef * * GStreamer libcamera Utility Function */ @@ -591,6 +594,31 @@ gst_task_resume(GstTask *task) } #endif +#if !GST_CHECK_VERSION(1, 22, 0) +gint gst_libcamera_extrapolate_stride(const GstVideoFormatInfo *finfo, gint plane, gint stride) +{ + gint estride; + gint comp[GST_VIDEO_MAX_COMPONENTS]; + gint i; + + /* there is nothing to extrapolate on first plane */ + if (plane == 0) + return stride; + + gst_video_format_info_component(finfo, plane, comp); + + /* For now, all planar formats have a single component on first plane, but + * if there was a planar format with more, we'd have to make a ratio of the + * number of component on the first plane against the number of component on + * the current plane. */ + estride = 0; + for (i = 0; i < GST_VIDEO_MAX_COMPONENTS && comp[i] >= 0; i++) + estride += GST_VIDEO_FORMAT_INFO_SCALE_WIDTH(finfo, comp[i], stride); + + return estride; +} +#endif + G_LOCK_DEFINE_STATIC(cm_singleton_lock); static std::weak_ptr cm_singleton_ptr; diff --git a/src/gstreamer/gstlibcamera-utils.h b/src/gstreamer/gstlibcamera-utils.h index cab1c814..14bf6664 100644 --- a/src/gstreamer/gstlibcamera-utils.h +++ b/src/gstreamer/gstlibcamera-utils.h @@ -2,6 +2,9 @@ /* * Copyright (C) 2020, Collabora Ltd. * Author: Nicolas Dufresne + * Copyright (C) <1999> Erik Walthinsen + * Library <2002> Ronald Bultje + * Copyright (C) <2007> David A. Schleef * * GStreamer libcamera Utility Functions */ @@ -35,6 +38,10 @@ static inline void gst_clear_event(GstEvent **event_ptr) #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task); #endif + +#if !GST_CHECK_VERSION(1, 22, 0) +gint gst_libcamera_extrapolate_stride(const GstVideoFormatInfo *finfo, gint plane, gint stride); +#endif std::shared_ptr gst_libcamera_get_camera_manager(int &ret); /** diff --git a/src/gstreamer/gstlibcamerapool.cpp b/src/gstreamer/gstlibcamerapool.cpp index 9cd7eccb..315f08ac 100644 --- a/src/gstreamer/gstlibcamerapool.cpp +++ b/src/gstreamer/gstlibcamerapool.cpp @@ -135,16 +135,40 @@ gst_libcamera_pool_class_init(GstLibcameraPoolClass *klass) } GstLibcameraPool * -gst_libcamera_pool_new(GstLibcameraAllocator *allocator, Stream *stream) +gst_libcamera_pool_new(GstLibcameraAllocator *allocator, const StreamConfiguration &stream_cfg, + GstCaps *caps, gboolean add_video_meta) { auto *pool = GST_LIBCAMERA_POOL(g_object_new(GST_TYPE_LIBCAMERA_POOL, nullptr)); + GstVideoInfo info; pool->allocator = GST_LIBCAMERA_ALLOCATOR(g_object_ref(allocator)); - pool->stream = stream; - - gsize pool_size = gst_libcamera_allocator_get_pool_size(allocator, stream); + pool->stream = stream_cfg.stream(); + + if (caps && gst_video_info_from_caps(&info, caps)) { + guint k, stride; + gsize offset = 0; + for (k = 0; k < GST_VIDEO_INFO_N_PLANES(&info); k++) { +#if !GST_CHECK_VERSION(1, 22, 0) + stride = gst_libcamera_extrapolate_stride(info.finfo, k, stream_cfg.stride); +#else + stride = gst_video_format_info_extrapolate_stride(info.finfo, k, stream_cfg.stride); +#endif + info.stride[k] = stride; + info.offset[k] = offset; + offset += stride * GST_VIDEO_FORMAT_INFO_SCALE_HEIGHT(info.finfo, k, GST_VIDEO_INFO_HEIGHT(&info)); + } + } else + add_video_meta = false; + + gsize pool_size = gst_libcamera_allocator_get_pool_size(allocator, stream_cfg.stream()); for (gsize i = 0; i < pool_size; i++) { GstBuffer *buffer = gst_buffer_new(); + if (add_video_meta) { + gst_buffer_add_video_meta_full(buffer, GST_VIDEO_FRAME_FLAG_NONE, + GST_VIDEO_INFO_FORMAT(&info), GST_VIDEO_INFO_WIDTH(&info), + GST_VIDEO_INFO_HEIGHT(&info), GST_VIDEO_INFO_N_PLANES(&info), + info.offset, info.stride); + } pool->queue->push_back(buffer); } diff --git a/src/gstreamer/gstlibcamerapool.h b/src/gstreamer/gstlibcamerapool.h index 2a7a9c77..8ad87cab 100644 --- a/src/gstreamer/gstlibcamerapool.h +++ b/src/gstreamer/gstlibcamerapool.h @@ -21,7 +21,7 @@ G_DECLARE_FINAL_TYPE(GstLibcameraPool, gst_libcamera_pool, GST_LIBCAMERA, POOL, GstBufferPool) GstLibcameraPool *gst_libcamera_pool_new(GstLibcameraAllocator *allocator, - libcamera::Stream *stream); + const libcamera::StreamConfiguration &stream_cfg, GstCaps *caps, gboolean add_video_meta); libcamera::Stream *gst_libcamera_pool_get_stream(GstLibcameraPool *self); diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 8efa25f4..c05a31e7 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -497,9 +497,21 @@ gst_libcamera_src_negotiate(GstLibcameraSrc *self) for (gsize i = 0; i < state->srcpads_.size(); i++) { GstPad *srcpad = state->srcpads_[i]; const StreamConfiguration &stream_cfg = state->config_->at(i); + GstQuery *query = NULL; + gboolean add_video_meta = false; + + g_autoptr(GstCaps) caps = gst_libcamera_stream_configuration_to_caps(stream_cfg); + gst_libcamera_framerate_to_caps(caps, element_caps); + + query = gst_query_new_allocation(caps, false); + if (!gst_pad_peer_query(srcpad, query)) + GST_DEBUG_OBJECT(self, "didn't get downstream ALLOCATION hints"); + else + add_video_meta = gst_query_find_allocation_meta(query, GST_VIDEO_META_API_TYPE, NULL); + gst_query_unref(query); GstLibcameraPool *pool = gst_libcamera_pool_new(self->allocator, - stream_cfg.stream()); + stream_cfg, caps, add_video_meta); g_signal_connect_swapped(pool, "buffer-notify", G_CALLBACK(gst_task_resume), self->task);