From patchwork Wed Dec 4 08:55:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Qi X-Patchwork-Id: 22157 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5BE16BDC71 for ; Wed, 4 Dec 2024 08:55:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EE1C9618B5; Wed, 4 Dec 2024 09:55:39 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=nxp.com header.i=@nxp.com header.b="W0xcb6oH"; dkim-atps=neutral Received: from EUR02-VI1-obe.outbound.protection.outlook.com (mail-vi1eur02on20615.outbound.protection.outlook.com [IPv6:2a01:111:f403:2607::615]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DB50A618B5 for ; Wed, 4 Dec 2024 09:55:36 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NIcUC4ZzNUHKUdpg2fLkfEDADBhWA93oxgf4NHnL5BE1AZyk/QMHOUbHDEIJEMgq0kAZfqaGmwM6TPUQ2xSapEoYD8ORBR0yqtbj+hQgariJvAINTeUh72yPZ0LpLjjA2sJX3FkAsJVGiaK/6XNlFfQhMNG5CDaWxta8TvSI9lI5NXR4uagFhfstFsi6/aNRy45Uu3LS4+xLuKsD66HOyKTnlHvyhx2HHyYfGs5UOyDU5QBZeOtLZm7lNmD4xIKlfFFeYTFngPLD/29/zQ2qM81jFMa8hm1XPLUsyoE/wv7i5m+hrAvV2xtf5iUiRvOedeQ9z7ikUxccMaulVRUu6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZrhxcvAAtbeOuWH8etRpfy0yezP8dYMj1efEcUa4DQg=; b=Aa3Bpnx64BjSxv6Z2e5F4loy3PnNGZmtz7Ca/FwzS6KMlT8JzxZ5KnYCIcbvJB07md88ykdFCCiooa4v4rsWHCAWerKi5PQ+OpSdImVNnSRrvPaxear252H1owvPdaT9w4HagxbKUY8t2frbPoy3pgvqdkIjr//aHMVFJ3J5sN4JPsBixpz4vFt93fmV1fv5rSx4CdJ7+G1Ikw1DcQQBiyidQjFa+7S/J6KfXSWxcAx+soe+nMmbfvOqBQMyizHrILRbW71md+nOhemGpsg1KyfWIHCv7nI6kdnvVwZxJ/pf0l748Hv6iJ7Pqg3uB/7ZanEvWHlC00e0z6HKUSTfnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZrhxcvAAtbeOuWH8etRpfy0yezP8dYMj1efEcUa4DQg=; b=W0xcb6oHc6K7dNzLStYCh9x064UG0daC/yOZN0Kt4opY6mR5rbtqebiO6oDx3I5encfU6aWWH0PUlpxGiLxmEC/o/4qh1dokFwFdc+TYOwARkw9AQLcWUtHBscR2cwKy0k9z6wdJYLUEo4SefJtLdiMLoMNMt2mpcdIvR9awPcvkmwiaowpNgVPBgqeNK+UsG1pqrgj0bfJDn8AdPrW+/8IDsVQ5Md66GQ/cDMpulbj3LO6dt6WTDU72t6biUM4SbuKBahnjOSV0iYgrvwEc/SayH9v8xbFM9b6VxjMziUbRaFVuVVi2gZs+EeiG6nwXDPHgWOy4WfhzNJjAmgFtww== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB8285.eurprd04.prod.outlook.com (2603:10a6:102:1ca::15) by PA4PR04MB7789.eurprd04.prod.outlook.com (2603:10a6:102:c3::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.11; Wed, 4 Dec 2024 08:55:34 +0000 Received: from PAXPR04MB8285.eurprd04.prod.outlook.com ([fe80::e003:8fb:64ea:acfd]) by PAXPR04MB8285.eurprd04.prod.outlook.com ([fe80::e003:8fb:64ea:acfd%6]) with mapi id 15.20.8207.017; Wed, 4 Dec 2024 08:55:34 +0000 From: Hou Qi To: libcamera-devel@lists.libcamera.org Cc: jared.hu@nxp.com, qi.hou@nxp.com, julien.vuillaumier@nxp.com Subject: [PATCH v6] gstreamer: Add GstVideoMeta support Date: Wed, 4 Dec 2024 17:55:11 +0900 Message-Id: <20241204085511.1733434-1-qi.hou@nxp.com> X-Mailer: git-send-email 2.34.1 X-ClientProxiedBy: SI2PR01CA0004.apcprd01.prod.exchangelabs.com (2603:1096:4:191::22) To PAXPR04MB8285.eurprd04.prod.outlook.com (2603:10a6:102:1ca::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR04MB8285:EE_|PA4PR04MB7789:EE_ X-MS-Office365-Filtering-Correlation-Id: de1c2ae9-60df-4d5d-5142-08dd1441694b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|366016|376014|52116014|38350700014; X-Microsoft-Antispam-Message-Info: tFp1VFQ6G2pp2yflTKZmdPgBIY74xRFdHJ3Y/wrA5O8qDG1xcUvdEOEsBRzZMJuXlzIRA8pf7OldT/Kg5y8HOL5f3TBLHCHhYNPFsBnskgagBaAQSCbdQzABBayYIWC42Y/RuA/hnJYsR+ryTtI7pbVzw6yiwTmcLxyj/kgKddbgKBuSClmS0axdvkTnyOKIcMyEJyCDvn9p7KVPHDiH3J5mCgn59XeEohl7yUiIZvEmt854Fmgkp5MWoUosTp/I7e+Ye1uijLtzU3Dw+pnx0qpYwngr8/qry4FgLshXWS1NTIV+N4GPPu3LKfzG3SiGhvdv1wsAqVQBdjlfROBfyrFPJoR5TBZ5BcwIlzeZu4q448Vb0ZDrJnoMRIrnhwQ7mnIWHXJSaPjckbq9U84wl57wnJQzoqxzpCkdSm5Gp0xe0WvGbGz17as8E9tl/ExonMMZoCGAO0AWNHkD6Cs8gXYfGsVmDx+IFKYz1A24sVBNFeBq/05qW9Tp42eUR0t85yTZhNAAasZwxkZ1dr+dZBS0x3mDVYjuSHi0zcEbkE12hyEGLMG1vnruXQe8RsDMoELSbvNBo8lwo0rnJnsx8d6avPjLPXNVVF9pToqT8SUXN57fKHXqztv4hm4L/2EfEOO7BTzsI32nz7iOlNCsxR7MzZaJ3yR51ZUR8/dd3AhLnhFkVraIVyEZc9jINLFJwfMkoayHPeKgBGOypMTbLy8sE+SimDPRaoszd4c/B9amYXGpeA7Sg+9mqUC7dBaP7S0Z4lRdE1AYR53F6C4kDLTk8vFSazI2/CnQZKxekhD7FbOL8ncdVMACPKHIGBg24w37rkWyf1h+lVuxFwubjTXpLYgB6mCPyj5borPulhSnv6BJFb9Cm0w9ebdtBXNjAkbIoOfuyXTMJ51Wn7MvKAaK9POizCZ2kOirJWjGnKuKJvs9ncgS7GZ1YhCKOUg/X7Nfu45LihUZyMPUHR7ECcZ+EOetu26E4djRrEe7tgx9GYTEnq6n6L7DJjx7LH7mx/P6Tk200u38oh3rXo+aOBBhk6OI31GlKt+ZbjlimXXTjrpSCcpOjlowZtl6mUTac7mcVW+WGh+PzBHIQNAyj40AfJ+o+Qg3qQx2TONptXKb7bhrjW6J01r+DLdLHjzDyqXdNYuw9NsWHtAYyR1/aMvgEm+/6nEv5Is+jPZO3Z60EXlojWXjJYwK45MtABUqo+E/DlGSaA6AaLY0ehen+BfDMQxfDMD5PHABaZYksrDm/w1rsRrn9hxysRAEtppzAJD4fv3EvKZS0CpigZay+jDN1ErjELkyJFhiVGzCIXr+4jCYDAdRxaUoRB3v9gxsax7F3uKflLE9DwmpDEKLcj0i25Dj2wy7PFoQCiMMnnNWOJRR4E3w4kyZK+vaPi3htb7s697QQ+NsZnStD4NRCg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR04MB8285.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(52116014)(38350700014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: zlWPt76T8zcfppqIg7Q8URvn0jF4OhFt1+In19+YcfM2VWMWhJVbM9vWEXmNSe3zcPC4lsoIKwCm/26iI2MzFGEzeUilEhpYU9i+GyO1APdlE1/tssmap0Vb40HNZCrEmmXMfwy5zjtYMeocP3LXHFoiwdPDDx0aD73weJSCcNNm0CsixryeCQHSCs9cxbzXExt0ePPpCa5jcRfotcVJvSjII0chNYtMzN6cAMluYaCzYLW72tlIcGaIjE18wJz6UDFsnFeAAYgvkWIm40ZqWGa53ffe6zkZFr9a5mhp66OEXtis2PVi8xQMJALS6Y2FFJ1xbEkOZW1sJq6jk1A/qjJamBszAdcjHZQnc+x5I7qwwdxObWOCU67UnMNR2GVj+WLZVRQ9yyfZ23N8xgqqXq2zgcczW0jCH0txDLCeMJhUcKxSx3UmwpvbbYaDFtLYFG4GHSg3CcTh4mtRHW8BLdic2VDV6VT1ucA6+EhvFdphVf2U3Dvt1v5H6fWJSVKBbKr0jDfyAwRV3WRmxYmA2s4CU4k/7z1Y/FjGjOLVtz6SOJ/K304yGUCbJY6H5G9xOuA8MqpVVckRg9U3N9Xpuqu/eJCbNbxvHiH3nZd6zBE3AhahS0DWfzJGdSW4rHiQhLPm5osb71nJAhU7tWkradgoMFmAZGCEwj1MTDelmAB12S4yU1IMbAnQPwHS3iLccXlGmjd2ytvKmidN87140ytphs2y4YLWah+T56mC3b0YjyoyabkD6SuJ1VNufCtYicn8LxVdZK6odB3Zr8+OuykB565kgU7xUMrVeUvX2cnmazxxApUOufW9FI28ACIr7h0M7NY1Mofhr/L1s7UNNnAbsSP3nqigQmKbmMk1LYheOTHlBJ2+EJS77jaLVuIieUs3z+fh2K8FGFEqJzftBKtTRhXMUFX8O0GsjEXQFXiifYkrIQMuSiRMO7U6qGfKIotsKlVcjCgAepF5XyCXm7qy+/CeWkROLvYI/lAKfbyu1bq+SFgZ5RI+mSy7MGNWZ0Vj4phjgIivANlmpWkfa7wRD3EzbFkNuO6hCwD3zMJGtExd9m0ixsfBFs651Vk2Rl7cwP39d4c4TL013EctrZjBSPN2mIUKGOV48AEwnWxQ59rgTjA8PBk9r5DXOOYG4a0MQ7MwsvHw7kYYx3CGSDwvVA4DUJrD9PTV24EpbmGyEEqUYyFKhmb5eUPTxB+QRRJadJ1iFLNP6jp7kXP9UvdYILbNOHPiFDbGPe0/3a8n3otxoWfkkNEJnMOeX3wL29otboB30/vqES5n29IRSyADPdltnA6pcJeBlwo0UqXnppKacIExQWMup5aDTa46nwNtTN/pYG+22pnMceT5ThQjDR3aQ6hkQaTknfVG5DLQUTcLAfBAlMiyQsjeZVYBU6lK+C57umOSaPY+jcpcQ5zeIwawbx8x9rlVQ77netc5atpm5qt7944LE1nDKh1OH2xNgMJrPLx0OD+Dr8bPg/vpJIlxVyNiBVYAiQxPDU5eTOhlgn6KWSFEu/zE89uG96NVFd9fBI0dtTGTrukI3yPukaZariUXc736LXq5PEU= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: de1c2ae9-60df-4d5d-5142-08dd1441694b X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB8285.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2024 08:55:34.2112 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: O7dLKNiAheKjEyzaobHL0Fb4ikSyevg4D4clp7Cxig2dt/EhN35nQFvdzFRDLmm5 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR04MB7789 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" GStreamer video-info calculated stride and offset may differ from those used by the camera. For stride and offset mismatch, this patch adds video meta to buffer if downstream supports VideoMeta through allocation query. Otherwise, create a internal VideoPool using the caps, and copy video frame to this system memory. Signed-off-by: Hou Qi --- src/gstreamer/gstlibcamera-utils.cpp | 33 +++++++ src/gstreamer/gstlibcamera-utils.h | 4 + src/gstreamer/gstlibcamerapad.cpp | 31 +++++++ src/gstreamer/gstlibcamerapad.h | 8 ++ src/gstreamer/gstlibcamerapool.cpp | 18 +++- src/gstreamer/gstlibcamerapool.h | 3 +- src/gstreamer/gstlibcamerasrc.cpp | 133 ++++++++++++++++++++++++++- 7 files changed, 226 insertions(+), 4 deletions(-) diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp index 732987ef..09af9204 100644 --- a/src/gstreamer/gstlibcamera-utils.cpp +++ b/src/gstreamer/gstlibcamera-utils.cpp @@ -591,6 +591,39 @@ gst_task_resume(GstTask *task) } #endif +#if !GST_CHECK_VERSION(1, 22, 0) +/* + * Copyright (C) <1999> Erik Walthinsen + * Library <2002> Ronald Bultje + * Copyright (C) <2007> David A. Schleef + */ +/* This function has been imported directly from the gstreamer project to + * support backwards compatibility and should be removed when the older + * version is no longer supported. */ +gint gst_video_format_info_extrapolate_stride(const GstVideoFormatInfo *finfo, gint plane, gint stride) +{ + gint estride; + gint comp[GST_VIDEO_MAX_COMPONENTS]; + gint i; + + /* there is nothing to extrapolate on first plane */ + if (plane == 0) + return stride; + + gst_video_format_info_component(finfo, plane, comp); + + /* For now, all planar formats have a single component on first plane, but + * if there was a planar format with more, we'd have to make a ratio of the + * number of component on the first plane against the number of component on + * the current plane. */ + estride = 0; + for (i = 0; i < GST_VIDEO_MAX_COMPONENTS && comp[i] >= 0; i++) + estride += GST_VIDEO_FORMAT_INFO_SCALE_WIDTH(finfo, comp[i], stride); + + return estride; +} +#endif + G_LOCK_DEFINE_STATIC(cm_singleton_lock); static std::weak_ptr cm_singleton_ptr; diff --git a/src/gstreamer/gstlibcamera-utils.h b/src/gstreamer/gstlibcamera-utils.h index cab1c814..81149280 100644 --- a/src/gstreamer/gstlibcamera-utils.h +++ b/src/gstreamer/gstlibcamera-utils.h @@ -35,6 +35,10 @@ static inline void gst_clear_event(GstEvent **event_ptr) #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task); #endif + +#if !GST_CHECK_VERSION(1, 22, 0) +gint gst_video_format_info_extrapolate_stride(const GstVideoFormatInfo *finfo, gint plane, gint stride); +#endif std::shared_ptr gst_libcamera_get_camera_manager(int &ret); /** diff --git a/src/gstreamer/gstlibcamerapad.cpp b/src/gstreamer/gstlibcamerapad.cpp index 7b22aebe..7fd11a8f 100644 --- a/src/gstreamer/gstlibcamerapad.cpp +++ b/src/gstreamer/gstlibcamerapad.cpp @@ -18,6 +18,8 @@ struct _GstLibcameraPad { GstPad parent; StreamRole role; GstLibcameraPool *pool; + GstBufferPool *video_pool; + GstVideoInfo info; GstClockTime latency; }; @@ -153,6 +155,35 @@ gst_libcamera_pad_set_pool(GstPad *pad, GstLibcameraPool *pool) self->pool = pool; } +GstBufferPool * +gst_libcamera_pad_get_video_pool(GstPad *pad) +{ + auto *self = GST_LIBCAMERA_PAD(pad); + return self->video_pool; +} + +void gst_libcamera_pad_set_video_pool(GstPad *pad, GstBufferPool *video_pool) +{ + auto *self = GST_LIBCAMERA_PAD(pad); + + if (self->video_pool) + g_object_unref(self->video_pool); + self->video_pool = video_pool; +} + +GstVideoInfo gst_libcamera_pad_get_video_info(GstPad *pad) +{ + auto *self = GST_LIBCAMERA_PAD(pad); + return self->info; +} + +void gst_libcamera_pad_set_video_info(GstPad *pad, GstVideoInfo info) +{ + auto *self = GST_LIBCAMERA_PAD(pad); + + self->info = info; +} + Stream * gst_libcamera_pad_get_stream(GstPad *pad) { diff --git a/src/gstreamer/gstlibcamerapad.h b/src/gstreamer/gstlibcamerapad.h index 630c168a..ed0beb1f 100644 --- a/src/gstreamer/gstlibcamerapad.h +++ b/src/gstreamer/gstlibcamerapad.h @@ -23,6 +23,14 @@ GstLibcameraPool *gst_libcamera_pad_get_pool(GstPad *pad); void gst_libcamera_pad_set_pool(GstPad *pad, GstLibcameraPool *pool); +GstBufferPool *gst_libcamera_pad_get_video_pool(GstPad *pad); + +void gst_libcamera_pad_set_video_pool(GstPad *pad, GstBufferPool *video_pool); + +GstVideoInfo gst_libcamera_pad_get_video_info(GstPad *pad); + +void gst_libcamera_pad_set_video_info(GstPad *pad, GstVideoInfo info); + libcamera::Stream *gst_libcamera_pad_get_stream(GstPad *pad); void gst_libcamera_pad_set_latency(GstPad *pad, GstClockTime latency); diff --git a/src/gstreamer/gstlibcamerapool.cpp b/src/gstreamer/gstlibcamerapool.cpp index 9cd7eccb..471f71da 100644 --- a/src/gstreamer/gstlibcamerapool.cpp +++ b/src/gstreamer/gstlibcamerapool.cpp @@ -14,6 +14,9 @@ #include "gstlibcamera-utils.h" +#include + + using namespace libcamera; enum { @@ -134,8 +137,20 @@ gst_libcamera_pool_class_init(GstLibcameraPoolClass *klass) G_TYPE_NONE, 0); } +static void +gst_libcamera_buffer_add_video_meta(GstBuffer *buffer, GstVideoInfo *info) +{ + GstVideoMeta *vmeta; + vmeta = gst_buffer_add_video_meta_full(buffer, GST_VIDEO_FRAME_FLAG_NONE, + GST_VIDEO_INFO_FORMAT(info), GST_VIDEO_INFO_WIDTH(info), + GST_VIDEO_INFO_HEIGHT(info), GST_VIDEO_INFO_N_PLANES(info), + info->offset, info->stride); + GST_META_FLAGS(vmeta) = (GstMetaFlags)(GST_META_FLAGS(vmeta) | GST_META_FLAG_POOLED); +} + GstLibcameraPool * -gst_libcamera_pool_new(GstLibcameraAllocator *allocator, Stream *stream) +gst_libcamera_pool_new(GstLibcameraAllocator *allocator, Stream *stream, + GstVideoInfo *info) { auto *pool = GST_LIBCAMERA_POOL(g_object_new(GST_TYPE_LIBCAMERA_POOL, nullptr)); @@ -145,6 +160,7 @@ gst_libcamera_pool_new(GstLibcameraAllocator *allocator, Stream *stream) gsize pool_size = gst_libcamera_allocator_get_pool_size(allocator, stream); for (gsize i = 0; i < pool_size; i++) { GstBuffer *buffer = gst_buffer_new(); + gst_libcamera_buffer_add_video_meta(buffer, info); pool->queue->push_back(buffer); } diff --git a/src/gstreamer/gstlibcamerapool.h b/src/gstreamer/gstlibcamerapool.h index 2a7a9c77..02ee4dd4 100644 --- a/src/gstreamer/gstlibcamerapool.h +++ b/src/gstreamer/gstlibcamerapool.h @@ -14,6 +14,7 @@ #include "gstlibcameraallocator.h" #include +#include #include @@ -21,7 +22,7 @@ G_DECLARE_FINAL_TYPE(GstLibcameraPool, gst_libcamera_pool, GST_LIBCAMERA, POOL, GstBufferPool) GstLibcameraPool *gst_libcamera_pool_new(GstLibcameraAllocator *allocator, - libcamera::Stream *stream); + libcamera::Stream *stream, GstVideoInfo *info); libcamera::Stream *gst_libcamera_pool_get_stream(GstLibcameraPool *self); diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 8efa25f4..82b06eb4 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -268,6 +268,52 @@ GstLibcameraSrcState::requestCompleted(Request *request) gst_task_resume(src_->task); } +static void +gst_libcamera_extrapolate_info(GstVideoInfo *info, guint32 stride) +{ + guint i, estride; + gsize offset = 0; + + /* this should be updated if tiled formats get added in the future. */ + for (i = 0; i < GST_VIDEO_INFO_N_PLANES(info); i++) { + estride = gst_video_format_info_extrapolate_stride(info->finfo, i, stride); + info->stride[i] = estride; + info->offset[i] = offset; + offset += estride * GST_VIDEO_FORMAT_INFO_SCALE_HEIGHT(info->finfo, i, + GST_VIDEO_INFO_HEIGHT(info)); + } +} + +static GstFlowReturn +gst_libcamera_video_frame_copy(GstBuffer *src, GstBuffer *dest, GstVideoInfo dest_info, guint32 stride) +{ + GstVideoInfo src_info = dest_info; + GstVideoFrame src_frame, dest_frame; + + gst_libcamera_extrapolate_info(&src_info, stride); + src_info.size = gst_buffer_get_size(src); + + if (!gst_video_frame_map(&src_frame, &src_info, src, GST_MAP_READ)) + goto invalid_buffer; + + if (!gst_video_frame_map(&dest_frame, &dest_info, dest, GST_MAP_WRITE)) { + gst_video_frame_unmap(&src_frame); + goto invalid_buffer; + } + + gst_video_frame_copy(&dest_frame, &src_frame); + + gst_video_frame_unmap(&src_frame); + gst_video_frame_unmap(&dest_frame); + + return GST_FLOW_OK; + +invalid_buffer : { + GST_ERROR("Could not map buffer"); + return GST_FLOW_ERROR; +} +} + /* Must be called with stream_lock held. */ int GstLibcameraSrcState::processRequest() { @@ -292,11 +338,36 @@ int GstLibcameraSrcState::processRequest() GstFlowReturn ret = GST_FLOW_OK; gst_flow_combiner_reset(src_->flow_combiner); - for (GstPad *srcpad : srcpads_) { + for (gsize i = 0; i < src_->state->srcpads_.size(); i++) { + GstPad *srcpad = src_->state->srcpads_[i]; Stream *stream = gst_libcamera_pad_get_stream(srcpad); GstBuffer *buffer = wrap->detachBuffer(stream); FrameBuffer *fb = gst_libcamera_buffer_get_frame_buffer(buffer); + const StreamConfiguration &stream_cfg = src_->state->config_->at(i); + GstBufferPool *video_pool = gst_libcamera_pad_get_video_pool(srcpad); + + if (video_pool) { + GstBuffer *copy = NULL; + GstVideoInfo info = gst_libcamera_pad_get_video_info(srcpad); + + ret = gst_buffer_pool_acquire_buffer(video_pool, ©, NULL); + if (ret != GST_FLOW_OK) { + GST_ERROR("Failed to acquire buffer"); + gst_buffer_unref(buffer); + return -EPIPE; + } + + ret = gst_libcamera_video_frame_copy(buffer, copy, info, stream_cfg.stride); + gst_buffer_unref(buffer); + if (ret != GST_FLOW_OK) { + GST_ERROR("Failed to copy buffer"); + gst_buffer_unref(copy); + return -EPIPE; + } + + buffer = copy; + } if (GST_CLOCK_TIME_IS_VALID(wrap->pts_)) { GST_BUFFER_PTS(buffer) = wrap->pts_; @@ -497,13 +568,65 @@ gst_libcamera_src_negotiate(GstLibcameraSrc *self) for (gsize i = 0; i < state->srcpads_.size(); i++) { GstPad *srcpad = state->srcpads_[i]; const StreamConfiguration &stream_cfg = state->config_->at(i); + GstBufferPool *video_pool = NULL; + GstVideoInfo info; + + g_autoptr(GstCaps) caps = gst_libcamera_stream_configuration_to_caps(stream_cfg); + + gst_video_info_init(&info); + gst_video_info_from_caps(&info, caps); + gst_libcamera_pad_set_video_info(srcpad, info); + + /* stride mismatch between camera stride and that calculated by video-info */ + if (static_cast(info.stride[0]) != stream_cfg.stride && + GST_VIDEO_INFO_FORMAT(&info) != GST_VIDEO_FORMAT_ENCODED) { + GstQuery *query = NULL; + gboolean need_pool = true; + gboolean has_video_meta = false; + + gst_libcamera_extrapolate_info(&info, stream_cfg.stride); + + query = gst_query_new_allocation(caps, need_pool); + if (!gst_pad_peer_query(srcpad, query)) + GST_DEBUG_OBJECT(self, "Didn't get downstream ALLOCATION hints"); + else + has_video_meta = gst_query_find_allocation_meta(query, GST_VIDEO_META_API_TYPE, NULL); + + if (!has_video_meta) { + GstBufferPool *pool = NULL; + + if (gst_query_get_n_allocation_pools(query) > 0) + gst_query_parse_nth_allocation_pool(query, 0, &pool, NULL, NULL, NULL); + + if (pool) + video_pool = pool; + else { + GstStructure *config; + guint min_buffers = 3; + video_pool = gst_video_buffer_pool_new(); + + config = gst_buffer_pool_get_config(video_pool); + gst_buffer_pool_config_set_params(config, caps, info.size, min_buffers, 0); + gst_buffer_pool_set_config(GST_BUFFER_POOL_CAST(video_pool), config); + GST_DEBUG_OBJECT(self, "Own pool config is %" GST_PTR_FORMAT, config); + } + + if (!gst_buffer_pool_set_active(video_pool, true)) { + GST_ERROR_OBJECT(self, "Failed to active buffer pool"); + gst_caps_unref(caps); + return false; + } + } + gst_query_unref(query); + } GstLibcameraPool *pool = gst_libcamera_pool_new(self->allocator, - stream_cfg.stream()); + stream_cfg.stream(), &info); g_signal_connect_swapped(pool, "buffer-notify", G_CALLBACK(gst_task_resume), self->task); gst_libcamera_pad_set_pool(srcpad, pool); + gst_libcamera_pad_set_video_pool(srcpad, video_pool); /* Clear all reconfigure flags. */ gst_pad_check_reconfigure(srcpad); @@ -920,6 +1043,12 @@ gst_libcamera_src_release_pad(GstElement *element, GstPad *pad) auto end_iterator = pads.end(); auto pad_iterator = std::find(begin_iterator, end_iterator, pad); + GstBufferPool *video_pool = gst_libcamera_pad_get_video_pool(pad); + if (video_pool) { + gst_buffer_pool_set_active(video_pool, false); + gst_object_unref(video_pool); + } + if (pad_iterator != end_iterator) { g_object_unref(*pad_iterator); pads.erase(pad_iterator);