From patchwork Wed Nov 22 13:43:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jaslo Ziska X-Patchwork-Id: 19225 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 09A7FC3220 for ; Wed, 22 Nov 2023 13:55:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BB461629BC; Wed, 22 Nov 2023 14:55:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1700661326; bh=VUBeSnAX+R8QfkSbKZ2jVILDxScnF1nIIKwhk2RJsUc=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=RH5IB2WKe+zh49l5w9BCmwz8l3QfqGnbBANFX2OVRd5Elp0PKQENqquiNla8NXLBq lp4zpiNI1RhsLMk56Br51BtOc4/01eoMa/fm0/SRi3LNefd2SwIjycaPo7XfP6KkRY OJ4QH+kOnXi/FkMeuq95xJEUHBnPCEiaSuiVvXhZ42kCQLzhZfrzuUTCzOKCZ3CP+2 +cqxdHhsiB8wa6S3cCqvhVO2nNStCkZZsfTXDLdscaJ8Vma5F3wa/kCjkUijMreerI rr/7CJA/zRqB05RYZkpCFqOuJ1QSOTrQYkNItUWzfvwCbQHf6S0YGo9YbE7utxREPf LlMSGN6xF6Wyg== Received: from mo4-p00-ob.smtp.rzone.de (mo4-p00-ob.smtp.rzone.de [85.215.255.21]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 45EC461DAD for ; Wed, 22 Nov 2023 14:55:25 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=ziska.de header.i=@ziska.de header.b="twLwLjS5"; dkim=permerror (0-bit key) header.d=ziska.de header.i=@ziska.de header.b="frwGEo4v"; dkim-atps=neutral ARC-Seal: i=1; a=rsa-sha256; t=1700661324; cv=none; d=strato.com; s=strato-dkim-0002; b=n5VCkFLMWuLmNixn6TIozXvJwiDWfqxnBdo8hMjhAYTmg0n8TBq1BpnYPQEAph7mL9 3GQTJbmsIYZtGstvQPVGUzsUdZDOufNrIIb3kh4Xex7vPey5IdT0UwjctHtn7hNRRgMq piZtOW98FrAY26QY+zsa1HttB6FRRD0ZOFtbGdvqcKFUlwfGLmUapUaKtiZwOk4R7sn4 hEZYle0un8whmKbAitx9C9qUfWQTb5iki8+yNMOm5mg4vRQCT6zjV2UbJhYnGgc+BA6W 1k3zlxKDxEDxQYPXdFAQpgPHMU1I4zkDWt1ArCKNncpltmJePJ/2bPm0ixAfXpbjUoEP zrdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1700661324; s=strato-dkim-0002; d=strato.com; h=References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Cc:Date: From:Subject:Sender; bh=NjoAQgXU5NeIvTedhSpZUyXJyHOywY+H8rZ/RMBpAxs=; b=PK+igTCF2V6aUEB7a5kAYK6fPIRAvHzIXdprilLxK6+4zPCfbSM6f3/Wa0MyaQ60uq R0fb1p5cXCszUXmnXb+NVi7ECbXYCQT8/LTOGYz44UjVMbVBA2LsA2llcjUX9N4bT0o8 1g9eVRRpo4A/wroT5YMai1ALwO3MRNOJ+tPELodvCnsT3tZd2c543WnRAAM4CNNZFKsZ /TmgC5rr/2LJozHTyHdNvddvq4gMi8WLdtrM2CReF15bZxXu2ahVViW+JYXacj7Xqs48 tqMYzregvzhIlpUNYThvjPuEyr4EvoFoMKJH8xVh6MPyKbbYL9pHQ6MxD2Y7X1kFyvJB 2fXA== ARC-Authentication-Results: i=1; strato.com; arc=none; dkim=none X-RZG-CLASS-ID: mo00 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1700661324; s=strato-dkim-0002; d=ziska.de; h=References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Cc:Date: From:Subject:Sender; bh=NjoAQgXU5NeIvTedhSpZUyXJyHOywY+H8rZ/RMBpAxs=; b=twLwLjS5ITVL46GBJ08s1Rbmii6wGBg8aSS/8VEe4CiAg7wzf8BkUNnmQYYX7R4ClP N0ZjEOGXVSweKsAmKxQvS2dJh6pSGh+SdadYmfwEAXOf9s0tHsLyWPsUJ6A7fD2sKPE2 OYQdqE0t/6YD10P580CxMEIm+ijeV0ardYhYs5RapolwlAHCaWFVISkhmUtEi1sbInH6 9XMC7BQ3EuMzMdQpoIH0MzRP1yA7mJuHr5wvhW2w4y8jThjv4VdsEn67ffZVhRshFH1I jkw/E67Znm3QpINLK/P57kQ+in+oSmA7piUvRLQRCD0FDcKH+XMo4aDw1bGFRbEF6NEq 6M0A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1700661324; s=strato-dkim-0003; d=ziska.de; h=References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Cc:Date: From:Subject:Sender; bh=NjoAQgXU5NeIvTedhSpZUyXJyHOywY+H8rZ/RMBpAxs=; b=frwGEo4vwwrhTbuIDPAn0LxmRfsATRvhMOB7CmbpUjsbcnj8RH7+w9LqWjSPlDJm1F fAcS7by1lf6QMgNtjmBg== X-RZG-AUTH: ":Jm0XeU+IYfb0x77LHmrjN5Wlb7TBwusDqIM6Hizy8VdfzvKi4yoFC9cG0Yq6UfJaraj+i8jcRUV7OSd6ZI2VJZzV+od9" Received: from archlinux.fritz.box by smtp.strato.de (RZmta 49.9.1 AUTH) with ESMTPSA id j3f4eezAMDtOnQ3 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate); Wed, 22 Nov 2023 14:55:24 +0100 (CET) To: libcamera-devel@lists.libcamera.org Date: Wed, 22 Nov 2023 14:43:55 +0100 Message-ID: <20231122135406.14921-3-jaslo@ziska.de> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231122135406.14921-1-jaslo@ziska.de> References: <20231122135406.14921-1-jaslo@ziska.de> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/2] gstreamer: Implement renegotiation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jaslo Ziska via libcamera-devel From: Jaslo Ziska Reply-To: Jaslo Ziska Cc: Jaslo Ziska Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This commit implements renegotiation of the camera configuration and source pad caps. A renegotiation can happen when a downstream element decides to change caps or the pipeline is dynamically changed. To handle the renegotiation the GST_FLOW_NOT_NEGOTIATED return value has to be handled in GstLibcameraSrcState::processRequest. Otherwise the default would be to print an error and stop streaming. To archive this in a clean way the if-statement is altered into a switch statement which now also has a case for GST_FLOW_NOT_NEGOTIATED. Just like the case for GST_FLOW_OK, the function will return without an error because the renegotiation will happen later in the running task. In gst_libcamera_src_task_run all the source pads are checked for the reconfigure flag by calling gst_pad_check_reconfigure. It is important to iterate all source pads and not break after one pad requested a reconfiguration because gst_pad_check_reconfigure clears the reconfigure flag and if two pads request a reconfiguration at the same time the renegotiation would happen twice. If a source pad requested a reconfiguration it is first checked whether the old caps are still sufficient. If they are not the renegotiation will happen. If any pad requested a reconfiguration the following will happen: 1. The camera is stopped because changing the configuration may not happen while running. 2. The completedRequests queue will be cleared because the completed buffers have the wrong configuration (see below). 3. The new caps are negotiated by calling gst_libcamera_src_negotiate. When the negotiation fails streaming will stop. 4. The camera is started again. Clearing the completedRequests queue is archived with a new method in GstLibcameraSrcState called clearRequests. This function is now also used in gst_libcamera_src_task_leave as the code there did the same thing. In gst_libcamera_src_task_enter, after the initial negotiation, a call to gst_pad_check_reconfigure was added to clear the reconfigure flag to avoid triggering a renegotiation when running the task for the first time. Signed-off-by: Jaslo Ziska --- src/gstreamer/gstlibcamerasrc.cpp | 72 +++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 17 deletions(-) diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index e7a49fef..868fa20a 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -11,7 +11,6 @@ * - Implement GstElement::send_event * + Allowing application to use FLUSH/FLUSH_STOP * + Prevent the main thread from accessing streaming thread - * - Implement renegotiation (even if slow) * - Implement GstElement::request-new-pad (multi stream) * + Evaluate if a single streaming thread is fine * - Add application driven request (snapshot) @@ -133,6 +132,7 @@ struct GstLibcameraSrcState { int queueRequest(); void requestCompleted(Request *request); int processRequest(); + void clearRequests(); }; struct _GstLibcameraSrc { @@ -301,23 +301,39 @@ int GstLibcameraSrcState::processRequest() srcpad, ret); } - if (ret != GST_FLOW_OK) { - if (ret == GST_FLOW_EOS) { - g_autoptr(GstEvent) eos = gst_event_new_eos(); - guint32 seqnum = gst_util_seqnum_next(); - gst_event_set_seqnum(eos, seqnum); - for (GstPad *srcpad : srcpads_) - gst_pad_push_event(srcpad, gst_event_ref(eos)); - } else if (ret != GST_FLOW_FLUSHING) { - GST_ELEMENT_FLOW_ERROR(src_, ret); - } + switch (ret) { + case GST_FLOW_OK: + case GST_FLOW_NOT_NEGOTIATED: + break; + case GST_FLOW_EOS: { + g_autoptr(GstEvent) eos = gst_event_new_eos(); + guint32 seqnum = gst_util_seqnum_next(); + gst_event_set_seqnum(eos, seqnum); + for (GstPad *srcpad : srcpads_) + gst_pad_push_event(srcpad, gst_event_ref(eos)); + + err = -EPIPE; + break; + } + case GST_FLOW_FLUSHING: + err = -EPIPE; + break; + default: + GST_ELEMENT_FLOW_ERROR(src_, ret); - return -EPIPE; + err = -EPIPE; + break; } return err; } +void GstLibcameraSrcState::clearRequests() +{ + GLibLocker locker(&lock_); + completedRequests_ = {}; +} + static bool gst_libcamera_src_open(GstLibcameraSrc *self) { @@ -488,6 +504,31 @@ gst_libcamera_src_task_run(gpointer user_data) return; } + // check if a srcpad requested a renegotiation + gboolean reconfigure = FALSE; + for (GstPad *srcpad : state->srcpads_) { + if (gst_pad_check_reconfigure(srcpad)) { + // check whether the caps even need changing + g_autoptr(GstCaps) caps = gst_pad_get_current_caps(srcpad); + g_autoptr(GstCaps) intersection = gst_pad_peer_query_caps(srcpad, caps); + + if (gst_caps_is_empty(intersection)) + reconfigure = TRUE; + } + } + + if (reconfigure) { + state->cam_->stop(); + state->clearRequests(); + + if (!gst_libcamera_src_negotiate(self)) { + GST_ELEMENT_FLOW_ERROR(self, GST_FLOW_NOT_NEGOTIATED); + gst_task_stop(self->task); + } + + state->cam_->start(&state->initControls_); + } + /* * Create and queue one request. If no buffers are available the * function returns -ENOBUFS, which we ignore here as that's not a @@ -594,6 +635,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_segment_init(&segment, GST_FORMAT_TIME); gst_pad_push_event(srcpad, gst_event_new_segment(&segment)); + gst_pad_check_reconfigure(srcpad); // clear reconfigure flag } if (self->auto_focus_mode != controls::AfModeManual) { @@ -629,11 +671,7 @@ gst_libcamera_src_task_leave([[maybe_unused]] GstTask *task, GST_DEBUG_OBJECT(self, "Streaming thread is about to stop"); state->cam_->stop(); - - { - GLibLocker locker(&state->lock_); - state->completedRequests_ = {}; - } + state->clearRequests(); { GLibRecLocker locker(&self->stream_lock);