From patchwork Mon Dec 6 23:39:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15064 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E8871C325B for ; Mon, 6 Dec 2021 23:40:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 55483608A0; Tue, 7 Dec 2021 00:40:02 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="W2m6sX1M"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7A6EF60897 for ; Tue, 7 Dec 2021 00:39:56 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1CFF15B0; Tue, 7 Dec 2021 00:39:56 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833996; bh=ccv875roBOef5VHCE9Pk42QtScQxh7XLZAFWnEY5n8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W2m6sX1M4RvTCuzP56/9PhNOk/a8n+gqLG9Vw8xNt6jYfoZeXZnPMxNRJPclI+Ori 61spHn/WAjzIGvnf+TvoKrycFu5Om1nZNzqjksTpitMgd1RK4Lu3mfnAMWYpDDWVnL LRmCd/YNBD7CS35zZarSyKd4OyMwTTCo42IgoIBM= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:48 +0000 Message-Id: <20211206233948.1351206-9-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 8/8] gstreamer: Use Sensor sequence numbers and detect frame drop X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The stream buffer sequence numbers might produce sequential monotonic sequence numbers from an ISP producing a frame for every input. This however, doesn't capture pipeline stalls that cause us to miss or drop frames from the sensor. Use the SensorSequence metadata to report sequence information, and report to the application if a frame drop is detected. Signed-off-by: Kieran Bingham --- RFC: This one likely needs better plumbing into gstreamer events. Is there anything we should specifically signal to gst ? Is there a better way to report the errors? --- src/gstreamer/gstlibcamerasrc.cpp | 42 ++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index c7292f66b17b..8653127f0d85 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -33,6 +33,7 @@ #include #include +#include #include #include @@ -111,6 +112,8 @@ struct GstLibcameraSrcState { std::queue> requests_; guint group_id_; + guint64 sequence_; + void requestCompleted(Request *request); }; @@ -165,13 +168,40 @@ GstLibcameraSrcState::requestCompleted(Request *request) return; } + /* Extract request metadata */ + uint64_t sequence = 0; + uint64_t timestamp = 0; + + for (const auto &ctrl : request->metadata()) { + const int id = ctrl.first; + const ControlValue &value = ctrl.second; + + if (id == controls::SensorSequence) { + /* Handle basic frame drop detection and reporting. */ + sequence = value.get(); + if (sequence_ == 0) + sequence_ = sequence - 1; + unsigned int drops = sequence - sequence_ - 1; + if (drops) + GST_ELEMENT_WARNING(src_, RESOURCE, BUSY, + ("Camera '%s' dropped %u frames.", cam_->id().c_str(), drops), + ("libcamera::Request.metadata() reports %u dropped frames at sequence %lu", drops, sequence)); + + sequence_ = sequence; + } + + if (id == controls::SensorTimestamp) { + timestamp = value.get(); + } + + /* \todo Handle all/other metadata types here. */ + } + GstBuffer *buffer; for (GstPad *srcpad : srcpads_) { Stream *stream = gst_libcamera_pad_get_stream(srcpad); buffer = wrap->detachBuffer(stream); - FrameBuffer *fb = gst_libcamera_buffer_get_frame_buffer(buffer); - if (GST_ELEMENT_CLOCK(src_)) { GstClockTime gst_base_time = GST_ELEMENT(src_)->base_time; GstClockTime gst_now = gst_clock_get_time(GST_ELEMENT_CLOCK(src_)); @@ -180,14 +210,14 @@ GstLibcameraSrcState::requestCompleted(Request *request) /* Deduced from: sys_now - sys_base_time == gst_now - gst_base_time */ GstClockTime sys_base_time = sys_now - (gst_now - gst_base_time); - GST_BUFFER_PTS(buffer) = fb->metadata().timestamp - sys_base_time; - gst_libcamera_pad_set_latency(srcpad, sys_now - fb->metadata().timestamp); + GST_BUFFER_PTS(buffer) = timestamp - sys_base_time; + gst_libcamera_pad_set_latency(srcpad, sys_now - timestamp); } else { GST_BUFFER_PTS(buffer) = 0; } - GST_BUFFER_OFFSET(buffer) = fb->metadata().sequence; - GST_BUFFER_OFFSET_END(buffer) = fb->metadata().sequence; + GST_BUFFER_OFFSET(buffer) = sequence; + GST_BUFFER_OFFSET_END(buffer) = sequence; gst_libcamera_pad_queue_buffer(srcpad, buffer); }