From patchwork Mon Dec 6 23:39:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15057 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 35BF6BF415 for ; Mon, 6 Dec 2021 23:39:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A30186087A; Tue, 7 Dec 2021 00:39:57 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="k7tSre0c"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 90427607DE for ; Tue, 7 Dec 2021 00:39:54 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 23BC85B0; Tue, 7 Dec 2021 00:39:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833994; bh=x2ujNUKNqqtxSYhUw/sSWerA77d8prx+H6BHd8JVuvU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=k7tSre0cQjOGGvU+rQbUsuVpakuwVH2V++VNeajnVjFC5ilU7hv/0NnbTUwdWsgeg m01jsM/V+lYFsUfiiYgXn+/hj6oBWrVBPOnUJRh+yDl+9kW/xPaEkbSwdGhoPl5h72 pVnqCJ8ywrgGZBhXVgIEaFrUvD4a9o7/hLSlTrP4= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:41 +0000 Message-Id: <20211206233948.1351206-2-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/8] libcamera: controls: Add SensorSequence metadata control X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Provide a metadata control that allows pipeline handlers to report the sequence number from the sensor for every request. This can be used to monitor and validate the sequence numbers of the captured images and detect any frame drops. Signed-off-by: Kieran Bingham Reviewed-by: Naushir Patuck --- src/libcamera/control_ids.yaml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 9d4638ae677e..8874fe75102c 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -381,6 +381,14 @@ controls: \todo Define how the sensor timestamp has to be used in the reprocessing use case. + - SensorSequence: + type: int64_t + description: | + The monotonic sequence number from the sensor. + + Any break in the sequence number is an indication of frame drop or error + in the captured stream. + # ---------------------------------------------------------------------------- # Draft controls section From patchwork Mon Dec 6 23:39:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15058 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 47AFAC324B for ; Mon, 6 Dec 2021 23:40:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 950F86088B; Tue, 7 Dec 2021 00:39:58 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="CiuwlI68"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CA5456086C for ; Tue, 7 Dec 2021 00:39:54 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 694E1A1A; Tue, 7 Dec 2021 00:39:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833994; bh=9sS+DqXITQIA53Wy/oaQIbdNzQPKykliP5txDOgCS84=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CiuwlI683/yX8a2P93VWVvf55Ph8piX2vKO5+HyvlgjHfLTCW4KNBTOsJA7+wsFIG HtxiQveoV1oKBRduTJaNPChcRWDW1kZFt9Q4HZic7xtlJDAjOp4Sh/D3xWatu/kVJy XH1oYY+pSFXYXqyACwXTU0Dh1G9QLn5M4Wn2essI= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:42 +0000 Message-Id: <20211206233948.1351206-3-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/8] libcamera: pipeline: Set the Sensor sequence number for all pipelines X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The SensorSequence metadata provides the sequence number associated with the image capture directly. While we already report sequence numbers in the metadata of the buffers, this can lead to mis-leading parsing of those sequence numbers which may be purely monotonic sequences from an ISP, and not represent the true sequence counts from a capture device. Use the newly added SensorSequence metadata control and populate completed requests with the sequence number from the appropriate stream. For ISP based devices, such as the IPU3, RPi, and RkISP1, this comes from the CSI2 receiver capture device, while for the Simple pipeline and the UVC Video pipeline, this comes from the single video capture device. Signed-off-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Naushir Patuck --- src/libcamera/pipeline/ipu3/ipu3.cpp | 4 +++- src/libcamera/pipeline/raspberrypi/raspberrypi.cpp | 3 +++ src/libcamera/pipeline/rkisp1/rkisp1.cpp | 4 +++- src/libcamera/pipeline/simple/simple.cpp | 12 ++++++++---- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 2 ++ 5 files changed, 19 insertions(+), 6 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 313220624aff..a7f35840afbb 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -1372,13 +1372,15 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer) } /* - * Record the sensor's timestamp in the request metadata. + * Record the sensor's timestamp and sequence in the request metadata. * * \todo The sensor timestamp should be better estimated by connecting * to the V4L2Device::frameStart signal. */ request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + request->metadata().set(controls::SensorSequence, + buffer->metadata().sequence); info->effectiveSensorControls = delayedCtrls_->get(buffer->metadata().sequence); diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 321b72adbbf7..f458587a8920 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1549,6 +1549,7 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) * as it does not receive the FrameBuffer object. */ ctrl.set(controls::SensorTimestamp, buffer->metadata().timestamp); + ctrl.set(controls::SensorSequence, buffer->metadata().sequence); bayerQueue_.push({ buffer, std::move(ctrl) }); } else { embeddedQueue_.push(buffer); @@ -1786,6 +1787,8 @@ void RPiCameraData::fillRequestMetadata(const ControlList &bufferControls, { request->metadata().set(controls::SensorTimestamp, bufferControls.get(controls::SensorTimestamp)); + request->metadata().set(controls::SensorSequence, + bufferControls.get(controls::SensorSequence)); request->metadata().set(controls::ScalerCrop, scalerCrop_); } diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 36ef6a02ae90..27592e473c14 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -1077,13 +1077,15 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer) Request *request = buffer->request(); /* - * Record the sensor's timestamp in the request metadata. + * Record the sensor's timestamp and sequence in the request metadata. * * \todo The sensor timestamp should be better estimated by connecting * to the V4L2Device::frameStart signal. */ request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + request->metadata().set(controls::SensorSequence, + buffer->metadata().sequence); completeBuffer(request, buffer); tryCompleteRequest(request); diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 701fb4be0b71..4c5d9c0b9919 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -667,9 +667,10 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) } /* - * Record the sensor's timestamp in the request metadata. The request - * needs to be obtained from the user-facing buffer, as internal - * buffers are free-wheeling and have no request associated with them. + * Record the sensor's timestamp and sequence in the request metadata. + * The request needs to be obtained from the user-facing buffer, as + * internal buffers are free-wheeling and have no request associated + * with them. * * \todo The sensor timestamp should be better estimated by connecting * to the V4L2Device::frameStart signal if the platform provides it. @@ -686,9 +687,12 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) } } - if (request) + if (request) { request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + request->metadata().set(controls::SensorSequence, + buffer->metadata().sequence); + } /* * Queue the captured and the request buffer to the converter if format diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 264f5370cf32..f5fab3b69e6d 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -666,6 +666,8 @@ void UVCCameraData::bufferReady(FrameBuffer *buffer) /* \todo Use the UVC metadata to calculate a more precise timestamp */ request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + request->metadata().set(controls::SensorSequence, + buffer->metadata().sequence); pipe()->completeBuffer(request, buffer); pipe()->completeRequest(request); From patchwork Mon Dec 6 23:39:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15059 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D7B94BF415 for ; Mon, 6 Dec 2021 23:40:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DDD8E60897; Tue, 7 Dec 2021 00:39:58 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="TYk2xxT3"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0F640607DE for ; Tue, 7 Dec 2021 00:39:55 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id ACCDD11FE; Tue, 7 Dec 2021 00:39:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833994; bh=jyFT+4JF+7jG1gz19qvWB1h+iBfIhy527w6TaN2IIJk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TYk2xxT31CtDlMBfj96d3zQD3XiOxHqJuoAjtzBtxtXB+DUdDZkIDAyyOzUFowaXq 8IJKgdRU6WTf8Tyir4muBN2GNcr//reABVRI0zvA2eTWC+tPOlTw/cCPwRGY2mv3Y5 2p46g5E4ekLr0Pd6lU/SnA8T9d81P9bm5fMZI5HE= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:43 +0000 Message-Id: <20211206233948.1351206-4-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/8] cam: Use SensorTimestamp rather than buffer metadata X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The SensorTimestamp is defined to be the time of capture of the image. While all streams should have the same timestamp, this is not always defined or guaranteed as ISP drivers may not forward sequence numbers and timestamps from their input buffer. Use the Request metadata to get the SensorTimestamp which must be set by the pipeline handlers according to the data from the capture device. Signed-off-by: Kieran Bingham Reviewed-by: Umang Jain --- src/cam/camera_session.cpp | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp index 1bf460fa3fb7..50170723c30f 100644 --- a/src/cam/camera_session.cpp +++ b/src/cam/camera_session.cpp @@ -359,10 +359,11 @@ void CameraSession::processRequest(Request *request) const Request::BufferMap &buffers = request->buffers(); /* - * Compute the frame rate. The timestamp is arbitrarily retrieved from - * the first buffer, as all buffers should have matching timestamps. + * Compute the frame rate. The timestamp is retrieved from the + * SensorTimestamp property, though all streams should have the + * same timestamp. */ - uint64_t ts = buffers.begin()->second->metadata().timestamp; + uint64_t ts = request->metadata().get(controls::SensorTimestamp); double fps = ts - last_; fps = last_ != 0 && fps ? 1000000000.0 / fps : 0.0; last_ = ts; From patchwork Mon Dec 6 23:39:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15060 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8727AC3258 for ; Mon, 6 Dec 2021 23:40:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AD6AA608E0; Tue, 7 Dec 2021 00:39:59 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ln9Xg4L1"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4A327607DE for ; Tue, 7 Dec 2021 00:39:55 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id EFE9F556; Tue, 7 Dec 2021 00:39:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833995; bh=FuGogmiA11ukCqWdkU6nDMdWFzqyUmYeDshrP/O2gkc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ln9Xg4L1fQzJg0ixGV9wOofibCEqRDuKP6U8EPW2vtzp8dqmaEPb9N2twrGWQ8jmG DFDQ2j7Cto+6hDu9HbbNOWy9aoeWK6xjzCPKOcfJz7z1HpJm+yIEyKsGDY5dvGkUoi JmnZdtrtjCc65WbyZCrao1ylHhSL3YeFdTkpTaOo= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:44 +0000 Message-Id: <20211206233948.1351206-5-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 4/8] cam: Use Sensor sequence numbers and detect frame drop X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The stream buffer sequence numbers might produce sequential monotonic sequence numbers from an ISP producing a frame for every input. This however, doesn't capture pipeline stalls that cause us to miss or drop frames from the sensor. Use the SensorSequence metadata to report sequence information, and add a warning to the summary line if a frame drop is detected. Signed-off-by: Kieran Bingham Reviewed-by: Umang Jain --- src/cam/camera_session.cpp | 17 +++++++++++++++-- src/cam/camera_session.h | 1 + 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp index 50170723c30f..f9495f7f7e42 100644 --- a/src/cam/camera_session.cpp +++ b/src/cam/camera_session.cpp @@ -29,8 +29,8 @@ CameraSession::CameraSession(CameraManager *cm, unsigned int cameraIndex, const OptionsParser::Options &options) : options_(options), cameraIndex_(cameraIndex), last_(0), - queueCount_(0), captureCount_(0), captureLimit_(0), - printMetadata_(false) + sequence_(0), queueCount_(0), captureCount_(0), + captureLimit_(0), printMetadata_(false) { char *endptr; unsigned long index = strtoul(cameraId.c_str(), &endptr, 10); @@ -375,6 +375,11 @@ void CameraSession::processRequest(Request *request) << std::setw(6) << std::setfill('0') << ts / 1000 % 1000000 << " (" << std::fixed << std::setprecision(2) << fps << " fps)"; + uint64_t sequence = request->metadata().get(controls::SensorSequence); + // Report capture sequence number + info << " [" << std::setw(6) << std::setfill('0') + << sequence << "]"; + for (auto it = buffers.begin(); it != buffers.end(); ++it) { const Stream *stream = it->first; FrameBuffer *buffer = it->second; @@ -398,6 +403,14 @@ void CameraSession::processRequest(Request *request) requeue = false; } + /* Handle basic frame drop detection and reporting. */ + if (sequence_ == 0) + sequence_ = sequence - 1; + unsigned int drops = sequence - sequence_ - 1; + if (drops) + info << " *" << drops << " frame drops detected* "; + sequence_ = sequence; + std::cout << info.str() << std::endl; if (printMetadata_) { diff --git a/src/cam/camera_session.h b/src/cam/camera_session.h index bf966bd15ab0..14dc1421da4d 100644 --- a/src/cam/camera_session.h +++ b/src/cam/camera_session.h @@ -65,6 +65,7 @@ private: unsigned int cameraIndex_; uint64_t last_; + uint64_t sequence_; unsigned int queueCount_; unsigned int captureCount_; From patchwork Mon Dec 6 23:39:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15061 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 09CFCC324B for ; Mon, 6 Dec 2021 23:40:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 979CE60882; Tue, 7 Dec 2021 00:40:00 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="uZ3l5kIX"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 965816086C for ; Tue, 7 Dec 2021 00:39:55 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 41A4A5B0; Tue, 7 Dec 2021 00:39:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833995; bh=jAuFgW3uGmumLGQnnchfLuWmm3zBGqHn9+ADZgD5aEU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uZ3l5kIXZu7zbjFU8hHgEfFVcdgQLH28raBnWA6xWfBJaNsESegSHR4wisuWgeT0O giEbPRsQ+1/op7UgknW7iUw/VtzAtoPXsg7CcHqwARyWZmOLpYrNikjOhadzKnlkqz pxXKi/I69vIvXTVSB8J858BI4HaiwvlQTvNyK3bM= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:45 +0000 Message-Id: <20211206233948.1351206-6-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 5/8] qcam: main_window: Fix include ordering X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The libcamera coding style has libcamera headers after system headers, and before any other library headers. Move the libcamera headers above the QT headers accordingly. Signed-off-by: Kieran Bingham Reviewed-by: Laurent Pinchart Reviewed-by: Umang Jain --- src/qcam/main_window.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp index ae51fa672a8c..dd0e51f55b70 100644 --- a/src/qcam/main_window.cpp +++ b/src/qcam/main_window.cpp @@ -11,6 +11,9 @@ #include #include +#include +#include + #include #include #include @@ -25,9 +28,6 @@ #include #include -#include -#include - #include "../cam/image.h" #include "dng_writer.h" #ifndef QT_NO_OPENGL From patchwork Mon Dec 6 23:39:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15062 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5EEFBC3259 for ; Mon, 6 Dec 2021 23:40:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 994C36087A; Tue, 7 Dec 2021 00:40:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="RCzz1FBC"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DE9B36086C for ; Tue, 7 Dec 2021 00:39:55 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 84E1CA1A; Tue, 7 Dec 2021 00:39:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833995; bh=c9W5V0c1b0I7kHfI/zzAV6kOyl6NOrnkM3mB7FO/ScA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RCzz1FBCsrqvVykG8AImWa5CjVYoA/g+vGSCCGFOlIMwNWowdP5QYGxd5ury/HiVP OKbJiFijNtGDjjAyCqfjL/Kwx0BJxV5T/lrxHN96JVMzSLGvWA6/azxz/NWxK9BskV iCG9RQZtRUis+LsZ3JXksGV8LqM4wkyZ4MN02gtU= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:46 +0000 Message-Id: <20211206233948.1351206-7-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 6/8] qcam: Use Sensor sequence numbers and detect frame drop X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The stream buffer sequence numbers might produce sequential monotonic sequence numbers from an ISP producing a frame for every input. This however, doesn't capture pipeline stalls that cause us to miss or drop frames from the sensor. Use the SensorSequence metadata to report sequence information, and report with qInfo if a frame drop is detected. Signed-off-by: Kieran Bingham Reviewed-by: Umang Jain --- src/qcam/main_window.cpp | 22 ++++++++++++++++++++++ src/qcam/main_window.h | 1 + 2 files changed, 23 insertions(+) diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp index dd0e51f55b70..8d1d5b275aed 100644 --- a/src/qcam/main_window.cpp +++ b/src/qcam/main_window.cpp @@ -12,6 +12,7 @@ #include #include +#include #include #include @@ -510,6 +511,7 @@ int MainWindow::startCapture() previousFrames_ = 0; framesCaptured_ = 0; lastBufferTime_ = 0; + sequence_ = 0; ret = camera_->start(); if (ret) { @@ -719,6 +721,26 @@ void MainWindow::processCapture() request = doneQueue_.dequeue(); } + /* Parse the request metadata for useful information */ + for (const auto &ctrl : request->metadata()) { + const int id = ctrl.first; + const ControlValue &value = ctrl.second; + + if (id == controls::SensorSequence) { + /* Handle basic frame drop detection and reporting. */ + int64_t sequence = value.get(); + if (sequence_ == 0) + sequence_ = sequence - 1; + unsigned int drops = sequence - sequence_ - 1; + if (drops) + qInfo() << "sequence: [" << sequence << "] *" + << drops << " frame drops detected *"; + sequence_ = sequence; + } + + /* \todo Handle all/other metadata types here. */ + } + /* Process buffers. */ if (request->buffers().count(vfStream_)) processViewfinder(request->buffers().at(vfStream_)); diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h index 3fbe872c0b5b..97568d6b619d 100644 --- a/src/qcam/main_window.h +++ b/src/qcam/main_window.h @@ -120,6 +120,7 @@ private: QMutex mutex_; /* Protects freeBuffers_, doneQueue_, and freeQueue_ */ uint64_t lastBufferTime_; + uint64_t sequence_; QElapsedTimer frameRateInterval_; uint32_t previousFrames_; uint32_t framesCaptured_; From patchwork Mon Dec 6 23:39:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15063 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AFAA7C325A for ; Mon, 6 Dec 2021 23:40:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F247960898; Tue, 7 Dec 2021 00:40:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="M1yHlJd3"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2C97B60890 for ; Tue, 7 Dec 2021 00:39:56 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id CAC4B556; Tue, 7 Dec 2021 00:39:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833995; bh=yRFK2wQI0wUltPLmYwPF+UXyW4E8vwk15FVtH27Mkos=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=M1yHlJd3Rxsn0Z4rt4vM+O0/2dtKY92SbSiWbxQ7kIJ9rllnDNiMtKbs/ANg9hk8/ ibwfhVUSfxpuE/piZB3ZbVvyKtT0eS+nrpfSq0rk1staUGMCquRtcz2XypdbO77LkR MPtw7/CIDMsIEHLv7JvfOEFtLW2u2bsXRZ8fZBfk= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:47 +0000 Message-Id: <20211206233948.1351206-8-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 7/8] gstreamer: gstlibcamerasrc: Fix include ordering X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The libcamera coding style has libcamera headers after system headers, and before any other library headers. Move the libcamera headers above the Gst headers accordingly. Signed-off-by: Kieran Bingham Reviewed-by: Laurent Pinchart Reviewed-by: Umang Jain --- src/gstreamer/gstlibcamerasrc.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 812ba7a2bd6e..c7292f66b17b 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -32,11 +32,11 @@ #include #include -#include - #include #include +#include + #include "gstlibcameraallocator.h" #include "gstlibcamerapad.h" #include "gstlibcamerapool.h" From patchwork Mon Dec 6 23:39:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15064 X-Patchwork-Delegate: kieran.bingham@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E8871C325B for ; Mon, 6 Dec 2021 23:40:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 55483608A0; Tue, 7 Dec 2021 00:40:02 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="W2m6sX1M"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7A6EF60897 for ; Tue, 7 Dec 2021 00:39:56 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1CFF15B0; Tue, 7 Dec 2021 00:39:56 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833996; bh=ccv875roBOef5VHCE9Pk42QtScQxh7XLZAFWnEY5n8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W2m6sX1M4RvTCuzP56/9PhNOk/a8n+gqLG9Vw8xNt6jYfoZeXZnPMxNRJPclI+Ori 61spHn/WAjzIGvnf+TvoKrycFu5Om1nZNzqjksTpitMgd1RK4Lu3mfnAMWYpDDWVnL LRmCd/YNBD7CS35zZarSyKd4OyMwTTCo42IgoIBM= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:48 +0000 Message-Id: <20211206233948.1351206-9-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> References: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 8/8] gstreamer: Use Sensor sequence numbers and detect frame drop X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The stream buffer sequence numbers might produce sequential monotonic sequence numbers from an ISP producing a frame for every input. This however, doesn't capture pipeline stalls that cause us to miss or drop frames from the sensor. Use the SensorSequence metadata to report sequence information, and report to the application if a frame drop is detected. Signed-off-by: Kieran Bingham --- RFC: This one likely needs better plumbing into gstreamer events. Is there anything we should specifically signal to gst ? Is there a better way to report the errors? --- src/gstreamer/gstlibcamerasrc.cpp | 42 ++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index c7292f66b17b..8653127f0d85 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -33,6 +33,7 @@ #include #include +#include #include #include @@ -111,6 +112,8 @@ struct GstLibcameraSrcState { std::queue> requests_; guint group_id_; + guint64 sequence_; + void requestCompleted(Request *request); }; @@ -165,13 +168,40 @@ GstLibcameraSrcState::requestCompleted(Request *request) return; } + /* Extract request metadata */ + uint64_t sequence = 0; + uint64_t timestamp = 0; + + for (const auto &ctrl : request->metadata()) { + const int id = ctrl.first; + const ControlValue &value = ctrl.second; + + if (id == controls::SensorSequence) { + /* Handle basic frame drop detection and reporting. */ + sequence = value.get(); + if (sequence_ == 0) + sequence_ = sequence - 1; + unsigned int drops = sequence - sequence_ - 1; + if (drops) + GST_ELEMENT_WARNING(src_, RESOURCE, BUSY, + ("Camera '%s' dropped %u frames.", cam_->id().c_str(), drops), + ("libcamera::Request.metadata() reports %u dropped frames at sequence %lu", drops, sequence)); + + sequence_ = sequence; + } + + if (id == controls::SensorTimestamp) { + timestamp = value.get(); + } + + /* \todo Handle all/other metadata types here. */ + } + GstBuffer *buffer; for (GstPad *srcpad : srcpads_) { Stream *stream = gst_libcamera_pad_get_stream(srcpad); buffer = wrap->detachBuffer(stream); - FrameBuffer *fb = gst_libcamera_buffer_get_frame_buffer(buffer); - if (GST_ELEMENT_CLOCK(src_)) { GstClockTime gst_base_time = GST_ELEMENT(src_)->base_time; GstClockTime gst_now = gst_clock_get_time(GST_ELEMENT_CLOCK(src_)); @@ -180,14 +210,14 @@ GstLibcameraSrcState::requestCompleted(Request *request) /* Deduced from: sys_now - sys_base_time == gst_now - gst_base_time */ GstClockTime sys_base_time = sys_now - (gst_now - gst_base_time); - GST_BUFFER_PTS(buffer) = fb->metadata().timestamp - sys_base_time; - gst_libcamera_pad_set_latency(srcpad, sys_now - fb->metadata().timestamp); + GST_BUFFER_PTS(buffer) = timestamp - sys_base_time; + gst_libcamera_pad_set_latency(srcpad, sys_now - timestamp); } else { GST_BUFFER_PTS(buffer) = 0; } - GST_BUFFER_OFFSET(buffer) = fb->metadata().sequence; - GST_BUFFER_OFFSET_END(buffer) = fb->metadata().sequence; + GST_BUFFER_OFFSET(buffer) = sequence; + GST_BUFFER_OFFSET_END(buffer) = sequence; gst_libcamera_pad_queue_buffer(srcpad, buffer); }