From patchwork Mon Dec 6 23:39:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 15056 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 85538BF415 for ; Mon, 6 Dec 2021 23:39:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8B02F60898; Tue, 7 Dec 2021 00:39:56 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ZxzsNUPD"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5FA3C60117 for ; Tue, 7 Dec 2021 00:39:54 +0100 (CET) Received: from Monstersaurus.local (cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D0FAE556; Tue, 7 Dec 2021 00:39:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1638833994; bh=ujAZAywscOsx91urn6hF+Mm7lSA6aqxdlP3eSwKsVIs=; h=From:To:Cc:Subject:Date:From; b=ZxzsNUPDdNkFMFftShxUPoctxuY/BAIAjdfpADqs0m2AoMUbOO2Bz6usWs1/uFSgX p51D3I1Y1HUpEShVj6oJDsOqBpKfaM0qi36mggtOEjEwdcmNxJOYM6qzY9Lbxnn2JR 44EU2OnwkzW05Z0XeTY57t3zmt9dy4z3L1xMzGGM= From: Kieran Bingham To: libcamera devel Date: Mon, 6 Dec 2021 23:39:40 +0000 Message-Id: <20211206233948.1351206-1-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 0/8] Request metadata: SensorSequence X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When completing a request, the individual stream buffers contain a sequence number. This number is generated by the device that ultimately fills that stream, but it might not be the sensor. Processing through an ISP could cause the sequence numbers and timestamp data associated with the completed buffer to be values representative of the ISP processing rather than the (intended) capture device. Provide a new metadata control, still to be fully sketched out which gives us a defined value to report the Camera sequence number. This allows pipeline handlers to correctly set the value according to the device that represents the capture from the sensor. This plumbing then allows applications to detect frame drops, which were otherwise going unnoticed, and as such some basic additions have been made to cam, qcam, and gstreamer to support this new data. Still possible: - Adding a validation to lc-compliance to make sure pipelines set the SensorSequence on every frame. - Probably expecting some better gstreamer event integration perhaps? - qcam should report more statistics on the processing overall - libcamera Tracepoints could be added as an event to track when frames are detected as dropped by the core framework. Anything else? Kieran Bingham (8): libcamera: controls: Add SensorSequence metadata control libcamera: pipeline: Set the Sensor sequence number for all pipelines cam: Use SensorTimestamp rather than buffer metadata cam: Use Sensor sequence numbers and detect frame drop qcam: main_window: Fix include ordering qcam: Use Sensor sequence numbers and detect frame drop gstreamer: gstlibcamerasrc: Fix include ordering gstreamer: Use Sensor sequence numbers and detect frame drop src/cam/camera_session.cpp | 24 ++++++++-- src/cam/camera_session.h | 1 + src/gstreamer/gstlibcamerasrc.cpp | 46 +++++++++++++++---- src/libcamera/control_ids.yaml | 8 ++++ src/libcamera/pipeline/ipu3/ipu3.cpp | 4 +- .../pipeline/raspberrypi/raspberrypi.cpp | 3 ++ src/libcamera/pipeline/rkisp1/rkisp1.cpp | 4 +- src/libcamera/pipeline/simple/simple.cpp | 12 +++-- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 2 + src/qcam/main_window.cpp | 28 +++++++++-- src/qcam/main_window.h | 1 + 11 files changed, 111 insertions(+), 22 deletions(-)