From patchwork Fri Aug 25 06:37:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18961 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CAC5AC3262 for ; Fri, 25 Aug 2023 06:37:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8013E61F19; Fri, 25 Aug 2023 08:37:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692945452; bh=CPhyN4E/4hqI8BZ3sYxX5MSPwp2ywtnMwIlrtJol5XY=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=xWqg4nrw2aiGQiiirf+G5ooOkE2qBfv7O+h6BZokVuM9B61kpy2UFyN8ar1K4fzI4 eyCpvHMBXdIPPckDZt2mBf0mjZxTZboi6c7TFj5k/23eekkdtDsV+qYkTgDaRgB0q7 9v5AcJTy0r7+xG29AVwKG2D6hFvDCNwTQLwCgv94FtFO/0n3Bu0lSdOQVV3Db0108h waRJv/KS95BepoM3dL1NCpqj4XC93LErXxLkuDOPiNnG4j7GimU+UnUBF0VO+qxh+c 9BfVpaDrfTmK4yp3lGVuAH9XkaFTiOMcmUEN8S1j/I66A1OOOdLGnCmDEC1z9WUeJh 3dTcJG/le1CMQ== Received: from mail-oo1-xc2d.google.com (mail-oo1-xc2d.google.com [IPv6:2607:f8b0:4864:20::c2d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AB19161E01 for ; Fri, 25 Aug 2023 08:37:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ET8powXy"; dkim-atps=neutral Received: by mail-oo1-xc2d.google.com with SMTP id 006d021491bc7-5732481b22eso421739eaf.3 for ; Thu, 24 Aug 2023 23:37:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692945447; x=1693550247; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2Onkgtntva6Jxzg6cR1B7ztmbjs3DRkoZm6St4PeQ3g=; b=ET8powXyYcxq/uYhpmkP22qFo7qOo79TBH/G+ZLBFDOmn/9LnOZT/P9RfcC/iepJ8K SirDRFcmbb+2aOwk942aagLXIRm2Ee5HWoB87odXLyZROKdW5TGXJNiUdm74X6Pi3zye ugFQyRbu5vXoZ1pK6qFOnwYC/j+LL21/EDTS3+9P50rkehbjn7OFydZeDcnca84qMfbp cr0nf+FtBeQjundYviVJntEjNZWq93cVTD5LFcf2z/cv/j/hdKHJubVcXfoDIADB8Csx lxw8jvu3g362CaFTPtERdlFyc5aVb7bIjBAbouGOb/ueY6gdksJN7JkX3dt3XMHFTRZy nTbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692945447; x=1693550247; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2Onkgtntva6Jxzg6cR1B7ztmbjs3DRkoZm6St4PeQ3g=; b=hx8eSE3xn7TfcvdN5elcDu2jcGu/HQGS2uJqUXGYR0dM9p4a9rsoPj0bmFpGwiBrTw TMJfckFgvxv966e2QxR72vCohO6BEKgafwePG2WLflt5RTvm53/rM7fWD/cyXbGAzrU9 ZiC6n0MqKXvd2ibDao3QlPZUu9Ut8wbNjSSsQZ4tgyLavFIEjsw+Myx+BK+A7ihFdCrD IowcL4H0p4zPoWxHn9UW9ne/RMLPAfICxJekesau8EqiMCVgdGBxY+77bl+A0dUzIadu bx3OWRTnyLvHZ+/bDgMyw6miEbkPceseQXNm8re1mXfTnqEQ0FmkLnNM7tKApXV6WfO3 eJTg== X-Gm-Message-State: AOJu0YzeWsPdkgsIBBltWdIDQF8sdcMr+ZSX2Si+IkjVJ8ugv14F//5s GBGTvdZihj8sp9xv/fu1JaScoju5OtI= X-Google-Smtp-Source: AGHT+IFp2crJPQ7H/yU9vR/4Y2xjdO9hZSbgZwxjKSTLkUlfctE6gdHJpxh2Yscv+63XrB8gXKPhFw== X-Received: by 2002:a4a:d2c8:0:b0:56d:2cbf:2315 with SMTP id j8-20020a4ad2c8000000b0056d2cbf2315mr4987011oos.9.1692945446948; Thu, 24 Aug 2023 23:37:26 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id u41-20020a4a8c2c000000b0056688eea98csm623239ooj.27.2023.08.24.23.37.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 23:37:26 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Thu, 24 Aug 2023 23:37:06 -0700 Message-Id: <20230825063707.8734-2-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230825063707.8734-1-gabbymg94@gmail.com> References: <20230825063707.8734-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/2] libcamera: pipeline: uvcvideo: Store time metadata X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Parse the raw timing data out of the metadata packets and store each sample in a circular buffer of size 32. To convert the latest frame's metadata into a system clock timestamp, use the metadata time information from 32 frames back stored in the back of the circular buffer. For every metadata buffer that comes in, store the timing information in the circular queue. The information at the head of the queue may be used instantly in the case where a corresponding video buffer has come in, or it may be used when the video buffer eventually does arrive. In any case, the information at the head of the queue will be used when it comes time for the timestamp to be set in the request metadata for the video buffer. If there has only been one frame of metadata, use the default (video buffer-provided) timestamp. If the circular buffer has not yet been filled with 32 frames, use the oldest frame. When the circular buffer reaches 32 frames, remove the oldest item. This approach creates synchronization between the metadata buffer that will be used to update the video buffer's requests' timestamp and the timing data itself, without having to perform checks ensuring synchronization. It also creates easy and efficient access to the front and back of the circular buffer. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 76 +++++++++++++++++--- 1 file changed, 67 insertions(+), 9 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 75a949b8..43ce4f8a 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -47,6 +48,17 @@ struct UVCTimingBuf { __u16 sofDevice; } __attribute__((packed)); +/* Raw timestamp input to the timestamp calculation function. */ +struct UVCTimestampData { + unsigned long long tsHost; /* System clock timestamp in nanoseconds*/ + unsigned short sofHost; /* The usb clock at the time tsHost was taken*/ + unsigned int stcDevice; /* The UVC device source timestamp */ + unsigned short sofDevice; /* the usb clock at the time the STC was taken*/ + + /* presentation time stamp to be converted into a system clock timestamp */ + unsigned int ptsDevice; +}; + class UVCCameraData : public Camera::Private { public: @@ -72,15 +84,18 @@ public: std::map> formats_; std::queue pendingVideoBuffers_; - std::queue> pendingMetadata_; + std::queue pendingMetadata_; private: int initMetadata(MediaDevice *media); void completeRequest(FrameBuffer *buffer, uint64_t timestamp); void endCorruptedStream(); + void addTimestampData(uvc_meta_buf &rawMetadata, UVCTimingBuf &packed); + std::deque timeSamples_; const unsigned int frameStart_ = 1; const unsigned int maxVidBuffersInQueue_ = 1; + const unsigned int bufferRingSize_ = 32; bool generateId(); @@ -899,6 +914,13 @@ void UVCCameraData::endCorruptedStream() << "UVC metadata stream corrupted. Reverting to driver timestamps."; } +unsigned long long calculateTimestamp([[maybe_unused]] const UVCTimestampData &p1, + const UVCTimestampData &p2, + [[maybe_unused]] const unsigned int PTS) +{ + return p2.tsHost; +} + /* * If there is a metadata buffer that hasn't been matched with a * video buffer, check to see if it matches this video buffer. @@ -929,9 +951,17 @@ void UVCCameraData::bufferReady(FrameBuffer *buffer) if (!pendingMetadata_.empty()) { /* A metadata buffer was ready first. */ - unsigned int mdSequence = std::get<0>(pendingMetadata_.front()) + frameStart_; + unsigned int mdSequence = pendingMetadata_.front() + frameStart_; if (mdSequence == buffer->metadata().sequence) { - completeRequest(buffer, std::get<1>(pendingMetadata_.front())); + unsigned long long timestamp; + if (timeSamples_.size() > 1) { + timestamp = calculateTimestamp(timeSamples_.front(), + timeSamples_.back(), + timeSamples_.back().ptsDevice); + } else { + timestamp = buffer->metadata().timestamp; + } + completeRequest(buffer, timestamp); pendingMetadata_.pop(); return; } else { @@ -951,6 +981,28 @@ void UVCCameraData::bufferReady(FrameBuffer *buffer) } } +void UVCCameraData::addTimestampData(uvc_meta_buf &rawMetadata, UVCTimingBuf &packed) +{ + /* + * Copy over the buffer packet from the raw Metadata + * into values we can use. Populate the storage struct + * with the data we need to calculate timestamps. + * Add to the circular queue. + */ + UVCTimestampData data; + data.ptsDevice = packed.pts; + data.sofDevice = packed.sofDevice; + data.stcDevice = packed.stc; + data.sofHost = rawMetadata.sof; + data.tsHost = rawMetadata.ns; + + if (timeSamples_.size() == bufferRingSize_) { + timeSamples_.pop_front(); + } + + timeSamples_.push_back(data); +} + void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) { if (!metadata_ || @@ -971,8 +1023,8 @@ void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) Span memMeta = mappedMetadataBuffers_.at(pos).planes()[0]; uvc_meta_buf *metaBuf = reinterpret_cast(memMeta.data()); - //Span memTime = mappedMetadataBuffers_.at(pos).planes()[0]; - //UVCTimingBuf * timeBuf = reinterpret_cast(&memTime.data()[sizeof(uvc_meta_buf)]); + Span memTime = mappedMetadataBuffers_.at(pos).planes()[0]; + UVCTimingBuf *timeBuf = reinterpret_cast(&memTime.data()[sizeof(uvc_meta_buf)]); size_t UVCPayloadHeaderSize = sizeof(metaBuf->length) + sizeof(metaBuf->flags) + sizeof(UVCTimingBuf); @@ -981,6 +1033,8 @@ void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) return; } + addTimestampData(*metaBuf, *timeBuf); + /* * Match a pending video buffer with this buffer's sequence. If * there is none available, put this timestamp information on the @@ -992,15 +1046,19 @@ void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) unsigned int vidSequence = vidBuffer->metadata().sequence; if (vidSequence == mdSequence) { - completeRequest(vidBuffer, static_cast(metaBuf->ns)); + unsigned long long timestamp; + if (timeSamples_.size() > 1) { + timestamp = calculateTimestamp(timeSamples_.front(), + timeSamples_.back(), + timeSamples_.back().ptsDevice); + } + completeRequest(vidBuffer, timestamp); pendingVideoBuffers_.pop(); } else { endCorruptedStream(); } } else { - pendingMetadata_.push( - std::make_pair(buffer->metadata().sequence, - static_cast(metaBuf->ns))); + pendingMetadata_.push(buffer->metadata().sequence); } metadata_->queueBuffer(buffer); }