{"id":18961,"url":"https://patchwork.libcamera.org/api/1.1/patches/18961/?format=json","web_url":"https://patchwork.libcamera.org/patch/18961/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20230825063707.8734-2-gabbymg94@gmail.com>","date":"2023-08-25T06:37:06","name":"[libcamera-devel,1/2] libcamera: pipeline: uvcvideo: Store time metadata","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"e727ba5a8cfa71a4e0360b4e12b446fc01793d6c","submitter":{"id":160,"url":"https://patchwork.libcamera.org/api/1.1/people/160/?format=json","name":"Gabrielle George","email":"gabbymg94@gmail.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/18961/mbox/","series":[{"id":4009,"url":"https://patchwork.libcamera.org/api/1.1/series/4009/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4009","date":"2023-08-25T06:37:05","name":"Calculation of new UVC timestamps","version":1,"mbox":"https://patchwork.libcamera.org/series/4009/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/18961/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/18961/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id CAC5AC3262\n\tfor ;\n\tFri, 25 Aug 2023 06:37:32 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8013E61F19;\n\tFri, 25 Aug 2023 08:37:32 +0200 (CEST)","from mail-oo1-xc2d.google.com (mail-oo1-xc2d.google.com\n\t[IPv6:2607:f8b0:4864:20::c2d])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AB19161E01\n\tfor ;\n\tFri, 25 Aug 2023 08:37:28 +0200 (CEST)","by mail-oo1-xc2d.google.com with SMTP id\n\t006d021491bc7-5732481b22eso421739eaf.3\n\tfor ;\n\tThu, 24 Aug 2023 23:37:28 -0700 (PDT)","from localhost.localdomain (97-115-76-16.ptld.qwest.net.\n\t[97.115.76.16]) by smtp.gmail.com with ESMTPSA id\n\tu41-20020a4a8c2c000000b0056688eea98csm623239ooj.27.2023.08.24.23.37.26\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tThu, 24 Aug 2023 23:37:26 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1692945452;\n\tbh=CPhyN4E/4hqI8BZ3sYxX5MSPwp2ywtnMwIlrtJol5XY=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=xWqg4nrw2aiGQiiirf+G5ooOkE2qBfv7O+h6BZokVuM9B61kpy2UFyN8ar1K4fzI4\n\teyCpvHMBXdIPPckDZt2mBf0mjZxTZboi6c7TFj5k/23eekkdtDsV+qYkTgDaRgB0q7\n\t9v5AcJTy0r7+xG29AVwKG2D6hFvDCNwTQLwCgv94FtFO/0n3Bu0lSdOQVV3Db0108h\n\twaRJv/KS95BepoM3dL1NCpqj4XC93LErXxLkuDOPiNnG4j7GimU+UnUBF0VO+qxh+c\n\t9BfVpaDrfTmK4yp3lGVuAH9XkaFTiOMcmUEN8S1j/I66A1OOOdLGnCmDEC1z9WUeJh\n\t3dTcJG/le1CMQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20221208; t=1692945447; x=1693550247;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:to:from:from:to:cc:subject:date:message-id\n\t:reply-to; bh=2Onkgtntva6Jxzg6cR1B7ztmbjs3DRkoZm6St4PeQ3g=;\n\tb=ET8powXyYcxq/uYhpmkP22qFo7qOo79TBH/G+ZLBFDOmn/9LnOZT/P9RfcC/iepJ8K\n\tSirDRFcmbb+2aOwk942aagLXIRm2Ee5HWoB87odXLyZROKdW5TGXJNiUdm74X6Pi3zye\n\tugFQyRbu5vXoZ1pK6qFOnwYC/j+LL21/EDTS3+9P50rkehbjn7OFydZeDcnca84qMfbp\n\tcr0nf+FtBeQjundYviVJntEjNZWq93cVTD5LFcf2z/cv/j/hdKHJubVcXfoDIADB8Csx\n\tlxw8jvu3g362CaFTPtERdlFyc5aVb7bIjBAbouGOb/ueY6gdksJN7JkX3dt3XMHFTRZy\n\tnTbQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"ET8powXy\"; dkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20221208; t=1692945447; x=1693550247;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:to:from:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=2Onkgtntva6Jxzg6cR1B7ztmbjs3DRkoZm6St4PeQ3g=;\n\tb=hx8eSE3xn7TfcvdN5elcDu2jcGu/HQGS2uJqUXGYR0dM9p4a9rsoPj0bmFpGwiBrTw\n\tTMJfckFgvxv966e2QxR72vCohO6BEKgafwePG2WLflt5RTvm53/rM7fWD/cyXbGAzrU9\n\tZiC6n0MqKXvd2ibDao3QlPZUu9Ut8wbNjSSsQZ4tgyLavFIEjsw+Myx+BK+A7ihFdCrD\n\tIowcL4H0p4zPoWxHn9UW9ne/RMLPAfICxJekesau8EqiMCVgdGBxY+77bl+A0dUzIadu\n\tbx3OWRTnyLvHZ+/bDgMyw6miEbkPceseQXNm8re1mXfTnqEQ0FmkLnNM7tKApXV6WfO3\n\teJTg==","X-Gm-Message-State":"AOJu0YzeWsPdkgsIBBltWdIDQF8sdcMr+ZSX2Si+IkjVJ8ugv14F//5s\n\tGBGTvdZihj8sp9xv/fu1JaScoju5OtI=","X-Google-Smtp-Source":"AGHT+IFp2crJPQ7H/yU9vR/4Y2xjdO9hZSbgZwxjKSTLkUlfctE6gdHJpxh2Yscv+63XrB8gXKPhFw==","X-Received":"by 2002:a4a:d2c8:0:b0:56d:2cbf:2315 with SMTP id\n\tj8-20020a4ad2c8000000b0056d2cbf2315mr4987011oos.9.1692945446948; \n\tThu, 24 Aug 2023 23:37:26 -0700 (PDT)","To":"libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com,\n\tvedantparanjape160201@gmail.com, gabbymg94@gmail.com","Date":"Thu, 24 Aug 2023 23:37:06 -0700","Message-Id":"<20230825063707.8734-2-gabbymg94@gmail.com>","X-Mailer":"git-send-email 2.34.1","In-Reply-To":"<20230825063707.8734-1-gabbymg94@gmail.com>","References":"<20230825063707.8734-1-gabbymg94@gmail.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 1/2] libcamera: pipeline: uvcvideo: Store\n\ttime metadata","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Gabby George via libcamera-devel ","Reply-To":"Gabby George ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Parse the raw timing data out of the metadata packets and store each\nsample in a circular buffer of size 32. To convert the latest frame's\nmetadata into a system clock timestamp, use the metadata time\ninformation from 32 frames back stored in the back of the circular\nbuffer.\n\nFor every metadata buffer that comes in, store the timing information\nin the circular queue. The information at the head of the queue may be\nused instantly in the case where a corresponding video buffer has come\nin, or it may be used when the video buffer eventually does arrive. In\nany case, the information at the head of the queue will be used when\nit comes time for the timestamp to be set in the request metadata for\nthe video buffer.\n\nIf there has only been one frame of metadata, use the default (video\nbuffer-provided) timestamp. If the circular buffer has not yet been\nfilled with 32 frames, use the oldest frame. When the circular buffer\nreaches 32 frames, remove the oldest item.\n\nThis approach creates synchronization between the metadata buffer that\nwill be used to update the video buffer's requests' timestamp and the\ntiming data itself, without having to perform checks ensuring\nsynchronization. It also creates easy and efficient access to the\nfront and back of the circular buffer.\n\nSigned-off-by: Gabby George \n---\n src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 76 +++++++++++++++++---\n 1 file changed, 67 insertions(+), 9 deletions(-)","diff":"diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp\nindex 75a949b8..43ce4f8a 100644\n--- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp\n+++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp\n@@ -6,6 +6,7 @@\n */\n \n #include \n+#include \n #include \n #include \n #include \n@@ -47,6 +48,17 @@ struct UVCTimingBuf {\n \t__u16 sofDevice;\n } __attribute__((packed));\n \n+/* Raw timestamp input to the timestamp calculation function. */\n+struct UVCTimestampData {\n+\tunsigned long long tsHost; /* System clock timestamp in nanoseconds*/\n+\tunsigned short sofHost; /* The usb clock at the time tsHost was taken*/\n+\tunsigned int stcDevice; /* The UVC device source timestamp */\n+\tunsigned short sofDevice; /* the usb clock at the time the STC was taken*/\n+\n+\t/* presentation time stamp to be converted into a system clock timestamp */\n+\tunsigned int ptsDevice;\n+};\n+\n class UVCCameraData : public Camera::Private\n {\n public:\n@@ -72,15 +84,18 @@ public:\n \n \tstd::map> formats_;\n \tstd::queue pendingVideoBuffers_;\n-\tstd::queue> pendingMetadata_;\n+\tstd::queue pendingMetadata_;\n \n private:\n \tint initMetadata(MediaDevice *media);\n \tvoid completeRequest(FrameBuffer *buffer, uint64_t timestamp);\n \tvoid endCorruptedStream();\n+\tvoid addTimestampData(uvc_meta_buf &rawMetadata, UVCTimingBuf &packed);\n \n+\tstd::deque timeSamples_;\n \tconst unsigned int frameStart_ = 1;\n \tconst unsigned int maxVidBuffersInQueue_ = 1;\n+\tconst unsigned int bufferRingSize_ = 32;\n \n \tbool generateId();\n \n@@ -899,6 +914,13 @@ void UVCCameraData::endCorruptedStream()\n \t\t<< \"UVC metadata stream corrupted. Reverting to driver timestamps.\";\n }\n \n+unsigned long long calculateTimestamp([[maybe_unused]] const UVCTimestampData &p1,\n+\t\t\t\t const UVCTimestampData &p2,\n+\t\t\t\t [[maybe_unused]] const unsigned int PTS)\n+{\n+\treturn p2.tsHost;\n+}\n+\n /*\n * If there is a metadata buffer that hasn't been matched with a\n * video buffer, check to see if it matches this video buffer.\n@@ -929,9 +951,17 @@ void UVCCameraData::bufferReady(FrameBuffer *buffer)\n \n \tif (!pendingMetadata_.empty()) {\n \t\t/* A metadata buffer was ready first. */\n-\t\tunsigned int mdSequence = std::get<0>(pendingMetadata_.front()) + frameStart_;\n+\t\tunsigned int mdSequence = pendingMetadata_.front() + frameStart_;\n \t\tif (mdSequence == buffer->metadata().sequence) {\n-\t\t\tcompleteRequest(buffer, std::get<1>(pendingMetadata_.front()));\n+\t\t\tunsigned long long timestamp;\n+\t\t\tif (timeSamples_.size() > 1) {\n+\t\t\t\ttimestamp = calculateTimestamp(timeSamples_.front(),\n+\t\t\t\t\t\t\t timeSamples_.back(),\n+\t\t\t\t\t\t\t timeSamples_.back().ptsDevice);\n+\t\t\t} else {\n+\t\t\t\ttimestamp = buffer->metadata().timestamp;\n+\t\t\t}\n+\t\t\tcompleteRequest(buffer, timestamp);\n \t\t\tpendingMetadata_.pop();\n \t\t\treturn;\n \t\t} else {\n@@ -951,6 +981,28 @@ void UVCCameraData::bufferReady(FrameBuffer *buffer)\n \t}\n }\n \n+void UVCCameraData::addTimestampData(uvc_meta_buf &rawMetadata, UVCTimingBuf &packed)\n+{\n+\t/*\n+\t * Copy over the buffer packet from the raw Metadata\n+\t * into values we can use. Populate the storage struct\n+\t * with the data we need to calculate timestamps.\n+\t * Add to the circular queue.\n+\t */\n+\tUVCTimestampData data;\n+\tdata.ptsDevice = packed.pts;\n+\tdata.sofDevice = packed.sofDevice;\n+\tdata.stcDevice = packed.stc;\n+\tdata.sofHost = rawMetadata.sof;\n+\tdata.tsHost = rawMetadata.ns;\n+\n+\tif (timeSamples_.size() == bufferRingSize_) {\n+\t\ttimeSamples_.pop_front();\n+\t}\n+\n+\ttimeSamples_.push_back(data);\n+}\n+\n void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer)\n {\n \tif (!metadata_ ||\n@@ -971,8 +1023,8 @@ void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer)\n \tSpan memMeta = mappedMetadataBuffers_.at(pos).planes()[0];\n \tuvc_meta_buf *metaBuf = reinterpret_cast(memMeta.data());\n \n-\t//Span memTime = mappedMetadataBuffers_.at(pos).planes()[0];\n-\t//UVCTimingBuf * timeBuf = reinterpret_cast(&memTime.data()[sizeof(uvc_meta_buf)]);\n+\tSpan memTime = mappedMetadataBuffers_.at(pos).planes()[0];\n+\tUVCTimingBuf *timeBuf = reinterpret_cast(&memTime.data()[sizeof(uvc_meta_buf)]);\n \n \tsize_t UVCPayloadHeaderSize = sizeof(metaBuf->length) +\n \t\t\t\t sizeof(metaBuf->flags) + sizeof(UVCTimingBuf);\n@@ -981,6 +1033,8 @@ void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer)\n \t\treturn;\n \t}\n \n+\taddTimestampData(*metaBuf, *timeBuf);\n+\n \t/*\n \t * Match a pending video buffer with this buffer's sequence. If\n \t * there is none available, put this timestamp information on the\n@@ -992,15 +1046,19 @@ void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer)\n \t\tunsigned int vidSequence = vidBuffer->metadata().sequence;\n \n \t\tif (vidSequence == mdSequence) {\n-\t\t\tcompleteRequest(vidBuffer, static_cast(metaBuf->ns));\n+\t\t\tunsigned long long timestamp;\n+\t\t\tif (timeSamples_.size() > 1) {\n+\t\t\t\ttimestamp = calculateTimestamp(timeSamples_.front(),\n+\t\t\t\t\t\t\t timeSamples_.back(),\n+\t\t\t\t\t\t\t timeSamples_.back().ptsDevice);\n+\t\t\t}\n+\t\t\tcompleteRequest(vidBuffer, timestamp);\n \t\t\tpendingVideoBuffers_.pop();\n \t\t} else {\n \t\t\tendCorruptedStream();\n \t\t}\n \t} else {\n-\t\tpendingMetadata_.push(\n-\t\t\tstd::make_pair(buffer->metadata().sequence,\n-\t\t\t\t static_cast(metaBuf->ns)));\n+\t\tpendingMetadata_.push(buffer->metadata().sequence);\n \t}\n \tmetadata_->queueBuffer(buffer);\n }\n","prefixes":["libcamera-devel","1/2"]}