From patchwork Fri Aug 25 06:37:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18962 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 908AFBF415 for ; Fri, 25 Aug 2023 06:37:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F2A8061E01; Fri, 25 Aug 2023 08:37:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692945453; bh=Gz/op4V/CEz9kxR/Pgp31gCIBHQ/CRS0cdt4iLZv7dk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=vX9LrNVxVHYmvmnu3rTatwoQgDLe2T1pVAaYBcInQNDv/fsoIb1My0ehn3uriciku LbJ3HfGxmFkNqTO1/CKibTDEoWA59l5PTW20kpU7t06Rh0MdRdLUEX/0C0Vbr3fkwg 4lvq8Ti6sEeEPiMzbSbJlaCGoESK2VcNtfzsxHqsD5H6kwgkfEmxOcKfWKV3h25oSZ AfM9nKV9g1hfVfMt2T3MkTetUYiG7aBvqu8aMyV/k3VWzKQqMzeEpxMDSyhPQ4p+Bn vbXljuZo8THgZOCOzjiNFC8QwTRcU32KLk7oDg2l6kUQz+NP2f4NVRqqHbYi+3yIQ2 H/lUsbeK7aAuQ== Received: from mail-oo1-xc35.google.com (mail-oo1-xc35.google.com [IPv6:2607:f8b0:4864:20::c35]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AB97F61E01 for ; Fri, 25 Aug 2023 08:37:29 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="EBNtYgin"; dkim-atps=neutral Received: by mail-oo1-xc35.google.com with SMTP id 006d021491bc7-5733bcf6eb6so372849eaf.0 for ; Thu, 24 Aug 2023 23:37:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692945448; x=1693550248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=LjTzj1CqlDJ8g91Ky0tegDwA35QuJF4yES22o5hmF0U=; b=EBNtYginfgsVHYTc0moNDLh/J578tWaPPzVrNzjKjjRqM9KytCh3vsNC8KRNV9rNc5 BFc4v5g1qm3i0Agc6avpMQfJP/SdnCsoCD30j1vdyW3iax5PaeO3aLxDxTq/7GU5Ki6T PsrIC6J6y8SyFdNeJ0/MK6i7WVOvx6iqOQcHv9O6pa+TF78XD4r4yNezarq+qCgZjBAn hWKfrVWc4p+ptmi0g+j3jSks+kvCpYFjtz6kJDMmcNsl0YFYYhjlC7lLRF62oRyr9+GM CjHGYVKX3Gill2fhZR9HZfM7R8qT7tn2aIwojJVQ7a7efoorNMZz0rjOXOEf1oOFsORI 0nIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692945448; x=1693550248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjTzj1CqlDJ8g91Ky0tegDwA35QuJF4yES22o5hmF0U=; b=cbBgjuFPbM5qaEPBhTnTmBsjXqM61KOyKySMe3h792MjH8ARjSTaY0FYqfLi9snQFA CUJ45gP6AKyYeZiIAE9iPiQufV+TyFlNgd/rsuXU/RAnSNh12dPtnCo8XMwJi1I50BeA KdonB9fM4iaY1//UOybe2opBUC3MWK7k64nTiOdvuBiwuu9/P5uW1JJk/kTY/C5oCWlc WTuGW0+7bYRvrdKirFHMMp7HRuo9VLEHxpIFg2KM6Qfw8McjVgbe6qefIMbi7gailQey 1+VZith2sGN2S4fFh/40P1Fa31vRpATTZk/TtbdEdfqWw1C6uTP2A1a0iydJdeb92HDt 5psQ== X-Gm-Message-State: AOJu0YxyBHdn4qn0YNEe8PpJ4M6Vx8NI4fPmyhBiSNMnfKfmetlVu29Y NhmSaVN/9gL3jcCncgj6sbFORtBrO20= X-Google-Smtp-Source: AGHT+IHIWmxtcqozip1mPXd43mA+5VRuWOJ/HF12rLxuDthdUcsQW0EWHELh69NgYs6FleMVL0ZmeQ== X-Received: by 2002:a4a:92cf:0:b0:573:4e21:5d25 with SMTP id j15-20020a4a92cf000000b005734e215d25mr16637ooh.9.1692945448053; Thu, 24 Aug 2023 23:37:28 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id u41-20020a4a8c2c000000b0056688eea98csm623239ooj.27.2023.08.24.23.37.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 23:37:27 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Thu, 24 Aug 2023 23:37:07 -0700 Message-Id: <20230825063707.8734-3-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230825063707.8734-1-gabbymg94@gmail.com> References: <20230825063707.8734-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/2] libcamera: pipeline: uvcvideo: Calculate UVC timestamps X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Convert the device clock presentation time stamp (pts) to a host system clock timestamp using the timing data provided by metadata buffers. This is done in two steps. The first step converts the pts into a usb clock time by linearly interpolating between two points where both the sof and the device clock time are known. The second step converts this sof into the host's system clock timestamp by interpolating the sof between two points where both the host timestamp and the usb clock timestamp (also known as frame number according to the UVCH field for it). These calculations take into account rollover potential of the metadata times. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 141 ++++++++++++++++++- 1 file changed, 137 insertions(+), 4 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 43ce4f8a..8cc31055 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -914,11 +914,145 @@ void UVCCameraData::endCorruptedStream() << "UVC metadata stream corrupted. Reverting to driver timestamps."; } -unsigned long long calculateTimestamp([[maybe_unused]] const UVCTimestampData &p1, +/** + * \brief Calculate a more accurate host SOF + * \param[in] sofHost The usb clock time taken by the host + * \param[in] sofDevice The usb clock time taken by the device + * that has full 11 bit precision. + * + * The precision of the host sof may be lower than the expected 11 bits. + * Obtain a more precise host sof by adding back in the lower 8 bits + * of the difference between the host sof and the more precise device SOF. + * + * \return An updated usb clock time for the host with 11 bits of precision + */ +static unsigned short recoverHostSOF(unsigned short sofHost, unsigned short sofDevice) +{ + char sofDelta; + + sofDelta = (sofHost - sofDevice) & 255; + + return (sofDevice + sofDelta) & 2047; +} + +/** + * \brief Convert the presentation timestamp recorded by the UVC device to + * a host timestamp. + * \param[in] p1 Timestamps taken by the usb clock, the device, and the host + * at an early point in time and provided by the UVC driver as metadata + * \param[in] p2 Timestamps taken by the usb clock, the device, and the host + * taken shortly after the presentation timestamp and provided by the UVC driver + * as metadata + * \param[in] PTS The presentation time stamp in device time to convert + * to a host timestamp. + * + * The following device to system clock timestamp conversion algorithm is based + * on the Linux kernel's implementation of UVC timestamp calculation. + * + * To convert the presentation time stamp provided by the device to a system + * clock timestamp, first convert the pts to the usb frame number (sof), + * and then from the usb sof to the system timestamp. The relationship between + * the device clock and the usb clock is linear, as is the relationship between + * the usb clock and the system clock. To calculate the equations needed for the + * conversion, V4L2 provides a metadata packet with a source timestamp (stc) + * and a usb clock sof taken at the same point in time, as well as a system + * timestamp and a usb clock sof pairing. + * + * Two sets of the timestamp metadata are used to recover this linear relationship + * and convert the pts into system clock time. + * + * \return The PTS timestamp converted to system clock time. + */ +unsigned long long calculateTimestamp(const UVCTimestampData &p1, const UVCTimestampData &p2, - [[maybe_unused]] const unsigned int PTS) + const unsigned int PTS) { - return p2.tsHost; + unsigned short sof1Device; + unsigned short sof2Device; + unsigned int stc1; + unsigned int stc2; + unsigned short sof1Host; + unsigned short sof2Host; + unsigned int mean; + unsigned int pts; + unsigned long long hostTS1; + unsigned long long hostTS2; + + float sof; + pts = PTS; + + /* Add 2048 to both sof values to prevent underflow */ + sof1Device = p1.sofDevice + 2048; + stc1 = p1.stcDevice; + sof2Device = p2.sofDevice + 2048; + stc2 = p2.stcDevice; + + /* Subtract out the first point's host timestamp for simplicity */ + hostTS1 = 0; + hostTS2 = p2.tsHost - p1.tsHost; + + /* + * Step 1: Convert the pts into an sof usb clock time + * from p1 and p2's stc and sof time pairs. + * + * The equation is: + * sof = ((sof1 - sof2) *pts + sof1 * stc2 - sof2 * stc1) / ( stc2 - stc1) + */ + + /* If the sof field rolled over, add 2048 so we can extrapolate the line. */ + if (sof2Device < sof1Device) { + sof2Device += 2048; + } + + /* + * If the stc value rolled over, subtract half the 32 bit range from the + * stc and pts values so they fit in the rollover window. + */ + if (stc2 < stc1) { + stc1 -= (1 << 31); + stc2 -= (1 << 31); + pts -= (1 << 31); + } + + /* Cast the values or they may overflow at the multiplication step */ + sof = static_cast( + (static_cast((sof2Device - sof1Device)) * static_cast(pts) + + static_cast(sof1Device) * static_cast(stc2) + - static_cast(sof2Device) * static_cast(stc1))) / + static_cast((stc2 - stc1)); + + /* + * Step 2: Similar to Step1, convert the calculated sof + * to system timestamp from p1 and p2's host timestamp and usb sof pairs + * + * The equation is: + * timestamp = ((ts2 - ts1) * sof + ts1 * sof2 - ts2 * sof1) / (sof2 - sof1) + */ + sof1Host = recoverHostSOF(p1.sofHost, p1.sofDevice) + 2048; + sof2Host = recoverHostSOF(p2.sofHost, p2.sofDevice) + 2048; + + if (sof2Host < sof1Host) { + sof2Host += 2048; + } + + /* + * This accounts for the possibility that the calculated sof + * rolled over and the host sof data fields did not. + */ + mean = (sof1Host + sof2Host) / 2; + + if ((mean - 1024) > sof) { + sof += 2048; + } else if (sof > mean + 1024) { + sof -= 2048; + } + + /* note: hostTS 1 has been set to zero so the difference is just the value of hostTS2.*/ + unsigned long long result = static_cast( + (hostTS2 * sof + hostTS1 * sof2Host - hostTS2 * sof1Host) / (sof2Host - sof1Host)); + + /* Add the subtracted system timestamp from p1 back in */ + return result + p1.tsHost; } /* @@ -943,7 +1077,6 @@ unsigned long long calculateTimestamp([[maybe_unused]] const UVCTimestampData &p */ void UVCCameraData::bufferReady(FrameBuffer *buffer) { - /* \todo Use the UVC metadata to calculate a more precise timestamp */ if (!metadata_ || buffer->metadata().sequence < frameStart_) { completeRequest(buffer, buffer->metadata().timestamp); return;