From patchwork Mon Aug 14 11:28:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18942 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A228FC32B2 for ; Mon, 14 Aug 2023 11:29:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AB059628E6; Mon, 14 Aug 2023 13:29:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692012541; bh=ABhfgmo5PUBSXwT8Jvvta5ZvStQKy++jlNCWwuPYcIg=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=sjDmf/IXHhTAslxv8iS45hmh5ECWzequXqFhFj+QNY29un+b0NMAqX+KPXni6LBbU MPX7FxuY3xXD73upwg6EAhvwHolUDZOaQp7KUCRO9qgrl2Ga7FFY/BEHcDywYltd4E 8d79Mtrazxj5lFEuTn5+sJuZWV0P+ce6bN5Vg8MiEAk6YyxxIxwq73HCyOPtj3bec/ umhQklBVaGvEuFgvOp7OPhJT9cWqaf1/xrOB99fTKB9iJ63tdLCGNH4TAnFtXT99Rl dGBhz04DiQek49Ex36ADTOXFzlwKyN96k1rSH28IrYjlhfd0FlcnmKBAsvy5jSflUd VgxejjCRYt7fg== Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 86309628DF for ; Mon, 14 Aug 2023 13:28:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aR/UQLML"; dkim-atps=neutral Received: by mail-ot1-x333.google.com with SMTP id 46e09a7af769-6bcd4b5ebbaso3730565a34.1 for ; Mon, 14 Aug 2023 04:28:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692012537; x=1692617337; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=pShnn9eKyZPi5uoMChY9/RWSzW6FXWMZIhCveXimmHI=; b=aR/UQLMLg62UNlaUiwZ/2c/PRPvyh3uYqNgtuaRz57pmC8GPoA4FurJLnD7vm+ZH8z CyvKDln+NXeKUNhp8KqXxODEY1KLRkq83uQQ3Y59F6ktcJkc0ISnERavZNNJiYve8KTx MvH4huLxfq6ggYWiMFq+7qlwNwv1BSM1cGry1d2oYVfxJKwO8IWstPfBe6YKxuaQ9EZn oAhezKTCit1syeo9FEvtFPU23OOYfgUKVFqZAmNGvuTlimDHMlvsWPMPRJno1AaSdnPZ W3VwbxnlgoFM9A4Vv4434Ocgnn5s9AW4KcatdncvlK2tH8ssjfb+bUYMML9V1Yc00vCB FaeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692012537; x=1692617337; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pShnn9eKyZPi5uoMChY9/RWSzW6FXWMZIhCveXimmHI=; b=cw3c/qU65fdAH78oPdD8bijZ048y402n7TAe6EdOqRM4y+U4lLmEd2G0d1lGnb717Y M+yZOV+nRs21mn2ZZK8RXczz3ktvFYzYSClXcI69Z+0nlgMxCSsC0VCNtTxeHBrZkIWk luKS55uCCwra5Uk0wjaxmKxXNJHMLxl+DFLSvjUXu1bQ+Qid4IwpMBCX36rjsI5z5o2j h0hWQd4sxg3ELG87XPBKKz5xM8d13gryCEkfcUYC1eZzf8R/khaIW1osdGDqVIIYTQMJ qEXxTGEGljX50bwd41vhGrBHfG55AsvxhJ/i2fSETGowXk57wqjw9ZGG10JbW0gdVV78 Bhrg== X-Gm-Message-State: AOJu0YxkxLsUQta6bo5+iKdwAjcAtCfv2D3N9IdBTckHvcLM9ss5OGCP NhlXtvz4uxB9l7ZYEgv26z9cQNaR7nK24w== X-Google-Smtp-Source: AGHT+IHsDct/sp9xSgzkJchQVT/FA8dlD5GvKjGS0USkEzX+SzBk8fJ5qd8p6u6A8HAkXH0Jo2BHng== X-Received: by 2002:a9d:63ce:0:b0:6b9:696d:716a with SMTP id e14-20020a9d63ce000000b006b9696d716amr9676790otl.18.1692012536904; Mon, 14 Aug 2023 04:28:56 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id c3-20020a9d6c83000000b006b92509e76esm4163817otr.32.2023.08.14.04.28.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 04:28:56 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 14 Aug 2023 04:28:49 -0700 Message-Id: <20230814112849.176943-6-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230814112849.176943-1-gabbymg94@gmail.com> References: <20230814112849.176943-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 5/5] libcamera: pipeline: uvcvideo: Handle metadata stream X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Register the metadata stream's buffer ready callback and start processing metadata buffers. Use the timestamp from the metadata buffer as the corresponding video buffer Requests' timestamp. Metadata buffers are synchronized with frames coming into the video stream using the sequence field of the buffers. They may come in either order (video buffer first or metadata buffer first), so store relevant information about the buffer required to set the metadata timestamp or complete the buffer request as soon as possible. The timestamp will improved upon in the next patch. For now, use the driver-provided metadata timestamp. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 157 ++++++++++++++++++- 1 file changed, 152 insertions(+), 5 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 51f30187..5c7ae064 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -12,6 +12,8 @@ #include #include +#include + #include #include @@ -34,6 +36,13 @@ namespace libcamera { LOG_DEFINE_CATEGORY(UVC) +/* This is used to memcpy */ +struct UVCMetadataPacked { + __u32 pts; + __u32 scr; + __u16 sofDevice; +} __attribute__((packed)); + class UVCCameraData : public Camera::Private { public: @@ -46,6 +55,7 @@ public: void addControl(uint32_t cid, const ControlInfo &v4l2info, ControlInfoMap::Map *ctrls); void bufferReady(FrameBuffer *buffer); + void bufferReadyMetadata(FrameBuffer *buffer); const std::string &id() const { return id_; } @@ -57,10 +67,14 @@ public: bool useMetadataStream_; std::map> formats_; + std::queue> waitingForVideoBuffer_; + std::queue> waitingForMDBuffer_; private: int initMetadata(MediaDevice *media); + const unsigned int frameStart_ = 1; + const unsigned int maxVidBuffersInQueue_ = 2; const unsigned int minLengthHeaderBuf_ = 10; bool generateId(); @@ -638,8 +652,16 @@ int UVCCameraData::init(MediaDevice *media) } controlInfo_ = ControlInfoMap(std::move(ctrls), controls::controls); + ret = initMetadata(media); + + if (!ret) { + metadata_->bufferReady.connect(this, &UVCCameraData::bufferReadyMetadata); + useMetadataStream_ = true; + } else { + useMetadataStream_ = false; + } - return initMetadata(media); + return 0; } bool UVCCameraData::generateId() @@ -824,16 +846,141 @@ void UVCCameraData::addControl(uint32_t cid, const ControlInfo &v4l2Info, ctrls->emplace(id, info); } +/* + * If there is a metadata buffer that hasn't been matched with a + * video buffer, check to see if it matches this video buffer. + * + * If there is a match, use the timestamp stored in the metadata queue + * for this video buffer's request. Complete this video buffer + * and its request. + * + * If there are no metadata buffers available to check for a match, + * push this video buffer's request object to the queue. It may + * be that the metadata buffer has not yet arrived. + * When the matching metadata buffer does come in, it will handle + * completion of the buffer and request. + * + * If more than maxVidBuffersInQueue_ video buffers have been added + * to the queue, something is wrong with the metadata stream and + * we can no longer use UVC metadata packets for timestamps. + * Complete all of the outstanding requests and turn off metadata + * stream use. + */ void UVCCameraData::bufferReady(FrameBuffer *buffer) { Request *request = buffer->request(); - - /* \todo Use the UVC metadata to calculate a more precise timestamp */ request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); - pipe()->completeBuffer(request, buffer); - pipe()->completeRequest(request); + if (useMetadataStream_) { + if (buffer->metadata().sequence == 0) { + /* \todo: we do not expect the first frame to have a + * metadata buffer associated with it. Why? + */ + pipe()->completeBuffer(request, buffer); + pipe()->completeRequest(request); + return; + } + + if (!waitingForMDBuffer_.empty()) { + unsigned int mdSequence = + std::get<0>(waitingForMDBuffer_.front()) + frameStart_; + if (mdSequence == buffer->metadata().sequence) { + request->metadata().set(controls::SensorTimestamp, + std::get<1>(waitingForMDBuffer_.front())); + pipe()->completeBuffer(request, buffer); + pipe()->completeRequest(request); + waitingForMDBuffer_.pop(); + return; + } + } else { + waitingForVideoBuffer_.push(std::make_pair(request, buffer)); + } + + if (waitingForVideoBuffer_.size() > maxVidBuffersInQueue_) { + while (!waitingForVideoBuffer_.empty()) { + Request *oldRequest = std::get<0>(waitingForVideoBuffer_.front()); + FrameBuffer *oldBuffer = std::get<1>(waitingForVideoBuffer_.front()); + oldRequest->metadata().set(controls::SensorTimestamp, + oldBuffer->metadata().timestamp); + pipe()->completeBuffer(oldRequest, oldBuffer); + pipe()->completeRequest(oldRequest); + waitingForVideoBuffer_.pop(); + } + } + } else { + pipe()->completeBuffer(request, buffer); + pipe()->completeRequest(request); + } +} + +void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) +{ + if (!useMetadataStream_ || buffer->metadata().status != FrameMetadata::Status::FrameSuccess) { + return; + } + + /* + * The metadata stream always starts at seq 1 and libcamera sets the start sequence to 0, + * so it's necessary to add one to match this buffer with the correct + * video frame buffer. + * + * \todo: Is there a better way to do this? What is the root cause? + */ + unsigned int mdSequence = buffer->metadata().sequence + frameStart_; + int pos = buffer->cookie(); + /* + * A UVC Metadata Block length field contains size of + * the header buf, length field, and flags field. + */ + uvc_meta_buf metadataBuf; + __u8 minLength = minLengthHeaderBuf_ + + sizeof(metadataBuf.length) + + sizeof(metadataBuf.flags); + size_t lenMDPacket = minLength + sizeof(metadataBuf.ns) + sizeof(metadataBuf.sof); + memcpy(&metadataBuf, mappedMetadataBuffers_.at(pos).planes()[0].data(), lenMDPacket); + + if (metadataBuf.length < minLength) { + LOG(UVC, Error) << "Received improper metadata packet. Using default timestamps."; + useMetadataStream_ = false; + return; + } + + /* + * If there is a video buffer that hasn't been matched with a + * metadata buffer, check to see if it matches this metadata buffer. + * + * If there is a match, use the timestamp associated with this + * metadata buffer as the timestamp for the video buffer's request. + * Complete that video buffer and its request. + * + * If there are no video buffers, push this metadata buffer's + * sequence number and timestamp to a shared queue. It may be that + * the metadata buffer came in before the video buffer. + * When the matching video buffer does come in, it will use this + * metadata buffer's timestamp. + */ + __u64 timestamp = metadataBuf.ns; + + if (!waitingForVideoBuffer_.empty()) { + Request *request = std::get<0>(waitingForVideoBuffer_.front()); + FrameBuffer *vidBuffer = std::get<1>(waitingForVideoBuffer_.front()); + unsigned int vidSequence = vidBuffer->metadata().sequence; + + if (vidSequence == mdSequence) { + request->metadata().set(controls::SensorTimestamp, + timestamp); + + pipe()->completeBuffer(request, vidBuffer); + pipe()->completeRequest(request); + waitingForVideoBuffer_.pop(); + } + } else { + waitingForMDBuffer_.push( + std::make_pair(buffer->metadata().sequence, + timestamp)); + } + metadata_->queueBuffer(buffer); } REGISTER_PIPELINE_HANDLER(PipelineHandlerUVC)