From patchwork Mon Aug 21 13:10:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18948 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7C2BAC32B2 for ; Mon, 21 Aug 2023 13:10:53 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AD39B628BE; Mon, 21 Aug 2023 15:10:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692623452; bh=6AxJ8c4c7HXmRYYCXvoO2RunB2yaegrKd1K9gPzMqbM=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=hOcVR+7i3cY2jtZz4cF5jmOesLQs0yZGpfgJRaGYbLnc8buuJcqLgKF5jumxbdSN+ JLcetcxLc5qFdOO2frZqmrDVJ/ZKhdVFo6X8d2PWPODmChTVLDvw5rGI9XZJVLw5aN QBfo7rN3vSg2kUxbYqj38uxvaUwJ3o2Kr/MC2a3qebsr3BFck7Lruipm4uxC2aVB9p 06YqpG+tBcDvXpZJxLTyRWw/yTfPtRcUc0IG1WzZGwB7CPxUx+15bERQdt2qTP8U7r L0ptpZy81VgBipbuegr+tYrmQ0eMxE7vtC+1vq4KYIeMli3mzIvVB2EtMBZrJva9US XPs9jLJE5/x1w== Received: from mail-oa1-x33.google.com (mail-oa1-x33.google.com [IPv6:2001:4860:4864:20::33]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2B045628C1 for ; Mon, 21 Aug 2023 15:10:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="UeDUtaXL"; dkim-atps=neutral Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-1a1fa977667so2192314fac.1 for ; Mon, 21 Aug 2023 06:10:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692623446; x=1693228246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=a0EOWnwLCT3Q7nV/PP4nZpbXd+qQfFMqXt5oWOXOEyI=; b=UeDUtaXLyVOBWsrk5nBz33ik5I6NwdjnamKH1yQTIQWc1ArsUp040jQ1qqR6qewcOw 6AjdXSvJOYQDt7a5MVS3OCLscInRiZG7afpM8N2KEuplo9WbP0cjwDR2SrxK1yFotjPA 4NDtnzZEsp5kegb6QVmQnUKgRC0o4fPedIO2NtlDmGclhO5ZF78IRtlqtdsj+7UeYiT5 dZ4j0Oov+B7bwUs5p8Ee6PnWiMp91aTVHmNsgOjk80cfix83KZ+5S7/u7sgiIOTPGIDy lBOzGagvRBV/kJZUTSgjSR0qfferWrngvMqInM5hBbFAymC0qHwuslspY1k6lJblU9RW oXhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692623446; x=1693228246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a0EOWnwLCT3Q7nV/PP4nZpbXd+qQfFMqXt5oWOXOEyI=; b=lXgmSx4v1UdvlIwHk03Th2fR7YHMyVs4OPSPzOQguZjI7AOTpNiSG7lH97lm/UUkJp +NOr26TxB1s3LJOKpHWZ0MQrqtDVp8Vl8YgCuDdZEGgNqRpnuZ87nrjrHEtNeDUxg23M YrrBHiHUOOm9YTXKAzta9AXw89PAovjhzdEY+Tdfg5r39OuRyHQScjUrK3v8FV8hPYo7 QP+kzmRtW6MoP4TOWfCo2PPjPJBgyS3gDTcsvUBRwcSFas6zA8T9WFh3fKEjkdSLK3N9 o6CoRTjVOAFSY6BA/JvqSozpcKJIGXYLbKmfr4jKNnUCMHQfxEGEwbjgay6wBdooyl0X +k4g== X-Gm-Message-State: AOJu0Yz0fNU81rQxEGmxkR2DpsO7wVsn7/u63Ztj04LY/uTSVZs3A0GF tgwgCFj1TpPo5MbqtxS3ddK/Gl7ZyLQ= X-Google-Smtp-Source: AGHT+IFbiiJai62GYO2Je+gLgIVtGay20sCY9TAG7cudF32BI3DCpfOX8tEzaey4OcO00oUdMPXchQ== X-Received: by 2002:a05:6870:4193:b0:1c0:2e8f:17fd with SMTP id y19-20020a056870419300b001c02e8f17fdmr10123197oac.40.1692623446638; Mon, 21 Aug 2023 06:10:46 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id ci1-20020a056871c48100b001bb51450d85sm4160283oac.4.2023.08.21.06.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 06:10:46 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 21 Aug 2023 06:10:39 -0700 Message-Id: <20230821131039.127370-6-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821131039.127370-1-gabbymg94@gmail.com> References: <20230821131039.127370-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 5/5] libcamera: pipeline: uvcvideo: Handle metadata stream X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Register the metadata stream's buffer ready callback and start processing metadata buffers. Use the timestamp from the metadata buffer as the corresponding video buffer Requests' timestamp. Metadata buffers are synchronized with frames coming into the video stream using the sequence field of the buffers. They may come in either order (video buffer first or metadata buffer first), so store relevant information about the buffer required to set the metadata timestamp or complete the buffer request as soon as possible. The timestamp will be improved upon in the next patch. For now, use the driver-provided metadata timestamp. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 168 ++++++++++++++++++- 1 file changed, 162 insertions(+), 6 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index c8d6633f..215435ec 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -12,6 +12,8 @@ #include #include +#include + #include #include @@ -34,6 +36,17 @@ namespace libcamera { LOG_DEFINE_CATEGORY(UVC) +/* + * The UVCH buffer contains an unsigned char array + * encoding UVC timing data that needs to be recast + * into usable data. + */ +struct UVCTimingBuf { + __u32 pts; + __u32 stc; + __u16 sofDevice; +} __attribute__((packed)); + class UVCCameraData : public Camera::Private { public: @@ -46,6 +59,8 @@ public: void addControl(uint32_t cid, const ControlInfo &v4l2info, ControlInfoMap::Map *ctrls); void bufferReady(FrameBuffer *buffer); + void bufferReadyMetadata(FrameBuffer *buffer); + void handleUnfinishedRequests(); const std::string &id() const { return id_; } @@ -56,9 +71,16 @@ public: std::map mappedMetadataBuffers_; std::map> formats_; + std::queue pendingVideoBuffers_; + std::queue> pendingMetadata_; private: int initMetadata(MediaDevice *media); + void completeRequest(FrameBuffer *buffer, uint64_t timestamp); + void endCorruptedStream(); + + const unsigned int frameStart_ = 1; + const unsigned int maxVidBuffersInQueue_ = 1; bool generateId(); @@ -242,7 +264,7 @@ int PipelineHandlerUVC::cleanupMetadataBuffers(Camera *camera) UVCCameraData *data = cameraData(camera); if (data->metadata_) - data->metadata_->releaseBuffers(); + ret = data->metadata_->releaseBuffers(); data->metadataBuffers_.clear(); data->mappedMetadataBuffers_.clear(); data->metadata_ = nullptr; @@ -253,7 +275,9 @@ int PipelineHandlerUVC::cleanupMetadataBuffers(Camera *camera) int PipelineHandlerUVC::cleanup(Camera *camera) { UVCCameraData *data = cameraData(camera); + cleanupMetadataBuffers(camera); + data->video_->releaseBuffers(); return 0; } @@ -354,6 +378,8 @@ void PipelineHandlerUVC::stopDevice(Camera *camera) if (data->metadata_) data->metadata_->streamOff(); + data->handleUnfinishedRequests(); + cleanup(camera); } @@ -646,8 +672,11 @@ int UVCCameraData::init(MediaDevice *media) if (ret) { metadata_ = nullptr; LOG(UVC, Error) << "Could not find a metadata video device."; + return 0; } + metadata_->bufferReady.connect(this, &UVCCameraData::bufferReadyMetadata); + return 0; } @@ -833,18 +862,145 @@ void UVCCameraData::addControl(uint32_t cid, const ControlInfo &v4l2Info, ctrls->emplace(id, info); } -void UVCCameraData::bufferReady(FrameBuffer *buffer) +void UVCCameraData::completeRequest(FrameBuffer *buffer, uint64_t timestamp) { Request *request = buffer->request(); - - /* \todo Use the UVC metadata to calculate a more precise timestamp */ - request->metadata().set(controls::SensorTimestamp, - buffer->metadata().timestamp); + request->metadata().set(controls::SensorTimestamp, timestamp); pipe()->completeBuffer(request, buffer); pipe()->completeRequest(request); } +void UVCCameraData::handleUnfinishedRequests() +{ + while (!pendingVideoBuffers_.empty()) { + FrameBuffer *oldBuffer = pendingVideoBuffers_.front(); + Request *oldRequest = oldBuffer->request(); + + oldRequest->metadata().set(controls::SensorTimestamp, + oldBuffer->metadata().timestamp); + + pipe()->completeBuffer(oldRequest, oldBuffer); + pipe()->completeRequest(oldRequest); + pendingVideoBuffers_.pop(); + } +} + +void UVCCameraData::endCorruptedStream() +{ + handleUnfinishedRequests(); + /* Close the metadata node so we don't get inaccurate timestamps*/ + metadata_ = nullptr; + LOG(UVC, Error) + << "UVC metadata stream corrupted. Reverting to driver timestamps."; +} + +/* + * If there is a metadata buffer that hasn't been matched with a + * video buffer, check to see if it matches this video buffer. + * + * If there is a match, use the timestamp stored in the metadata queue + * for this video buffer's request. Complete this video buffer + * and its request. + * + * If there are no metadata buffers available to check for a match, + * push this video buffer's request object to the queue. It may + * be that the metadata buffer has not yet arrived. + * When the matching metadata buffer does come in, it will handle + * completion of the buffer and request. + * + * If more than maxVidBuffersInQueue_ video buffers have been added + * to the queue, something is wrong with the metadata stream and + * we can no longer use UVC metadata packets for timestamps. + * Complete all of the outstanding requests and turn off metadata + * stream use. + */ +void UVCCameraData::bufferReady(FrameBuffer *buffer) +{ + /* \todo Use the UVC metadata to calculate a more precise timestamp */ + if (!metadata_ || buffer->metadata().sequence < frameStart_) { + completeRequest(buffer, buffer->metadata().timestamp); + return; + } + + if (!pendingMetadata_.empty()) { + /* A metadata buffer was ready first. */ + unsigned int mdSequence = std::get<0>(pendingMetadata_.front()) + frameStart_; + if (mdSequence == buffer->metadata().sequence) { + completeRequest(buffer, std::get<1>(pendingMetadata_.front())); + pendingMetadata_.pop(); + return; + } else { + /* \todo: Is there a reason metadata buffers can arrive out of order? */ + endCorruptedStream(); + return; + } + } + + pendingVideoBuffers_.push(buffer); + /* + * Deal with video buffers that haven't been completed, and with + * buffers whose metadata information arrived out of order. + */ + if (pendingVideoBuffers_.size() > maxVidBuffersInQueue_) { + endCorruptedStream(); + } +} + +void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) +{ + if (!metadata_ || + buffer->metadata().status != FrameMetadata::Status::FrameSuccess) { + return; + } + + /* + * The metadata stream seems to start at seq 1 and libcamera + * sets the start sequence to 0, so it's necessary to add one + * to match this buffer with the correct video frame buffer. + * + * \todo: Is there a better way to do this? What is the root cause? + */ + unsigned int mdSequence = buffer->metadata().sequence + frameStart_; + int pos = buffer->cookie(); + + Span memMeta = mappedMetadataBuffers_.at(pos).planes()[0]; + uvc_meta_buf *metaBuf = reinterpret_cast(memMeta.data()); + + //Span memTime = mappedMetadataBuffers_.at(pos).planes()[0]; + //UVCTimingBuf * timeBuf = reinterpret_cast(&memTime.data()[sizeof(uvc_meta_buf)]); + + size_t UVCPayloadHeaderSize = sizeof(metaBuf->length) + + sizeof(metaBuf->flags) + sizeof(UVCTimingBuf); + if (metaBuf->length < UVCPayloadHeaderSize) { + endCorruptedStream(); + return; + } + + /* + * Match a pending video buffer with this buffer's sequence. If + * there is none available, put this timestamp information on the + * queue. When the matching video buffer does come in, it will use + * a timestamp from this metadata. + */ + if (!pendingVideoBuffers_.empty()) { + FrameBuffer *vidBuffer = pendingVideoBuffers_.front(); + unsigned int vidSequence = vidBuffer->metadata().sequence; + + if (vidSequence == mdSequence) { + completeRequest(vidBuffer, static_cast(metaBuf->ns)); + pendingVideoBuffers_.pop(); + } else { + endCorruptedStream(); + } + } else { + pendingMetadata_.push( + std::make_pair(buffer->metadata().sequence, + static_cast(metaBuf->ns))); + } + metadata_->queueBuffer(buffer); +} + REGISTER_PIPELINE_HANDLER(PipelineHandlerUVC) } /* namespace libcamera */