From patchwork Mon Aug 14 11:28:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18938 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C1ADEC3257 for ; Mon, 14 Aug 2023 11:29:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2892D628DF; Mon, 14 Aug 2023 13:28:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692012539; bh=tITfKvnBlkEJdz07jEYOKbwfWNQmea4GTx911uz2yNk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=MH8i+rlnnFvRzS6N0+nVKdM0bQNNVuhbnPIo6ULGvsayj5currCVaBMRRV6LKSUCL SgYXs+naI8RbJqQgNftbj7YCrtZzpMu3WZu2DXLg96bNLKT8WHSt8cgifWSqZES+lg BP0UbTLnuQqYiolPloXh8ypJCbenixpSELH7A50WBqbMSYLgw1NYxY/h6AtZgHzcVi 8c9x+wP70lrT4/hFpq62mZPJ7hGwiQXWse8+XAJSA57Xu5ktF+3VfbcNYLvw7YrDxk sy+S1ihhTHFFnaX1+j2MoI9BaZBftjDJpaCL2AXyeEE0o92mCb7tLNVaeWUF7x/CXx ZY9ykFYPN2Hyg== Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3A82B61E0B for ; Mon, 14 Aug 2023 13:28:54 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hyfKpH92"; dkim-atps=neutral Received: by mail-ot1-x32f.google.com with SMTP id 46e09a7af769-6bd045336c6so3840193a34.2 for ; Mon, 14 Aug 2023 04:28:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692012532; x=1692617332; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=qhmizX4FuLCSyb8453O84uUId5fqc6MvHIneCkyAl7Q=; b=hyfKpH92reKEo+zItwGxcONaR33ya1fu7VheiyzswFIYdsEu3XR7YB3tieIoD6y/mw nk7qfJvLDjzUbNRGAClGcflIz2oZe/6Olog8gTR7s/rhvLSXz4k72bsWUpE+Gcrc2eQk ySYTH3gbsnfNoo+QLzsfgDPf+stSoBiOoe99txHMip0FmUKfU8rnITViudTg8C+q2SkA 0/ng6x3wxHR1X2SBdAcmtDacuuYDds9uT+BGOvMuWC4qSIdHo78MNXR7U6Z0ynhWvW4N ecN0RvtCLYo+3R1oG08L+hT4dlYZErLGEHbqKnwU79gMJRAkcg+QrTJ/j/BuTfDiqcdX P8Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692012532; x=1692617332; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qhmizX4FuLCSyb8453O84uUId5fqc6MvHIneCkyAl7Q=; b=g5zOofQRKcFBaYdgZj54soktxBizvnEeSFcSEMu3upX3PBgDHDxnu4YTD8/jOd3M9j jx5On6ifkSnPLlwkKmsZ9Z4g2+cFj3M2KD++UUE0ET9kcenCo9k5qEKombv+WhXC+9KU VyslsivzkPjWH4qTBw1Hwdm7m2LyiqE+EFB/ZsloxBPDwzEf9eDPrC84xBLQWeP0mHTz s4jAR1B65E40nvYMUmmH1BSML6cLwJRUtrXhzqjYgA3Pbz4z13kRUMm+nj4CtmijNjve hOKkHi/ugpLUiJkvuqTGTfgUUbnLRIHJUYi7shtAwtoV1RwxEjk8ksEmt4S/EMJVnuGD kTLg== X-Gm-Message-State: AOJu0YwsqpRL/IckF+rgJv0jG2AfHW7WpTGcnEj9ohH3ybxqYn57mj4j Bxl9M4qq3f/dk+O/CNuQW9len2mrNZ0ynA== X-Google-Smtp-Source: AGHT+IGpqZZfiso8ESWgGoUb+YfXoZXAqy9f6NhCjcaWCrMxwMWO2/dxriTaJ4Eb7tIHfhMU6YnLmQ== X-Received: by 2002:a9d:7dd6:0:b0:6b9:9288:613c with SMTP id k22-20020a9d7dd6000000b006b99288613cmr8539153otn.13.1692012532772; Mon, 14 Aug 2023 04:28:52 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id c3-20020a9d6c83000000b006b92509e76esm4163817otr.32.2023.08.14.04.28.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 04:28:52 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 14 Aug 2023 04:28:45 -0700 Message-Id: <20230814112849.176943-2-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230814112849.176943-1-gabbymg94@gmail.com> References: <20230814112849.176943-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 1/5] libcamera: pipeline: uvcvideo: Add UVC metadata node X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Identify and open the UVC metadata's video node. This will give us access to metadata associated with the video stream of the uvc camera. The user will not have access to this video node and will not need to manage its data in any way. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 36 +++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 38f48a5d..4470d8a2 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -49,10 +49,13 @@ public: const std::string &id() const { return id_; } std::unique_ptr video_; + std::unique_ptr metadata_; Stream stream_; std::map> formats_; private: + int initMetadata(MediaDevice *media); + bool generateId(); std::string id_; @@ -411,6 +414,37 @@ bool PipelineHandlerUVC::match(DeviceEnumerator *enumerator) return true; } +int UVCCameraData::initMetadata(MediaDevice *media) +{ + int ret; + + const std::vector &entities = media->entities(); + std::string dev_node_name = video_->deviceNode(); + auto metadata = std::find_if(entities.begin(), entities.end(), + [&dev_node_name](MediaEntity *e) { + return e->type() == MediaEntity::Type::V4L2VideoDevice && !(e->flags() & MEDIA_ENT_FL_DEFAULT); + }); + + if (metadata == entities.end()) { + LOG(UVC, Error) << "Could not find a metadata video device."; + return -ENODEV; + } + + /* configure the metadata node */ + metadata_ = std::make_unique(*metadata); + ret = metadata_->open(); + if (ret) + return ret; + + if (!(metadata_->caps().isMeta())) { + /* if the caps do not have the metadata attribute + * (shouldn't happen) */ + metadata_ = NULL; + return -EINVAL; + } + return 0; +} + int UVCCameraData::init(MediaDevice *media) { int ret; @@ -512,7 +546,7 @@ int UVCCameraData::init(MediaDevice *media) controlInfo_ = ControlInfoMap(std::move(ctrls), controls::controls); - return 0; + return initMetadata(media); } bool UVCCameraData::generateId() From patchwork Mon Aug 14 11:28:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18939 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2BAE0BF415 for ; Mon, 14 Aug 2023 11:29:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 027F1628E2; Mon, 14 Aug 2023 13:28:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692012540; bh=rW76to3CewSxJAQvDPi+DUc/D/dhgJYr6JQ8WoIjpTg=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=pWJb8VAUmS9xCWpqYPs+Jm+Rm29cCNU0blTTqEC4DvYgk8ZY5m4hw1O/dVpbycefe 1K+lhcO6hcNXlFEjW1uOwxxmnK3Nl21fZyt50vVPjQ66gp6FPR9S/1oRKZhrVo22Xs Zhw4ueR71CWKZ+0893UhWpWdeOzQqwqYRx7NKTIFD9Y0HU+7DEaDgyTM+Xe3KQVMIi 7MMwIj92ujGvaTwEx+sAe1JzyNHXnluPyTNG97KbhIPc/fZLFljY7Kh+FQBNxev+21 3Xb9scJK4sowju3+nEpSP4k8yS0TPRYp6Pujb5VaFeO0I3cqxmnEQbMGppjHwkr2PO FNYpZFkT1Fy9A== Received: from mail-ot1-x331.google.com (mail-ot1-x331.google.com [IPv6:2607:f8b0:4864:20::331]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4566761E0B for ; Mon, 14 Aug 2023 13:28:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="rY8pnJ5M"; dkim-atps=neutral Received: by mail-ot1-x331.google.com with SMTP id 46e09a7af769-6bd0a0a6766so3376165a34.2 for ; Mon, 14 Aug 2023 04:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692012534; x=1692617334; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=71lcLD82o1iAaxJ3Re4tzyvkOeuk5oyOwipvfyqdST0=; b=rY8pnJ5Mi2DyNfkXS2wigqfEe/vtvH3t+Go/DGs2PCfQpOotuUnAl6KGv2P9NaxYg7 cYQuEy4M0yMwkF6kRkGDVnmiZS10HIS0Xfc1newMZMJv2iS8eCA5t7p6wisF8zc+mM3V L6JrG9LGhAP0+ik3Rw0Phhn4Bq5OW5iFK3AONxvL3dNUwyeJXpzTcvxt07uQ54Fhy9e8 R3hAKSFawgyDT/tT0NJ1gPq/ARiyG4MGA8UEz/nn8nahnj0jwDakDf1duFnXya7nmi34 mX1ksCw7GdovYSFBGq2z3QdpRNs7fH4SQLPrRustsnRRSK2rLLGKTNxgee1yFi4moYs8 +Y0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692012534; x=1692617334; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=71lcLD82o1iAaxJ3Re4tzyvkOeuk5oyOwipvfyqdST0=; b=HNgm7IASyqaIURPqDZEp628Rur7khxLK6t9J6N0JOL6DWdqO/vItCQI+WV7biUah+4 38uGpqygKVeMmtY2w8qpBZ83kK950sRfPE5fs6BqBa77xHqcBEXt4jxLaKIcOhVZ/1ni 5/3j7BougkN6/72B9rcMUloYSsoSsZ4Rn0uujj/767NwpSneM+Y6jZvfcC9jcP9gEkNK 8PeZtsKJTmk2Q45FXqPgDXnZTq9NiNCbxODVBUnE12vdvJeQBsBSzP53v3M103/krZ75 x+9FlvsBl+7vhFHUKRwg+xV7BV5b+VSbYIWLXuEYXOHA7KjjeLm8ERnoLj3wXPOi91Fo Gscg== X-Gm-Message-State: AOJu0YyYR944pjo3Nskdx4vRTyopT0X/L3gR7zyy3/PUQaGmLUtS2SER 7D/a9K1vyzXotZg4iPsgNait2b5z8/4HTQ== X-Google-Smtp-Source: AGHT+IFv6cfQhSTa63M6Aj8mwrEdcXweH6Asj5AUdP5Dxu3PXz3DHGYt+yr7U/WrcPbgBlQU3wqCzg== X-Received: by 2002:a05:6830:1007:b0:6b9:6712:d4ad with SMTP id a7-20020a056830100700b006b96712d4admr7951864otp.36.1692012533808; Mon, 14 Aug 2023 04:28:53 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id c3-20020a9d6c83000000b006b92509e76esm4163817otr.32.2023.08.14.04.28.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 04:28:53 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 14 Aug 2023 04:28:46 -0700 Message-Id: <20230814112849.176943-3-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230814112849.176943-1-gabbymg94@gmail.com> References: <20230814112849.176943-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 2/5] libcamera: MappedFrameBuffer: Use stored plane offset X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When calling mmap, the mapped frame buffer class hard-codes the offset value to 0. TODO: EDIT ME WHEN YOU HAVE MORE VERBAL REASONING SKILLS Signed-off-by: Gabby George --- include/libcamera/internal/mapped_framebuffer.h | 2 +- src/libcamera/mapped_framebuffer.cpp | 16 ++++++++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/include/libcamera/internal/mapped_framebuffer.h b/include/libcamera/internal/mapped_framebuffer.h index fb39adbf..fac86344 100644 --- a/include/libcamera/internal/mapped_framebuffer.h +++ b/include/libcamera/internal/mapped_framebuffer.h @@ -54,7 +54,7 @@ public: using MapFlags = Flags; - MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags); + MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags, bool usePlaneOffset = false); }; LIBCAMERA_FLAGS_ENABLE_OPERATORS(MappedFrameBuffer::MapFlag) diff --git a/src/libcamera/mapped_framebuffer.cpp b/src/libcamera/mapped_framebuffer.cpp index 6860069b..fcbb38ec 100644 --- a/src/libcamera/mapped_framebuffer.cpp +++ b/src/libcamera/mapped_framebuffer.cpp @@ -172,12 +172,13 @@ MappedBuffer::~MappedBuffer() * \brief Map all planes of a FrameBuffer * \param[in] buffer FrameBuffer to be mapped * \param[in] flags Protection flags to apply to map + * \param[in] usePlaneOffset Use offset stored in buffer's plane; default false * * Construct an object to map a frame buffer for CPU access. The mapping can be * made as Read only, Write only or support Read and Write operations by setting * the MapFlag flags accordingly. */ -MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags) +MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags, bool usePlaneOffset) { ASSERT(!buffer->planes().empty()); planes_.reserve(buffer->planes().size()); @@ -223,8 +224,14 @@ MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags) const int fd = plane.fd.get(); auto &info = mappedBuffers[fd]; if (!info.address) { - void *address = mmap(nullptr, info.mapLength, mmapFlags, - MAP_SHARED, fd, 0); + void *address; + if (usePlaneOffset) { + address = mmap(nullptr, plane.length, mmapFlags, + MAP_SHARED, fd, plane.offset); + } else { + address = mmap(nullptr, info.mapLength, mmapFlags, + MAP_SHARED, fd, 0); + } if (address == MAP_FAILED) { error_ = -errno; LOG(Buffer, Error) << "Failed to mmap plane: " @@ -236,7 +243,8 @@ MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags) maps_.emplace_back(info.address, info.mapLength); } - planes_.emplace_back(info.address + plane.offset, plane.length); + uint8_t *storedAddress = usePlaneOffset ? info.address : info.address + plane.offset; + planes_.emplace_back(storedAddress, plane.length); } } From patchwork Mon Aug 14 11:28:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18940 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D6AE7C32B0 for ; Mon, 14 Aug 2023 11:29:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7C255628E4; Mon, 14 Aug 2023 13:29:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692012540; bh=267zk0bj+HNkmSZr/ZXjLE2OK+LNcaCshBj12zeX6rY=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=uMrkP/3mq+ET60L50/Bg1o0T1wDt1G0iFaogUtLAO4bk/21JQ7rg8uOlgnPHtyxfU QmbhhKNlH5vkAilO7xRayzoBX+M7oxYAZTBQZSUFRnRHVIRUmAdD8OTCO6qGyqibe5 JWGaf+wQkXGsSJkGMa9RA77ZN62egRUuQzKGfOO5x1xTj+/1Q6QWhIXCsvoEE1NbvX BEJZ5LgFAPjZwvg5RI3/niG4h/t2krjnlRddp+Q99/jAFvg5yjm7lN42EcAQAjQQVJ 49Kangqg8y5IZo7wi/s2Z331fPYUUHsNwhOq4QRVJgU2L+MWzdCv1TEsC4jO8Z+shp k/gICcieAo9mg== Received: from mail-oa1-x30.google.com (mail-oa1-x30.google.com [IPv6:2001:4860:4864:20::30]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 76E2C61E0B for ; Mon, 14 Aug 2023 13:28:56 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QVT6zUUq"; dkim-atps=neutral Received: by mail-oa1-x30.google.com with SMTP id 586e51a60fabf-1c4d67f493bso826148fac.2 for ; Mon, 14 Aug 2023 04:28:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692012535; x=1692617335; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=6tSJ7JOO2z+X4Zmaf/GuvpJdVyVmlcb8Z4R8sdhfdp0=; b=QVT6zUUqZwnTsMh30wZpvk+vfMd6edwdSTgs/oYPitNFB1+0gM/xLRgudF0+L5foMT cEWmnnbURBgwjUTfBE3XSNhV4405xPAUH46CSKy+Q/g9tn137MJw5CgWeg71ng6ZiW9h uS/r2haaN21xB+FpjFZwXVTm874z8cCG5AhbEJfKFctqqF41lrGX5YPIoMJFRhMcs3y5 TnBVnr6l5VLInEs6Zi8tv1N6RaPnidGjNKCErBbhSwsNz9gTK/fee06/puQUNCthJ0f2 BZpYtk3ehte+vnD6mbK4C7NbG1dZI/NubZrwLTnDTkxuiOJFWr6D93bAZ7ZPILeJ8HHL kn9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692012535; x=1692617335; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6tSJ7JOO2z+X4Zmaf/GuvpJdVyVmlcb8Z4R8sdhfdp0=; b=ipUkByu9k0JsO1o4wqYp7ksIxKuLWmUtRWLrtJ6oynmHiozgkMHkZdPiUERXXuwh4s 7pzY/a5Hj7i4A1umqdmuGbosjVyJCuKJDFO/rWh0sUH86+SIJaffSJjhHh+9+YAUXjYI XIdnWahQtJSwOp5iAZ1JhTIhK5BUhyhGgzXtd7oxF9g4mR05mTTN2OAlyC7iqAKE15Hh JIIgNht39tgGs5Qy01KkfmQ7shXxWv89Gk1k0YCULcdH4vIgNKVFtr/sCK3QLH8wop9e jUeOmMZatjKuwIspkRx2EUPAxiroD+GDY8kkAukKt7udnYTNz2eOfywxyrbpDi0wBOaR HkLA== X-Gm-Message-State: AOJu0YwtKxY4c6Aj/S/JyuSMg2xsLSd7RQFir7bPseT8uZhgGXVav6Be 9qo10qtJRy9KoiJENfgBuzWw7RZN34L7FQ== X-Google-Smtp-Source: AGHT+IGe7Gn43V1JUUIxPdiMSegfVqXn6us7yfiXzI4QExjH9pwI0ateCkZPZaMcR6EqEa4qqQcc6w== X-Received: by 2002:a05:6871:886:b0:1c4:ee87:d3ea with SMTP id r6-20020a056871088600b001c4ee87d3eamr3424797oaq.36.1692012534860; Mon, 14 Aug 2023 04:28:54 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id c3-20020a9d6c83000000b006b92509e76esm4163817otr.32.2023.08.14.04.28.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 04:28:54 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 14 Aug 2023 04:28:47 -0700 Message-Id: <20230814112849.176943-4-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230814112849.176943-1-gabbymg94@gmail.com> References: <20230814112849.176943-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 3/5] libcamera: v4l2 device: Store buffer info in planes X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To perform a memory mapping using mmap, the MappedFrameBuffer class needs the plane offset and file descriptor information of the frame buffer's plane(s). This information is provided in the response to REQBUF, which happens during buffer allocation. Store the plane offset and file descriptor information in the buffer's plane at the time of allocation. Currently, there is a metadata buffer type (metadata format UVCH) that does not support exporting buffers using EXPBUF, so this should only be done if the buffer type is metadata capture. Signed-off-by: Gabby George --- src/libcamera/v4l2_videodevice.cpp | 32 ++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp index a72ef64d..8cf427c0 100644 --- a/src/libcamera/v4l2_videodevice.cpp +++ b/src/libcamera/v4l2_videodevice.cpp @@ -1402,18 +1402,28 @@ std::unique_ptr V4L2VideoDevice::createBuffer(unsigned int index) std::vector planes; for (unsigned int nplane = 0; nplane < numPlanes; nplane++) { - UniqueFD fd = exportDmabufFd(buf.index, nplane); - if (!fd.isValid()) - return nullptr; - FrameBuffer::Plane plane; - plane.fd = SharedFD(std::move(fd)); - /* - * V4L2 API doesn't provide dmabuf offset information of plane. - * Set 0 as a placeholder offset. - * \todo Set the right offset once V4L2 API provides a way. - */ - plane.offset = 0; + + if (buf.type != V4L2_BUF_TYPE_META_CAPTURE) { + UniqueFD fd = exportDmabufFd(buf.index, nplane); + if (!fd.isValid()) + return nullptr; + plane.fd = SharedFD(std::move(fd)); + + /* + * V4L2 API doesn't provide dmabuf offset information of plane. + * Set 0 as a placeholder offset. + * \todo Set the right offset once V4L2 API provides a way. + */ + plane.offset = 0; + } else { + /* Dmabuf fd is not exported for metadata, so store + * the offset from the querybuf call and this device's fd. + */ + SharedFD tmp(this->fd()); + plane.fd = tmp; + plane.offset = buf.m.offset; + } plane.length = multiPlanar ? buf.m.planes[nplane].length : buf.length; planes.push_back(std::move(plane)); From patchwork Mon Aug 14 11:28:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18941 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 56798C32B1 for ; Mon, 14 Aug 2023 11:29:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2541E628DD; Mon, 14 Aug 2023 13:29:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692012541; bh=szpRlwvfHuwJi9e/cV+qIhsj0OHhxLHDghQNsEK/8f4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=SfK9cHj4V1t5fFQlKzRcROAnFVTPYCYz4Wtg3WmRfMmY8nKhU/Q2fvH9RXGp9/jOU qrZMbbozEJ4lOqeZv+d6Q/HUsB96GnJJRovTfdYBRmdq3Ani7fAXzAJY/YaujGmJbT DKJzptm5ROqGErQ/sz/y55oPg3QshEP97+CpXQ8ZSiIC3x85CJ9mmmuPjpfZlL92rk o7/iP7rtiXg/ydePRtfHdMih2hYbmyKMnLY5c2qQI3Hzivr7LOiIB4BNslTTZbBae0 WNjYP2yJbu1+FzHS0BaF50A0Hm9ZK2WPrWhkKTd70Kw6jt8Cavebs2Mcak9WvPbesY 9dpz8/z58w+kQ== Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5A8C7628DB for ; Mon, 14 Aug 2023 13:28:57 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Not+KZnU"; dkim-atps=neutral Received: by mail-ot1-x32f.google.com with SMTP id 46e09a7af769-6bca66e6c44so3735614a34.0 for ; Mon, 14 Aug 2023 04:28:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692012536; x=1692617336; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2paCm8n5nk6/G3ElxO9IuZP3Z4eBx88uZvY2ElGxhDk=; b=Not+KZnUubJMvjuTOcwvhfnSmib04EyzzXYOXH5+hNJpnH7/Fwbn5RDspj3r5knCrx 6c1wWfb9AUYV3ICVNNbl4fKTBZ0ORdYYvYF72wBEsOrTiEh1Yb/D+eS5pwq5Fc3eviE4 Uxt1Saodr1uFX5161fJOXkjNu1TT939Wy/9Ovzl8rjHMdDGwVxnl+b2B2k9iRB6e+2uA Bb1sPUu2MTG+apDsTM/m7YK3VyE2DLspxl+JcWiUmNObTpkGX0rFDIsd8tUq7ACVOHd3 O63s6Yh0cdlKsRBInGzPyIbfCult/SHprM29UC1p+Up6H7hJOSbq85yFlEFHhPqrYwug BdqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692012536; x=1692617336; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2paCm8n5nk6/G3ElxO9IuZP3Z4eBx88uZvY2ElGxhDk=; b=FxEu7rJvqKojAaRX89pZ3s9hMU+J7Ckcj4KXNlyAxYwjaaBfdFQmuTJBaMk/kuwN97 3twNjjiHHUNSgQasOKo/3xiVYn38al2/rPEMtabCJBN3epBLeYgobcp4OoygPuFFvHnd zywzgJBDpXxdTmVh0ISBLWXCorpO7vKIdgcNBDjg8I6IARiMIqGNpsnFcF9GzDhvR0my shrVVAyz99miZmVafuE+xV66PJlWE/f5UZ6xT9HDEVQXB/GqBepWJhRN0CbxF7ocTqUp RJp9wlOjgiBXP/3WD03M1Os/NDGbuWrG++C2ymzFUPpRw26x0+n7ow2AeVs55AenYuxm omhQ== X-Gm-Message-State: AOJu0YzX3o3TjJDmA5jZuqSfsF4Ym6ANNNkhTv+yNquESeoLR/xgFvuw jjI8lE8xx09ajDTrm3x1sJ5W/iprOAOYuQ== X-Google-Smtp-Source: AGHT+IHjrbpcs3YSyLE8L1EbRDdGRPuif37IATGwiWpMmjU596ZQCgTiT7+Uf+SEaF7M8L/RYAFczg== X-Received: by 2002:a9d:7585:0:b0:6b5:f457:adaa with SMTP id s5-20020a9d7585000000b006b5f457adaamr8688818otk.29.1692012535912; Mon, 14 Aug 2023 04:28:55 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id c3-20020a9d6c83000000b006b92509e76esm4163817otr.32.2023.08.14.04.28.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 04:28:55 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 14 Aug 2023 04:28:48 -0700 Message-Id: <20230814112849.176943-5-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230814112849.176943-1-gabbymg94@gmail.com> References: <20230814112849.176943-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 4/5] libcamera: pipeline: uvcvideo: Allocate metadata buffers X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Perform the allocation and mapping of metadata buffers into libcamera's virtual address space. UVC metadata buffers cannot be exported as DMA buffer file descriptors, so use the MappedFrameBuffer class to map them into memory directly. This will give the UVC pipeline access to buffer data to extract timestamp information. Metadata buffers are internal to the UVC pipeline, so buffer memory should not be exposed to the user. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 97 +++++++++++++++++++- 1 file changed, 95 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 4470d8a2..51f30187 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -24,6 +24,7 @@ #include "libcamera/internal/camera.h" #include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/mapped_framebuffer.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" #include "libcamera/internal/sysfs.h" @@ -51,11 +52,17 @@ public: std::unique_ptr video_; std::unique_ptr metadata_; Stream stream_; + std::vector> metadataBuffers_; + std::map mappedMetadataBuffers_; + bool useMetadataStream_; + std::map> formats_; private: int initMetadata(MediaDevice *media); + const unsigned int minLengthHeaderBuf_ = 10; + bool generateId(); std::string id_; @@ -96,6 +103,10 @@ private: const ControlValue &value); int processControls(UVCCameraData *data, Request *request); + int createMetadataBuffers(Camera *camera, unsigned int count); + int cleanupMetadataBuffers(Camera *camera); + int cleanup(Camera *camera); + UVCCameraData *cameraData(Camera *camera) { return static_cast(camera->_d()); @@ -225,10 +236,66 @@ int PipelineHandlerUVC::configure(Camera *camera, CameraConfiguration *config) return -EINVAL; cfg.setStream(&data->stream_); + return 0; +} +int PipelineHandlerUVC::cleanupMetadataBuffers(Camera *camera) +{ + int ret = 0; + UVCCameraData *data = cameraData(camera); + + ret = data->metadata_->releaseBuffers(); + data->metadataBuffers_.clear(); //call the destructor for the frame buffers + data->mappedMetadataBuffers_.clear(); + data->useMetadataStream_ = false; + + return ret; +} + +int PipelineHandlerUVC::cleanup(Camera *camera) +{ + UVCCameraData *data = cameraData(camera); + cleanupMetadataBuffers(camera); + data->video_->releaseBuffers(); return 0; } +/* + * UVC Metadata stream does not support exporting buffers via EXPBUF, + * so it is necessary to create and store mmap-ed addresses. + * Metadata buffers are internal to libcamera. They are not, and + * cannot be, exposed to the user. + * + * Returns the number of buffers allocated and mapped. + * + * \return The number of buffers allocated, or a negative error code if + * the number of buffers allocated was not equal to "count" + * \retval -EINVAL if "count" buffers were not successfully allocated. + * \retval -ENOMEM if mmap failed. + */ +int PipelineHandlerUVC::createMetadataBuffers(Camera *camera, unsigned int count) +{ + UVCCameraData *data = cameraData(camera); + int ret = data->metadata_->allocateBuffers(count, &data->metadataBuffers_); + if (ret < 0) + return -EINVAL; + + for (unsigned int i = 0; i < count; i++) { + MappedFrameBuffer mappedBuffer(data->metadataBuffers_[i].get(), + MappedFrameBuffer::MapFlag::Read, true); + if (!mappedBuffer.isValid()) { + LOG(UVC, Error) + << "Failed to mmap metadata buffer: " + << strerror(mappedBuffer.error()); + return mappedBuffer.error(); + } + + data->mappedMetadataBuffers_.emplace(i, std::move(mappedBuffer)); + data->metadataBuffers_[i]->setCookie(i); + } + return ret; +} + int PipelineHandlerUVC::exportFrameBuffers(Camera *camera, Stream *stream, std::vector> *buffers) { @@ -247,20 +314,46 @@ int PipelineHandlerUVC::start(Camera *camera, [[maybe_unused]] const ControlList if (ret < 0) return ret; + if (data->useMetadataStream_) { + if (createMetadataBuffers(camera, count) < 0) { + LOG(UVC, Error) << "Unable to allocate buffers for UVC metadata stream."; + data->useMetadataStream_ = false; + } + } + ret = data->video_->streamOn(); if (ret < 0) { - data->video_->releaseBuffers(); + cleanup(camera); return ret; } + if (data->useMetadataStream_) { + ret = data->metadata_->streamOn(); + if (ret) { + LOG(UVC, Error) << "Failed to start metadata stream"; + return ret; + } + + for (std::unique_ptr &buf : data->metadataBuffers_) { + ret = data->metadata_->queueBuffer(buf.get()); + if (ret < 0) { + cleanupMetadataBuffers(camera); + return ret; + } + } + } return 0; } void PipelineHandlerUVC::stopDevice(Camera *camera) { UVCCameraData *data = cameraData(camera); + data->video_->streamOff(); - data->video_->releaseBuffers(); + + data->metadata_->streamOff(); + + cleanup(camera); } int PipelineHandlerUVC::processControl(ControlList *controls, unsigned int id, From patchwork Mon Aug 14 11:28:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18942 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A228FC32B2 for ; Mon, 14 Aug 2023 11:29:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AB059628E6; Mon, 14 Aug 2023 13:29:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692012541; bh=ABhfgmo5PUBSXwT8Jvvta5ZvStQKy++jlNCWwuPYcIg=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=sjDmf/IXHhTAslxv8iS45hmh5ECWzequXqFhFj+QNY29un+b0NMAqX+KPXni6LBbU MPX7FxuY3xXD73upwg6EAhvwHolUDZOaQp7KUCRO9qgrl2Ga7FFY/BEHcDywYltd4E 8d79Mtrazxj5lFEuTn5+sJuZWV0P+ce6bN5Vg8MiEAk6YyxxIxwq73HCyOPtj3bec/ umhQklBVaGvEuFgvOp7OPhJT9cWqaf1/xrOB99fTKB9iJ63tdLCGNH4TAnFtXT99Rl dGBhz04DiQek49Ex36ADTOXFzlwKyN96k1rSH28IrYjlhfd0FlcnmKBAsvy5jSflUd VgxejjCRYt7fg== Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 86309628DF for ; Mon, 14 Aug 2023 13:28:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aR/UQLML"; dkim-atps=neutral Received: by mail-ot1-x333.google.com with SMTP id 46e09a7af769-6bcd4b5ebbaso3730565a34.1 for ; Mon, 14 Aug 2023 04:28:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692012537; x=1692617337; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=pShnn9eKyZPi5uoMChY9/RWSzW6FXWMZIhCveXimmHI=; b=aR/UQLMLg62UNlaUiwZ/2c/PRPvyh3uYqNgtuaRz57pmC8GPoA4FurJLnD7vm+ZH8z CyvKDln+NXeKUNhp8KqXxODEY1KLRkq83uQQ3Y59F6ktcJkc0ISnERavZNNJiYve8KTx MvH4huLxfq6ggYWiMFq+7qlwNwv1BSM1cGry1d2oYVfxJKwO8IWstPfBe6YKxuaQ9EZn oAhezKTCit1syeo9FEvtFPU23OOYfgUKVFqZAmNGvuTlimDHMlvsWPMPRJno1AaSdnPZ W3VwbxnlgoFM9A4Vv4434Ocgnn5s9AW4KcatdncvlK2tH8ssjfb+bUYMML9V1Yc00vCB FaeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692012537; x=1692617337; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pShnn9eKyZPi5uoMChY9/RWSzW6FXWMZIhCveXimmHI=; b=cw3c/qU65fdAH78oPdD8bijZ048y402n7TAe6EdOqRM4y+U4lLmEd2G0d1lGnb717Y M+yZOV+nRs21mn2ZZK8RXczz3ktvFYzYSClXcI69Z+0nlgMxCSsC0VCNtTxeHBrZkIWk luKS55uCCwra5Uk0wjaxmKxXNJHMLxl+DFLSvjUXu1bQ+Qid4IwpMBCX36rjsI5z5o2j h0hWQd4sxg3ELG87XPBKKz5xM8d13gryCEkfcUYC1eZzf8R/khaIW1osdGDqVIIYTQMJ qEXxTGEGljX50bwd41vhGrBHfG55AsvxhJ/i2fSETGowXk57wqjw9ZGG10JbW0gdVV78 Bhrg== X-Gm-Message-State: AOJu0YxkxLsUQta6bo5+iKdwAjcAtCfv2D3N9IdBTckHvcLM9ss5OGCP NhlXtvz4uxB9l7ZYEgv26z9cQNaR7nK24w== X-Google-Smtp-Source: AGHT+IHsDct/sp9xSgzkJchQVT/FA8dlD5GvKjGS0USkEzX+SzBk8fJ5qd8p6u6A8HAkXH0Jo2BHng== X-Received: by 2002:a9d:63ce:0:b0:6b9:696d:716a with SMTP id e14-20020a9d63ce000000b006b9696d716amr9676790otl.18.1692012536904; Mon, 14 Aug 2023 04:28:56 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id c3-20020a9d6c83000000b006b92509e76esm4163817otr.32.2023.08.14.04.28.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Aug 2023 04:28:56 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 14 Aug 2023 04:28:49 -0700 Message-Id: <20230814112849.176943-6-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230814112849.176943-1-gabbymg94@gmail.com> References: <20230814112849.176943-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 5/5] libcamera: pipeline: uvcvideo: Handle metadata stream X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Register the metadata stream's buffer ready callback and start processing metadata buffers. Use the timestamp from the metadata buffer as the corresponding video buffer Requests' timestamp. Metadata buffers are synchronized with frames coming into the video stream using the sequence field of the buffers. They may come in either order (video buffer first or metadata buffer first), so store relevant information about the buffer required to set the metadata timestamp or complete the buffer request as soon as possible. The timestamp will improved upon in the next patch. For now, use the driver-provided metadata timestamp. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 157 ++++++++++++++++++- 1 file changed, 152 insertions(+), 5 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 51f30187..5c7ae064 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -12,6 +12,8 @@ #include #include +#include + #include #include @@ -34,6 +36,13 @@ namespace libcamera { LOG_DEFINE_CATEGORY(UVC) +/* This is used to memcpy */ +struct UVCMetadataPacked { + __u32 pts; + __u32 scr; + __u16 sofDevice; +} __attribute__((packed)); + class UVCCameraData : public Camera::Private { public: @@ -46,6 +55,7 @@ public: void addControl(uint32_t cid, const ControlInfo &v4l2info, ControlInfoMap::Map *ctrls); void bufferReady(FrameBuffer *buffer); + void bufferReadyMetadata(FrameBuffer *buffer); const std::string &id() const { return id_; } @@ -57,10 +67,14 @@ public: bool useMetadataStream_; std::map> formats_; + std::queue> waitingForVideoBuffer_; + std::queue> waitingForMDBuffer_; private: int initMetadata(MediaDevice *media); + const unsigned int frameStart_ = 1; + const unsigned int maxVidBuffersInQueue_ = 2; const unsigned int minLengthHeaderBuf_ = 10; bool generateId(); @@ -638,8 +652,16 @@ int UVCCameraData::init(MediaDevice *media) } controlInfo_ = ControlInfoMap(std::move(ctrls), controls::controls); + ret = initMetadata(media); + + if (!ret) { + metadata_->bufferReady.connect(this, &UVCCameraData::bufferReadyMetadata); + useMetadataStream_ = true; + } else { + useMetadataStream_ = false; + } - return initMetadata(media); + return 0; } bool UVCCameraData::generateId() @@ -824,16 +846,141 @@ void UVCCameraData::addControl(uint32_t cid, const ControlInfo &v4l2Info, ctrls->emplace(id, info); } +/* + * If there is a metadata buffer that hasn't been matched with a + * video buffer, check to see if it matches this video buffer. + * + * If there is a match, use the timestamp stored in the metadata queue + * for this video buffer's request. Complete this video buffer + * and its request. + * + * If there are no metadata buffers available to check for a match, + * push this video buffer's request object to the queue. It may + * be that the metadata buffer has not yet arrived. + * When the matching metadata buffer does come in, it will handle + * completion of the buffer and request. + * + * If more than maxVidBuffersInQueue_ video buffers have been added + * to the queue, something is wrong with the metadata stream and + * we can no longer use UVC metadata packets for timestamps. + * Complete all of the outstanding requests and turn off metadata + * stream use. + */ void UVCCameraData::bufferReady(FrameBuffer *buffer) { Request *request = buffer->request(); - - /* \todo Use the UVC metadata to calculate a more precise timestamp */ request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); - pipe()->completeBuffer(request, buffer); - pipe()->completeRequest(request); + if (useMetadataStream_) { + if (buffer->metadata().sequence == 0) { + /* \todo: we do not expect the first frame to have a + * metadata buffer associated with it. Why? + */ + pipe()->completeBuffer(request, buffer); + pipe()->completeRequest(request); + return; + } + + if (!waitingForMDBuffer_.empty()) { + unsigned int mdSequence = + std::get<0>(waitingForMDBuffer_.front()) + frameStart_; + if (mdSequence == buffer->metadata().sequence) { + request->metadata().set(controls::SensorTimestamp, + std::get<1>(waitingForMDBuffer_.front())); + pipe()->completeBuffer(request, buffer); + pipe()->completeRequest(request); + waitingForMDBuffer_.pop(); + return; + } + } else { + waitingForVideoBuffer_.push(std::make_pair(request, buffer)); + } + + if (waitingForVideoBuffer_.size() > maxVidBuffersInQueue_) { + while (!waitingForVideoBuffer_.empty()) { + Request *oldRequest = std::get<0>(waitingForVideoBuffer_.front()); + FrameBuffer *oldBuffer = std::get<1>(waitingForVideoBuffer_.front()); + oldRequest->metadata().set(controls::SensorTimestamp, + oldBuffer->metadata().timestamp); + pipe()->completeBuffer(oldRequest, oldBuffer); + pipe()->completeRequest(oldRequest); + waitingForVideoBuffer_.pop(); + } + } + } else { + pipe()->completeBuffer(request, buffer); + pipe()->completeRequest(request); + } +} + +void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) +{ + if (!useMetadataStream_ || buffer->metadata().status != FrameMetadata::Status::FrameSuccess) { + return; + } + + /* + * The metadata stream always starts at seq 1 and libcamera sets the start sequence to 0, + * so it's necessary to add one to match this buffer with the correct + * video frame buffer. + * + * \todo: Is there a better way to do this? What is the root cause? + */ + unsigned int mdSequence = buffer->metadata().sequence + frameStart_; + int pos = buffer->cookie(); + /* + * A UVC Metadata Block length field contains size of + * the header buf, length field, and flags field. + */ + uvc_meta_buf metadataBuf; + __u8 minLength = minLengthHeaderBuf_ + + sizeof(metadataBuf.length) + + sizeof(metadataBuf.flags); + size_t lenMDPacket = minLength + sizeof(metadataBuf.ns) + sizeof(metadataBuf.sof); + memcpy(&metadataBuf, mappedMetadataBuffers_.at(pos).planes()[0].data(), lenMDPacket); + + if (metadataBuf.length < minLength) { + LOG(UVC, Error) << "Received improper metadata packet. Using default timestamps."; + useMetadataStream_ = false; + return; + } + + /* + * If there is a video buffer that hasn't been matched with a + * metadata buffer, check to see if it matches this metadata buffer. + * + * If there is a match, use the timestamp associated with this + * metadata buffer as the timestamp for the video buffer's request. + * Complete that video buffer and its request. + * + * If there are no video buffers, push this metadata buffer's + * sequence number and timestamp to a shared queue. It may be that + * the metadata buffer came in before the video buffer. + * When the matching video buffer does come in, it will use this + * metadata buffer's timestamp. + */ + __u64 timestamp = metadataBuf.ns; + + if (!waitingForVideoBuffer_.empty()) { + Request *request = std::get<0>(waitingForVideoBuffer_.front()); + FrameBuffer *vidBuffer = std::get<1>(waitingForVideoBuffer_.front()); + unsigned int vidSequence = vidBuffer->metadata().sequence; + + if (vidSequence == mdSequence) { + request->metadata().set(controls::SensorTimestamp, + timestamp); + + pipe()->completeBuffer(request, vidBuffer); + pipe()->completeRequest(request); + waitingForVideoBuffer_.pop(); + } + } else { + waitingForMDBuffer_.push( + std::make_pair(buffer->metadata().sequence, + timestamp)); + } + metadata_->queueBuffer(buffer); } REGISTER_PIPELINE_HANDLER(PipelineHandlerUVC)