From patchwork Mon Aug 21 13:10:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18944 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1ADC4C3261 for ; Mon, 21 Aug 2023 13:10:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9E228627E4; Mon, 21 Aug 2023 15:10:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692623445; bh=V7dvhyL1dJls6TnTqwFfnwsQqRVVoRsc/vzZXaJCdJk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ykCMcihxYJkslOM4Bwsvt149Ik/+u9iwoGliAhsyMf9MebE85LkdyT4VsIC5vPIGs QFAjyrXddSUaZ5KtSMZZNRflIgtpFeGlO8SPIaIxyCxWyKN9eTM1x2WtN+DeEsnqW+ Lc9lUF1rcxX6Z6kUrm7wQQCPehhXkwtBxOJiW+eMap9zI2UcBz72MsGXjmArc1xtgo QjoNaIydUwozyfwcd24kAAJdU3tQkRrgxU97b/qIZ7131WUD2LYq6X9KS0k2d7KEOU S+ji3iyN3/EDefCJ3ucdY+UIsadexAykFY/yDEn///9hOWssX2c/As6+Wx3S/FQAt+ 54b9EuFP+RjrA== Received: from mail-oa1-x36.google.com (mail-oa1-x36.google.com [IPv6:2001:4860:4864:20::36]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E360661E09 for ; Mon, 21 Aug 2023 15:10:43 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="pp6zpTOF"; dkim-atps=neutral Received: by mail-oa1-x36.google.com with SMTP id 586e51a60fabf-1c0fff40ec6so2037825fac.2 for ; Mon, 21 Aug 2023 06:10:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692623442; x=1693228242; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=OmfY6WyS9NxuJh3gBonJnWAumuZMXL/N1xt6mkK1am8=; b=pp6zpTOFzscdasJLhGkGfBjKxZ5OdtUbhlrc3fCv0gBfRvUzLwB/yiHvR/UHxNBP8O mMYR1KiD5T+Y1N0reKNO1YTCvw6Mwaz/Tln/GD95yz0X3Ju+LW2iU8BLPmpJxb4BFhuH RFx/9hks5hKqOugt/oW+Lv7g4Ce+kzbmfis+R0gG69VUpGLAeImA22xnAakbzVQ+Xm6k 43PnqJHGJriPsA+U547KgmMeW9ZYuPTZsNtWsOchPJ4KIUJVGgc6eIMfcn7VTvDzJm1b AnB3KzMN+Pke+w1P2Tq13A1GK05/qxVcd+SSQDskopBzT+CGDWFAUXFOXSx3TTGV3x2X rx+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692623442; x=1693228242; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OmfY6WyS9NxuJh3gBonJnWAumuZMXL/N1xt6mkK1am8=; b=k1UxmMFJVbCXj6DpmHG7ZWwmws1PzZvP0yJdvr/9gbVy0kHbWSDlfzo/9It78abQm+ 13qUkJt/tyV/K8hAa/UjUW6WdN9f2yfwHHE5w9LHNEhX20eVWdxXRSCu+eTZ7sZkLV2l 5LYCaGA+auWCp1D2QB/uFUFWmVDAmG2emfsQ0oAM0CJI3arjrHw80uTToRmdNsO6a2Ku ta4/At0YuhWSVgXjsb4oJpWSjuZx9smdACBZ7HdZdoP0wAswVFdNJwQsORmrGOkbQdr/ 85H4YtqBsBrzrYe9F/TRn3isKW1ugaezUHc5NY0JHZ1OiBFUzXqdQqNRivMFjKKiAcvO sB0g== X-Gm-Message-State: AOJu0YwWXhssVM5xAf4Wt0y8B456pa/RyC0aSAJlTCPa4HOlKnw480is pMZkClbsEuYs7vQsyyG8DSoYDSw8zDY= X-Google-Smtp-Source: AGHT+IGc8asc16gReMOCbrjMICJSWpupZ7LniilQ52PK0Y6byGaNy7xPTfdjoZgB9YNZPaZ/yGvobQ== X-Received: by 2002:a05:6870:e313:b0:1b0:222f:9cda with SMTP id z19-20020a056870e31300b001b0222f9cdamr8864169oad.0.1692623442403; Mon, 21 Aug 2023 06:10:42 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id ci1-20020a056871c48100b001bb51450d85sm4160283oac.4.2023.08.21.06.10.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 06:10:42 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 21 Aug 2023 06:10:35 -0700 Message-Id: <20230821131039.127370-2-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821131039.127370-1-gabbymg94@gmail.com> References: <20230821131039.127370-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 1/5] libcamera: pipeline: uvcvideo: Add UVC metadata node X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Identify and open the UVC metadata's video node. This will give us access to metadata associated with the video stream of the uvc camera. The user will not have access to this video node and will not need to manage its data in any way. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 42 ++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 38f48a5d..dbe0cc8c 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -49,10 +49,13 @@ public: const std::string &id() const { return id_; } std::unique_ptr video_; + std::unique_ptr metadata_; Stream stream_; std::map> formats_; private: + int initMetadata(MediaDevice *media); + bool generateId(); std::string id_; @@ -411,6 +414,39 @@ bool PipelineHandlerUVC::match(DeviceEnumerator *enumerator) return true; } +int UVCCameraData::initMetadata(MediaDevice *media) +{ + int ret; + + const std::vector &entities = media->entities(); + std::string dev_node_name = video_->deviceNode(); + auto metadata = std::find_if(entities.begin(), entities.end(), + [&dev_node_name](MediaEntity *e) { + return e->type() == MediaEntity::Type::V4L2VideoDevice + && !(e->flags() & MEDIA_ENT_FL_DEFAULT); + }); + + if (metadata == entities.end()) { + return -ENODEV; + } + + /* configure the metadata node */ + metadata_ = std::make_unique(*metadata); + ret = metadata_->open(); + if (ret) + return ret; + + if (!(metadata_->caps().isMeta())) { + /* + * UVC Devices are usually only anticipated to expose two + * devices, so we assume the non-default device is the metadata + * device node + */ + return -EINVAL; + } + return 0; +} + int UVCCameraData::init(MediaDevice *media) { int ret; @@ -512,6 +548,12 @@ int UVCCameraData::init(MediaDevice *media) controlInfo_ = ControlInfoMap(std::move(ctrls), controls::controls); + ret = initMetadata(media); + if (ret) { + metadata_ = nullptr; + LOG(UVC, Error) << "Could not find a metadata video device."; + } + return 0; } From patchwork Mon Aug 21 13:10:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18945 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9532AC32B0 for ; Mon, 21 Aug 2023 13:10:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 28305627E2; Mon, 21 Aug 2023 15:10:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692623448; bh=DMCXw05gQo2OpLrAzcVL4l/cyzLBkRuuVkgP6P5OnJs=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=R6ET98cO75JqwbCkjIefTgi6VnWjRvU+g3bNPpzFb7qBxmSqoe+9nAXXd08M34C/B McZUl6CtrafHJL9Xpn7yYzQMuovMiwRUiAoZdUXvfnw7oraLGPp7FDvuqYcdf9D4G0 /uP76GFrrmROWD/WVVO/Me1hyG0pfUZ9E522ucIBYI+hoHTMvh0gtgM5lfZ7d09YNC lujzGF2TY+0tQKTBdBZ/c1+UZAqjA6JeyOwJDBxhAEZN2l8LPKYTbtMgHlwo1azDz+ xnVD57mhXbKVdHOi9gE1ml4ZQaGQFRFM2wf/oUmGjlTS91ib9agrgbmVglEobWMuF4 FXLVEQa1l/WMQ== Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 064B561E09 for ; Mon, 21 Aug 2023 15:10:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hqBJSkqa"; dkim-atps=neutral Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-1cca0a1b3c7so18548fac.2 for ; Mon, 21 Aug 2023 06:10:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692623443; x=1693228243; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=gLWaKmx/cdcNVCylxEJWFcipivTebMrGpD7wATBJy5o=; b=hqBJSkqa8Ng97Vz8RDYeQtRC/UYBC+PcpnOSFjDCGPLC3b4/lIe1rTyf45eeTYedPO 6kqW2qFshq42FcAosQaF2seOXkkLwwMDfNnDBlxJbzhl0p+bh5MJv5DtXpvlwqtbcxq1 uZsz6rbUXzSIKJ8asSJbPY8Tioy+MO15u0BdPIK/zfk7pJ7SOAhHIY1inRIsh8+nS+8J uzQLm7LDeEXwbJYcmSzgEJ7sdMleTmt+cSZB5ipwxsUunxzAzscWnDQoMDWzjMr+R+aa 360999vvFvYMVSS9sqtxRFhJnsshKb+UUVIr/DlUdWFb9xOifI+rLyKqym8pmOpyrfhi WqHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692623443; x=1693228243; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gLWaKmx/cdcNVCylxEJWFcipivTebMrGpD7wATBJy5o=; b=EGg+SLEEqeQfxLNTDKotmc1c3GPLBcaKbmY6Lc+7NALZgrhusNOw9OS6Qh3MkOZZAv WNnUdv++5VpevVB8jnkVYdWHnKyPirrIIjXHwrvMLU1QbxdnIfOD+A6ExaVQRl/7aSYI OVtrDF/bFyXPaOXM9tDXwRGfQWewIzLFajBZgwb8XJkUO4pFtbp310ytP66WPaIcoaej akVRIhT2dNkTtjejALgHThpgQYj5dNqfna6Z7T7kbGuc0WNzCRATT7six2+IgzDU+bV3 w1lzSmM6xDt0+YA9q8lLBYOEUzRQTTJT+E6wh2mYv8tBLekMAKT8NC117/Z1T2EgVpkw SIDw== X-Gm-Message-State: AOJu0Yz6V6zLBhVkb6+jCCYTWkrRygq7MOJJxtMIPRnSYhQyDvkJ7ssD lKogWalcpVdVSQwBJuocWmLpUDUT4xY= X-Google-Smtp-Source: AGHT+IFe737xihYZsVt677Rbiqp7a+FY84GHAlVGiU7VbU5CLiDUo6Rn9c9HDbqDe9RzwV6+uP+u/w== X-Received: by 2002:a05:6870:40ca:b0:1bc:3f6:579b with SMTP id l10-20020a05687040ca00b001bc03f6579bmr10132146oal.22.1692623443533; Mon, 21 Aug 2023 06:10:43 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id ci1-20020a056871c48100b001bb51450d85sm4160283oac.4.2023.08.21.06.10.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 06:10:43 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 21 Aug 2023 06:10:36 -0700 Message-Id: <20230821131039.127370-3-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821131039.127370-1-gabbymg94@gmail.com> References: <20230821131039.127370-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 2/5] libcamera: MappedFrameBuffer: Use stored plane offset X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" As it is written, the MappedFrameBuffer causes a failure in mmap when attempting to map UVC metadata planes. UVC metadata (four character code UVCH) does not support exporting buffers as file descriptors, so mmap can be used to give libcamera access to the metadata buffer planes. It is convenient to use the already-existing MappedFrameBuffers class, but the class must be modified to support mapping using file descriptors of the video node itself. To do this, mmap needs information obtained through a call to QUERYBUF, namely, the plane offset for buffer planes. Modify the constructor of a MappedFrameBuffer to use the plane offset directly in the call to mmap, rather than the hard-coded 0 value. The current version does not work when a buffer cannot be exported as a dma buf fd. The fd argument to mmap must be the one obtained on a call to open() for the video node (ie, /dev/videoX). This method of mapping buffer planes requires the arguments to mmap to be exactly the length and offset QUERYBUF provides. Mmap will return a -EINVAL if this is not the case. Signed-off-by: Gabby George --- .../libcamera/internal/mapped_framebuffer.h | 3 ++- src/libcamera/mapped_framebuffer.cpp | 20 ++++++++++++++----- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/include/libcamera/internal/mapped_framebuffer.h b/include/libcamera/internal/mapped_framebuffer.h index fb39adbf..04096d1b 100644 --- a/include/libcamera/internal/mapped_framebuffer.h +++ b/include/libcamera/internal/mapped_framebuffer.h @@ -54,7 +54,8 @@ public: using MapFlags = Flags; - MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags); + MappedFrameBuffer(const FrameBuffer *buffer, + MapFlags flags, bool usePlaneOffset = false); }; LIBCAMERA_FLAGS_ENABLE_OPERATORS(MappedFrameBuffer::MapFlag) diff --git a/src/libcamera/mapped_framebuffer.cpp b/src/libcamera/mapped_framebuffer.cpp index 6860069b..3d2cc332 100644 --- a/src/libcamera/mapped_framebuffer.cpp +++ b/src/libcamera/mapped_framebuffer.cpp @@ -172,12 +172,13 @@ MappedBuffer::~MappedBuffer() * \brief Map all planes of a FrameBuffer * \param[in] buffer FrameBuffer to be mapped * \param[in] flags Protection flags to apply to map + * \param[in] usePlaneOffset Use offset stored in buffer's plane; default false * * Construct an object to map a frame buffer for CPU access. The mapping can be * made as Read only, Write only or support Read and Write operations by setting * the MapFlag flags accordingly. */ -MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags) +MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags, bool usePlaneOffset) { ASSERT(!buffer->planes().empty()); planes_.reserve(buffer->planes().size()); @@ -223,8 +224,14 @@ MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags) const int fd = plane.fd.get(); auto &info = mappedBuffers[fd]; if (!info.address) { - void *address = mmap(nullptr, info.mapLength, mmapFlags, - MAP_SHARED, fd, 0); + void *address; + if (usePlaneOffset) { + address = mmap(nullptr, plane.length, mmapFlags, + MAP_SHARED, fd, plane.offset); + } else { + address = mmap(nullptr, info.mapLength, mmapFlags, + MAP_SHARED, fd, 0); + } if (address == MAP_FAILED) { error_ = -errno; LOG(Buffer, Error) << "Failed to mmap plane: " @@ -233,10 +240,13 @@ MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags) } info.address = static_cast(address); - maps_.emplace_back(info.address, info.mapLength); + maps_.emplace_back(info.address, usePlaneOffset ? plane.length : + info.mapLength); } - planes_.emplace_back(info.address + plane.offset, plane.length); + uint8_t *storedAddress = usePlaneOffset ? info.address : + info.address + plane.offset; + planes_.emplace_back(storedAddress, plane.length); } } From patchwork Mon Aug 21 13:10:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18946 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5077EC32B1 for ; Mon, 21 Aug 2023 13:10:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D79B8628C0; Mon, 21 Aug 2023 15:10:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692623448; bh=x0m3Q6Zm5HNEYCWrPDKCa4gssb4L4IG3sXuRHUIlQiY=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=IMix/AsxD5AZoeQ5jqN62BP4sh2PiEwjpJjQfm8Z5Gt/5ecm5N9Z+k48O8KmUXr+y VRhtgM5c4oTq1mdi35LiODTRSnuTpzHpDAI9voZFylsGvKAMgcRNfoYJ5DwPoop2rj iVKmGCaa9JgYPcHx7TxyIccdgj6dgl9tz0GJ/ewc5Zyk98C0zjZpfVZTTk/zZYvR3X wy/sh0BubSnCj8MubZgRojOGi0HpP8fLt8G+asY+WyG6FhT63zIAxO5Y1p7nW53ZD/ 5K9xO5kwjVu2rrIHABf9k1uLCRyPW3BWMmvHZfO0z2wKBoN/SfqhLvGLc0XNgD9Cc2 boavBcKnB44kw== Received: from mail-oa1-x36.google.com (mail-oa1-x36.google.com [IPv6:2001:4860:4864:20::36]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1B8FB627E1 for ; Mon, 21 Aug 2023 15:10:46 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="GvwjXhmZ"; dkim-atps=neutral Received: by mail-oa1-x36.google.com with SMTP id 586e51a60fabf-1c4dd644cf8so2202463fac.0 for ; Mon, 21 Aug 2023 06:10:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692623444; x=1693228244; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=nS1565gi8HITXrBfkzEzaklAo8ZiCurdIwCfe5mRzKg=; b=GvwjXhmZnKwXgiHvvIBI7WFqCpCfIyKjbtkvDD73oz7XlGjJ4FU5rr4Exm31In2rDM RLJq6/xPRWav9LbZ9vxWePdU1XUds+Ij/KFLwF10ILCULdHUQBi0L0UWIsZ5n5i2Mv5H kCdp008rEa46aSjDLgLLFjgTsOW60efp6XSFZVYheR/FhFLQ48xWfzmYO4Vu57s3qd2R 7DT1WTwligYoXGpF0570uIiFT/hJpB655OurMUqQJwYRvUNNFV4aTI8JxxO5fmB/ReWT LqUQlR8UE6CpLOv2QIIaUdxm9aR5Idhbm8BqhIkrPk7SWm+oHMaNjFToTx9BnvuN5i+G DsUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692623444; x=1693228244; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nS1565gi8HITXrBfkzEzaklAo8ZiCurdIwCfe5mRzKg=; b=aFkBcZLKsxFRO9D0uQS3hPUper1757TUHzWUN7MeJBq5FoarRRu5V3hZF0LtoYpbLG PEH872Y5EODIJeG6z1pKSAAorOzdNFd4byrU8lqXzh1N42TIyzwIpJy+o8VK7nveox14 yNTvOUuKi/cQ3vEPFHculCW1NpPKDd4+79aKYNCm+LRsjNEnn6kUOPG9CNP7/6AkWnXr ej9BXQezNzsRlscuB3eNdawRzaio+VLa7dpDf5SpBR2HkbZe/RMsBKyyudpQlyco/dmd CdU6dQ5sNgayi+Vm7Ao0r2JCcwLlOEbMlemuBkCDYtYhvAry4zWPS9fdZosE22C+ehOt p8QQ== X-Gm-Message-State: AOJu0Ywd/CTuos+Ohv0pSoq3tVuobRccvNkS8Jy4Y/FuqYJU8xU9e4hW ah0/O3JpVz+aSaq+0iK3YEyf1INTvlg= X-Google-Smtp-Source: AGHT+IEJmgsGy9aZkt9MLs2B85KK4lKLaxgd7vEqCTitMK+t+4iqvf7ZgM/61brhB9CqPgPEHh7tjw== X-Received: by 2002:a05:6870:470e:b0:1be:deef:748a with SMTP id b14-20020a056870470e00b001bedeef748amr9832354oaq.50.1692623444515; Mon, 21 Aug 2023 06:10:44 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id ci1-20020a056871c48100b001bb51450d85sm4160283oac.4.2023.08.21.06.10.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 06:10:44 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 21 Aug 2023 06:10:37 -0700 Message-Id: <20230821131039.127370-4-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821131039.127370-1-gabbymg94@gmail.com> References: <20230821131039.127370-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 3/5] libcamera: v4l2 device: Store buffer info in planes X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To perform a memory mapping using mmap, the MappedFrameBuffer class needs the plane offset and file descriptor information of the frame buffer's plane(s). This information is provided in the response to REQBUF, which happens during buffer allocation. Store the plane offset and file descriptor information in the buffer's plane at the time of allocation. Currently, there is a metadata buffer type (metadata format UVCH) that does not support exporting buffers using EXPBUF, so this should only be done if the buffer type is metadata capture. Signed-off-by: Gabby George Reviewed-by: Vedant Paranjape --- src/libcamera/v4l2_videodevice.cpp | 31 +++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp index a72ef64d..e57cb131 100644 --- a/src/libcamera/v4l2_videodevice.cpp +++ b/src/libcamera/v4l2_videodevice.cpp @@ -1402,18 +1402,27 @@ std::unique_ptr V4L2VideoDevice::createBuffer(unsigned int index) std::vector planes; for (unsigned int nplane = 0; nplane < numPlanes; nplane++) { - UniqueFD fd = exportDmabufFd(buf.index, nplane); - if (!fd.isValid()) - return nullptr; - FrameBuffer::Plane plane; - plane.fd = SharedFD(std::move(fd)); - /* - * V4L2 API doesn't provide dmabuf offset information of plane. - * Set 0 as a placeholder offset. - * \todo Set the right offset once V4L2 API provides a way. - */ - plane.offset = 0; + + if (buf.type == V4L2_BUF_TYPE_META_CAPTURE) { + /* + * Dmabuf fd is not exported for metadata, so store + * the offset from the querybuf call and this device's fd. + */ + plane.fd = SharedFD(this->fd()); + plane.offset = buf.m.offset; + } else { + UniqueFD fd = exportDmabufFd(buf.index, nplane); + if (!fd.isValid()) + return nullptr; + plane.fd = SharedFD(std::move(fd)); + /* + * V4L2 API doesn't provide dmabuf offset information of plane. + * Set 0 as a placeholder offset. + * \todo Set the right offset once V4L2 API provides a way. + */ + plane.offset = 0; + } plane.length = multiPlanar ? buf.m.planes[nplane].length : buf.length; planes.push_back(std::move(plane)); From patchwork Mon Aug 21 13:10:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18947 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0FE42BF415 for ; Mon, 21 Aug 2023 13:10:52 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A5BE3628D2; Mon, 21 Aug 2023 15:10:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692623451; bh=rPAqqsyOSUamSnOAgOEB1nNiDlbbH8F5K4TXQGSNDmc=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=sUo540ObJSr8FW6SD6b3QyA0zh6jXmUCkLqnv0iRVKSa+cHBOOS2FiNIyImHQD2pu 11r+KhB8LeuUKo04cTo8pNkguFoMmKk/ia94/fNTDsM6hkaYGgLA0vEIWerP49Dq05 +vfqwkJjBZeiDG6oei6YiwvYKH2rSKXo2KaC2zplIwO0XisXzzO3bKhtRRiDi8n88/ yhwvXMjW9Lly/g3G+Q0tAErCAYSfs44On1c4taVgXRvK4z6E4x6uup68zg5r4eOFgZ rL1sCSx1S5iARcJwtUWyCSIg4VaLfXjTeHn4J/A3to41gTrhrr7XcjE9u6zQpX1Nb/ e/XBBcf1HX0Ow== Received: from mail-oa1-x2f.google.com (mail-oa1-x2f.google.com [IPv6:2001:4860:4864:20::2f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1D70661E09 for ; Mon, 21 Aug 2023 15:10:47 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="XAz0LZjj"; dkim-atps=neutral Received: by mail-oa1-x2f.google.com with SMTP id 586e51a60fabf-1c4b4c40281so2370819fac.1 for ; Mon, 21 Aug 2023 06:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692623445; x=1693228245; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=EU6t+fl/QqCK94Dhs7LxVZPCFNQK/h1wuTeKXag0jbU=; b=XAz0LZjj4dTHE/R1BR7lULrWmujiUM98d4ZkfHUU3t/tvnzGJ4I5Pa7EQibsBw63Bg 3mSD1gBz4uh7R5+M/GsDexOmIzTmdFJU8ORzC9/yC0nMUXYpDOkvjfGQ4BqX2INIfuEo yFTZevWacZ51RcgViD1EZxJFausOglLubc/JtOl41wi8M/VvBJzfKel4/B81ieaLGXD7 /VYvVwBFLbkQgJtEQi540DktT+2Y0v9/ckXoLt/OCzn9DtxlHYIhhrCfvrHfDGFl0Ldr YAu7Ol7NTa+KM5VqsSBK/Nu7MG67phcq+ke8VN8EjHzMpsyZQ2Xt3qHZdLnVEPi843Mp FM2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692623445; x=1693228245; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EU6t+fl/QqCK94Dhs7LxVZPCFNQK/h1wuTeKXag0jbU=; b=GSm+YBU69C583HUJaMm/ZXgTgniGKHJUGfpaQ75Glapu1/KGwztgDZUN4Sc9XhgYeT 6FE1SF7c3vnGE+XoBXLoZXv3XNmGuAugWDeNMzeS4LKFFimz6e6SvFohSzdXGE+B1CFu cYk/xbwErW5Ka3UcqYqXEqppUDYVQLZ6hi6pkv95sy//J1HW16vIBkfpHwVVT9m+RfbJ Jks7UHRf9qMYzSiwZYdh3Wj+Q3IK1wrq4KhGSvtDGI6p4A/9IERdVLRGUg0rt+zI/B/9 EPmzHpumku9upv41F+kBbcpZQUfpByXwxvA5Z9JXwicvG/yD9IUliq0Q3DocZ/QvNa3p ha0Q== X-Gm-Message-State: AOJu0YyuOfnvSPLslurOkzjIPer0E5Pb2/TeF7RDcrbWjNKgII80TQju OP/hPrGlvYDb+6vjdyAEgjt52CvMyTc= X-Google-Smtp-Source: AGHT+IGpoqNewMqbPmpHwQMEdvahBu2zXq1QIDCtHpvenaN8k9yyPlgf6IpgEGPAyaWHPWHOyGEO0g== X-Received: by 2002:a05:6870:2198:b0:1b7:273d:780a with SMTP id l24-20020a056870219800b001b7273d780amr4653240oae.22.1692623445541; Mon, 21 Aug 2023 06:10:45 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id ci1-20020a056871c48100b001bb51450d85sm4160283oac.4.2023.08.21.06.10.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 06:10:45 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 21 Aug 2023 06:10:38 -0700 Message-Id: <20230821131039.127370-5-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821131039.127370-1-gabbymg94@gmail.com> References: <20230821131039.127370-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 4/5] libcamera: pipeline: uvcvideo: Allocate metadata buffers X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Perform the allocation and mapping of metadata buffers into libcamera's virtual address space. UVC metadata buffers cannot be exported as DMA buffer file descriptors, so use the MappedFrameBuffer class to map them into memory directly. This will give the UVC pipeline access to buffer data to extract timestamp information. Metadata buffers are internal to the UVC pipeline, so buffer memory should not be exposed to the user. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 98 +++++++++++++++++++- 1 file changed, 96 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index dbe0cc8c..c8d6633f 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -24,6 +24,7 @@ #include "libcamera/internal/camera.h" #include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/mapped_framebuffer.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" #include "libcamera/internal/sysfs.h" @@ -51,6 +52,9 @@ public: std::unique_ptr video_; std::unique_ptr metadata_; Stream stream_; + std::vector> metadataBuffers_; + std::map mappedMetadataBuffers_; + std::map> formats_; private: @@ -96,6 +100,10 @@ private: const ControlValue &value); int processControls(UVCCameraData *data, Request *request); + int createMetadataBuffers(Camera *camera, unsigned int count); + int cleanupMetadataBuffers(Camera *camera); + int cleanup(Camera *camera); + UVCCameraData *cameraData(Camera *camera) { return static_cast(camera->_d()); @@ -225,10 +233,67 @@ int PipelineHandlerUVC::configure(Camera *camera, CameraConfiguration *config) return -EINVAL; cfg.setStream(&data->stream_); + return 0; +} + +int PipelineHandlerUVC::cleanupMetadataBuffers(Camera *camera) +{ + int ret = 0; + UVCCameraData *data = cameraData(camera); + + if (data->metadata_) + data->metadata_->releaseBuffers(); + data->metadataBuffers_.clear(); + data->mappedMetadataBuffers_.clear(); + data->metadata_ = nullptr; + return ret; +} + +int PipelineHandlerUVC::cleanup(Camera *camera) +{ + UVCCameraData *data = cameraData(camera); + cleanupMetadataBuffers(camera); + data->video_->releaseBuffers(); return 0; } +/** + * UVC Metadata stream does not support exporting buffers via EXPBUF, + * so it is necessary to create and store mmap-ed addresses. + * Metadata buffers are internal to libcamera. They are not, and + * cannot be, exposed to the user. + * + * Returns the number of buffers allocated and mapped. + * + * \return The number of buffers allocated, or a negative error code if + * the number of buffers allocated was not equal to "count" + * \retval -EINVAL if "count" buffers were not successfully allocated. + * \retval MappedFrameBuffer::error() + */ +int PipelineHandlerUVC::createMetadataBuffers(Camera *camera, unsigned int count) +{ + UVCCameraData *data = cameraData(camera); + int ret = data->metadata_->allocateBuffers(count, &data->metadataBuffers_); + if (ret < 0) + return -EINVAL; + + for (unsigned int i = 0; i < count; i++) { + MappedFrameBuffer mappedBuffer(data->metadataBuffers_[i].get(), + MappedFrameBuffer::MapFlag::Read, true); + if (!mappedBuffer.isValid()) { + LOG(UVC, Warning) + << "Failed to mmap metadata buffer: " + << strerror(mappedBuffer.error()); + return mappedBuffer.error(); + } + + data->mappedMetadataBuffers_.emplace(i, std::move(mappedBuffer)); + data->metadataBuffers_[i]->setCookie(i); + } + return ret; +} + int PipelineHandlerUVC::exportFrameBuffers(Camera *camera, Stream *stream, std::vector> *buffers) { @@ -249,18 +314,47 @@ int PipelineHandlerUVC::start(Camera *camera, [[maybe_unused]] const ControlList ret = data->video_->streamOn(); if (ret < 0) { - data->video_->releaseBuffers(); + cleanup(camera); return ret; } + /* + * If metadata allocation fails, exit this function but + * do not return a failure as video started successfully. + * Fall back on using driver timestamps. + */ + if (data->metadata_) { + if (createMetadataBuffers(camera, count) < 0 || + data->metadata_->streamOn()) { + LOG(UVC, Warning) + << "Metadata stream unavailable. Using driver timestamps."; + data->metadata_ = nullptr; + return 0; + } + + for (std::unique_ptr &buf : data->metadataBuffers_) { + ret = data->metadata_->queueBuffer(buf.get()); + if (ret < 0) { + LOG(UVC, Warning) + << "Metadata stream unavailable. Using driver timestamps."; + cleanupMetadataBuffers(camera); + return 0; + } + } + } return 0; } void PipelineHandlerUVC::stopDevice(Camera *camera) { UVCCameraData *data = cameraData(camera); + data->video_->streamOff(); - data->video_->releaseBuffers(); + + if (data->metadata_) + data->metadata_->streamOff(); + + cleanup(camera); } int PipelineHandlerUVC::processControl(ControlList *controls, unsigned int id, From patchwork Mon Aug 21 13:10:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gabrielle George X-Patchwork-Id: 18948 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7C2BAC32B2 for ; Mon, 21 Aug 2023 13:10:53 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AD39B628BE; Mon, 21 Aug 2023 15:10:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1692623452; bh=6AxJ8c4c7HXmRYYCXvoO2RunB2yaegrKd1K9gPzMqbM=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=hOcVR+7i3cY2jtZz4cF5jmOesLQs0yZGpfgJRaGYbLnc8buuJcqLgKF5jumxbdSN+ JLcetcxLc5qFdOO2frZqmrDVJ/ZKhdVFo6X8d2PWPODmChTVLDvw5rGI9XZJVLw5aN QBfo7rN3vSg2kUxbYqj38uxvaUwJ3o2Kr/MC2a3qebsr3BFck7Lruipm4uxC2aVB9p 06YqpG+tBcDvXpZJxLTyRWw/yTfPtRcUc0IG1WzZGwB7CPxUx+15bERQdt2qTP8U7r L0ptpZy81VgBipbuegr+tYrmQ0eMxE7vtC+1vq4KYIeMli3mzIvVB2EtMBZrJva9US XPs9jLJE5/x1w== Received: from mail-oa1-x33.google.com (mail-oa1-x33.google.com [IPv6:2001:4860:4864:20::33]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2B045628C1 for ; Mon, 21 Aug 2023 15:10:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="UeDUtaXL"; dkim-atps=neutral Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-1a1fa977667so2192314fac.1 for ; Mon, 21 Aug 2023 06:10:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692623446; x=1693228246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=a0EOWnwLCT3Q7nV/PP4nZpbXd+qQfFMqXt5oWOXOEyI=; b=UeDUtaXLyVOBWsrk5nBz33ik5I6NwdjnamKH1yQTIQWc1ArsUp040jQ1qqR6qewcOw 6AjdXSvJOYQDt7a5MVS3OCLscInRiZG7afpM8N2KEuplo9WbP0cjwDR2SrxK1yFotjPA 4NDtnzZEsp5kegb6QVmQnUKgRC0o4fPedIO2NtlDmGclhO5ZF78IRtlqtdsj+7UeYiT5 dZ4j0Oov+B7bwUs5p8Ee6PnWiMp91aTVHmNsgOjk80cfix83KZ+5S7/u7sgiIOTPGIDy lBOzGagvRBV/kJZUTSgjSR0qfferWrngvMqInM5hBbFAymC0qHwuslspY1k6lJblU9RW oXhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692623446; x=1693228246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a0EOWnwLCT3Q7nV/PP4nZpbXd+qQfFMqXt5oWOXOEyI=; b=lXgmSx4v1UdvlIwHk03Th2fR7YHMyVs4OPSPzOQguZjI7AOTpNiSG7lH97lm/UUkJp +NOr26TxB1s3LJOKpHWZ0MQrqtDVp8Vl8YgCuDdZEGgNqRpnuZ87nrjrHEtNeDUxg23M YrrBHiHUOOm9YTXKAzta9AXw89PAovjhzdEY+Tdfg5r39OuRyHQScjUrK3v8FV8hPYo7 QP+kzmRtW6MoP4TOWfCo2PPjPJBgyS3gDTcsvUBRwcSFas6zA8T9WFh3fKEjkdSLK3N9 o6CoRTjVOAFSY6BA/JvqSozpcKJIGXYLbKmfr4jKNnUCMHQfxEGEwbjgay6wBdooyl0X +k4g== X-Gm-Message-State: AOJu0Yz0fNU81rQxEGmxkR2DpsO7wVsn7/u63Ztj04LY/uTSVZs3A0GF tgwgCFj1TpPo5MbqtxS3ddK/Gl7ZyLQ= X-Google-Smtp-Source: AGHT+IFbiiJai62GYO2Je+gLgIVtGay20sCY9TAG7cudF32BI3DCpfOX8tEzaey4OcO00oUdMPXchQ== X-Received: by 2002:a05:6870:4193:b0:1c0:2e8f:17fd with SMTP id y19-20020a056870419300b001c02e8f17fdmr10123197oac.40.1692623446638; Mon, 21 Aug 2023 06:10:46 -0700 (PDT) Received: from localhost.localdomain (97-115-76-16.ptld.qwest.net. [97.115.76.16]) by smtp.gmail.com with ESMTPSA id ci1-20020a056871c48100b001bb51450d85sm4160283oac.4.2023.08.21.06.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 06:10:46 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, kieran.bingham@ideasonboard.com, vedantparanjape160201@gmail.com, gabbymg94@gmail.com Date: Mon, 21 Aug 2023 06:10:39 -0700 Message-Id: <20230821131039.127370-6-gabbymg94@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230821131039.127370-1-gabbymg94@gmail.com> References: <20230821131039.127370-1-gabbymg94@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 5/5] libcamera: pipeline: uvcvideo: Handle metadata stream X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Gabby George via libcamera-devel From: Gabrielle George Reply-To: Gabby George Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Register the metadata stream's buffer ready callback and start processing metadata buffers. Use the timestamp from the metadata buffer as the corresponding video buffer Requests' timestamp. Metadata buffers are synchronized with frames coming into the video stream using the sequence field of the buffers. They may come in either order (video buffer first or metadata buffer first), so store relevant information about the buffer required to set the metadata timestamp or complete the buffer request as soon as possible. The timestamp will be improved upon in the next patch. For now, use the driver-provided metadata timestamp. Signed-off-by: Gabby George --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 168 ++++++++++++++++++- 1 file changed, 162 insertions(+), 6 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index c8d6633f..215435ec 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -12,6 +12,8 @@ #include #include +#include + #include #include @@ -34,6 +36,17 @@ namespace libcamera { LOG_DEFINE_CATEGORY(UVC) +/* + * The UVCH buffer contains an unsigned char array + * encoding UVC timing data that needs to be recast + * into usable data. + */ +struct UVCTimingBuf { + __u32 pts; + __u32 stc; + __u16 sofDevice; +} __attribute__((packed)); + class UVCCameraData : public Camera::Private { public: @@ -46,6 +59,8 @@ public: void addControl(uint32_t cid, const ControlInfo &v4l2info, ControlInfoMap::Map *ctrls); void bufferReady(FrameBuffer *buffer); + void bufferReadyMetadata(FrameBuffer *buffer); + void handleUnfinishedRequests(); const std::string &id() const { return id_; } @@ -56,9 +71,16 @@ public: std::map mappedMetadataBuffers_; std::map> formats_; + std::queue pendingVideoBuffers_; + std::queue> pendingMetadata_; private: int initMetadata(MediaDevice *media); + void completeRequest(FrameBuffer *buffer, uint64_t timestamp); + void endCorruptedStream(); + + const unsigned int frameStart_ = 1; + const unsigned int maxVidBuffersInQueue_ = 1; bool generateId(); @@ -242,7 +264,7 @@ int PipelineHandlerUVC::cleanupMetadataBuffers(Camera *camera) UVCCameraData *data = cameraData(camera); if (data->metadata_) - data->metadata_->releaseBuffers(); + ret = data->metadata_->releaseBuffers(); data->metadataBuffers_.clear(); data->mappedMetadataBuffers_.clear(); data->metadata_ = nullptr; @@ -253,7 +275,9 @@ int PipelineHandlerUVC::cleanupMetadataBuffers(Camera *camera) int PipelineHandlerUVC::cleanup(Camera *camera) { UVCCameraData *data = cameraData(camera); + cleanupMetadataBuffers(camera); + data->video_->releaseBuffers(); return 0; } @@ -354,6 +378,8 @@ void PipelineHandlerUVC::stopDevice(Camera *camera) if (data->metadata_) data->metadata_->streamOff(); + data->handleUnfinishedRequests(); + cleanup(camera); } @@ -646,8 +672,11 @@ int UVCCameraData::init(MediaDevice *media) if (ret) { metadata_ = nullptr; LOG(UVC, Error) << "Could not find a metadata video device."; + return 0; } + metadata_->bufferReady.connect(this, &UVCCameraData::bufferReadyMetadata); + return 0; } @@ -833,18 +862,145 @@ void UVCCameraData::addControl(uint32_t cid, const ControlInfo &v4l2Info, ctrls->emplace(id, info); } -void UVCCameraData::bufferReady(FrameBuffer *buffer) +void UVCCameraData::completeRequest(FrameBuffer *buffer, uint64_t timestamp) { Request *request = buffer->request(); - - /* \todo Use the UVC metadata to calculate a more precise timestamp */ - request->metadata().set(controls::SensorTimestamp, - buffer->metadata().timestamp); + request->metadata().set(controls::SensorTimestamp, timestamp); pipe()->completeBuffer(request, buffer); pipe()->completeRequest(request); } +void UVCCameraData::handleUnfinishedRequests() +{ + while (!pendingVideoBuffers_.empty()) { + FrameBuffer *oldBuffer = pendingVideoBuffers_.front(); + Request *oldRequest = oldBuffer->request(); + + oldRequest->metadata().set(controls::SensorTimestamp, + oldBuffer->metadata().timestamp); + + pipe()->completeBuffer(oldRequest, oldBuffer); + pipe()->completeRequest(oldRequest); + pendingVideoBuffers_.pop(); + } +} + +void UVCCameraData::endCorruptedStream() +{ + handleUnfinishedRequests(); + /* Close the metadata node so we don't get inaccurate timestamps*/ + metadata_ = nullptr; + LOG(UVC, Error) + << "UVC metadata stream corrupted. Reverting to driver timestamps."; +} + +/* + * If there is a metadata buffer that hasn't been matched with a + * video buffer, check to see if it matches this video buffer. + * + * If there is a match, use the timestamp stored in the metadata queue + * for this video buffer's request. Complete this video buffer + * and its request. + * + * If there are no metadata buffers available to check for a match, + * push this video buffer's request object to the queue. It may + * be that the metadata buffer has not yet arrived. + * When the matching metadata buffer does come in, it will handle + * completion of the buffer and request. + * + * If more than maxVidBuffersInQueue_ video buffers have been added + * to the queue, something is wrong with the metadata stream and + * we can no longer use UVC metadata packets for timestamps. + * Complete all of the outstanding requests and turn off metadata + * stream use. + */ +void UVCCameraData::bufferReady(FrameBuffer *buffer) +{ + /* \todo Use the UVC metadata to calculate a more precise timestamp */ + if (!metadata_ || buffer->metadata().sequence < frameStart_) { + completeRequest(buffer, buffer->metadata().timestamp); + return; + } + + if (!pendingMetadata_.empty()) { + /* A metadata buffer was ready first. */ + unsigned int mdSequence = std::get<0>(pendingMetadata_.front()) + frameStart_; + if (mdSequence == buffer->metadata().sequence) { + completeRequest(buffer, std::get<1>(pendingMetadata_.front())); + pendingMetadata_.pop(); + return; + } else { + /* \todo: Is there a reason metadata buffers can arrive out of order? */ + endCorruptedStream(); + return; + } + } + + pendingVideoBuffers_.push(buffer); + /* + * Deal with video buffers that haven't been completed, and with + * buffers whose metadata information arrived out of order. + */ + if (pendingVideoBuffers_.size() > maxVidBuffersInQueue_) { + endCorruptedStream(); + } +} + +void UVCCameraData::bufferReadyMetadata(FrameBuffer *buffer) +{ + if (!metadata_ || + buffer->metadata().status != FrameMetadata::Status::FrameSuccess) { + return; + } + + /* + * The metadata stream seems to start at seq 1 and libcamera + * sets the start sequence to 0, so it's necessary to add one + * to match this buffer with the correct video frame buffer. + * + * \todo: Is there a better way to do this? What is the root cause? + */ + unsigned int mdSequence = buffer->metadata().sequence + frameStart_; + int pos = buffer->cookie(); + + Span memMeta = mappedMetadataBuffers_.at(pos).planes()[0]; + uvc_meta_buf *metaBuf = reinterpret_cast(memMeta.data()); + + //Span memTime = mappedMetadataBuffers_.at(pos).planes()[0]; + //UVCTimingBuf * timeBuf = reinterpret_cast(&memTime.data()[sizeof(uvc_meta_buf)]); + + size_t UVCPayloadHeaderSize = sizeof(metaBuf->length) + + sizeof(metaBuf->flags) + sizeof(UVCTimingBuf); + if (metaBuf->length < UVCPayloadHeaderSize) { + endCorruptedStream(); + return; + } + + /* + * Match a pending video buffer with this buffer's sequence. If + * there is none available, put this timestamp information on the + * queue. When the matching video buffer does come in, it will use + * a timestamp from this metadata. + */ + if (!pendingVideoBuffers_.empty()) { + FrameBuffer *vidBuffer = pendingVideoBuffers_.front(); + unsigned int vidSequence = vidBuffer->metadata().sequence; + + if (vidSequence == mdSequence) { + completeRequest(vidBuffer, static_cast(metaBuf->ns)); + pendingVideoBuffers_.pop(); + } else { + endCorruptedStream(); + } + } else { + pendingMetadata_.push( + std::make_pair(buffer->metadata().sequence, + static_cast(metaBuf->ns))); + } + metadata_->queueBuffer(buffer); +} + REGISTER_PIPELINE_HANDLER(PipelineHandlerUVC) } /* namespace libcamera */