{"id":11339,"url":"https://patchwork.libcamera.org/api/patches/11339/?format=json","web_url":"https://patchwork.libcamera.org/patch/11339/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210218170126.2060783-4-naush@raspberrypi.com>","date":"2021-02-18T17:01:25","name":"[libcamera-devel,v3,3/4] pipeline: raspberrypi: Only enabled embedded stream when available","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"ae3835d9dc0daaa087627d0fb3b4f9c5f9b2f7a4","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/?format=json","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/11339/mbox/","series":[{"id":1708,"url":"https://patchwork.libcamera.org/api/series/1708/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1708","date":"2021-02-18T17:01:22","name":"Raspberry Pi: Embedded data usage","version":3,"mbox":"https://patchwork.libcamera.org/series/1708/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/11339/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/11339/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 124E5BD1F1\n\tfor ;\n\tThu, 18 Feb 2021 17:01:37 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D327E689C7;\n\tThu, 18 Feb 2021 18:01:36 +0100 (CET)","from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com\n\t[IPv6:2a00:1450:4864:20::32c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id F245668370\n\tfor ;\n\tThu, 18 Feb 2021 18:01:33 +0100 (CET)","by mail-wm1-x32c.google.com with SMTP id x16so4382783wmk.3\n\tfor ;\n\tThu, 18 Feb 2021 09:01:33 -0800 (PST)","from naush-laptop.patuck.local ([88.97.76.4])\n\tby smtp.gmail.com with ESMTPSA id\n\tt30sm10504747wra.56.2021.02.18.09.01.32\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tThu, 18 Feb 2021 09:01:32 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"LeTSqS8k\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=from:to:cc:subject:date:message-id:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=G9aibhqp+Si9LQKERAmiI660bAvlasfj1ec+UDjUjW4=;\n\tb=LeTSqS8kU4F3+3Mb9gAmJLR7RkdxbFJAi5+sXxT4/H+j0E16HnPIDXZiVbGzwrgtAT\n\tTJeA3lEimASbg7vT50LuiYNiElevyVmOJR9RWwCH61s8GBdC1+a2AfqGe8P8TkFxinhj\n\td9caHczfvHUWrKgZxR/A13nLLWy/sOqw47feJLREkJFTvuL1TOnLO6HNJm4HqhkN0YAM\n\t0OTlv6fHWhfGe2kNow7oPhyHoLLmL8K9rYNAGmLM823TAT5rg+8DoJkFtyrXLnb8iYbt\n\t0WPWvZWRJJdnC2hNByseI772tLBBImOvtb3D+w2NTsJziOXyQz2Abi0GpoB1Op5Vybt3\n\tAF8g==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=G9aibhqp+Si9LQKERAmiI660bAvlasfj1ec+UDjUjW4=;\n\tb=r44efpNQ+lpEW3MwxasiI+n9LHvBd55MYYhvhhBtv9SBVWoIhBkokwb03aqj/3lckm\n\tSSKZBwgKeqomBBpyeToJUSiUhj3J6os7mu53n/MEFRzccBpAtgORHhjfn13bu4mBtQIu\n\tscJX+KLit0ONmO5gR/S8uv2zywFlLxHIxqA/PsGksNVWZuDXHVp2Gh5qB1jnRK9/gNWi\n\trTIN5ryWfRDCRy8Xb684frepTCwro6JWKho5nKEivIMC6U9ryMdcQi4WrNW0ZqAFjYOv\n\tTOGzqEGQ83o/GE5TR9lfmb2pOKhGDE0pbuvyP5JQABh6xThC7HN1Ekg2xlDy7+eJJcDa\n\tL4Uw==","X-Gm-Message-State":"AOAM531NKWgKvCzD5TdnaZRGKaLV24c4xZEODHq2kZkPeWoiYb7SSD0G\n\tSCjk7WMKNnvnoinascw42co8VLzs4PX5lkUq","X-Google-Smtp-Source":"ABdhPJz7EGhjT95KridSbxiuLTd5pwsUladEE/BpfQIKnSJyuqHX+V+JvoGMMIyLKMxW2QOdDaXVkg==","X-Received":"by 2002:a1c:4006:: with SMTP id n6mr4462163wma.177.1613667693261;\n\tThu, 18 Feb 2021 09:01:33 -0800 (PST)","From":"Naushir Patuck ","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 18 Feb 2021 17:01:25 +0000","Message-Id":"<20210218170126.2060783-4-naush@raspberrypi.com>","X-Mailer":"git-send-email 2.25.1","In-Reply-To":"<20210218170126.2060783-1-naush@raspberrypi.com>","References":"<20210218170126.2060783-1-naush@raspberrypi.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v3 3/4] pipeline: raspberrypi: Only\n\tenabled embedded stream when available","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"The pipeline handler would enable and use the Unicam embedded data stream\neven if the sensor did not support it. This was to allow a means to\npass exposure and gain values for the frame to the IPA in a synchronised\nway.\n\nThe recent changes to get the pipeline handler to pass a ControlList\nwith exposure and gain values means this is no longer required. Disable\nthe use of the embedded data stream when a sensor does not support it.\n\nThis change also removes the mappedEmbeddedBuffers_ map as it is no\nlonger used.\n\nSigned-off-by: Naushir Patuck \n---\n .../pipeline/raspberrypi/raspberrypi.cpp | 112 +++++++++++-------\n 1 file changed, 70 insertions(+), 42 deletions(-)","diff":"diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\nindex b43d86166c63..d969c77993eb 100644\n--- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n+++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n@@ -177,12 +177,6 @@ public:\n \t/* Stores the ids of the buffers mapped in the IPA. */\n \tstd::unordered_set ipaBuffers_;\n \n-\t/*\n-\t * Map of (internal) mmaped embedded data buffers, to avoid having to\n-\t * map/unmap on every frame.\n-\t */\n-\tstd::map mappedEmbeddedBuffers_;\n-\n \t/* DMAHEAP allocation helper. */\n \tRPi::DmaHeap dmaHeap_;\n \tFileDescriptor lsTable_;\n@@ -636,14 +630,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n \n \t\tif (isRaw(cfg.pixelFormat)) {\n \t\t\tcfg.setStream(&data->unicam_[Unicam::Image]);\n-\t\t\t/*\n-\t\t\t * We must set both Unicam streams as external, even\n-\t\t\t * though the application may only request RAW frames.\n-\t\t\t * This is because we match timestamps on both streams\n-\t\t\t * to synchronise buffers.\n-\t\t\t */\n \t\t\tdata->unicam_[Unicam::Image].setExternal(true);\n-\t\t\tdata->unicam_[Unicam::Embedded].setExternal(true);\n \t\t\tcontinue;\n \t\t}\n \n@@ -715,17 +702,6 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n \t\treturn ret;\n \t}\n \n-\t/* Unicam embedded data output format. */\n-\tformat = {};\n-\tformat.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA);\n-\tLOG(RPI, Debug) << \"Setting embedded data format.\";\n-\tret = data->unicam_[Unicam::Embedded].dev()->setFormat(&format);\n-\tif (ret) {\n-\t\tLOG(RPI, Error) << \"Failed to set format on Unicam embedded: \"\n-\t\t\t\t<< format.toString();\n-\t\treturn ret;\n-\t}\n-\n \t/* Figure out the smallest selection the ISP will allow. */\n \tRectangle testCrop(0, 0, 1, 1);\n \tdata->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop);\n@@ -742,6 +718,41 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n \tif (ret)\n \t\tLOG(RPI, Error) << \"Failed to configure the IPA: \" << ret;\n \n+\t/*\n+\t * The IPA will set data->sensorMetadata_ to true if embedded data is\n+\t * supported on this sensor. If so, open the Unicam embedded data\n+\t * node and configure the output format.\n+\t */\n+\tif (data->sensorMetadata_) {\n+\t\tformat = {};\n+\t\tformat.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA);\n+\t\tLOG(RPI, Debug) << \"Setting embedded data format.\";\n+\t\tdata->unicam_[Unicam::Embedded].dev()->open();\n+\t\tret = data->unicam_[Unicam::Embedded].dev()->setFormat(&format);\n+\t\tif (ret) {\n+\t\t\tLOG(RPI, Error) << \"Failed to set format on Unicam embedded: \"\n+\t\t\t\t\t<< format.toString();\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If a RAW/Bayer stream has been requested by the application,\n+\t\t * we must set both Unicam streams as external, even though the\n+\t\t * application may only request RAW frames. This is because we\n+\t\t * match timestamps on both streams to synchronise buffers.\n+\t\t */\n+\t\tif (rawStream)\n+\t\t\tdata->unicam_[Unicam::Embedded].setExternal(true);\n+\t} else {\n+\t\t/*\n+\t\t * No embedded data present, so we do not want to iterate over\n+\t\t * the embedded data stream when starting and stopping.\n+\t\t */\n+\t\tdata->streams_.erase(std::remove(data->streams_.begin(), data->streams_.end(),\n+\t\t\t\t\t\t &data->unicam_[Unicam::Embedded]),\n+\t\t\t\t data->streams_.end());\n+\t}\n+\n \t/*\n \t * Update the ScalerCropMaximum to the correct value for this camera mode.\n \t * For us, it's the same as the \"analogue crop\".\n@@ -949,10 +960,16 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator)\n \tfor (auto &stream : data->isp_)\n \t\tdata->streams_.push_back(&stream);\n \n-\t/* Open all Unicam and ISP streams. */\n+\t/*\n+\t * Open all Unicam and ISP streams. The exception is the embedded data\n+\t * stream, which only gets opened if the IPA reports that the sensor\n+\t * supports embedded data. This happens in RPiCameraData::configureIPA().\n+\t */\n \tfor (auto const stream : data->streams_) {\n-\t\tif (stream->dev()->open())\n-\t\t\treturn false;\n+\t\tif (stream != &data->unicam_[Unicam::Embedded]) {\n+\t\t\tif (stream->dev()->open())\n+\t\t\t\treturn false;\n+\t\t}\n \t}\n \n \t/* Wire up all the buffer connections. */\n@@ -1109,19 +1126,13 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera)\n \t\t\treturn ret;\n \t}\n \n-\tif (!data->sensorMetadata_) {\n-\t\tfor (auto const &it : data->unicam_[Unicam::Embedded].getBuffers()) {\n-\t\t\tMappedFrameBuffer fb(it.second, PROT_READ | PROT_WRITE);\n-\t\t\tdata->mappedEmbeddedBuffers_.emplace(it.first, std::move(fb));\n-\t\t}\n-\t}\n-\n \t/*\n \t * Pass the stats and embedded data buffers to the IPA. No other\n \t * buffers need to be passed.\n \t */\n \tmapBuffers(camera, data->isp_[Isp::Stats].getBuffers(), ipa::RPi::MaskStats);\n-\tmapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), ipa::RPi::MaskEmbeddedData);\n+\tif (data->sensorMetadata_)\n+\t\tmapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), ipa::RPi::MaskEmbeddedData);\n \n \treturn 0;\n }\n@@ -1154,7 +1165,6 @@ void PipelineHandlerRPi::freeBuffers(Camera *camera)\n \tstd::vector ipaBuffers(data->ipaBuffers_.begin(), data->ipaBuffers_.end());\n \tdata->ipa_->unmapBuffers(ipaBuffers);\n \tdata->ipaBuffers_.clear();\n-\tdata->mappedEmbeddedBuffers_.clear();\n \n \tfor (auto const stream : data->streams_)\n \t\tstream->releaseBuffers();\n@@ -1652,7 +1662,7 @@ void RPiCameraData::tryRunPipeline()\n \n \t/* If any of our request or buffer queues are empty, we cannot proceed. */\n \tif (state_ != State::Idle || requestQueue_.empty() ||\n-\t bayerQueue_.empty() || embeddedQueue_.empty())\n+\t bayerQueue_.empty() || (embeddedQueue_.empty() && sensorMetadata_))\n \t\treturn;\n \n \tif (!findMatchingBuffers(bayerFrame, embeddedBuffer))\n@@ -1675,17 +1685,24 @@ void RPiCameraData::tryRunPipeline()\n \tstate_ = State::Busy;\n \n \tunsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerFrame.buffer);\n-\tunsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer);\n \n \tLOG(RPI, Debug) << \"Signalling signalIspPrepare:\"\n-\t\t\t<< \" Bayer buffer id: \" << bayerId\n-\t\t\t<< \" Embedded buffer id: \" << embeddedId;\n+\t\t\t<< \" Bayer buffer id: \" << bayerId;\n \n \tipa::RPi::ISPConfig ispPrepare;\n-\tispPrepare.embeddedBufferId = ipa::RPi::MaskEmbeddedData | embeddedId;\n \tispPrepare.bayerBufferId = ipa::RPi::MaskBayerData | bayerId;\n-\tispPrepare.embeddedBufferPresent = true;\n \tispPrepare.controls = std::move(bayerFrame.controls);\n+\n+\tif (embeddedBuffer) {\n+\t\tunsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer);\n+\n+\t\tispPrepare.embeddedBufferId = ipa::RPi::MaskEmbeddedData | embeddedId;\n+\t\tispPrepare.embeddedBufferPresent = true;\n+\n+\t\tLOG(RPI, Debug) << \"Signalling signalIspPrepare:\"\n+\t\t\t\t<< \" Bayer buffer id: \" << embeddedId;\n+\t}\n+\n \tipa_->signalIspPrepare(ispPrepare);\n }\n \n@@ -1727,6 +1744,17 @@ bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&em\n \n \t\t\tLOG(RPI, Debug) << \"Could not find matching embedded buffer\";\n \n+\t\t\tif (!sensorMetadata_) {\n+\t\t\t\t/*\n+\t\t\t\t * If there is no sensor metadata, simply return the\n+\t\t\t\t * first bayer frame in the queue.\n+\t\t\t\t */\n+\t\t\t\tLOG(RPI, Debug) << \"Returning bayer frame without a match\";\n+\t\t\t\tbayerQueue_.pop();\n+\t\t\t\tembeddedBuffer = nullptr;\n+\t\t\t\treturn true;\n+\t\t\t}\n+\n \t\t\tif (!embeddedQueue_.empty()) {\n \t\t\t\t/*\n \t\t\t\t * Not found a matching embedded buffer for the bayer buffer in\n","prefixes":["libcamera-devel","v3","3/4"]}