{"id":9177,"url":"https://patchwork.libcamera.org/api/1.1/patches/9177/?format=json","web_url":"https://patchwork.libcamera.org/patch/9177/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20200804095850.275499-13-naush@raspberrypi.com>","date":"2020-08-04T09:58:50","name":"[libcamera-devel,v6,12/12] pipeline: ipa: raspberrypi: Handle any externally allocated FrameBuffer","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"38ea696a38778524bde72f36491ffd680770c3c7","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/1.1/people/34/?format=json","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/9177/mbox/","series":[{"id":1187,"url":"https://patchwork.libcamera.org/api/1.1/series/1187/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1187","date":"2020-08-04T09:58:38","name":"Zero-copy RAW stream work","version":6,"mbox":"https://patchwork.libcamera.org/series/1187/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/9177/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/9177/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7956DBD86F\n\tfor ;\n\tTue, 4 Aug 2020 09:59:12 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 430EB6049B;\n\tTue, 4 Aug 2020 11:59:12 +0200 (CEST)","from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com\n\t[IPv6:2a00:1450:4864:20::42c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1005B60550\n\tfor ;\n\tTue, 4 Aug 2020 11:59:09 +0200 (CEST)","by mail-wr1-x42c.google.com with SMTP id f7so36827481wrw.1\n\tfor ;\n\tTue, 04 Aug 2020 02:59:09 -0700 (PDT)","from naushir-VirtualBox.patuck.local ([88.97.76.4])\n\tby smtp.gmail.com with ESMTPSA id\n\t32sm34047234wrh.18.2020.08.04.02.59.07\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tTue, 04 Aug 2020 02:59:07 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"aS/ATfxi\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=from:to:cc:subject:date:message-id:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=VrnZVRzAFcTlfpKwg6N0sIpXHNob2AVqtYJvsi6G/f4=;\n\tb=aS/ATfxiQDBlhTPdF9BqRoymrGLfHs3LwduRyuAy2pzxZQ29PPd7q9ReZwdMwncv1e\n\tYr+5tH/B+dH3Uxau/+jvQTTtmW2vdk1oa7sdTG17eChzlHzVLItp0h48bV/0zNyS2Ae6\n\tmWRyNZ77TWZlgAOSOTJeedSZD/EedBL2cgRt0WU+/vYpn7VRt37sVJSJfZm4F246zg2S\n\twfuNs0xUVCKAUsGr4IQENJ3i7LjU93xlTsJiDljGOB2EC8z93g3D1nQMxlP0YVeN4/4j\n\tnrGTLDXzDvJGl8P+L2gaCYw8CIwplenJLLWLzRMyOUqHGVd4batplq4NafEsxrPl9qfZ\n\tKA9w==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=VrnZVRzAFcTlfpKwg6N0sIpXHNob2AVqtYJvsi6G/f4=;\n\tb=TL1nqVZKiR80uAGzWq/EEH+OAw6Pl2rb8GZZltM1Y5s5a76jeDa0SGiMbOU31j3o7y\n\tGvBnQXB7L5T65CP4cPxQaR+6rWwWHILECtqPTmeLupPRIzrIJ1V82zyEHECT5jBWnrIW\n\tnaaNC4AjHFahylFg5nllFUEFilOVV52ouzf+N4qIzNbW0cjxC86A9J7DVmpILxTMp7H9\n\tfby9VDeCbxKxmGx5RUfF+h6eXH5b8uGnB7/7OohJ9kXvv4P8DT1VA8hKmgG/EJy+MXvt\n\t9hzx9v19AltkF39DC2oPjQO9j5sFQxaLAaNIHwie9Osi9AoBqwN3hN74ekxOe+qHcjQ1\n\twUnA==","X-Gm-Message-State":"AOAM533S9YO49Hz+TuBQQtTsoQAwIuBSmovCXrOOIJKBRARUUp4Jiv1B\n\tokkCokFSK2lgcdf94g4urAxIwpT3QMBGBA==","X-Google-Smtp-Source":"ABdhPJzt9EiSGL8oe6ZjQxDaB5G4zJ7FMr4/lK6xqmEA2+Let7409S1qLH/AqHlCu2S6Ax+ew4TRAw==","X-Received":"by 2002:a5d:4802:: with SMTP id\n\tl2mr17877581wrq.183.1596535148230; \n\tTue, 04 Aug 2020 02:59:08 -0700 (PDT)","From":"Naushir Patuck ","To":"libcamera-devel@lists.libcamera.org","Date":"Tue, 4 Aug 2020 10:58:50 +0100","Message-Id":"<20200804095850.275499-13-naush@raspberrypi.com>","X-Mailer":"git-send-email 2.25.1","In-Reply-To":"<20200804095850.275499-1-naush@raspberrypi.com>","References":"<20200804095850.275499-1-naush@raspberrypi.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v6 12/12] pipeline: ipa: raspberrypi:\n\tHandle any externally allocated FrameBuffer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Handle the case where a FrameBuffer that has been externally allocated\n(i.e. not through the v4l2 video device) is passed into a Request.\n\nWe must store the buffer pointer in the stream internal buffer list to\nidentify when used, as well as mmap the buffer in the IPA if needed.\n\nSigned-off-by: Naushir Patuck \n---\n include/libcamera/ipa/raspberrypi.h | 11 +--\n src/ipa/raspberrypi/raspberrypi.cpp | 6 +-\n .../pipeline/raspberrypi/raspberrypi.cpp | 82 ++++++++++++++-----\n .../pipeline/raspberrypi/rpi_stream.cpp | 30 ++++++-\n .../pipeline/raspberrypi/rpi_stream.h | 3 +\n 5 files changed, 103 insertions(+), 29 deletions(-)","diff":"diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h\nindex 262fc6f3..dd6ebeac 100644\n--- a/include/libcamera/ipa/raspberrypi.h\n+++ b/include/libcamera/ipa/raspberrypi.h\n@@ -28,11 +28,12 @@ enum RPiOperations {\n \tRPI_IPA_EVENT_QUEUE_REQUEST,\n };\n \n-enum RPiIpaMask {\n-\tID\t\t= 0x0ffff,\n-\tSTATS\t\t= 0x10000,\n-\tEMBEDDED_DATA\t= 0x20000,\n-\tBAYER_DATA\t= 0x40000\n+enum RPiBufferMask {\n+\tID\t\t= 0x00ffff,\n+\tSTATS\t\t= 0x010000,\n+\tEMBEDDED_DATA\t= 0x020000,\n+\tBAYER_DATA\t= 0x040000,\n+\tEXTERNAL_BUFFER\t= 0x100000,\n };\n \n /* Size of the LS grid allocation. */\ndiff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp\nindex cd49de80..93dfef27 100644\n--- a/src/ipa/raspberrypi/raspberrypi.cpp\n+++ b/src/ipa/raspberrypi/raspberrypi.cpp\n@@ -352,7 +352,7 @@ void IPARPi::processEvent(const IPAOperationData &event)\n \n \t\tIPAOperationData op;\n \t\top.operation = RPI_IPA_ACTION_STATS_METADATA_COMPLETE;\n-\t\top.data = { bufferId & RPiIpaMask::ID };\n+\t\top.data = { bufferId & RPiBufferMask::ID };\n \t\top.controls = { libcameraMetadata_ };\n \t\tqueueFrameAction.emit(0, op);\n \t\tbreak;\n@@ -373,7 +373,7 @@ void IPARPi::processEvent(const IPAOperationData &event)\n \t\t/* Ready to push the input buffer into the ISP. */\n \t\tIPAOperationData op;\n \t\top.operation = RPI_IPA_ACTION_RUN_ISP;\n-\t\top.data = { bayerbufferId & RPiIpaMask::ID };\n+\t\top.data = { bayerbufferId & RPiBufferMask::ID };\n \t\tqueueFrameAction.emit(0, op);\n \t\tbreak;\n \t}\n@@ -700,7 +700,7 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId)\n {\n \tIPAOperationData op;\n \top.operation = RPI_IPA_ACTION_EMBEDDED_COMPLETE;\n-\top.data = { bufferId & RPiIpaMask::ID };\n+\top.data = { bufferId & RPiBufferMask::ID };\n \tqueueFrameAction.emit(0, op);\n }\n \ndiff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\nindex b059c582..e3240914 100644\n--- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n+++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n@@ -151,6 +151,7 @@ public:\n \n \tvoid clearIncompleteRequests();\n \tvoid handleStreamBuffer(FrameBuffer *buffer, RPi::RPiStream *stream);\n+\tvoid handleExternalBuffer(FrameBuffer *buffer, RPi::RPiStream *stream);\n \tvoid handleState();\n \n \tCameraSensor *sensor_;\n@@ -726,23 +727,41 @@ int PipelineHandlerRPi::queueRequestDevice(Camera *camera, Request *request)\n \n \t/* Push all buffers supplied in the Request to the respective streams. */\n \tfor (auto stream : data->streams_) {\n-\t\tif (stream->isExternal()) {\n-\t\t\tFrameBuffer *buffer = request->findBuffer(stream);\n+\t\tif (!stream->isExternal())\n+\t\t\tcontinue;\n+\n+\t\tFrameBuffer *buffer = request->findBuffer(stream);\n+\t\tif (buffer && stream->getBufferId(buffer) == -1) {\n \t\t\t/*\n-\t\t\t * If no buffer is provided by the request for this stream, we\n-\t\t\t * queue a nullptr to the stream to signify that it must use an\n-\t\t\t * internally allocated buffer for this capture request. This\n-\t\t\t * buffer will not be given back to the application, but is used\n-\t\t\t * to support the internal pipeline flow.\n-\t\t\t *\n-\t\t\t * The below queueBuffer() call will do nothing if there are not\n-\t\t\t * enough internal buffers allocated, but this will be handled by\n-\t\t\t * queuing the request for buffers in the RPiStream object.\n+\t\t\t * This buffer is not recognised, so it must have been allocated\n+\t\t\t * outside the v4l2 device. Store it in the stream buffer list\n+\t\t\t * so we can track it.\n \t\t\t */\n-\t\t\tint ret = stream->queueBuffer(buffer);\n-\t\t\tif (ret)\n-\t\t\t\treturn ret;\n+\t\t\tstream->setExternalBuffer(buffer);\n+\n+\t\t\t/* Also get the IPA to mmap it if needed. */\n+\t\t\tif (stream == &data->unicam_[Unicam::Embedded] ||\n+\t\t\t stream == &data->isp_[Isp::Stats]) {\n+\t\t\t\tunsigned int id = stream->getBufferId(buffer);\n+\t\t\t\tunsigned int mask = stream == &data->unicam_[Unicam::Embedded] ?\n+\t\t\t\t\t\t RPiBufferMask::EMBEDDED_DATA : RPiBufferMask::STATS;\n+\t\t\t\tmapBuffers(camera, {{ id, buffer }}, mask);\n+\t\t\t}\n \t\t}\n+\t\t/*\n+\t\t * If no buffer is provided by the request for this stream, we\n+\t\t * queue a nullptr to the stream to signify that it must use an\n+\t\t * internally allocated buffer for this capture request. This\n+\t\t * buffer will not be given back to the application, but is used\n+\t\t * to support the internal pipeline flow.\n+\t\t *\n+\t\t * The below queueBuffer() call will do nothing if there are not\n+\t\t * enough internal buffers allocated, but this will be handled by\n+\t\t * queuing the request for buffers in the RPiStream object.\n+\t\t */\n+\t\tint ret = stream->queueBuffer(buffer);\n+\t\tif (ret)\n+\t\t\treturn ret;\n \t}\n \n \t/* Push the request to the back of the queue. */\n@@ -906,8 +925,8 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera)\n \t * Pass the stats and embedded data buffers to the IPA. No other\n \t * buffers need to be passed.\n \t */\n-\tmapBuffers(camera, data->isp_[Isp::Stats].getBuffers(), RPiIpaMask::STATS);\n-\tmapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), RPiIpaMask::EMBEDDED_DATA);\n+\tmapBuffers(camera, data->isp_[Isp::Stats].getBuffers(), RPiBufferMask::STATS);\n+\tmapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), RPiBufferMask::EMBEDDED_DATA);\n \n \treturn 0;\n }\n@@ -1216,7 +1235,7 @@ void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer)\n \tif (stream == &isp_[Isp::Stats]) {\n \t\tIPAOperationData op;\n \t\top.operation = RPI_IPA_EVENT_SIGNAL_STAT_READY;\n-\t\top.data = { RPiIpaMask::STATS | static_cast(index) };\n+\t\top.data = { RPiBufferMask::STATS | static_cast(index) };\n \t\tipa_->processEvent(op);\n \t} else {\n \t\t/* Any other ISP output can be handed back to the application now. */\n@@ -1290,6 +1309,12 @@ void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, RPi::RPiStream *stre\n \t\tRequest *request = requestQueue_.front();\n \n \t\tif (!dropFrameCount_ && request->findBuffer(stream) == buffer) {\n+\t\t\t/*\n+\t\t\t * Check if this is an externally provided buffer, and if\n+\t\t\t * so, we must get the IPA to unmap it and stop tracking\n+\t\t\t * it in the pipeline handler.\n+\t\t\t */\n+\t\t\thandleExternalBuffer(buffer, stream);\n \t\t\t/*\n \t\t\t * Tag the buffer as completed, returning it to the\n \t\t\t * application.\n@@ -1308,6 +1333,25 @@ void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, RPi::RPiStream *stre\n \t}\n }\n \n+void RPiCameraData::handleExternalBuffer(FrameBuffer *buffer, RPi::RPiStream *stream)\n+{\n+\tunsigned int id = stream->getBufferId(buffer);\n+\n+\tif (!(id & RPiBufferMask::EXTERNAL_BUFFER))\n+\t\treturn;\n+\n+\tif (stream == &unicam_[Unicam::Embedded] || stream == &isp_[Isp::Stats]) {\n+\t\tunsigned int mask = stream == &unicam_[Unicam::Embedded] ?\n+\t\t\t\t RPiBufferMask::EMBEDDED_DATA : RPiBufferMask::STATS;\n+\t\t/* Ask the IPA to unmap the buffer. */\n+\t\tipa_->unmapBuffers({ mask | id });\n+\t\tipaBuffers_.erase(mask | id);\n+\t}\n+\n+\t/* Stop the Stream object from tracking the buffer. */\n+\tstream->removeExternalBuffer(buffer);\n+}\n+\n void RPiCameraData::handleState()\n {\n \tswitch (state_) {\n@@ -1440,8 +1484,8 @@ void RPiCameraData::tryRunPipeline()\n \t\t\t<< \" Embedded buffer id: \" << embeddedIndex;\n \n \top.operation = RPI_IPA_EVENT_SIGNAL_ISP_PREPARE;\n-\top.data = { RPiIpaMask::EMBEDDED_DATA | embeddedIndex,\n-\t\t RPiIpaMask::BAYER_DATA | bayerIndex };\n+\top.data = { RPiBufferMask::EMBEDDED_DATA | embeddedIndex,\n+\t\t RPiBufferMask::BAYER_DATA | bayerIndex };\n \tipa_->processEvent(op);\n }\n \ndiff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp\nindex 879e25ba..c09f14c9 100644\n--- a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp\n+++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp\n@@ -6,6 +6,8 @@\n */\n #include \"rpi_stream.h\"\n \n+#include \n+\n #include \"libcamera/internal/log.h\"\n \n namespace libcamera {\n@@ -44,8 +46,13 @@ bool RPiStream::isExternal() const\n \n void RPiStream::setExportedBuffers(std::vector> *buffers)\n {\n-\tfor (auto const &buffer : *buffers)\n-\t\tbufferMap_.emplace(id_++, buffer.get());\n+\t/* Ensure we are using a sensible number of buffers. */\n+\tASSERT(id_ < RPiBufferMask::ID);\n+\n+\tfor (auto const &buffer : *buffers) {\n+\t\tbufferMap_.emplace(id_ & RPiBufferMask::ID, buffer.get());\n+\t\tid_++;\n+\t}\n }\n \n const BufferMap &RPiStream::getBuffers() const\n@@ -68,6 +75,25 @@ int RPiStream::getBufferId(FrameBuffer *buffer) const\n \treturn it->first;\n }\n \n+void RPiStream::setExternalBuffer(FrameBuffer *buffer)\n+{\n+\t/* Ensure we are using a sensible number of buffers. */\n+\tASSERT(id_ < RPiBufferMask::ID);\n+\n+\tbufferMap_.emplace(RPiBufferMask::EXTERNAL_BUFFER | (id_ & RPiBufferMask::ID),\n+\t\t\t buffer);\n+\tid_++;\n+}\n+\n+void RPiStream::removeExternalBuffer(FrameBuffer *buffer)\n+{\n+\tint id = getBufferId(buffer);\n+\n+\t/* Ensure we have this buffer in the stream, and it is marked external. */\n+\tASSERT(id != -1 && (id & RPiBufferMask::EXTERNAL_BUFFER));\n+\tbufferMap_.erase(id);\n+}\n+\n int RPiStream::prepareBuffers(unsigned int count)\n {\n \tint ret;\ndiff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.h b/src/libcamera/pipeline/raspberrypi/rpi_stream.h\nindex ed517c22..8b23c4b2 100644\n--- a/src/libcamera/pipeline/raspberrypi/rpi_stream.h\n+++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.h\n@@ -51,6 +51,9 @@ public:\n \tconst BufferMap &getBuffers() const;\n \tint getBufferId(FrameBuffer *buffer) const;\n \n+\tvoid setExternalBuffer(FrameBuffer *buffer);\n+\tvoid removeExternalBuffer(FrameBuffer *buffer);\n+\n \tint prepareBuffers(unsigned int count);\n \tint queueBuffer(FrameBuffer *buffer);\n \tvoid returnBuffer(FrameBuffer *buffer);\n","prefixes":["libcamera-devel","v6","12/12"]}