{"id":8844,"url":"https://patchwork.libcamera.org/api/1.1/patches/8844/?format=json","web_url":"https://patchwork.libcamera.org/patch/8844/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20200717085410.732308-7-naush@raspberrypi.com>","date":"2020-07-17T08:54:06","name":"[libcamera-devel,v3,06/10] libcamera: pipeline: raspberrypi: Rework stream buffer logic for zero-copy","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"321d64f7d7ebaf68a7b79ad4e31baf69af0678ed","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/1.1/people/34/?format=json","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/8844/mbox/","series":[{"id":1114,"url":"https://patchwork.libcamera.org/api/1.1/series/1114/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1114","date":"2020-07-17T08:54:00","name":"Zero-copy RAW stream work","version":3,"mbox":"https://patchwork.libcamera.org/series/1114/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/8844/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/8844/checks/","tags":{},"headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 6B381C0109\n\tfor <parsemail@patchwork.libcamera.org>;\n\tFri, 17 Jul 2020 08:54:25 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1D80160539;\n\tFri, 17 Jul 2020 10:54:25 +0200 (CEST)","from mail-wm1-x335.google.com (mail-wm1-x335.google.com\n\t[IPv6:2a00:1450:4864:20::335])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id F388D6053B\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 17 Jul 2020 10:54:21 +0200 (CEST)","by mail-wm1-x335.google.com with SMTP id 22so13861359wmg.1\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 17 Jul 2020 01:54:21 -0700 (PDT)","from naushir-VirtualBox.patuck.local ([88.97.76.4])\n\tby smtp.gmail.com with ESMTPSA id\n\t92sm14582375wrr.96.2020.07.17.01.54.19\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 17 Jul 2020 01:54:19 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"OgwScW5m\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=from:to:cc:subject:date:message-id:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=ByNhL3Xa4AJmPZURHeKtXCROTfQCLsH8plZUc/ktua0=;\n\tb=OgwScW5m34LLAmlBQTG3z1rdPaFb6+6ZMwpmtGhYVvPRnxS7g1ev5hwyOHpPgOhPWG\n\t8vo5EcOxusEjq6ChbjC6Dnb3tjkfZt9HszKSLnlHyhUTQEL6qotyBcNSxG5EDrC+nPzl\n\txnYfrqfR1YxILbNVBvhDOmzMET02WhuXKhYRTUGpgyjhfCjYeUu2Krl90LrpmSvN/9i7\n\tkNjLabAs4AuOiJF1J8a2aq0YbIH1ONNdHb9TcNr2d0s3peEhnwGnfiQ8vBjEQHKXF5LB\n\t3EkB7kE60WnfTlnCO1o7BB+tPEhlXelulDiuxzLxknnqQ2dx1VtUsBuF7VVHueaxxOyi\n\tHYHQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=ByNhL3Xa4AJmPZURHeKtXCROTfQCLsH8plZUc/ktua0=;\n\tb=Crb1mpKtQ6Fq9cOcO4FcoFm0GSSO6t4lnaNMvTOs7SQJdpjSVAFGJYOv6MfSVDDb15\n\t7K0iktwSzPaaHQvJeIJbs7gOVcqhN7dcANl6QY5TwcMlKHKH6CleK6G8UZYaEM7sX1oB\n\twxFTN7Owj6e9gmTcXa81OC2bMDGp1uYOY9yACc6gEQ7qZR9SkYoSVUWlRJoT6SQ72N4Q\n\te1UNk2y7WYt948sBGB7HDyt2zhjW4bm2e/S7B4uzQnz94TQDgKpWMzgEWslbnd4ZEBMj\n\tztwxtsRMxLtgTsw+tQRDas5d33umcm7hY4R2XyBWxrhW8lITOKZ8odsibqVFfINzrtbv\n\t4/qA==","X-Gm-Message-State":"AOAM532nwaOC8BpXN64bGven+yCaVTgkv9wE8IaaBfynZTLmxyf9NgFJ\n\t0IEfXbJIuECy35jsQI3MWs/kFBuL054=","X-Google-Smtp-Source":"ABdhPJyRxtEwBzE65aAcSlncSeNGPGsiH0ydUbiKYRnOERzAJlCRmBft/MOar744SL2sHMpF1pct0A==","X-Received":"by 2002:a1c:d8:: with SMTP id 207mr7949923wma.81.1594976060656; \n\tFri, 17 Jul 2020 01:54:20 -0700 (PDT)","From":"Naushir Patuck <naush@raspberrypi.com>","To":"libcamera-devel@lists.libcamera.org","Date":"Fri, 17 Jul 2020 09:54:06 +0100","Message-Id":"<20200717085410.732308-7-naush@raspberrypi.com>","X-Mailer":"git-send-email 2.25.1","In-Reply-To":"<20200717085410.732308-1-naush@raspberrypi.com>","References":"<20200717085410.732308-1-naush@raspberrypi.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v3 06/10] libcamera: pipeline:\n\traspberrypi: Rework stream buffer logic for zero-copy","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"},"content":"Stop using v4l2_videodevice::allocateBuffer() for internal buffers and\ninstead export/import all buffers. This allows the pipeline to return\nany stream buffer requested by the application as zero-copy.\n\nAdvertise the Unicam Image stream as the RAW capture stream now.\n\nThe RPiStream object now maintains a new list of buffers that are\navailable to queue into a device. This is needed to distinguish between\nFrameBuffers allocated for internal use vs externally provided buffers.\nWhen a Request comes in, if a buffer is not provided for an exported\nstream, we re-use a buffer from this list.\n\nSigned-off-by: Naushir Patuck <naush@raspberrypi.com>\n---\n .../pipeline/raspberrypi/raspberrypi.cpp      | 228 ++++++++++--------\n .../pipeline/raspberrypi/rpi_stream.cpp       | 122 +++++++---\n .../pipeline/raspberrypi/rpi_stream.h         |  30 ++-\n 3 files changed, 226 insertions(+), 154 deletions(-)","diff":"diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\nindex ca8434a3..f63bf497 100644\n--- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n+++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n@@ -199,7 +199,8 @@ private:\n \tvoid checkRequestCompleted();\n \tvoid tryRunPipeline();\n \tvoid tryFlushQueues();\n-\tFrameBuffer *updateQueue(std::queue<FrameBuffer *> &q, uint64_t timestamp, V4L2VideoDevice *dev);\n+\tFrameBuffer *updateQueue(std::queue<FrameBuffer *> &q, uint64_t timestamp,\n+\t\t\t\t RPi::RPiStream *stream);\n \n \tunsigned int ispOutputCount_;\n };\n@@ -520,8 +521,15 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n \t\tStreamConfiguration &cfg = config->at(i);\n \n \t\tif (isRaw(cfg.pixelFormat)) {\n-\t\t\tcfg.setStream(&data->isp_[Isp::Input]);\n-\t\t\tdata->isp_[Isp::Input].setExternal(true);\n+\t\t\tcfg.setStream(&data->unicam_[Unicam::Image]);\n+\t\t\t/*\n+\t\t\t * We must set both Unicam streams as external, even\n+\t\t\t * though the application may only request RAW frames.\n+\t\t\t * This is because we match timestamps on both streams\n+\t\t\t * to synchronise buffers.\n+\t\t\t */\n+\t\t\tdata->unicam_[Unicam::Image].setExternal(true);\n+\t\t\tdata->unicam_[Unicam::Embedded].setExternal(true);\n \t\t\tcontinue;\n \t\t}\n \n@@ -624,7 +632,7 @@ int PipelineHandlerRPi::exportFrameBuffers(Camera *camera, Stream *stream,\n \tunsigned int count = stream->configuration().bufferCount;\n \tint ret = s->dev()->exportBuffers(count, buffers);\n \n-\ts->setExternalBuffers(buffers);\n+\ts->setExportedBuffers(buffers);\n \n \treturn ret;\n }\n@@ -724,14 +732,23 @@ int PipelineHandlerRPi::queueRequestDevice(Camera *camera, Request *request)\n \tif (data->state_ == RPiCameraData::State::Stopped)\n \t\treturn -EINVAL;\n \n-\t/* Ensure all external streams have associated buffers! */\n-\tfor (auto &stream : data->isp_) {\n-\t\tif (!stream.isExternal())\n-\t\t\tcontinue;\n+\tLOG(RPI, Debug) << \"queueRequestDevice: New request.\";\n \n-\t\tif (!request->findBuffer(&stream)) {\n-\t\t\tLOG(RPI, Error) << \"Attempt to queue request with invalid stream.\";\n-\t\t\treturn -ENOENT;\n+\t/* Push all buffers supplied in the Request to the respective streams. */\n+\tfor (auto stream : data->streams_) {\n+\t\tif (stream->isExternal()) {\n+\t\t\tFrameBuffer *buffer = request->findBuffer(stream);\n+\t\t\t/*\n+\t\t\t * A nullptr in buffer means that we should queue an internally\n+\t\t\t * allocated buffer.\n+\t\t\t *\n+\t\t\t * The below queueBuffer() call will do nothing if there are not\n+\t\t\t * enough internal buffers allocated, but this will be handled by\n+\t\t\t * queuing the request for buffers in the RPiStream object.\n+\t\t\t */\n+\t\t\tint ret = stream->queueBuffer(buffer);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n \t\t}\n \t}\n \n@@ -820,12 +837,10 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator)\n \t/* Initialize the camera properties. */\n \tdata->properties_ = data->sensor_->properties();\n \n-\t/*\n-\t * List the available output streams.\n-\t * Currently cannot do Unicam streams!\n-\t */\n+\t/*List the available streams an application may request. */\n \tstd::set<Stream *> streams;\n-\tstreams.insert(&data->isp_[Isp::Input]);\n+\tstreams.insert(&data->unicam_[Unicam::Image]);\n+\tstreams.insert(&data->unicam_[Unicam::Embedded]);\n \tstreams.insert(&data->isp_[Isp::Output0]);\n \tstreams.insert(&data->isp_[Isp::Output1]);\n \tstreams.insert(&data->isp_[Isp::Stats]);\n@@ -843,9 +858,28 @@ int PipelineHandlerRPi::queueAllBuffers(Camera *camera)\n \tint ret;\n \n \tfor (auto const stream : data->streams_) {\n-\t\tret = stream->queueBuffers();\n-\t\tif (ret < 0)\n-\t\t\treturn ret;\n+\t\tif (!stream->isExternal()) {\n+\t\t\tret = stream->queueAllBuffers();\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * For external streams, we must queue up a set of internal\n+\t\t\t * buffers to handle the number of drop frames requested by\n+\t\t\t * the IPA. This is done by passing nullptr in queueBuffer().\n+\t\t\t *\n+\t\t\t * The below queueBuffer() call will do nothing if there\n+\t\t\t * are not enough internal buffers allocated, but this will\n+\t\t\t * be handled by queuing the request for buffers in the\n+\t\t\t * RPiStream object.\n+\t\t\t */\n+\t\t\tunsigned int i;\n+\t\t\tfor (i = 0; i < data->dropFrameCount_; i++) {\n+\t\t\t\tint ret = stream->queueBuffer(nullptr);\n+\t\t\t\tif (ret)\n+\t\t\t\t\treturn ret;\n+\t\t\t}\n+\t\t}\n \t}\n \n \treturn 0;\n@@ -859,7 +893,8 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera)\n \t/*\n \t * Decide how many internal buffers to allocate. For now, simply look\n \t * at how many external buffers will be provided. Will need to improve\n-\t * this logic.\n+\t * this logic. However, we really must have all stream allocate the same\n+\t * number of buffers to simplify error handling in queueRequestDevice().\n \t */\n \tunsigned int maxBuffers = 0;\n \tfor (const Stream *s : camera->streams())\n@@ -867,33 +902,9 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera)\n \t\t\tmaxBuffers = std::max(maxBuffers, s->configuration().bufferCount);\n \n \tfor (auto const stream : data->streams_) {\n-\t\tif (stream->isExternal() || stream->isImporter()) {\n-\t\t\t/*\n-\t\t\t * If a stream is marked as external reserve memory to\n-\t\t\t * prepare to import as many buffers are requested in\n-\t\t\t * the stream configuration.\n-\t\t\t *\n-\t\t\t * If a stream is an internal stream with importer\n-\t\t\t * role, reserve as many buffers as possible.\n-\t\t\t */\n-\t\t\tunsigned int count = stream->isExternal()\n-\t\t\t\t\t\t     ? stream->configuration().bufferCount\n-\t\t\t\t\t\t     : maxBuffers;\n-\t\t\tret = stream->importBuffers(count);\n-\t\t\tif (ret < 0)\n-\t\t\t\treturn ret;\n-\t\t} else {\n-\t\t\t/*\n-\t\t\t * If the stream is an internal exporter allocate and\n-\t\t\t * export as many buffers as possible to its internal\n-\t\t\t * pool.\n-\t\t\t */\n-\t\t\tret = stream->allocateBuffers(maxBuffers);\n-\t\t\tif (ret < 0) {\n-\t\t\t\tfreeBuffers(camera);\n-\t\t\t\treturn ret;\n-\t\t\t}\n-\t\t}\n+\t\tret = stream->prepareBuffers(maxBuffers);\n+\t\tif (ret < 0)\n+\t\t\treturn ret;\n \t}\n \n \t/*\n@@ -901,7 +912,7 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera)\n \t * the IPA and RPI_IPA_EVENT_SIGNAL_ISP_PREPARE event.\n \t */\n \tcount = 0;\n-\tfor (auto const &b : *data->unicam_[Unicam::Image].getBuffers()) {\n+\tfor (auto const &b : data->unicam_[Unicam::Image].getBuffers()) {\n \t\tb->setCookie(count++);\n \t}\n \n@@ -910,14 +921,14 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera)\n \t * the IPA.\n \t */\n \tcount = 0;\n-\tfor (auto const &b : *data->isp_[Isp::Stats].getBuffers()) {\n+\tfor (auto const &b : data->isp_[Isp::Stats].getBuffers()) {\n \t\tb->setCookie(count++);\n \t\tdata->ipaBuffers_.push_back({ .id = RPiIpaMask::STATS | b->cookie(),\n \t\t\t\t\t      .planes = b->planes() });\n \t}\n \n \tcount = 0;\n-\tfor (auto const &b : *data->unicam_[Unicam::Embedded].getBuffers()) {\n+\tfor (auto const &b : data->unicam_[Unicam::Embedded].getBuffers()) {\n \t\tb->setCookie(count++);\n \t\tdata->ipaBuffers_.push_back({ .id = RPiIpaMask::EMBEDDED_DATA | b->cookie(),\n \t\t\t\t\t      .planes = b->planes() });\n@@ -1089,7 +1100,7 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData\n \tswitch (action.operation) {\n \tcase RPI_IPA_ACTION_STATS_METADATA_COMPLETE: {\n \t\tunsigned int bufferId = action.data[0];\n-\t\tFrameBuffer *buffer = isp_[Isp::Stats].getBuffers()->at(bufferId).get();\n+\t\tFrameBuffer *buffer = isp_[Isp::Stats].getBuffers().at(bufferId);\n \n \t\thandleStreamBuffer(buffer, &isp_[Isp::Stats]);\n \t\t/* Fill the Request metadata buffer with what the IPA has provided */\n@@ -1100,19 +1111,19 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData\n \n \tcase RPI_IPA_ACTION_EMBEDDED_COMPLETE: {\n \t\tunsigned int bufferId = action.data[0];\n-\t\tFrameBuffer *buffer = unicam_[Unicam::Embedded].getBuffers()->at(bufferId).get();\n+\t\tFrameBuffer *buffer = unicam_[Unicam::Embedded].getBuffers().at(bufferId);\n \t\thandleStreamBuffer(buffer, &unicam_[Unicam::Embedded]);\n \t\tbreak;\n \t}\n \n \tcase RPI_IPA_ACTION_RUN_ISP: {\n \t\tunsigned int bufferId = action.data[0];\n-\t\tFrameBuffer *buffer = unicam_[Unicam::Image].getBuffers()->at(bufferId).get();\n+\t\tFrameBuffer *buffer = unicam_[Unicam::Image].getBuffers().at(bufferId);\n \n \t\tLOG(RPI, Debug) << \"Input re-queue to ISP, buffer id \" << buffer->cookie()\n \t\t\t\t<< \", timestamp: \" << buffer->metadata().timestamp;\n \n-\t\tisp_[Isp::Input].dev()->queueBuffer(buffer);\n+\t\tisp_[Isp::Input].queueBuffer(buffer);\n \t\tispOutputCount_ = 0;\n \t\tbreak;\n \t}\n@@ -1213,22 +1224,27 @@ void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer)\n \t\t\t<< \", buffer id \" << buffer->cookie()\n \t\t\t<< \", timestamp: \" << buffer->metadata().timestamp;\n \n-\thandleStreamBuffer(buffer, stream);\n-\n \t/*\n-\t * Increment the number of ISP outputs generated.\n-\t * This is needed to track dropped frames.\n+\t * ISP statistics buffer must not be re-queued or sent back to the\n+\t * application until after the IPA signals so.\n \t */\n-\tispOutputCount_++;\n-\n-\t/* If this is a stats output, hand it to the IPA now. */\n \tif (stream == &isp_[Isp::Stats]) {\n \t\tIPAOperationData op;\n \t\top.operation = RPI_IPA_EVENT_SIGNAL_STAT_READY;\n \t\top.data = { RPiIpaMask::STATS | buffer->cookie() };\n \t\tipa_->processEvent(op);\n+\t} else {\n+\t\t/* Any other ISP output can be handed back to the application now. */\n+\t\thandleStreamBuffer(buffer, stream);\n \t}\n \n+\t/*\n+\t * Increment the number of ISP outputs generated.\n+\t * This is needed to track dropped frames.\n+\t */\n+\tispOutputCount_++;\n+\n+\n \thandleState();\n }\n \n@@ -1241,8 +1257,12 @@ void RPiCameraData::clearIncompleteRequests()\n \t */\n \tfor (auto const request : requestQueue_) {\n \t\tfor (auto const stream : streams_) {\n-\t\t\tif (stream->isExternal())\n-\t\t\t\tstream->dev()->queueBuffer(request->findBuffer(stream));\n+\t\t\tif (!stream->isExternal())\n+\t\t\t\tcontinue;\n+\n+\t\t\tFrameBuffer *buffer = request->findBuffer(stream);\n+\t\t\tif (buffer)\n+\t\t\t\tstream->queueBuffer(buffer);\n \t\t}\n \t}\n \n@@ -1271,7 +1291,7 @@ void RPiCameraData::clearIncompleteRequests()\n \t\t\t * Has the buffer already been handed back to the\n \t\t\t * request? If not, do so now.\n \t\t\t */\n-\t\t\tif (buffer->request())\n+\t\t\tif (buffer && buffer->request())\n \t\t\t\tpipe_->completeBuffer(camera_, request, buffer);\n \t\t}\n \n@@ -1283,30 +1303,24 @@ void RPiCameraData::clearIncompleteRequests()\n void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, RPi::RPiStream *stream)\n {\n \tif (stream->isExternal()) {\n-\t\tif (!dropFrameCount_) {\n-\t\t\tRequest *request = buffer->request();\n+\t\tRequest *request = requestQueue_.front();\n+\n+\t\tif (!dropFrameCount_ && request->findBuffer(stream) == buffer) {\n+\t\t\t/*\n+\t\t\t * Tag the buffer as completed, returning it to the\n+\t\t\t * application.\n+\t\t\t */\n \t\t\tpipe_->completeBuffer(camera_, request, buffer);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * This buffer was not part of the Request, so we can\n+\t\t\t * recycle it.\n+\t\t\t */\n+\t\t\tstream->returnBuffer(buffer);\n \t\t}\n \t} else {\n-\t\t/* Special handling for RAW buffer Requests.\n-\t\t *\n-\t\t * The ISP input stream is alway an import stream, but if the\n-\t\t * current Request has been made for a buffer on the stream,\n-\t\t * simply memcpy to the Request buffer and requeue back to the\n-\t\t * device.\n-\t\t */\n-\t\tif (stream == &unicam_[Unicam::Image] && !dropFrameCount_) {\n-\t\t\tconst Stream *rawStream = static_cast<const Stream *>(&isp_[Isp::Input]);\n-\t\t\tRequest *request = requestQueue_.front();\n-\t\t\tFrameBuffer *raw = request->findBuffer(const_cast<Stream *>(rawStream));\n-\t\t\tif (raw) {\n-\t\t\t\traw->copyFrom(buffer);\n-\t\t\t\tpipe_->completeBuffer(camera_, request, raw);\n-\t\t\t}\n-\t\t}\n-\n-\t\t/* Simply requeue the buffer. */\n-\t\tstream->dev()->queueBuffer(buffer);\n+\t\t/* Simply re-queue the buffer to the requested stream. */\n+\t\tstream->queueBuffer(buffer);\n \t}\n }\n \n@@ -1390,7 +1404,7 @@ void RPiCameraData::tryRunPipeline()\n \t * current bayer buffer will be removed and re-queued to the driver.\n \t */\n \tembeddedBuffer = updateQueue(embeddedQueue_, bayerBuffer->metadata().timestamp,\n-\t\t\t\t     unicam_[Unicam::Embedded].dev());\n+\t\t\t\t     &unicam_[Unicam::Embedded]);\n \n \tif (!embeddedBuffer) {\n \t\tLOG(RPI, Debug) << \"Could not find matching embedded buffer\";\n@@ -1409,7 +1423,7 @@ void RPiCameraData::tryRunPipeline()\n \n \t\tembeddedBuffer = embeddedQueue_.front();\n \t\tbayerBuffer = updateQueue(bayerQueue_, embeddedBuffer->metadata().timestamp,\n-\t\t\t\t\t  unicam_[Unicam::Image].dev());\n+\t\t\t\t\t  &unicam_[Unicam::Image]);\n \n \t\tif (!bayerBuffer) {\n \t\t\tLOG(RPI, Debug) << \"Could not find matching bayer buffer - ending.\";\n@@ -1417,11 +1431,7 @@ void RPiCameraData::tryRunPipeline()\n \t\t}\n \t}\n \n-\t/*\n-\t * Take the first request from the queue and action the IPA.\n-\t * Unicam buffers for the request have already been queued as they come\n-\t * in.\n-\t */\n+\t/* Take the first request from the queue and action the IPA. */\n \tRequest *request = requestQueue_.front();\n \n \t/*\n@@ -1433,12 +1443,6 @@ void RPiCameraData::tryRunPipeline()\n \top.controls = { request->controls() };\n \tipa_->processEvent(op);\n \n-\t/* Queue up any ISP buffers passed into the request. */\n-\tfor (auto &stream : isp_) {\n-\t\tif (stream.isExternal())\n-\t\t\tstream.dev()->queueBuffer(request->findBuffer(&stream));\n-\t}\n-\n \t/* Ready to use the buffers, pop them off the queue. */\n \tbayerQueue_.pop();\n \tembeddedQueue_.pop();\n@@ -1468,32 +1472,42 @@ void RPiCameraData::tryFlushQueues()\n \t * and give a chance for the hardware to return to lock-step. We do have\n \t * to drop all interim frames.\n \t */\n-\tif (unicam_[Unicam::Image].getBuffers()->size() == bayerQueue_.size() &&\n-\t    unicam_[Unicam::Embedded].getBuffers()->size() == embeddedQueue_.size()) {\n+\tif (unicam_[Unicam::Image].getBuffers().size() == bayerQueue_.size() &&\n+\t    unicam_[Unicam::Embedded].getBuffers().size() == embeddedQueue_.size()) {\n+\t\t/* This cannot happen when Unicam streams are external. */\n+\t\tassert(!unicam_[Unicam::Image].isExternal());\n+\n \t\tLOG(RPI, Warning) << \"Flushing all buffer queues!\";\n \n \t\twhile (!bayerQueue_.empty()) {\n-\t\t\tunicam_[Unicam::Image].dev()->queueBuffer(bayerQueue_.front());\n+\t\t\tunicam_[Unicam::Image].queueBuffer(bayerQueue_.front());\n \t\t\tbayerQueue_.pop();\n \t\t}\n \n \t\twhile (!embeddedQueue_.empty()) {\n-\t\t\tunicam_[Unicam::Embedded].dev()->queueBuffer(embeddedQueue_.front());\n+\t\t\tunicam_[Unicam::Embedded].queueBuffer(embeddedQueue_.front());\n \t\t\tembeddedQueue_.pop();\n \t\t}\n \t}\n }\n \n FrameBuffer *RPiCameraData::updateQueue(std::queue<FrameBuffer *> &q, uint64_t timestamp,\n-\t\t\t\t\tV4L2VideoDevice *dev)\n+\t\t\t\t\tRPi::RPiStream *stream)\n {\n+\t/*\n+\t * If the unicam streams are external (both have to the same), then we\n+\t * can only return out the top buffer in the queue, and assume they have\n+\t * been synced by queuing at the same time. We cannot drop these frames,\n+\t * as they may have been provided externally.\n+\t */\n \twhile (!q.empty()) {\n \t\tFrameBuffer *b = q.front();\n-\t\tif (b->metadata().timestamp < timestamp) {\n+\t\tif (!stream->isExternal() && b->metadata().timestamp < timestamp) {\n \t\t\tq.pop();\n-\t\t\tdev->queueBuffer(b);\n-\t\t\tLOG(RPI, Warning) << \"Dropping input frame!\";\n-\t\t} else if (b->metadata().timestamp == timestamp) {\n+\t\t\tstream->queueBuffer(b);\n+\t\t\tLOG(RPI, Warning) << \"Dropping unmatched input frame in stream \"\n+\t\t\t\t\t  << stream->name();\n+\t\t} else if (stream->isExternal() || b->metadata().timestamp == timestamp) {\n \t\t\t/* The calling function will pop the item from the queue. */\n \t\t\treturn b;\n \t\t} else {\ndiff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp\nindex 57e5cf72..53a335e3 100644\n--- a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp\n+++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp\n@@ -21,30 +21,20 @@ V4L2VideoDevice *RPiStream::dev() const\n \n void RPiStream::setExternal(bool external)\n {\n+\t/* Import streams cannot be external. */\n+\tassert(!external || !importOnly_);\n \texternal_ = external;\n }\n \n bool RPiStream::isExternal() const\n {\n-\t/*\n-\t * Import streams cannot be external.\n-\t *\n-\t * RAW capture is a special case where we simply copy the RAW\n-\t * buffer out of the request. All other buffer handling happens\n-\t * as if the stream is internal.\n-\t */\n-\treturn external_ && !importOnly_;\n-}\n-\n-bool RPiStream::isImporter() const\n-{\n-\treturn importOnly_;\n+\treturn external_;\n }\n \n void RPiStream::reset()\n {\n \texternal_ = false;\n-\tinternalBuffers_.clear();\n+\tclearBuffers();\n }\n \n std::string RPiStream::name() const\n@@ -52,65 +42,123 @@ std::string RPiStream::name() const\n \treturn name_;\n }\n \n-void RPiStream::setExternalBuffers(std::vector<std::unique_ptr<FrameBuffer>> *buffers)\n+void RPiStream::setExportedBuffers(std::vector<std::unique_ptr<FrameBuffer>> *buffers)\n {\n-\texternalBuffers_ = buffers;\n+\tstd::transform(buffers->begin(), buffers->end(), std::back_inserter(bufferList_),\n+\t\t       [](std::unique_ptr<FrameBuffer> &b) { return b.get(); });\n }\n \n-const std::vector<std::unique_ptr<FrameBuffer>> *RPiStream::getBuffers() const\n+const std::vector<FrameBuffer *> &RPiStream::getBuffers() const\n {\n-\treturn external_ ? externalBuffers_ : &internalBuffers_;\n+\treturn bufferList_;\n }\n \n void RPiStream::releaseBuffers()\n {\n \tdev_->releaseBuffers();\n-\tif (!external_ && !importOnly_)\n-\t\tinternalBuffers_.clear();\n+\tclearBuffers();\n }\n \n-int RPiStream::importBuffers(unsigned int count)\n+int RPiStream::prepareBuffers(unsigned int count)\n {\n+\tint ret;\n+\n+\tif (!importOnly_) {\n+\t\tif (count) {\n+\t\t\t/* Export some frame buffers for internal use. */\n+\t\t\tret = dev_->exportBuffers(count, &internalBuffers_);\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\n+\t\t\t/* Add these exported buffers to the internal/external buffer list. */\n+\t\t\tsetExportedBuffers(&internalBuffers_);\n+\n+\t\t\t/* Add these buffers to the queue of internal usable buffers. */\n+\t\t\tfor (auto const &buffer : internalBuffers_)\n+\t\t\t\tavailableBuffers_.push(buffer.get());\n+\t\t}\n+\n+\t\t/* We must import all internal/external exported buffers. */\n+\t\tcount = bufferList_.size();\n+\t}\n+\n \treturn dev_->importBuffers(count);\n }\n \n-int RPiStream::allocateBuffers(unsigned int count)\n+int RPiStream::queueAllBuffers()\n {\n-\treturn dev_->allocateBuffers(count, &internalBuffers_);\n-}\n+\tint ret;\n \n-int RPiStream::queueBuffers()\n-{\n \tif (external_)\n \t\treturn 0;\n \n-\tfor (auto &b : internalBuffers_) {\n-\t\tint ret = dev_->queueBuffer(b.get());\n-\t\tif (ret) {\n-\t\t\tLOG(RPISTREAM, Error) << \"Failed to queue buffers for \"\n-\t\t\t\t\t      << name_;\n+\twhile (!availableBuffers_.empty()) {\n+\t\tret = queueBuffer(availableBuffers_.front());\n+\t\tif (ret < 0)\n \t\t\treturn ret;\n-\t\t}\n+\n+\t\tavailableBuffers_.pop();\n \t}\n \n \treturn 0;\n }\n \n-bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const\n+int RPiStream::queueBuffer(FrameBuffer *buffer)\n+{\n+\t/*\n+\t * A nullptr buffer implies an external stream, but no external\n+\t * buffer has been supplied. So, pick one from the availableBuffers_\n+\t * queue.\n+\t */\n+\tif (!buffer) {\n+\t\tif (availableBuffers_.empty()) {\n+\t\t\tLOG(RPISTREAM, Warning) << \"No buffers available for \"\n+\t\t\t\t\t\t<< name_;\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\n+\t\tbuffer = availableBuffers_.front();\n+\t\tavailableBuffers_.pop();\n+\t}\n+\n+\tLOG(RPISTREAM, Debug) << \"Queuing buffer \" << buffer->cookie()\n+\t\t\t      << \" for \" << name_;\n+\n+\tint ret = dev_->queueBuffer(buffer);\n+\tif (ret) {\n+\t\tLOG(RPISTREAM, Error) << \"Failed to queue buffer for \"\n+\t\t\t\t      << name_;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+void RPiStream::returnBuffer(FrameBuffer *buffer)\n {\n-\tauto start = external_ ? externalBuffers_->begin() : internalBuffers_.begin();\n-\tauto end = external_ ? externalBuffers_->end() : internalBuffers_.end();\n+\t/* This can only be called for external streams. */\n+\tassert(external_);\n+\n+\tavailableBuffers_.push(buffer);\n+}\n \n+bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const\n+{\n \tif (importOnly_)\n \t\treturn false;\n \n-\tif (std::find_if(start, end,\n-\t\t\t [buffer](std::unique_ptr<FrameBuffer> const &ref) { return ref.get() == buffer; }) != end)\n+\tif (std::find(bufferList_.begin(), bufferList_.end(), buffer) != bufferList_.end())\n \t\treturn true;\n \n \treturn false;\n }\n \n+void RPiStream::clearBuffers()\n+{\n+\tavailableBuffers_ = std::queue<FrameBuffer *>{};\n+\tinternalBuffers_.clear();\n+\tbufferList_.clear();\n+}\n+\n } /* namespace RPi */\n \n } /* namespace libcamera */\ndiff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.h b/src/libcamera/pipeline/raspberrypi/rpi_stream.h\nindex 40fff81d..019e236d 100644\n--- a/src/libcamera/pipeline/raspberrypi/rpi_stream.h\n+++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.h\n@@ -38,21 +38,22 @@ public:\n \tV4L2VideoDevice *dev() const;\n \tvoid setExternal(bool external);\n \tbool isExternal() const;\n-\tbool isImporter() const;\n \tvoid reset();\n \tstd::string name() const;\n-\tvoid setExternalBuffers(std::vector<std::unique_ptr<FrameBuffer>> *buffers);\n-\tconst std::vector<std::unique_ptr<FrameBuffer>> *getBuffers() const;\n+\tvoid setExportedBuffers(std::vector<std::unique_ptr<FrameBuffer>> *buffers);\n+\tconst std::vector<FrameBuffer *> &getBuffers() const;\n \tvoid releaseBuffers();\n-\tint importBuffers(unsigned int count);\n-\tint allocateBuffers(unsigned int count);\n-\tint queueBuffers();\n+\tint prepareBuffers(unsigned int count);\n+\tint queueAllBuffers();\n+\tint queueBuffer(FrameBuffer *buffer);\n+\tvoid returnBuffer(FrameBuffer *buffer);\n \tbool findFrameBuffer(FrameBuffer *buffer) const;\n \n private:\n+\tvoid clearBuffers();\n \t/*\n \t * Indicates that this stream is active externally, i.e. the buffers\n-\t * are provided by the application.\n+\t * might be provided by (and returned to) the application.\n \t */\n \tbool external_;\n \t/* Indicates that this stream only imports buffers, e.g. ISP input. */\n@@ -61,10 +62,19 @@ private:\n \tstd::string name_;\n \t/* The actual device stream. */\n \tstd::unique_ptr<V4L2VideoDevice> dev_;\n-\t/* Internally allocated framebuffers associated with this device stream. */\n+\t/* All framebuffers associated with this device stream. */\n+\tstd::vector<FrameBuffer *> bufferList_;\n+\t/*\n+\t * List of frame buffer that we can use if none have been provided by\n+\t * the application for external streams. This is populated by the\n+\t * buffers exported internally.\n+\t */\n+\tstd::queue<FrameBuffer *> availableBuffers_;\n+\t/*\n+\t * This is a list of buffers exported internally. Need to keep this around\n+\t * as the stream needs to maintain ownership of these buffers.\n+\t */\n \tstd::vector<std::unique_ptr<FrameBuffer>> internalBuffers_;\n-\t/* Externally allocated framebuffers associated with this device stream. */\n-\tstd::vector<std::unique_ptr<FrameBuffer>> *externalBuffers_;\n };\n \n /*\n","prefixes":["libcamera-devel","v3","06/10"]}