From patchwork Mon Jul 20 09:13:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8863 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 588EABD792 for ; Mon, 20 Jul 2020 09:13:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1E6E0605B2; Mon, 20 Jul 2020 11:13:31 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="MwzD1QIp"; dkim-atps=neutral Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9709760554 for ; Mon, 20 Jul 2020 11:13:28 +0200 (CEST) Received: by mail-wm1-x32a.google.com with SMTP id g10so11466763wmc.1 for ; Mon, 20 Jul 2020 02:13:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cXiNsMbIkFPmFbzw2rBupCuOWAWpbZRv7jNYIs6RHKw=; b=MwzD1QIpDAgoE+CjpOWpYY8nhiDHbQ1AOrgRNiHjqCONesoXEZYbi3gkv+TajeODJY NOrXTNtv6CR3hUTeDx2CvFe2KBXt93w+X5bkg2pbhBveHADl6nbfl0uWUhpKFOwqRVWW nhv71mpmvFxlniZ5TZKmruDqtlrhOrweEFTe5/epQvz4YumjRw2lUQhBzWZdFPGVyPP4 gw/ctSehfS0OixSC2LoHVO8qiCwBrmgUp/IpWT0l/9FlZR7LJ5bZG4d3IIjgaR6/JFeM b3Ef7BcrlLhekw9yfT2W2P8zUSOeup3bJh+iWBC08BbTWz4h4cltYcJcKN3mFt8KC3SR WmwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cXiNsMbIkFPmFbzw2rBupCuOWAWpbZRv7jNYIs6RHKw=; b=iva5TTD78r9TLSdDdPTzyeB1g7nEqrxMLEphA9tEkIUd+4ykeolYBMGZsp6Q+IlF5n VpqGzbVrzQrXf9/tOasBRX/fRE0HJNdStIw4imZD+htzpdZU7YVrD4wFXtOB512BaSph z7T+zJZli2H78m9F+nG5ZZzZknXtUXWmBw0jJu5yUxa+7u6aXC+rJbeLdfSka0fesX9x sQG5QpJqvePQAyLI0p/M1SkI5JI8L/Ydu8ayD5QxjfigWG31jktvIA4MGgO3AFUwaxbI tFQhdH7rXbdjW7dCS9mzXAO/kTISgDv75KMkT3Pz4SK648ewb6K+bet592Zy89q2P/BH y7oQ== X-Gm-Message-State: AOAM531mJn1PV3IBO0XZjdVhbHG2xcauU9qgPdnosgR18F2lvlseTtMR hwH2AyR58j2GSqZvY2oHeSE0lffkVvj5tw== X-Google-Smtp-Source: ABdhPJxc/dLnS6iEB5RWgkjBYQ7WYoGFeGVsrUrNFLROO5x6HUlPG6VMcn4C/Ro78zCISh7pWsLgQg== X-Received: by 2002:a1c:1f09:: with SMTP id f9mr21641375wmf.137.1595236407720; Mon, 20 Jul 2020 02:13:27 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:26 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:03 +0100 Message-Id: <20200720091311.805092-2-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 1/9] libcamera: pipeline: raspberrypi: Move RPiStream into a separate file X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Put RPiStream into the RPi namespace and add a new log category (RPISTREAM). There are no functional changes in this commit. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- .../pipeline/raspberrypi/meson.build | 1 + .../pipeline/raspberrypi/raspberrypi.cpp | 193 ++---------------- .../pipeline/raspberrypi/rpi_stream.cpp | 116 +++++++++++ .../pipeline/raspberrypi/rpi_stream.h | 98 +++++++++ 4 files changed, 234 insertions(+), 174 deletions(-) create mode 100644 src/libcamera/pipeline/raspberrypi/rpi_stream.cpp create mode 100644 src/libcamera/pipeline/raspberrypi/rpi_stream.h diff --git a/src/libcamera/pipeline/raspberrypi/meson.build b/src/libcamera/pipeline/raspberrypi/meson.build index ae0aed3b..7c5b6ff7 100644 --- a/src/libcamera/pipeline/raspberrypi/meson.build +++ b/src/libcamera/pipeline/raspberrypi/meson.build @@ -3,5 +3,6 @@ libcamera_sources += files([ 'dma_heaps.cpp', 'raspberrypi.cpp', + 'rpi_stream.cpp', 'staggered_ctrl.cpp', ]) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index bf1c7714..6630ef57 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -19,7 +19,6 @@ #include #include #include -#include #include @@ -33,6 +32,7 @@ #include "libcamera/internal/v4l2_videodevice.h" #include "dma_heaps.h" +#include "rpi_stream.h" #include "staggered_ctrl.h" namespace libcamera { @@ -123,165 +123,10 @@ V4L2DeviceFormat findBestMode(V4L2PixFmtMap &formatsMap, const Size &req) return bestMode; } -} /* namespace */ - -/* - * Device stream abstraction for either an internal or external stream. - * Used for both Unicam and the ISP. - */ -class RPiStream : public Stream -{ -public: - RPiStream() - { - } - - RPiStream(const char *name, MediaEntity *dev, bool importOnly = false) - : external_(false), importOnly_(importOnly), name_(name), - dev_(std::make_unique(dev)) - { - } - - V4L2VideoDevice *dev() const - { - return dev_.get(); - } - - void setExternal(bool external) - { - external_ = external; - } - - bool isExternal() const - { - /* - * Import streams cannot be external. - * - * RAW capture is a special case where we simply copy the RAW - * buffer out of the request. All other buffer handling happens - * as if the stream is internal. - */ - return external_ && !importOnly_; - } - - bool isImporter() const - { - return importOnly_; - } - - void reset() - { - external_ = false; - internalBuffers_.clear(); - } - - std::string name() const - { - return name_; - } - - void setExternalBuffers(std::vector> *buffers) - { - externalBuffers_ = buffers; - } - - const std::vector> *getBuffers() const - { - return external_ ? externalBuffers_ : &internalBuffers_; - } - - void releaseBuffers() - { - dev_->releaseBuffers(); - if (!external_ && !importOnly_) - internalBuffers_.clear(); - } - - int importBuffers(unsigned int count) - { - return dev_->importBuffers(count); - } - - int allocateBuffers(unsigned int count) - { - return dev_->allocateBuffers(count, &internalBuffers_); - } - - int queueBuffers() - { - if (external_) - return 0; - - for (auto &b : internalBuffers_) { - int ret = dev_->queueBuffer(b.get()); - if (ret) { - LOG(RPI, Error) << "Failed to queue buffers for " - << name_; - return ret; - } - } - - return 0; - } - - bool findFrameBuffer(FrameBuffer *buffer) const - { - auto start = external_ ? externalBuffers_->begin() : internalBuffers_.begin(); - auto end = external_ ? externalBuffers_->end() : internalBuffers_.end(); - - if (importOnly_) - return false; - - if (std::find_if(start, end, - [buffer](std::unique_ptr const &ref) { return ref.get() == buffer; }) != end) - return true; - - return false; - } - -private: - /* - * Indicates that this stream is active externally, i.e. the buffers - * are provided by the application. - */ - bool external_; - /* Indicates that this stream only imports buffers, e.g. ISP input. */ - bool importOnly_; - /* Stream name identifier. */ - std::string name_; - /* The actual device stream. */ - std::unique_ptr dev_; - /* Internally allocated framebuffers associated with this device stream. */ - std::vector> internalBuffers_; - /* Externally allocated framebuffers associated with this device stream. */ - std::vector> *externalBuffers_; -}; - -/* - * The following class is just a convenient (and typesafe) array of device - * streams indexed with an enum class. - */ enum class Unicam : unsigned int { Image, Embedded }; enum class Isp : unsigned int { Input, Output0, Output1, Stats }; -template -class RPiDevice : public std::array -{ -private: - constexpr auto index(E e) const noexcept - { - return static_cast>(e); - } -public: - RPiStream &operator[](E e) - { - return std::array::operator[](index(e)); - } - const RPiStream &operator[](E e) const - { - return std::array::operator[](index(e)); - } -}; +} /* namespace */ class RPiCameraData : public CameraData { @@ -305,15 +150,15 @@ public: void ispOutputDequeue(FrameBuffer *buffer); void clearIncompleteRequests(); - void handleStreamBuffer(FrameBuffer *buffer, const RPiStream *stream); + void handleStreamBuffer(FrameBuffer *buffer, const RPi::RPiStream *stream); void handleState(); CameraSensor *sensor_; /* Array of Unicam and ISP device streams and associated buffers/streams. */ - RPiDevice unicam_; - RPiDevice isp_; + RPi::RPiDevice unicam_; + RPi::RPiDevice isp_; /* The vector below is just for convenience when iterating over all streams. */ - std::vector streams_; + std::vector streams_; /* Buffers passed to the IPA. */ std::vector ipaBuffers_; @@ -762,7 +607,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) int PipelineHandlerRPi::exportFrameBuffers(Camera *camera, Stream *stream, std::vector> *buffers) { - RPiStream *s = static_cast(stream); + RPi::RPiStream *s = static_cast(stream); unsigned int count = stream->configuration().bufferCount; int ret = s->dev()->exportBuffers(count, buffers); @@ -908,14 +753,14 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator) std::unique_ptr data = std::make_unique(this); /* Locate and open the unicam video streams. */ - data->unicam_[Unicam::Embedded] = RPiStream("Unicam Embedded", unicam_->getEntityByName("unicam-embedded")); - data->unicam_[Unicam::Image] = RPiStream("Unicam Image", unicam_->getEntityByName("unicam-image")); + data->unicam_[Unicam::Embedded] = RPi::RPiStream("Unicam Embedded", unicam_->getEntityByName("unicam-embedded")); + data->unicam_[Unicam::Image] = RPi::RPiStream("Unicam Image", unicam_->getEntityByName("unicam-image")); /* Tag the ISP input stream as an import stream. */ - data->isp_[Isp::Input] = RPiStream("ISP Input", isp_->getEntityByName("bcm2835-isp0-output0"), true); - data->isp_[Isp::Output0] = RPiStream("ISP Output0", isp_->getEntityByName("bcm2835-isp0-capture1")); - data->isp_[Isp::Output1] = RPiStream("ISP Output1", isp_->getEntityByName("bcm2835-isp0-capture2")); - data->isp_[Isp::Stats] = RPiStream("ISP Stats", isp_->getEntityByName("bcm2835-isp0-capture3")); + data->isp_[Isp::Input] = RPi::RPiStream("ISP Input", isp_->getEntityByName("bcm2835-isp0-output0"), true); + data->isp_[Isp::Output0] = RPi::RPiStream("ISP Output0", isp_->getEntityByName("bcm2835-isp0-capture1")); + data->isp_[Isp::Output1] = RPi::RPiStream("ISP Output1", isp_->getEntityByName("bcm2835-isp0-capture2")); + data->isp_[Isp::Stats] = RPi::RPiStream("ISP Stats", isp_->getEntityByName("bcm2835-isp0-capture3")); /* This is just for convenience so that we can easily iterate over all streams. */ for (auto &stream : data->unicam_) @@ -1005,7 +850,7 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) */ unsigned int maxBuffers = 0; for (const Stream *s : camera->streams()) - if (static_cast(s)->isExternal()) + if (static_cast(s)->isExternal()) maxBuffers = std::max(maxBuffers, s->configuration().bufferCount); for (auto const stream : data->streams_) { @@ -1255,12 +1100,12 @@ done: void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) { - const RPiStream *stream = nullptr; + const RPi::RPiStream *stream = nullptr; if (state_ == State::Stopped) return; - for (RPiStream const &s : unicam_) { + for (RPi::RPiStream const &s : unicam_) { if (s.findFrameBuffer(buffer)) { stream = &s; break; @@ -1316,12 +1161,12 @@ void RPiCameraData::ispInputDequeue(FrameBuffer *buffer) void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer) { - const RPiStream *stream = nullptr; + const RPi::RPiStream *stream = nullptr; if (state_ == State::Stopped) return; - for (RPiStream const &s : isp_) { + for (RPi::RPiStream const &s : isp_) { if (s.findFrameBuffer(buffer)) { stream = &s; break; @@ -1402,7 +1247,7 @@ void RPiCameraData::clearIncompleteRequests() } } -void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, const RPiStream *stream) +void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, const RPi::RPiStream *stream) { if (stream->isExternal()) { if (!dropFrame_) { diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp new file mode 100644 index 00000000..2edb8b59 --- /dev/null +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp @@ -0,0 +1,116 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Raspberry Pi (Trading) Ltd. + * + * rpi_stream.cpp - Raspberry Pi device stream abstraction class. + */ +#include "rpi_stream.h" + +#include "libcamera/internal/log.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(RPISTREAM) + +namespace RPi { + +V4L2VideoDevice *RPiStream::dev() const +{ + return dev_.get(); +} + +void RPiStream::setExternal(bool external) +{ + external_ = external; +} + +bool RPiStream::isExternal() const +{ + /* + * Import streams cannot be external. + * + * RAW capture is a special case where we simply copy the RAW + * buffer out of the request. All other buffer handling happens + * as if the stream is internal. + */ + return external_ && !importOnly_; +} + +bool RPiStream::isImporter() const +{ + return importOnly_; +} + +void RPiStream::reset() +{ + external_ = false; + internalBuffers_.clear(); +} + +std::string RPiStream::name() const +{ + return name_; +} + +void RPiStream::setExternalBuffers(std::vector> *buffers) +{ + externalBuffers_ = buffers; +} + +const std::vector> *RPiStream::getBuffers() const +{ + return external_ ? externalBuffers_ : &internalBuffers_; +} + +void RPiStream::releaseBuffers() +{ + dev_->releaseBuffers(); + if (!external_ && !importOnly_) + internalBuffers_.clear(); +} + +int RPiStream::importBuffers(unsigned int count) +{ + return dev_->importBuffers(count); +} + +int RPiStream::allocateBuffers(unsigned int count) +{ + return dev_->allocateBuffers(count, &internalBuffers_); +} + +int RPiStream::queueBuffers() +{ + if (external_) + return 0; + + for (auto &b : internalBuffers_) { + int ret = dev_->queueBuffer(b.get()); + if (ret) { + LOG(RPISTREAM, Error) << "Failed to queue buffers for " + << name_; + return ret; + } + } + + return 0; +} + +bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const +{ + auto start = external_ ? externalBuffers_->begin() : internalBuffers_.begin(); + auto end = external_ ? externalBuffers_->end() : internalBuffers_.end(); + + if (importOnly_) + return false; + + if (std::find_if(start, end, + [buffer](std::unique_ptr const &ref) { return ref.get() == buffer; }) != end) + return true; + + return false; +} + +} /* namespace RPi */ + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.h b/src/libcamera/pipeline/raspberrypi/rpi_stream.h new file mode 100644 index 00000000..3957e342 --- /dev/null +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.h @@ -0,0 +1,98 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Raspberry Pi (Trading) Ltd. + * + * rpi_stream.h - Raspberry Pi device stream abstraction class. + */ +#ifndef __LIBCAMERA_PIPELINE_RPI_STREAM_H__ +#define __LIBCAMERA_PIPELINE_RPI_STREAM_H__ + +#include +#include +#include + +#include + +#include "libcamera/internal/v4l2_videodevice.h" + +namespace libcamera { + +namespace RPi { + +/* + * Device stream abstraction for either an internal or external stream. + * Used for both Unicam and the ISP. + */ +class RPiStream : public Stream +{ +public: + RPiStream() + { + } + + RPiStream(const char *name, MediaEntity *dev, bool importOnly = false) + : external_(false), importOnly_(importOnly), name_(name), + dev_(std::make_unique(dev)) + { + } + + V4L2VideoDevice *dev() const; + void setExternal(bool external); + bool isExternal() const; + bool isImporter() const; + void reset(); + std::string name() const; + void setExternalBuffers(std::vector> *buffers); + const std::vector> *getBuffers() const; + void releaseBuffers(); + int importBuffers(unsigned int count); + int allocateBuffers(unsigned int count); + int queueBuffers(); + bool findFrameBuffer(FrameBuffer *buffer) const; + +private: + /* + * Indicates that this stream is active externally, i.e. the buffers + * are provided by the application. + */ + bool external_; + /* Indicates that this stream only imports buffers, e.g. ISP input. */ + bool importOnly_; + /* Stream name identifier. */ + std::string name_; + /* The actual device stream. */ + std::unique_ptr dev_; + /* Internally allocated framebuffers associated with this device stream. */ + std::vector> internalBuffers_; + /* Externally allocated framebuffers associated with this device stream. */ + std::vector> *externalBuffers_; +}; + +/* + * The following class is just a convenient (and typesafe) array of device + * streams indexed with an enum class. + */ +template +class RPiDevice : public std::array +{ +private: + constexpr auto index(E e) const noexcept + { + return static_cast>(e); + } +public: + RPiStream &operator[](E e) + { + return std::array::operator[](index(e)); + } + const RPiStream &operator[](E e) const + { + return std::array::operator[](index(e)); + } +}; + +} /* namespace RPi */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_PIPELINE_RPI_STREAM_H__ */ From patchwork Mon Jul 20 09:13:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8864 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9DCAEBD792 for ; Mon, 20 Jul 2020 09:13:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A54926071C; Mon, 20 Jul 2020 11:13:31 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="EAETjlIW"; dkim-atps=neutral Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6F482605B2 for ; Mon, 20 Jul 2020 11:13:29 +0200 (CEST) Received: by mail-wm1-x330.google.com with SMTP id f18so24371917wml.3 for ; Mon, 20 Jul 2020 02:13:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BnNjRQGVdFTo5m4c8B+o7cFqI9rsLcIoUeoK5+uiA5U=; b=EAETjlIWAE0E+taybz0KTM1AlTl3WxNEdr6OcF0JSKITyBcW8rT2klLVPZ4wEGsPAR PuevMXo1rTdHF4uaavDE926g0BJoFaECZQf3FW3TrH46mh3c0qvQ85FSrMF1nL4PnmCL zLMLmY09cRJVBm08cFjrfihm0QV6FkvfnXRHHkEA9xhHlOvwQsURBRyGVUiVG/VcH1Lu HOqFZ7e68a4lwQbgwj/ne0lEvoNlMgdP/B/fDyHuU+FHbHHTKKrJ94bVvau2EuZ17VpV BfMdkGh1rha5lL1rSyEWPTTPRQKLtsWpIImC3DMY1NXRJeSGRovLp2iqvkQTN6SS2s1D pgxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BnNjRQGVdFTo5m4c8B+o7cFqI9rsLcIoUeoK5+uiA5U=; b=AJ+jb93EPf2iCQHrxQc0BBNVOsmKJ0fJkHvH2BNR43yScZg2FTV2yQ21OkOvOoqEUE 2bJK/BIvq0X3ZuolpqrLW3oFw6mPoUwMR7YI7Mcx+eJAOhnUeIETGgAwEU4xcmzP1Plk ccvicbLUv0EKPmkQ/Aq9yBSqD7uLJDIO7/dt2FSBHecXxotkiS6iraXg75phBwYD/Zew j6P9gbwgKb48qtAlSPZSTqmb+iwhT+4+ilI8j3MWqwAxmM3u+QI86VMUQZhEAz6m6OJ0 x5rPZChCRlQzs1nM0NgEHuehfrxL3FO5YBbL1sE/DDYGPNmG/FYJaTrv0vZSKAx8Z6KR CnPQ== X-Gm-Message-State: AOAM531XBu8Ze3M+j58pClYUaFDPX0Bk2CgolfnKUBzlSPBYvW/UnNkV ub7pHrDnCRf2d8zQDz7uZmam4NgD8bnDYw== X-Google-Smtp-Source: ABdhPJwD094E3HUQKt7qAg0neKy2kKpUPZo2DLBVAy1J3XzpGf4I2VNx1B8W9MlDpM5MQFwCRqr/JA== X-Received: by 2002:a7b:ca4c:: with SMTP id m12mr20784862wml.33.1595236408527; Mon, 20 Jul 2020 02:13:28 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:27 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:04 +0100 Message-Id: <20200720091311.805092-3-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 2/9] libcamera: pipeline: ipa: raspberrypi: Rework drop frame signalling X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The IPA now signals up front how many frames it wants the pipeline handler to drop. This makes it easier to handle up-coming changes to the buffer handling for import/export buffers. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- include/libcamera/ipa/raspberrypi.h | 2 +- src/ipa/raspberrypi/raspberrypi.cpp | 20 +++++------ .../pipeline/raspberrypi/raspberrypi.cpp | 36 +++++++++++-------- 3 files changed, 33 insertions(+), 25 deletions(-) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index a4937769..0f31b6ea 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -14,6 +14,7 @@ enum RPiConfigParameters { RPI_IPA_CONFIG_LS_TABLE = (1 << 0), RPI_IPA_CONFIG_STAGGERED_WRITE = (1 << 1), RPI_IPA_CONFIG_SENSOR = (1 << 2), + RPI_IPA_CONFIG_DROP_FRAMES = (1 << 3), }; enum RPiOperations { @@ -21,7 +22,6 @@ enum RPiOperations { RPI_IPA_ACTION_V4L2_SET_ISP, RPI_IPA_ACTION_STATS_METADATA_COMPLETE, RPI_IPA_ACTION_RUN_ISP, - RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME, RPI_IPA_ACTION_EMBEDDED_COMPLETE, RPI_IPA_EVENT_SIGNAL_STAT_READY, RPI_IPA_EVENT_SIGNAL_ISP_PREPARE, diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 7bd04880..2d9fb9d8 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -65,8 +65,8 @@ class IPARPi : public IPAInterface public: IPARPi() : lastMode_({}), controller_(), controllerInit_(false), - frame_count_(0), check_count_(0), hide_count_(0), - mistrust_count_(0), lsTable_(nullptr) + frame_count_(0), check_count_(0), mistrust_count_(0), + lsTableHandle_(0), lsTable_(nullptr) { } @@ -137,8 +137,6 @@ private: uint64_t frame_count_; /* For checking the sequencing of Prepare/Process calls. */ uint64_t check_count_; - /* How many frames the pipeline handler should hide, or "drop". */ - unsigned int hide_count_; /* How many frames we should avoid running control algos on. */ unsigned int mistrust_count_; /* LS table allocation passed in from the pipeline handler. */ @@ -242,14 +240,18 @@ void IPARPi::configure(const CameraSensorInfo &sensorInfo, */ frame_count_ = 0; check_count_ = 0; + int drop_frame = 0; if (controllerInit_) { - hide_count_ = helper_->HideFramesModeSwitch(); + drop_frame = helper_->HideFramesModeSwitch(); mistrust_count_ = helper_->MistrustFramesModeSwitch(); } else { - hide_count_ = helper_->HideFramesStartup(); + drop_frame = helper_->HideFramesStartup(); mistrust_count_ = helper_->MistrustFramesStartup(); } + result->data.push_back(drop_frame); + result->operation |= RPI_IPA_CONFIG_DROP_FRAMES; + struct AgcStatus agcStatus; /* These zero values mean not program anything (unless overwritten). */ agcStatus.shutter_time = 0.0; @@ -366,13 +368,11 @@ void IPARPi::processEvent(const IPAOperationData &event) * they are "unreliable". */ prepareISP(embeddedbufferId); + frame_count_++; /* Ready to push the input buffer into the ISP. */ IPAOperationData op; - if (++frame_count_ > hide_count_) - op.operation = RPI_IPA_ACTION_RUN_ISP; - else - op.operation = RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME; + op.operation = RPI_IPA_ACTION_RUN_ISP; op.data = { bayerbufferId & RPiIpaMask::ID }; queueFrameAction.emit(0, op); break; diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 6630ef57..d3639ce1 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -133,7 +133,7 @@ class RPiCameraData : public CameraData public: RPiCameraData(PipelineHandler *pipe) : CameraData(pipe), sensor_(nullptr), state_(State::Stopped), - dropFrame_(false), ispOutputCount_(0) + dropFrameCount_(0), ispOutputCount_(0) { } @@ -181,14 +181,15 @@ public: std::queue embeddedQueue_; std::deque requestQueue_; + unsigned int dropFrameCount_; + private: void checkRequestCompleted(); void tryRunPipeline(); void tryFlushQueues(); FrameBuffer *updateQueue(std::queue &q, uint64_t timestamp, V4L2VideoDevice *dev); - bool dropFrame_; - int ispOutputCount_; + unsigned int ispOutputCount_; }; class RPiCameraConfiguration : public CameraConfiguration @@ -999,6 +1000,7 @@ int RPiCameraData::configureIPA() ipa_->configure(sensorInfo, streamConfig, entityControls, ipaConfig, &result); + unsigned int resultIdx = 0; if (result.operation & RPI_IPA_CONFIG_STAGGERED_WRITE) { /* * Setup our staggered control writer with the sensor default @@ -1006,9 +1008,9 @@ int RPiCameraData::configureIPA() */ if (!staggeredCtrl_) { staggeredCtrl_.init(unicam_[Unicam::Image].dev(), - { { V4L2_CID_ANALOGUE_GAIN, result.data[0] }, - { V4L2_CID_EXPOSURE, result.data[1] } }); - sensorMetadata_ = result.data[2]; + { { V4L2_CID_ANALOGUE_GAIN, result.data[resultIdx++] }, + { V4L2_CID_EXPOSURE, result.data[resultIdx++] } }); + sensorMetadata_ = result.data[resultIdx++]; } /* Configure the H/V flip controls based on the sensor rotation. */ @@ -1025,6 +1027,11 @@ int RPiCameraData::configureIPA() LOG(RPI, Error) << "V4L2 staggered set failed"; } + if (result.operation & RPI_IPA_CONFIG_DROP_FRAMES) { + /* Configure the number of dropped frames required on startup. */ + dropFrameCount_ = result.data[resultIdx++]; + } + return 0; } @@ -1075,7 +1082,6 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData break; } - case RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME: case RPI_IPA_ACTION_RUN_ISP: { unsigned int bufferId = action.data[0]; FrameBuffer *buffer = unicam_[Unicam::Image].getBuffers()->at(bufferId).get(); @@ -1084,7 +1090,6 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData << ", timestamp: " << buffer->metadata().timestamp; isp_[Isp::Input].dev()->queueBuffer(buffer); - dropFrame_ = (action.operation == RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME) ? true : false; ispOutputCount_ = 0; break; } @@ -1250,7 +1255,7 @@ void RPiCameraData::clearIncompleteRequests() void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, const RPi::RPiStream *stream) { if (stream->isExternal()) { - if (!dropFrame_) { + if (!dropFrameCount_) { Request *request = buffer->request(); pipe_->completeBuffer(camera_, request, buffer); } @@ -1262,7 +1267,7 @@ void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, const RPi::RPiStream * simply memcpy to the Request buffer and requeue back to the * device. */ - if (stream == &unicam_[Unicam::Image] && !dropFrame_) { + if (stream == &unicam_[Unicam::Image] && !dropFrameCount_) { const Stream *rawStream = static_cast(&isp_[Isp::Input]); Request *request = requestQueue_.front(); FrameBuffer *raw = request->findBuffer(const_cast(rawStream)); @@ -1307,7 +1312,7 @@ void RPiCameraData::checkRequestCompleted() * If we are dropping this frame, do not touch the request, simply * change the state to IDLE when ready. */ - if (!dropFrame_) { + if (!dropFrameCount_) { Request *request = requestQueue_.front(); if (request->hasPendingBuffers()) return; @@ -1326,10 +1331,13 @@ void RPiCameraData::checkRequestCompleted() * frame. */ if (state_ == State::IpaComplete && - ((ispOutputCount_ == 3 && dropFrame_) || requestCompleted)) { + ((ispOutputCount_ == 3 && dropFrameCount_) || requestCompleted)) { state_ = State::Idle; - if (dropFrame_) - LOG(RPI, Info) << "Dropping frame at the request of the IPA"; + if (dropFrameCount_) { + dropFrameCount_--; + LOG(RPI, Info) << "Dropping frame at the request of the IPA (" + << dropFrameCount_ << " left)"; + } } } From patchwork Mon Jul 20 09:13:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8865 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0C57FC2E67 for ; Mon, 20 Jul 2020 09:13:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4AFC860724; Mon, 20 Jul 2020 11:13:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="hMsxiGef"; dkim-atps=neutral Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0BB8A60554 for ; Mon, 20 Jul 2020 11:13:30 +0200 (CEST) Received: by mail-wm1-x343.google.com with SMTP id j18so21509730wmi.3 for ; Mon, 20 Jul 2020 02:13:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h2usrW3kOg/PwxnrQi+Bi9R18693ZRttLpAYPr1ayMA=; b=hMsxiGefJL63ssdh0NoTVjQT6h80uttSozroJAHfsBNVzlO7TCsBP5qz/BXJ7I95TQ UPKqroXUOQe+pQxdhFg6D6zmpNr/SkBo3GTwBPuJcBym3JBM6gdntPZFnx6PWekzA2L5 c+wsIMh1WRAs5DcuswPhu+mIXirvVTd2iVcJxPFyst4BGPCDh7D1kCkTwLtuDeb6CokP nyGoJ98oquz8iWFDFvBo2HKq+nak2uFaUfjgpJsYC13KreRejNq5xt+YRhrAO3ZX1mpH 0VzEk+dFhauJ97reHSjcABeMzCG/R/WajPjPlhTj1tmCoNf6QrPNCcIWTsIppPoztxv7 Y+Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h2usrW3kOg/PwxnrQi+Bi9R18693ZRttLpAYPr1ayMA=; b=aqPV63x4RnLDyR9McmN9yBb7dH0vuEvKisGQrAoFSkqBAn7wBhvoLMAUwQBWzVcIFq YoPMxV4+1lddDusOdLo96D3Vz8BCGC7PQQ1U+tp8VhQIfrFuX+FZGkJYzglDjB3/imIE p2rNpuVTc4ZjFeSz7iekPF4tVx4LeTL0tByyuPg+BxuQQ+6KDwv2L85OxmrY3gR1g5Ta rDFh1VWU0Nf+C3RFub2233nrIL6jaKfea5o6/GVhAmWRpM6UxxyFJZwruFQs0obnvWhS TLHIQI0MvdzELn6rR/PaPC6/53lh4fQMunPqk9OwSGWNMumr0ZNzHFlUwUNV+X17ufqH 9fPg== X-Gm-Message-State: AOAM533RaRXQJhH9D0yK/FThHAuMuzeyGv+hJg6nPCH43mFVrl2pc1VA jkD66D64XMKPmy71XpPej7nJh6qecvgIIQ== X-Google-Smtp-Source: ABdhPJyctXnzSXmxW85EZyjXMqMxwnKbTYrV+vitzMgCsGQFBywK+tGm0D8hptqwIrEXqvabMk5Y4w== X-Received: by 2002:a1c:a181:: with SMTP id k123mr20861498wme.172.1595236409429; Mon, 20 Jul 2020 02:13:29 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:28 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:05 +0100 Message-Id: <20200720091311.805092-4-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 3/9] libcamera: pipeline: raspberrypi: Add some debug logging X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" No functional changes, only added some more trace points. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham --- src/libcamera/pipeline/raspberrypi/raspberrypi.cpp | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index d3639ce1..35f14dda 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1160,6 +1160,11 @@ void RPiCameraData::ispInputDequeue(FrameBuffer *buffer) if (state_ == State::Stopped) return; + LOG(RPI, Debug) << "Stream ISP Input buffer complete" + << ", buffer id " << buffer->cookie() + << ", timestamp: " << buffer->metadata().timestamp; + + /* The ISP input buffer gets re-queued into Unicam. */ handleStreamBuffer(buffer, &unicam_[Unicam::Image]); handleState(); } @@ -1324,6 +1329,8 @@ void RPiCameraData::checkRequestCompleted() pipe_->completeRequest(camera_, request); requestQueue_.pop_front(); requestCompleted = true; + + LOG(RPI, Debug) << "Request is complete"; } /* @@ -1462,7 +1469,7 @@ FrameBuffer *RPiCameraData::updateQueue(std::queue &q, uint64_t t if (b->metadata().timestamp < timestamp) { q.pop(); dev->queueBuffer(b); - LOG(RPI, Error) << "Dropping input frame!"; + LOG(RPI, Warning) << "Dropping input frame!"; } else if (b->metadata().timestamp == timestamp) { /* The calling function will pop the item from the queue. */ return b; From patchwork Mon Jul 20 09:13:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8866 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 09CCABD792 for ; Mon, 20 Jul 2020 09:13:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C9F9060729; Mon, 20 Jul 2020 11:13:33 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="JGpeFBBk"; dkim-atps=neutral Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2FEEB60717 for ; Mon, 20 Jul 2020 11:13:31 +0200 (CEST) Received: by mail-wm1-x333.google.com with SMTP id o2so24447669wmh.2 for ; Mon, 20 Jul 2020 02:13:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6HHVIiW2Zt33xaBdxR4KVZKDUQQdUezTVtI0Akxm8s4=; b=JGpeFBBkGR6oP9JT4e5rNe4x60EZzKiuIUtDA7kuNyn5oO2UZHuYglmmldpuAHm8es 5XBzzOM3/O8jvikCz701H5qQr96DaGyNsz7/+6Eh4ldqgN+0G7AxIuWL93fL7ylCrD6x AYZ6gGY7r8JIriIqQkDy+PFOLONsEVR1R6ucuAAE58tu2BRWCa3bzc63P5Uxs9otZKgJ 6POz6CAscvFscj0N84e7VcacjRXup5uCd6Vruy8vhpaA1Wx9W2V6uWUti6RwqvpOADSx TSOD7Khku+Q1NQ63S9CLXwFP5hIwnErys0MJICcx4dlUdm7IlamfojAPP9r7PGsAh8IE LuLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6HHVIiW2Zt33xaBdxR4KVZKDUQQdUezTVtI0Akxm8s4=; b=m8LTDCbr+4/OZ8FP8xFZsdJxDbwKA64qIv7lCRj1WuMvFAiks7TEMVyujIUAOm6zXb HcHLjgte+fI6VTb/Msy9Gl48OVZFTEyBlZU91GODntu2av7zOqwfePKgnCuFBpvqU/jS QzFYFncZLqXiaHZab5ZtejBEXqB6MiBb/XWcvs2A20VZTpFAS+yNPlzdSpbgZ/GThGbY 6jhMNi5TDTk0y1OQsf7Nu2kCFlyyDpE5GaDsKeZd4AJ+6x2zf0i/Vld3thcY4OaFHkL+ 3XcaLVj2u7YFAB/ucqQ11nsGaUZrXdfANeVl9Q+4mXsiNf3plSOQ8nPIZcqoTJVW+oP5 ndig== X-Gm-Message-State: AOAM532Q+jJSMf4pnJLQ19kNxJ+jjqLYILiddpoIE3sF7tDh75QO9DIW dMyJ9M5+x+eWkigDQCwMP3PCIJEqWrBvvA== X-Google-Smtp-Source: ABdhPJwzcHPCHWs8yDMFfxGcfQepyT3uvXNcWxyTkoNiKGpuT23CvkcNrjokJHYJDGPQ+fFoRFXIEw== X-Received: by 2002:a05:600c:4114:: with SMTP id j20mr20293949wmi.74.1595236410564; Mon, 20 Jul 2020 02:13:30 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:29 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:06 +0100 Message-Id: <20200720091311.805092-5-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 4/9] libcamera: pipeline: raspberrypi: Increase the number of RAW buffers X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Increase the number of expected RAW buffers in the stream configuration to 2. This will avoid dropping Unicam frames when exporting RAW streams. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/libcamera/pipeline/raspberrypi/raspberrypi.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 35f14dda..486345fa 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -379,7 +379,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, sensorFormat = findBestMode(fmts, size); pixelFormat = sensorFormat.fourcc.toPixelFormat(); ASSERT(pixelFormat.isValid()); - bufferCount = 1; + bufferCount = 2; rawCount++; break; From patchwork Mon Jul 20 09:13:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8867 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9B45EBD792 for ; Mon, 20 Jul 2020 09:13:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4A617605B2; Mon, 20 Jul 2020 11:13:34 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="HC8r2PZZ"; dkim-atps=neutral Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 12093605BB for ; Mon, 20 Jul 2020 11:13:32 +0200 (CEST) Received: by mail-wm1-x32f.google.com with SMTP id 17so24394775wmo.1 for ; Mon, 20 Jul 2020 02:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mlnlsKaKP2HEw7g4p82W82aRH/V0hITuA/ZGL2qH+mQ=; b=HC8r2PZZyjvzHjdYfqMEWR+IlZjy8pAerCNwpwhfBLjadxiodLBymR2CXJ4xUHf+q9 gz4b4IggV0/HUznX2/ZV7xXaPus2HVywCZM6g3FDxkLB0fSxprPOgtlJfGHAUVxWl+VQ sfUEkrgWgy/J3ZT62YIwmWtA330u3IJluI0DmdblqmCutZOPzls23fAUW/Stm1jhzngR g1gboEXVLl2PlAgYt0j0JUoGEoR+ItWy32etkk+6vvxxeHemdbNcDL8d5Xfxh5FE6ksV H3ekQMoLZo4lMzojeOQ7EGiFZRdT6rCXz/ECAD8gSdeKjhQj6WnI5+fm3NYA1BpcQbq7 YuSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mlnlsKaKP2HEw7g4p82W82aRH/V0hITuA/ZGL2qH+mQ=; b=Mb+o0Q/YATYYi5Bm0xK1Srq2siNC5jCHLBANi4S9yQyRUaXhODepSlhFyroExoWVFG UJHhAfW3WnGw8hxGNzkuMxGyZ+sPbCzXEfaDr8Nl0SYnrRPVVcHldjvff2mecD9uZPYW rlYdXWcbAJLu3zez7WYjOIzBBhYh7851Ny+CpGxa4zIHafH6M0b9gmiQV/cTsudMMoOQ Yc4sNf1xYlEGeVnbW3AFddgjx59QGWYCMLECNo6t08O9RCvaQCuG0JZioneL5yjxv5ga caADZfCLcMR0oJ7Eoru6MtwLNRWyLt+bDCF4dhcc1+LhgNKVYaQmKG3DSsSfRKLP1RmL xMIA== X-Gm-Message-State: AOAM531AcPTqvRxdFz4oUXOsy9HDcDoBcwsN+yz7oPBlEotisfgOwWa9 ak0NJH2OKMEgNbYQj01bMddV1kKotxybdw== X-Google-Smtp-Source: ABdhPJzET71JUjMTHS1JHY5Nfvafj7xyfFQcq1HvqRloHYQedmaTniNGNMu9ySIBUBVW6c+Ev1611A== X-Received: by 2002:a1c:f219:: with SMTP id s25mr19938168wmc.2.1595236411517; Mon, 20 Jul 2020 02:13:31 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:30 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:07 +0100 Message-Id: <20200720091311.805092-6-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 5/9] libcamera: pipeline: raspberrypi: Remove const qualifier from RPiStream X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" For the zero-copy RAW capture feature, the RPiStream will have to be modified. Remove the const qualifier in anticipation of the future commits for this feature. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/libcamera/pipeline/raspberrypi/raspberrypi.cpp | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 486345fa..8f6a999b 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -150,7 +150,7 @@ public: void ispOutputDequeue(FrameBuffer *buffer); void clearIncompleteRequests(); - void handleStreamBuffer(FrameBuffer *buffer, const RPi::RPiStream *stream); + void handleStreamBuffer(FrameBuffer *buffer, RPi::RPiStream *stream); void handleState(); CameraSensor *sensor_; @@ -1105,12 +1105,12 @@ done: void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) { - const RPi::RPiStream *stream = nullptr; + RPi::RPiStream *stream = nullptr; if (state_ == State::Stopped) return; - for (RPi::RPiStream const &s : unicam_) { + for (RPi::RPiStream &s : unicam_) { if (s.findFrameBuffer(buffer)) { stream = &s; break; @@ -1171,12 +1171,12 @@ void RPiCameraData::ispInputDequeue(FrameBuffer *buffer) void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer) { - const RPi::RPiStream *stream = nullptr; + RPi::RPiStream *stream = nullptr; if (state_ == State::Stopped) return; - for (RPi::RPiStream const &s : isp_) { + for (RPi::RPiStream &s : isp_) { if (s.findFrameBuffer(buffer)) { stream = &s; break; @@ -1257,7 +1257,7 @@ void RPiCameraData::clearIncompleteRequests() } } -void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, const RPi::RPiStream *stream) +void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, RPi::RPiStream *stream) { if (stream->isExternal()) { if (!dropFrameCount_) { From patchwork Mon Jul 20 09:13:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8868 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2A744BD792 for ; Mon, 20 Jul 2020 09:13:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EB0C16072E; Mon, 20 Jul 2020 11:13:35 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="odvYu4bk"; dkim-atps=neutral Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E7B446072E for ; Mon, 20 Jul 2020 11:13:33 +0200 (CEST) Received: by mail-wm1-x32b.google.com with SMTP id o8so21498769wmh.4 for ; Mon, 20 Jul 2020 02:13:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hvIw1keLnbe7wLTbUm/SfDG3LXHs5sst4RkHnSaxYn0=; b=odvYu4bkDgmFzDTKj8amMOvJSbFTC4bVplhnw882nTZqrb7Zt4PxXq4Atw1U2GTQj7 /1oUM0no/9pwA6FfR+cxAWtgBK0LiWUGK6sy1z/e3YRtlwXoEWnln30m4gpa2NThiZGp kd8kpmU/3PPAzS5cc0OzsXMrer6iwjrCR2SS6gJ2PVloj1t7uFIz331nbfHt1YwNjaqO 5PcbNOwwb1lOYw8kn0ZFoiYYriyiATlJs3ib3FvWsWEQhrChhZq6WtdY//rXwt1VEYtQ I6hx6CMroEsuSNJJfdqzagxaeYB9tPiRN3XsW01/ZxjkGR2YpzkyZ/tNaUhAh0PK9eoq eLxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hvIw1keLnbe7wLTbUm/SfDG3LXHs5sst4RkHnSaxYn0=; b=sRqCpgAkFtPzpGdy0PSX1sNgRGbCAt3ZxH6WREVkpojvymRxy20tiE9bMy0JGbcA7O VXO8CUQ8YpUpDMEje0NH1E8mBEQ3+aY4lEsUre/5E4aFJlgdArfGnn/8fMlFwNpO/Rfw KKe2FBS5Kd5p+uhuidkoPwbt2fMGj2eY5FHoeHWo1z8xVDtgu7AcuGilHpyde5F5xZDJ Xfzy3aewD0dTePUuZ9D2AWd6Ero5ZKgifPkUNHu6OautgQlt/sMUPHFhdGhziEHXg5zC LMm1Ta8jyayBQ6RpEt6j8DKRdt/Es4GA6NaVnjiH7nIwWa/75zN+Px6hl6VPNnNqp7Sy iRLQ== X-Gm-Message-State: AOAM531pauj9X3uk/iVJAEA1zS7CbPvrv87Jmh7Lwax6WZkJcKmc6Wf+ /89JGXvULLcff9SrCPD0heWHvPE04THoKg== X-Google-Smtp-Source: ABdhPJz0AkQmRjXuEoM+JDmhrqDPZCk4ucDyaezRjgnDVPExh7ZlGcef9lzPsv/e6nDdnKdAle+CuQ== X-Received: by 2002:a05:600c:4143:: with SMTP id h3mr21654280wmm.131.1595236412541; Mon, 20 Jul 2020 02:13:32 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:31 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:08 +0100 Message-Id: <20200720091311.805092-7-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 6/9] libcamera: pipeline: raspberrypi: Rework stream buffer logic for zero-copy X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Stop using v4l2_videodevice::allocateBuffer() for internal buffers and instead export/import all buffers. This allows the pipeline to return any stream buffer requested by the application as zero-copy. Advertise the Unicam Image stream as the RAW capture stream now. The RPiStream object now maintains a new list of buffers that are available to queue into a device. This is needed to distinguish between FrameBuffers allocated for internal use vs externally provided buffers. When a Request comes in, if a buffer is not provided for an exported stream, we re-use a buffer from this list. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham --- .../pipeline/raspberrypi/raspberrypi.cpp | 228 ++++++++++-------- .../pipeline/raspberrypi/rpi_stream.cpp | 122 +++++++--- .../pipeline/raspberrypi/rpi_stream.h | 30 ++- 3 files changed, 226 insertions(+), 154 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 8f6a999b..dbc22521 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -187,7 +187,8 @@ private: void checkRequestCompleted(); void tryRunPipeline(); void tryFlushQueues(); - FrameBuffer *updateQueue(std::queue &q, uint64_t timestamp, V4L2VideoDevice *dev); + FrameBuffer *updateQueue(std::queue &q, uint64_t timestamp, + RPi::RPiStream *stream); unsigned int ispOutputCount_; }; @@ -508,8 +509,15 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) StreamConfiguration &cfg = config->at(i); if (isRaw(cfg.pixelFormat)) { - cfg.setStream(&data->isp_[Isp::Input]); - data->isp_[Isp::Input].setExternal(true); + cfg.setStream(&data->unicam_[Unicam::Image]); + /* + * We must set both Unicam streams as external, even + * though the application may only request RAW frames. + * This is because we match timestamps on both streams + * to synchronise buffers. + */ + data->unicam_[Unicam::Image].setExternal(true); + data->unicam_[Unicam::Embedded].setExternal(true); continue; } @@ -612,7 +620,7 @@ int PipelineHandlerRPi::exportFrameBuffers(Camera *camera, Stream *stream, unsigned int count = stream->configuration().bufferCount; int ret = s->dev()->exportBuffers(count, buffers); - s->setExternalBuffers(buffers); + s->setExportedBuffers(buffers); return ret; } @@ -712,14 +720,23 @@ int PipelineHandlerRPi::queueRequestDevice(Camera *camera, Request *request) if (data->state_ == RPiCameraData::State::Stopped) return -EINVAL; - /* Ensure all external streams have associated buffers! */ - for (auto &stream : data->isp_) { - if (!stream.isExternal()) - continue; + LOG(RPI, Debug) << "queueRequestDevice: New request."; - if (!request->findBuffer(&stream)) { - LOG(RPI, Error) << "Attempt to queue request with invalid stream."; - return -ENOENT; + /* Push all buffers supplied in the Request to the respective streams. */ + for (auto stream : data->streams_) { + if (stream->isExternal()) { + FrameBuffer *buffer = request->findBuffer(stream); + /* + * A nullptr in buffer means that we should queue an internally + * allocated buffer. + * + * The below queueBuffer() call will do nothing if there are not + * enough internal buffers allocated, but this will be handled by + * queuing the request for buffers in the RPiStream object. + */ + int ret = stream->queueBuffer(buffer); + if (ret) + return ret; } } @@ -808,12 +825,10 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator) /* Initialize the camera properties. */ data->properties_ = data->sensor_->properties(); - /* - * List the available output streams. - * Currently cannot do Unicam streams! - */ + /*List the available streams an application may request. */ std::set streams; - streams.insert(&data->isp_[Isp::Input]); + streams.insert(&data->unicam_[Unicam::Image]); + streams.insert(&data->unicam_[Unicam::Embedded]); streams.insert(&data->isp_[Isp::Output0]); streams.insert(&data->isp_[Isp::Output1]); streams.insert(&data->isp_[Isp::Stats]); @@ -831,9 +846,28 @@ int PipelineHandlerRPi::queueAllBuffers(Camera *camera) int ret; for (auto const stream : data->streams_) { - ret = stream->queueBuffers(); - if (ret < 0) - return ret; + if (!stream->isExternal()) { + ret = stream->queueAllBuffers(); + if (ret < 0) + return ret; + } else { + /* + * For external streams, we must queue up a set of internal + * buffers to handle the number of drop frames requested by + * the IPA. This is done by passing nullptr in queueBuffer(). + * + * The below queueBuffer() call will do nothing if there + * are not enough internal buffers allocated, but this will + * be handled by queuing the request for buffers in the + * RPiStream object. + */ + unsigned int i; + for (i = 0; i < data->dropFrameCount_; i++) { + int ret = stream->queueBuffer(nullptr); + if (ret) + return ret; + } + } } return 0; @@ -847,7 +881,8 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) /* * Decide how many internal buffers to allocate. For now, simply look * at how many external buffers will be provided. Will need to improve - * this logic. + * this logic. However, we really must have all stream allocate the same + * number of buffers to simplify error handling in queueRequestDevice(). */ unsigned int maxBuffers = 0; for (const Stream *s : camera->streams()) @@ -855,33 +890,9 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) maxBuffers = std::max(maxBuffers, s->configuration().bufferCount); for (auto const stream : data->streams_) { - if (stream->isExternal() || stream->isImporter()) { - /* - * If a stream is marked as external reserve memory to - * prepare to import as many buffers are requested in - * the stream configuration. - * - * If a stream is an internal stream with importer - * role, reserve as many buffers as possible. - */ - unsigned int count = stream->isExternal() - ? stream->configuration().bufferCount - : maxBuffers; - ret = stream->importBuffers(count); - if (ret < 0) - return ret; - } else { - /* - * If the stream is an internal exporter allocate and - * export as many buffers as possible to its internal - * pool. - */ - ret = stream->allocateBuffers(maxBuffers); - if (ret < 0) { - freeBuffers(camera); - return ret; - } - } + ret = stream->prepareBuffers(maxBuffers); + if (ret < 0) + return ret; } /* @@ -889,7 +900,7 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) * the IPA and RPI_IPA_EVENT_SIGNAL_ISP_PREPARE event. */ count = 0; - for (auto const &b : *data->unicam_[Unicam::Image].getBuffers()) { + for (auto const &b : data->unicam_[Unicam::Image].getBuffers()) { b->setCookie(count++); } @@ -898,14 +909,14 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) * the IPA. */ count = 0; - for (auto const &b : *data->isp_[Isp::Stats].getBuffers()) { + for (auto const &b : data->isp_[Isp::Stats].getBuffers()) { b->setCookie(count++); data->ipaBuffers_.push_back({ .id = RPiIpaMask::STATS | b->cookie(), .planes = b->planes() }); } count = 0; - for (auto const &b : *data->unicam_[Unicam::Embedded].getBuffers()) { + for (auto const &b : data->unicam_[Unicam::Embedded].getBuffers()) { b->setCookie(count++); data->ipaBuffers_.push_back({ .id = RPiIpaMask::EMBEDDED_DATA | b->cookie(), .planes = b->planes() }); @@ -1066,7 +1077,7 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData switch (action.operation) { case RPI_IPA_ACTION_STATS_METADATA_COMPLETE: { unsigned int bufferId = action.data[0]; - FrameBuffer *buffer = isp_[Isp::Stats].getBuffers()->at(bufferId).get(); + FrameBuffer *buffer = isp_[Isp::Stats].getBuffers().at(bufferId); handleStreamBuffer(buffer, &isp_[Isp::Stats]); /* Fill the Request metadata buffer with what the IPA has provided */ @@ -1077,19 +1088,19 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData case RPI_IPA_ACTION_EMBEDDED_COMPLETE: { unsigned int bufferId = action.data[0]; - FrameBuffer *buffer = unicam_[Unicam::Embedded].getBuffers()->at(bufferId).get(); + FrameBuffer *buffer = unicam_[Unicam::Embedded].getBuffers().at(bufferId); handleStreamBuffer(buffer, &unicam_[Unicam::Embedded]); break; } case RPI_IPA_ACTION_RUN_ISP: { unsigned int bufferId = action.data[0]; - FrameBuffer *buffer = unicam_[Unicam::Image].getBuffers()->at(bufferId).get(); + FrameBuffer *buffer = unicam_[Unicam::Image].getBuffers().at(bufferId); LOG(RPI, Debug) << "Input re-queue to ISP, buffer id " << buffer->cookie() << ", timestamp: " << buffer->metadata().timestamp; - isp_[Isp::Input].dev()->queueBuffer(buffer); + isp_[Isp::Input].queueBuffer(buffer); ispOutputCount_ = 0; break; } @@ -1190,22 +1201,27 @@ void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer) << ", buffer id " << buffer->cookie() << ", timestamp: " << buffer->metadata().timestamp; - handleStreamBuffer(buffer, stream); - /* - * Increment the number of ISP outputs generated. - * This is needed to track dropped frames. + * ISP statistics buffer must not be re-queued or sent back to the + * application until after the IPA signals so. */ - ispOutputCount_++; - - /* If this is a stats output, hand it to the IPA now. */ if (stream == &isp_[Isp::Stats]) { IPAOperationData op; op.operation = RPI_IPA_EVENT_SIGNAL_STAT_READY; op.data = { RPiIpaMask::STATS | buffer->cookie() }; ipa_->processEvent(op); + } else { + /* Any other ISP output can be handed back to the application now. */ + handleStreamBuffer(buffer, stream); } + /* + * Increment the number of ISP outputs generated. + * This is needed to track dropped frames. + */ + ispOutputCount_++; + + handleState(); } @@ -1218,8 +1234,12 @@ void RPiCameraData::clearIncompleteRequests() */ for (auto const request : requestQueue_) { for (auto const stream : streams_) { - if (stream->isExternal()) - stream->dev()->queueBuffer(request->findBuffer(stream)); + if (!stream->isExternal()) + continue; + + FrameBuffer *buffer = request->findBuffer(stream); + if (buffer) + stream->queueBuffer(buffer); } } @@ -1248,7 +1268,7 @@ void RPiCameraData::clearIncompleteRequests() * Has the buffer already been handed back to the * request? If not, do so now. */ - if (buffer->request()) + if (buffer && buffer->request()) pipe_->completeBuffer(camera_, request, buffer); } @@ -1260,30 +1280,24 @@ void RPiCameraData::clearIncompleteRequests() void RPiCameraData::handleStreamBuffer(FrameBuffer *buffer, RPi::RPiStream *stream) { if (stream->isExternal()) { - if (!dropFrameCount_) { - Request *request = buffer->request(); + Request *request = requestQueue_.front(); + + if (!dropFrameCount_ && request->findBuffer(stream) == buffer) { + /* + * Tag the buffer as completed, returning it to the + * application. + */ pipe_->completeBuffer(camera_, request, buffer); + } else { + /* + * This buffer was not part of the Request, so we can + * recycle it. + */ + stream->returnBuffer(buffer); } } else { - /* Special handling for RAW buffer Requests. - * - * The ISP input stream is alway an import stream, but if the - * current Request has been made for a buffer on the stream, - * simply memcpy to the Request buffer and requeue back to the - * device. - */ - if (stream == &unicam_[Unicam::Image] && !dropFrameCount_) { - const Stream *rawStream = static_cast(&isp_[Isp::Input]); - Request *request = requestQueue_.front(); - FrameBuffer *raw = request->findBuffer(const_cast(rawStream)); - if (raw) { - raw->copyFrom(buffer); - pipe_->completeBuffer(camera_, request, raw); - } - } - - /* Simply requeue the buffer. */ - stream->dev()->queueBuffer(buffer); + /* Simply re-queue the buffer to the requested stream. */ + stream->queueBuffer(buffer); } } @@ -1367,7 +1381,7 @@ void RPiCameraData::tryRunPipeline() * current bayer buffer will be removed and re-queued to the driver. */ embeddedBuffer = updateQueue(embeddedQueue_, bayerBuffer->metadata().timestamp, - unicam_[Unicam::Embedded].dev()); + &unicam_[Unicam::Embedded]); if (!embeddedBuffer) { LOG(RPI, Debug) << "Could not find matching embedded buffer"; @@ -1386,7 +1400,7 @@ void RPiCameraData::tryRunPipeline() embeddedBuffer = embeddedQueue_.front(); bayerBuffer = updateQueue(bayerQueue_, embeddedBuffer->metadata().timestamp, - unicam_[Unicam::Image].dev()); + &unicam_[Unicam::Image]); if (!bayerBuffer) { LOG(RPI, Debug) << "Could not find matching bayer buffer - ending."; @@ -1394,11 +1408,7 @@ void RPiCameraData::tryRunPipeline() } } - /* - * Take the first request from the queue and action the IPA. - * Unicam buffers for the request have already been queued as they come - * in. - */ + /* Take the first request from the queue and action the IPA. */ Request *request = requestQueue_.front(); /* @@ -1410,12 +1420,6 @@ void RPiCameraData::tryRunPipeline() op.controls = { request->controls() }; ipa_->processEvent(op); - /* Queue up any ISP buffers passed into the request. */ - for (auto &stream : isp_) { - if (stream.isExternal()) - stream.dev()->queueBuffer(request->findBuffer(&stream)); - } - /* Ready to use the buffers, pop them off the queue. */ bayerQueue_.pop(); embeddedQueue_.pop(); @@ -1445,32 +1449,42 @@ void RPiCameraData::tryFlushQueues() * and give a chance for the hardware to return to lock-step. We do have * to drop all interim frames. */ - if (unicam_[Unicam::Image].getBuffers()->size() == bayerQueue_.size() && - unicam_[Unicam::Embedded].getBuffers()->size() == embeddedQueue_.size()) { + if (unicam_[Unicam::Image].getBuffers().size() == bayerQueue_.size() && + unicam_[Unicam::Embedded].getBuffers().size() == embeddedQueue_.size()) { + /* This cannot happen when Unicam streams are external. */ + assert(!unicam_[Unicam::Image].isExternal()); + LOG(RPI, Warning) << "Flushing all buffer queues!"; while (!bayerQueue_.empty()) { - unicam_[Unicam::Image].dev()->queueBuffer(bayerQueue_.front()); + unicam_[Unicam::Image].queueBuffer(bayerQueue_.front()); bayerQueue_.pop(); } while (!embeddedQueue_.empty()) { - unicam_[Unicam::Embedded].dev()->queueBuffer(embeddedQueue_.front()); + unicam_[Unicam::Embedded].queueBuffer(embeddedQueue_.front()); embeddedQueue_.pop(); } } } FrameBuffer *RPiCameraData::updateQueue(std::queue &q, uint64_t timestamp, - V4L2VideoDevice *dev) + RPi::RPiStream *stream) { + /* + * If the unicam streams are external (both have to the same), then we + * can only return out the top buffer in the queue, and assume they have + * been synced by queuing at the same time. We cannot drop these frames, + * as they may have been provided externally. + */ while (!q.empty()) { FrameBuffer *b = q.front(); - if (b->metadata().timestamp < timestamp) { + if (!stream->isExternal() && b->metadata().timestamp < timestamp) { q.pop(); - dev->queueBuffer(b); - LOG(RPI, Warning) << "Dropping input frame!"; - } else if (b->metadata().timestamp == timestamp) { + stream->queueBuffer(b); + LOG(RPI, Warning) << "Dropping unmatched input frame in stream " + << stream->name(); + } else if (stream->isExternal() || b->metadata().timestamp == timestamp) { /* The calling function will pop the item from the queue. */ return b; } else { diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp index 2edb8b59..02f8d3e0 100644 --- a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp @@ -21,30 +21,20 @@ V4L2VideoDevice *RPiStream::dev() const void RPiStream::setExternal(bool external) { + /* Import streams cannot be external. */ + assert(!external || !importOnly_); external_ = external; } bool RPiStream::isExternal() const { - /* - * Import streams cannot be external. - * - * RAW capture is a special case where we simply copy the RAW - * buffer out of the request. All other buffer handling happens - * as if the stream is internal. - */ - return external_ && !importOnly_; -} - -bool RPiStream::isImporter() const -{ - return importOnly_; + return external_; } void RPiStream::reset() { external_ = false; - internalBuffers_.clear(); + clearBuffers(); } std::string RPiStream::name() const @@ -52,65 +42,123 @@ std::string RPiStream::name() const return name_; } -void RPiStream::setExternalBuffers(std::vector> *buffers) +void RPiStream::setExportedBuffers(std::vector> *buffers) { - externalBuffers_ = buffers; + std::transform(buffers->begin(), buffers->end(), std::back_inserter(bufferList_), + [](std::unique_ptr &b) { return b.get(); }); } -const std::vector> *RPiStream::getBuffers() const +const std::vector &RPiStream::getBuffers() const { - return external_ ? externalBuffers_ : &internalBuffers_; + return bufferList_; } void RPiStream::releaseBuffers() { dev_->releaseBuffers(); - if (!external_ && !importOnly_) - internalBuffers_.clear(); + clearBuffers(); } -int RPiStream::importBuffers(unsigned int count) +int RPiStream::prepareBuffers(unsigned int count) { + int ret; + + if (!importOnly_) { + if (count) { + /* Export some frame buffers for internal use. */ + ret = dev_->exportBuffers(count, &internalBuffers_); + if (ret < 0) + return ret; + + /* Add these exported buffers to the internal/external buffer list. */ + setExportedBuffers(&internalBuffers_); + + /* Add these buffers to the queue of internal usable buffers. */ + for (auto const &buffer : internalBuffers_) + availableBuffers_.push(buffer.get()); + } + + /* We must import all internal/external exported buffers. */ + count = bufferList_.size(); + } + return dev_->importBuffers(count); } -int RPiStream::allocateBuffers(unsigned int count) +int RPiStream::queueAllBuffers() { - return dev_->allocateBuffers(count, &internalBuffers_); -} + int ret; -int RPiStream::queueBuffers() -{ if (external_) return 0; - for (auto &b : internalBuffers_) { - int ret = dev_->queueBuffer(b.get()); - if (ret) { - LOG(RPISTREAM, Error) << "Failed to queue buffers for " - << name_; + while (!availableBuffers_.empty()) { + ret = queueBuffer(availableBuffers_.front()); + if (ret < 0) return ret; - } + + availableBuffers_.pop(); } return 0; } -bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const +int RPiStream::queueBuffer(FrameBuffer *buffer) +{ + /* + * A nullptr buffer implies an external stream, but no external + * buffer has been supplied. So, pick one from the availableBuffers_ + * queue. + */ + if (!buffer) { + if (availableBuffers_.empty()) { + LOG(RPISTREAM, Warning) << "No buffers available for " + << name_; + return -EINVAL; + } + + buffer = availableBuffers_.front(); + availableBuffers_.pop(); + } + + LOG(RPISTREAM, Debug) << "Queuing buffer " << buffer->cookie() + << " for " << name_; + + int ret = dev_->queueBuffer(buffer); + if (ret) { + LOG(RPISTREAM, Error) << "Failed to queue buffer for " + << name_; + } + + return ret; +} + +void RPiStream::returnBuffer(FrameBuffer *buffer) { - auto start = external_ ? externalBuffers_->begin() : internalBuffers_.begin(); - auto end = external_ ? externalBuffers_->end() : internalBuffers_.end(); + /* This can only be called for external streams. */ + assert(external_); + + availableBuffers_.push(buffer); +} +bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const +{ if (importOnly_) return false; - if (std::find_if(start, end, - [buffer](std::unique_ptr const &ref) { return ref.get() == buffer; }) != end) + if (std::find(bufferList_.begin(), bufferList_.end(), buffer) != bufferList_.end()) return true; return false; } +void RPiStream::clearBuffers() +{ + availableBuffers_ = std::queue{}; + internalBuffers_.clear(); + bufferList_.clear(); +} + } /* namespace RPi */ } /* namespace libcamera */ diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.h b/src/libcamera/pipeline/raspberrypi/rpi_stream.h index 3957e342..af9c2ad2 100644 --- a/src/libcamera/pipeline/raspberrypi/rpi_stream.h +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.h @@ -39,21 +39,22 @@ public: V4L2VideoDevice *dev() const; void setExternal(bool external); bool isExternal() const; - bool isImporter() const; void reset(); std::string name() const; - void setExternalBuffers(std::vector> *buffers); - const std::vector> *getBuffers() const; + void setExportedBuffers(std::vector> *buffers); + const std::vector &getBuffers() const; void releaseBuffers(); - int importBuffers(unsigned int count); - int allocateBuffers(unsigned int count); - int queueBuffers(); + int prepareBuffers(unsigned int count); + int queueAllBuffers(); + int queueBuffer(FrameBuffer *buffer); + void returnBuffer(FrameBuffer *buffer); bool findFrameBuffer(FrameBuffer *buffer) const; private: + void clearBuffers(); /* * Indicates that this stream is active externally, i.e. the buffers - * are provided by the application. + * might be provided by (and returned to) the application. */ bool external_; /* Indicates that this stream only imports buffers, e.g. ISP input. */ @@ -62,10 +63,19 @@ private: std::string name_; /* The actual device stream. */ std::unique_ptr dev_; - /* Internally allocated framebuffers associated with this device stream. */ + /* All framebuffers associated with this device stream. */ + std::vector bufferList_; + /* + * List of frame buffer that we can use if none have been provided by + * the application for external streams. This is populated by the + * buffers exported internally. + */ + std::queue availableBuffers_; + /* + * This is a list of buffers exported internally. Need to keep this around + * as the stream needs to maintain ownership of these buffers. + */ std::vector> internalBuffers_; - /* Externally allocated framebuffers associated with this device stream. */ - std::vector> *externalBuffers_; }; /* From patchwork Mon Jul 20 09:13:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8869 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A5503BD792 for ; Mon, 20 Jul 2020 09:13:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 591CB60724; Mon, 20 Jul 2020 11:13:36 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="QhLoulLM"; dkim-atps=neutral Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2E33E60732 for ; Mon, 20 Jul 2020 11:13:34 +0200 (CEST) Received: by mail-wr1-x42c.google.com with SMTP id f7so17068602wrw.1 for ; Mon, 20 Jul 2020 02:13:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0ZvNILQsT/lrTa1SsqTTVdBjGKFPz1KeBl7JphindJQ=; b=QhLoulLMa2cNuszm97/RgrOnersc41RquF1jSChsns6+WaiPYOaxcuxAV0mjyI5LDJ 1pc+YKBx+jGszvaAEOTn7QbENySyIrrsvTh3odRIjDpkZfcL4B+7KMzXRYBDm3g85/SQ sfI9bxdk/aJ7AGVyiWajX2C7o8vOjvFgzQKSJr9mA9lwaQEDlrpiAkBFkKPSelKOkdHJ YzawiyXI0/ZLR/zd1q0VpOw9N2e238tlG9eJdAiV8mXy3zikbQeMFkC4MrJIJ5gMr9rI kWQA+U9eKMwlzMwRxRmQ8T3VyM27C7xEgWMkqhLFx4ANhTro179ThMg1EemWBx7niEEm fn+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0ZvNILQsT/lrTa1SsqTTVdBjGKFPz1KeBl7JphindJQ=; b=XIGm1xhthNXI5XpbuRAIt83s1/CAIJJ2P4RkaXw5YOCambzxSZPY+T0KbO09PUiwDy ozP9KYqt5u7aeU87/laLJFH7IutRywlsF68dlQ4psF1P6EJIN7tV0qIWvRaOK4RZDXF5 u/XxF6r8WW19+b+eIs55O431dHxrsCntt2VJOMeh6NslcQgyVleU0gWftIAc+B9EcgJv cHf+WiHLlTXUiNCcxXqTQlIXOKwvVGXrZVG63HuDe+RR4koJTjenGL4f4qYJfLQBrZGC bBDwaHLC/laQPktejDZU1RjZ6djh+1XycXUoAtbIUhj59qkxXJGXEv6ZvkOqg9chPWl/ vOEA== X-Gm-Message-State: AOAM532avDF5Bj7xqX3ByZnoswJo69VgGkMploBuuwaOhraJvXGCwsRn QlbsttCpSO0vaGVCW6jRqH1QYDH3fa1XNw== X-Google-Smtp-Source: ABdhPJzYkngW4Mq30toJwbQzFbs+7+h7Y9SVRvCJ/F7G+mqvmAr8LLrVwKsV8cbGtYv2DLi6S/Molw== X-Received: by 2002:a05:6000:124c:: with SMTP id j12mr4455843wrx.83.1595236413431; Mon, 20 Jul 2020 02:13:33 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:32 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:09 +0100 Message-Id: <20200720091311.805092-8-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 7/9] libcamera: pipeline: raspberrypi: Fix bug in passing configuration to IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The counter was not incremented, so multiple streams would only pass the last stream config to the IPA. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/libcamera/pipeline/raspberrypi/raspberrypi.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index dbc22521..c28fe997 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -976,14 +976,14 @@ int RPiCameraData::configureIPA() unicam_[Unicam::Image].dev()->getFormat(&sensorFormat); /* Inform IPA of stream configuration and sensor controls. */ unsigned int i = 0; - for (auto const &stream : isp_) { + for (auto const &stream : isp_) if (stream.isExternal()) { - streamConfig[i] = { + streamConfig[i++] = { .pixelFormat = stream.configuration().pixelFormat, .size = stream.configuration().size }; } - } + entityControls.emplace(0, unicam_[Unicam::Image].dev()->controls()); entityControls.emplace(1, isp_[Isp::Input].dev()->controls()); From patchwork Mon Jul 20 09:13:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8870 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 93787BD792 for ; Mon, 20 Jul 2020 09:13:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6203B605C9; Mon, 20 Jul 2020 11:13:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="m4dxU722"; dkim-atps=neutral Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7EEBE605BB for ; Mon, 20 Jul 2020 11:13:35 +0200 (CEST) Received: by mail-wm1-x330.google.com with SMTP id c80so21520478wme.0 for ; Mon, 20 Jul 2020 02:13:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xz6y/lKwinaj+jtgprrtx2kUePux8NUENK0+5gnGFGw=; b=m4dxU7225Xt5amA0nuZLMvradiY87iiTtMDfxEs+zpudAUM05DBUx6cJm4lPM9IClE 7O9w+422RV2kZ6fv5vC71JW6UXw/6tFhn7QUCaN/JxlrSgGYWjA27Wk5nHUIuSdSTQ6W mybSzdqqTa75uqQkACmhTstua34fePYPRPH00dRx4N3vEKr1dPRldxP+nyi7EmfxgyBN pz5v0wr5PnTAYZtkXaLDlZVMWYyQZafcE7yIDVkZ0P+rHXFrLzmUou0Agj/jG95AGPu3 PadZRjw5KHGMsUfBKE1UPfQlv+b+Bi0SMsTFO7uzwQHteCyccCdKdrAJBUoYz7d5A3yz oKGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xz6y/lKwinaj+jtgprrtx2kUePux8NUENK0+5gnGFGw=; b=lDcAIbmzXZwyF6f49US074wRfWQFxJo53vUuzxTvPrR9d+NdmJFIOwqh0hrbXzzVrf H+0aOlPtKKhbvcRRwMHwxbZXDjpRWXx5Cuq18lPHaR6UNQPo8epitdv0fWUf0ZYX71xU UVUFP3cTReaiRJnbLggxS986Y/Fy1akN3zbzudLT+ZCnN/kyihL+S+Lxg4jLpzw+NA7C s8PXCnzswtBcMagb3IDWkyhnch9JzG/VNxrEmcYMnVd0gK4G+DCiOGdnUWFMTQOER8ux UfsZA+QPUpe1damvqNd1xisaHMZout0f4j1W2n3IR1k01DHsx0Vj0HhC8XVl7iWI2QIO r6dg== X-Gm-Message-State: AOAM530js1G3Up+UpJSlKRcuBLeG2H1aRoOJQu/+hE6UFOR1R9B2G8ai twHEvHVI1RqfQreX5iHvDCNtyTxXsbG+iQ== X-Google-Smtp-Source: ABdhPJwsVwKrq77BlXk+pCoVfs6NXg5osAwBBkN7H3X3G3cfoF06FsMevKWerSgg0iZWU89W8PTbRw== X-Received: by 2002:a1c:1f54:: with SMTP id f81mr20484254wmf.4.1595236414662; Mon, 20 Jul 2020 02:13:34 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:34 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:10 +0100 Message-Id: <20200720091311.805092-9-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 8/9] libcamera: pipeline: raspberrypi: Add more robust stream buffer logic X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add further queueing into the RPiStream object to ensure that we always follow the buffer ordering (be it internal or external) given by incoming Requests. This is essential, otherwise we risk dropping frames that are meant to be part of a Request, and can cause the pipeline to stall indefinitely. This also prevents any possibility of mismatched frame buffers going through the pipeline and out to the application. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund --- .../pipeline/raspberrypi/rpi_stream.cpp | 70 ++++++++++++++++--- .../pipeline/raspberrypi/rpi_stream.h | 12 +++- 2 files changed, 71 insertions(+), 11 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp index 02f8d3e0..6635a751 100644 --- a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp @@ -112,25 +112,34 @@ int RPiStream::queueBuffer(FrameBuffer *buffer) */ if (!buffer) { if (availableBuffers_.empty()) { - LOG(RPISTREAM, Warning) << "No buffers available for " + LOG(RPISTREAM, Info) << "No buffers available for " << name_; - return -EINVAL; + /* + * Note that we need to requeue an internal buffer as soon + * as one becomes available. + */ + requeueBuffers_.push(nullptr); + return 0; } buffer = availableBuffers_.front(); availableBuffers_.pop(); } - LOG(RPISTREAM, Debug) << "Queuing buffer " << buffer->cookie() - << " for " << name_; + /* + * If no earlier requests are pending to be queued we can go ahead and + * queue the buffer into the device. + */ + if (requeueBuffers_.empty()) + return queueToDevice(buffer); - int ret = dev_->queueBuffer(buffer); - if (ret) { - LOG(RPISTREAM, Error) << "Failed to queue buffer for " - << name_; - } + /* + * There are earlier buffers to be queued, so this bufferm ust go on + * the waiting list. + */ + requeueBuffers_.push(buffer); - return ret; + return 0; } void RPiStream::returnBuffer(FrameBuffer *buffer) @@ -138,7 +147,35 @@ void RPiStream::returnBuffer(FrameBuffer *buffer) /* This can only be called for external streams. */ assert(external_); + /* Push this buffer back into the queue to be used again. */ availableBuffers_.push(buffer); + + /* + * Do we have any buffers that are waiting to be queued? + * If so, do it now as availableBuffers_ will not be empty. + */ + while (!requeueBuffers_.empty()) { + FrameBuffer *buffer = requeueBuffers_.front(); + + if (!buffer) { + /* + * We want to queue an internal buffer, but none + * are available. Can't do anything, quit the loop. + */ + if (availableBuffers_.empty()) + break; + + /* + * We want to queue an internal buffer, and at least one + * is available. + */ + buffer = availableBuffers_.front(); + availableBuffers_.pop(); + } + + requeueBuffers_.pop(); + queueToDevice(buffer); + } } bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const @@ -155,10 +192,23 @@ bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const void RPiStream::clearBuffers() { availableBuffers_ = std::queue{}; + requeueBuffers_ = std::queue{}; internalBuffers_.clear(); bufferList_.clear(); } +int RPiStream::queueToDevice(FrameBuffer *buffer) +{ + LOG(RPISTREAM, Debug) << "Queuing buffer " << buffer->cookie() + << " for " << name_; + + int ret = dev_->queueBuffer(buffer); + if (ret) + LOG(RPISTREAM, Error) << "Failed to queue buffer for " + << name_; + return ret; +} + } /* namespace RPi */ } /* namespace libcamera */ diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.h b/src/libcamera/pipeline/raspberrypi/rpi_stream.h index af9c2ad2..6222c867 100644 --- a/src/libcamera/pipeline/raspberrypi/rpi_stream.h +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.h @@ -52,6 +52,7 @@ public: private: void clearBuffers(); + int queueToDevice(FrameBuffer *buffer); /* * Indicates that this stream is active externally, i.e. the buffers * might be provided by (and returned to) the application. @@ -63,7 +64,7 @@ private: std::string name_; /* The actual device stream. */ std::unique_ptr dev_; - /* All framebuffers associated with this device stream. */ + /* All frame buffers associated with this device stream. */ std::vector bufferList_; /* * List of frame buffer that we can use if none have been provided by @@ -71,6 +72,15 @@ private: * buffers exported internally. */ std::queue availableBuffers_; + /* + * List of frame buffer that are to be re-queued into the device. + * A nullptr indicates any internal buffer can be used (from availableBuffers_), + * whereas a valid pointer indicates an external buffer to be queued. + * + * Ordering buffers to be queued is important here as it must match the + * requests coming from the application. + */ + std::queue requeueBuffers_; /* * This is a list of buffers exported internally. Need to keep this around * as the stream needs to maintain ownership of these buffers. From patchwork Mon Jul 20 09:13:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 8871 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id ECD82C2E67 for ; Mon, 20 Jul 2020 09:13:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B5A8B6071A; Mon, 20 Jul 2020 11:13:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="oy0Lqf0k"; dkim-atps=neutral Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3C83060740 for ; Mon, 20 Jul 2020 11:13:36 +0200 (CEST) Received: by mail-wr1-x42f.google.com with SMTP id o11so17044281wrv.9 for ; Mon, 20 Jul 2020 02:13:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Duv/Vjlm7mcp+9mCFKeMyXs8naSwOJ+ashD0v9Kh0ps=; b=oy0Lqf0ko68LkZcj9Qk5AjcS7MWMG+EW05Wj+AmXG/VC/N9NQxmGnxtkGA0QBWwRlh EitcSdkkVi/frGE4+HQa6D1BUosLPyFp/1b6dDywMOWOJ1hC6zAZwL6bKamfHx9Rt39t fUlxOJHj+Tu994aUJHTk0B5bNvUFkJ0cNTUQ6NURKjFPaEIiuQ9DWVTBv5bD0R3PE/bm mTwjHmEqYTldVSJAjD5Co+aU6sQu7YLn3qJH0rlBtC+qN/2P9uXPKUDJ2qlzmOzg4KBS /nYHA5GHJLUp/kjJhXQDqOffbrXjnWJPe1XYNOD/xPx44weJzQgOG5vUT1EagCyPK4u9 nx/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Duv/Vjlm7mcp+9mCFKeMyXs8naSwOJ+ashD0v9Kh0ps=; b=ma8Z4Bg8tMOcFiR+qfK+3vRcC2rdConvkmYNM2lYkl8MRsbBhxBvbVyzoWC+crlwny Q0rwrQj30u/DeXPCt0/gJwQMcjlBFDrcDexAoKF0W+cvmTNQTQ1lSw6fCQybgXbZWpnS 2rka23JXideOVX1bGRJ0f8g3CvFSntVgHjACMsSVY/HAeH1koqQ77c8eNePUn5jhTyZW ip7Pm2u94toeKwZGJNRfEZHUiR1NkC7I33KfmX6XjlLKAUKk3mrWH0al7VFW3lVUeP85 n+Ch0buFnscrTc2MQeZ/T50SL0Uqkn2uZz2B2dvo9ehdg2zAPKdnD6VK0Em5J9ckmCh7 0MWw== X-Gm-Message-State: AOAM533OO2/nGMmblJAtllQ+DtfLtjwDIYIx/BK1TWXy5+j9nMfYyE/w eoD5mW9JTpBEMY9qKqllHuVgbeY5Am0IvQ== X-Google-Smtp-Source: ABdhPJzY2MYM54nK7JsQTH21hc9cN/QVym+AaA0MP89GtsWQK7GsBlF/kDxLFBFDP1KJIZE0dn3LfQ== X-Received: by 2002:adf:d08d:: with SMTP id y13mr1017446wrh.313.1595236415487; Mon, 20 Jul 2020 02:13:35 -0700 (PDT) Received: from naushir-VirtualBox.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id u1sm38137185wrb.78.2020.07.20.02.13.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 02:13:34 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jul 2020 10:13:11 +0100 Message-Id: <20200720091311.805092-10-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200720091311.805092-1-naush@raspberrypi.com> References: <20200720091311.805092-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 9/9] libcamera: pipeline: ipa: raspberrypi: Remove use of FrameBuffer cookie X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The FrameBuffer cookie may be set by the application, so this cannot be set by the pipeline handler as well. Revert to using a simple index into the buffer list to identify buffers passing to and from the IPA. Signed-off-by: Naushir Patuck Reviewed-by: Niklas Söderlund Reviewed-by: Kieran Bingham --- .../pipeline/raspberrypi/raspberrypi.cpp | 60 ++++++++++--------- .../pipeline/raspberrypi/rpi_stream.cpp | 14 +++-- .../pipeline/raspberrypi/rpi_stream.h | 2 +- 3 files changed, 40 insertions(+), 36 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index c28fe997..e4fc5ac7 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -876,7 +876,8 @@ int PipelineHandlerRPi::queueAllBuffers(Camera *camera) int PipelineHandlerRPi::prepareBuffers(Camera *camera) { RPiCameraData *data = cameraData(camera); - int count, ret; + unsigned int index; + int ret; /* * Decide how many internal buffers to allocate. For now, simply look @@ -896,30 +897,24 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) } /* - * Add cookies to the ISP Input buffers so that we can link them with - * the IPA and RPI_IPA_EVENT_SIGNAL_ISP_PREPARE event. - */ - count = 0; - for (auto const &b : data->unicam_[Unicam::Image].getBuffers()) { - b->setCookie(count++); - } - - /* - * Add cookies to the stats and embedded data buffers and link them with - * the IPA. + * Link the FrameBuffers with the index of their position in the vector + * stored in the RPi stream object. + * + * This will allow us to identify buffers passed between the pipeline + * handler and the IPA. */ - count = 0; + index = 0; for (auto const &b : data->isp_[Isp::Stats].getBuffers()) { - b->setCookie(count++); - data->ipaBuffers_.push_back({ .id = RPiIpaMask::STATS | b->cookie(), + data->ipaBuffers_.push_back({ .id = RPiIpaMask::STATS | index, .planes = b->planes() }); + index++; } - count = 0; + index = 0; for (auto const &b : data->unicam_[Unicam::Embedded].getBuffers()) { - b->setCookie(count++); - data->ipaBuffers_.push_back({ .id = RPiIpaMask::EMBEDDED_DATA | b->cookie(), + data->ipaBuffers_.push_back({ .id = RPiIpaMask::EMBEDDED_DATA | index, .planes = b->planes() }); + index++; } data->ipa_->mapBuffers(data->ipaBuffers_); @@ -1097,7 +1092,7 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData unsigned int bufferId = action.data[0]; FrameBuffer *buffer = unicam_[Unicam::Image].getBuffers().at(bufferId); - LOG(RPI, Debug) << "Input re-queue to ISP, buffer id " << buffer->cookie() + LOG(RPI, Debug) << "Input re-queue to ISP, buffer id " << bufferId << ", timestamp: " << buffer->metadata().timestamp; isp_[Isp::Input].queueBuffer(buffer); @@ -1117,12 +1112,14 @@ done: void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) { RPi::RPiStream *stream = nullptr; + int index; if (state_ == State::Stopped) return; for (RPi::RPiStream &s : unicam_) { - if (s.findFrameBuffer(buffer)) { + index = s.getBufferIndex(buffer); + if (index != -1) { stream = &s; break; } @@ -1132,7 +1129,7 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) ASSERT(stream); LOG(RPI, Debug) << "Stream " << stream->name() << " buffer dequeue" - << ", buffer id " << buffer->cookie() + << ", buffer id " << index << ", timestamp: " << buffer->metadata().timestamp; if (stream == &unicam_[Unicam::Image]) { @@ -1172,7 +1169,7 @@ void RPiCameraData::ispInputDequeue(FrameBuffer *buffer) return; LOG(RPI, Debug) << "Stream ISP Input buffer complete" - << ", buffer id " << buffer->cookie() + << ", buffer id " << unicam_[Unicam::Image].getBufferIndex(buffer) << ", timestamp: " << buffer->metadata().timestamp; /* The ISP input buffer gets re-queued into Unicam. */ @@ -1183,12 +1180,14 @@ void RPiCameraData::ispInputDequeue(FrameBuffer *buffer) void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer) { RPi::RPiStream *stream = nullptr; + int index; if (state_ == State::Stopped) return; for (RPi::RPiStream &s : isp_) { - if (s.findFrameBuffer(buffer)) { + index = s.getBufferIndex(buffer); + if (index != -1) { stream = &s; break; } @@ -1198,7 +1197,7 @@ void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer) ASSERT(stream); LOG(RPI, Debug) << "Stream " << stream->name() << " buffer complete" - << ", buffer id " << buffer->cookie() + << ", buffer id " << index << ", timestamp: " << buffer->metadata().timestamp; /* @@ -1208,7 +1207,7 @@ void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer) if (stream == &isp_[Isp::Stats]) { IPAOperationData op; op.operation = RPI_IPA_EVENT_SIGNAL_STAT_READY; - op.data = { RPiIpaMask::STATS | buffer->cookie() }; + op.data = { RPiIpaMask::STATS | static_cast(index) }; ipa_->processEvent(op); } else { /* Any other ISP output can be handed back to the application now. */ @@ -1427,13 +1426,16 @@ void RPiCameraData::tryRunPipeline() /* Set our state to say the pipeline is active. */ state_ = State::Busy; + unsigned int bayerIndex = unicam_[Unicam::Image].getBufferIndex(bayerBuffer); + unsigned int embeddedIndex = unicam_[Unicam::Embedded].getBufferIndex(embeddedBuffer); + LOG(RPI, Debug) << "Signalling RPI_IPA_EVENT_SIGNAL_ISP_PREPARE:" - << " Bayer buffer id: " << bayerBuffer->cookie() - << " Embedded buffer id: " << embeddedBuffer->cookie(); + << " Bayer buffer id: " << bayerIndex + << " Embedded buffer id: " << embeddedIndex; op.operation = RPI_IPA_EVENT_SIGNAL_ISP_PREPARE; - op.data = { RPiIpaMask::EMBEDDED_DATA | embeddedBuffer->cookie(), - RPiIpaMask::BAYER_DATA | bayerBuffer->cookie() }; + op.data = { RPiIpaMask::EMBEDDED_DATA | embeddedIndex, + RPiIpaMask::BAYER_DATA | bayerIndex }; ipa_->processEvent(op); } diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp index 6635a751..73eb04a1 100644 --- a/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.cpp @@ -178,15 +178,17 @@ void RPiStream::returnBuffer(FrameBuffer *buffer) } } -bool RPiStream::findFrameBuffer(FrameBuffer *buffer) const +int RPiStream::getBufferIndex(FrameBuffer *buffer) const { if (importOnly_) - return false; + return -1; - if (std::find(bufferList_.begin(), bufferList_.end(), buffer) != bufferList_.end()) - return true; + /* Find the buffer in the list, and return the index position. */ + auto it = std::find(bufferList_.begin(), bufferList_.end(), buffer); + if (it != bufferList_.end()) + return std::distance(bufferList_.begin(), it); - return false; + return -1; } void RPiStream::clearBuffers() @@ -199,7 +201,7 @@ void RPiStream::clearBuffers() int RPiStream::queueToDevice(FrameBuffer *buffer) { - LOG(RPISTREAM, Debug) << "Queuing buffer " << buffer->cookie() + LOG(RPISTREAM, Debug) << "Queuing buffer " << getBufferIndex(buffer) << " for " << name_; int ret = dev_->queueBuffer(buffer); diff --git a/src/libcamera/pipeline/raspberrypi/rpi_stream.h b/src/libcamera/pipeline/raspberrypi/rpi_stream.h index 6222c867..45cf5237 100644 --- a/src/libcamera/pipeline/raspberrypi/rpi_stream.h +++ b/src/libcamera/pipeline/raspberrypi/rpi_stream.h @@ -48,7 +48,7 @@ public: int queueAllBuffers(); int queueBuffer(FrameBuffer *buffer); void returnBuffer(FrameBuffer *buffer); - bool findFrameBuffer(FrameBuffer *buffer) const; + int getBufferIndex(FrameBuffer *buffer) const; private: void clearBuffers();