From patchwork Mon May 10 09:58:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 12239 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2382DBF829 for ; Mon, 10 May 2021 09:59:14 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B47C668926; Mon, 10 May 2021 11:59:13 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="inIbQJsR"; dkim-atps=neutral Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BC90D6890C for ; Mon, 10 May 2021 11:59:10 +0200 (CEST) Received: by mail-wm1-x333.google.com with SMTP id y124-20020a1c32820000b029010c93864955so10787448wmy.5 for ; Mon, 10 May 2021 02:59:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fmFRyvKG40VFNgvY7s5IVKLxkKqN5DwxSDI/eMitC4I=; b=inIbQJsRbdaz3bZNW+8Ay47vaaALuQDwvMCvX5cjzMjE9KpQAdCD7//rVc7AfQy7vJ AySjpD4vnMdoTLuHz7bRVxhbCrCThtjCcJ7LN8xzA/dKYlq7DuD6Rq9nor9+EL4u1jbO oDx8ytdcZZIqq6fVEMGj5gacP6QT0+oAEv/4hmOgLkEnI4XfhDFE6p25toqQjv75WNJ7 RmOMEVNZ7oFYY2V9FAkkCVaxYfS0ts9UZ8kzTLUft//HbR+Wu6Hx17WyXyIFaxhOcA5D j0vh/PJ+EgqN2bX1WEfMnsCBvWCjVLDIjwqSvvC7mmTE0HE09WM8CozxwW5sepCoO5fS bQIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fmFRyvKG40VFNgvY7s5IVKLxkKqN5DwxSDI/eMitC4I=; b=nNqx2JH/7NwB4AIFhIXNNG7TsQr5vzTa3j2e2nb49FRbf/ET66x/EEM0pFt7EdRhJd eW0z8zVie3QGBxVazc644x+/RZe1fLCFRjFMzeXBYSRfZhNN79Tlw+2GN++wQqYmck3G X1ccXJdOqIcvQC8JaNnTco0WiSmyFOnMBE2//1GWRdjkmGBpV7AT7JADo5UCPwa6zoEk BauG2Z0/Vlk9l4fkk/wNsaKQyl53tLuAXmcJMMgbIT341qUErY6RrYc8jwX/5lOyT63Y oH9QPvhN75TZFBxXmg9I+jfmjLrHi1NxK5Pm+sWNv+zGXd7TGz4z8cu0sT+RwFPJzXYz bkdw== X-Gm-Message-State: AOAM531+Ih6HC3dk1/wSO4DwvO5Dvk/pf22U1+T65VNf0Jh6kGUSeFgi ms2Tx5+PTCFgAgBH5UM7+j/Xt9ekx/whJw== X-Google-Smtp-Source: ABdhPJzoCHHfMCK6qULSshNl970toyxie62p6nni30TannPChpFWn/k1JouwmEUB1+al+ftJyaTzXQ== X-Received: by 2002:a1c:7402:: with SMTP id p2mr35839969wmc.88.1620640750116; Mon, 10 May 2021 02:59:10 -0700 (PDT) Received: from naush-laptop.pitowers.org ([2a00:1098:3142:14:e710:3ec:6dad:dc2]) by smtp.gmail.com with ESMTPSA id y17sm24767917wrw.90.2021.05.10.02.59.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 May 2021 02:59:09 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Mon, 10 May 2021 10:58:15 +0100 Message-Id: <20210510095814.3732400-7-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210510095814.3732400-1-naush@raspberrypi.com> References: <20210510095814.3732400-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v6 6/6] ipa: raspberrypi: Rate-limit the controller algorithms X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The controller algorithms currently run on every frame provided to the IPA by the pipeline handler. This may be undesirable for very fast fps operating modes where it could significantly increase the computation cycles (per unit time) without providing any significant changes to the IQ parameters. The added latencies could also cause dropped frames. Pass the FrameBuffer timestamp to the IPA through the controls. This timestamp will be used to rate-limit the controller algorithms to run with a minimum inter-frame time given by a compile time constant, currently set to 16.66ms. On startup, we don't rate-limit the algorithms until after the number of frames required for convergence. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman Reviewed-by: Laurent Pinchart --- src/ipa/raspberrypi/raspberrypi.cpp | 42 +++++++++++++++++-- .../pipeline/raspberrypi/raspberrypi.cpp | 5 +++ 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 07409621a03a..52d91db282ea 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -61,6 +61,14 @@ constexpr unsigned int DefaultExposureTime = 20000; constexpr double defaultMinFrameDuration = 1e6 / 30.0; constexpr double defaultMaxFrameDuration = 1e6 / 0.01; +/* + * Determine the minimum allowable inter-frame duration (in us) to run the + * controller algorithms. If the pipeline handler provider frames at a rate + * higher than this, we rate-limit the controller Prepare() and Process() calls + * to lower than or equal to this rate. + */ +constexpr double controllerMinFrameDuration = 1e6 / 60.0; + LOG_DEFINE_CATEGORY(IPARPI) class IPARPi : public ipa::RPi::IPARPiInterface @@ -68,7 +76,7 @@ class IPARPi : public ipa::RPi::IPARPiInterface public: IPARPi() : controller_(), frameCount_(0), checkCount_(0), mistrustCount_(0), - lsTable_(nullptr), firstStart_(true) + lastRunTimestamp_(0), lsTable_(nullptr), firstStart_(true) { } @@ -146,6 +154,12 @@ private: /* Number of frames that need to be dropped on startup. */ unsigned int dropFrameCount_; + /* Frame timestamp for the last run of the controller. */ + uint64_t lastRunTimestamp_; + + /* Do we run a Controller::process() for this frame? */ + bool processPending_; + /* LS table allocation passed in from the pipeline handler. */ FileDescriptor lsTableHandle_; void *lsTable_; @@ -262,6 +276,7 @@ void IPARPi::start(const ControlList &controls, ipa::RPi::StartConfig *startConf startConfig->dropFrameCount = dropFrameCount_; firstStart_ = false; + lastRunTimestamp_ = 0; } void IPARPi::setMode(const CameraSensorInfo &sensorInfo) @@ -406,7 +421,7 @@ void IPARPi::signalStatReady(uint32_t bufferId) { if (++checkCount_ != frameCount_) /* assert here? */ LOG(IPARPI, Error) << "WARNING: Prepare/Process mismatch!!!"; - if (frameCount_ > mistrustCount_) + if (processPending_ && frameCount_ > mistrustCount_) processStats(bufferId); reportMetadata(); @@ -894,10 +909,11 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId) void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data) { + int64_t frameTimestamp = data.controls.get(controls::SensorTimestamp); + RPiController::Metadata lastMetadata; Span embeddedBuffer; - rpiMetadata_.Clear(); - + lastMetadata = std::move(rpiMetadata_); fillDeviceStatus(data.controls); if (data.embeddedBufferPresent) { @@ -920,6 +936,24 @@ void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data) if (data.embeddedBufferPresent) returnEmbeddedBuffer(data.embeddedBufferId); + /* Allow a 10% margin on the comparison below. */ + constexpr double eps = controllerMinFrameDuration * 1e3 * 0.1; + if (lastRunTimestamp_ && frameCount_ > dropFrameCount_ && + frameTimestamp - lastRunTimestamp_ + eps < controllerMinFrameDuration * 1e3) { + /* + * Ensure we merge the previous frame's metadata with the current + * frame. This will not overwrite exposure/gain values for the + * current frame, or any other bits of metadata that were added + * in helper_->Prepare(). + */ + rpiMetadata_.Merge(lastMetadata); + processPending_ = false; + return; + } + + lastRunTimestamp_ = frameTimestamp; + processPending_ = true; + ControlList ctrls(ispCtrls_); controller_.Prepare(&rpiMetadata_); diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index b22564938cdc..d26ae2a9772a 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1426,6 +1426,11 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) * DelayedControl and queue them along with the frame buffer. */ ControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence); + /* + * Add the frame timestamp to the ControlList for the IPA to use + * as it does not receive the FrameBuffer object. + */ + ctrl.set(controls::SensorTimestamp, buffer->metadata().timestamp); bayerQueue_.push({ buffer, std::move(ctrl) }); } else { embeddedQueue_.push(buffer);