From patchwork Fri May 7 08:40:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 12226 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0631ABF835 for ; Fri, 7 May 2021 08:40:56 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BD98B6890C; Fri, 7 May 2021 10:40:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="TfVgXges"; dkim-atps=neutral Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C245668921 for ; Fri, 7 May 2021 10:40:52 +0200 (CEST) Received: by mail-wm1-x333.google.com with SMTP id l18-20020a1ced120000b029014c1adff1edso6668371wmh.4 for ; Fri, 07 May 2021 01:40:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4S717ZsySRiN4h+aZW4nhYMwXVC+cgf8UZnjbQ6phO8=; b=TfVgXgesW084VhTrb1+XqCWAT9vl1OaSWlVcNYTp6Z2VfmLtUQGrHL5uEW+mPpm5lV xEz3yQObidj4IFM8C/PmZF4q0WZB6LO8qUukgXqyTrVD8bpk5UgsSu9w0GuDliuZF8uT bAHQS2HWagUJSUFogXo/UYAzw0kmRKdIxCP66Dwf2Qz9DOX2Go6tyqmapw15kJFfu5Ws ARHCR63JlsH8pIUaRF0Ies/QhCcTk2i5tznIYfsTJ3KFYb9jhx8GGBRXDk9WtywZohIm FB4J5ViPR6r/TxTzoyvDPtTYabC+/iZSpOMOtHwkOXDExHgMAQUGReh+XcmFcyRKF8OC 0Lig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4S717ZsySRiN4h+aZW4nhYMwXVC+cgf8UZnjbQ6phO8=; b=mTll8zYt3/Gp3dozyZVMQ6XhXhpkbxUcT4eQiX5TQb9y8ZEfkPyu5D/Ue9RTIMtnl0 zv0ZIcexhJwGayX/dIA9Z808v3a4Aj+lisUI4R/mWuZzFkUTkg3bqrd0QyOQK2wDppF5 cbOvt5/1mcznYafAx4tMWNMgyng5UtMKEmV2sWx6+9ToFRu9/HBFZV4iMWKxnS2v88cT 3qq42vHfyFjM2zPrCDQ+ElNcpSac0QMpOB54AUnuAFiocwWLtyP/TrnONOMWTX+RxsIP Azj63VDTEQHz2CvkzMWDhH6GkusuDy5B3Yll8YvcNKnrc3Gy0QFddoRlhRkj112UlScV 81Ow== X-Gm-Message-State: AOAM531I2klwNxW11A83i5QoZlZ4ppo910l6rDN6ShCXidBxKmV1Mfe4 hu7FZJ3NOUWfnbOjVl51v/MreAqXz4nH9Q== X-Google-Smtp-Source: ABdhPJxeL3uesUMTy12RipniR9k9xi0PJmpYBu6vvZI2Xuzn1c8ZBeJEHz9TRSKB4gKmvDA22r5vnA== X-Received: by 2002:a7b:c74d:: with SMTP id w13mr8455645wmk.25.1620376852053; Fri, 07 May 2021 01:40:52 -0700 (PDT) Received: from naush-laptop.pitowers.org ([2a00:1098:3142:14:81ab:95ee:9715:e9fb]) by smtp.gmail.com with ESMTPSA id l12sm7604098wrm.76.2021.05.07.01.40.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 01:40:51 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Fri, 7 May 2021 09:40:42 +0100 Message-Id: <20210507084042.31879-6-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210507084042.31879-1-naush@raspberrypi.com> References: <20210507084042.31879-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v6 5/5] ipa: raspberrypi: Rate-limit the controller algorithms X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The controller algorithms currently run on every frame provided to the IPA by the pipeline handler. This may be undesirable for very fast fps operating modes where it could significantly increase the computation cycles (per unit time) without providing any significant changes to the IQ parameters. The added latencies could also cause dropped frames. Pass the FrameBuffer timestamp to the IPA through the controls. This timestamp will be used to rate-limit the controller algorithms to run with a minimum inter-frame time given by a compile time constant, currently set to 16.66ms. On startup, we don't rate-limit the algorithms until after the number of frames required for convergence. Signed-off-by: Naushir Patuck Reviewed-by: Laurent Pinchart --- src/ipa/raspberrypi/raspberrypi.cpp | 48 +++++++++++++++++-- .../pipeline/raspberrypi/raspberrypi.cpp | 5 ++ 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index f6d1ab16a290..57f5497a6e6b 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -61,6 +61,14 @@ constexpr unsigned int DefaultExposureTime = 20000; constexpr double defaultMinFrameDuration = 1e6 / 30.0; constexpr double defaultMaxFrameDuration = 1e6 / 0.01; +/* + * Determine the minimum allowable inter-frame duration (in us) to run the + * controller algorithms. If the pipeline handler provider frames at a rate + * higher than this, we rate-limit the controller Prepare() and Process() calls + * to lower than or equal to this rate. + */ +constexpr double controllerMinFrameDuration = 1e6 / 60.0; + LOG_DEFINE_CATEGORY(IPARPI) class IPARPi : public ipa::RPi::IPARPiInterface @@ -68,7 +76,7 @@ class IPARPi : public ipa::RPi::IPARPiInterface public: IPARPi() : controller_(), frameCount_(0), checkCount_(0), mistrustCount_(0), - lsTable_(nullptr), firstStart_(true) + lastRunTimestamp_(0), lsTable_(nullptr), firstStart_(true) { } @@ -146,6 +154,12 @@ private: /* Number of frames that need to be dropped on startup. */ unsigned int dropFrameCount_; + /* Frame timestamp for the last run of the controller. */ + uint64_t lastRunTimestamp_; + + /* Do we run a Controller::process() for this frame? */ + bool processPending_; + /* LS table allocation passed in from the pipeline handler. */ FileDescriptor lsTableHandle_; void *lsTable_; @@ -262,6 +276,7 @@ void IPARPi::start(const ControlList &controls, ipa::RPi::StartConfig *startConf startConfig->dropFrameCount = dropFrameCount_; firstStart_ = false; + lastRunTimestamp_ = 0; } void IPARPi::setMode(const CameraSensorInfo &sensorInfo) @@ -406,7 +421,7 @@ void IPARPi::signalStatReady(uint32_t bufferId) { if (++checkCount_ != frameCount_) /* assert here? */ LOG(IPARPI, Error) << "WARNING: Prepare/Process mismatch!!!"; - if (frameCount_ > mistrustCount_) + if (processPending_ && frameCount_ > mistrustCount_) processStats(bufferId); reportMetadata(); @@ -894,10 +909,11 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId) void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data) { + int64_t frameTimestamp = data.controls.get(controls::draft::SensorTimestamp); + RPiController::Metadata lastMetadata; Span embeddedBuffer; - rpiMetadata_.Clear(); - + lastMetadata = std::move(rpiMetadata_); fillDeviceStatus(data.controls); if (data.embeddedBufferPresent) { @@ -920,6 +936,30 @@ void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data) if (data.embeddedBufferPresent) returnEmbeddedBuffer(data.embeddedBufferId); + /* Allow a 10% margin on the comparison below. */ + constexpr double eps = controllerMinFrameDuration * 1e3 * 0.1; + if (lastRunTimestamp_ && frameCount_ > dropFrameCount_ && + frameTimestamp - lastRunTimestamp_ + eps < controllerMinFrameDuration * 1e3) { + /* + * Ensure we update the controller metadata with the new frame's + * exposure/gain values so that the correct values are returned + * out in libcamera metadata later on. Any other bits of metadata + * that were added in helper_->Prepare() will also be moved across. + * All other metadata values must remain the same as the last frame. + * + * We do this by overwriting the current frame's metadata into the + * previous frame's metadata object, and then swapping the latter + * with the former. + */ + lastMetadata.Overwrite(rpiMetadata_); + std::swap(rpiMetadata_, lastMetadata); + processPending_ = false; + return; + } + + lastRunTimestamp_ = frameTimestamp; + processPending_ = true; + ControlList ctrls(ispCtrls_); controller_.Prepare(&rpiMetadata_); diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 2a917455500f..9cf9c8c6cebd 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1414,6 +1414,11 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) * DelayedControl and queue them along with the frame buffer. */ ControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence); + /* + * Add the frame timestamp to the ControlList for the IPA to use + * as it does not receive the FrameBuffer object. + */ + ctrl.set(controls::draft::SensorTimestamp, buffer->metadata().timestamp); bayerQueue_.push({ buffer, std::move(ctrl) }); } else { embeddedQueue_.push(buffer);