From patchwork Fri Apr 16 10:31:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11965 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 42216BD235 for ; Fri, 16 Apr 2021 10:31:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A2C3D68813; Fri, 16 Apr 2021 12:31:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="jKp3YlC3"; dkim-atps=neutral Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 73F386880A for ; Fri, 16 Apr 2021 12:31:56 +0200 (CEST) Received: by mail-wr1-x435.google.com with SMTP id h4so17145604wrt.12 for ; Fri, 16 Apr 2021 03:31:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ISdsVA4HeIRxKBQupjIswGfDjzyE6VkT7Xub0Q0Fgvc=; b=jKp3YlC3NXSg3G9D/N5Mhmk72aBJb9jphmg/EPwcfkDtgll1crVTjTfGcx87fiBxT5 nqk4qbtLVYgTQSQDHQ/ZJr8kzQeqGAbab845L2do4ZVaqATKaVRo1DpLwFnV01WZgGMj jYL8kBr7SPx+u/MvZFbdsctHZNvLaakv4AfQ13rCYaXTphmPGzqZXvsaJacTAtazgqEo +Th7TxHKt3zOmEeOPxy1zHWGUgMyrgaQfZ3sCGiyCe2y9VhwAQerqnxfZsoXfdsNLUhE CmJGAnKbki6C4qAcsNJw5/vIApKEW1arDIgrpnxhDPpKnR1LoUOgCQL66aSghEC4nwF5 vUZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ISdsVA4HeIRxKBQupjIswGfDjzyE6VkT7Xub0Q0Fgvc=; b=Xxleeag4Lo6kK/NX2QU5flbdt52n8Y3rDZSuKMPf/CuSOPCeW7vKaA31z/EV2WYxUY 3kzp81XMJ69YTxDFDGuVZJqXcplw274qr9+YTGeZ0sVBgN8TKhx9vWZjTK+stTCrTA15 aQYxiYbtX51oE4oLqNSA8jDUFB+0kq4WmFLchidULcla6dEnLAmmsH61IGRmNM71TvLt oBZ9eZf2GOy9HjCL17NsULrUEQ0EYvn/jq82CVjznNxJfqs6OtNz2MlAqGJjAgBUS7Sc 4ogL/X83iBWlT/aojoFRL8qI1L6K6/gnic2APkPumVoV9SzqX2RLIFR9ZtSIQvZUQceH ZsCw== X-Gm-Message-State: AOAM530+heRPvFpVptKAt+Zzx4sUNiApfMnm28YVddVrQBBIo5IrkBb+ iNd8VUE9amSLWq6hgU85d6wV6iZAV0whlw== X-Google-Smtp-Source: ABdhPJywUbXGhlWe7HznCIOaaCTZ+trOcMCxFhEzUI/ZCybc9c42WbMrHdTVxWovdkxIJmpTdliazQ== X-Received: by 2002:adf:df0a:: with SMTP id y10mr8514604wrl.246.1618569115827; Fri, 16 Apr 2021 03:31:55 -0700 (PDT) Received: from naush-laptop.pitowers.org ([2a00:1098:3142:15:4af5:8d45:b5d1:889d]) by smtp.gmail.com with ESMTPSA id d133sm8234500wmf.9.2021.04.16.03.31.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Apr 2021 03:31:55 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Fri, 16 Apr 2021 11:31:41 +0100 Message-Id: <20210416103141.1483745-4-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210416103141.1483745-1-naush@raspberrypi.com> References: <20210416103141.1483745-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 3/3] ipa: raspberrypi: Rate-limit the controller algorithms X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The controller algorithms currently run on every frame provided to the IPA by the pipeline handler. This may be undesirable for very fast fps operating modes where it could significantly increase the computation cycles (per unit time) without providing any significant changes to the IQ parameters. The added latencies could also cause dropped frames. Pass the FrameBuffer timestamp to the IPA through the controls. This timestamp will be used to rate-limit the controller algorithms to run with a minimum inter-frame time given by a compile time constant, currently set to 16.66ms. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman --- src/ipa/raspberrypi/raspberrypi.cpp | 48 +++++++++++++++++-- .../pipeline/raspberrypi/raspberrypi.cpp | 5 ++ 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index f6d1ab16a290..e96b169ca612 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -61,6 +61,14 @@ constexpr unsigned int DefaultExposureTime = 20000; constexpr double defaultMinFrameDuration = 1e6 / 30.0; constexpr double defaultMaxFrameDuration = 1e6 / 0.01; +/* + * Determine the minimum allowable inter-frame duration (in us) to run the + * controller algorithms. If the pipeline handler provider frames at a rate + * higher than this, we rate-limit the controller prepare() and process() calls + * to lower than or equal to this rate. + */ +constexpr double controllerMinFrameDuration = 1e6 / 60.0; + LOG_DEFINE_CATEGORY(IPARPI) class IPARPi : public ipa::RPi::IPARPiInterface @@ -68,7 +76,7 @@ class IPARPi : public ipa::RPi::IPARPiInterface public: IPARPi() : controller_(), frameCount_(0), checkCount_(0), mistrustCount_(0), - lsTable_(nullptr), firstStart_(true) + lastRunTimestamp_(0), lsTable_(nullptr), firstStart_(true) { } @@ -146,6 +154,12 @@ private: /* Number of frames that need to be dropped on startup. */ unsigned int dropFrameCount_; + /* Frame timestamp for the last run of the controller. */ + uint64_t lastRunTimestamp_; + + /* Do we run a Controller::process() for this frame? */ + bool processPending_; + /* LS table allocation passed in from the pipeline handler. */ FileDescriptor lsTableHandle_; void *lsTable_; @@ -262,6 +276,7 @@ void IPARPi::start(const ControlList &controls, ipa::RPi::StartConfig *startConf startConfig->dropFrameCount = dropFrameCount_; firstStart_ = false; + lastRunTimestamp_ = 0; } void IPARPi::setMode(const CameraSensorInfo &sensorInfo) @@ -406,7 +421,7 @@ void IPARPi::signalStatReady(uint32_t bufferId) { if (++checkCount_ != frameCount_) /* assert here? */ LOG(IPARPI, Error) << "WARNING: Prepare/Process mismatch!!!"; - if (frameCount_ > mistrustCount_) + if (processPending_ && frameCount_ > mistrustCount_) processStats(bufferId); reportMetadata(); @@ -894,10 +909,11 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId) void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data) { + int64_t frameTimestamp = data.controls.get(controls::draft::SensorTimestamp); + RPiController::Metadata lastMetadata; Span embeddedBuffer; - rpiMetadata_.Clear(); - + lastMetadata = std::move(rpiMetadata_); fillDeviceStatus(data.controls); if (data.embeddedBufferPresent) { @@ -920,6 +936,30 @@ void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data) if (data.embeddedBufferPresent) returnEmbeddedBuffer(data.embeddedBufferId); + if (lastRunTimestamp_ && frameCount_ > dropFrameCount_ && + frameTimestamp - lastRunTimestamp_ < controllerMinFrameDuration * 1e3) { + /* + * Ensure we update the controller metadata with the new frame's + * exposure/gain values so that the correct values are returned + * out in libcamera metadata later on. All other metadata values + * must remain the same as the last frame. + */ + DeviceStatus currentDeviceStatus; + + rpiMetadata_.Get("device.status", currentDeviceStatus); + rpiMetadata_ = std::move(lastMetadata); + rpiMetadata_.Set("device.status", currentDeviceStatus); + processPending_ = false; + LOG(IPARPI, Debug) << "Rate-limiting the controller! inter-frame duration: " + << frameTimestamp - lastRunTimestamp_ + << ", min duration " + << controllerMinFrameDuration * 1e3; + return; + } + + lastRunTimestamp_ = frameTimestamp; + processPending_ = true; + ControlList ctrls(ispCtrls_); controller_.Prepare(&rpiMetadata_); diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 2a917455500f..9cf9c8c6cebd 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1414,6 +1414,11 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) * DelayedControl and queue them along with the frame buffer. */ ControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence); + /* + * Add the frame timestamp to the ControlList for the IPA to use + * as it does not receive the FrameBuffer object. + */ + ctrl.set(controls::draft::SensorTimestamp, buffer->metadata().timestamp); bayerQueue_.push({ buffer, std::move(ctrl) }); } else { embeddedQueue_.push(buffer);