From patchwork Thu Feb 18 17:01:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11337 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D9790BD1F1 for ; Thu, 18 Feb 2021 17:01:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A703B689C1; Thu, 18 Feb 2021 18:01:34 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="a0t8Kzyi"; dkim-atps=neutral Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BBFA968370 for ; Thu, 18 Feb 2021 18:01:32 +0100 (CET) Received: by mail-wm1-x330.google.com with SMTP id x4so4590396wmi.3 for ; Thu, 18 Feb 2021 09:01:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BGf6FqDf+QKZ9XQ/XwwRcdUDqGpiZdrGU8Nuikcqs10=; b=a0t8KzyiRc5ic+B+Tlt6ARj7VvH2Wy5P5yArm9/Pa/NUAL1GWM+wn93fFtS+69wTEr Cx7SVy1ZG4ZgfbDTwPZdWNbPBBWpejuKtPnncNUyYIC86P4bekT5TaxCl/tuU/NIdvHd jzEJ3jEv5NcDVrMrEa1N3X0pzzpvBDU+ZffSiWxudgPGX3bfFzK7iCYp5etnfE8eKSTY 20Bc3G9pdLSqY2j+mZSrMkRXaeHI8R4l+oIqQjyulD6gxEVmdUXTqiLIKg0piS5nRT0a 5Dfc9iS4uQSp7v9NOsnJCWBzUXvYHokRuBLqXmeGfqkXOM0/QGJe7gXkbRzN366Cgh9l yY6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BGf6FqDf+QKZ9XQ/XwwRcdUDqGpiZdrGU8Nuikcqs10=; b=gjpo2mQxU6QlCWkZp1CnuOP9aiR/hOSVQIX02euE5ncOB35A5aXT4baRhKqU86Oplt Dt1VFjyg0CoJQR6kKh16BKEGOkEiYVzYHx0jqE29DROWbX8ISCB81fxF8qzSn9DsyDlF XB8sGIPTcmjhF+ZgitkK4oV47GgfOpEyTwRHVIfG0DevsFQkELicBX3wGmN/KrLOZVlK Zj64oig4KioLvhEvYmSsLNZ0tVEuNpCBiSe2lrlt1n6pjD1gGeyrxeai7xcYGebDK1qO rVpdXUI3wkqIxLWNodTh1xCjDcAuvz/8CjAvH8/SbWuuJPdetxCV0fxDkV+X5Hx276Ob YZxg== X-Gm-Message-State: AOAM532lAcBIr7nbpcrI3iazHuXYCXvW/x3VUOZrycxmkl5+dHE8Xu7c +3iuiVkxcwsk8X2fW7/Vi3tm5i8tZQsfewYY X-Google-Smtp-Source: ABdhPJzrfbokt6r7OxRgOhiXz4CXyCTaWAZ37tUy/Wn0yA1ZcU8er2W5CkE9MTYvoLc9rVsMo60fYw== X-Received: by 2002:a05:600c:4857:: with SMTP id j23mr4493482wmo.66.1613667691733; Thu, 18 Feb 2021 09:01:31 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id t30sm10504747wra.56.2021.02.18.09.01.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Feb 2021 09:01:31 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Thu, 18 Feb 2021 17:01:23 +0000 Message-Id: <20210218170126.2060783-2-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210218170126.2060783-1-naush@raspberrypi.com> References: <20210218170126.2060783-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 1/4] pipeline: ipa: raspberrypi: Pass exposure/gain values to IPA though controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When running with sensors that had no embedded data, the pipeline handler would fill a dummy embedded data buffer with gain/exposure values, and pass this buffer to the IPA together with the bayer buffer. The IPA would extract these values for use in the controller algorithms. Rework this logic entirely by having a new RPiCameraData::BayerFrame queue to replace the existing bayer queue. In addition to storing the FrameBuffer pointer, this also stores all the controls tracked by DelayedControls for that frame in a ControlList. This includes include exposure and gain values. On signalling RPi::IPA_EVENT_SIGNAL_ISP_PREPARE IPA event, the pipeline handler now passes this ControlList from the RPiCameraData::BayerFrame queue. The IPA now extracts the gain and exposure values from the ControlList instead of using RPiController::MdParserRPi to parse the embedded data buffer. Signed-off-by: Naushir Patuck --- include/libcamera/ipa/raspberrypi.mojom | 2 + src/ipa/raspberrypi/raspberrypi.cpp | 132 +++++++++++------- .../pipeline/raspberrypi/raspberrypi.cpp | 58 ++++---- 3 files changed, 108 insertions(+), 84 deletions(-) diff --git a/include/libcamera/ipa/raspberrypi.mojom b/include/libcamera/ipa/raspberrypi.mojom index 5a27b1e4fc2d..f733a2cd2a40 100644 --- a/include/libcamera/ipa/raspberrypi.mojom +++ b/include/libcamera/ipa/raspberrypi.mojom @@ -29,6 +29,8 @@ struct SensorConfig { struct ISPConfig { uint32 embeddedBufferId; uint32 bayerBufferId; + bool embeddedBufferPresent; + ControlList controls; }; struct ConfigInput { diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 1226ea514521..f9ab6417866e 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: BSD-2-Clause */ /* - * Copyright (C) 2019-2020, Raspberry Pi (Trading) Ltd. + * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd. * * rpi.cpp - Raspberry Pi Image Processing Algorithms */ @@ -101,9 +101,11 @@ private: bool validateIspControls(); void queueRequest(const ControlList &controls); void returnEmbeddedBuffer(unsigned int bufferId); - void prepareISP(unsigned int bufferId); + void prepareISP(const ipa::RPi::ISPConfig &data); void reportMetadata(); bool parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &deviceStatus); + void fillDeviceStatus(uint32_t exposureLines, uint32_t gainCode, + struct DeviceStatus &deviceStatus); void processStats(unsigned int bufferId); void applyFrameDurations(double minFrameDuration, double maxFrameDuration); void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls); @@ -447,7 +449,7 @@ void IPARPi::signalIspPrepare(const ipa::RPi::ISPConfig &data) * avoid running the control algos for a few frames in case * they are "unreliable". */ - prepareISP(data.embeddedBufferId); + prepareISP(data); frameCount_++; /* Ready to push the input buffer into the ISP. */ @@ -909,67 +911,84 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId) embeddedComplete.emit(bufferId & ipa::RPi::MaskID); } -void IPARPi::prepareISP(unsigned int bufferId) +void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data) { struct DeviceStatus deviceStatus = {}; - bool success = parseEmbeddedData(bufferId, deviceStatus); + bool success = false; - /* Done with embedded data now, return to pipeline handler asap. */ - returnEmbeddedBuffer(bufferId); + if (data.embeddedBufferPresent) { + /* + * Pipeline handler has supplied us with an embedded data buffer, + * so parse it and extract the exposure and gain. + */ + success = parseEmbeddedData(data.embeddedBufferId, deviceStatus); - if (success) { - ControlList ctrls(ispCtrls_); + /* Done with embedded data now, return to pipeline handler asap. */ + returnEmbeddedBuffer(data.embeddedBufferId); + } - rpiMetadata_.Clear(); - rpiMetadata_.Set("device.status", deviceStatus); - controller_.Prepare(&rpiMetadata_); + if (!success) { + /* + * Pipeline handler has not supplied an embedded data buffer, + * or embedded data buffer parsing has failed for some reason, + * so pull the exposure and gain values from the control list. + */ + int32_t exposureLines = data.controls.get(V4L2_CID_EXPOSURE).get(); + int32_t gainCode = data.controls.get(V4L2_CID_ANALOGUE_GAIN).get(); + fillDeviceStatus(exposureLines, gainCode, deviceStatus); + } - /* Lock the metadata buffer to avoid constant locks/unlocks. */ - std::unique_lock lock(rpiMetadata_); + ControlList ctrls(ispCtrls_); - AwbStatus *awbStatus = rpiMetadata_.GetLocked("awb.status"); - if (awbStatus) - applyAWB(awbStatus, ctrls); + rpiMetadata_.Clear(); + rpiMetadata_.Set("device.status", deviceStatus); + controller_.Prepare(&rpiMetadata_); - CcmStatus *ccmStatus = rpiMetadata_.GetLocked("ccm.status"); - if (ccmStatus) - applyCCM(ccmStatus, ctrls); + /* Lock the metadata buffer to avoid constant locks/unlocks. */ + std::unique_lock lock(rpiMetadata_); - AgcStatus *dgStatus = rpiMetadata_.GetLocked("agc.status"); - if (dgStatus) - applyDG(dgStatus, ctrls); + AwbStatus *awbStatus = rpiMetadata_.GetLocked("awb.status"); + if (awbStatus) + applyAWB(awbStatus, ctrls); - AlscStatus *lsStatus = rpiMetadata_.GetLocked("alsc.status"); - if (lsStatus) - applyLS(lsStatus, ctrls); + CcmStatus *ccmStatus = rpiMetadata_.GetLocked("ccm.status"); + if (ccmStatus) + applyCCM(ccmStatus, ctrls); - ContrastStatus *contrastStatus = rpiMetadata_.GetLocked("contrast.status"); - if (contrastStatus) - applyGamma(contrastStatus, ctrls); + AgcStatus *dgStatus = rpiMetadata_.GetLocked("agc.status"); + if (dgStatus) + applyDG(dgStatus, ctrls); - BlackLevelStatus *blackLevelStatus = rpiMetadata_.GetLocked("black_level.status"); - if (blackLevelStatus) - applyBlackLevel(blackLevelStatus, ctrls); + AlscStatus *lsStatus = rpiMetadata_.GetLocked("alsc.status"); + if (lsStatus) + applyLS(lsStatus, ctrls); - GeqStatus *geqStatus = rpiMetadata_.GetLocked("geq.status"); - if (geqStatus) - applyGEQ(geqStatus, ctrls); + ContrastStatus *contrastStatus = rpiMetadata_.GetLocked("contrast.status"); + if (contrastStatus) + applyGamma(contrastStatus, ctrls); - DenoiseStatus *denoiseStatus = rpiMetadata_.GetLocked("denoise.status"); - if (denoiseStatus) - applyDenoise(denoiseStatus, ctrls); + BlackLevelStatus *blackLevelStatus = rpiMetadata_.GetLocked("black_level.status"); + if (blackLevelStatus) + applyBlackLevel(blackLevelStatus, ctrls); - SharpenStatus *sharpenStatus = rpiMetadata_.GetLocked("sharpen.status"); - if (sharpenStatus) - applySharpen(sharpenStatus, ctrls); + GeqStatus *geqStatus = rpiMetadata_.GetLocked("geq.status"); + if (geqStatus) + applyGEQ(geqStatus, ctrls); - DpcStatus *dpcStatus = rpiMetadata_.GetLocked("dpc.status"); - if (dpcStatus) - applyDPC(dpcStatus, ctrls); + DenoiseStatus *denoiseStatus = rpiMetadata_.GetLocked("denoise.status"); + if (denoiseStatus) + applyDenoise(denoiseStatus, ctrls); - if (!ctrls.empty()) - setIspControls.emit(ctrls); - } + SharpenStatus *sharpenStatus = rpiMetadata_.GetLocked("sharpen.status"); + if (sharpenStatus) + applySharpen(sharpenStatus, ctrls); + + DpcStatus *dpcStatus = rpiMetadata_.GetLocked("dpc.status"); + if (dpcStatus) + applyDPC(dpcStatus, ctrls); + + if (!ctrls.empty()) + setIspControls.emit(ctrls); } bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &deviceStatus) @@ -985,6 +1004,7 @@ bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &devic RPiController::MdParser::Status status = helper_->Parser().Parse(mem.data()); if (status != RPiController::MdParser::Status::OK) { LOG(IPARPI, Error) << "Embedded Buffer parsing failed, error " << status; + return false; } else { uint32_t exposureLines, gainCode; if (helper_->Parser().GetExposureLines(exposureLines) != RPiController::MdParser::Status::OK) { @@ -992,21 +1012,29 @@ bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &devic return false; } - deviceStatus.shutter_speed = helper_->Exposure(exposureLines); if (helper_->Parser().GetGainCode(gainCode) != RPiController::MdParser::Status::OK) { LOG(IPARPI, Error) << "Gain failed"; return false; } - deviceStatus.analogue_gain = helper_->Gain(gainCode); - LOG(IPARPI, Debug) << "Metadata - Exposure : " - << deviceStatus.shutter_speed << " Gain : " - << deviceStatus.analogue_gain; + fillDeviceStatus(exposureLines, gainCode, deviceStatus); } return true; } +void IPARPi::fillDeviceStatus(uint32_t exposureLines, uint32_t gainCode, + struct DeviceStatus &deviceStatus) +{ + deviceStatus.shutter_speed = helper_->Exposure(exposureLines); + deviceStatus.analogue_gain = helper_->Gain(gainCode); + + LOG(IPARPI, Debug) << "Metadata - Exposure : " + << deviceStatus.shutter_speed + << " Gain : " + << deviceStatus.analogue_gain; +} + void IPARPi::processStats(unsigned int bufferId) { auto it = buffers_.find(bufferId); diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index ba74ace183db..b43d86166c63 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: LGPL-2.1-or-later */ /* - * Copyright (C) 2019-2020, Raspberry Pi (Trading) Ltd. + * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd. * * raspberrypi.cpp - Pipeline handler for Raspberry Pi devices */ @@ -197,7 +197,13 @@ public: */ enum class State { Stopped, Idle, Busy, IpaComplete }; State state_; - std::queue bayerQueue_; + + struct BayerFrame { + FrameBuffer *buffer; + ControlList controls; + }; + + std::queue bayerQueue_; std::queue embeddedQueue_; std::deque requestQueue_; @@ -222,7 +228,7 @@ public: private: void checkRequestCompleted(); void tryRunPipeline(); - bool findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *&embeddedBuffer); + bool findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&embeddedBuffer); unsigned int ispOutputCount_; }; @@ -1383,29 +1389,14 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) << ", timestamp: " << buffer->metadata().timestamp; if (stream == &unicam_[Unicam::Image]) { - bayerQueue_.push(buffer); - } else { - embeddedQueue_.push(buffer); - - ControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence); - /* - * Sensor metadata is unavailable, so put the expected ctrl - * values (accounting for the staggered delays) into the empty - * metadata buffer. + * Lookup the sensor controls used for this frame sequence from + * StaggeredCtrl and queue them along with the frame buffer. */ - if (!sensorMetadata_) { - unsigned int bufferId = unicam_[Unicam::Embedded].getBufferId(buffer); - auto it = mappedEmbeddedBuffers_.find(bufferId); - if (it != mappedEmbeddedBuffers_.end()) { - uint32_t *mem = reinterpret_cast(it->second.maps()[0].data()); - mem[0] = ctrl.get(V4L2_CID_EXPOSURE).get(); - mem[1] = ctrl.get(V4L2_CID_ANALOGUE_GAIN).get(); - } else { - LOG(RPI, Warning) << "Failed to find embedded buffer " - << bufferId; - } - } + ControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence); + bayerQueue_.push({ buffer, std::move(ctrl) }); + } else { + embeddedQueue_.push(buffer); } handleState(); @@ -1656,14 +1647,15 @@ void RPiCameraData::applyScalerCrop(const ControlList &controls) void RPiCameraData::tryRunPipeline() { - FrameBuffer *bayerBuffer, *embeddedBuffer; + FrameBuffer *embeddedBuffer; + BayerFrame bayerFrame; /* If any of our request or buffer queues are empty, we cannot proceed. */ if (state_ != State::Idle || requestQueue_.empty() || bayerQueue_.empty() || embeddedQueue_.empty()) return; - if (!findMatchingBuffers(bayerBuffer, embeddedBuffer)) + if (!findMatchingBuffers(bayerFrame, embeddedBuffer)) return; /* Take the first request from the queue and action the IPA. */ @@ -1682,7 +1674,7 @@ void RPiCameraData::tryRunPipeline() /* Set our state to say the pipeline is active. */ state_ = State::Busy; - unsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerBuffer); + unsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerFrame.buffer); unsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer); LOG(RPI, Debug) << "Signalling signalIspPrepare:" @@ -1692,24 +1684,26 @@ void RPiCameraData::tryRunPipeline() ipa::RPi::ISPConfig ispPrepare; ispPrepare.embeddedBufferId = ipa::RPi::MaskEmbeddedData | embeddedId; ispPrepare.bayerBufferId = ipa::RPi::MaskBayerData | bayerId; + ispPrepare.embeddedBufferPresent = true; + ispPrepare.controls = std::move(bayerFrame.controls); ipa_->signalIspPrepare(ispPrepare); } -bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *&embeddedBuffer) +bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&embeddedBuffer) { unsigned int embeddedRequeueCount = 0, bayerRequeueCount = 0; /* Loop until we find a matching bayer and embedded data buffer. */ while (!bayerQueue_.empty()) { /* Start with the front of the bayer queue. */ - bayerBuffer = bayerQueue_.front(); + bayerFrame = bayerQueue_.front(); /* * Find the embedded data buffer with a matching timestamp to pass to * the IPA. Any embedded buffers with a timestamp lower than the * current bayer buffer will be removed and re-queued to the driver. */ - uint64_t ts = bayerBuffer->metadata().timestamp; + uint64_t ts = bayerFrame.buffer->metadata().timestamp; embeddedBuffer = nullptr; while (!embeddedQueue_.empty()) { FrameBuffer *b = embeddedQueue_.front(); @@ -1739,7 +1733,7 @@ bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer * * the front of the queue. This buffer is now orphaned, so requeue * it back to the device. */ - unicam_[Unicam::Image].queueBuffer(bayerQueue_.front()); + unicam_[Unicam::Image].queueBuffer(bayerQueue_.front().buffer); bayerQueue_.pop(); bayerRequeueCount++; LOG(RPI, Warning) << "Dropping unmatched input frame in stream " @@ -1757,7 +1751,7 @@ bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer * LOG(RPI, Warning) << "Flushing bayer stream!"; while (!bayerQueue_.empty()) { - unicam_[Unicam::Image].queueBuffer(bayerQueue_.front()); + unicam_[Unicam::Image].queueBuffer(bayerQueue_.front().buffer); bayerQueue_.pop(); } flushedBuffers = true; From patchwork Thu Feb 18 17:01:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11338 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EDF1ABD1F1 for ; Thu, 18 Feb 2021 17:01:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2F349689C5; Thu, 18 Feb 2021 18:01:35 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="fOo3doD4"; dkim-atps=neutral Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A4E7568370 for ; Thu, 18 Feb 2021 18:01:33 +0100 (CET) Received: by mail-wm1-x331.google.com with SMTP id x4so4590434wmi.3 for ; Thu, 18 Feb 2021 09:01:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3HvJuESFWALCqMFAwUgJ9kgCVsW7NPMfzuFxlTN4+C4=; b=fOo3doD4cKGuhfawJs58hoj6sMYzOjJtqg/nGqTPpVqacZ5juOw1F3eYFjrojcObQr bZ9TbL6P0lQnluyjbT7Jtr3yAeQ1YO9pGWtpJwslLi7CP2MZphYBBdIxcIVwKyZAGri6 bpcJk/UkiC/Qo0S+NrGOrL1U2pcjbLW3NLvf4lEgoe+Q932f+0H9X8V79lVB6+Re+ifM RoH8yFT3ArCEkyv311d6AnEyKPNMkL6CGqDz6cs8OicepzuPppcisrkvnQurHcocZnVO f5smFCXrkVoLdhBofxmrKpBpmhiGZ5RQEM4lPeYvm0BGxzKTo+AQqA7IQqeTU/N/1/pI iylg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3HvJuESFWALCqMFAwUgJ9kgCVsW7NPMfzuFxlTN4+C4=; b=I0cVcDBA2vxuikcnRwxOc9b6EnWvKyN4n1/uCoJ463lfF/YAANJCeH07d65QYsMggN BG7DXzjJO02AOecVELsemVLAlgEDiPZGKzXwhNzDG0g652wBW/SasBw2DW+tqQX6/PfC J7N2Aab+3kv0LqCrj2/Wl8ND44a+DE/CqoEPsRtohwHJixt1HNMpk3tsqOnVhKhDsuSw f0M2TZEuSoGTVij9Kqin4aAeHYewNB4vgIzp5aF1RUwjK2AL9sh2BMQD67x7zvpmhYkN eWZeFzE2nEX2OTC/W1ooiXJCtUcIYUBzKoNqlN1C69MaV73cGAr9AkJs5PvpdbtMzfOw yKIg== X-Gm-Message-State: AOAM532C5lGdrJtLH7FAu68aQu2JMhG4nS/d9GB2paOp1GScpN6AQE7d uWrf6yUgfkU2ZHIQWquGbA6xIUHQRSafNg7o X-Google-Smtp-Source: ABdhPJyPIsg9M0Gah/mBWTU5w21yNRb31Glf0raCepkdMXDOxqfjO7Em4CE2YlHWGwX0/qO+nJqqXQ== X-Received: by 2002:a05:600c:19cf:: with SMTP id u15mr4370528wmq.139.1613667692493; Thu, 18 Feb 2021 09:01:32 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id t30sm10504747wra.56.2021.02.18.09.01.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Feb 2021 09:01:32 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Thu, 18 Feb 2021 17:01:24 +0000 Message-Id: <20210218170126.2060783-3-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210218170126.2060783-1-naush@raspberrypi.com> References: <20210218170126.2060783-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 2/4] ipa: raspberrypi: Remove MdParserRPi X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With the recent change to pass a ControlList to the IPA with exposure and gain values used for a frame, RPiController::MdParserRPi is not needed any more. Remove all traces of it. The derived CamHelper objects now pass nullptr values for the parser to the base CamHelper class when sensors do not use metadata. Signed-off-by: Naushir Patuck Reviewed-by: Laurent Pinchart --- src/ipa/raspberrypi/cam_helper.cpp | 9 ++++-- src/ipa/raspberrypi/cam_helper_imx219.cpp | 4 +-- src/ipa/raspberrypi/cam_helper_ov5647.cpp | 3 +- src/ipa/raspberrypi/md_parser_rpi.cpp | 37 ----------------------- src/ipa/raspberrypi/md_parser_rpi.hpp | 32 -------------------- src/ipa/raspberrypi/meson.build | 1 - 6 files changed, 8 insertions(+), 78 deletions(-) delete mode 100644 src/ipa/raspberrypi/md_parser_rpi.cpp delete mode 100644 src/ipa/raspberrypi/md_parser_rpi.hpp diff --git a/src/ipa/raspberrypi/cam_helper.cpp b/src/ipa/raspberrypi/cam_helper.cpp index 93d1b7b0296a..c9cdc39c5932 100644 --- a/src/ipa/raspberrypi/cam_helper.cpp +++ b/src/ipa/raspberrypi/cam_helper.cpp @@ -42,7 +42,8 @@ CamHelper::CamHelper(MdParser *parser, unsigned int frameIntegrationDiff) CamHelper::~CamHelper() { - delete parser_; + if (parser_) + delete parser_; } uint32_t CamHelper::ExposureLines(double exposure_us) const @@ -88,8 +89,10 @@ uint32_t CamHelper::GetVBlanking(double &exposure, double minFrameDuration, void CamHelper::SetCameraMode(const CameraMode &mode) { mode_ = mode; - parser_->SetBitsPerPixel(mode.bitdepth); - parser_->SetLineLengthBytes(0); /* We use SetBufferSize. */ + if (parser_) { + parser_->SetBitsPerPixel(mode.bitdepth); + parser_->SetLineLengthBytes(0); /* We use SetBufferSize. */ + } initialized_ = true; } diff --git a/src/ipa/raspberrypi/cam_helper_imx219.cpp b/src/ipa/raspberrypi/cam_helper_imx219.cpp index 95b8e698fe3b..0e454d0de2dc 100644 --- a/src/ipa/raspberrypi/cam_helper_imx219.cpp +++ b/src/ipa/raspberrypi/cam_helper_imx219.cpp @@ -19,8 +19,6 @@ #include "cam_helper.hpp" #if ENABLE_EMBEDDED_DATA #include "md_parser.hpp" -#else -#include "md_parser_rpi.hpp" #endif using namespace RPiController; @@ -62,7 +60,7 @@ CamHelperImx219::CamHelperImx219() #if ENABLE_EMBEDDED_DATA : CamHelper(new MdParserImx219(), frameIntegrationDiff) #else - : CamHelper(new MdParserRPi(), frameIntegrationDiff) + : CamHelper(nullptr, frameIntegrationDiff) #endif { } diff --git a/src/ipa/raspberrypi/cam_helper_ov5647.cpp b/src/ipa/raspberrypi/cam_helper_ov5647.cpp index a7f417324048..75486e900d31 100644 --- a/src/ipa/raspberrypi/cam_helper_ov5647.cpp +++ b/src/ipa/raspberrypi/cam_helper_ov5647.cpp @@ -8,7 +8,6 @@ #include #include "cam_helper.hpp" -#include "md_parser_rpi.hpp" using namespace RPiController; @@ -38,7 +37,7 @@ private: */ CamHelperOv5647::CamHelperOv5647() - : CamHelper(new MdParserRPi(), frameIntegrationDiff) + : CamHelper(nullptr, frameIntegrationDiff) { } diff --git a/src/ipa/raspberrypi/md_parser_rpi.cpp b/src/ipa/raspberrypi/md_parser_rpi.cpp deleted file mode 100644 index 2b0bcfc5f034..000000000000 --- a/src/ipa/raspberrypi/md_parser_rpi.cpp +++ /dev/null @@ -1,37 +0,0 @@ -/* SPDX-License-Identifier: BSD-2-Clause */ -/* - * Copyright (C) 2020, Raspberry Pi (Trading) Limited - * - * md_parser_rpi.cpp - Metadata parser for generic Raspberry Pi metadata - */ - -#include - -#include "md_parser_rpi.hpp" - -using namespace RPiController; - -MdParserRPi::MdParserRPi() -{ -} - -MdParser::Status MdParserRPi::Parse(void *data) -{ - if (buffer_size_bytes_ < sizeof(rpiMetadata)) - return ERROR; - - memcpy(&metadata, data, sizeof(rpiMetadata)); - return OK; -} - -MdParser::Status MdParserRPi::GetExposureLines(unsigned int &lines) -{ - lines = metadata.exposure; - return OK; -} - -MdParser::Status MdParserRPi::GetGainCode(unsigned int &gain_code) -{ - gain_code = metadata.gain; - return OK; -} diff --git a/src/ipa/raspberrypi/md_parser_rpi.hpp b/src/ipa/raspberrypi/md_parser_rpi.hpp deleted file mode 100644 index 52f54f008056..000000000000 --- a/src/ipa/raspberrypi/md_parser_rpi.hpp +++ /dev/null @@ -1,32 +0,0 @@ -/* SPDX-License-Identifier: BSD-2-Clause */ -/* - * Copyright (C) 2019, Raspberry Pi (Trading) Limited - * - * md_parser_rpi.hpp - Raspberry Pi metadata parser interface - */ -#pragma once - -#include "md_parser.hpp" - -namespace RPiController { - -class MdParserRPi : public MdParser -{ -public: - MdParserRPi(); - Status Parse(void *data) override; - Status GetExposureLines(unsigned int &lines) override; - Status GetGainCode(unsigned int &gain_code) override; - -private: - // This must be the same struct that is filled into the metadata buffer - // in the pipeline handler. - struct rpiMetadata - { - uint32_t exposure; - uint32_t gain; - }; - rpiMetadata metadata; -}; - -} diff --git a/src/ipa/raspberrypi/meson.build b/src/ipa/raspberrypi/meson.build index 9af755259747..59e49686bc09 100644 --- a/src/ipa/raspberrypi/meson.build +++ b/src/ipa/raspberrypi/meson.build @@ -17,7 +17,6 @@ rpi_ipa_includes = [ rpi_ipa_sources = files([ 'raspberrypi.cpp', 'md_parser.cpp', - 'md_parser_rpi.cpp', 'cam_helper.cpp', 'cam_helper_ov5647.cpp', 'cam_helper_imx219.cpp', From patchwork Thu Feb 18 17:01:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11339 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 124E5BD1F1 for ; Thu, 18 Feb 2021 17:01:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D327E689C7; Thu, 18 Feb 2021 18:01:36 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="LeTSqS8k"; dkim-atps=neutral Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F245668370 for ; Thu, 18 Feb 2021 18:01:33 +0100 (CET) Received: by mail-wm1-x32c.google.com with SMTP id x16so4382783wmk.3 for ; Thu, 18 Feb 2021 09:01:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G9aibhqp+Si9LQKERAmiI660bAvlasfj1ec+UDjUjW4=; b=LeTSqS8kU4F3+3Mb9gAmJLR7RkdxbFJAi5+sXxT4/H+j0E16HnPIDXZiVbGzwrgtAT TJeA3lEimASbg7vT50LuiYNiElevyVmOJR9RWwCH61s8GBdC1+a2AfqGe8P8TkFxinhj d9caHczfvHUWrKgZxR/A13nLLWy/sOqw47feJLREkJFTvuL1TOnLO6HNJm4HqhkN0YAM 0OTlv6fHWhfGe2kNow7oPhyHoLLmL8K9rYNAGmLM823TAT5rg+8DoJkFtyrXLnb8iYbt 0WPWvZWRJJdnC2hNByseI772tLBBImOvtb3D+w2NTsJziOXyQz2Abi0GpoB1Op5Vybt3 AF8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G9aibhqp+Si9LQKERAmiI660bAvlasfj1ec+UDjUjW4=; b=r44efpNQ+lpEW3MwxasiI+n9LHvBd55MYYhvhhBtv9SBVWoIhBkokwb03aqj/3lckm SSKZBwgKeqomBBpyeToJUSiUhj3J6os7mu53n/MEFRzccBpAtgORHhjfn13bu4mBtQIu scJX+KLit0ONmO5gR/S8uv2zywFlLxHIxqA/PsGksNVWZuDXHVp2Gh5qB1jnRK9/gNWi rTIN5ryWfRDCRy8Xb684frepTCwro6JWKho5nKEivIMC6U9ryMdcQi4WrNW0ZqAFjYOv TOGzqEGQ83o/GE5TR9lfmb2pOKhGDE0pbuvyP5JQABh6xThC7HN1Ekg2xlDy7+eJJcDa L4Uw== X-Gm-Message-State: AOAM531NKWgKvCzD5TdnaZRGKaLV24c4xZEODHq2kZkPeWoiYb7SSD0G SCjk7WMKNnvnoinascw42co8VLzs4PX5lkUq X-Google-Smtp-Source: ABdhPJz7EGhjT95KridSbxiuLTd5pwsUladEE/BpfQIKnSJyuqHX+V+JvoGMMIyLKMxW2QOdDaXVkg== X-Received: by 2002:a1c:4006:: with SMTP id n6mr4462163wma.177.1613667693261; Thu, 18 Feb 2021 09:01:33 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id t30sm10504747wra.56.2021.02.18.09.01.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Feb 2021 09:01:32 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Thu, 18 Feb 2021 17:01:25 +0000 Message-Id: <20210218170126.2060783-4-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210218170126.2060783-1-naush@raspberrypi.com> References: <20210218170126.2060783-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 3/4] pipeline: raspberrypi: Only enabled embedded stream when available X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The pipeline handler would enable and use the Unicam embedded data stream even if the sensor did not support it. This was to allow a means to pass exposure and gain values for the frame to the IPA in a synchronised way. The recent changes to get the pipeline handler to pass a ControlList with exposure and gain values means this is no longer required. Disable the use of the embedded data stream when a sensor does not support it. This change also removes the mappedEmbeddedBuffers_ map as it is no longer used. Signed-off-by: Naushir Patuck --- .../pipeline/raspberrypi/raspberrypi.cpp | 112 +++++++++++------- 1 file changed, 70 insertions(+), 42 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index b43d86166c63..d969c77993eb 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -177,12 +177,6 @@ public: /* Stores the ids of the buffers mapped in the IPA. */ std::unordered_set ipaBuffers_; - /* - * Map of (internal) mmaped embedded data buffers, to avoid having to - * map/unmap on every frame. - */ - std::map mappedEmbeddedBuffers_; - /* DMAHEAP allocation helper. */ RPi::DmaHeap dmaHeap_; FileDescriptor lsTable_; @@ -636,14 +630,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) if (isRaw(cfg.pixelFormat)) { cfg.setStream(&data->unicam_[Unicam::Image]); - /* - * We must set both Unicam streams as external, even - * though the application may only request RAW frames. - * This is because we match timestamps on both streams - * to synchronise buffers. - */ data->unicam_[Unicam::Image].setExternal(true); - data->unicam_[Unicam::Embedded].setExternal(true); continue; } @@ -715,17 +702,6 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) return ret; } - /* Unicam embedded data output format. */ - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); - LOG(RPI, Debug) << "Setting embedded data format."; - ret = data->unicam_[Unicam::Embedded].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on Unicam embedded: " - << format.toString(); - return ret; - } - /* Figure out the smallest selection the ISP will allow. */ Rectangle testCrop(0, 0, 1, 1); data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop); @@ -742,6 +718,41 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) if (ret) LOG(RPI, Error) << "Failed to configure the IPA: " << ret; + /* + * The IPA will set data->sensorMetadata_ to true if embedded data is + * supported on this sensor. If so, open the Unicam embedded data + * node and configure the output format. + */ + if (data->sensorMetadata_) { + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); + LOG(RPI, Debug) << "Setting embedded data format."; + data->unicam_[Unicam::Embedded].dev()->open(); + ret = data->unicam_[Unicam::Embedded].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on Unicam embedded: " + << format.toString(); + return ret; + } + + /* + * If a RAW/Bayer stream has been requested by the application, + * we must set both Unicam streams as external, even though the + * application may only request RAW frames. This is because we + * match timestamps on both streams to synchronise buffers. + */ + if (rawStream) + data->unicam_[Unicam::Embedded].setExternal(true); + } else { + /* + * No embedded data present, so we do not want to iterate over + * the embedded data stream when starting and stopping. + */ + data->streams_.erase(std::remove(data->streams_.begin(), data->streams_.end(), + &data->unicam_[Unicam::Embedded]), + data->streams_.end()); + } + /* * Update the ScalerCropMaximum to the correct value for this camera mode. * For us, it's the same as the "analogue crop". @@ -949,10 +960,16 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator) for (auto &stream : data->isp_) data->streams_.push_back(&stream); - /* Open all Unicam and ISP streams. */ + /* + * Open all Unicam and ISP streams. The exception is the embedded data + * stream, which only gets opened if the IPA reports that the sensor + * supports embedded data. This happens in RPiCameraData::configureIPA(). + */ for (auto const stream : data->streams_) { - if (stream->dev()->open()) - return false; + if (stream != &data->unicam_[Unicam::Embedded]) { + if (stream->dev()->open()) + return false; + } } /* Wire up all the buffer connections. */ @@ -1109,19 +1126,13 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) return ret; } - if (!data->sensorMetadata_) { - for (auto const &it : data->unicam_[Unicam::Embedded].getBuffers()) { - MappedFrameBuffer fb(it.second, PROT_READ | PROT_WRITE); - data->mappedEmbeddedBuffers_.emplace(it.first, std::move(fb)); - } - } - /* * Pass the stats and embedded data buffers to the IPA. No other * buffers need to be passed. */ mapBuffers(camera, data->isp_[Isp::Stats].getBuffers(), ipa::RPi::MaskStats); - mapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), ipa::RPi::MaskEmbeddedData); + if (data->sensorMetadata_) + mapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), ipa::RPi::MaskEmbeddedData); return 0; } @@ -1154,7 +1165,6 @@ void PipelineHandlerRPi::freeBuffers(Camera *camera) std::vector ipaBuffers(data->ipaBuffers_.begin(), data->ipaBuffers_.end()); data->ipa_->unmapBuffers(ipaBuffers); data->ipaBuffers_.clear(); - data->mappedEmbeddedBuffers_.clear(); for (auto const stream : data->streams_) stream->releaseBuffers(); @@ -1652,7 +1662,7 @@ void RPiCameraData::tryRunPipeline() /* If any of our request or buffer queues are empty, we cannot proceed. */ if (state_ != State::Idle || requestQueue_.empty() || - bayerQueue_.empty() || embeddedQueue_.empty()) + bayerQueue_.empty() || (embeddedQueue_.empty() && sensorMetadata_)) return; if (!findMatchingBuffers(bayerFrame, embeddedBuffer)) @@ -1675,17 +1685,24 @@ void RPiCameraData::tryRunPipeline() state_ = State::Busy; unsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerFrame.buffer); - unsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer); LOG(RPI, Debug) << "Signalling signalIspPrepare:" - << " Bayer buffer id: " << bayerId - << " Embedded buffer id: " << embeddedId; + << " Bayer buffer id: " << bayerId; ipa::RPi::ISPConfig ispPrepare; - ispPrepare.embeddedBufferId = ipa::RPi::MaskEmbeddedData | embeddedId; ispPrepare.bayerBufferId = ipa::RPi::MaskBayerData | bayerId; - ispPrepare.embeddedBufferPresent = true; ispPrepare.controls = std::move(bayerFrame.controls); + + if (embeddedBuffer) { + unsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer); + + ispPrepare.embeddedBufferId = ipa::RPi::MaskEmbeddedData | embeddedId; + ispPrepare.embeddedBufferPresent = true; + + LOG(RPI, Debug) << "Signalling signalIspPrepare:" + << " Bayer buffer id: " << embeddedId; + } + ipa_->signalIspPrepare(ispPrepare); } @@ -1727,6 +1744,17 @@ bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&em LOG(RPI, Debug) << "Could not find matching embedded buffer"; + if (!sensorMetadata_) { + /* + * If there is no sensor metadata, simply return the + * first bayer frame in the queue. + */ + LOG(RPI, Debug) << "Returning bayer frame without a match"; + bayerQueue_.pop(); + embeddedBuffer = nullptr; + return true; + } + if (!embeddedQueue_.empty()) { /* * Not found a matching embedded buffer for the bayer buffer in From patchwork Thu Feb 18 17:01:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11340 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9AB81BD1F1 for ; Thu, 18 Feb 2021 17:01:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 31C28689CA; Thu, 18 Feb 2021 18:01:37 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="Ww1jAaqj"; dkim-atps=neutral Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 00D12689C4 for ; Thu, 18 Feb 2021 18:01:35 +0100 (CET) Received: by mail-wm1-x32b.google.com with SMTP id o82so4433751wme.1 for ; Thu, 18 Feb 2021 09:01:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QOvqICc8HaszzrrUhEyZFOx1hnvZtHh8GWAPTiZOFQc=; b=Ww1jAaqjSAkt9u1JNCBFwGs8V8XNoXEa5/ofSqCBwuzWhmZE9OmhCutXjodrXPVRmF 6VIoD6SgPhX6ZVAeT03YuwYTv09FpjZLmYpifvp9XhthsL2aAy+utNoyccljgsAAOA8M YU9mBo6a4T3+z1mKkme86aRtfUGb+dTqA5agQoklLsp3R7tUISyj82DPeDnLuud8a1rL 1D8YkLoB2Y7ZryEdyef6XqrhdtQehF4HNhkuBITwGfTn9SgiPI1kgTBuYzhIopxJFHjN /9qhOJKD/EjCMGqn9bSjqfBE6OcAXOXlYUx2CLo/9m73Qc8jFFsCq60SavamLmIaDb0s 4vIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QOvqICc8HaszzrrUhEyZFOx1hnvZtHh8GWAPTiZOFQc=; b=DDe5QO+UW/53NAUN3FWmvGpqGHdNU+lcgnpStGW/R68ShuxeUmJhju8t8nD1fPzEJ5 SOBYToGKk8f/gfZHeH+Z1GVS3+N3xPFn0pvrFeidAkmKO3hYcDUTZAowaCtodSUyZwhB JTUyy73RGVsfDK/mM9ypqn0jJP95vEfsIj4w+mfAHvlbQcxfEqZxBB3nGmJUHLXcXq7A qYq3e/dB52+TLeKPP3JxNiboEAwFJJwVH4XkbubRtu7Gk6QH7lqVhqjmREbCWLvxru7r nZe87qF2kIo3gVO96MJ/PzCnPiTccAJK8GgR4K5nOOleV7tNtrRX9E6F8SVxyf38sJ7M 5uew== X-Gm-Message-State: AOAM533yv+/pDTWpcgUxqnZW6DtG2s0ulF6mk73Q9RK9EFGVD+hYESQR /AlCzBa5I/QwxoFEtumddnQ0fhEXrPkFLNCT X-Google-Smtp-Source: ABdhPJzRLFwpwcy1dvYftgMOeTD5zlcFNLbHYYJkl3/D9bF6feG1urbJyUV1gqf7BHkK70f+Ivup1g== X-Received: by 2002:a1c:771a:: with SMTP id t26mr4452037wmi.37.1613667694287; Thu, 18 Feb 2021 09:01:34 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id t30sm10504747wra.56.2021.02.18.09.01.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Feb 2021 09:01:33 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Thu, 18 Feb 2021 17:01:26 +0000 Message-Id: <20210218170126.2060783-5-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210218170126.2060783-1-naush@raspberrypi.com> References: <20210218170126.2060783-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 4/4] pipeline: raspberrypi: Allow either strict or non-strict buffer matching X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A new flag, StrictBufferMatching, is used to control the behavior of the embedded and bayer buffer matching routine. If set to true (default), we reject and drop all bayer frames that do not have a matching embedded data buffer and vice-versa. This guarantees the IPA will always have the correct frame exposure and gain values to use. If set to false, we use bayer frames that do not have a matching embedded data buffer. In this case, IPA will use use the ControlList passed to it for gain and exposure values. Additionally, allow external stream buffers to behave as if StrictBufferMatching = false since we are not allowed to drop them. Signed-off-by: Naushir Patuck --- .../pipeline/raspberrypi/raspberrypi.cpp | 31 ++++++++++++++++--- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index d969c77993eb..7f66d6995176 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -46,6 +46,22 @@ namespace libcamera { LOG_DEFINE_CATEGORY(RPI) +/* + * Set this to true to reject and drop all bayer frames that do not have a + * matching embedded data buffer and vice-versa. This guarantees the IPA will + * always have the correct frame exposure and gain values to use. + * + * Set this to false to use bayer frames that do not have a matching embedded + * data buffer. In this case, IPA will use use our local history for gain and + * exposure values, occasional frame drops may cause these number to be out of + * sync for a short period. + * + * \todo Ideally, this property should be set by the application, but we don't + * have any mechanism to pass generic properties into a pipeline handler at + * present. + */ +static const bool StrictBufferMatching = true; + namespace { bool isRaw(PixelFormat &pixFmt) @@ -1724,13 +1740,13 @@ bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&em embeddedBuffer = nullptr; while (!embeddedQueue_.empty()) { FrameBuffer *b = embeddedQueue_.front(); - if (!unicam_[Unicam::Embedded].isExternal() && b->metadata().timestamp < ts) { + if (b->metadata().timestamp < ts) { embeddedQueue_.pop(); unicam_[Unicam::Embedded].queueBuffer(b); embeddedRequeueCount++; LOG(RPI, Warning) << "Dropping unmatched input frame in stream " << unicam_[Unicam::Embedded].name(); - } else if (unicam_[Unicam::Embedded].isExternal() || b->metadata().timestamp == ts) { + } else if (b->metadata().timestamp == ts) { /* We pop the item from the queue lower down. */ embeddedBuffer = b; break; @@ -1744,10 +1760,15 @@ bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&em LOG(RPI, Debug) << "Could not find matching embedded buffer"; - if (!sensorMetadata_) { + if (unicam_[Unicam::Embedded].isExternal() || + !StrictBufferMatching || !sensorMetadata_) { /* - * If there is no sensor metadata, simply return the - * first bayer frame in the queue. + * If any of the follwing is true: + * - This is an external stream buffer (so cannot be dropped). + * - We do not care about strict buffer matching. + * - There is no sensor metadata present. + * we can simply return the first bayer frame in the queue + * without a matching embedded buffer. */ LOG(RPI, Debug) << "Returning bayer frame without a match"; bayerQueue_.pop();