From patchwork Tue Feb 16 10:31:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11307 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BDD51BD160 for ; Tue, 16 Feb 2021 10:32:14 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 72C70637E6; Tue, 16 Feb 2021 11:32:14 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="pyPvDH5w"; dkim-atps=neutral Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5A4F5637D6 for ; Tue, 16 Feb 2021 11:32:12 +0100 (CET) Received: by mail-wm1-x335.google.com with SMTP id l17so8711461wmq.2 for ; Tue, 16 Feb 2021 02:32:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=El5Uie/+IQnGOiQKboBAz6Cg/G6vkVYzBqWd3DSJDlc=; b=pyPvDH5wEFmX6y8IJsPRhvbSq6Gxu5NYXAm8/jGaLcc1ImlK1TwQwLceM4DiaS6rQS mXIwwV3LUw+DgsC9vpd/aszUZpSh4g4Rcf8Run2ZB1AUSoCjeg33JNlos7vK7bw5VCt5 9Qi8u/g31FcXEGfhvrsIXOGBfQi8qXevCmkY2eWpykCDETgrR37nA0PBLeP01q+bFgk+ hUOOpm1e0ma07OwF0o36RXQIiefTGCy8NXiI8PDezm35HDU7MpdyDQPTloS0LtNwbWcS R4NBvix2alceVPJtZUQGptcaFAmzV7ltt9ds0Dr1F/T7baHR1tSf/OXAvyQOWSqA4KQq YNpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=El5Uie/+IQnGOiQKboBAz6Cg/G6vkVYzBqWd3DSJDlc=; b=omSjduDJLJUBHpSX94pPteAWwOrrMntgBGpnDovw7OGDa5QbmGYcLxgUZshhn9wzuI XB/KDrmdmj43bGMhb8I2qBWV6lf+Lux2NdqCL8oOSuYRqgulmXU1GCPRnnXE22w10EfU P+t++D+Cw7IVVXTABgYJcE1RKArQa130j54Te/Egp9TGbOqI87ZpsYblmKPwRBB6109B ItTdOGUO+kUiwUn8/6fr5WvvMZOM3nLnHSTvPfdKvOGPo45bRtMGzrVhF8D24T7cwppv NpcEQF2ho4+m7LwKgnmZSz5a0dadyT+Tj/ltUCim3NRQFqy64pqpWyjfmkwCGMEzKSEK /poQ== X-Gm-Message-State: AOAM5316sDUMpaoiv4pc/hYUXeyIjIgNmigRm3kA6U5V9K5xc2ybo18j S6BzzyJUlZrMLOAOdqVdGKfKbbpVW6cnbUmD X-Google-Smtp-Source: ABdhPJwdyGYsalnApzZkYEiLi+xG0UTZo/ASlYp8lhvNMJ+zYwABWWoUKUlu/8t/sneVc5G+fC7LyA== X-Received: by 2002:a1c:545d:: with SMTP id p29mr2689810wmi.54.1613471531445; Tue, 16 Feb 2021 02:32:11 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id a186sm3018054wme.17.2021.02.16.02.32.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Feb 2021 02:32:10 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Tue, 16 Feb 2021 10:31:37 +0000 Message-Id: <20210216103140.1077307-2-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210216103140.1077307-1-naush@raspberrypi.com> References: <20210216103140.1077307-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/4] pipeline: ipa: raspberrypi: Pass exposure/gain values to IPA though controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When running with sensors that had no embedded data, the pipeline handler would fill a dummy embedded data buffer with gain/exposure values, and pass this buffer to the IPA together with the bayer buffer. The IPA would extract these values for use in the controller algorithms. Rework this logic entirely by having a new RPiCameraData::BayerFrame queue to replace the existing bayer queue. In addition to storing the FrameBuffer pointer, this also stores all the controls tracked by DelayedControls for that frame in a ControlList. This includes include exposure and gain values. On signalling RPi::IPA_EVENT_SIGNAL_ISP_PREPARE IPA event, the pipeline handler now passes this ControlList from the RPiCameraData::BayerFrame queue. The IPA now extracts the gain and exposure values from the ControlList instead of using RPiController::MdParserRPi to parse the embedded data buffer. Signed-off-by: Naushir Patuck --- src/ipa/raspberrypi/raspberrypi.cpp | 142 +++++++++++------- .../pipeline/raspberrypi/raspberrypi.cpp | 62 ++++---- 2 files changed, 113 insertions(+), 91 deletions(-) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index ef8b19baa754..30db1235953b 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: BSD-2-Clause */ /* - * Copyright (C) 2019-2020, Raspberry Pi (Trading) Ltd. + * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd. * * rpi.cpp - Raspberry Pi Image Processing Algorithms */ @@ -99,9 +99,11 @@ private: bool validateIspControls(); void queueRequest(const ControlList &controls); void returnEmbeddedBuffer(unsigned int bufferId); - void prepareISP(unsigned int bufferId); + void prepareISP(const IPAOperationData &event); void reportMetadata(); bool parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &deviceStatus); + void fillDeviceStatus(uint32_t exposureLines, uint32_t gainCode, + struct DeviceStatus &deviceStatus); void processStats(unsigned int bufferId); void applyFrameDurations(double minFrameDuration, double maxFrameDuration); void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls); @@ -451,15 +453,14 @@ void IPARPi::processEvent(const IPAOperationData &event) } case RPi::IPA_EVENT_SIGNAL_ISP_PREPARE: { - unsigned int embeddedbufferId = event.data[0]; - unsigned int bayerbufferId = event.data[1]; + unsigned int bayerbufferId = event.data[0]; /* * At start-up, or after a mode-switch, we may want to * avoid running the control algos for a few frames in case * they are "unreliable". */ - prepareISP(embeddedbufferId); + prepareISP(event); frameCount_++; /* Ready to push the input buffer into the ISP. */ @@ -939,70 +940,88 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId) queueFrameAction.emit(0, op); } -void IPARPi::prepareISP(unsigned int bufferId) +void IPARPi::prepareISP(const IPAOperationData &event) { struct DeviceStatus deviceStatus = {}; - bool success = parseEmbeddedData(bufferId, deviceStatus); + bool success = false; - /* Done with embedded data now, return to pipeline handler asap. */ - returnEmbeddedBuffer(bufferId); + if (event.data.size() == 2) { + /* + * Pipeline handler has supplied us with an embedded data buffer, + * so parse it and extract the exposure and gain. + */ + unsigned int bufferId = event.data[1]; + success = parseEmbeddedData(bufferId, deviceStatus); - if (success) { - ControlList ctrls(ispCtrls_); + /* Done with embedded data now, return to pipeline handler asap. */ + returnEmbeddedBuffer(bufferId); + } - rpiMetadata_.Clear(); - rpiMetadata_.Set("device.status", deviceStatus); - controller_.Prepare(&rpiMetadata_); + if (!success) { + /* + * Pipeline handler has not supplied an embedded data buffer, + * or embedded data buffer parsing has failed for some reason, + * so pull the exposure and gain values from the control list. + */ + int32_t exposureLines = event.controls[0].get(V4L2_CID_EXPOSURE).get(); + int32_t gainCode = event.controls[0].get(V4L2_CID_ANALOGUE_GAIN).get(); + fillDeviceStatus(exposureLines, gainCode, deviceStatus); + } - /* Lock the metadata buffer to avoid constant locks/unlocks. */ - std::unique_lock lock(rpiMetadata_); + ControlList ctrls(ispCtrls_); - AwbStatus *awbStatus = rpiMetadata_.GetLocked("awb.status"); - if (awbStatus) - applyAWB(awbStatus, ctrls); + rpiMetadata_.Clear(); + rpiMetadata_.Set("device.status", deviceStatus); + controller_.Prepare(&rpiMetadata_); - CcmStatus *ccmStatus = rpiMetadata_.GetLocked("ccm.status"); - if (ccmStatus) - applyCCM(ccmStatus, ctrls); + /* Lock the metadata buffer to avoid constant locks/unlocks. */ + std::unique_lock lock(rpiMetadata_); - AgcStatus *dgStatus = rpiMetadata_.GetLocked("agc.status"); - if (dgStatus) - applyDG(dgStatus, ctrls); + AwbStatus *awbStatus = rpiMetadata_.GetLocked("awb.status"); + if (awbStatus) + applyAWB(awbStatus, ctrls); - AlscStatus *lsStatus = rpiMetadata_.GetLocked("alsc.status"); - if (lsStatus) - applyLS(lsStatus, ctrls); + CcmStatus *ccmStatus = rpiMetadata_.GetLocked("ccm.status"); + if (ccmStatus) + applyCCM(ccmStatus, ctrls); - ContrastStatus *contrastStatus = rpiMetadata_.GetLocked("contrast.status"); - if (contrastStatus) - applyGamma(contrastStatus, ctrls); + AgcStatus *dgStatus = rpiMetadata_.GetLocked("agc.status"); + if (dgStatus) + applyDG(dgStatus, ctrls); - BlackLevelStatus *blackLevelStatus = rpiMetadata_.GetLocked("black_level.status"); - if (blackLevelStatus) - applyBlackLevel(blackLevelStatus, ctrls); + AlscStatus *lsStatus = rpiMetadata_.GetLocked("alsc.status"); + if (lsStatus) + applyLS(lsStatus, ctrls); - GeqStatus *geqStatus = rpiMetadata_.GetLocked("geq.status"); - if (geqStatus) - applyGEQ(geqStatus, ctrls); + ContrastStatus *contrastStatus = rpiMetadata_.GetLocked("contrast.status"); + if (contrastStatus) + applyGamma(contrastStatus, ctrls); - DenoiseStatus *denoiseStatus = rpiMetadata_.GetLocked("denoise.status"); - if (denoiseStatus) - applyDenoise(denoiseStatus, ctrls); + BlackLevelStatus *blackLevelStatus = rpiMetadata_.GetLocked("black_level.status"); + if (blackLevelStatus) + applyBlackLevel(blackLevelStatus, ctrls); - SharpenStatus *sharpenStatus = rpiMetadata_.GetLocked("sharpen.status"); - if (sharpenStatus) - applySharpen(sharpenStatus, ctrls); + GeqStatus *geqStatus = rpiMetadata_.GetLocked("geq.status"); + if (geqStatus) + applyGEQ(geqStatus, ctrls); - DpcStatus *dpcStatus = rpiMetadata_.GetLocked("dpc.status"); - if (dpcStatus) - applyDPC(dpcStatus, ctrls); + DenoiseStatus *denoiseStatus = rpiMetadata_.GetLocked("denoise.status"); + if (denoiseStatus) + applyDenoise(denoiseStatus, ctrls); - if (!ctrls.empty()) { - IPAOperationData op; - op.operation = RPi::IPA_ACTION_V4L2_SET_ISP; - op.controls.push_back(ctrls); - queueFrameAction.emit(0, op); - } + SharpenStatus *sharpenStatus = rpiMetadata_.GetLocked("sharpen.status"); + if (sharpenStatus) + applySharpen(sharpenStatus, ctrls); + + DpcStatus *dpcStatus = rpiMetadata_.GetLocked("dpc.status"); + if (dpcStatus) + applyDPC(dpcStatus, ctrls); + + if (!ctrls.empty()) { + IPAOperationData op; + op.operation = RPi::IPA_ACTION_V4L2_SET_ISP; + op.controls.push_back(ctrls); + queueFrameAction.emit(0, op); } } @@ -1019,6 +1038,7 @@ bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &devic RPiController::MdParser::Status status = helper_->Parser().Parse(mem.data()); if (status != RPiController::MdParser::Status::OK) { LOG(IPARPI, Error) << "Embedded Buffer parsing failed, error " << status; + return false; } else { uint32_t exposureLines, gainCode; if (helper_->Parser().GetExposureLines(exposureLines) != RPiController::MdParser::Status::OK) { @@ -1026,21 +1046,29 @@ bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &devic return false; } - deviceStatus.shutter_speed = helper_->Exposure(exposureLines); if (helper_->Parser().GetGainCode(gainCode) != RPiController::MdParser::Status::OK) { LOG(IPARPI, Error) << "Gain failed"; return false; } - deviceStatus.analogue_gain = helper_->Gain(gainCode); - LOG(IPARPI, Debug) << "Metadata - Exposure : " - << deviceStatus.shutter_speed << " Gain : " - << deviceStatus.analogue_gain; + fillDeviceStatus(exposureLines, gainCode, deviceStatus); } return true; } +void IPARPi::fillDeviceStatus(uint32_t exposureLines, uint32_t gainCode, + struct DeviceStatus &deviceStatus) +{ + deviceStatus.shutter_speed = helper_->Exposure(exposureLines); + deviceStatus.analogue_gain = helper_->Gain(gainCode); + + LOG(IPARPI, Debug) << "Metadata - Exposure : " + << deviceStatus.shutter_speed + << " Gain : " + << deviceStatus.analogue_gain; +} + void IPARPi::processStats(unsigned int bufferId) { auto it = buffers_.find(bufferId); diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index e47646818c73..7e744ce13172 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: LGPL-2.1-or-later */ /* - * Copyright (C) 2019-2020, Raspberry Pi (Trading) Ltd. + * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd. * * raspberrypi.cpp - Pipeline handler for Raspberry Pi devices */ @@ -188,7 +188,13 @@ public: */ enum class State { Stopped, Idle, Busy, IpaComplete }; State state_; - std::queue bayerQueue_; + + struct BayerFrame { + FrameBuffer *buffer; + ControlList controls; + }; + + std::queue bayerQueue_; std::queue embeddedQueue_; std::deque requestQueue_; @@ -213,7 +219,7 @@ public: private: void checkRequestCompleted(); void tryRunPipeline(); - bool findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *&embeddedBuffer); + bool findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&embeddedBuffer); unsigned int ispOutputCount_; }; @@ -1386,29 +1392,14 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer) << ", timestamp: " << buffer->metadata().timestamp; if (stream == &unicam_[Unicam::Image]) { - bayerQueue_.push(buffer); - } else { - embeddedQueue_.push(buffer); - - ControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence); - /* - * Sensor metadata is unavailable, so put the expected ctrl - * values (accounting for the staggered delays) into the empty - * metadata buffer. + * Lookup the sensor controls used for this frame sequence from + * StaggeredCtrl and queue them along with the frame buffer. */ - if (!sensorMetadata_) { - unsigned int bufferId = unicam_[Unicam::Embedded].getBufferId(buffer); - auto it = mappedEmbeddedBuffers_.find(bufferId); - if (it != mappedEmbeddedBuffers_.end()) { - uint32_t *mem = reinterpret_cast(it->second.maps()[0].data()); - mem[0] = ctrl.get(V4L2_CID_EXPOSURE).get(); - mem[1] = ctrl.get(V4L2_CID_ANALOGUE_GAIN).get(); - } else { - LOG(RPI, Warning) << "Failed to find embedded buffer " - << bufferId; - } - } + ControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence); + bayerQueue_.push({ buffer, std::move(ctrl) }); + } else { + embeddedQueue_.push(buffer); } handleState(); @@ -1662,7 +1653,8 @@ void RPiCameraData::applyScalerCrop(const ControlList &controls) void RPiCameraData::tryRunPipeline() { - FrameBuffer *bayerBuffer, *embeddedBuffer; + FrameBuffer *embeddedBuffer; + BayerFrame bayerFrame; IPAOperationData op; /* If any of our request or buffer queues are empty, we cannot proceed. */ @@ -1670,7 +1662,7 @@ void RPiCameraData::tryRunPipeline() bayerQueue_.empty() || embeddedQueue_.empty()) return; - if (!findMatchingBuffers(bayerBuffer, embeddedBuffer)) + if (!findMatchingBuffers(bayerFrame, embeddedBuffer)) return; /* Take the first request from the queue and action the IPA. */ @@ -1691,7 +1683,7 @@ void RPiCameraData::tryRunPipeline() /* Set our state to say the pipeline is active. */ state_ = State::Busy; - unsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerBuffer); + unsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerFrame.buffer); unsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer); LOG(RPI, Debug) << "Signalling RPi::IPA_EVENT_SIGNAL_ISP_PREPARE:" @@ -1699,26 +1691,28 @@ void RPiCameraData::tryRunPipeline() << " Embedded buffer id: " << embeddedId; op.operation = RPi::IPA_EVENT_SIGNAL_ISP_PREPARE; - op.data = { RPi::BufferMask::EMBEDDED_DATA | embeddedId, - RPi::BufferMask::BAYER_DATA | bayerId }; + op.data.push_back(RPi::BufferMask::BAYER_DATA | bayerId); + op.data.push_back(RPi::BufferMask::EMBEDDED_DATA | embeddedId); + op.controls.emplace_back(std::move(bayerFrame.controls)); + ipa_->processEvent(op); } -bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *&embeddedBuffer) +bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&embeddedBuffer) { unsigned int embeddedRequeueCount = 0, bayerRequeueCount = 0; /* Loop until we find a matching bayer and embedded data buffer. */ while (!bayerQueue_.empty()) { /* Start with the front of the bayer queue. */ - bayerBuffer = bayerQueue_.front(); + bayerFrame = bayerQueue_.front(); /* * Find the embedded data buffer with a matching timestamp to pass to * the IPA. Any embedded buffers with a timestamp lower than the * current bayer buffer will be removed and re-queued to the driver. */ - uint64_t ts = bayerBuffer->metadata().timestamp; + uint64_t ts = bayerFrame.buffer->metadata().timestamp; embeddedBuffer = nullptr; while (!embeddedQueue_.empty()) { FrameBuffer *b = embeddedQueue_.front(); @@ -1748,7 +1742,7 @@ bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer * * the front of the queue. This buffer is now orphaned, so requeue * it back to the device. */ - unicam_[Unicam::Image].queueBuffer(bayerQueue_.front()); + unicam_[Unicam::Image].queueBuffer(bayerQueue_.front().buffer); bayerQueue_.pop(); bayerRequeueCount++; LOG(RPI, Warning) << "Dropping unmatched input frame in stream " @@ -1766,7 +1760,7 @@ bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer * LOG(RPI, Warning) << "Flushing bayer stream!"; while (!bayerQueue_.empty()) { - unicam_[Unicam::Image].queueBuffer(bayerQueue_.front()); + unicam_[Unicam::Image].queueBuffer(bayerQueue_.front().buffer); bayerQueue_.pop(); } flushedBuffers = true; From patchwork Tue Feb 16 10:31:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11308 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C4E55BD160 for ; Tue, 16 Feb 2021 10:32:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3749C637E5; Tue, 16 Feb 2021 11:32:15 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="pSoUytr6"; dkim-atps=neutral Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 99D82637D7 for ; Tue, 16 Feb 2021 11:32:12 +0100 (CET) Received: by mail-wr1-x42d.google.com with SMTP id n8so12264294wrm.10 for ; Tue, 16 Feb 2021 02:32:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qmnEAoO4i7QBLDEGgA6LFmWZLM34sydbmOWer6mxaWY=; b=pSoUytr6z+CFLilg2ARCsxoe6OTXcv/7JZuto6A7mgURY5NrCs1Q7Az4yjKZ46nnqE fD0nJaGsO8bPTzHGGeMINqh2ZvKOuL8yqGZLubrQEwqDvVwEPqVbHkcHwG/SmAmxuuNk 2RqN/pN3Vh6K1DXt24gce1EM922rnozITczO6VIv8ADG7NlTSWTytE/CNPTe3JZuVqVa rbcUA4Oy2GqJpsDV7D0mR085Wdtat2XSLULHrWvjqPfVIqpsKs1yJGNOKhdnLJtB7vrh gQeGD2HDlBgDTNnZlm/rG5kS2bah6npF3YNDPW8lg5GHcYLUTxSWQK+LbnTrUcUhgbxH OgLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qmnEAoO4i7QBLDEGgA6LFmWZLM34sydbmOWer6mxaWY=; b=KvLovm+VnYTMAVXB7fYf9d9lXVen9QxaEEAjJQ9ijs3ApZVaSdqRoC1ZB6vIDzyVKO 7MXqPHrIGR+FVE5AVvfjeXOPQAvZJtunjZ60p4eRqoUBa+b3Rjf9EQbFjDBUJh6qhqpp HahXJqgnGCbR9cpD2TWtu1hJK7/2KneNiqiPD0S+fTxJeDoWHlf3DstvsAECifFp3+PT 3EPyqrHg+I5i5eWJYUMqYRFEd/UEcGFaI+kJlmtK8iqYrnnDaxXczidZt4fhaOtcIiTd S2QzXaDPRLVbCv5Mxu4r07xNiBvY1+Qqc4z3cXgyCyXPKycIV7CV6wNCbbsAYYF/iarH QgUA== X-Gm-Message-State: AOAM530lzXpQeQKjasGE1adE9OhYTlk9+sb9wLNMHgIBCDePThCJ9Ebu yPecD6hkrjgw7it3Ub35Ov8wFHbfW+DweFdH X-Google-Smtp-Source: ABdhPJwb8Fr0yOpmyKVY1BZdaHnXHBgj7UsedqYQdIWE68icUHiS1NYvTisxr2AD5jam59dilDd0Eg== X-Received: by 2002:adf:d082:: with SMTP id y2mr1351419wrh.293.1613471532138; Tue, 16 Feb 2021 02:32:12 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id a186sm3018054wme.17.2021.02.16.02.32.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Feb 2021 02:32:11 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Tue, 16 Feb 2021 10:31:38 +0000 Message-Id: <20210216103140.1077307-3-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210216103140.1077307-1-naush@raspberrypi.com> References: <20210216103140.1077307-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/4] ipa: raspberrypi: Remove MdParserRPi X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With the recent change to pass a ControlList to the IPA with exposure and gain values used for a frame, RPiController::MdParserRPi is not needed any more. Remove all traces of it. The derived CamHelper objects now pass nullptr values for the parser to the base CamHelper class when sensors do not use metadata. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman Reviewed-by: Laurent Pinchart --- src/ipa/raspberrypi/cam_helper.cpp | 9 ++++-- src/ipa/raspberrypi/cam_helper_imx219.cpp | 4 +-- src/ipa/raspberrypi/cam_helper_ov5647.cpp | 3 +- src/ipa/raspberrypi/md_parser_rpi.cpp | 37 ----------------------- src/ipa/raspberrypi/md_parser_rpi.hpp | 32 -------------------- src/ipa/raspberrypi/meson.build | 1 - 6 files changed, 8 insertions(+), 78 deletions(-) delete mode 100644 src/ipa/raspberrypi/md_parser_rpi.cpp delete mode 100644 src/ipa/raspberrypi/md_parser_rpi.hpp diff --git a/src/ipa/raspberrypi/cam_helper.cpp b/src/ipa/raspberrypi/cam_helper.cpp index 93d1b7b0296a..c9cdc39c5932 100644 --- a/src/ipa/raspberrypi/cam_helper.cpp +++ b/src/ipa/raspberrypi/cam_helper.cpp @@ -42,7 +42,8 @@ CamHelper::CamHelper(MdParser *parser, unsigned int frameIntegrationDiff) CamHelper::~CamHelper() { - delete parser_; + if (parser_) + delete parser_; } uint32_t CamHelper::ExposureLines(double exposure_us) const @@ -88,8 +89,10 @@ uint32_t CamHelper::GetVBlanking(double &exposure, double minFrameDuration, void CamHelper::SetCameraMode(const CameraMode &mode) { mode_ = mode; - parser_->SetBitsPerPixel(mode.bitdepth); - parser_->SetLineLengthBytes(0); /* We use SetBufferSize. */ + if (parser_) { + parser_->SetBitsPerPixel(mode.bitdepth); + parser_->SetLineLengthBytes(0); /* We use SetBufferSize. */ + } initialized_ = true; } diff --git a/src/ipa/raspberrypi/cam_helper_imx219.cpp b/src/ipa/raspberrypi/cam_helper_imx219.cpp index 95b8e698fe3b..0e454d0de2dc 100644 --- a/src/ipa/raspberrypi/cam_helper_imx219.cpp +++ b/src/ipa/raspberrypi/cam_helper_imx219.cpp @@ -19,8 +19,6 @@ #include "cam_helper.hpp" #if ENABLE_EMBEDDED_DATA #include "md_parser.hpp" -#else -#include "md_parser_rpi.hpp" #endif using namespace RPiController; @@ -62,7 +60,7 @@ CamHelperImx219::CamHelperImx219() #if ENABLE_EMBEDDED_DATA : CamHelper(new MdParserImx219(), frameIntegrationDiff) #else - : CamHelper(new MdParserRPi(), frameIntegrationDiff) + : CamHelper(nullptr, frameIntegrationDiff) #endif { } diff --git a/src/ipa/raspberrypi/cam_helper_ov5647.cpp b/src/ipa/raspberrypi/cam_helper_ov5647.cpp index a7f417324048..75486e900d31 100644 --- a/src/ipa/raspberrypi/cam_helper_ov5647.cpp +++ b/src/ipa/raspberrypi/cam_helper_ov5647.cpp @@ -8,7 +8,6 @@ #include #include "cam_helper.hpp" -#include "md_parser_rpi.hpp" using namespace RPiController; @@ -38,7 +37,7 @@ private: */ CamHelperOv5647::CamHelperOv5647() - : CamHelper(new MdParserRPi(), frameIntegrationDiff) + : CamHelper(nullptr, frameIntegrationDiff) { } diff --git a/src/ipa/raspberrypi/md_parser_rpi.cpp b/src/ipa/raspberrypi/md_parser_rpi.cpp deleted file mode 100644 index 2b0bcfc5f034..000000000000 --- a/src/ipa/raspberrypi/md_parser_rpi.cpp +++ /dev/null @@ -1,37 +0,0 @@ -/* SPDX-License-Identifier: BSD-2-Clause */ -/* - * Copyright (C) 2020, Raspberry Pi (Trading) Limited - * - * md_parser_rpi.cpp - Metadata parser for generic Raspberry Pi metadata - */ - -#include - -#include "md_parser_rpi.hpp" - -using namespace RPiController; - -MdParserRPi::MdParserRPi() -{ -} - -MdParser::Status MdParserRPi::Parse(void *data) -{ - if (buffer_size_bytes_ < sizeof(rpiMetadata)) - return ERROR; - - memcpy(&metadata, data, sizeof(rpiMetadata)); - return OK; -} - -MdParser::Status MdParserRPi::GetExposureLines(unsigned int &lines) -{ - lines = metadata.exposure; - return OK; -} - -MdParser::Status MdParserRPi::GetGainCode(unsigned int &gain_code) -{ - gain_code = metadata.gain; - return OK; -} diff --git a/src/ipa/raspberrypi/md_parser_rpi.hpp b/src/ipa/raspberrypi/md_parser_rpi.hpp deleted file mode 100644 index 52f54f008056..000000000000 --- a/src/ipa/raspberrypi/md_parser_rpi.hpp +++ /dev/null @@ -1,32 +0,0 @@ -/* SPDX-License-Identifier: BSD-2-Clause */ -/* - * Copyright (C) 2019, Raspberry Pi (Trading) Limited - * - * md_parser_rpi.hpp - Raspberry Pi metadata parser interface - */ -#pragma once - -#include "md_parser.hpp" - -namespace RPiController { - -class MdParserRPi : public MdParser -{ -public: - MdParserRPi(); - Status Parse(void *data) override; - Status GetExposureLines(unsigned int &lines) override; - Status GetGainCode(unsigned int &gain_code) override; - -private: - // This must be the same struct that is filled into the metadata buffer - // in the pipeline handler. - struct rpiMetadata - { - uint32_t exposure; - uint32_t gain; - }; - rpiMetadata metadata; -}; - -} diff --git a/src/ipa/raspberrypi/meson.build b/src/ipa/raspberrypi/meson.build index 9e9ea80b93ad..0baf7b7e8d4a 100644 --- a/src/ipa/raspberrypi/meson.build +++ b/src/ipa/raspberrypi/meson.build @@ -17,7 +17,6 @@ rpi_ipa_includes = [ rpi_ipa_sources = files([ 'raspberrypi.cpp', 'md_parser.cpp', - 'md_parser_rpi.cpp', 'cam_helper.cpp', 'cam_helper_ov5647.cpp', 'cam_helper_imx219.cpp', From patchwork Tue Feb 16 10:31:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11309 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0B704BD160 for ; Tue, 16 Feb 2021 10:32:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C7408637EA; Tue, 16 Feb 2021 11:32:16 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="bZbmMYVV"; dkim-atps=neutral Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A9302637E1 for ; Tue, 16 Feb 2021 11:32:13 +0100 (CET) Received: by mail-wm1-x32a.google.com with SMTP id v62so4416520wmg.4 for ; Tue, 16 Feb 2021 02:32:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X+yw7DYbSrwMABbJpyfIkob1Kdhd5yi2bvfKEpqoZoM=; b=bZbmMYVVa5c6V9VqVMr2W2Z3twwdaf4y1fIDvEtqvqQ6guk4hSXf94m7DP8amf+xE8 kXou8oolKoJXRlXAU6M67vRKAu7fksWsjxxzvpjcEj5iDzy1W559mE3yZe8Q9n630h1G 42NM1vijoEIAjyNZuYnYv3jggd6rhdbC5ESPY/uteq+xGhNJSP/RbcL+pOpkj/fPAKXh 5dzM0y13cXM0ZabMxb590Xr4wbtMbN2T6M4+6p0bZUKHuxyUQhDIuXkoRUVYjgqOn2CT iQOV1peWrBpPE79FlfmBJn/XJ510vgBY/bAOmTbFDCWHMbCT4jQvbRjmIvv1sMiV6U3V /X7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X+yw7DYbSrwMABbJpyfIkob1Kdhd5yi2bvfKEpqoZoM=; b=VWW53gMfA5MQGLU0P54UHCwWqFBOfr15GCl2B+7IhI3edJ33fxhmQo5sx2p5XdkS3i F6L2Yksgle0Gr9LJobRfdTq62gEOaSdF9k0xMDTPRmqpqI1SPMvl03bSXk/mwr+BG2/e LPbG3Mn0nRjSPgjVnSDgjPILRD0uWIa/ldPMEKahQm0EnCdA5ZPP7KHORPq9A9sRr5su SnDTmP2rKCgoAVZQQtHFsav34FVRi0UYG8a9T41OoWt/Ym9LY2pCQfJUU9qk0RHhmXhy U3WE2YRMvfJ0TlxRQZuVdwkWu7xcp87iLgEwWSnVb1jToP/j3t5uZPfpkGFybM1ljgEZ PIcw== X-Gm-Message-State: AOAM531v/9lI2a2Bsw1goPgsxXoUwrzZxIE2QG/qsZ7i3CapwcqWJEJQ Hu2K9k//+f6d+e40zRfAZFqAUVhDjOYQAWPE X-Google-Smtp-Source: ABdhPJxJUiNQEK4bKwuds7TRJhcXtsE+2ZJs/bmO3hD4JTpySG5nO4TKHcDSu71Sqay/p5da5y8PMg== X-Received: by 2002:a7b:c8cc:: with SMTP id f12mr1146676wml.1.1613471533059; Tue, 16 Feb 2021 02:32:13 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id a186sm3018054wme.17.2021.02.16.02.32.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Feb 2021 02:32:12 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Tue, 16 Feb 2021 10:31:39 +0000 Message-Id: <20210216103140.1077307-4-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210216103140.1077307-1-naush@raspberrypi.com> References: <20210216103140.1077307-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/4] pipeline: raspberrypi: Only enabled embedded stream when available X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The pipeline handler would enable and use the Unicam embedded data stream even if the sensor did not support it. This was to allow a means to pass exposure and gain values for the frame to the IPA in a synchronised way. The recent changes to get the pipeline handler to pass a ControlList with exposure and gain values means this is no longer required. Disable the use of the embedded data stream when a sensor does not support it. This change also removes the mappedEmbeddedBuffers_ map as it is no longer used. Signed-off-by: Naushir Patuck --- .../pipeline/raspberrypi/raspberrypi.cpp | 109 +++++++++++------- 1 file changed, 68 insertions(+), 41 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 7e744ce13172..b6ee014b620e 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -168,12 +168,6 @@ public: /* Stores the ids of the buffers mapped in the IPA. */ std::unordered_set ipaBuffers_; - /* - * Map of (internal) mmaped embedded data buffers, to avoid having to - * map/unmap on every frame. - */ - std::map mappedEmbeddedBuffers_; - /* DMAHEAP allocation helper. */ RPi::DmaHeap dmaHeap_; FileDescriptor lsTable_; @@ -627,14 +621,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) if (isRaw(cfg.pixelFormat)) { cfg.setStream(&data->unicam_[Unicam::Image]); - /* - * We must set both Unicam streams as external, even - * though the application may only request RAW frames. - * This is because we match timestamps on both streams - * to synchronise buffers. - */ data->unicam_[Unicam::Image].setExternal(true); - data->unicam_[Unicam::Embedded].setExternal(true); continue; } @@ -706,17 +693,6 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) return ret; } - /* Unicam embedded data output format. */ - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); - LOG(RPI, Debug) << "Setting embedded data format."; - ret = data->unicam_[Unicam::Embedded].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on Unicam embedded: " - << format.toString(); - return ret; - } - /* Figure out the smallest selection the ISP will allow. */ Rectangle testCrop(0, 0, 1, 1); data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop); @@ -733,6 +709,41 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) if (ret) LOG(RPI, Error) << "Failed to configure the IPA: " << ret; + /* + * The IPA will set data->sensorMetadata_ to true if embedded data is + * supported on this sensor. If so, open the Unicam embedded data + * node and configure the output format. + */ + if (data->sensorMetadata_) { + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); + LOG(RPI, Debug) << "Setting embedded data format."; + data->unicam_[Unicam::Embedded].dev()->open(); + ret = data->unicam_[Unicam::Embedded].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on Unicam embedded: " + << format.toString(); + return ret; + } + + /* + * If a RAW/Bayer stream has been requested by the application, + * we must set both Unicam streams as external, even though the + * application may only request RAW frames. This is because we + * match timestamps on both streams to synchronise buffers. + */ + if (rawStream) + data->unicam_[Unicam::Embedded].setExternal(true); + } else { + /* + * No embedded data present, so we do not want to iterate over + * the embedded data stream when starting and stopping. + */ + data->streams_.erase(std::remove(data->streams_.begin(), data->streams_.end(), + &data->unicam_[Unicam::Embedded]), + data->streams_.end()); + } + /* * Update the ScalerCropMaximum to the correct value for this camera mode. * For us, it's the same as the "analogue crop". @@ -950,10 +961,16 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator) for (auto &stream : data->isp_) data->streams_.push_back(&stream); - /* Open all Unicam and ISP streams. */ + /* + * Open all Unicam and ISP streams. The exception is the embedded data + * stream, which only gets opened if the IPA reports that the sensor + * supports embedded data. This happens in RPiCameraData::configureIPA(). + */ for (auto const stream : data->streams_) { - if (stream->dev()->open()) - return false; + if (stream != &data->unicam_[Unicam::Embedded]) { + if (stream->dev()->open()) + return false; + } } /* Wire up all the buffer connections. */ @@ -1110,19 +1127,13 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) return ret; } - if (!data->sensorMetadata_) { - for (auto const &it : data->unicam_[Unicam::Embedded].getBuffers()) { - MappedFrameBuffer fb(it.second, PROT_READ | PROT_WRITE); - data->mappedEmbeddedBuffers_.emplace(it.first, std::move(fb)); - } - } - /* * Pass the stats and embedded data buffers to the IPA. No other * buffers need to be passed. */ mapBuffers(camera, data->isp_[Isp::Stats].getBuffers(), RPi::BufferMask::STATS); - mapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), RPi::BufferMask::EMBEDDED_DATA); + if (data->sensorMetadata_) + mapBuffers(camera, data->unicam_[Unicam::Embedded].getBuffers(), RPi::BufferMask::EMBEDDED_DATA); return 0; } @@ -1155,7 +1166,6 @@ void PipelineHandlerRPi::freeBuffers(Camera *camera) std::vector ipaBuffers(data->ipaBuffers_.begin(), data->ipaBuffers_.end()); data->ipa_->unmapBuffers(ipaBuffers); data->ipaBuffers_.clear(); - data->mappedEmbeddedBuffers_.clear(); for (auto const stream : data->streams_) stream->releaseBuffers(); @@ -1659,7 +1669,7 @@ void RPiCameraData::tryRunPipeline() /* If any of our request or buffer queues are empty, we cannot proceed. */ if (state_ != State::Idle || requestQueue_.empty() || - bayerQueue_.empty() || embeddedQueue_.empty()) + bayerQueue_.empty() || (embeddedQueue_.empty() && sensorMetadata_)) return; if (!findMatchingBuffers(bayerFrame, embeddedBuffer)) @@ -1684,17 +1694,23 @@ void RPiCameraData::tryRunPipeline() state_ = State::Busy; unsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerFrame.buffer); - unsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer); LOG(RPI, Debug) << "Signalling RPi::IPA_EVENT_SIGNAL_ISP_PREPARE:" - << " Bayer buffer id: " << bayerId - << " Embedded buffer id: " << embeddedId; + << " Bayer buffer id: " << bayerId; + op = {}; op.operation = RPi::IPA_EVENT_SIGNAL_ISP_PREPARE; op.data.push_back(RPi::BufferMask::BAYER_DATA | bayerId); - op.data.push_back(RPi::BufferMask::EMBEDDED_DATA | embeddedId); op.controls.emplace_back(std::move(bayerFrame.controls)); + if (embeddedBuffer) { + unsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer); + + op.data.push_back(RPi::BufferMask::EMBEDDED_DATA | embeddedId); + LOG(RPI, Debug) << "Signalling RPi::IPA_EVENT_SIGNAL_ISP_PREPARE:" + << " Embedded buffer id: " << embeddedId; + } + ipa_->processEvent(op); } @@ -1736,6 +1752,17 @@ bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&em LOG(RPI, Debug) << "Could not find matching embedded buffer"; + if (!sensorMetadata_) { + /* + * If there is no sensor metadata, simply return the + * first bayer frame in the queue. + */ + LOG(RPI, Debug) << "Returning bayer frame without a match"; + bayerQueue_.pop(); + embeddedBuffer = nullptr; + return true; + } + if (!embeddedQueue_.empty()) { /* * Not found a matching embedded buffer for the bayer buffer in From patchwork Tue Feb 16 10:31:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 11310 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 73D5DBD160 for ; Tue, 16 Feb 2021 10:32:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 34CFF637EE; Tue, 16 Feb 2021 11:32:17 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="fsB5dkMx"; dkim-atps=neutral Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 61613637E5 for ; Tue, 16 Feb 2021 11:32:14 +0100 (CET) Received: by mail-wm1-x32f.google.com with SMTP id o24so13903226wmh.5 for ; Tue, 16 Feb 2021 02:32:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LZl4tMiV9T5lpjTuzgxl4XgOZ5IV08IN5LTVWylkY4s=; b=fsB5dkMxsC/1iSwoCo7Nh3xL6vBSyKjF+0juCBHzP66KaBnCWIMWT5vIwXchbavf8w 31yGVzKOkQBBh7AcMnV08DBb2nJtZnC6uHIx63d1+gOpccxSCv42U/nM2wElC6bYkPJa QtrDtz6VsOfdbHgtwKoOtTFR1rlh2Lc1ec53CG0OJPrmniSy5Dqa0Fk1/kFQlg5MViFr OEfd4xZojXsdZIk9qUdli4iTWn7Oqte1IP9mBaUs/xRDOW/TmappRYU6NuiYhCD94c2z TtAZBukVGXDoj1C/kx9EOxzQrCOaRu11Exlpw5MDlx+nT+RwpKbipemE/PBwXMghSYTd aYDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LZl4tMiV9T5lpjTuzgxl4XgOZ5IV08IN5LTVWylkY4s=; b=IgnjHsfCj+5Fvw+0RE8OTYIkvqlinfzg/TFT/uBlYMdthNvPCk4zgSj07kJsed0N+V 7oU3TN4hPeW+MmlHvh0Eu2rlrFqPZjSq0DCLgqS8nyhyh3X8LQ6QIgE8etW0HjyD3REb 0TGKFZdtfXPqiFlBCr/fPZ6H71XPtqGJ4i8oOhzE/Da0VxmvtkfpKzZSOf3JRd2L2l3t E4AaFf9wLZ6cLxFZ2lxc0GZSRWieZFGLHDl/6HsyUjnOmzaUmXVfvoD/gWzaxv6DR5sI uSY1f+487ravS3nW6rmf8fC3w2QvbR/2pe/mHzHbpeSgZucj4lm9C89e+suKiXz4PU5A dh0Q== X-Gm-Message-State: AOAM530NfWxPTvEWMiB723HtKggbgRvTRjZLGihxeFXPDsUQdfGBMjOy f2TuVP7jPoCxLpFzbdibA9dQxPSU90BnA7NT X-Google-Smtp-Source: ABdhPJxGat7M0IOdW7May/GEifsMSZnkvfnsmciMZ447+Sz6R3peWQQQKnh0PDon/sDm4Qb+Bgjk5g== X-Received: by 2002:a05:600c:4656:: with SMTP id n22mr2755579wmo.130.1613471533874; Tue, 16 Feb 2021 02:32:13 -0800 (PST) Received: from naush-laptop.patuck.local ([88.97.76.4]) by smtp.gmail.com with ESMTPSA id a186sm3018054wme.17.2021.02.16.02.32.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Feb 2021 02:32:13 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Tue, 16 Feb 2021 10:31:40 +0000 Message-Id: <20210216103140.1077307-5-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210216103140.1077307-1-naush@raspberrypi.com> References: <20210216103140.1077307-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 4/4] pipeline: raspberrypi: Allow either strict or non-strict buffer matching X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A new flag, StrictBufferMatching, is used to control the behavior of the embedded and bayer buffer matching routine. If set to true (default), we reject and drop all bayer frames that do not have a matching embedded data buffer and vice-versa. This guarantees the IPA will always have the correct frame exposure and gain values to use. If set to false, we use bayer frames that do not have a matching embedded data buffer. In this case, IPA will use use the ControlList passed to it for gain and exposure values. Additionally, allow external stream buffers to behave as if StrictBufferMatching = false since we are not allowed to drop them. Signed-off-by: Naushir Patuck --- .../pipeline/raspberrypi/raspberrypi.cpp | 31 ++++++++++++++++--- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index b6ee014b620e..b0bb12ab0f37 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -43,6 +43,22 @@ namespace libcamera { LOG_DEFINE_CATEGORY(RPI) +/* + * Set this to true to reject and drop all bayer frames that do not have a + * matching embedded data buffer and vice-versa. This guarantees the IPA will + * always have the correct frame exposure and gain values to use. + * + * Set this to false to use bayer frames that do not have a matching embedded + * data buffer. In this case, IPA will use use our local history for gain and + * exposure values, occasional frame drops may cause these number to be out of + * sync for a short period. + * + * \todo Ideally, this property should be set by the application, but we don't + * have any mechanism to pass generic properties into a pipeline handler at + * present. + */ +static const bool StrictBufferMatching = true; + namespace { bool isRaw(PixelFormat &pixFmt) @@ -1732,13 +1748,13 @@ bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&em embeddedBuffer = nullptr; while (!embeddedQueue_.empty()) { FrameBuffer *b = embeddedQueue_.front(); - if (!unicam_[Unicam::Embedded].isExternal() && b->metadata().timestamp < ts) { + if (b->metadata().timestamp < ts) { embeddedQueue_.pop(); unicam_[Unicam::Embedded].queueBuffer(b); embeddedRequeueCount++; LOG(RPI, Warning) << "Dropping unmatched input frame in stream " << unicam_[Unicam::Embedded].name(); - } else if (unicam_[Unicam::Embedded].isExternal() || b->metadata().timestamp == ts) { + } else if (b->metadata().timestamp == ts) { /* We pop the item from the queue lower down. */ embeddedBuffer = b; break; @@ -1752,10 +1768,15 @@ bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&em LOG(RPI, Debug) << "Could not find matching embedded buffer"; - if (!sensorMetadata_) { + if (unicam_[Unicam::Embedded].isExternal() || + !StrictBufferMatching || !sensorMetadata_) { /* - * If there is no sensor metadata, simply return the - * first bayer frame in the queue. + * If any of the follwing is true: + * - This is an external stream buffer (so cannot be dropped). + * - We do not care about strict buffer matching. + * - There is no sensor metadata present. + * we can simply return the first bayer frame in the queue + * without a matching embedded buffer. */ LOG(RPI, Debug) << "Returning bayer frame without a match"; bayerQueue_.pop();