From patchwork Mon Sep 5 07:39:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 17301 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 03D1BC327E for ; Mon, 5 Sep 2022 07:40:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9911D62070; Mon, 5 Sep 2022 09:40:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1662363609; bh=VfIc76xxCBkQTU6St66/eRRU9AtqP02ewDytavvodXI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Eh1gMniuj12ODuB+83gBrSB8sHWBO0MFbiiRSnAqydDMfA94utJU3oug1UtcA5SIr 7W85Yl44rqFBEb0XNv9x3EpOlwMKYjT+Kvgn+sLKtGwrArgtIeLJIZOtJQdRUDlMki sJoQ8tHrRwIKYhTGVpty4yyPX05ws9TNeq+hrUb0MlGZlHH9kMFGNDj18OlWgOrbuu yaKgd/c5H8hJiyhPNNXFx0ifTSB8p5zqau7NYcL17b4aCPf7vLn5Pt2rnYR0+HK01x 3RHz0R2EyK+CeGnkO+tuAuu0Ai4Qn8szEF552ITe0rEznYtki7HYyuJVodFr1thLUj Y3XHhIUb1no8g== Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DFD1A62067 for ; Mon, 5 Sep 2022 09:40:07 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="XlKEgAn/"; dkim-atps=neutral Received: by mail-wr1-x42f.google.com with SMTP id e13so10144731wrm.1 for ; Mon, 05 Sep 2022 00:40:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Qqc/dg1qIaUBbIhfBkvNsu6duUiVZA3Iti9oNkXI1rw=; b=XlKEgAn//gRHcABQNgOvR336PGHhtsG6B90b98lkTAy7f99y/qOf3tY/sXkILrioYB J/qehj2NQXbHn43XJhGNn07Y77eOylVFAlTriee/nRzObk5aeDy2VGlBKPOeUXKQJLmp I1h7hhkhErl0h/ErTYe0bwfuYfbTQJCWpXkzkaSfwsWPFTI9rt1kdJSbTkJzb6fTVIKo rSacNNwdHPomnFwYWGUQRcXd3lokbmx4FCYfhAQ5aviAggQqi4z3Xg9boV/zYPl/Nn1n z3cP6lsBZFGROP20UQKEN3Xbf5JqsGD8Vdvi+aowU09YPvct4yeehX6nsih/7w6G3kn9 oGNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Qqc/dg1qIaUBbIhfBkvNsu6duUiVZA3Iti9oNkXI1rw=; b=X9Oijikj/BATXTFLyrL20IDRWFoPDtxNuF9Spx3i6rt0GneUue89jqESapTLhNalqw YnwQ7TJ/XOpepB9zLaJlQDAhQmKlxJLVafsSdqKBBm/kcXdqCvqaiJylc8m5nCqrLsoM 1PPQyY4QjDTaTqqMusvtULhtEh8W95lL7F9HTwLUIJpzKNRqTDEevzcdAS569sBv2nX4 bcGiVxGIbVW/mIVmtkCwzKPUUNBczrCHBK6J7e4FTLJ97FpCwmPLBXb6Mcetx3Hh8HrZ R/LkUBSJnSK6dTBu7eoaARXNdqdWFToZTPyJvbpzsdeUfD/gnMGdxeTIjTVDvPPuxQdP 5SCg== X-Gm-Message-State: ACgBeo1CJegVyc2SSsk5yzu+MAQhtwUTdpknHgbBXhdLZfJ3h7ugZeLV 4K4VSCQW0FiRxhUGyhX29PHcct7tu+yoxDLP X-Google-Smtp-Source: AA6agR6wROAsk8puiJOJbaC5rW8UGT5UqO0I+Dyx1lepjO/vL0CtHw/YLfGkmJrXLJyMHsf3wpBMHw== X-Received: by 2002:a5d:64a4:0:b0:226:df7c:2e5d with SMTP id m4-20020a5d64a4000000b00226df7c2e5dmr18520057wrp.416.1662363607324; Mon, 05 Sep 2022 00:40:07 -0700 (PDT) Received: from naush-laptop.localdomain ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id v16-20020a5d5910000000b00228a6ce17b4sm1342841wrd.37.2022.09.05.00.40.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Sep 2022 00:40:06 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Mon, 5 Sep 2022 08:39:54 +0100 Message-Id: <20220905073956.7342-6-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220905073956.7342-1-naush@raspberrypi.com> References: <20220905073956.7342-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 5/7] ipa: raspberrypi: Use an array of RPiController::Metadata objects X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Naushir Patuck via libcamera-devel From: Naushir Patuck Reply-To: Naushir Patuck Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Cycle through an array of RPiController::Metadata objects, one per prepare()/process() invocation, when running the controller algorithms. This allows historical metadata objects to be retained, and subsequently passed into the controller algorithms on future frames. This change provides a route to fixing a problem with the AGC algorithm, where if a manual shutter/gain is requested, the algorithm does not currently retain any context of the total exposure that it has calculated. As a result, the wrong digital gain would be applied when the frame with the manual shutter/gain is processed by the ISP. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman Tested-by: David Plowman --- src/ipa/raspberrypi/raspberrypi.cpp | 69 +++++++++++++++++------------ 1 file changed, 40 insertions(+), 29 deletions(-) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 8d731435764e..63cccda901c1 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -57,6 +57,9 @@ namespace libcamera { using namespace std::literals::chrono_literals; using utils::Duration; +/* Number of metadata objects available in the context list. */ +constexpr unsigned int numMetadataContexts = 16; + /* Configure the sensor with these values initially. */ constexpr double defaultAnalogueGain = 1.0; constexpr Duration defaultExposureTime = 20.0ms; @@ -163,7 +166,8 @@ private: /* Raspberry Pi controller specific defines. */ std::unique_ptr helper_; RPiController::Controller controller_; - RPiController::Metadata rpiMetadata_; + std::array rpiMetadata_; + unsigned int metadataIdx_; /* * We count frames to decide if the frame must be hidden (e.g. from @@ -319,6 +323,7 @@ void IPARPi::start(const ControlList &controls, StartConfig *startConfig) firstStart_ = false; lastRunTimestamp_ = 0; + metadataIdx_ = 0; } void IPARPi::setMode(const IPACameraSensorInfo &sensorInfo) @@ -513,6 +518,7 @@ void IPARPi::signalStatReady(uint32_t bufferId) reportMetadata(); statsMetadataComplete.emit(bufferId & MaskID, libcameraMetadata_); + metadataIdx_ = (metadataIdx_ + 1) % rpiMetadata_.size(); } void IPARPi::signalQueueRequest(const ControlList &controls) @@ -536,14 +542,15 @@ void IPARPi::signalIspPrepare(const ISPConfig &data) void IPARPi::reportMetadata() { - std::unique_lock lock(rpiMetadata_); + RPiController::Metadata &rpiMetadata = rpiMetadata_[metadataIdx_]; + std::unique_lock lock(rpiMetadata); /* * Certain information about the current frame and how it will be * processed can be extracted and placed into the libcamera metadata * buffer, where an application could query it. */ - DeviceStatus *deviceStatus = rpiMetadata_.getLocked("device.status"); + DeviceStatus *deviceStatus = rpiMetadata.getLocked("device.status"); if (deviceStatus) { libcameraMetadata_.set(controls::ExposureTime, deviceStatus->shutterSpeed.get()); @@ -554,17 +561,17 @@ void IPARPi::reportMetadata() libcameraMetadata_.set(controls::SensorTemperature, *deviceStatus->sensorTemperature); } - AgcStatus *agcStatus = rpiMetadata_.getLocked("agc.status"); + AgcStatus *agcStatus = rpiMetadata.getLocked("agc.status"); if (agcStatus) { libcameraMetadata_.set(controls::AeLocked, agcStatus->locked); libcameraMetadata_.set(controls::DigitalGain, agcStatus->digitalGain); } - LuxStatus *luxStatus = rpiMetadata_.getLocked("lux.status"); + LuxStatus *luxStatus = rpiMetadata.getLocked("lux.status"); if (luxStatus) libcameraMetadata_.set(controls::Lux, luxStatus->lux); - AwbStatus *awbStatus = rpiMetadata_.getLocked("awb.status"); + AwbStatus *awbStatus = rpiMetadata.getLocked("awb.status"); if (awbStatus) { libcameraMetadata_.set(controls::ColourGains, Span({ static_cast(awbStatus->gainR), @@ -572,7 +579,7 @@ void IPARPi::reportMetadata() libcameraMetadata_.set(controls::ColourTemperature, awbStatus->temperatureK); } - BlackLevelStatus *blackLevelStatus = rpiMetadata_.getLocked("black_level.status"); + BlackLevelStatus *blackLevelStatus = rpiMetadata.getLocked("black_level.status"); if (blackLevelStatus) libcameraMetadata_.set(controls::SensorBlackLevels, Span({ static_cast(blackLevelStatus->blackLevelR), @@ -580,7 +587,7 @@ void IPARPi::reportMetadata() static_cast(blackLevelStatus->blackLevelG), static_cast(blackLevelStatus->blackLevelB) })); - FocusStatus *focusStatus = rpiMetadata_.getLocked("focus.status"); + FocusStatus *focusStatus = rpiMetadata.getLocked("focus.status"); if (focusStatus && focusStatus->num == 12) { /* * We get a 4x3 grid of regions by default. Calculate the average @@ -591,7 +598,7 @@ void IPARPi::reportMetadata() libcameraMetadata_.set(controls::FocusFoM, focusFoM); } - CcmStatus *ccmStatus = rpiMetadata_.getLocked("ccm.status"); + CcmStatus *ccmStatus = rpiMetadata.getLocked("ccm.status"); if (ccmStatus) { float m[9]; for (unsigned int i = 0; i < 9; i++) @@ -1002,10 +1009,10 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId) void IPARPi::prepareISP(const ISPConfig &data) { int64_t frameTimestamp = data.controls.get(controls::SensorTimestamp).value_or(0); - RPiController::Metadata lastMetadata; + RPiController::Metadata &rpiMetadata = rpiMetadata_[metadataIdx_]; Span embeddedBuffer; - lastMetadata = std::move(rpiMetadata_); + rpiMetadata.clear(); fillDeviceStatus(data.controls); if (data.embeddedBufferPresent) { @@ -1022,7 +1029,7 @@ void IPARPi::prepareISP(const ISPConfig &data) * This may overwrite the DeviceStatus using values from the sensor * metadata, and may also do additional custom processing. */ - helper_->prepare(embeddedBuffer, rpiMetadata_); + helper_->prepare(embeddedBuffer, rpiMetadata); /* Done with embedded data now, return to pipeline handler asap. */ if (data.embeddedBufferPresent) @@ -1038,7 +1045,9 @@ void IPARPi::prepareISP(const ISPConfig &data) * current frame, or any other bits of metadata that were added * in helper_->Prepare(). */ - rpiMetadata_.merge(lastMetadata); + RPiController::Metadata &lastMetadata = + rpiMetadata_[metadataIdx_ == 0 ? rpiMetadata_.size() - 1 : metadataIdx_ - 1]; + rpiMetadata.mergeCopy(lastMetadata); processPending_ = false; return; } @@ -1048,48 +1057,48 @@ void IPARPi::prepareISP(const ISPConfig &data) ControlList ctrls(ispCtrls_); - controller_.prepare(&rpiMetadata_); + controller_.prepare(&rpiMetadata); /* Lock the metadata buffer to avoid constant locks/unlocks. */ - std::unique_lock lock(rpiMetadata_); + std::unique_lock lock(rpiMetadata); - AwbStatus *awbStatus = rpiMetadata_.getLocked("awb.status"); + AwbStatus *awbStatus = rpiMetadata.getLocked("awb.status"); if (awbStatus) applyAWB(awbStatus, ctrls); - CcmStatus *ccmStatus = rpiMetadata_.getLocked("ccm.status"); + CcmStatus *ccmStatus = rpiMetadata.getLocked("ccm.status"); if (ccmStatus) applyCCM(ccmStatus, ctrls); - AgcStatus *dgStatus = rpiMetadata_.getLocked("agc.status"); + AgcStatus *dgStatus = rpiMetadata.getLocked("agc.status"); if (dgStatus) applyDG(dgStatus, ctrls); - AlscStatus *lsStatus = rpiMetadata_.getLocked("alsc.status"); + AlscStatus *lsStatus = rpiMetadata.getLocked("alsc.status"); if (lsStatus) applyLS(lsStatus, ctrls); - ContrastStatus *contrastStatus = rpiMetadata_.getLocked("contrast.status"); + ContrastStatus *contrastStatus = rpiMetadata.getLocked("contrast.status"); if (contrastStatus) applyGamma(contrastStatus, ctrls); - BlackLevelStatus *blackLevelStatus = rpiMetadata_.getLocked("black_level.status"); + BlackLevelStatus *blackLevelStatus = rpiMetadata.getLocked("black_level.status"); if (blackLevelStatus) applyBlackLevel(blackLevelStatus, ctrls); - GeqStatus *geqStatus = rpiMetadata_.getLocked("geq.status"); + GeqStatus *geqStatus = rpiMetadata.getLocked("geq.status"); if (geqStatus) applyGEQ(geqStatus, ctrls); - DenoiseStatus *denoiseStatus = rpiMetadata_.getLocked("denoise.status"); + DenoiseStatus *denoiseStatus = rpiMetadata.getLocked("denoise.status"); if (denoiseStatus) applyDenoise(denoiseStatus, ctrls); - SharpenStatus *sharpenStatus = rpiMetadata_.getLocked("sharpen.status"); + SharpenStatus *sharpenStatus = rpiMetadata.getLocked("sharpen.status"); if (sharpenStatus) applySharpen(sharpenStatus, ctrls); - DpcStatus *dpcStatus = rpiMetadata_.getLocked("dpc.status"); + DpcStatus *dpcStatus = rpiMetadata.getLocked("dpc.status"); if (dpcStatus) applyDPC(dpcStatus, ctrls); @@ -1111,11 +1120,13 @@ void IPARPi::fillDeviceStatus(const ControlList &sensorControls) LOG(IPARPI, Debug) << "Metadata - " << deviceStatus; - rpiMetadata_.set("device.status", deviceStatus); + rpiMetadata_[metadataIdx_].set("device.status", deviceStatus); } void IPARPi::processStats(unsigned int bufferId) { + RPiController::Metadata &rpiMetadata = rpiMetadata_[metadataIdx_]; + auto it = buffers_.find(bufferId); if (it == buffers_.end()) { LOG(IPARPI, Error) << "Could not find stats buffer!"; @@ -1125,11 +1136,11 @@ void IPARPi::processStats(unsigned int bufferId) Span mem = it->second.planes()[0]; bcm2835_isp_stats *stats = reinterpret_cast(mem.data()); RPiController::StatisticsPtr statistics = std::make_shared(*stats); - helper_->process(statistics, rpiMetadata_); - controller_.process(statistics, &rpiMetadata_); + helper_->process(statistics, rpiMetadata); + controller_.process(statistics, &rpiMetadata); struct AgcStatus agcStatus; - if (rpiMetadata_.get("agc.status", agcStatus) == 0) { + if (rpiMetadata.get("agc.status", agcStatus) == 0) { ControlList ctrls(sensorCtrls_); applyAGC(&agcStatus, ctrls);