From patchwork Tue Nov 15 09:07:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 17798 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 36034BE08B for ; Tue, 15 Nov 2022 09:08:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E316563089; Tue, 15 Nov 2022 10:08:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1668503290; bh=jlRfxeMNJmHpGoiVrybdPgGhig7tY5AWa9GoNAXL7UI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=UlrBDyOiYjup7vPPYfs24omnJcAffF/47OyjHlpAK2U8M0isVP8iMTBmN4CfPwRJC E8eUYAMlhyB0+VOPPhhuKqmsYej7ZQtNQcawsUBVxJJZGMYMIdYxZYOUUB/geySz4o zkxppRI8+tz5U/2hxWX0aK+ZveRjc4znXhtJngajHbcH6NX2VJOgEtEYWzsNi1XEIm IJWNqwt9CQbrJPsASkyLNqverGwfCX0vzVxsnLgO1B1ZNdLyPGh0mbk4u1bdvkXI9r qeCi6ylcyL4a97WzYUYA/vmZVLn875nIdvLQpZ7t/Pw2fEqxqFceOTzuOHPTtEkxeR MNchDahARPsGg== Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D0A8863097 for ; Tue, 15 Nov 2022 10:08:05 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="XwLnmbtt"; dkim-atps=neutral Received: by mail-wr1-x431.google.com with SMTP id z14so22945398wrn.7 for ; Tue, 15 Nov 2022 01:08:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7pCZVHEGkH4rkfudxa5QoNd2EeOZUCc279G5kmfq7Bk=; b=XwLnmbttdCZAw1Ab1iumLNvE3cbdXp3aQmRmOUuIYf9GA2bHwf2lacfFB37jvZ+Gu4 Qp1I8ePXu7c+D/UEQX7b8hUOwYuqg4INQlyAu7fvEr4G63R/SBaEidQ+hsb0wpQN84vL ioBk6yrC4NZ310lztXCh77auiLKs4xsQHXh6ixb5BdTos7zwgds0kHmIZ/NKKh+xAKMb FzlyNtKd8lpgHIpggDJoUNueCrmt4PUJYgvYBF+SgYuZbqOnr/fznzwFOzdT8HvpNNB/ WCin3uA/EybsHARZo47j6h4GpX6hlaa/lh+48QCCf5kWkKJmPfR/sBHydDvWqiIfwWmH 4z0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7pCZVHEGkH4rkfudxa5QoNd2EeOZUCc279G5kmfq7Bk=; b=awtT142bEtYoM87abqosl2Hz8tQ6CvUFEGqxIu6RKDjlfu+/h+jgKnfuvKlyNgiJHG ijSpKqc3zMQedomtPix+q4WzWMxabsi68EjeJkvwUHoHTW+7qu4cfJyWNhimH9Y8BKJ0 xW6cwSNV8HCIyiWho7kVHxUFjnszLW2pkBK1UayXPMWzf15MxYDYxNV6cSS0Xc0PF9g+ +S7StY3CbsqspRqSB0MmdGn6f5/q/tqHKkWQBwvBiC3c0hs6xLqEL1XkJQaM4Q0kC/b/ wJri578Rw5QxPozcCTr+6kjqUZidBuU5Gb29aGwFtRRxwVOZUWVyjXzNQSJfxsU89daG 1Cfg== X-Gm-Message-State: ANoB5pkmtX1udu8fCPN3K4aYal6sIFrXnXfg2bqMYr2ES3hFiaLKulE4 im7kdm0PnSQumdWs3JCDdA9xmhOhDo/uCQ== X-Google-Smtp-Source: AA0mqf49O+mFJiDRfrcBwN7wQIVAicNQ2nFaqctkuluGLqnHyt2V+oWGWSiOl9v5xWdVGDtto3KPzA== X-Received: by 2002:a5d:46c6:0:b0:236:8710:41f2 with SMTP id g6-20020a5d46c6000000b00236871041f2mr9968225wrs.239.1668503285110; Tue, 15 Nov 2022 01:08:05 -0800 (PST) Received: from naush-laptop.localdomain ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id p25-20020a05600c1d9900b003c64c186206sm15143696wms.16.2022.11.15.01.08.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 01:08:04 -0800 (PST) To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Nov 2022 09:07:53 +0000 Message-Id: <20221115090755.2921-7-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221115090755.2921-1-naush@raspberrypi.com> References: <20221115090755.2921-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v6 6/8] ipa: raspberrypi: Use an array of RPiController::Metadata objects X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Naushir Patuck via libcamera-devel From: Naushir Patuck Reply-To: Naushir Patuck Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Allow the IPA to cycle through an array of RPiController::Metadata objects, one per prepare()/process() invocation, when running the controller algorithms. This allows historical metadata objects to be retained, and subsequently passed into the controller algorithms on future frames. At present, only a single index into this array is used. This change provides a route to fixing a problem with the AGC algorithm, where if a manual shutter/gain is requested, the algorithm does not currently retain any context of the total exposure that it has calculated. As a result, the wrong digital gain would be applied when the frame with the manual shutter/gain is processed by the ISP. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman Tested-by: David Plowman Reviewed-by: Kieran Bingham --- src/ipa/raspberrypi/raspberrypi.cpp | 63 ++++++++++++++++------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index b74f1ecf738f..799a4fe70000 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -57,6 +57,9 @@ namespace libcamera { using namespace std::literals::chrono_literals; using utils::Duration; +/* Number of metadata objects available in the context list. */ +constexpr unsigned int numMetadataContexts = 16; + /* Configure the sensor with these values initially. */ constexpr double defaultAnalogueGain = 1.0; constexpr Duration defaultExposureTime = 20.0ms; @@ -163,7 +166,7 @@ private: /* Raspberry Pi controller specific defines. */ std::unique_ptr helper_; RPiController::Controller controller_; - RPiController::Metadata rpiMetadata_; + std::array rpiMetadata_; /* * We count frames to decide if the frame must be hidden (e.g. from @@ -539,14 +542,15 @@ void IPARPi::signalIspPrepare(const ISPConfig &data) void IPARPi::reportMetadata() { - std::unique_lock lock(rpiMetadata_); + RPiController::Metadata &rpiMetadata = rpiMetadata_[0]; + std::unique_lock lock(rpiMetadata); /* * Certain information about the current frame and how it will be * processed can be extracted and placed into the libcamera metadata * buffer, where an application could query it. */ - DeviceStatus *deviceStatus = rpiMetadata_.getLocked("device.status"); + DeviceStatus *deviceStatus = rpiMetadata.getLocked("device.status"); if (deviceStatus) { libcameraMetadata_.set(controls::ExposureTime, deviceStatus->shutterSpeed.get()); @@ -557,24 +561,24 @@ void IPARPi::reportMetadata() libcameraMetadata_.set(controls::SensorTemperature, *deviceStatus->sensorTemperature); } - AgcStatus *agcStatus = rpiMetadata_.getLocked("agc.status"); + AgcStatus *agcStatus = rpiMetadata.getLocked("agc.status"); if (agcStatus) { libcameraMetadata_.set(controls::AeLocked, agcStatus->locked); libcameraMetadata_.set(controls::DigitalGain, agcStatus->digitalGain); } - LuxStatus *luxStatus = rpiMetadata_.getLocked("lux.status"); + LuxStatus *luxStatus = rpiMetadata.getLocked("lux.status"); if (luxStatus) libcameraMetadata_.set(controls::Lux, luxStatus->lux); - AwbStatus *awbStatus = rpiMetadata_.getLocked("awb.status"); + AwbStatus *awbStatus = rpiMetadata.getLocked("awb.status"); if (awbStatus) { libcameraMetadata_.set(controls::ColourGains, { static_cast(awbStatus->gainR), static_cast(awbStatus->gainB) }); libcameraMetadata_.set(controls::ColourTemperature, awbStatus->temperatureK); } - BlackLevelStatus *blackLevelStatus = rpiMetadata_.getLocked("black_level.status"); + BlackLevelStatus *blackLevelStatus = rpiMetadata.getLocked("black_level.status"); if (blackLevelStatus) libcameraMetadata_.set(controls::SensorBlackLevels, { static_cast(blackLevelStatus->blackLevelR), @@ -582,7 +586,7 @@ void IPARPi::reportMetadata() static_cast(blackLevelStatus->blackLevelG), static_cast(blackLevelStatus->blackLevelB) }); - FocusStatus *focusStatus = rpiMetadata_.getLocked("focus.status"); + FocusStatus *focusStatus = rpiMetadata.getLocked("focus.status"); if (focusStatus && focusStatus->num == 12) { /* * We get a 4x3 grid of regions by default. Calculate the average @@ -593,7 +597,7 @@ void IPARPi::reportMetadata() libcameraMetadata_.set(controls::FocusFoM, focusFoM); } - CcmStatus *ccmStatus = rpiMetadata_.getLocked("ccm.status"); + CcmStatus *ccmStatus = rpiMetadata.getLocked("ccm.status"); if (ccmStatus) { float m[9]; for (unsigned int i = 0; i < 9; i++) @@ -1006,9 +1010,10 @@ void IPARPi::prepareISP(const ISPConfig &data) { int64_t frameTimestamp = data.controls.get(controls::SensorTimestamp).value_or(0); RPiController::Metadata lastMetadata; + RPiController::Metadata &rpiMetadata = rpiMetadata_[0]; Span embeddedBuffer; - lastMetadata = std::move(rpiMetadata_); + lastMetadata = std::move(rpiMetadata); fillDeviceStatus(data.controls); if (data.embeddedBufferPresent) { @@ -1025,7 +1030,7 @@ void IPARPi::prepareISP(const ISPConfig &data) * This may overwrite the DeviceStatus using values from the sensor * metadata, and may also do additional custom processing. */ - helper_->prepare(embeddedBuffer, rpiMetadata_); + helper_->prepare(embeddedBuffer, rpiMetadata); /* Done with embedded data now, return to pipeline handler asap. */ if (data.embeddedBufferPresent) @@ -1041,7 +1046,7 @@ void IPARPi::prepareISP(const ISPConfig &data) * current frame, or any other bits of metadata that were added * in helper_->Prepare(). */ - rpiMetadata_.merge(lastMetadata); + rpiMetadata.merge(lastMetadata); processPending_ = false; return; } @@ -1051,48 +1056,48 @@ void IPARPi::prepareISP(const ISPConfig &data) ControlList ctrls(ispCtrls_); - controller_.prepare(&rpiMetadata_); + controller_.prepare(&rpiMetadata); /* Lock the metadata buffer to avoid constant locks/unlocks. */ - std::unique_lock lock(rpiMetadata_); + std::unique_lock lock(rpiMetadata); - AwbStatus *awbStatus = rpiMetadata_.getLocked("awb.status"); + AwbStatus *awbStatus = rpiMetadata.getLocked("awb.status"); if (awbStatus) applyAWB(awbStatus, ctrls); - CcmStatus *ccmStatus = rpiMetadata_.getLocked("ccm.status"); + CcmStatus *ccmStatus = rpiMetadata.getLocked("ccm.status"); if (ccmStatus) applyCCM(ccmStatus, ctrls); - AgcStatus *dgStatus = rpiMetadata_.getLocked("agc.status"); + AgcStatus *dgStatus = rpiMetadata.getLocked("agc.status"); if (dgStatus) applyDG(dgStatus, ctrls); - AlscStatus *lsStatus = rpiMetadata_.getLocked("alsc.status"); + AlscStatus *lsStatus = rpiMetadata.getLocked("alsc.status"); if (lsStatus) applyLS(lsStatus, ctrls); - ContrastStatus *contrastStatus = rpiMetadata_.getLocked("contrast.status"); + ContrastStatus *contrastStatus = rpiMetadata.getLocked("contrast.status"); if (contrastStatus) applyGamma(contrastStatus, ctrls); - BlackLevelStatus *blackLevelStatus = rpiMetadata_.getLocked("black_level.status"); + BlackLevelStatus *blackLevelStatus = rpiMetadata.getLocked("black_level.status"); if (blackLevelStatus) applyBlackLevel(blackLevelStatus, ctrls); - GeqStatus *geqStatus = rpiMetadata_.getLocked("geq.status"); + GeqStatus *geqStatus = rpiMetadata.getLocked("geq.status"); if (geqStatus) applyGEQ(geqStatus, ctrls); - DenoiseStatus *denoiseStatus = rpiMetadata_.getLocked("denoise.status"); + DenoiseStatus *denoiseStatus = rpiMetadata.getLocked("denoise.status"); if (denoiseStatus) applyDenoise(denoiseStatus, ctrls); - SharpenStatus *sharpenStatus = rpiMetadata_.getLocked("sharpen.status"); + SharpenStatus *sharpenStatus = rpiMetadata.getLocked("sharpen.status"); if (sharpenStatus) applySharpen(sharpenStatus, ctrls); - DpcStatus *dpcStatus = rpiMetadata_.getLocked("dpc.status"); + DpcStatus *dpcStatus = rpiMetadata.getLocked("dpc.status"); if (dpcStatus) applyDPC(dpcStatus, ctrls); @@ -1116,11 +1121,13 @@ void IPARPi::fillDeviceStatus(const ControlList &sensorControls) LOG(IPARPI, Debug) << "Metadata - " << deviceStatus; - rpiMetadata_.set("device.status", deviceStatus); + rpiMetadata_[0].set("device.status", deviceStatus); } void IPARPi::processStats(unsigned int bufferId) { + RPiController::Metadata &rpiMetadata = rpiMetadata_[0]; + auto it = buffers_.find(bufferId); if (it == buffers_.end()) { LOG(IPARPI, Error) << "Could not find stats buffer!"; @@ -1130,11 +1137,11 @@ void IPARPi::processStats(unsigned int bufferId) Span mem = it->second.planes()[0]; bcm2835_isp_stats *stats = reinterpret_cast(mem.data()); RPiController::StatisticsPtr statistics = std::make_shared(*stats); - helper_->process(statistics, rpiMetadata_); - controller_.process(statistics, &rpiMetadata_); + helper_->process(statistics, rpiMetadata); + controller_.process(statistics, &rpiMetadata); struct AgcStatus agcStatus; - if (rpiMetadata_.get("agc.status", agcStatus) == 0) { + if (rpiMetadata.get("agc.status", agcStatus) == 0) { ControlList ctrls(sensorCtrls_); applyAGC(&agcStatus, ctrls);