From patchwork Wed Oct 19 09:01:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 17630 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 39AF9C327C for ; Wed, 19 Oct 2022 09:01:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D6F6462E55; Wed, 19 Oct 2022 11:01:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1666170085; bh=j5PQ+nf/o1EwX49vwnSLKLQnrh26xJfINWtBYZg6WO4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=IdprpAqH59Q8mT+q9WBO9AQ6hJXmBQgGz5/m5zm34UoxjwZh/l8z7WWkYwCUFVmE9 kJF7mmjD8mgnY64B2muv5RfvemfjDxIezRAYh+BCf6cYvXiEkHSCOKm5iReMuulf7h sYlwpPP7oDb2/7C0CUgswUfiSVhDf8tRdzcwYZgf4bDzlYaHFw/J60GbbY2+hNK9wJ 9KwwqnJGV492OTDSY+eLlNYZWJxj/0o0cj501J5+R86zgf2hlYsfGO3wzVBkV+vREu k4pXRDf5zz7EQtZfJf4LJGGQabjXk9Rit24t90jMazXPn4wOQPIih1rH7hU/cidN+x DOfqLZVMWlqSw== Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2A54762E4C for ; Wed, 19 Oct 2022 11:01:22 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="OjFV/vKR"; dkim-atps=neutral Received: by mail-wr1-x432.google.com with SMTP id b4so27944138wrs.1 for ; Wed, 19 Oct 2022 02:01:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C08pD0BBWsfh+mVfb1uG3tXySSr8+tqVZu1o8cdDMyA=; b=OjFV/vKRlGl+jLlYa2z5S/nRsvjFR0f9Bh3v8SFMeWEe52knR/UMNPg+yms5dfzPlQ xq4N1vTBlKhT6zcmaNUOhqFu9/q8P0VjDQgvASXSAYJWma5REcOExfgv5uAotySLTAkS Im0Sbzi4cCycRyjVBl/i1yKfzLKK1HYnCOFY/mRTZTmRVvTTkbi9QXNo5cupMT71EQpQ weayX8M//yVu5CWN189B0/VDbJAjfaREyvsyPu7fv7drh/x+mtKx4VUYGF3U6qfpUT1q KKgwErmohFmLmL2aYSms1PWecpTj/+nNAqxivi0nDDpX2Ilp0XfXhckISg6qO6rMKqxs DyZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C08pD0BBWsfh+mVfb1uG3tXySSr8+tqVZu1o8cdDMyA=; b=P5RxQKWGsgiN05pJ3lLwebJYThviTFWYQxurp2UGl98V+lA8yqnZIPC26NYTp4szwS Fba1RUWls6K0/q3PslXZV/SBT8txGLkWpRMoM39O70MHb+Vuy364RjXBXAD+nUYPW10K WAKKIr6ybeHlw/Qq4+REjMCtLbCpuODwp6M3uc1jQnxZKsy+KFra1Bw7VwgOvaRZY1gj jFjtjGEhSco6Wag83K1faRcASa8W6KNGtjrAHHZSZNh+bmZMqXVwhe7a2gY1lzo7rZMB Jq5xPcYs+XPOyT3EclC3zNza7DZaF/yn6oVCIDjaKtB26z/3ql9A6y4MuhxJNHy3b5Dg u7Iw== X-Gm-Message-State: ACrzQf0Q3AGmk+ryTre7Qoj2Um5b5fYqezAOM7rAIg1ozI7KCf0QqmF6 zOeIxtOtXEISSn3UaqBMjZ7SrnySdlxtkg== X-Google-Smtp-Source: AMsMyM6iQdwk9Pq5pmOjDs96Qotima4SJeYHbiHY4UaggRsZlxKsglX9wJT4bUEWZJFKNlI06UKEdQ== X-Received: by 2002:adf:ce0e:0:b0:22e:525:c090 with SMTP id p14-20020adfce0e000000b0022e0525c090mr4395820wrn.238.1666170081376; Wed, 19 Oct 2022 02:01:21 -0700 (PDT) Received: from naush-laptop.localdomain ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id r9-20020a05600c458900b003b47b913901sm1017595wmo.1.2022.10.19.02.01.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Oct 2022 02:01:20 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 19 Oct 2022 10:01:05 +0100 Message-Id: <20221019090107.19975-6-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221019090107.19975-1-naush@raspberrypi.com> References: <20221019090107.19975-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 5/7] ipa: raspberrypi: Use an array of RPiController::Metadata objects X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Naushir Patuck via libcamera-devel From: Naushir Patuck Reply-To: Naushir Patuck Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Allow the IPA to cycle through an array of RPiController::Metadata objects, one per prepare()/process() invocation, when running the controller algorithms. This allows historical metadata objects to be retained, and subsequently passed into the controller algorithms on future frames. At present, only a single index into this array is used. This change provides a route to fixing a problem with the AGC algorithm, where if a manual shutter/gain is requested, the algorithm does not currently retain any context of the total exposure that it has calculated. As a result, the wrong digital gain would be applied when the frame with the manual shutter/gain is processed by the ISP. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman Tested-by: David Plowman Reviewed-by: Kieran Bingham --- src/ipa/raspberrypi/raspberrypi.cpp | 65 ++++++++++++++++------------- 1 file changed, 37 insertions(+), 28 deletions(-) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index b74f1ecf738f..9e7792f5dfbe 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -57,6 +57,9 @@ namespace libcamera { using namespace std::literals::chrono_literals; using utils::Duration; +/* Number of metadata objects available in the context list. */ +constexpr unsigned int numMetadataContexts = 16; + /* Configure the sensor with these values initially. */ constexpr double defaultAnalogueGain = 1.0; constexpr Duration defaultExposureTime = 20.0ms; @@ -163,7 +166,8 @@ private: /* Raspberry Pi controller specific defines. */ std::unique_ptr helper_; RPiController::Controller controller_; - RPiController::Metadata rpiMetadata_; + std::array rpiMetadata_; + unsigned int metadataIdx_; /* * We count frames to decide if the frame must be hidden (e.g. from @@ -320,6 +324,7 @@ void IPARPi::start(const ControlList &controls, StartConfig *startConfig) firstStart_ = false; lastRunTimestamp_ = 0; + metadataIdx_ = 0; } void IPARPi::setMode(const IPACameraSensorInfo &sensorInfo) @@ -539,14 +544,15 @@ void IPARPi::signalIspPrepare(const ISPConfig &data) void IPARPi::reportMetadata() { - std::unique_lock lock(rpiMetadata_); + RPiController::Metadata &rpiMetadata = rpiMetadata_[metadataIdx_]; + std::unique_lock lock(rpiMetadata); /* * Certain information about the current frame and how it will be * processed can be extracted and placed into the libcamera metadata * buffer, where an application could query it. */ - DeviceStatus *deviceStatus = rpiMetadata_.getLocked("device.status"); + DeviceStatus *deviceStatus = rpiMetadata.getLocked("device.status"); if (deviceStatus) { libcameraMetadata_.set(controls::ExposureTime, deviceStatus->shutterSpeed.get()); @@ -557,24 +563,24 @@ void IPARPi::reportMetadata() libcameraMetadata_.set(controls::SensorTemperature, *deviceStatus->sensorTemperature); } - AgcStatus *agcStatus = rpiMetadata_.getLocked("agc.status"); + AgcStatus *agcStatus = rpiMetadata.getLocked("agc.status"); if (agcStatus) { libcameraMetadata_.set(controls::AeLocked, agcStatus->locked); libcameraMetadata_.set(controls::DigitalGain, agcStatus->digitalGain); } - LuxStatus *luxStatus = rpiMetadata_.getLocked("lux.status"); + LuxStatus *luxStatus = rpiMetadata.getLocked("lux.status"); if (luxStatus) libcameraMetadata_.set(controls::Lux, luxStatus->lux); - AwbStatus *awbStatus = rpiMetadata_.getLocked("awb.status"); + AwbStatus *awbStatus = rpiMetadata.getLocked("awb.status"); if (awbStatus) { libcameraMetadata_.set(controls::ColourGains, { static_cast(awbStatus->gainR), static_cast(awbStatus->gainB) }); libcameraMetadata_.set(controls::ColourTemperature, awbStatus->temperatureK); } - BlackLevelStatus *blackLevelStatus = rpiMetadata_.getLocked("black_level.status"); + BlackLevelStatus *blackLevelStatus = rpiMetadata.getLocked("black_level.status"); if (blackLevelStatus) libcameraMetadata_.set(controls::SensorBlackLevels, { static_cast(blackLevelStatus->blackLevelR), @@ -582,7 +588,7 @@ void IPARPi::reportMetadata() static_cast(blackLevelStatus->blackLevelG), static_cast(blackLevelStatus->blackLevelB) }); - FocusStatus *focusStatus = rpiMetadata_.getLocked("focus.status"); + FocusStatus *focusStatus = rpiMetadata.getLocked("focus.status"); if (focusStatus && focusStatus->num == 12) { /* * We get a 4x3 grid of regions by default. Calculate the average @@ -593,7 +599,7 @@ void IPARPi::reportMetadata() libcameraMetadata_.set(controls::FocusFoM, focusFoM); } - CcmStatus *ccmStatus = rpiMetadata_.getLocked("ccm.status"); + CcmStatus *ccmStatus = rpiMetadata.getLocked("ccm.status"); if (ccmStatus) { float m[9]; for (unsigned int i = 0; i < 9; i++) @@ -1006,9 +1012,10 @@ void IPARPi::prepareISP(const ISPConfig &data) { int64_t frameTimestamp = data.controls.get(controls::SensorTimestamp).value_or(0); RPiController::Metadata lastMetadata; + RPiController::Metadata &rpiMetadata = rpiMetadata_[metadataIdx_]; Span embeddedBuffer; - lastMetadata = std::move(rpiMetadata_); + lastMetadata = std::move(rpiMetadata); fillDeviceStatus(data.controls); if (data.embeddedBufferPresent) { @@ -1025,7 +1032,7 @@ void IPARPi::prepareISP(const ISPConfig &data) * This may overwrite the DeviceStatus using values from the sensor * metadata, and may also do additional custom processing. */ - helper_->prepare(embeddedBuffer, rpiMetadata_); + helper_->prepare(embeddedBuffer, rpiMetadata); /* Done with embedded data now, return to pipeline handler asap. */ if (data.embeddedBufferPresent) @@ -1041,7 +1048,7 @@ void IPARPi::prepareISP(const ISPConfig &data) * current frame, or any other bits of metadata that were added * in helper_->Prepare(). */ - rpiMetadata_.merge(lastMetadata); + rpiMetadata.merge(lastMetadata); processPending_ = false; return; } @@ -1051,48 +1058,48 @@ void IPARPi::prepareISP(const ISPConfig &data) ControlList ctrls(ispCtrls_); - controller_.prepare(&rpiMetadata_); + controller_.prepare(&rpiMetadata); /* Lock the metadata buffer to avoid constant locks/unlocks. */ - std::unique_lock lock(rpiMetadata_); + std::unique_lock lock(rpiMetadata); - AwbStatus *awbStatus = rpiMetadata_.getLocked("awb.status"); + AwbStatus *awbStatus = rpiMetadata.getLocked("awb.status"); if (awbStatus) applyAWB(awbStatus, ctrls); - CcmStatus *ccmStatus = rpiMetadata_.getLocked("ccm.status"); + CcmStatus *ccmStatus = rpiMetadata.getLocked("ccm.status"); if (ccmStatus) applyCCM(ccmStatus, ctrls); - AgcStatus *dgStatus = rpiMetadata_.getLocked("agc.status"); + AgcStatus *dgStatus = rpiMetadata.getLocked("agc.status"); if (dgStatus) applyDG(dgStatus, ctrls); - AlscStatus *lsStatus = rpiMetadata_.getLocked("alsc.status"); + AlscStatus *lsStatus = rpiMetadata.getLocked("alsc.status"); if (lsStatus) applyLS(lsStatus, ctrls); - ContrastStatus *contrastStatus = rpiMetadata_.getLocked("contrast.status"); + ContrastStatus *contrastStatus = rpiMetadata.getLocked("contrast.status"); if (contrastStatus) applyGamma(contrastStatus, ctrls); - BlackLevelStatus *blackLevelStatus = rpiMetadata_.getLocked("black_level.status"); + BlackLevelStatus *blackLevelStatus = rpiMetadata.getLocked("black_level.status"); if (blackLevelStatus) applyBlackLevel(blackLevelStatus, ctrls); - GeqStatus *geqStatus = rpiMetadata_.getLocked("geq.status"); + GeqStatus *geqStatus = rpiMetadata.getLocked("geq.status"); if (geqStatus) applyGEQ(geqStatus, ctrls); - DenoiseStatus *denoiseStatus = rpiMetadata_.getLocked("denoise.status"); + DenoiseStatus *denoiseStatus = rpiMetadata.getLocked("denoise.status"); if (denoiseStatus) applyDenoise(denoiseStatus, ctrls); - SharpenStatus *sharpenStatus = rpiMetadata_.getLocked("sharpen.status"); + SharpenStatus *sharpenStatus = rpiMetadata.getLocked("sharpen.status"); if (sharpenStatus) applySharpen(sharpenStatus, ctrls); - DpcStatus *dpcStatus = rpiMetadata_.getLocked("dpc.status"); + DpcStatus *dpcStatus = rpiMetadata.getLocked("dpc.status"); if (dpcStatus) applyDPC(dpcStatus, ctrls); @@ -1116,11 +1123,13 @@ void IPARPi::fillDeviceStatus(const ControlList &sensorControls) LOG(IPARPI, Debug) << "Metadata - " << deviceStatus; - rpiMetadata_.set("device.status", deviceStatus); + rpiMetadata_[metadataIdx_].set("device.status", deviceStatus); } void IPARPi::processStats(unsigned int bufferId) { + RPiController::Metadata &rpiMetadata = rpiMetadata_[metadataIdx_]; + auto it = buffers_.find(bufferId); if (it == buffers_.end()) { LOG(IPARPI, Error) << "Could not find stats buffer!"; @@ -1130,11 +1139,11 @@ void IPARPi::processStats(unsigned int bufferId) Span mem = it->second.planes()[0]; bcm2835_isp_stats *stats = reinterpret_cast(mem.data()); RPiController::StatisticsPtr statistics = std::make_shared(*stats); - helper_->process(statistics, rpiMetadata_); - controller_.process(statistics, &rpiMetadata_); + helper_->process(statistics, rpiMetadata); + controller_.process(statistics, &rpiMetadata); struct AgcStatus agcStatus; - if (rpiMetadata_.get("agc.status", agcStatus) == 0) { + if (rpiMetadata.get("agc.status", agcStatus) == 0) { ControlList ctrls(sensorCtrls_); applyAGC(&agcStatus, ctrls);