From patchwork Sat Nov 13 08:49:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14591 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3C2E8BDB1C for ; Sat, 13 Nov 2021 08:49:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C77DB603EE; Sat, 13 Nov 2021 09:49:54 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="m73++KkD"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2A9816034D for ; Sat, 13 Nov 2021 09:49:51 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A1618A59; Sat, 13 Nov 2021 09:49:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793390; bh=N81w8/G3lWtPy4LdH5xSsM77aOlGhlOyxo0vtEPgaEo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m73++KkDGTLlyTL4InzD0KmhuXorC0u/YxZhIhKPtcPjnAJTNsFtxakc127aaMgNo C3Uvfwhu1HolDhpW7K29GPJrPYnL0XvoagslSAu+76nh6jVcoSGClgrl+S3dWuzuYS C8MRXk4L+g1K77wJk20eyyXGfVHNKTgyh8s4wZQM= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:34 +0100 Message-Id: <20211113084947.21892-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 01/14] ipa: ipu3: Extend ipu3 ipa interface for sensor controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen IPU3Event and IPU3Action use single ControlList for both libcamera and V4L2 controls, and it's content could be either one based on the context. Extend IPU3Event and IPU3Action for sensor V4L2 controls, and preserve the original one for only libcamera Controls to make the content of an event more specific. Signed-off-by: Han-Lin Chen [Jean-Michel: remove lensControls from the original patch] Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- include/libcamera/ipa/ipu3.mojom | 2 ++ src/ipa/ipu3/ipu3.cpp | 2 +- src/libcamera/pipeline/ipu3/ipu3.cpp | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/libcamera/ipa/ipu3.mojom b/include/libcamera/ipa/ipu3.mojom index 2f254ed4..16e3462e 100644 --- a/include/libcamera/ipa/ipu3.mojom +++ b/include/libcamera/ipa/ipu3.mojom @@ -23,11 +23,13 @@ struct IPU3Event { int64 frameTimestamp; uint32 bufferId; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; struct IPU3Action { IPU3Operations op; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; struct IPAConfigInfo { diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 93b700bd..bcc3863b 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -653,7 +653,7 @@ void IPAIPU3::setControls(unsigned int frame) ControlList ctrls(ctrls_); ctrls.set(V4L2_CID_EXPOSURE, static_cast(exposure_)); ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(gain_)); - op.controls = ctrls; + op.sensorControls = ctrls; queueFrameAction.emit(frame, op); } diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index eb714aa6..8816efc5 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -1248,7 +1248,7 @@ void IPU3CameraData::queueFrameAction(unsigned int id, { switch (action.op) { case ipa::ipu3::ActionSetSensorControls: { - const ControlList &controls = action.controls; + const ControlList &controls = action.sensorControls; delayedCtrls_->push(controls); break; } From patchwork Sat Nov 13 08:49:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14592 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 05943BDB1C for ; Sat, 13 Nov 2021 08:49:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DBD99603CB; Sat, 13 Nov 2021 09:49:55 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rsRQGYNv"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 328E660361 for ; Sat, 13 Nov 2021 09:49:51 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D50ADD89; Sat, 13 Nov 2021 09:49:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793390; bh=um/q0FWP1kb931vE+L4SfSpg2QMOmgzm/QyoW+8kTig=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rsRQGYNvE7+n8mMvAnJ/dDMFf7lU87GmonWV5eoZCYokK+tzeBz/QK2O/IvKC0ZES FYZQ3h4EG0/MZ7Fu0xYqAy3kmwsNZGwNeDoEWX2ch0v+VGbTH2FrqEu7Pui+Lmgjmt Zq2qjpas1d5P4cYo4U/Qu4QD6+2aJd/7l2s9OxP8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:35 +0100 Message-Id: <20211113084947.21892-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 02/14] ipu3: ipa: Report effective sensor controls with statistics to IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen The Intel close sourced IPA requires the effective controls applied to the sensor when the statistics are generated. Report effective sensor controls with the statistics to IPA. Signed-off-by: Han-Lin Chen [Jean-Michel: Reword s/stastistics/statistics and move reset after IPA start] Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- src/libcamera/pipeline/ipu3/frames.h | 4 ++++ src/libcamera/pipeline/ipu3/ipu3.cpp | 5 +++++ 2 files changed, 9 insertions(+) diff --git a/src/libcamera/pipeline/ipu3/frames.h b/src/libcamera/pipeline/ipu3/frames.h index 3ef7e445..614701e2 100644 --- a/src/libcamera/pipeline/ipu3/frames.h +++ b/src/libcamera/pipeline/ipu3/frames.h @@ -14,6 +14,8 @@ #include +#include + namespace libcamera { class FrameBuffer; @@ -34,6 +36,8 @@ public: FrameBuffer *paramBuffer; FrameBuffer *statBuffer; + ControlList effectiveSensorControls; + bool paramDequeued; bool metadataProcessed; }; diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 8816efc5..c984ed12 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -767,6 +767,8 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis if (ret) goto error; + data->delayedCtrls_->reset(); + /* * Start the ImgU video devices, buffers will be queued to the * ImgU output and viewfinder when requests will be queued. @@ -1363,6 +1365,8 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer) request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + info->effectiveSensorControls = delayedCtrls_->get(buffer->metadata().sequence); + if (request->findBuffer(&rawStream_)) pipe()->completeBuffer(request, buffer); @@ -1419,6 +1423,7 @@ void IPU3CameraData::statBufferReady(FrameBuffer *buffer) ev.frame = info->id; ev.bufferId = info->statBuffer->cookie(); ev.frameTimestamp = request->metadata().get(controls::SensorTimestamp); + ev.sensorControls = info->effectiveSensorControls; ipa_->processEvent(ev); } From patchwork Sat Nov 13 08:49:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14593 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 53A79C324E for ; Sat, 13 Nov 2021 08:49:58 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5C1EE603B5; Sat, 13 Nov 2021 09:49:56 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Nw+cLrrT"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5E63460368 for ; Sat, 13 Nov 2021 09:49:51 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 132C4E0C; Sat, 13 Nov 2021 09:49:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793391; bh=2bJqhll3vqe8ksNtFPj2X2w6mRPKr3rqtiWbi97iWSc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Nw+cLrrTY8k1oiIDQA/a9CFbopHrvrx/cF/77Kc5dq9418CqDLiNcPOMDJOKxYwvy yHY5LAt5SVHqYKRaZA0t3LLSixtbmFwqN3wNlWj3eemPPQCopDxHQpsTrmV17xDGlb JbgDQ65GtDjAPlSGuNZbszgit43xetbW1XeItTa8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:36 +0100 Message-Id: <20211113084947.21892-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 03/14] ipa: ipu3: Use sensor controls to update frameContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The pipeline handler populates the new sensorControls ControlList, to have the effective exposure and gain values for the current frame. This is done when a statistics buffer is received. Make those values the frameContext::sensor values for the frame when the EventStatReady event is received. AGC also needs to use frameContext.sensor as its input values and frameContext.agc as its output values. Modify computeExposure by passing it the frameContext instead of individual exposure and gain values. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 19 ++++++++++--------- src/ipa/ipu3/algorithms/agc.h | 2 +- src/ipa/ipu3/ipa_context.cpp | 11 +++++++++++ src/ipa/ipu3/ipa_context.h | 5 +++++ src/ipa/ipu3/ipu3.cpp | 3 +++ 5 files changed, 30 insertions(+), 10 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index b5d736c1..5723f6f3 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -169,10 +169,9 @@ void Agc::filterExposure() /** * \brief Estimate the new exposure and gain values - * \param[inout] exposure The exposure value reference as a number of lines - * \param[inout] gain The gain reference to be updated + * \param[inout] frameContext The shared IPA frame Context */ -void Agc::computeExposure(uint32_t &exposure, double &analogueGain) +void Agc::computeExposure(IPAFrameContext &frameContext) { /* Algorithm initialization should wait for first valid frames */ /* \todo - have a number of frames given by DelayedControls ? @@ -189,6 +188,10 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) return; } + /* Get the effective exposure and gain applied on the sensor. */ + uint32_t exposure = frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + /* Estimate the gain needed to have the proportion wanted */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; @@ -233,8 +236,9 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) << shutterTime << " and " << stepGain; - exposure = shutterTime / lineDuration_; - analogueGain = stepGain; + /* Update the estimated exposure and gain. */ + frameContext.agc.exposure = shutterTime / lineDuration_; + frameContext.agc.gain = stepGain; /* * Update the exposure value for the next process call. @@ -257,11 +261,8 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) */ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { - /* Get the latest exposure and gain applied */ - uint32_t &exposure = context.frameContext.agc.exposure; - double &analogueGain = context.frameContext.agc.gain; measureBrightness(stats, context.configuration.grid.bdsGrid); - computeExposure(exposure, analogueGain); + computeExposure(context.frameContext); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 69e0b831..f0db25ee 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -34,7 +34,7 @@ private: void measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); void filterExposure(); - void computeExposure(uint32_t &exposure, double &gain); + void computeExposure(IPAFrameContext &frameContext); uint64_t frameCount_; uint64_t lastFrame_; diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 2355a9c7..a7ff957d 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -119,6 +119,17 @@ namespace libcamera::ipa::ipu3 { * \brief White balance gain for B channel */ +/** + * \var IPAFrameContext::sensor + * \brief Effective sensor values + * + * \var IPAFrameContext::sensor.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::sensor.gain + * \brief Analogue gain multiplier + */ + /** * \var IPAFrameContext::toneMapping * \brief Context for ToneMapping and Gamma control diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 1e46c61a..a5a19800 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -47,6 +47,11 @@ struct IPAFrameContext { } gains; } awb; + struct { + uint32_t exposure; + double gain; + } sensor; + struct { double gamma; struct ipu3_uapi_gamma_corr_lut gammaCorrection; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index bcc3863b..38e86e58 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -549,6 +549,9 @@ void IPAIPU3::processEvent(const IPU3Event &event) const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); + context_.frameContext.sensor.exposure = event.sensorControls.get(V4L2_CID_EXPOSURE).get(); + context_.frameContext.sensor.gain = camHelper_->gain(event.sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + parseStatistics(event.frame, event.frameTimestamp, stats); break; } From patchwork Sat Nov 13 08:49:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14594 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8D013BDB1C for ; Sat, 13 Nov 2021 08:49:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 39BD860451; Sat, 13 Nov 2021 09:49:57 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="mKHVN0gY"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 939296036C for ; Sat, 13 Nov 2021 09:49:51 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4A2FB1490; Sat, 13 Nov 2021 09:49:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793391; bh=1+ZpyVr8E4p//ZS1A4+q8pMHMpddDrE2Nbise2ZoWj8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mKHVN0gYysefe1M5R0kGMIKBKWzJl2mhWqeEkGOiW4e5QfZofv8ZhA4RGn3YqjtnJ vRnU5ULJ88vLY+zDYjSD/lvvVZGsDrzYyiDNF1Vwg7I5E2OOBotKqHd5ecgnC6R67K LZmQPfZZvZ7mDlNl4j4UoHWT2Bi/qZOiYCSP4O7o= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:37 +0100 Message-Id: <20211113084947.21892-5-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 04/14] ipa: ipu3: agc: Limit the number of saturated cells X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the histogram is calculated, we check if a cell is saturated or not before cumulating its green value. This is wrong, and it can lead to an empty histogram in case of a fully saturated frame. Use a constant to limit the amount of pixels within a cell before considering it saturated. If at the end of the loop we still have an empty histogram, then make it a fully saturated one. Bug: https://bugs.libcamera.org/show_bug.cgi?id=84 Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 5723f6f3..83aa3676 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -63,6 +63,12 @@ static constexpr uint32_t knumHistogramBins = 256; /* Target value to reach for the top 2% of the histogram */ static constexpr double kEvGainTarget = 0.5; +/* + * Maximum ratio of saturated pixels in a cell for the cell to be considered + * non-saturated and counted by the AGC algorithm. + */ +static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; + Agc::Agc() : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), @@ -124,7 +130,7 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, &stats->awb_raw_buffer.meta_data[cellPosition] ); - if (cell->sat_ratio == 0) { + if (cell->sat_ratio <= kMinCellsPerZoneRatio) { uint8_t gr = cell->Gr_avg; uint8_t gb = cell->Gb_avg; /* @@ -137,8 +143,14 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, } } + Histogram cumulativeHist = Histogram(Span(hist)); /* Estimate the quantile mean of the top 2% of the histogram */ - iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); + if (cumulativeHist.total() == 0) { + /* Force the value as histogram is empty */ + iqMean_ = knumHistogramBins - 0.5; + } else { + iqMean_ = cumulativeHist.interQuantileMean(0.98, 1.0); + } } /** From patchwork Sat Nov 13 08:49:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14595 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 59E3AC324E for ; Sat, 13 Nov 2021 08:50:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1583A6036B; Sat, 13 Nov 2021 09:49:58 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="A8ej15Q+"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C7E046036E for ; Sat, 13 Nov 2021 09:49:51 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7B90714D4; Sat, 13 Nov 2021 09:49:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793391; bh=NPkKl64NFgcRyNRNm3kmc+7yMoKbCTwDFvSsJ4bcQFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=A8ej15Q+HPP9Lr9dDp7iT8ASMZrg4mwYajXAn5RaBKODFyI5P2CjUdWzEkxmeVnmo ZTiKrbRWnfMhF2zBIATY+uoBw4inHIuyGLlmkx+Hj4bVCOLkAtE2rETp0PD1BHuzFY 18TqG/csWfV1dSW0l0kFflut3N/NBRJ72ZCuCi3w= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:38 +0100 Message-Id: <20211113084947.21892-6-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 05/14] ipa: ipu3: agc: Compute the gain for each frame X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have the real exposure applied at each frame, remove the early return based on a frame counter and compute the gain for each frame. Introduce a number of startup frames during which the filter speed is 1.0, meaning we apply instantly the exposure value calculated and not a slower filtered one. This is used to have a faster convergence, and those frames may be dropped in a future development to hide the convergance process from the viewer. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 83aa3676..74bce7bb 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -45,11 +45,6 @@ namespace ipa::ipu3::algorithms { LOG_DEFINE_CATEGORY(IPU3Agc) -/* Number of frames to wait before calculating stats on minimum exposure */ -static constexpr uint32_t kInitialFrameMinAECount = 4; -/* Number of frames to wait between new gain/shutter time estimations */ -static constexpr uint32_t kFrameSkipCount = 6; - /* Limits for analogue gain values */ static constexpr double kMinAnalogueGain = 1.0; static constexpr double kMaxAnalogueGain = 8.0; @@ -69,10 +64,13 @@ static constexpr double kEvGainTarget = 0.5; */ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; +/* Number of frames to wait before calculating stats on minimum exposure */ +static constexpr uint32_t kNumStartupFrames = 10; + Agc::Agc() - : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), - minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), - currentExposure_(0s), prevExposureValue_(0s) + : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), + maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), + prevExposureValue_(0s) { } @@ -159,6 +157,11 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, void Agc::filterExposure() { double speed = 0.2; + + /* Adapt instantly if we are in startup phase */ + if (frameCount_ < kNumStartupFrames) + speed = 1.0; + if (filteredExposure_ == 0s) { /* DG stands for digital gain.*/ filteredExposure_ = currentExposure_; @@ -185,13 +188,6 @@ void Agc::filterExposure() */ void Agc::computeExposure(IPAFrameContext &frameContext) { - /* Algorithm initialization should wait for first valid frames */ - /* \todo - have a number of frames given by DelayedControls ? - * - implement a function for IIR */ - if ((frameCount_ < kInitialFrameMinAECount) || (frameCount_ - lastFrame_ < kFrameSkipCount)) - return; - - lastFrame_ = frameCount_; /* Are we correctly exposed ? */ if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { From patchwork Sat Nov 13 08:49:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14596 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0E80BBDB1C for ; Sat, 13 Nov 2021 08:50:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0DE01603BE; Sat, 13 Nov 2021 09:49:59 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="af6abFfk"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 016D960371 for ; Sat, 13 Nov 2021 09:49:52 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B14FA1875; Sat, 13 Nov 2021 09:49:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793391; bh=CYzs0lL4OB90psMr5a0ZMVciVBGNeZuxkR+LeIfhqEA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=af6abFfk+5nFFk7WUGZ8ULaX/ZCRNU1wAJushrUBm1HGF4rKVlAq5yKRds+EJpCjc xPy3Wb97cOkyvHbCQgb5xznlzPsNNf+2dZcHfI0CUT3exSt7D6+Pg8ANjbmYD/DTVn wKsV//34bjXDndK+1ua3HJ9AjOJjV5Bew1ZO6z70= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:39 +0100 Message-Id: <20211113084947.21892-7-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 06/14] ipa: ipu3: agc: Refactor ev gain calculation and testing X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When we compute the new gain, we use the iqMean_ and estimate an exposure value gain to apply. Return early when the gain is less than 1%. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 74bce7bb..61ca8b3f 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -188,14 +188,6 @@ void Agc::filterExposure() */ void Agc::computeExposure(IPAFrameContext &frameContext) { - - /* Are we correctly exposed ? */ - if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { - LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " - << iqMean_ << ")"; - return; - } - /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; double analogueGain = frameContext.sensor.gain; @@ -203,6 +195,12 @@ void Agc::computeExposure(IPAFrameContext &frameContext) /* Estimate the gain needed to have the proportion wanted */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; + if (std::abs(evGain - 1.0) < 0.01) { + LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " + << iqMean_ << ")"; + return; + } + /* extracted from Rpi::Agc::computeTargetExposure */ /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; From patchwork Sat Nov 13 08:49:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14597 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A759DC324F for ; Sat, 13 Nov 2021 08:50:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 16A2860378; Sat, 13 Nov 2021 09:50:00 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="gbpccngi"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3BBE56037A for ; Sat, 13 Nov 2021 09:49:52 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E3FDF55C; Sat, 13 Nov 2021 09:49:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793392; bh=W225EcVU6hbpwGKw9tcVP/QWB86TcOz3cnYTPXGyL18=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gbpccngihnVrZ7h9ZciyM7pkNscoTmBKJUB+8D+tnkxJigrDLjrj/st9VxvixEM1H I3mNKz83SmwLX5HBqPfvWXjUxLhRrbyimd7QfAF8H/riaBng6I3axSi10UIqneLX0F HgSa6D3e1oo6ZKM5Yvoe7Iiqg3B0ecvL4Am0rP+0= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:40 +0100 Message-Id: <20211113084947.21892-8-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 07/14] ipa: ipu3: agc: Improve gain calculation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When an image is partially saturated, its brightness is not increasing linearly when the shutter time or gain increases. It is a big issue with a backlight as the algorithm is fading to darkness right now. Introduce a function to estimate the brightness of the frame, based on the current exposure/gain and loop on it several times to estimate it again and approach the non linear function. Inspired-by: 7de5506c30b3 ("libcamera: src: ipa: raspberrypi: agc: Improve gain update calculation for partly saturated images") Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder Reviewed-by: Umang Jain Reviewed-by: Umang Jain --- src/ipa/ipu3/algorithms/agc.cpp | 104 ++++++++++++++++++++++++++++++-- src/ipa/ipu3/algorithms/agc.h | 6 +- 2 files changed, 103 insertions(+), 7 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 61ca8b3f..a9f9244b 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -67,6 +67,17 @@ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; /* Number of frames to wait before calculating stats on minimum exposure */ static constexpr uint32_t kNumStartupFrames = 10; +/* Maximum luminance used for brightness normalization */ +static constexpr uint32_t kMaxLuminance = 255; + +/* + * Normalized luma value target. + * + * It's a number that's chosen so that, when the camera points at a grey + * target, the resulting image brightness is considered right. + */ +static constexpr double kNormalizedLumaTarget = 0.16; + Agc::Agc() : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), @@ -185,23 +196,32 @@ void Agc::filterExposure() /** * \brief Estimate the new exposure and gain values * \param[inout] frameContext The shared IPA frame Context + * \param[in] currentYGain The gain calculated on the current brightness level */ -void Agc::computeExposure(IPAFrameContext &frameContext) +void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) { /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; double analogueGain = frameContext.sensor.gain; - /* Estimate the gain needed to have the proportion wanted */ + /* + * Estimate the gain needed to have the proportion of pixels in a given + * range wanted. iqMean_ returns the mean value of the top 2% of the + * cumulative histogram, and we want it to be as close as possible to a + * configured target. + */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; - if (std::abs(evGain - 1.0) < 0.01) { + if (evGain < currentYGain) + evGain = currentYGain; + + /* Consider within 1% of the target as correctly exposed */ + if (std::abs(evGain - 1.0) < 0.01) LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " << iqMean_ << ")"; - return; - } /* extracted from Rpi::Agc::computeTargetExposure */ + /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain @@ -257,6 +277,56 @@ void Agc::computeExposure(IPAFrameContext &frameContext) prevExposureValue_ = shutterTime * analogueGain; } +/** + * \brief Estimate the average brightness of the frame + * \param[in] frameContext The shared IPA frame context + * \param[in] grid The grid used to store the statistics in the IPU3 + * \param[in] stats The IPU3 statistics and ISP results + * \param[in] currentYGain The gain calculated on the current brightness level + * \return The normalized luma + * + * Luma is the weighted sum of gamma-compressed R′G′B′ components of a color + * video. The luma values are normalized as 0.0 to 1.0, with 1.0 being a + * theoretical perfect reflector of 100% reference white. We use the Rec. 601 + * luma here. + * + * More detailed information can be found in: + * https://en.wikipedia.org/wiki/Luma_(video) + */ +double Agc::computeInitialY(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain) +{ + double redSum = 0, greenSum = 0, blueSum = 0; + + for (unsigned int cellY = 0; cellY < grid.height; cellY++) { + for (unsigned int cellX = 0; cellX < grid.width; cellX++) { + uint32_t cellPosition = cellY * stride_ + cellX; + + const ipu3_uapi_awb_set_item *cell = + reinterpret_cast( + &stats->awb_raw_buffer.meta_data[cellPosition] + ); + + redSum += cell->R_avg * currentYGain; + greenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * currentYGain; + blueSum += cell->B_avg * currentYGain; + } + } + + /* + * Estimate the sum of the brightness values, weighted with the gains + * applied on the channels in AWB as the Rec. 601 luma. + */ + double Y_sum = redSum * frameContext.awb.gains.red * .299 + + greenSum * frameContext.awb.gains.green * .587 + + blueSum * frameContext.awb.gains.blue * .114; + + /* Return the normalized relative luminance. */ + return Y_sum / (grid.height * grid.width) / kMaxLuminance; +} + /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -268,7 +338,29 @@ void Agc::computeExposure(IPAFrameContext &frameContext) void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { measureBrightness(stats, context.configuration.grid.bdsGrid); - computeExposure(context.frameContext); + + double currentYGain = 1.0; + double targetY = kNormalizedLumaTarget; + + /* + * Do this calculation a few times as brightness increase can be + * non-linear when there are saturated regions. + */ + for (int i = 0; i < 8; i++) { + double initialY = computeInitialY(context.frameContext, + context.configuration.grid.bdsGrid, + stats, currentYGain); + double extra_gain = std::min(10.0, targetY / (initialY + .001)); + + currentYGain *= extra_gain; + LOG(IPU3Agc, Debug) << "Initial Y " << initialY + << " target " << targetY + << " gives gain " << currentYGain; + if (extra_gain < 1.01) + break; + } + + computeExposure(context.frameContext, currentYGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index f0db25ee..79736283 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -34,7 +34,11 @@ private: void measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); void filterExposure(); - void computeExposure(IPAFrameContext &frameContext); + void computeExposure(IPAFrameContext &frameContext, double currentYGain); + double computeInitialY(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain); uint64_t frameCount_; uint64_t lastFrame_; From patchwork Sat Nov 13 08:49:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14598 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 006ECC324E for ; Sat, 13 Nov 2021 08:50:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 21CF96038B; Sat, 13 Nov 2021 09:50:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="uS2dylEp"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6FF7760361 for ; Sat, 13 Nov 2021 09:49:52 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 23C3AA59; Sat, 13 Nov 2021 09:49:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793392; bh=m6GOjqFnfCd7ScEQMjXEwqvnr4QEjrc3f7fYRteZFSo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uS2dylEpfuDqUwmVYX/M1N003oE5EXM//ihR0HIw8NwnSqvt1RPnKBT25TO0AxzFa RsJI423gfCH4lnVbFHL6wjNrvjYHSgMLoairfiOHDqPT+OilDAnxgb2F/zk7fnUCnK j5N8EiSBXdV6czLeRnLgiGkJ40tWNlFIvipAfp1k= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:41 +0100 Message-Id: <20211113084947.21892-9-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 08/14] ipa: ipu3: agc: Update previous exposure value X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Previously, the exposure value was calculated based on the estimated shutter time and gain applied. Now that we have the real values for the current frame, use those before estimating the next one and rename the variable accordingly. As the exposure value is updated in the beginning of the computation, there is no need to initialize effectiveExposureValue anymore in the configure call, and it can be a local variable and not a class variable anymore. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 26 +++++++++----------------- src/ipa/ipu3/algorithms/agc.h | 1 - 2 files changed, 9 insertions(+), 18 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index a9f9244b..4d2450fc 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -80,8 +80,7 @@ static constexpr double kNormalizedLumaTarget = 0.16; Agc::Agc() : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), - maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), - prevExposureValue_(0s) + maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s) { } @@ -112,10 +111,6 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) context.frameContext.agc.gain = minAnalogueGain_; context.frameContext.agc.exposure = minExposureLines_; - prevExposureValue_ = context.frameContext.agc.gain - * context.frameContext.agc.exposure - * lineDuration_; - return 0; } @@ -224,6 +219,13 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; + + /* + * Update the exposure value for the next computation using the values + * of exposure and gain really used by the sensor. + */ + utils::Duration effectiveExposureValue = currentShutter * analogueGain; + LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain << " Shutter speed " << currentShutter << " Gain " << analogueGain @@ -233,7 +235,7 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * Calculate the current exposure value for the scene as the latest * exposure value applied multiplied by the new estimated gain. */ - currentExposure_ = prevExposureValue_ * evGain; + currentExposure_ = effectiveExposureValue * evGain; utils::Duration minShutterSpeed = minExposureLines_ * lineDuration_; utils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_; @@ -265,16 +267,6 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Update the estimated exposure and gain. */ frameContext.agc.exposure = shutterTime / lineDuration_; frameContext.agc.gain = stepGain; - - /* - * Update the exposure value for the next process call. - * - * \todo Obtain the values of the exposure time and analog gain - * that were actually used by the sensor, either from embedded - * data when available, or from the delayed controls - * infrastructure in case a slow down caused a mismatch. - */ - prevExposureValue_ = shutterTime * analogueGain; } /** diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 79736283..1d085b40 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -54,7 +54,6 @@ private: utils::Duration filteredExposure_; utils::Duration currentExposure_; - utils::Duration prevExposureValue_; uint32_t stride_; }; From patchwork Sat Nov 13 08:49:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14599 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6E8B9C3250 for ; Sat, 13 Nov 2021 08:50:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D565860451; Sat, 13 Nov 2021 09:50:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pxpaR9/v"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A0E516038B for ; Sat, 13 Nov 2021 09:49:52 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 589AFD89; Sat, 13 Nov 2021 09:49:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793392; bh=v9d+/ELAZJnzhIawUaDoM5inmBpwqteDirstqqx0rrI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pxpaR9/vkCtWi9Ybxus/OM78JDCCeXwKehxY1SDxg13iLzm0jBplLDCy9rXL30IRl hzsGczofE+BBYb2VgN+wC72oXD89j0inYQRFK7RGs7SJYFzXMwntZG5WWCSmf/odWQ DLaRdCE70GxiaUXcWoqnegflrICA22raZy4B8+4A= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:42 +0100 Message-Id: <20211113084947.21892-10-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 09/14] ipa: ipu3: agc: Store exposure in units of time X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The minimum and maximum exposure are stored in lines. Replace it by values in time to simplify the calculations. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 21 +++++++++------------ src/ipa/ipu3/algorithms/agc.h | 4 ++-- 2 files changed, 11 insertions(+), 14 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 4d2450fc..1e6f2b64 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -79,8 +79,8 @@ static constexpr uint32_t kMaxLuminance = 255; static constexpr double kNormalizedLumaTarget = 0.16; Agc::Agc() - : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), - maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s) + : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minShutterSpeed_(0s), + maxShutterSpeed_(0s), filteredExposure_(0s), currentExposure_(0s) { } @@ -99,17 +99,16 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) lineDuration_ = configInfo.sensorInfo.lineLength * 1.0s / configInfo.sensorInfo.pixelRate; - /* \todo replace the exposure in lines storage with time based ones. */ - minExposureLines_ = context.configuration.agc.minShutterSpeed / lineDuration_; - maxExposureLines_ = std::min(context.configuration.agc.maxShutterSpeed / lineDuration_, - kMaxShutterSpeed / lineDuration_); + minShutterSpeed_ = context.configuration.agc.minShutterSpeed; + maxShutterSpeed_ = std::min(context.configuration.agc.maxShutterSpeed, + kMaxShutterSpeed); minAnalogueGain_ = std::max(context.configuration.agc.minAnalogueGain, kMinAnalogueGain); maxAnalogueGain_ = std::min(context.configuration.agc.maxAnalogueGain, kMaxAnalogueGain); /* Configure the default exposure and gain. */ context.frameContext.agc.gain = minAnalogueGain_; - context.frameContext.agc.exposure = minExposureLines_; + context.frameContext.agc.exposure = minShutterSpeed_ / lineDuration_; return 0; } @@ -236,11 +235,9 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * exposure value applied multiplied by the new estimated gain. */ currentExposure_ = effectiveExposureValue * evGain; - utils::Duration minShutterSpeed = minExposureLines_ * lineDuration_; - utils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_; /* Clamp the exposure value to the min and max authorized */ - utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain_; + utils::Duration maxTotalExposure = maxShutterSpeed_ * maxAnalogueGain_; currentExposure_ = std::min(currentExposure_, maxTotalExposure); LOG(IPU3Agc, Debug) << "Target total exposure " << currentExposure_ << ", maximum is " << maxTotalExposure; @@ -250,14 +247,14 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Divide the exposure value as new exposure and gain values */ utils::Duration exposureValue = filteredExposure_; - utils::Duration shutterTime = minShutterSpeed; + utils::Duration shutterTime; /* * Push the shutter time up to the maximum first, and only then * increase the gain. */ shutterTime = std::clamp(exposureValue / minAnalogueGain_, - minShutterSpeed, maxShutterSpeed); + minShutterSpeed_, maxShutterSpeed_); double stepGain = std::clamp(exposureValue / shutterTime, minAnalogueGain_, maxAnalogueGain_); LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 1d085b40..f14d56b7 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -46,8 +46,8 @@ private: double iqMean_; utils::Duration lineDuration_; - uint32_t minExposureLines_; - uint32_t maxExposureLines_; + utils::Duration minShutterSpeed_; + utils::Duration maxShutterSpeed_; double minAnalogueGain_; double maxAnalogueGain_; From patchwork Sat Nov 13 08:49:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14600 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E4697C3251 for ; Sat, 13 Nov 2021 08:50:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 76C496044F; Sat, 13 Nov 2021 09:50:02 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rEJPvtS3"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D353D6036B for ; Sat, 13 Nov 2021 09:49:52 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8D1ADE0C; Sat, 13 Nov 2021 09:49:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793392; bh=cSMjPTzkHWd0gpDn7FUtm4fLB+5H5w+vMhEXeUBk4U4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rEJPvtS3e/ACOFbqfzPqwJTRjf74Oxxu4JvT7YGuYOyot1wltqNHB2U9rSrNylq9Y XOshL6NjsrENF5IXJaoF0dhKczrcjqb5iEKJlSxo9R71yMTwXDLehpEaapNnBD1ZqF AkcA2HbXojhWU0/eRS7/0w7sunP00QNZG082oz68= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:43 +0100 Message-Id: <20211113084947.21892-11-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 10/14] ipa: ipu3: awb: Add support for color temperature X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AWB estimates the color temperature, but it is not used at all. It can be useful for debug purpose at least, but also for lux estimation later, to be able to know the temperature estimated for a given frame. Add a new member to the IPAFrameContext::awb for this purpose, and update the value in AWB. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/awb.cpp | 1 + src/ipa/ipu3/ipa_context.cpp | 3 +++ src/ipa/ipu3/ipa_context.h | 2 ++ 3 files changed, 6 insertions(+) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index a4114659..c7bcb20e 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -385,6 +385,7 @@ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) context.frameContext.awb.gains.blue = asyncResults_.blueGain; context.frameContext.awb.gains.green = asyncResults_.greenGain; context.frameContext.awb.gains.red = asyncResults_.redGain; + context.frameContext.awb.temperatureK = asyncResults_.temperatureK; } constexpr uint16_t Awb::threshold(float value) diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index a7ff957d..99caf9ad 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -117,6 +117,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPAFrameContext::awb.gains.blue * \brief White balance gain for B channel + * + * \var IPAFrameContext::awb.temperatureK + * \brief Estimated color temperature */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index a5a19800..fd97e240 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -45,6 +45,8 @@ struct IPAFrameContext { double green; double blue; } gains; + + double temperatureK; } awb; struct { From patchwork Sat Nov 13 08:49:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14601 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7EDEDBDB1C for ; Sat, 13 Nov 2021 08:50:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2C18D6038C; Sat, 13 Nov 2021 09:50:03 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hFRiMyj7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 14EAE60376 for ; Sat, 13 Nov 2021 09:49:53 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C31F01490; Sat, 13 Nov 2021 09:49:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793392; bh=NgF1q5SlJ0cU/YYa4IU5nmw01PCbqD8M1CPmJhlW/kw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hFRiMyj7ZpOMatEmUr+QID7p11E5d2ocinIvrwADhXHmlAb/zC8h8Lx+hg+nXMtVv 3uDrm61FPpZkXQdk+Pxd/Pd3hsLdH+Ukmy1OlHrMhCKErAWfZuvZsobWTvp9VaEZlb OUr/LHIJewTAr1doW0HdjDU3cfp/D8DeIq1chDxQ= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:44 +0100 Message-Id: <20211113084947.21892-12-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 11/14] ipa: ipu3: Send color temperature in the metadata X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the color temperature is updated per-frame, use the value and set the corresponding controls::ColourTemperature. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 38e86e58..d3195de6 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -631,6 +631,16 @@ void IPAIPU3::parseStatistics(unsigned int frame, (sensorInfo_.pixelRate / 1e6); ctrls.set(controls::FrameDuration, frameDuration); + ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + + /* + * \todo The Metadata provides a path to getting extended data + * out to the application. Further data such as a simplifed Histogram + * might have value to be exposed, however such data may be + * difficult to report in a generically parsable way and we + * likely want to avoid putting platform specific metadata in. + */ + IPU3Action op; op.op = ActionMetadataReady; op.controls = ctrls; From patchwork Sat Nov 13 08:49:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14602 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1BBC1C3252 for ; Sat, 13 Nov 2021 08:50:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BEEF26038A; Sat, 13 Nov 2021 09:50:03 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="cX3UEMWo"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 560F760368 for ; Sat, 13 Nov 2021 09:49:53 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0591414D4; Sat, 13 Nov 2021 09:49:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793393; bh=md1LCPt5ebZQvuvRwPSooLkR8Dk9lhGjoAEseT2yuNs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cX3UEMWoG57jDPo2w/tCyg+0OQDmc6XfSynrpUtofwuE2hf7M1UNOWIi65UciuVK9 /LodPjveHnuRVgiaaKYX/UWkGc2D3QQcZGCSYJc9D81tSnJkkdBX5jvbuQCehF/Hty DVfh4S1dg3S/cq9WBV6BtTcpnCfSqr+p76KiFYuA= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:45 +0100 Message-Id: <20211113084947.21892-13-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 12/14] ipa: ipu3: Cache line duration at configure call X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We use the line duration several times in the IPAIPU3. Instead of recalculating it each time, cache the value as a utils::Duration. ExposureTime and FrameDuration units are in micro-seconds, cast the Duration accordingly. As sensorInfo is no longer used in updateSessionConfiguration remove the reference to it. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index d3195de6..f1597f78 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -148,8 +148,7 @@ private: void updateControls(const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls); - void updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, - const ControlInfoMap &sensorControls); + void updateSessionConfiguration(const ControlInfoMap &sensorControls); void processControls(unsigned int frame, const ControlList &controls); void fillParams(unsigned int frame, ipu3_uapi_params *params); void parseStatistics(unsigned int frame, @@ -174,6 +173,8 @@ private: uint32_t minGain_; uint32_t maxGain_; + utils::Duration lineDuration_; + /* Interface to the Camera Helper */ std::unique_ptr camHelper_; @@ -188,16 +189,12 @@ private: * \brief Compute IPASessionConfiguration using the sensor information and the * sensor V4L2 controls */ -void IPAIPU3::updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, - const ControlInfoMap &sensorControls) +void IPAIPU3::updateSessionConfiguration(const ControlInfoMap &sensorControls) { const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; int32_t minExposure = v4l2Exposure.min().get(); int32_t maxExposure = v4l2Exposure.max().get(); - utils::Duration lineDuration = sensorInfo.lineLength * 1.0s - / sensorInfo.pixelRate; - const ControlInfo &v4l2Gain = sensorControls.find(V4L2_CID_ANALOGUE_GAIN)->second; int32_t minGain = v4l2Gain.min().get(); int32_t maxGain = v4l2Gain.max().get(); @@ -209,8 +206,8 @@ void IPAIPU3::updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, * * \todo take VBLANK into account for maximum shutter speed */ - context_.configuration.agc.minShutterSpeed = minExposure * lineDuration; - context_.configuration.agc.maxShutterSpeed = maxExposure * lineDuration; + context_.configuration.agc.minShutterSpeed = minExposure * lineDuration_; + context_.configuration.agc.maxShutterSpeed = maxExposure * lineDuration_; context_.configuration.agc.minAnalogueGain = camHelper_->gain(minGain); context_.configuration.agc.maxAnalogueGain = camHelper_->gain(maxGain); } @@ -239,11 +236,10 @@ void IPAIPU3::updateControls(const IPACameraSensorInfo &sensorInfo, * exposure min, max and default and convert it from lines to * microseconds. */ - double lineDuration = sensorInfo.lineLength / (sensorInfo.pixelRate / 1e6); const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; - int32_t minExposure = v4l2Exposure.min().get() * lineDuration; - int32_t maxExposure = v4l2Exposure.max().get() * lineDuration; - int32_t defExposure = v4l2Exposure.def().get() * lineDuration; + int32_t minExposure = v4l2Exposure.min().get() * lineDuration_.get(); + int32_t maxExposure = v4l2Exposure.max().get() * lineDuration_.get(); + int32_t defExposure = v4l2Exposure.def().get() * lineDuration_.get(); controls[&controls::ExposureTime] = ControlInfo(minExposure, maxExposure, defExposure); @@ -463,11 +459,13 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, calculateBdsGrid(configInfo.bdsOutputSize); + lineDuration_ = sensorInfo_.lineLength * 1.0s / sensorInfo_.pixelRate; + /* Update the camera controls using the new sensor settings. */ updateControls(sensorInfo_, ctrls_, ipaControls); /* Update the IPASessionConfiguration using the sensor settings. */ - updateSessionConfiguration(sensorInfo_, ctrls_); + updateSessionConfiguration(ctrls_); for (auto const &algo : algorithms_) { int ret = algo->configure(context_, configInfo); @@ -627,8 +625,7 @@ void IPAIPU3::parseStatistics(unsigned int frame, setControls(frame); /* \todo Use VBlank value calculated from each frame exposure. */ - int64_t frameDuration = sensorInfo_.lineLength * (defVBlank_ + sensorInfo_.outputSize.height) / - (sensorInfo_.pixelRate / 1e6); + int64_t frameDuration = (defVBlank_ + sensorInfo_.outputSize.height) * lineDuration_.get(); ctrls.set(controls::FrameDuration, frameDuration); ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); From patchwork Sat Nov 13 08:49:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14603 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B673AC324F for ; Sat, 13 Nov 2021 08:50:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5A56B6036C; Sat, 13 Nov 2021 09:50:04 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rEIr/ugD"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 866AF60394 for ; Sat, 13 Nov 2021 09:49:53 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3D28A55C; Sat, 13 Nov 2021 09:49:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793393; bh=QWekOye9ZV8IwyJ1PCC5qBnXNK+YkUVkIconh9+UD1Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rEIr/ugDOj7KgDSdG+0U1YLyvuXteoMo4LmZj7aISfw75d3Rr+8GtufrzXbB2jAFy wVcnQ+Jv4U0sKl3/bjYGqa80eDXlfJVYqF5U2UUzMu9H+j4+cdfu1Ls3YDnyO3HzxR cAV42wYeUiHm71pFU9LX9BsoXeuNL8BFJ3lBqGgw= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:46 +0100 Message-Id: <20211113084947.21892-14-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 13/14] ipa: ipu3: Move ExposureTime to IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have the exposure time calculated, report it in the controls::ExposureTime and don't use the pipeline handler for it anymore. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 2 ++ src/libcamera/pipeline/ipu3/ipu3.cpp | 16 +--------------- 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index f1597f78..dc6f2ced 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -630,6 +630,8 @@ void IPAIPU3::parseStatistics(unsigned int frame, ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration_.get()); + /* * \todo The Metadata provides a path to getting extended data * out to the application. Further data such as a simplifed Histogram diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index c984ed12..319bb1a3 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -47,7 +47,7 @@ class IPU3CameraData : public Camera::Private { public: IPU3CameraData(PipelineHandler *pipe) - : Camera::Private(pipe), exposureTime_(0), supportsFlips_(false) + : Camera::Private(pipe), supportsFlips_(false) { } @@ -67,7 +67,6 @@ public: Stream vfStream_; Stream rawStream_; - uint32_t exposureTime_; Rectangle cropRegion_; bool supportsFlips_; Transform rotationTransform_; @@ -1045,17 +1044,6 @@ int PipelineHandlerIPU3::updateControls(IPU3CameraData *data) controls[&controls::ScalerCrop] = ControlInfo(minCrop, maxCrop, maxCrop); - /* - * \todo Report the actual exposure time, use the default for the - * moment. - */ - const auto exposureInfo = data->ipaControls_.find(&controls::ExposureTime); - if (exposureInfo == data->ipaControls_.end()) { - LOG(IPU3, Error) << "Exposure control not initialized by the IPA"; - return -EINVAL; - } - data->exposureTime_ = exposureInfo->second.def().get(); - /* Add the IPA registered controls to list of camera controls. */ for (const auto &ipaControl : data->ipaControls_) controls[ipaControl.first] = ipaControl.second; @@ -1317,8 +1305,6 @@ void IPU3CameraData::imguOutputBufferReady(FrameBuffer *buffer) pipe()->completeBuffer(request, buffer); request->metadata().set(controls::draft::PipelineDepth, 3); - /* \todo Move the ExposureTime control to the IPA. */ - request->metadata().set(controls::ExposureTime, exposureTime_); /* \todo Actually apply the scaler crop region to the ImgU. */ if (request->controls().contains(controls::ScalerCrop)) cropRegion_ = request->controls().get(controls::ScalerCrop); From patchwork Sat Nov 13 08:49:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14604 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 767FFC3253 for ; Sat, 13 Nov 2021 08:50:05 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0D4AB60378; Sat, 13 Nov 2021 09:50:05 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="WxVILABH"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C180D6038C for ; Sat, 13 Nov 2021 09:49:53 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:4d35:9445:6881:4ffd]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 73509A59; Sat, 13 Nov 2021 09:49:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636793393; bh=S4lMfGGUWczpZZwfgV05GaDbQDnhbkbE1xqBWhI79x0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WxVILABH7K+1bnkIio/U1hT23aDJpy4n1foywoXAunHItOIWGCKtZDxghOikUUBEA kvPec1MHO236aQRPhh7aLIeHKQBXQTIrokBPtjV/b6xryR+NjTegWioxUq3Kv+Q9ix zJAfjQkKXHwm5BVeufg2VqWIdHmawNdYoVuMdx6s= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Sat, 13 Nov 2021 09:49:47 +0100 Message-Id: <20211113084947.21892-15-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> References: <20211113084947.21892-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 14/14] ipa: ipu3: Set the AnalogueGain control X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We can report the controls::AnalogueGain metadata now that AGC is updating it correctly. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index dc6f2ced..a8d54a5d 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -628,6 +628,8 @@ void IPAIPU3::parseStatistics(unsigned int frame, int64_t frameDuration = (defVBlank_ + sensorInfo_.outputSize.height) * lineDuration_.get(); ctrls.set(controls::FrameDuration, frameDuration); + ctrls.set(controls::AnalogueGain, context_.frameContext.agc.gain); + ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration_.get());