From patchwork Thu Nov 11 14:09:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14571 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CBA16BDB1C for ; Thu, 11 Nov 2021 14:09:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D213B6038A; Thu, 11 Nov 2021 15:09:34 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="aSgK+HCf"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6E10260345 for ; Thu, 11 Nov 2021 15:09:31 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 03FC7E51; Thu, 11 Nov 2021 15:09:30 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639771; bh=N81w8/G3lWtPy4LdH5xSsM77aOlGhlOyxo0vtEPgaEo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aSgK+HCfrw0huWKUPpP5V5KOjlFFEC8/OE7KzyCVatwPg0w0SBJmM/2lW4fUgKzYr 9ttXtdUjMlTXjAlq9cvHxx88f6jC9lhma9sPXM8lEO3OKiqwqipARAmCVs77Ivw/a/ MrQMjsvvTpbBH5/N84FuC8VXlE6s/6xE0sytRzkQ= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:15 +0100 Message-Id: <20211111140928.136111-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 01/14] ipa: ipu3: Extend ipu3 ipa interface for sensor controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen IPU3Event and IPU3Action use single ControlList for both libcamera and V4L2 controls, and it's content could be either one based on the context. Extend IPU3Event and IPU3Action for sensor V4L2 controls, and preserve the original one for only libcamera Controls to make the content of an event more specific. Signed-off-by: Han-Lin Chen [Jean-Michel: remove lensControls from the original patch] Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- include/libcamera/ipa/ipu3.mojom | 2 ++ src/ipa/ipu3/ipu3.cpp | 2 +- src/libcamera/pipeline/ipu3/ipu3.cpp | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/libcamera/ipa/ipu3.mojom b/include/libcamera/ipa/ipu3.mojom index 2f254ed4..16e3462e 100644 --- a/include/libcamera/ipa/ipu3.mojom +++ b/include/libcamera/ipa/ipu3.mojom @@ -23,11 +23,13 @@ struct IPU3Event { int64 frameTimestamp; uint32 bufferId; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; struct IPU3Action { IPU3Operations op; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; struct IPAConfigInfo { diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 93b700bd..bcc3863b 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -653,7 +653,7 @@ void IPAIPU3::setControls(unsigned int frame) ControlList ctrls(ctrls_); ctrls.set(V4L2_CID_EXPOSURE, static_cast(exposure_)); ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(gain_)); - op.controls = ctrls; + op.sensorControls = ctrls; queueFrameAction.emit(frame, op); } diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index eb714aa6..8816efc5 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -1248,7 +1248,7 @@ void IPU3CameraData::queueFrameAction(unsigned int id, { switch (action.op) { case ipa::ipu3::ActionSetSensorControls: { - const ControlList &controls = action.controls; + const ControlList &controls = action.sensorControls; delayedCtrls_->push(controls); break; } From patchwork Thu Nov 11 14:09:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14572 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A40CBBDB1C for ; Thu, 11 Nov 2021 14:09:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 53035603BE; Thu, 11 Nov 2021 15:09:35 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="RaWAoleY"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 958416034D for ; Thu, 11 Nov 2021 15:09:31 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3B5711253; Thu, 11 Nov 2021 15:09:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639771; bh=Mu1TU/yh1YDWahDJAs7Sa7vaoW7Kmpa57HarVGJRSUw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RaWAoleYpJIQGsBj56hJk6B24Xhv4yYeAPP+Sz1rkZTuFYD38hbB0WRSQL5ROCC8q X2qJnUK2TzmFS62dcQoMbaDqznoU1NTr+nP35pJkEsRCna0GB6kXbTckRYTmfb3vuS 923oVaFTcI5p4XxLkCq3yMg2G+N/dFWfJEjwJ1Os= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:16 +0100 Message-Id: <20211111140928.136111-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 02/14] ipu3: ipa: Report effective sensor controls with statistics to IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen The Intel close sourced IPA requires the effective controls applied to the sensor when the statistics are generated. Report effective sensor controls with the statistics to IPA. Signed-off-by: Han-Lin Chen [Jean-Michel: Reword s/stastistics/statistics and move reset after IPA start] Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- src/libcamera/pipeline/ipu3/frames.h | 4 ++++ src/libcamera/pipeline/ipu3/ipu3.cpp | 5 +++++ 2 files changed, 9 insertions(+) diff --git a/src/libcamera/pipeline/ipu3/frames.h b/src/libcamera/pipeline/ipu3/frames.h index 3ef7e445..614701e2 100644 --- a/src/libcamera/pipeline/ipu3/frames.h +++ b/src/libcamera/pipeline/ipu3/frames.h @@ -14,6 +14,8 @@ #include +#include + namespace libcamera { class FrameBuffer; @@ -34,6 +36,8 @@ public: FrameBuffer *paramBuffer; FrameBuffer *statBuffer; + ControlList effectiveSensorControls; + bool paramDequeued; bool metadataProcessed; }; diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 8816efc5..c984ed12 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -767,6 +767,8 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis if (ret) goto error; + data->delayedCtrls_->reset(); + /* * Start the ImgU video devices, buffers will be queued to the * ImgU output and viewfinder when requests will be queued. @@ -1363,6 +1365,8 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer) request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + info->effectiveSensorControls = delayedCtrls_->get(buffer->metadata().sequence); + if (request->findBuffer(&rawStream_)) pipe()->completeBuffer(request, buffer); @@ -1419,6 +1423,7 @@ void IPU3CameraData::statBufferReady(FrameBuffer *buffer) ev.frame = info->id; ev.bufferId = info->statBuffer->cookie(); ev.frameTimestamp = request->metadata().get(controls::SensorTimestamp); + ev.sensorControls = info->effectiveSensorControls; ipa_->processEvent(ev); } From patchwork Thu Nov 11 14:09:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14573 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5C5E9BDB1C for ; Thu, 11 Nov 2021 14:09:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1CBBE6039C; Thu, 11 Nov 2021 15:09:38 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="iVf3J0n6"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B57956035A for ; Thu, 11 Nov 2021 15:09:31 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6CD7EDEE; Thu, 11 Nov 2021 15:09:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639771; bh=ocXOEWT9pg8hQVzQCDtfylDmjAYemhxFouDIRAhEhis=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iVf3J0n6MWU0TFPZvH0qDnYJxirvKofLJyoRsIOfWRmMEFFX0XuFqItmUrDpnabaL hG3oROqkIGU0RdLhSbH6/hzG6WxZ7gct4CI4ZQsKldkgw4aQ92Kyu1ai7X+KJIeU4Y Nug+FV1iivrMPsVOdYM4bwpnwbs5i8e4Lp55J0k0= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:17 +0100 Message-Id: <20211111140928.136111-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 03/14] ipa: ipu3: Use sensor controls to update frameContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The pipeline handler populates the new sensorControls ControlList, to have the effective exposure and gain values for the current frame. This is done when a statistics buffer is received. Make those values the frameContext::sensor values for the frame when the EventStatReady event is received. AGC also needs to use frameContext.sensor as its input values and frameContext.agc as its output values. Modify computeExposure by passing it the frameContext instead of individual exposure and gain values. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 19 ++++++++++--------- src/ipa/ipu3/algorithms/agc.h | 2 +- src/ipa/ipu3/ipa_context.cpp | 11 +++++++++++ src/ipa/ipu3/ipa_context.h | 5 +++++ src/ipa/ipu3/ipu3.cpp | 3 +++ 5 files changed, 30 insertions(+), 10 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index b5d736c1..5723f6f3 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -169,10 +169,9 @@ void Agc::filterExposure() /** * \brief Estimate the new exposure and gain values - * \param[inout] exposure The exposure value reference as a number of lines - * \param[inout] gain The gain reference to be updated + * \param[inout] frameContext The shared IPA frame Context */ -void Agc::computeExposure(uint32_t &exposure, double &analogueGain) +void Agc::computeExposure(IPAFrameContext &frameContext) { /* Algorithm initialization should wait for first valid frames */ /* \todo - have a number of frames given by DelayedControls ? @@ -189,6 +188,10 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) return; } + /* Get the effective exposure and gain applied on the sensor. */ + uint32_t exposure = frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + /* Estimate the gain needed to have the proportion wanted */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; @@ -233,8 +236,9 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) << shutterTime << " and " << stepGain; - exposure = shutterTime / lineDuration_; - analogueGain = stepGain; + /* Update the estimated exposure and gain. */ + frameContext.agc.exposure = shutterTime / lineDuration_; + frameContext.agc.gain = stepGain; /* * Update the exposure value for the next process call. @@ -257,11 +261,8 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) */ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { - /* Get the latest exposure and gain applied */ - uint32_t &exposure = context.frameContext.agc.exposure; - double &analogueGain = context.frameContext.agc.gain; measureBrightness(stats, context.configuration.grid.bdsGrid); - computeExposure(exposure, analogueGain); + computeExposure(context.frameContext); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 69e0b831..f0db25ee 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -34,7 +34,7 @@ private: void measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); void filterExposure(); - void computeExposure(uint32_t &exposure, double &gain); + void computeExposure(IPAFrameContext &frameContext); uint64_t frameCount_; uint64_t lastFrame_; diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 2355a9c7..a7ff957d 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -119,6 +119,17 @@ namespace libcamera::ipa::ipu3 { * \brief White balance gain for B channel */ +/** + * \var IPAFrameContext::sensor + * \brief Effective sensor values + * + * \var IPAFrameContext::sensor.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::sensor.gain + * \brief Analogue gain multiplier + */ + /** * \var IPAFrameContext::toneMapping * \brief Context for ToneMapping and Gamma control diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 1e46c61a..a5a19800 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -47,6 +47,11 @@ struct IPAFrameContext { } gains; } awb; + struct { + uint32_t exposure; + double gain; + } sensor; + struct { double gamma; struct ipu3_uapi_gamma_corr_lut gammaCorrection; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index bcc3863b..38e86e58 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -549,6 +549,9 @@ void IPAIPU3::processEvent(const IPU3Event &event) const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); + context_.frameContext.sensor.exposure = event.sensorControls.get(V4L2_CID_EXPOSURE).get(); + context_.frameContext.sensor.gain = camHelper_->gain(event.sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + parseStatistics(event.frame, event.frameTimestamp, stats); break; } From patchwork Thu Nov 11 14:09:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14574 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5DF8AC324E for ; Thu, 11 Nov 2021 14:09:40 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0B16860369; Thu, 11 Nov 2021 15:09:39 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="CJrO0Uiq"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E320A6035D for ; Thu, 11 Nov 2021 15:09:31 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9EE491516; Thu, 11 Nov 2021 15:09:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639771; bh=UEu+Tja8cZ/7BpqPpndVk3JHoHnEsVebkaJgeLPCkHQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CJrO0Uiq9PhWG/tFkMUXK00WsKakPVZ9zXQ8sYfiaJSDISXCeK3a5KohP52PkOxes zALZ2ju6JVSdX9JfCXDhPSPkVsz5plUwyywdJBwo6DOx0XxtmT518YYGgT6nH4gqOh Y8JcC29dV/aLYb7Cw7v4438bpSoBbOzRJGk/M0a4= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:18 +0100 Message-Id: <20211111140928.136111-5-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 04/14] ipa: ipu3: agc: Limit the number of saturated cells X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the histogram is calculated, we check if a cell is saturated or not before cumulating its green value. This is wrong, and it can lead to an empty histogram in case of a fully saturated frame. Use a constant to limit the amount of pixels within a cell before considering it saturated. If at the end of the loop we still have an empty histogram, then make it a fully saturated one. Bug: https://bugs.libcamera.org/show_bug.cgi?id=84 Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 5723f6f3..83aa3676 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -63,6 +63,12 @@ static constexpr uint32_t knumHistogramBins = 256; /* Target value to reach for the top 2% of the histogram */ static constexpr double kEvGainTarget = 0.5; +/* + * Maximum ratio of saturated pixels in a cell for the cell to be considered + * non-saturated and counted by the AGC algorithm. + */ +static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; + Agc::Agc() : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), @@ -124,7 +130,7 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, &stats->awb_raw_buffer.meta_data[cellPosition] ); - if (cell->sat_ratio == 0) { + if (cell->sat_ratio <= kMinCellsPerZoneRatio) { uint8_t gr = cell->Gr_avg; uint8_t gb = cell->Gb_avg; /* @@ -137,8 +143,14 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, } } + Histogram cumulativeHist = Histogram(Span(hist)); /* Estimate the quantile mean of the top 2% of the histogram */ - iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); + if (cumulativeHist.total() == 0) { + /* Force the value as histogram is empty */ + iqMean_ = knumHistogramBins - 0.5; + } else { + iqMean_ = cumulativeHist.interQuantileMean(0.98, 1.0); + } } /** From patchwork Thu Nov 11 14:09:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14575 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 167E5BDB1C for ; Thu, 11 Nov 2021 14:09:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 88C0E603D9; Thu, 11 Nov 2021 15:09:40 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="XUVHVvXM"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2CD5C6035F for ; Thu, 11 Nov 2021 15:09:32 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D1ED1E51; Thu, 11 Nov 2021 15:09:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639771; bh=uNzmGF8HAfJr6FPm9Ld+AdRJ5aqu0oMeosheuod0h5Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XUVHVvXMbSNbHdw7EsPu7Z38AxKeJUvtD987eahvyYkyLOpYMrn/57P8s94SZaMCK AV/uFe5znSjtjHYo+oCCo4p2pzxU6rlHftnw5qoHeOA5zcBkAriXY+kjLKn4RKD314 W6kz0hwzJcfyM2VL3oGpvyrmzwzd4PTgrxxv6Qxw= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:19 +0100 Message-Id: <20211111140928.136111-6-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 05/14] ipa: ipu3: agc: Compute the gain for each frame X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have the real exposure applied at each frame, remove the early return based on a frame counter and compute the gain for each frame. Instead of that, introduce a number of startup frames during which the filter speed is 1.0, meaning we apply instantly the exposure value calculated and not a slower filtered one. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 83aa3676..74bce7bb 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -45,11 +45,6 @@ namespace ipa::ipu3::algorithms { LOG_DEFINE_CATEGORY(IPU3Agc) -/* Number of frames to wait before calculating stats on minimum exposure */ -static constexpr uint32_t kInitialFrameMinAECount = 4; -/* Number of frames to wait between new gain/shutter time estimations */ -static constexpr uint32_t kFrameSkipCount = 6; - /* Limits for analogue gain values */ static constexpr double kMinAnalogueGain = 1.0; static constexpr double kMaxAnalogueGain = 8.0; @@ -69,10 +64,13 @@ static constexpr double kEvGainTarget = 0.5; */ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; +/* Number of frames to wait before calculating stats on minimum exposure */ +static constexpr uint32_t kNumStartupFrames = 10; + Agc::Agc() - : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), - minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), - currentExposure_(0s), prevExposureValue_(0s) + : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), + maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), + prevExposureValue_(0s) { } @@ -159,6 +157,11 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, void Agc::filterExposure() { double speed = 0.2; + + /* Adapt instantly if we are in startup phase */ + if (frameCount_ < kNumStartupFrames) + speed = 1.0; + if (filteredExposure_ == 0s) { /* DG stands for digital gain.*/ filteredExposure_ = currentExposure_; @@ -185,13 +188,6 @@ void Agc::filterExposure() */ void Agc::computeExposure(IPAFrameContext &frameContext) { - /* Algorithm initialization should wait for first valid frames */ - /* \todo - have a number of frames given by DelayedControls ? - * - implement a function for IIR */ - if ((frameCount_ < kInitialFrameMinAECount) || (frameCount_ - lastFrame_ < kFrameSkipCount)) - return; - - lastFrame_ = frameCount_; /* Are we correctly exposed ? */ if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { From patchwork Thu Nov 11 14:09:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14576 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A21E2C324F for ; Thu, 11 Nov 2021 14:09:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F2F11603BE; Thu, 11 Nov 2021 15:09:40 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="FxzRQzCe"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5AB4760369 for ; Thu, 11 Nov 2021 15:09:32 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 138911253; Thu, 11 Nov 2021 15:09:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639772; bh=F8JFsx9svwRYYGLy8xgyz2n9HlIqYVqb3L6FVnk+ruk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FxzRQzCetmV8l/59NgO9xjzaj9MOstpfAfUQp8y1HlIvB5HID0lV21k4+E1VNU8E5 1cgkopCGpofS+efzMv2rsj3fGi/LUyWiSwXJMHUZDCV8X4iXBi2RPWIM69aZZrCkMN nUhh5OO1vKbY9bBnmhl3iOaTD6L5UEoffikaZIa8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:20 +0100 Message-Id: <20211111140928.136111-7-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 06/14] ipa: ipu3: agc: Refactor ev gain calculation and testing X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When we compute the new gain, we use the iqMean_ and estimate an exposure value gain to apply. Return early when the gain is less than 1%. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 74bce7bb..61ca8b3f 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -188,14 +188,6 @@ void Agc::filterExposure() */ void Agc::computeExposure(IPAFrameContext &frameContext) { - - /* Are we correctly exposed ? */ - if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { - LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " - << iqMean_ << ")"; - return; - } - /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; double analogueGain = frameContext.sensor.gain; @@ -203,6 +195,12 @@ void Agc::computeExposure(IPAFrameContext &frameContext) /* Estimate the gain needed to have the proportion wanted */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; + if (std::abs(evGain - 1.0) < 0.01) { + LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " + << iqMean_ << ")"; + return; + } + /* extracted from Rpi::Agc::computeTargetExposure */ /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; From patchwork Thu Nov 11 14:09:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14578 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A5345C3250 for ; Thu, 11 Nov 2021 14:09:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0F7ED60441; Thu, 11 Nov 2021 15:09:42 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="BYm6HyZP"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8D6CC6036C for ; Thu, 11 Nov 2021 15:09:32 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 463C61543; Thu, 11 Nov 2021 15:09:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639772; bh=BeJz93F6GMcTUgsLx2/ns6lOLKAugtTQPDs0TL8H4uc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BYm6HyZPjf0JqgyAyG4KV1Dw8ZUE3NsRj/X5MM6xPyJubXY62/uVZrDuRPfjQYTmA 0PyAWnYI/Wk0AhJIuXunEuFwLshfU0N3PJ/S+tFQRY8O6Fj6i14K3l8+wKzBFyw18G 0qGz6L+6KIGSOo1fdDhh7XSHKL+l9a4W0NRXMG4U= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:21 +0100 Message-Id: <20211111140928.136111-8-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 07/14] ipa: ipu3: agc: Improve gain calculation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When an image is partially saturated, its brightness is not increasing linearly when the shutter time or gain increases. It is a big issue with a backlight as the algorithm is fading to darkness right now. Introduce a function to estimate the brightness of the frame, based on the current exposure/gain and loop on it several times to estimate it again and approach the non linear function. Inspired-by: 7de5506c30b3 ("libcamera: src: ipa: raspberrypi: agc: Improve gain update calculation for partly saturated images") Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 104 +++++++++++++++++++++++++++++++- src/ipa/ipu3/algorithms/agc.h | 6 +- 2 files changed, 106 insertions(+), 4 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 61ca8b3f..a34f070e 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -67,6 +67,17 @@ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; /* Number of frames to wait before calculating stats on minimum exposure */ static constexpr uint32_t kNumStartupFrames = 10; +/* Maximum luminance used for brightness normalization */ +static constexpr uint32_t kMaxLuminance = 255; + +/* + * Normalized luma value target. + * + * It's a number that's chosen so that, when the camera points at a grey + * target, the resulting image brightness is considered right. + */ +static constexpr double kNormalizedLumaTarget = 0.16; + Agc::Agc() : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), @@ -185,14 +196,20 @@ void Agc::filterExposure() /** * \brief Estimate the new exposure and gain values * \param[inout] frameContext The shared IPA frame Context + * \param[in] currentYGain The gain calculated on the current brightness level */ -void Agc::computeExposure(IPAFrameContext &frameContext) +void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) { /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; double analogueGain = frameContext.sensor.gain; - /* Estimate the gain needed to have the proportion wanted */ + /* + * Estimate the gain needed to have the proportion of pixels in a given + * range wanted. iqMean_ returns the mean value of the top 2% of the + * cumulative histogram, and we want it to be as close as possible to a + * configured target. + */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; if (std::abs(evGain - 1.0) < 0.01) { @@ -202,6 +219,7 @@ void Agc::computeExposure(IPAFrameContext &frameContext) } /* extracted from Rpi::Agc::computeTargetExposure */ + /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain @@ -209,6 +227,14 @@ void Agc::computeExposure(IPAFrameContext &frameContext) << " Gain " << analogueGain << " Needed ev gain " << evGain; + if (evGain < currentYGain) + evGain = currentYGain; + + /* Consider within 1% of the target as correctly exposed */ + if (std::abs(evGain - 1.0) < 0.01) + LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " + << iqMean_ << ")"; + /* * Calculate the current exposure value for the scene as the latest * exposure value applied multiplied by the new estimated gain. @@ -257,6 +283,56 @@ void Agc::computeExposure(IPAFrameContext &frameContext) prevExposureValue_ = shutterTime * analogueGain; } +/** + * \brief Estimate the average brightness of the frame + * \param[in] frameContext The shared IPA frame context + * \param[in] grid The grid used to store the statistics in the IPU3 + * \param[in] stats The IPU3 statistics and ISP results + * \param[in] currentYGain The gain calculated on the current brightness level + * \return The normalized luma + * + * Luma is the weighted sum of gamma-compressed R′G′B′ components of a color + * video. The luma values are normalized as 0.0 to 1.0, with 1.0 being a + * theoretical perfect reflector of 100% reference white. We use the Rec. 601 + * luma here. + * + * More detailed information can be found in: + * https://en.wikipedia.org/wiki/Luma_(video) + */ +double Agc::computeInitialY(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain) +{ + double redSum = 0, greenSum = 0, blueSum = 0; + + for (unsigned int cellY = 0; cellY < grid.height; cellY++) { + for (unsigned int cellX = 0; cellX < grid.width; cellX++) { + uint32_t cellPosition = cellY * stride_ + cellX; + + const ipu3_uapi_awb_set_item *cell = + reinterpret_cast( + &stats->awb_raw_buffer.meta_data[cellPosition] + ); + + redSum += cell->R_avg * currentYGain; + greenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * currentYGain; + blueSum += cell->B_avg * currentYGain; + } + } + + /* + * Estimate the sum of the brightness values, weighted with the gains + * applied on the channels in AWB as the Rec. 601 luma. + */ + double Y_sum = redSum * frameContext.awb.gains.red * .299 + + greenSum * frameContext.awb.gains.green * .587 + + blueSum * frameContext.awb.gains.blue * .114; + + /* Return the normalized relative luminance. */ + return Y_sum / (grid.height * grid.width) / kMaxLuminance; +} + /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -268,7 +344,29 @@ void Agc::computeExposure(IPAFrameContext &frameContext) void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { measureBrightness(stats, context.configuration.grid.bdsGrid); - computeExposure(context.frameContext); + + double currentYGain = 1.0; + double targetY = kNormalizedLumaTarget; + + /* + * Do this calculation a few times as brightness increase can be + * non-linear when there are saturated regions. + */ + for (int i = 0; i < 8; i++) { + double initialY = computeInitialY(context.frameContext, + context.configuration.grid.bdsGrid, + stats, currentYGain); + double extra_gain = std::min(10.0, targetY / (initialY + .001)); + + currentYGain *= extra_gain; + LOG(IPU3Agc, Debug) << "Initial Y " << initialY + << " target " << targetY + << " gives gain " << currentYGain; + if (extra_gain < 1.01) + break; + } + + computeExposure(context.frameContext, currentYGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index f0db25ee..79736283 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -34,7 +34,11 @@ private: void measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); void filterExposure(); - void computeExposure(IPAFrameContext &frameContext); + void computeExposure(IPAFrameContext &frameContext, double currentYGain); + double computeInitialY(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain); uint64_t frameCount_; uint64_t lastFrame_; From patchwork Thu Nov 11 14:09:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14577 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 21388C324E for ; Thu, 11 Nov 2021 14:09:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8ED9760423; Thu, 11 Nov 2021 15:09:41 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="WsF34VAD"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B620360371 for ; Thu, 11 Nov 2021 15:09:32 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 79032DEE; Thu, 11 Nov 2021 15:09:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639772; bh=xg9dwKCABj8ypq5L89gja33/nZOvu25bnOUjVpDgLFg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WsF34VADs44xO42260JsSw2OskVLD0jXZ/DKJAcHONyO3gs9N87yrYWUwW4A3JBZf lPRopfrzYe+19CZBqWdFf0wcJlb3agN3Ns9AYouvvlbfpEiK1LKdYDYHsqcg/uD9XN BOX/gXRvoFbhSd/IUyGlAD3E5dEAV2eVnKattz4U= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:22 +0100 Message-Id: <20211111140928.136111-9-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 08/14] ipa: ipu3: agc: Update previous exposure value X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Previously, the exposure value was calculated based on the estimated shutter time and gain applied. Now that we have the real values for the current frame, use those before estimating the next one and rename the variable accordingly. As the exposure value is updated in the beginning of the computation, there is no need to initialize effectiveExposureValue anymore in the configure call, and it can be a local variable and not a class variable anymore. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- v3: make effectiveExposureValue a local variable --- src/ipa/ipu3/algorithms/agc.cpp | 26 +++++++++----------------- src/ipa/ipu3/algorithms/agc.h | 1 - 2 files changed, 9 insertions(+), 18 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index a34f070e..d736f21c 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -80,8 +80,7 @@ static constexpr double kNormalizedLumaTarget = 0.16; Agc::Agc() : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), - maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), - prevExposureValue_(0s) + maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s) { } @@ -112,10 +111,6 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) context.frameContext.agc.gain = minAnalogueGain_; context.frameContext.agc.exposure = minExposureLines_; - prevExposureValue_ = context.frameContext.agc.gain - * context.frameContext.agc.exposure - * lineDuration_; - return 0; } @@ -222,6 +217,13 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; + + /* + * Update the exposure value for the next computation using the values + * of exposure and gain really used by the sensor. + */ + utils::Duration effectiveExposureValue = currentShutter * analogueGain; + LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain << " Shutter speed " << currentShutter << " Gain " << analogueGain @@ -239,7 +241,7 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * Calculate the current exposure value for the scene as the latest * exposure value applied multiplied by the new estimated gain. */ - currentExposure_ = prevExposureValue_ * evGain; + currentExposure_ = effectiveExposureValue * evGain; utils::Duration minShutterSpeed = minExposureLines_ * lineDuration_; utils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_; @@ -271,16 +273,6 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Update the estimated exposure and gain. */ frameContext.agc.exposure = shutterTime / lineDuration_; frameContext.agc.gain = stepGain; - - /* - * Update the exposure value for the next process call. - * - * \todo Obtain the values of the exposure time and analog gain - * that were actually used by the sensor, either from embedded - * data when available, or from the delayed controls - * infrastructure in case a slow down caused a mismatch. - */ - prevExposureValue_ = shutterTime * analogueGain; } /** diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 79736283..1d085b40 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -54,7 +54,6 @@ private: utils::Duration filteredExposure_; utils::Duration currentExposure_; - utils::Duration prevExposureValue_; uint32_t stride_; }; From patchwork Thu Nov 11 14:09:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14579 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B0F11BDB1C for ; Thu, 11 Nov 2021 14:09:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5E9BD60378; Thu, 11 Nov 2021 15:09:43 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="UkTi4oem"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F39A860361 for ; Thu, 11 Nov 2021 15:09:32 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A74411516; Thu, 11 Nov 2021 15:09:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639772; bh=yWkJdvRSgfXh4qjq9NioC2/JvKEeiIrDVDhZ80AwLCg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UkTi4oem9Taopc8U7zypgt0SXlz+mv79Bl//W70aeIpYCKmHfGKH/qyzc9OUJcc8v Lex3uhWAPg1Pqfrf7hEtrvhtubGvBhK4DoNyLOR78EN59IWq6BOAOQBRN0Ud62nhgO RqekHgI2fotnPDL9Cm5h3DWRXroOE/1V5EQMWVWA= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:23 +0100 Message-Id: <20211111140928.136111-10-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 09/14] ipa: ipu3: agc: Use exposure in time for storage X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The minimum and maximum exposure are stored in lines. Replace it by values in time to simplify the calculations. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 21 +++++++++------------ src/ipa/ipu3/algorithms/agc.h | 4 ++-- 2 files changed, 11 insertions(+), 14 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index d736f21c..86b71886 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -79,8 +79,8 @@ static constexpr uint32_t kMaxLuminance = 255; static constexpr double kNormalizedLumaTarget = 0.16; Agc::Agc() - : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), - maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s) + : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minShutterSpeed_(0s), + maxShutterSpeed_(0s), filteredExposure_(0s), currentExposure_(0s) { } @@ -99,17 +99,16 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) lineDuration_ = configInfo.sensorInfo.lineLength * 1.0s / configInfo.sensorInfo.pixelRate; - /* \todo replace the exposure in lines storage with time based ones. */ - minExposureLines_ = context.configuration.agc.minShutterSpeed / lineDuration_; - maxExposureLines_ = std::min(context.configuration.agc.maxShutterSpeed / lineDuration_, - kMaxShutterSpeed / lineDuration_); + minShutterSpeed_ = context.configuration.agc.minShutterSpeed; + maxShutterSpeed_ = std::min(context.configuration.agc.maxShutterSpeed, + kMaxShutterSpeed); minAnalogueGain_ = std::max(context.configuration.agc.minAnalogueGain, kMinAnalogueGain); maxAnalogueGain_ = std::min(context.configuration.agc.maxAnalogueGain, kMaxAnalogueGain); /* Configure the default exposure and gain. */ context.frameContext.agc.gain = minAnalogueGain_; - context.frameContext.agc.exposure = minExposureLines_; + context.frameContext.agc.exposure = minShutterSpeed_ / lineDuration_; return 0; } @@ -242,11 +241,9 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * exposure value applied multiplied by the new estimated gain. */ currentExposure_ = effectiveExposureValue * evGain; - utils::Duration minShutterSpeed = minExposureLines_ * lineDuration_; - utils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_; /* Clamp the exposure value to the min and max authorized */ - utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain_; + utils::Duration maxTotalExposure = maxShutterSpeed_ * maxAnalogueGain_; currentExposure_ = std::min(currentExposure_, maxTotalExposure); LOG(IPU3Agc, Debug) << "Target total exposure " << currentExposure_ << ", maximum is " << maxTotalExposure; @@ -256,14 +253,14 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Divide the exposure value as new exposure and gain values */ utils::Duration exposureValue = filteredExposure_; - utils::Duration shutterTime = minShutterSpeed; + utils::Duration shutterTime; /* * Push the shutter time up to the maximum first, and only then * increase the gain. */ shutterTime = std::clamp(exposureValue / minAnalogueGain_, - minShutterSpeed, maxShutterSpeed); + minShutterSpeed_, maxShutterSpeed_); double stepGain = std::clamp(exposureValue / shutterTime, minAnalogueGain_, maxAnalogueGain_); LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 1d085b40..f14d56b7 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -46,8 +46,8 @@ private: double iqMean_; utils::Duration lineDuration_; - uint32_t minExposureLines_; - uint32_t maxExposureLines_; + utils::Duration minShutterSpeed_; + utils::Duration maxShutterSpeed_; double minAnalogueGain_; double maxAnalogueGain_; From patchwork Thu Nov 11 14:09:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14580 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6B8CAC324F for ; Thu, 11 Nov 2021 14:09:44 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 12252603C2; Thu, 11 Nov 2021 15:09:44 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="WZDbrs1c"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2CDA760345 for ; Thu, 11 Nov 2021 15:09:33 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D8EE4180D; Thu, 11 Nov 2021 15:09:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639772; bh=4tNa6FTLtligxhLvhqy7ErCQZN9FVjZQl++ok5GJQiU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WZDbrs1cq4cIl+PtS99WDqlKUUcaH7+szCUhYYEzQag4LW0I8OmmRDMeErSS0c8Pu /8f9+zYRzzDxQBg5CpzA1+wpUbCWz68Gg3JqM2+0c1u4dp/lo9h7qButmp9Q3VORbo 8X/bi06Evh3uJXLKrKzarfRaMu9Foe42hZvGvtG0= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:24 +0100 Message-Id: <20211111140928.136111-11-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 10/14] ipa: ipu3: awb: Add support for color temperature X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AWB estimates the color temperature, but it is not used at all. It can be useful for debug purpose at least, but also for lux estimation later, to be able to know the temperature estimated for a given frame. Add a new member to the IPAFrameContext::awb for this purpose, and update the value in AWB. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/awb.cpp | 1 + src/ipa/ipu3/ipa_context.cpp | 3 +++ src/ipa/ipu3/ipa_context.h | 2 ++ 3 files changed, 6 insertions(+) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index a4114659..c7bcb20e 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -385,6 +385,7 @@ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) context.frameContext.awb.gains.blue = asyncResults_.blueGain; context.frameContext.awb.gains.green = asyncResults_.greenGain; context.frameContext.awb.gains.red = asyncResults_.redGain; + context.frameContext.awb.temperatureK = asyncResults_.temperatureK; } constexpr uint16_t Awb::threshold(float value) diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index a7ff957d..99caf9ad 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -117,6 +117,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPAFrameContext::awb.gains.blue * \brief White balance gain for B channel + * + * \var IPAFrameContext::awb.temperatureK + * \brief Estimated color temperature */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index a5a19800..fd97e240 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -45,6 +45,8 @@ struct IPAFrameContext { double green; double blue; } gains; + + double temperatureK; } awb; struct { From patchwork Thu Nov 11 14:09:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14581 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0E46CC324E for ; Thu, 11 Nov 2021 14:09:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AA1D060362; Thu, 11 Nov 2021 15:09:46 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rS5WJYG2"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 68569600B5 for ; Thu, 11 Nov 2021 15:09:33 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 15F00E51; Thu, 11 Nov 2021 15:09:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639773; bh=JMQm9uZfy7p4vYGC16HYi0ldVVfHp8KTz889bsLCWqk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rS5WJYG2JNs2mgmG4T1kSZoLzb+d6Yin1UB15mFyP+c16ATsAO5g5NYAs15FqmUp9 9oHaNE5HNwomfsbEAOKo9KKZ1nRuSEfvWc7pBPQzkRxbLTfabP33XzrWGIwJBosF9H jK9wNpbXtLpXbIKDJG21kxSoaae47LS6Y1ZCFFNI= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:25 +0100 Message-Id: <20211111140928.136111-12-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 11/14] ipa: ipu3: Send color temperature in the metadata X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the color temperature is updated per-frame, use the value and set the corresponding controls::ColourTemperature. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 38e86e58..d3195de6 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -631,6 +631,16 @@ void IPAIPU3::parseStatistics(unsigned int frame, (sensorInfo_.pixelRate / 1e6); ctrls.set(controls::FrameDuration, frameDuration); + ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + + /* + * \todo The Metadata provides a path to getting extended data + * out to the application. Further data such as a simplifed Histogram + * might have value to be exposed, however such data may be + * difficult to report in a generically parsable way and we + * likely want to avoid putting platform specific metadata in. + */ + IPU3Action op; op.op = ActionMetadataReady; op.controls = ctrls; From patchwork Thu Nov 11 14:09:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14582 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A6E0ABDB1C for ; Thu, 11 Nov 2021 14:09:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3F9D960369; Thu, 11 Nov 2021 15:09:47 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="CkDYtgE8"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8C21260362 for ; Thu, 11 Nov 2021 15:09:33 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 492C21253; Thu, 11 Nov 2021 15:09:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639773; bh=liWKDZrU6Id0fLcs6Ntfnu0PW3DmSxq/jNie3jkZ74k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CkDYtgE8bNCGLBqu/eVNfsGHP6GuYIpuQ/LTSm8wXnwF5Gw8vecVYp9/dEpnT6Ynm Wa+/tqzatjnlFJjQCcoEgKqxEAXy4il9dolJRW8FPUcAp6GEyaHKiI86smE7hQ0/Sj nwwKps9n4TOUBGjfunuxNHCRAL1vp+UVI+uoV/yk= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:26 +0100 Message-Id: <20211111140928.136111-13-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 12/14] ipa: ipu3: Cache line duration at configure call X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We use the line duration several times in the IPAIPU3. Instead of recalculating it each time, cache the value as a utils::Duration. When the value is needed, cast it to a std::micro value. As sensorInfo is no longer used in updateSessionConfiguration remove the reference to it. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index d3195de6..f1597f78 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -148,8 +148,7 @@ private: void updateControls(const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls); - void updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, - const ControlInfoMap &sensorControls); + void updateSessionConfiguration(const ControlInfoMap &sensorControls); void processControls(unsigned int frame, const ControlList &controls); void fillParams(unsigned int frame, ipu3_uapi_params *params); void parseStatistics(unsigned int frame, @@ -174,6 +173,8 @@ private: uint32_t minGain_; uint32_t maxGain_; + utils::Duration lineDuration_; + /* Interface to the Camera Helper */ std::unique_ptr camHelper_; @@ -188,16 +189,12 @@ private: * \brief Compute IPASessionConfiguration using the sensor information and the * sensor V4L2 controls */ -void IPAIPU3::updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, - const ControlInfoMap &sensorControls) +void IPAIPU3::updateSessionConfiguration(const ControlInfoMap &sensorControls) { const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; int32_t minExposure = v4l2Exposure.min().get(); int32_t maxExposure = v4l2Exposure.max().get(); - utils::Duration lineDuration = sensorInfo.lineLength * 1.0s - / sensorInfo.pixelRate; - const ControlInfo &v4l2Gain = sensorControls.find(V4L2_CID_ANALOGUE_GAIN)->second; int32_t minGain = v4l2Gain.min().get(); int32_t maxGain = v4l2Gain.max().get(); @@ -209,8 +206,8 @@ void IPAIPU3::updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, * * \todo take VBLANK into account for maximum shutter speed */ - context_.configuration.agc.minShutterSpeed = minExposure * lineDuration; - context_.configuration.agc.maxShutterSpeed = maxExposure * lineDuration; + context_.configuration.agc.minShutterSpeed = minExposure * lineDuration_; + context_.configuration.agc.maxShutterSpeed = maxExposure * lineDuration_; context_.configuration.agc.minAnalogueGain = camHelper_->gain(minGain); context_.configuration.agc.maxAnalogueGain = camHelper_->gain(maxGain); } @@ -239,11 +236,10 @@ void IPAIPU3::updateControls(const IPACameraSensorInfo &sensorInfo, * exposure min, max and default and convert it from lines to * microseconds. */ - double lineDuration = sensorInfo.lineLength / (sensorInfo.pixelRate / 1e6); const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; - int32_t minExposure = v4l2Exposure.min().get() * lineDuration; - int32_t maxExposure = v4l2Exposure.max().get() * lineDuration; - int32_t defExposure = v4l2Exposure.def().get() * lineDuration; + int32_t minExposure = v4l2Exposure.min().get() * lineDuration_.get(); + int32_t maxExposure = v4l2Exposure.max().get() * lineDuration_.get(); + int32_t defExposure = v4l2Exposure.def().get() * lineDuration_.get(); controls[&controls::ExposureTime] = ControlInfo(minExposure, maxExposure, defExposure); @@ -463,11 +459,13 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, calculateBdsGrid(configInfo.bdsOutputSize); + lineDuration_ = sensorInfo_.lineLength * 1.0s / sensorInfo_.pixelRate; + /* Update the camera controls using the new sensor settings. */ updateControls(sensorInfo_, ctrls_, ipaControls); /* Update the IPASessionConfiguration using the sensor settings. */ - updateSessionConfiguration(sensorInfo_, ctrls_); + updateSessionConfiguration(ctrls_); for (auto const &algo : algorithms_) { int ret = algo->configure(context_, configInfo); @@ -627,8 +625,7 @@ void IPAIPU3::parseStatistics(unsigned int frame, setControls(frame); /* \todo Use VBlank value calculated from each frame exposure. */ - int64_t frameDuration = sensorInfo_.lineLength * (defVBlank_ + sensorInfo_.outputSize.height) / - (sensorInfo_.pixelRate / 1e6); + int64_t frameDuration = (defVBlank_ + sensorInfo_.outputSize.height) * lineDuration_.get(); ctrls.set(controls::FrameDuration, frameDuration); ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); From patchwork Thu Nov 11 14:09:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14583 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 590F3C324F for ; Thu, 11 Nov 2021 14:09:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F1DB960394; Thu, 11 Nov 2021 15:09:47 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="vxqbDnJu"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C20C060376 for ; Thu, 11 Nov 2021 15:09:33 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7B883DEE; Thu, 11 Nov 2021 15:09:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639773; bh=ZZG8ygpZqmBTVkiRCnAYqcI17OWq8hHPemPMSXoMexI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vxqbDnJun3ILvtozstzBdlSym+j9+hG8IiD31hF0UOU5EKx02VxGEri+ldLBqV2ND zNWPwnsIDj6LYoMM/eN4iiEU+HogohOKo2zSH7XEPwk7F/6M2qrts97XcZmsLI/mjQ qMc989sJDzt5KPTQHXoIOwxVWxzmeVXBzVzKp8E0= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:27 +0100 Message-Id: <20211111140928.136111-14-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 13/14] ipa: ipu3: Move ExposureTime to IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have the exposure time calculated, pass it to the controls::ExposureTime and don't use the pipeline handler for it anymore. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 2 ++ src/libcamera/pipeline/ipu3/ipu3.cpp | 16 +--------------- 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index f1597f78..dc6f2ced 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -630,6 +630,8 @@ void IPAIPU3::parseStatistics(unsigned int frame, ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration_.get()); + /* * \todo The Metadata provides a path to getting extended data * out to the application. Further data such as a simplifed Histogram diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index c984ed12..319bb1a3 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -47,7 +47,7 @@ class IPU3CameraData : public Camera::Private { public: IPU3CameraData(PipelineHandler *pipe) - : Camera::Private(pipe), exposureTime_(0), supportsFlips_(false) + : Camera::Private(pipe), supportsFlips_(false) { } @@ -67,7 +67,6 @@ public: Stream vfStream_; Stream rawStream_; - uint32_t exposureTime_; Rectangle cropRegion_; bool supportsFlips_; Transform rotationTransform_; @@ -1045,17 +1044,6 @@ int PipelineHandlerIPU3::updateControls(IPU3CameraData *data) controls[&controls::ScalerCrop] = ControlInfo(minCrop, maxCrop, maxCrop); - /* - * \todo Report the actual exposure time, use the default for the - * moment. - */ - const auto exposureInfo = data->ipaControls_.find(&controls::ExposureTime); - if (exposureInfo == data->ipaControls_.end()) { - LOG(IPU3, Error) << "Exposure control not initialized by the IPA"; - return -EINVAL; - } - data->exposureTime_ = exposureInfo->second.def().get(); - /* Add the IPA registered controls to list of camera controls. */ for (const auto &ipaControl : data->ipaControls_) controls[ipaControl.first] = ipaControl.second; @@ -1317,8 +1305,6 @@ void IPU3CameraData::imguOutputBufferReady(FrameBuffer *buffer) pipe()->completeBuffer(request, buffer); request->metadata().set(controls::draft::PipelineDepth, 3); - /* \todo Move the ExposureTime control to the IPA. */ - request->metadata().set(controls::ExposureTime, exposureTime_); /* \todo Actually apply the scaler crop region to the ImgU. */ if (request->controls().contains(controls::ScalerCrop)) cropRegion_ = request->controls().get(controls::ScalerCrop); From patchwork Thu Nov 11 14:09:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14584 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DFA5BC3250 for ; Thu, 11 Nov 2021 14:09:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 89B94603BE; Thu, 11 Nov 2021 15:09:48 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="YsbfDcrj"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F3C5F60378 for ; Thu, 11 Nov 2021 15:09:33 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AD5C41543; Thu, 11 Nov 2021 15:09:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636639773; bh=BirNg8WHuzLbZZUuU8vZb8XsA3wsf4Xa1TTuL25YHvM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YsbfDcrjWf51fo6LGfyPEVhkYWk2/zG2XMaCKzSamHFxqyUGJEmDb0/ZB7eLrIeyo G0wX4yq0DdM1qk7wXV9QzFpbkSCtiKBMErEgfRgyZxKUgBeVuSp7gSSjRRxaIpvBBH 5ty2XKfszrhjenQtX+eS9ESD8ix16AthWUfo0eMg= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 15:09:28 +0100 Message-Id: <20211111140928.136111-15-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111140928.136111-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 14/14] ipa: ipu3: Pass the AnalogueGain control X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We can set the controls::AnalogueGain metadata now that AGC is updating it correctly. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index dc6f2ced..a8d54a5d 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -628,6 +628,8 @@ void IPAIPU3::parseStatistics(unsigned int frame, int64_t frameDuration = (defVBlank_ + sensorInfo_.outputSize.height) * lineDuration_.get(); ctrls.set(controls::FrameDuration, frameDuration); + ctrls.set(controls::AnalogueGain, context_.frameContext.agc.gain); + ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration_.get());