From patchwork Thu Nov 11 11:05:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14556 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DD3ECBF415 for ; Thu, 11 Nov 2021 11:06:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CC9B06038B; Thu, 11 Nov 2021 12:06:11 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="dYfkQOro"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D47B960345 for ; Thu, 11 Nov 2021 12:06:09 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 61B12556; Thu, 11 Nov 2021 12:06:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628769; bh=N81w8/G3lWtPy4LdH5xSsM77aOlGhlOyxo0vtEPgaEo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dYfkQOro0XGATwtcGt2znAsmxMy+JDaNE6wbg3R6GG0pCrKZP7jc/FiDeZCW+/dJK vYt7en4zxqcRy1ovKLmls0eIBUXnJuu6xkuh8XypUFokggsny/Yibf5RjR/uFYR1C7 wvf3U9JEZQKTC4sPyiA5/CX1vIfnUeBh4/aArMiA= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:52 +0100 Message-Id: <20211111110605.105202-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 01/14] ipa: ipu3: Extend ipu3 ipa interface for sensor controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen IPU3Event and IPU3Action use single ControlList for both libcamera and V4L2 controls, and it's content could be either one based on the context. Extend IPU3Event and IPU3Action for sensor V4L2 controls, and preserve the original one for only libcamera Controls to make the content of an event more specific. Signed-off-by: Han-Lin Chen [Jean-Michel: remove lensControls from the original patch] Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- include/libcamera/ipa/ipu3.mojom | 2 ++ src/ipa/ipu3/ipu3.cpp | 2 +- src/libcamera/pipeline/ipu3/ipu3.cpp | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/libcamera/ipa/ipu3.mojom b/include/libcamera/ipa/ipu3.mojom index 2f254ed4..16e3462e 100644 --- a/include/libcamera/ipa/ipu3.mojom +++ b/include/libcamera/ipa/ipu3.mojom @@ -23,11 +23,13 @@ struct IPU3Event { int64 frameTimestamp; uint32 bufferId; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; struct IPU3Action { IPU3Operations op; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; struct IPAConfigInfo { diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 93b700bd..bcc3863b 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -653,7 +653,7 @@ void IPAIPU3::setControls(unsigned int frame) ControlList ctrls(ctrls_); ctrls.set(V4L2_CID_EXPOSURE, static_cast(exposure_)); ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(gain_)); - op.controls = ctrls; + op.sensorControls = ctrls; queueFrameAction.emit(frame, op); } diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index eb714aa6..8816efc5 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -1248,7 +1248,7 @@ void IPU3CameraData::queueFrameAction(unsigned int id, { switch (action.op) { case ipa::ipu3::ActionSetSensorControls: { - const ControlList &controls = action.controls; + const ControlList &controls = action.sensorControls; delayedCtrls_->push(controls); break; } From patchwork Thu Nov 11 11:05:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14557 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9761BBF415 for ; Thu, 11 Nov 2021 11:06:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B7DC360368; Thu, 11 Nov 2021 12:06:12 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="i0OxHa/p"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E2C4A600B5 for ; Thu, 11 Nov 2021 12:06:09 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9A3B1E51; Thu, 11 Nov 2021 12:06:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628769; bh=zN1A2JCDmbXNy9LOEdVyYSE6rbn1dmLIfjJAeVVCjPQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i0OxHa/phobVvMn5/Pi2Mr7KOF2T3G7PwP4K3rWAaZ7/G0caPaoG5ZQCoBox+D6Rd vUDpPupLs8T7CK4+AJU8sEQr6z+RKabeTeSlvMa+eSWZPj/7oTFXOomLXOV79QaO/X Ylrdi38RP/QFYwv9qDG2cICqwO620qu5kPF1EV3I= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:53 +0100 Message-Id: <20211111110605.105202-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 02/14] ipu3: ipa: Report effective sensor controls with statistics to IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen The Intel close sourced IPA requires the effective controls applied to the sensor when the statistics are generated. Report effective sensor controls with the statistics to IPA. Signed-off-by: Han-Lin Chen [Jean-Michel: Reword s/stastistics/statistics and move reset after IPA start] Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/libcamera/pipeline/ipu3/frames.h | 4 ++++ src/libcamera/pipeline/ipu3/ipu3.cpp | 9 +++++++++ 2 files changed, 13 insertions(+) diff --git a/src/libcamera/pipeline/ipu3/frames.h b/src/libcamera/pipeline/ipu3/frames.h index 3ef7e445..614701e2 100644 --- a/src/libcamera/pipeline/ipu3/frames.h +++ b/src/libcamera/pipeline/ipu3/frames.h @@ -14,6 +14,8 @@ #include +#include + namespace libcamera { class FrameBuffer; @@ -34,6 +36,8 @@ public: FrameBuffer *paramBuffer; FrameBuffer *statBuffer; + ControlList effectiveSensorControls; + bool paramDequeued; bool metadataProcessed; }; diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 8816efc5..3fcfa777 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -767,6 +767,12 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis if (ret) goto error; + /* + * Reset the delayed controls with the gain and exposure values set by + * the IPA. + */ + data->delayedCtrls_->reset(); + /* * Start the ImgU video devices, buffers will be queued to the * ImgU output and viewfinder when requests will be queued. @@ -1363,6 +1369,8 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer) request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + info->effectiveSensorControls = delayedCtrls_->get(buffer->metadata().sequence); + if (request->findBuffer(&rawStream_)) pipe()->completeBuffer(request, buffer); @@ -1419,6 +1427,7 @@ void IPU3CameraData::statBufferReady(FrameBuffer *buffer) ev.frame = info->id; ev.bufferId = info->statBuffer->cookie(); ev.frameTimestamp = request->metadata().get(controls::SensorTimestamp); + ev.sensorControls = info->effectiveSensorControls; ipa_->processEvent(ev); } From patchwork Thu Nov 11 11:05:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14558 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D40B8BF415 for ; Thu, 11 Nov 2021 11:06:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DED6060362; Thu, 11 Nov 2021 12:06:13 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="o86rjCG7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0C9A26035A for ; Thu, 11 Nov 2021 12:06:10 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C7ECB1253; Thu, 11 Nov 2021 12:06:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628769; bh=U6B1s0nyvT5OD4Vi4dFpWNTizD+S776sNyQUai1p+qA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o86rjCG79IAYqvmHGmjd4B+rl0Mo40/ZYDnx2Vxb0f9SXxxF0PtwVYFavGuZ9MaS3 BSYkXVBudzINOeJoQJlWfBxRCpLcKxLIeBFNx8J4HoE6H0xem1hTqfRYq+BYW2fQo0 8BsaTLVPq7UtRgt10vyJKjHq0t80o7BzuRyJuAf8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:54 +0100 Message-Id: <20211111110605.105202-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 03/14] ipa: ipu3: Use sensor controls to update frameContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The pipeline handler populates the new sensorControls ControlList, to have the effective exposure and gain values for the current frame. This is done when a statistics buffer is received. Make those values the frameContext::sensor values for the frame when the EventStatReady event is received. AGC also needs to use frameContext.sensor as its input values and frameContext.agc as its output values. Modify computeExposure by passing it the frameContext instead of individual exposure and gain values. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 19 ++++++++++--------- src/ipa/ipu3/algorithms/agc.h | 2 +- src/ipa/ipu3/ipa_context.cpp | 11 +++++++++++ src/ipa/ipu3/ipa_context.h | 5 +++++ src/ipa/ipu3/ipu3.cpp | 4 ++++ 5 files changed, 31 insertions(+), 10 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index b5d736c1..825b35d4 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -169,10 +169,9 @@ void Agc::filterExposure() /** * \brief Estimate the new exposure and gain values - * \param[inout] exposure The exposure value reference as a number of lines - * \param[inout] gain The gain reference to be updated + * \param[inout] frameContext The shared IPA frame Context */ -void Agc::computeExposure(uint32_t &exposure, double &analogueGain) +void Agc::computeExposure(IPAFrameContext &frameContext) { /* Algorithm initialization should wait for first valid frames */ /* \todo - have a number of frames given by DelayedControls ? @@ -189,6 +188,10 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) return; } + /* Get the effective exposure and gain applied on the sensor. */ + uint32_t &exposure = frameContext.sensor.exposure; + double &analogueGain = frameContext.sensor.gain; + /* Estimate the gain needed to have the proportion wanted */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; @@ -233,8 +236,9 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) << shutterTime << " and " << stepGain; - exposure = shutterTime / lineDuration_; - analogueGain = stepGain; + /* Update the estimated exposure and gain. */ + frameContext.agc.exposure = shutterTime / lineDuration_; + frameContext.agc.gain = stepGain; /* * Update the exposure value for the next process call. @@ -257,11 +261,8 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain) */ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { - /* Get the latest exposure and gain applied */ - uint32_t &exposure = context.frameContext.agc.exposure; - double &analogueGain = context.frameContext.agc.gain; measureBrightness(stats, context.configuration.grid.bdsGrid); - computeExposure(exposure, analogueGain); + computeExposure(context.frameContext); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 69e0b831..f0db25ee 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -34,7 +34,7 @@ private: void measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); void filterExposure(); - void computeExposure(uint32_t &exposure, double &gain); + void computeExposure(IPAFrameContext &frameContext); uint64_t frameCount_; uint64_t lastFrame_; diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 2355a9c7..a7ff957d 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -119,6 +119,17 @@ namespace libcamera::ipa::ipu3 { * \brief White balance gain for B channel */ +/** + * \var IPAFrameContext::sensor + * \brief Effective sensor values + * + * \var IPAFrameContext::sensor.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::sensor.gain + * \brief Analogue gain multiplier + */ + /** * \var IPAFrameContext::toneMapping * \brief Context for ToneMapping and Gamma control diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 1e46c61a..a5a19800 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -47,6 +47,11 @@ struct IPAFrameContext { } gains; } awb; + struct { + uint32_t exposure; + double gain; + } sensor; + struct { double gamma; struct ipu3_uapi_gamma_corr_lut gammaCorrection; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index bcc3863b..dee002a5 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -549,6 +549,10 @@ void IPAIPU3::processEvent(const IPU3Event &event) const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); + /* \todo move those into processControls ? */ + context_.frameContext.sensor.exposure = event.sensorControls.get(V4L2_CID_EXPOSURE).get(); + context_.frameContext.sensor.gain = camHelper_->gain(event.sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + parseStatistics(event.frame, event.frameTimestamp, stats); break; } From patchwork Thu Nov 11 11:05:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14559 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D682DC324E for ; Thu, 11 Nov 2021 11:06:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9BA0C60361; Thu, 11 Nov 2021 12:06:15 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="fCH09suV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 47ECC6035D for ; Thu, 11 Nov 2021 12:06:10 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 014FA1516; Thu, 11 Nov 2021 12:06:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628770; bh=lnAOQaPrbGcZ6oTYL7fUiUyq7QQmH6uxS9qDF5v9TW4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fCH09suV/6N5t9bBfmG2N1ZxuGUvFeHYzdrcNCAiLQJQCpwxcnv+lKACAFN/qcwCn tUiYGlFzm9wCAW5S63kziIUyVVcGA0OHsxq2g69UhKHO63FtG1jk7XzgXTw+qp3nCl VFD27OVMQY3uEMBrPlwAv8RIVoK+yoAd/GjnDXKU= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:55 +0100 Message-Id: <20211111110605.105202-5-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 04/14] ipa: ipu3: agc: Limit the number of saturated cells X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the histogram is calculated, we check if a cell is saturated or not before cumulating its green value. This is wrong, and it can lead to an empty histogram in case of a fully saturated frame. Use a constant to limit the amount of pixels within a cell before considering it saturated. If at the end of the loop we still have an empty histogram, then make it a fully saturated one. Bug: https://bugs.libcamera.org/show_bug.cgi?id=84 Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 825b35d4..0ae575a3 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -63,6 +63,12 @@ static constexpr uint32_t knumHistogramBins = 256; /* Target value to reach for the top 2% of the histogram */ static constexpr double kEvGainTarget = 0.5; +/* + * Maximum ratio of saturated pixels in a cell for the cell to be considered + * non-saturated and counted by the AGC algorithm. + */ +static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; + Agc::Agc() : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), @@ -124,7 +130,7 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, &stats->awb_raw_buffer.meta_data[cellPosition] ); - if (cell->sat_ratio == 0) { + if (cell->sat_ratio <= kMinCellsPerZoneRatio) { uint8_t gr = cell->Gr_avg; uint8_t gb = cell->Gb_avg; /* @@ -137,8 +143,14 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, } } + Histogram cumulativeHist = Histogram(Span(hist)); /* Estimate the quantile mean of the top 2% of the histogram */ - iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); + if (cumulativeHist.total() == 0) { + /* Force the value as histogram is empty */ + iqMean_ = knumHistogramBins - 0.5; + } else { + iqMean_ = cumulativeHist.interQuantileMean(0.98, 1.0); + } } /** From patchwork Thu Nov 11 11:05:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14560 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 86FCFBF415 for ; Thu, 11 Nov 2021 11:06:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 14F6460369; Thu, 11 Nov 2021 12:06:16 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rBYGfTx4"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7ACD460345 for ; Thu, 11 Nov 2021 12:06:10 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 335531543; Thu, 11 Nov 2021 12:06:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628770; bh=LlZGDXH5lxpj6GorSDNn50EUKU2Bjq0JdYuHBX9ctJs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rBYGfTx4iPfxlqtVtRr7GFZoZrvn/b5Qz7yVFiE0Tqh9cN2Dr6EuYvK+MswNZ28ul YkKmSCPiCEBUFOG3E/Ct5nec1yLosuMuN8jmbKxlVqO/ukms9AfxA39xJZs6TSyFSo vMuoCTwM3zb1Yo83NJQ6UHURoTr/ycSUhsDd4Ttg= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:56 +0100 Message-Id: <20211111110605.105202-6-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 05/14] ipa: ipu3: agc: Compute the gain for each frame X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have the real exposure applied at each frame, remove the early return based on a frame counter and compute the gain for each frame. Instead of that, introduce a number of startup frames during which the filter speed is 1.0, meaning we apply instantly the exposure value calculated and not a slower filtered one. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 0ae575a3..862366ac 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -45,11 +45,6 @@ namespace ipa::ipu3::algorithms { LOG_DEFINE_CATEGORY(IPU3Agc) -/* Number of frames to wait before calculating stats on minimum exposure */ -static constexpr uint32_t kInitialFrameMinAECount = 4; -/* Number of frames to wait between new gain/shutter time estimations */ -static constexpr uint32_t kFrameSkipCount = 6; - /* Limits for analogue gain values */ static constexpr double kMinAnalogueGain = 1.0; static constexpr double kMaxAnalogueGain = 8.0; @@ -69,10 +64,13 @@ static constexpr double kEvGainTarget = 0.5; */ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; +/* Number of frames to wait before calculating stats on minimum exposure */ +static constexpr uint32_t kNumStartupFrames = 10; + Agc::Agc() - : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), - minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), - currentExposure_(0s), prevExposureValue_(0s) + : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), + maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), + prevExposureValue_(0s) { } @@ -159,6 +157,11 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, void Agc::filterExposure() { double speed = 0.2; + + /* Adapt instantly if we are in startup phase */ + if (frameCount_ < kNumStartupFrames) + speed = 1.0; + if (filteredExposure_ == 0s) { /* DG stands for digital gain.*/ filteredExposure_ = currentExposure_; @@ -185,13 +188,6 @@ void Agc::filterExposure() */ void Agc::computeExposure(IPAFrameContext &frameContext) { - /* Algorithm initialization should wait for first valid frames */ - /* \todo - have a number of frames given by DelayedControls ? - * - implement a function for IIR */ - if ((frameCount_ < kInitialFrameMinAECount) || (frameCount_ - lastFrame_ < kFrameSkipCount)) - return; - - lastFrame_ = frameCount_; /* Are we correctly exposed ? */ if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { From patchwork Thu Nov 11 11:05:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14561 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 35900C324E for ; Thu, 11 Nov 2021 11:06:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 580206036F; Thu, 11 Nov 2021 12:06:19 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="jDvNJMnc"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AD7D060369 for ; Thu, 11 Nov 2021 12:06:10 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6604C4A6; Thu, 11 Nov 2021 12:06:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628770; bh=EG1KRtC2FK51EjXX/dSiaGYIrwZ9w0KhblE0WnJdxpc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jDvNJMnchk3eO+k5Ps0EeugqmDrPrb3kz45XX/Sk/OAsmwpbjnN5XMHOMbcBmX2ER Tp2+HiJlfA5CTUUSaA84jHNZt11AY5EKDmfDghVLuYqYCG85U9rq31Edz2iWdt0Iza l0FnmCGouztdbvzyQhxqx336WE2sWKDkYyFt6CbU= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:57 +0100 Message-Id: <20211111110605.105202-7-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 06/14] ipa: ipu3: agc: Refactor ev gain calculation and testing X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When we compute the new gain, we use the iqMean_ and estimate an exposure value gain to apply. Return early when the gain is less than 1%. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 862366ac..71a7f39a 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -188,14 +188,6 @@ void Agc::filterExposure() */ void Agc::computeExposure(IPAFrameContext &frameContext) { - - /* Are we correctly exposed ? */ - if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { - LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " - << iqMean_ << ")"; - return; - } - /* Get the effective exposure and gain applied on the sensor. */ uint32_t &exposure = frameContext.sensor.exposure; double &analogueGain = frameContext.sensor.gain; @@ -203,6 +195,12 @@ void Agc::computeExposure(IPAFrameContext &frameContext) /* Estimate the gain needed to have the proportion wanted */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; + if (std::abs(evGain - 1.0) < 0.01) { + LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " + << iqMean_ << ")"; + return; + } + /* extracted from Rpi::Agc::computeTargetExposure */ /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; From patchwork Thu Nov 11 11:05:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14562 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BA5D1C324F for ; Thu, 11 Nov 2021 11:06:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 40B666038C; Thu, 11 Nov 2021 12:06:20 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="jc/wCmxR"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DB323600B5 for ; Thu, 11 Nov 2021 12:06:10 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 97F21556; Thu, 11 Nov 2021 12:06:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628770; bh=Am5HIUg9QZGRHn+WCUWC8PSuDnD5NZrCxp3jOHlKm+Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jc/wCmxRCiCbFx++pDF+T/uD0YhLzYIIoUEHrb5qyIcTAtJuxqVql8AADCjRn7tYG XKQKWj2xfADDjTyLE6Ci29jX5i+Olcvzka34wY+KqIInH9AXoXcZhASV+XyumRYc5Y CLwIPpbs9gkZT5wlGAl+YV8B1OUDfig0lSow+tvg= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:58 +0100 Message-Id: <20211111110605.105202-8-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 07/14] ipa: ipu3: agc: Improve gain calculation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When an image is partially saturated, its brightness is not increasing linearly when the shutter time or gain increases. It is a big issue with a backlight as the algorithm is fading to darkness right now. Introduce a function to estimate the brightness of the frame, based on the current exposure/gain and loop on it several times to estimate it again and approach the non linear function. Inspired-by: 7de5506c30b3 ("libcamera: src: ipa: raspberrypi: agc: Improve gain update calculation for partly saturated images") Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 105 +++++++++++++++++++++++++++++++- src/ipa/ipu3/algorithms/agc.h | 6 +- 2 files changed, 107 insertions(+), 4 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 71a7f39a..fd05822b 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -67,6 +67,17 @@ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 20 / 100; /* Number of frames to wait before calculating stats on minimum exposure */ static constexpr uint32_t kNumStartupFrames = 10; +/* Maximum luminance used for brightness normalization */ +static constexpr uint32_t kMaxLuminance = 255; + +/* + * Normalized luma value target. + * + * It's a number that's chosen so that, when the camera points at a grey + * target, the resulting image brightness is considered right. + */ +static constexpr double kNormalizedLumaTarget = 0.16; + Agc::Agc() : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), @@ -185,14 +196,20 @@ void Agc::filterExposure() /** * \brief Estimate the new exposure and gain values * \param[inout] frameContext The shared IPA frame Context + * \param[in] currentYGain The gain calculated on the current brightness level */ -void Agc::computeExposure(IPAFrameContext &frameContext) +void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) { /* Get the effective exposure and gain applied on the sensor. */ uint32_t &exposure = frameContext.sensor.exposure; double &analogueGain = frameContext.sensor.gain; - /* Estimate the gain needed to have the proportion wanted */ + /* + * Estimate the gain needed to have the proportion of pixels in a given + * range wanted. iqMean_ returns the mean value of the top 2% of the + * cumulative histogram, and we want it to be as close as possible to a + * configured target. + */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; if (std::abs(evGain - 1.0) < 0.01) { @@ -202,6 +219,7 @@ void Agc::computeExposure(IPAFrameContext &frameContext) } /* extracted from Rpi::Agc::computeTargetExposure */ + /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain @@ -209,6 +227,14 @@ void Agc::computeExposure(IPAFrameContext &frameContext) << " Gain " << analogueGain << " Needed ev gain " << evGain; + if (evGain < currentYGain) + evGain = currentYGain; + + /* Consider within 1% of the target as correctly exposed */ + if (std::abs(evGain - 1.0) < 0.01) + LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " + << iqMean_ << ")"; + /* * Calculate the current exposure value for the scene as the latest * exposure value applied multiplied by the new estimated gain. @@ -257,6 +283,57 @@ void Agc::computeExposure(IPAFrameContext &frameContext) prevExposureValue_ = shutterTime * analogueGain; } +/** + * \brief Estimate the average brightness of the frame + * \param[in] frameContext The shared IPA frame context + * \param[in] grid The grid used to store the statistics in the IPU3 + * \param[in] stats The IPU3 statistics and ISP results + * \param[in] currentYGain The gain calculated on the current brightness level + * \return The normalized luma + * + * Luma is the weighted sum of gamma-compressed R′G′B′ components of a color + * video. The luma values are normalized as 0.0 to 1.0, with 1.0 being a + * theoretical perfect reflector of 100% reference white. We use the Rec. 601 + * luma here. + * + * More detailed information can be found in: + * https://en.wikipedia.org/wiki/Luma_(video) + */ +double Agc::computeInitialY(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain) +{ + double redSum = 0, greenSum = 0, blueSum = 0; + + for (unsigned int cellY = 0; cellY < grid.height; cellY++) { + for (unsigned int cellX = 0; cellX < grid.width; cellX++) { + uint32_t cellPosition = cellY * stride_ + cellX; + + const ipu3_uapi_awb_set_item *cell = + reinterpret_cast( + &stats->awb_raw_buffer.meta_data[cellPosition] + ); + + redSum += cell->R_avg * currentYGain; + greenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * currentYGain; + blueSum += cell->B_avg * currentYGain; + } + } + + /* + * Estimate the sum of the brightness values, weighted with the gains + * applied on the channels in AWB as the Rec. 601 luma. + */ + double Y_sum = redSum * frameContext.awb.gains.red * .299 + + greenSum * frameContext.awb.gains.green * .587 + + blueSum * frameContext.awb.gains.blue * .114; + + /* Return the normalized relative luminance. */ + return Y_sum / (grid.height * grid.width) / kMaxLuminance; +} + + /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -268,7 +345,29 @@ void Agc::computeExposure(IPAFrameContext &frameContext) void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { measureBrightness(stats, context.configuration.grid.bdsGrid); - computeExposure(context.frameContext); + + double currentYGain = 1.0; + double targetY = kNormalizedLumaTarget; + + /* + * Do this calculation a few times as brightness increase can be + * non-linear when there are saturated regions. + */ + for (int i = 0; i < 8; i++) { + double initialY = computeInitialY(context.frameContext, + context.configuration.grid.bdsGrid, + stats, currentYGain); + double extra_gain = std::min(10.0, targetY / (initialY + .001)); + + currentYGain *= extra_gain; + LOG(IPU3Agc, Debug) << "Initial Y " << initialY + << " target " << targetY + << " gives gain " << currentYGain; + if (extra_gain < 1.01) + break; + } + + computeExposure(context.frameContext, currentYGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index f0db25ee..79736283 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -34,7 +34,11 @@ private: void measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); void filterExposure(); - void computeExposure(IPAFrameContext &frameContext); + void computeExposure(IPAFrameContext &frameContext, double currentYGain); + double computeInitialY(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain); uint64_t frameCount_; uint64_t lastFrame_; From patchwork Thu Nov 11 11:05:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14563 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0956EC3250 for ; Thu, 11 Nov 2021 11:06:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 26A2D60394; Thu, 11 Nov 2021 12:06:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ob0gO/wO"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 183E86035A for ; Thu, 11 Nov 2021 12:06:11 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C9F52E51; Thu, 11 Nov 2021 12:06:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628770; bh=G9jYSqLoa1fGQNKUJPHJfkmc4GroDOgky/Ev4jTzDaI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ob0gO/wOtu5wjSHmEESY8kXegW5gIZvDHgk0JA8h7BCINpN1ALQ+3CmVeCCFNp0jv eSVyAPM/6lOxOmLzaMRtnLygFU3Uio6SfmzVO13IXCWxoRgebDZ7CSkFm5tpVJZxC7 5yc1A949qhWcSoEmKntkHTJFQJJmd1VN1QAfP8Ow= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:05:59 +0100 Message-Id: <20211111110605.105202-9-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 08/14] ipa: ipu3: agc: Update previous exposure value X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Previously, the exposure value was calculated based on the estimated shutter time and gain applied. Now that we have the real values for the current frame, use those before estimating the next one and rename the variable accordingly. As the exposure value is updated in the beginning of the computation, there is no need to initialize effectiveExposureValue anymore in the configure call. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/agc.cpp | 25 +++++++++---------------- src/ipa/ipu3/algorithms/agc.h | 2 +- 2 files changed, 10 insertions(+), 17 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index fd05822b..e828d9cb 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -81,7 +81,7 @@ static constexpr double kNormalizedLumaTarget = 0.16; Agc::Agc() : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), - prevExposureValue_(0s) + effectiveExposureValue_(0s) { } @@ -112,10 +112,6 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) context.frameContext.agc.gain = minAnalogueGain_; context.frameContext.agc.exposure = minExposureLines_; - prevExposureValue_ = context.frameContext.agc.gain - * context.frameContext.agc.exposure - * lineDuration_; - return 0; } @@ -222,6 +218,13 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Calculate the shutter time in seconds */ utils::Duration currentShutter = exposure * lineDuration_; + + /* + * Update the exposure value for the next computation using the values + * of exposure and gain really used by the sensor. + */ + effectiveExposureValue_ = currentShutter * analogueGain; + LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain << " Shutter speed " << currentShutter << " Gain " << analogueGain @@ -239,7 +242,7 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * Calculate the current exposure value for the scene as the latest * exposure value applied multiplied by the new estimated gain. */ - currentExposure_ = prevExposureValue_ * evGain; + currentExposure_ = effectiveExposureValue_ * evGain; utils::Duration minShutterSpeed = minExposureLines_ * lineDuration_; utils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_; @@ -271,16 +274,6 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Update the estimated exposure and gain. */ frameContext.agc.exposure = shutterTime / lineDuration_; frameContext.agc.gain = stepGain; - - /* - * Update the exposure value for the next process call. - * - * \todo Obtain the values of the exposure time and analog gain - * that were actually used by the sensor, either from embedded - * data when available, or from the delayed controls - * infrastructure in case a slow down caused a mismatch. - */ - prevExposureValue_ = shutterTime * analogueGain; } /** diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 79736283..1a58c79c 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -54,7 +54,7 @@ private: utils::Duration filteredExposure_; utils::Duration currentExposure_; - utils::Duration prevExposureValue_; + utils::Duration effectiveExposureValue_; uint32_t stride_; }; From patchwork Thu Nov 11 11:06:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14564 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4D17BBF415 for ; Thu, 11 Nov 2021 11:06:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E0AB76036B; Thu, 11 Nov 2021 12:06:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="PwdK8gwU"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4EE816036F for ; Thu, 11 Nov 2021 12:06:11 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 036191253; Thu, 11 Nov 2021 12:06:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628771; bh=DI9LAruhDgU9GRKu64HGNzKYbiAUo23pF4WcSWyOl0E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PwdK8gwUxuIgHPkOjHlxO5cIISP6W+MnfXYKWou4AFFGExYaoe6QTkxHehoa5EZxr hMhBSKTQmdktD9EPCXbSWqE+yrd3XCRLJr4ZXMtPt+M2ufaMIowAjbCbXDQurXHwyY aumzEQBF3NX7GVTlQphyTLO1EwpdUjeHoCOA5fiI= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:06:00 +0100 Message-Id: <20211111110605.105202-10-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 09/14] ipa: ipu3: agc: Use exposure in time for storage X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The minimum and maximum exposure are stored in lines. Replace it by values in time to simplify the calculations. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 21 +++++++++------------ src/ipa/ipu3/algorithms/agc.h | 4 ++-- 2 files changed, 11 insertions(+), 14 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index e828d9cb..32a7882b 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -79,8 +79,8 @@ static constexpr uint32_t kMaxLuminance = 255; static constexpr double kNormalizedLumaTarget = 0.16; Agc::Agc() - : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minExposureLines_(0), - maxExposureLines_(0), filteredExposure_(0s), currentExposure_(0s), + : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minShutterSpeed_(0s), + maxShutterSpeed_(0s), filteredExposure_(0s), currentExposure_(0s), effectiveExposureValue_(0s) { } @@ -100,17 +100,16 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) lineDuration_ = configInfo.sensorInfo.lineLength * 1.0s / configInfo.sensorInfo.pixelRate; - /* \todo replace the exposure in lines storage with time based ones. */ - minExposureLines_ = context.configuration.agc.minShutterSpeed / lineDuration_; - maxExposureLines_ = std::min(context.configuration.agc.maxShutterSpeed / lineDuration_, - kMaxShutterSpeed / lineDuration_); + minShutterSpeed_ = context.configuration.agc.minShutterSpeed; + maxShutterSpeed_ = std::min(context.configuration.agc.maxShutterSpeed, + kMaxShutterSpeed); minAnalogueGain_ = std::max(context.configuration.agc.minAnalogueGain, kMinAnalogueGain); maxAnalogueGain_ = std::min(context.configuration.agc.maxAnalogueGain, kMaxAnalogueGain); /* Configure the default exposure and gain. */ context.frameContext.agc.gain = minAnalogueGain_; - context.frameContext.agc.exposure = minExposureLines_; + context.frameContext.agc.exposure = minShutterSpeed_ / lineDuration_; return 0; } @@ -243,11 +242,9 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * exposure value applied multiplied by the new estimated gain. */ currentExposure_ = effectiveExposureValue_ * evGain; - utils::Duration minShutterSpeed = minExposureLines_ * lineDuration_; - utils::Duration maxShutterSpeed = maxExposureLines_ * lineDuration_; /* Clamp the exposure value to the min and max authorized */ - utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain_; + utils::Duration maxTotalExposure = maxShutterSpeed_ * maxAnalogueGain_; currentExposure_ = std::min(currentExposure_, maxTotalExposure); LOG(IPU3Agc, Debug) << "Target total exposure " << currentExposure_ << ", maximum is " << maxTotalExposure; @@ -257,14 +254,14 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) /* Divide the exposure value as new exposure and gain values */ utils::Duration exposureValue = filteredExposure_; - utils::Duration shutterTime = minShutterSpeed; + utils::Duration shutterTime; /* * Push the shutter time up to the maximum first, and only then * increase the gain. */ shutterTime = std::clamp(exposureValue / minAnalogueGain_, - minShutterSpeed, maxShutterSpeed); + minShutterSpeed_, maxShutterSpeed_); double stepGain = std::clamp(exposureValue / shutterTime, minAnalogueGain_, maxAnalogueGain_); LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 1a58c79c..2a4cfdad 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -46,8 +46,8 @@ private: double iqMean_; utils::Duration lineDuration_; - uint32_t minExposureLines_; - uint32_t maxExposureLines_; + utils::Duration minShutterSpeed_; + utils::Duration maxShutterSpeed_; double minAnalogueGain_; double maxAnalogueGain_; From patchwork Thu Nov 11 11:06:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14565 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B23B4C3251 for ; Thu, 11 Nov 2021 11:06:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 57690603B5; Thu, 11 Nov 2021 12:06:22 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hrM3mETO"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 77C9B60376 for ; Thu, 11 Nov 2021 12:06:11 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3502D180D; Thu, 11 Nov 2021 12:06:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628771; bh=4tNa6FTLtligxhLvhqy7ErCQZN9FVjZQl++ok5GJQiU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hrM3mETOJxJb0mYIEf7dbbF7Q+q5C7jv6syIFdv628LmGbDAFWssedfTrYtFV76xa GaBh7v6VzwYXgm1RI7HMSprbizIhXtSUWZheA9blZ+iUzjZOZ/U7SQnrFExBdgYiLD oie5jHBZX2QtOB5Vz8mCNtqvcziG9JgdKJXeF554= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:06:01 +0100 Message-Id: <20211111110605.105202-11-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 10/14] ipa: ipu3: awb: Add support for color temperature X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AWB estimates the color temperature, but it is not used at all. It can be useful for debug purpose at least, but also for lux estimation later, to be able to know the temperature estimated for a given frame. Add a new member to the IPAFrameContext::awb for this purpose, and update the value in AWB. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/awb.cpp | 1 + src/ipa/ipu3/ipa_context.cpp | 3 +++ src/ipa/ipu3/ipa_context.h | 2 ++ 3 files changed, 6 insertions(+) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index a4114659..c7bcb20e 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -385,6 +385,7 @@ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) context.frameContext.awb.gains.blue = asyncResults_.blueGain; context.frameContext.awb.gains.green = asyncResults_.greenGain; context.frameContext.awb.gains.red = asyncResults_.redGain; + context.frameContext.awb.temperatureK = asyncResults_.temperatureK; } constexpr uint16_t Awb::threshold(float value) diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index a7ff957d..99caf9ad 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -117,6 +117,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPAFrameContext::awb.gains.blue * \brief White balance gain for B channel + * + * \var IPAFrameContext::awb.temperatureK + * \brief Estimated color temperature */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index a5a19800..fd97e240 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -45,6 +45,8 @@ struct IPAFrameContext { double green; double blue; } gains; + + double temperatureK; } awb; struct { From patchwork Thu Nov 11 11:06:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14566 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 487E1C324E for ; Thu, 11 Nov 2021 11:06:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DDAE66039C; Thu, 11 Nov 2021 12:06:22 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="RAPCgdPN"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B493C6036B for ; Thu, 11 Nov 2021 12:06:11 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 65FB81516; Thu, 11 Nov 2021 12:06:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628771; bh=YX16MVOhRGH9pRktKox1inFdSvi5ffQz01PP1FzGKO4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RAPCgdPNP3qiqp/1GIRRyLIwi8RmGwWzdwCVOFQArujy7mIJOkftUwcOaMdUTD8lQ 08B8UlGkJAYi0bML6CKm5FMDREKilt0BT7p1fr6HqXYAzvsiQJ3/V7EaIwawkXWgwy +uw3cTyaLG8KUSUSJHUf2PNl6iNhfFBQdDQPlLP8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:06:02 +0100 Message-Id: <20211111110605.105202-12-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 11/14] ipa: ipu3: Send color temperature in the metadata X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the color temperature is updated per-frame, use the value and set the corresponding controls::ColourTemperature. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index dee002a5..7762c5ec 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -632,6 +632,16 @@ void IPAIPU3::parseStatistics(unsigned int frame, (sensorInfo_.pixelRate / 1e6); ctrls.set(controls::FrameDuration, frameDuration); + ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + + /* + * \todo The Metadata provides a path to getting extended data + * out to the application. Further data such as a simplifed Histogram + * might have value to be exposed, however such data may be + * difficult to report in a generically parsable way and we + * likely want to avoid putting platform specific metadata in. + */ + IPU3Action op; op.op = ActionMetadataReady; op.controls = ctrls; From patchwork Thu Nov 11 11:06:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14568 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7B67EC324F for ; Thu, 11 Nov 2021 11:06:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 350996037A; Thu, 11 Nov 2021 12:06:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Fg9PDOI9"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DB68C6038C for ; Thu, 11 Nov 2021 12:06:11 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9836F1543; Thu, 11 Nov 2021 12:06:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628771; bh=Ibcoz8ZT0qS7NH3RAzlzA1DrCQiVsEEY+1t0CHwxd2Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Fg9PDOI9sLYr8qMvLzVFFJ5gH8gM3PDLSCOkLRFgxY2fFgh1GY6W0LN+5ByDEs+9d EeW55kwKtQ5HL9pEh/Vht0cfeILGdQe2mtbO+RM9A3m/5x/DuCY/iv+Gw2K+XEXmAd PL+8bwXPPVM1m9EqduynoEbivNwl1VomGoghQnQM= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:06:03 +0100 Message-Id: <20211111110605.105202-13-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 12/14] ipa: ipu3: Cache line duration at configure call X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We use the line duration several times in the IPAIPU3. Instead of recalculating it each time, cache the value as a utils::Duration. When the value is needed in double, cast it to a std::micro value. As sensorInfo is no longer used in updateSessionConfiguration remove the reference to it. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 7762c5ec..3c1aa5ad 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -148,8 +148,7 @@ private: void updateControls(const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls); - void updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, - const ControlInfoMap &sensorControls); + void updateSessionConfiguration(const ControlInfoMap &sensorControls); void processControls(unsigned int frame, const ControlList &controls); void fillParams(unsigned int frame, ipu3_uapi_params *params); void parseStatistics(unsigned int frame, @@ -174,6 +173,8 @@ private: uint32_t minGain_; uint32_t maxGain_; + utils::Duration lineDuration_; + /* Interface to the Camera Helper */ std::unique_ptr camHelper_; @@ -188,16 +189,12 @@ private: * \brief Compute IPASessionConfiguration using the sensor information and the * sensor V4L2 controls */ -void IPAIPU3::updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, - const ControlInfoMap &sensorControls) +void IPAIPU3::updateSessionConfiguration(const ControlInfoMap &sensorControls) { const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; int32_t minExposure = v4l2Exposure.min().get(); int32_t maxExposure = v4l2Exposure.max().get(); - utils::Duration lineDuration = sensorInfo.lineLength * 1.0s - / sensorInfo.pixelRate; - const ControlInfo &v4l2Gain = sensorControls.find(V4L2_CID_ANALOGUE_GAIN)->second; int32_t minGain = v4l2Gain.min().get(); int32_t maxGain = v4l2Gain.max().get(); @@ -209,8 +206,8 @@ void IPAIPU3::updateSessionConfiguration(const IPACameraSensorInfo &sensorInfo, * * \todo take VBLANK into account for maximum shutter speed */ - context_.configuration.agc.minShutterSpeed = minExposure * lineDuration; - context_.configuration.agc.maxShutterSpeed = maxExposure * lineDuration; + context_.configuration.agc.minShutterSpeed = minExposure * lineDuration_; + context_.configuration.agc.maxShutterSpeed = maxExposure * lineDuration_; context_.configuration.agc.minAnalogueGain = camHelper_->gain(minGain); context_.configuration.agc.maxAnalogueGain = camHelper_->gain(maxGain); } @@ -239,11 +236,10 @@ void IPAIPU3::updateControls(const IPACameraSensorInfo &sensorInfo, * exposure min, max and default and convert it from lines to * microseconds. */ - double lineDuration = sensorInfo.lineLength / (sensorInfo.pixelRate / 1e6); const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; - int32_t minExposure = v4l2Exposure.min().get() * lineDuration; - int32_t maxExposure = v4l2Exposure.max().get() * lineDuration; - int32_t defExposure = v4l2Exposure.def().get() * lineDuration; + int32_t minExposure = v4l2Exposure.min().get() * lineDuration_.get(); + int32_t maxExposure = v4l2Exposure.max().get() * lineDuration_.get(); + int32_t defExposure = v4l2Exposure.def().get() * lineDuration_.get(); controls[&controls::ExposureTime] = ControlInfo(minExposure, maxExposure, defExposure); @@ -463,11 +459,13 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, calculateBdsGrid(configInfo.bdsOutputSize); + lineDuration_ = sensorInfo_.lineLength * 1.0s / sensorInfo_.pixelRate; + /* Update the camera controls using the new sensor settings. */ updateControls(sensorInfo_, ctrls_, ipaControls); /* Update the IPASessionConfiguration using the sensor settings. */ - updateSessionConfiguration(sensorInfo_, ctrls_); + updateSessionConfiguration(ctrls_); for (auto const &algo : algorithms_) { int ret = algo->configure(context_, configInfo); @@ -628,8 +626,7 @@ void IPAIPU3::parseStatistics(unsigned int frame, setControls(frame); /* \todo Use VBlank value calculated from each frame exposure. */ - int64_t frameDuration = sensorInfo_.lineLength * (defVBlank_ + sensorInfo_.outputSize.height) / - (sensorInfo_.pixelRate / 1e6); + int64_t frameDuration = (defVBlank_ + sensorInfo_.outputSize.height) * lineDuration_.get(); ctrls.set(controls::FrameDuration, frameDuration); ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); From patchwork Thu Nov 11 11:06:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14567 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 21E91C3252 for ; Thu, 11 Nov 2021 11:06:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BC560603C9; Thu, 11 Nov 2021 12:06:23 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="vNBMnvg4"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1660A60398 for ; Thu, 11 Nov 2021 12:06:12 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C5E2B4A6; Thu, 11 Nov 2021 12:06:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628771; bh=3IRMIAWUezeGngzYle6fpefQ2VGWHfQcgIEfF2OIcGY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vNBMnvg4AcSZi6IKlPscz7UEH3+1Jwkov73wn3HtF2OJX50kZ47GoyMUYwuZ96/LQ e/lYFNdkK9krCeOi5c6klUhnBT+Kj6xloecTDhO5cDkSJMtfSoDsWeLTeDEOrMkKwh bMpymXzelUNa3KXo2E251CgWsLktTqCcqVHyzIBY= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:06:04 +0100 Message-Id: <20211111110605.105202-14-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 13/14] ipa: ipu3: Move ExposureTime to IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have the exposure time calculated, pass it to the controls::ExposureTime and don't use the pipeline handler for it anymore. While at it, use the same line duration value for ExposureTime and FrameDuration. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 2 ++ src/libcamera/pipeline/ipu3/ipu3.cpp | 16 +--------------- 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 3c1aa5ad..a91550b9 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -631,6 +631,8 @@ void IPAIPU3::parseStatistics(unsigned int frame, ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); + ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration_.get()); + /* * \todo The Metadata provides a path to getting extended data * out to the application. Further data such as a simplifed Histogram diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 3fcfa777..fe37e9c7 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -47,7 +47,7 @@ class IPU3CameraData : public Camera::Private { public: IPU3CameraData(PipelineHandler *pipe) - : Camera::Private(pipe), exposureTime_(0), supportsFlips_(false) + : Camera::Private(pipe), supportsFlips_(false) { } @@ -67,7 +67,6 @@ public: Stream vfStream_; Stream rawStream_; - uint32_t exposureTime_; Rectangle cropRegion_; bool supportsFlips_; Transform rotationTransform_; @@ -1049,17 +1048,6 @@ int PipelineHandlerIPU3::updateControls(IPU3CameraData *data) controls[&controls::ScalerCrop] = ControlInfo(minCrop, maxCrop, maxCrop); - /* - * \todo Report the actual exposure time, use the default for the - * moment. - */ - const auto exposureInfo = data->ipaControls_.find(&controls::ExposureTime); - if (exposureInfo == data->ipaControls_.end()) { - LOG(IPU3, Error) << "Exposure control not initialized by the IPA"; - return -EINVAL; - } - data->exposureTime_ = exposureInfo->second.def().get(); - /* Add the IPA registered controls to list of camera controls. */ for (const auto &ipaControl : data->ipaControls_) controls[ipaControl.first] = ipaControl.second; @@ -1321,8 +1309,6 @@ void IPU3CameraData::imguOutputBufferReady(FrameBuffer *buffer) pipe()->completeBuffer(request, buffer); request->metadata().set(controls::draft::PipelineDepth, 3); - /* \todo Move the ExposureTime control to the IPA. */ - request->metadata().set(controls::ExposureTime, exposureTime_); /* \todo Actually apply the scaler crop region to the ImgU. */ if (request->controls().contains(controls::ScalerCrop)) cropRegion_ = request->controls().get(controls::ScalerCrop); From patchwork Thu Nov 11 11:06:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14569 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D8B4AC3253 for ; Thu, 11 Nov 2021 11:06:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 92B37603CB; Thu, 11 Nov 2021 12:06:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="TGXrdwe1"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4DDB86039C for ; Thu, 11 Nov 2021 12:06:12 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:e627:8337:a781:d98]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id F3691556; Thu, 11 Nov 2021 12:06:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1636628772; bh=YADvt0/tz2UfER+GEmn71Ro5nHLMCnwqGgvOBV2lF8E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TGXrdwe1iPCC/DdN/MrVyBW+AwlYiACxqSBYDif92+Sfg59kWsWhZkyCfiBEOxne8 NMxmd7+g4hctsJAD8aaG/zSOPp1/F5GFUac8ZeZ3WSLy6vNDlY5c9iFdxdLo9yUiJK HKULznt9hsMKanSAZ6zS0dHODBdKjlAL9tHYP800= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Nov 2021 12:06:05 +0100 Message-Id: <20211111110605.105202-15-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> References: <20211111110605.105202-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 14/14] ipa: ipu3: Pass the AnalogueGain control X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We can set the controls::AnalogueGain metadata now that AGC is updating it correctly. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index a91550b9..9c862114 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -629,6 +629,8 @@ void IPAIPU3::parseStatistics(unsigned int frame, int64_t frameDuration = (defVBlank_ + sensorInfo_.outputSize.height) * lineDuration_.get(); ctrls.set(controls::FrameDuration, frameDuration); + ctrls.set(controls::AnalogueGain, context_.frameContext.agc.gain); + ctrls.set(controls::ColourTemperature, context_.frameContext.awb.temperatureK); ctrls.set(controls::ExposureTime, context_.frameContext.sensor.exposure * lineDuration_.get());