From patchwork Wed Oct 19 20:56:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 17641 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 68414C327C for ; Wed, 19 Oct 2022 20:56:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8A21A62E65; Wed, 19 Oct 2022 22:56:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1666213001; bh=prmlCGxeEhrPEMzwd6NcmlMsXzpOxQ5G6eErNmWO2S8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=dQvtQK/M45GPOOgvTJkl1k1YljBjmgK+cXjKhrFitr6ysU3NIoD7qkjxMc73jrgxK Va0iB3cMR/b84p5iOCs0BivGBBxYxCLqIFpc0ik9i5VbmNg2BC2dKqv6TiYj3KPycu 0zAeWT8Dhcc47V0aF8sNJ4CrxhCQLlysaYQI223G5UEMf/yg+oXk692Bmm9Inc05Np 2bis3WKBhv4M35Wiw5EP9d4gcl31dlIc0IC+g3d3FYujZjw19HTJPBttYQxY4luA97 OkqWygeorbqDhBU7Xk27Ztp388JftmxLYWtfV3VJmFg/1jJhqt1fPOGDbJgtMVVO5b Y+p6gg7bWM2/g== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3E3D362DFA for ; Wed, 19 Oct 2022 22:56:40 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Hm3jMd1f"; dkim-atps=neutral Received: from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C366D570 for ; Wed, 19 Oct 2022 22:56:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1666213000; bh=prmlCGxeEhrPEMzwd6NcmlMsXzpOxQ5G6eErNmWO2S8=; h=From:To:Subject:Date:In-Reply-To:References:From; b=Hm3jMd1fjICspX96Ul0KhfVvE19k+MiSPX1x5QD8z2T8tubCPfsMmm64MukyRFSAK 6qn5mFAa+doldXf4MX9zp6GYKW0jomcDrhfRoI+RRdIGaBCWfYdXUenozoBU30DLuH Du265fVeIX5fLzBm5ANiqIAgUUj10IuGM7Yo5/kg= To: libcamera-devel@lists.libcamera.org Date: Wed, 19 Oct 2022 23:56:12 +0300 Message-Id: <20221019205614.25751-2-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019205614.25751-1-laurent.pinchart@ideasonboard.com> References: <20221019205614.25751-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/3] ipa: Pass metadata to Algorithm::process() function X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Laurent Pinchart via libcamera-devel From: Laurent Pinchart Reply-To: Laurent Pinchart Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Extend the Algorithm::process() function with a metadata control list, to be filled by individual algorithms with frame metadata. Update the rkisp1 and ipu3 IPA modules accordingly, and drop the dead code in the IPARkISP1::prepareMetadata() function while at it. This only creates the infrastructure, filling metadata in individual algorithms will be handled separately. Signed-off-by: Laurent Pinchart Reviewed-by: Kieran Bingham Reviewed-by: Jacopo Mondi --- Changes since v1: - Pass metadata as last argument to process() - Add missing documentation update in src/ipa/rkisp1/algorithms/agc.cpp - Correctly send metadata to pipeline handler in IPU3 IPA module --- src/ipa/ipu3/algorithms/af.cpp | 4 +++- src/ipa/ipu3/algorithms/af.h | 3 ++- src/ipa/ipu3/algorithms/agc.cpp | 4 +++- src/ipa/ipu3/algorithms/agc.h | 3 ++- src/ipa/ipu3/algorithms/awb.cpp | 3 ++- src/ipa/ipu3/algorithms/awb.h | 3 ++- src/ipa/ipu3/algorithms/tone_mapping.cpp | 4 +++- src/ipa/ipu3/algorithms/tone_mapping.h | 3 ++- src/ipa/ipu3/ipu3.cpp | 14 +++++++------- src/ipa/libipa/algorithm.cpp | 5 +++-- src/ipa/libipa/algorithm.h | 3 ++- src/ipa/rkisp1/algorithms/agc.cpp | 4 +++- src/ipa/rkisp1/algorithms/agc.h | 3 ++- src/ipa/rkisp1/algorithms/awb.cpp | 3 ++- src/ipa/rkisp1/algorithms/awb.h | 5 +++-- src/ipa/rkisp1/rkisp1.cpp | 17 +++-------------- 16 files changed, 44 insertions(+), 37 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index 75632aa39d21..5a0452a5719b 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -409,6 +409,7 @@ bool Af::afIsOutOfFocus(IPAContext &context) * \param[in] frame The frame context sequence number * \param[in] frameContext The current frame context * \param[in] stats The statistics buffer of IPU3 + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * * Ideally, a clear image also has a relatively higher contrast. So, every * image for each focus step should be tested to find an optimal focus step. @@ -423,7 +424,8 @@ bool Af::afIsOutOfFocus(IPAContext &context) */ void Af::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - const ipu3_uapi_stats_3a *stats) + const ipu3_uapi_stats_3a *stats, + [[maybe_unused]] ControlList &metadata) { /* Evaluate the AF buffer length */ uint32_t afRawBufferLen = context.configuration.af.afGrid.width * diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h index 89d37ac18615..c6168e30fb2e 100644 --- a/src/ipa/ipu3/algorithms/af.h +++ b/src/ipa/ipu3/algorithms/af.h @@ -36,7 +36,8 @@ public: ipu3_uapi_params *params) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - const ipu3_uapi_stats_3a *stats) override; + const ipu3_uapi_stats_3a *stats, + ControlList &metadata) override; private: void afCoarseScan(IPAContext &context); diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index a1a3c38ffe84..f4e559bf8b84 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -320,13 +320,15 @@ double Agc::estimateLuminance(IPAActiveState &activeState, * \param[in] frame The current frame sequence number * \param[in] frameContext The current frame context * \param[in] stats The IPU3 statistics and ISP results + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * * Identify the current image brightness, and use that to estimate the optimal * new exposure and gain for the scene. */ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - const ipu3_uapi_stats_3a *stats) + const ipu3_uapi_stats_3a *stats, + [[maybe_unused]] ControlList &metadata) { /* * Estimate the gain needed to have the proportion of pixels in a given diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 59b4b9843c2f..9d6e3ff1a687 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -30,7 +30,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - const ipu3_uapi_stats_3a *stats) override; + const ipu3_uapi_stats_3a *stats, + ControlList &metadata) override; private: double measureBrightness(const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 0dbd7d4c374f..6452b6a1acd2 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -389,7 +389,8 @@ void Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats) */ void Awb::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - const ipu3_uapi_stats_3a *stats) + const ipu3_uapi_stats_3a *stats, + [[maybe_unused]] ControlList &metadata) { calculateWBGains(stats); diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index 28e2d38a711c..7a70854e8999 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -44,7 +44,8 @@ public: ipu3_uapi_params *params) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - const ipu3_uapi_stats_3a *stats) override; + const ipu3_uapi_stats_3a *stats, + ControlList &metadata) override; private: /* \todo Make these structs available to all the ISPs ? */ diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index eac3d4064443..a169894cb628 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -79,13 +79,15 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, * \param[in] frame The current frame sequence number * \param[in] frameContext The current frame context * \param[in] stats The IPU3 statistics and ISP results + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * * The tone mapping look up table is generated as an inverse power curve from * our gamma setting. */ void ToneMapping::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - [[maybe_unused]] const ipu3_uapi_stats_3a *stats) + [[maybe_unused]] const ipu3_uapi_stats_3a *stats, + [[maybe_unused]] ControlList &metadata) { /* * Hardcode gamma to 1.1 as a default for now. diff --git a/src/ipa/ipu3/algorithms/tone_mapping.h b/src/ipa/ipu3/algorithms/tone_mapping.h index 822e5168df82..5ae35da55e87 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.h +++ b/src/ipa/ipu3/algorithms/tone_mapping.h @@ -23,7 +23,8 @@ public: IPAFrameContext &frameContext, ipu3_uapi_params *params) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - const ipu3_uapi_stats_3a *stats) override; + const ipu3_uapi_stats_3a *stats, + ControlList &metadata) override; private: double gamma_; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 891643e057b8..bc0f6007baca 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -630,22 +630,22 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, double lineDuration = context_.configuration.sensor.lineDuration.get(); int32_t vBlank = context_.configuration.sensor.defVBlank; - ControlList ctrls(controls::controls); + ControlList metadata(controls::controls); for (auto const &algo : algorithms()) - algo->process(context_, frame, frameContext, stats); + algo->process(context_, frame, frameContext, stats, metadata); setControls(frame); /* \todo Use VBlank value calculated from each frame exposure. */ int64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration; - ctrls.set(controls::FrameDuration, frameDuration); + metadata.set(controls::FrameDuration, frameDuration); - ctrls.set(controls::AnalogueGain, frameContext.sensor.gain); + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); - ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); + metadata.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); - ctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); + metadata.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); /* * \todo The Metadata provides a path to getting extended data @@ -655,7 +655,7 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, * likely want to avoid putting platform specific metadata in. */ - metadataReady.emit(frame, ctrls); + metadataReady.emit(frame, metadata); } /** diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp index 55abddab2b0a..71f583840a5a 100644 --- a/src/ipa/libipa/algorithm.cpp +++ b/src/ipa/libipa/algorithm.cpp @@ -107,11 +107,12 @@ namespace ipa { * \param[in] frame The frame context sequence number * \param[in] frameContext The current frame's context * \param[in] stats The IPA statistics and ISP results + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * * This function is called while camera is running for every frame processed by * the ISP, to process statistics generated from that frame by the ISP. - * Algorithms shall use this data to run calculations and update their state - * accordingly. + * Algorithms shall use this data to run calculations, update their state + * accordingly, and fill the frame metadata. * * Processing shall not take an undue amount of time, and any extended or * computationally expensive calculations or operations must be handled diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h index d8601f9ccaff..38b3231c5d0e 100644 --- a/src/ipa/libipa/algorithm.h +++ b/src/ipa/libipa/algorithm.h @@ -55,7 +55,8 @@ public: virtual void process([[maybe_unused]] typename Module::Context &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] typename Module::FrameContext &frameContext, - [[maybe_unused]] const typename Module::Stats *stats) + [[maybe_unused]] const typename Module::Stats *stats, + [[maybe_unused]] ControlList &metadata) { } }; diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 04062a364e7f..37cd10f9b302 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -283,12 +283,14 @@ double Agc::measureBrightness(const rkisp1_cif_isp_hist_stat *hist) const * \param[in] frame The frame context sequence number * \param[in] frameContext The current frame context * \param[in] stats The RKISP1 statistics and ISP results + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * * Identify the current image brightness, and use that to estimate the optimal * new exposure and gain for the scene. */ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, - IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats) + IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, + [[maybe_unused]] ControlList &metadata) { /* * \todo Verify that the exposure and gain applied by the sensor for diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index 9ad5c32fd6f6..da4d2d4e8359 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -31,7 +31,8 @@ public: rkisp1_params_cfg *params) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - const rkisp1_stat_buffer *stats) override; + const rkisp1_stat_buffer *stats, + ControlList &metadata) override; private: void computeExposure(IPAContext &Context, IPAFrameContext &frameContext, diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index 3349948a3acf..f597ded800c2 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -208,7 +208,8 @@ void Awb::queueRequest(IPAContext &context, void Awb::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - const rkisp1_stat_buffer *stats) + const rkisp1_stat_buffer *stats, + [[maybe_unused]] ControlList &metadata) { const rkisp1_cif_isp_stat *params = &stats->params; const rkisp1_cif_isp_awb_stat *awb = ¶ms->awb; diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h index d76b538288ec..f5633b1171a0 100644 --- a/src/ipa/rkisp1/algorithms/awb.h +++ b/src/ipa/rkisp1/algorithms/awb.h @@ -27,8 +27,9 @@ public: IPAFrameContext &frameContext, const ControlList &controls) override; void process(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameCtx, - const rkisp1_stat_buffer *stats) override; + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) override; private: uint32_t estimateCCT(double red, double green, double blue); diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index ba3c547e02f6..494d8c891509 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -69,7 +69,6 @@ protected: private: void setControls(unsigned int frame); - void prepareMetadata(unsigned int frame, unsigned int aeState); std::map buffers_; std::map mappedBuffers_; @@ -338,14 +337,14 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); - unsigned int aeState = 0; + ControlList metadata(controls::controls); for (auto const &algo : algorithms()) - algo->process(context_, frame, frameContext, stats); + algo->process(context_, frame, frameContext, stats, metadata); setControls(frame); - prepareMetadata(frame, aeState); + metadataReady.emit(frame, metadata); } void IPARkISP1::setControls(unsigned int frame) @@ -366,16 +365,6 @@ void IPARkISP1::setControls(unsigned int frame) setSensorControls.emit(frame, ctrls); } -void IPARkISP1::prepareMetadata(unsigned int frame, unsigned int aeState) -{ - ControlList ctrls(controls::controls); - - if (aeState) - ctrls.set(controls::AeLocked, aeState == 2); - - metadataReady.emit(frame, ctrls); -} - } /* namespace ipa::rkisp1 */ /* From patchwork Wed Oct 19 20:56:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 17642 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4C7BDC0DA4 for ; Wed, 19 Oct 2022 20:56:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0D3D862E70; Wed, 19 Oct 2022 22:56:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1666213005; bh=T69Fg7V5bOCmQmWqtUUO8FPim6u00zY40ZESKAS4hAc=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=RZ2EubhaUpobuDf4q3r7uRHtJrwVSQDeVfpV7WyF0LutCOXucBFQka23W11yjwWyk Bb8Akc4wXO9Uq14aThE8e0bqgVj0Fqr8Ng47Q1IkqoVzRaMvHI78aV6hAjN7wFAgM/ lA8H2spOS+sCYBvGkSFBvXr22OR6rlvQAxZ3Imcbn+S/t4hClzDGY/MVgw2ON/LK19 XpXAuFE5nzsDCPR1V/v5XZ3HemmDyE5ckGTB+tOmRXdYfW+8i6mIh84tr1EPcLl/zc soZI/KpuK7M12OYLT0otM4YD+dS0BD7l0Y63er6hZXkiGM5DF7HsySQtimJPA1dzRD 4zrwwgB+thsog== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9B6C662E6B for ; Wed, 19 Oct 2022 22:56:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="cd9kKHgJ"; dkim-atps=neutral Received: from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 131F2570 for ; Wed, 19 Oct 2022 22:56:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1666213001; bh=T69Fg7V5bOCmQmWqtUUO8FPim6u00zY40ZESKAS4hAc=; h=From:To:Subject:Date:In-Reply-To:References:From; b=cd9kKHgJghXYXyhPZ7TjEuRIT5rCO64i5aYBIPVxsGGSlG1weUK/OkL6jZ83NfaJQ VoGGcx85bcQ/GvHmGm1+LRlwHksT9Pc1gPs2J6csZSD1ulrdSYY+dKNVaOfO+U54Dv pDsS5IgzSJluUkb6IibdMmTsJvyACcfDD0MBKLtk= To: libcamera-devel@lists.libcamera.org Date: Wed, 19 Oct 2022 23:56:13 +0300 Message-Id: <20221019205614.25751-3-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019205614.25751-1-laurent.pinchart@ideasonboard.com> References: <20221019205614.25751-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/3] ipa: rkisp1: Fill AGC and AWB metadata X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Laurent Pinchart via libcamera-devel From: Laurent Pinchart Reply-To: Laurent Pinchart Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Fill the frame metadata in the AGC and AWB algorithm's prepare() function. Additional metadata for other algorithms will be addressed later. Signed-off-by: Laurent Pinchart Reviewed-by: Kieran Bingham Reviewed-by: Jacopo Mondi Reviewed-by: Paul Elder --- Changes since v1: - Report the ColourTemperature and FrameDuration controls as well --- src/ipa/rkisp1/algorithms/agc.cpp | 15 ++++++++++++++- src/ipa/rkisp1/algorithms/awb.cpp | 9 ++++++++- src/ipa/rkisp1/ipa_context.cpp | 3 +++ src/ipa/rkisp1/ipa_context.h | 1 + src/ipa/rkisp1/rkisp1.cpp | 2 ++ 5 files changed, 28 insertions(+), 2 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 37cd10f9b302..b4fc7aed4d9d 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -14,6 +14,7 @@ #include #include +#include #include #include "libipa/histogram.h" @@ -290,7 +291,7 @@ double Agc::measureBrightness(const rkisp1_cif_isp_hist_stat *hist) const */ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, - [[maybe_unused]] ControlList &metadata) + ControlList &metadata) { /* * \todo Verify that the exposure and gain applied by the sensor for @@ -333,6 +334,18 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); + metadata.set(controls::ExposureTime, exposureTime.get()); + + /* \todo Use VBlank value calculated from each frame exposure. */ + uint32_t vTotal = context.configuration.sensor.size.height + + context.configuration.sensor.defVBlank; + utils::Duration frameDuration = context.configuration.sensor.lineDuration + * vTotal; + metadata.set(controls::FrameDuration, frameDuration.get()); } /** diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index f597ded800c2..083c2d982c52 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -209,7 +209,7 @@ void Awb::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, - [[maybe_unused]] ControlList &metadata) + ControlList &metadata) { const rkisp1_cif_isp_stat *params = &stats->params; const rkisp1_cif_isp_awb_stat *awb = ¶ms->awb; @@ -307,6 +307,13 @@ void Awb::process(IPAContext &context, frameContext.awb.temperatureK = activeState.awb.temperatureK; + metadata.set(controls::AwbEnable, frameContext.awb.autoEnabled); + metadata.set(controls::ColourGains, { + static_cast(frameContext.awb.gains.red), + static_cast(frameContext.awb.gains.blue) + }); + metadata.set(controls::ColourTemperature, frameContext.awb.temperatureK); + LOG(RkISP1Awb, Debug) << std::showpoint << "Means [" << redMean << ", " << greenMean << ", " << blueMean << "], gains [" << activeState.awb.gains.automatic.red << ", " diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp index b00dc29c1713..3c14cf3476ce 100644 --- a/src/ipa/rkisp1/ipa_context.cpp +++ b/src/ipa/rkisp1/ipa_context.cpp @@ -77,6 +77,9 @@ namespace libcamera::ipa::rkisp1 { * \var IPASessionConfiguration::sensor * \brief Sensor-specific configuration of the IPA * + * \var IPASessionConfiguration::sensor.defVBlank + * \brief The default vblank value of the sensor + * * \var IPASessionConfiguration::sensor.lineDuration * \brief Line duration in microseconds * diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index c85d8abeca71..60e8a7e11196 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -39,6 +39,7 @@ struct IPASessionConfiguration { } lsc; struct { + int32_t defVBlank; utils::Duration lineDuration; Size size; } sensor; diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 494d8c891509..069c901bb141 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -244,6 +244,8 @@ int IPARkISP1::configure([[maybe_unused]] const IPACameraSensorInfo &info, /* Set the hardware revision for the algorithms. */ context_.configuration.hw.revision = hwRevision_; + const ControlInfo vBlank = ctrls_.find(V4L2_CID_VBLANK)->second; + context_.configuration.sensor.defVBlank = vBlank.def().get(); context_.configuration.sensor.size = info.outputSize; context_.configuration.sensor.lineDuration = info.minLineLength * 1.0s / info.pixelRate; From patchwork Wed Oct 19 20:56:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 17643 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5B586C327C for ; Wed, 19 Oct 2022 20:56:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 071D762E69; Wed, 19 Oct 2022 22:56:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1666213006; bh=outIGZNDFhfw+v/f0Xrt5Zk6YNJRbV269r4O/NHohC4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Ev7smDNZkK0Nof8ATP/4N6lf58A5d9/gOqYTAU8bvPokPaZWA4dKxiLZr+TsHjZHg XWeUON82MWinVhQh9W5CMfqy3RSxykZklK4V95LhnyJ6+nGKXI19Ra1dllU5m5Ma7Z DF8i3+Djd90fH67PSCdzLwJNCMZKcoYlR1GO2UDTdoxlIZXpHcoXmBbtnTUnH51oyj pGSD0qJk36LJrBLVMXPpkdBh1v8txR4vPJsfo5xlu9DQ1M4OFVScQp5XmbUsyGv0Bx jQoIHUuv60OzRCt2y5fl5TN1vJ/q5S/fEtHFR96Kl5hRoQayIJc7dZyv9wBRJ52djK TOWzUEVb8stDw== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CFDF362E61 for ; Wed, 19 Oct 2022 22:56:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="YpHCkjFM"; dkim-atps=neutral Received: from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5FE13903 for ; Wed, 19 Oct 2022 22:56:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1666213002; bh=outIGZNDFhfw+v/f0Xrt5Zk6YNJRbV269r4O/NHohC4=; h=From:To:Subject:Date:In-Reply-To:References:From; b=YpHCkjFMa1e1JGIC2YxAbH0UswVSRS6y+X8efXZEWhDoRM3JuN8sQ0u0DypvQI4+5 cIs2Ibl3kUwL/KgIABbwne7FAL9O+QoFqkBSNR+msuX93QoBc9zAF3fiaXo+1fGl6w B7YgRCFx+JUa+fmPxWI4czkNs/6jp33EL8bDlMRY= To: libcamera-devel@lists.libcamera.org Date: Wed, 19 Oct 2022 23:56:14 +0300 Message-Id: <20221019205614.25751-4-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019205614.25751-1-laurent.pinchart@ideasonboard.com> References: <20221019205614.25751-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 3/3] ipa: ipu3: Fill AGC and AWB metadata in algorithms X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Laurent Pinchart via libcamera-devel From: Laurent Pinchart Reply-To: Laurent Pinchart Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Fill the frame metadata in the AGC and AWB algorithm's prepare() function. This removes the need to fill metadata manually in the IPA module's processStatsBuffer() function. Signed-off-by: Laurent Pinchart Reviewed-by: Jacopo Mondi Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 16 +++++++++++++++- src/ipa/ipu3/algorithms/awb.cpp | 10 ++++++++++ src/ipa/ipu3/ipa_context.cpp | 3 +++ src/ipa/ipu3/ipa_context.h | 1 + src/ipa/ipu3/ipu3.cpp | 13 +------------ 5 files changed, 30 insertions(+), 13 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index f4e559bf8b84..b5309bdbea25 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -14,6 +14,7 @@ #include #include +#include #include #include "libipa/histogram.h" @@ -328,7 +329,7 @@ double Agc::estimateLuminance(IPAActiveState &activeState, void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, const ipu3_uapi_stats_3a *stats, - [[maybe_unused]] ControlList &metadata) + ControlList &metadata) { /* * Estimate the gain needed to have the proportion of pixels in a given @@ -365,6 +366,19 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); + metadata.set(controls::ExposureTime, exposureTime.get()); + + /* \todo Use VBlank value calculated from each frame exposure. */ + uint32_t vTotal = context.configuration.sensor.size.height + + context.configuration.sensor.defVBlank; + utils::Duration frameDuration = context.configuration.sensor.lineDuration + * vTotal; + metadata.set(controls::FrameDuration, frameDuration.get()); + } REGISTER_IPA_ALGORITHM(Agc, "Agc") diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 6452b6a1acd2..dd7ebc07c534 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -11,6 +11,8 @@ #include +#include + /** * \file awb.h */ @@ -403,6 +405,14 @@ void Awb::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, context.activeState.awb.gains.green = asyncResults_.greenGain; context.activeState.awb.gains.red = asyncResults_.redGain; context.activeState.awb.temperatureK = asyncResults_.temperatureK; + + metadata.set(controls::AwbEnable, true); + metadata.set(controls::ColourGains, { + static_cast(context.activeState.awb.gains.red), + static_cast(context.activeState.awb.gains.blue) + }); + metadata.set(controls::ColourTemperature, + context.activeState.awb.temperatureK); } constexpr uint16_t Awb::threshold(float value) diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 68f017b04751..959f314f5452 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -111,6 +111,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPASessionConfiguration::sensor.defVBlank * \brief The default vblank value of the sensor + * + * \var IPASessionConfiguration::sensor.size + * \brief Sensor output resolution */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 36099353e9f2..e9a3863b1693 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -41,6 +41,7 @@ struct IPASessionConfiguration { struct { int32_t defVBlank; utils::Duration lineDuration; + Size size; } sensor; }; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index bc0f6007baca..a9a2b49ca95b 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -502,6 +502,7 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, /* Initialise the sensor configuration. */ context_.configuration.sensor.lineDuration = sensorInfo_.minLineLength * 1.0s / sensorInfo_.pixelRate; + context_.configuration.sensor.size = sensorInfo_.outputSize; /* * Compute the sensor V4L2 controls to be used by the algorithms and @@ -628,8 +629,6 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, frameContext.sensor.exposure = sensorControls.get(V4L2_CID_EXPOSURE).get(); frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); - double lineDuration = context_.configuration.sensor.lineDuration.get(); - int32_t vBlank = context_.configuration.sensor.defVBlank; ControlList metadata(controls::controls); for (auto const &algo : algorithms()) @@ -637,16 +636,6 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, setControls(frame); - /* \todo Use VBlank value calculated from each frame exposure. */ - int64_t frameDuration = (vBlank + sensorInfo_.outputSize.height) * lineDuration; - metadata.set(controls::FrameDuration, frameDuration); - - metadata.set(controls::AnalogueGain, frameContext.sensor.gain); - - metadata.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); - - metadata.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); - /* * \todo The Metadata provides a path to getting extended data * out to the application. Further data such as a simplifed Histogram