From patchwork Wed Oct 19 11:04:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 17635 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 70571BD16B for ; Wed, 19 Oct 2022 11:05:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B155662E59; Wed, 19 Oct 2022 13:05:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1666177503; bh=I0pSobrHpsk5LfES/QvWcPKwDxBmazYuDuK3ZzGhAB8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Ql0aefxxSxEuAbdMgpr5/SLIuzDw3zcf8gn77lsUcVz2L2oZRv12Lmkt2lQipJbx9 9reUjBzimnCHbDJ03Xm69aWG3Vpt1UtKLqiAEl1vrBQ0uA/iNx9u2h+Tib5s/iTIcS sGJUg6Lz3GTf9DJ4XuAubRImlgmKVhlDmouQIf8/Df3a2G5OFQs6kpHbNYwICfh+pw PWmD4hiNdDY+pj73ZtyjzYRaGB3mZxtx0q9imCsuAdDFnX1MBRLyhSgTFBaqJF4x2C aErxYQ5oJhL21zT0TQLFe76tBfRRKNynrO0Yn6ZfTvgG9j8FkeDEYzcizKT2x7UEbR ZMmQaX8drXFYg== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 148A162E45 for ; Wed, 19 Oct 2022 13:05:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="I1V0lrU+"; dkim-atps=neutral Received: from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A38A05A4 for ; Wed, 19 Oct 2022 13:05:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1666177500; bh=I0pSobrHpsk5LfES/QvWcPKwDxBmazYuDuK3ZzGhAB8=; h=From:To:Subject:Date:In-Reply-To:References:From; b=I1V0lrU+D9K6cBzuTMMlnoWVk9OYFHeW8VDJB8JRQAj9N8l5ojMkTe9ZecozftD58 /CGNyAgX1Ucwvf8CWuciwkSkKIz3MAUgjfriaXGATuUm4UytyGj4dQREE0XAhnv5Gk wuhAFhzbKIKt0PycruB8SlBtYQafSgZfJztes1ns= To: libcamera-devel@lists.libcamera.org Date: Wed, 19 Oct 2022 14:04:32 +0300 Message-Id: <20221019110434.17767-3-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019110434.17767-1-laurent.pinchart@ideasonboard.com> References: <20221019110434.17767-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 2/4] ipa: Pass metadata to Algorithm::process() function X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Laurent Pinchart via libcamera-devel From: Laurent Pinchart Reply-To: Laurent Pinchart Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Extend the Algorithm::process() function with a metadata control list, to be filled by individual algorithms with frame metadata. Update the rkisp1 and ipu3 IPA modules accordingly, and drop the dead code in the IPARkISP1::prepareMetadata() function while at it. This only creates the infrastructure, filling metadata in individual algorithms will be handled separately. Signed-off-by: Laurent Pinchart Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/af.cpp | 2 ++ src/ipa/ipu3/algorithms/af.h | 2 +- src/ipa/ipu3/algorithms/agc.cpp | 2 ++ src/ipa/ipu3/algorithms/agc.h | 2 +- src/ipa/ipu3/algorithms/awb.cpp | 1 + src/ipa/ipu3/algorithms/awb.h | 2 +- src/ipa/ipu3/algorithms/tone_mapping.cpp | 2 ++ src/ipa/ipu3/algorithms/tone_mapping.h | 2 +- src/ipa/ipu3/ipu3.cpp | 3 ++- src/ipa/libipa/algorithm.cpp | 5 +++-- src/ipa/libipa/algorithm.h | 1 + src/ipa/rkisp1/algorithms/agc.cpp | 4 +++- src/ipa/rkisp1/algorithms/agc.h | 2 +- src/ipa/rkisp1/algorithms/awb.cpp | 1 + src/ipa/rkisp1/algorithms/awb.h | 2 +- src/ipa/rkisp1/rkisp1.cpp | 17 +++-------------- 16 files changed, 26 insertions(+), 24 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index 75632aa39d21..2d728871a63b 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -408,6 +408,7 @@ bool Af::afIsOutOfFocus(IPAContext &context) * \param[in] context The IPA context * \param[in] frame The frame context sequence number * \param[in] frameContext The current frame context + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * \param[in] stats The statistics buffer of IPU3 * * Ideally, a clear image also has a relatively higher contrast. So, every @@ -423,6 +424,7 @@ bool Af::afIsOutOfFocus(IPAContext &context) */ void Af::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, + [[maybe_unused]] ControlList &metadata, const ipu3_uapi_stats_3a *stats) { /* Evaluate the AF buffer length */ diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h index 89d37ac18615..ea7ca647df12 100644 --- a/src/ipa/ipu3/algorithms/af.h +++ b/src/ipa/ipu3/algorithms/af.h @@ -35,7 +35,7 @@ public: IPAFrameContext &frameContext, ipu3_uapi_params *params) override; void process(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, + IPAFrameContext &frameContext, ControlList &metadata, const ipu3_uapi_stats_3a *stats) override; private: diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index a1a3c38ffe84..44dd5c809e26 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -319,6 +319,7 @@ double Agc::estimateLuminance(IPAActiveState &activeState, * \param[in] context The shared IPA context * \param[in] frame The current frame sequence number * \param[in] frameContext The current frame context + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * \param[in] stats The IPU3 statistics and ISP results * * Identify the current image brightness, and use that to estimate the optimal @@ -326,6 +327,7 @@ double Agc::estimateLuminance(IPAActiveState &activeState, */ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, + [[maybe_unused]] ControlList &metadata, const ipu3_uapi_stats_3a *stats) { /* diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 59b4b9843c2f..9106cbefe87a 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -29,7 +29,7 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void process(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, + IPAFrameContext &frameContext, ControlList &metadata, const ipu3_uapi_stats_3a *stats) override; private: diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 0dbd7d4c374f..836eb0687181 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -389,6 +389,7 @@ void Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats) */ void Awb::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, + [[maybe_unused]] ControlList &metadata, const ipu3_uapi_stats_3a *stats) { calculateWBGains(stats); diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index 28e2d38a711c..c38c425c3654 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -43,7 +43,7 @@ public: IPAFrameContext &frameContext, ipu3_uapi_params *params) override; void process(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, + IPAFrameContext &frameContext, ControlList &metadata, const ipu3_uapi_stats_3a *stats) override; private: diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index eac3d4064443..6fcc03f3fcf7 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -78,6 +78,7 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, * \param[in] context The shared IPA context * \param[in] frame The current frame sequence number * \param[in] frameContext The current frame context + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * \param[in] stats The IPU3 statistics and ISP results * * The tone mapping look up table is generated as an inverse power curve from @@ -85,6 +86,7 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, */ void ToneMapping::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, + [[maybe_unused]] ControlList &metadata, [[maybe_unused]] const ipu3_uapi_stats_3a *stats) { /* diff --git a/src/ipa/ipu3/algorithms/tone_mapping.h b/src/ipa/ipu3/algorithms/tone_mapping.h index 822e5168df82..ab2a1c000f2f 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.h +++ b/src/ipa/ipu3/algorithms/tone_mapping.h @@ -22,7 +22,7 @@ public: void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, ipu3_uapi_params *params) override; void process(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, + IPAFrameContext &frameContext, ControlList &metadata, const ipu3_uapi_stats_3a *stats) override; private: diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 891643e057b8..0ccc6bf5c8af 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -631,9 +631,10 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, double lineDuration = context_.configuration.sensor.lineDuration.get(); int32_t vBlank = context_.configuration.sensor.defVBlank; ControlList ctrls(controls::controls); + ControlList metadata(controls::controls); for (auto const &algo : algorithms()) - algo->process(context_, frame, frameContext, stats); + algo->process(context_, frame, frameContext, metadata, stats); setControls(frame); diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp index 55abddab2b0a..a34c583d947e 100644 --- a/src/ipa/libipa/algorithm.cpp +++ b/src/ipa/libipa/algorithm.cpp @@ -106,12 +106,13 @@ namespace ipa { * \param[in] context The shared IPA context * \param[in] frame The frame context sequence number * \param[in] frameContext The current frame's context + * \param[out] metadata Metadata for the frame, to be filled by the algorithm * \param[in] stats The IPA statistics and ISP results * * This function is called while camera is running for every frame processed by * the ISP, to process statistics generated from that frame by the ISP. - * Algorithms shall use this data to run calculations and update their state - * accordingly. + * Algorithms shall use this data to run calculations, update their state + * accordingly, and fill the frame metadata. * * Processing shall not take an undue amount of time, and any extended or * computationally expensive calculations or operations must be handled diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h index d8601f9ccaff..793247be8129 100644 --- a/src/ipa/libipa/algorithm.h +++ b/src/ipa/libipa/algorithm.h @@ -55,6 +55,7 @@ public: virtual void process([[maybe_unused]] typename Module::Context &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] typename Module::FrameContext &frameContext, + [[maybe_unused]] ControlList &metadata, [[maybe_unused]] const typename Module::Stats *stats) { } diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 04062a364e7f..9de11a2a3bca 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -288,7 +288,9 @@ double Agc::measureBrightness(const rkisp1_cif_isp_hist_stat *hist) const * new exposure and gain for the scene. */ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, - IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats) + IPAFrameContext &frameContext, + [[maybe_unused]] ControlList &metadata, + const rkisp1_stat_buffer *stats) { /* * \todo Verify that the exposure and gain applied by the sensor for diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index 9ad5c32fd6f6..99af09167331 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -30,7 +30,7 @@ public: IPAFrameContext &frameContext, rkisp1_params_cfg *params) override; void process(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, + IPAFrameContext &frameContext, ControlList &metadata, const rkisp1_stat_buffer *stats) override; private: diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index 3349948a3acf..9b97db7d5c5a 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -208,6 +208,7 @@ void Awb::queueRequest(IPAContext &context, void Awb::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, + [[maybe_unused]] ControlList &metadata, const rkisp1_stat_buffer *stats) { const rkisp1_cif_isp_stat *params = &stats->params; diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h index d76b538288ec..3659b2d5405f 100644 --- a/src/ipa/rkisp1/algorithms/awb.h +++ b/src/ipa/rkisp1/algorithms/awb.h @@ -27,7 +27,7 @@ public: IPAFrameContext &frameContext, const ControlList &controls) override; void process(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameCtx, + IPAFrameContext &frameContext, ControlList &metadata, const rkisp1_stat_buffer *stats) override; private: diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 3f5c1a58695c..bef5211d5c34 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -69,7 +69,6 @@ protected: private: void setControls(unsigned int frame); - void prepareMetadata(unsigned int frame, unsigned int aeState); std::map buffers_; std::map mappedBuffers_; @@ -338,14 +337,14 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); - unsigned int aeState = 0; + ControlList metadata(controls::controls); for (auto const &algo : algorithms()) - algo->process(context_, frame, frameContext, stats); + algo->process(context_, frame, frameContext, metadata, stats); setControls(frame); - prepareMetadata(frame, aeState); + metadataReady.emit(frame, metadata); } void IPARkISP1::setControls(unsigned int frame) @@ -366,16 +365,6 @@ void IPARkISP1::setControls(unsigned int frame) setSensorControls.emit(frame, ctrls); } -void IPARkISP1::prepareMetadata(unsigned int frame, unsigned int aeState) -{ - ControlList ctrls(controls::controls); - - if (aeState) - ctrls.set(controls::AeLocked, aeState == 2); - - metadataReady.emit(frame, ctrls); -} - } /* namespace ipa::rkisp1 */ /*