From patchwork Fri Jul 16 14:32:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13030 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1B462C3229 for ; Fri, 16 Jul 2021 14:31:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AFAE76853A; Fri, 16 Jul 2021 16:31:34 +0200 (CEST) Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 820F668536 for ; Fri, 16 Jul 2021 16:31:32 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 0F58A24000B; Fri, 16 Jul 2021 14:31:31 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 16 Jul 2021 16:32:14 +0200 Message-Id: <20210716143215.67454-2-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210716143215.67454-1-jacopo@jmondi.org> References: <20210716143215.67454-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/2] libcamera: ipu3: Initialize controls in the IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" All the IPU3 Camera controls are currently initialized by the pipeline handler which initializes them using the camera sensor configuration and platform specific requirements. However, some controls are better initialized by the IPA, which might, in example, cap the exposure times and frame duration to the constraints of its algorithms implementation. Also, moving forward, the IPA should register controls to report its capabilities, in example the ability to enable/disable 3A algorithms on request. Move the existing controls initialization to the IPA, by providing the sensor configuration and its controls to the IPU3IPA::init() function, which initializes controls and returns them to the pipeline through an output parameter. The existing controls initialization has been copied verbatim from the pipeline handler to the IPA, if not a for few line breaks adjustments and the resulting Camera controls values are not changed . Signed-off-by: Jacopo Mondi Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart --- include/libcamera/ipa/ipu3.mojom | 8 ++- src/ipa/ipu3/ipu3.cpp | 71 ++++++++++++++++++-- src/libcamera/pipeline/ipu3/ipu3.cpp | 96 ++++++++++++---------------- 3 files changed, 116 insertions(+), 59 deletions(-) diff --git a/include/libcamera/ipa/ipu3.mojom b/include/libcamera/ipa/ipu3.mojom index 911a3a072464..eafefa8b7fe3 100644 --- a/include/libcamera/ipa/ipu3.mojom +++ b/include/libcamera/ipa/ipu3.mojom @@ -30,6 +30,11 @@ struct IPU3Action { libcamera.ControlList controls; }; +struct IPAInitInfo { + libcamera.IPACameraSensorInfo sensorInfo; + libcamera.ControlInfoMap sensorControls; +}; + struct IPAConfigInfo { libcamera.IPACameraSensorInfo sensorInfo; map entityControls; @@ -38,7 +43,8 @@ struct IPAConfigInfo { }; interface IPAIPU3Interface { - init(libcamera.IPASettings settings) => (int32 ret); + init(IPAInitInfo initInfo) + => (int32 ret, libcamera.ControlInfoMap ipaControls); start() => (int32 ret); stop(); diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 71698d36e50f..d3c69bc07bd0 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -5,8 +5,10 @@ * ipu3.cpp - IPU3 Image Processing Algorithms */ +#include #include #include +#include #include #include @@ -38,7 +40,8 @@ namespace ipa::ipu3 { class IPAIPU3 : public IPAIPU3Interface { public: - int init(const IPASettings &settings) override; + int init(const IPAInitInfo &initInfo, ControlInfoMap *ipaControls) override; + int start() override; void stop() override {} @@ -86,14 +89,74 @@ private: struct ipu3_uapi_grid_config bdsGrid_; }; -int IPAIPU3::init(const IPASettings &settings) +/** + * Initialize the IPA module and its controls. + * + * This function receives the camera sensor information from the pipeline + * handler, computes the limits of the controls it handles and returns + * them in the \a ipaControls output parameter. + */ +int IPAIPU3::init(const IPAInitInfo &initInfo, ControlInfoMap *ipaControls) { - camHelper_ = CameraSensorHelperFactory::create(settings.sensorModel); + const IPACameraSensorInfo &sensorInfo = initInfo.sensorInfo; + + /* Initialize the camera sensor helper. */ + camHelper_ = CameraSensorHelperFactory::create(sensorInfo.model); if (camHelper_ == nullptr) { - LOG(IPAIPU3, Error) << "Failed to create camera sensor helper for " << settings.sensorModel; + LOG(IPAIPU3, Error) << "Failed to create camera sensor helper for " + << sensorInfo.model; return -ENODEV; } + /* Initialize Controls. */ + const ControlInfoMap &sensorControls = initInfo.sensorControls; + ControlInfoMap::Map controls{}; + + /* + * Compute exposure time limits. + * + * Initialize the control using the line length and pixel rate of the + * current configuration converted to microseconds. Use the + * V4L2_CID_EXPOSURE control to get exposure min, max and default and + * convert it from lines to microseconds. + */ + double lineDuration = sensorInfo.lineLength / (sensorInfo.pixelRate / 1e6); + const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; + int32_t minExposure = v4l2Exposure.min().get() * lineDuration; + int32_t maxExposure = v4l2Exposure.max().get() * lineDuration; + int32_t defExposure = v4l2Exposure.def().get() * lineDuration; + controls[&controls::ExposureTime] = ControlInfo(minExposure, maxExposure, + defExposure); + + /* + * Compute the frame duration limits. + * + * The frame length is computed assuming a fixed line length combined + * with the vertical frame sizes. + */ + const ControlInfo &v4l2HBlank = sensorControls.find(V4L2_CID_HBLANK)->second; + uint32_t hblank = v4l2HBlank.def().get(); + uint32_t lineLength = sensorInfo.outputSize.width + hblank; + + const ControlInfo &v4l2VBlank = sensorControls.find(V4L2_CID_VBLANK)->second; + std::array frameHeights{ + v4l2VBlank.min().get() + sensorInfo.outputSize.height, + v4l2VBlank.max().get() + sensorInfo.outputSize.height, + v4l2VBlank.def().get() + sensorInfo.outputSize.height, + }; + + std::array frameDurations; + for (unsigned int i = 0; i < frameHeights.size(); ++i) { + uint64_t frameSize = lineLength * frameHeights[i]; + frameDurations[i] = frameSize / (sensorInfo.pixelRate / 1000000U); + } + + controls[&controls::FrameDurationLimits] = ControlInfo(frameDurations[0], + frameDurations[1], + frameDurations[2]); + + *ipaControls = std::move(controls); + return 0; } diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 76c3bb3d8aa9..22df9c3650af 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -88,6 +88,8 @@ public: std::queue pendingRequests_; + ControlInfoMap ipaControls_; + private: void queueFrameAction(unsigned int id, const ipa::ipu3::IPU3Action &action); @@ -940,7 +942,6 @@ int PipelineHandlerIPU3::initControls(IPU3CameraData *data) return ret; ControlInfoMap::Map controls = IPU3Controls; - const ControlInfoMap &sensorControls = sensor->controls(); const std::vector &testPatternModes = sensor->testPatternModes(); if (!testPatternModes.empty()) { std::vector values; @@ -952,58 +953,6 @@ int PipelineHandlerIPU3::initControls(IPU3CameraData *data) controls[&controls::draft::TestPatternMode] = ControlInfo(values); } - /* - * Compute exposure time limits. - * - * Initialize the control using the line length and pixel rate of the - * current configuration converted to microseconds. Use the - * V4L2_CID_EXPOSURE control to get exposure min, max and default and - * convert it from lines to microseconds. - */ - double lineDuration = sensorInfo.lineLength - / (sensorInfo.pixelRate / 1e6); - const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; - int32_t minExposure = v4l2Exposure.min().get() * lineDuration; - int32_t maxExposure = v4l2Exposure.max().get() * lineDuration; - int32_t defExposure = v4l2Exposure.def().get() * lineDuration; - - /* - * \todo Report the actual exposure time, use the default for the - * moment. - */ - data->exposureTime_ = defExposure; - - controls[&controls::ExposureTime] = ControlInfo(minExposure, maxExposure, - defExposure); - - /* - * Compute the frame duration limits. - * - * The frame length is computed assuming a fixed line length combined - * with the vertical frame sizes. - */ - const ControlInfo &v4l2HBlank = sensorControls.find(V4L2_CID_HBLANK)->second; - uint32_t hblank = v4l2HBlank.def().get(); - uint32_t lineLength = sensorInfo.outputSize.width + hblank; - - const ControlInfo &v4l2VBlank = sensorControls.find(V4L2_CID_VBLANK)->second; - std::array frameHeights{ - v4l2VBlank.min().get() + sensorInfo.outputSize.height, - v4l2VBlank.max().get() + sensorInfo.outputSize.height, - v4l2VBlank.def().get() + sensorInfo.outputSize.height, - }; - - std::array frameDurations; - for (unsigned int i = 0; i < frameHeights.size(); ++i) { - uint64_t frameSize = lineLength * frameHeights[i]; - frameDurations[i] = frameSize / (sensorInfo.pixelRate / 1000000U); - } - - controls[&controls::FrameDurationLimits] = - ControlInfo(frameDurations[0], - frameDurations[1], - frameDurations[2]); - /* * Compute the scaler crop limits. * @@ -1057,6 +1006,10 @@ int PipelineHandlerIPU3::initControls(IPU3CameraData *data) controls[&controls::ScalerCrop] = ControlInfo(minCrop, maxCrop, maxCrop); + /* Add the IPA registered controls to list of camera controls. */ + for (const auto &ipaControl : data->ipaControls_) + controls[ipaControl.first] = ipaControl.second; + data->controlInfo_ = std::move(controls); return 0; @@ -1208,13 +1161,48 @@ int IPU3CameraData::loadIPA() ipa_->queueFrameAction.connect(this, &IPU3CameraData::queueFrameAction); + /* + * Pass the sensor info the the IPA to initialize controls. + * + * \todo The limits of the registered controls depend on the current + * sensor configuration. Initialize the sensor using its resolution as + * its initial configuration and use it to compute the controls limits + * in the IPA. + */ CameraSensor *sensor = cio2_.sensor(); - int ret = ipa_->init(IPASettings{ "", sensor->model() }); + V4L2SubdeviceFormat sensorFormat = {}; + sensorFormat.size = sensor->resolution(); + int ret = sensor->setFormat(&sensorFormat); + if (ret) + return ret; + + IPACameraSensorInfo sensorInfo{}; + ret = sensor->sensorInfo(&sensorInfo); + if (ret) + return ret; + + ipa::ipu3::IPAInitInfo initInfo{ + sensorInfo, + sensor->controls(), + }; + ipaControls_ = {}; + ret = ipa_->init(initInfo, &ipaControls_); if (ret) { LOG(IPU3, Error) << "Failed to initialise the IPU3 IPA"; return ret; } + /* + * \todo Report the actual exposure time, use the default for the + * moment. + */ + const auto exposureInfo = ipaControls_.find(&controls::ExposureTime); + if (exposureInfo == ipaControls_.end()) { + LOG(IPU3, Error) << "Exposure control not initializaed by the IPA"; + return -EINVAL; + } + exposureTime_ = exposureInfo->second.def().get(); + return 0; } From patchwork Fri Jul 16 14:32:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13031 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B1A7EC322A for ; Fri, 16 Jul 2021 14:31:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 06C3C68542; Fri, 16 Jul 2021 16:31:35 +0200 (CEST) Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 37C8368536 for ; Fri, 16 Jul 2021 16:31:33 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id AC13624000C; Fri, 16 Jul 2021 14:31:32 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 16 Jul 2021 16:32:15 +0200 Message-Id: <20210716143215.67454-3-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210716143215.67454-1-jacopo@jmondi.org> References: <20210716143215.67454-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/2] ipa: ipu3: Tidy-up includes X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Tidy-up a bit the inclusions directive in the IPU3 IPA module. In detail: - ipu3.cpp is missing inclusions for: std::abs from std::map from std::min/max from std::numeric_limits from std::unique_ptr from std::vector from and does not require - ipu3_agc has two not used inclusions in the header file and one the cpp file and is missing for std::literals::chrono_literals - ipu3_awb is missing for std::sort and does not use or Signed-off-by: Jacopo Mondi Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart --- src/ipa/ipu3/ipu3.cpp | 7 ++++++- src/ipa/ipu3/ipu3_agc.cpp | 2 +- src/ipa/ipu3/ipu3_agc.h | 3 --- src/ipa/ipu3/ipu3_awb.cpp | 3 +-- 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index d3c69bc07bd0..dc22acd4fd08 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -5,10 +5,15 @@ * ipu3.cpp - IPU3 Image Processing Algorithms */ +#include #include +#include +#include +#include +#include #include -#include #include +#include #include #include diff --git a/src/ipa/ipu3/ipu3_agc.cpp b/src/ipa/ipu3/ipu3_agc.cpp index 6253ab94cff1..408eb849b428 100644 --- a/src/ipa/ipu3/ipu3_agc.cpp +++ b/src/ipa/ipu3/ipu3_agc.cpp @@ -8,8 +8,8 @@ #include "ipu3_agc.h" #include +#include #include -#include #include diff --git a/src/ipa/ipu3/ipu3_agc.h b/src/ipa/ipu3/ipu3_agc.h index 3deca3ae6933..9f3d4257d945 100644 --- a/src/ipa/ipu3/ipu3_agc.h +++ b/src/ipa/ipu3/ipu3_agc.h @@ -7,9 +7,6 @@ #ifndef __LIBCAMERA_IPU3_AGC_H__ #define __LIBCAMERA_IPU3_AGC_H__ -#include -#include - #include #include diff --git a/src/ipa/ipu3/ipu3_awb.cpp b/src/ipa/ipu3/ipu3_awb.cpp index 9b409c8ffad9..4bb321b377a2 100644 --- a/src/ipa/ipu3/ipu3_awb.cpp +++ b/src/ipa/ipu3/ipu3_awb.cpp @@ -6,9 +6,8 @@ */ #include "ipu3_awb.h" +#include #include -#include -#include #include