From patchwork Fri Aug 27 12:07:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13545 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 65803BD87D for ; Fri, 27 Aug 2021 12:07:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 21EC66893C; Fri, 27 Aug 2021 14:07:16 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D30A16890C for ; Fri, 27 Aug 2021 14:07:13 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 1C12E2000A; Fri, 27 Aug 2021 12:07:12 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:42 +0200 Message-Id: <20210827120757.110615-2-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 01/16] libcamera: ipu3: Use the optimal sensor size X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" As reported by commit 7208e70211a6 ("libcamera: ipu3: Always use sensor full frame size") the current implementation of the IPU3 pipeline handler always use the sensor resolution as the ImgU input frame size in order to work around an issue with the ImgU configuration procedure. Now that the frame selection policy has been modified in the CIO2Device class implementation to comply with the requirements of the ImgU configuration script we can remove the workaround and select the most opportune sensor size to feed the ImgU with. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder Reviewed-by: Umang Jain --- src/libcamera/pipeline/ipu3/ipu3.cpp | 38 ++++++++++++++-------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index c287bf86e79a..b321c94e9cb0 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -229,7 +229,16 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate() status = Adjusted; } - /* Validate the requested stream configuration */ + /* + * Validate the requested stream configuration and select the sensor + * format by collecting the maximum RAW stream width and height and + * picking the closest larger match. + * + * If no RAW stream is requested use the one of the largest YUV stream, + * plus margin pixels for the IF and BDS rectangle to downscale. + * + * \todo Clarify the IF and BDS margins requirements. + */ unsigned int rawCount = 0; unsigned int yuvCount = 0; Size maxYuvSize; @@ -240,7 +249,7 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate() if (info.colourEncoding == PixelFormatInfo::ColourEncodingRAW) { rawCount++; - rawSize = cfg.size; + rawSize.expandTo(cfg.size); } else { yuvCount++; maxYuvSize.expandTo(cfg.size); @@ -269,24 +278,15 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate() /* * Generate raw configuration from CIO2. * - * \todo The image sensor frame size should be selected to optimize - * operations based on the sizes of the requested streams. However such - * a selection makes the pipeline configuration procedure fail for small - * resolutions (for example: 640x480 with OV5670) and causes the capture - * operations to stall for some stream size combinations (see the - * commit message of the patch that introduced this comment for more - * failure examples). - * - * Until the sensor frame size calculation criteria are clarified, when - * capturing from ImgU always use the largest possible size which - * guarantees better results at the expense of the frame rate and CSI-2 - * bus bandwidth. When only a raw stream is requested use the requested - * size instead, as the ImgU is not involved. + * The output YUV streams will be limited in size to the + * maximum frame size requested for the RAW stream. */ - if (!yuvCount) - cio2Configuration_ = data_->cio2_.generateConfiguration(rawSize); - else - cio2Configuration_ = data_->cio2_.generateConfiguration({}); + if (rawSize.isNull()) + rawSize = maxYuvSize.alignedUpTo(40, 540) + .alignedUpTo(IMGU_OUTPUT_WIDTH_MARGIN, + IMGU_OUTPUT_HEIGHT_MARGIN) + .boundedTo(data_->cio2_.sensor()->resolution()); + cio2Configuration_ = data_->cio2_.generateConfiguration(rawSize); if (!cio2Configuration_.pixelFormat.isValid()) return Invalid; From patchwork Fri Aug 27 12:07:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13546 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7EA69C3241 for ; Fri, 27 Aug 2021 12:07:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6CA026893F; Fri, 27 Aug 2021 14:07:16 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 98A7C6891F for ; Fri, 27 Aug 2021 14:07:14 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 0DDC820006; Fri, 27 Aug 2021 12:07:13 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:43 +0200 Message-Id: <20210827120757.110615-3-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 02/16] libcamera: ipu3: Split controls init/update X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In order to prepare to updating the Camera controls limits when anew camera configuration is applied, split the initControls() function in 2: - updateControls() to actually compute controls values - initControls() to initialize the sensor configuration and call updateControls Update the functions documentation accordingly. No functional changes intended. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder Reviewed-by: Umang Jain --- src/libcamera/pipeline/ipu3/ipu3.cpp | 33 ++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index b321c94e9cb0..885f5ddce139 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -154,6 +154,7 @@ private: } int initControls(IPU3CameraData *data); + int updateControls(IPU3CameraData *data); int registerCameras(); int allocateBuffers(Camera *camera); @@ -927,9 +928,11 @@ bool PipelineHandlerIPU3::match(DeviceEnumerator *enumerator) * \brief Initialize the camera controls * \param[in] data The camera data * - * Initialize the camera controls as the union of the static pipeline handler - * controls (IPU3Controls) and controls created dynamically from the sensor - * capabilities. + * Initialize the camera controls by calculating controls which the pipeline + * is reponsible for and merge them with the controls computed by the IPA. + * + * This function needs data->ipaControls_ to be initialized by the IPA init() + * function at camera creation time. Always call this function after IPA init(). * * \return 0 on success or a negative error code otherwise */ @@ -950,8 +953,30 @@ int PipelineHandlerIPU3::initControls(IPU3CameraData *data) if (ret) return ret; + return updateControls(data); +} + +/** + * \brief Update the camera controls + * \param[in] data The camera data + * + * Compute the camera controls by calculating controls which the pipeline + * is reponsible for and merge them with the controls computed by the IPA. + * + * This function needs data->ipaControls_ to be refreshed when a new + * configuration is applied to the camera by the IPA configure() function. + * + * Always call this function after IPA configure() to make sure to have a + * properly refreshed IPA controls list. + * + * \return 0 on success or a negative error code otherwise + */ +int PipelineHandlerIPU3::updateControls(IPU3CameraData *data) +{ + CameraSensor *sensor = data->cio2_.sensor(); IPACameraSensorInfo sensorInfo{}; - ret = sensor->sensorInfo(&sensorInfo); + + int ret = sensor->sensorInfo(&sensorInfo); if (ret) return ret; From patchwork Fri Aug 27 12:07:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13547 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4CF93BD87D for ; Fri, 27 Aug 2021 12:07:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0E42B6893C; Fri, 27 Aug 2021 14:07:19 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9E55E6891F for ; Fri, 27 Aug 2021 14:07:15 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id CDA6E20005; Fri, 27 Aug 2021 12:07:14 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:44 +0200 Message-Id: <20210827120757.110615-4-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 03/16] ipa: ipu3: Update camera controls in configure() X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a new CameraConfiguration is applied to the Camera the IPA is configured as well, using the newly applied sensor configuration and its updated V4L2 controls. Also update the Camera controls at IPA::configure() time by re-computing the controls::ExposureTime and controls::FrameDurationLimits limits and update the controls on the pipeline handler side after having configured the IPA. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder Reviewed-by: Umang Jain --- include/libcamera/ipa/ipu3.mojom | 3 +- src/ipa/ipu3/ipu3.cpp | 82 +++++++++++++++++++--------- src/libcamera/pipeline/ipu3/ipu3.cpp | 4 +- 3 files changed, 60 insertions(+), 29 deletions(-) diff --git a/include/libcamera/ipa/ipu3.mojom b/include/libcamera/ipa/ipu3.mojom index d561c2244907..714f58ab8a42 100644 --- a/include/libcamera/ipa/ipu3.mojom +++ b/include/libcamera/ipa/ipu3.mojom @@ -45,7 +45,8 @@ interface IPAIPU3Interface { start() => (int32 ret); stop(); - configure(IPAConfigInfo configInfo) => (int32 ret); + configure(IPAConfigInfo configInfo) + => (int32 ret, libcamera.ControlInfoMap ipaControls); mapBuffers(array buffers); unmapBuffers(array ids); diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 0ed0a6f1a355..fc5f69ed5ddc 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -156,13 +156,17 @@ public: int start() override; void stop() override {} - int configure(const IPAConfigInfo &configInfo) override; + int configure(const IPAConfigInfo &configInfo, + ControlInfoMap *ipaControls) override; void mapBuffers(const std::vector &buffers) override; void unmapBuffers(const std::vector &ids) override; void processEvent(const IPU3Event &event) override; private: + void updateControls(const IPACameraSensorInfo &sensorInfo, + const ControlInfoMap &sensorControls, + ControlInfoMap *ipaControls); void processControls(unsigned int frame, const ControlList &controls); void fillParams(unsigned int frame, ipu3_uapi_params *params); void parseStatistics(unsigned int frame, @@ -197,36 +201,30 @@ private: struct IPAContext context_; }; -/** - * Initialize the IPA module and its controls. + +/* + * Compute camera controls using the sensor information and the sensor + * v4l2 controls. * - * This function receives the camera sensor information from the pipeline - * handler, computes the limits of the controls it handles and returns - * them in the \a ipaControls output parameter. + * Some of the camera controls are computed by the pipeline handler, some others + * by the IPA module which is in charge of handling, in example, the exposure + * time and the frame duration. + * + * This functions computes: + * - controls::ExposureTime + * - controls::FrameDurationLimits */ -int IPAIPU3::init(const IPASettings &settings, - const IPACameraSensorInfo &sensorInfo, - const ControlInfoMap &sensorControls, - ControlInfoMap *ipaControls) +void IPAIPU3::updateControls(const IPACameraSensorInfo &sensorInfo, + const ControlInfoMap &sensorControls, + ControlInfoMap *ipaControls) { - camHelper_ = CameraSensorHelperFactory::create(settings.sensorModel); - if (camHelper_ == nullptr) { - LOG(IPAIPU3, Error) - << "Failed to create camera sensor helper for " - << settings.sensorModel; - return -ENODEV; - } - - /* Initialize Controls. */ ControlInfoMap::Map controls{}; /* - * Compute exposure time limits. - * - * Initialize the control using the line length and pixel rate of the - * current configuration converted to microseconds. Use the - * V4L2_CID_EXPOSURE control to get exposure min, max and default and - * convert it from lines to microseconds. + * Compute exposure time limits by using line length and pixel rate + * converted to microseconds. Use the V4L2_CID_EXPOSURE control to get + * exposure min, max and default and convert it from lines to + * microseconds. */ double lineDuration = sensorInfo.lineLength / (sensorInfo.pixelRate / 1e6); const ControlInfo &v4l2Exposure = sensorControls.find(V4L2_CID_EXPOSURE)->second; @@ -264,12 +262,36 @@ int IPAIPU3::init(const IPASettings &settings, frameDurations[2]); *ipaControls = ControlInfoMap(std::move(controls), controls::controls); +} + +/** + * Initialize the IPA module and its controls. + * + * This function receives the camera sensor information from the pipeline + * handler, computes the limits of the controls it handles and returns + * them in the \a ipaControls output parameter. + */ +int IPAIPU3::init(const IPASettings &settings, + const IPACameraSensorInfo &sensorInfo, + const ControlInfoMap &sensorControls, + ControlInfoMap *ipaControls) +{ + camHelper_ = CameraSensorHelperFactory::create(settings.sensorModel); + if (camHelper_ == nullptr) { + LOG(IPAIPU3, Error) + << "Failed to create camera sensor helper for " + << settings.sensorModel; + return -ENODEV; + } /* Construct our Algorithms */ algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); + /* Initialize controls. */ + updateControls(sensorInfo, sensorControls, ipaControls); + return 0; } @@ -335,7 +357,8 @@ void IPAIPU3::calculateBdsGrid(const Size &bdsOutputSize) << (int)bdsGrid.height << " << " << (int)bdsGrid.block_height_log2 << ")"; } -int IPAIPU3::configure(const IPAConfigInfo &configInfo) +int IPAIPU3::configure(const IPAConfigInfo &configInfo, + ControlInfoMap *ipaControls) { if (configInfo.entityControls.empty()) { LOG(IPAIPU3, Error) << "No controls provided"; @@ -344,6 +367,10 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo) sensorInfo_ = configInfo.sensorInfo; + /* + * Compute the sensor V4L2 controls to be used by the algorithms and + * to be set on the sensor. + */ ctrls_ = configInfo.entityControls.at(0); const auto itExp = ctrls_.find(V4L2_CID_EXPOSURE); @@ -385,6 +412,9 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo) return ret; } + /* Update the camera controls using the new sensor settings. */ + updateControls(sensorInfo_, ctrls_, ipaControls); + return 0; } diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 885f5ddce139..f3ca52811df5 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -660,14 +660,14 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) configInfo.bdsOutputSize = config->imguConfig().bds; configInfo.iif = config->imguConfig().iif; - ret = data->ipa_->configure(configInfo); + ret = data->ipa_->configure(configInfo, &data->ipaControls_); if (ret) { LOG(IPU3, Error) << "Failed to configure IPA: " << strerror(-ret); return ret; } - return 0; + return updateControls(data); } int PipelineHandlerIPU3::exportFrameBuffers(Camera *camera, Stream *stream, From patchwork Fri Aug 27 12:07:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13548 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A6095C3241 for ; Fri, 27 Aug 2021 12:07:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3A42868947; Fri, 27 Aug 2021 14:07:19 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C0EE368941 for ; Fri, 27 Aug 2021 14:07:16 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id D255A20005; Fri, 27 Aug 2021 12:07:15 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:45 +0200 Message-Id: <20210827120757.110615-5-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 04/16] android: capabilities: Collect per-stream frame durations X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Collect the per-stream frame durations while building the list of supported stream formats and resolutions. In order to get an updated list of controls it is necessary to apply to the Camera the configuration we're testing, which was so far only validated. The per-configuration durations will be used to populate the Android ANDROID_SCALER_AVAILABLE_MIN_FRAME_DURATIONS static metadata. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder Reviewed-by: Umang Jain --- src/android/camera_capabilities.cpp | 36 ++++++++++++++++++++++++++--- src/android/camera_capabilities.h | 2 ++ 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index c16e755f4695..fdda90379ce2 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -608,7 +608,29 @@ int CameraCapabilities::initializeStreamConfigurations() } for (const Size &res : resolutions) { - streamConfigurations_.push_back({ res, androidFormat }); + /* + * Configure the Camera with the collected format and + * resolution to get an updated list of controls. + */ + cfg.size = res; + int ret = camera_->configure(cameraConfig.get()); + if (ret) + return ret; + + const ControlInfoMap &controls = camera_->controls(); + const auto frameDurations = controls.find( + &controls::FrameDurationLimits); + if (frameDurations == controls.end()) { + LOG(HAL, Error) + << "Camera does not report frame durations"; + return -EINVAL; + } + + int64_t minFrameDuration = frameDurations->second.min().get() * 1000; + int64_t maxFrameDuration = frameDurations->second.max().get() * 1000; + streamConfigurations_.push_back({ + res, androidFormat, minFrameDuration, maxFrameDuration, + }); /* * If the format is HAL_PIXEL_FORMAT_YCbCr_420_888 @@ -620,10 +642,18 @@ int CameraCapabilities::initializeStreamConfigurations() * * \todo Support JPEG streams produced by the camera * natively. + * + * \todo HAL_PIXEL_FORMAT_BLOB is a 'stalling' format, + * its duration should take into account the time + * required for the YUV to JPEG encoding. For now + * use the same frame durations as collected for + * the YUV/RGB streams. */ if (androidFormat == HAL_PIXEL_FORMAT_YCbCr_420_888) { - streamConfigurations_.push_back( - { res, HAL_PIXEL_FORMAT_BLOB }); + streamConfigurations_.push_back({ + res, HAL_PIXEL_FORMAT_BLOB, + minFrameDuration, maxFrameDuration, + }); maxJpegSize = std::max(maxJpegSize, res); } } diff --git a/src/android/camera_capabilities.h b/src/android/camera_capabilities.h index a12596993ee5..6e55ddab445e 100644 --- a/src/android/camera_capabilities.h +++ b/src/android/camera_capabilities.h @@ -43,6 +43,8 @@ private: struct Camera3StreamConfiguration { libcamera::Size resolution; int androidFormat; + int64_t minFrameDurationNsec; + int64_t maxFrameDurationNsec; }; bool validateManualSensorCapability(); From patchwork Fri Aug 27 12:07:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13549 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D847AC3244 for ; Fri, 27 Aug 2021 12:07:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3678568945; Fri, 27 Aug 2021 14:07:24 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B2F6668932 for ; Fri, 27 Aug 2021 14:07:19 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id F255820005; Fri, 27 Aug 2021 12:07:18 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:48 +0200 Message-Id: <20210827120757.110615-8-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 07/16] android: capabilities: Use per-configuration durations X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Use the per-configuration stream durations as collected during initializeStreamConfigurations() to populate the ANDROID_SCALER_AVAILABLE_STREAM_CONFIGURATIONS_OUTPUT static metadata. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder Reviewed-by: Umang Jain --- src/android/camera_capabilities.cpp | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index c45d74189aaa..061e65241806 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1269,19 +1269,16 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_STALL_DURATIONS, availableStallDurations); - /* Use the minimum frame duration for all the YUV/RGB formats. */ - if (minFrameDurationNsec > 0) { - std::vector minFrameDurations; - minFrameDurations.reserve(streamConfigurations_.size() * 4); - for (const auto &entry : streamConfigurations_) { - minFrameDurations.push_back(entry.androidFormat); - minFrameDurations.push_back(entry.resolution.width); - minFrameDurations.push_back(entry.resolution.height); - minFrameDurations.push_back(minFrameDurationNsec); - } - staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_MIN_FRAME_DURATIONS, - minFrameDurations); + std::vector minFrameDurations; + minFrameDurations.reserve(streamConfigurations_.size() * 4); + for (const auto &entry : streamConfigurations_) { + minFrameDurations.push_back(entry.androidFormat); + minFrameDurations.push_back(entry.resolution.width); + minFrameDurations.push_back(entry.resolution.height); + minFrameDurations.push_back(entry.minFrameDurationNsec); } + staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_MIN_FRAME_DURATIONS, + minFrameDurations); uint8_t croppingType = ANDROID_SCALER_CROPPING_TYPE_CENTER_ONLY; staticMetadata_->addEntry(ANDROID_SCALER_CROPPING_TYPE, croppingType); From patchwork Fri Aug 27 12:07:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13550 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C2A33C3245 for ; Fri, 27 Aug 2021 12:07:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 557106894D; Fri, 27 Aug 2021 14:07:24 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BBF0668895 for ; Fri, 27 Aug 2021 14:07:20 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 048D520005; Fri, 27 Aug 2021 12:07:19 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:49 +0200 Message-Id: <20210827120757.110615-9-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 08/16] android: capabilties: Correctly populate STALL durations X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We currently only record 2560x1920@30FPS as stalling frame duration. This is of course not correct, and all the required information to properly populate the ANDROID_SCALER_AVAILABLE_STALL_DURATIONS static metadata are available from initializeStaticMetadata(). Use the collected stalling durations and sizes to properly popoulate the static property. Signed-off-by: Jacopo Mondi Reviewed-by: Umang Jain Reviewed-by: Paul Elder --- src/android/camera_capabilities.cpp | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 061e65241806..23a9a3fbec50 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1263,12 +1263,6 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_STREAM_CONFIGURATIONS, availableStreamConfigurations); - std::vector availableStallDurations = { - ANDROID_SCALER_AVAILABLE_FORMATS_BLOB, 2560, 1920, 33333333, - }; - staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_STALL_DURATIONS, - availableStallDurations); - std::vector minFrameDurations; minFrameDurations.reserve(streamConfigurations_.size() * 4); for (const auto &entry : streamConfigurations_) { @@ -1280,6 +1274,19 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_MIN_FRAME_DURATIONS, minFrameDurations); + std::vector availableStallDurations; + for (const auto &entry : streamConfigurations_) { + if (entry.androidFormat != HAL_PIXEL_FORMAT_BLOB) + continue; + + availableStallDurations.push_back(entry.androidFormat); + availableStallDurations.push_back(entry.resolution.width); + availableStallDurations.push_back(entry.resolution.height); + availableStallDurations.push_back(entry.minFrameDurationNsec); + } + staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_STALL_DURATIONS, + availableStallDurations); + uint8_t croppingType = ANDROID_SCALER_CROPPING_TYPE_CENTER_ONLY; staticMetadata_->addEntry(ANDROID_SCALER_CROPPING_TYPE, croppingType); From patchwork Fri Aug 27 12:07:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13551 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 86B66C3246 for ; Fri, 27 Aug 2021 12:07:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C4A8F68954; Fri, 27 Aug 2021 14:07:24 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A030A68895 for ; Fri, 27 Aug 2021 14:07:22 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 07A7320008; Fri, 27 Aug 2021 12:07:21 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:51 +0200 Message-Id: <20210827120757.110615-11-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 10/16] android: Filter preview streams on FPS X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Register as preview streams only streams capable of producing at least 30 FPS. This requirement comes from inspecting the existing HAL implementation on Intel IPU3 platform. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder Reviewed-by: Umang Jain --- src/android/camera_capabilities.cpp | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 388552963c47..c1ce63018aa8 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1257,6 +1257,21 @@ int CameraCapabilities::initializeStaticMetadata() std::vector availableStreamConfigurations; availableStreamConfigurations.reserve(streamConfigurations_.size() * 4); for (const auto &entry : streamConfigurations_) { + /* + * Filter out YUV streams not capable of running at 30 FPS. + * + * This requirement comes from inspecting the Intel IPU3 + * HAL implementation but no reference has been found in the + * metadata documentation. + * + * Calculate FPS as CTS does: see + * Camera2SurfaceViewTestCase.java:getSuitableFpsRangeForDuration() + */ + unsigned int fps = static_cast + (floor(1e9 / entry.minFrameDurationNsec + 0.05f)); + if (entry.androidFormat != HAL_PIXEL_FORMAT_BLOB && fps < 30) + continue; + availableStreamConfigurations.push_back(entry.androidFormat); availableStreamConfigurations.push_back(entry.resolution.width); availableStreamConfigurations.push_back(entry.resolution.height); @@ -1269,6 +1284,11 @@ int CameraCapabilities::initializeStaticMetadata() std::vector minFrameDurations; minFrameDurations.reserve(streamConfigurations_.size() * 4); for (const auto &entry : streamConfigurations_) { + unsigned int fps = static_cast + (floor(1e9 / entry.minFrameDurationNsec + 0.05f)); + if (entry.androidFormat != HAL_PIXEL_FORMAT_BLOB && fps < 30) + continue; + minFrameDurations.push_back(entry.androidFormat); minFrameDurations.push_back(entry.resolution.width); minFrameDurations.push_back(entry.resolution.height); From patchwork Fri Aug 27 12:07:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13552 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 14E0FC3248 for ; Fri, 27 Aug 2021 12:07:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 723CB68940; Fri, 27 Aug 2021 14:07:25 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7348668895 for ; Fri, 27 Aug 2021 14:07:23 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id DF0FC20002; Fri, 27 Aug 2021 12:07:22 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:52 +0200 Message-Id: <20210827120757.110615-12-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 11/16] android: capabilities: Print output stream list X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a debug statement to print out the list of collected output stream and their characteristics. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder Reviewed-by: Umang Jain --- src/android/camera_capabilities.cpp | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index c1ce63018aa8..484242d0ad81 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1277,6 +1277,13 @@ int CameraCapabilities::initializeStaticMetadata() availableStreamConfigurations.push_back(entry.resolution.height); availableStreamConfigurations.push_back( ANDROID_SCALER_AVAILABLE_STREAM_CONFIGURATIONS_OUTPUT); + + LOG(HAL, Debug) + << "Output Stream:" << utils::hex(entry.androidFormat) + << "(" << entry.resolution.width << "x" + << entry.resolution.height << ")[" + << entry.minFrameDurationNsec << "]" + << "@" << fps; } staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_STREAM_CONFIGURATIONS, availableStreamConfigurations); From patchwork Fri Aug 27 12:07:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13553 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9BE01C324A for ; Fri, 27 Aug 2021 12:07:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4007168948; Fri, 27 Aug 2021 14:07:31 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4857668956 for ; Fri, 27 Aug 2021 14:07:25 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 9685220002; Fri, 27 Aug 2021 12:07:24 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:54 +0200 Message-Id: <20210827120757.110615-14-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 13/16] android: capabilties: Fix ANDROID_CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" As reported by the CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES documentation in the Android developer reference: "For devices advertising any color filter arrangement other than NIR, or devices not advertising color filter arrangement, this list will always include (min, max) and (max, max) where min <= 15 and max = the maximum output frame rate of the maximum YUV_420_888 output size." Collect the higher FPS of the larger YUV stream and use 15 FPS as the minimum value, if the camera can go slower than that. Populate the ANDROID_CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES static metadata with the newly computed values. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder --- src/android/camera_capabilities.cpp | 75 ++++++++++------------------- 1 file changed, 26 insertions(+), 49 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 8bde06e824ef..01cdbcf1395d 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -862,55 +862,6 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_CONTROL_AE_AVAILABLE_MODES, aeAvailableModes); - /* Initialize the AE frame duration limits. */ - int64_t minFrameDurationNsec = -1; - int64_t maxFrameDurationNsec = -1; - const auto frameDurationsInfo = controlsInfo.find(&controls::FrameDurationLimits); - minFrameDurationNsec = frameDurationsInfo->second.min().get() * 1000; - maxFrameDurationNsec = frameDurationsInfo->second.max().get() * 1000; - - /* - * Adjust the minimum frame duration to comply with Android - * requirements. The camera service mandates all preview/record - * streams to have a minimum frame duration < 33,366 milliseconds - * (see MAX_PREVIEW_RECORD_DURATION_NS in the camera service - * implementation). - * - * If we're close enough (+ 500 useconds) to that value, round - * the minimum frame duration of the camera to an accepted - * value. - */ - static constexpr int64_t MAX_PREVIEW_RECORD_DURATION_NS = 1e9 / 29.97; - if (minFrameDurationNsec > MAX_PREVIEW_RECORD_DURATION_NS && - minFrameDurationNsec < MAX_PREVIEW_RECORD_DURATION_NS + 500000) - minFrameDurationNsec = MAX_PREVIEW_RECORD_DURATION_NS - 1000; - - /* - * The AE routine frame rate limits are computed using the frame - * duration limits, as libcamera clips the AE routine to the - * frame durations. - */ - int32_t maxFps = std::round(1e9 / minFrameDurationNsec); - int32_t minFps = std::round(1e9 / maxFrameDurationNsec); - minFps = std::max(1, minFps); - - /* - * Force rounding errors so that we have the proper frame - * durations for when we reuse these variables later - */ - minFrameDurationNsec = 1e9 / maxFps; - maxFrameDurationNsec = 1e9 / minFps; - - /* - * Register to the camera service {min, max} and {max, max} - * intervals as requested by the metadata documentation. - */ - int32_t availableAeFpsTarget[] = { - minFps, maxFps, maxFps, maxFps - }; - staticMetadata_->addEntry(ANDROID_CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES, - availableAeFpsTarget); - std::vector aeCompensationRange = { 0, 0, }; @@ -1256,8 +1207,12 @@ int CameraCapabilities::initializeStaticMetadata() std::vector availableStreamConfigurations; std::vector minFrameDurations; + int maxYUVFps = 0; + Size maxYUVSize; + availableStreamConfigurations.reserve(streamConfigurations_.size() * 4); minFrameDurations.reserve(streamConfigurations_.size() * 4); + for (const auto &entry : streamConfigurations_) { /* * Filter out YUV streams not capable of running at 30 FPS. @@ -1274,6 +1229,16 @@ int CameraCapabilities::initializeStaticMetadata() if (entry.androidFormat != HAL_PIXEL_FORMAT_BLOB && fps < 30) continue; + /* + * Collect the FPS of the maximum YUV output size to populate + * AE_AVAILABLE_TARGET_FPS_RANGE + */ + if (entry.androidFormat == HAL_PIXEL_FORMAT_YCbCr_420_888 && + entry.resolution > maxYUVSize) { + maxYUVSize = entry.resolution; + maxYUVFps = fps; + } + /* Stream configuration map. */ availableStreamConfigurations.push_back(entry.androidFormat); availableStreamConfigurations.push_back(entry.resolution.width); @@ -1301,6 +1266,18 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_AVAILABLE_MIN_FRAME_DURATIONS, minFrameDurations); + /* + * Register to the camera service {min, max} and {max, max} with + * max being the larger YUV stream maximum frame rate and min being + * 15 if the camera can do less than that. + */ + int32_t minFps = std::max(1e9 / maxFrameDuration_, 15.0); + int32_t availableAeFpsTarget[] = { + minFps, maxYUVFps, maxYUVFps, maxYUVFps, + }; + staticMetadata_->addEntry(ANDROID_CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES, + availableAeFpsTarget); + std::vector availableStallDurations; for (const auto &entry : streamConfigurations_) { if (entry.androidFormat != HAL_PIXEL_FORMAT_BLOB) From patchwork Fri Aug 27 12:07:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13554 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9A4D0C324B for ; Fri, 27 Aug 2021 12:07:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5A78D6894C; Fri, 27 Aug 2021 14:07:31 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3DA2768941 for ; Fri, 27 Aug 2021 14:07:26 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 79E7E20007; Fri, 27 Aug 2021 12:07:25 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:55 +0200 Message-Id: <20210827120757.110615-15-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 14/16] android: capabilities: Cap frame rate to 30 FPS X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Limit the reported minumum frame duration to 30 FPS. The reason to do is to bring the libcamra HAL in par with the Intel HAL implementation on IPU3 platform, where 30FPS is the frame rate used to perform quality tuning in the closed-source IPA module and has been validated as the most efficient rate for the power/performace budget. This change bring into the HAL a platform specific constraints, which might be opportune for most platforms but should rather be configurable by system integrators. Record that with a \todo entry. Also record that, even if we report a lower frame rate, we currently do not limit what the camera actually produce. Signed-off-by: Jacopo Mondi --- src/android/camera_capabilities.cpp | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 01cdbcf1395d..6890bfecab29 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -632,6 +632,29 @@ int CameraCapabilities::initializeStreamConfigurations() int64_t minFrameDuration = frameDurations->second.min().get() * 1000; int64_t maxFrameDuration = frameDurations->second.max().get() * 1000; + + /* + * Cap min frame duration to 30 FPS. + * + * 30 frames per second has been validated as the most + * opportune frame rate for quality tuning, and power + * vs performances budget on Intel IPU3. + * + * \todo This is a platform-specific decision that needs + * to be abstracted and delegated to the configuration + * file. + * + * \todo libcamera only allows to control frame duration + * through the per-request controls::FrameDuration + * control. If we cap the durations here, we should be + * capable of configuring the camera to operate at such + * duration without requiring to have the FrameDuration + * control to be specified for each Request. Defer this + * to the in-development configuration API rework. + */ + if (minFrameDuration < 1e9 / 30.0) + minFrameDuration = 1e9 / 30.0; + streamConfigurations_.push_back({ res, androidFormat, minFrameDuration, maxFrameDuration, }); From patchwork Fri Aug 27 12:07:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13555 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7D758C324C for ; Fri, 27 Aug 2021 12:07:58 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 80C7868954; Fri, 27 Aug 2021 14:07:31 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 09CA06893C for ; Fri, 27 Aug 2021 14:07:27 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 6A4D920007; Fri, 27 Aug 2021 12:07:26 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:56 +0200 Message-Id: <20210827120757.110615-16-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 15/16] Revert "android: capabilities: Cap frame rate to 30 FPS" X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This reverts commit 257df463ad0e0862c211b03742140a99dd8cc0ef. --- src/android/camera_capabilities.cpp | 23 ----------------------- 1 file changed, 23 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 6890bfecab29..01cdbcf1395d 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -632,29 +632,6 @@ int CameraCapabilities::initializeStreamConfigurations() int64_t minFrameDuration = frameDurations->second.min().get() * 1000; int64_t maxFrameDuration = frameDurations->second.max().get() * 1000; - - /* - * Cap min frame duration to 30 FPS. - * - * 30 frames per second has been validated as the most - * opportune frame rate for quality tuning, and power - * vs performances budget on Intel IPU3. - * - * \todo This is a platform-specific decision that needs - * to be abstracted and delegated to the configuration - * file. - * - * \todo libcamera only allows to control frame duration - * through the per-request controls::FrameDuration - * control. If we cap the durations here, we should be - * capable of configuring the camera to operate at such - * duration without requiring to have the FrameDuration - * control to be specified for each Request. Defer this - * to the in-development configuration API rework. - */ - if (minFrameDuration < 1e9 / 30.0) - minFrameDuration = 1e9 / 30.0; - streamConfigurations_.push_back({ res, androidFormat, minFrameDuration, maxFrameDuration, }); From patchwork Fri Aug 27 12:07:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 13556 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A8279C324D for ; Fri, 27 Aug 2021 12:08:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A35A568958; Fri, 27 Aug 2021 14:07:31 +0200 (CEST) Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DCD726893B for ; Fri, 27 Aug 2021 14:07:27 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 36B0320007; Fri, 27 Aug 2021 12:07:27 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Fri, 27 Aug 2021 14:07:57 +0200 Message-Id: <20210827120757.110615-17-jacopo@jmondi.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210827120757.110615-1-jacopo@jmondi.org> References: <20210827120757.110615-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 16/16] ipa: ipu3: Cap frame duration to 30 FPS X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Limit the IPU3 frame rate to 30 FPS. The reason to do is to bring the IPU3 IPA in par with the Intel HAL implementation on IPU3 platform, where 30FPS is the frame rate used to perform quality tuning in the closed-source IPA module and has been validated as the most efficient rate for the power/performace budget. Compute the vertical blanking to maintain such frame rate and configure the sensor with that. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder --- src/ipa/ipu3/ipu3.cpp | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index fc5f69ed5ddc..0e5d5e479e20 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -183,7 +183,7 @@ private: IPACameraSensorInfo sensorInfo_; /* Camera sensor controls. */ - uint32_t defVBlank_; + uint32_t vBlank_; uint32_t exposure_; uint32_t minExposure_; uint32_t maxExposure_; @@ -257,10 +257,39 @@ void IPAIPU3::updateControls(const IPACameraSensorInfo &sensorInfo, frameDurations[i] = frameSize / (sensorInfo.pixelRate / 1000000U); } + /* + * Cap minimum frame duration to 30FPS. + * + * 30 FPS has been validated in the closed source Intel 3A module as the + * most opportune frame rate for quality tuning, and power + * vs performances budget on Intel IPU3. + * + * Reduce the minimum achievable frame rate to 30 FPS and compute the + * vertical blanking to maintain that rate. + */ + int64_t *minFrameDuration = &frameDurations[0]; + if (*minFrameDuration < 1e6 / 30.0) + *minFrameDuration = 1e6 / 30.0; + controls[&controls::FrameDurationLimits] = ControlInfo(frameDurations[0], frameDurations[1], frameDurations[2]); + /* + * Adjust the vertical blanking to obtain the desired frame duration. + * + * Assume a fixed line length as horizontal blanking is seldom + * controllable. + * + * \todo Support making this overridable by the application through + * controls::FrameDuration. + * + * \todo Clamp exposure to frame duration. + */ + vBlank_ = *minFrameDuration * (sensorInfo.pixelRate / 1000000U); + vBlank_ /= lineLength; + vBlank_ -= sensorInfo.outputSize.height; + *ipaControls = ControlInfoMap(std::move(controls), controls::controls); } @@ -399,8 +428,6 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, maxGain_ = itGain->second.max().get(); gain_ = minGain_; - defVBlank_ = itVBlank->second.def().get(); - /* Clean context at configuration */ context_ = {}; @@ -511,8 +538,7 @@ void IPAIPU3::parseStatistics(unsigned int frame, setControls(frame); - /* \todo Use VBlank value calculated from each frame exposure. */ - int64_t frameDuration = sensorInfo_.lineLength * (defVBlank_ + sensorInfo_.outputSize.height) / + int64_t frameDuration = sensorInfo_.lineLength * (vBlank_ + sensorInfo_.outputSize.height) / (sensorInfo_.pixelRate / 1e6); ctrls.set(controls::FrameDuration, frameDuration); @@ -534,6 +560,7 @@ void IPAIPU3::setControls(unsigned int frame) ControlList ctrls(ctrls_); ctrls.set(V4L2_CID_EXPOSURE, static_cast(exposure_)); ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(gain_)); + ctrls.set(V4L2_CID_VBLANK, static_cast(vBlank_)); op.controls = ctrls; queueFrameAction.emit(frame, op);