From patchwork Mon Apr 28 09:02:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Sven_P=C3=BCschel?= X-Patchwork-Id: 23273 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CC48CC327D for ; Mon, 28 Apr 2025 09:05:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5CEBB68B26; Mon, 28 Apr 2025 11:05:08 +0200 (CEST) Received: from metis.whiteo.stw.pengutronix.de (metis.whiteo.stw.pengutronix.de [IPv6:2a0a:edc0:2:b01:1d::104]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D527668ACF for ; Mon, 28 Apr 2025 11:05:05 +0200 (CEST) Received: from ptz.office.stw.pengutronix.de ([2a0a:edc0:0:900:1d::77] helo=peter.guest.stw.pengutronix.de) by metis.whiteo.stw.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1u9KQD-0001au-Hp; Mon, 28 Apr 2025 11:05:05 +0200 From: =?utf-8?q?Sven_P=C3=BCschel?= To: libcamera-devel@lists.libcamera.org Cc: =?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= , Paul Elder , Umang Jain , =?utf-8?q?Sven_P=C3=BCschel?= Subject: [PATCH v11 01/19] libcamera: property: Add MinimumRequests property Date: Mon, 28 Apr 2025 11:02:26 +0200 Message-ID: <20250428090413.38234-2-s.pueschel@pengutronix.de> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250428090413.38234-1-s.pueschel@pengutronix.de> References: <20250428090413.38234-1-s.pueschel@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2a0a:edc0:0:900:1d::77 X-SA-Exim-Mail-From: s.pueschel@pengutronix.de X-SA-Exim-Scanned: No (on metis.whiteo.stw.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: libcamera-devel@lists.libcamera.org X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Nícolas F. R. A. Prado The MinimumRequests property reports the minimum number of capture requests that the camera pipeline requires to have queued in order to sustain capture operations without frame drops. At this quantity, there's no guarantee that manual per-frame controls will apply correctly. The quantity needed for that may be added as a separate property in the future. The mali-c55 driver defines min_queued_buffers = 1 [1]. Therefore set set the minimum requests to 2 to account one request in the userspace. The dcmipp and j721e drivers both defines min_queued_buffers = 1 in the kernel under drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-bytecap.c and drivers/media/platform/ti/j721e-csi2rx/j721e-csi2rx.c . Therefore use these values, as they are added 1 more. For the intel-ipu6, mtk-seninf simple devices and the virtual pipeline use a conservative value of 3 minimumBuffers, as no further requirements are known about them. [1] https://lore.kernel.org/linux-media/20240131174355.GB20792@pendragon.ideasonboard.com/T/#t Signed-off-by: Nícolas F. R. A. Prado Signed-off-by: Paul Elder Acked-by: Umang Jain Signed-off-by: Sven Püschel --- Changes in v11 - Add mali-c55, dcmipp, virtual, j721e, intel-ipu6 and mtk-seninf - Hold only the minimum buffers instead of the whole deviceInfo pointer in SimpleCameraData - Adjusted min_buffers_needed property to min_reqbufs_allocation in docs - Relax property description from no frame drops -> only minimal frame drops, based on the comment from Naush [10] - Changed property type from int32_t -> uint32_t to match all of the indirect casts to an unsigned value throughout the existing patchset. - Removed Reviewed-by: Laurent Pinchart - Removed Reviewed-by: Paul Elder [10] https://lists.libcamera.org/pipermail/libcamera-devel/2022-December/035872.html Changes in v10: - ipu3: add a constant to populate MinimumRequests, as we'll also use it elsewhere - update pipeline handler guide to set vivid' MinimumRequests to 2 - expand the MinimumRequests property description to include a line about startup - add isi Changes in v9: - rebased Changes in v8: - Changed the MinimumRequests property meaning to require that frames aren't dropped - Set MinimumRequests on SimplePipeline depending on device and usage of converter - Undid definition of default MinimumRequests on CameraSensor constructor - Updated pipeline-handler guides with new MinimumRequests property Changes in v7: - Renamed property from MinNumRequests to MinimumRequests - Changed MinimumRequests property's description Changes in v6: - Removed comment from Raspberrypi MinNumRequests setting - Removed an extra blank line --- Documentation/guides/pipeline-handler.rst | 20 ++++++--- src/libcamera/pipeline/imx8-isi/imx8-isi.cpp | 2 + src/libcamera/pipeline/ipu3/ipu3.cpp | 4 ++ src/libcamera/pipeline/mali-c55/mali-c55.cpp | 1 + src/libcamera/pipeline/rkisp1/rkisp1.cpp | 1 + .../pipeline/rpi/common/pipeline_base.cpp | 2 + src/libcamera/pipeline/simple/simple.cpp | 41 +++++++++++++++---- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 2 + src/libcamera/pipeline/vimc/vimc.cpp | 2 + src/libcamera/pipeline/virtual/virtual.cpp | 1 + src/libcamera/property_ids_core.yaml | 22 ++++++++++ 11 files changed, 85 insertions(+), 13 deletions(-) diff --git a/Documentation/guides/pipeline-handler.rst b/Documentation/guides/pipeline-handler.rst index 9a15c20a..1079d998 100644 --- a/Documentation/guides/pipeline-handler.rst +++ b/Documentation/guides/pipeline-handler.rst @@ -663,19 +663,29 @@ associated with immutable values, which represent static characteristics that ca be used by applications to identify camera devices in the system. Properties can be registered by inspecting the values of V4L2 controls from the video devices and camera sensor (for example to retrieve the position and orientation of a camera) -or to express other immutable characteristics. The example pipeline handler does -not register any property, but examples are available in the libcamera code -base. +or to express other immutable characteristics. -.. TODO: Add a property example to the pipeline handler. At least the model. +A required property is ``MinimumRequests``, which indicates how many requests +need to be queued in the pipeline for capture without frame drops to be +possible. + +In our case, the vivid driver requires two buffers before it'll start streaming +(can be seen in the ``min_reqbufs_allocation`` property for the ``vid_cap`` queue in +vivid's driver code). Therefore we will set our ``MinimumRequests`` to two. +Append the following line to ``init()``: + +.. code-block:: cpp + + properties_.set(properties::MinimumRequests, 2); At this point you need to add the following includes to the top of the file for -handling controls: +handling controls and properties: .. code-block:: cpp #include #include + #include Vendor-specific controls and properties ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/src/libcamera/pipeline/imx8-isi/imx8-isi.cpp b/src/libcamera/pipeline/imx8-isi/imx8-isi.cpp index ecda426a..2b8a583e 100644 --- a/src/libcamera/pipeline/imx8-isi/imx8-isi.cpp +++ b/src/libcamera/pipeline/imx8-isi/imx8-isi.cpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include "libcamera/internal/bayer_format.h" @@ -165,6 +166,7 @@ int ISICameraData::init() return ret; properties_ = sensor_->properties(); + properties_.set(properties::MinimumRequests, 2); return 0; } diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index e31e3879..356ca2e1 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -168,6 +168,8 @@ private: MediaDevice *imguMediaDev_; std::vector ipaBuffers_; + + static constexpr unsigned int kMinimumRequests = 3; }; IPU3CameraConfiguration::IPU3CameraConfiguration(IPU3CameraData *data) @@ -1074,6 +1076,8 @@ int PipelineHandlerIPU3::registerCameras() /* Initialize the camera properties. */ data->properties_ = cio2->sensor()->properties(); + data->properties_.set(properties::MinimumRequests, kMinimumRequests); + ret = initControls(data.get()); if (ret) continue; diff --git a/src/libcamera/pipeline/mali-c55/mali-c55.cpp b/src/libcamera/pipeline/mali-c55/mali-c55.cpp index a05e11fc..f0ab3d2e 100644 --- a/src/libcamera/pipeline/mali-c55/mali-c55.cpp +++ b/src/libcamera/pipeline/mali-c55/mali-c55.cpp @@ -1602,6 +1602,7 @@ bool PipelineHandlerMaliC55::registerSensorCamera(MediaLink *ispLink) return false; data->properties_ = data->sensor_->properties(); + data->properties_.set(properties::MinimumRequests, 2); const CameraSensorProperties::SensorDelays &delays = data->sensor_->sensorDelays(); std::unordered_map params = { diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 194dfce7..5e3a4dba 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -1318,6 +1318,7 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor) /* Initialize the camera properties. */ data->properties_ = data->sensor_->properties(); + data->properties_.set(properties::MinimumRequests, 3); scalerMaxCrop_ = Rectangle(data->sensor_->resolution()); diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp index 1f13e523..ef51d530 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp @@ -848,6 +848,8 @@ int PipelineHandlerBase::registerCamera(std::unique_ptr &camera */ data->properties_.set(properties::ScalerCropMaximum, Rectangle{}); + data->properties_.set(properties::MinimumRequests, 3); + ret = platformRegister(cameraData, frontend, backend); if (ret) return ret; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index efb07051..7abf0fc9 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -26,6 +26,7 @@ #include #include +#include #include #include @@ -233,6 +234,10 @@ SimpleFrameInfo *SimpleFrames::find(uint32_t frame) struct SimplePipelineInfo { const char *driver; + /* + * Minimum number of buffers required by the driver to start streaming. + */ + unsigned int minimumBuffers; /* * Each converter in the list contains the name * and the number of streams it supports. @@ -249,14 +254,14 @@ struct SimplePipelineInfo { namespace { static const SimplePipelineInfo supportedDevices[] = { - { "dcmipp", {}, false }, - { "imx7-csi", { { "pxp", 1 } }, false }, - { "intel-ipu6", {}, true }, - { "j721e-csi2rx", {}, true }, - { "mtk-seninf", { { "mtk-mdp", 3 } }, false }, - { "mxc-isi", {}, false }, - { "qcom-camss", {}, true }, - { "sun6i-csi", {}, false }, + { "dcmipp", 1, {}, false }, + { "imx7-csi", 2, { { "pxp", 1 } }, false }, + { "intel-ipu6", 3, {}, true }, // \todo Check if the minimumBuffers can be lowered + { "j721e-csi2rx", 1, {}, true }, + { "mtk-seninf", 3, { { "mtk-mdp", 3 } }, false }, // \todo Check if the minimumBuffers can be lowered + { "mxc-isi", 3, {}, false }, + { "qcom-camss", 1, {}, true }, + { "sun6i-csi", 3, {}, false }, }; } /* namespace */ @@ -346,6 +351,8 @@ public: std::unique_ptr swIsp_; SimpleFrames frameInfo_; + unsigned int deviceInfoMinimumBuffers; + private: void tryPipeline(unsigned int code, const Size &size); static std::vector routedSourcePads(MediaPad *sink); @@ -1402,8 +1409,24 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) inputCfg.bufferCount = kNumInternalBuffers; if (data->converter_) { + /* + * The application will interact only with the capture node of + * the converter. Require two buffers for a frame drop free + * conversion, plus one extra to account for requeue delays. + */ + data->properties_.set(properties::MinimumRequests, 3); + return data->converter_->configure(inputCfg, outputCfgs); } else { + /* + * The application will interact directly with the video capture + * device. Require the minimum required by the driver, plus one + * extra to account for requeue delays. Force at least three + * buffers in order to not drop frames. + */ + data->properties_.set(properties::MinimumRequests, std::max(data->deviceInfoMinimumBuffers + 1, + 3U)); + ipa::soft::IPAConfigInfo configInfo; configInfo.sensorControls = data->sensor_->controls(); return data->swIsp_->configure(inputCfg, outputCfgs, configInfo); @@ -1787,6 +1810,8 @@ bool SimplePipelineHandler::match(DeviceEnumerator *enumerator) bool registered = false; for (std::unique_ptr &data : pipelines) { + data->deviceInfoMinimumBuffers = info->minimumBuffers; + int ret = data->init(); if (ret < 0) continue; diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index 586e932d..31951a12 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -586,6 +586,8 @@ int UVCCameraData::init(MediaDevice *media) properties_.set(properties::PixelArraySize, resolution); properties_.set(properties::PixelArrayActiveAreas, { Rectangle(resolution) }); + properties_.set(properties::MinimumRequests, 3); + /* Initialise the supported controls. */ ControlInfoMap::Map ctrls; diff --git a/src/libcamera/pipeline/vimc/vimc.cpp b/src/libcamera/pipeline/vimc/vimc.cpp index 07273bd2..01685a64 100644 --- a/src/libcamera/pipeline/vimc/vimc.cpp +++ b/src/libcamera/pipeline/vimc/vimc.cpp @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -592,6 +593,7 @@ int VimcCameraData::init() /* Initialize the camera properties. */ properties_ = sensor_->properties(); + properties_.set(properties::MinimumRequests, 3); return 0; } diff --git a/src/libcamera/pipeline/virtual/virtual.cpp b/src/libcamera/pipeline/virtual/virtual.cpp index 049ebcba..c33f1a5e 100644 --- a/src/libcamera/pipeline/virtual/virtual.cpp +++ b/src/libcamera/pipeline/virtual/virtual.cpp @@ -126,6 +126,7 @@ VirtualCameraData::VirtualCameraData(PipelineHandler *pipe, properties_.set(properties::PixelArrayActiveAreas, { Rectangle(config_.maxResolutionSize) }); + properties_.set(properties::MinimumRequests, 3); /* \todo Support multiple streams and pass multi_stream_test */ streamConfigs_.resize(kMaxStream); diff --git a/src/libcamera/property_ids_core.yaml b/src/libcamera/property_ids_core.yaml index 834454a4..cc28d677 100644 --- a/src/libcamera/property_ids_core.yaml +++ b/src/libcamera/property_ids_core.yaml @@ -701,4 +701,26 @@ controls: Different cameras may report identical devices. + - MinimumRequests: + type: uint32_t + description: | + The minimum number of capture requests that the camera pipeline requires + to have queued in order to sustain capture operations with only minimal + frame drops. At this quantity, there's no guarantee that manual per + frame controls will apply correctly. This is also the minimum number of + requests that must be queued before capture starts. + + This property is based on the minimum number of memory buffers + needed to fill the capture pipeline composed of hardware processing + blocks plus as many buffers as needed to take into account propagation + delays and avoid dropping frames. + + \todo Should this be a per-stream property? + + \todo Extend this property to expose the different minimum buffer and + request counts (the minimum number of buffers to be able to capture at + all, the minimum number of buffers to sustain capture without frame + drop, and the minimum number of requests to ensure proper operation of + per-frame controls). + ...