From patchwork Wed Nov 2 13:56:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 17746 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E7155C3285 for ; Wed, 2 Nov 2022 13:56:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 51B8463075; Wed, 2 Nov 2022 14:56:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1667397388; bh=5J5VroM+40BTYAyivO//Zbssg3S73q4p/fVwcAjZERY=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=SlNIPIbW8lc/jOfMtMv4j1Yrt1vXB5E5XTxk2uSDmRQklYX2sx3DkbKkDp6gpLeGZ 01KYPAoVw9W/nrfFYf3+N9CfzySMWMyG2oedMvtbwnqeXmMR4eRoxtcex480RHAdDr H37NiklstmsWZLUl2kxpfV98ZOXwoY0ItB1v90aKxDteq/+Hw0SK8zxNW2UJ637eME v4JldW0e80/eFalh4jiddilD268qP+VM9PQnxpu2i3Sk4aHlGsJ9QP4826dGq735A/ +0VIXSE0kLbIz6VOn1yPIBPkDGc/e0Ij9EyKFY7lkFcGykb2Sz0w6aeE9OsSw5Y6nF dh36v2bAOD70Q== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 06E7F63075 for ; Wed, 2 Nov 2022 14:56:27 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="UJk5+QOc"; dkim-atps=neutral Received: from umang.jainideasonboard.com (unknown [103.251.226.107]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 488ED1226; Wed, 2 Nov 2022 14:56:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1667397386; bh=5J5VroM+40BTYAyivO//Zbssg3S73q4p/fVwcAjZERY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UJk5+QOcoZuQSxaMwIH9s9a+wXGnkKAnqcja/q8A9+oBJHLf7uAewqQ3U4Wi5XnPM AOYIcGtr9eK1chD66le7XBkqtKGg0FeMzglGEkQ0eGGDPuZwous3mk+c4F81iXbICX eQD7ZJ1VX+WN8H/RbW30K38nL6dh3ySAdnUonvq8= To: libcamera-devel@lists.libcamera.org Date: Wed, 2 Nov 2022 19:26:13 +0530 Message-Id: <20221102135614.657444-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221102135614.657444-1-umang.jain@ideasonboard.com> References: <20221102135614.657444-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v6 1/2] gstreamer: Do not expose the caps before configuring the camera X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Rishikesh Donadkar Configure the camera before exposing the caps valid controls values (and bounds) are available. These control values might be of interest to be exposed on the capabilites, which otherwise, would not be available if the camera is configured after the update caps event. For instance, the FrameDurationLimits are computed by RPi's IPA in its configure(). Hence, we need to Camera::configure() to happen in order to know the FrameDurationLimits, that can be exposed in the caps. This ties into the framerate support for libcamerasrc which will happen in a follow-up commit. Signed-off-by: Rishikesh Donadkar Signed-off-by: Umang Jain Tested-by: Umang Jain Reviewed-by: Umang Jain --- src/gstreamer/gstlibcamerasrc.cpp | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 16d70fea..60032236 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -515,6 +515,16 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, goto done; } + ret = state->cam_->configure(state->config_.get()); + if (ret) { + GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, + ("Failed to configure camera: %s", g_strerror(-ret)), + ("Camera::configure() failed with error code %i", ret)); + gst_task_stop(task); + flow_ret = GST_FLOW_NOT_NEGOTIATED; + goto done; + } + /* * Regardless if it has been modified, create clean caps and push the * caps event. Downstream will decide if the caps are acceptable. @@ -535,15 +545,6 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_pad_push_event(srcpad, gst_event_new_segment(&segment)); } - ret = state->cam_->configure(state->config_.get()); - if (ret) { - GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, - ("Failed to configure camera: %s", g_strerror(-ret)), - ("Camera::configure() failed with error code %i", ret)); - gst_task_stop(task); - return; - } - self->allocator = gst_libcamera_allocator_new(state->cam_, state->config_.get()); if (!self->allocator) { GST_ELEMENT_ERROR(self, RESOURCE, NO_SPACE_LEFT, From patchwork Wed Nov 2 13:56:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 17747 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2857DBD16B for ; Wed, 2 Nov 2022 13:56:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C101063080; Wed, 2 Nov 2022 14:56:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1667397391; bh=Pn7J48k/yYKWXnlUrI9OnY6j4RworjZkje1VK59bdfc=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=zmidIl6Zm/k+yXWq8f75fFGfXo559sm1uPyBH2E4hq1rceKuhtWOxEgTp6lAMnSV+ u+j8bhKjFMyM4atzkd+oeDXOyqQgvaHG+ZtwUYsujy6awcPKt04jpYYLogPCYJFfrN PAWh69sUWVskL/Rn3OoyxaUISwvONloivqI/8Oxnav0U7kwcaSrSNeo7Vn3aI+JQ2i NqWKW/SMq2GYfCiziAt7vmXbCUhsqjXBVX7Na7jIDwMaXhm0+7wzeyPkt5ms4VjW+7 NEtMaf21FwFuhB95OfYxoiSx0PTBCQn8mYJjAt330w2klNyx+kWdFj7ir1Sa9iWiR9 1GD1u/8q5Zq9A== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 11C2A63078 for ; Wed, 2 Nov 2022 14:56:30 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="XJa/G79P"; dkim-atps=neutral Received: from umang.jainideasonboard.com (unknown [103.251.226.107]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3265F1226; Wed, 2 Nov 2022 14:56:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1667397389; bh=Pn7J48k/yYKWXnlUrI9OnY6j4RworjZkje1VK59bdfc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XJa/G79PK6JnC/olrQg5gCvdL7/xZiix5t6kczEfSQ8+bc20WLFNDGHfy0sxsHyQr wBMtz1kq3WWFVFG/YcBSWx4HM2/wUcrXVPPyRAhxNTJq/SoHz3QyCmPnp/vjxsjk8o p0q5lr3/klZWWZ6FpVOLl/6o5NnKwqaat1/ku5NU= To: libcamera-devel@lists.libcamera.org Date: Wed, 2 Nov 2022 19:26:14 +0530 Message-Id: <20221102135614.657444-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221102135614.657444-1-umang.jain@ideasonboard.com> References: <20221102135614.657444-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v6 2/2] gstreamer: Provide framerate support for libcamerasrc X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Rishikesh Donadkar Control the framerate by passing the controls::FrameDurationLimits during Camera::start(). Framerate in gstreamer is expressed as GST_TYPE_FRACTION so we maximise on maintaining it as a fraction throughout and only do arithematic computations as and when required (to compute frame-duration and vice-versa). To weed out abritrary framerate as input, place the clamping via the controls::FrameDurationLimits provided after camera::configure() phase. This is handled by a helper function gst_libcamera_clamp_and_set_frameduration(). Set the bound checked framerate (done in the above mentioned helper) into the caps and pass the ControlList containing the frame-duration to Camera::start(ctrls). Signed-off-by: Rishikesh Donadkar Signed-off-by: Umang Jain Tested-by: Umang Jain --- src/gstreamer/gstlibcamera-utils.cpp | 78 ++++++++++++++++++++++++++++ src/gstreamer/gstlibcamera-utils.h | 6 +++ src/gstreamer/gstlibcamerasrc.cpp | 15 +++++- 3 files changed, 98 insertions(+), 1 deletion(-) diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp index 244a4a79..c14f72ec 100644 --- a/src/gstreamer/gstlibcamera-utils.cpp +++ b/src/gstreamer/gstlibcamera-utils.cpp @@ -8,6 +8,7 @@ #include "gstlibcamera-utils.h" +#include #include using namespace libcamera; @@ -407,6 +408,83 @@ gst_libcamera_configure_stream_from_caps(StreamConfiguration &stream_cfg, } } +void gst_libcamera_get_framerate_from_caps(GstCaps *caps, + GstStructure *container) +{ + GstStructure *s = gst_caps_get_structure(caps, 0); + /* + * Default to 30 fps. If the "framerate" fraction is invalid below, + * libcamerasrc will set 30fps as the framerate. + */ + gint fps_n = 30, fps_d = 1; + + if (gst_structure_has_field_typed(s, "framerate", GST_TYPE_FRACTION)) { + if (!gst_structure_get_fraction(s, "framerate", &fps_n, &fps_d)) + GST_WARNING("Invalid framerate in the caps"); + } + + gst_structure_set(container, "framerate", GST_TYPE_FRACTION, + fps_n, fps_d, nullptr); +} + +void gst_libcamera_clamp_and_set_frameduration(ControlList &initCtrls, + const ControlInfoMap &cam_ctrls, + GstStructure *container) +{ + gboolean in_bounds = false; + gint fps_caps_n, fps_caps_d; + + if (!gst_structure_has_field_typed(container, "framerate", GST_TYPE_FRACTION)) + return; + + auto iterFrameDuration = cam_ctrls.find(controls::FrameDurationLimits.id()); + if (iterFrameDuration == cam_ctrls.end()) { + GST_WARNING("FrameDurationLimits not found in camera controls."); + return; + } + + const GValue *framerate = gst_structure_get_value(container, "framerate"); + + fps_caps_n = gst_value_get_fraction_numerator(framerate); + fps_caps_d = gst_value_get_fraction_denominator(framerate); + + int64_t frame_duration = (fps_caps_d * 1000000.0) / fps_caps_n; + int64_t min_frame_duration = iterFrameDuration->second.min().get(); + int64_t max_frame_duration = iterFrameDuration->second.max().get(); + + if (frame_duration < min_frame_duration) { + frame_duration = min_frame_duration; + } else if (frame_duration > max_frame_duration) { + frame_duration = max_frame_duration; + } else { + in_bounds = true; + } + + if (!in_bounds) { + gint framerate_clamped = 1000000 / frame_duration; + + /* Update the framerate which then will be exposed in caps. */ + gst_structure_set(container, "framerate", GST_TYPE_FRACTION, + framerate_clamped, 1, nullptr); + } + + initCtrls.set(controls::FrameDurationLimits, + { frame_duration, frame_duration }); +} + +void gst_libcamera_framerate_to_caps(GstCaps *caps, const GValue *container) +{ + if (!GST_VALUE_HOLDS_FRACTION(container)) + return; + + GstStructure *s = gst_caps_get_structure(caps, 0); + gint fps_caps_n, fps_caps_d; + + fps_caps_n = gst_value_get_fraction_numerator(container); + fps_caps_d = gst_value_get_fraction_denominator(container); + gst_structure_set(s, "framerate", GST_TYPE_FRACTION, fps_caps_n, fps_caps_d, nullptr); +} + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task) diff --git a/src/gstreamer/gstlibcamera-utils.h b/src/gstreamer/gstlibcamera-utils.h index 164189a2..3d217fcf 100644 --- a/src/gstreamer/gstlibcamera-utils.h +++ b/src/gstreamer/gstlibcamera-utils.h @@ -9,6 +9,7 @@ #pragma once #include +#include #include #include @@ -18,6 +19,11 @@ GstCaps *gst_libcamera_stream_formats_to_caps(const libcamera::StreamFormats &fo GstCaps *gst_libcamera_stream_configuration_to_caps(const libcamera::StreamConfiguration &stream_cfg); void gst_libcamera_configure_stream_from_caps(libcamera::StreamConfiguration &stream_cfg, GstCaps *caps); +void gst_libcamera_get_framerate_from_caps(GstCaps *caps, GstStructure *container); +void gst_libcamera_clamp_and_set_frameduration(libcamera::ControlList &controls, + const libcamera::ControlInfoMap &camera_controls, GstStructure *container); +void gst_libcamera_framerate_to_caps(GstCaps *caps, const GValue *container); + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task); #endif diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 60032236..955d6eac 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -131,6 +131,7 @@ struct GstLibcameraSrcState { std::queue> completedRequests_ LIBCAMERA_TSA_GUARDED_BY(lock_); + ControlList initControls_; guint group_id_; int queueRequest(); @@ -462,6 +463,8 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, GstFlowReturn flow_ret = GST_FLOW_OK; gint ret; + GstStructure *container = gst_structure_new_empty("container"); + GST_DEBUG_OBJECT(self, "Streaming thread has started"); gint stream_id_num = 0; @@ -504,6 +507,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, /* Fixate caps and configure the stream. */ caps = gst_caps_make_writable(caps); gst_libcamera_configure_stream_from_caps(stream_cfg, caps); + gst_libcamera_get_framerate_from_caps(caps, container); } if (flow_ret != GST_FLOW_OK) @@ -525,6 +529,10 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, goto done; } + /* Check frame duration bounds within controls::FrameDurationLimits */ + gst_libcamera_clamp_and_set_frameduration(state->initControls_, + state->cam_->controls(), container); + /* * Regardless if it has been modified, create clean caps and push the * caps event. Downstream will decide if the caps are acceptable. @@ -532,8 +540,11 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, for (gsize i = 0; i < state->srcpads_.size(); i++) { GstPad *srcpad = state->srcpads_[i]; const StreamConfiguration &stream_cfg = state->config_->at(i); + const GValue *framerate = gst_structure_get_value(container, "framerate"); g_autoptr(GstCaps) caps = gst_libcamera_stream_configuration_to_caps(stream_cfg); + gst_libcamera_framerate_to_caps(caps, framerate); + if (!gst_pad_push_event(srcpad, gst_event_new_caps(caps))) { flow_ret = GST_FLOW_NOT_NEGOTIATED; break; @@ -567,7 +578,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_flow_combiner_add_pad(self->flow_combiner, srcpad); } - ret = state->cam_->start(); + ret = state->cam_->start(&state->initControls_); if (ret) { GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, ("Failed to start the camera: %s", g_strerror(-ret)), @@ -577,6 +588,8 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, } done: + state->initControls_.clear(); + g_free(container); switch (flow_ret) { case GST_FLOW_NOT_NEGOTIATED: GST_ELEMENT_FLOW_ERROR(self, flow_ret);