From patchwork Thu Sep 15 11:47:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rishikesh Donadkar X-Patchwork-Id: 17370 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B6C56C3272 for ; Thu, 15 Sep 2022 11:47:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 71A7161FB6; Thu, 15 Sep 2022 13:47:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1663242479; bh=2yW/T08lgkNZWJF/Wz917bnmL36SVgvJ/9PffarehGo=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=vLdVuKBXvmOp9YQ9uNRyqdiUul8VBX3EEnp2UaUSqiJpyIoTr3OrHOTqcyMPzuInP SkvDpRkQYSzMsGiT3GskY4frGxbY4guxZBvuO3TUeyMNpGo/VbHMql3wYdgJd64wj2 N93dJ5P2LcETvymxGtX7dMQe4j30FD/kHntQwWc/vEvsuuVLvpwXJFSC178/RCM661 +YTN/3B6NYkfHaIDsN3xd1AX/k5Ge36Rtq4sZ/R6u4zcQW70MdATBRsJidLBx3B7lK //MS4GdBOlyEgbvseL15wEq3/5bjzqkJwBOaqYuhr1WBY9ABVftpeykSre43Hz8FSj urivNa+MONT8g== Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 60D7A61FB0 for ; Thu, 15 Sep 2022 13:47:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NJ875NRQ"; dkim-atps=neutral Received: by mail-pg1-x532.google.com with SMTP id 207so8685912pgc.7 for ; Thu, 15 Sep 2022 04:47:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=snGuuKLWj8mTNv76jKgQ92n9lMhXdXR/s2vS5Dmao1o=; b=NJ875NRQcRTB+Z9lr3Cr5defWU5+dF9f6dawdujJBvrztpULVXa3wYiwdGjVb3Cnxb wVk67umzvCE+k+8bxS/ZqQ+2x8lFTTv+BTyWJDDo5LdhyErPjbW+s1Zxg/e9lO1Abv2L U6NMP2Uh+BezsEBxBY9gHa3swCCwxSRCDbCtWGBnx9SKjKCo1vruG18z+sIZg6qb8Zxs 2yrNZnTRWp6m6K6v2PbooRITAyS1fPH4//FQncn4HOTrdX/rvUxi5VtKYs30cY1+qJDM /PR+mkCNzUFKumZBMwhbyInZxMqsyxr9pGYBh8dv7FQ+y63/JGyk3b8XH/1g0sGZORYa 45hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=snGuuKLWj8mTNv76jKgQ92n9lMhXdXR/s2vS5Dmao1o=; b=2v2o9usczUOcU2gMGK5U0woTcsFNi36ATSciGk6IJTCeU6IbnD/eTrlCV3BNxKAv6T R+TcsEX+Dw9bgOzP5F0jp8bpXmuHrNdYHQgjPtL0BJOqGH1dC51xCSqlUKF3IksUo/GH f71gedQLEWmddP1cyn5e0KBAypUGO70+FTV7pk1dynru4D6RnO4wdk8uIglnsD/x7UW2 s4A0X6MOoRKN9TA9C6G4wDeqjvUMtQ8bxoFd61RcdlKHOeS8aq/Hrxha0wMB6aB73Yvg 85UqZvfnWbH4wDwKuDK2+fagkI2HzvycI+EwE9GEHVN/TQrlbWVH8FHjNtSFbdEfZhCR flOw== X-Gm-Message-State: ACgBeo0we/s6eBQkur0xYvfdS4yB9scWDvk8RwqyQMvY+/RQ71qMPn2M foA3wTBtbNBeNaLH6uhnVAYaYiKqTe1H8g== X-Google-Smtp-Source: AA6agR5zO6U/jvrDUOBl//idXH226F0QfXwhyGvbVaqj4jg1fwM7zZaQk18ObSHKxQPf3AdUs61vaA== X-Received: by 2002:a05:6a00:804:b0:544:4e98:3ff3 with SMTP id m4-20020a056a00080400b005444e983ff3mr18031048pfk.36.1663242476668; Thu, 15 Sep 2022 04:47:56 -0700 (PDT) Received: from localhost.localdomain ([49.36.99.120]) by smtp.googlemail.com with ESMTPSA id x15-20020a170902a38f00b001785dddc703sm3718994pla.120.2022.09.15.04.47.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Sep 2022 04:47:56 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Thu, 15 Sep 2022 17:17:34 +0530 Message-Id: <20220915114734.115572-3-rishikeshdonadkar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220915114734.115572-1-rishikeshdonadkar@gmail.com> References: <20220915114734.115572-1-rishikeshdonadkar@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 2/2] gstreamer: Provide framerate support for libcamerasrc. X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Rishikesh Donadkar via libcamera-devel From: Rishikesh Donadkar Reply-To: Rishikesh Donadkar Cc: nicolas.dufresne@collabora.com, vedantparanjape160201@gmail.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Control the framerate by setting the controls::FrameDurationLimits in an ControlList and passing the control list at camera::start(). This requires converting the framerate which of the type GST_TYPE_FRACTION into FrameDuration which of the type int64_t. This conversion causes loss of precision and the precise framerate cannot be obtained by Inverting the control::FrameDurationLimits value. Example - * Suppose the framerate requested is 35/1. The framerate is read form the caps in the from of fraction that has a numerator and denominator. * Converting to FrameDuration (in microseconds) (1 * 10^6) / 35 = 28571.4286 int64_t frame_duration = 28571 (the precision here is lost.) * To expose the framerate in caps, Inverting the frame_duration to get back the framerate and converting to Seconds. double framerate = 10^6 / 28571 and 10^6/28571 which is close but not equal to 35/1 will fail the negotiation. Get the framerate from the caps and convert it to FrameDuration. Clamp the frame_duration between the min/max FrameDuration supported by the camera. Set the FrameDurationLimits control in the initControls_. Store the clamped/unclamped framerate in the container to be retrieved later. Set the bound checked framerate from the container into the caps. Pass the ControlList initControls_ at camera->start(). Signed-off-by: Rishikesh Donadkar --- src/gstreamer/gstlibcamera-utils.cpp | 70 ++++++++++++++++++++++++++++ src/gstreamer/gstlibcamera-utils.h | 6 +++ src/gstreamer/gstlibcamerasrc.cpp | 16 ++++++- 3 files changed, 91 insertions(+), 1 deletion(-) diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp index 244a4a79..89f116ba 100644 --- a/src/gstreamer/gstlibcamera-utils.cpp +++ b/src/gstreamer/gstlibcamera-utils.cpp @@ -8,6 +8,7 @@ #include "gstlibcamera-utils.h" +#include #include using namespace libcamera; @@ -407,6 +408,75 @@ gst_libcamera_configure_stream_from_caps(StreamConfiguration &stream_cfg, } } +void gst_libcamera_get_framerate_from_caps([[maybe_unused]] GstCaps *caps, + GstStructure *container) +{ + GstStructure *s = gst_caps_get_structure(caps, 0); + gint fps_n = 0, fps_d = 1; + + if (gst_structure_has_field_typed(s, "framerate", GST_TYPE_FRACTION)) { + if (gst_structure_get_fraction(s, "framerate", &fps_n, &fps_d)) + gst_structure_set(container, "framerate", GST_TYPE_FRACTION, fps_n, fps_d, nullptr); + else + GST_WARNING("invalid framerate in the caps."); + } +} + +void gst_libcamera_clamp_and_set_frameduration(ControlList &controls, const ControlInfoMap &camera_controls, + GstStructure *container) +{ + gboolean in_bounds = false; + gint fps_caps_n, fps_caps_d, fps_n, fps_d; + + if (!gst_structure_has_field_typed(container, "framerate", GST_TYPE_FRACTION)) + return; + + auto iterFrameDuration = camera_controls.find(controls::FrameDurationLimits.id()); + if (iterFrameDuration == camera_controls.end()) { + GST_WARNING("Valid bounds for FrameDuration not found"); + return; + } + + const GValue *framerate = gst_structure_get_value(container, "framerate"); + + fps_caps_n = gst_value_get_fraction_numerator(framerate); + fps_caps_d = gst_value_get_fraction_denominator(framerate); + + int64_t frame_duration = (fps_caps_d * 1000000.0) / fps_caps_n; + int64_t min_frame_duration = iterFrameDuration->second.min().get(); + int64_t max_frame_duration = iterFrameDuration->second.max().get(); + + if (frame_duration < min_frame_duration) { + frame_duration = min_frame_duration; + } else if (frame_duration > max_frame_duration) { + frame_duration = max_frame_duration; + } else { + in_bounds = true; + } + + if (!in_bounds) { + double framerate_clamped = 1000000.0 / frame_duration; + gst_util_double_to_fraction(framerate_clamped, &fps_n, &fps_d); + gst_structure_set(container, "framerate", GST_TYPE_FRACTION, fps_n, fps_d, nullptr); + } + + controls.set(controls::FrameDurationLimits, + Span({ frame_duration, frame_duration })); +} + +void gst_libcamera_framerate_to_caps(GstCaps *caps, const GValue *container) +{ + if (!GST_VALUE_HOLDS_FRACTION(container)) + return; + + GstStructure *s = gst_caps_get_structure(caps, 0); + gint fps_caps_n, fps_caps_d; + + fps_caps_n = gst_value_get_fraction_numerator(container); + fps_caps_d = gst_value_get_fraction_denominator(container); + gst_structure_set(s, "framerate", GST_TYPE_FRACTION, fps_caps_n, fps_caps_d, nullptr); +} + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task) diff --git a/src/gstreamer/gstlibcamera-utils.h b/src/gstreamer/gstlibcamera-utils.h index 164189a2..3d217fcf 100644 --- a/src/gstreamer/gstlibcamera-utils.h +++ b/src/gstreamer/gstlibcamera-utils.h @@ -9,6 +9,7 @@ #pragma once #include +#include #include #include @@ -18,6 +19,11 @@ GstCaps *gst_libcamera_stream_formats_to_caps(const libcamera::StreamFormats &fo GstCaps *gst_libcamera_stream_configuration_to_caps(const libcamera::StreamConfiguration &stream_cfg); void gst_libcamera_configure_stream_from_caps(libcamera::StreamConfiguration &stream_cfg, GstCaps *caps); +void gst_libcamera_get_framerate_from_caps(GstCaps *caps, GstStructure *container); +void gst_libcamera_clamp_and_set_frameduration(libcamera::ControlList &controls, + const libcamera::ControlInfoMap &camera_controls, GstStructure *container); +void gst_libcamera_framerate_to_caps(GstCaps *caps, const GValue *container); + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task); #endif diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 60032236..37f07676 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -131,6 +131,7 @@ struct GstLibcameraSrcState { std::queue> completedRequests_ LIBCAMERA_TSA_GUARDED_BY(lock_); + ControlList initControls_; guint group_id_; int queueRequest(); @@ -462,6 +463,8 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, GstFlowReturn flow_ret = GST_FLOW_OK; gint ret; + GstStructure *container = gst_structure_new_empty("container"); + GST_DEBUG_OBJECT(self, "Streaming thread has started"); gint stream_id_num = 0; @@ -496,6 +499,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, /* Retrieve the supported caps. */ g_autoptr(GstCaps) filter = gst_libcamera_stream_formats_to_caps(stream_cfg.formats()); g_autoptr(GstCaps) caps = gst_pad_peer_query_caps(srcpad, filter); + if (gst_caps_is_empty(caps)) { flow_ret = GST_FLOW_NOT_NEGOTIATED; break; @@ -504,6 +508,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, /* Fixate caps and configure the stream. */ caps = gst_caps_make_writable(caps); gst_libcamera_configure_stream_from_caps(stream_cfg, caps); + gst_libcamera_get_framerate_from_caps(caps, container); } if (flow_ret != GST_FLOW_OK) @@ -525,6 +530,10 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, goto done; } + /* Check frame duration bounds within controls::FrameDurationLimits */ + gst_libcamera_clamp_and_set_frameduration(state->initControls_, + state->cam_->controls(), container); + /* * Regardless if it has been modified, create clean caps and push the * caps event. Downstream will decide if the caps are acceptable. @@ -532,8 +541,11 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, for (gsize i = 0; i < state->srcpads_.size(); i++) { GstPad *srcpad = state->srcpads_[i]; const StreamConfiguration &stream_cfg = state->config_->at(i); + const GValue *framerate = gst_structure_get_value(container, "framerate"); g_autoptr(GstCaps) caps = gst_libcamera_stream_configuration_to_caps(stream_cfg); + gst_libcamera_framerate_to_caps(caps, framerate); + if (!gst_pad_push_event(srcpad, gst_event_new_caps(caps))) { flow_ret = GST_FLOW_NOT_NEGOTIATED; break; @@ -567,7 +579,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_flow_combiner_add_pad(self->flow_combiner, srcpad); } - ret = state->cam_->start(); + ret = state->cam_->start(&state->initControls_); if (ret) { GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, ("Failed to start the camera: %s", g_strerror(-ret)), @@ -577,6 +589,8 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, } done: + state->initControls_.clear(); + g_free(container); switch (flow_ret) { case GST_FLOW_NOT_NEGOTIATED: GST_ELEMENT_FLOW_ERROR(self, flow_ret);