From patchwork Mon Sep 12 09:56:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rishikesh Donadkar X-Patchwork-Id: 17363 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 48BA3C3272 for ; Mon, 12 Sep 2022 09:57:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 077C761F95; Mon, 12 Sep 2022 11:57:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1662976656; bh=TRIx0JdiGmmpv3T8gBGWMGXbcFCU+NDnhoqZd9KjXZY=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=j3kkto63G+5qNhw1rUpOZgtfm47vLSlvKP3YA1RuDtC1CxtLw4yUzAbkbB3amsHgN RwCOdBNlQ7La2tqeUM8wax97XdKozAlkXSchRmNKZf/RbEmHL7pbInzuyslR9elCY9 fQNT/xuekiIgFl0kqlT/gxjNi+Ty49ufcIjM4nSo61SMYcmIKypeie2O9bDNFApF7E Q4gCTHw48x73rD16mCjh3H7FGo2SkSZi4P/lkgSdMhM705oIqRPkoyNCVy9rDB3LMQ LmOZ5hXnNag+9IDKzSXNMxtM+i9A6ZJO8TUBhhVkVDZmGb/uILJRmvFWbHvXM9wKrw IlY1YTMQXJWPQ== Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1924161F8F for ; Mon, 12 Sep 2022 11:57:35 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WFlX8kQj"; dkim-atps=neutral Received: by mail-pl1-x62d.google.com with SMTP id iw17so8109713plb.0 for ; Mon, 12 Sep 2022 02:57:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=UcrUIUCtr/KO4vo4yPcHd7mFPsvV2uUA56Wbnl+hvpE=; b=WFlX8kQjzdQDI/P05ElsXHGQ6g7FEf9OYdDKYCFVVaAuFottYtrPL6+y7iY6eVzW1R YCNvAFWLHbPoeCwx2dKULbUZLSSUeRudoiubrJbNO5iVMMH/7A80Btpa779y8Wktr5nc HitsJ578Wq/cAZDToh97dSF46BzDLcP3VYIJKmJ0dryRl8st55PsoCG2poRkPQv1gPrm gy3n2Y+IXJpVL9WJjvGFBCmtrwnG3RAHtTvWcHJ7yg3SMjbmfi2Rf0qS1ROdTVSzkq4a eZ10jZIocAFAY+6Px4dDGI9lGTmYi5sqop/MgHXlV93yA6ay5SAROrAQj3Jns/aOdHWz 1+Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=UcrUIUCtr/KO4vo4yPcHd7mFPsvV2uUA56Wbnl+hvpE=; b=pze5g7/vSU/EDH9nmOaWozD6Ec/m1lDspKLRXOsoOhxWe5piUGPVYapBST15euk5If GXyrQDvZkboloRsuHyq5hpJk0m9I1s27vDg3nZzY0qWdWCjmkdM1P/EcJNPFbY0hEaUi wyaKfU4R0cStRVolLAFDZgCU8aIEo3PHqOBsJisrVbrLNaSnjolhMbilmFRifLyilcKY 8aV8qRz9NXzlrTy1bgJodkjeDAiQcNc8UZcaVJ4AHimFmXzDZACpEWzzdqwaLoxtcjtE qefVAuqMxN3nfe0iBy7KG76E4nOyT9uqrP3aNBwMkJjchPKIEYrToKo0cLDNPrPl5Yxc 4FOw== X-Gm-Message-State: ACgBeo0TVIMT0BiO1N73F/ZzpbtlmSFDr5Dh2sv+TjXIa5u/1gq4MJi4 qAEmZDErW0N1m6zswQArbMnWRttpXCnJpg== X-Google-Smtp-Source: AA6agR7ZsLfPSb/hs+K9ig3ErlyBGfDBd8LXHMIqKJ6evgBfd9MVhH5Jjg2hMi+VcpVPCwactfd+sg== X-Received: by 2002:a17:903:54:b0:176:cdf8:b791 with SMTP id l20-20020a170903005400b00176cdf8b791mr24903668pla.24.1662976653388; Mon, 12 Sep 2022 02:57:33 -0700 (PDT) Received: from localhost.localdomain ([49.36.99.4]) by smtp.googlemail.com with ESMTPSA id t14-20020aa7946e000000b00540dbae6272sm4959495pfq.213.2022.09.12.02.57.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Sep 2022 02:57:33 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Mon, 12 Sep 2022 15:26:56 +0530 Message-Id: <20220912095656.19013-3-rishikeshdonadkar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220912095656.19013-1-rishikeshdonadkar@gmail.com> References: <20220912095656.19013-1-rishikeshdonadkar@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 2/2] gstreamer: Provide framerate support for libcamerasrc. X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Rishikesh Donadkar via libcamera-devel From: Rishikesh Donadkar Reply-To: Rishikesh Donadkar Cc: nicolas.dufresne@collabora.com, vedantparanjape160201@gmail.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a field of the type ControlList to GstLibcameraSrcState. Get the framerate from the caps and convert it to FrameDuration. Clamp the frame_duration between the min/max FrameDuration supported by the camera and set the FrameDurationLimits control in the initControls_ Solve the complications in exposing the correct framerate due to loss in precision as a result of casting the frameduration to int64_t(which is required in libcamera to set the FrameDurationLimits control). Example - * Suppose the framerate requested is 35/1. The framerate is read form the caps in the from of fraction that has a numerator and denominator. * Converting to FrameDuration (in microseconds) (1 * 10^6) / 35 = 28571.4286 int64_t frame_duration = 28571 (the precision here is lost.) * To expose the framerate in caps, Inverting the frame_duration to get back the framerate and converting to Seconds. double framerate = 10^6 / 28571 and 10^6/28571 which is close but not equal to 35/1 will fail the negotiation. To solve the above problem, Store the framerate requested in the peer element caps in a container GstStructure, to be retrieved later. Get the control::FrameDurationLimits value that was set previously, convert to framerate and if it is equal to the one requested, set the framerate in the caps, else set the framerate to whatever is obtained from inverting the controls::FrameDurationLimits. Pass the initControls_ at camera->start(). Signed-off-by: Rishikesh Donadkar --- src/gstreamer/gstlibcamera-utils.cpp | 69 ++++++++++++++++++++++++++++ src/gstreamer/gstlibcamera-utils.h | 7 +++ src/gstreamer/gstlibcamerasrc.cpp | 16 ++++++- 3 files changed, 91 insertions(+), 1 deletion(-) diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp index 4df5dd6c..907ceab9 100644 --- a/src/gstreamer/gstlibcamera-utils.cpp +++ b/src/gstreamer/gstlibcamera-utils.cpp @@ -8,6 +8,7 @@ #include "gstlibcamera-utils.h" +#include #include using namespace libcamera; @@ -405,6 +406,74 @@ gst_libcamera_configure_stream_from_caps(StreamConfiguration &stream_cfg, } } +void gst_libcamera_get_framerate_from_caps([[maybe_unused]] GstCaps *caps, + GstStructure *container) +{ + GstStructure *s = gst_caps_get_structure(caps, 0); + gint fps_n = -1, fps_d = -1; + + if (gst_structure_has_field_typed(s, "framerate", GST_TYPE_FRACTION)) { + if (!gst_structure_get_fraction(s, "framerate", &fps_n, &fps_d)) { + GST_WARNING("invalid framerate in the caps."); + return; + } + } + + if (fps_n < 0 || fps_d < 0) + return; + + gst_structure_set(container, "framerate", GST_TYPE_FRACTION, fps_n, fps_d, nullptr); +} + +void gst_libcamera_clamp_and_set_frameduration(ControlList &controls, const ControlInfoMap &camera_controls, + const GstStructure *container) +{ + gint fps_caps_n, fps_caps_d; + const GValue *framerate = gst_structure_get_value(container, "framerate"); + + if (!gst_structure_has_field_typed(container, "framerate", GST_TYPE_FRACTION)) + return; + + auto iterFrameDuration = camera_controls.find(controls::FrameDurationLimits.id()); + if (iterFrameDuration == camera_controls.end()) + return; + + fps_caps_n = gst_value_get_fraction_numerator(framerate); + fps_caps_d = gst_value_get_fraction_denominator(framerate); + + int64_t frame_duration = (fps_caps_d * 1000000.0) / fps_caps_n; + int64_t min_frame_duration = iterFrameDuration->second.min().get(); + int64_t max_frame_duration = iterFrameDuration->second.max().get(); + + if (frame_duration < min_frame_duration) { + frame_duration = min_frame_duration; + } else if (frame_duration > max_frame_duration) { + frame_duration = max_frame_duration; + } + + controls.set(controls::FrameDurationLimits, + Span({ frame_duration, frame_duration })); +} + +void gst_libcamera_framerate_to_caps(const ControlList &controls, GstCaps *caps, const GValue *container) +{ + if (!GST_VALUE_HOLDS_FRACTION(container)) + return; + + GstStructure *s = gst_caps_get_structure(caps, 0); + gint fps_n, fps_d, fps_caps_n, fps_caps_d; + + fps_caps_n = gst_value_get_fraction_numerator(container); + fps_caps_d = gst_value_get_fraction_denominator(container); + double framerate = 1000000 / controls.get(controls::FrameDurationLimits).value()[0]; + gst_util_double_to_fraction(framerate, &fps_n, &fps_d); + + if (fps_caps_n / fps_caps_d == fps_n / fps_d) + gst_structure_set(s, "framerate", GST_TYPE_FRACTION, fps_caps_n, fps_caps_d, nullptr); + else + gst_structure_set(s, "framerate", GST_TYPE_FRACTION, fps_n, fps_d, nullptr); +} + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task) diff --git a/src/gstreamer/gstlibcamera-utils.h b/src/gstreamer/gstlibcamera-utils.h index 164189a2..bc04b073 100644 --- a/src/gstreamer/gstlibcamera-utils.h +++ b/src/gstreamer/gstlibcamera-utils.h @@ -9,6 +9,7 @@ #pragma once #include +#include #include #include @@ -18,6 +19,12 @@ GstCaps *gst_libcamera_stream_formats_to_caps(const libcamera::StreamFormats &fo GstCaps *gst_libcamera_stream_configuration_to_caps(const libcamera::StreamConfiguration &stream_cfg); void gst_libcamera_configure_stream_from_caps(libcamera::StreamConfiguration &stream_cfg, GstCaps *caps); +void gst_libcamera_get_framerate_from_caps(GstCaps *caps, GstStructure *container); +void gst_libcamera_clamp_and_set_frameduration(libcamera::ControlList &controls, + const libcamera::ControlInfoMap &camera_controls, const GstStructure *container); +void gst_libcamera_framerate_to_caps(const libcamera::ControlList &controls, GstCaps *caps, + const GValue *container); + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task); #endif diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 1ee1d08a..5cdf23a1 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -131,6 +131,7 @@ struct GstLibcameraSrcState { std::queue> completedRequests_ LIBCAMERA_TSA_GUARDED_BY(lock_); + ControlList initControls_; guint group_id_; int queueRequest(); @@ -462,6 +463,8 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, GstFlowReturn flow_ret = GST_FLOW_OK; gint ret; + GstStructure *container = gst_structure_new_empty("container"); + GST_DEBUG_OBJECT(self, "Streaming thread has started"); gint stream_id_num = 0; @@ -496,6 +499,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, /* Retrieve the supported caps. */ g_autoptr(GstCaps) filter = gst_libcamera_stream_formats_to_caps(stream_cfg.formats()); g_autoptr(GstCaps) caps = gst_pad_peer_query_caps(srcpad, filter); + if (gst_caps_is_empty(caps)) { flow_ret = GST_FLOW_NOT_NEGOTIATED; break; @@ -504,6 +508,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, /* Fixate caps and configure the stream. */ caps = gst_caps_make_writable(caps); gst_libcamera_configure_stream_from_caps(stream_cfg, caps); + gst_libcamera_get_framerate_from_caps(caps, container); } if (flow_ret != GST_FLOW_OK) @@ -524,6 +529,10 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, return; } + /* Check frame duration bounds within controls::FrameDurationLimits */ + gst_libcamera_clamp_and_set_frameduration(state->initControls_, + state->cam_->controls(), container); + /* * Regardless if it has been modified, create clean caps and push the * caps event. Downstream will decide if the caps are acceptable. @@ -531,8 +540,11 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, for (gsize i = 0; i < state->srcpads_.size(); i++) { GstPad *srcpad = state->srcpads_[i]; const StreamConfiguration &stream_cfg = state->config_->at(i); + const GValue *framerate = gst_structure_get_value(container, "framerate"); g_autoptr(GstCaps) caps = gst_libcamera_stream_configuration_to_caps(stream_cfg); + gst_libcamera_framerate_to_caps(state->initControls_, caps, framerate); + if (!gst_pad_push_event(srcpad, gst_event_new_caps(caps))) { flow_ret = GST_FLOW_NOT_NEGOTIATED; break; @@ -566,7 +578,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_flow_combiner_add_pad(self->flow_combiner, srcpad); } - ret = state->cam_->start(); + ret = state->cam_->start(&state->initControls_); if (ret) { GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, ("Failed to start the camera: %s", g_strerror(-ret)), @@ -576,6 +588,8 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, } done: + state->initControls_.clear(); + g_free(container); switch (flow_ret) { case GST_FLOW_NOT_NEGOTIATED: GST_ELEMENT_FLOW_ERROR(self, flow_ret);