From patchwork Fri Sep 2 05:50:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rishikesh Donadkar X-Patchwork-Id: 17272 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D35E7C3272 for ; Fri, 2 Sep 2022 05:51:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2EAE561FDA; Fri, 2 Sep 2022 07:51:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1662097871; bh=YZhWPtrCBSyJjRr7SKpLTzIlaSEdbc0Mfi1qrq4vQGY=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=3lVi1ViY4FUxIMEg24VPfRTQur/UXNqEoNKYCdKfgAqk+HUnLhMr4lZGn5ks1wYkW jWzurvCF7frdyhjNfNLfum9g/D0/NsifAp2D6SSOgxG0E2kGY+2igJiy7y2b69L0Y+ KidkqFX+/c1TjCtUNakqR6vnkyI91yxvPbTzy+CZEbSSCSDsYkXo/IcZoHuDUuYY4F B3z2UerKMxkBXXKvl0I3uehuy85uzbMzDVJvIrh2eJW/g+I1+LWPqQPf44nXh6yWz4 8kC8C3zvTBar5VEBcjpk3yQlf1GSq9UzSjxxoJbZQCueu6o3v4dhx0pYNDNtTZgmQe XOAx0Fxsk4s7A== Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7E0DF61F99 for ; Fri, 2 Sep 2022 07:51:10 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="qzssOw0A"; dkim-atps=neutral Received: by mail-pj1-x102c.google.com with SMTP id z3-20020a17090abd8300b001fd803e34f1so4530759pjr.1 for ; Thu, 01 Sep 2022 22:51:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date; bh=bP9RXE5UQsl8VPOeVrdR9c/RC6UhFv9CGnys6DQju7A=; b=qzssOw0AukM2LoK6vpq/p1THTLKpnbdfHGzbp5iX8noVSRZUB0CVPDigd3z9hhvQp+ qKusRJ80fA0kWdTFJNt2x5nPzrVftcROvkyVHU86hopha/etx6Od4uFrsXZRyCqHGJsP 1PS1SwMhGsgNoVuOAbKRiIOdH/ExTDY3BGAhYNneb1zUqmhbOcmSyArW8M5D0c3iHipa N0BLOCesVHwzxDKvZ8YRNeRO9RJSfK+wTkfWVh+oDDgY8vobs9RL3+Gi2qOrcYS6mk1C wzR9JACyWRrfyzMyQY291TkH8LRi8+vgRuvtBkvU9t8Q2floPYWo3pnrczPAYDZ2WQuZ B5cA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date; bh=bP9RXE5UQsl8VPOeVrdR9c/RC6UhFv9CGnys6DQju7A=; b=TLx3k86NiNWKaPNiTHpA1FcAGaSFLanMrUnCxquoXme9WGo9gpyxXY5YtWJIux3b53 WE7ezBWbPXz0g2nqaZr/BgbE24vMIwY3dHk8dMM6WWPykQQRb25E5PqTSbDSmckP4BB8 bXhD82Lmk6RAVA7NFl+QNysXGFTRPUZ2Hnj8ngh2Z1f+3ex1WYJu9uEooKrKjS9ULQNv 1Po/q30PmxDcDEekmSqjKpK+jlXSLXDDBlsqiJUhJRoV3wWdtgKkXwk7ktBBGGSRg+FR GJ2EcxHRAissCHjvFQDpVtPR7O7IkFYCw8QdBmHDvPtvOAwYnpycuIV/Cq+z+FlqE8vE jakg== X-Gm-Message-State: ACgBeo3/Sdk3wiM32xFab2Q9sEc8HxRC21UkNUMzv0EQqrnaFIC5hx76 Ar28kh8Vcze6Y7vjFZKdMpJn9VrBXA8= X-Google-Smtp-Source: AA6agR7+6bupM4y8Kf7VlZqRYHCiT/jzZIlQqsIC2c52gyGeq8vax8zCQMJmHgwcmg3drH2O3ZGlGg== X-Received: by 2002:a17:902:ab55:b0:173:4116:8d3d with SMTP id ij21-20020a170902ab5500b0017341168d3dmr33451088plb.157.1662097868433; Thu, 01 Sep 2022 22:51:08 -0700 (PDT) Received: from localhost.localdomain ([49.36.101.190]) by smtp.googlemail.com with ESMTPSA id fy10-20020a17090b020a00b001fb18855440sm610496pjb.31.2022.09.01.22.51.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 22:51:08 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 2 Sep 2022 11:20:49 +0530 Message-Id: <20220902055049.15183-1-rishikeshdonadkar@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2] gstreamer: Provide framerate support for libcamerasrc. X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Rishikesh Donadkar via libcamera-devel From: Rishikesh Donadkar Reply-To: Rishikesh Donadkar Cc: Rishikesh Donadkar , nicolas.dufresne@collabora.com, vedantparanjape160201@gmail.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a field of the type ControlList to GstLibcameraSrcState. Get the framerate from the caps and convert it to FrameDuration. Set the FrameDurationLimits control in the initControls_ Passing initCtrls_ at camera::start() doesn't tell us if the controls have been applied to the camera successfully or not (or they have adjusted in some fashion). Until that is introduced in camera::start() API, we need to carry out such handling on the application side. Check the frame_duration whether it is in-bound to the max / min frame-duration supported by the camera. If the framer_duartion is out of bounds, bind the frame_duration between the max and min FrameDuration that is supported by the camera. Pass the initControls_ at the time of starting camera. Change in the existing code structure[RFC]: Shift the call to camera::cofigure() above the for loop which does not touch anything form libcamera just reads from it. This is done because the bounds checking for frame duration cannot be done without the call to camera::configure(). This will ease exposing the framerate to GStreamer. The framerate can be exposed with the same new CAPS event which is exposing the width, height and the colorimetry. Solve the complications in exposing the correct framerate due to loss in precision as a result of casting the frameduration to int64_t(which is required in libcamera to set the FrameDurationLimits control). Example - * Suppose the framerate requested is 35/1. The framerate is read form the caps in the from of fraction that has a numerator and denominator. * Converting to FrameDuration (in microseconds) (1 * 10^6) / 35 = 28571.4286 int64_t frame_duration = 28571 (the precision here is lost.) * To expose the framerate in caps, Inverting the frame_duration to get back the framerate and converting to Seconds. double framerate = 10^6 / 28571 and 10^6/28571 which is close but not equal to 35/1 will fail the negotiation. To solve the above problem, introduce a global std::pair variable that stores the numerator and denominator of the framerate in the negotiatied caps. Try to apply the framerate and check if the floor value of framerate that gets applied closely matches with the floor value framerate requested in the negotiatied caps. If the value matches set the framerate that is stored in the global container, else set the framerate to 0/1. --- Testing using fpsdisplaysink element on RPi 4 + OV5647. https://paste.debian.net/1252473/ https://paste.debian.net/1252474/ framerate out of max/min bounds supported by the camera: https://paste.debian.net/1252475/ --- Signed-off-by: Umang Jain Signed-off-by: Rishikesh Donadkar --- src/gstreamer/gstlibcamera-utils.cpp | 63 ++++++++++++++++++++++++++++ src/gstreamer/gstlibcamera-utils.h | 5 +++ src/gstreamer/gstlibcamerasrc.cpp | 27 +++++++----- 3 files changed, 85 insertions(+), 10 deletions(-) diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp index 4df5dd6c..d657be70 100644 --- a/src/gstreamer/gstlibcamera-utils.cpp +++ b/src/gstreamer/gstlibcamera-utils.cpp @@ -8,10 +8,13 @@ #include "gstlibcamera-utils.h" +#include #include using namespace libcamera; +static std::pair framerate_container = { -1, -1 }; + static struct { GstVideoFormat gst_format; PixelFormat format; @@ -405,6 +408,66 @@ gst_libcamera_configure_stream_from_caps(StreamConfiguration &stream_cfg, } } +void +gst_libcamera_get_init_controls_from_caps(ControlList &controls, [[maybe_unused]] GstCaps *caps) +{ + /* read framerate from caps - convert to integer and set to frame_time. */ + GstStructure *s = gst_caps_get_structure(caps, 0); + gint fps_n = -1, fps_d = -1; + + if (gst_structure_has_field_typed(s, "framerate", GST_TYPE_FRACTION)) { + if (!gst_structure_get_fraction(s, "framerate", &fps_n, &fps_d)) { + GST_WARNING("invalid framerate in the caps."); + return; + } + } + + if (fps_n < 0 || fps_d < 0) + return; + + framerate_container = { fps_n, fps_d }; + int64_t frame_duration = (fps_d * 1000000.0) / fps_n; + controls.set(controls::FrameDurationLimits, + Span({ frame_duration, frame_duration })); +} + +void +gst_libcamera_set_frame_duration_bounds(const ControlInfoMap &defaults, ControlList ¤t) +{ + int64_t min_frame_duration, max_frame_duration; + + auto iterFrameDuration = defaults.find(controls::FrameDurationLimits.id()); + if (!(iterFrameDuration != defaults.end() && + current.contains(controls::FRAME_DURATION_LIMITS))) + return; + + min_frame_duration = std::max(iterFrameDuration->second.min().get(), + current.get(controls::FrameDurationLimits).value()[0]); + max_frame_duration = std::min(iterFrameDuration->second.max().get(), + current.get(controls::FrameDurationLimits).value()[1]); + + current.set(controls::FrameDurationLimits, + Span({ min_frame_duration, max_frame_duration })); +} + +void +gst_libcamera_controllist_to_caps(const ControlList &controls, GstCaps *caps) +{ + gint numerator, denominator; + if (!controls.contains(controls::FRAME_DURATION_LIMITS)) + return; + + GstStructure *s = gst_caps_get_structure(caps, 0); + double framerate = 1000000 / static_cast(controls.get(controls::FrameDurationLimits).value()[0]); + gst_util_double_to_fraction(framerate, &numerator, &denominator); + + if (framerate_container.first / framerate_container.second == numerator / denominator) + gst_structure_set(s, "framerate", GST_TYPE_FRACTION, + framerate_container.first, framerate_container.second, nullptr); + else + gst_structure_set(s, "framerate", GST_TYPE_FRACTION, 0, 1, nullptr); +} + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task) diff --git a/src/gstreamer/gstlibcamera-utils.h b/src/gstreamer/gstlibcamera-utils.h index 164189a2..a3594942 100644 --- a/src/gstreamer/gstlibcamera-utils.h +++ b/src/gstreamer/gstlibcamera-utils.h @@ -9,6 +9,7 @@ #pragma once #include +#include #include #include @@ -18,6 +19,10 @@ GstCaps *gst_libcamera_stream_formats_to_caps(const libcamera::StreamFormats &fo GstCaps *gst_libcamera_stream_configuration_to_caps(const libcamera::StreamConfiguration &stream_cfg); void gst_libcamera_configure_stream_from_caps(libcamera::StreamConfiguration &stream_cfg, GstCaps *caps); +void gst_libcamera_get_init_controls_from_caps(libcamera::ControlList &controls, GstCaps *caps); +void gst_libcamera_set_frame_duration_bounds(const libcamera::ControlInfoMap &defaults, libcamera::ControlList ¤t); +void gst_libcamera_controllist_to_caps(const libcamera::ControlList &controls, GstCaps *caps); + #if !GST_CHECK_VERSION(1, 17, 1) gboolean gst_task_resume(GstTask *task); #endif diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index 16d70fea..ee9fa5e9 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -131,6 +131,7 @@ struct GstLibcameraSrcState { std::queue> completedRequests_ LIBCAMERA_TSA_GUARDED_BY(lock_); + ControlList initControls_; guint group_id_; int queueRequest(); @@ -496,6 +497,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, /* Retrieve the supported caps. */ g_autoptr(GstCaps) filter = gst_libcamera_stream_formats_to_caps(stream_cfg.formats()); g_autoptr(GstCaps) caps = gst_pad_peer_query_caps(srcpad, filter); + if (gst_caps_is_empty(caps)) { flow_ret = GST_FLOW_NOT_NEGOTIATED; break; @@ -504,6 +506,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, /* Fixate caps and configure the stream. */ caps = gst_caps_make_writable(caps); gst_libcamera_configure_stream_from_caps(stream_cfg, caps); + gst_libcamera_get_init_controls_from_caps(state->initControls_, caps); } if (flow_ret != GST_FLOW_OK) @@ -515,6 +518,16 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, goto done; } + ret = state->cam_->configure(state->config_.get()); + if (ret) { + GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, + ("Failed to configure camera: %s", g_strerror(-ret)), + ("Camera::configure() failed with error code %i", ret)); + gst_task_stop(task); + return; + } + gst_libcamera_set_frame_duration_bounds(state->cam_->controls(), state->initControls_); + /* * Regardless if it has been modified, create clean caps and push the * caps event. Downstream will decide if the caps are acceptable. @@ -524,6 +537,8 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, const StreamConfiguration &stream_cfg = state->config_->at(i); g_autoptr(GstCaps) caps = gst_libcamera_stream_configuration_to_caps(stream_cfg); + gst_libcamera_controllist_to_caps(state->initControls_, caps); + if (!gst_pad_push_event(srcpad, gst_event_new_caps(caps))) { flow_ret = GST_FLOW_NOT_NEGOTIATED; break; @@ -535,15 +550,6 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_pad_push_event(srcpad, gst_event_new_segment(&segment)); } - ret = state->cam_->configure(state->config_.get()); - if (ret) { - GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, - ("Failed to configure camera: %s", g_strerror(-ret)), - ("Camera::configure() failed with error code %i", ret)); - gst_task_stop(task); - return; - } - self->allocator = gst_libcamera_allocator_new(state->cam_, state->config_.get()); if (!self->allocator) { GST_ELEMENT_ERROR(self, RESOURCE, NO_SPACE_LEFT, @@ -566,7 +572,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_flow_combiner_add_pad(self->flow_combiner, srcpad); } - ret = state->cam_->start(); + ret = state->cam_->start(&state->initControls_); if (ret) { GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS, ("Failed to start the camera: %s", g_strerror(-ret)), @@ -576,6 +582,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, } done: + state->initControls_.clear(); switch (flow_ret) { case GST_FLOW_NOT_NEGOTIATED: GST_ELEMENT_FLOW_ERROR(self, flow_ret);