From patchwork Fri Mar 24 17:40:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Dufresne X-Patchwork-Id: 18455 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B63ABC3260 for ; Fri, 24 Mar 2023 17:40:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 49AAA6271E; Fri, 24 Mar 2023 18:40:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1679679626; bh=3TBrSS9uw2wn1TRoj9H+1NJrUcoOVikNvu1z7r6yHD4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=qf5FBCVci8eFguyakP0Hv8DMuKAx82aXVryHvvCoAF3GgsXPIRaHZxHZRm6PsDN8J ExGUuGJLb4TBjqD97aie5JkJJCq9mHz3OQrQUmx6VZD6pYgPJT/YxVYnvfg1+fChUV tZauO9HFHgUlNzkwrffbROpZ8yyjO9LRkUKlOFxeKWvxdRXjf5UGwakHIMS20C7ptI 8SLHXoYjhYeXt8AvPaS1iYklOt+PCxh0WmnHhlbw11mSluyIhCcIWzV0PvP+E5Ah90 9ZhyiGmRXlGNrYwef7ZPWm67Rgn1z4n5N1sWrXVunOb8/F+d/mqDUjESsIEFB108H2 qNjRYlDmzFllw== Received: from madras.collabora.co.uk (madras.collabora.co.uk [IPv6:2a00:1098:0:82:1000:25:2eeb:e5ab]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D8B4C603AA for ; Fri, 24 Mar 2023 18:40:23 +0100 (CET) Received: from nicolas-tpx395.lan (192-222-136-102.qc.cable.ebox.net [192.222.136.102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: nicolas) by madras.collabora.co.uk (Postfix) with ESMTPSA id 64FA6660312E; Fri, 24 Mar 2023 17:40:23 +0000 (GMT) To: libcamera-devel@lists.libcamera.org Date: Fri, 24 Mar 2023 13:40:07 -0400 Message-Id: <20230324174009.300123-2-nicolas@ndufresne.ca> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230324174009.300123-1-nicolas@ndufresne.ca> References: <20230324174009.300123-1-nicolas@ndufresne.ca> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 1/3] gstreamer: Add sensor mode selection X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Nicolas Dufresne via libcamera-devel From: Nicolas Dufresne Reply-To: Nicolas Dufresne Cc: Nicolas Dufresne Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Nicolas Dufresne This add support for selecting the sensor mode. A new read-only property called sensor-modes is filled when the element reaches READY state. It contains the list of all available sensor modes, including the supported framerate range. This is exposed as GstCaps in the form of sensor/mode,width=X,height=Y,format=Y,framerate=[...]. The format string matches the libcamera format string representation. The application can then select a mode using the read/write sensor-mode control. The selected mode is also a caps, it will be intersected with the supported mode and the "best" match will be picked. This allows application to use simple filter when they want to pick a mode for lets say a specific framerate (e.g. sensor/mode,framerate=60/1). Signed-off-by: Nicolas Dufresne --- src/gstreamer/gstlibcamera-utils.cpp | 4 + src/gstreamer/gstlibcamerasrc.cpp | 123 ++++++++++++++++++++++++++- 2 files changed, 126 insertions(+), 1 deletion(-) diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp index 750ec351..c8a8df09 100644 --- a/src/gstreamer/gstlibcamera-utils.cpp +++ b/src/gstreamer/gstlibcamera-utils.cpp @@ -416,6 +416,10 @@ gst_libcamera_configure_stream_from_caps(StreamConfiguration &stream_cfg, stream_cfg.pixelFormat = gst_format_to_pixel_format(gst_format); } else if (gst_structure_has_name(s, "image/jpeg")) { stream_cfg.pixelFormat = formats::MJPEG; + } else if (gst_structure_has_name(s, "sensor/mode")) { + gst_structure_fixate_field(s, "format"); + const gchar *format = gst_structure_get_string(s, "format"); + stream_cfg.pixelFormat = PixelFormat::fromString(format); } else { g_critical("Unsupported media type: %s", gst_structure_get_name(s)); } diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp index a10cbd4f..c448a9fe 100644 --- a/src/gstreamer/gstlibcamerasrc.cpp +++ b/src/gstreamer/gstlibcamerasrc.cpp @@ -147,6 +147,15 @@ struct _GstLibcameraSrc { gchar *camera_name; +<<<<<<< HEAD +======= + gboolean hflip; + gboolean vflip; + + GstCaps *sensor_modes; + GstCaps *sensor_mode; + +>>>>>>> 210f0a67 (gstreamer: Add sensor mode selection) GstLibcameraSrcState *state; GstLibcameraAllocator *allocator; GstFlowCombiner *flow_combiner; @@ -154,7 +163,16 @@ struct _GstLibcameraSrc { enum { PROP_0, +<<<<<<< HEAD PROP_CAMERA_NAME +======= + PROP_CAMERA_NAME, + PROP_HFLIP, + PROP_VFLIP, + PROP_SENSOR_MODES, + PROP_SENSOR_MODE, + PROP_LAST +>>>>>>> 210f0a67 (gstreamer: Add sensor mode selection) }; G_DEFINE_TYPE_WITH_CODE(GstLibcameraSrc, gst_libcamera_src, GST_TYPE_ELEMENT, @@ -318,6 +336,61 @@ int GstLibcameraSrcState::processRequest() return err; } +static GstCaps * +gst_libcamera_src_enumerate_sensor_modes(GstLibcameraSrc *self) +{ + GstCaps *modes = gst_caps_new_empty(); + GstLibcameraSrcState *state = self->state; + auto config = state->cam_->generateConfiguration({ libcamera::StreamRole::Raw, + libcamera::StreamRole::VideoRecording }); + if (config == nullptr) { + GST_DEBUG_OBJECT(self, "Sensor mode selection is not supported, skipping enumeration."); + return modes; + } + + const libcamera::StreamFormats &formats = config->at(0).formats(); + + for (const auto &pixfmt : formats.pixelformats()) { + for (const auto &size : formats.sizes(pixfmt)) { + config->at(0).size = size; + config->at(0).pixelFormat = pixfmt; + + if (config->validate() == CameraConfiguration::Invalid) + continue; + + if (state->cam_->configure(config.get())) + continue; + + auto fd_ctrl = state->cam_->controls().find(&controls::FrameDurationLimits); + if (fd_ctrl == state->cam_->controls().end()) + continue; + + int minrate_num, minrate_denom; + int maxrate_num, maxrate_denom; + double min_framerate = gst_util_guint64_to_gdouble(1.0e6) / + gst_util_guint64_to_gdouble(fd_ctrl->second.max().get()); + double max_framerate = gst_util_guint64_to_gdouble(1.0e6) / + gst_util_guint64_to_gdouble(fd_ctrl->second.min().get()); + gst_util_double_to_fraction(min_framerate, &minrate_num, &minrate_denom); + gst_util_double_to_fraction(max_framerate, &maxrate_num, &maxrate_denom); + + GstStructure *s = gst_structure_new("sensor/mode", + "format", G_TYPE_STRING, pixfmt.toString().c_str(), + "width", G_TYPE_INT, size.width, + "height", G_TYPE_INT, size.height, + "framerate", GST_TYPE_FRACTION_RANGE, + minrate_num, minrate_denom, + maxrate_num, maxrate_denom, + nullptr); + gst_caps_append_structure(modes, s); + } + } + + GST_DEBUG_OBJECT(self, "Camera sensor modes: %" GST_PTR_FORMAT, modes); + + return modes; +} + static bool gst_libcamera_src_open(GstLibcameraSrc *self) { @@ -375,6 +448,7 @@ gst_libcamera_src_open(GstLibcameraSrc *self) /* No need to lock here, we didn't start our threads yet. */ self->state->cm_ = cm; self->state->cam_ = cam; + self->sensor_modes = gst_libcamera_src_enumerate_sensor_modes(self); return true; } @@ -462,6 +536,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, GstLibcameraSrcState *state = self->state; GstFlowReturn flow_ret = GST_FLOW_OK; gint ret; + g_autoptr(GstCaps) sensor_mode = nullptr; g_autoptr(GstStructure) element_caps = gst_structure_new_empty("caps"); @@ -481,6 +556,16 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, roles.push_back(gst_libcamera_pad_get_role(srcpad)); } + if (!gst_caps_is_any(self->sensor_mode)) { + sensor_mode = gst_caps_intersect(self->sensor_mode, self->sensor_modes); + if (!gst_caps_is_empty(sensor_mode)) { + roles.push_back(libcamera::StreamRole::Raw); + } else { + GST_WARNING_OBJECT(self, "No sensor mode matching the selection, ignoring."); + gst_clear_caps(&sensor_mode); + } + } + /* Generate the stream configurations, there should be one per pad. */ state->config_ = state->cam_->generateConfiguration(roles); if (state->config_ == nullptr) { @@ -490,7 +575,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_task_stop(task); return; } - g_assert(state->config_->size() == state->srcpads_.size()); + g_assert(state->config_->size() == state->srcpads_.size() + (!!sensor_mode)); for (gsize i = 0; i < state->srcpads_.size(); i++) { GstPad *srcpad = state->srcpads_[i]; @@ -510,6 +595,12 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread, gst_libcamera_get_framerate_from_caps(caps, element_caps); } + if (sensor_mode) { + StreamConfiguration &stream_cfg = state->config_->at(state->srcpads_.size()); + g_assert(gst_caps_is_writable(sensor_mode)); + gst_libcamera_configure_stream_from_caps(stream_cfg, sensor_mode); + } + if (flow_ret != GST_FLOW_OK) goto done; @@ -624,6 +715,7 @@ gst_libcamera_src_task_leave([[maybe_unused]] GstTask *task, g_clear_object(&self->allocator); g_clear_pointer(&self->flow_combiner, (GDestroyNotify)gst_flow_combiner_free); + gst_clear_caps(&self->sensor_modes); } static void @@ -659,6 +751,10 @@ gst_libcamera_src_set_property(GObject *object, guint prop_id, g_free(self->camera_name); self->camera_name = g_value_dup_string(value); break; + case PROP_SENSOR_MODE: + gst_clear_caps(&self->sensor_mode); + self->sensor_mode = GST_CAPS(g_value_dup_boxed(value)); + break; default: G_OBJECT_WARN_INVALID_PROPERTY_ID(object, prop_id, pspec); break; @@ -676,6 +772,12 @@ gst_libcamera_src_get_property(GObject *object, guint prop_id, GValue *value, case PROP_CAMERA_NAME: g_value_set_string(value, self->camera_name); break; + case PROP_SENSOR_MODES: + g_value_set_boxed(value, self->sensor_modes); + break; + case PROP_SENSOR_MODE: + g_value_set_boxed(value, self->sensor_mode); + break; default: G_OBJECT_WARN_INVALID_PROPERTY_ID(object, prop_id, pspec); break; @@ -763,6 +865,7 @@ gst_libcamera_src_init(GstLibcameraSrc *self) /* C-style friend. */ state->src_ = self; self->state = state; + self->sensor_mode = gst_caps_new_any(); } static GstPad * @@ -844,4 +947,22 @@ gst_libcamera_src_class_init(GstLibcameraSrcClass *klass) | G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)); g_object_class_install_property(object_class, PROP_CAMERA_NAME, spec); + + spec = g_param_spec_boxed("sensor-modes", "Sensor Modes", + "GstCaps representing available sensor modes.", + GST_TYPE_CAPS, + (GParamFlags)(G_PARAM_READABLE + | G_PARAM_STATIC_STRINGS)); + g_object_class_install_property(object_class, PROP_SENSOR_MODES, spec); + + spec = g_param_spec_boxed("sensor-mode", "Sensor Mode", + "GstCaps representing selected sensor mode.", + GST_TYPE_CAPS, + (GParamFlags)(GST_PARAM_MUTABLE_READY + | G_PARAM_READWRITE + | G_PARAM_STATIC_STRINGS)); + g_object_class_install_property(object_class, PROP_SENSOR_MODE, spec); + + + GstCameraControls::installProperties(object_class, PROP_LAST); } From patchwork Fri Mar 24 17:40:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Dufresne X-Patchwork-Id: 18456 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CEB14C0F1B for ; Fri, 24 Mar 2023 17:40:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B11226273E; Fri, 24 Mar 2023 18:40:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1679679626; bh=j+n25JxjeqYb70RPV0hDqHUcYZb2xmnVd87QWprcmGM=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=CIvDhWh2NnV+ivnhycqWq3bL006Qwz/0kYBXpaRZs3fDDMgDs+k/wO80+UPaIFb+Y yXZOYGAkr1SHDD3zgXihyYEbdJP1QyJB/N5Fm0lHJuap9MTBlEbjDhIe1NsMZco72o 77f3V5bVluRZ6Ld5YKFPG3lhyYx6NxkdKDdQr5log19uVnV/0pjfHw9OXybd5zWE33 ga2cayxi7LmL8YgIb78uF6pLbNg7K8WoR2SPs2m8NNdtcN06cg2L6bkt0BX0X44Ul+ /DjvdogESlod3Ja7pAEJOUbhoPouW/Wn9zoTePNgvxvXMUOpiu2B7ZkwE0/ltCZtzJ jPyDnawy+tU1Q== Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 65A83603AA for ; Fri, 24 Mar 2023 18:40:24 +0100 (CET) Received: from nicolas-tpx395.lan (192-222-136-102.qc.cable.ebox.net [192.222.136.102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: nicolas) by madras.collabora.co.uk (Postfix) with ESMTPSA id E99C2660312C; Fri, 24 Mar 2023 17:40:23 +0000 (GMT) To: libcamera-devel@lists.libcamera.org Date: Fri, 24 Mar 2023 13:40:08 -0400 Message-Id: <20230324174009.300123-3-nicolas@ndufresne.ca> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230324174009.300123-1-nicolas@ndufresne.ca> References: <20230324174009.300123-1-nicolas@ndufresne.ca> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 2/3] doc: gstreamer: Add missing queues X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Nicolas Dufresne via libcamera-devel From: Nicolas Dufresne Reply-To: Nicolas Dufresne Cc: Nicolas Dufresne Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Nicolas Dufresne As libcamerasrc reports some latency, a queue is needed in order to store the data in case the buffers has been produced slightly ahead of our reported latency. Signed-off-by: Nicolas Dufresne --- README.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 34b6b49f..52cde91c 100644 --- a/README.rst +++ b/README.rst @@ -133,7 +133,7 @@ onto the OpenGL accelerated display element on your system. .. code:: - gst-launch-1.0 libcamerasrc camera-name="Camera 1" ! glimagesink + gst-launch-1.0 libcamerasrc camera-name="Camera 1" ! queue ! glimagesink To show the first camera found you can omit the camera-name property, or you can list the cameras and their capabilities using: @@ -164,7 +164,7 @@ the following example could be used as a starting point: gst-launch-1.0 libcamerasrc ! \ video/x-raw,colorimetry=bt709,format=NV12,width=1280,height=720,framerate=30/1 ! \ - jpegenc ! multipartmux ! \ + queue ! jpegenc ! multipartmux ! \ tcpserversink host=0.0.0.0 port=5000 Which can be received on another device over the network with: From patchwork Fri Mar 24 17:40:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Dufresne X-Patchwork-Id: 18457 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1D8EEC329C for ; Fri, 24 Mar 2023 17:40:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 204726271F; Fri, 24 Mar 2023 18:40:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1679679627; bh=AG+K99daHMuCR/dO/FlC82WLKrNzoMeLd8FlNJLpYW0=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=sR9ntVcz/HNT3HpN0jzl3QSbSS2PVpJLXbUgkP8oJt60HwT1BlFnGunNkxtWE5Gzy B/rfFV/S6MHN+kdTZFSTOWwnVxUU/oMEekFSoCzN1V86InGnNlV5worTTPZ/O4TwQu xCi8IR0ArOrbZKUt2gwQGV7oEeT3ORSYJONBRunjzwnDNiTrBRca8O0+MeF3Ccsq4e FNkEMnoAUdpMwoIAmFvseuuEDyvd1G9vPOgYFcwnQxxsI5y56cldCbCAliAKaiQlZG w36570XmW89F5Vwm892kATgp3SABdnfWntsY0Lf3kwEBIw006bAzWr6BooK9Trtrux g47PGdSoZLD6w== Received: from madras.collabora.co.uk (madras.collabora.co.uk [IPv6:2a00:1098:0:82:1000:25:2eeb:e5ab]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F2FCF603AA for ; Fri, 24 Mar 2023 18:40:24 +0100 (CET) Received: from nicolas-tpx395.lan (192-222-136-102.qc.cable.ebox.net [192.222.136.102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: nicolas) by madras.collabora.co.uk (Postfix) with ESMTPSA id 80AA0660312C; Fri, 24 Mar 2023 17:40:24 +0000 (GMT) To: libcamera-devel@lists.libcamera.org Date: Fri, 24 Mar 2023 13:40:09 -0400 Message-Id: <20230324174009.300123-4-nicolas@ndufresne.ca> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230324174009.300123-1-nicolas@ndufresne.ca> References: <20230324174009.300123-1-nicolas@ndufresne.ca> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 3/3] doc: gstreamer: Document the sensor-mode(s) properties X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Nicolas Dufresne via libcamera-devel From: Nicolas Dufresne Reply-To: Nicolas Dufresne Cc: Nicolas Dufresne Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Nicolas Dufresne Signed-off-by: Nicolas Dufresne --- README.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.rst b/README.rst index 52cde91c..21a4d127 100644 --- a/README.rst +++ b/README.rst @@ -174,6 +174,20 @@ Which can be received on another device over the network with: gst-launch-1.0 tcpclientsrc host=$DEVICE_IP port=5000 ! \ multipartdemux ! jpegdec ! autovideosink +Some pipeline managers have support for selecting the sensor mode. This consist +of locking the sensor into a specific resolution/format in order to obtains a +specific field or view, quality or get access to different frame duration +ranges. Applications can read the enumerate mode through the ``sensor-modes`` +properties and select (or filter) the selected mode using ``sensor-mode`` +property. As an example, an application that needs 60 frame per second can +ensure this with: + +.. code:: + + gst-launch-1.0 libcamerasrc sensor-mode="sensor/mode,framerate=60/1" ! \ + video/x-raw,framerate=60/1,format=NV12 ! \ + queue ! videoconvert ! autovideosink + .. section-end-getting-started Troubleshooting