{"id":18459,"url":"https://patchwork.libcamera.org/api/1.1/patches/18459/?format=json","web_url":"https://patchwork.libcamera.org/patch/18459/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20230324181247.302586-2-nicolas@ndufresne.ca>","date":"2023-03-24T18:12:45","name":"[libcamera-devel,v2,1/3] gstreamer: Add sensor mode selection","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"aa44e8aa51d525772d8345e2235ba5890873683d","submitter":{"id":30,"url":"https://patchwork.libcamera.org/api/1.1/people/30/?format=json","name":"Nicolas Dufresne","email":"nicolas@ndufresne.ca"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/18459/mbox/","series":[{"id":3815,"url":"https://patchwork.libcamera.org/api/1.1/series/3815/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3815","date":"2023-03-24T18:12:44","name":"Add sensor mode selection to GStreamer","version":2,"mbox":"https://patchwork.libcamera.org/series/3815/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/18459/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/18459/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 916EAC3260\n\tfor ;\n\tFri, 24 Mar 2023 18:12:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 382296271E;\n\tFri, 24 Mar 2023 19:12:59 +0100 (CET)","from madras.collabora.co.uk (madras.collabora.co.uk\n\t[IPv6:2a00:1098:0:82:1000:25:2eeb:e5ab])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 505E962715\n\tfor ;\n\tFri, 24 Mar 2023 19:12:57 +0100 (CET)","from nicolas-tpx395.lan (192-222-136-102.qc.cable.ebox.net\n\t[192.222.136.102])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (4096 bits)\n\tserver-digest SHA256)\n\t(No client certificate requested) (Authenticated sender: nicolas)\n\tby madras.collabora.co.uk (Postfix) with ESMTPSA id C8751660312C;\n\tFri, 24 Mar 2023 18:12:56 +0000 (GMT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1679681579;\n\tbh=OD0ahQzIbwPZKN0O8W+b6n38WzK5KqZ4lBNlRtpsKmA=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=Ii7ikJpmjdJXNnTSpAxYbJBcqVPkGzwSZ7p4G0vs/fglfInyqe2uQ3feb1jyLHrrH\n\tElB8gOiUoe0wxJw+fZ3vE9dJq/ZcIWWvuef4kGjXKfjYjfeRGPwRc6x1hbVC4xhQc8\n\t3SOifsuQhdqpHXPvi3uHzrMgJJ4an0QY4DVHCcS8RJT5RP7748a+x1oaRbCylOv9Ng\n\tg+4xvpNVuCv+KtleqhQVIPkfjF1JZd28z+qIE+psm3hiU88GZXxo4Cn3HdHIKBgpUq\n\tsV8vf1jJGIVo9kx5QwkHR2TBZHlhOeUaKQfcpJYGoWGVyP4gbklCkpTqduen/8+YCl\n\thk7acnsN/Vecw==","To":"libcamera-devel@lists.libcamera.org","Date":"Fri, 24 Mar 2023 14:12:45 -0400","Message-Id":"<20230324181247.302586-2-nicolas@ndufresne.ca>","X-Mailer":"git-send-email 2.39.2","In-Reply-To":"<20230324181247.302586-1-nicolas@ndufresne.ca>","References":"<20230324181247.302586-1-nicolas@ndufresne.ca>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v2 1/3] gstreamer: Add sensor mode\n\tselection","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Nicolas Dufresne via libcamera-devel\n\t","Reply-To":"Nicolas Dufresne ","Cc":"Nicolas Dufresne ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"From: Nicolas Dufresne \n\nThis add support for selecting the sensor mode. A new read-only\nproperty called sensor-modes is filled when the element reaches\nREADY state. It contains the list of all available sensor modes,\nincluding the supported framerate range. This is exposed as GstCaps\nin the form of sensor/mode,width=X,height=Y,format=Y,framerate=[...].\nThe format string matches the libcamera format string representation.\n\nThe application can then select a mode using the read/write sensor-mode\ncontrol. The selected mode is also a caps, it will be intersected with\nthe supported mode and the \"best\" match will be picked. This allows\napplication to use simple filter when they want to pick a mode for lets\nsay a specific framerate (e.g. sensor/mode,framerate=60/1).\n\nSigned-off-by: Nicolas Dufresne \n---\n src/gstreamer/gstlibcamera-utils.cpp | 4 +\n src/gstreamer/gstlibcamerasrc.cpp | 110 ++++++++++++++++++++++++++-\n 2 files changed, 112 insertions(+), 2 deletions(-)","diff":"diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp\nindex 750ec351..c8a8df09 100644\n--- a/src/gstreamer/gstlibcamera-utils.cpp\n+++ b/src/gstreamer/gstlibcamera-utils.cpp\n@@ -416,6 +416,10 @@ gst_libcamera_configure_stream_from_caps(StreamConfiguration &stream_cfg,\n \t\tstream_cfg.pixelFormat = gst_format_to_pixel_format(gst_format);\n \t} else if (gst_structure_has_name(s, \"image/jpeg\")) {\n \t\tstream_cfg.pixelFormat = formats::MJPEG;\n+\t} else if (gst_structure_has_name(s, \"sensor/mode\")) {\n+\t\tgst_structure_fixate_field(s, \"format\");\n+\t\tconst gchar *format = gst_structure_get_string(s, \"format\");\n+\t\tstream_cfg.pixelFormat = PixelFormat::fromString(format);\n \t} else {\n \t\tg_critical(\"Unsupported media type: %s\", gst_structure_get_name(s));\n \t}\ndiff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp\nindex a10cbd4f..2f05a03f 100644\n--- a/src/gstreamer/gstlibcamerasrc.cpp\n+++ b/src/gstreamer/gstlibcamerasrc.cpp\n@@ -147,6 +147,9 @@ struct _GstLibcameraSrc {\n \n \tgchar *camera_name;\n \n+\tGstCaps *sensor_modes;\n+\tGstCaps *sensor_mode;\n+\n \tGstLibcameraSrcState *state;\n \tGstLibcameraAllocator *allocator;\n \tGstFlowCombiner *flow_combiner;\n@@ -154,7 +157,10 @@ struct _GstLibcameraSrc {\n \n enum {\n \tPROP_0,\n-\tPROP_CAMERA_NAME\n+\tPROP_CAMERA_NAME,\n+\tPROP_SENSOR_MODES,\n+\tPROP_SENSOR_MODE,\n+\tPROP_LAST\n };\n \n G_DEFINE_TYPE_WITH_CODE(GstLibcameraSrc, gst_libcamera_src, GST_TYPE_ELEMENT,\n@@ -318,6 +324,61 @@ int GstLibcameraSrcState::processRequest()\n \treturn err;\n }\n \n+static GstCaps *\n+gst_libcamera_src_enumerate_sensor_modes(GstLibcameraSrc *self)\n+{\n+\tGstCaps *modes = gst_caps_new_empty();\n+\tGstLibcameraSrcState *state = self->state;\n+\tauto config = state->cam_->generateConfiguration({ libcamera::StreamRole::Raw,\n+\t\t\t\t\t\t\t libcamera::StreamRole::VideoRecording });\n+\tif (config == nullptr) {\n+\t\tGST_DEBUG_OBJECT(self, \"Sensor mode selection is not supported, skipping enumeration.\");\n+\t\treturn modes;\n+\t}\n+\n+\tconst libcamera::StreamFormats &formats = config->at(0).formats();\n+\n+\tfor (const auto &pixfmt : formats.pixelformats()) {\n+\t\tfor (const auto &size : formats.sizes(pixfmt)) {\n+\t\t\tconfig->at(0).size = size;\n+\t\t\tconfig->at(0).pixelFormat = pixfmt;\n+\n+\t\t\tif (config->validate() == CameraConfiguration::Invalid)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (state->cam_->configure(config.get()))\n+\t\t\t\tcontinue;\n+\n+\t\t\tauto fd_ctrl = state->cam_->controls().find(&controls::FrameDurationLimits);\n+\t\t\tif (fd_ctrl == state->cam_->controls().end())\n+\t\t\t\tcontinue;\n+\n+\t\t\tint minrate_num, minrate_denom;\n+\t\t\tint maxrate_num, maxrate_denom;\n+\t\t\tdouble min_framerate = gst_util_guint64_to_gdouble(1.0e6) /\n+\t\t\t\t\t gst_util_guint64_to_gdouble(fd_ctrl->second.max().get());\n+\t\t\tdouble max_framerate = gst_util_guint64_to_gdouble(1.0e6) /\n+\t\t\t\t\t gst_util_guint64_to_gdouble(fd_ctrl->second.min().get());\n+\t\t\tgst_util_double_to_fraction(min_framerate, &minrate_num, &minrate_denom);\n+\t\t\tgst_util_double_to_fraction(max_framerate, &maxrate_num, &maxrate_denom);\n+\n+\t\t\tGstStructure *s = gst_structure_new(\"sensor/mode\",\n+\t\t\t\t\t\t\t \"format\", G_TYPE_STRING, pixfmt.toString().c_str(),\n+\t\t\t\t\t\t\t \"width\", G_TYPE_INT, size.width,\n+\t\t\t\t\t\t\t \"height\", G_TYPE_INT, size.height,\n+\t\t\t\t\t\t\t \"framerate\", GST_TYPE_FRACTION_RANGE,\n+\t\t\t\t\t\t\t minrate_num, minrate_denom,\n+\t\t\t\t\t\t\t maxrate_num, maxrate_denom,\n+\t\t\t\t\t\t\t nullptr);\n+\t\t\tgst_caps_append_structure(modes, s);\n+\t\t}\n+\t}\n+\n+\tGST_DEBUG_OBJECT(self, \"Camera sensor modes: %\" GST_PTR_FORMAT, modes);\n+\n+\treturn modes;\n+}\n+\n static bool\n gst_libcamera_src_open(GstLibcameraSrc *self)\n {\n@@ -375,6 +436,7 @@ gst_libcamera_src_open(GstLibcameraSrc *self)\n \t/* No need to lock here, we didn't start our threads yet. */\n \tself->state->cm_ = cm;\n \tself->state->cam_ = cam;\n+\tself->sensor_modes = gst_libcamera_src_enumerate_sensor_modes(self);\n \n \treturn true;\n }\n@@ -462,6 +524,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,\n \tGstLibcameraSrcState *state = self->state;\n \tGstFlowReturn flow_ret = GST_FLOW_OK;\n \tgint ret;\n+\tg_autoptr(GstCaps) sensor_mode = nullptr;\n \n \tg_autoptr(GstStructure) element_caps = gst_structure_new_empty(\"caps\");\n \n@@ -481,6 +544,16 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,\n \t\troles.push_back(gst_libcamera_pad_get_role(srcpad));\n \t}\n \n+\tif (!gst_caps_is_any(self->sensor_mode)) {\n+\t\tsensor_mode = gst_caps_intersect(self->sensor_mode, self->sensor_modes);\n+\t\tif (!gst_caps_is_empty(sensor_mode)) {\n+\t\t\troles.push_back(libcamera::StreamRole::Raw);\n+\t\t} else {\n+\t\t\tGST_WARNING_OBJECT(self, \"No sensor mode matching the selection, ignoring.\");\n+\t\t\tgst_clear_caps(&sensor_mode);\n+\t\t}\n+\t}\n+\n \t/* Generate the stream configurations, there should be one per pad. */\n \tstate->config_ = state->cam_->generateConfiguration(roles);\n \tif (state->config_ == nullptr) {\n@@ -490,7 +563,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,\n \t\tgst_task_stop(task);\n \t\treturn;\n \t}\n-\tg_assert(state->config_->size() == state->srcpads_.size());\n+\tg_assert(state->config_->size() == state->srcpads_.size() + (!!sensor_mode));\n \n \tfor (gsize i = 0; i < state->srcpads_.size(); i++) {\n \t\tGstPad *srcpad = state->srcpads_[i];\n@@ -510,6 +583,12 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,\n \t\tgst_libcamera_get_framerate_from_caps(caps, element_caps);\n \t}\n \n+\tif (sensor_mode) {\n+\t\tStreamConfiguration &stream_cfg = state->config_->at(state->srcpads_.size());\n+\t\tg_assert(gst_caps_is_writable(sensor_mode));\n+\t\tgst_libcamera_configure_stream_from_caps(stream_cfg, sensor_mode);\n+\t}\n+\n \tif (flow_ret != GST_FLOW_OK)\n \t\tgoto done;\n \n@@ -624,6 +703,7 @@ gst_libcamera_src_task_leave([[maybe_unused]] GstTask *task,\n \tg_clear_object(&self->allocator);\n \tg_clear_pointer(&self->flow_combiner,\n \t\t\t(GDestroyNotify)gst_flow_combiner_free);\n+\tgst_clear_caps(&self->sensor_modes);\n }\n \n static void\n@@ -659,6 +739,10 @@ gst_libcamera_src_set_property(GObject *object, guint prop_id,\n \t\tg_free(self->camera_name);\n \t\tself->camera_name = g_value_dup_string(value);\n \t\tbreak;\n+\tcase PROP_SENSOR_MODE:\n+\t\tgst_clear_caps(&self->sensor_mode);\n+\t\tself->sensor_mode = GST_CAPS(g_value_dup_boxed(value));\n+\t\tbreak;\n \tdefault:\n \t\tG_OBJECT_WARN_INVALID_PROPERTY_ID(object, prop_id, pspec);\n \t\tbreak;\n@@ -676,6 +760,12 @@ gst_libcamera_src_get_property(GObject *object, guint prop_id, GValue *value,\n \tcase PROP_CAMERA_NAME:\n \t\tg_value_set_string(value, self->camera_name);\n \t\tbreak;\n+\tcase PROP_SENSOR_MODES:\n+\t\tg_value_set_boxed(value, self->sensor_modes);\n+\t\tbreak;\n+\tcase PROP_SENSOR_MODE:\n+\t\tg_value_set_boxed(value, self->sensor_mode);\n+\t\tbreak;\n \tdefault:\n \t\tG_OBJECT_WARN_INVALID_PROPERTY_ID(object, prop_id, pspec);\n \t\tbreak;\n@@ -763,6 +853,7 @@ gst_libcamera_src_init(GstLibcameraSrc *self)\n \t/* C-style friend. */\n \tstate->src_ = self;\n \tself->state = state;\n+\tself->sensor_mode = gst_caps_new_any();\n }\n \n static GstPad *\n@@ -844,4 +935,19 @@ gst_libcamera_src_class_init(GstLibcameraSrcClass *klass)\n \t\t\t\t\t\t\t | G_PARAM_READWRITE\n \t\t\t\t\t\t\t | G_PARAM_STATIC_STRINGS));\n \tg_object_class_install_property(object_class, PROP_CAMERA_NAME, spec);\n+\n+\tspec = g_param_spec_boxed(\"sensor-modes\", \"Sensor Modes\",\n+\t\t\t\t \"GstCaps representing available sensor modes.\",\n+\t\t\t\t GST_TYPE_CAPS,\n+\t\t\t\t (GParamFlags)(G_PARAM_READABLE\n+\t\t\t\t\t | G_PARAM_STATIC_STRINGS));\n+\tg_object_class_install_property(object_class, PROP_SENSOR_MODES, spec);\n+\n+\tspec = g_param_spec_boxed(\"sensor-mode\", \"Sensor Mode\",\n+\t\t\t\t \"GstCaps representing selected sensor mode.\",\n+\t\t\t\t GST_TYPE_CAPS,\n+\t\t\t\t (GParamFlags)(GST_PARAM_MUTABLE_READY\n+\t\t\t\t\t | G_PARAM_READWRITE\n+\t\t\t\t\t | G_PARAM_STATIC_STRINGS));\n+\tg_object_class_install_property(object_class, PROP_SENSOR_MODE, spec);\n }\n","prefixes":["libcamera-devel","v2","1/3"]}