[{"id":11491,"web_url":"https://patchwork.libcamera.org/comment/11491/","msgid":"<e80fc0e0-d7f4-c77d-f98a-b7d86d773298@ideasonboard.com>","date":"2020-07-22T13:49:38","subject":"Re: [libcamera-devel] media: vimc: VIMC multiple streams","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Kaaira,\n\nOn 22/07/2020 14:43, Kaaira Gupta wrote:\n> This is a patchset by Niklas (https://patchwork.kernel.org/cover/10948831/)\n> that I have rebased on the current media-tree, I have not added anything\n> to it and it is just a compressed version of his patchset into a single\n> patch, based on the latest master to aid testing multiple stream support\n> in the VIMC pipeline.\n> \n> Signed-off-by: Kaaira Gupta <kgupta@es.iitr.ac.in>\n\nGreat, it's good to hear that Niklas' patches still work and are simple\nto rebase.\n\nI think his patches would need to be kept separate though, and submitted\nto the linux kernel through the linux-media mailing list, with any\ncomments from the previous series addressed.\n\nYou can keep Niklas' patches and his authorship, but add your signed-off\nby tag, and document any changes you make in the commit message before\nresubmitting to linux-media.\n\n\nThen when we get to a point that the kernel can support multiple\nstreams, we can then enable that support in libcamera (we might have to\nadd a variable that enables multiple streams based on the kernel\nversion, but lets see how it goes).\n\n\n--\nKieran\n\n\n\n> ---\n>  .../media/test-drivers/vimc/vimc-capture.c    | 35 ++++++++++++++++++-\n>  .../media/test-drivers/vimc/vimc-debayer.c    |  8 +++++\n>  drivers/media/test-drivers/vimc/vimc-scaler.c |  8 +++++\n>  drivers/media/test-drivers/vimc/vimc-sensor.c |  8 +++++\n>  .../media/test-drivers/vimc/vimc-streamer.c   | 22 +++++++-----\n>  5 files changed, 71 insertions(+), 10 deletions(-)\n> \n> diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c\n> index c63496b17b9a..5fc56582af83 100644\n> --- a/drivers/media/test-drivers/vimc/vimc-capture.c\n> +++ b/drivers/media/test-drivers/vimc/vimc-capture.c\n> @@ -237,16 +237,49 @@ static void vimc_cap_return_all_buffers(struct vimc_cap_device *vcap,\n>  \tspin_unlock(&vcap->qlock);\n>  }\n>  \n> +static struct media_entity *vimc_cap_get_sensor(struct vimc_cap_device *vcap)\n> +{\n> +\tstruct media_entity *entity = &vcap->vdev.entity;\n> +\tstruct media_device *mdev = entity->graph_obj.mdev;\n> +\tstruct media_graph graph;\n> +\n> +\tmutex_lock(&mdev->graph_mutex);\n> +\tif (media_graph_walk_init(&graph, mdev)) {\n> +\t\tmutex_unlock(&mdev->graph_mutex);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tmedia_graph_walk_start(&graph, entity);\n> +\n> +\twhile ((entity = media_graph_walk_next(&graph)))\n> +\t\tif (entity->function == MEDIA_ENT_F_CAM_SENSOR)\n> +\t\t\tbreak;\n> +\n> +\tmutex_unlock(&mdev->graph_mutex);\n> +\n> +\tmedia_graph_walk_cleanup(&graph);\n> +\n> +\treturn entity;\n> +}\n> +\n>  static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)\n>  {\n>  \tstruct vimc_cap_device *vcap = vb2_get_drv_priv(vq);\n>  \tstruct media_entity *entity = &vcap->vdev.entity;\n> +\tstruct media_pipeline *pipe = NULL;\n> +\tstruct media_entity *sensorent;\n>  \tint ret;\n>  \n>  \tvcap->sequence = 0;\n>  \n>  \t/* Start the media pipeline */\n> -\tret = media_pipeline_start(entity, &vcap->stream.pipe);\n> +\tsensorent = vimc_cap_get_sensor(vcap);\n> +\tif (sensorent && sensorent->pipe)\n> +\t\tpipe = sensorent->pipe;\n> +\telse\n> +\t\tpipe = &vcap->stream.pipe;\n> +\n> +\tret = media_pipeline_start(entity, pipe);\n>  \tif (ret) {\n>  \t\tvimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);\n>  \t\treturn ret;\n> diff --git a/drivers/media/test-drivers/vimc/vimc-debayer.c b/drivers/media/test-drivers/vimc/vimc-debayer.c\n> index c3f6fef34f68..93fe19d8d2b4 100644\n> --- a/drivers/media/test-drivers/vimc/vimc-debayer.c\n> +++ b/drivers/media/test-drivers/vimc/vimc-debayer.c\n> @@ -29,6 +29,7 @@ struct vimc_deb_pix_map {\n>  struct vimc_deb_device {\n>  \tstruct vimc_ent_device ved;\n>  \tstruct v4l2_subdev sd;\n> +\tatomic_t use_count;\n>  \t/* The active format */\n>  \tstruct v4l2_mbus_framefmt sink_fmt;\n>  \tu32 src_code;\n> @@ -343,6 +344,9 @@ static int vimc_deb_s_stream(struct v4l2_subdev *sd, int enable)\n>  \t\tconst struct vimc_pix_map *vpix;\n>  \t\tunsigned int frame_size;\n>  \n> +\t\tif (atomic_inc_return(&vdeb->use_count) != 1)\n> +\t\t\treturn 0;\n> +\n>  \t\tif (vdeb->src_frame)\n>  \t\t\treturn 0;\n>  \n> @@ -368,6 +372,9 @@ static int vimc_deb_s_stream(struct v4l2_subdev *sd, int enable)\n>  \t\t\treturn -ENOMEM;\n>  \n>  \t} else {\n> +\t\tif (atomic_dec_return(&vdeb->use_count) != 0)\n> +\t\t\treturn 0;\n> +\n>  \t\tif (!vdeb->src_frame)\n>  \t\t\treturn 0;\n>  \n> @@ -595,6 +602,7 @@ static struct vimc_ent_device *vimc_deb_add(struct vimc_device *vimc,\n>  \tvdeb->ved.process_frame = vimc_deb_process_frame;\n>  \tvdeb->ved.dev = vimc->mdev.dev;\n>  \tvdeb->mean_win_size = vimc_deb_ctrl_mean_win_size.def;\n> +\tatomic_set(&vdeb->use_count, 0);\n>  \n>  \t/* Initialize the frame format */\n>  \tvdeb->sink_fmt = sink_fmt_default;\n> diff --git a/drivers/media/test-drivers/vimc/vimc-scaler.c b/drivers/media/test-drivers/vimc/vimc-scaler.c\n> index 121fa7d62a2e..9b8458dbe57c 100644\n> --- a/drivers/media/test-drivers/vimc/vimc-scaler.c\n> +++ b/drivers/media/test-drivers/vimc/vimc-scaler.c\n> @@ -25,6 +25,7 @@ MODULE_PARM_DESC(sca_mult, \" the image size multiplier\");\n>  struct vimc_sca_device {\n>  \tstruct vimc_ent_device ved;\n>  \tstruct v4l2_subdev sd;\n> +\tatomic_t use_count;\n>  \t/* NOTE: the source fmt is the same as the sink\n>  \t * with the width and hight multiplied by mult\n>  \t */\n> @@ -340,6 +341,9 @@ static int vimc_sca_s_stream(struct v4l2_subdev *sd, int enable)\n>  \t\tconst struct vimc_pix_map *vpix;\n>  \t\tunsigned int frame_size;\n>  \n> +\t\tif (atomic_inc_return(&vsca->use_count) != 1)\n> +\t\t\treturn 0;\n> +\n>  \t\tif (vsca->src_frame)\n>  \t\t\treturn 0;\n>  \n> @@ -363,6 +367,9 @@ static int vimc_sca_s_stream(struct v4l2_subdev *sd, int enable)\n>  \t\t\treturn -ENOMEM;\n>  \n>  \t} else {\n> +\t\tif (atomic_dec_return(&vsca->use_count) != 0)\n> +\t\t\treturn 0;\n> +\n>  \t\tif (!vsca->src_frame)\n>  \t\t\treturn 0;\n>  \n> @@ -506,6 +513,7 @@ static struct vimc_ent_device *vimc_sca_add(struct vimc_device *vimc,\n>  \n>  \tvsca->ved.process_frame = vimc_sca_process_frame;\n>  \tvsca->ved.dev = vimc->mdev.dev;\n> +\tatomic_set(&vsca->use_count, 0);\n>  \n>  \t/* Initialize the frame format */\n>  \tvsca->sink_fmt = sink_fmt_default;\n> diff --git a/drivers/media/test-drivers/vimc/vimc-sensor.c b/drivers/media/test-drivers/vimc/vimc-sensor.c\n> index a2f09ac9a360..d68113886290 100644\n> --- a/drivers/media/test-drivers/vimc/vimc-sensor.c\n> +++ b/drivers/media/test-drivers/vimc/vimc-sensor.c\n> @@ -18,6 +18,7 @@ struct vimc_sen_device {\n>  \tstruct vimc_ent_device ved;\n>  \tstruct v4l2_subdev sd;\n>  \tstruct tpg_data tpg;\n> +\tatomic_t use_count;\n>  \tu8 *frame;\n>  \t/* The active format */\n>  \tstruct v4l2_mbus_framefmt mbus_format;\n> @@ -201,6 +202,9 @@ static int vimc_sen_s_stream(struct v4l2_subdev *sd, int enable)\n>  \t\tconst struct vimc_pix_map *vpix;\n>  \t\tunsigned int frame_size;\n>  \n> +\t\tif (atomic_inc_return(&vsen->use_count) != 1)\n> +\t\t\treturn 0;\n> +\n>  \t\t/* Calculate the frame size */\n>  \t\tvpix = vimc_pix_map_by_code(vsen->mbus_format.code);\n>  \t\tframe_size = vsen->mbus_format.width * vpix->bpp *\n> @@ -219,6 +223,9 @@ static int vimc_sen_s_stream(struct v4l2_subdev *sd, int enable)\n>  \n>  \t} else {\n>  \n> +\t\tif (atomic_dec_return(&vsen->use_count) != 0)\n> +\t\t\treturn 0;\n> +\n>  \t\tvfree(vsen->frame);\n>  \t\tvsen->frame = NULL;\n>  \t}\n> @@ -359,6 +366,7 @@ static struct vimc_ent_device *vimc_sen_add(struct vimc_device *vimc,\n>  \n>  \tvsen->ved.process_frame = vimc_sen_process_frame;\n>  \tvsen->ved.dev = vimc->mdev.dev;\n> +\tatomic_set(&vsen->use_count, 0);\n>  \n>  \t/* Initialize the frame format */\n>  \tvsen->mbus_format = fmt_default;\n> diff --git a/drivers/media/test-drivers/vimc/vimc-streamer.c b/drivers/media/test-drivers/vimc/vimc-streamer.c\n> index 451a32c0d034..d5b966a99f7b 100644\n> --- a/drivers/media/test-drivers/vimc/vimc-streamer.c\n> +++ b/drivers/media/test-drivers/vimc/vimc-streamer.c\n> @@ -192,33 +192,35 @@ int vimc_streamer_s_stream(struct vimc_stream *stream,\n>  \t\t\t   struct vimc_ent_device *ved,\n>  \t\t\t   int enable)\n>  {\n> +\tstatic DEFINE_MUTEX(vimc_streamer_lock);\n>  \tint ret;\n>  \n>  \tif (!stream || !ved)\n>  \t\treturn -EINVAL;\n>  \n> +\tret = mutex_lock_interruptible(&vimc_streamer_lock);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n>  \tif (enable) {\n>  \t\tif (stream->kthread)\n> -\t\t\treturn 0;\n> +\t\t\tgoto out;\n>  \n>  \t\tret = vimc_streamer_pipeline_init(stream, ved);\n>  \t\tif (ret)\n> -\t\t\treturn ret;\n> +\t\t\tgoto out;\n>  \n>  \t\tstream->kthread = kthread_run(vimc_streamer_thread, stream,\n>  \t\t\t\t\t      \"vimc-streamer thread\");\n>  \n>  \t\tif (IS_ERR(stream->kthread)) {\n>  \t\t\tret = PTR_ERR(stream->kthread);\n> -\t\t\tdev_err(ved->dev, \"kthread_run failed with %d\\n\", ret);\n> -\t\t\tvimc_streamer_pipeline_terminate(stream);\n> -\t\t\tstream->kthread = NULL;\n> -\t\t\treturn ret;\n> +\t\t\tgoto out;\n>  \t\t}\n>  \n>  \t} else {\n>  \t\tif (!stream->kthread)\n> -\t\t\treturn 0;\n> +\t\t\tgoto out;\n>  \n>  \t\tret = kthread_stop(stream->kthread);\n>  \t\t/*\n> @@ -228,12 +230,14 @@ int vimc_streamer_s_stream(struct vimc_stream *stream,\n>  \t\t * pipeline.\n>  \t\t */\n>  \t\tif (ret)\n> -\t\t\tdev_dbg(ved->dev, \"kthread_stop returned '%d'\\n\", ret);\n> +\t\t\tgoto out;\n>  \n>  \t\tstream->kthread = NULL;\n>  \n>  \t\tvimc_streamer_pipeline_terminate(stream);\n>  \t}\n> +out:\n> +\tmutex_unlock(&vimc_streamer_lock);\n>  \n> -\treturn 0;\n> +\treturn ret;\n>  }\n>","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B324BC2E68\n\tfor <parsemail@patchwork.libcamera.org>;\n\tWed, 22 Jul 2020 13:49:44 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2656060983;\n\tWed, 22 Jul 2020 15:49:44 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6A87E6053C\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed, 22 Jul 2020 15:49:42 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id C9FDC329;\n\tWed, 22 Jul 2020 15:49:41 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"ORAv1pCz\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1595425782;\n\tbh=60en5oYW4ntHYHiC/ROkv/dq+Ao1d6sDg+0Nj6xyjVI=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=ORAv1pCzdY04JLx0IShfi/roqc4OIcrzRRLhbVqpCMWy8js8Uz/YwCcOrJgoEeiqO\n\tTwZfly6feKcPvWsfqmYOylUvfajbAiigCz8Q4EMd011QIpNJeVWCQ5y6BmOXXaDEx0\n\t1yDfbjXRwj5RffXt25BK1/dc6ehodh4vGnkjpxzU=","To":"Kaaira Gupta <kgupta@es.iitr.ac.in>, libcamera-devel@lists.libcamera.org","References":"<20200722134303.GA27212@kaaira-HP-Pavilion-Notebook>","From":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<e80fc0e0-d7f4-c77d-f98a-b7d86d773298@ideasonboard.com>","Date":"Wed, 22 Jul 2020 14:49:38 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"<20200722134303.GA27212@kaaira-HP-Pavilion-Notebook>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] media: vimc: VIMC multiple streams","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"}},{"id":11500,"web_url":"https://patchwork.libcamera.org/comment/11500/","msgid":"<20200722144831.GA4865@kaaira-HP-Pavilion-Notebook>","date":"2020-07-22T14:48:31","subject":"Re: [libcamera-devel] media: vimc: VIMC multiple streams","submitter":{"id":39,"url":"https://patchwork.libcamera.org/api/people/39/","name":"Kaaira Gupta","email":"kgupta@es.iitr.ac.in"},"content":"On Wed, Jul 22, 2020 at 02:49:38PM +0100, Kieran Bingham wrote:\n> Hi Kaaira,\n> \n> On 22/07/2020 14:43, Kaaira Gupta wrote:\n> > This is a patchset by Niklas (https://patchwork.kernel.org/cover/10948831/)\n> > that I have rebased on the current media-tree, I have not added anything\n> > to it and it is just a compressed version of his patchset into a single\n> > patch, based on the latest master to aid testing multiple stream support\n> > in the VIMC pipeline.\n> > \n> > Signed-off-by: Kaaira Gupta <kgupta@es.iitr.ac.in>\n> \n> Great, it's good to hear that Niklas' patches still work and are simple\n> to rebase.\n> \n> I think his patches would need to be kept separate though, and submitted\n\nYes agreed, but I sent it here just for it to become easier for others to\nreview and test the patchset of multiples streams in the pipeline. I did not\nsend it as a patch, rather an option to help implement multiple streams\nsupport in the driver easily, and hence test the previous patchset :-D\n\n> to the linux kernel through the linux-media mailing list, with any\n> comments from the previous series addressed.\n> \n> You can keep Niklas' patches and his authorship, but add your signed-off\n> by tag, and document any changes you make in the commit message before\n> resubmitting to linux-media.\n\nI did not make any changes to the original patchset, as there are\nsome requirements (like if the counter for stream start/stop will be in\nthe driver or in v4l2-common or someplace else) of which I am not sure\nyet.\n\n> \n> \n> Then when we get to a point that the kernel can support multiple\n> streams, we can then enable that support in libcamera (we might have to\n> add a variable that enables multiple streams based on the kernel\n> version, but lets see how it goes).\n> \n> \n> --\n> Kieran\n\nThanks,\nKaaira\n\n> \n> \n> \n> > ---\n> >  .../media/test-drivers/vimc/vimc-capture.c    | 35 ++++++++++++++++++-\n> >  .../media/test-drivers/vimc/vimc-debayer.c    |  8 +++++\n> >  drivers/media/test-drivers/vimc/vimc-scaler.c |  8 +++++\n> >  drivers/media/test-drivers/vimc/vimc-sensor.c |  8 +++++\n> >  .../media/test-drivers/vimc/vimc-streamer.c   | 22 +++++++-----\n> >  5 files changed, 71 insertions(+), 10 deletions(-)\n> > \n> > diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c\n> > index c63496b17b9a..5fc56582af83 100644\n> > --- a/drivers/media/test-drivers/vimc/vimc-capture.c\n> > +++ b/drivers/media/test-drivers/vimc/vimc-capture.c\n> > @@ -237,16 +237,49 @@ static void vimc_cap_return_all_buffers(struct vimc_cap_device *vcap,\n> >  \tspin_unlock(&vcap->qlock);\n> >  }\n> >  \n> > +static struct media_entity *vimc_cap_get_sensor(struct vimc_cap_device *vcap)\n> > +{\n> > +\tstruct media_entity *entity = &vcap->vdev.entity;\n> > +\tstruct media_device *mdev = entity->graph_obj.mdev;\n> > +\tstruct media_graph graph;\n> > +\n> > +\tmutex_lock(&mdev->graph_mutex);\n> > +\tif (media_graph_walk_init(&graph, mdev)) {\n> > +\t\tmutex_unlock(&mdev->graph_mutex);\n> > +\t\treturn NULL;\n> > +\t}\n> > +\n> > +\tmedia_graph_walk_start(&graph, entity);\n> > +\n> > +\twhile ((entity = media_graph_walk_next(&graph)))\n> > +\t\tif (entity->function == MEDIA_ENT_F_CAM_SENSOR)\n> > +\t\t\tbreak;\n> > +\n> > +\tmutex_unlock(&mdev->graph_mutex);\n> > +\n> > +\tmedia_graph_walk_cleanup(&graph);\n> > +\n> > +\treturn entity;\n> > +}\n> > +\n> >  static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)\n> >  {\n> >  \tstruct vimc_cap_device *vcap = vb2_get_drv_priv(vq);\n> >  \tstruct media_entity *entity = &vcap->vdev.entity;\n> > +\tstruct media_pipeline *pipe = NULL;\n> > +\tstruct media_entity *sensorent;\n> >  \tint ret;\n> >  \n> >  \tvcap->sequence = 0;\n> >  \n> >  \t/* Start the media pipeline */\n> > -\tret = media_pipeline_start(entity, &vcap->stream.pipe);\n> > +\tsensorent = vimc_cap_get_sensor(vcap);\n> > +\tif (sensorent && sensorent->pipe)\n> > +\t\tpipe = sensorent->pipe;\n> > +\telse\n> > +\t\tpipe = &vcap->stream.pipe;\n> > +\n> > +\tret = media_pipeline_start(entity, pipe);\n> >  \tif (ret) {\n> >  \t\tvimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);\n> >  \t\treturn ret;\n> > diff --git a/drivers/media/test-drivers/vimc/vimc-debayer.c b/drivers/media/test-drivers/vimc/vimc-debayer.c\n> > index c3f6fef34f68..93fe19d8d2b4 100644\n> > --- a/drivers/media/test-drivers/vimc/vimc-debayer.c\n> > +++ b/drivers/media/test-drivers/vimc/vimc-debayer.c\n> > @@ -29,6 +29,7 @@ struct vimc_deb_pix_map {\n> >  struct vimc_deb_device {\n> >  \tstruct vimc_ent_device ved;\n> >  \tstruct v4l2_subdev sd;\n> > +\tatomic_t use_count;\n> >  \t/* The active format */\n> >  \tstruct v4l2_mbus_framefmt sink_fmt;\n> >  \tu32 src_code;\n> > @@ -343,6 +344,9 @@ static int vimc_deb_s_stream(struct v4l2_subdev *sd, int enable)\n> >  \t\tconst struct vimc_pix_map *vpix;\n> >  \t\tunsigned int frame_size;\n> >  \n> > +\t\tif (atomic_inc_return(&vdeb->use_count) != 1)\n> > +\t\t\treturn 0;\n> > +\n> >  \t\tif (vdeb->src_frame)\n> >  \t\t\treturn 0;\n> >  \n> > @@ -368,6 +372,9 @@ static int vimc_deb_s_stream(struct v4l2_subdev *sd, int enable)\n> >  \t\t\treturn -ENOMEM;\n> >  \n> >  \t} else {\n> > +\t\tif (atomic_dec_return(&vdeb->use_count) != 0)\n> > +\t\t\treturn 0;\n> > +\n> >  \t\tif (!vdeb->src_frame)\n> >  \t\t\treturn 0;\n> >  \n> > @@ -595,6 +602,7 @@ static struct vimc_ent_device *vimc_deb_add(struct vimc_device *vimc,\n> >  \tvdeb->ved.process_frame = vimc_deb_process_frame;\n> >  \tvdeb->ved.dev = vimc->mdev.dev;\n> >  \tvdeb->mean_win_size = vimc_deb_ctrl_mean_win_size.def;\n> > +\tatomic_set(&vdeb->use_count, 0);\n> >  \n> >  \t/* Initialize the frame format */\n> >  \tvdeb->sink_fmt = sink_fmt_default;\n> > diff --git a/drivers/media/test-drivers/vimc/vimc-scaler.c b/drivers/media/test-drivers/vimc/vimc-scaler.c\n> > index 121fa7d62a2e..9b8458dbe57c 100644\n> > --- a/drivers/media/test-drivers/vimc/vimc-scaler.c\n> > +++ b/drivers/media/test-drivers/vimc/vimc-scaler.c\n> > @@ -25,6 +25,7 @@ MODULE_PARM_DESC(sca_mult, \" the image size multiplier\");\n> >  struct vimc_sca_device {\n> >  \tstruct vimc_ent_device ved;\n> >  \tstruct v4l2_subdev sd;\n> > +\tatomic_t use_count;\n> >  \t/* NOTE: the source fmt is the same as the sink\n> >  \t * with the width and hight multiplied by mult\n> >  \t */\n> > @@ -340,6 +341,9 @@ static int vimc_sca_s_stream(struct v4l2_subdev *sd, int enable)\n> >  \t\tconst struct vimc_pix_map *vpix;\n> >  \t\tunsigned int frame_size;\n> >  \n> > +\t\tif (atomic_inc_return(&vsca->use_count) != 1)\n> > +\t\t\treturn 0;\n> > +\n> >  \t\tif (vsca->src_frame)\n> >  \t\t\treturn 0;\n> >  \n> > @@ -363,6 +367,9 @@ static int vimc_sca_s_stream(struct v4l2_subdev *sd, int enable)\n> >  \t\t\treturn -ENOMEM;\n> >  \n> >  \t} else {\n> > +\t\tif (atomic_dec_return(&vsca->use_count) != 0)\n> > +\t\t\treturn 0;\n> > +\n> >  \t\tif (!vsca->src_frame)\n> >  \t\t\treturn 0;\n> >  \n> > @@ -506,6 +513,7 @@ static struct vimc_ent_device *vimc_sca_add(struct vimc_device *vimc,\n> >  \n> >  \tvsca->ved.process_frame = vimc_sca_process_frame;\n> >  \tvsca->ved.dev = vimc->mdev.dev;\n> > +\tatomic_set(&vsca->use_count, 0);\n> >  \n> >  \t/* Initialize the frame format */\n> >  \tvsca->sink_fmt = sink_fmt_default;\n> > diff --git a/drivers/media/test-drivers/vimc/vimc-sensor.c b/drivers/media/test-drivers/vimc/vimc-sensor.c\n> > index a2f09ac9a360..d68113886290 100644\n> > --- a/drivers/media/test-drivers/vimc/vimc-sensor.c\n> > +++ b/drivers/media/test-drivers/vimc/vimc-sensor.c\n> > @@ -18,6 +18,7 @@ struct vimc_sen_device {\n> >  \tstruct vimc_ent_device ved;\n> >  \tstruct v4l2_subdev sd;\n> >  \tstruct tpg_data tpg;\n> > +\tatomic_t use_count;\n> >  \tu8 *frame;\n> >  \t/* The active format */\n> >  \tstruct v4l2_mbus_framefmt mbus_format;\n> > @@ -201,6 +202,9 @@ static int vimc_sen_s_stream(struct v4l2_subdev *sd, int enable)\n> >  \t\tconst struct vimc_pix_map *vpix;\n> >  \t\tunsigned int frame_size;\n> >  \n> > +\t\tif (atomic_inc_return(&vsen->use_count) != 1)\n> > +\t\t\treturn 0;\n> > +\n> >  \t\t/* Calculate the frame size */\n> >  \t\tvpix = vimc_pix_map_by_code(vsen->mbus_format.code);\n> >  \t\tframe_size = vsen->mbus_format.width * vpix->bpp *\n> > @@ -219,6 +223,9 @@ static int vimc_sen_s_stream(struct v4l2_subdev *sd, int enable)\n> >  \n> >  \t} else {\n> >  \n> > +\t\tif (atomic_dec_return(&vsen->use_count) != 0)\n> > +\t\t\treturn 0;\n> > +\n> >  \t\tvfree(vsen->frame);\n> >  \t\tvsen->frame = NULL;\n> >  \t}\n> > @@ -359,6 +366,7 @@ static struct vimc_ent_device *vimc_sen_add(struct vimc_device *vimc,\n> >  \n> >  \tvsen->ved.process_frame = vimc_sen_process_frame;\n> >  \tvsen->ved.dev = vimc->mdev.dev;\n> > +\tatomic_set(&vsen->use_count, 0);\n> >  \n> >  \t/* Initialize the frame format */\n> >  \tvsen->mbus_format = fmt_default;\n> > diff --git a/drivers/media/test-drivers/vimc/vimc-streamer.c b/drivers/media/test-drivers/vimc/vimc-streamer.c\n> > index 451a32c0d034..d5b966a99f7b 100644\n> > --- a/drivers/media/test-drivers/vimc/vimc-streamer.c\n> > +++ b/drivers/media/test-drivers/vimc/vimc-streamer.c\n> > @@ -192,33 +192,35 @@ int vimc_streamer_s_stream(struct vimc_stream *stream,\n> >  \t\t\t   struct vimc_ent_device *ved,\n> >  \t\t\t   int enable)\n> >  {\n> > +\tstatic DEFINE_MUTEX(vimc_streamer_lock);\n> >  \tint ret;\n> >  \n> >  \tif (!stream || !ved)\n> >  \t\treturn -EINVAL;\n> >  \n> > +\tret = mutex_lock_interruptible(&vimc_streamer_lock);\n> > +\tif (ret)\n> > +\t\treturn ret;\n> > +\n> >  \tif (enable) {\n> >  \t\tif (stream->kthread)\n> > -\t\t\treturn 0;\n> > +\t\t\tgoto out;\n> >  \n> >  \t\tret = vimc_streamer_pipeline_init(stream, ved);\n> >  \t\tif (ret)\n> > -\t\t\treturn ret;\n> > +\t\t\tgoto out;\n> >  \n> >  \t\tstream->kthread = kthread_run(vimc_streamer_thread, stream,\n> >  \t\t\t\t\t      \"vimc-streamer thread\");\n> >  \n> >  \t\tif (IS_ERR(stream->kthread)) {\n> >  \t\t\tret = PTR_ERR(stream->kthread);\n> > -\t\t\tdev_err(ved->dev, \"kthread_run failed with %d\\n\", ret);\n> > -\t\t\tvimc_streamer_pipeline_terminate(stream);\n> > -\t\t\tstream->kthread = NULL;\n> > -\t\t\treturn ret;\n> > +\t\t\tgoto out;\n> >  \t\t}\n> >  \n> >  \t} else {\n> >  \t\tif (!stream->kthread)\n> > -\t\t\treturn 0;\n> > +\t\t\tgoto out;\n> >  \n> >  \t\tret = kthread_stop(stream->kthread);\n> >  \t\t/*\n> > @@ -228,12 +230,14 @@ int vimc_streamer_s_stream(struct vimc_stream *stream,\n> >  \t\t * pipeline.\n> >  \t\t */\n> >  \t\tif (ret)\n> > -\t\t\tdev_dbg(ved->dev, \"kthread_stop returned '%d'\\n\", ret);\n> > +\t\t\tgoto out;\n> >  \n> >  \t\tstream->kthread = NULL;\n> >  \n> >  \t\tvimc_streamer_pipeline_terminate(stream);\n> >  \t}\n> > +out:\n> > +\tmutex_unlock(&vimc_streamer_lock);\n> >  \n> > -\treturn 0;\n> > +\treturn ret;\n> >  }\n> > \n> \n> -- \n> Regards\n> --\n> Kieran","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0EB15BDB1B\n\tfor <parsemail@patchwork.libcamera.org>;\n\tWed, 22 Jul 2020 14:48:41 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7F79F6099C;\n\tWed, 22 Jul 2020 16:48:40 +0200 (CEST)","from mail-pl1-x642.google.com (mail-pl1-x642.google.com\n\t[IPv6:2607:f8b0:4864:20::642])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5E0E36039F\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed, 22 Jul 2020 16:48:39 +0200 (CEST)","by mail-pl1-x642.google.com with SMTP id m16so1106579pls.5\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed, 22 Jul 2020 07:48:39 -0700 (PDT)","from kaaira-HP-Pavilion-Notebook ([103.113.213.178])\n\tby smtp.gmail.com with ESMTPSA id\n\tbg6sm6982393pjb.51.2020.07.22.07.48.35\n\t(version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);\n\tWed, 22 Jul 2020 07:48:37 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=es-iitr-ac-in.20150623.gappssmtp.com\n\theader.i=@es-iitr-ac-in.20150623.gappssmtp.com\n\theader.b=\"X48rlCBG\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=es-iitr-ac-in.20150623.gappssmtp.com; s=20150623;\n\th=from:date:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:in-reply-to:user-agent;\n\tbh=xRKE6fxA2SCX7Enjhz4sPQxNHsrEiwrnl36AYY0llgk=;\n\tb=X48rlCBGLsdxarkxiink4iCtO7epIPLo6U9kHmLkPchtwRGjwCRPcen05pPjB3SDj8\n\tVXSIP+8Lm1sr5OUQNstzPMW4IkgQqSIKKAinuhc/+1Uoae9LffTkpOwNadKg06lmfsUM\n\tlCCsPLXJXygvFsI543qxvC5/N3V3plNEqYnl3U+PB4A9gDcamZ0yGkbZxpkCyDTBdX2c\n\tnPSigaV52uj2OBAXncTSWyPmD28a4zW9paVu3RCXPj8HH4Ik7YRuHOe54J9zlq7naQLk\n\tr8lvwvBfW08VmJzKLyHZyj1cNWXXzTPbdLxT3l7i0t3W/rRYW51gszIdN0XblKcD1nY9\n\tb53w==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:date:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:in-reply-to:user-agent;\n\tbh=xRKE6fxA2SCX7Enjhz4sPQxNHsrEiwrnl36AYY0llgk=;\n\tb=oXWAD/hU6HqvXjt95qPAKvpB1xyq/p8lAY234SQMKhr3lIosMYT3eUWFuhpaXcYpRK\n\t3QXjx7+7mbUJBhHXQGoj8UgQkGMFtBrfhKxTHYYCDxUjXDwCw2kJoQ3NoNjvxk+9zdM1\n\trNSY0pGD6F6fAVGQmivAUKe6CNB3MKOSOKBXu4VMdwVTrZbaXNST7jjqKbvFq0++Ff7O\n\tCD+TmeFIBBCZVCQve0aKNM4mhdULvXmgZ3c4PRI5sHrMnIHTgog7k0mMhI27sNrl/zGG\n\tHi3KHgvxBNv1YoYapq9Pb2R9vj0dxyIzOIwz9m4zPQMQfEB1+I0t7UWMGxRqjkk+vSQZ\n\tA+dg==","X-Gm-Message-State":"AOAM532vOH9NEMZDv324ewHBVyrKEjVe4J1vecyWIVPuIO3CKDWsFwyp\n\t1H974XoMVu1F1OQjHxWpK6Unsr4Jj3w=","X-Google-Smtp-Source":"ABdhPJxseX61rgYZwaDnxcaVvodCTzX1FEqXg6npCqQBagQTuivEP77uYHx8lPkOWHBHuffd7bBNQg==","X-Received":"by 2002:a17:90a:c68e:: with SMTP id\n\tn14mr10022761pjt.182.1595429317725; \n\tWed, 22 Jul 2020 07:48:37 -0700 (PDT)","From":"Kaaira Gupta <kgupta@es.iitr.ac.in>","X-Google-Original-From":"Kaaira Gupta <Kaairakgupta@es.iitr.ac.in>","Date":"Wed, 22 Jul 2020 20:18:31 +0530","To":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Message-ID":"<20200722144831.GA4865@kaaira-HP-Pavilion-Notebook>","References":"<20200722134303.GA27212@kaaira-HP-Pavilion-Notebook>\n\t<e80fc0e0-d7f4-c77d-f98a-b7d86d773298@ideasonboard.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<e80fc0e0-d7f4-c77d-f98a-b7d86d773298@ideasonboard.com>","User-Agent":"Mutt/1.9.4 (2018-02-28)","Subject":"Re: [libcamera-devel] media: vimc: VIMC multiple streams","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Cc":"Kaaira Gupta <kgupta@es.iitr.ac.in>, libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"}}]