[{"id":21795,"web_url":"https://patchwork.libcamera.org/comment/21795/","msgid":"<20211215102509.2969578d@coco.lan>","date":"2021-12-15T09:25:09","subject":"Re: [libcamera-devel] [PATCH 0/5] Introduce ancillary links","submitter":{"id":110,"url":"https://patchwork.libcamera.org/api/people/110/","name":"Mauro Carvalho Chehab","email":"mchehab+huawei@kernel.org"},"content":"Em Mon, 13 Dec 2021 23:28:44 +0000\nDaniel Scally escreveu:\n\n> Hello all\n> \n> At present there's no means in the kernel of describing the supporting\n> relationship between subdevices that work together to form an effective single\n> unit - the type example in this case being a camera sensor and its\n> corresponding vcm. To attempt to solve that, this series adds a new type of\n> media link called MEDIA_LNK_FL_ANCILLARY_LINK, which connects two instances of\n> struct media_entity.\n> \n> The mechanism of connection I have modelled as a notifier and async subdev,\n> which seemed the best route since sensor drivers already typically will call\n> v4l2_async_register_subdev_sensor() on probe, and that function already looks\n> for a reference to a firmware node with the reference named \"lens-focus\". To\n> avoid boilerplate in the sensor drivers, I added some new functions in\n> v4l2-async that are called in v4l2_async_match_notify() to create the ancillary\n> links - checking the entity.function of both notifier and subdev to make sure\n> that's appropriate. I haven't gone further than that yet, but I suspect we could\n> cut down on code elsewhere by, for example, also creating pad-to-pad links in\n> the same place.\n> \n> Thoughts and comments very welcome \n\nThe idea of ancillary link sounds interesting. I did a quick look at\nthe series. \n\nLaurent already did some good points during his review.\nBesides that, one thing it is missing, though, is an implementation on\na driver. At least vimc should gain an implementation at this series,\nin order to allow media developers to test and see how the graph will\nbe after the patch series.\n\nRegards,\nMauro\n\n> \n> Dan\n> \n> Daniel Scally (5):\n> media: media.h: Add new media link type\n> media: entity: Add link_type() helper\n> media: entity: Skip non-data links in graph iteration\n> media: entity: Add support for ancillary links\n> media: v4l2-async: Create links during v4l2_async_match_notify()\n> \n> drivers/media/mc/mc-entity.c | 56 ++++++++++++++++++++++++++--\n> drivers/media/v4l2-core/v4l2-async.c | 51 +++++++++++++++++++++++++\n> include/media/media-entity.h | 29 ++++++++++++++\n> include/uapi/linux/media.h | 1 +\n> 4 files changed, 134 insertions(+), 3 deletions(-)\n> \n\n\n\nThanks,\nMauro","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3149CBF415\n\tfor ;\n\tWed, 15 Dec 2021 09:33:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9048960114;\n\tWed, 15 Dec 2021 10:33:58 +0100 (CET)","from dfw.source.kernel.org (dfw.source.kernel.org\n\t[IPv6:2604:1380:4641:c500::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 98FE160114\n\tfor ;\n\tWed, 15 Dec 2021 10:25:16 +0100 (CET)","from smtp.kernel.org (relay.kernel.org [52.25.139.140])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby dfw.source.kernel.org (Postfix) with ESMTPS id 3CEDF6185D;\n\tWed, 15 Dec 2021 09:25:15 +0000 (UTC)","by smtp.kernel.org (Postfix) with ESMTPSA id 95117C34600;\n\tWed, 15 Dec 2021 09:25:12 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=kernel.org header.i=@kernel.org\n\theader.b=\"uddsRJD6\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;\n\ts=k20201202; t=1639560314;\n\tbh=Boi5KTv+fWa939z6smBHV7KAPcNZQ8kMnVrmfnBWFlI=;\n\th=Date:From:To:Cc:Subject:In-Reply-To:References:From;\n\tb=uddsRJD6hj0c8/VgEn8k8fIcbDPzGfvheB9m3WA0i2jhpKKEzGIJblPpx82HXLv/6\n\tjMAS6KVanLLcrADyzBJHfRR5yCnrnziSnIoHnkTZBZs7jLapCXuUYZK91KL0dHDuXP\n\t2kSrEhpQn7bRYOTHGIUqdEWcP4iPXeirfWTtJ7FkmHx1xJ5Hz/1i1EfVAdNCHSCf2Z\n\tvAB/UnIaIbNk1EHWa9n1ZlqoNfqbIIlLKiZVCRuvMYhG8/WLLdmkmLTZLFShQgV0D+\n\ttSs2X6KkvLU/kn7MbC1I8idmhFpQx93U2PVs2u/J0sbLRBw2dIPVOuE6jHszfY//8b\n\tmxHMKFV2a+BBg==","Date":"Wed, 15 Dec 2021 10:25:09 +0100","From":"Mauro Carvalho Chehab ","To":"Daniel Scally ","Message-ID":"<20211215102509.2969578d@coco.lan>","In-Reply-To":"<20211213232849.40071-1-djrscally@gmail.com>","References":"<20211213232849.40071-1-djrscally@gmail.com>","X-Mailer":"Claws Mail 4.0.0 (GTK+ 3.24.30; x86_64-redhat-linux-gnu)","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Mailman-Approved-At":"Wed, 15 Dec 2021 10:33:56 +0100","Subject":"Re: [libcamera-devel] [PATCH 0/5] Introduce ancillary links","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, sakari.ailus@linux.intel.com,\n\tlinux-media@vger.kernel.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21796,"web_url":"https://patchwork.libcamera.org/comment/21796/","msgid":"<621f03ed-4137-ab6e-a5c8-3c11fbf70aff@gmail.com>","date":"2021-12-15T09:36:10","subject":"Re: [libcamera-devel] [PATCH 0/5] Introduce ancillary links","submitter":{"id":90,"url":"https://patchwork.libcamera.org/api/people/90/","name":"Daniel Scally","email":"djrscally@gmail.com"},"content":"Hi Mauro\n\nOn 15/12/2021 09:25, Mauro Carvalho Chehab wrote:\n> Em Mon, 13 Dec 2021 23:28:44 +0000\n> Daniel Scally escreveu:\n>\n>> Hello all\n>>\n>> At present there's no means in the kernel of describing the supporting\n>> relationship between subdevices that work together to form an effective single\n>> unit - the type example in this case being a camera sensor and its\n>> corresponding vcm. To attempt to solve that, this series adds a new type of\n>> media link called MEDIA_LNK_FL_ANCILLARY_LINK, which connects two instances of\n>> struct media_entity.\n>>\n>> The mechanism of connection I have modelled as a notifier and async subdev,\n>> which seemed the best route since sensor drivers already typically will call\n>> v4l2_async_register_subdev_sensor() on probe, and that function already looks\n>> for a reference to a firmware node with the reference named \"lens-focus\". To\n>> avoid boilerplate in the sensor drivers, I added some new functions in\n>> v4l2-async that are called in v4l2_async_match_notify() to create the ancillary\n>> links - checking the entity.function of both notifier and subdev to make sure\n>> that's appropriate. I haven't gone further than that yet, but I suspect we could\n>> cut down on code elsewhere by, for example, also creating pad-to-pad links in\n>> the same place.\n>>\n>> Thoughts and comments very welcome \n> The idea of ancillary link sounds interesting. I did a quick look at\n> the series. \n>\n> Laurent already did some good points during his review.\n> Besides that, one thing it is missing, though, is an implementation on\n> a driver. At least vimc should gain an implementation at this series,\n> in order to allow media developers to test and see how the graph will\n> be after the patch series.\n\n\nWe have this running through libcamera at the moment; this series piggy\nbacks onto the notifier that's set up by\nv4l2_async_create_sensor_subdev() so that the connection is made and the\nlinks created when a lens controller is linked to the sensor via the\nlens-focus firmware property. I've been testing this using the dw9719\ndriver I posted [1], plus a series that adds support to libcamera [2]. I\nbelieve that some folks from Cros have also tested it with the dw9714\ndriver too.\n\n\n[1]\nhttps://lore.kernel.org/linux-media/20211128232115.38833-1-djrscally@gmail.com/\n\n[2]\nhttps://lists.libcamera.org/pipermail/libcamera-devel/2021-December/028082.html\n\n>\n> Regards,\n> Mauro\n>\n>> Dan\n>>\n>> Daniel Scally (5):\n>> media: media.h: Add new media link type\n>> media: entity: Add link_type() helper\n>> media: entity: Skip non-data links in graph iteration\n>> media: entity: Add support for ancillary links\n>> media: v4l2-async: Create links during v4l2_async_match_notify()\n>>\n>> drivers/media/mc/mc-entity.c | 56 ++++++++++++++++++++++++++--\n>> drivers/media/v4l2-core/v4l2-async.c | 51 +++++++++++++++++++++++++\n>> include/media/media-entity.h | 29 ++++++++++++++\n>> include/uapi/linux/media.h | 1 +\n>> 4 files changed, 134 insertions(+), 3 deletions(-)\n>>\n>\n>\n> Thanks,\n> Mauro","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D5B07BF415\n\tfor ;\n\tWed, 15 Dec 2021 09:36:13 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2D91D608A4;\n\tWed, 15 Dec 2021 10:36:13 +0100 (CET)","from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com\n\t[IPv6:2a00:1450:4864:20::42e])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E7DCA60114\n\tfor ;\n\tWed, 15 Dec 2021 10:36:11 +0100 (CET)","by mail-wr1-x42e.google.com with SMTP id i22so7258041wrb.13\n\tfor ;\n\tWed, 15 Dec 2021 01:36:11 -0800 (PST)","from [192.168.0.14]\n\t(cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net. [86.13.91.161])\n\tby smtp.gmail.com with ESMTPSA id\n\tt4sm4221585wmi.48.2021.12.15.01.36.10\n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tWed, 15 Dec 2021 01:36:11 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"BkxD3+iB\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;\n\th=subject:to:cc:references:from:message-id:date:user-agent\n\t:mime-version:in-reply-to:content-transfer-encoding:content-language; \n\tbh=hGhiOZoNFezPBSG6DBLOpLXgwVZkfwjw+NMMaFsgmEQ=;\n\tb=BkxD3+iBhYlaulChNb3Y9OWAMvuCV0ci8yywNPwq/8TtaQC0M4/4BswCo437rh4gqA\n\t6VGkU7/JYzEoGg8UBxbmc6A1B1tWr/GD/DR0kC8oeqThb5GArPWtkNU2CniqYIykFyC9\n\tdGz+99MrsM6x/KaryP7QVZVnSkqwUxEM970yD/f5VkEBchskNgz+Ea2yXubTSHbFL9uv\n\ti9M5XQZy09YRM3eUrpJC/ZJXo2pt3DxXcGe63Q0eSU9ZA0Xj1NQu05t1ofb3O2zu/4wH\n\tqzVKzLKlWKc3U5SPuGA0Jz7emvgLIz0zZELQXnBgU5G4dBS8vkNr0B1hzM+ooSQiJzqj\n\t/iSQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:subject:to:cc:references:from:message-id:date\n\t:user-agent:mime-version:in-reply-to:content-transfer-encoding\n\t:content-language;\n\tbh=hGhiOZoNFezPBSG6DBLOpLXgwVZkfwjw+NMMaFsgmEQ=;\n\tb=6+9lUGTSIxFOwjkLmkkj/LW3WALUATNFi1BDEhJ3exIFOBh+1Ws6DYjEqTyxWzErP3\n\tt7ldj0EJZJjB4nTB26cpt1ekSp2dRjtSFG4zBeHgJ7Si2hwDaVfzFmc3ZfxLRKZePkzR\n\txItfU2e+FYHkm8GGSWx8d8oUoLgTeWCUdGWm46d127w/dZcRqDfOYufAb++Z8VWbfplV\n\trJ65yogGdqgUJBrUCBtW86yH0hS1USl3bhXIm5X42D0IY8YZ4q/DlOvGRmJQkp3duMV1\n\t9+wQWEYF1QyFyn8PYqp3iHY79fsT2nD8Ie+1HIdRh/tZx+KJhLcLxfYibAqwWpw/BMfx\n\tP9Cw==","X-Gm-Message-State":"AOAM5339/BU6n8jo9FXFqxlLuaVS9OTJOBHcqNTZknYLr9xnBUaLhKh3\n\tDWewHHM8Etiiz+6Ujz9fRJc=","X-Google-Smtp-Source":"ABdhPJwhliBYrdN6QmS38dFfMgfcuWWK7J0GBxdKWUIsqpjJXXZJV4mRc89cEIA8ImjjlXys+0pS8w==","X-Received":"by 2002:adf:eac8:: with SMTP id o8mr2364595wrn.93.1639560971522; \n\tWed, 15 Dec 2021 01:36:11 -0800 (PST)","To":"Mauro Carvalho Chehab ","References":"<20211213232849.40071-1-djrscally@gmail.com>\n\t<20211215102509.2969578d@coco.lan>","From":"Daniel Scally ","Message-ID":"<621f03ed-4137-ab6e-a5c8-3c11fbf70aff@gmail.com>","Date":"Wed, 15 Dec 2021 09:36:10 +0000","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101\n\tThunderbird/78.14.0","MIME-Version":"1.0","In-Reply-To":"<20211215102509.2969578d@coco.lan>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"7bit","Content-Language":"en-US","Subject":"Re: [libcamera-devel] [PATCH 0/5] Introduce ancillary links","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, sakari.ailus@linux.intel.com,\n\tlinux-media@vger.kernel.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":21798,"web_url":"https://patchwork.libcamera.org/comment/21798/","msgid":"","date":"2021-12-15T09:52:38","subject":"Re: [libcamera-devel] [PATCH 0/5] Introduce ancillary links","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Mauro,\n\nOn Wed, Dec 15, 2021 at 10:25:09AM +0100, Mauro Carvalho Chehab wrote:\n> Em Mon, 13 Dec 2021 23:28:44 +0000 Daniel Scally escreveu:\n> \n> > Hello all\n> > \n> > At present there's no means in the kernel of describing the supporting\n> > relationship between subdevices that work together to form an effective single\n> > unit - the type example in this case being a camera sensor and its\n> > corresponding vcm. To attempt to solve that, this series adds a new type of\n> > media link called MEDIA_LNK_FL_ANCILLARY_LINK, which connects two instances of\n> > struct media_entity.\n> > \n> > The mechanism of connection I have modelled as a notifier and async subdev,\n> > which seemed the best route since sensor drivers already typically will call\n> > v4l2_async_register_subdev_sensor() on probe, and that function already looks\n> > for a reference to a firmware node with the reference named \"lens-focus\". To\n> > avoid boilerplate in the sensor drivers, I added some new functions in\n> > v4l2-async that are called in v4l2_async_match_notify() to create the ancillary\n> > links - checking the entity.function of both notifier and subdev to make sure\n> > that's appropriate. I haven't gone further than that yet, but I suspect we could\n> > cut down on code elsewhere by, for example, also creating pad-to-pad links in\n> > the same place.\n> > \n> > Thoughts and comments very welcome \n> \n> The idea of ancillary link sounds interesting. I did a quick look at\n> the series. \n> \n> Laurent already did some good points during his review.\n> Besides that, one thing it is missing, though, is an implementation on\n> a driver. At least vimc should gain an implementation at this series,\n> in order to allow media developers to test and see how the graph will\n> be after the patch series.\n\nThe whole point of patch 4/4 is to create those links in the core\nwithout requiring manual involvement of the drivers :-)\n\n> > Daniel Scally (5):\n> > media: media.h: Add new media link type\n> > media: entity: Add link_type() helper\n> > media: entity: Skip non-data links in graph iteration\n> > media: entity: Add support for ancillary links\n> > media: v4l2-async: Create links during v4l2_async_match_notify()\n> > \n> > drivers/media/mc/mc-entity.c | 56 ++++++++++++++++++++++++++--\n> > drivers/media/v4l2-core/v4l2-async.c | 51 +++++++++++++++++++++++++\n> > include/media/media-entity.h | 29 ++++++++++++++\n> > include/uapi/linux/media.h | 1 +\n> > 4 files changed, 134 insertions(+), 3 deletions(-)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E1848BF415\n\tfor ;\n\tWed, 15 Dec 2021 09:52:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 27884608E5;\n\tWed, 15 Dec 2021 10:52:43 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CC9B260114\n\tfor ;\n\tWed, 15 Dec 2021 10:52:41 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 0F2C6292;\n\tWed, 15 Dec 2021 10:52:40 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"t3NKqhHk\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1639561961;\n\tbh=78AHd+3m++S+gX2fZ38+9vvLLDBXS9tzJ+NOscEhWFA=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=t3NKqhHkKnrOL1AB7jpXcP7lEu6kC/m3rdTkYDOMQ5A5Bylzsj9t6DEm249DzJU0j\n\tXBNNj0M9tVv8xKMYeGKiI4MlV3D2oGD07u5c6/Yz2MvYPEGVvxBcl6xBHYtf0QyKhI\n\tEKGNx8+TVVDM0PFYjIy0NqnCmDZJXkCZS7kiaG6A=","Date":"Wed, 15 Dec 2021 11:52:38 +0200","From":"Laurent Pinchart ","To":"Mauro Carvalho Chehab ","Message-ID":"","References":"<20211213232849.40071-1-djrscally@gmail.com>\n\t<20211215102509.2969578d@coco.lan>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20211215102509.2969578d@coco.lan>","Subject":"Re: [libcamera-devel] [PATCH 0/5] Introduce ancillary links","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, sakari.ailus@linux.intel.com,\n\tlinux-media@vger.kernel.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]