From patchwork Tue Jan 4 12:57:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 15243 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C936CC3258 for ; Tue, 4 Jan 2022 12:57:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 671886092F; Tue, 4 Jan 2022 13:57:19 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="ChhKve4S"; dkim-atps=neutral Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 04D44604F4 for ; Tue, 4 Jan 2022 13:57:17 +0100 (CET) Received: by mail-wm1-x335.google.com with SMTP id p1-20020a1c7401000000b00345c2d068bdso20139909wmc.3 for ; Tue, 04 Jan 2022 04:57:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=isbLew+Hfz3OpvSyqfWQp0MtX+L2fQ4xOz39/GgziQQ=; b=ChhKve4SPkNEedsSBhMdCHBx5YIxKVm+SUjemYATuT4M7+tyebaUtp3GUp1a1glUTo GXS0AsOhJABIPsRBst2yAMXJm3Ow/nWmO9Fj0hw0MVtlpqyVZ6mUSqkRdjscTpAWg11B ycFQnFwvq9IU3sVvI73X6AmhMD3ZZ5SV0MP2iw+CNZhodky8zxAN0+WrhSDaujrOSYOH f91KrLHb73lVtVAGnQnvJx+L00nR3ghWuwWGKNVoJ3ba7moWIvQ7pSBLd10HhJmZRvEy +YUwcSc0vkspWJnBvnWfG/wXMPLTAltJ8JSjf0/Ak4g0q5Vc4SEBFSUKTHs/tLPVoWYv ygLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=isbLew+Hfz3OpvSyqfWQp0MtX+L2fQ4xOz39/GgziQQ=; b=2MoYHtIwuxwSVSO5buMT6tPd94EB7+bL/4Xpkgh76OXTk2Ndvqnh1nTJ34Va6k9KPh ampPcqGCwbylH67dgEuoyeAAdGSkQ6zhIERFQ+U4xGAgttrZey7rP67X/Ud1KZJXq90E S+FdG4it5jxMYCnKQzdNGQu0Z9dKu1iTx5vXrMWOO26MkjRoyDcc/aZlGJsgfAaIdmZ9 TSO9E+zro+LsAB56ButiZN+GIcZNe44ES/muhKWkXbSSgjF7kZFue2KdjavZySKHYcRX G1U6BY+JRSKOVb+dL9aM6YZJVaK6S0R+C6OxHB6wUg6qvMigPGAh97CLFo9EWcfQR9nC tKWQ== X-Gm-Message-State: AOAM5326tS29EJ6UyIWKbFf4rC9jpJMUXVSvIlJLx39+8VtqA5yY9xqb G5qKdE3K2OHEkFbgOYxhWrbOMBBSfEaVmg== X-Google-Smtp-Source: ABdhPJzegxBGYBEIqKtylSQQNFA7M1i84Oo1F3oP6o3JAwnTyXRotEutTBcpFveu0SPq7x/0PbXZSg== X-Received: by 2002:a7b:c007:: with SMTP id c7mr42423899wmb.27.1641301036525; Tue, 04 Jan 2022 04:57:16 -0800 (PST) Received: from naush-laptop.pitowers.org ([2a00:1098:3142:14:cf71:3c12:f0dd:e082]) by smtp.gmail.com with ESMTPSA id p15sm35673753wmj.46.2022.01.04.04.57.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jan 2022 04:57:16 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Tue, 4 Jan 2022 12:57:08 +0000 Message-Id: <20220104125709.1273546-2-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220104125709.1273546-1-naush@raspberrypi.com> References: <20220104125709.1273546-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 1/2] pipeline: raspberrypi: Move sensor entity detection out of registerCamera() X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Enumerate the sensor device entities in PipelineHandlerRPi::match() and loop over PipelineHandlerRPi::registerCamera() for each sensor found. This will allow the pipeline handler to register multiple cameras attached to a single Unicam instance with a Video Mux device. Signed-off-by: Naushir Patuck Reviewed-by: Kieran Bingham Reviewed-by: David Plowman Reviewed-by: Jacopo Mondi --- .../pipeline/raspberrypi/raspberrypi.cpp | 35 +++++++++++-------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index b5c687da467f..a6d1e8570abf 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -312,7 +312,7 @@ private: return static_cast(camera->_d()); } - int registerCamera(MediaDevice *unicam, MediaDevice *isp); + int registerCamera(MediaDevice *unicam, MediaDevice *isp, MediaEntity *sensorEntity); int queueAllBuffers(Camera *camera); int prepareBuffers(Camera *camera); void freeBuffers(Camera *camera); @@ -1069,16 +1069,28 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator) return false; } - int ret = registerCamera(unicamDevice, ispDevice); - if (ret) { - LOG(RPI, Error) << "Failed to register camera: " << ret; - return false; + /* + * The loop below is used to register multiple cameras behind one or more + * video mux devices that are attached to a particular Unicam instance. + * Obviously these cameras cannot be used simultaneously. + */ + unsigned int numCameras = 0; + for (MediaEntity *entity : unicamDevice->entities()) { + if (entity->function() != MEDIA_ENT_F_CAM_SENSOR) + continue; + + int ret = registerCamera(unicamDevice, ispDevice, entity); + if (ret) + LOG(RPI, Error) << "Failed to register camera " + << entity->name() << ": " << ret; + else + numCameras++; } - return true; + return !!numCameras; } -int PipelineHandlerRPi::registerCamera(MediaDevice *unicam, MediaDevice *isp) +int PipelineHandlerRPi::registerCamera(MediaDevice *unicam, MediaDevice *isp, MediaEntity *sensorEntity) { std::unique_ptr data = std::make_unique(this); @@ -1119,14 +1131,7 @@ int PipelineHandlerRPi::registerCamera(MediaDevice *unicam, MediaDevice *isp) data->isp_[Isp::Output1].dev()->bufferReady.connect(data.get(), &RPiCameraData::ispOutputDequeue); data->isp_[Isp::Stats].dev()->bufferReady.connect(data.get(), &RPiCameraData::ispOutputDequeue); - /* Identify the sensor. */ - for (MediaEntity *entity : unicam->entities()) { - if (entity->function() == MEDIA_ENT_F_CAM_SENSOR) { - data->sensor_ = std::make_unique(entity); - break; - } - } - + data->sensor_ = std::make_unique(sensorEntity); if (!data->sensor_) return -EINVAL; From patchwork Tue Jan 4 12:57:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 15244 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 75594C3259 for ; Tue, 4 Jan 2022 12:57:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2410E60927; Tue, 4 Jan 2022 13:57:20 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="EkaVfDzu"; dkim-atps=neutral Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D588B604F5 for ; Tue, 4 Jan 2022 13:57:17 +0100 (CET) Received: by mail-wr1-x436.google.com with SMTP id s1so76116370wrg.1 for ; Tue, 04 Jan 2022 04:57:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ky1hhFEI9Nur/24AXsXA4ap2z+oByp8M83gncjnbIac=; b=EkaVfDzueRPAa9gVADh/Ho6b4rXme9Cb+MBGQrVXdYDeaC06rVq5CCsn19RmTGmkSZ z5OlMLJ5Fs4U1lbdCOC+4OyAhj5OoEwy4IO6UqJnWFEZBOBhUjkLFHB4e/pRPNJZEvu1 mz1ImslkChjCweIbA86wvNM68lN2jVGCkyLrmGEk14ULRPre6O71IJQjollJ2UXhG9wt imYIKiCdDHUzWjn23J9pQquUluI0EnZ0Ocpt96cN9ldticIPsanqiXwaP6zSVdg0F4m4 uAdEowrqT+6q6OzxG9VcDTj6drtxBm0kKlJafXa/J7tTndjXwP2GPXvY/oNAr1o8M8Le DPiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ky1hhFEI9Nur/24AXsXA4ap2z+oByp8M83gncjnbIac=; b=3ETLPPm4vbDdEbUUq9rKGBqJrvaEaBYMA+nm5Yn9whUkakJHEGHTggzc6fwFMOjEsc WzZOax28B/7Te+UgvinVi3RXFv6G7uyQyx7xfvy1sKogAbwwCrTAChZgPT4vNXr6Cm7E pG7s67chv6v7ZJ2C9KxEhcoRbhIkMfKFvAqcbW6M1s5RuUWpeMXupgWBgSzHe6+xf1Qc PHwV2qvY9VEb9X4vIH3GzF6znKq8/NzQbetpAKU3FmF3Q63AqrcVBKOY/Ij1zc9EzGyB Yzw0ZgxtL1lo2yydADf6pLDA/yNi/0tNuzkQv803Bpk2dchejK+cCW79VqA73/lZYBC0 jemg== X-Gm-Message-State: AOAM530qZGaQo9Q2Y5qwj1Fc33+qI/YMhdJfvyQFRYQR1pzucWVKaQYG cTj8QRZ8CLHoYgJbO6N03EAekMyrjMWvhw== X-Google-Smtp-Source: ABdhPJz3mKd983gtSyaJdreDdo6qukc8RIvFSZyzOlP9DMqgrkDkKdF3QVl9YcxEAx/sD02ZLm/SxQ== X-Received: by 2002:a05:6000:178b:: with SMTP id e11mr38432525wrg.348.1641301037291; Tue, 04 Jan 2022 04:57:17 -0800 (PST) Received: from naush-laptop.pitowers.org ([2a00:1098:3142:14:cf71:3c12:f0dd:e082]) by smtp.gmail.com with ESMTPSA id p15sm35673753wmj.46.2022.01.04.04.57.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jan 2022 04:57:16 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Tue, 4 Jan 2022 12:57:09 +0000 Message-Id: <20220104125709.1273546-3-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220104125709.1273546-1-naush@raspberrypi.com> References: <20220104125709.1273546-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 2/2] pipeline: raspberrypi: Add support for Video Mux and Bridge devices X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This change will allow the pipeline handler to enumerate and control Video Mux or Bridge devices that may be attached between sensors and a particular Unicam instance. Cascaded mux or bridge devices are also handled. A new member function RPiCameraData::enumerateVideoDevices(), called from PipelineHandlerRPi::registerCamera(), is used to identify and open all mux and bridge subdevices present in the sensor -> Unicam link. Relevant links are enabled/disabled and pad formats correctly set in PipelineHandlerRPi::configure() before the camera is started. Signed-off-by: Naushir Patuck Reviewed-by: Kieran Bingham Reviewed-by: David Plowman --- .../pipeline/raspberrypi/raspberrypi.cpp | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index a6d1e8570abf..168bbcef819c 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -193,6 +194,8 @@ public: int loadIPA(ipa::RPi::SensorConfig *sensorConfig); int configureIPA(const CameraConfiguration *config); + void enumerateVideoDevices(MediaLink *link); + void statsMetadataComplete(uint32_t bufferId, const ControlList &controls); void runIsp(uint32_t bufferId); void embeddedComplete(uint32_t bufferId); @@ -222,6 +225,11 @@ public: std::vector streams_; /* Stores the ids of the buffers mapped in the IPA. */ std::unordered_set ipaBuffers_; + /* + * Stores a cascade of Video Mux or Bridge devices between the sensor and + * Unicam together with media link across the entities. + */ + std::vector, MediaLink *>> bridgeDevices_; /* DMAHEAP allocation helper. */ RPi::DmaHeap dmaHeap_; @@ -908,6 +916,43 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) */ data->properties_.set(properties::ScalerCropMaximum, data->sensorInfo_.analogCrop); + /* Setup the Video Mux/Bridge entities. */ + for (auto &[device, link] : data->bridgeDevices_) { + /* + * Start by disabling all the sink pad links on the devices in the + * cascade, with the exception of the link connecting the device. + */ + for (const MediaPad *p : device->entity()->pads()) { + if (!(p->flags() & MEDIA_PAD_FL_SINK)) + continue; + + for (MediaLink *l : p->links()) { + if (l != link) + l->setEnabled(false); + } + } + + /* + * Next, enable the entity -> entity links, and setup the pad format. + * + * \todo Some bridge devices may chainge the media bus code, so we + * ought to read the source pad format and propagate it to the sink pad. + */ + link->setEnabled(true); + const MediaPad *sinkPad = link->sink(); + ret = device->setFormat(sinkPad->index(), &sensorFormat); + if (ret) { + LOG(RPI, Error) << "Failed to set format on " << device->entity()->name() + << " pad " << sinkPad->index() + << " with format " << format.toString() + << ": " << ret; + return ret; + } + + LOG(RPI, Debug) << "Configured media link on device " << device->entity()->name() + << " on pad " << sinkPad->index(); + } + return ret; } @@ -1138,6 +1183,13 @@ int PipelineHandlerRPi::registerCamera(MediaDevice *unicam, MediaDevice *isp, Me if (data->sensor_->init()) return -EINVAL; + /* + * Enumerate all the Video Mux/Bridge devices across the sensor -> unicam + * chain. There may be a cascade of devices in this chain! + */ + MediaLink *link = sensorEntity->getPadByIndex(0)->links()[0]; + data->enumerateVideoDevices(link); + data->sensorFormats_ = populateSensorFormats(data->sensor_); ipa::RPi::SensorConfig sensorConfig; @@ -1487,6 +1539,98 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config) return 0; } +/* + * enumerateVideoDevices() iterates over the Media Controller topology, starting + * at the sensor and finishing at Unicam. For each sensor, RPiCameraData stores + * a unique list of any intermediate video mux or bridge devices connected in a + * cascade, together with the entity to entity link. + * + * Entity pad configuration and link enabling happens at the end of configure(). + * We first disable all pad links on each entity device in the chain, and then + * selectively enabling the specific links to link sensor to Unicam across all + * intermediate muxes and bridges. + * + * In the cascaded topology below, if Sensor1 is used, the Mux2 -> Mux1 link + * will be disabled, and Sensor1 -> Mux1 -> Unicam links enabled. Alternatively, + * if Sensor3 is used, the Sensor2 -> Mux2 and Sensor1 -> Mux1 links are disabled, + * and Sensor3 -> Mux2 -> Mux1 -> Unicam links are enabled. All other links will + * remain unchanged. + * + * +----------+ + * | Unicam | + * +-----^----+ + * | + * +---+---+ + * | Mux1 <-------+ + * +--^----+ | + * | | + * +-----+---+ +---+---+ + * | Sensor1 | | Mux2 |<--+ + * +---------+ +-^-----+ | + * | | + * +-------+-+ +---+-----+ + * | Sensor2 | | Sensor3 | + * +---------+ +---------+ + */ +void RPiCameraData::enumerateVideoDevices(MediaLink *link) +{ + const MediaPad *sinkPad = link->sink(); + const MediaEntity *entity = sinkPad->entity(); + bool unicamFound = false; + + /* We only deal with Video Mux and Bridge devices in cascade. */ + if (entity->function() != MEDIA_ENT_F_VID_MUX && + entity->function() != MEDIA_ENT_F_VID_IF_BRIDGE) + return; + + /* Find the source pad for this Video Mux or Bridge device. */ + const MediaPad *sourcePad = nullptr; + for (const MediaPad *pad : entity->pads()) { + if (pad->flags() & MEDIA_PAD_FL_SOURCE) { + /* + * We can only deal with devices that have a single source + * pad. If this device has multiple source pads, ignore it + * and this branch in the cascade. + */ + if (sourcePad) + return; + + sourcePad = pad; + } + } + + LOG(RPI, Debug) << "Found video mux device " << entity->name() + << " linked to sink pad " << sinkPad->index(); + + bridgeDevices_.emplace_back(std::make_unique(entity), link); + bridgeDevices_.back().first->open(); + + /* + * Iterate through all the sink pad links down the cascade to find any + * other Video Mux and Bridge devices. + */ + for (MediaLink *l : sourcePad->links()) { + enumerateVideoDevices(l); + /* Once we reach the Unicam entity, we are done. */ + if (l->sink()->entity()->name() == "unicam-image") { + unicamFound = true; + break; + } + } + + /* This identifies the end of our entity enumeration recursion. */ + if (link->source()->entity()->function() == MEDIA_ENT_F_CAM_SENSOR) { + /* + * If Unicam is not at the end of this cascade, we cannot configure + * this topology automatically, so remove all entity references. + */ + if (!unicamFound) { + LOG(RPI, Warning) << "Cannot automatically configure this MC topology!"; + bridgeDevices_.clear(); + } + } +} + void RPiCameraData::statsMetadataComplete(uint32_t bufferId, const ControlList &controls) { if (state_ == State::Stopped)