{"id":14359,"url":"https://patchwork.libcamera.org/api/1.1/patches/14359/?format=json","web_url":"https://patchwork.libcamera.org/patch/14359/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20211027092803.3671096-6-naush@raspberrypi.com>","date":"2021-10-27T09:27:59","name":"[libcamera-devel,v3,5/9] pipeline: raspberrypi: Convert the pipeline handler to use media controller","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"c25954d48eb19469c1a4c7bc3ea0ccbfb82e8a5c","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/1.1/people/34/?format=json","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/14359/mbox/","series":[{"id":2660,"url":"https://patchwork.libcamera.org/api/1.1/series/2660/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2660","date":"2021-10-27T09:27:54","name":"Raspberry Pi: Conversion to media controller","version":3,"mbox":"https://patchwork.libcamera.org/series/2660/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/14359/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/14359/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BFC91BDB1C\n\tfor ;\n\tWed, 27 Oct 2021 09:28:17 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4B60364893;\n\tWed, 27 Oct 2021 11:28:17 +0200 (CEST)","from mail-wm1-x334.google.com (mail-wm1-x334.google.com\n\t[IPv6:2a00:1450:4864:20::334])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A94F664889\n\tfor ;\n\tWed, 27 Oct 2021 11:28:11 +0200 (CEST)","by mail-wm1-x334.google.com with SMTP id\n\to4-20020a1c7504000000b0032cab7473caso2541199wmc.1\n\tfor ;\n\tWed, 27 Oct 2021 02:28:11 -0700 (PDT)","from naush-laptop.pitowers.org\n\t([2a00:1098:3142:14:6561:eb24:3f00:ce98])\n\tby smtp.gmail.com with ESMTPSA id\n\tx2sm2861360wmj.3.2021.10.27.02.28.10\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tWed, 27 Oct 2021 02:28:10 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"VA9RWh+M\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=from:to:cc:subject:date:message-id:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=OCYzorYT+INRNlFq8S/Bq/qNDj5LEcDEMz5THw7OaoE=;\n\tb=VA9RWh+Mp8bL743YOO+nwgYnVzEsjVCKXuiwPDs7207RcglbkToeLOZtbdNbyxKnEz\n\tXgvu81TzEHweLtZrGLO31TEWQ86njAex/EqKb8alxsue+fSgVLhMIFaXPq02ZI2uEnJS\n\t38XpwJ9+LwDeV0IUtKl/36GIpM5oBPR/cpUzkHmmXN06LyLkPup/AvZZHYn5eP/z5gAl\n\tA2gnf4qZmX0G3udNaAvd6Xyplc5+vKbTe5XbDPcO6hgRifY6305OwqjWpEuGoJ57+W1r\n\ton4Wi6dLg6IyQWQRmynZ6vL2mhBT99dcHm1RtI6byGpKjo0fbxlWIw3RDBOaC0Wo/7xx\n\tENXA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=OCYzorYT+INRNlFq8S/Bq/qNDj5LEcDEMz5THw7OaoE=;\n\tb=brq7klI/LqWei1smU1rQ90oVYnGj6GTkoHEQG1oeClmVCsIs+mRF6iKtRwPcmEGpg4\n\t9CO2D9shnEpYRAkT0qGcd5B6z/VLvJScK9n8zrLW4nZfC9uFM/tJcRGsq9Qxean1PvUH\n\t2C58cQ57hMSe1fSrJT0d6JahFipxOp3kS75OPXCwFTEbu9hFFdwsIP8AvqV2hta60qbL\n\tjc2TCt/UakTE7uxdYKZ+0y+zkqI995ZbOdJsnB/L/6VUs3YaVG2S7eIm1y//j2J+DvkY\n\tM/z7g1mcVLy3nj3gRFN247WU3SbxuGSuKk06Uej3cbFg1lpBXQ5t0ympskf24NZBUcNa\n\t3xnw==","X-Gm-Message-State":"AOAM533+lmL6xgQp96EeSAWhUXJ8Wxz/ZZOjK58DRvVicCgsyJidiOdW\n\tniXhm21qXzxOh/NsgK1PfXmOD8LzuZK4xCga","X-Google-Smtp-Source":"ABdhPJyKD9qZZ54YtnRw3DovLUXVMk70+xA42y63egne0UH3UdfWKEjd+Z+jnYZuipZlJ12Vn+5AHg==","X-Received":"by 2002:a1c:9d56:: with SMTP id g83mr379737wme.58.1635326890761; \n\tWed, 27 Oct 2021 02:28:10 -0700 (PDT)","From":"Naushir Patuck ","To":"libcamera-devel@lists.libcamera.org","Date":"Wed, 27 Oct 2021 10:27:59 +0100","Message-Id":"<20211027092803.3671096-6-naush@raspberrypi.com>","X-Mailer":"git-send-email 2.25.1","In-Reply-To":"<20211027092803.3671096-1-naush@raspberrypi.com>","References":"<20211027092803.3671096-1-naush@raspberrypi.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v3 5/9] pipeline: raspberrypi: Convert the\n\tpipeline handler to use media controller","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Switch the pipeline handler to use the new Unicam media controller based driver.\nWith this change, we directly talk to the sensor device driver to set controls\nand set/get formats in the pipeline handler.\n\nThis change requires the accompanying Raspberry Pi linux kernel change at\nhttps://github.com/raspberrypi/linux/pull/4645. If this kernel change is not\npresent, the pipeline handler will fail to run with an error message informing\nthe user to update the kernel build.\n\nSigned-off-by: Naushir Patuck \nReviewed-by: David Plowman \n---\n .../pipeline/raspberrypi/raspberrypi.cpp | 185 ++++++++++++------\n 1 file changed, 127 insertions(+), 58 deletions(-)","diff":"diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\nindex 1634ca98f481..48f561d31a31 100644\n--- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n+++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n@@ -26,6 +26,7 @@\n #include \n \n #include \n+#include \n #include \n \n #include \"libcamera/internal/bayer_format.h\"\n@@ -48,6 +49,66 @@ LOG_DEFINE_CATEGORY(RPI)\n \n namespace {\n \n+/* Map of mbus codes to supported sizes reported by the sensor. */\n+using SensorFormats = std::map>;\n+\n+SensorFormats populateSensorFormats(std::unique_ptr &sensor)\n+{\n+\tSensorFormats formats;\n+\n+\tfor (auto const mbusCode : sensor->mbusCodes())\n+\t\tformats.emplace(mbusCode, sensor->sizes(mbusCode));\n+\n+\treturn formats;\n+}\n+\n+BayerFormat mbusCodeToBayerFormat(unsigned int mbusCode)\n+{\n+\tstatic const std::unordered_map mbusCodeToBayer {\n+\t\t{ MEDIA_BUS_FMT_SBGGR8_1X8, { BayerFormat::BGGR, 8, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGBRG8_1X8, { BayerFormat::GBRG, 8, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGRBG8_1X8, { BayerFormat::GRBG, 8, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SRGGB8_1X8, { BayerFormat::RGGB, 8, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SBGGR10_1X10, { BayerFormat::BGGR, 10, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGBRG10_1X10, { BayerFormat::GBRG, 10, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGRBG10_1X10, { BayerFormat::GRBG, 10, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SRGGB10_1X10, { BayerFormat::RGGB, 10, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SBGGR12_1X12, { BayerFormat::BGGR, 12, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGBRG12_1X12, { BayerFormat::GBRG, 12, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGRBG12_1X12, { BayerFormat::GRBG, 12, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SRGGB12_1X12, { BayerFormat::RGGB, 12, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SBGGR14_1X14, { BayerFormat::BGGR, 14, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGBRG14_1X14, { BayerFormat::GBRG, 14, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGRBG14_1X14, { BayerFormat::GRBG, 14, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SRGGB14_1X14, { BayerFormat::RGGB, 14, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SBGGR16_1X16, { BayerFormat::BGGR, 16, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGBRG16_1X16, { BayerFormat::GBRG, 16, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SGRBG16_1X16, { BayerFormat::GRBG, 16, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_SRGGB16_1X16, { BayerFormat::RGGB, 16, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_Y8_1X8, { BayerFormat::MONO, 8, BayerFormat::None } },\n+\t\t{ MEDIA_BUS_FMT_Y10_1X10, { BayerFormat::MONO, 10, BayerFormat::None } },\n+\t};\n+\n+\tconst auto it = mbusCodeToBayer.find(mbusCode);\n+\tif (it != mbusCodeToBayer.end())\n+\t\treturn it->second;\n+\n+\treturn BayerFormat{};\n+}\n+\n+V4L2DeviceFormat toV4L2DeviceFormat(V4L2SubdeviceFormat &format)\n+{\n+\tV4L2DeviceFormat deviceFormat;\n+\tBayerFormat bayer = mbusCodeToBayerFormat(format.mbus_code);\n+\n+\tASSERT(bayer.isValid());\n+\n+\tbayer.packing = BayerFormat::Packing::CSI2Packed;\n+\tdeviceFormat.fourcc = bayer.toV4L2PixelFormat();\n+\tdeviceFormat.size = format.size;\n+\treturn deviceFormat;\n+}\n+\n bool isRaw(PixelFormat &pixFmt)\n {\n \t/*\n@@ -74,11 +135,10 @@ double scoreFormat(double desired, double actual)\n \treturn score;\n }\n \n-V4L2DeviceFormat findBestMode(V4L2VideoDevice::Formats &formatsMap,\n-\t\t\t const Size &req)\n+V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size &req)\n {\n \tdouble bestScore = std::numeric_limits::max(), score;\n-\tV4L2DeviceFormat bestMode;\n+\tV4L2SubdeviceFormat bestFormat;\n \n #define PENALTY_AR\t\t1500.0\n #define PENALTY_8BIT\t\t2000.0\n@@ -88,19 +148,18 @@ V4L2DeviceFormat findBestMode(V4L2VideoDevice::Formats &formatsMap,\n \n \t/* Calculate the closest/best mode from the user requested size. */\n \tfor (const auto &iter : formatsMap) {\n-\t\tV4L2PixelFormat v4l2Format = iter.first;\n-\t\tconst PixelFormatInfo &info = PixelFormatInfo::info(v4l2Format);\n+\t\tconst unsigned int mbusCode = iter.first;\n+\t\tconst PixelFormat format = mbusCodeToBayerFormat(mbusCode).toPixelFormat();\n+\t\tconst PixelFormatInfo &info = PixelFormatInfo::info(format);\n \n-\t\tfor (const SizeRange &sz : iter.second) {\n-\t\t\tdouble modeWidth = sz.contains(req) ? req.width : sz.max.width;\n-\t\t\tdouble modeHeight = sz.contains(req) ? req.height : sz.max.height;\n+\t\tfor (const Size &size : iter.second) {\n \t\t\tdouble reqAr = static_cast(req.width) / req.height;\n-\t\t\tdouble modeAr = modeWidth / modeHeight;\n+\t\t\tdouble fmtAr = size.width / size.height;\n \n \t\t\t/* Score the dimensions for closeness. */\n-\t\t\tscore = scoreFormat(req.width, modeWidth);\n-\t\t\tscore += scoreFormat(req.height, modeHeight);\n-\t\t\tscore += PENALTY_AR * scoreFormat(reqAr, modeAr);\n+\t\t\tscore = scoreFormat(req.width, size.width);\n+\t\t\tscore += scoreFormat(req.height, size.height);\n+\t\t\tscore += PENALTY_AR * scoreFormat(reqAr, fmtAr);\n \n \t\t\t/* Add any penalties... this is not an exact science! */\n \t\t\tif (!info.packed)\n@@ -115,18 +174,18 @@ V4L2DeviceFormat findBestMode(V4L2VideoDevice::Formats &formatsMap,\n \n \t\t\tif (score <= bestScore) {\n \t\t\t\tbestScore = score;\n-\t\t\t\tbestMode.fourcc = v4l2Format;\n-\t\t\t\tbestMode.size = Size(modeWidth, modeHeight);\n+\t\t\t\tbestFormat.mbus_code = mbusCode;\n+\t\t\t\tbestFormat.size = size;\n \t\t\t}\n \n-\t\t\tLOG(RPI, Info) << \"Mode: \" << modeWidth << \"x\" << modeHeight\n-\t\t\t\t << \" fmt \" << v4l2Format.toString()\n+\t\t\tLOG(RPI, Info) << \"Format: \" << size.toString()\n+\t\t\t\t << \" fmt \" << format.toString()\n \t\t\t\t << \" Score: \" << score\n \t\t\t\t << \" (best \" << bestScore << \")\";\n \t\t}\n \t}\n \n-\treturn bestMode;\n+\treturn bestFormat;\n }\n \n enum class Unicam : unsigned int { Image, Embedded };\n@@ -170,6 +229,7 @@ public:\n \tstd::unique_ptr ipa_;\n \n \tstd::unique_ptr sensor_;\n+\tSensorFormats sensorFormats_;\n \t/* Array of Unicam and ISP device streams and associated buffers/streams. */\n \tRPi::Device unicam_;\n \tRPi::Device isp_;\n@@ -352,9 +412,9 @@ CameraConfiguration::Status RPiCameraConfiguration::validate()\n \t\t\t * Calculate the best sensor mode we can use based on\n \t\t\t * the user request.\n \t\t\t */\n-\t\t\tV4L2VideoDevice::Formats fmts = data_->unicam_[Unicam::Image].dev()->formats();\n-\t\t\tV4L2DeviceFormat sensorFormat = findBestMode(fmts, cfg.size);\n-\t\t\tint ret = data_->unicam_[Unicam::Image].dev()->tryFormat(&sensorFormat);\n+\t\t\tV4L2SubdeviceFormat sensorFormat = findBestFormat(data_->sensorFormats_, cfg.size);\n+\t\t\tV4L2DeviceFormat unicamFormat = toV4L2DeviceFormat(sensorFormat);\n+\t\t\tint ret = data_->unicam_[Unicam::Image].dev()->tryFormat(&unicamFormat);\n \t\t\tif (ret)\n \t\t\t\treturn Invalid;\n \n@@ -366,7 +426,7 @@ CameraConfiguration::Status RPiCameraConfiguration::validate()\n \t\t\t * fetch the \"native\" (i.e. untransformed) Bayer order,\n \t\t\t * because the sensor may currently be flipped!\n \t\t\t */\n-\t\t\tV4L2PixelFormat fourcc = sensorFormat.fourcc;\n+\t\t\tV4L2PixelFormat fourcc = unicamFormat.fourcc;\n \t\t\tif (data_->flipsAlterBayerOrder_) {\n \t\t\t\tBayerFormat bayer = BayerFormat::fromV4L2PixelFormat(fourcc);\n \t\t\t\tbayer.order = data_->nativeBayerOrder_;\n@@ -375,15 +435,15 @@ CameraConfiguration::Status RPiCameraConfiguration::validate()\n \t\t\t}\n \n \t\t\tPixelFormat sensorPixFormat = fourcc.toPixelFormat();\n-\t\t\tif (cfg.size != sensorFormat.size ||\n+\t\t\tif (cfg.size != unicamFormat.size ||\n \t\t\t cfg.pixelFormat != sensorPixFormat) {\n-\t\t\t\tcfg.size = sensorFormat.size;\n+\t\t\t\tcfg.size = unicamFormat.size;\n \t\t\t\tcfg.pixelFormat = sensorPixFormat;\n \t\t\t\tstatus = Adjusted;\n \t\t\t}\n \n-\t\t\tcfg.stride = sensorFormat.planes[0].bpl;\n-\t\t\tcfg.frameSize = sensorFormat.planes[0].size;\n+\t\t\tcfg.stride = unicamFormat.planes[0].bpl;\n+\t\t\tcfg.frameSize = unicamFormat.planes[0].size;\n \n \t\t\trawCount++;\n \t\t} else {\n@@ -472,7 +532,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera,\n {\n \tRPiCameraData *data = cameraData(camera);\n \tCameraConfiguration *config = new RPiCameraConfiguration(data);\n-\tV4L2DeviceFormat sensorFormat;\n+\tV4L2SubdeviceFormat sensorFormat;\n \tunsigned int bufferCount;\n \tPixelFormat pixelFormat;\n \tV4L2VideoDevice::Formats fmts;\n@@ -487,9 +547,8 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera,\n \t\tswitch (role) {\n \t\tcase StreamRole::Raw:\n \t\t\tsize = data->sensor_->resolution();\n-\t\t\tfmts = data->unicam_[Unicam::Image].dev()->formats();\n-\t\t\tsensorFormat = findBestMode(fmts, size);\n-\t\t\tpixelFormat = sensorFormat.fourcc.toPixelFormat();\n+\t\t\tsensorFormat = findBestFormat(data->sensorFormats_, size);\n+\t\t\tpixelFormat = mbusCodeToBayerFormat(sensorFormat.mbus_code).toPixelFormat();\n \t\t\tASSERT(pixelFormat.isValid());\n \t\t\tbufferCount = 2;\n \t\t\trawCount++;\n@@ -599,32 +658,29 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n \t}\n \n \t/* First calculate the best sensor mode we can use based on the user request. */\n-\tV4L2VideoDevice::Formats fmts = data->unicam_[Unicam::Image].dev()->formats();\n-\tV4L2DeviceFormat sensorFormat = findBestMode(fmts, rawStream ? sensorSize : maxSize);\n+\tV4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize);\n+\tret = data->sensor_->setFormat(&sensorFormat);\n+\tif (ret)\n+\t\treturn ret;\n \n \t/*\n \t * Unicam image output format. The ISP input format gets set at start,\n \t * just in case we have swapped bayer orders due to flips.\n \t */\n-\tret = data->unicam_[Unicam::Image].dev()->setFormat(&sensorFormat);\n+\tV4L2DeviceFormat unicamFormat = toV4L2DeviceFormat(sensorFormat);\n+\tret = data->unicam_[Unicam::Image].dev()->setFormat(&unicamFormat);\n \tif (ret)\n \t\treturn ret;\n \n-\t/*\n-\t * The control ranges associated with the sensor may need updating\n-\t * after a format change.\n-\t * \\todo Use the CameraSensor::setFormat API instead.\n-\t */\n-\tdata->sensor_->updateControlInfo();\n-\n \tLOG(RPI, Info) << \"Sensor: \" << camera->id()\n-\t\t << \" - Selected mode: \" << sensorFormat.toString();\n+\t\t << \" - Selected sensor format: \" << sensorFormat.toString()\n+\t\t << \" - Selected unicam format: \" << unicamFormat.toString();\n \n \t/*\n \t * This format may be reset on start() if the bayer order has changed\n \t * because of flips in the sensor.\n \t */\n-\tret = data->isp_[Isp::Input].dev()->setFormat(&sensorFormat);\n+\tret = data->isp_[Isp::Input].dev()->setFormat(&unicamFormat);\n \tif (ret)\n \t\treturn ret;\n \n@@ -746,8 +802,8 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n \tdata->ispMinCropSize_ = testCrop.size();\n \n \t/* Adjust aspect ratio by providing crops on the input image. */\n-\tSize size = sensorFormat.size.boundedToAspectRatio(maxSize);\n-\tRectangle crop = size.centeredTo(Rectangle(sensorFormat.size).center());\n+\tSize size = unicamFormat.size.boundedToAspectRatio(maxSize);\n+\tRectangle crop = size.centeredTo(Rectangle(unicamFormat.size).center());\n \tdata->ispCrop_ = crop;\n \n \tdata->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop);\n@@ -761,8 +817,11 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n \t * supports it.\n \t */\n \tif (data->sensorMetadata_) {\n-\t\tformat = {};\n+\t\tV4L2SubdeviceFormat embeddedFormat;\n+\n+\t\tdata->sensor_->device()->getFormat(1, &embeddedFormat);\n \t\tformat.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA);\n+\t\tformat.planes[0].size = embeddedFormat.size.width * embeddedFormat.size.height;\n \n \t\tLOG(RPI, Debug) << \"Setting embedded data format.\";\n \t\tret = data->unicam_[Unicam::Embedded].dev()->setFormat(&format);\n@@ -847,9 +906,14 @@ int PipelineHandlerRPi::start(Camera *camera, const ControlList *controls)\n \t * IPA configure may have changed the sensor flips - hence the bayer\n \t * order. Get the sensor format and set the ISP input now.\n \t */\n-\tV4L2DeviceFormat sensorFormat;\n-\tdata->unicam_[Unicam::Image].dev()->getFormat(&sensorFormat);\n-\tret = data->isp_[Isp::Input].dev()->setFormat(&sensorFormat);\n+\tV4L2SubdeviceFormat sensorFormat;\n+\tdata->sensor_->device()->getFormat(0, &sensorFormat);\n+\n+\tV4L2DeviceFormat ispFormat;\n+\tispFormat.fourcc = mbusCodeToBayerFormat(sensorFormat.mbus_code).toV4L2PixelFormat();\n+\tispFormat.size = sensorFormat.size;\n+\n+\tret = data->isp_[Isp::Input].dev()->setFormat(&ispFormat);\n \tif (ret) {\n \t\tstop(camera);\n \t\treturn ret;\n@@ -1004,6 +1068,8 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator)\n \tif (data->sensor_->init())\n \t\treturn false;\n \n+\tdata->sensorFormats_ = populateSensorFormats(data->sensor_);\n+\n \tipa::RPi::SensorConfig sensorConfig;\n \tif (data->loadIPA(&sensorConfig)) {\n \t\tLOG(RPI, Error) << \"Failed to load a suitable IPA library\";\n@@ -1030,6 +1096,11 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator)\n \t\t\treturn false;\n \t}\n \n+\tif (!(data->unicam_[Unicam::Image].dev()->caps().device_caps() & V4L2_CAP_IO_MC)) {\n+\t\tLOG(RPI, Error) << \"Unicam driver did not advertise V4L2_CAP_IO_MC, please update your kernel!\";\n+\t\treturn false;\n+\t}\n+\n \t/*\n \t * Setup our delayed control writer with the sensor default\n \t * gain and exposure delays. Mark VBLANK for priority write.\n@@ -1039,7 +1110,7 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator)\n \t\t{ V4L2_CID_EXPOSURE, { sensorConfig.exposureDelay, false } },\n \t\t{ V4L2_CID_VBLANK, { sensorConfig.vblankDelay, true } }\n \t};\n-\tdata->delayedCtrls_ = std::make_unique(data->unicam_[Unicam::Image].dev(), params);\n+\tdata->delayedCtrls_ = std::make_unique(data->sensor_->device(), params);\n \tdata->sensorMetadata_ = sensorConfig.sensorMetadata;\n \n \t/* Register the controls that the Raspberry Pi IPA can handle. */\n@@ -1066,15 +1137,14 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator)\n \t * As part of answering the final question, we reset the camera to\n \t * no transform at all.\n \t */\n-\n-\tV4L2VideoDevice *dev = data->unicam_[Unicam::Image].dev();\n-\tconst struct v4l2_query_ext_ctrl *hflipCtrl = dev->controlInfo(V4L2_CID_HFLIP);\n+\tconst V4L2Subdevice *sensor = data->sensor_->device();\n+\tconst struct v4l2_query_ext_ctrl *hflipCtrl = sensor->controlInfo(V4L2_CID_HFLIP);\n \tif (hflipCtrl) {\n \t\t/* We assume it will support vflips too... */\n \t\tdata->supportsFlips_ = true;\n \t\tdata->flipsAlterBayerOrder_ = hflipCtrl->flags & V4L2_CTRL_FLAG_MODIFY_LAYOUT;\n \n-\t\tControlList ctrls(dev->controls());\n+\t\tControlList ctrls(data->sensor_->controls());\n \t\tctrls.set(V4L2_CID_HFLIP, 0);\n \t\tctrls.set(V4L2_CID_VFLIP, 0);\n \t\tdata->setSensorControls(ctrls);\n@@ -1082,9 +1152,8 @@ bool PipelineHandlerRPi::match(DeviceEnumerator *enumerator)\n \n \t/* Look for a valid Bayer format. */\n \tBayerFormat bayerFormat;\n-\tfor (const auto &iter : dev->formats()) {\n-\t\tV4L2PixelFormat v4l2Format = iter.first;\n-\t\tbayerFormat = BayerFormat::fromV4L2PixelFormat(v4l2Format);\n+\tfor (const auto &iter : data->sensorFormats_) {\n+\t\tbayerFormat = mbusCodeToBayerFormat(iter.first);\n \t\tif (bayerFormat.isValid())\n \t\t\tbreak;\n \t}\n@@ -1271,7 +1340,7 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config)\n \t\t}\n \t}\n \n-\tentityControls.emplace(0, unicam_[Unicam::Image].dev()->controls());\n+\tentityControls.emplace(0, sensor_->controls());\n \tentityControls.emplace(1, isp_[Isp::Input].dev()->controls());\n \n \t/* Always send the user transform to the IPA. */\n@@ -1406,10 +1475,10 @@ void RPiCameraData::setSensorControls(ControlList &controls)\n \t\tControlList vblank_ctrl;\n \n \t\tvblank_ctrl.set(V4L2_CID_VBLANK, controls.get(V4L2_CID_VBLANK));\n-\t\tunicam_[Unicam::Image].dev()->setControls(&vblank_ctrl);\n+\t\tsensor_->setControls(&vblank_ctrl);\n \t}\n \n-\tunicam_[Unicam::Image].dev()->setControls(&controls);\n+\tsensor_->setControls(&controls);\n }\n \n void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer)\n","prefixes":["libcamera-devel","v3","5/9"]}