From patchwork Wed Dec 10 16:15:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 25475 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 80693C32DE for ; Wed, 10 Dec 2025 16:41:14 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3F42A61595; Wed, 10 Dec 2025 17:41:14 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="iMgkf0wR"; dkim-atps=neutral Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0C2E0614B1 for ; Wed, 10 Dec 2025 17:41:08 +0100 (CET) Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-47795f6f5c0so46406035e9.1 for ; Wed, 10 Dec 2025 08:41:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1765384867; x=1765989667; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ltnT0SLnKxShoSiZLAZpMr3VNz5MfJiqnPTdurewV0=; b=iMgkf0wRD2/UvybjARqfItqdvLOqFBVuAnYV/qe6wmlXa/SP6By8PeJqdAxlOdfq4j JOdAzR8GFe7wyHhEOZNNZ5A3bhlx040sjiSixxpIoeDbimyq1CVX77WTG7Frmn4lf8mg crOWhbwAstPf/lN5DZPw0gtALBxqebBGA6Pxep8EA/yKGUFh9FKgq+35y8IlbW4DsubC dLOUzsd0P50lktO2lSvzLkV82qLSH4WHZ3zZ+PzfTV6JOEaMajJVWvCzOKMghmpoiLWt Is/xVmpo4VYTGQWtbK/EFjibPXlvZvrxh8VpRc7gyWxXlzm2EAHSabb0DiP87riPiH4L tSrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765384867; x=1765989667; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=4ltnT0SLnKxShoSiZLAZpMr3VNz5MfJiqnPTdurewV0=; b=ubrOZmdELWCZ5jRbDpagBCpeu9Sckx10r5gYHiAIlNskYRe9AEYMfxYKO4J2LiXRsl nxpST7tbWU7jGFEru9mn0XwyFT6A6Z/6tV2Vayafpo1WpBH9yRZU92EhKK2E4Nn61UHj hdAbfKwgI6uUUCuW5wDWANd4d8de55tp4tdlTFDrWhlaMgEk3tAZLsmBzxNqCkA8CKGk heakEhXBcX5G5+e80Uuq9u8BSStLf6v2SsMj8b5695QeaTf9BPS/sqW0KthSL7uhu/Dy UEJUpQnqXoAxBBOjY5JdmDTkSeEfU2jk0wXwnUOHyPlQ1TSTzy/EoWaRNCiWd9Xd+gXE RJTQ== X-Gm-Message-State: AOJu0YxMV3s4UGoeVpx+TJl+r8e/EfZHhGeJOeJdW4MMxsA/ylN+3uYT SR27gq6ZRbjyel1JZooQBrBxOuu/aUyBcm7dlnTcDqlLyPudOUz+9ckS9tMsduYLB5izSkuoCV7 iK8+/ X-Gm-Gg: ASbGncv9+dTw6T18tOmq6r7NJHVZqc9GfIKGN7jgnBzt/i/CMLdy72cYHL7u/wks1CY A+8pGhW08ulO7VonL47CcbNBidC2bt0Fxl1/V/KzDzUMySKLwQBIgIHQWyyUuBxV2HMhaf/7Ac2 Un6oje5+o4b/Duzh5wsdg6En3AUmKb9PYHjYexcEb4Ns7Dh6kluPf9GNPMjljeCHArRxFcpVvYZ cmX3S2VxVGc/zUKe+ks33TXaDGFg51hbuuMzfAo0SkLAO6uY+SAZRaVM4YxHB/VT1BnxbRJOpap 3crsRh0PdNetp2yuahgoPJXLwZ8122fsuVgEIGO8fQst8LcPLmZ128PGDkB4xrk2ncvLD6F5SY1 zNrYqXmAgSQckOFFdzMftHmWdhsfpFbeVsFIpRFE0ftAKScZssyNxQEhDBOIoX0Zp8F6mQJ/PJ9 rUUKxnn7HcidXPcC5cLkxkY56zvqCUdZQaeSXif/CHPcQDKCp6yldHbiw67fAFIN7JJsyBPg6kb 6eQUq3H4pXGIxarPe4z5dylVkOrFS4g2WthM/qm X-Google-Smtp-Source: AGHT+IF1maZPO5gPAzBl4nUdrtf6FV1kCXUNsvAQCQP2+YJKs5CAypjFEHy4aW0NyeVnwfOPGPHFFA== X-Received: by 2002:a05:600c:3587:b0:477:7b9a:bb0a with SMTP id 5b1f17b1804b1-47a85ea3ff7mr14888085e9.21.1765384867116; Wed, 10 Dec 2025 08:41:07 -0800 (PST) Received: from davidp-pi5.pitowers.org ([2a00:1098:3142:1f:88ea:c658:5b20:5e46]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-47a88371e13sm1270415e9.11.2025.12.10.08.41.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Dec 2025 08:41:06 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH 10/11] pipeline: rpi: Support memory cameras Date: Wed, 10 Dec 2025 16:15:25 +0000 Message-ID: <20251210164055.17856-11-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251210164055.17856-1-david.plowman@raspberrypi.com> References: <20251210164055.17856-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Mostly small changes to support memory cameras (for Bayer reprocessing) in the pipeline handler. Firstly, we make a "CameraSensorMemory" as the sensor as soon as we have the raw input buffer configuration, which helps the rest of the code to work with fewer changes. The PiSP platformConfigure method is refactored to split the Cfe and Isp parts into separate functions. Memory cameras then omit platformConfigureCfe but run platformConfigureIsp as before. Signed-off-by: David Plowman --- .../pipeline/rpi/common/pipeline_base.cpp | 36 ++++- .../pipeline/rpi/common/pipeline_base.h | 4 +- src/libcamera/pipeline/rpi/pisp/pisp.cpp | 141 ++++++++++++------ 3 files changed, 124 insertions(+), 57 deletions(-) diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp index 76bcb2a4..2f02ec4c 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp @@ -20,6 +20,7 @@ #include #include "libcamera/internal/camera_lens.h" +#include "libcamera/internal/camera_sensor_memory.h" #include "libcamera/internal/ipa_manager.h" #include "libcamera/internal/v4l2_subdevice.h" @@ -170,15 +171,9 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() status = validateColorSpaces(ColorSpaceFlag::StreamsShareColorSpace); /* - * Validate the requested transform against the sensor capabilities and - * rotation and store the final combined transform that configure() will - * need to apply to the sensor to save us working it out again. + * Separate the raw and output streams first, to make it easier to + * detect the raw reprocessing use case. */ - Orientation requestedOrientation = orientation; - combinedTransform_ = data_->sensor_->computeTransform(&orientation); - if (orientation != requestedOrientation) - status = Adjusted; - rawStreams_.clear(); outStreams_.clear(); unsigned int rawStreamIndex = 0; @@ -191,6 +186,31 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() outStreams_.emplace_back(outStreamIndex++, &cfg); } + /* + * For the reprocessing use case, make a "memory camera" to match the + * raw input buffer. This will make all the subsequent code run more like + * the regular sensor case. + */ + if (!rawStreams_.empty() && rawStreams_[0].cfg->isInput()) { + LOG(RPI, Debug) << "Raw reprocessing use case for " << *rawStreams_[0].cfg; + const StreamConfiguration &rawInput = *rawStreams_[0].cfg; + BayerFormat bayerFormat = BayerFormat::fromPixelFormat(rawInput.pixelFormat); + unsigned int mbusCode = PipelineHandlerBase::bayerToMbusCode(bayerFormat); + data_->sensor_ = std::make_unique(rawInput, mbusCode); + /* We can fill in the only sensor format we support! */ + data_->sensorFormats_.emplace(mbusCode, data_->sensor_->sizes(mbusCode)); + } + + /* + * Validate the requested transform against the sensor capabilities and + * rotation and store the final combined transform that configure() will + * need to apply to the sensor to save us working it out again. + */ + Orientation requestedOrientation = orientation; + combinedTransform_ = data_->sensor_->computeTransform(&orientation); + if (orientation != requestedOrientation) + status = Adjusted; + /* Sort the streams so the highest resolution is first. */ std::sort(rawStreams_.begin(), rawStreams_.end(), [](auto &l, auto &r) { return l.cfg->size > r.cfg->size; }); diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.h b/src/libcamera/pipeline/rpi/common/pipeline_base.h index 9453ae7e..76a269bd 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.h +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.h @@ -250,7 +250,7 @@ private: class RPiCameraConfiguration final : public CameraConfiguration { public: - RPiCameraConfiguration(const CameraData *data) + RPiCameraConfiguration(CameraData *data) : CameraConfiguration(), data_(data) { } @@ -292,7 +292,7 @@ public: std::optional rgbColorSpace_; private: - const CameraData *data_; + CameraData *data_; }; } /* namespace RPi */ diff --git a/src/libcamera/pipeline/rpi/pisp/pisp.cpp b/src/libcamera/pipeline/rpi/pisp/pisp.cpp index 726ab063..da3f5f40 100644 --- a/src/libcamera/pipeline/rpi/pisp/pisp.cpp +++ b/src/libcamera/pipeline/rpi/pisp/pisp.cpp @@ -756,6 +756,10 @@ public: bool adjustDeviceFormat(V4L2DeviceFormat &format) const; private: + int platformConfigureCfe(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat &cfeFormat); + int platformConfigureIsp(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat cfeFormat); int platformConfigure(const RPi::RPiCameraConfiguration *rpiConfig) override; int platformConfigureIpa([[maybe_unused]] ipa::RPi::ConfigParams ¶ms) override @@ -995,6 +999,12 @@ PipelineHandlerPiSP::createMemoryCamera(DeviceEnumerator *enumerator, std::shared_ptr camera = platformCreateCamera(cameraData, nullptr, ispDevice.get()); + int ret = pisp->loadPipelineConfiguration(); + if (ret) { + LOG(RPI, Error) << "Unable to load pipeline configuration"; + return nullptr; + } + return camera; } @@ -1274,10 +1284,14 @@ PiSPCameraData::platformValidate(RPi::RPiCameraConfiguration *rpiConfig) const } if (!rawStreams.empty()) { - rawStreams[0].dev = cfe_[Cfe::Output0].dev(); - StreamConfiguration *rawStream = rawStreams[0].cfg; BayerFormat bayer = BayerFormat::fromPixelFormat(rawStream->pixelFormat); + + if (rawStream->isInput()) + rawStreams[0].dev = isp_[Isp::Input].dev(); + else + rawStreams[0].dev = cfe_[Cfe::Output0].dev(); + /* * We cannot output CSI2 packed or non 16-bit output from the frontend, * so signal the output as unpacked 16-bits in these cases. @@ -1297,7 +1311,7 @@ PiSPCameraData::platformValidate(RPi::RPiCameraConfiguration *rpiConfig) const } rawStreams[0].format = - RPi::PipelineHandlerBase::toV4L2DeviceFormat(cfe_[Cfe::Output0].dev(), rawStream); + RPi::PipelineHandlerBase::toV4L2DeviceFormat(rawStreams[0].dev, rawStream); computeOptimalStride(rawStreams[0].format); } @@ -1491,13 +1505,13 @@ bool PiSPCameraData::adjustDeviceFormat(V4L2DeviceFormat &format) const return false; } -int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConfig) +int PiSPCameraData::platformConfigureCfe(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat &cfeFormat) { const std::vector &rawStreams = rpiConfig->rawStreams_; - const std::vector &outStreams = rpiConfig->outStreams_; int ret; V4L2VideoDevice *cfe = cfe_[Cfe::Output0].dev(); - V4L2DeviceFormat cfeFormat; + V4L2DeviceFormat format; /* * See which streams are requested, and route the user @@ -1542,8 +1556,59 @@ int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConf } ret = cfe->setFormat(&cfeFormat); - if (ret) + + /* CFE statistics output format. */ + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_STATS); + ret = cfe_[Cfe::Stats].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on CFE stats stream: " + << format.toString(); + return ret; + } + + /* CFE config format. */ + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_CFG); + ret = cfe_[Cfe::Config].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on CFE config stream: " + << format.toString(); return ret; + } + + /* + * Configure the CFE embedded data output format only if the sensor + * supports it. + */ + V4L2SubdeviceFormat embeddedFormat; + if (sensorMetadata_) { + sensor_->device()->getFormat(1, &embeddedFormat); + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); + format.planes[0].size = embeddedFormat.size.width * embeddedFormat.size.height; + + LOG(RPI, Debug) << "Setting embedded data format " << format.toString(); + ret = cfe_[Cfe::Embedded].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on CFE embedded: " + << format; + return ret; + } + } + + configureEntities(rpiConfig->sensorFormat_, embeddedFormat); + configureCfe(); + + return 0; +} + +int PiSPCameraData::platformConfigureIsp(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat cfeFormat) +{ + int ret; + + const std::vector &outStreams = rpiConfig->outStreams_; /* Set the TDN and Stitch node formats in case they are turned on. */ isp_[Isp::TdnOutput].dev()->setFormat(&cfeFormat); @@ -1679,53 +1744,35 @@ int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConf beEnabled_ = beEnables & (PISP_BE_RGB_ENABLE_OUTPUT0 | PISP_BE_RGB_ENABLE_OUTPUT1); - /* CFE statistics output format. */ - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_STATS); - ret = cfe_[Cfe::Stats].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on CFE stats stream: " - << format.toString(); - return ret; - } + if (beEnabled_) + configureBe(rpiConfig->yuvColorSpace_); - /* CFE config format. */ - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_CFG); - ret = cfe_[Cfe::Config].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on CFE config stream: " - << format.toString(); - return ret; - } + return 0; +} + +int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConfig) +{ + /* What we call the cfeFormat here is also the input format for the ISP (Back End). */ + V4L2DeviceFormat cfeFormat; /* - * Configure the CFE embedded data output format only if the sensor - * supports it. + * First configure the CFE (if it's being used). In the case of memory cameras + * (Bayer reprocessing), there's no CFE but the raw input stream should already + * contain the correct ISP input format. */ - V4L2SubdeviceFormat embeddedFormat; - if (sensorMetadata_) { - sensor_->device()->getFormat(1, &embeddedFormat); - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); - format.planes[0].size = embeddedFormat.size.width * embeddedFormat.size.height; - - LOG(RPI, Debug) << "Setting embedded data format " << format.toString(); - ret = cfe_[Cfe::Embedded].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on CFE embedded: " - << format; + if (cfe_[Cfe::Output0].dev()) { + /* Regular sensor. */ + int ret = platformConfigureCfe(rpiConfig, cfeFormat); + if (ret) return ret; - } + } else { + /* Memory camera. */ + cfeFormat = rpiConfig->rawStreams_[0].format; + rpiConfig->rawStreams_[0].cfg->setStream(&isp_[Isp::Input]); } - configureEntities(rpiConfig->sensorFormat_, embeddedFormat); - configureCfe(); - - if (beEnabled_) - configureBe(rpiConfig->yuvColorSpace_); - - return 0; + /* Finally configure the back end ISP. */ + return platformConfigureIsp(rpiConfig, cfeFormat); } void PiSPCameraData::platformStart()