From patchwork Wed Aug 27 09:07:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 24248 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D3858BD87C for ; Wed, 27 Aug 2025 09:08:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4C4F2692F9; Wed, 27 Aug 2025 11:08:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="kSgHokGk"; dkim-atps=neutral Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E558869300 for ; Wed, 27 Aug 2025 11:07:52 +0200 (CEST) Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-45a286135c8so3536485e9.0 for ; Wed, 27 Aug 2025 02:07:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1756285672; x=1756890472; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WcH7qPcpnlexlEHUZ82vhB+0IeLIPHEAtdRfHfWypxo=; b=kSgHokGk9YcXpqBf1LL/STSscn6Yu1Zb1Ck+VzNqpsZQqP8uChGVdUXn+UPhDq8QQk aX5Hti1v573akB0+oVqPE9MnKunBYB8s6EHvLxn4x13oGBZF5LBB4sRTEcVbyQp+sg8K 8cPK0ROlxKQgWmIkuTRWYbR0trfpzi5HW4tSgrMndXtjekoMVq4U41+G/njaYuYRUnrj yQA4WizqC9ZAG06SrUNwNViEegxoDSHMlhJG5qZH8i5DB1MCGbBRoYWdpYDg6JpbAzAJ a3G+SLOsx29AsqePC9ZtCvWt6eVEQrbtC50bmswjbm6+w6Wavyy4Dn5flNM5gnrST/IN o1hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756285672; x=1756890472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WcH7qPcpnlexlEHUZ82vhB+0IeLIPHEAtdRfHfWypxo=; b=qVbMM/sbl5BB4YaD8qLehs18v/HDMT99QiRmeQw8L5FYq94qfVwqbD4RUGDHZZRLNS KZFcfPgErucal5sxIqjGZCSZd3eNukwAs3LOcD/Ubi1GJNj0YuC0xDenMBF9nkQwZq7L Qhe0hv6FSwZ/qamIUBVEX8TUkW2ldLlYNEG2rtm4d/FOy8p3W3JyZNib3lJMPyYkjUS4 vtXze+l3HzUkLsRQ2grV0am4rQtzFjjO9szxZ6wU/D7mw7xkIqhASAU4G24yaVzcjen2 MfEOM8CFJQtNtHkoIAb22l0eSNQfXTqDTJ6A7GU40M+mQBh4uMYz7xcnbK0WJ0K4mbL7 KDVA== X-Gm-Message-State: AOJu0Yx+4TWrsN4XDlLVmaSuZc71jijBwl4X8Oc66XhvQLQI+MdoGLfk ycPDwk+qC/qKXCcZCcd/npCynP9lWKEKcZkydoPnfON7wDNZZ1zP75BkCPQ/orhJ58hKjjX/+FB IdcI7 X-Gm-Gg: ASbGncvtr99E2rJBBJVU8RWOOCGeFKB/asjyw6Xp98gyTzZBWyn86GKdq3hct4/o5E7 RV1g30k2lo0fYwELB9vLdO9BQNoXmuioKOyPfQKjXPePwew6qPGsQRhp/2OgnhHAVt2hISLBhB5 E50e+oLbhfRxWl5E7nxh1+RzVfypmrjykDHjH9nyvxfXEUsgzQUgJ25ED3cEzylsegMf5STfVys FYEh01Jc50JLtZLFuRxpqx/yJPNjW7T+6MZbKkgy6bA2FPUjjCY8BEsBO82I8Ftyn47dncuEox7 hKDyRvkcofaIdidVYzFKei/43g7L6X6H6ZRSxvNhDmajxMC5DtiqUXYE60LRBqUBUTfqdXmJaUC e2hq/rO4RCAHEvlmFslCHH1GHsnzq9Zx0dv5K8wE6XHUX+1ZuDJxmXszm2YTWeMJhVndAl+hwOv T4HSAQ2TTL8VydDY+uaYFLyDgrcDj5GfMJr4mhthFxLly6CMOeKtm4zltFH14V X-Google-Smtp-Source: AGHT+IE98sXfpWSukxZDWQJbuUxVTl0vaUG9RKGu+/TA2aYhc4GtqOrvwdIan713BZyt05IUkkWbHA== X-Received: by 2002:a05:600c:5246:b0:458:6f13:aa4a with SMTP id 5b1f17b1804b1-45b68b79262mr42107925e9.6.1756285671671; Wed, 27 Aug 2025 02:07:51 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-45b6f30fe02sm21498675e9.18.2025.08.27.02.07.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Aug 2025 02:07:51 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [RFC PATCH 11/12] pipeline: rpi: Support memory cameras Date: Wed, 27 Aug 2025 10:07:38 +0100 Message-Id: <20250827090739.86955-12-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250827090739.86955-1-david.plowman@raspberrypi.com> References: <20250827090739.86955-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Mostly small changes to support memory cameras (for Bayer reprocessing) in the pipeline handler. Firstly, we make a "CameraSensorMemory" as the sensor as soon as we have the raw input buffer configuration, which helps the rest of the to work with fewer changes. The PiSP platformConfigure method is refactored to split the Cfe and Isp parts into separate functions. Memory cameras then omit platformConfigureCfe but run platformConfigureIsp as before. Signed-off-by: David Plowman --- .../pipeline/rpi/common/pipeline_base.cpp | 34 ++++- .../pipeline/rpi/common/pipeline_base.h | 4 +- src/libcamera/pipeline/rpi/pisp/pisp.cpp | 141 ++++++++++++------ 3 files changed, 122 insertions(+), 57 deletions(-) diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp index e7b01f9f..31bacc7c 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp @@ -20,6 +20,7 @@ #include #include "libcamera/internal/camera_lens.h" +#include "libcamera/internal/camera_sensor_memory.h" #include "libcamera/internal/ipa_manager.h" #include "libcamera/internal/v4l2_subdevice.h" @@ -170,15 +171,9 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() status = validateColorSpaces(ColorSpaceFlag::StreamsShareColorSpace); /* - * Validate the requested transform against the sensor capabilities and - * rotation and store the final combined transform that configure() will - * need to apply to the sensor to save us working it out again. + * Separate the raw and output streams first, to make it easier to + * detect the raw reprocessing use case. */ - Orientation requestedOrientation = orientation; - combinedTransform_ = data_->sensor_->computeTransform(&orientation); - if (orientation != requestedOrientation) - status = Adjusted; - rawStreams_.clear(); outStreams_.clear(); unsigned int rawStreamIndex = 0; @@ -191,6 +186,29 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() outStreams_.emplace_back(outStreamIndex++, &cfg); } + /* + * For the reprocessing use case, make a "memory camera" to match the + * raw input buffer. This will make all the subsequent code run more like + * the regular sensor case. + */ + if (rawStreams_[0].cfg->isInput()) { + LOG(RPI, Debug) << "Raw reprocessing use case for " << *rawStreams_[0].cfg; + data_->sensor_ = std::make_unique(*rawStreams_[0].cfg); + /* We can fill in the only sensor format we support! */ + auto const mbusCode = data_->sensor_->mbusCodes()[0]; + data_->sensorFormats_.emplace(mbusCode, data_->sensor_->sizes(mbusCode)); + } + + /* + * Validate the requested transform against the sensor capabilities and + * rotation and store the final combined transform that configure() will + * need to apply to the sensor to save us working it out again. + */ + Orientation requestedOrientation = orientation; + combinedTransform_ = data_->sensor_->computeTransform(&orientation); + if (orientation != requestedOrientation) + status = Adjusted; + /* Sort the streams so the highest resolution is first. */ std::sort(rawStreams_.begin(), rawStreams_.end(), [](auto &l, auto &r) { return l.cfg->size > r.cfg->size; }); diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.h b/src/libcamera/pipeline/rpi/common/pipeline_base.h index 397ad6f8..c53abaa7 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.h +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.h @@ -246,7 +246,7 @@ private: class RPiCameraConfiguration final : public CameraConfiguration { public: - RPiCameraConfiguration(const CameraData *data) + RPiCameraConfiguration(CameraData *data) : CameraConfiguration(), data_(data) { } @@ -288,7 +288,7 @@ public: std::optional rgbColorSpace_; private: - const CameraData *data_; + CameraData *data_; }; } /* namespace RPi */ diff --git a/src/libcamera/pipeline/rpi/pisp/pisp.cpp b/src/libcamera/pipeline/rpi/pisp/pisp.cpp index d18035ec..bb22e6d2 100644 --- a/src/libcamera/pipeline/rpi/pisp/pisp.cpp +++ b/src/libcamera/pipeline/rpi/pisp/pisp.cpp @@ -756,6 +756,10 @@ public: bool adjustDeviceFormat(V4L2DeviceFormat &format) const; private: + int platformConfigureCfe(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat &cfeFormat); + int platformConfigureIsp(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat cfeFormat); int platformConfigure(const RPi::RPiCameraConfiguration *rpiConfig) override; int platformConfigureIpa([[maybe_unused]] ipa::RPi::ConfigParams ¶ms) override @@ -994,6 +998,12 @@ PipelineHandlerPiSP::createMemoryCamera(DeviceEnumerator *enumerator, std::shared_ptr camera = platformCreateCamera(cameraData, nullptr, ispDevice); + int ret = pisp->loadPipelineConfiguration(); + if (ret) { + LOG(RPI, Error) << "Unable to load pipeline configuration"; + return nullptr; + } + return camera; } @@ -1272,10 +1282,14 @@ PiSPCameraData::platformValidate(RPi::RPiCameraConfiguration *rpiConfig) const } if (!rawStreams.empty()) { - rawStreams[0].dev = cfe_[Cfe::Output0].dev(); - StreamConfiguration *rawStream = rawStreams[0].cfg; BayerFormat bayer = BayerFormat::fromPixelFormat(rawStream->pixelFormat); + + if (rawStream->isInput()) + rawStreams[0].dev = isp_[Isp::Input].dev(); + else + rawStreams[0].dev = cfe_[Cfe::Output0].dev(); + /* * We cannot output CSI2 packed or non 16-bit output from the frontend, * so signal the output as unpacked 16-bits in these cases. @@ -1295,7 +1309,7 @@ PiSPCameraData::platformValidate(RPi::RPiCameraConfiguration *rpiConfig) const } rawStreams[0].format = - RPi::PipelineHandlerBase::toV4L2DeviceFormat(cfe_[Cfe::Output0].dev(), rawStream); + RPi::PipelineHandlerBase::toV4L2DeviceFormat(rawStreams[0].dev, rawStream); computeOptimalStride(rawStreams[0].format); } @@ -1489,13 +1503,13 @@ bool PiSPCameraData::adjustDeviceFormat(V4L2DeviceFormat &format) const return false; } -int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConfig) +int PiSPCameraData::platformConfigureCfe(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat &cfeFormat) { const std::vector &rawStreams = rpiConfig->rawStreams_; - const std::vector &outStreams = rpiConfig->outStreams_; int ret; V4L2VideoDevice *cfe = cfe_[Cfe::Output0].dev(); - V4L2DeviceFormat cfeFormat; + V4L2DeviceFormat format; /* * See which streams are requested, and route the user @@ -1540,8 +1554,59 @@ int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConf } ret = cfe->setFormat(&cfeFormat); - if (ret) + + /* CFE statistics output format. */ + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_STATS); + ret = cfe_[Cfe::Stats].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on CFE stats stream: " + << format.toString(); + return ret; + } + + /* CFE config format. */ + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_CFG); + ret = cfe_[Cfe::Config].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on CFE config stream: " + << format.toString(); return ret; + } + + /* + * Configure the CFE embedded data output format only if the sensor + * supports it. + */ + V4L2SubdeviceFormat embeddedFormat; + if (sensorMetadata_) { + sensor_->device()->getFormat(1, &embeddedFormat); + format = {}; + format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); + format.planes[0].size = embeddedFormat.size.width * embeddedFormat.size.height; + + LOG(RPI, Debug) << "Setting embedded data format " << format.toString(); + ret = cfe_[Cfe::Embedded].dev()->setFormat(&format); + if (ret) { + LOG(RPI, Error) << "Failed to set format on CFE embedded: " + << format; + return ret; + } + } + + configureEntities(rpiConfig->sensorFormat_, embeddedFormat); + configureCfe(); + + return 0; +} + +int PiSPCameraData::platformConfigureIsp(const RPi::RPiCameraConfiguration *rpiConfig, + V4L2DeviceFormat cfeFormat) +{ + int ret; + + const std::vector &outStreams = rpiConfig->outStreams_; /* Set the TDN and Stitch node formats in case they are turned on. */ isp_[Isp::TdnOutput].dev()->setFormat(&cfeFormat); @@ -1677,53 +1742,35 @@ int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConf beEnabled_ = beEnables & (PISP_BE_RGB_ENABLE_OUTPUT0 | PISP_BE_RGB_ENABLE_OUTPUT1); - /* CFE statistics output format. */ - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_STATS); - ret = cfe_[Cfe::Stats].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on CFE stats stream: " - << format.toString(); - return ret; - } + if (beEnabled_) + configureBe(rpiConfig->yuvColorSpace_); - /* CFE config format. */ - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_RPI_FE_CFG); - ret = cfe_[Cfe::Config].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on CFE config stream: " - << format.toString(); - return ret; - } + return 0; +} + +int PiSPCameraData::platformConfigure(const RPi::RPiCameraConfiguration *rpiConfig) +{ + /* What we call the cfeFormat here is also the input format for the ISP (Back End). */ + V4L2DeviceFormat cfeFormat; /* - * Configure the CFE embedded data output format only if the sensor - * supports it. + * First configure the CFE (if it's being used). In the case of memory cameras + * (Bayer reprocessing), there's no CFE but the raw input stream should already + * contain the correct ISP input format. */ - V4L2SubdeviceFormat embeddedFormat; - if (sensorMetadata_) { - sensor_->device()->getFormat(1, &embeddedFormat); - format = {}; - format.fourcc = V4L2PixelFormat(V4L2_META_FMT_SENSOR_DATA); - format.planes[0].size = embeddedFormat.size.width * embeddedFormat.size.height; - - LOG(RPI, Debug) << "Setting embedded data format " << format.toString(); - ret = cfe_[Cfe::Embedded].dev()->setFormat(&format); - if (ret) { - LOG(RPI, Error) << "Failed to set format on CFE embedded: " - << format; + if (cfe_[Cfe::Output0].dev()) { + /* Regular sensor. */ + int ret = platformConfigureCfe(rpiConfig, cfeFormat); + if (ret) return ret; - } + } else { + /* Memory camera. */ + cfeFormat = rpiConfig->rawStreams_[0].format; + rpiConfig->rawStreams_[0].cfg->setStream(&isp_[Isp::Input]); } - configureEntities(rpiConfig->sensorFormat_, embeddedFormat); - configureCfe(); - - if (beEnabled_) - configureBe(rpiConfig->yuvColorSpace_); - - return 0; + /* Finally configure the back end ISP. */ + return platformConfigureIsp(rpiConfig, cfeFormat); } void PiSPCameraData::platformStart()