| Message ID | 20260325-mali-cru-v6-4-b16b0c49819a@ideasonboard.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series |
|
| Related | show |
2026. 03. 25. 15:44 keltezéssel, Jacopo Mondi írta: > Add support to Mali C55 pipeline handler for memory-to-memory camera > mode. > > The Mali-C55, as integrated in the Renesas RZ/V2H(P) SoC, is fed with > images saved to memory by the CRU unit and read on behalf of the ISP > by the IVC. > > Update the pipeline handler's match() function to search for the CRU > media entity necessary to register a memory input camera. Create and > initialize a CameraData instance for the memory camera use case if a > valid CRU is found and initialize the IVC components used to feed the > ISP with images read from memory. > > Do not implement camera configuration for the time being to reuce > the patch size. It will be provided in the following patches. > > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > --- > src/libcamera/pipeline/mali-c55/mali-c55.cpp | 135 +++++++++++++++++++++++---- > 1 file changed, 117 insertions(+), 18 deletions(-) > > diff --git a/src/libcamera/pipeline/mali-c55/mali-c55.cpp b/src/libcamera/pipeline/mali-c55/mali-c55.cpp > index dca82564e2eb..26d368bb9132 100644 > --- a/src/libcamera/pipeline/mali-c55/mali-c55.cpp > +++ b/src/libcamera/pipeline/mali-c55/mali-c55.cpp > [...] > @@ -1682,14 +1753,14 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > * The TPG and the downscale pipe are both optional blocks and may not > * be fitted. > */ > - DeviceMatch dm("mali-c55"); > - dm.add("mali-c55 isp"); > - dm.add("mali-c55 resizer fr"); > - dm.add("mali-c55 fr"); > - dm.add("mali-c55 3a stats"); > - dm.add("mali-c55 3a params"); > - > - media_ = acquireMediaDevice(enumerator, dm); > + DeviceMatch c55_dm("mali-c55"); Minor thing, this should probably be `c55Dm`. > + c55_dm.add("mali-c55 isp"); > + c55_dm.add("mali-c55 resizer fr"); > + c55_dm.add("mali-c55 fr"); > + c55_dm.add("mali-c55 3a stats"); > + c55_dm.add("mali-c55 3a params"); > + > + media_ = acquireMediaDevice(enumerator, c55_dm); > if (!media_) > return false; > > @@ -1749,6 +1820,25 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > stats_->bufferReady.connect(this, &PipelineHandlerMaliC55::statsBufferReady); > params_->bufferReady.connect(this, &PipelineHandlerMaliC55::paramsBufferReady); > > + /* > + * We also need to search for the rzg2l-cru CSI-2 receiver. If we find > + * that then we need to work in memory input mode instead of the inline > + * mode. The absence of this match is not necessarily a failure at this > + * point...it depends on the media links that we investigate momentarily. > + * > + * This is a bit hacky, because there could be multiple of these media > + * devices and we're just taking the first. We need modular pipelines to > + * properly solve the issue. > + */ > + static const std::regex cruCsi2Regex("csi-[0-9a-f]{8}.csi2"); > + static const std::regex cruIpRegex("cru-ip-[0-9a-f]{8}.cru[0-9]"); > + > + DeviceMatch cruDm("rzg2l_cru"); > + cruDm.add(cruCsi2Regex); > + cruDm.add(cruIpRegex); > + cruDm.add("CRU output"); > + cruMedia_ = acquireMediaDevice(enumerator, cruDm); > + > ispSink = isp_->entity()->getPadByIndex(0); > if (!ispSink || ispSink->links().empty()) { > LOG(MaliC55, Error) << "ISP sink pad error"; > @@ -1762,13 +1852,6 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > * MEDIA_ENT_F_CAM_SENSOR - The test pattern generator > * MEDIA_ENT_F_VID_IF_BRIDGE - A CSI-2 receiver > * MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER - An input device > - * > - * The last one will be unsupported for now. The TPG is relatively easy, > - * we just register a Camera for it. If we have a CSI-2 receiver we need > - * to check its sink pad and register Cameras for anything connected to > - * it (probably...there are some complex situations in which that might > - * not be true but let's pretend they don't exist until we come across > - * them) > */ > bool registered; > for (MediaLink *link : ispSink->links()) { > @@ -1788,7 +1871,23 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > > break; > case MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER: > - LOG(MaliC55, Warning) << "Memory input not yet supported"; > + if (!cruMedia_) > + return false; I think it could make sense to do this similarly to `registerTPGCamera()` and check the name, print warning, and `continue` if it's not a match? > + > + ivcSd_ = V4L2Subdevice::fromEntityName(media_.get(), > + "rzv2h ivc block"); > + if (ivcSd_->open() < 0) !ivcSd_ || ivcSd_->open() < 0 and similarly below? > + return false; > + > + ivc_ = V4L2VideoDevice::fromEntityName(media_.get(), > + "rzv2h-ivc"); > + if (ivc_->open() < 0) > + return false; > + > + registered = registerMemoryInputCamera(); > + if (!registered) > + return registered; > + > break; > default: > LOG(MaliC55, Error) << "Unsupported entity function"; >
On Tue, Mar 31, 2026 at 03:07:19PM +0200, Barnabás Pőcze wrote: > 2026. 03. 25. 15:44 keltezéssel, Jacopo Mondi írta: > > Add support to Mali C55 pipeline handler for memory-to-memory camera > > mode. > > > > The Mali-C55, as integrated in the Renesas RZ/V2H(P) SoC, is fed with > > images saved to memory by the CRU unit and read on behalf of the ISP > > by the IVC. > > > > Update the pipeline handler's match() function to search for the CRU > > media entity necessary to register a memory input camera. Create and > > initialize a CameraData instance for the memory camera use case if a > > valid CRU is found and initialize the IVC components used to feed the > > ISP with images read from memory. > > > > Do not implement camera configuration for the time being to reuce > > the patch size. It will be provided in the following patches. > > > > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > --- > > src/libcamera/pipeline/mali-c55/mali-c55.cpp | 135 +++++++++++++++++++++++---- > > 1 file changed, 117 insertions(+), 18 deletions(-) > > > > diff --git a/src/libcamera/pipeline/mali-c55/mali-c55.cpp b/src/libcamera/pipeline/mali-c55/mali-c55.cpp > > index dca82564e2eb..26d368bb9132 100644 > > --- a/src/libcamera/pipeline/mali-c55/mali-c55.cpp > > +++ b/src/libcamera/pipeline/mali-c55/mali-c55.cpp > > [...] > > @@ -1682,14 +1753,14 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > > * The TPG and the downscale pipe are both optional blocks and may not > > * be fitted. > > */ > > - DeviceMatch dm("mali-c55"); > > - dm.add("mali-c55 isp"); > > - dm.add("mali-c55 resizer fr"); > > - dm.add("mali-c55 fr"); > > - dm.add("mali-c55 3a stats"); > > - dm.add("mali-c55 3a params"); > > - > > - media_ = acquireMediaDevice(enumerator, dm); > > + DeviceMatch c55_dm("mali-c55"); > > Minor thing, this should probably be `c55Dm`. > > > > + c55_dm.add("mali-c55 isp"); > > + c55_dm.add("mali-c55 resizer fr"); > > + c55_dm.add("mali-c55 fr"); > > + c55_dm.add("mali-c55 3a stats"); > > + c55_dm.add("mali-c55 3a params"); > > + > > + media_ = acquireMediaDevice(enumerator, c55_dm); > > if (!media_) > > return false; > > @@ -1749,6 +1820,25 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > > stats_->bufferReady.connect(this, &PipelineHandlerMaliC55::statsBufferReady); > > params_->bufferReady.connect(this, &PipelineHandlerMaliC55::paramsBufferReady); > > + /* > > + * We also need to search for the rzg2l-cru CSI-2 receiver. If we find > > + * that then we need to work in memory input mode instead of the inline > > + * mode. The absence of this match is not necessarily a failure at this > > + * point...it depends on the media links that we investigate momentarily. > > + * > > + * This is a bit hacky, because there could be multiple of these media > > + * devices and we're just taking the first. We need modular pipelines to > > + * properly solve the issue. > > + */ > > + static const std::regex cruCsi2Regex("csi-[0-9a-f]{8}.csi2"); > > + static const std::regex cruIpRegex("cru-ip-[0-9a-f]{8}.cru[0-9]"); > > + > > + DeviceMatch cruDm("rzg2l_cru"); > > + cruDm.add(cruCsi2Regex); > > + cruDm.add(cruIpRegex); > > + cruDm.add("CRU output"); > > + cruMedia_ = acquireMediaDevice(enumerator, cruDm); > > + > > ispSink = isp_->entity()->getPadByIndex(0); > > if (!ispSink || ispSink->links().empty()) { > > LOG(MaliC55, Error) << "ISP sink pad error"; > > @@ -1762,13 +1852,6 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > > * MEDIA_ENT_F_CAM_SENSOR - The test pattern generator > > * MEDIA_ENT_F_VID_IF_BRIDGE - A CSI-2 receiver > > * MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER - An input device > > - * > > - * The last one will be unsupported for now. The TPG is relatively easy, > > - * we just register a Camera for it. If we have a CSI-2 receiver we need > > - * to check its sink pad and register Cameras for anything connected to > > - * it (probably...there are some complex situations in which that might > > - * not be true but let's pretend they don't exist until we come across > > - * them) > > */ > > bool registered; > > for (MediaLink *link : ispSink->links()) { > > @@ -1788,7 +1871,23 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) > > break; > > case MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER: > > - LOG(MaliC55, Warning) << "Memory input not yet supported"; > > + if (!cruMedia_) > > + return false; > > I think it could make sense to do this similarly to `registerTPGCamera()` and check > the name, print warning, and `continue` if it's not a match? > I can move this part to registerMemoryInputCamera(). I'm not sure how it will work with multi camera, but we'll evaluate once there > > > + > > + ivcSd_ = V4L2Subdevice::fromEntityName(media_.get(), > > + "rzv2h ivc block"); > > + if (ivcSd_->open() < 0) > > !ivcSd_ || ivcSd_->open() < 0 > > and similarly below? Thanks, better! > > > > + return false; > > + > > + ivc_ = V4L2VideoDevice::fromEntityName(media_.get(), > > + "rzv2h-ivc"); > > + if (ivc_->open() < 0) > > + return false; > > + > > + registered = registerMemoryInputCamera(); > > + if (!registered) > > + return registered; > > + > > break; > > default: > > LOG(MaliC55, Error) << "Unsupported entity function"; > > >
diff --git a/src/libcamera/pipeline/mali-c55/mali-c55.cpp b/src/libcamera/pipeline/mali-c55/mali-c55.cpp index dca82564e2eb..26d368bb9132 100644 --- a/src/libcamera/pipeline/mali-c55/mali-c55.cpp +++ b/src/libcamera/pipeline/mali-c55/mali-c55.cpp @@ -44,6 +44,8 @@ #include "libcamera/internal/v4l2_subdevice.h" #include "libcamera/internal/v4l2_videodevice.h" +#include "rzg2l-cru.h" + namespace { bool isFormatRaw(const libcamera::PixelFormat &pixFmt) @@ -105,7 +107,12 @@ public: std::unique_ptr<V4L2Subdevice> csi2_; std::unique_ptr<CameraSensor> sensor_; }; - using CameraType = std::variant<Tpg, Inline>; + + struct Memory { + std::unique_ptr<RZG2LCRU> cru_; + }; + + using CameraType = std::variant<Tpg, Inline, Memory>; MaliC55CameraData(PipelineHandler *pipe) : Camera::Private(pipe) @@ -116,6 +123,7 @@ public: Tpg *initTpg(MediaEntity *entity); Inline *initInline(MediaEntity *entity); + Memory *initMemory(MediaDevice *cruMedia); std::vector<Size> sizes(unsigned int mbusCode) const { @@ -125,7 +133,10 @@ public: }, [&](const Inline &in) -> std::vector<Size> { return in.sensor_->sizes(mbusCode); - } + }, + [&](const Memory &mem) -> std::vector<Size> { + return mem.cru_->sizes(); + }, }, input_); } @@ -138,6 +149,9 @@ public: [&](const Inline &in) -> V4L2Subdevice * { return in.sensor_->device(); }, + [&](const Memory &mem) -> V4L2Subdevice * { + return mem.cru_->sensor()->device(); + }, }, input_); } @@ -151,6 +165,9 @@ public: [&](const Inline &in) -> CameraSensor * { return in.sensor_.get(); }, + [&](const Memory &mem) -> CameraSensor * { + return mem.cru_->sensor(); + }, }, input_); } @@ -163,6 +180,9 @@ public: [&](const Inline &in) -> Size { return in.sensor_->resolution(); }, + [&](const Memory &mem) -> Size { + return mem.cru_->resolution(); + }, }, input_); } @@ -241,6 +261,19 @@ MaliC55CameraData::Inline *MaliC55CameraData::initInline(MediaEntity *sensor) return &input_.emplace<Inline>(std::move(in)); } +MaliC55CameraData::Memory *MaliC55CameraData::initMemory(MediaDevice *cruMedia) +{ + Memory mem; + + mem.cru_ = std::make_unique<RZG2LCRU>(); + + int ret = mem.cru_->init(cruMedia); + if (ret) + return nullptr; + + return &input_.emplace<Memory>(std::move(mem)); +} + std::vector<Size> MaliC55CameraData::Tpg::sizes(unsigned int mbusCode) const { V4L2Subdevice::Formats formats = sd_->formats(0); @@ -721,11 +754,15 @@ private: const std::string &name); bool registerTPGCamera(MediaLink *link); bool registerSensorCamera(MediaLink *link); + bool registerMemoryInputCamera(); std::shared_ptr<MediaDevice> media_; + std::shared_ptr<MediaDevice> cruMedia_; std::unique_ptr<V4L2Subdevice> isp_; std::unique_ptr<V4L2VideoDevice> stats_; std::unique_ptr<V4L2VideoDevice> params_; + std::unique_ptr<V4L2Subdevice> ivcSd_; + std::unique_ptr<V4L2VideoDevice> ivc_; std::vector<std::unique_ptr<FrameBuffer>> statsBuffers_; std::queue<FrameBuffer *> availableStatsBuffers_; @@ -976,6 +1013,9 @@ int PipelineHandlerMaliC55::configure(Camera *camera, const MediaEntity *csi2Entity = in.csi2_->entity(); return csi2Entity->getPadByIndex(1)->links()[0]->setEnabled(true); }, + [](MaliC55CameraData::Memory &) { + return 0; + }, }, data->input_); if (ret) return ret; @@ -1001,6 +1041,9 @@ int PipelineHandlerMaliC55::configure(Camera *camera, return in.csi2_->getFormat(1, &subdevFormat); }, + [](MaliC55CameraData::Memory &) { + return 0; + }, }, data->input_); if (ret) return ret; @@ -1673,6 +1716,34 @@ bool PipelineHandlerMaliC55::registerSensorCamera(MediaLink *ispLink) return true; } +bool PipelineHandlerMaliC55::registerMemoryInputCamera() +{ + std::unique_ptr<MaliC55CameraData> data = + std::make_unique<MaliC55CameraData>(this); + + auto *mem = data->initMemory(cruMedia_.get()); + if (!mem) + return false; + + CameraSensor *sensor = data->sensor(); + data->properties_ = sensor->properties(); + + const CameraSensorProperties::SensorDelays &delays = sensor->sensorDelays(); + std::unordered_map<uint32_t, DelayedControls::ControlParams> params = { + { V4L2_CID_ANALOGUE_GAIN, { delays.gainDelay, false } }, + { V4L2_CID_EXPOSURE, { delays.exposureDelay, false } }, + }; + + data->delayedCtrls_ = + std::make_unique<DelayedControls>(sensor->device(), params); + isp_->frameStart.connect(data->delayedCtrls_.get(), + &DelayedControls::applyControls); + + ivc_->bufferReady.connect(mem->cru_.get(), &RZG2LCRU::returnBuffer); + + return registerMaliCamera(std::move(data), sensor->device()->entity()->name()); +} + bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) { const MediaPad *ispSink; @@ -1682,14 +1753,14 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) * The TPG and the downscale pipe are both optional blocks and may not * be fitted. */ - DeviceMatch dm("mali-c55"); - dm.add("mali-c55 isp"); - dm.add("mali-c55 resizer fr"); - dm.add("mali-c55 fr"); - dm.add("mali-c55 3a stats"); - dm.add("mali-c55 3a params"); - - media_ = acquireMediaDevice(enumerator, dm); + DeviceMatch c55_dm("mali-c55"); + c55_dm.add("mali-c55 isp"); + c55_dm.add("mali-c55 resizer fr"); + c55_dm.add("mali-c55 fr"); + c55_dm.add("mali-c55 3a stats"); + c55_dm.add("mali-c55 3a params"); + + media_ = acquireMediaDevice(enumerator, c55_dm); if (!media_) return false; @@ -1749,6 +1820,25 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) stats_->bufferReady.connect(this, &PipelineHandlerMaliC55::statsBufferReady); params_->bufferReady.connect(this, &PipelineHandlerMaliC55::paramsBufferReady); + /* + * We also need to search for the rzg2l-cru CSI-2 receiver. If we find + * that then we need to work in memory input mode instead of the inline + * mode. The absence of this match is not necessarily a failure at this + * point...it depends on the media links that we investigate momentarily. + * + * This is a bit hacky, because there could be multiple of these media + * devices and we're just taking the first. We need modular pipelines to + * properly solve the issue. + */ + static const std::regex cruCsi2Regex("csi-[0-9a-f]{8}.csi2"); + static const std::regex cruIpRegex("cru-ip-[0-9a-f]{8}.cru[0-9]"); + + DeviceMatch cruDm("rzg2l_cru"); + cruDm.add(cruCsi2Regex); + cruDm.add(cruIpRegex); + cruDm.add("CRU output"); + cruMedia_ = acquireMediaDevice(enumerator, cruDm); + ispSink = isp_->entity()->getPadByIndex(0); if (!ispSink || ispSink->links().empty()) { LOG(MaliC55, Error) << "ISP sink pad error"; @@ -1762,13 +1852,6 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) * MEDIA_ENT_F_CAM_SENSOR - The test pattern generator * MEDIA_ENT_F_VID_IF_BRIDGE - A CSI-2 receiver * MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER - An input device - * - * The last one will be unsupported for now. The TPG is relatively easy, - * we just register a Camera for it. If we have a CSI-2 receiver we need - * to check its sink pad and register Cameras for anything connected to - * it (probably...there are some complex situations in which that might - * not be true but let's pretend they don't exist until we come across - * them) */ bool registered; for (MediaLink *link : ispSink->links()) { @@ -1788,7 +1871,23 @@ bool PipelineHandlerMaliC55::match(DeviceEnumerator *enumerator) break; case MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER: - LOG(MaliC55, Warning) << "Memory input not yet supported"; + if (!cruMedia_) + return false; + + ivcSd_ = V4L2Subdevice::fromEntityName(media_.get(), + "rzv2h ivc block"); + if (ivcSd_->open() < 0) + return false; + + ivc_ = V4L2VideoDevice::fromEntityName(media_.get(), + "rzv2h-ivc"); + if (ivc_->open() < 0) + return false; + + registered = registerMemoryInputCamera(); + if (!registered) + return registered; + break; default: LOG(MaliC55, Error) << "Unsupported entity function";