From patchwork Tue Mar 31 16:36:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 26397 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 90D42C32F7 for ; Tue, 31 Mar 2026 16:37:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4A54462D2C; Tue, 31 Mar 2026 18:36:59 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hHgnP17D"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 31DE462D26 for ; Tue, 31 Mar 2026 18:36:52 +0200 (CEST) Received: from [100.93.44.16] (net-93-65-100-155.cust.vodafonedsl.it [93.65.100.155]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A27A825B5; Tue, 31 Mar 2026 18:35:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1774974929; bh=CzTYhqk8LUFjjiKF0NNAHl3BVN+64xBZR3KkW8s/LhE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=hHgnP17DHFPgHjuQ0L0sPui9NpRrPS2n+Ad6o+nh07Xg/vRzfaQkD4l7z4CR8QLUS XRqQ6curHJ5Pnkcc7ATr3A8RmESwgg8/81tSNf3K2nwYlFDsqsK9ZsZPSvtqZmh3Xm e0Lufk6UDHs4b9Yc8kWfMspFQSjsWwpowGsxNtrY= From: Jacopo Mondi Date: Tue, 31 Mar 2026 18:36:35 +0200 Subject: [PATCH v7 7/8] libcamera: mali-c55: Implement capture for memory-to-memory MIME-Version: 1.0 Message-Id: <20260331-mali-cru-v7-7-4caedc898a0e@ideasonboard.com> References: <20260331-mali-cru-v7-0-4caedc898a0e@ideasonboard.com> In-Reply-To: <20260331-mali-cru-v7-0-4caedc898a0e@ideasonboard.com> To: Daniel Scally , libcamera-devel@lists.libcamera.org Cc: =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Jacopo Mondi X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=12755; i=jacopo.mondi@ideasonboard.com; h=from:subject:message-id; bh=CzTYhqk8LUFjjiKF0NNAHl3BVN+64xBZR3KkW8s/LhE=; b=kA0DAAoBcjQGjxahVjwByyZiAGnL+CHItFm5aPiZpl05SMS+611dcziVXOlKcTyNO8k8wEseM YkCMwQAAQoAHRYhBLXEPUiAmiCKj1g4wHI0Bo8WoVY8BQJpy/ghAAoJEHI0Bo8WoVY85poP/25v kUS905GnrgDSDfmGI3ILnuEDZ8PRwgvCqamj3n672wthm+UUYGHG2M0MjALJj8GThbYgEbg9vdB 1m2fWnGVV2veLbO1/83Sp18pR0TSbdxKfAHJSNPciALeNNCN5UG1YxNJcZD/BQ89ZVpnDlWSxvy EZ0CvOv+7H6XnXeh8Z17aw+HsYFOyTZ9dN8MF/VtT4mkAC1cwiOQ0+RJOqLe+2AhgEHaiLGYzAv DvrYipBJSfMguqITNsohavNnquv2TsMh5uc5cgpxMTKjvUpkAoM1PEUqEDVnJD40el/NQEWHPaI WXKEEI96d8lV/g37rJCtojWoUp9fitK3XTxtLKdAhfFMpfw8YgoA6BX88I7tTqW6I52W44J4uox PbbMPKvre/NqtmsXFzdVkBnk7n/WTiYHxfCdyAn7LCHx3RLvyd41n6Oy5dcOxCkrXW3VY7kph4i EBqo9nUId7yrLOOo8kORSyfecZtSLHQ++24s156jBlp3E29dCfldeWk3j4VFh9/i6n1Mt674gVf 7qJ+uDzPIDzUq56TMKjA8/tdhxKweaqheJOxTp0FXrSYMC9A5tKT8Q9AutgbHjsfcbS4t3D87qn PWm4/fFpB3yo61k7D78Fr9J9J6i7Y5Lj7uAnRucBDEK384SRvahGcD3IU1Li4jwQOqNX22OI0ph CTcxY X-Developer-Key: i=jacopo.mondi@ideasonboard.com; a=openpgp; fpr=72392EDC88144A65C701EA9BA5826A2587AD026B X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Plumb in the MaliC55 pipeline handler support for capturing frames from memory using the CRU. Introduce a data flow which uses the CRU to feed the ISP through the IVC. In detail: - push incoming request to a pending queue until a buffer from the CRU is available - delay the call to ipa_->fillParams() to the CRU buffer ready event - once the IPA has computed parameters feed the ISP through the IVC with buffers from the CRU, params and statistics - Handle completion of in-flight requests: in m2m mode buffers are queued earlier to the ISP capture devices. If the camera is stopped these buffers are returned earlier in error state: complete the Request bypassing the IPA operations - As cancelled Requests might now be completed earlier then IPA events, modify the IPA event handler to ignore Requests that have been completed already Signed-off-by: Daniel Scally Signed-off-by: Jacopo Mondi --- src/libcamera/pipeline/mali-c55/mali-c55.cpp | 211 ++++++++++++++++++++++----- 1 file changed, 174 insertions(+), 37 deletions(-) diff --git a/src/libcamera/pipeline/mali-c55/mali-c55.cpp b/src/libcamera/pipeline/mali-c55/mali-c55.cpp index ce1f0b65cd6e..c3856f7c0a11 100644 --- a/src/libcamera/pipeline/mali-c55/mali-c55.cpp +++ b/src/libcamera/pipeline/mali-c55/mali-c55.cpp @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -90,6 +91,7 @@ struct MaliC55FrameInfo { FrameBuffer *paramBuffer; FrameBuffer *statBuffer; + FrameBuffer *rawBuffer; bool paramsDone; bool statsDone; @@ -693,6 +695,7 @@ public: void imageBufferReady(FrameBuffer *buffer); void paramsBufferReady(FrameBuffer *buffer); void statsBufferReady(FrameBuffer *buffer); + void cruBufferReady(FrameBuffer *buffer); void paramsComputed(unsigned int requestId, uint32_t bytesused); void statsProcessed(unsigned int requestId, const ControlList &metadata); @@ -741,7 +744,7 @@ private: MaliC55FrameInfo *findFrameInfo(FrameBuffer *buffer); MaliC55FrameInfo *findFrameInfo(Request *request); - void tryComplete(MaliC55FrameInfo *info); + void tryComplete(MaliC55FrameInfo *info, bool cancelled = false); int configureRawStream(MaliC55CameraData *data, const StreamConfiguration &config, @@ -750,6 +753,9 @@ private: const StreamConfiguration &config, V4L2SubdeviceFormat &subdevFormat); + MaliC55FrameInfo *prepareFrameInfo(Request *request); + void queueRequestToCru(MaliC55CameraData *data, Request *request); + void applyScalerCrop(Camera *camera, const ControlList &controls); bool registerMaliCamera(std::unique_ptr data, @@ -1222,6 +1228,13 @@ void PipelineHandlerMaliC55::freeBuffers(Camera *camera) if (params_->releaseBuffers()) LOG(MaliC55, Error) << "Failed to release params buffers"; + if (auto *mem = std::get_if(&data->input_)) { + if (ivc_->releaseBuffers()) + LOG(MaliC55, Error) << "Failed to release input buffers"; + if (mem->cru_->freeBuffers()) + LOG(MaliC55, Error) << "Failed to release CRU buffers"; + } + return; } @@ -1245,6 +1258,12 @@ int PipelineHandlerMaliC55::allocateBuffers(Camera *camera) } }; + if (std::holds_alternative(data->input_)) { + ret = ivc_->importBuffers(kMaliC55BufferCount); + if (ret < 0) + return ret; + } + ret = stats_->allocateBuffers(kMaliC55BufferCount, &statsBuffers_); if (ret < 0) return ret; @@ -1276,6 +1295,24 @@ int PipelineHandlerMaliC55::start(Camera *camera, [[maybe_unused]] const Control if (ret) return ret; + if (auto *mem = std::get_if(&data->input_)) { + ret = mem->cru_->start(kMaliC55BufferCount); + if (ret) { + LOG(MaliC55, Error) + << "Failed to start CRU " << camera->id(); + freeBuffers(camera); + return ret; + } + + ret = ivc_->streamOn(); + if (ret) { + LOG(MaliC55, Error) + << "Failed to start IVC" << camera->id(); + freeBuffers(camera); + return ret; + } + } + if (data->ipa_) { ret = data->ipa_->start(); if (ret) { @@ -1357,6 +1394,11 @@ void PipelineHandlerMaliC55::stopDevice(Camera *camera) isp_->setFrameStartEnabled(false); + if (auto *mem = std::get_if(&data->input_)) { + ivc_->streamOff(); + mem->cru_->stop(); + } + for (MaliC55Pipe &pipe : pipes_) { if (!pipe.stream) continue; @@ -1468,10 +1510,67 @@ void PipelineHandlerMaliC55::applyScalerCrop(Camera *camera, } } +MaliC55FrameInfo *PipelineHandlerMaliC55::prepareFrameInfo(Request *request) +{ + if (availableStatsBuffers_.empty()) { + LOG(MaliC55, Error) << "Stats buffer underrun"; + return nullptr; + } + + if (availableParamsBuffers_.empty()) { + LOG(MaliC55, Error) << "Params buffer underrun"; + return nullptr; + } + + MaliC55FrameInfo &frameInfo = frameInfoMap_[request->sequence()]; + frameInfo.request = request; + frameInfo.statBuffer = availableStatsBuffers_.front(); + availableStatsBuffers_.pop(); + frameInfo.paramBuffer = availableParamsBuffers_.front(); + availableParamsBuffers_.pop(); + + frameInfo.paramsDone = false; + frameInfo.statsDone = false; + + return &frameInfo; +} + +void PipelineHandlerMaliC55::queueRequestToCru(MaliC55CameraData *data, + Request *request) +{ + auto *mem = std::get_if(&data->input_); + ASSERT(mem); + + FrameBuffer *cruBuffer = mem->cru_->queueBuffer(request); + ASSERT(cruBuffer); + + auto frameInfo = prepareFrameInfo(request); + ASSERT(frameInfo); + frameInfo->rawBuffer = cruBuffer; + + for (auto &[stream, buffer] : request->buffers()) { + MaliC55Pipe *pipe = pipeFromStream(data, stream); + + pipe->cap->queueBuffer(buffer); + } + + data->ipa_->queueRequest(request->sequence(), request->controls()); +} + int PipelineHandlerMaliC55::queueRequestDevice(Camera *camera, Request *request) { MaliC55CameraData *data = cameraData(camera); + /* + * If we're in memory input mode, we need to queue the Request to the + * CRU, otherwise we can just do everything immediately. + */ + if (std::holds_alternative(data->input_)) { + queueRequestToCru(data, request); + + return 0; + } + /* Do not run the IPA if the TPG is in use. */ if (!data->ipa_) { MaliC55FrameInfo frameInfo; @@ -1492,32 +1591,13 @@ int PipelineHandlerMaliC55::queueRequestDevice(Camera *camera, Request *request) return 0; } - if (availableStatsBuffers_.empty()) { - LOG(MaliC55, Error) << "Stats buffer underrun"; + auto frameInfo = prepareFrameInfo(request); + if (!frameInfo) return -ENOENT; - } - - if (availableParamsBuffers_.empty()) { - LOG(MaliC55, Error) << "Params buffer underrun"; - return -ENOENT; - } - - MaliC55FrameInfo frameInfo; - frameInfo.request = request; - - frameInfo.statBuffer = availableStatsBuffers_.front(); - availableStatsBuffers_.pop(); - frameInfo.paramBuffer = availableParamsBuffers_.front(); - availableParamsBuffers_.pop(); - - frameInfo.paramsDone = false; - frameInfo.statsDone = false; - - frameInfoMap_[request->sequence()] = frameInfo; data->ipa_->queueRequest(request->sequence(), request->controls()); data->ipa_->fillParams(request->sequence(), - frameInfo.paramBuffer->cookie()); + frameInfo->paramBuffer->cookie()); return 0; } @@ -1536,18 +1616,21 @@ MaliC55FrameInfo *PipelineHandlerMaliC55::findFrameInfo(FrameBuffer *buffer) { for (auto &[sequence, info] : frameInfoMap_) { if (info.paramBuffer == buffer || - info.statBuffer == buffer) + info.statBuffer == buffer || + info.rawBuffer == buffer) return &info; } return nullptr; } -void PipelineHandlerMaliC55::tryComplete(MaliC55FrameInfo *info) +void PipelineHandlerMaliC55::tryComplete(MaliC55FrameInfo *info, bool cancelled) { - if (!info->paramsDone) - return; - if (!info->statsDone) + /* + * If the buffer has been cancelled, we complete the request without + * waiting for the IPA. + */ + if (!cancelled && (!info->paramsDone || !info->statsDone)) return; Request *request = info->request; @@ -1571,13 +1654,15 @@ void PipelineHandlerMaliC55::imageBufferReady(FrameBuffer *buffer) ASSERT(info); if (completeBuffer(request, buffer)) - tryComplete(info); + tryComplete(info, + buffer->metadata().status == FrameMetadata::FrameCancelled); } void PipelineHandlerMaliC55::paramsBufferReady(FrameBuffer *buffer) { MaliC55FrameInfo *info = findFrameInfo(buffer); - ASSERT(info); + if (!info) + return; info->paramsDone = true; @@ -1587,7 +1672,8 @@ void PipelineHandlerMaliC55::paramsBufferReady(FrameBuffer *buffer) void PipelineHandlerMaliC55::statsBufferReady(FrameBuffer *buffer) { MaliC55FrameInfo *info = findFrameInfo(buffer); - ASSERT(info); + if (!info) + return; Request *request = info->request; MaliC55CameraData *data = cameraData(request->_d()->camera()); @@ -1598,32 +1684,80 @@ void PipelineHandlerMaliC55::statsBufferReady(FrameBuffer *buffer) sensorControls); } +void PipelineHandlerMaliC55::cruBufferReady(FrameBuffer *buffer) +{ + /* + * If the buffer has been cancelled, do not ask the IPA to prepare + * parameters. + * + * The Request this cancelled buffer belongs to will be handled by + * imageBufferReady() as we have queued buffers to the ISP capture + * devices at the same time we have queued this buffer to the CRU + * when running in m2m mode. + */ + if (buffer->metadata().status == FrameMetadata::FrameCancelled) + return; + + MaliC55FrameInfo *info = findFrameInfo(buffer); + ASSERT(info); + + Request *request = info->request; + request->_d()->metadata().set(controls::SensorTimestamp, + buffer->metadata().timestamp); + + /* Ought we do something with the sensor's controls here...? */ + MaliC55CameraData *data = cameraData(request->_d()->camera()); + data->ipa_->fillParams(request->sequence(), info->paramBuffer->cookie()); +} + void PipelineHandlerMaliC55::paramsComputed(unsigned int requestId, uint32_t bytesused) { - MaliC55FrameInfo &frameInfo = frameInfoMap_[requestId]; + auto it = frameInfoMap_.find(requestId); + if (it == frameInfoMap_.end()) + return; + + MaliC55FrameInfo &frameInfo = it->second; Request *request = frameInfo.request; + if (!request) + return; + MaliC55CameraData *data = cameraData(request->_d()->camera()); /* * Queue buffers for stats and params, then queue buffers to the capture - * video devices. + * video devices if we're running in Inline mode or with the TPG. + * + * If we're running in M2M buffers have been queued to the capture + * devices at queueRequestToCru() time and here we only have to queue + * buffers to the IVC input to start a transfer. */ frameInfo.paramBuffer->_d()->metadata().planes()[0].bytesused = bytesused; params_->queueBuffer(frameInfo.paramBuffer); stats_->queueBuffer(frameInfo.statBuffer); - for (auto &[stream, buffer] : request->buffers()) { - MaliC55Pipe *pipe = pipeFromStream(data, stream); + if (!std::holds_alternative(data->input_)) { + for (auto &[stream, buffer] : request->buffers()) { + MaliC55Pipe *pipe = pipeFromStream(data, stream); - pipe->cap->queueBuffer(buffer); + pipe->cap->queueBuffer(buffer); + } + } else { + ivc_->queueBuffer(frameInfo.rawBuffer); + frameInfo.rawBuffer = nullptr; } } void PipelineHandlerMaliC55::statsProcessed(unsigned int requestId, const ControlList &metadata) { - MaliC55FrameInfo &frameInfo = frameInfoMap_[requestId]; + auto it = frameInfoMap_.find(requestId); + if (it == frameInfoMap_.end()) + return; + + MaliC55FrameInfo &frameInfo = it->second; + if (!frameInfo.request) + return; frameInfo.statsDone = true; frameInfo.request->_d()->metadata().merge(metadata); @@ -1785,6 +1919,9 @@ bool PipelineHandlerMaliC55::registerMemoryInputCamera(MediaLink *link) ivc_->bufferReady.connect(mem->cru_.get(), &RZG2LCRU::returnBuffer); + V4L2VideoDevice *cruOutput = mem->cru_->output(); + cruOutput->bufferReady.connect(this, &PipelineHandlerMaliC55::cruBufferReady); + return registerMaliCamera(std::move(data), sensor->device()->entity()->name()); }