From patchwork Wed Mar 25 14:44:13 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 26340 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 748C5C32F8 for ; Wed, 25 Mar 2026 14:44:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B03376283C; Wed, 25 Mar 2026 15:44:26 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="l7HOhznP"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B2A246274D for ; Wed, 25 Mar 2026 15:44:20 +0100 (CET) Received: from [192.168.1.104] (net-93-65-100-155.cust.vodafonedsl.it [93.65.100.155]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AE9211ADE; Wed, 25 Mar 2026 15:43:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1774449782; bh=flq4ge5QIzfQeoUke8dLji5yEiMp+FGTl/TsXLIuW3M=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=l7HOhznPjmwC4thbZvDloByz3yiPN675TqSavFDcBAJz8aKvHgI7ikNonwE0cX8Ef eBlBsws59UrNd4uZYwxOPY5IIoaoVcEKao38PYupNV3hJuYIXdXw8EZUHSmPZbYsJ2 V5OA5Lh5jPWHkSKOnZuGWZgycVwskzNq1bcyz21o= From: Jacopo Mondi Date: Wed, 25 Mar 2026 15:44:13 +0100 Subject: [PATCH v6 6/7] libcamera: mali-c55: Implement capture for memory-to-memory MIME-Version: 1.0 Message-Id: <20260325-mali-cru-v6-6-b16b0c49819a@ideasonboard.com> References: <20260325-mali-cru-v6-0-b16b0c49819a@ideasonboard.com> In-Reply-To: <20260325-mali-cru-v6-0-b16b0c49819a@ideasonboard.com> To: Daniel Scally , libcamera-devel@lists.libcamera.org Cc: =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Jacopo Mondi X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=13937; i=jacopo.mondi@ideasonboard.com; h=from:subject:message-id; bh=pHJmjnkudjHJjFFuc+UJG4KlbidDyD0DgZrmZ5fOwbQ=; b=owEBbQKS/ZANAwAKAXI0Bo8WoVY8AcsmYgBpw/TCUWHsn7psmT4Uig1WfvGxmoqC6265FkuOF iKZBUFJjRuJAjMEAAEKAB0WIQS1xD1IgJogio9YOMByNAaPFqFWPAUCacP0wgAKCRByNAaPFqFW PE6hD/9yhdTQuOhs4+8fRgqddfkoSm2bOVccSAa/a9yPNvXgI40QdWCbe+v+GIarlKSEhixnmZJ xXVanvKkj7ofWsGHuSXycEgv0f6Z6fxhdV6wEfnOejN8eZV88nTH67l+JfTqhGEB4meB13iiUeG /0UFrfu7jTsMRwMjtDQ73JvJcbQRqJzTox6Ykooy82ipPxNtaKOdZuoO2ZO1pXJIkfg+4ijci2u BvXFVSG0LZQJGmqorrU1d7IT9yR1IKgC0NRShDo407vVUBoJg0krnK6KPgfn5n/xQRul88aExWE r4UZub6DH1Z7Sx5v1U1wjoX46hb3tv4QVzHDblopsuAh7ld3YJKKYTr/x5vUzQ64B0ubKbzTOFb tl6Kp4HZjoV5pf+bGZOB2eFbB7f0SbFNg6xWwnNgdXiYB26xAMLhX7WDGJfhMmQhPW6aGf8QkjL Yl9dFsQ30EzC9IIF52vU3REFUfUTEE7Djp1dfqFwMboev7G8rH1UcKmtpm1GQtXy2NCgstOhZf2 B9rowozmWz2M++Y9zCMUil1SxXtt63906EvYXBB8T0zcM6Fo2vrTK19dd/dBk7qVZYwXauo2c+B rGf+UPoYJWRUH6c1bSU9r1gzgZJ9Rus7C8RwYgRRd7Llc9xxEIxU/A+hpUg5zChtZKbKkNo2btP /WGkuiF7PDUBf2A== X-Developer-Key: i=jacopo.mondi@ideasonboard.com; a=openpgp; fpr=72392EDC88144A65C701EA9BA5826A2587AD026B X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Daniel Scally Plumb in the MaliC55 pipeline handler support for capturing frames from memory using the CRU. Introduce a data flow which uses the CRU to feed the ISP through the IVC. In detail: - push incoming request to a pending queue until a buffer from the CRU is available - delay the call to ipa_->fillParams() to the CRU buffer ready even - once the IPA has computed parameters feed the ISP through the IVC with buffers from the CRU, params and statistics. - Handle completion of in-flight requests: in m2m mode buffers are queued earlier to the ISP capture devices. If the camera is stopped these buffers are returned earlier in error state: complete the Request bypassing the IPA operations. - As cancelled Requests might now be completed earlier then IPA events, modify the IPA event handler to ignore Requests that have been completed already Signed-off-by: Daniel Scally Signed-off-by: Jacopo Mondi --- src/libcamera/pipeline/mali-c55/mali-c55.cpp | 244 +++++++++++++++++++++++---- 1 file changed, 207 insertions(+), 37 deletions(-) diff --git a/src/libcamera/pipeline/mali-c55/mali-c55.cpp b/src/libcamera/pipeline/mali-c55/mali-c55.cpp index be3e38da95ab..10848c412849 100644 --- a/src/libcamera/pipeline/mali-c55/mali-c55.cpp +++ b/src/libcamera/pipeline/mali-c55/mali-c55.cpp @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -88,6 +89,7 @@ struct MaliC55FrameInfo { FrameBuffer *paramBuffer; FrameBuffer *statBuffer; + FrameBuffer *rawBuffer; bool paramsDone; bool statsDone; @@ -691,6 +693,7 @@ public: void imageBufferReady(FrameBuffer *buffer); void paramsBufferReady(FrameBuffer *buffer); void statsBufferReady(FrameBuffer *buffer); + void cruBufferReady(FrameBuffer *buffer); void paramsComputed(unsigned int requestId, uint32_t bytesused); void statsProcessed(unsigned int requestId, const ControlList &metadata); @@ -739,7 +742,7 @@ private: MaliC55FrameInfo *findFrameInfo(FrameBuffer *buffer); MaliC55FrameInfo *findFrameInfo(Request *request); - void tryComplete(MaliC55FrameInfo *info); + void tryComplete(MaliC55FrameInfo *info, bool cancelled = false); int configureRawStream(MaliC55CameraData *data, const StreamConfiguration &config, @@ -747,6 +750,11 @@ private: int configureProcessedStream(MaliC55CameraData *data, const StreamConfiguration &config, V4L2SubdeviceFormat &subdevFormat); + void cancelPendingRequests(); + + MaliC55FrameInfo * + prepareFrameInfo(Request *request, RZG2LCRU *cru = nullptr); + int queuePendingRequests(MaliC55CameraData *data); void applyScalerCrop(Camera *camera, const ControlList &controls); @@ -772,6 +780,9 @@ private: std::map frameInfoMap_; + /* Requests for which no buffer has been queued to the CRU device yet. */ + std::queue pendingRequests_; + std::array pipes_; bool dsFitted_; @@ -1220,6 +1231,13 @@ void PipelineHandlerMaliC55::freeBuffers(Camera *camera) if (params_->releaseBuffers()) LOG(MaliC55, Error) << "Failed to release params buffers"; + if (auto *mem = std::get_if(&data->input_)) { + if (ivc_->releaseBuffers()) + LOG(MaliC55, Error) << "Failed to release input buffers"; + if (mem->cru_->freeBuffers()) + LOG(MaliC55, Error) << "Failed to release CRU buffers"; + } + return; } @@ -1249,6 +1267,12 @@ int PipelineHandlerMaliC55::allocateBuffers(Camera *camera) } }; + if (std::holds_alternative(data->input_)) { + ret = ivc_->importBuffers(RZG2LCRU::kBufferCount); + if (ret < 0) + return ret; + } + ret = stats_->allocateBuffers(bufferCount, &statsBuffers_); if (ret < 0) return ret; @@ -1280,6 +1304,24 @@ int PipelineHandlerMaliC55::start(Camera *camera, [[maybe_unused]] const Control if (ret) return ret; + if (auto *mem = std::get_if(&data->input_)) { + ret = mem->cru_->start(); + if (ret) { + LOG(MaliC55, Error) + << "Failed to start CRU " << camera->id(); + freeBuffers(camera); + return ret; + } + + ret = ivc_->streamOn(); + if (ret) { + LOG(MaliC55, Error) + << "Failed to start IVC" << camera->id(); + freeBuffers(camera); + return ret; + } + } + if (data->ipa_) { ret = data->ipa_->start(); if (ret) { @@ -1355,12 +1397,25 @@ int PipelineHandlerMaliC55::start(Camera *camera, [[maybe_unused]] const Control return 0; } +void PipelineHandlerMaliC55::cancelPendingRequests() +{ + while (!pendingRequests_.empty()) { + cancelRequest(pendingRequests_.front()); + pendingRequests_.pop(); + } +} + void PipelineHandlerMaliC55::stopDevice(Camera *camera) { MaliC55CameraData *data = cameraData(camera); isp_->setFrameStartEnabled(false); + if (auto *mem = std::get_if(&data->input_)) { + ivc_->streamOff(); + mem->cru_->stop(); + } + for (MaliC55Pipe &pipe : pipes_) { if (!pipe.stream) continue; @@ -1374,6 +1429,8 @@ void PipelineHandlerMaliC55::stopDevice(Camera *camera) if (data->ipa_) data->ipa_->stop(); freeBuffers(camera); + + cancelPendingRequests(); } void PipelineHandlerMaliC55::applyScalerCrop(Camera *camera, @@ -1472,10 +1529,85 @@ void PipelineHandlerMaliC55::applyScalerCrop(Camera *camera, } } +MaliC55FrameInfo * +PipelineHandlerMaliC55::prepareFrameInfo(Request *request, RZG2LCRU *cru) +{ + if (availableStatsBuffers_.empty()) { + LOG(MaliC55, Error) << "Stats buffer underrun"; + return nullptr; + } + + if (availableParamsBuffers_.empty()) { + LOG(MaliC55, Error) << "Params buffer underrun"; + return nullptr; + } + + MaliC55FrameInfo &frameInfo = frameInfoMap_[request->sequence()]; + frameInfo.request = request; + + if (cru) { + frameInfo.rawBuffer = cru->queueBuffer(request); + if (!frameInfo.rawBuffer) + return nullptr; + } + + frameInfo.statBuffer = availableStatsBuffers_.front(); + availableStatsBuffers_.pop(); + frameInfo.paramBuffer = availableParamsBuffers_.front(); + availableParamsBuffers_.pop(); + + frameInfo.paramsDone = false; + frameInfo.statsDone = false; + + return &frameInfo; +} + +int PipelineHandlerMaliC55::queuePendingRequests(MaliC55CameraData *data) +{ + auto *mem = std::get_if(&data->input_); + ASSERT(mem); + + while (!pendingRequests_.empty()) { + Request *request = pendingRequests_.front(); + + if (!prepareFrameInfo(request, mem->cru_.get())) + return -ENOENT; + + for (auto &[stream, buffer] : request->buffers()) { + MaliC55Pipe *pipe = pipeFromStream(data, stream); + + pipe->cap->queueBuffer(buffer); + } + + data->ipa_->queueRequest(request->sequence(), request->controls()); + + pendingRequests_.pop(); + } + + return 0; +} + int PipelineHandlerMaliC55::queueRequestDevice(Camera *camera, Request *request) { MaliC55CameraData *data = cameraData(camera); + /* + * If we're in memory input mode, we need to pop the requests onto the + * pending list until a CRU buffer is ready...otherwise we can just do + * everything immediately. + */ + if (std::holds_alternative(data->input_)) { + pendingRequests_.push(request); + + int ret = queuePendingRequests(data); + if (ret) { + pendingRequests_.pop(); + return ret; + } + + return 0; + } + /* Do not run the IPA if the TPG is in use. */ if (!data->ipa_) { MaliC55FrameInfo frameInfo; @@ -1496,32 +1628,13 @@ int PipelineHandlerMaliC55::queueRequestDevice(Camera *camera, Request *request) return 0; } - if (availableStatsBuffers_.empty()) { - LOG(MaliC55, Error) << "Stats buffer underrun"; - return -ENOENT; - } - - if (availableParamsBuffers_.empty()) { - LOG(MaliC55, Error) << "Params buffer underrun"; + auto frameInfo = prepareFrameInfo(request); + if (!frameInfo) return -ENOENT; - } - - MaliC55FrameInfo frameInfo; - frameInfo.request = request; - - frameInfo.statBuffer = availableStatsBuffers_.front(); - availableStatsBuffers_.pop(); - frameInfo.paramBuffer = availableParamsBuffers_.front(); - availableParamsBuffers_.pop(); - - frameInfo.paramsDone = false; - frameInfo.statsDone = false; - - frameInfoMap_[request->sequence()] = frameInfo; data->ipa_->queueRequest(request->sequence(), request->controls()); data->ipa_->fillParams(request->sequence(), - frameInfo.paramBuffer->cookie()); + frameInfo->paramBuffer->cookie()); return 0; } @@ -1540,18 +1653,21 @@ MaliC55FrameInfo *PipelineHandlerMaliC55::findFrameInfo(FrameBuffer *buffer) { for (auto &[sequence, info] : frameInfoMap_) { if (info.paramBuffer == buffer || - info.statBuffer == buffer) + info.statBuffer == buffer || + info.rawBuffer == buffer) return &info; } return nullptr; } -void PipelineHandlerMaliC55::tryComplete(MaliC55FrameInfo *info) +void PipelineHandlerMaliC55::tryComplete(MaliC55FrameInfo *info, bool cancelled) { - if (!info->paramsDone) - return; - if (!info->statsDone) + /* + * If the buffer has been cancelled, we complete the request without + * waiting for the IPA. + */ + if (!cancelled && (!info->paramsDone || !info->statsDone)) return; Request *request = info->request; @@ -1575,13 +1691,15 @@ void PipelineHandlerMaliC55::imageBufferReady(FrameBuffer *buffer) ASSERT(info); if (completeBuffer(request, buffer)) - tryComplete(info); + tryComplete(info, + buffer->metadata().status == FrameMetadata::FrameCancelled); } void PipelineHandlerMaliC55::paramsBufferReady(FrameBuffer *buffer) { MaliC55FrameInfo *info = findFrameInfo(buffer); - ASSERT(info); + if (!info) + return; info->paramsDone = true; @@ -1591,7 +1709,8 @@ void PipelineHandlerMaliC55::paramsBufferReady(FrameBuffer *buffer) void PipelineHandlerMaliC55::statsBufferReady(FrameBuffer *buffer) { MaliC55FrameInfo *info = findFrameInfo(buffer); - ASSERT(info); + if (!info) + return; Request *request = info->request; MaliC55CameraData *data = cameraData(request->_d()->camera()); @@ -1602,32 +1721,80 @@ void PipelineHandlerMaliC55::statsBufferReady(FrameBuffer *buffer) sensorControls); } +void PipelineHandlerMaliC55::cruBufferReady(FrameBuffer *buffer) +{ + /* + * If the buffer has been cancelled, do not ask the IPA to prepare + * parameters. + * + * The Request this cancelled buffer belongs to will be handled by + * imageBufferReady() as we have queued buffers to the ISP capture + * devices at the same time we have queued this buffer to the CRU + * when running in m2m mode. + */ + if (buffer->metadata().status == FrameMetadata::FrameCancelled) + return; + + MaliC55FrameInfo *info = findFrameInfo(buffer); + ASSERT(info); + + Request *request = info->request; + request->_d()->metadata().set(controls::SensorTimestamp, + buffer->metadata().timestamp); + + /* Ought we do something with the sensor's controls here...? */ + MaliC55CameraData *data = cameraData(request->_d()->camera()); + data->ipa_->fillParams(request->sequence(), info->paramBuffer->cookie()); +} + void PipelineHandlerMaliC55::paramsComputed(unsigned int requestId, uint32_t bytesused) { - MaliC55FrameInfo &frameInfo = frameInfoMap_[requestId]; + auto it = frameInfoMap_.find(requestId); + if (it == frameInfoMap_.end()) + return; + + MaliC55FrameInfo &frameInfo = it->second; Request *request = frameInfo.request; + if (!request) + return; + MaliC55CameraData *data = cameraData(request->_d()->camera()); /* * Queue buffers for stats and params, then queue buffers to the capture - * video devices. + * video devices if we're running in Inline mode or with the TPG. + * + * If we're running in M2M buffers have been queued to the capture + * devices at queuePendingRequests() time and here we only have to queue + * buffers to the IVC input to start a transfer. */ frameInfo.paramBuffer->_d()->metadata().planes()[0].bytesused = bytesused; params_->queueBuffer(frameInfo.paramBuffer); stats_->queueBuffer(frameInfo.statBuffer); - for (auto &[stream, buffer] : request->buffers()) { - MaliC55Pipe *pipe = pipeFromStream(data, stream); + if (!std::holds_alternative(data->input_)) { + for (auto &[stream, buffer] : request->buffers()) { + MaliC55Pipe *pipe = pipeFromStream(data, stream); - pipe->cap->queueBuffer(buffer); + pipe->cap->queueBuffer(buffer); + } + } else { + ivc_->queueBuffer(frameInfo.rawBuffer); + frameInfo.rawBuffer = nullptr; } } void PipelineHandlerMaliC55::statsProcessed(unsigned int requestId, const ControlList &metadata) { - MaliC55FrameInfo &frameInfo = frameInfoMap_[requestId]; + auto it = frameInfoMap_.find(requestId); + if (it == frameInfoMap_.end()) + return; + + MaliC55FrameInfo &frameInfo = it->second; + if (!frameInfo.request) + return; frameInfo.statsDone = true; frameInfo.request->_d()->metadata().merge(metadata); @@ -1757,6 +1924,9 @@ bool PipelineHandlerMaliC55::registerMemoryInputCamera() ivc_->bufferReady.connect(mem->cru_.get(), &RZG2LCRU::returnBuffer); + V4L2VideoDevice *cruOutput = mem->cru_->output(); + cruOutput->bufferReady.connect(this, &PipelineHandlerMaliC55::cruBufferReady); + return registerMaliCamera(std::move(data), sensor->device()->entity()->name()); }