From patchwork Thu Sep 11 09:05:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 24306 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 16BA1BDB13 for ; Thu, 11 Sep 2025 09:06:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A30A96936F; Thu, 11 Sep 2025 11:06:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=ragnatech.se header.i=@ragnatech.se header.b="QqnIOx0V"; dkim=pass (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.b="P7FtWy2H"; dkim-atps=neutral Received: from fhigh-b3-smtp.messagingengine.com (fhigh-b3-smtp.messagingengine.com [202.12.124.154]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2313569367 for ; Thu, 11 Sep 2025 11:06:40 +0200 (CEST) Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfhigh.stl.internal (Postfix) with ESMTP id 094FD7A0084; Thu, 11 Sep 2025 05:06:39 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Thu, 11 Sep 2025 05:06:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ragnatech.se; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1757581598; x=1757667998; bh=Jnu2Z22fs4fwa5o7KxBnWD+w6kcF3m78ye34K3rGdaQ=; b= QqnIOx0VAV9N0BON5Nc++gR8mnP2gwkLg0jnkyoeTyOENnahf/uoghVwW6zcsrhP VDaMSDeee+Ka2JYAG1s07g71U67GdTyCYRmlR1+XV8GZmQsv+rBk2UolK5lucTcm BOF0ijJS9wUGZlbXtYBZn+4mn8XDHq1dr+Vz92u2NSSYqqChcZEm/MgnUh5L4BK2 IWDStkKWUyn2cIlKdVKK/4RnJ8SbdihV1GfIIQysCQD9tqiLsCQNFSTwzDVjbTAq y51XZobNOm2rLpDgPtlDo3ejchyNRKEh9OG72+6qMgkXCBgO4jOQp508Orp4zoMu VpuVXCGeZkEoZO5tLmnNyA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1757581598; x= 1757667998; bh=Jnu2Z22fs4fwa5o7KxBnWD+w6kcF3m78ye34K3rGdaQ=; b=P 7FtWy2HJkkbgkkf/V64tMoId/6Yn+OLFAzdzJUcGysYaY6eMClDnf6PmJJZ5x+7f NQ8TaRRfqVvNHrwjQbejQht/Fr3d4I7Dt1bm3myFV5k3q9U1zjlBwzv8m2/33+16 pT0qnLhzGl/FIWq/iZ2D+ZqK0m5xong7IOD6SvYlFVdaigpnnQBcTzZXFAUU6tW1 YwbnBaq/qQl7NEW073UEaaeuOcWrxXTyp8CjW8cjNhk3J+B+loADoGzuX5WjJQsx ab4C8aEgRznDQwn9YubOo9jSA2nYiDA8NvrxWd/Is1II/D4xsHbth9xpZJfUytBS GOAoMAIIbNew2Oogrmf2w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggddvheejjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfgggtgfesthekredtredtjeenucfhrhhomheppfhikhhlrghs ucfunpguvghrlhhunhguuceonhhikhhlrghsrdhsohguvghrlhhunhguodhrvghnvghsrg hssehrrghgnhgrthgvtghhrdhsvgeqnecuggftrfgrthhtvghrnhepheeigfeuveeutdef hfehgeekvedtleeuueekveefudehhffhjeffgfegffelfeegnecuvehluhhsthgvrhfuih iivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepnhhikhhlrghsrdhsohguvghrlhhu nhgusehrrghgnhgrthgvtghhrdhsvgdpnhgspghrtghpthhtohepgedpmhhouggvpehsmh htphhouhhtpdhrtghpthhtohepjhgrtghophhordhmohhnughisehiuggvrghsohhnsgho rghrugdrtghomhdprhgtphhtthhopehlrghurhgvnhhtrdhpihhntghhrghrthesihguvg grshhonhgsohgrrhgurdgtohhmpdhrtghpthhtoheplhhisggtrghmvghrrgdquggvvhgv lheslhhishhtshdrlhhisggtrghmvghrrgdrohhrghdprhgtphhtthhopehnihhklhgrsh drshhouggvrhhluhhnugdorhgvnhgvshgrshesrhgrghhnrghtvggthhdrshgv X-ME-Proxy: Feedback-ID: i80c9496c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 11 Sep 2025 05:06:37 -0400 (EDT) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: Jacopo Mondi , Laurent Pinchart , libcamera-devel@lists.libcamera.org Cc: =?utf-8?q?Niklas_S=C3=B6derlund?= Subject: [PATCH v2 2/2] libcamera: pipeline: Add R-Car Gen4 ISP pipeline Date: Thu, 11 Sep 2025 11:05:47 +0200 Message-ID: <20250911090547.1860863-3-niklas.soderlund+renesas@ragnatech.se> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250911090547.1860863-1-niklas.soderlund+renesas@ragnatech.se> References: <20250911090547.1860863-1-niklas.soderlund+renesas@ragnatech.se> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a pipeline handler for R-Car Gen4. The pipeline makes use of the RkISP1 IPA and, depending on the kernel R-Car ISP driver, support the same features as the RkISP1 pipeline handler. The pipeline and IPA is tested with the AWB measurement, Histogram Measurement, Auto Exposure Measurement, AWB gain, Black Level Correction, Color Correction Matrix, Lens Shade Correction and Gamma Correction algorithms and produce a stable and good image. Tested on R-Car V4H Sparrow-Hawk together with IMX219 and IMX462 sensors. Signed-off-by: Niklas Söderlund --- * Changes since v1 - Fix camera names. Was 'imx462 2-001a', is now '/base/soc/i2c@e6510000/cam@1a'. - Fix a compiler issue with some versions of gcc. - Add dependency on rkisp1.mojom. - Rebase on latest master branch which requires reworking some interfaces. --- include/libcamera/ipa/meson.build | 1 + meson.build | 1 + meson_options.txt | 1 + src/libcamera/pipeline/rcar-gen4/frames.cpp | 280 ++++++ src/libcamera/pipeline/rcar-gen4/frames.h | 85 ++ src/libcamera/pipeline/rcar-gen4/isp.cpp | 227 +++++ src/libcamera/pipeline/rcar-gen4/isp.h | 44 + src/libcamera/pipeline/rcar-gen4/meson.build | 8 + .../pipeline/rcar-gen4/rcar-gen4.cpp | 814 ++++++++++++++++++ src/libcamera/pipeline/rcar-gen4/vin.cpp | 391 +++++++++ src/libcamera/pipeline/rcar-gen4/vin.h | 66 ++ 11 files changed, 1918 insertions(+) create mode 100644 src/libcamera/pipeline/rcar-gen4/frames.cpp create mode 100644 src/libcamera/pipeline/rcar-gen4/frames.h create mode 100644 src/libcamera/pipeline/rcar-gen4/isp.cpp create mode 100644 src/libcamera/pipeline/rcar-gen4/isp.h create mode 100644 src/libcamera/pipeline/rcar-gen4/meson.build create mode 100644 src/libcamera/pipeline/rcar-gen4/rcar-gen4.cpp create mode 100644 src/libcamera/pipeline/rcar-gen4/vin.cpp create mode 100644 src/libcamera/pipeline/rcar-gen4/vin.h diff --git a/include/libcamera/ipa/meson.build b/include/libcamera/ipa/meson.build index 3ee3ada303c0..48eb19b2f0f6 100644 --- a/include/libcamera/ipa/meson.build +++ b/include/libcamera/ipa/meson.build @@ -66,6 +66,7 @@ pipeline_ipa_mojom_mapping = { 'ipu3': 'ipu3.mojom', 'mali-c55': 'mali-c55.mojom', 'rkisp1': 'rkisp1.mojom', + 'rcar-gen4': 'rkisp1.mojom', 'rpi/pisp': 'raspberrypi.mojom', 'rpi/vc4': 'raspberrypi.mojom', 'simple': 'soft.mojom', diff --git a/meson.build b/meson.build index fd508fd7f6b5..054e209e8b92 100644 --- a/meson.build +++ b/meson.build @@ -218,6 +218,7 @@ pipelines_support = { 'imx8-isi': arch_arm, 'ipu3': arch_x86, 'mali-c55': arch_arm, + 'rcar-gen4': arch_arm, 'rkisp1': arch_arm, 'rpi/pisp': arch_arm, 'rpi/vc4': arch_arm, diff --git a/meson_options.txt b/meson_options.txt index 2104469e3793..eba9458487ff 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -51,6 +51,7 @@ option('pipelines', 'imx8-isi', 'ipu3', 'mali-c55', + 'rcar-gen4', 'rkisp1', 'rpi/pisp', 'rpi/vc4', diff --git a/src/libcamera/pipeline/rcar-gen4/frames.cpp b/src/libcamera/pipeline/rcar-gen4/frames.cpp new file mode 100644 index 000000000000..c604924b7127 --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/frames.cpp @@ -0,0 +1,280 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright 2025 Renesas Electronics Co + * Copyright 2025 Niklas Söderlund + * + * Renesas R-Car Gen4 VIN pipeline + */ + +#include "frames.h" + +#include + +#include +#include + +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/pipeline_handler.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(RCar4) + +int RCar4Frames::start(class RCarISPDevice *isp, class ipa::rkisp1::IPAProxyRkISP1 *ipa) +{ + unsigned int ipaBufferId = 1; + unsigned int bufferCount; + int ret; + + auto pushBuffers = [&](const std::vector> &buffers, + std::queue &queue) { + for (const std::unique_ptr &buffer : buffers) { + Span planes = buffer->planes(); + + buffer->setCookie(ipaBufferId++); + ipaBuffers_.emplace_back(buffer->cookie(), + std::vector{ planes.begin(), + planes.end() }); + queue.push(buffer.get()); + } + }; + + frameInfo_.clear(); + + bufferCount = std::max({ + rawStream_.configuration().bufferCount, + outputStream_.configuration().bufferCount, + }); + + ret = isp->input_->exportBuffers(bufferCount, &inputBuffers_); + if (ret < 0) { + LOG(RCar4, Error) << "Failed to allocate ISP input buffers"; + goto error; + } + + ret = isp->param_->allocateBuffers(bufferCount, ¶mBuffers_); + if (ret < 0) { + LOG(RCar4, Error) << "Failed to allocate ISP param buffers"; + goto error; + } + + ret = isp->stat_->allocateBuffers(bufferCount, &statBuffers_); + if (ret < 0) { + LOG(RCar4, Error) << "Failed to allocate ISP stat buffers"; + goto error; + } + + ret = isp->output_->exportBuffers(bufferCount, &outputBuffers_); + if (ret < 0) { + LOG(RCar4, Error) << "Failed to allocate ISP output buffers"; + goto error; + } + + for (const std::unique_ptr &buffer : inputBuffers_) + availableInputBuffers_.push(buffer.get()); + + pushBuffers(paramBuffers_, availableParamBuffers_); + pushBuffers(statBuffers_, availableStatBuffers_); + + for (const std::unique_ptr &buffer : outputBuffers_) + availableOutputBuffers_.push(buffer.get()); + + ipa->mapBuffers(ipaBuffers_); + + return 0; +error: + stop(isp, ipa); + return ret; +} + +void RCar4Frames::stop(class RCarISPDevice *isp, class ipa::rkisp1::IPAProxyRkISP1 *ipa) +{ + std::vector ids; + + availableInputBuffers_ = {}; + availableParamBuffers_ = {}; + availableStatBuffers_ = {}; + availableOutputBuffers_ = {}; + + outputBuffers_.clear(); + statBuffers_.clear(); + paramBuffers_.clear(); + inputBuffers_.clear(); + + for (IPABuffer &ipabuf : ipaBuffers_) + ids.push_back(ipabuf.id); + + ipa->unmapBuffers(ids); + ipaBuffers_.clear(); + + if (isp->output_->releaseBuffers()) + LOG(RCar4, Error) << "Failed to release ISP output buffers"; + + if (isp->stat_->releaseBuffers()) + LOG(RCar4, Error) << "Failed to release ISP stat buffers"; + + if (isp->param_->releaseBuffers()) + LOG(RCar4, Error) << "Failed to release ISP param buffers"; + + if (isp->input_->releaseBuffers()) + LOG(RCar4, Error) << "Failed to release ISP input buffers"; +} + +RCar4Frames::Info *RCar4Frames::create(Request *request) +{ + unsigned int frame = request->sequence(); + + /* Try to get input and output buffers from request. */ + FrameBuffer *inputBuffer = request->findBuffer(&rawStream_); + FrameBuffer *outputBuffer = request->findBuffer(&outputStream_); + + /* Make sure we have enough internal buffers. */ + if (!inputBuffer && availableInputBuffers_.empty()) { + LOG(RCar4, Debug) << "Input buffer underrun"; + return nullptr; + } + + if (availableParamBuffers_.empty()) { + LOG(RCar4, Debug) << "Parameters buffer underrun"; + return nullptr; + } + + if (availableStatBuffers_.empty()) { + LOG(RCar4, Debug) << "Statistics buffer underrun"; + return nullptr; + } + + if (!outputBuffer && availableOutputBuffers_.empty()) { + LOG(RCar4, Debug) << "Output buffer underrun"; + return nullptr; + } + + /* Select buffers to use. */ + if (!inputBuffer) { + inputBuffer = availableInputBuffers_.front(); + availableInputBuffers_.pop(); + inputBuffer->_d()->setRequest(request); + } + + FrameBuffer *paramBuffer = availableParamBuffers_.front(); + availableParamBuffers_.pop(); + paramBuffer->_d()->setRequest(request); + + FrameBuffer *statBuffer = availableStatBuffers_.front(); + availableStatBuffers_.pop(); + statBuffer->_d()->setRequest(request); + + if (!outputBuffer) { + outputBuffer = availableOutputBuffers_.front(); + availableOutputBuffers_.pop(); + outputBuffer->_d()->setRequest(request); + } + + /* Recored the info needed to process one frame. */ + auto [it, inserted] = frameInfo_.try_emplace(frame); + if (!inserted) + return nullptr; + + auto &info = it->second; + + info.frame = frame; + info.request = request; + info.inputBuffer = inputBuffer; + info.paramBuffer = paramBuffer; + info.statBuffer = statBuffer; + info.outputBuffer = outputBuffer; + info.rawDequeued = false; + info.paramDequeued = false; + info.metadataProcessed = false; + info.outputDequeued = false; + + return &info; +} + +void RCar4Frames::remove(RCar4Frames::Info *info) +{ + /* If internal input buffer used, return for reuse. */ + for (const std::unique_ptr &buf : inputBuffers_) { + if (info->inputBuffer == buf.get()) { + availableInputBuffers_.push(info->inputBuffer); + break; + } + } + + /* Return param and stat buffer for reuse. */ + availableParamBuffers_.push(info->paramBuffer); + availableStatBuffers_.push(info->statBuffer); + + /* If internal output buffer used, return for reuse. */ + for (const std::unique_ptr &buf : outputBuffers_) { + if (info->outputBuffer == buf.get()) { + availableOutputBuffers_.push(info->outputBuffer); + break; + } + } + + /* Delete the extended frame information. */ + frameInfo_.erase(info->frame); +} + +bool RCar4Frames::tryComplete(RCar4Frames::Info *info) +{ + Request *request = info->request; + + if (request->hasPendingBuffers()) + return false; + + if (!info->rawDequeued) + return false; + + if (!info->metadataProcessed) + return false; + + if (!info->paramDequeued) + return false; + + if (!info->outputDequeued) + return false; + + remove(info); + + /* Signal new internal buffers available. */ + bufferAvailable.emit(); + + return true; +} + +RCar4Frames::Info *RCar4Frames::find(unsigned int frame) +{ + const auto &itInfo = frameInfo_.find(frame); + + if (itInfo != frameInfo_.end()) + return &itInfo->second; + + LOG(RCar4, Fatal) << "Can't find tracking information for frame " << frame; + + return nullptr; +} + +RCar4Frames::Info *RCar4Frames::find(FrameBuffer *buffer) +{ + for (auto &itInfo : frameInfo_) { + Info *info = &itInfo.second; + + for (auto const itBuffers : info->request->buffers()) + if (itBuffers.second == buffer) + return info; + + if (info->inputBuffer == buffer || + info->paramBuffer == buffer || + info->statBuffer == buffer || + info->outputBuffer == buffer) + return info; + } + + LOG(RCar4, Fatal) << "Can't find tracking information from buffer"; + + return nullptr; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rcar-gen4/frames.h b/src/libcamera/pipeline/rcar-gen4/frames.h new file mode 100644 index 000000000000..24573fe07a25 --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/frames.h @@ -0,0 +1,85 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright 2025 Renesas Electronics Co + * Copyright 2025 Niklas Söderlund + * + * Renesas R-Car Gen4 VIN pipeline + */ + +#pragma once + +#include +#include +#include +#include + +#include + +#include +#include + +#include +#include +#include + +#include "isp.h" + +namespace libcamera { + +class FrameBuffer; +class Request; + +class RCar4Frames +{ +public: + struct Info { + unsigned int frame; + Request *request; + + FrameBuffer *inputBuffer; + FrameBuffer *paramBuffer; + FrameBuffer *statBuffer; + FrameBuffer *outputBuffer; + + ControlList effectiveSensorControls; + + bool rawDequeued; + bool paramDequeued; + bool metadataProcessed; + bool outputDequeued; + }; + + int start(class RCarISPDevice *isp, class ipa::rkisp1::IPAProxyRkISP1 *ipa); + void stop(class RCarISPDevice *isp, class ipa::rkisp1::IPAProxyRkISP1 *ipa); + + Info *create(Request *request); + void remove(Info *info); + bool tryComplete(Info *info); + + Info *find(unsigned int frame); + Info *find(FrameBuffer *buffer); + + Signal<> bufferAvailable; + + Stream rawStream_; + Stream outputStream_; +private: + std::map frameInfo_; + + /* Buffers for internal use, if none is provided in request. */ + std::vector> inputBuffers_; + std::vector> paramBuffers_; + std::vector> statBuffers_; + std::vector> outputBuffers_; + + /* Queues of available internal buffers. */ + std::queue availableInputBuffers_; + std::queue availableParamBuffers_; + std::queue availableStatBuffers_; + std::queue availableOutputBuffers_; + + /* Buffers mapped and shared with IPA. */ + std::vector ipaBuffers_; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rcar-gen4/isp.cpp b/src/libcamera/pipeline/rcar-gen4/isp.cpp new file mode 100644 index 000000000000..1d9d7f98429c --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/isp.cpp @@ -0,0 +1,227 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright 2025 Renesas Electronics Co + * Copyright 2025 Niklas Söderlund + * + * Renesas R-Car Gen4 ISP pipeline + */ + +#include "isp.h" + +#include +#include +#include + +#include + +#include +#include + +#include +#include + +#include "libcamera/internal/media_device.h" +#include "libcamera/internal/v4l2_subdevice.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(RCar4) + +int RCarISPDevice::init(const MediaDevice *media, const std::string &pipeId) +{ + const MediaEntity *entity; + const MediaPad *pad, *next; + int ret; + + /* Locate IPSCORE, e.g. rcar_isp fed00000.isp core */ + std::unique_ptr core = + V4L2Subdevice::fromEntityName(media, pipeId + " core"); + if (!core) { + LOG(RCar4, Error) << "Failed to find ISPCORE " << pipeId; + return -EINVAL; + } + + entity = core->entity(); + + /* Use the media links to find all video devices. */ + pad = entity->getPadByIndex(0); + next = pad->links()[0]->source(); + input_ = V4L2VideoDevice::fromEntityName(media, next->entity()->name()); + if (!input_) { + LOG(RCar4, Error) << "Failed to find ISP input entity"; + return -EINVAL; + } + + pad = entity->getPadByIndex(1); + next = pad->links()[0]->source(); + param_ = V4L2VideoDevice::fromEntityName(media, next->entity()->name()); + if (!param_) { + LOG(RCar4, Error) << "Failed to find ISP param entity"; + return -EINVAL; + } + + pad = entity->getPadByIndex(2); + next = pad->links()[0]->sink(); + stat_ = V4L2VideoDevice::fromEntityName(media, next->entity()->name()); + if (!stat_) { + LOG(RCar4, Error) << "Failed to find ISP stat entity"; + return -EINVAL; + } + + pad = entity->getPadByIndex(3); + next = pad->links()[0]->sink(); + output_ = V4L2VideoDevice::fromEntityName(media, next->entity()->name()); + if (!output_) { + LOG(RCar4, Error) << "Failed to find ISP output entity"; + return -EINVAL; + } + + /* Open all devices. */ + ret = input_->open(); + if (ret) + return ret; + + ret = param_->open(); + if (ret) + return ret; + + ret = stat_->open(); + if (ret) + return ret; + + ret = output_->open(); + if (ret) + return ret; + + return 0; +} + +std::vector RCarISPDevice::formats() const +{ + std::vector formats; + for (const auto &[format, sizes] : output_->formats()) + formats.push_back(format.toPixelFormat()); + + return formats; +} + +StreamConfiguration +RCarISPDevice::generateConfiguration(PixelFormat format, Size size) const +{ + StreamConfiguration cfg; + + bool found = false; + for (const auto &pixelFormat : formats()) { + if (pixelFormat == format) + found = true; + } + + cfg.size = size; + cfg.bufferCount = kBufferCount; + cfg.pixelFormat = found ? format : formats::XRGB8888; + + /* Get stride and frame size from device. */ + V4L2DeviceFormat fmt; + fmt.fourcc = output_->toV4L2PixelFormat(cfg.pixelFormat); + fmt.size = cfg.size; + + if (output_->tryFormat(&fmt)) + return {}; + + cfg.stride = fmt.planes[0].bpl; + cfg.frameSize = fmt.planes[0].size; + + return cfg; +} + +int RCarISPDevice::configure(V4L2DeviceFormat *inputFormat, + const PixelFormat &outputPixelFormat) +{ + int ret; + + /* Configure the RAW input. */ + ret = input_->setFormat(inputFormat); + if (ret) + return ret; + + LOG(RCar4, Debug) << "ISP input format = " << *inputFormat; + + /* Configure the image output. */ + V4L2DeviceFormat outputFormat; + outputFormat.fourcc = output_->toV4L2PixelFormat(outputPixelFormat); + outputFormat.size = inputFormat->size; + ret = output_->setFormat(&outputFormat); + if (ret) + return ret; + + LOG(RCar4, Debug) << "ISP output format = " << outputFormat; + + /* Configure paramaters. */ + V4L2DeviceFormat paramFormat; + paramFormat.fourcc = V4L2PixelFormat(V4L2_META_FMT_RK_ISP1_EXT_PARAMS); + ret = param_->setFormat(¶mFormat); + if (ret) + return ret; + + /* Configure statistics. */ + V4L2DeviceFormat statFormat; + statFormat.fourcc = V4L2PixelFormat(V4L2_META_FMT_RK_ISP1_STAT_3A); + ret = stat_->setFormat(&statFormat); + if (ret) + return ret; + + return 0; +} + +int RCarISPDevice::start() +{ + int ret; + + ret = input_->importBuffers(kBufferCount); + if (ret) { + LOG(RCar4, Error) << "Failed to import ISP input buffers"; + return ret; + } + + ret = output_->importBuffers(kBufferCount); + if (ret) { + LOG(RCar4, Error) << "Failed to import ISP output buffers"; + return ret; + } + + ret = output_->streamOn(); + if (ret) { + LOG(RCar4, Error) << "Failed to start ISP output"; + return ret; + } + + ret = param_->streamOn(); + if (ret) { + LOG(RCar4, Error) << "Failed to start ISP param"; + return ret; + } + + ret = stat_->streamOn(); + if (ret) { + LOG(RCar4, Error) << "Failed to start ISP stat"; + return ret; + } + + ret = input_->streamOn(); + if (ret) { + LOG(RCar4, Error) << "Failed to start ISP input"; + return ret; + } + + return 0; +} + +void RCarISPDevice::stop() +{ + output_->streamOff(); + param_->streamOff(); + stat_->streamOff(); + input_->streamOff(); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rcar-gen4/isp.h b/src/libcamera/pipeline/rcar-gen4/isp.h new file mode 100644 index 000000000000..57148dc43041 --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/isp.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright 2025 Renesas Electronics Co + * Copyright 2025 Niklas Söderlund + * + * Renesas R-Car Gen4 ISP pipeline + */ + +#pragma once + +#include +#include + +#include "libcamera/internal/v4l2_videodevice.h" + +namespace libcamera { + +class MediaDevice; +class Size; +struct StreamConfiguration; + +class RCarISPDevice +{ +public: + static constexpr unsigned int kBufferCount = 4; + + std::vector formats() const; + + int init(const MediaDevice *media, const std::string &pipeId); + + StreamConfiguration generateConfiguration(PixelFormat format, Size size) const; + + int configure(V4L2DeviceFormat *inputFormat, const PixelFormat &outputPixelFormat); + + int start(); + void stop(); + + std::unique_ptr input_; + std::unique_ptr param_; + std::unique_ptr stat_; + std::unique_ptr output_; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rcar-gen4/meson.build b/src/libcamera/pipeline/rcar-gen4/meson.build new file mode 100644 index 000000000000..431eb54e2803 --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/meson.build @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: CC0-1.0 + +libcamera_internal_sources += files([ + 'frames.cpp', + 'isp.cpp', + 'rcar-gen4.cpp', + 'vin.cpp', +]) diff --git a/src/libcamera/pipeline/rcar-gen4/rcar-gen4.cpp b/src/libcamera/pipeline/rcar-gen4/rcar-gen4.cpp new file mode 100644 index 000000000000..b75197b24ae3 --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/rcar-gen4.cpp @@ -0,0 +1,814 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright 2025 Renesas Electronics Co + * Copyright 2025 Niklas Söderlund + * + * Renesas R-Car Gen4 ISP pipeline + */ + +#include +#include +#include +#include + +#include + +#include +#include + +#include +#include +#include + +#include "libcamera/internal/camera.h" +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/delayed_controls.h" +#include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/ipa_manager.h" +#include "libcamera/internal/mapped_framebuffer.h" +#include "libcamera/internal/media_device.h" +#include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/v4l2_device.h" +#include "libcamera/internal/v4l2_subdevice.h" +#include "libcamera/internal/v4l2_videodevice.h" + +#include "frames.h" +#include "isp.h" +#include "vin.h" + +namespace libcamera { + +namespace { + +const std::map formatToMediaBus = { + { formats::SBGGR10, MEDIA_BUS_FMT_SBGGR10_1X10 }, + { formats::SGBRG10, MEDIA_BUS_FMT_SGBRG10_1X10 }, + { formats::SGRBG10, MEDIA_BUS_FMT_SGRBG10_1X10 }, + { formats::SRGGB10, MEDIA_BUS_FMT_SRGGB10_1X10 }, +}; + +/* Max supported resolution of VIN and ISP. */ +static constexpr Size MaxResolution = { 4096, 4096 }; +static constexpr unsigned int nBuffers = 4; + +} /* namespace */ + +LOG_DEFINE_CATEGORY(RCar4) + +/* ----------------------------------------------------------------------------- + * Camera Data + */ + +class RCar4CameraData : public Camera::Private +{ +public: + RCar4CameraData(PipelineHandler *pipe) + : Camera::Private(pipe) + { + } + + RCarVINDevice vin_; + RCarISPDevice isp_; + std::unique_ptr ipa_; + + RCar4Frames frames_; + std::unique_ptr delayedCtrls_; + ControlInfoMap ipaControls_; + + void queuePendingRequests(); + void cancelPendingRequests(); + + /* Requests for which no buffer has been queued to the VIN device yet. */ + std::queue pendingRequests_; + + /* Slots for processing ready buffers. */ + void vinBufferReady(FrameBuffer *buffer); + void inputBufferReady(FrameBuffer *buffer); + void paramBufferReady(FrameBuffer *buffer); + void statBufferReady(FrameBuffer *buffer); + void outputBufferReady(FrameBuffer *buffer); + + /* Slots for processing IPA interactions. */ + void paramsComputed(unsigned int frame, unsigned int bytesused); + void setSensorControls(unsigned int frame, + const ControlList &sensorControls); + void metadataReady(unsigned int frame, const ControlList &metadata); +}; + +void RCar4CameraData::queuePendingRequests() +{ + while (!pendingRequests_.empty()) { + Request *request = pendingRequests_.front(); + + RCar4Frames::Info *info = frames_.create(request); + if (!info) + break; + + if (vin_.queueBuffer(info->inputBuffer)) { + /* Remove if raw buffer failed, should not happen. */ + frames_.remove(info); + break; + } + + ipa_->queueRequest(info->frame, request->controls()); + + pendingRequests_.pop(); + } +} + +void RCar4CameraData::cancelPendingRequests() +{ + while (!pendingRequests_.empty()) { + Request *request = pendingRequests_.front(); + + for (auto it : request->buffers()) { + FrameBuffer *buffer = it.second; + buffer->_d()->cancel(); + pipe()->completeBuffer(request, buffer); + } + + pipe()->completeRequest(request); + pendingRequests_.pop(); + } +} + +void RCar4CameraData::vinBufferReady(FrameBuffer *buffer) +{ + RCar4Frames::Info *info = frames_.find(buffer); + if (!info) + return; + + Request *request = info->request; + + /* If the buffer is cancelled force a complete of the whole request. */ + if (buffer->metadata().status == FrameMetadata::FrameCancelled) { + for (auto it : request->buffers()) { + FrameBuffer *b = it.second; + b->_d()->cancel(); + pipe()->completeBuffer(request, b); + } + + frames_.remove(info); + pipe()->completeRequest(request); + return; + } + + /* Record the sensor's timestamp in the request metadata. */ + request->metadata().set(controls::SensorTimestamp, + buffer->metadata().timestamp); + + ipa_->computeParams(info->frame, info->paramBuffer->cookie()); +} + +void RCar4CameraData::inputBufferReady(FrameBuffer *buffer) +{ + RCar4Frames::Info *info = frames_.find(buffer); + if (!info) + return; + + Request *request = info->request; + + if (request->findBuffer(&frames_.rawStream_)) + pipe()->completeBuffer(request, buffer); + + info->rawDequeued = true; + + if (frames_.tryComplete(info)) + pipe()->completeRequest(request); +} + +void RCar4CameraData::paramBufferReady(FrameBuffer *buffer) +{ + RCar4Frames::Info *info = frames_.find(buffer); + if (!info) + return; + + Request *request = info->request; + + info->paramDequeued = true; + + if (frames_.tryComplete(info)) + pipe()->completeRequest(request); +} + +void RCar4CameraData::statBufferReady(FrameBuffer *buffer) +{ + RCar4Frames::Info *info = frames_.find(buffer); + if (!info) + return; + + Request *request = info->request; + + if (buffer->metadata().status == FrameMetadata::FrameCancelled) { + info->metadataProcessed = true; + + if (frames_.tryComplete(info)) + pipe()->completeRequest(request); + + return; + } + + ipa_->processStats(info->frame, info->statBuffer->cookie(), + delayedCtrls_->get(buffer->metadata().sequence)); +} + +void RCar4CameraData::outputBufferReady(FrameBuffer *buffer) +{ + RCar4Frames::Info *info = frames_.find(buffer); + if (!info) + return; + + Request *request = info->request; + + if (request->findBuffer(&frames_.outputStream_)) + pipe()->completeBuffer(request, buffer); + + request->metadata().set(controls::draft::PipelineDepth, 3); + + info->outputDequeued = true; + + if (frames_.tryComplete(info)) + pipe()->completeRequest(request); +} + +void RCar4CameraData::paramsComputed(unsigned int frame, unsigned int bytesused) +{ + RCar4Frames::Info *info = frames_.find(frame); + if (!info) + return; + + info->paramBuffer->_d()->metadata().planes()[0].bytesused = bytesused; + + isp_.output_->queueBuffer(info->outputBuffer); + isp_.param_->queueBuffer(info->paramBuffer); + isp_.stat_->queueBuffer(info->statBuffer); + isp_.input_->queueBuffer(info->inputBuffer); +} + +void RCar4CameraData::setSensorControls([[maybe_unused]] unsigned int frame, + const ControlList &sensorControls) +{ + delayedCtrls_->push(sensorControls); +} + +void RCar4CameraData::metadataReady(unsigned int frame, const ControlList &metadata) +{ + RCar4Frames::Info *info = frames_.find(frame); + if (!info) + return; + + Request *request = info->request; + + info->request->metadata().merge(metadata); + info->metadataProcessed = true; + + if (frames_.tryComplete(info)) + pipe()->completeRequest(request); +} + +/* ----------------------------------------------------------------------------- + * Camera Configuration + */ + +class RCar4CameraConfiguration : public CameraConfiguration +{ +public: + RCar4CameraConfiguration(Camera *camera, RCar4CameraData *data); + + Status validate() override; + + const V4L2SubdeviceFormat &sensorFormat() { return sensorFormat_; } + const Transform &combinedTransform() { return combinedTransform_; } + const PixelFormat &ispOutputFormat() { return ispOutputFormat_; } +private: + std::shared_ptr camera_; + RCar4CameraData *data_; + + V4L2SubdeviceFormat sensorFormat_; + Transform combinedTransform_; + PixelFormat ispOutputFormat_; +}; + +RCar4CameraConfiguration::RCar4CameraConfiguration(Camera *camera, + RCar4CameraData *data) + : CameraConfiguration(), camera_(camera->shared_from_this()), data_(data) +{ +} + +CameraConfiguration::Status RCar4CameraConfiguration::validate() +{ + Status status; + + if (config_.empty()) + return Invalid; + + status = validateColorSpaces(ColorSpaceFlag::StreamsShareColorSpace); + + /* Cap the number of entries to the available streams. */ + if (config_.size() > 2) { + config_.resize(2); + status = Adjusted; + } + + Orientation requestedOrientation = orientation; + combinedTransform_ = data_->vin_.sensor()->computeTransform(&orientation); + if (orientation != requestedOrientation) + status = Adjusted; + + /* Figure out the VIN configuration based on the first stream size. */ + StreamConfiguration vinCfg = data_->vin_.generateConfiguration(config_.at(0).size); + + /* Default ISP output format. */ + ispOutputFormat_ = formats::XRGB8888; + + /* + * Validate there are at max two streams, one output and one RAW. The + * size of two streams must match each other and the sensor output as we + * have no scaler. + */ + unsigned int outputStreams = 0; + unsigned int rawStreams = 0; + for (unsigned int i = 0; i < config_.size(); i++) { + StreamConfiguration &cfg = config_.at(i); + StreamConfiguration newCfg = {}; + const StreamConfiguration originalCfg = cfg; + const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat); + + LOG(RCar4, Debug) << "Validating stream: " << cfg.toString(); + + if (info.colourEncoding == PixelFormatInfo::ColourEncodingRAW) { + if (rawStreams++) { + LOG(RCar4, Error) + << "Camera configuration support only one RAW stream"; + return Invalid; + } + + newCfg = vinCfg; + + cfg.setStream(&data_->frames_.rawStream_); + LOG(RCar4, Debug) << "Assigned " << newCfg.toString() + << " to the raw stream"; + } else { + if (outputStreams++) { + LOG(RCar4, Error) + << "Camera configuration support only one output stream"; + return Invalid; + } + + newCfg = data_->isp_.generateConfiguration(cfg.pixelFormat, vinCfg.size); + ispOutputFormat_ = newCfg.pixelFormat; + + cfg.setStream(&data_->frames_.outputStream_); + LOG(RCar4, Debug) << "Assigned " << newCfg.toString() + << " to the output stream"; + } + + cfg.size = newCfg.size; + cfg.bufferCount = newCfg.bufferCount; + cfg.pixelFormat = newCfg.pixelFormat; + cfg.stride = newCfg.stride; + cfg.frameSize = newCfg.frameSize; + + if (!cfg.pixelFormat.isValid()) { + LOG(RCar4, Error) + << "Stream " << i << " can not generate cfg"; + return Invalid; + } + + if (cfg.pixelFormat != originalCfg.pixelFormat || + cfg.size != originalCfg.size) { + LOG(RCar4, Debug) + << "Stream " << i << " configuration adjusted to " + << cfg.toString(); + status = Adjusted; + } + } + + /* Select the sensor format. */ + std::vector mbusCodes = { formatToMediaBus.at(vinCfg.pixelFormat) }; + sensorFormat_ = + data_->vin_.sensor()->getFormat(mbusCodes, vinCfg.size, vinCfg.size); + + return status; +} + +/* ----------------------------------------------------------------------------- + * Pipeline Handler + */ + +class PipelineHandlerRCar4 : public PipelineHandler +{ +public: + PipelineHandlerRCar4(CameraManager *manager); + + std::unique_ptr generateConfiguration(Camera *camera, + Span roles) override; + int configure(Camera *camera, CameraConfiguration *config) override; + + int exportFrameBuffers(Camera *camera, Stream *stream, + std::vector> *buffers) override; + + int start(Camera *camera, const ControlList *controls) override; + void stopDevice(Camera *camera) override; + + int queueRequestDevice(Camera *camera, Request *request) override; + + bool match(DeviceEnumerator *enumerator) override; + + int updateControls(RCar4CameraData *data); + +private: + RCar4CameraData *cameraData(Camera *camera) + { + return static_cast(camera->_d()); + } + + int createCamera(MediaDevice *mdev, const std::string &pipeId); + + StreamConfiguration generateStreamConfiguration(RCar4CameraData *data, + StreamRole role); +}; + +PipelineHandlerRCar4::PipelineHandlerRCar4(CameraManager *manager) + : PipelineHandler(manager) +{ +} + +StreamConfiguration +PipelineHandlerRCar4::generateStreamConfiguration(RCar4CameraData *data, + StreamRole role) +{ + const std::vector &mbusCodes = data->vin_.sensor()->mbusCodes(); + + /* Create the list of supported RAW stream formats. */ + std::map> rawFormats; + unsigned int rawBitsPerPixel = 0; + PixelFormat rawFormat; + Size rawSize = { 0, 0 }; + std::vector rawSizes; + + for (const auto &format : data->vin_.formats()) { + const PixelFormatInfo &info = PixelFormatInfo::info(format); + + /* Populate stream formats for RAW configurations. */ + uint32_t mbusCode = formatToMediaBus.at(format); + + /* Skip formats not supported by sensor. */ + if (std::find(mbusCodes.begin(), mbusCodes.end(), mbusCode) == mbusCodes.end()) + continue; + + /* Add all the RAW sizes the sensor can produce for this code. */ + for (const auto &rawSizeByCode : data->vin_.sensor()->sizes(mbusCode)) { + if (rawSizeByCode.width > MaxResolution.width || + rawSizeByCode.height > MaxResolution.height) + continue; + + rawSizes.push_back({ rawSizeByCode, rawSizeByCode }); + + rawFormats[format].push_back({ rawSizeByCode, rawSizeByCode }); + + /* Cache for later default format. */ + if (info.bitsPerPixel >= rawBitsPerPixel) { + rawBitsPerPixel = info.bitsPerPixel; + rawFormat = format; + + if (rawSizeByCode > rawSize) + rawSize = rawSizeByCode; + } + } + } + + /* If generating for RAW role we are done. */ + if (role == StreamRole::Raw) { + StreamFormats rawStreamFormats(rawFormats); + StreamConfiguration rawCfg(rawStreamFormats); + rawCfg.pixelFormat = rawFormat; + rawCfg.size = rawSize; + rawCfg.bufferCount = nBuffers; + + return rawCfg; + } + + /* Create the list of supported other stream formats. */ + std::map> outputFormats; + std::vector outputSizes(rawSizes.begin(), rawSizes.end()); + + for (const auto &format : data->isp_.formats()) { + const PixelFormatInfo &info = PixelFormatInfo::info(format); + + /* Skip RAW formats. */ + if (info.colourEncoding == PixelFormatInfo::ColourEncodingRAW) + continue; + + outputFormats[format] = { outputSizes }; + } + + StreamFormats outputStreamFormats(outputFormats); + StreamConfiguration outputCfg(outputStreamFormats); + outputCfg.pixelFormat = formats::XRGB8888; + outputCfg.size = rawSize; + + return outputCfg; +} + +std::unique_ptr +PipelineHandlerRCar4::generateConfiguration(Camera *camera, + Span roles) +{ + RCar4CameraData *data = cameraData(camera); + + std::unique_ptr config = + std::make_unique(camera, data); + + if (roles.empty()) + return config; + + for (const StreamRole role : roles) { + std::optional colorSpace; + + switch (role) { + case StreamRole::Raw: + colorSpace = ColorSpace::Raw; + break; + default: + colorSpace = ColorSpace::Rec709; + break; + } + + StreamConfiguration cfg = + generateStreamConfiguration(data, role); + if (!cfg.pixelFormat.isValid()) + return nullptr; + + cfg.colorSpace = colorSpace; + cfg.bufferCount = nBuffers; + config->addConfiguration(cfg); + } + + if (config->validate() == CameraConfiguration::Invalid) + return nullptr; + + return config; +} + +int PipelineHandlerRCar4::configure(Camera *camera, CameraConfiguration *c) +{ + RCar4CameraConfiguration *config = static_cast(c); + RCar4CameraData *data = cameraData(camera); + + V4L2DeviceFormat vinFormat; + int ret; + + /* Configure VIN and propagate format to ISP. */ + ret = data->vin_.configure(config->sensorFormat().size, + config->combinedTransform(), &vinFormat); + if (ret) + return ret; + + ret = data->isp_.configure(&vinFormat, config->ispOutputFormat()); + if (ret) + return ret; + + /* Inform IPA of stream configuration and sensor controls. */ + IPACameraSensorInfo sensorInfo; + ret = data->vin_.sensor()->sensorInfo(&sensorInfo); + if (ret) + return ret; + + ipa::rkisp1::IPAConfigInfo ipaConfig{ + sensorInfo, data->vin_.sensor()->controls(), + V4L2_META_FMT_RK_ISP1_EXT_PARAMS + }; + + std::map streamConfig; + streamConfig[0] = + IPAStream(PixelFormat(config->ispOutputFormat().fourcc()), + config->sensorFormat().size); + + ret = data->ipa_->configure(ipaConfig, streamConfig, &data->ipaControls_); + if (ret) { + LOG(RCar4, Error) << "failed configuring IPA (" << ret << ")"; + return ret; + } + + return updateControls(data); +} + +int PipelineHandlerRCar4::exportFrameBuffers(Camera *camera, Stream *stream, + std::vector> *buffers) +{ + RCar4CameraData *data = cameraData(camera); + unsigned int count = stream->configuration().bufferCount; + + if (stream == &data->frames_.outputStream_) + return data->isp_.output_->exportBuffers(count, buffers); + + if (stream == &data->frames_.rawStream_) + return data->isp_.input_->exportBuffers(count, buffers); + + return -EINVAL; +} + +int PipelineHandlerRCar4::start(Camera *camera, + [[maybe_unused]] const ControlList *controls) +{ + RCar4CameraData *data = cameraData(camera); + int ret; + + data->delayedCtrls_->reset(); + + ret = data->frames_.start(&data->isp_, data->ipa_.get()); + if (ret) + goto error; + + ret = data->ipa_->start(); + if (ret) + goto error; + + ret = data->vin_.start(); + if (ret) + goto error; + + ret = data->isp_.start(); + if (ret) + goto error; + + return 0; +error: + stop(camera); + + return ret; +} + +void PipelineHandlerRCar4::stopDevice(Camera *camera) +{ + RCar4CameraData *data = cameraData(camera); + + data->cancelPendingRequests(); + + data->isp_.stop(); + data->vin_.stop(); + data->ipa_->stop(); + + data->frames_.stop(&data->isp_, data->ipa_.get()); +} + +int PipelineHandlerRCar4::queueRequestDevice(Camera *camera, Request *request) +{ + RCar4CameraData *data = cameraData(camera); + + data->pendingRequests_.push(request); + data->queuePendingRequests(); + + return 0; +} + +int PipelineHandlerRCar4::updateControls(RCar4CameraData *data) +{ + ControlInfoMap::Map controls; + + for (const auto &ipaControl : data->ipaControls_) + controls[ipaControl.first] = ipaControl.second; + + data->controlInfo_ = ControlInfoMap(std::move(controls), + controls::controls); + return 0; +} + +int PipelineHandlerRCar4::createCamera(MediaDevice *mdev, + const std::string &pipeId) +{ + std::unique_ptr data = std::make_unique(this); + IPACameraSensorInfo sensorInfo{}; + int ret; + + ret = data->vin_.init(mdev, pipeId); + if (ret) + return ret; + + ret = data->isp_.init(mdev, pipeId); + if (ret) + return ret; + + /* + * Load RkISP1 IPA for use with RCar4 + */ + data->ipa_ = IPAManager::createIPA(this, 1, 1, "rkisp1"); + if (!data->ipa_) { + LOG(RCar4, Error) << "No IPA module found"; + return -ENOENT; + } + + /* The IPA tuning file is made from the sensor name. */ + std::string ipaTuningFile = + data->ipa_->configurationFile(data->vin_.sensor()->model() + ".yaml", "uncalibrated.yaml"); + + ret = data->vin_.sensor()->sensorInfo(&sensorInfo); + if (ret) { + LOG(RCar4, Error) << "Camera sensor information not available"; + return ret; + } + + ret = data->ipa_->init({ ipaTuningFile, data->vin_.sensor()->model() }, + libcamera::ipa::rkisp1::HwRevisionExternalRppX1, + sensorInfo, data->vin_.sensor()->controls(), &data->ipaControls_); + if (ret < 0) { + LOG(RCar4, Error) << "IPA initialization failure"; + return ret; + } + + updateControls(data.get()); + + /* + * Initialize the camera properties. + */ + data->properties_ = data->vin_.sensor()->properties(); + const CameraSensorProperties::SensorDelays &delays = data->vin_.sensor()->sensorDelays(); + std::unordered_map params = { + { V4L2_CID_ANALOGUE_GAIN, { delays.gainDelay, false } }, + { V4L2_CID_EXPOSURE, { delays.exposureDelay, false } }, + { V4L2_CID_VBLANK, { delays.vblankDelay, false } }, + }; + + data->delayedCtrls_ = + std::make_unique(data->vin_.sensor()->device(), + params); + + std::set streams{ + &data->frames_.rawStream_, + &data->frames_.outputStream_, + }; + + /* + * Connect signals to slots to drive the pipeline. + */ + + /* When internal buffers become available try to queue more jobs. */ + data->frames_.bufferAvailable.connect(data.get(), + &RCar4CameraData::queuePendingRequests); + + /* Connect bufferReady for each video device to a handler. */ + data->vin_.bufferReady().connect(data.get(), + &RCar4CameraData::vinBufferReady); + data->isp_.input_->bufferReady.connect(data.get(), + &RCar4CameraData::inputBufferReady); + data->isp_.param_->bufferReady.connect(data.get(), + &RCar4CameraData::paramBufferReady); + data->isp_.stat_->bufferReady.connect(data.get(), + &RCar4CameraData::statBufferReady); + data->isp_.output_->bufferReady.connect(data.get(), + &RCar4CameraData::outputBufferReady); + + /* Connect IPA signals. */ + data->ipa_->setSensorControls.connect(data.get(), + &RCar4CameraData::setSensorControls); + data->ipa_->paramsComputed.connect(data.get(), + &RCar4CameraData::paramsComputed); + data->ipa_->metadataReady.connect(data.get(), + &RCar4CameraData::metadataReady); + + /* Apply controls at start at exposure. */ + data->vin_.frameStart().connect(data->delayedCtrls_.get(), + &DelayedControls::applyControls); + + /* + * Register the camera. + */ + const std::string &id = data->vin_.sensor()->id(); + std::shared_ptr camera = Camera::create(std::move(data), id, streams); + + registerCamera(std::move(camera)); + + return 0; +} + +bool PipelineHandlerRCar4::match(DeviceEnumerator *enumerator) +{ + DeviceMatch dm("rcar_vin"); + + MediaDevice *media = acquireMediaDevice(enumerator, dm); + if (!media) + return false; + + bool registered = false; + for (const MediaEntity *entity : media->entities()) { + if (entity->name().substr(0, 8) == "rcar_isp" && + entity->name().rfind("core") != std::string::npos) { + /* + * Isolate the unit address that identifies one ISP + * instance. pipeId will look like + * 'rcar_isp fed00000.isp'. + */ + std::string pipeId = entity->name().substr(0, 21); + if (!createCamera(media, pipeId)) + registered = true; + } + } + + return registered; +} + +REGISTER_PIPELINE_HANDLER(PipelineHandlerRCar4, "rcar-gen4") + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rcar-gen4/vin.cpp b/src/libcamera/pipeline/rcar-gen4/vin.cpp new file mode 100644 index 000000000000..76b94a434d84 --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/vin.cpp @@ -0,0 +1,391 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright 2025 Renesas Electronics Co + * Copyright 2025 Niklas Söderlund + * + * Renesas R-Car Gen4 VIN pipeline + */ + +#include "vin.h" + +#include +#include + +#include + +#include +#include +#include +#include + +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/media_device.h" +#include "libcamera/internal/v4l2_subdevice.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(RCar4) + +namespace { + +const std::map mbusCodesToPixelFormat = { + { MEDIA_BUS_FMT_SBGGR10_1X10, formats::SBGGR10 }, + { MEDIA_BUS_FMT_SGBRG10_1X10, formats::SGBRG10 }, + { MEDIA_BUS_FMT_SGRBG10_1X10, formats::SGRBG10 }, + { MEDIA_BUS_FMT_SRGGB10_1X10, formats::SRGGB10 }, +}; + +} /* namespace */ + +/** + * \brief Retrieve the list of supported PixelFormats + * + * Retrieve the list of supported pixel formats by matching the sensor produced + * media bus codes with the formats supported by the VIN unit. + * + * \return The list of supported PixelFormat + */ +std::vector RCarVINDevice::formats() const +{ + if (!sensor_) + return {}; + + std::vector formats; + for (unsigned int code : sensor_->mbusCodes()) { + auto it = mbusCodesToPixelFormat.find(code); + if (it != mbusCodesToPixelFormat.end()) + formats.push_back(it->second); + } + + return formats; +} + +/** + * \brief Retrieve the list of supported size ranges + * \param[in] format The pixel format + * + * Retrieve the list of supported sizes for a particular \a format by matching + * the sensor produced media bus codes formats supported by the VIN unit. + * + * \return A list of supported sizes for the \a format or an empty list + * otherwise + */ +std::vector RCarVINDevice::sizes(const PixelFormat &format) const +{ + int mbusCode = -1; + + if (!sensor_) + return {}; + + std::vector sizes; + for (const auto &iter : mbusCodesToPixelFormat) { + if (iter.second != format) + continue; + + mbusCode = iter.first; + break; + } + + if (mbusCode == -1) + return {}; + + for (const Size &sz : sensor_->sizes(mbusCode)) + sizes.emplace_back(sz); + + return sizes; +} + +int RCarVINDevice::init(const MediaDevice *media, const std::string &pipeId) +{ + const MediaEntity *entity; + const MediaPad *pad, *next; + int ret; + + /* Locate IPS Channel Selector, e.g. rcar_isp fed00000.isp */ + csisp_ = V4L2Subdevice::fromEntityName(media, pipeId); + if (!csisp_) { + LOG(RCar4, Error) << "Failed to find Channel Selector " << pipeId; + return -EINVAL; + } + + /* Use the Channel Selector links to find CSI-2 Rx and Sensor. */ + entity = csisp_->entity(); + pad = entity->getPadByIndex(0); + next = pad->links()[0]->source(); + csi2_ = V4L2Subdevice::fromEntityName(media, next->entity()->name()); + if (!csi2_) { + LOG(RCar4, Error) << "Failed to find CSI-2 Rx entity"; + return -EINVAL; + } + + entity = csi2_->entity(); + pad = entity->getPadByIndex(0); + next = pad->links()[0]->source(); + sensor_ = CameraSensorFactoryBase::create(next->entity()); + if (!sensor_) { + LOG(RCar4, Error) << "Failed to find sensor entity"; + return -EINVAL; + } + + /* + * Make sure the sensor produces at least one format compatible with + * the VIN requirements. + */ + std::vector vinCodes = utils::map_keys(mbusCodesToPixelFormat); + const std::vector &sensorCodes = sensor_->mbusCodes(); + if (!utils::set_overlap(sensorCodes.begin(), sensorCodes.end(), + vinCodes.begin(), vinCodes.end())) { + LOG(RCar4, Error) + << "Sensor " << sensor_->entity()->name() + << " has not format compatible with the VIN"; + return -EINVAL; + } + + /* Use the Channel Selector links to find VIN. */ + entity = csisp_->entity(); + pad = entity->getPadByIndex(1); + next = pad->links()[0]->sink(); + output_ = V4L2VideoDevice::fromEntityName(media, next->entity()->name()); + if (!output_) { + LOG(RCar4, Error) << "Failed to find VIN entity"; + return -EINVAL; + } + + /* Open all devices. */ + ret = csi2_->open(); + if (ret) + return ret; + + ret = csisp_->open(); + if (ret) + return ret; + + ret = output_->open(); + if (ret) + return ret; + + return 0; +} + +int RCarVINDevice::configure(const Size &size, const Transform &transform, + V4L2DeviceFormat *outputFormat) +{ + V4L2SubdeviceFormat sensorFormat; + int ret; + + /* Configure sensor */ + std::vector mbusCodes = utils::map_keys(mbusCodesToPixelFormat); + sensorFormat = getSensorFormat(mbusCodes, size); + ret = sensor_->setFormat(&sensorFormat, transform); + if (ret) + return ret; + + /* Configure CSI-2 */ + ret = csi2_->setFormat(0, &sensorFormat); + if (ret) + return ret; + + if (mbusCodesToPixelFormat.find(sensorFormat.code) == mbusCodesToPixelFormat.end()) + return -EINVAL; + + /* Configure Channel selector. */ + ret = csisp_->setFormat(0, &sensorFormat); + if (ret) + return ret; + + if (mbusCodesToPixelFormat.find(sensorFormat.code) == mbusCodesToPixelFormat.end()) + return -EINVAL; + + /* Configure VIN */ + const auto &itInfo = mbusCodesToPixelFormat.find(sensorFormat.code); + outputFormat->fourcc = output_->toV4L2PixelFormat(itInfo->second); + outputFormat->size = sensorFormat.size; + outputFormat->planesCount = 1; + + ret = output_->setFormat(outputFormat); + if (ret) + return ret; + + LOG(RCar4, Debug) << "VIN output format " << *outputFormat; + + return 0; +} + +StreamConfiguration RCarVINDevice::generateConfiguration(Size size) const +{ + StreamConfiguration cfg; + + /* If no desired size use the sensor resolution. */ + if (size.isNull()) + size = sensor_->resolution(); + + /* Query the sensor static information for closest match. */ + std::vector mbusCodes = utils::map_keys(mbusCodesToPixelFormat); + V4L2SubdeviceFormat sensorFormat = getSensorFormat(mbusCodes, size); + if (!sensorFormat.code) { + LOG(RCar4, Error) << "Sensor does not support mbus code"; + return {}; + } + + cfg.size = sensorFormat.size; + cfg.pixelFormat = mbusCodesToPixelFormat.at(sensorFormat.code); + cfg.bufferCount = kBufferCount; + + /* Get stride and frame size from device. */ + V4L2DeviceFormat fmt; + fmt.fourcc = output_->toV4L2PixelFormat(cfg.pixelFormat); + fmt.size = cfg.size; + + int ret = output_->tryFormat(&fmt); + if (ret) + return {}; + + cfg.stride = fmt.planes[0].bpl; + cfg.frameSize = fmt.planes[0].size; + + return cfg; +} + +/** + * \brief Retrieve the best sensor format for a desired output + * \param[in] mbusCodes The list of acceptable media bus codes + * \param[in] size The desired size + * + * Media bus codes are selected from \a mbusCodes, which lists all acceptable + * codes in decreasing order of preference. Media bus codes supported by the + * sensor but not listed in \a mbusCodes are ignored. If none of the desired + * codes is supported, it returns an error. + * + * \a size indicates the desired size at the output of the sensor. This method + * selects the best media bus code and size supported by the sensor according + * to the following criteria. + * + * - The desired \a size shall fit in the sensor output size to avoid the need + * to up-scale. + * - The aspect ratio of sensor output size shall be as close as possible to + * the sensor's native resolution field of view. + * - The sensor output size shall be as small as possible to lower the required + * bandwidth. + * - The desired \a size shall be supported by one of the media bus code listed + * in \a mbusCodes. + * + * When multiple media bus codes can produce the same size, the code at the + * lowest position in \a mbusCodes is selected. + * + * The returned sensor output format is guaranteed to be acceptable by the + * setFormat() method without any modification. + * + * \return The best sensor output format matching the desired media bus codes + * and size on success, or an empty format otherwise. + */ +V4L2SubdeviceFormat RCarVINDevice::getSensorFormat(const std::vector &mbusCodes, + const Size &size) const +{ + unsigned int desiredArea = size.width * size.height; + unsigned int bestArea = std::numeric_limits::max(); + const Size &resolution = sensor_->resolution(); + float desiredRatio = static_cast(resolution.width) / + resolution.height; + float bestRatio = std::numeric_limits::max(); + Size bestSize; + uint32_t bestCode = 0; + + for (unsigned int code : mbusCodes) { + const auto sizes = sensor_->sizes(code); + if (!sizes.size()) + continue; + + for (const Size &sz : sizes) { + /* No need to check ratios if we have an exact match. */ + if (sz == size) { + bestRatio = 0; + bestArea = 0; + bestSize = sz; + bestCode = code; + break; + } + + if (sz.width < size.width || sz.height < size.height) + continue; + + float ratio = static_cast(sz.width) / sz.height; + /* + * Ratios can differ by small mantissa difference which + * can affect the selection of the sensor output size + * wildly. We are interested in selection of the closest + * size with respect to the desired output size, hence + * comparing it with a single precision digit is enough. + */ + ratio = static_cast(ratio * 10) / 10.0; + float ratioDiff = std::abs(ratio - desiredRatio); + unsigned int area = sz.width * sz.height; + unsigned int areaDiff = area - desiredArea; + + if (ratioDiff > bestRatio) + continue; + + if (ratioDiff < bestRatio || areaDiff < bestArea) { + bestRatio = ratioDiff; + bestArea = areaDiff; + bestSize = sz; + bestCode = code; + } + } + } + + if (bestSize.isNull()) { + LOG(RCar4, Debug) << "No supported format or size found"; + return {}; + } + + V4L2SubdeviceFormat format{}; + format.code = bestCode; + format.size = bestSize; + + return format; +} + +int RCarVINDevice::start() +{ + int ret; + + ret = output_->importBuffers(kBufferCount); + if (ret) { + LOG(RCar4, Error) << "Failed to import VIN buffers"; + return ret; + } + + ret = output_->streamOn(); + if (ret) { + LOG(RCar4, Error) << "Failed to start VIN"; + stop(); + return ret; + } + + ret = output_->setFrameStartEnabled(true); + if (ret) { + LOG(RCar4, Error) << "Failed to enable Frame Start"; + stop(); + return ret; + } + + return 0; +} + +void RCarVINDevice::stop() +{ + output_->setFrameStartEnabled(false); + + output_->streamOff(); + + if (output_->releaseBuffers()) + LOG(RCar4, Error) << "Failed to release VIN buffers"; +} + +int RCarVINDevice::queueBuffer(FrameBuffer *buffer) +{ + return output_->queueBuffer(buffer); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rcar-gen4/vin.h b/src/libcamera/pipeline/rcar-gen4/vin.h new file mode 100644 index 000000000000..5ac97be0e4bc --- /dev/null +++ b/src/libcamera/pipeline/rcar-gen4/vin.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright 2025 Renesas Electronics Co + * Copyright 2025 Niklas Söderlund + * + * Renesas R-Car Gen4 VIN pipeline + */ + +#pragma once + +#include +#include +#include + +#include + +#include "libcamera/internal/v4l2_subdevice.h" +#include "libcamera/internal/v4l2_videodevice.h" + +namespace libcamera { + +class CameraSensor; +class FrameBuffer; +class MediaDevice; +class PixelFormat; +class Request; +class Size; +class SizeRange; +struct StreamConfiguration; +enum class Transform; + +class RCarVINDevice +{ +public: + static constexpr unsigned int kBufferCount = 4; + + std::vector formats() const; + std::vector sizes(const PixelFormat &format) const; + + int init(const MediaDevice *media, const std::string &pipeId); + int configure(const Size &size, const Transform &transform, + V4L2DeviceFormat *outputFormat); + + StreamConfiguration generateConfiguration(Size size) const; + + int start(); + void stop(); + + CameraSensor *sensor() { return sensor_.get(); } + const CameraSensor *sensor() const { return sensor_.get(); } + + int queueBuffer(FrameBuffer *buffer); + + Signal &bufferReady() { return output_->bufferReady; } + Signal &frameStart() { return output_->frameStart; } +private: + V4L2SubdeviceFormat getSensorFormat(const std::vector &mbusCodes, + const Size &size) const; + + std::unique_ptr sensor_; + std::unique_ptr csi2_; + std::unique_ptr csisp_; + std::unique_ptr output_; +}; + +} /* namespace libcamera */