{"id":20069,"url":"https://patchwork.libcamera.org/api/patches/20069/?format=json","web_url":"https://patchwork.libcamera.org/patch/20069/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20240519081539.29832-5-umang.jain@ideasonboard.com>","date":"2024-05-19T08:15:39","name":"[4/4] libcamera: rkisp1: Plumb through RkISP1Dewarper","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"10710e57c32391b029c8ff9b03f9ec57b699b103","submitter":{"id":86,"url":"https://patchwork.libcamera.org/api/people/86/?format=json","name":"Umang Jain","email":"umang.jain@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/20069/mbox/","series":[{"id":4314,"url":"https://patchwork.libcamera.org/api/series/4314/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4314","date":"2024-05-19T08:15:35","name":"libcamera: rkisp1: Plumb i.MX8MP DW100 dewarper","version":1,"mbox":"https://patchwork.libcamera.org/series/4314/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/20069/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/20069/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id CF168BD78E\n\tfor ;\n\tSun, 19 May 2024 08:16:00 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 28FE46348C;\n\tSun, 19 May 2024 10:16:00 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4FA9963480\n\tfor ;\n\tSun, 19 May 2024 10:15:54 +0200 (CEST)","from fedora.local (unknown\n\t[IPv6:2405:201:2015:f873:9278:2c85:fd02:c5f5])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id D01615A4;\n\tSun, 19 May 2024 10:15:42 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"YyxL0pIy\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1716106543;\n\tbh=GYw8e5c6SwEqPSRAU7bykC6UN5d540tMj8/ObNyExks=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=YyxL0pIy8nGMuldiVmjHOLCv3qtqb42FmbPS2K/JhSHq+BeK+0EJrCUbcr8DZUCbZ\n\tqFmg9dpCjNj/oO0SutrwK2nAyxz0bhhltq1i319GVGvgw25A1xVqmugNAB+1EBGbgw\n\t2pwq6KMe1VLB2FxxXplv/or5bxkaaaHy6hypSJNY=","From":"Umang Jain ","To":"libcamera-devel@lists.libcamera.org","Cc":"Umang Jain ","Subject":"[PATCH 4/4] libcamera: rkisp1: Plumb through RkISP1Dewarper","Date":"Sun, 19 May 2024 13:45:39 +0530","Message-ID":"<20240519081539.29832-5-umang.jain@ideasonboard.com>","X-Mailer":"git-send-email 2.44.0","In-Reply-To":"<20240519081539.29832-1-umang.jain@ideasonboard.com>","References":"<20240519081539.29832-1-umang.jain@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Plumb the DW100 RkISP1Dewarper in the rkisp1 pipeline handler.\nIf the dewarper is found, it is instantiated and buffers are exported\nfrom it, instead of RkISP1Path. Internal buffers are allocated for the\nRkISP1Path in case where dewarper is going to be used.\n\nThe RKISP1 pipeline handler now supports scaler crop control through\ndewarper. Register the ScalerCrop control for the cameras created in the\nRKISP1 pipeline handler.\n\nSigned-off-by: Umang Jain \n---\n src/libcamera/pipeline/rkisp1/rkisp1.cpp | 134 +++++++++++++++++-\n src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 12 +-\n src/libcamera/pipeline/rkisp1/rkisp1_path.h | 14 ++\n 3 files changed, 152 insertions(+), 8 deletions(-)","diff":"diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\nindex bfc44239..8bb014a7 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n@@ -8,9 +8,11 @@\n #include \n #include \n #include \n+#include \n #include \n #include \n #include \n+#include \n \n #include \n #include \n@@ -42,6 +44,7 @@\n #include \"libcamera/internal/v4l2_subdevice.h\"\n #include \"libcamera/internal/v4l2_videodevice.h\"\n \n+#include \"rkisp1_dewarper.h\"\n #include \"rkisp1_path.h\"\n \n namespace libcamera {\n@@ -62,6 +65,8 @@ struct RkISP1FrameInfo {\n \n \tbool paramDequeued;\n \tbool metadataProcessed;\n+\n+\tstd::optional scalerCrop;\n };\n \n class RkISP1Frames\n@@ -165,6 +170,8 @@ public:\n \n private:\n \tstatic constexpr Size kRkISP1PreviewSize = { 1920, 1080 };\n+\tstatic constexpr Size kMinDewarpSize = { 176, 144 };\n+\tstatic constexpr Size kMaxDewarpSize = { 4096, 3072 };\n \n \tRkISP1CameraData *cameraData(Camera *camera)\n \t{\n@@ -181,6 +188,7 @@ private:\n \tvoid bufferReady(FrameBuffer *buffer);\n \tvoid paramReady(FrameBuffer *buffer);\n \tvoid statReady(FrameBuffer *buffer);\n+\tvoid dewarpBufferReady(FrameBuffer *buffer);\n \tvoid frameStart(uint32_t sequence);\n \n \tint allocateBuffers(Camera *camera);\n@@ -200,6 +208,13 @@ private:\n \tRkISP1MainPath mainPath_;\n \tRkISP1SelfPath selfPath_;\n \n+\tstd::unique_ptr dewarper_;\n+\tstd::map dewarpOutputBufs_;\n+\n+\t/* Internal buffers used when dewarper is being used */\n+\tstd::vector> mainPathBuffers_;\n+\tstd::queue availableMainPathBuffers_;\n+\n \tstd::vector> paramBuffers_;\n \tstd::vector> statBuffers_;\n \tstd::queue availableParamBuffers_;\n@@ -222,6 +237,8 @@ RkISP1FrameInfo *RkISP1Frames::create(const RkISP1CameraData *data, Request *req\n \n \tFrameBuffer *paramBuffer = nullptr;\n \tFrameBuffer *statBuffer = nullptr;\n+\tFrameBuffer *mainPathBuffer = nullptr;\n+\tFrameBuffer *selfPathBuffer = nullptr;\n \n \tif (!isRaw) {\n \t\tif (pipe_->availableParamBuffers_.empty()) {\n@@ -239,10 +256,16 @@ RkISP1FrameInfo *RkISP1Frames::create(const RkISP1CameraData *data, Request *req\n \n \t\tstatBuffer = pipe_->availableStatBuffers_.front();\n \t\tpipe_->availableStatBuffers_.pop();\n+\n+\t\tif (pipe_->dewarper_) {\n+\t\t\tmainPathBuffer = pipe_->availableMainPathBuffers_.front();\n+\t\t\tpipe_->availableMainPathBuffers_.pop();\n+\t\t}\n \t}\n \n-\tFrameBuffer *mainPathBuffer = request->findBuffer(&data->mainPathStream_);\n-\tFrameBuffer *selfPathBuffer = request->findBuffer(&data->selfPathStream_);\n+\tif (!mainPathBuffer)\n+\t\tmainPathBuffer = request->findBuffer(&data->mainPathStream_);\n+\tselfPathBuffer = request->findBuffer(&data->selfPathStream_);\n \n \tRkISP1FrameInfo *info = new RkISP1FrameInfo;\n \n@@ -268,6 +291,7 @@ int RkISP1Frames::destroy(unsigned int frame)\n \n \tpipe_->availableParamBuffers_.push(info->paramBuffer);\n \tpipe_->availableStatBuffers_.push(info->statBuffer);\n+\tpipe_->availableMainPathBuffers_.push(info->mainPathBuffer);\n \n \tframeInfo_.erase(info->frame);\n \n@@ -283,6 +307,7 @@ void RkISP1Frames::clear()\n \n \t\tpipe_->availableParamBuffers_.push(info->paramBuffer);\n \t\tpipe_->availableStatBuffers_.push(info->statBuffer);\n+\t\tpipe_->availableMainPathBuffers_.push(info->mainPathBuffer);\n \n \t\tdelete info;\n \t}\n@@ -785,12 +810,23 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n \t\t<< \" crop \" << rect;\n \n \tstd::map streamConfig;\n+\tstd::vector> outputCfgs;\n \n \tfor (const StreamConfiguration &cfg : *config) {\n \t\tif (cfg.stream() == &data->mainPathStream_) {\n \t\t\tret = mainPath_.configure(cfg, format);\n \t\t\tstreamConfig[0] = IPAStream(cfg.pixelFormat,\n \t\t\t\t\t\t cfg.size);\n+\t\t\t/* Configure dewarp */\n+\t\t\tif (dewarper_) {\n+\t\t\t\t/*\n+\t\t\t\t * \\todo Converter::configure() API should be changed\n+\t\t\t\t * to use std::vector> ?\n+\t\t\t\t */\n+\t\t\t\toutputCfgs.push_back(const_cast(cfg));\n+\t\t\t\tdewarper_->configure(cfg, outputCfgs);\n+\t\t\t}\n+\n \t\t} else if (hasSelfPath_) {\n \t\t\tret = selfPath_.configure(cfg, format);\n \t\t\tstreamConfig[1] = IPAStream(cfg.pixelFormat,\n@@ -839,6 +875,9 @@ int PipelineHandlerRkISP1::exportFrameBuffers([[maybe_unused]] Camera *camera, S\n \tRkISP1CameraData *data = cameraData(camera);\n \tunsigned int count = stream->configuration().bufferCount;\n \n+\tif (!isRaw_ && dewarper_)\n+\t\treturn dewarper_->exportBuffers(0, count, buffers);\n+\n \tif (stream == &data->mainPathStream_)\n \t\treturn mainPath_.exportBuffers(count, buffers);\n \telse if (hasSelfPath_ && stream == &data->selfPathStream_)\n@@ -866,6 +905,16 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera)\n \t\tret = stat_->allocateBuffers(maxCount, &statBuffers_);\n \t\tif (ret < 0)\n \t\t\tgoto error;\n+\n+\t\t/* If the dewarper is being used, allocate internal buffers for ISP */\n+\t\tif (dewarper_) {\n+\t\t\tret = mainPath_.allocateBuffers(maxCount, &mainPathBuffers_);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto error;\n+\n+\t\t\tfor (std::unique_ptr &buffer : mainPathBuffers_)\n+\t\t\t\tavailableMainPathBuffers_.push(buffer.get());\n+\t\t}\n \t}\n \n \tfor (std::unique_ptr &buffer : paramBuffers_) {\n@@ -889,6 +938,7 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera)\n error:\n \tparamBuffers_.clear();\n \tstatBuffers_.clear();\n+\tmainPathBuffers_.clear();\n \n \treturn ret;\n }\n@@ -903,8 +953,12 @@ int PipelineHandlerRkISP1::freeBuffers(Camera *camera)\n \twhile (!availableParamBuffers_.empty())\n \t\tavailableParamBuffers_.pop();\n \n+\twhile (!availableMainPathBuffers_.empty())\n+\t\tavailableMainPathBuffers_.pop();\n+\n \tparamBuffers_.clear();\n \tstatBuffers_.clear();\n+\tmainPathBuffers_.clear();\n \n \tstd::vector ids;\n \tfor (IPABuffer &ipabuf : data->ipaBuffers_)\n@@ -919,6 +973,9 @@ int PipelineHandlerRkISP1::freeBuffers(Camera *camera)\n \tif (stat_->releaseBuffers())\n \t\tLOG(RkISP1, Error) << \"Failed to release stat buffers\";\n \n+\tif (mainPath_.releaseBuffers())\n+\t\tLOG(RkISP1, Error) << \"Failed to release main path buffers\";\n+\n \treturn 0;\n }\n \n@@ -961,6 +1018,14 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL\n \t\t\t\t<< \"Failed to start statistics \" << camera->id();\n \t\t\treturn ret;\n \t\t}\n+\n+\t\tif (dewarper_) {\n+\t\t\tret = dewarper_->start();\n+\t\t\tif (ret) {\n+\t\t\t\tLOG(RkISP1, Error) << \"Failed to start dewarper\";\n+\t\t\t\treturn ret;\n+\t\t\t}\n+\t\t}\n \t}\n \n \tif (data->mainPath_->isEnabled()) {\n@@ -1015,6 +1080,9 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera)\n \t\tif (ret)\n \t\t\tLOG(RkISP1, Warning)\n \t\t\t\t<< \"Failed to stop parameters for \" << camera->id();\n+\n+\t\tif (dewarper_)\n+\t\t\tdewarper_->stop();\n \t}\n \n \tASSERT(data->queuedRequests_.empty());\n@@ -1045,6 +1113,13 @@ int PipelineHandlerRkISP1::queueRequestDevice(Camera *camera, Request *request)\n \t\t\t\t\t info->paramBuffer->cookie());\n \t}\n \n+\tconst auto &crop = request->controls().get(controls::ScalerCrop);\n+\tif (crop && !isRaw_) {\n+\t\tint ret = dewarper_->setScalerCrop(0, crop.value());\n+\t\tif (!ret)\n+\t\t\tinfo->scalerCrop = crop;\n+\t}\n+\n \tdata->frame_++;\n \n \treturn 0;\n@@ -1110,6 +1185,13 @@ int PipelineHandlerRkISP1::updateControls(RkISP1CameraData *data)\n {\n \tControlInfoMap::Map rkisp1Controls;\n \n+\tif (dewarper_) {\n+\t\tRectangle maxCrop(kMaxDewarpSize);\n+\t\tRectangle minCrop = kMinDewarpSize.centeredTo(maxCrop.center());\n+\n+\t\trkisp1Controls[&controls::ScalerCrop] = ControlInfo(minCrop, maxCrop, maxCrop);\n+\t}\n+\n \t/* Add the IPA registered controls to list of camera controls. */\n \tfor (const auto &ipaControl : data->ipaControls_)\n \t\trkisp1Controls[ipaControl.first] = ipaControl.second;\n@@ -1173,6 +1255,7 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor)\n \n bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator)\n {\n+\tstd::shared_ptr dwpMediaDevice;\n \tconst MediaPad *pad;\n \n \tDeviceMatch dm(\"rkisp1\");\n@@ -1237,6 +1320,26 @@ bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator)\n \tstat_->bufferReady.connect(this, &PipelineHandlerRkISP1::statReady);\n \tparam_->bufferReady.connect(this, &PipelineHandlerRkISP1::paramReady);\n \n+\t/* If dewarper is present, create its instance. */\n+\tDeviceMatch dwp(\"dw100\");\n+\tdwp.add(\"dw100-source\");\n+\tdwp.add(\"dw100-sink\");\n+\n+\tdwpMediaDevice = enumerator->search(dwp);\n+\tif (dwpMediaDevice) {\n+\t\tdewarper_ = std::make_unique(std::move(dwpMediaDevice));\n+\t\tif (dewarper_->isValid()) {\n+\t\t\tdewarper_->outputBufferReady.connect(\n+\t\t\t\tthis, &PipelineHandlerRkISP1::dewarpBufferReady);\n+\n+\t\t\tLOG(RkISP1, Info) << \"Using DW100 dewarper \" << dewarper_->deviceNode();\n+\t\t} else {\n+\t\t\tLOG(RkISP1, Debug) << \"Found DW100 dewarper \" << dewarper_->deviceNode()\n+\t\t\t\t\t << \" but invalid\";\n+\t\t\tdewarper_.reset();\n+\t\t}\n+\t}\n+\n \t/*\n \t * Enumerate all sensors connected to the ISP and create one\n \t * camera instance for each of them.\n@@ -1283,7 +1386,7 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer)\n \t\treturn;\n \n \tconst FrameMetadata &metadata = buffer->metadata();\n-\tRequest *request = buffer->request();\n+\tRequest *request = info->request;\n \n \tif (metadata.status != FrameMetadata::FrameCancelled) {\n \t\t/*\n@@ -1300,11 +1403,36 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer)\n \t\t\t\tdata->delayedCtrls_->get(metadata.sequence);\n \t\t\tdata->ipa_->processStatsBuffer(info->frame, 0, ctrls);\n \t\t}\n+\n \t} else {\n \t\tif (isRaw_)\n \t\t\tinfo->metadataProcessed = true;\n \t}\n \n+\tif (dewarper_) {\n+\t\tdewarpOutputBufs_[0] = request->findBuffer(&data->mainPathStream_);\n+\n+\t\tdewarper_->queueBuffers(buffer, dewarpOutputBufs_);\n+\t\treturn;\n+\t}\n+\n+\tcompleteBuffer(request, buffer);\n+\ttryCompleteRequest(info);\n+}\n+\n+void PipelineHandlerRkISP1::dewarpBufferReady(FrameBuffer *buffer)\n+{\n+\tASSERT(activeCamera_);\n+\tRkISP1CameraData *data = cameraData(activeCamera_);\n+\tRequest *request = buffer->request();\n+\n+\tRkISP1FrameInfo *info = data->frameInfo_.find(buffer->request());\n+\tif (!info)\n+\t\treturn;\n+\n+\tif (info->scalerCrop)\n+\t\trequest->metadata().set(controls::ScalerCrop, info->scalerCrop.value());\n+\n \tcompleteBuffer(request, buffer);\n \ttryCompleteRequest(info);\n }\ndiff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\nindex c49017d1..c96ec1d6 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n@@ -56,7 +56,7 @@ const std::map formatToMediaBus = {\n \n RkISP1Path::RkISP1Path(const char *name, const Span &formats,\n \t\t const Size &minResolution, const Size &maxResolution)\n-\t: name_(name), running_(false), formats_(formats),\n+\t: name_(name), running_(false), internalBufs_(false), formats_(formats),\n \t minResolution_(minResolution), maxResolution_(maxResolution),\n \t link_(nullptr)\n {\n@@ -402,10 +402,12 @@ int RkISP1Path::start()\n \tif (running_)\n \t\treturn -EBUSY;\n \n-\t/* \\todo Make buffer count user configurable. */\n-\tret = video_->importBuffers(RKISP1_BUFFER_COUNT);\n-\tif (ret)\n-\t\treturn ret;\n+\tif (!internalBufs_) {\n+\t\t/* \\todo Make buffer count user configurable. */\n+\t\tret = video_->importBuffers(RKISP1_BUFFER_COUNT);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n \n \tret = video_->streamOn();\n \tif (ret) {\ndiff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.h b/src/libcamera/pipeline/rkisp1/rkisp1_path.h\nindex 08edefec..b7fa82d0 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.h\n@@ -55,6 +55,19 @@ public:\n \t\treturn video_->exportBuffers(bufferCount, buffers);\n \t}\n \n+\tint allocateBuffers(unsigned int bufferCount,\n+\t\t\t std::vector> *buffers)\n+\t{\n+\t\tinternalBufs_ = true;\n+\t\treturn video_->allocateBuffers(bufferCount, buffers);\n+\t}\n+\n+\tint releaseBuffers()\n+\t{\n+\t\tASSERT(internalBufs_);\n+\t\treturn video_->releaseBuffers();\n+\t}\n+\n \tint start();\n \tvoid stop();\n \n@@ -68,6 +81,7 @@ private:\n \n \tconst char *name_;\n \tbool running_;\n+\tbool internalBufs_;\n \n \tconst Span formats_;\n \tstd::set streamFormats_;\n","prefixes":["4/4"]}