{"id":20714,"url":"https://patchwork.libcamera.org/api/patches/20714/?format=json","web_url":"https://patchwork.libcamera.org/patch/20714/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20240726114715.226468-6-umang.jain@ideasonboard.com>","date":"2024-07-26T11:47:15","name":"[v6,5/5] libcamera: rkisp1: Plumb the ConverterDW100 converter","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"5feba71619ee322ce608eaaa12d37c18d56344f0","submitter":{"id":86,"url":"https://patchwork.libcamera.org/api/people/86/?format=json","name":"Umang Jain","email":"umang.jain@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/20714/mbox/","series":[{"id":4462,"url":"https://patchwork.libcamera.org/api/series/4462/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4462","date":"2024-07-26T11:47:10","name":"libcamera: rkisp1: Plumb the ConverterDW100 converter","version":6,"mbox":"https://patchwork.libcamera.org/series/4462/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/20714/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/20714/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4B70BBDC71\n\tfor ;\n\tFri, 26 Jul 2024 11:47:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AB8F86337E;\n\tFri, 26 Jul 2024 13:47:42 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AE97D6336B\n\tfor ;\n\tFri, 26 Jul 2024 13:47:33 +0200 (CEST)","from localhost.localdomain (unknown\n\t[IPv6:2405:201:2015:f873:55f8:639e:8e9f:12ec])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id B493983F;\n\tFri, 26 Jul 2024 13:46:48 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"t4eEXgtE\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1721994409;\n\tbh=rOebwUw8mptPzhY3dggJ8H7dvqP41iw+t1zbpPXXXh0=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=t4eEXgtEfuz+cLs+56hwHYzOrdLYYDz0KGDRE8q8NVWZJmrLGZD3ua9bmggl/f6wJ\n\txtpB7c4q6OrIsmr3JJ60Y1Kh2w351UE/BDBSX+q/Jsv+E47BSIX1Ruav69KXSHHIU+\n\tcoqnmzvYNFQrFMoZHKe7zZdAw1QIWBLdrzzEDSfc=","From":"Umang Jain ","To":"libcamera-devel@lists.libcamera.org","Cc":"Umang Jain ","Subject":"[PATCH v6 5/5] libcamera: rkisp1: Plumb the ConverterDW100 converter","Date":"Fri, 26 Jul 2024 17:17:15 +0530","Message-ID":"<20240726114715.226468-6-umang.jain@ideasonboard.com>","X-Mailer":"git-send-email 2.45.0","In-Reply-To":"<20240726114715.226468-1-umang.jain@ideasonboard.com>","References":"<20240726114715.226468-1-umang.jain@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Plumb the ConverterDW100 converter in the rkisp1 pipeline handler.\nIf the dewarper is found, it is instantiated and buffers are exported\nfrom it, instead of RkISP1Path. Internal buffers are allocated for the\nRkISP1Path in case where dewarper is going to be used.\n\nThe RKISP1 pipeline handler now supports scaler crop control through\nConverterDW100. Register the ScalerCrop control for the cameras created\nin the RKISP1 pipeline handler.\n\nSigned-off-by: Umang Jain \n---\n src/libcamera/pipeline/rkisp1/rkisp1.cpp | 154 ++++++++++++++++++++++-\n 1 file changed, 150 insertions(+), 4 deletions(-)","diff":"diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\nindex 25f2cc97..32ec0fdf 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n@@ -8,9 +8,11 @@\n #include \n #include \n #include \n+#include \n #include \n #include \n #include \n+#include \n \n #include \n #include \n@@ -33,6 +35,7 @@\n \n #include \"libcamera/internal/camera.h\"\n #include \"libcamera/internal/camera_sensor.h\"\n+#include \"libcamera/internal/converter/converter_dw100.h\"\n #include \"libcamera/internal/delayed_controls.h\"\n #include \"libcamera/internal/device_enumerator.h\"\n #include \"libcamera/internal/framebuffer.h\"\n@@ -62,6 +65,8 @@ struct RkISP1FrameInfo {\n \n \tbool paramDequeued;\n \tbool metadataProcessed;\n+\n+\tstd::optional scalerCrop;\n };\n \n class RkISP1Frames\n@@ -181,6 +186,7 @@ private:\n \tvoid bufferReady(FrameBuffer *buffer);\n \tvoid paramReady(FrameBuffer *buffer);\n \tvoid statReady(FrameBuffer *buffer);\n+\tvoid dewarpBufferReady(FrameBuffer *buffer);\n \tvoid frameStart(uint32_t sequence);\n \n \tint allocateBuffers(Camera *camera);\n@@ -200,6 +206,13 @@ private:\n \tRkISP1MainPath mainPath_;\n \tRkISP1SelfPath selfPath_;\n \n+\tstd::unique_ptr dewarper_;\n+\tbool useDewarper_;\n+\n+\t/* Internal buffers used when dewarper is being used */\n+\tstd::vector> mainPathBuffers_;\n+\tstd::queue availableMainPathBuffers_;\n+\n \tstd::vector> paramBuffers_;\n \tstd::vector> statBuffers_;\n \tstd::queue availableParamBuffers_;\n@@ -222,6 +235,8 @@ RkISP1FrameInfo *RkISP1Frames::create(const RkISP1CameraData *data, Request *req\n \n \tFrameBuffer *paramBuffer = nullptr;\n \tFrameBuffer *statBuffer = nullptr;\n+\tFrameBuffer *mainPathBuffer = nullptr;\n+\tFrameBuffer *selfPathBuffer = nullptr;\n \n \tif (!isRaw) {\n \t\tif (pipe_->availableParamBuffers_.empty()) {\n@@ -239,10 +254,16 @@ RkISP1FrameInfo *RkISP1Frames::create(const RkISP1CameraData *data, Request *req\n \n \t\tstatBuffer = pipe_->availableStatBuffers_.front();\n \t\tpipe_->availableStatBuffers_.pop();\n+\n+\t\tif (pipe_->useDewarper_) {\n+\t\t\tmainPathBuffer = pipe_->availableMainPathBuffers_.front();\n+\t\t\tpipe_->availableMainPathBuffers_.pop();\n+\t\t}\n \t}\n \n-\tFrameBuffer *mainPathBuffer = request->findBuffer(&data->mainPathStream_);\n-\tFrameBuffer *selfPathBuffer = request->findBuffer(&data->selfPathStream_);\n+\tif (!mainPathBuffer)\n+\t\tmainPathBuffer = request->findBuffer(&data->mainPathStream_);\n+\tselfPathBuffer = request->findBuffer(&data->selfPathStream_);\n \n \tRkISP1FrameInfo *info = new RkISP1FrameInfo;\n \n@@ -268,6 +289,7 @@ int RkISP1Frames::destroy(unsigned int frame)\n \n \tpipe_->availableParamBuffers_.push(info->paramBuffer);\n \tpipe_->availableStatBuffers_.push(info->statBuffer);\n+\tpipe_->availableMainPathBuffers_.push(info->mainPathBuffer);\n \n \tframeInfo_.erase(info->frame);\n \n@@ -283,6 +305,7 @@ void RkISP1Frames::clear()\n \n \t\tpipe_->availableParamBuffers_.push(info->paramBuffer);\n \t\tpipe_->availableStatBuffers_.push(info->statBuffer);\n+\t\tpipe_->availableMainPathBuffers_.push(info->mainPathBuffer);\n \n \t\tdelete info;\n \t}\n@@ -607,7 +630,7 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate()\n */\n \n PipelineHandlerRkISP1::PipelineHandlerRkISP1(CameraManager *manager)\n-\t: PipelineHandler(manager), hasSelfPath_(true)\n+\t: PipelineHandler(manager), hasSelfPath_(true), useDewarper_(false)\n {\n }\n \n@@ -785,12 +808,19 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n \t\t<< \" crop \" << rect;\n \n \tstd::map streamConfig;\n+\tstd::vector> outputCfgs;\n \n \tfor (const StreamConfiguration &cfg : *config) {\n \t\tif (cfg.stream() == &data->mainPathStream_) {\n \t\t\tret = mainPath_.configure(cfg, format);\n \t\t\tstreamConfig[0] = IPAStream(cfg.pixelFormat,\n \t\t\t\t\t\t cfg.size);\n+\t\t\t/* Configure dewarp */\n+\t\t\tif (dewarper_ && !isRaw_) {\n+\t\t\t\toutputCfgs.push_back(const_cast(cfg));\n+\t\t\t\tret = dewarper_->configure(cfg, outputCfgs);\n+\t\t\t\tuseDewarper_ = ret ? false : true;\n+\t\t\t}\n \t\t} else if (hasSelfPath_) {\n \t\t\tret = selfPath_.configure(cfg, format);\n \t\t\tstreamConfig[1] = IPAStream(cfg.pixelFormat,\n@@ -839,6 +869,9 @@ int PipelineHandlerRkISP1::exportFrameBuffers([[maybe_unused]] Camera *camera, S\n \tRkISP1CameraData *data = cameraData(camera);\n \tunsigned int count = stream->configuration().bufferCount;\n \n+\tif (useDewarper_)\n+\t\treturn dewarper_->exportBuffers(&data->mainPathStream_, count, buffers);\n+\n \tif (stream == &data->mainPathStream_)\n \t\treturn mainPath_.exportBuffers(count, buffers);\n \telse if (hasSelfPath_ && stream == &data->selfPathStream_)\n@@ -866,6 +899,16 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera)\n \t\tret = stat_->allocateBuffers(maxCount, &statBuffers_);\n \t\tif (ret < 0)\n \t\t\tgoto error;\n+\n+\t\t/* If the dewarper is being used, allocate internal buffers for ISP */\n+\t\tif (useDewarper_) {\n+\t\t\tret = mainPath_.exportBuffers(maxCount, &mainPathBuffers_);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto error;\n+\n+\t\t\tfor (std::unique_ptr &buffer : mainPathBuffers_)\n+\t\t\t\tavailableMainPathBuffers_.push(buffer.get());\n+\t\t}\n \t}\n \n \tfor (std::unique_ptr &buffer : paramBuffers_) {\n@@ -889,6 +932,7 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera)\n error:\n \tparamBuffers_.clear();\n \tstatBuffers_.clear();\n+\tmainPathBuffers_.clear();\n \n \treturn ret;\n }\n@@ -903,8 +947,12 @@ int PipelineHandlerRkISP1::freeBuffers(Camera *camera)\n \twhile (!availableParamBuffers_.empty())\n \t\tavailableParamBuffers_.pop();\n \n+\twhile (!availableMainPathBuffers_.empty())\n+\t\tavailableMainPathBuffers_.pop();\n+\n \tparamBuffers_.clear();\n \tstatBuffers_.clear();\n+\tmainPathBuffers_.clear();\n \n \tstd::vector ids;\n \tfor (IPABuffer &ipabuf : data->ipaBuffers_)\n@@ -961,6 +1009,14 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL\n \t\t\t\t<< \"Failed to start statistics \" << camera->id();\n \t\t\treturn ret;\n \t\t}\n+\n+\t\tif (useDewarper_) {\n+\t\t\tret = dewarper_->start();\n+\t\t\tif (ret) {\n+\t\t\t\tLOG(RkISP1, Error) << \"Failed to start dewarper\";\n+\t\t\t\treturn ret;\n+\t\t\t}\n+\t\t}\n \t}\n \n \tif (data->mainPath_->isEnabled()) {\n@@ -1015,6 +1071,9 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera)\n \t\tif (ret)\n \t\t\tLOG(RkISP1, Warning)\n \t\t\t\t<< \"Failed to stop parameters for \" << camera->id();\n+\n+\t\tif (useDewarper_)\n+\t\t\tdewarper_->stop();\n \t}\n \n \tASSERT(data->queuedRequests_.empty());\n@@ -1045,6 +1104,25 @@ int PipelineHandlerRkISP1::queueRequestDevice(Camera *camera, Request *request)\n \t\t\t\t\t info->paramBuffer->cookie());\n \t}\n \n+\tconst auto &crop = request->controls().get(controls::ScalerCrop);\n+\tif (crop && useDewarper_) {\n+\t\tRectangle appliedRect = crop.value();\n+\t\tint ret = dewarper_->setInputCrop(&data->mainPathStream_, &appliedRect);\n+\t\tif (!ret) {\n+\t\t\tif (appliedRect != crop.value()) {\n+\t\t\t\t/*\n+\t\t\t\t * \\todo How to handle these case?\n+\t\t\t\t * Do we aim for pixel perfect set rectangles?\n+\t\t\t\t */\n+\t\t\t\tLOG(RkISP1, Warning)\n+\t\t\t\t\t<< \"Applied rectangle \" << appliedRect.toString()\n+\t\t\t\t\t<< \" differs from requested \" << crop.value().toString();\n+\t\t\t}\n+\n+\t\t\tinfo->scalerCrop = appliedRect;\n+\t\t}\n+\t}\n+\n \tdata->frame_++;\n \n \treturn 0;\n@@ -1110,6 +1188,12 @@ int PipelineHandlerRkISP1::updateControls(RkISP1CameraData *data)\n {\n \tControlInfoMap::Map rkisp1Controls;\n \n+\tif (dewarper_) {\n+\t\tauto [minCrop, maxCrop] = dewarper_->inputCropBounds(&data->mainPathStream_);\n+\n+\t\trkisp1Controls[&controls::ScalerCrop] = ControlInfo(minCrop, maxCrop, maxCrop);\n+\t}\n+\n \t/* Add the IPA registered controls to list of camera controls. */\n \tfor (const auto &ipaControl : data->ipaControls_)\n \t\trkisp1Controls[ipaControl.first] = ipaControl.second;\n@@ -1173,6 +1257,7 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor)\n \n bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator)\n {\n+\tstd::shared_ptr dwpMediaDevice;\n \tconst MediaPad *pad;\n \n \tDeviceMatch dm(\"rkisp1\");\n@@ -1250,6 +1335,26 @@ bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator)\n \tstat_->bufferReady.connect(this, &PipelineHandlerRkISP1::statReady);\n \tparam_->bufferReady.connect(this, &PipelineHandlerRkISP1::paramReady);\n \n+\t/* If dewarper is present, create its instance. */\n+\tDeviceMatch dwp(\"dw100\");\n+\tdwp.add(\"dw100-source\");\n+\tdwp.add(\"dw100-sink\");\n+\n+\tdwpMediaDevice = enumerator->search(dwp);\n+\tif (dwpMediaDevice) {\n+\t\tdewarper_ = std::make_unique(std::move(dwpMediaDevice));\n+\t\tif (dewarper_->isValid()) {\n+\t\t\tdewarper_->outputBufferReady.connect(\n+\t\t\t\tthis, &PipelineHandlerRkISP1::dewarpBufferReady);\n+\n+\t\t\tLOG(RkISP1, Info) << \"Using DW100 dewarper \" << dewarper_->deviceNode();\n+\t\t} else {\n+\t\t\tLOG(RkISP1, Debug) << \"Found DW100 dewarper \" << dewarper_->deviceNode()\n+\t\t\t\t\t << \" but invalid\";\n+\t\t\tdewarper_.reset();\n+\t\t}\n+\t}\n+\n \t/*\n \t * Enumerate all sensors connected to the ISP and create one\n \t * camera instance for each of them.\n@@ -1296,7 +1401,7 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer)\n \t\treturn;\n \n \tconst FrameMetadata &metadata = buffer->metadata();\n-\tRequest *request = buffer->request();\n+\tRequest *request = info->request;\n \n \tif (metadata.status != FrameMetadata::FrameCancelled) {\n \t\t/*\n@@ -1313,11 +1418,52 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer)\n \t\t\t\tdata->delayedCtrls_->get(metadata.sequence);\n \t\t\tdata->ipa_->processStatsBuffer(info->frame, 0, ctrls);\n \t\t}\n+\n \t} else {\n \t\tif (isRaw_)\n \t\t\tinfo->metadataProcessed = true;\n \t}\n \n+\tif (useDewarper_) {\n+\t\t/* Do not queue cancelled frames to dewarper. */\n+\t\tif (metadata.status == FrameMetadata::FrameCancelled) {\n+\t\t\tfor (auto it : request->buffers())\n+\t\t\t\tcompleteBuffer(request, it.second);\n+\n+\t\t\ttryCompleteRequest(info);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Queue input and output buffers to the dewarper. The output\n+\t\t * buffers for the dewarper are the buffers of the request, supplied\n+\t\t * by the application.\n+\t\t */\n+\t\tint ret = dewarper_->queueBuffers(buffer, request->buffers());\n+\t\tif (ret < 0)\n+\t\t\tLOG(RkISP1, Error) << \"Cannot queue buffers to dewarper: \"\n+\t\t\t\t\t << strerror(-ret);\n+\n+\t\treturn;\n+\t}\n+\n+\tcompleteBuffer(request, buffer);\n+\ttryCompleteRequest(info);\n+}\n+\n+void PipelineHandlerRkISP1::dewarpBufferReady(FrameBuffer *buffer)\n+{\n+\tASSERT(activeCamera_);\n+\tRkISP1CameraData *data = cameraData(activeCamera_);\n+\tRequest *request = buffer->request();\n+\n+\tRkISP1FrameInfo *info = data->frameInfo_.find(buffer->request());\n+\tif (!info)\n+\t\treturn;\n+\n+\tif (info->scalerCrop)\n+\t\trequest->metadata().set(controls::ScalerCrop, info->scalerCrop.value());\n+\n \tcompleteBuffer(request, buffer);\n \ttryCompleteRequest(info);\n }\n","prefixes":["v6","5/5"]}