[{"id":26899,"web_url":"https://patchwork.libcamera.org/comment/26899/","msgid":"<20230419055508.GP30837@pendragon.ideasonboard.com>","date":"2023-04-19T05:55:08","subject":"Re: [libcamera-devel] [PATCH 06/11] pipeline: ipu3: Support sensors\n\tusing only CIO2","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Dan,\n\nThank you for the patch.\n\nOn Sat, Mar 18, 2023 at 11:40:09PM +0000, Daniel Scally via libcamera-devel wrote:\n> Some sensors that need to be supported by the IPU3 pipeline are not\n> supported by the Imgu's kernel driver, but can be used by simply\n> treating the CIO2's capture device as the source of image data. In\n> that case much of the processing in the IPU3 pipeline needs to be\n> disabled. Guard it behind checks that query the CIO2Device::needsImgu\n> field.\n\nConceptually speaking, I don't think this should be gated by\nCIO2Device::needsImgu(). It's not really the CIO2 itself that needs the\nImgU. Could we store that flag in the IPU3CameraData class instead ?\nMoving the code that checks if the sensor format is compatible with the\nImgU to the IPU3CameraData class would also be nice. All decisions\nrelated to whether or not to use the ImgU in the pipeline for a camera\nwould then be done in the top-level pipeline handler code instead of the\nindividual components. I would also then rename the needsImgU field to\nuseImgU.\n\n> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>\n> ---\n>  src/libcamera/pipeline/ipu3/ipu3.cpp | 214 +++++++++++++++------------\n>  1 file changed, 118 insertions(+), 96 deletions(-)\n> \n> diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp\n> index 3077c6bb..cc7da299 100644\n> --- a/src/libcamera/pipeline/ipu3/ipu3.cpp\n> +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp\n> @@ -489,6 +489,34 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c)\n>  \tV4L2DeviceFormat outputFormat;\n>  \tint ret;\n>  \n> +\t/*\n> +\t * Pass the requested stream size to the CIO2 unit and get back the\n> +\t * adjusted format to be propagated to the ImgU output devices.\n> +\t */\n> +\tconst Size &sensorSize = config->cio2Format().size;\n> +\tV4L2DeviceFormat cio2Format;\n> +\tret = cio2->configure(sensorSize, config->combinedTransform_, &cio2Format);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tIPACameraSensorInfo sensorInfo;\n> +\tcio2->sensor()->sensorInfo(&sensorInfo);\n> +\tdata->cropRegion_ = sensorInfo.analogCrop;\n> +\n> +\t/*\n> +\t * If the ImgU gets configured, its driver seems to expect that\n> +\t * buffers will be queued to its outputs, as otherwise the next\n> +\t * capture session that uses the ImgU fails when queueing\n> +\t * buffers to its input.\n> +\t *\n> +\t * If no ImgU configuration has been computed, it means only a RAW\n> +\t * stream has been requested: return here to skip the ImgU configuration\n> +\t * part.\n> +\t */\n> +\tImgUDevice::PipeConfig imguConfig = config->imguConfig();\n> +\tif (imguConfig.isNull())\n> +\t\treturn 0;\n> +\n>  \t/*\n>  \t * FIXME: enabled links in one ImgU pipe interfere with capture\n>  \t * operations on the other one. This can be easily triggered by\n> @@ -531,34 +559,6 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c)\n>  \tif (ret)\n>  \t\treturn ret;\n>  \n> -\t/*\n> -\t * Pass the requested stream size to the CIO2 unit and get back the\n> -\t * adjusted format to be propagated to the ImgU output devices.\n> -\t */\n> -\tconst Size &sensorSize = config->cio2Format().size;\n> -\tV4L2DeviceFormat cio2Format;\n> -\tret = cio2->configure(sensorSize, config->combinedTransform_, &cio2Format);\n> -\tif (ret)\n> -\t\treturn ret;\n> -\n> -\tIPACameraSensorInfo sensorInfo;\n> -\tcio2->sensor()->sensorInfo(&sensorInfo);\n> -\tdata->cropRegion_ = sensorInfo.analogCrop;\n> -\n> -\t/*\n> -\t * If the ImgU gets configured, its driver seems to expect that\n> -\t * buffers will be queued to its outputs, as otherwise the next\n> -\t * capture session that uses the ImgU fails when queueing\n> -\t * buffers to its input.\n> -\t *\n> -\t * If no ImgU configuration has been computed, it means only a RAW\n> -\t * stream has been requested: return here to skip the ImgU configuration\n> -\t * part.\n> -\t */\n> -\tImgUDevice::PipeConfig imguConfig = config->imguConfig();\n> -\tif (imguConfig.isNull())\n> -\t\treturn 0;\n> -\n>  \tret = imgu->configure(imguConfig, &cio2Format);\n>  \tif (ret)\n>  \t\treturn ret;\n> @@ -660,36 +660,39 @@ int PipelineHandlerIPU3::exportFrameBuffers(Camera *camera, Stream *stream,\n>  int PipelineHandlerIPU3::allocateBuffers(Camera *camera)\n>  {\n>  \tIPU3CameraData *data = cameraData(camera);\n> +\tCIO2Device *cio2 = &data->cio2_;\n>  \tImgUDevice *imgu = data->imgu_;\n>  \tunsigned int bufferCount;\n>  \tint ret;\n>  \n> -\tbufferCount = std::max({\n> -\t\tdata->outStream_.configuration().bufferCount,\n> -\t\tdata->vfStream_.configuration().bufferCount,\n> -\t\tdata->rawStream_.configuration().bufferCount,\n> -\t});\n> +\tif (cio2->needsImgu()) {\n> +\t\tbufferCount = std::max({\n> +\t\t\tdata->outStream_.configuration().bufferCount,\n> +\t\t\tdata->vfStream_.configuration().bufferCount,\n> +\t\t\tdata->rawStream_.configuration().bufferCount,\n> +\t\t});\n>  \n> -\tret = imgu->allocateBuffers(bufferCount);\n> -\tif (ret < 0)\n> -\t\treturn ret;\n> +\t\tret = imgu->allocateBuffers(bufferCount);\n> +\t\tif (ret < 0)\n> +\t\t\treturn ret;\n>  \n> -\t/* Map buffers to the IPA. */\n> -\tunsigned int ipaBufferId = 1;\n> +\t\t/* Map buffers to the IPA. */\n> +\t\tunsigned int ipaBufferId = 1;\n>  \n> -\tfor (const std::unique_ptr<FrameBuffer> &buffer : imgu->paramBuffers_) {\n> -\t\tbuffer->setCookie(ipaBufferId++);\n> -\t\tipaBuffers_.emplace_back(buffer->cookie(), buffer->planes());\n> -\t}\n> +\t\tfor (const std::unique_ptr<FrameBuffer> &buffer : imgu->paramBuffers_) {\n> +\t\t\tbuffer->setCookie(ipaBufferId++);\n> +\t\t\tipaBuffers_.emplace_back(buffer->cookie(), buffer->planes());\n> +\t\t}\n>  \n> -\tfor (const std::unique_ptr<FrameBuffer> &buffer : imgu->statBuffers_) {\n> -\t\tbuffer->setCookie(ipaBufferId++);\n> -\t\tipaBuffers_.emplace_back(buffer->cookie(), buffer->planes());\n> -\t}\n> +\t\tfor (const std::unique_ptr<FrameBuffer> &buffer : imgu->statBuffers_) {\n> +\t\t\tbuffer->setCookie(ipaBufferId++);\n> +\t\t\tipaBuffers_.emplace_back(buffer->cookie(), buffer->planes());\n> +\t\t}\n>  \n> -\tdata->ipa_->mapBuffers(ipaBuffers_);\n> +\t\tdata->ipa_->mapBuffers(ipaBuffers_);\n> +\t\tdata->frameInfos_.init(imgu->paramBuffers_, imgu->statBuffers_);\n> +\t}\n>  \n> -\tdata->frameInfos_.init(imgu->paramBuffers_, imgu->statBuffers_);\n>  \tdata->frameInfos_.bufferAvailable.connect(\n>  \t\tdata, &IPU3CameraData::queuePendingRequests);\n>  \n> @@ -699,17 +702,20 @@ int PipelineHandlerIPU3::allocateBuffers(Camera *camera)\n>  int PipelineHandlerIPU3::freeBuffers(Camera *camera)\n>  {\n>  \tIPU3CameraData *data = cameraData(camera);\n> +\tCIO2Device *cio2 = &data->cio2_;\n>  \n>  \tdata->frameInfos_.clear();\n>  \n> -\tstd::vector<unsigned int> ids;\n> -\tfor (IPABuffer &ipabuf : ipaBuffers_)\n> -\t\tids.push_back(ipabuf.id);\n> +\tif (cio2->needsImgu()) {\n> +\t\tstd::vector<unsigned int> ids;\n> +\t\tfor (IPABuffer &ipabuf : ipaBuffers_)\n> +\t\t\tids.push_back(ipabuf.id);\n>  \n> -\tdata->ipa_->unmapBuffers(ids);\n> -\tipaBuffers_.clear();\n> +\t\tdata->ipa_->unmapBuffers(ids);\n> +\t\tdata->imgu_->freeBuffers();\n>  \n> -\tdata->imgu_->freeBuffers();\n> +\t\tipaBuffers_.clear();\n> +\t}\n>  \n>  \treturn 0;\n>  }\n> @@ -736,10 +742,6 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis\n>  \tif (ret)\n>  \t\treturn ret;\n>  \n> -\tret = data->ipa_->start();\n> -\tif (ret)\n> -\t\tgoto error;\n> -\n>  \tdata->delayedCtrls_->reset();\n>  \n>  \t/*\n> @@ -750,16 +752,24 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis\n>  \tif (ret)\n>  \t\tgoto error;\n>  \n> -\tret = imgu->start();\n> -\tif (ret)\n> -\t\tgoto error;\n> +\tif (cio2->needsImgu()) {\n> +\t\tret = data->ipa_->start();\n> +\t\tif (ret)\n> +\t\t\tgoto error;\n> +\n> +\t\tret = imgu->start();\n> +\t\tif (ret)\n> +\t\t\tgoto error;\n> +\t}\n>  \n>  \treturn 0;\n>  \n>  error:\n> -\timgu->stop();\n>  \tcio2->stop();\n> -\tdata->ipa_->stop();\n> +\tif (cio2->needsImgu()) {\n> +\t\timgu->stop();\n> +\t\tdata->ipa_->stop();\n> +\t}\n>  \tfreeBuffers(camera);\n>  \tLOG(IPU3, Error) << \"Failed to start camera \" << camera->id();\n>  \n> @@ -773,9 +783,11 @@ void PipelineHandlerIPU3::stopDevice(Camera *camera)\n>  \n>  \tdata->cancelPendingRequests();\n>  \n> -\tdata->ipa_->stop();\n> +\tif (data->cio2_.needsImgu()) {\n> +\t\tdata->ipa_->stop();\n> +\t\tret |= data->imgu_->stop();\n> +\t}\n>  \n> -\tret |= data->imgu_->stop();\n>  \tret |= data->cio2_.stop();\n>  \tif (ret)\n>  \t\tLOG(IPU3, Warning) << \"Failed to stop camera \" << camera->id();\n> @@ -806,7 +818,7 @@ void IPU3CameraData::queuePendingRequests()\n>  \twhile (!pendingRequests_.empty()) {\n>  \t\tRequest *request = pendingRequests_.front();\n>  \n> -\t\tIPU3Frames::Info *info = frameInfos_.create(request, true);\n> +\t\tIPU3Frames::Info *info = frameInfos_.create(request, cio2_.needsImgu());\n>  \t\tif (!info)\n>  \t\t\tbreak;\n>  \n> @@ -829,7 +841,8 @@ void IPU3CameraData::queuePendingRequests()\n>  \n>  \t\tinfo->rawBuffer = rawBuffer;\n>  \n> -\t\tipa_->queueRequest(info->id, request->controls());\n> +\t\tif (cio2_.needsImgu())\n> +\t\t\tipa_->queueRequest(info->id, request->controls());\n>  \n>  \t\tpendingRequests_.pop();\n>  \t\tprocessingRequests_.push(request);\n> @@ -1070,10 +1083,6 @@ int PipelineHandlerIPU3::registerCameras()\n>  \t\tif (ret)\n>  \t\t\tcontinue;\n>  \n> -\t\tret = data->loadIPA();\n> -\t\tif (ret)\n> -\t\t\tcontinue;\n> -\n>  \t\t/* Initialize the camera properties. */\n>  \t\tdata->properties_ = cio2->sensor()->properties();\n>  \n> @@ -1104,36 +1113,44 @@ int PipelineHandlerIPU3::registerCameras()\n>  \t\t\t\t\t   << cio2->sensor()->id()\n>  \t\t\t\t\t   << \". Assume rotation 0\";\n>  \n> -\t\t/**\n> -\t\t * \\todo Dynamically assign ImgU and output devices to each\n> -\t\t * stream and camera; as of now, limit support to two cameras\n> -\t\t * only, and assign imgu0 to the first one and imgu1 to the\n> -\t\t * second.\n> -\t\t */\n> -\t\tdata->imgu_ = numCameras ? &imgu1_ : &imgu0_;\n> -\n>  \t\t/*\n>  \t\t * Connect video devices' 'bufferReady' signals to their\n>  \t\t * slot to implement the image processing pipeline.\n> -\t\t *\n> -\t\t * Frames produced by the CIO2 unit are passed to the\n> -\t\t * associated ImgU input where they get processed and\n> -\t\t * returned through the ImgU main and secondary outputs.\n>  \t\t */\n>  \t\tdata->cio2_.bufferReady().connect(data.get(),\n> -\t\t\t\t\t&IPU3CameraData::cio2BufferReady);\n> +\t\t\t\t\t\t  &IPU3CameraData::cio2BufferReady);\n>  \t\tdata->cio2_.bufferAvailable.connect(\n>  \t\t\tdata.get(), &IPU3CameraData::queuePendingRequests);\n> -\t\tdata->imgu_->input_->bufferReady.connect(&data->cio2_,\n> -\t\t\t\t\t&CIO2Device::tryReturnBuffer);\n> -\t\tdata->imgu_->output_->bufferReady.connect(data.get(),\n> -\t\t\t\t\t&IPU3CameraData::imguOutputBufferReady);\n> -\t\tdata->imgu_->viewfinder_->bufferReady.connect(data.get(),\n> -\t\t\t\t\t&IPU3CameraData::imguOutputBufferReady);\n> -\t\tdata->imgu_->param_->bufferReady.connect(data.get(),\n> -\t\t\t\t\t&IPU3CameraData::paramBufferReady);\n> -\t\tdata->imgu_->stat_->bufferReady.connect(data.get(),\n> -\t\t\t\t\t&IPU3CameraData::statBufferReady);\n> +\n> +\t\tif (cio2->needsImgu()) {\n> +\t\t\tret = data->loadIPA();\n> +\t\t\tif (ret)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\t/**\n> +\t\t\t * \\todo Dynamically assign ImgU and output devices to each\n> +\t\t\t * stream and camera; as of now, limit support to two cameras\n> +\t\t\t * only, and assign imgu0 to the first one and imgu1 to the\n> +\t\t\t * second.\n> +\t\t\t */\n> +\t\t\tdata->imgu_ = numCameras ? &imgu1_ : &imgu0_;\n> +\n> +\t\t\t/*\n> +\t\t\t* Frames produced by the CIO2 unit are passed to the\n> +\t\t\t* associated ImgU input where they get processed and\n> +\t\t\t* returned through the ImgU main and secondary outputs.\n> +\t\t\t*/\n> +\t\t\tdata->imgu_->input_->bufferReady.connect(&data->cio2_,\n> +\t\t\t\t\t\t&CIO2Device::tryReturnBuffer);\n> +\t\t\tdata->imgu_->output_->bufferReady.connect(data.get(),\n> +\t\t\t\t\t\t&IPU3CameraData::imguOutputBufferReady);\n> +\t\t\tdata->imgu_->viewfinder_->bufferReady.connect(data.get(),\n> +\t\t\t\t\t\t&IPU3CameraData::imguOutputBufferReady);\n> +\t\t\tdata->imgu_->param_->bufferReady.connect(data.get(),\n> +\t\t\t\t\t\t&IPU3CameraData::paramBufferReady);\n> +\t\t\tdata->imgu_->stat_->bufferReady.connect(data.get(),\n> +\t\t\t\t\t\t&IPU3CameraData::statBufferReady);\n> +\t\t}\n>  \n>  \t\t/* Create and register the Camera instance. */\n>  \t\tconst std::string &cameraId = cio2->sensor()->id();\n> @@ -1330,10 +1347,15 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer)\n>  \n>  \tinfo->effectiveSensorControls = delayedCtrls_->get(buffer->metadata().sequence);\n>  \n> -\tif (request->findBuffer(&rawStream_))\n> +\tif (request->findBuffer(&rawStream_)) {\n>  \t\tpipe()->completeBuffer(request, buffer);\n>  \n> -\tipa_->fillParamsBuffer(info->id, info->paramBuffer->cookie());\n> +\t\tif (frameInfos_.tryComplete(info))\n> +\t\t\tpipe()->completeRequest(request);\n> +\t}\n> +\n> +\tif (cio2_.needsImgu())\n> +\t\tipa_->fillParamsBuffer(info->id, info->paramBuffer->cookie());\n>  }\n>  \n>  void IPU3CameraData::paramBufferReady(FrameBuffer *buffer)","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2B509BD1F1\n\tfor <parsemail@patchwork.libcamera.org>;\n\tWed, 19 Apr 2023 05:54:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 92D6B627B8;\n\tWed, 19 Apr 2023 07:54:58 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A8B4D603A0\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed, 19 Apr 2023 07:54:56 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(133-32-181-51.west.xps.vectant.ne.jp [133.32.181.51])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 4C27B10A;\n\tWed, 19 Apr 2023 07:54:49 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1681883698;\n\tbh=2TVDNbwMU463XRmA0IuA/2VYskHpD4e26HKn3TbnWN4=;\n\th=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:\n\tFrom;\n\tb=vY8ERxMRx5jc+cVIlT/xxDbFWNW2C1WY291RCJ9EdY+WOWsqyTvcdosOl0PRoHV0u\n\tOQXwXC3sN/Bk6yBDQKAeGE5AM7f4h1l7BW/WXancKLvFVxb2T1ES6Ff/BSGJEst5oZ\n\tct3RfSKQ/pV310GWIlAzISKnjr1fwpR/Lg3kNdQqqmp7DDxXSvRJhJ8A1AX0UcWxIW\n\t+JEF0ojN2F/bGgaIZD0Qmbo68CIz1ZycHEAWIDZiCibVznnM1XtXchEtnCzUadKRd2\n\tvqIzzzCUIv4cepLdNl4Jkm4W6KqeBzg3K+EFW9x4L5UsEFHpY845FVzOw+Ue/46HDo\n\tlLOB8O0TbDzig==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1681883690;\n\tbh=2TVDNbwMU463XRmA0IuA/2VYskHpD4e26HKn3TbnWN4=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=e3XX88l9XWmmyH6Z9IX1Su/z0iDw/u0F6tEX4ueQlwn69tfiINDXESSp+r1oc7rrw\n\tTzybzXav429b4c38y4oKYtuGsD0FRprhLwgnRnmSLAZYgkvY3EKhDoQNm9TJV6HxUb\n\tdwnGKAlgEdV7Fzgzv1aq5AUTFX08/ptmNUbnhKHg="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"e3XX88l9\"; dkim-atps=neutral","Date":"Wed, 19 Apr 2023 08:55:08 +0300","To":"Daniel Scally <dan.scally@ideasonboard.com>","Message-ID":"<20230419055508.GP30837@pendragon.ideasonboard.com>","References":"<20230318234014.29506-1-dan.scally@ideasonboard.com>\n\t<20230318234014.29506-7-dan.scally@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20230318234014.29506-7-dan.scally@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH 06/11] pipeline: ipu3: Support sensors\n\tusing only CIO2","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","From":"Laurent Pinchart via libcamera-devel\n\t<libcamera-devel@lists.libcamera.org>","Reply-To":"Laurent Pinchart <laurent.pinchart@ideasonboard.com>","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"}}]