From patchwork Thu May 12 10:32:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Harvey Yang X-Patchwork-Id: 15902 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D0ACFC326C for ; Thu, 12 May 2022 10:33:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 71E2665670; Thu, 12 May 2022 12:33:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1652351596; bh=VP2nw55uW/QB4dezhVhvnuEfHpreQgLzYfYwy7PCD+Y=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=xNICuhVe85cswJo9sdyHYBoEAgOkZut7WvSVtF9/4WM/h9RYsfkuJbrsLkBx5yQJ0 xj6sTPfqxH8o1+Ko40d4k+ghItca1YRng3JdITKZAgpi/Wse2tRLNF4GjLFuC+1hQR x5tAcrIkIqtj8oHFGSFTOVDcjeLNUmTMthGhVf8oR//sniRB00sv3iDl7HMTLp2KOB AQhVrg9cHdrpfG35A4I39/ihiHxr6r2c7kvRL6alIo9ONWo93W7hVRbsqvyUIvewD0 WJNSVl1HP45OKObnDNElh4+FJTcI0CzLbSfPaQPemntbv/GwlKMujoJved7jn+LI8o RveZgCQcsRAbQ== Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DB87165657 for ; Thu, 12 May 2022 12:33:14 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="T6OhWMWH"; dkim-atps=neutral Received: by mail-pl1-x634.google.com with SMTP id x18so4467279plg.6 for ; Thu, 12 May 2022 03:33:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DUzZlYpdbMeE74ukQP/CyVaVtD+Vet5FhfkFoC2n4bw=; b=T6OhWMWHzJy59BplgHZvi8xJb7uaLRO/FrrHmPm4keEtmaYu1p7v/oUofse811Y4MJ ZCY2aNra2dC6Ss//tXiXu87eX4nOqJKPcw2c04QdaCSzacZp4rTLSDwrryY8Nea/v3yT TuryYsDw4XBuQ6rpFs5Fukka76pKfsGVrN5xc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DUzZlYpdbMeE74ukQP/CyVaVtD+Vet5FhfkFoC2n4bw=; b=VedINhCOthJ/dmsA+SU3UfbWYdnffRikJ9UH9jVD9xPrBm/EToMZOArqFjSPfEkQXW /xBbuzMZx1MTukc0f86lbPe15Gf+pAvxWytOKH0v2B5H2xk0zp7XgIsMO5zTT1ScVdAF K8aFojhWTT/tdcNVuBMoj2BpbKy+a+Kr4z9fYSavO713cgwKlvn/cCl93euu+AbKSxP3 HeCkXVVu0UTO6+3NVQlEq2PFq28qMvCTcb3A8smVU2fw4BeXQp34u1FNkKMhv2wK/uqR 6QKm7fh/yn83G+KU3W1iBYFrbzPxScZhluHdXE9afdPpWpX6ktVO25rB+0eqhgvlI2W3 uHQA== X-Gm-Message-State: AOAM531I4LDp4s/9L0ZXZ7Vnazd2pGKncaI9x72ka3ESZKUc7iaOvBfS 6kj2j3bofCQT8axkSrjxkOzepvRJf6gbOQ6a X-Google-Smtp-Source: ABdhPJzPgxkhewf5JekBqacrDARL9eroLDLOkmyFhMWJfaA/W3DWASosYOBBjK1rDm08kltZ6ScOsQ== X-Received: by 2002:a17:90a:9483:b0:1dc:5fb2:9b89 with SMTP id s3-20020a17090a948300b001dc5fb29b89mr9998497pjo.235.1652351592975; Thu, 12 May 2022 03:33:12 -0700 (PDT) Received: from chenghaoyang-low.c.googlers.com.com (174.71.80.34.bc.googleusercontent.com. [34.80.71.174]) by smtp.gmail.com with ESMTPSA id t19-20020a62d153000000b005087c23ad8dsm3402219pfl.0.2022.05.12.03.33.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 May 2022 03:33:12 -0700 (PDT) X-Google-Original-From: Harvey Yang To: libcamera-devel@lists.libcamera.org Date: Thu, 12 May 2022 10:32:56 +0000 Message-Id: <20220512103258.324339-7-chenghaoyang@google.com> X-Mailer: git-send-email 2.36.0.512.ge40c2bad7a-goog In-Reply-To: <20220512103258.324339-1-chenghaoyang@google.com> References: <20220512103258.324339-1-chenghaoyang@google.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 6/8] Configure imgu1 when necessary X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Harvey Yang via libcamera-devel From: Harvey Yang Reply-To: Harvey Yang Cc: Harvey Yang , Harvey Yang Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch configure and setup imgu1 when still capture and other streams are enabled. If only still capture is enabled, imgu0 is still being used with IPU3PipeModeStillCapture. Still capture stream will be enabled in the following patches though. Signed-off-by: Harvey Yang --- src/libcamera/pipeline/ipu3/ipu3.cpp | 179 +++++++++++++++++++++++---- 1 file changed, 155 insertions(+), 24 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 8ce1968f..9498b1b0 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -55,9 +55,11 @@ public: int loadIPA(); + void tryReturnBuffer(FrameBuffer *buffer); void imguOutputBufferReady(FrameBuffer *buffer); void cio2BufferReady(FrameBuffer *buffer); void paramBufferReady(FrameBuffer *buffer); + void captureParamBufferReady(FrameBuffer *buffer); void statBufferReady(FrameBuffer *buffer); void queuePendingRequests(); void cancelPendingRequests(); @@ -93,6 +95,8 @@ private: void paramsBufferReady(unsigned int id); void setSensorControls(unsigned int id, const ControlList &sensorControls, const ControlList &lensControls); + + std::map inputReturnCounters; }; class IPU3CameraConfiguration : public CameraConfiguration @@ -106,11 +110,14 @@ public: Status validate() override; const StreamConfiguration &cio2Format() const { return cio2Configuration_; } - const ImgUDevice::PipeConfig imguConfig() const { return pipeConfig_; } + const ImgUDevice::PipeConfig imguConfig0() const { return pipeConfig0_; } + const ImgUDevice::PipeConfig imguConfig1() const { return pipeConfig1_; } /* Cache the combinedTransform_ that will be applied to the sensor */ Transform combinedTransform_; + bool onlyStillCapture_ = false; + private: /* * The IPU3CameraData instance is guaranteed to be valid as long as the @@ -120,7 +127,8 @@ private: const IPU3CameraData *data_; StreamConfiguration cio2Configuration_; - ImgUDevice::PipeConfig pipeConfig_; + ImgUDevice::PipeConfig pipeConfig0_; + ImgUDevice::PipeConfig pipeConfig1_; }; class PipelineHandlerIPU3 : public PipelineHandler @@ -168,6 +176,9 @@ private: MediaDevice *cio2MediaDev_; MediaDevice *imguMediaDev_; + bool onlyStillCapture_ = false; + bool useImgu1_ = false; + Camera *inUseCamera_ = nullptr; std::vector ipaBuffers_; @@ -408,8 +419,8 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate() /* Only compute the ImgU configuration if a YUV stream has been requested. */ if (yuvCount) { - pipeConfig_ = data_->imgu0_->calculatePipeConfig(&pipe); - if (pipeConfig_.isNull()) { + pipeConfig0_ = data_->imgu0_->calculatePipeConfig(&pipe); + if (pipeConfig0_.isNull()) { LOG(IPU3, Error) << "Failed to calculate pipe configuration: " << "unsupported resolutions."; return Invalid; @@ -519,8 +530,14 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) IPU3CameraData *data = cameraData(camera); Stream *outStream = &data->outStream_; Stream *vfStream = &data->vfStream_; + Stream *outCaptureStream = &data->outCaptureStream_; CIO2Device *cio2 = &data->cio2_; V4L2DeviceFormat outputFormat; + + onlyStillCapture_ = config->onlyStillCapture_; + ImgUDevice::PipeConfig imguConfig1 = config->imguConfig1(); + useImgu1_ = !imguConfig1.isNull(); + int ret; /* @@ -565,6 +582,12 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) if (ret) return ret; + if (useImgu1_) { + ret = imgu1_.enableLinks(true); + if (ret) + return ret; + } + /* * Pass the requested stream size to the CIO2 unit and get back the * adjusted format to be propagated to the ImgU output devices. @@ -607,14 +630,20 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) * stream has been requested: return here to skip the ImgU configuration * part. */ - ImgUDevice::PipeConfig imguConfig = config->imguConfig(); - if (imguConfig.isNull()) + ImgUDevice::PipeConfig imguConfig0 = config->imguConfig0(); + if (imguConfig0.isNull()) return 0; - ret = imgu0_.configure(imguConfig, &cio2Format); + ret = imgu0_.configure(imguConfig0, &cio2Format); if (ret) return ret; + if (useImgu1_) { + ret = imgu1_.configure(imguConfig1, &cio2Format); + if (ret) + return ret; + } + /* Apply the format to the configured streams output devices. */ StreamConfiguration *mainCfg = nullptr; StreamConfiguration *vfCfg = nullptr; @@ -633,6 +662,15 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) ret = imgu0_.configureViewfinder(cfg, &outputFormat); if (ret) return ret; + } else if (stream == outCaptureStream) { + ASSERT(useImgu1_); + ret = imgu1_.configureOutput(cfg, &outputFormat); + if (ret) + return ret; + + ret = imgu1_.configureViewfinder(cfg, &outputFormat); + if (ret) + return ret; } } @@ -648,22 +686,35 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) } /* Apply the "pipe_mode" control to the ImgU subdevice. */ - ControlList ctrls(imgu0_.imgu_->controls()); + ControlList ctrls0(imgu0_.imgu_->controls()); /* * Set the ImgU pipe mode to 'Video' unconditionally to have statistics * generated. - * - * \todo Figure out what the 'Still Capture' mode is meant for, and use - * it accordingly. */ - ctrls.set(V4L2_CID_IPU3_PIPE_MODE, - static_cast(IPU3PipeModeVideo)); - ret = imgu0_.imgu_->setControls(&ctrls); + ctrls0.set(V4L2_CID_IPU3_PIPE_MODE, + static_cast(onlyStillCapture_ ? + IPU3PipeModeStillCapture : IPU3PipeModeVideo)); + ret = imgu0_.imgu_->setControls(&ctrls0); if (ret) { LOG(IPU3, Error) << "Unable to set pipe_mode control"; return ret; } + if (useImgu1_) { + ControlList ctrls1(imgu1_.imgu_->controls()); + /* + * \todo Figure out what the 'Still Capture' mode is meant for, + * and use it accordingly. + */ + ctrls1.set(V4L2_CID_IPU3_PIPE_MODE, + static_cast(IPU3PipeModeStillCapture)); + ret = imgu1_.imgu_->setControls(&ctrls1); + if (ret) { + LOG(IPU3, Error) << "Unable to set pipe_mode control"; + return ret; + } + } + ipa::ipu3::IPAConfigInfo configInfo; configInfo.sensorControls = data->cio2_.sensor()->controls(); @@ -672,8 +723,8 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) configInfo.lensControls = lens->controls(); configInfo.sensorInfo = sensorInfo; - configInfo.bdsOutputSize = config->imguConfig().bds; - configInfo.iif = config->imguConfig().iif; + configInfo.bdsOutputSize = config->imguConfig0().bds; + configInfo.iif = config->imguConfig0().iif; ret = data->ipa_->configure(configInfo, &data->ipaControls_); if (ret) { @@ -789,8 +840,8 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis CIO2Device *cio2 = &data->cio2_; int ret; - imgu0_.input_->bufferReady.connect(&data->cio2_, - &CIO2Device::tryReturnBuffer); + imgu0_.input_->bufferReady.connect(data, + &IPU3CameraData::tryReturnBuffer); imgu0_.output_->bufferReady.connect(data, &IPU3CameraData::imguOutputBufferReady); imgu0_.viewfinder_->bufferReady.connect(data, @@ -799,6 +850,16 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis &IPU3CameraData::paramBufferReady); imgu0_.stat_->bufferReady.connect(data, &IPU3CameraData::statBufferReady); + + if (useImgu1_) { + imgu1_.input_->bufferReady.connect(data, + &IPU3CameraData::tryReturnBuffer); + imgu1_.output_->bufferReady.connect(data, + &IPU3CameraData::imguOutputBufferReady); + imgu1_.param_->bufferReady.connect(data, + &IPU3CameraData::captureParamBufferReady); + } + /* Disable test pattern mode on the sensor, if any. */ ret = cio2->sensor()->setTestPatternMode( controls::draft::TestPatternModeEnum::TestPatternModeOff); @@ -830,10 +891,17 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis if (ret) goto error; + if (useImgu1_) { + ret = imgu1_.start(); + if (ret) + goto error; + } + return 0; error: imgu0_.stop(); + imgu1_.stop(); cio2->stop(); data->ipa_->stop(); freeBuffers(camera); @@ -844,6 +912,12 @@ error: imgu0_.viewfinder_->bufferReady.disconnect(); imgu0_.param_->bufferReady.disconnect(); imgu0_.stat_->bufferReady.disconnect(); + imgu1_.input_->bufferReady.disconnect(); + imgu1_.output_->bufferReady.disconnect(); + imgu1_.param_->bufferReady.disconnect(); + + onlyStillCapture_ = false; + useImgu1_ = false; inUseCamera_ = nullptr; return ret; @@ -859,6 +933,7 @@ void PipelineHandlerIPU3::stopDevice(Camera *camera) data->ipa_->stop(); ret |= imgu0_.stop(); + ret |= imgu1_.stop(); ret |= data->cio2_.stop(); if (ret) LOG(IPU3, Warning) << "Failed to stop camera " << camera->id(); @@ -870,7 +945,12 @@ void PipelineHandlerIPU3::stopDevice(Camera *camera) data->imgu0_->viewfinder_->bufferReady.disconnect(); data->imgu0_->param_->bufferReady.disconnect(); data->imgu0_->stat_->bufferReady.disconnect(); + data->imgu1_->input_->bufferReady.disconnect(); + data->imgu1_->output_->bufferReady.disconnect(); + data->imgu1_->param_->bufferReady.disconnect(); + onlyStillCapture_ = false; + useImgu1_ = false; inUseCamera_ = nullptr; } @@ -1316,22 +1396,45 @@ void IPU3CameraData::paramsBufferReady(unsigned int id) if (!info) return; + bool hasYuv = false; + bool hasCapture = false; /* Queue all buffers from the request aimed for the ImgU. */ for (auto it : info->request->buffers()) { const Stream *stream = it.first; FrameBuffer *outbuffer = it.second; - if (stream == &outStream_) + if (stream == &outStream_) { + hasYuv = true; imgu0_->output_->queueBuffer(outbuffer); - else if (stream == &vfStream_) + } else if (stream == &vfStream_) { + hasYuv = true; imgu0_->viewfinder_->queueBuffer(outbuffer); + } else if (stream == &outCaptureStream_) { + hasCapture = true; + + imgu1_->output_->queueBuffer(outbuffer); + } } - imgu0_->param_->queueBuffer(info->paramBuffer); - imgu0_->stat_->queueBuffer(info->statBuffer); - imgu0_->input_->queueBuffer(info->rawBuffer); + if (hasYuv) { + inputReturnCounters[info->rawBuffer] += 1; - info->captureParamDequeued = true; + imgu0_->param_->queueBuffer(info->paramBuffer); + imgu0_->stat_->queueBuffer(info->statBuffer); + imgu0_->input_->queueBuffer(info->rawBuffer); + } else { + info->paramDequeued = true; + info->metadataProcessed = true; + } + + if (hasCapture) { + inputReturnCounters[info->rawBuffer] += 1; + + imgu1_->param_->queueBuffer(info->captureParamBuffer); + imgu1_->input_->queueBuffer(info->rawBuffer); + } else { + info->captureParamDequeued = true; + } } void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) @@ -1352,6 +1455,14 @@ void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) * Buffer Ready slots */ +void IPU3CameraData::tryReturnBuffer(FrameBuffer *buffer) +{ + if (--inputReturnCounters[buffer] == 0) { + inputReturnCounters.erase(buffer); + cio2_.tryReturnBuffer(buffer); + } +} + /** * \brief Handle buffers completion at the ImgU output * \param[in] buffer The completed buffer @@ -1443,6 +1554,26 @@ void IPU3CameraData::paramBufferReady(FrameBuffer *buffer) pipe()->completeRequest(request); } +void IPU3CameraData::captureParamBufferReady(FrameBuffer *buffer) +{ + IPU3Frames::Info *info = frameInfos_.find(buffer); + if (!info) + return; + + info->captureParamDequeued = true; + + /* + * tryComplete() will delete info if it completes the IPU3Frame. + * In that event, we must have obtained the Request before hand. + * + * \todo Improve the FrameInfo API to avoid this type of issue + */ + Request *request = info->request; + + if (frameInfos_.tryComplete(info)) + pipe()->completeRequest(request); +} + void IPU3CameraData::statBufferReady(FrameBuffer *buffer) { IPU3Frames::Info *info = frameInfos_.find(buffer);