From patchwork Wed Jun 29 10:30:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Harvey Yang X-Patchwork-Id: 16432 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 71A40BE173 for ; Wed, 29 Jun 2022 10:30:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 243566564A; Wed, 29 Jun 2022 12:30:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1656498637; bh=tCsDkvLIRPMp8NkbSM+fZjmRxr1XZq/dIBbbebw9z2U=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=s2LshpeHrg52a2he6ER54HPkmPRfcu8Vp3Dhs6+6UAZ7mfOCexE6kNyOvvcdGct6D CfcFDzLje3CH+jn7N9pKTh1tUruVSSgkacp0SPzTHVNDCt0sxWwvtbfGEt82fkCkqV FYNY4T06d4P0kF73lB1bNub1x8E2j/Dl+vk2RvZc5fbSwB6K7TKAlSoW9GBcW97TPX +uMl0yl3hvAgkygQrFkrN7wX4quwlgKVapS8XfxpdvdtLga3x8EAttJYhbn+tr/LOb pL0JRtvrho/Hp/jb5UgJ0No7MbIYsgRpE7qvn7d5STtz4sIXVgXbql7pT63U6aOEbE Z//utCvxCcX8g== Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9694265643 for ; Wed, 29 Jun 2022 12:30:31 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="GWwBpi/Z"; dkim-atps=neutral Received: by mail-pg1-x52f.google.com with SMTP id e63so14913392pgc.5 for ; Wed, 29 Jun 2022 03:30:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lLE8xkogtcdUmvJ6QKLyJkUAIsGf0UlLlM995CLC/gY=; b=GWwBpi/ZunZdG5o0f8oPs6ozzxn3cyZdtAih6cuNpto0HOIxXil65TTZNVRiA5UQ9O OT670Udpb6MhDd1lLJ7+43jzb1/Ro7l4Ttd6x/R9lQ5fbwM5ydmk4/wDyqlR4Otxx2cE wyX1wmTO1e9N8FDARKr/L3dLTKov6XzIBbRiQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lLE8xkogtcdUmvJ6QKLyJkUAIsGf0UlLlM995CLC/gY=; b=4sj0hZVBFlzBvphCnpmvYgfMejzV15ckEAqZyV+HlOwtAIcBLHhBC5b3iXe234b+V9 W4gtII6aMuBcMMqan6WXVyIBLGuziYtwzdizO5bOB6hFwl3Rzipltrv9/WjgQ4zZKJKh r8DZtH3igVL2j+UC0c/WBNvm5VajYvmvRqWF8ZJ7oG/oJX+39H2Jv+gMiYtG+Rwd5Lyw +MNUexOfjO4wQNnICglYcVF/td8uE7cU49bPs3AB9yQvydQ5LcoQ/iCd8eEVDds848EF kuW8zC+5hteErNpa3Bac90XhwqEThS2DDMBqo3zjNxPRYi6ie4qB+uHmqdOqbRcTossb TYVQ== X-Gm-Message-State: AJIora9A1RmyA1/Snez0dexhzrFt6m510Hk3a6d1ycFCeRpfN06erf5w UttjkdMcsbikaVIpn5zB+qpVK+3vs+n0gJ68 X-Google-Smtp-Source: AGRyM1tNe7vUmw6FqW6dypY6pK6t8ejlIA8CMUbZIKq5phh+9YFIhT6aLwv3tcuv0VuBIzZUMqi1Vg== X-Received: by 2002:a63:803:0:b0:40d:dfde:a822 with SMTP id 3-20020a630803000000b0040ddfdea822mr2451326pgi.93.1656498629886; Wed, 29 Jun 2022 03:30:29 -0700 (PDT) Received: from chenghaoyang-low.c.googlers.com.com (208.158.221.35.bc.googleusercontent.com. [35.221.158.208]) by smtp.gmail.com with ESMTPSA id jb4-20020a170903258400b00169071538a0sm10946059plb.267.2022.06.29.03.30.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Jun 2022 03:30:29 -0700 (PDT) X-Google-Original-From: Harvey Yang To: libcamera-devel@lists.libcamera.org Date: Wed, 29 Jun 2022 10:30:15 +0000 Message-Id: <20220629103018.4025635-7-chenghaoyang@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog In-Reply-To: <20220629103018.4025635-1-chenghaoyang@google.com> References: <20220629103018.4025635-1-chenghaoyang@google.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 6/9] ipu3: Configure imgu1 when necessary X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Harvey Yang via libcamera-devel From: Harvey Yang Reply-To: Harvey Yang Cc: Harvey Yang , Harvey Yang Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch configure and setup imgu1 when still capture and other streams are enabled. If only still capture is enabled, imgu0 is still being used with IPU3PipeModeVideo (i.e. VideoSnapshot) to ensure 3A is working. Still capture stream will be enabled in the following patches. Signed-off-by: Harvey Yang --- src/libcamera/pipeline/ipu3/ipu3.cpp | 173 +++++++++++++++++++++++---- 1 file changed, 149 insertions(+), 24 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index a13fb881..ec9d14d1 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -55,9 +55,11 @@ public: int loadIPA(); + void tryReturnBuffer(FrameBuffer *buffer); void imguOutputBufferReady(FrameBuffer *buffer); void cio2BufferReady(FrameBuffer *buffer); void paramBufferReady(FrameBuffer *buffer); + void captureParamBufferReady(FrameBuffer *buffer); void statBufferReady(FrameBuffer *buffer); void queuePendingRequests(); void cancelPendingRequests(); @@ -93,6 +95,8 @@ private: void paramsBufferReady(unsigned int id); void setSensorControls(unsigned int id, const ControlList &sensorControls, const ControlList &lensControls); + + std::map bufferReturnCounters; }; class IPU3CameraConfiguration : public CameraConfiguration @@ -106,7 +110,8 @@ public: Status validate() override; const StreamConfiguration &cio2Format() const { return cio2Configuration_; } - const ImgUDevice::PipeConfig imguConfig() const { return pipeConfig_; } + const ImgUDevice::PipeConfig imguConfig0() const { return pipeConfig0_; } + const ImgUDevice::PipeConfig imguConfig1() const { return pipeConfig1_; } /* Cache the combinedTransform_ that will be applied to the sensor */ Transform combinedTransform_; @@ -120,7 +125,8 @@ private: const IPU3CameraData *data_; StreamConfiguration cio2Configuration_; - ImgUDevice::PipeConfig pipeConfig_; + ImgUDevice::PipeConfig pipeConfig0_; + ImgUDevice::PipeConfig pipeConfig1_; }; class PipelineHandlerIPU3 : public PipelineHandler @@ -168,6 +174,8 @@ private: MediaDevice *cio2MediaDev_; MediaDevice *imguMediaDev_; + bool useImgu1_ = false; + Camera *inUseCamera_ = nullptr; std::vector ipaBuffers_; @@ -408,8 +416,8 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate() /* Only compute the ImgU configuration if a YUV stream has been requested. */ if (yuvCount) { - pipeConfig_ = data_->imgu0_->calculatePipeConfig(&pipe); - if (pipeConfig_.isNull()) { + pipeConfig0_ = data_->imgu0_->calculatePipeConfig(&pipe); + if (pipeConfig0_.isNull()) { LOG(IPU3, Error) << "Failed to calculate pipe configuration: " << "unsupported resolutions."; return Invalid; @@ -519,8 +527,13 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) IPU3CameraData *data = cameraData(camera); Stream *outStream = &data->outStream_; Stream *vfStream = &data->vfStream_; + Stream *outCaptureStream = &data->outCaptureStream_; CIO2Device *cio2 = &data->cio2_; V4L2DeviceFormat outputFormat; + + ImgUDevice::PipeConfig imguConfig1 = config->imguConfig1(); + useImgu1_ = !imguConfig1.isNull(); + int ret; /* @@ -565,6 +578,12 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) if (ret) return ret; + if (useImgu1_) { + ret = imgu1_.enableLinks(true); + if (ret) + return ret; + } + /* * Pass the requested stream size to the CIO2 unit and get back the * adjusted format to be propagated to the ImgU output devices. @@ -607,14 +626,20 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) * stream has been requested: return here to skip the ImgU configuration * part. */ - ImgUDevice::PipeConfig imguConfig = config->imguConfig(); - if (imguConfig.isNull()) + ImgUDevice::PipeConfig imguConfig0 = config->imguConfig0(); + if (imguConfig0.isNull()) return 0; - ret = imgu0_.configure(imguConfig, &cio2Format); + ret = imgu0_.configure(imguConfig0, &cio2Format); if (ret) return ret; + if (useImgu1_) { + ret = imgu1_.configure(imguConfig1, &cio2Format); + if (ret) + return ret; + } + /* Apply the format to the configured streams output devices. */ StreamConfiguration *mainCfg = nullptr; StreamConfiguration *vfCfg = nullptr; @@ -633,6 +658,15 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) ret = imgu0_.configureViewfinder(cfg, &outputFormat); if (ret) return ret; + } else if (stream == outCaptureStream) { + ASSERT(useImgu1_); + ret = imgu1_.configureOutput(cfg, &outputFormat); + if (ret) + return ret; + + ret = imgu1_.configureViewfinder(cfg, &outputFormat); + if (ret) + return ret; } } @@ -648,22 +682,34 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) } /* Apply the "pipe_mode" control to the ImgU subdevice. */ - ControlList ctrls(imgu0_.imgu_->controls()); + ControlList ctrls0(imgu0_.imgu_->controls()); /* * Set the ImgU pipe mode to 'Video' unconditionally to have statistics * generated. - * - * \todo Figure out what the 'Still Capture' mode is meant for, and use - * it accordingly. */ - ctrls.set(V4L2_CID_IPU3_PIPE_MODE, - static_cast(IPU3PipeModeVideo)); - ret = imgu0_.imgu_->setControls(&ctrls); + ctrls0.set(V4L2_CID_IPU3_PIPE_MODE, + static_cast(IPU3PipeModeVideo)); + ret = imgu0_.imgu_->setControls(&ctrls0); if (ret) { LOG(IPU3, Error) << "Unable to set pipe_mode control"; return ret; } + if (useImgu1_) { + ControlList ctrls1(imgu1_.imgu_->controls()); + /* + * \todo Figure out what the 'Still Capture' mode is meant for, + * and use it accordingly. + */ + ctrls1.set(V4L2_CID_IPU3_PIPE_MODE, + static_cast(IPU3PipeModeStillCapture)); + ret = imgu1_.imgu_->setControls(&ctrls1); + if (ret) { + LOG(IPU3, Error) << "Unable to set pipe_mode control"; + return ret; + } + } + ipa::ipu3::IPAConfigInfo configInfo; configInfo.sensorControls = data->cio2_.sensor()->controls(); @@ -672,8 +718,8 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) configInfo.lensControls = lens->controls(); configInfo.sensorInfo = sensorInfo; - configInfo.bdsOutputSize = config->imguConfig().bds; - configInfo.iif = config->imguConfig().iif; + configInfo.bdsOutputSize = config->imguConfig0().bds; + configInfo.iif = config->imguConfig0().iif; ret = data->ipa_->configure(configInfo, &data->ipaControls_); if (ret) { @@ -793,8 +839,8 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis CIO2Device *cio2 = &data->cio2_; int ret; - imgu0_.input_->bufferReady.connect(&data->cio2_, - &CIO2Device::tryReturnBuffer); + imgu0_.input_->bufferReady.connect(data, + &IPU3CameraData::tryReturnBuffer); imgu0_.output_->bufferReady.connect(data, &IPU3CameraData::imguOutputBufferReady); imgu0_.viewfinder_->bufferReady.connect(data, @@ -803,6 +849,16 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis &IPU3CameraData::paramBufferReady); imgu0_.stat_->bufferReady.connect(data, &IPU3CameraData::statBufferReady); + + if (useImgu1_) { + imgu1_.input_->bufferReady.connect(data, + &IPU3CameraData::tryReturnBuffer); + imgu1_.output_->bufferReady.connect(data, + &IPU3CameraData::imguOutputBufferReady); + imgu1_.param_->bufferReady.connect(data, + &IPU3CameraData::captureParamBufferReady); + } + /* Disable test pattern mode on the sensor, if any. */ ret = cio2->sensor()->setTestPatternMode( controls::draft::TestPatternModeEnum::TestPatternModeOff); @@ -834,10 +890,17 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis if (ret) goto error; + if (useImgu1_) { + ret = imgu1_.start(); + if (ret) + goto error; + } + return 0; error: imgu0_.stop(); + imgu1_.stop(); cio2->stop(); data->ipa_->stop(); freeBuffers(camera); @@ -848,6 +911,11 @@ error: imgu0_.viewfinder_->bufferReady.disconnect(); imgu0_.param_->bufferReady.disconnect(); imgu0_.stat_->bufferReady.disconnect(); + imgu1_.input_->bufferReady.disconnect(); + imgu1_.output_->bufferReady.disconnect(); + imgu1_.param_->bufferReady.disconnect(); + + useImgu1_ = false; inUseCamera_ = nullptr; return ret; @@ -863,6 +931,7 @@ void PipelineHandlerIPU3::stopDevice(Camera *camera) data->ipa_->stop(); ret |= imgu0_.stop(); + ret |= imgu1_.stop(); ret |= data->cio2_.stop(); if (ret) LOG(IPU3, Warning) << "Failed to stop camera " << camera->id(); @@ -874,7 +943,11 @@ void PipelineHandlerIPU3::stopDevice(Camera *camera) data->imgu0_->viewfinder_->bufferReady.disconnect(); data->imgu0_->param_->bufferReady.disconnect(); data->imgu0_->stat_->bufferReady.disconnect(); + data->imgu1_->input_->bufferReady.disconnect(); + data->imgu1_->output_->bufferReady.disconnect(); + data->imgu1_->param_->bufferReady.disconnect(); + useImgu1_ = false; inUseCamera_ = nullptr; } @@ -1320,22 +1393,45 @@ void IPU3CameraData::paramsBufferReady(unsigned int id) if (!info) return; + bool hasYuv = false; + bool hasCapture = false; /* Queue all buffers from the request aimed for the ImgU. */ for (auto it : info->request->buffers()) { const Stream *stream = it.first; FrameBuffer *outbuffer = it.second; - if (stream == &outStream_) + if (stream == &outStream_) { + hasYuv = true; imgu0_->output_->queueBuffer(outbuffer); - else if (stream == &vfStream_) + } else if (stream == &vfStream_) { + hasYuv = true; imgu0_->viewfinder_->queueBuffer(outbuffer); + } else if (stream == &outCaptureStream_) { + hasCapture = true; + + imgu1_->output_->queueBuffer(outbuffer); + } } - imgu0_->param_->queueBuffer(info->paramBuffer); - imgu0_->stat_->queueBuffer(info->statBuffer); - imgu0_->input_->queueBuffer(info->rawBuffer); + if (hasYuv) { + bufferReturnCounters[info->rawBuffer] += 1; - info->captureParamDequeued = true; + imgu0_->param_->queueBuffer(info->paramBuffer); + imgu0_->stat_->queueBuffer(info->statBuffer); + imgu0_->input_->queueBuffer(info->rawBuffer); + } else { + info->paramDequeued = true; + info->metadataProcessed = true; + } + + if (hasCapture) { + bufferReturnCounters[info->rawBuffer] += 1; + + imgu1_->param_->queueBuffer(info->captureParamBuffer); + imgu1_->input_->queueBuffer(info->rawBuffer); + } else { + info->captureParamDequeued = true; + } } void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) @@ -1356,6 +1452,15 @@ void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) * Buffer Ready slots */ +void IPU3CameraData::tryReturnBuffer(FrameBuffer *buffer) +{ + if (--bufferReturnCounters[buffer] > 0) + return; + + bufferReturnCounters.erase(buffer); + cio2_.tryReturnBuffer(buffer); +} + /** * \brief Handle buffers completion at the ImgU output * \param[in] buffer The completed buffer @@ -1448,6 +1553,26 @@ void IPU3CameraData::paramBufferReady(FrameBuffer *buffer) pipe()->completeRequest(request); } +void IPU3CameraData::captureParamBufferReady(FrameBuffer *buffer) +{ + IPU3Frames::Info *info = frameInfos_.find(buffer); + if (!info) + return; + + info->captureParamDequeued = true; + + /* + * tryComplete() will delete info if it completes the IPU3Frame. + * In that event, we must have obtained the Request before hand. + * + * \todo Improve the FrameInfo API to avoid this type of issue + */ + Request *request = info->request; + + if (frameInfos_.tryComplete(info)) + pipe()->completeRequest(request); +} + void IPU3CameraData::statBufferReady(FrameBuffer *buffer) { IPU3Frames::Info *info = frameInfos_.find(buffer);