From patchwork Thu May 26 08:27:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Harvey Yang X-Patchwork-Id: 16055 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8794ABD161 for ; Thu, 26 May 2022 08:28:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E082A6566B; Thu, 26 May 2022 10:27:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1653553679; bh=NilUXQlh17c9nyL0n8+RDM5lzFv3afTxeR0DKKO/yUc=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Q68z+ztW7HXeB2pwq+7KfMGmY4mUAtXHe1nH3zGWk8fr5SRtlmSvHQqMs7RNRCpIw 2hLCmpt0bYGWGyfHREXvaOpA8j+HSphqYyjGuxHYiU3LJ9TSippYZmfFfza8jeGueF aoo98A1LfU+QDK10iPEAwrxfQ3/r7EbJLy0P4DTSOFbbTX6Q267XOCTtXIqbebPoLC LH23jTQmosJWeRSWBEF2BaIX+I/AkFK7giolPQVjLUTYEhryiXX2y544r/X1IHq7Mr TxvzfXhe/dygHWmAglLf17o2fsYD+xD56iGQdHE/cw33od31LPhOjFDhYASDgA9kvl wLUua4334STHw== Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8898265663 for ; Thu, 26 May 2022 10:27:56 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="N/ovMBc9"; dkim-atps=neutral Received: by mail-pg1-x534.google.com with SMTP id v15so767675pgk.11 for ; Thu, 26 May 2022 01:27:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5E3jNNUwiKdnag9tyxtSoX+l4avZ7kjXyEWrGbZbUFM=; b=N/ovMBc9SLdlw+1RTsBOFATNQtl0quGlnMfhsMKpGSQKqxetcN5qHD1WgbxBC/zaxk 6M32Di47UkCKxGFd1yPhC7jRf7t2LQ+cJCP5ozM9PwWQd7hfIQYFmP/9je+PPdj1Mq9R LpIUc7dEbbt+8BbSoe5pnwg45fmLsxMMtj/I0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5E3jNNUwiKdnag9tyxtSoX+l4avZ7kjXyEWrGbZbUFM=; b=SW+6lrNdFa1qxGLioz6jx4F+Aojv76OoywGEDC3Ek/MOYj70KlA52gTaXjmkXtZAH1 kJf3wQa49nHPgLpIvOuOHAUCX7mKB+SrIpyzr9TS4dKDuP/3EtHcvbmZM64DZjmRmlb6 XlYe8uIyKoYg8WIU2WziKGOiZ9p0xwzxryaEgF2Sbz2bwx0MlI7B5+wO3YWNok+sx8Kx VvPoC9jr6mM1M4LNV2mNfbU6aEgpKDgWlOy84+o/WJyemZYCRomYzRuDUu2tt8LCo5vs bAahwfHfkxdLyPHU2bcEwQTTMdgTCWZ0GGTa4Sse7vVJTREp2xE1Kl6Q80qRwgqvpkrW XNFg== X-Gm-Message-State: AOAM532MIea4LIxYUaUWrmPhMMnvn8gOxQ/ng5HNpaW6R4GlonscfyWS jtmJ7J0qX7rlAhF4JuvIUxvBuHMoy4BhwqM7 X-Google-Smtp-Source: ABdhPJwU4l1HPj99xS2LFdz2oUZgMr9D2gLoEtBtAqF9P5avpwU1l3xrTDQx/d9AqTLpMkL9Ag1Qzw== X-Received: by 2002:a63:dd4c:0:b0:3fa:a80e:8ba0 with SMTP id g12-20020a63dd4c000000b003faa80e8ba0mr11039466pgj.167.1653553674382; Thu, 26 May 2022 01:27:54 -0700 (PDT) Received: from chenghaoyang-low.c.googlers.com.com (21.160.199.104.bc.googleusercontent.com. [104.199.160.21]) by smtp.gmail.com with ESMTPSA id m2-20020a170902bb8200b0016191b843e2sm834429pls.235.2022.05.26.01.27.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 May 2022 01:27:53 -0700 (PDT) X-Google-Original-From: Harvey Yang To: libcamera-devel@lists.libcamera.org Date: Thu, 26 May 2022 08:27:34 +0000 Message-Id: <20220526082737.1081262-7-chenghaoyang@google.com> X-Mailer: git-send-email 2.36.1.124.g0e6072fb45-goog In-Reply-To: <20220526082737.1081262-1-chenghaoyang@google.com> References: <20220526082737.1081262-1-chenghaoyang@google.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 6/9] ipu3: Configure imgu1 when necessary X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Harvey Yang via libcamera-devel From: Harvey Yang Reply-To: Harvey Yang Cc: Harvey Yang , Harvey Yang Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch configure and setup imgu1 when still capture and other streams are enabled. If only still capture is enabled, imgu0 is still being used with IPU3PipeModeVideo (i.e. VideoSnapshot) to ensure 3A is working. Still capture stream will be enabled in the following patches. Signed-off-by: Harvey Yang --- src/libcamera/pipeline/ipu3/ipu3.cpp | 173 +++++++++++++++++++++++---- 1 file changed, 149 insertions(+), 24 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index a13fb881..ec9d14d1 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -55,9 +55,11 @@ public: int loadIPA(); + void tryReturnBuffer(FrameBuffer *buffer); void imguOutputBufferReady(FrameBuffer *buffer); void cio2BufferReady(FrameBuffer *buffer); void paramBufferReady(FrameBuffer *buffer); + void captureParamBufferReady(FrameBuffer *buffer); void statBufferReady(FrameBuffer *buffer); void queuePendingRequests(); void cancelPendingRequests(); @@ -93,6 +95,8 @@ private: void paramsBufferReady(unsigned int id); void setSensorControls(unsigned int id, const ControlList &sensorControls, const ControlList &lensControls); + + std::map bufferReturnCounters; }; class IPU3CameraConfiguration : public CameraConfiguration @@ -106,7 +110,8 @@ public: Status validate() override; const StreamConfiguration &cio2Format() const { return cio2Configuration_; } - const ImgUDevice::PipeConfig imguConfig() const { return pipeConfig_; } + const ImgUDevice::PipeConfig imguConfig0() const { return pipeConfig0_; } + const ImgUDevice::PipeConfig imguConfig1() const { return pipeConfig1_; } /* Cache the combinedTransform_ that will be applied to the sensor */ Transform combinedTransform_; @@ -120,7 +125,8 @@ private: const IPU3CameraData *data_; StreamConfiguration cio2Configuration_; - ImgUDevice::PipeConfig pipeConfig_; + ImgUDevice::PipeConfig pipeConfig0_; + ImgUDevice::PipeConfig pipeConfig1_; }; class PipelineHandlerIPU3 : public PipelineHandler @@ -168,6 +174,8 @@ private: MediaDevice *cio2MediaDev_; MediaDevice *imguMediaDev_; + bool useImgu1_ = false; + Camera *inUseCamera_ = nullptr; std::vector ipaBuffers_; @@ -408,8 +416,8 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate() /* Only compute the ImgU configuration if a YUV stream has been requested. */ if (yuvCount) { - pipeConfig_ = data_->imgu0_->calculatePipeConfig(&pipe); - if (pipeConfig_.isNull()) { + pipeConfig0_ = data_->imgu0_->calculatePipeConfig(&pipe); + if (pipeConfig0_.isNull()) { LOG(IPU3, Error) << "Failed to calculate pipe configuration: " << "unsupported resolutions."; return Invalid; @@ -519,8 +527,13 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) IPU3CameraData *data = cameraData(camera); Stream *outStream = &data->outStream_; Stream *vfStream = &data->vfStream_; + Stream *outCaptureStream = &data->outCaptureStream_; CIO2Device *cio2 = &data->cio2_; V4L2DeviceFormat outputFormat; + + ImgUDevice::PipeConfig imguConfig1 = config->imguConfig1(); + useImgu1_ = !imguConfig1.isNull(); + int ret; /* @@ -565,6 +578,12 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) if (ret) return ret; + if (useImgu1_) { + ret = imgu1_.enableLinks(true); + if (ret) + return ret; + } + /* * Pass the requested stream size to the CIO2 unit and get back the * adjusted format to be propagated to the ImgU output devices. @@ -607,14 +626,20 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) * stream has been requested: return here to skip the ImgU configuration * part. */ - ImgUDevice::PipeConfig imguConfig = config->imguConfig(); - if (imguConfig.isNull()) + ImgUDevice::PipeConfig imguConfig0 = config->imguConfig0(); + if (imguConfig0.isNull()) return 0; - ret = imgu0_.configure(imguConfig, &cio2Format); + ret = imgu0_.configure(imguConfig0, &cio2Format); if (ret) return ret; + if (useImgu1_) { + ret = imgu1_.configure(imguConfig1, &cio2Format); + if (ret) + return ret; + } + /* Apply the format to the configured streams output devices. */ StreamConfiguration *mainCfg = nullptr; StreamConfiguration *vfCfg = nullptr; @@ -633,6 +658,15 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) ret = imgu0_.configureViewfinder(cfg, &outputFormat); if (ret) return ret; + } else if (stream == outCaptureStream) { + ASSERT(useImgu1_); + ret = imgu1_.configureOutput(cfg, &outputFormat); + if (ret) + return ret; + + ret = imgu1_.configureViewfinder(cfg, &outputFormat); + if (ret) + return ret; } } @@ -648,22 +682,34 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) } /* Apply the "pipe_mode" control to the ImgU subdevice. */ - ControlList ctrls(imgu0_.imgu_->controls()); + ControlList ctrls0(imgu0_.imgu_->controls()); /* * Set the ImgU pipe mode to 'Video' unconditionally to have statistics * generated. - * - * \todo Figure out what the 'Still Capture' mode is meant for, and use - * it accordingly. */ - ctrls.set(V4L2_CID_IPU3_PIPE_MODE, - static_cast(IPU3PipeModeVideo)); - ret = imgu0_.imgu_->setControls(&ctrls); + ctrls0.set(V4L2_CID_IPU3_PIPE_MODE, + static_cast(IPU3PipeModeVideo)); + ret = imgu0_.imgu_->setControls(&ctrls0); if (ret) { LOG(IPU3, Error) << "Unable to set pipe_mode control"; return ret; } + if (useImgu1_) { + ControlList ctrls1(imgu1_.imgu_->controls()); + /* + * \todo Figure out what the 'Still Capture' mode is meant for, + * and use it accordingly. + */ + ctrls1.set(V4L2_CID_IPU3_PIPE_MODE, + static_cast(IPU3PipeModeStillCapture)); + ret = imgu1_.imgu_->setControls(&ctrls1); + if (ret) { + LOG(IPU3, Error) << "Unable to set pipe_mode control"; + return ret; + } + } + ipa::ipu3::IPAConfigInfo configInfo; configInfo.sensorControls = data->cio2_.sensor()->controls(); @@ -672,8 +718,8 @@ int PipelineHandlerIPU3::configure(Camera *camera, CameraConfiguration *c) configInfo.lensControls = lens->controls(); configInfo.sensorInfo = sensorInfo; - configInfo.bdsOutputSize = config->imguConfig().bds; - configInfo.iif = config->imguConfig().iif; + configInfo.bdsOutputSize = config->imguConfig0().bds; + configInfo.iif = config->imguConfig0().iif; ret = data->ipa_->configure(configInfo, &data->ipaControls_); if (ret) { @@ -793,8 +839,8 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis CIO2Device *cio2 = &data->cio2_; int ret; - imgu0_.input_->bufferReady.connect(&data->cio2_, - &CIO2Device::tryReturnBuffer); + imgu0_.input_->bufferReady.connect(data, + &IPU3CameraData::tryReturnBuffer); imgu0_.output_->bufferReady.connect(data, &IPU3CameraData::imguOutputBufferReady); imgu0_.viewfinder_->bufferReady.connect(data, @@ -803,6 +849,16 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis &IPU3CameraData::paramBufferReady); imgu0_.stat_->bufferReady.connect(data, &IPU3CameraData::statBufferReady); + + if (useImgu1_) { + imgu1_.input_->bufferReady.connect(data, + &IPU3CameraData::tryReturnBuffer); + imgu1_.output_->bufferReady.connect(data, + &IPU3CameraData::imguOutputBufferReady); + imgu1_.param_->bufferReady.connect(data, + &IPU3CameraData::captureParamBufferReady); + } + /* Disable test pattern mode on the sensor, if any. */ ret = cio2->sensor()->setTestPatternMode( controls::draft::TestPatternModeEnum::TestPatternModeOff); @@ -834,10 +890,17 @@ int PipelineHandlerIPU3::start(Camera *camera, [[maybe_unused]] const ControlLis if (ret) goto error; + if (useImgu1_) { + ret = imgu1_.start(); + if (ret) + goto error; + } + return 0; error: imgu0_.stop(); + imgu1_.stop(); cio2->stop(); data->ipa_->stop(); freeBuffers(camera); @@ -848,6 +911,11 @@ error: imgu0_.viewfinder_->bufferReady.disconnect(); imgu0_.param_->bufferReady.disconnect(); imgu0_.stat_->bufferReady.disconnect(); + imgu1_.input_->bufferReady.disconnect(); + imgu1_.output_->bufferReady.disconnect(); + imgu1_.param_->bufferReady.disconnect(); + + useImgu1_ = false; inUseCamera_ = nullptr; return ret; @@ -863,6 +931,7 @@ void PipelineHandlerIPU3::stopDevice(Camera *camera) data->ipa_->stop(); ret |= imgu0_.stop(); + ret |= imgu1_.stop(); ret |= data->cio2_.stop(); if (ret) LOG(IPU3, Warning) << "Failed to stop camera " << camera->id(); @@ -874,7 +943,11 @@ void PipelineHandlerIPU3::stopDevice(Camera *camera) data->imgu0_->viewfinder_->bufferReady.disconnect(); data->imgu0_->param_->bufferReady.disconnect(); data->imgu0_->stat_->bufferReady.disconnect(); + data->imgu1_->input_->bufferReady.disconnect(); + data->imgu1_->output_->bufferReady.disconnect(); + data->imgu1_->param_->bufferReady.disconnect(); + useImgu1_ = false; inUseCamera_ = nullptr; } @@ -1320,22 +1393,45 @@ void IPU3CameraData::paramsBufferReady(unsigned int id) if (!info) return; + bool hasYuv = false; + bool hasCapture = false; /* Queue all buffers from the request aimed for the ImgU. */ for (auto it : info->request->buffers()) { const Stream *stream = it.first; FrameBuffer *outbuffer = it.second; - if (stream == &outStream_) + if (stream == &outStream_) { + hasYuv = true; imgu0_->output_->queueBuffer(outbuffer); - else if (stream == &vfStream_) + } else if (stream == &vfStream_) { + hasYuv = true; imgu0_->viewfinder_->queueBuffer(outbuffer); + } else if (stream == &outCaptureStream_) { + hasCapture = true; + + imgu1_->output_->queueBuffer(outbuffer); + } } - imgu0_->param_->queueBuffer(info->paramBuffer); - imgu0_->stat_->queueBuffer(info->statBuffer); - imgu0_->input_->queueBuffer(info->rawBuffer); + if (hasYuv) { + bufferReturnCounters[info->rawBuffer] += 1; - info->captureParamDequeued = true; + imgu0_->param_->queueBuffer(info->paramBuffer); + imgu0_->stat_->queueBuffer(info->statBuffer); + imgu0_->input_->queueBuffer(info->rawBuffer); + } else { + info->paramDequeued = true; + info->metadataProcessed = true; + } + + if (hasCapture) { + bufferReturnCounters[info->rawBuffer] += 1; + + imgu1_->param_->queueBuffer(info->captureParamBuffer); + imgu1_->input_->queueBuffer(info->rawBuffer); + } else { + info->captureParamDequeued = true; + } } void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) @@ -1356,6 +1452,15 @@ void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) * Buffer Ready slots */ +void IPU3CameraData::tryReturnBuffer(FrameBuffer *buffer) +{ + if (--bufferReturnCounters[buffer] > 0) + return; + + bufferReturnCounters.erase(buffer); + cio2_.tryReturnBuffer(buffer); +} + /** * \brief Handle buffers completion at the ImgU output * \param[in] buffer The completed buffer @@ -1448,6 +1553,26 @@ void IPU3CameraData::paramBufferReady(FrameBuffer *buffer) pipe()->completeRequest(request); } +void IPU3CameraData::captureParamBufferReady(FrameBuffer *buffer) +{ + IPU3Frames::Info *info = frameInfos_.find(buffer); + if (!info) + return; + + info->captureParamDequeued = true; + + /* + * tryComplete() will delete info if it completes the IPU3Frame. + * In that event, we must have obtained the Request before hand. + * + * \todo Improve the FrameInfo API to avoid this type of issue + */ + Request *request = info->request; + + if (frameInfos_.tryComplete(info)) + pipe()->completeRequest(request); +} + void IPU3CameraData::statBufferReady(FrameBuffer *buffer) { IPU3Frames::Info *info = frameInfos_.find(buffer);