From patchwork Thu Nov 24 02:51:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 17859 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CD02FBDE6B for ; Thu, 24 Nov 2022 02:52:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8706463326; Thu, 24 Nov 2022 03:52:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1669258323; bh=vBrMrU2YIZwERR5/f3yxMrEXrqpQ7RG72DN+vkmfEGI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=U59bJ9SkwLFgt81qIFXUN/0yNg+v0lTAFP1NTMqpRa+hMo4yd/HWLeUTcw5Y/4uvy hKf0nk8TiffKeZE4iP2+nLFN5BztUiLp8T8/ZtgwF+7GXk0CPZjVlIuM7H0YZyFOQQ E3Yzh77V/cB7dPfSm5w/oTM8koUEenwPvoZQQqUrvbB3PJfOXeYIfHtogD5fvud6EH rM4Fh1tSDcOJSSMHmMwgP+85e150lwWSRd7dkjsm33G8aC76pVNMnyvNiPUQME32Zq Ogvk040dXuLxFU7d0xVABnrwzpQi2DKkJ/z1nw5ry6Cikq+PnDjR3eUc2qp/YM1Lxe ndZj5NE71rdyw== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 956706331D for ; Thu, 24 Nov 2022 03:52:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="bc02A7G6"; dkim-atps=neutral Received: from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C9524496; Thu, 24 Nov 2022 03:52:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1669258321; bh=vBrMrU2YIZwERR5/f3yxMrEXrqpQ7RG72DN+vkmfEGI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bc02A7G6qbg3l6DvaMhxOKhV7fjKxDewj2MPsalPQbJGhwzpC4sp0+u9ONU+iFh8p Qs5StEuH8Sg9Q+nuR2ihQ9IHbRdQXPzE1ZB3ZQRlwRFJHWQK98MJJSreISNXXu1iEO EoUOxvnwsPtioZnGRW53vvHnCuXBCfDY+n3pBFXQ= To: libcamera-devel@lists.libcamera.org Date: Thu, 24 Nov 2022 04:51:32 +0200 Message-Id: <20221124025133.17875-9-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.37.4 In-Reply-To: <20221124025133.17875-1-laurent.pinchart@ideasonboard.com> References: <20221124025133.17875-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 8/9] pipeline: rkisp1: Support raw Bayer capture at runtime X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Laurent Pinchart via libcamera-devel From: Laurent Pinchart Reply-To: Laurent Pinchart Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Florian Sylvestre Implement support for raw Bayer capture at runtime, from start() to stop(). Support of raw formats in the camera configuration is split to a subsequent change to ease review. In raw mode, the ISP is bypassed. There is no need to provide parameter buffers, and the ISP will not generate statistics. This requires multiple changes in the buffer handling: - The params and stats buffers don't need to be allocated, and the corresponding video nodes don't need to be started or stopped. - The IPA module fillParamsBuffer() operation must not be called in queueRequestDevice(). As a result, the IPA module thus doesn't emit the paramsBufferReady signal. The main and self path video buffers must thus be queued directly in queueRequestDevice(). - The tryCompleteRequest() function must not to wait until the params buffer has been dequeued. - When the frame buffer has been captured, the IPA module processStatsBuffer() operation must be called directly to fill request metadata. Signed-off-by: Florian Sylvestre Signed-off-by: Paul Elder Signed-off-by: Laurent Pinchart Reviewed-by: Paul Elder Tested-by: Jacopo Mondi Reviewed-by: Jacopo Mondi --- Changes since v2: - Split from "pipeline: rkisp1: Support raw Bayer capture" - Drop new completeRaw() IPA operation - Don't queue params buffers in raw capture mode - Fix assertion failure when stopping capture --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 148 ++++++++++++++--------- 1 file changed, 93 insertions(+), 55 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 3eb31a49bd92..6073fee8ec20 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -67,7 +67,8 @@ class RkISP1Frames public: RkISP1Frames(PipelineHandler *pipe); - RkISP1FrameInfo *create(const RkISP1CameraData *data, Request *request); + RkISP1FrameInfo *create(const RkISP1CameraData *data, Request *request, + bool isRaw); int destroy(unsigned int frame); void clear(); @@ -184,6 +185,7 @@ private: std::unique_ptr csi_; bool hasSelfPath_; + bool isRaw_; RkISP1MainPath mainPath_; RkISP1SelfPath selfPath_; @@ -203,28 +205,35 @@ RkISP1Frames::RkISP1Frames(PipelineHandler *pipe) { } -RkISP1FrameInfo *RkISP1Frames::create(const RkISP1CameraData *data, Request *request) +RkISP1FrameInfo *RkISP1Frames::create(const RkISP1CameraData *data, Request *request, + bool isRaw) { unsigned int frame = data->frame_; - if (pipe_->availableParamBuffers_.empty()) { - LOG(RkISP1, Error) << "Parameters buffer underrun"; - return nullptr; - } - FrameBuffer *paramBuffer = pipe_->availableParamBuffers_.front(); + FrameBuffer *paramBuffer = nullptr; + FrameBuffer *statBuffer = nullptr; + + if (!isRaw) { + if (pipe_->availableParamBuffers_.empty()) { + LOG(RkISP1, Error) << "Parameters buffer underrun"; + return nullptr; + } + + if (pipe_->availableStatBuffers_.empty()) { + LOG(RkISP1, Error) << "Statisitc buffer underrun"; + return nullptr; + } + + paramBuffer = pipe_->availableParamBuffers_.front(); + pipe_->availableParamBuffers_.pop(); - if (pipe_->availableStatBuffers_.empty()) { - LOG(RkISP1, Error) << "Statisitc buffer underrun"; - return nullptr; + statBuffer = pipe_->availableStatBuffers_.front(); + pipe_->availableStatBuffers_.pop(); } - FrameBuffer *statBuffer = pipe_->availableStatBuffers_.front(); FrameBuffer *mainPathBuffer = request->findBuffer(&data->mainPathStream_); FrameBuffer *selfPathBuffer = request->findBuffer(&data->selfPathStream_); - pipe_->availableParamBuffers_.pop(); - pipe_->availableStatBuffers_.pop(); - RkISP1FrameInfo *info = new RkISP1FrameInfo; info->frame = frame; @@ -672,6 +681,8 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c) << "ISP input pad configured with " << format << " crop " << rect; + isRaw_ = false; + /* YUYV8_2X8 is required on the ISP source path pad for YUV output. */ format.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8; LOG(RkISP1, Debug) @@ -764,13 +775,15 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) data->selfPathStream_.configuration().bufferCount, }); - ret = param_->allocateBuffers(maxCount, ¶mBuffers_); - if (ret < 0) - goto error; + if (!isRaw_) { + ret = param_->allocateBuffers(maxCount, ¶mBuffers_); + if (ret < 0) + goto error; - ret = stat_->allocateBuffers(maxCount, &statBuffers_); - if (ret < 0) - goto error; + ret = stat_->allocateBuffers(maxCount, &statBuffers_); + if (ret < 0) + goto error; + } for (std::unique_ptr &buffer : paramBuffers_) { buffer->setCookie(ipaBufferId++); @@ -846,23 +859,25 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL data->frame_ = 0; - ret = param_->streamOn(); - if (ret) { - data->ipa_->stop(); - freeBuffers(camera); - LOG(RkISP1, Error) - << "Failed to start parameters " << camera->id(); - return ret; - } + if (!isRaw_) { + ret = param_->streamOn(); + if (ret) { + data->ipa_->stop(); + freeBuffers(camera); + LOG(RkISP1, Error) + << "Failed to start parameters " << camera->id(); + return ret; + } - ret = stat_->streamOn(); - if (ret) { - param_->streamOff(); - data->ipa_->stop(); - freeBuffers(camera); - LOG(RkISP1, Error) - << "Failed to start statistics " << camera->id(); - return ret; + ret = stat_->streamOn(); + if (ret) { + param_->streamOff(); + data->ipa_->stop(); + freeBuffers(camera); + LOG(RkISP1, Error) + << "Failed to start statistics " << camera->id(); + return ret; + } } if (data->mainPath_->isEnabled()) { @@ -907,15 +922,17 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera) selfPath_.stop(); mainPath_.stop(); - ret = stat_->streamOff(); - if (ret) - LOG(RkISP1, Warning) - << "Failed to stop statistics for " << camera->id(); + if (!isRaw_) { + ret = stat_->streamOff(); + if (ret) + LOG(RkISP1, Warning) + << "Failed to stop statistics for " << camera->id(); - ret = param_->streamOff(); - if (ret) - LOG(RkISP1, Warning) - << "Failed to stop parameters for " << camera->id(); + ret = param_->streamOff(); + if (ret) + LOG(RkISP1, Warning) + << "Failed to stop parameters for " << camera->id(); + } ASSERT(data->queuedRequests_.empty()); data->frameInfo_.clear(); @@ -929,12 +946,21 @@ int PipelineHandlerRkISP1::queueRequestDevice(Camera *camera, Request *request) { RkISP1CameraData *data = cameraData(camera); - RkISP1FrameInfo *info = data->frameInfo_.create(data, request); + RkISP1FrameInfo *info = data->frameInfo_.create(data, request, isRaw_); if (!info) return -ENOENT; data->ipa_->queueRequest(data->frame_, request->controls()); - data->ipa_->fillParamsBuffer(data->frame_, info->paramBuffer->cookie()); + if (isRaw_) { + if (info->mainPathBuffer) + data->mainPath_->queueBuffer(info->mainPathBuffer); + + if (data->selfPath_ && info->selfPathBuffer) + data->selfPath_->queueBuffer(info->selfPathBuffer); + } else { + data->ipa_->fillParamsBuffer(data->frame_, + info->paramBuffer->cookie()); + } data->frame_++; @@ -1139,7 +1165,7 @@ void PipelineHandlerRkISP1::tryCompleteRequest(RkISP1FrameInfo *info) if (!info->metadataProcessed) return; - if (!info->paramDequeued) + if (!isRaw_ && !info->paramDequeued) return; data->frameInfo_.destroy(info->frame); @@ -1156,16 +1182,28 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer) if (!info) return; + const FrameMetadata &metadata = buffer->metadata(); Request *request = buffer->request(); - /* - * Record the sensor's timestamp in the request metadata. - * - * \todo The sensor timestamp should be better estimated by connecting - * to the V4L2Device::frameStart signal. - */ - request->metadata().set(controls::SensorTimestamp, - buffer->metadata().timestamp); + if (metadata.status != FrameMetadata::FrameCancelled) { + /* + * Record the sensor's timestamp in the request metadata. + * + * \todo The sensor timestamp should be better estimated by connecting + * to the V4L2Device::frameStart signal. + */ + request->metadata().set(controls::SensorTimestamp, + metadata.timestamp); + + if (isRaw_) { + const ControlList &ctrls = + data->delayedCtrls_->get(metadata.sequence); + data->ipa_->processStatsBuffer(info->frame, 0, ctrls); + } + } else { + if (isRaw_) + info->metadataProcessed = true; + } completeBuffer(request, buffer); tryCompleteRequest(info);