From patchwork Fri Sep 23 12:55:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Sylvestre X-Patchwork-Id: 17392 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 442F7C327E for ; Fri, 23 Sep 2022 12:55:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E66786223E; Fri, 23 Sep 2022 14:55:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1663937754; bh=6wVoyG2gI6jnwwvuHRm8VNLy5Q9Jxeoa5ttgU+tcUMQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=PJ9EUGNHtfA+W7x3xd/LjYpIWyegk6keBgUrVitizzbfGboMkzddWDCHm/FgbWAqv Ma64n9YzG2ejZp+7rPDSOlR2iQ8OzDguFDLSJMjzgX3s2flkHMAy9sPadWfsl3YtQT FRWRtEf/7uK8jns0C9J+fmNrIYdcJK/BOypfcGKhf4tvyTIu2zDeLtR9A29YHiOQP7 DEpstOJgNDXjf6ueQKRye04P0rp6qc9M5wG4iho8TBwqODha+DpuMGtt/xiqdHYpKo OOqFHua8B51oXmoFMDkBhtX0KO+I1WpXwB1oueeac2e8wxYZd793LpUbV62wvkJOCo e/M61ybdcFHrQ== Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9E9A662239 for ; Fri, 23 Sep 2022 14:55:52 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20210112.gappssmtp.com header.i=@baylibre-com.20210112.gappssmtp.com header.b="uIibL/oI"; dkim-atps=neutral Received: by mail-wm1-x331.google.com with SMTP id u16-20020a05600c211000b003b5152ebf09so702537wml.5 for ; Fri, 23 Sep 2022 05:55:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=NMu1jY3op4sydSlNEwiugx8CLjbQpuEjMz2NJ3tLmiE=; b=uIibL/oIR9wlGb4Gzka72dI+ESh9kUnh5YBvgTCkxYkSCkIboeZCdNWizNuN2YQNsR CC5JCNWeldhlmfllCDNpKb333hfpWoPX/A3WTAkb+9I5diL9Q9hRwzapg4rxGuvzOTwf DwILZiqOJr0qnO8eyppHg+OWD3vOfxlnfAKmCwN5GsXzigx5Pjk/a8dfH2aJPPDEBdNd c4dXu/a9oL0fEqfPwbJR5wT+IEC6F7AQrktwJ3b1A8cu3bfPhLn+PIy3hZnbzv6lDC1N IMs1RADsITNj7g+txoQWwUYcJ5/KhuIF/YQv1Mt9NkoZbRAurLjIEEpbrHCIqaEIwvBl iVjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=NMu1jY3op4sydSlNEwiugx8CLjbQpuEjMz2NJ3tLmiE=; b=KvR0GyfR+jK26hBYzgCNmTMUuTGyH7yMon8pUOqV4XkKR/1XpvRw942Vy7jthA7Q1l 05qjdquSS99DjREs74RhI1evxHxi3j+XI6uRS1JZAelQR2nvnjlcUJpUj9Ml897VHCVl WzDnlE/sGOXMBqTKFepF71NLG1xqIWMfvKIc/LAoC1c/h0FLkGmr9MB/xwY7OGLCf9nP cVl7FfPs8W0xcGhN067NcDflG8p8VQiUuOVoM0YWLM+qQmpZKyjR5wZgQal5VL9e65oP HYb4iXWhoTq799zDy7RR/6/oPlyJvmW0Ue5n0l5D4SYRD1u26m04wPNSuEdFJmCMTILA 1m6g== X-Gm-Message-State: ACrzQf1L9C42mTXkMNdMVw/JYzuObHqwA6rLI6lPAzcEFSeJNWWrgd0O vHssLtkp5rmHnii1uz1+RcHl1ui74PovHA== X-Google-Smtp-Source: AMsMyM4Q/ZyA4g0lSW47E9zjJCBccF66PO01RaEgGSCzJkSS3cf7kQb3ylyfpvirM3nxwcwe5DzTAw== X-Received: by 2002:a05:600c:42c9:b0:3b4:b0c0:d614 with SMTP id j9-20020a05600c42c900b003b4b0c0d614mr12825172wme.90.1663937751942; Fri, 23 Sep 2022 05:55:51 -0700 (PDT) Received: from BL087.. ([2a01:e34:eea9:e630:db4f:e82d:37b3:6a74]) by smtp.gmail.com with ESMTPSA id bn27-20020a056000061b00b0022762b0e2a2sm8251353wrb.6.2022.09.23.05.55.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Sep 2022 05:55:51 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 23 Sep 2022 14:55:46 +0200 Message-Id: <20220923125546.903671-2-fsylvestre@baylibre.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220923125546.903671-1-fsylvestre@baylibre.com> References: <20220923125546.903671-1-fsylvestre@baylibre.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/1] pipeline: rkisp1: Implement Bayer formats support X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Sylvestre via libcamera-devel From: Florian Sylvestre Reply-To: Florian Sylvestre Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Implement raw mode for RkISP1: - The ISP resizer doesn't support raw formats, so when in raw mode we force the output resolution to be the same as the sensor one. - In raw mode, the ISP is bypassed, so we never get statistics buffers. This means that the IPA is never instructed to set the controls nor the metadata. Add a completeRaw() function to the IPA for the purpose of instructing the IPA to set controls and metadata when a frame is ready, as opposed to when the statistics are ready. We also need to skip queueing the stats buffer when in raw mode to prevent the statistics bufferReady slot to be triggered at stream off. Signed-off-by: Florian Sylvestre Signed-off-by: Paul Elder --- include/libcamera/ipa/rkisp1.mojom | 1 + src/ipa/rkisp1/rkisp1.cpp | 10 +++ src/libcamera/pipeline/rkisp1/rkisp1.cpp | 63 ++++++++++++++++++- src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 51 ++++++++++++++- 4 files changed, 120 insertions(+), 5 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index eaf3955e..931ef357 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -27,6 +27,7 @@ interface IPARkISP1Interface { [async] fillParamsBuffer(uint32 frame, uint32 bufferId); [async] processStatsBuffer(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); + [async] completeRaw(uint32 frame); }; interface IPARkISP1EventInterface { diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 61a091e6..4c784e8f 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -63,6 +63,7 @@ public: void fillParamsBuffer(const uint32_t frame, const uint32_t bufferId) override; void processStatsBuffer(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; + void completeRaw(const uint32_t frame) override; protected: std::string logPrefix() const override; @@ -352,6 +353,15 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId prepareMetadata(frame, aeState); } +void IPARkISP1::completeRaw(const uint32_t frame) +{ + unsigned int aeState = 0; + + setControls(frame); + + prepareMetadata(frame, aeState); +} + void IPARkISP1::setControls(unsigned int frame) { RkISP1FrameContext &frameContext = context_.frameContexts.get(frame); diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 25fbf9f1..bade024d 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -182,6 +182,7 @@ private: std::unique_ptr csi_; bool hasSelfPath_; + bool isRaw_; RkISP1MainPath mainPath_; RkISP1SelfPath selfPath_; @@ -363,7 +364,9 @@ void RkISP1CameraData::paramFilled(unsigned int frame) return; pipe->param_->queueBuffer(info->paramBuffer); - pipe->stat_->queueBuffer(info->statBuffer); + + if (!pipe->isRaw_) + pipe->stat_->queueBuffer(info->statBuffer); if (info->mainPathBuffer) mainPath_->queueBuffer(info->mainPathBuffer); @@ -413,6 +416,21 @@ bool RkISP1CameraConfiguration::fitsAllPaths(const StreamConfiguration &cfg) return true; } +std::map rawFormats = { + { formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 }, + { formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 }, + { formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 }, + { formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 }, + { formats::SRGGB10, MEDIA_BUS_FMT_SRGGB10_1X10 }, + { formats::SGRBG10, MEDIA_BUS_FMT_SGRBG10_1X10 }, + { formats::SGBRG10, MEDIA_BUS_FMT_SGBRG10_1X10 }, + { formats::SBGGR10, MEDIA_BUS_FMT_SBGGR10_1X10 }, + { formats::SRGGB12, MEDIA_BUS_FMT_SRGGB12_1X12 }, + { formats::SGRBG12, MEDIA_BUS_FMT_SGRBG12_1X12 }, + { formats::SGBRG12, MEDIA_BUS_FMT_SGBRG12_1X12 }, + { formats::SBGGR12, MEDIA_BUS_FMT_SBGGR12_1X12 }, +}; + CameraConfiguration::Status RkISP1CameraConfiguration::validate() { const CameraSensor *sensor = data_->sensor_.get(); @@ -504,8 +522,23 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate() /* Select the sensor format. */ Size maxSize; - for (const StreamConfiguration &cfg : config_) + PixelFormat rawFormat; + bool hasRawFormat = false; + for (StreamConfiguration &cfg : config_) { + if (PixelFormatInfo::info(cfg.pixelFormat).colourEncoding == + PixelFormatInfo::ColourEncodingRAW) { + hasRawFormat = true; + rawFormat = cfg.pixelFormat; + + /* Raw format cannot be resized by ISP. */ + if (cfg.size != sensor->resolution()) { + cfg.size = sensor->resolution(); + status = Adjusted; + } + } + maxSize = std::max(maxSize, cfg.size); + } sensorFormat_ = sensor->getFormat({ MEDIA_BUS_FMT_SBGGR12_1X12, MEDIA_BUS_FMT_SGBRG12_1X12, @@ -520,6 +553,13 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate() MEDIA_BUS_FMT_SGRBG8_1X8, MEDIA_BUS_FMT_SRGGB8_1X8 }, maxSize); + + if (hasRawFormat) { + auto mbus = rawFormats.find(rawFormat); + if (mbus != rawFormats.end()) + sensorFormat_ = sensor->getFormat({ mbus->second }, maxSize); + } + if (sensorFormat_.size.isNull()) sensorFormat_.size = sensor->resolution(); @@ -659,8 +699,14 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c) << "ISP input pad configured with " << format << " crop " << rect; + const PixelFormat &streamFormat = config->at(0).pixelFormat; + const PixelFormatInfo &info = PixelFormatInfo::info(streamFormat); + isRaw_ = info.colourEncoding == PixelFormatInfo::ColourEncodingRAW; + /* YUYV8_2X8 is required on the ISP source path pad for YUV output. */ - format.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8; + if (!isRaw_) + format.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8; + LOG(RkISP1, Debug) << "Configuring ISP output pad with " << format << " crop " << rect; @@ -1152,6 +1198,17 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer) request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + if (isRaw_) { + ASSERT(activeCamera_); + RkISP1CameraData *data = cameraData(activeCamera_); + + RkISP1FrameInfo *info = data->frameInfo_.find(buffer); + if (!info) + return; + + data->ipa_->completeRaw(info->frame); + } + completeBuffer(request, buffer); tryCompleteRequest(request); } diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp index 2d38f0fb..0a33d9ed 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp @@ -132,6 +132,42 @@ int RkISP1Path::configure(const StreamConfiguration &config, case formats::NV21: ispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_1_5X8; break; + case formats::SRGGB8: + ispFormat.mbus_code = MEDIA_BUS_FMT_SRGGB8_1X8; + break; + case formats::SGRBG8: + ispFormat.mbus_code = MEDIA_BUS_FMT_SGRBG8_1X8; + break; + case formats::SGBRG8: + ispFormat.mbus_code = MEDIA_BUS_FMT_SGBRG8_1X8; + break; + case formats::SBGGR8: + ispFormat.mbus_code = MEDIA_BUS_FMT_SBGGR8_1X8; + break; + case formats::SRGGB10: + ispFormat.mbus_code = MEDIA_BUS_FMT_SRGGB10_1X10; + break; + case formats::SGRBG10: + ispFormat.mbus_code = MEDIA_BUS_FMT_SGRBG10_1X10; + break; + case formats::SGBRG10: + ispFormat.mbus_code = MEDIA_BUS_FMT_SGBRG10_1X10; + break; + case formats::SBGGR10: + ispFormat.mbus_code = MEDIA_BUS_FMT_SBGGR10_1X10; + break; + case formats::SRGGB12: + ispFormat.mbus_code = MEDIA_BUS_FMT_SRGGB12_1X12; + break; + case formats::SGRBG12: + ispFormat.mbus_code = MEDIA_BUS_FMT_SGRBG12_1X12; + break; + case formats::SGBRG12: + ispFormat.mbus_code = MEDIA_BUS_FMT_SGBRG12_1X12; + break; + case formats::SBGGR12: + ispFormat.mbus_code = MEDIA_BUS_FMT_SBGGR12_1X12; + break; default: ispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8; break; @@ -207,14 +243,25 @@ void RkISP1Path::stop() namespace { constexpr Size RKISP1_RSZ_MP_SRC_MIN{ 32, 16 }; constexpr Size RKISP1_RSZ_MP_SRC_MAX{ 4416, 3312 }; -constexpr std::array RKISP1_RSZ_MP_FORMATS{ +constexpr std::array RKISP1_RSZ_MP_FORMATS{ formats::YUYV, formats::NV16, formats::NV61, formats::NV21, formats::NV12, formats::R8, - /* \todo Add support for RAW formats. */ + formats::SRGGB8, + formats::SGRBG8, + formats::SGBRG8, + formats::SBGGR8, + formats::SRGGB10, + formats::SGRBG10, + formats::SGBRG10, + formats::SBGGR10, + formats::SRGGB12, + formats::SGRBG12, + formats::SGBRG12, + formats::SBGGR12, }; constexpr Size RKISP1_RSZ_SP_SRC_MIN{ 32, 16 };