From patchwork Mon Oct 3 14:36:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Sylvestre X-Patchwork-Id: 17506 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1C17EC327C for ; Mon, 3 Oct 2022 14:36:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E59C0601C8; Mon, 3 Oct 2022 16:36:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1664807806; bh=dXabrNMqDZI/0SIpS/swLgpvG3HuDTkoloBjXPgCTLw=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=eLUECYO/3z62Y4Ei2LEHTmsjCyEaBRfPhR/612EN4VJqQQmoA0zuseuv5onMRsM4C sHF11kN4ECAUlgyHq2Fb9+DraCB6bICv1Kt1PKD6g8jgreX4119TSEf6Mkw32l7CNK CZmg2z7/09BR2egBErfXr1NOeVmF2aB1pCtwNXYR6JNhqj79i92q6etVARQ7ZciPFE zqoaAXajQqCQpZH0RxgyVdEwaIVH6XenwIDUpYaJ4GUeq9mjgkvWOflf6ZG1bEiPbv UibXLVPDslD6gWzfM22fAVAb6GmMYgIzqbzt05iJlM+StQpuQGdJ091V4KmHkRTgRN hfqW58Sksctfg== Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 936DB601BC for ; Mon, 3 Oct 2022 16:36:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20210112.gappssmtp.com header.i=@baylibre-com.20210112.gappssmtp.com header.b="sGrkhzp3"; dkim-atps=neutral Received: by mail-wr1-x431.google.com with SMTP id f11so14404073wrm.6 for ; Mon, 03 Oct 2022 07:36:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=WWf3Y/AV7pPyo4N8jAJIBTBDWwWQYAeG9N74jPggf9k=; b=sGrkhzp3SU5MusW710MPzTTDvEkYVd3/r6eREzWmYnuNrVTDV03U63PCNKlO+0xGsG sTdFfJnz3M0JFLz0Cmi6WFHPCaMUtXS/sbok9XhrJXqIAFBJq3cZwpZxUh4BF0Y/dQ+C idXPywlxAdB1hzWVWG8icu2s7P3YLDrnYRMeD1bw0qSgWQhUK0hjeeaqtT33bbDL5za2 bIKkQYwYQ39I6wbPE7dsd/caljuwUk2Jn7CxWxmIFzcSZjrs9E4Hm9UtNcHiXXmoVDV/ +cpkvAXz0jQxTX9dZN4fYiQcvLMyAASP5Y8E/69M2hEE5oHPnqOjs5N24FBu0N/ez/pC eThg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=WWf3Y/AV7pPyo4N8jAJIBTBDWwWQYAeG9N74jPggf9k=; b=lYhgtZzeKK1lYxvj91492/rOs5Lv0EpEvmqHQjqTFRagR2znBNobjVTx3ZYVMKTPfT 902Hd6RiLaK0LCfr8ewwSeXr6MdUom12qCMyPig9hS5c8obzAAFRpdc4r2fVH+xqXqmk y+0bj5VPF/IZD1OB1MqDo0IaCijGebA3NwehBRxiqOHmLQCnEFOg9UzIPjUAfk7RSY9f X213ct+938XMZ++0iRNXxJ+ifxT3jT98EmTbh5fwK+fRSURPxFJWmC8dPCXNk6D4OGSD c0N4WLNsBPP3vB8JSatP2bR0wvzaV/dFoARu5aL7wEQGvWqfh0Ut47+qhVPEwWtMZCHO KNew== X-Gm-Message-State: ACrzQf2yMDaowsDpSD9YFnkZCRIsDCZoHaYw2Qx5dkPxjdmHcQsun2JK JV5c4LlhXHllsq/eBmwBbbjoVfZp3fzBMA== X-Google-Smtp-Source: AMsMyM4sS7h1GsEDA1gFj2sSXpA8Lxq6QhUmmjmrXpi3ZsKEPXGjMnCYIqt9rtIgwwr4aO3XHcyd9w== X-Received: by 2002:a5d:47a1:0:b0:22a:2979:4899 with SMTP id 1-20020a5d47a1000000b0022a29794899mr12694265wrb.349.1664807804061; Mon, 03 Oct 2022 07:36:44 -0700 (PDT) Received: from BL087.. (rqp06-2-78-234-158-99.fbx.proxad.net. [78.234.158.99]) by smtp.gmail.com with ESMTPSA id p8-20020adf9d88000000b002252884cc91sm10202590wre.43.2022.10.03.07.36.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Oct 2022 07:36:43 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Mon, 3 Oct 2022 16:36:38 +0200 Message-Id: <20221003143638.604343-2-fsylvestre@baylibre.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221003143638.604343-1-fsylvestre@baylibre.com> References: <20221003143638.604343-1-fsylvestre@baylibre.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/1] pipeline: rkisp1: Implement Bayer formats support X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Sylvestre via libcamera-devel From: Florian Sylvestre Reply-To: Florian Sylvestre Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Implement raw mode for RkISP1: - The ISP resizer doesn't support raw formats, so when in raw mode we force the output resolution to be the same as the sensor one. - In raw mode, the ISP is bypassed, so we never get statistics buffers. This means that the IPA is never instructed to set the controls nor the metadata. Add a completeRaw() function to the IPA for the purpose of instructing the IPA to set controls and metadata when a frame is ready, as opposed to when the statistics are ready. We also need to skip queueing the stats buffer when in raw mode to prevent the statistics bufferReady slot to be triggered at stream off. Signed-off-by: Florian Sylvestre Signed-off-by: Paul Elder --- include/libcamera/ipa/rkisp1.mojom | 1 + src/ipa/rkisp1/rkisp1.cpp | 10 +++ src/libcamera/pipeline/rkisp1/rkisp1.cpp | 64 ++++++++++++++++++- src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 44 ++++++++++--- 4 files changed, 106 insertions(+), 13 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index eaf3955e..931ef357 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -27,6 +27,7 @@ interface IPARkISP1Interface { [async] fillParamsBuffer(uint32 frame, uint32 bufferId); [async] processStatsBuffer(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); + [async] completeRaw(uint32 frame); }; interface IPARkISP1EventInterface { diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 1335e3d1..4fd12333 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -63,6 +63,7 @@ public: void fillParamsBuffer(const uint32_t frame, const uint32_t bufferId) override; void processStatsBuffer(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; + void completeRaw(const uint32_t frame) override; protected: std::string logPrefix() const override; @@ -349,6 +350,15 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId prepareMetadata(frame, aeState); } +void IPARkISP1::completeRaw(const uint32_t frame) +{ + unsigned int aeState = 0; + + setControls(frame); + + prepareMetadata(frame, aeState); +} + void IPARkISP1::setControls(unsigned int frame) { /* diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 25fbf9f1..80ca5547 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -182,6 +182,7 @@ private: std::unique_ptr csi_; bool hasSelfPath_; + bool isRaw_; RkISP1MainPath mainPath_; RkISP1SelfPath selfPath_; @@ -363,7 +364,10 @@ void RkISP1CameraData::paramFilled(unsigned int frame) return; pipe->param_->queueBuffer(info->paramBuffer); - pipe->stat_->queueBuffer(info->statBuffer); + + if (!pipe->isRaw_) { + pipe->stat_->queueBuffer(info->statBuffer); + } if (info->mainPathBuffer) mainPath_->queueBuffer(info->mainPathBuffer); @@ -413,6 +417,21 @@ bool RkISP1CameraConfiguration::fitsAllPaths(const StreamConfiguration &cfg) return true; } +std::map rawFormats = { + { formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 }, + { formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 }, + { formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 }, + { formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 }, + { formats::SBGGR10, MEDIA_BUS_FMT_SBGGR10_1X10 }, + { formats::SGBRG10, MEDIA_BUS_FMT_SGBRG10_1X10 }, + { formats::SGRBG10, MEDIA_BUS_FMT_SGRBG10_1X10 }, + { formats::SRGGB10, MEDIA_BUS_FMT_SRGGB10_1X10 }, + { formats::SBGGR12, MEDIA_BUS_FMT_SBGGR12_1X12 }, + { formats::SGBRG12, MEDIA_BUS_FMT_SGBRG12_1X12 }, + { formats::SGRBG12, MEDIA_BUS_FMT_SGRBG12_1X12 }, + { formats::SRGGB12, MEDIA_BUS_FMT_SRGGB12_1X12 }, +}; + CameraConfiguration::Status RkISP1CameraConfiguration::validate() { const CameraSensor *sensor = data_->sensor_.get(); @@ -504,8 +523,23 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate() /* Select the sensor format. */ Size maxSize; - for (const StreamConfiguration &cfg : config_) + PixelFormat rawFormat; + bool hasRawFormat = false; + for (StreamConfiguration &cfg : config_) { + if (PixelFormatInfo::info(cfg.pixelFormat).colourEncoding == + PixelFormatInfo::ColourEncodingRAW) { + hasRawFormat = true; + rawFormat = cfg.pixelFormat; + + /* Raw format cannot be resized by ISP. */ + if (cfg.size != sensor->resolution()) { + cfg.size = sensor->resolution(); + status = Adjusted; + } + } + maxSize = std::max(maxSize, cfg.size); + } sensorFormat_ = sensor->getFormat({ MEDIA_BUS_FMT_SBGGR12_1X12, MEDIA_BUS_FMT_SGBRG12_1X12, @@ -520,6 +554,13 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate() MEDIA_BUS_FMT_SGRBG8_1X8, MEDIA_BUS_FMT_SRGGB8_1X8 }, maxSize); + + if (hasRawFormat) { + auto mbus = rawFormats.find(rawFormat); + if (mbus != rawFormats.end()) + sensorFormat_ = sensor->getFormat({ mbus->second }, maxSize); + } + if (sensorFormat_.size.isNull()) sensorFormat_.size = sensor->resolution(); @@ -659,8 +700,14 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c) << "ISP input pad configured with " << format << " crop " << rect; + const PixelFormat &streamFormat = config->at(0).pixelFormat; + const PixelFormatInfo &info = PixelFormatInfo::info(streamFormat); + isRaw_ = info.colourEncoding == PixelFormatInfo::ColourEncodingRAW; + /* YUYV8_2X8 is required on the ISP source path pad for YUV output. */ - format.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8; + if (!isRaw_) + format.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8; + LOG(RkISP1, Debug) << "Configuring ISP output pad with " << format << " crop " << rect; @@ -1152,6 +1199,17 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer) request->metadata().set(controls::SensorTimestamp, buffer->metadata().timestamp); + if (isRaw_) { + ASSERT(activeCamera_); + RkISP1CameraData *data = cameraData(activeCamera_); + + RkISP1FrameInfo *info = data->frameInfo_.find(buffer); + if (!info) + return; + + data->ipa_->completeRaw(info->frame); + } + completeBuffer(request, buffer); tryCompleteRequest(request); } diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp index 2d38f0fb..0ae006e8 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp @@ -101,6 +101,23 @@ CameraConfiguration::Status RkISP1Path::validate(StreamConfiguration *cfg) return status; } +std::map formatToMediaBus = { + { formats::NV12, MEDIA_BUS_FMT_YUYV8_1_5X8 }, + { formats::NV21, MEDIA_BUS_FMT_YUYV8_1_5X8 }, + { formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 }, + { formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 }, + { formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 }, + { formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 }, + { formats::SBGGR10, MEDIA_BUS_FMT_SBGGR10_1X10 }, + { formats::SGBRG10, MEDIA_BUS_FMT_SGBRG10_1X10 }, + { formats::SGRBG10, MEDIA_BUS_FMT_SGRBG10_1X10 }, + { formats::SRGGB10, MEDIA_BUS_FMT_SRGGB10_1X10 }, + { formats::SBGGR12, MEDIA_BUS_FMT_SBGGR12_1X12 }, + { formats::SGBRG12, MEDIA_BUS_FMT_SGBRG12_1X12 }, + { formats::SGRBG12, MEDIA_BUS_FMT_SGRBG12_1X12 }, + { formats::SRGGB12, MEDIA_BUS_FMT_SRGGB12_1X12 }, +}; + int RkISP1Path::configure(const StreamConfiguration &config, const V4L2SubdeviceFormat &inputFormat) { @@ -127,15 +144,11 @@ int RkISP1Path::configure(const StreamConfiguration &config, << "Configuring " << name_ << " resizer output pad with " << ispFormat; - switch (config.pixelFormat) { - case formats::NV12: - case formats::NV21: - ispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_1_5X8; - break; - default: + auto mbus = formatToMediaBus.find(config.pixelFormat); + if (mbus != formatToMediaBus.end()) + ispFormat.mbus_code = mbus->second; + else ispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8; - break; - } ret = resizer_->setFormat(1, &ispFormat); if (ret < 0) @@ -207,14 +220,25 @@ void RkISP1Path::stop() namespace { constexpr Size RKISP1_RSZ_MP_SRC_MIN{ 32, 16 }; constexpr Size RKISP1_RSZ_MP_SRC_MAX{ 4416, 3312 }; -constexpr std::array RKISP1_RSZ_MP_FORMATS{ +constexpr std::array RKISP1_RSZ_MP_FORMATS{ formats::YUYV, formats::NV16, formats::NV61, formats::NV21, formats::NV12, formats::R8, - /* \todo Add support for RAW formats. */ + formats::SRGGB8, + formats::SGRBG8, + formats::SGBRG8, + formats::SBGGR8, + formats::SRGGB10, + formats::SGRBG10, + formats::SGBRG10, + formats::SBGGR10, + formats::SRGGB12, + formats::SGRBG12, + formats::SGBRG12, + formats::SBGGR12, }; constexpr Size RKISP1_RSZ_SP_SRC_MIN{ 32, 16 };