{"id":17392,"url":"https://patchwork.libcamera.org/api/1.1/patches/17392/?format=json","web_url":"https://patchwork.libcamera.org/patch/17392/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220923125546.903671-2-fsylvestre@baylibre.com>","date":"2022-09-23T12:55:46","name":"[libcamera-devel,1/1] pipeline: rkisp1: Implement Bayer formats support","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"bd1ec2637ac5cdad82edfaafa355fea38b0c3e38","submitter":{"id":123,"url":"https://patchwork.libcamera.org/api/1.1/people/123/?format=json","name":"Florian Sylvestre","email":"fsylvestre@baylibre.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/17392/mbox/","series":[{"id":3498,"url":"https://patchwork.libcamera.org/api/1.1/series/3498/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3498","date":"2022-09-23T12:55:45","name":"Add Bayer format support for RkISP1","version":1,"mbox":"https://patchwork.libcamera.org/series/3498/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/17392/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/17392/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 442F7C327E\n\tfor ;\n\tFri, 23 Sep 2022 12:55:55 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E66786223E;\n\tFri, 23 Sep 2022 14:55:54 +0200 (CEST)","from mail-wm1-x331.google.com (mail-wm1-x331.google.com\n\t[IPv6:2a00:1450:4864:20::331])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9E9A662239\n\tfor ;\n\tFri, 23 Sep 2022 14:55:52 +0200 (CEST)","by mail-wm1-x331.google.com with SMTP id\n\tu16-20020a05600c211000b003b5152ebf09so702537wml.5\n\tfor ;\n\tFri, 23 Sep 2022 05:55:52 -0700 (PDT)","from BL087.. ([2a01:e34:eea9:e630:db4f:e82d:37b3:6a74])\n\tby smtp.gmail.com with ESMTPSA id\n\tbn27-20020a056000061b00b0022762b0e2a2sm8251353wrb.6.2022.09.23.05.55.51\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 23 Sep 2022 05:55:51 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1663937754;\n\tbh=6wVoyG2gI6jnwwvuHRm8VNLy5Q9Jxeoa5ttgU+tcUMQ=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=PJ9EUGNHtfA+W7x3xd/LjYpIWyegk6keBgUrVitizzbfGboMkzddWDCHm/FgbWAqv\n\tMa64n9YzG2ejZp+7rPDSOlR2iQ8OzDguFDLSJMjzgX3s2flkHMAy9sPadWfsl3YtQT\n\tFRWRtEf/7uK8jns0C9J+fmNrIYdcJK/BOypfcGKhf4tvyTIu2zDeLtR9A29YHiOQP7\n\tDEpstOJgNDXjf6ueQKRye04P0rp6qc9M5wG4iho8TBwqODha+DpuMGtt/xiqdHYpKo\n\tOOqFHua8B51oXmoFMDkBhtX0KO+I1WpXwB1oueeac2e8wxYZd793LpUbV62wvkJOCo\n\te/M61ybdcFHrQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=baylibre-com.20210112.gappssmtp.com; s=20210112;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date;\n\tbh=NMu1jY3op4sydSlNEwiugx8CLjbQpuEjMz2NJ3tLmiE=;\n\tb=uIibL/oIR9wlGb4Gzka72dI+ESh9kUnh5YBvgTCkxYkSCkIboeZCdNWizNuN2YQNsR\n\tCC5JCNWeldhlmfllCDNpKb333hfpWoPX/A3WTAkb+9I5diL9Q9hRwzapg4rxGuvzOTwf\n\tDwILZiqOJr0qnO8eyppHg+OWD3vOfxlnfAKmCwN5GsXzigx5Pjk/a8dfH2aJPPDEBdNd\n\tc4dXu/a9oL0fEqfPwbJR5wT+IEC6F7AQrktwJ3b1A8cu3bfPhLn+PIy3hZnbzv6lDC1N\n\tIMs1RADsITNj7g+txoQWwUYcJ5/KhuIF/YQv1Mt9NkoZbRAurLjIEEpbrHCIqaEIwvBl\n\tiVjQ=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected)\n\theader.d=baylibre-com.20210112.gappssmtp.com\n\theader.i=@baylibre-com.20210112.gappssmtp.com header.b=\"uIibL/oI\"; \n\tdkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date;\n\tbh=NMu1jY3op4sydSlNEwiugx8CLjbQpuEjMz2NJ3tLmiE=;\n\tb=KvR0GyfR+jK26hBYzgCNmTMUuTGyH7yMon8pUOqV4XkKR/1XpvRw942Vy7jthA7Q1l\n\t05qjdquSS99DjREs74RhI1evxHxi3j+XI6uRS1JZAelQR2nvnjlcUJpUj9Ml897VHCVl\n\tWzDnlE/sGOXMBqTKFepF71NLG1xqIWMfvKIc/LAoC1c/h0FLkGmr9MB/xwY7OGLCf9nP\n\tcVl7FfPs8W0xcGhN067NcDflG8p8VQiUuOVoM0YWLM+qQmpZKyjR5wZgQal5VL9e65oP\n\tHYb4iXWhoTq799zDy7RR/6/oPlyJvmW0Ue5n0l5D4SYRD1u26m04wPNSuEdFJmCMTILA\n\t1m6g==","X-Gm-Message-State":"ACrzQf1L9C42mTXkMNdMVw/JYzuObHqwA6rLI6lPAzcEFSeJNWWrgd0O\n\tvHssLtkp5rmHnii1uz1+RcHl1ui74PovHA==","X-Google-Smtp-Source":"AMsMyM4Q/ZyA4g0lSW47E9zjJCBccF66PO01RaEgGSCzJkSS3cf7kQb3ylyfpvirM3nxwcwe5DzTAw==","X-Received":"by 2002:a05:600c:42c9:b0:3b4:b0c0:d614 with SMTP id\n\tj9-20020a05600c42c900b003b4b0c0d614mr12825172wme.90.1663937751942; \n\tFri, 23 Sep 2022 05:55:51 -0700 (PDT)","To":"libcamera-devel@lists.libcamera.org","Date":"Fri, 23 Sep 2022 14:55:46 +0200","Message-Id":"<20220923125546.903671-2-fsylvestre@baylibre.com>","X-Mailer":"git-send-email 2.34.1","In-Reply-To":"<20220923125546.903671-1-fsylvestre@baylibre.com>","References":"<20220923125546.903671-1-fsylvestre@baylibre.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 1/1] pipeline: rkisp1: Implement Bayer\n\tformats support","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Florian Sylvestre via libcamera-devel\n\t","Reply-To":"Florian Sylvestre ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Implement raw mode for RkISP1:\n- The ISP resizer doesn't support raw formats, so when in raw mode we force the\noutput resolution to be the same as the sensor one.\n- In raw mode, the ISP is bypassed, so we never get statistics buffers.\nThis means that the IPA is never instructed to set the controls nor the\nmetadata.\nAdd a completeRaw() function to the IPA for the purpose of instructing the IPA\nto set controls and metadata when a frame is ready, as opposed to when the\nstatistics are ready.\nWe also need to skip queueing the stats buffer when in raw mode to prevent the\nstatistics bufferReady slot to be triggered at stream off.\n\nSigned-off-by: Florian Sylvestre \nSigned-off-by: Paul Elder \n---\n include/libcamera/ipa/rkisp1.mojom | 1 +\n src/ipa/rkisp1/rkisp1.cpp | 10 +++\n src/libcamera/pipeline/rkisp1/rkisp1.cpp | 63 ++++++++++++++++++-\n src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 51 ++++++++++++++-\n 4 files changed, 120 insertions(+), 5 deletions(-)","diff":"diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom\nindex eaf3955e..931ef357 100644\n--- a/include/libcamera/ipa/rkisp1.mojom\n+++ b/include/libcamera/ipa/rkisp1.mojom\n@@ -27,6 +27,7 @@ interface IPARkISP1Interface {\n \t[async] fillParamsBuffer(uint32 frame, uint32 bufferId);\n \t[async] processStatsBuffer(uint32 frame, uint32 bufferId,\n \t\t\t\t libcamera.ControlList sensorControls);\n+\t[async] completeRaw(uint32 frame);\n };\n \n interface IPARkISP1EventInterface {\ndiff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp\nindex 61a091e6..4c784e8f 100644\n--- a/src/ipa/rkisp1/rkisp1.cpp\n+++ b/src/ipa/rkisp1/rkisp1.cpp\n@@ -63,6 +63,7 @@ public:\n \tvoid fillParamsBuffer(const uint32_t frame, const uint32_t bufferId) override;\n \tvoid processStatsBuffer(const uint32_t frame, const uint32_t bufferId,\n \t\t\t\tconst ControlList &sensorControls) override;\n+\tvoid completeRaw(const uint32_t frame) override;\n \n protected:\n \tstd::string logPrefix() const override;\n@@ -352,6 +353,15 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId\n \tprepareMetadata(frame, aeState);\n }\n \n+void IPARkISP1::completeRaw(const uint32_t frame)\n+{\n+\tunsigned int aeState = 0;\n+\n+\tsetControls(frame);\n+\n+\tprepareMetadata(frame, aeState);\n+}\n+\n void IPARkISP1::setControls(unsigned int frame)\n {\n \tRkISP1FrameContext &frameContext = context_.frameContexts.get(frame);\ndiff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\nindex 25fbf9f1..bade024d 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n@@ -182,6 +182,7 @@ private:\n \tstd::unique_ptr csi_;\n \n \tbool hasSelfPath_;\n+\tbool isRaw_;\n \n \tRkISP1MainPath mainPath_;\n \tRkISP1SelfPath selfPath_;\n@@ -363,7 +364,9 @@ void RkISP1CameraData::paramFilled(unsigned int frame)\n \t\treturn;\n \n \tpipe->param_->queueBuffer(info->paramBuffer);\n-\tpipe->stat_->queueBuffer(info->statBuffer);\n+\n+\tif (!pipe->isRaw_)\n+\t\tpipe->stat_->queueBuffer(info->statBuffer);\n \n \tif (info->mainPathBuffer)\n \t\tmainPath_->queueBuffer(info->mainPathBuffer);\n@@ -413,6 +416,21 @@ bool RkISP1CameraConfiguration::fitsAllPaths(const StreamConfiguration &cfg)\n \treturn true;\n }\n \n+std::map rawFormats = {\n+\t{ formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 },\n+\t{ formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 },\n+\t{ formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 },\n+\t{ formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 },\n+\t{ formats::SRGGB10, MEDIA_BUS_FMT_SRGGB10_1X10 },\n+\t{ formats::SGRBG10, MEDIA_BUS_FMT_SGRBG10_1X10 },\n+\t{ formats::SGBRG10, MEDIA_BUS_FMT_SGBRG10_1X10 },\n+\t{ formats::SBGGR10, MEDIA_BUS_FMT_SBGGR10_1X10 },\n+\t{ formats::SRGGB12, MEDIA_BUS_FMT_SRGGB12_1X12 },\n+\t{ formats::SGRBG12, MEDIA_BUS_FMT_SGRBG12_1X12 },\n+\t{ formats::SGBRG12, MEDIA_BUS_FMT_SGBRG12_1X12 },\n+\t{ formats::SBGGR12, MEDIA_BUS_FMT_SBGGR12_1X12 },\n+};\n+\n CameraConfiguration::Status RkISP1CameraConfiguration::validate()\n {\n \tconst CameraSensor *sensor = data_->sensor_.get();\n@@ -504,8 +522,23 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate()\n \n \t/* Select the sensor format. */\n \tSize maxSize;\n-\tfor (const StreamConfiguration &cfg : config_)\n+\tPixelFormat rawFormat;\n+\tbool hasRawFormat = false;\n+\tfor (StreamConfiguration &cfg : config_) {\n+\t\tif (PixelFormatInfo::info(cfg.pixelFormat).colourEncoding ==\n+\t\t PixelFormatInfo::ColourEncodingRAW) {\n+\t\t\thasRawFormat = true;\n+\t\t\trawFormat = cfg.pixelFormat;\n+\n+\t\t\t/* Raw format cannot be resized by ISP. */\n+\t\t\tif (cfg.size != sensor->resolution()) {\n+\t\t\t\tcfg.size = sensor->resolution();\n+\t\t\t\tstatus = Adjusted;\n+\t\t\t}\n+\t\t}\n+\n \t\tmaxSize = std::max(maxSize, cfg.size);\n+\t}\n \n \tsensorFormat_ = sensor->getFormat({ MEDIA_BUS_FMT_SBGGR12_1X12,\n \t\t\t\t\t MEDIA_BUS_FMT_SGBRG12_1X12,\n@@ -520,6 +553,13 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate()\n \t\t\t\t\t MEDIA_BUS_FMT_SGRBG8_1X8,\n \t\t\t\t\t MEDIA_BUS_FMT_SRGGB8_1X8 },\n \t\t\t\t\t maxSize);\n+\n+\tif (hasRawFormat) {\n+\t\tauto mbus = rawFormats.find(rawFormat);\n+\t\tif (mbus != rawFormats.end())\n+\t\t\tsensorFormat_ = sensor->getFormat({ mbus->second }, maxSize);\n+\t}\n+\n \tif (sensorFormat_.size.isNull())\n \t\tsensorFormat_.size = sensor->resolution();\n \n@@ -659,8 +699,14 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n \t\t<< \"ISP input pad configured with \" << format\n \t\t<< \" crop \" << rect;\n \n+\tconst PixelFormat &streamFormat = config->at(0).pixelFormat;\n+\tconst PixelFormatInfo &info = PixelFormatInfo::info(streamFormat);\n+\tisRaw_ = info.colourEncoding == PixelFormatInfo::ColourEncodingRAW;\n+\n \t/* YUYV8_2X8 is required on the ISP source path pad for YUV output. */\n-\tformat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8;\n+\tif (!isRaw_)\n+\t\tformat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8;\n+\n \tLOG(RkISP1, Debug)\n \t\t<< \"Configuring ISP output pad with \" << format\n \t\t<< \" crop \" << rect;\n@@ -1152,6 +1198,17 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer)\n \trequest->metadata().set(controls::SensorTimestamp,\n \t\t\t\tbuffer->metadata().timestamp);\n \n+\tif (isRaw_) {\n+\t\tASSERT(activeCamera_);\n+\t\tRkISP1CameraData *data = cameraData(activeCamera_);\n+\n+\t\tRkISP1FrameInfo *info = data->frameInfo_.find(buffer);\n+\t\tif (!info)\n+\t\t\treturn;\n+\n+\t\tdata->ipa_->completeRaw(info->frame);\n+\t}\n+\n \tcompleteBuffer(request, buffer);\n \ttryCompleteRequest(request);\n }\ndiff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\nindex 2d38f0fb..0a33d9ed 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n@@ -132,6 +132,42 @@ int RkISP1Path::configure(const StreamConfiguration &config,\n \tcase formats::NV21:\n \t\tispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_1_5X8;\n \t\tbreak;\n+\tcase formats::SRGGB8:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SRGGB8_1X8;\n+\t\tbreak;\n+\tcase formats::SGRBG8:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SGRBG8_1X8;\n+\t\tbreak;\n+\tcase formats::SGBRG8:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SGBRG8_1X8;\n+\t\tbreak;\n+\tcase formats::SBGGR8:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SBGGR8_1X8;\n+\t\tbreak;\n+\tcase formats::SRGGB10:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SRGGB10_1X10;\n+\t\tbreak;\n+\tcase formats::SGRBG10:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SGRBG10_1X10;\n+\t\tbreak;\n+\tcase formats::SGBRG10:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SGBRG10_1X10;\n+\t\tbreak;\n+\tcase formats::SBGGR10:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SBGGR10_1X10;\n+\t\tbreak;\n+\tcase formats::SRGGB12:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SRGGB12_1X12;\n+\t\tbreak;\n+\tcase formats::SGRBG12:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SGRBG12_1X12;\n+\t\tbreak;\n+\tcase formats::SGBRG12:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SGBRG12_1X12;\n+\t\tbreak;\n+\tcase formats::SBGGR12:\n+\t\tispFormat.mbus_code = MEDIA_BUS_FMT_SBGGR12_1X12;\n+\t\tbreak;\n \tdefault:\n \t\tispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8;\n \t\tbreak;\n@@ -207,14 +243,25 @@ void RkISP1Path::stop()\n namespace {\n constexpr Size RKISP1_RSZ_MP_SRC_MIN{ 32, 16 };\n constexpr Size RKISP1_RSZ_MP_SRC_MAX{ 4416, 3312 };\n-constexpr std::array RKISP1_RSZ_MP_FORMATS{\n+constexpr std::array RKISP1_RSZ_MP_FORMATS{\n \tformats::YUYV,\n \tformats::NV16,\n \tformats::NV61,\n \tformats::NV21,\n \tformats::NV12,\n \tformats::R8,\n-\t/* \\todo Add support for RAW formats. */\n+\tformats::SRGGB8,\n+\tformats::SGRBG8,\n+\tformats::SGBRG8,\n+\tformats::SBGGR8,\n+\tformats::SRGGB10,\n+\tformats::SGRBG10,\n+\tformats::SGBRG10,\n+\tformats::SBGGR10,\n+\tformats::SRGGB12,\n+\tformats::SGRBG12,\n+\tformats::SGBRG12,\n+\tformats::SBGGR12,\n };\n \n constexpr Size RKISP1_RSZ_SP_SRC_MIN{ 32, 16 };\n","prefixes":["libcamera-devel","1/1"]}