{"id":17506,"url":"https://patchwork.libcamera.org/api/1.1/patches/17506/?format=json","web_url":"https://patchwork.libcamera.org/patch/17506/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20221003143638.604343-2-fsylvestre@baylibre.com>","date":"2022-10-03T14:36:38","name":"[libcamera-devel,v2,1/1] pipeline: rkisp1: Implement Bayer formats support","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"0b8ffa82b4aa97b4b2542450cddb51c907f1a44c","submitter":{"id":123,"url":"https://patchwork.libcamera.org/api/1.1/people/123/?format=json","name":"Florian Sylvestre","email":"fsylvestre@baylibre.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/17506/mbox/","series":[{"id":3526,"url":"https://patchwork.libcamera.org/api/1.1/series/3526/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3526","date":"2022-10-03T14:36:37","name":"Add Bayer format support for RkISP1","version":2,"mbox":"https://patchwork.libcamera.org/series/3526/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/17506/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/17506/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1C17EC327C\n\tfor ;\n\tMon, 3 Oct 2022 14:36:47 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E59C0601C8;\n\tMon, 3 Oct 2022 16:36:45 +0200 (CEST)","from mail-wr1-x431.google.com (mail-wr1-x431.google.com\n\t[IPv6:2a00:1450:4864:20::431])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 936DB601BC\n\tfor ;\n\tMon, 3 Oct 2022 16:36:44 +0200 (CEST)","by mail-wr1-x431.google.com with SMTP id f11so14404073wrm.6\n\tfor ;\n\tMon, 03 Oct 2022 07:36:44 -0700 (PDT)","from BL087.. (rqp06-2-78-234-158-99.fbx.proxad.net.\n\t[78.234.158.99]) by smtp.gmail.com with ESMTPSA id\n\tp8-20020adf9d88000000b002252884cc91sm10202590wre.43.2022.10.03.07.36.43\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tMon, 03 Oct 2022 07:36:43 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1664807806;\n\tbh=dXabrNMqDZI/0SIpS/swLgpvG3HuDTkoloBjXPgCTLw=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=eLUECYO/3z62Y4Ei2LEHTmsjCyEaBRfPhR/612EN4VJqQQmoA0zuseuv5onMRsM4C\n\tsHF11kN4ECAUlgyHq2Fb9+DraCB6bICv1Kt1PKD6g8jgreX4119TSEf6Mkw32l7CNK\n\tCZmg2z7/09BR2egBErfXr1NOeVmF2aB1pCtwNXYR6JNhqj79i92q6etVARQ7ZciPFE\n\tzqoaAXajQqCQpZH0RxgyVdEwaIVH6XenwIDUpYaJ4GUeq9mjgkvWOflf6ZG1bEiPbv\n\tUibXLVPDslD6gWzfM22fAVAb6GmMYgIzqbzt05iJlM+StQpuQGdJ091V4KmHkRTgRN\n\thfqW58Sksctfg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=baylibre-com.20210112.gappssmtp.com; s=20210112;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date;\n\tbh=WWf3Y/AV7pPyo4N8jAJIBTBDWwWQYAeG9N74jPggf9k=;\n\tb=sGrkhzp3SU5MusW710MPzTTDvEkYVd3/r6eREzWmYnuNrVTDV03U63PCNKlO+0xGsG\n\tsTdFfJnz3M0JFLz0Cmi6WFHPCaMUtXS/sbok9XhrJXqIAFBJq3cZwpZxUh4BF0Y/dQ+C\n\tidXPywlxAdB1hzWVWG8icu2s7P3YLDrnYRMeD1bw0qSgWQhUK0hjeeaqtT33bbDL5za2\n\tbIKkQYwYQ39I6wbPE7dsd/caljuwUk2Jn7CxWxmIFzcSZjrs9E4Hm9UtNcHiXXmoVDV/\n\t+cpkvAXz0jQxTX9dZN4fYiQcvLMyAASP5Y8E/69M2hEE5oHPnqOjs5N24FBu0N/ez/pC\n\teThg=="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key; \n\tunprotected)\n\theader.d=baylibre-com.20210112.gappssmtp.com\n\theader.i=@baylibre-com.20210112.gappssmtp.com header.b=\"sGrkhzp3\"; \n\tdkim-atps=neutral","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20210112;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date;\n\tbh=WWf3Y/AV7pPyo4N8jAJIBTBDWwWQYAeG9N74jPggf9k=;\n\tb=lYhgtZzeKK1lYxvj91492/rOs5Lv0EpEvmqHQjqTFRagR2znBNobjVTx3ZYVMKTPfT\n\t902Hd6RiLaK0LCfr8ewwSeXr6MdUom12qCMyPig9hS5c8obzAAFRpdc4r2fVH+xqXqmk\n\ty+0bj5VPF/IZD1OB1MqDo0IaCijGebA3NwehBRxiqOHmLQCnEFOg9UzIPjUAfk7RSY9f\n\tX213ct+938XMZ++0iRNXxJ+ifxT3jT98EmTbh5fwK+fRSURPxFJWmC8dPCXNk6D4OGSD\n\tc0N4WLNsBPP3vB8JSatP2bR0wvzaV/dFoARu5aL7wEQGvWqfh0Ut47+qhVPEwWtMZCHO\n\tKNew==","X-Gm-Message-State":"ACrzQf2yMDaowsDpSD9YFnkZCRIsDCZoHaYw2Qx5dkPxjdmHcQsun2JK\n\tJV5c4LlhXHllsq/eBmwBbbjoVfZp3fzBMA==","X-Google-Smtp-Source":"AMsMyM4sS7h1GsEDA1gFj2sSXpA8Lxq6QhUmmjmrXpi3ZsKEPXGjMnCYIqt9rtIgwwr4aO3XHcyd9w==","X-Received":"by 2002:a5d:47a1:0:b0:22a:2979:4899 with SMTP id\n\t1-20020a5d47a1000000b0022a29794899mr12694265wrb.349.1664807804061; \n\tMon, 03 Oct 2022 07:36:44 -0700 (PDT)","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 3 Oct 2022 16:36:38 +0200","Message-Id":"<20221003143638.604343-2-fsylvestre@baylibre.com>","X-Mailer":"git-send-email 2.34.1","In-Reply-To":"<20221003143638.604343-1-fsylvestre@baylibre.com>","References":"<20221003143638.604343-1-fsylvestre@baylibre.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v2 1/1] pipeline: rkisp1: Implement Bayer\n\tformats support","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Florian Sylvestre via libcamera-devel\n\t","Reply-To":"Florian Sylvestre ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Implement raw mode for RkISP1:\n- The ISP resizer doesn't support raw formats, so when in raw mode we force the\noutput resolution to be the same as the sensor one.\n- In raw mode, the ISP is bypassed, so we never get statistics buffers.\nThis means that the IPA is never instructed to set the controls nor the\nmetadata.\nAdd a completeRaw() function to the IPA for the purpose of instructing the IPA\nto set controls and metadata when a frame is ready, as opposed to when the\nstatistics are ready.\nWe also need to skip queueing the stats buffer when in raw mode to prevent the\nstatistics bufferReady slot to be triggered at stream off.\n\nSigned-off-by: Florian Sylvestre \nSigned-off-by: Paul Elder \n---\n include/libcamera/ipa/rkisp1.mojom | 1 +\n src/ipa/rkisp1/rkisp1.cpp | 10 +++\n src/libcamera/pipeline/rkisp1/rkisp1.cpp | 64 ++++++++++++++++++-\n src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 44 ++++++++++---\n 4 files changed, 106 insertions(+), 13 deletions(-)","diff":"diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom\nindex eaf3955e..931ef357 100644\n--- a/include/libcamera/ipa/rkisp1.mojom\n+++ b/include/libcamera/ipa/rkisp1.mojom\n@@ -27,6 +27,7 @@ interface IPARkISP1Interface {\n \t[async] fillParamsBuffer(uint32 frame, uint32 bufferId);\n \t[async] processStatsBuffer(uint32 frame, uint32 bufferId,\n \t\t\t\t libcamera.ControlList sensorControls);\n+\t[async] completeRaw(uint32 frame);\n };\n \n interface IPARkISP1EventInterface {\ndiff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp\nindex 1335e3d1..4fd12333 100644\n--- a/src/ipa/rkisp1/rkisp1.cpp\n+++ b/src/ipa/rkisp1/rkisp1.cpp\n@@ -63,6 +63,7 @@ public:\n \tvoid fillParamsBuffer(const uint32_t frame, const uint32_t bufferId) override;\n \tvoid processStatsBuffer(const uint32_t frame, const uint32_t bufferId,\n \t\t\t\tconst ControlList &sensorControls) override;\n+\tvoid completeRaw(const uint32_t frame) override;\n \n protected:\n \tstd::string logPrefix() const override;\n@@ -349,6 +350,15 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId\n \tprepareMetadata(frame, aeState);\n }\n \n+void IPARkISP1::completeRaw(const uint32_t frame)\n+{\n+\tunsigned int aeState = 0;\n+\n+\tsetControls(frame);\n+\n+\tprepareMetadata(frame, aeState);\n+}\n+\n void IPARkISP1::setControls(unsigned int frame)\n {\n \t/*\ndiff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\nindex 25fbf9f1..80ca5547 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp\n@@ -182,6 +182,7 @@ private:\n \tstd::unique_ptr csi_;\n \n \tbool hasSelfPath_;\n+\tbool isRaw_;\n \n \tRkISP1MainPath mainPath_;\n \tRkISP1SelfPath selfPath_;\n@@ -363,7 +364,10 @@ void RkISP1CameraData::paramFilled(unsigned int frame)\n \t\treturn;\n \n \tpipe->param_->queueBuffer(info->paramBuffer);\n-\tpipe->stat_->queueBuffer(info->statBuffer);\n+\n+\tif (!pipe->isRaw_) {\n+\t\tpipe->stat_->queueBuffer(info->statBuffer);\n+\t}\n \n \tif (info->mainPathBuffer)\n \t\tmainPath_->queueBuffer(info->mainPathBuffer);\n@@ -413,6 +417,21 @@ bool RkISP1CameraConfiguration::fitsAllPaths(const StreamConfiguration &cfg)\n \treturn true;\n }\n \n+std::map rawFormats = {\n+\t{ formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 },\n+\t{ formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 },\n+\t{ formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 },\n+\t{ formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 },\n+\t{ formats::SBGGR10, MEDIA_BUS_FMT_SBGGR10_1X10 },\n+\t{ formats::SGBRG10, MEDIA_BUS_FMT_SGBRG10_1X10 },\n+\t{ formats::SGRBG10, MEDIA_BUS_FMT_SGRBG10_1X10 },\n+\t{ formats::SRGGB10, MEDIA_BUS_FMT_SRGGB10_1X10 },\n+\t{ formats::SBGGR12, MEDIA_BUS_FMT_SBGGR12_1X12 },\n+\t{ formats::SGBRG12, MEDIA_BUS_FMT_SGBRG12_1X12 },\n+\t{ formats::SGRBG12, MEDIA_BUS_FMT_SGRBG12_1X12 },\n+\t{ formats::SRGGB12, MEDIA_BUS_FMT_SRGGB12_1X12 },\n+};\n+\n CameraConfiguration::Status RkISP1CameraConfiguration::validate()\n {\n \tconst CameraSensor *sensor = data_->sensor_.get();\n@@ -504,8 +523,23 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate()\n \n \t/* Select the sensor format. */\n \tSize maxSize;\n-\tfor (const StreamConfiguration &cfg : config_)\n+\tPixelFormat rawFormat;\n+\tbool hasRawFormat = false;\n+\tfor (StreamConfiguration &cfg : config_) {\n+\t\tif (PixelFormatInfo::info(cfg.pixelFormat).colourEncoding ==\n+\t\t PixelFormatInfo::ColourEncodingRAW) {\n+\t\t\thasRawFormat = true;\n+\t\t\trawFormat = cfg.pixelFormat;\n+\n+\t\t\t/* Raw format cannot be resized by ISP. */\n+\t\t\tif (cfg.size != sensor->resolution()) {\n+\t\t\t\tcfg.size = sensor->resolution();\n+\t\t\t\tstatus = Adjusted;\n+\t\t\t}\n+\t\t}\n+\n \t\tmaxSize = std::max(maxSize, cfg.size);\n+\t}\n \n \tsensorFormat_ = sensor->getFormat({ MEDIA_BUS_FMT_SBGGR12_1X12,\n \t\t\t\t\t MEDIA_BUS_FMT_SGBRG12_1X12,\n@@ -520,6 +554,13 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate()\n \t\t\t\t\t MEDIA_BUS_FMT_SGRBG8_1X8,\n \t\t\t\t\t MEDIA_BUS_FMT_SRGGB8_1X8 },\n \t\t\t\t\t maxSize);\n+\n+\tif (hasRawFormat) {\n+\t\tauto mbus = rawFormats.find(rawFormat);\n+\t\tif (mbus != rawFormats.end())\n+\t\t\tsensorFormat_ = sensor->getFormat({ mbus->second }, maxSize);\n+\t}\n+\n \tif (sensorFormat_.size.isNull())\n \t\tsensorFormat_.size = sensor->resolution();\n \n@@ -659,8 +700,14 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c)\n \t\t<< \"ISP input pad configured with \" << format\n \t\t<< \" crop \" << rect;\n \n+\tconst PixelFormat &streamFormat = config->at(0).pixelFormat;\n+\tconst PixelFormatInfo &info = PixelFormatInfo::info(streamFormat);\n+\tisRaw_ = info.colourEncoding == PixelFormatInfo::ColourEncodingRAW;\n+\n \t/* YUYV8_2X8 is required on the ISP source path pad for YUV output. */\n-\tformat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8;\n+\tif (!isRaw_)\n+\t\tformat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8;\n+\n \tLOG(RkISP1, Debug)\n \t\t<< \"Configuring ISP output pad with \" << format\n \t\t<< \" crop \" << rect;\n@@ -1152,6 +1199,17 @@ void PipelineHandlerRkISP1::bufferReady(FrameBuffer *buffer)\n \trequest->metadata().set(controls::SensorTimestamp,\n \t\t\t\tbuffer->metadata().timestamp);\n \n+\tif (isRaw_) {\n+\t\tASSERT(activeCamera_);\n+\t\tRkISP1CameraData *data = cameraData(activeCamera_);\n+\n+\t\tRkISP1FrameInfo *info = data->frameInfo_.find(buffer);\n+\t\tif (!info)\n+\t\t\treturn;\n+\n+\t\tdata->ipa_->completeRaw(info->frame);\n+\t}\n+\n \tcompleteBuffer(request, buffer);\n \ttryCompleteRequest(request);\n }\ndiff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\nindex 2d38f0fb..0ae006e8 100644\n--- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n+++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp\n@@ -101,6 +101,23 @@ CameraConfiguration::Status RkISP1Path::validate(StreamConfiguration *cfg)\n \treturn status;\n }\n \n+std::map formatToMediaBus = {\n+\t{ formats::NV12, MEDIA_BUS_FMT_YUYV8_1_5X8 },\n+\t{ formats::NV21, MEDIA_BUS_FMT_YUYV8_1_5X8 },\n+\t{ formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 },\n+\t{ formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 },\n+\t{ formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 },\n+\t{ formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 },\n+\t{ formats::SBGGR10, MEDIA_BUS_FMT_SBGGR10_1X10 },\n+\t{ formats::SGBRG10, MEDIA_BUS_FMT_SGBRG10_1X10 },\n+\t{ formats::SGRBG10, MEDIA_BUS_FMT_SGRBG10_1X10 },\n+\t{ formats::SRGGB10, MEDIA_BUS_FMT_SRGGB10_1X10 },\n+\t{ formats::SBGGR12, MEDIA_BUS_FMT_SBGGR12_1X12 },\n+\t{ formats::SGBRG12, MEDIA_BUS_FMT_SGBRG12_1X12 },\n+\t{ formats::SGRBG12, MEDIA_BUS_FMT_SGRBG12_1X12 },\n+\t{ formats::SRGGB12, MEDIA_BUS_FMT_SRGGB12_1X12 },\n+};\n+\n int RkISP1Path::configure(const StreamConfiguration &config,\n \t\t\t const V4L2SubdeviceFormat &inputFormat)\n {\n@@ -127,15 +144,11 @@ int RkISP1Path::configure(const StreamConfiguration &config,\n \t\t<< \"Configuring \" << name_ << \" resizer output pad with \"\n \t\t<< ispFormat;\n \n-\tswitch (config.pixelFormat) {\n-\tcase formats::NV12:\n-\tcase formats::NV21:\n-\t\tispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_1_5X8;\n-\t\tbreak;\n-\tdefault:\n+\tauto mbus = formatToMediaBus.find(config.pixelFormat);\n+\tif (mbus != formatToMediaBus.end())\n+\t\tispFormat.mbus_code = mbus->second;\n+\telse\n \t\tispFormat.mbus_code = MEDIA_BUS_FMT_YUYV8_2X8;\n-\t\tbreak;\n-\t}\n \n \tret = resizer_->setFormat(1, &ispFormat);\n \tif (ret < 0)\n@@ -207,14 +220,25 @@ void RkISP1Path::stop()\n namespace {\n constexpr Size RKISP1_RSZ_MP_SRC_MIN{ 32, 16 };\n constexpr Size RKISP1_RSZ_MP_SRC_MAX{ 4416, 3312 };\n-constexpr std::array RKISP1_RSZ_MP_FORMATS{\n+constexpr std::array RKISP1_RSZ_MP_FORMATS{\n \tformats::YUYV,\n \tformats::NV16,\n \tformats::NV61,\n \tformats::NV21,\n \tformats::NV12,\n \tformats::R8,\n-\t/* \\todo Add support for RAW formats. */\n+\tformats::SRGGB8,\n+\tformats::SGRBG8,\n+\tformats::SGBRG8,\n+\tformats::SBGGR8,\n+\tformats::SRGGB10,\n+\tformats::SGRBG10,\n+\tformats::SGBRG10,\n+\tformats::SBGGR10,\n+\tformats::SRGGB12,\n+\tformats::SGRBG12,\n+\tformats::SGBRG12,\n+\tformats::SBGGR12,\n };\n \n constexpr Size RKISP1_RSZ_SP_SRC_MIN{ 32, 16 };\n","prefixes":["libcamera-devel","v2","1/1"]}