From patchwork Wed Dec 1 10:15:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 14954 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2314DC3250 for ; Wed, 1 Dec 2021 10:15:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B597360726; Wed, 1 Dec 2021 11:15:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="ES6MRyMV"; dkim-atps=neutral Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 802D96011D for ; Wed, 1 Dec 2021 11:15:19 +0100 (CET) Received: by mail-wm1-x32c.google.com with SMTP id az34-20020a05600c602200b0033bf8662572so741613wmb.0 for ; Wed, 01 Dec 2021 02:15:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+5tbuXZiLzJgDNl3HF0j/KsQSSn76KepjHhFHoG2/N0=; b=ES6MRyMVTxgpba6yzEUFw7Vb3a0979G0PcTAPjBxFxwln3N+MB8XXxL8SBiJ5jLS6Z OJsI0i+3FG2cCMc9HgnMu7kkA0p2Y2x1XCpHMma9thzIwx0KMEEdxxyKyx7mnd7ZknYX 4W8KbZHCcPwlhvqzfqxEXYvXNSzFo+OVXUn6wEqQvwtsDqSbqK9OXELZOI5f6908a3WS QVhZ20hy60nhRA0XGHB0gBv+f3uJ/Qpew/U9g8K3Az+4evHuy0dwyu1hvNx5YA8t4rW+ 61jWbVh86dmH2OKzGbaDiO7v76I9zqYvFLLzsS48zPyus75BjjxlNPUXhW8862Ngoq2K WBgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+5tbuXZiLzJgDNl3HF0j/KsQSSn76KepjHhFHoG2/N0=; b=btimOorGcxQBkWM+gMO7tRqjtDOkJ+Ii28CwJClzB5KbqYc83zXzBCi7F0RY/AvFgo cFugJ0Sc25JMZ7RpQV2I+vl3yV7FNNlcaen0f/+RB0Jd9QPnn1LGWudL/97RKs+IhXd5 4XqEMiS4+eC5TGaVKg2nJQ2Ddy/9Z23t4X0a8qEOYJpSp2WOsun9deOPMp4dhiF4j/dY kSfHUK/bmB/NqHcIH/2MrWIBF8lyaVEXqzwrlWGF+RTg22ERNh9ivBVrMcR4tmtHpYhS dAGqzK/PUXnZyzKF3OQtuJaiAp/zj54zgyYB5OUFZyAaT60mRISkSHI7PasLMsAukYoq /bnA== X-Gm-Message-State: AOAM531+7MnaPac9iCnhblLW20e2O1qvfvQ5/jebankWPmHFrCDjFxYH d6eF+r0nXVhqQQCrwkiiiceAKstETezYEbkv X-Google-Smtp-Source: ABdhPJxoxN2gvYiSz4KVvTvGyesZOFYsJliTDYeKSWhW5okpEel+gOsIEI+gdMfgV3AJjUt81ooHUA== X-Received: by 2002:a05:600c:1987:: with SMTP id t7mr5716432wmq.24.1638353719007; Wed, 01 Dec 2021 02:15:19 -0800 (PST) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:e4a2:3070:eea4:e434]) by smtp.gmail.com with ESMTPSA id z18sm18815535wrq.11.2021.12.01.02.15.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 02:15:18 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Wed, 1 Dec 2021 10:15:07 +0000 Message-Id: <20211201101508.10619-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211201101508.10619-1-david.plowman@raspberrypi.com> References: <20211201101508.10619-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/2] pipeline: raspberrypi: Fix under-allocation of embedded data buffers X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The code was reducing the number of raw stream buffers allocated when the application is providing some of its own. However, it was not taking account of the fact that the application cannot supply embedded data buffers, so it must always allocate a reasonable minimum number of these buffers (possibly more than the number of raw stream buffers) to prevent frame drops. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck --- .../pipeline/raspberrypi/raspberrypi.cpp | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index e31fa3b2..045725dd 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1248,18 +1248,25 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) /* Decide how many internal buffers to allocate. */ for (auto const stream : data->streams_) { unsigned int numBuffers; - - if (stream == &data->unicam_[Unicam::Image] || - stream == &data->unicam_[Unicam::Embedded]) { + /* + * For Unicam, allocate a minimum of 4 buffers as we want + * to avoid any frame drops. + */ + constexpr unsigned int minBuffers = 4; + if (stream == &data->unicam_[Unicam::Image]) { /* - * For Unicam, allocate a minimum of 4 buffers as we want - * to avoid any frame drops. If an application has configured - * a RAW stream, allocate additional buffers to make up the - * minimum, but ensure we have at least 2 sets of internal - * buffers to use to minimise frame drops. + * If an application has configured a RAW stream, allocate + * additional buffers to make up the minimum, but ensure + * we have at least 2 sets of internal buffers to use to + * minimise frame drops. */ - constexpr unsigned int minBuffers = 4; numBuffers = std::max(2, minBuffers - numRawBuffers); + } else if (stream == &data->unicam_[Unicam::Embedded]) { + /* + * Embedded data buffers are (currently) for internal use, + * so allocate the minimum required to avoid frame drops. + */ + numBuffers = minBuffers; } else { /* * Since the ISP runs synchronous with the IPA and requests, From patchwork Wed Dec 1 10:15:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 14955 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9F2E5C3251 for ; Wed, 1 Dec 2021 10:15:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0B92D60729; Wed, 1 Dec 2021 11:15:22 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="c0gIV4fM"; dkim-atps=neutral Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2EE1B6011D for ; Wed, 1 Dec 2021 11:15:20 +0100 (CET) Received: by mail-wr1-x435.google.com with SMTP id v11so51010251wrw.10 for ; Wed, 01 Dec 2021 02:15:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5zCiFuQtuj7Sh6qS4ID/TIMwe//x9gGRyINlvmZvs2Y=; b=c0gIV4fMmU9wWytZDYUkkmm0w4pwOEYoqRJsys89wUE7Y2sub8jRCqu6Qjrt826FRD WaRTnqgrjHQqMaOmZav+xqa/CVL/JRliidig3LtwjGEQMvUAuW0X1/Ls7C0pn5CZlpOV SzyKwnaYWtRceTXIeGYX8hAoqQSnEdKItohO7lmJCB0u67waeSqdC+N9IvhWSPs/j7wp spt1CONNk6OkYW4BjVpkOTtHrDAqBFL95eZ7IdeldHtE3w6xFsbLy8lT2Mj8Xz7UzI4d Bv489FrjkStbwAmktXdXDRr5mYqlBusgTXKq2/nrAkfUeMYA8nb0VEeXXSlmI0pakUe0 0Myg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5zCiFuQtuj7Sh6qS4ID/TIMwe//x9gGRyINlvmZvs2Y=; b=pwB3rF3asshpYjxx9UTSspKNVcQ+53VqtUY6/cKO04MHfF9Di+9/U0TeJYE2iOkF1i r/HeQ6ASXgwKqmG0wPq9EUsrUwmOsuRUCJhkRKY+jf1GvVm1WlpcOKKZdcdgHzPp6cnS vJS9GknIeM01RJSUv6q/y+Gu2dt7+/l8dY6fJy0krMyZuMSz/yEIOlVxkl/NM6J9tX7J d0ljZOaRcKtXVvuWs15nm0Eufgdl+gPlQTR+TMblRR0DatKFDrQRi73nSrp6/q0zr/C7 29t19k+qna0mJD2WKi97q8/THBBAsViGk8PniIDT8+BzDdIhdh6GQN82nfcGaFLE/git J89A== X-Gm-Message-State: AOAM531TMj2HymlmZOEEt4D7U33Rm1lp/2NbceSUdPr2YSw1EzOmTeEC TycZ2KV1L99s/O8Ef+zo3I3o1viIhkVF5Q0I X-Google-Smtp-Source: ABdhPJwB9Kqp9h8H2bMwtuSQM+3zVZm561EBtflJ0nOlxA9MbQXH+EiReH2gH8U7d3edGffHlPqz3g== X-Received: by 2002:a5d:52c3:: with SMTP id r3mr5618937wrv.115.1638353719701; Wed, 01 Dec 2021 02:15:19 -0800 (PST) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:e4a2:3070:eea4:e434]) by smtp.gmail.com with ESMTPSA id z18sm18815535wrq.11.2021.12.01.02.15.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Dec 2021 02:15:19 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Wed, 1 Dec 2021 10:15:08 +0000 Message-Id: <20211201101508.10619-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211201101508.10619-1-david.plowman@raspberrypi.com> References: <20211201101508.10619-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/2] pipeline: raspberrypi: Choose bit depth and packing according to raw stream X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a raw stream is specified, the bit depth and packing requested should influence our choice of camera mode to match (if possible). Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck --- .../pipeline/raspberrypi/raspberrypi.cpp | 37 ++++++++++--------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 045725dd..c01e0769 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -49,6 +49,8 @@ LOG_DEFINE_CATEGORY(RPI) namespace { +constexpr unsigned int defaultRawBitDepth = 12; + /* Map of mbus codes to supported sizes reported by the sensor. */ using SensorFormats = std::map>; @@ -125,15 +127,13 @@ double scoreFormat(double desired, double actual) return score; } -V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size &req) +V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size &req, unsigned int bitDepth) { double bestScore = std::numeric_limits::max(), score; V4L2SubdeviceFormat bestFormat; -#define PENALTY_AR 1500.0 -#define PENALTY_8BIT 2000.0 -#define PENALTY_10BIT 1000.0 -#define PENALTY_12BIT 0.0 + constexpr float penaltyAr = 1500.0; + constexpr float penaltyBitDepth = 500.0; /* Calculate the closest/best mode from the user requested size. */ for (const auto &iter : formatsMap) { @@ -149,15 +149,10 @@ V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size & /* Score the dimensions for closeness. */ score = scoreFormat(req.width, size.width); score += scoreFormat(req.height, size.height); - score += PENALTY_AR * scoreFormat(reqAr, fmtAr); + score += penaltyAr * scoreFormat(reqAr, fmtAr); /* Add any penalties... this is not an exact science! */ - if (info.bitsPerPixel == 12) - score += PENALTY_12BIT; - else if (info.bitsPerPixel == 10) - score += PENALTY_10BIT; - else if (info.bitsPerPixel == 8) - score += PENALTY_8BIT; + score += abs(info.bitsPerPixel - bitDepth) * penaltyBitDepth; if (score <= bestScore) { bestScore = score; @@ -397,9 +392,14 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() * Calculate the best sensor mode we can use based on * the user request. */ - V4L2SubdeviceFormat sensorFormat = findBestFormat(data_->sensorFormats_, cfg.size); + const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat); + unsigned int bitDepth = info.isValid() ? info.bitsPerPixel : defaultRawBitDepth; + V4L2SubdeviceFormat sensorFormat = findBestFormat(data_->sensorFormats_, cfg.size, bitDepth); + BayerFormat::Packing packing = BayerFormat::Packing::CSI2; + if (info.isValid() && !info.packed) + packing = BayerFormat::Packing::None; V4L2DeviceFormat unicamFormat = toV4L2DeviceFormat(sensorFormat, - BayerFormat::Packing::CSI2); + packing); int ret = data_->unicam_[Unicam::Image].dev()->tryFormat(&unicamFormat); if (ret) return Invalid; @@ -533,7 +533,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, switch (role) { case StreamRole::Raw: size = data->sensor_->resolution(); - sensorFormat = findBestFormat(data->sensorFormats_, size); + sensorFormat = findBestFormat(data->sensorFormats_, size, defaultRawBitDepth); pixelFormat = mbusCodeToPixelFormat(sensorFormat.mbus_code, BayerFormat::Packing::CSI2); ASSERT(pixelFormat.isValid()); @@ -622,6 +622,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) Size maxSize, sensorSize; unsigned int maxIndex = 0; bool rawStream = false; + unsigned int bitDepth = defaultRawBitDepth; /* * Look for the RAW stream (if given) size as well as the largest @@ -638,7 +639,9 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) sensorSize = cfg.size; rawStream = true; /* Check if the user has explicitly set an unpacked format. */ - packing = BayerFormat::fromPixelFormat(cfg.pixelFormat).packing; + BayerFormat bayerFormat = BayerFormat::fromPixelFormat(cfg.pixelFormat); + packing = bayerFormat.packing; + bitDepth = bayerFormat.bitDepth; } else { if (cfg.size > maxSize) { maxSize = config->at(i).size; @@ -664,7 +667,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) } /* First calculate the best sensor mode we can use based on the user request. */ - V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize); + V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize, bitDepth); ret = data->sensor_->setFormat(&sensorFormat); if (ret) return ret;