From patchwork Tue Nov 30 11:22:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 14892 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 61EEBBDB13 for ; Tue, 30 Nov 2021 11:23:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 41E51605AA; Tue, 30 Nov 2021 12:23:18 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="rTLgyZrW"; dkim-atps=neutral Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 923776011A for ; Tue, 30 Nov 2021 12:23:15 +0100 (CET) Received: by mail-wr1-x434.google.com with SMTP id q3so20772383wru.5 for ; Tue, 30 Nov 2021 03:23:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fUpF1cChapciAMV2VFUunyAxPS/CfPVaIVJTfV2jEsw=; b=rTLgyZrW/VEBkgFHVHoXTkcx8OV+tXhEsM9ynJlRn+X21wwajQLsvUo60SxTVKgEeQ Vod4MsDV4+t7WDVUaw8XfIFTcVFrrYI6ZCwdan+LefcAWHKhfbYmorooPnpwjWrWc9mD ZyiOtBMUDvC52hZNsTq9io6O3Y/mumjXoayCu16lo7b96Amwbvg3xI7dQLX1YAqG2W2t LQOL2OO/IKL12n6JD/q7J9bX3rWPB5pf4/haegwSOv467+GbF9NPV5b2GyZO9CTNdk+l KZGkmr28R7MBaTWACeyWM82xGR1P3b53g55XL/kcNel2cSMVW5aPd7noNmdicw9xV6UT k90w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fUpF1cChapciAMV2VFUunyAxPS/CfPVaIVJTfV2jEsw=; b=T71IvV8zCVVSjm/McZklg2th4opkR8OR8IU7n6EjuaQg1hIQ3sCYwI8Enb6lKYsL9E S/dxePv95RIaegfOmkQ7GJAGVZno0fW4EulmCoym0wMI+22ByGavA7/CNfitgkjljMWa HAMaF65h7Xxi7Kp+38L4cYRftYnBr2QM0HUrUgxmDMeRHeGxLQNfiq98tUOXU2petsUm rM5KAeJxwP/WnPLW3LPHs1S2tB5eZFKhOsRAI/jEfDQj5qeyTgSxIBIZUugGqD2scoyI UvycTDrVGNAlGUSNZG32yJwwdWSg2q52UBQV+SSY+YgwdsrJvwbHgQxKzNZ7zJwn0b6k ZIrg== X-Gm-Message-State: AOAM533kjcz03avm3cYQS8xG7/5HzdTuHtGprLz0ew5m7I2lFpyh9Hej q9isQF14iIJjL5sPIZRk8dkcAn3i4EYInQqY X-Google-Smtp-Source: ABdhPJzK+s3zevOTmihXMDO5aKnLpHs7DMyx7nl+whS9tHL5uZ0/FNmNGdRkztq/Cn6xHu/5tV4apg== X-Received: by 2002:adf:ef4f:: with SMTP id c15mr41203016wrp.226.1638271395138; Tue, 30 Nov 2021 03:23:15 -0800 (PST) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:e4a2:3070:eea4:e434]) by smtp.gmail.com with ESMTPSA id u2sm19054923wrs.17.2021.11.30.03.23.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Nov 2021 03:23:14 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Tue, 30 Nov 2021 11:22:58 +0000 Message-Id: <20211130112259.18723-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211130112259.18723-1-david.plowman@raspberrypi.com> References: <20211130112259.18723-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/2] pipeline: raspberrypi: Fix under-allocation of embedded data buffers X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The code was reducing the number of raw stream buffers allocated when the application is providing some of its own. However, it was not taking account of the fact that the application cannot supply embedded data buffers, so it must always allocate a reasonable minimum number of these buffers (possibly more than the number of raw stream buffers) to prevent frame drops. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Reviewed-by: Jacopo Mondi --- .../pipeline/raspberrypi/raspberrypi.cpp | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index e31fa3b2..045725dd 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1248,18 +1248,25 @@ int PipelineHandlerRPi::prepareBuffers(Camera *camera) /* Decide how many internal buffers to allocate. */ for (auto const stream : data->streams_) { unsigned int numBuffers; - - if (stream == &data->unicam_[Unicam::Image] || - stream == &data->unicam_[Unicam::Embedded]) { + /* + * For Unicam, allocate a minimum of 4 buffers as we want + * to avoid any frame drops. + */ + constexpr unsigned int minBuffers = 4; + if (stream == &data->unicam_[Unicam::Image]) { /* - * For Unicam, allocate a minimum of 4 buffers as we want - * to avoid any frame drops. If an application has configured - * a RAW stream, allocate additional buffers to make up the - * minimum, but ensure we have at least 2 sets of internal - * buffers to use to minimise frame drops. + * If an application has configured a RAW stream, allocate + * additional buffers to make up the minimum, but ensure + * we have at least 2 sets of internal buffers to use to + * minimise frame drops. */ - constexpr unsigned int minBuffers = 4; numBuffers = std::max(2, minBuffers - numRawBuffers); + } else if (stream == &data->unicam_[Unicam::Embedded]) { + /* + * Embedded data buffers are (currently) for internal use, + * so allocate the minimum required to avoid frame drops. + */ + numBuffers = minBuffers; } else { /* * Since the ISP runs synchronous with the IPA and requests, From patchwork Tue Nov 30 11:22:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 14893 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6E778C324F for ; Tue, 30 Nov 2021 11:23:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1503F605BA; Tue, 30 Nov 2021 12:23:20 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="E5ACehJh"; dkim-atps=neutral Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7541C60587 for ; Tue, 30 Nov 2021 12:23:16 +0100 (CET) Received: by mail-wr1-x430.google.com with SMTP id j3so43647377wrp.1 for ; Tue, 30 Nov 2021 03:23:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2F2qAO8MKIglq4NnZW8/UIdKUY8Q5e13Dbp8ZkesB50=; b=E5ACehJh0qEMo9B01HrdDk/eLzKp0NymBawFVLS/bS9CX0TgzgB+MlsZXr6NcY8j2d GE+C8RipTDeAVybLrb9kG3zaJFDcEHke6ybiBx4sr82yc9OvieoDKVPrWOM6KMvky7K+ K4DChkrT+kFomKDI9m+KIhUes5m3OxSjWyo0Y7twkrY6bCCpc797c2qNM3/Laag6f8Bh qO6f0EK9TRWoq3DMUDzrNs51Z7Q2wN7DqTLg4nl6szNjmtqMiS8k3jwM2rcC2eH4WMmt GR9NkoihdFhTOICtdT2u+v9ManDP1N8F7QY44bG3OzuInP9BzfG1V8zHkhCtjA/2Ecis hc7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2F2qAO8MKIglq4NnZW8/UIdKUY8Q5e13Dbp8ZkesB50=; b=XtdIPeC0qB/7ahdSNby03xyAjJa8/1apWiaS8C4EBABLdRg5r2Wa5leGvShWDZJPex tI6T67eoHXlcDoaIB93SO0PXyDIeBDX+2R9WmOGldRwtM9QnEDZ802THcM//254/CJZp aLUm3mJ3kOPc2Yuv46RSq/Z24rmbdxstz/SwX4Ub42XlPB4UYcy+/7WGwuolxJKzMn9x V+xO5Yz42KYz1uhi0BT3rSYYnNcL0N3cSoKNjo6pldvKb03/+KaOnqwJWy6xwiw/xcJZ RqNXOs9QQFld057U8uhEha+0BLCwpsU2hc8GwWe6pcYPnK6lFO5JGkiNfg5KxmNGamtK d7Dg== X-Gm-Message-State: AOAM532b2atFQd3f9kMJ+w77f7vauhetNpFR4/jSA9wk6snEj/NOq8rK X6tmvWoGTXxgU3PsRtE29UTfnAPsCm4k3Vcp X-Google-Smtp-Source: ABdhPJybUtltRbENGo2sXml7UXxsALol7mLq0AM1wwDqfOmfSeLQadSpPAdsQtmXJQ4yk78Rt283qg== X-Received: by 2002:adf:f1cc:: with SMTP id z12mr31470613wro.395.1638271395996; Tue, 30 Nov 2021 03:23:15 -0800 (PST) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:e4a2:3070:eea4:e434]) by smtp.gmail.com with ESMTPSA id u2sm19054923wrs.17.2021.11.30.03.23.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Nov 2021 03:23:15 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Tue, 30 Nov 2021 11:22:59 +0000 Message-Id: <20211130112259.18723-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211130112259.18723-1-david.plowman@raspberrypi.com> References: <20211130112259.18723-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/2] pipeline: raspberrypi: Choose bit depth and packing according to raw stream X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a raw stream is specified, the bit depth and packing requested should influence our choice of camera mode to match (if possible). Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck --- .../pipeline/raspberrypi/raspberrypi.cpp | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 045725dd..03012e89 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -49,6 +49,8 @@ LOG_DEFINE_CATEGORY(RPI) namespace { +unsigned int DEFAULT_RAW_BIT_DEPTH = 12; + /* Map of mbus codes to supported sizes reported by the sensor. */ using SensorFormats = std::map>; @@ -125,15 +127,13 @@ double scoreFormat(double desired, double actual) return score; } -V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size &req) +V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size &req, unsigned int bitDepth) { double bestScore = std::numeric_limits::max(), score; V4L2SubdeviceFormat bestFormat; #define PENALTY_AR 1500.0 -#define PENALTY_8BIT 2000.0 -#define PENALTY_10BIT 1000.0 -#define PENALTY_12BIT 0.0 +#define PENALTY_BIT_DEPTH 500.0 /* Calculate the closest/best mode from the user requested size. */ for (const auto &iter : formatsMap) { @@ -152,12 +152,7 @@ V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size & score += PENALTY_AR * scoreFormat(reqAr, fmtAr); /* Add any penalties... this is not an exact science! */ - if (info.bitsPerPixel == 12) - score += PENALTY_12BIT; - else if (info.bitsPerPixel == 10) - score += PENALTY_10BIT; - else if (info.bitsPerPixel == 8) - score += PENALTY_8BIT; + score += abs(info.bitsPerPixel - bitDepth) * PENALTY_BIT_DEPTH; if (score <= bestScore) { bestScore = score; @@ -397,9 +392,14 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() * Calculate the best sensor mode we can use based on * the user request. */ - V4L2SubdeviceFormat sensorFormat = findBestFormat(data_->sensorFormats_, cfg.size); + const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat); + unsigned int bitDepth = info.isValid() ? info.bitsPerPixel : DEFAULT_RAW_BIT_DEPTH; + V4L2SubdeviceFormat sensorFormat = findBestFormat(data_->sensorFormats_, cfg.size, bitDepth); + BayerFormat::Packing packing = BayerFormat::Packing::CSI2; + if (info.isValid() && !info.packed) + packing = BayerFormat::Packing::None; V4L2DeviceFormat unicamFormat = toV4L2DeviceFormat(sensorFormat, - BayerFormat::Packing::CSI2); + packing); int ret = data_->unicam_[Unicam::Image].dev()->tryFormat(&unicamFormat); if (ret) return Invalid; @@ -533,7 +533,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, switch (role) { case StreamRole::Raw: size = data->sensor_->resolution(); - sensorFormat = findBestFormat(data->sensorFormats_, size); + sensorFormat = findBestFormat(data->sensorFormats_, size, DEFAULT_RAW_BIT_DEPTH); pixelFormat = mbusCodeToPixelFormat(sensorFormat.mbus_code, BayerFormat::Packing::CSI2); ASSERT(pixelFormat.isValid()); @@ -622,6 +622,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) Size maxSize, sensorSize; unsigned int maxIndex = 0; bool rawStream = false; + unsigned int bitDepth = DEFAULT_RAW_BIT_DEPTH; /* * Look for the RAW stream (if given) size as well as the largest @@ -638,7 +639,9 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) sensorSize = cfg.size; rawStream = true; /* Check if the user has explicitly set an unpacked format. */ - packing = BayerFormat::fromPixelFormat(cfg.pixelFormat).packing; + BayerFormat bayerFormat = BayerFormat::fromPixelFormat(cfg.pixelFormat); + packing = bayerFormat.packing; + bitDepth = bayerFormat.bitDepth; } else { if (cfg.size > maxSize) { maxSize = config->at(i).size; @@ -664,7 +667,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) } /* First calculate the best sensor mode we can use based on the user request. */ - V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize); + V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize, bitDepth); ret = data->sensor_->setFormat(&sensorFormat); if (ret) return ret;