From patchwork Fri Oct 14 13:18:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 17597 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B4370C327C for ; Fri, 14 Oct 2022 13:18:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2926462DC4; Fri, 14 Oct 2022 15:18:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1665753536; bh=ExRPpGRFqjnrmv9X2kH7z0Ok9SOOjLAPX+Olr3TDYnQ=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=JaBFlN7n7EMeka8PFxhKQzUJbAohfd2dxJTp7yihhGVJjgTblZb1iPzDCdCfIbb4O dHWjt6NMa695BwqOjxHeOk41tSWPUZ+ePOWGLcHJPkQfx7l88KC07cm+NGc7juiKBy KZ+0CjERQcWFaWVLjVVSZCAL1ACjSFpeO3j9hR7nhrXqoEPkFsjeJ4XGsv+fE58ijw pCfAmq7cPp349AWRfTf4RCGDCfzgHMMpQ2lHkv8vRHWnWbSQa0ZzgIW3CRB++WA+3u uB9g+8ztLr7KExnkK6U8xUsED5APfQBcbd9resLOip8OpUb/RYKpbDpjS/ynA8AQ/X KZbqqjwy8SVuA== Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2434562D8E for ; Fri, 14 Oct 2022 15:18:51 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="WUCruMWc"; dkim-atps=neutral Received: by mail-wr1-x435.google.com with SMTP id r13so7492229wrj.11 for ; Fri, 14 Oct 2022 06:18:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=L+1yEG0ZioKF335bKGvQvDrhQdwlXiacOlC6EMfirlU=; b=WUCruMWcTV9xIRGiSlphwQKDU7Ed5ZQj1TEnIQCknMBlY894XWkqdMTqNJt6ZpYufG iM6M02ZjvZp7lhBF8Ef7rb5R499B/EerzgpXhXEpbHMnBXPAWa7RO27tA4c+S+1nXO6B g0GKloRpTa0AqT+QRz+Oie3JQhY43H80pL8nf4LVe+YMbuBAUGKCvvSd/Pzvl9efalTJ etICXOpVvMaFOl3SykwyOOH/KP0MidQMGupyJnWs8osV3z0ocMjwqluLh6U8X5NACSFw pXERnTqNarUVbihDCA+ZxiHjpOPZWJdk+VagcKT9Yx0x9cXcbDAozsv/Ir1iEXERtpxZ XOBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=L+1yEG0ZioKF335bKGvQvDrhQdwlXiacOlC6EMfirlU=; b=E9R92UGOiDdPmD4sdmfU8T5e03rTb3Ft/5J8RVk80z3vGhCrN6RCVGGOunqna8a0fV g9NDbUgjq9Th/wf4whpgkxvaEpGWzrXXkOyiOwPY2+wo8/tY8+Ps7/AquIGBf/4MqplX zJmRqw5U7H81hYkilKoNCKgEADfoL/iPdUQdnt4Clw/mj2v1wh0nGPsxywlpN+1ao/Sz aQOE6EPJoy5W1aeZslzqj2KZ6FRF/IhNzbXgi6n9y0kdSTN0DaXhf7qCXsSgsYTpuwjd lxK8k2rodHg0l3i2BWnMoOgDpADItizsFlF1ta0CeJ0BKFf8YPVxKuqI5tMipzgXE5xp iDzQ== X-Gm-Message-State: ACrzQf0ajeAXf95H8e73duDAP9MlI4l2BKKRH8kXhAe2Iv5BsUResHX2 0RkU+u2XPjWmORTOf+fGf+hapHGNYTy8oA== X-Google-Smtp-Source: AMsMyM7828FME1Kj1Ye7L5pDhnp6j5RrAsVFGkBo6OZukj5kKBPVoZxGZDcFka3VFPmpK5M0S2O+Ug== X-Received: by 2002:a05:6000:1d8a:b0:22f:2235:93cb with SMTP id bk10-20020a0560001d8a00b0022f223593cbmr3357775wrb.701.1665753530204; Fri, 14 Oct 2022 06:18:50 -0700 (PDT) Received: from naush-laptop.localdomain ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id g17-20020a05600c001100b003c6bbe910fdsm8282458wmc.9.2022.10.14.06.18.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 06:18:49 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 14 Oct 2022 14:18:36 +0100 Message-Id: <20221014131846.27169-1-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform configuration and buffer allocation improvements X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Naushir Patuck via libcamera-devel From: Naushir Patuck Reply-To: Naushir Patuck Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi, One big issue that is plaguing libcamera users on Raspberry Pi is memory usage. All our frame buffers have to be allocated in CMA space because of hardware restrictions. Unfortunately CMA space is very limited (particularly on the Raspberry Pi Zero), and is prone to frequent fragmentation as display/rendering buffers also use this pool. Not to mention that the kernel is also allowed to allocate in the CMA pool for its own needs. As a result, some of our libcamera applications intermittently fail to run because of buffer allocation failure, and some use-cases (e.g. > 20MPix captures) will simply not be able to run on some platforms at all. This patch series attempts to try and fix this by easing the memory usage of the Raspberry Pi pipeline handler. It does this by using a platform configuration file which (amongst other things) instructs the pipeline handler on how many internal buffers it should allocate. There may be a performance penality (e.g dropped frames, or longer time taken for initial AE/AWB convergence) that have been documented. But if the application can guarantee certain behavior, we can ensure the pipeline can still process requests with the same performance as we have today. The Raspberry Pi libcamera-apps and picamera2 framework already use this behavior, so the change should be transparent to applications built on these, apart from using significantly less memory. Patches 1 to 4 and the framework for using a configuration file to setup the buffer allocation parameters Patch 5 to 7 improve the buffer handling dealing with dropped frames, allowing us to use fewer internal buffers. Patch 8 adds a parameter to modify drop frame behavior - not related to this problem, but a nice to have. Patches 9 and 10 add configuration files for minimal memory configuration, and another for optimal memory usage for libcamera-apps and picamera2. I've considered how these various performance tuning parameters could be made part of the core StreamConfiguration structure or even as a Control. However, they are entirely platform specific, and I am struggling to come up with a clean way to put this into libcamera core with polluting it with various platform specific parameters. If others have any ideas or suggestions on using alternate means of passing platform specific parameters into the pipeline handler, please do comment on this review. Thanks, Naush Naushir Patuck (10): libcamera: pipeline: Add a platform configuration file helper pipeline: raspberrypi: Add a pipeline config structure pipeline: raspberrypi: Split out ISP Output0 buffer allocation pipeline: raspberrypi: Read config parameters from a file pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when dropping frames pipeline: raspberrypi: Reorder startup drop frame initialisation pipeline: raspberrypi: Add a parameter to disable startup drop frames pipeline: raspberrypi: Allow pipeline handler to always use the newest frame pipeline: raspberrypi: Add minimal memory usage config file pipeline: raspberrypi: Add libcamera-apps and picamera2 config file include/libcamera/internal/pipeline_handler.h | 2 + .../pipeline/raspberrypi/data/default.json | 28 +++ .../pipeline/raspberrypi/data/meson.build | 10 + .../raspberrypi/data/minimal_mem.json | 28 +++ .../pipeline/raspberrypi/data/rpi_apps.json | 28 +++ .../pipeline/raspberrypi/meson.build | 2 + .../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++--- src/libcamera/pipeline_handler.cpp | 57 +++++ 8 files changed, 339 insertions(+), 31 deletions(-) create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build create mode 100644 src/libcamera/pipeline/raspberrypi/data/minimal_mem.json create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json