From patchwork Wed May 7 07:52:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 23341 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9A52FC3226 for ; Wed, 7 May 2025 08:28:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E82AD68B2E; Wed, 7 May 2025 10:27:59 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="cngenAga"; dkim-atps=neutral Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 360B568AD3 for ; Wed, 7 May 2025 10:27:57 +0200 (CEST) Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-43cee550af2so7047195e9.1 for ; Wed, 07 May 2025 01:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1746606476; x=1747211276; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=v/Foq4rHHSRZIsPO5URXdBlZZw4YnjS05CZfgNIOXz8=; b=cngenAgalqguFR1TxJWjm/ZA3c11SErqDzVu5naiMbvWgUgSJPgW0d4mWGLMdtiANh 3kIm4XN2ViqU29643KNyL68lyoF5HYGwZrvWFVaRN+u3gse/5p0rfJrd85uPAf/ZeNIQ 6vFH88zgvWQqQwefu3AJmwMuZomiBEqWluoF1BqB5608mXWA2V6mhkC92/BudVUKLS2C A+oV4yqD9n4TPSki9BdhjdmmCU2PmQ05/tqtpt40ZcSaW5xksNoOoVeKW3/XKq1vUAQH qYE/a/ynDH+7cuVBb0BaL+gGVRlpztSCeE237CxotnZt5uo/Ju0HSMxkOCVmFewsR/Bd M1ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746606476; x=1747211276; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=v/Foq4rHHSRZIsPO5URXdBlZZw4YnjS05CZfgNIOXz8=; b=dRJD9nIQdHAEzifWDGDXDdpNyLL7Yni3v1utEvW4KCZ5XP1KfcmBqPRoFz7/1IGlmT 9r/g4E2x/TCLQgLaV0koWfosHVZvRbQeuf+gCt7I8mB78yfUGnOW/+lfzNlboAeJajcU R2A/agYzoRWfmIVmVc7PB30fdi5M/IM2MoCcJwGdXyQlB6tGfnBPZ1yXjGtZk7fqH7d/ lS4UhRN/sgpxZxS6UbfJJ3WTKa4DubCE4wFRUiHETOMZ9hO44wQNOAJq147l+8eRLpWa qhv8irHUBpkPi35nFZmEAGyDiYs3bRitZ8GU7/ZF8IdaU5N8b0WNwudhaon3yIQZcap2 vCAw== X-Gm-Message-State: AOJu0YzMmEHnQ/Isrw7dUzgxNEMYh8uKvZPaXPv3VexloRlqFRO/lQrb P5YrLW8NZl6FGRVwSHNzHgRMiUJgugcXmL0kBzBlcj6DISbWjeiVuCqEF3y59A4DPEpor6y8OPh b X-Gm-Gg: ASbGncuiJKDE/tu9jDrGTHgV5AuEsNaX6nCTmMR1/2xGIsPwOFB3FCQk5z/qHt/2RKC 61FoU4Bv86tViQiEk86QRKVjzhDgHRXuxVd/IQ2PVDbkMOB+cTC5pDnmkjIa7NRhm3tWzyy/dW2 tFFsbC3dY5fj+REh+qVUHhpwwFwJUYbvolkMamibv2XLuZwPKdfv+ythv56r1ZdrbQNDceec0ZE ++pysySY99+XEpJgTZgWuURiNRbo1xpdxuX7wEEiVsTWEXpVruUrsknwFQjdXx5oYXMAQglVQ5+ xz5XkwS8rhMM4qxJXxBl8UJ97HQwnZxuN/c5vwhwGGqZELHFWCUuLZsmFU1h X-Google-Smtp-Source: AGHT+IELZKStGDOewbMhMKQXYjEgojB3YTHEncj6bkT9hXQVpVd17G8D5h8ZPCxfXozhzI5viSbfnw== X-Received: by 2002:a05:600c:3e18:b0:439:930a:58a6 with SMTP id 5b1f17b1804b1-441d44ed3c7mr6738575e9.8.1746606475997; Wed, 07 May 2025 01:27:55 -0700 (PDT) Received: from NAUSH-P-DELL.pitowers.org ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-441d440fff6sm22182575e9.25.2025.05.07.01.27.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 May 2025 01:27:55 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Cc: Naushir Patuck Subject: [RFC PATCH 0/6] Eliminating startup frames Date: Wed, 7 May 2025 08:52:07 +0100 Message-ID: <20250507082753.2306743-1-naush@raspberrypi.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi, From the earliest days, the Raspberry Pi pipeline handler had the ability for the pipeline handler to drop a number of frames on startup or re-configuration. This is used to hide frames that should not be consumed by application (for reasons discussed below). However, this code to handle dropping frames silently complicates the pipeline handler, and more crucially requires additional buffer allocations that we don't really want to add for more memory-limited platforms such as the Raspberry Pi Zero. Just removing the drop frame handling from the pipeline handler is not enough as we have to have a way to signal applications not to consume these frames. This RFC patch series removes our drop frame handling logic, but still provides the user or application with the necessary metadata on whether a frame should be consumed or not. There are 2 main reasons for these frame drops on startup. First, some sensors produce complete garbage output when startup up, either on a power cycle, or going from stream-off to stream-on. These frames are completely unusable and should be discarded. Second, some IQ algorithms, e.g. AWB/AGC/ALSC, will take a finite number of frames to converge to a stable state and before this time the IQ will likely be wildly oscillating. Note that this initial startup converging phase is quite different from the runtime converging that can be determined by, for example the AeState == AeStateConverged metadata control. The latter condition is generally a smooth softly filtered transition whereas the former is much more aggressive to converge as quickly as possible. Also, an application may not know what algorithms are running and need converging on startup, so they cannot rely solely on these controls changing state to determine if a frame is usable. For this RFC patch, I propose to use FrameMetadata::status for pipeline handler to signal either of these states, and rely on the application to drop the frames if it requires. Status::FrameError is overloaded to include the sensor producing bad frames on startup, and the new Status::FrameStartup is used to indicate the IQ algorithms are in a startup state. Patch 1 updates the libcamera API to add the new state, and all subsequent patches strip out the drop frame logic in the Raspberry Pi pipeline handler and IPA. As always, thoughts and comments are welcome, either through email or perhaps during our F2F next week! Regards, Naush Naushir Patuck (6): libcamera: framebuffer: Add FrameMetadata::Status::FrameStartup ipa: rpi: Replace dropFrameCount in the IPA -> PH interface pipeline: ipa: rpi: Split RPiCameraData::dropFrameCount_ pipeline: rpi: Remove disable_startup_frame_drops config option pipeline: rpi: Remove ispOutputCount_ and ispOutputTotal_ ipa: rpi: Rename dropFrameCount_ to startupCount_ include/libcamera/framebuffer.h | 1 + include/libcamera/ipa/raspberrypi.mojom | 3 +- src/ipa/rpi/common/ipa_base.cpp | 20 ++-- src/ipa/rpi/common/ipa_base.h | 4 +- src/libcamera/framebuffer.cpp | 11 ++- .../pipeline/rpi/common/pipeline_base.cpp | 95 ++++++++----------- .../pipeline/rpi/common/pipeline_base.h | 14 +-- .../pipeline/rpi/pisp/data/example.yaml | 5 - src/libcamera/pipeline/rpi/pisp/pisp.cpp | 12 --- .../pipeline/rpi/vc4/data/example.yaml | 5 - src/libcamera/pipeline/rpi/vc4/vc4.cpp | 15 --- 11 files changed, 66 insertions(+), 119 deletions(-)