[RFC,0/6] Eliminating startup frames
mbox series

Message ID 20250507082753.2306743-1-naush@raspberrypi.com
Headers show
Series
  • Eliminating startup frames
Related show

Message

Naushir Patuck May 7, 2025, 7:52 a.m. UTC 
Hi,

From the earliest days, the Raspberry Pi pipeline handler had the ability for
the pipeline handler to drop a number of frames on startup or re-configuration.
This is used to hide frames that should not be consumed by application (for
reasons discussed below).  However, this code to handle dropping frames silently
complicates the pipeline handler, and more crucially requires additional buffer
allocations that we don't really want to add for more memory-limited platforms
such as the Raspberry Pi Zero. Just removing the drop frame handling from the
pipeline handler is not enough as we have to have a way to signal applications
not to consume these frames.

This RFC patch series removes our drop frame handling logic, but still provides
the user or application with the necessary metadata on whether a frame should be
consumed or not.

There are 2 main reasons for these frame drops on startup.  First, some sensors
produce complete garbage output when startup up, either on a power cycle, or
going from stream-off to stream-on. These frames are completely unusable and
should be discarded.

Second, some IQ algorithms, e.g. AWB/AGC/ALSC, will take a finite number of
frames to converge to a stable state and before this time the IQ will likely be
wildly oscillating.  Note that this initial startup converging phase is quite
different from the runtime converging that can be determined by, for example the
AeState == AeStateConverged metadata control.  The latter condition is generally
a smooth softly filtered transition whereas the former is much more aggressive
to converge as quickly as possible. Also, an application may not know what
algorithms are running and need converging on startup, so they cannot rely
solely on these controls changing state to determine if a frame is usable.

For this RFC patch, I propose to use FrameMetadata::status for pipeline handler
to signal either of these states, and rely on the application to drop the frames
if it requires. Status::FrameError is overloaded to include the sensor producing
bad frames on startup, and the new Status::FrameStartup is used to indicate the
IQ algorithms are in a startup state.

Patch 1 updates the libcamera API to add the new state, and all subsequent
patches strip out the drop frame logic in the Raspberry Pi pipeline handler and
IPA.

As always, thoughts and comments are welcome, either through email or perhaps
during our F2F next week!

Regards,
Naush

Naushir Patuck (6):
  libcamera: framebuffer: Add FrameMetadata::Status::FrameStartup
  ipa: rpi: Replace dropFrameCount in the IPA -> PH interface
  pipeline: ipa: rpi: Split RPiCameraData::dropFrameCount_
  pipeline: rpi: Remove disable_startup_frame_drops config option
  pipeline: rpi: Remove ispOutputCount_ and ispOutputTotal_
  ipa: rpi: Rename dropFrameCount_ to startupCount_

 include/libcamera/framebuffer.h               |  1 +
 include/libcamera/ipa/raspberrypi.mojom       |  3 +-
 src/ipa/rpi/common/ipa_base.cpp               | 20 ++--
 src/ipa/rpi/common/ipa_base.h                 |  4 +-
 src/libcamera/framebuffer.cpp                 | 11 ++-
 .../pipeline/rpi/common/pipeline_base.cpp     | 95 ++++++++-----------
 .../pipeline/rpi/common/pipeline_base.h       | 14 +--
 .../pipeline/rpi/pisp/data/example.yaml       |  5 -
 src/libcamera/pipeline/rpi/pisp/pisp.cpp      | 12 ---
 .../pipeline/rpi/vc4/data/example.yaml        |  5 -
 src/libcamera/pipeline/rpi/vc4/vc4.cpp        | 15 ---
 11 files changed, 66 insertions(+), 119 deletions(-)