[libcamera-devel,00/22] IPA: IPU3: Introduce per-frame controls

14469 mbox series

Hello,

This patch series is not only introducing per-frame controls, it also
improves the AGC in some difficult cases (saturated scenes, backlight,
etc.) thanks to the possibility to use the effective exposure and gains
values for a given frame.

The first two patches 1/22 and 2/22 are Han-Lin Chan series, splitted to
keep only the sensor controls part. I have modified the first patch to
remove the lens controls references, and another series containing only
the lens controls should be sent separately as it requires a bit more
review. Han-Lin, I hope you are ok with this plan :-) ?

Next, from patches 3/22 to 9/22 we use those sensor controls as input
when a EventStatReady event is received, by updating the frameContext
values. With those values, we can be better at estimating the exposure
value, and we introduce a brightness measure to have a better behaviour
in saturated cases.

Patches 10/22 to 18/22 are the per-frame control usage. Instead of using
the current input frame value, we populate a frameContext queue which is
introduced in patch 14/22 giving the ability to know the values in a
request lifetime. In order to not break the previous usage of
EventStatReady to pass the sensor controls, we keep those in the
pipeline handler.

Patches 19/22 to 22/22 are here to improve the per-frame control updates
by the IPA, introducing the colour temperature control, and updating the
ExposureTime and AnalogueGain controls.

Han-Lin Chen (2):
  ipa: ipu3: Extend ipu3 ipa interface for sensor controls
  ipu3: ipa: Report effective sensor controls with statistics to IPA

Jean-Michel Hautbois (20):
  ipa: ipu3: Use sensor controls to update frameContext
  ipa: ipu3: agc: Limit the number of saturated cells
  ipa: ipu3: agc: Compute the gain for each frame
  ipa: ipu3: agc: Refactor ev gain calculation and testing
  ipa: ipu3: agc: Improve gain calculation
  ipa: ipu3: agc: Update previous exposure value
  ipa: ipu3: agc: Use exposure in time for storage
  ipu3: Move delayedControls reset after IPA is started
  ipu3: Populate the EventProcessCall with the sensor controls
  ipa: ipu3: Introduce a frame index for the IPAFrameContext
  ipa: ipu3: Do not access IPAFrameContext in configure
  ipa: ipu3: Introduce a queue of IPAFrameContext
  ipa: ipu3: Mark the beginning and and of a frame
  ipa: ipu3: Introduce sensor controls
  ipa: ipu3: Use a IPAFrameContext pointer from the per-frame queue
  ipa: ipu3: Move the sensor update controls
  ipa: ipu3: awb: Add support for color temperature
  ipa: ipu3: Send color temperature in the metadata
  ipa: ipu3: Move ExposureTime to IPA
  ipa: ipu3: Pass the AnalogueGain control

 include/libcamera/ipa/ipu3.mojom         |   2 +
 src/ipa/ipu3/algorithms/agc.cpp          | 161 +++++++++++++++--------
 src/ipa/ipu3/algorithms/agc.h            |  10 +-
 src/ipa/ipu3/algorithms/awb.cpp          |  15 ++-
 src/ipa/ipu3/algorithms/tone_mapping.cpp |  13 +-
 src/ipa/ipu3/ipa_context.cpp             |   8 ++
 src/ipa/ipu3/ipa_context.h               |   6 +-
 src/ipa/ipu3/ipu3.cpp                    | 101 ++++++++++++--
 src/libcamera/pipeline/ipu3/frames.h     |   3 +
 src/libcamera/pipeline/ipu3/ipu3.cpp     |  14 +-
 10 files changed, 247 insertions(+), 86 deletions(-)