[v8,45/47] ipa: libipa: agc: Work without `CameraSensorHelper`

28077 diff mbox series

`AgcMeanLuminance` can operate without a `CameraSensorHelper`, but in that
case it assumes an "ideal" gain model, where gains are truly continuous,
and a gain of x will apply a gain of exactly x to the result. This is not
a good fit when only gain codes are available.

So now that the agc algorithm from the simple ipa has been extracted (AgcMSV),
let's use that in `AgcAlgorithm` to be able to always provide some level of
automatic exposure/gain control. Even though the main use case for operating
without a known gain model is empirically determining the gain model using
the manual controls.

Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 src/ipa/ipu3/algorithms/agc.cpp     |   4 +-
 src/ipa/libipa/agc.cpp              | 202 +++++++++++++++++++---------
 src/ipa/libipa/agc.h                |  10 +-
 src/ipa/mali-c55/algorithms/agc.cpp |   4 +-
 src/ipa/rkisp1/algorithms/agc.cpp   |   4 +-
 5 files changed, 148 insertions(+), 76 deletions(-)

Hi Barnabás,

Quoting Barnabás Pőcze (2026-08-24 11:14:04)
> `AgcMeanLuminance` can operate without a `CameraSensorHelper`, but in that
> case it assumes an "ideal" gain model, where gains are truly continuous,
> and a gain of x will apply a gain of exactly x to the result. This is not
> a good fit when only gain codes are available.
> 
> So now that the agc algorithm from the simple ipa has been extracted (AgcMSV),
> let's use that in `AgcAlgorithm` to be able to always provide some level of
> automatic exposure/gain control. Even though the main use case for operating
> without a known gain model is empirically determining the gain model using
> the manual controls.
> 
> Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
> Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>

Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>

Best regards,
Stefan

> ---
>  src/ipa/ipu3/algorithms/agc.cpp     |   4 +-
>  src/ipa/libipa/agc.cpp              | 202 +++++++++++++++++++---------
>  src/ipa/libipa/agc.h                |  10 +-
>  src/ipa/mali-c55/algorithms/agc.cpp |   4 +-
>  src/ipa/rkisp1/algorithms/agc.cpp   |   4 +-
>  5 files changed, 148 insertions(+), 76 deletions(-)
> 
> diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp
> index 29a562d667..22846514be 100644
> --- a/src/ipa/ipu3/algorithms/agc.cpp
> +++ b/src/ipa/ipu3/algorithms/agc.cpp
> @@ -66,8 +66,7 @@ Agc::Agc()
>   */
>  int Agc::init(IPAContext &context, const ValueNode &tuningData)
>  {
> -       return agc_.init(tuningData, {
> -               .sensor = context.camHelper.get(),
> +       return agc_.init(tuningData, context.camHelper.get(), {
>                 .sensorInfo = context.sensorInfo,
>                 .sensorControls = context.sensorControls,
>                 .ctrlMap = context.ctrlMap,
> @@ -88,7 +87,6 @@ int Agc::configure(IPAContext &context,
>         bdsGrid_ = context.configuration.grid.bdsGrid;
>  
>         return agc_.configure(context.configuration.agc, context.activeState.agc, {
> -               .sensor = context.camHelper.get(),
>                 .sensorInfo = context.sensorInfo,
>                 .sensorControls = context.sensorControls,
>                 .ctrlMap = context.ctrlMap,
> diff --git a/src/ipa/libipa/agc.cpp b/src/ipa/libipa/agc.cpp
> index 1c096c984d..9f37e1b8ee 100644
> --- a/src/ipa/libipa/agc.cpp
> +++ b/src/ipa/libipa/agc.cpp
> @@ -13,11 +13,13 @@
>  #include <cmath>
>  #include <optional>
>  #include <ratio>
> +#include <variant>
>  
>  #include <linux/v4l2-controls.h>
>  
>  #include <libcamera/base/log.h>
>  #include <libcamera/base/span.h>
> +#include <libcamera/base/utils.h>
>  
>  #include <libcamera/control_ids.h>
>  #include <libcamera/controls.h>
> @@ -80,6 +82,9 @@ namespace agc {
>   * \var Session::maxAnalogueGain
>   * \brief Maximum analogue gain for the streaming session
>   *
> + * \var Session::defAnalogueGain
> + * \brief Default analogue gain of the configured sensor
> + *
>   * \var Session::minFrameDuration
>   * \brief Minimum frame duration for the streaming session
>   *
> @@ -222,10 +227,9 @@ namespace agc {
>   *
>   * The AgcAlgorithm class can be used to implement automatic exposure/gain
>   * control in an IPA module, conforming to the prescribed Algorithm interface.
> - * Internally AgcMeanLuminance is used, this class merely concerns itself with
> - * processing the sensor properties, establishing limits, managing controls,
> - * providing the resulting metadata, and driving the actual algorithm in
> - * process().
> + * Internally AgcMeanLuminance or AgcMSV is used, this class merely concerns
> + * itself with processing the sensor properties, establishing limits, managing
> + * controls, and providing the resulting metadata.
>   *
>   * Users should compose the agc::Session, agc::ActiveState, and agc::FrameContext
>   * structures into their session configuration, active state, and frame contexts,
> @@ -241,9 +245,6 @@ namespace agc {
>   * \struct AgcAlgorithm::ConfigurationParams
>   * \brief Parameters for AgcAlgorithm::configure()
>   *
> - * \var AgcAlgorithm::ConfigurationParams::sensor
> - * \brief CameraSensorHelper for the sensor
> - *
>   * \var AgcAlgorithm::ConfigurationParams::sensorInfo
>   * \brief Current configuration of the sensor
>   *
> @@ -288,24 +289,35 @@ namespace agc {
>  /**
>   * \brief Load tuning data and configure
>   * \param[in] tuningData The tuning data
> + * \param[in] sensor The camera sensor helper
>   * \param[in] config The algorithm configuration
>   *
>   * This function loads the tuning data and configures the algorithm as if
>   * by a call to configure(), in order to provide the available controls
>   * in ConfigurationParams::ctrlMap.
>   *
> - * The tuning data format is that of AgcMeanLuminance. Refer to
> - * AgcMeanLuminance::parseTuningData() for more details.
> + * If a CameraSensorHelper is provided in \a sensor, an instance of
> + * AgcMeanLuminance is instantiated and the tuning data is loaded by
> + * calling AgcMeanLuminance::parseTuningData(). Otherwise an instance
> + * of AgcMSV is instantiated, and in this case the tuning data is ignored.
>   *
>   * \return 0 on success, or a negative error code
>   *
>   * \sa Algorithm::init()
>   */
> -int AgcAlgorithm::init(const ValueNode &tuningData, const ConfigurationParams &config)
> +int AgcAlgorithm::init(const ValueNode &tuningData, CameraSensorHelper *sensor,
> +                      const ConfigurationParams &config)
>  {
> -       int ret = impl_.parseTuningData(tuningData);
> -       if (ret)
> -               return ret;
> +       if (sensor) {
> +               auto &impl = impl_.emplace<AgcMeanLuminance>();
> +               int ret = impl.parseTuningData(tuningData);
> +               if (ret)
> +                       return ret;
> +       } else {
> +               impl_.emplace<AgcMSV>();
> +       }
> +
> +       sensor_ = sensor;
>  
>         /*
>          * The purpose of this `configure()` is merely to provide the
> @@ -361,10 +373,14 @@ int AgcAlgorithm::configure(agc::Session &session, agc::ActiveState &state,
>         int32_t defExposure = v4l2Exposure.def().get<int32_t>();
>  
>         /* Compute the analogue gain limits. */
> +       const auto extractGain = [&](const ControlValue &v) {
> +               auto gainCode = v.get<int32_t>();
> +               return sensor_ ? sensor_->gain(gainCode) : gainCode;
> +       };
>         const ControlInfo &v4l2Gain = config.sensorControls.find(V4L2_CID_ANALOGUE_GAIN)->second;
> -       float minGain = config.sensor->gain(v4l2Gain.min().get<int32_t>());
> -       float maxGain = config.sensor->gain(v4l2Gain.max().get<int32_t>());
> -       float defGain = config.sensor->gain(v4l2Gain.def().get<int32_t>());
> +       float minGain = extractGain(v4l2Gain.min());
> +       float maxGain = extractGain(v4l2Gain.max());
> +       float defGain = extractGain(v4l2Gain.def());
>  
>         LOG(Agc, Debug)
>                 << "exposure: [" << minExposure << ',' << maxExposure << "], "
> @@ -403,28 +419,21 @@ int AgcAlgorithm::configure(agc::Session &session, agc::ActiveState &state,
>         session.maxExposureTime = maxExposure * session.lineDuration;
>         session.minAnalogueGain = minGain;
>         session.maxAnalogueGain = maxGain;
> +       session.defAnalogueGain = defGain;
>         session.minFrameDuration = std::chrono::microseconds(frameDurations[0]);
>         session.maxFrameDuration = std::chrono::microseconds(frameDurations[1]);
>  
> -       impl_.configure(session.lineDuration, config.sensor);
> -       impl_.resetFrameCount();
> -
>         /* Configure the default exposure and gain. */
>         state = {};
>         state.automatic.gain = session.minAnalogueGain;
>         state.automatic.exposure = defExposure;
>         state.automatic.quantizationGain = 1;
>         state.automatic.digitalGain = 1;
> -       state.automatic.yTarget = impl_.effectiveYTarget(0, 1);
>         state.manual.gain = state.automatic.gain;
>         state.manual.exposure = state.automatic.exposure;
>         state.autoExposureEnabled = session.autoAllowed;
>         state.autoGainEnabled = session.autoAllowed;
>         state.exposureValue = 0;
> -       state.constraintMode =
> -               static_cast<controls::AeConstraintModeEnum>(impl_.constraintModes().begin()->first);
> -       state.exposureMode =
> -               static_cast<controls::AeExposureModeEnum>(impl_.exposureModeHelpers().begin()->first);
>         state.minFrameDuration = session.minFrameDuration;
>         state.maxFrameDuration = session.maxFrameDuration;
>  
> @@ -475,19 +484,52 @@ int AgcAlgorithm::configure(agc::Session &session, agc::ActiveState &state,
>         add(controls::AnalogueGainMode,
>             controls::AnalogueGainModeAuto, controls::AnalogueGainModeManual);
>  
> -       if (session.autoAllowed) {
> -               config.ctrlMap[&controls::ExposureValue] = ControlInfo(-8.0f, 8.0f, 0.0f);
> +       std::visit(utils::overloaded{
> +               [&](AgcMSV &) {
> +                       /* No constraint/exposure mode support. */
> +                       state.constraintMode = controls::AeConstraintModeEnum::ConstraintNormal;
> +                       state.exposureMode = controls::AeExposureModeEnum::ExposureNormal;
>  
> -               std::vector<ControlValue> options;
> -               for (const auto &[id, _] : impl_.constraintModes())
> -                       options.emplace_back(id);
> -               config.ctrlMap[&controls::AeConstraintMode] = ControlInfo(options);
> +                       state.automatic.yTarget = 0; /* Not supported. */
>  
> -               options.clear();
> -               for (const auto &[id, _] : impl_.exposureModeHelpers())
> -                       options.emplace_back(id);
> -               config.ctrlMap[&controls::AeExposureMode] = ControlInfo(options);
> -       }
> +                       if (!session.autoAllowed)
> +                               return;
> +
> +                       config.ctrlMap[&controls::AeConstraintMode] = ControlInfo(
> +                               std::array{ ControlValue(state.constraintMode) }
> +                       );
> +
> +                       config.ctrlMap[&controls::AeExposureMode] = ControlInfo(
> +                               std::array{ ControlValue(state.exposureMode) }
> +                       );
> +               },
> +               [&](AgcMeanLuminance &impl) {
> +                       state.constraintMode =
> +                               static_cast<controls::AeConstraintModeEnum>(impl.constraintModes().begin()->first);
> +                       state.exposureMode =
> +                               static_cast<controls::AeExposureModeEnum>(impl.exposureModeHelpers().begin()->first);
> +
> +                       state.automatic.yTarget = impl.effectiveYTarget(0, 1);
> +
> +                       impl.configure(session.lineDuration, sensor_);
> +                       impl.resetFrameCount();
> +
> +                       if (!session.autoAllowed)
> +                               return;
> +
> +                       config.ctrlMap[&controls::ExposureValue] = ControlInfo(-8.0f, 8.0f, 0.0f);
> +
> +                       std::vector<ControlValue> options;
> +                       for (const auto &[id, _] : impl.constraintModes())
> +                               options.emplace_back(id);
> +                       config.ctrlMap[&controls::AeConstraintMode] = ControlInfo(options);
> +
> +                       options.clear();
> +                       for (const auto &[id, _] : impl.exposureModeHelpers())
> +                               options.emplace_back(id);
> +                       config.ctrlMap[&controls::AeExposureMode] = ControlInfo(options);
> +               },
> +       }, impl_);
>  
>         return 0;
>  }
> @@ -718,36 +760,68 @@ void AgcAlgorithm::process(const agc::Session &session, agc::ActiveState &state,
>                 maxAnalogueGain = frameContext.gain;
>         }
>  
> -       /*
> -        * The Agc algorithm needs to know the effective exposure value that was
> -        * applied to the sensor when the statistics were collected.
> -        */
> -       utils::Duration effectiveExposureValue =
> -               lineDuration * params->exposure * params->gain;
> -
> -       impl_.setLimits(minExposureTime, maxExposureTime,
> -                       minAnalogueGain, maxAnalogueGain,
> -                       std::move(params->additionalConstraints));
> -
> -       const auto &newEv = impl_.calculateNewEv({
> -               .traits = params->traits,
> -               .yHist = params->yHist,
> -               .effectiveExposureValue = effectiveExposureValue,
> -               .constraintModeIndex = frameContext.constraintMode,
> -               .exposureModeIndex = frameContext.exposureMode,
> -               .lux = params->lux,
> -               .exposureCompensation = std::pow(2.0, frameContext.exposureValue),
> -       });
> -
> -       /* Update the estimated exposure and gain. */
> -       state.automatic.exposure = newEv.exposureTime / lineDuration;
> -       state.automatic.gain = newEv.analogueGain;
> -       state.automatic.quantizationGain = newEv.quantizationGain;
> -       state.automatic.digitalGain = newEv.digitalGain;
> -       state.automatic.yTarget = newEv.yTarget;
> +       std::visit(utils::overloaded{
> +               [&](AgcMSV& impl) {
> +                       impl.setLimits({
> +                               .exposure = {
> +                                       static_cast<uint32_t>(minExposureTime / lineDuration),
> +                                       static_cast<uint32_t>(maxExposureTime / lineDuration),
> +                               },
> +                               .gain = {
> +                                       minAnalogueGain,
> +                                       maxAnalogueGain,
> +                               },
> +                               /* gain codes -> step size of 1 */
> +                               .gainMinStep = 1,
> +                               /* assume default gain is close to 1.0 */
> +                               .gain1 = session.defAnalogueGain,
> +                       });
> +
> +                       const auto& newEv = impl.calculateNewEv({
> +                               .yHist = params->yHist,
> +                               .exposure = params->exposure,
> +                               .gain = params->gain,
> +                       });
> +
> +                       state.automatic.exposure = newEv.exposure;
> +                       state.automatic.gain = newEv.analogueGain;
> +               },
> +               [&](AgcMeanLuminance& impl) {
> +                       /*
> +                        * The Agc algorithm needs to know the effective exposure
> +                        * value that was applied to the sensor when the statistics
> +                        * were collected.
> +                        */
> +                       utils::Duration effectiveExposureValue =
> +                               lineDuration * params->exposure * params->gain;
> +
> +                       impl.setLimits(minExposureTime, maxExposureTime,
> +                                      minAnalogueGain, maxAnalogueGain,
> +                                      std::move(params->additionalConstraints));
> +
> +                       const auto &newEv = impl.calculateNewEv({
> +                               .traits = params->traits,
> +                               .yHist = params->yHist,
> +                               .effectiveExposureValue = effectiveExposureValue,
> +                               .constraintModeIndex = frameContext.constraintMode,
> +                               .exposureModeIndex = frameContext.exposureMode,
> +                               .lux = params->lux,
> +                               .exposureCompensation = std::pow(2.0, frameContext.exposureValue),
> +                       });
> +
> +                       /* Update the estimated exposure and gain. */
> +                       state.automatic.exposure = newEv.exposureTime / lineDuration;
> +                       state.automatic.gain = newEv.analogueGain;
> +                       state.automatic.quantizationGain = newEv.quantizationGain;
> +                       state.automatic.digitalGain = newEv.digitalGain;
> +                       state.automatic.yTarget = newEv.yTarget;
> +               },
> +       }, impl_);
> +
> +       const utils::Duration newExposureTime = state.automatic.exposure * lineDuration;
>  
>         LOG(Agc, Debug)
> -               << "exposure-time: " << newEv.exposureTime << ", "
> +               << "exposure-time: " << newExposureTime << ", "
>                 << "analogue-gain: " << state.automatic.gain << ", "
>                 << "quantization-gain: " << state.automatic.quantizationGain << ", "
>                 << "digital-gain: " << state.automatic.digitalGain;
> @@ -757,7 +831,7 @@ void AgcAlgorithm::process(const agc::Session &session, agc::ActiveState &state,
>          * the minimum frame duration when we have short exposures.
>          */
>         processFrameDuration(session, frameContext,
> -                            std::max(frameContext.minFrameDuration, newEv.exposureTime));
> +                            std::max(frameContext.minFrameDuration, newExposureTime));
>  
>         fillMetadata(session, frameContext, metadata);
>  }
> diff --git a/src/ipa/libipa/agc.h b/src/ipa/libipa/agc.h
> index 7c5a7c3518..e820077e7a 100644
> --- a/src/ipa/libipa/agc.h
> +++ b/src/ipa/libipa/agc.h
> @@ -11,6 +11,7 @@
>  #include <stdint.h>
>  #include <utility>
>  #include <vector>
> +#include <variant>
>  
>  #include <linux/v4l2-controls.h>
>  
> @@ -21,6 +22,7 @@
>  #include <libcamera/geometry.h>
>  
>  #include "agc_mean_luminance.h"
> +#include "agc_msv.h"
>  #include "camera_sensor_helper.h"
>  
>  namespace libcamera {
> @@ -38,6 +40,7 @@ struct Session {
>         utils::Duration maxExposureTime;
>         double minAnalogueGain;
>         double maxAnalogueGain;
> +       double defAnalogueGain;
>         utils::Duration minFrameDuration;
>         utils::Duration maxFrameDuration;
>         utils::Duration lineDuration;
> @@ -119,7 +122,6 @@ class AgcAlgorithm
>  {
>  public:
>         struct ConfigurationParams {
> -               const CameraSensorHelper *sensor;
>                 const IPACameraSensorInfo &sensorInfo;
>                 const ControlInfoMap &sensorControls;
>                 ControlInfoMap::Map &ctrlMap;
> @@ -135,7 +137,8 @@ public:
>                 double lux = 0;
>         };
>  
> -       int init(const ValueNode &tuningData, const ConfigurationParams &config);
> +       int init(const ValueNode &tuningData, CameraSensorHelper *sensor,
> +                const ConfigurationParams &config);
>  
>         int configure(agc::Session &session, agc::ActiveState &state,
>                       const ConfigurationParams &config);
> @@ -157,7 +160,8 @@ private:
>                           const agc::FrameContext &frameContext,
>                           ControlList &metadata);
>  
> -       AgcMeanLuminance impl_;
> +       std::variant<AgcMSV, AgcMeanLuminance> impl_;
> +       CameraSensorHelper *sensor_ = nullptr;
>  };
>  
>  } /* namespace ipa */
> diff --git a/src/ipa/mali-c55/algorithms/agc.cpp b/src/ipa/mali-c55/algorithms/agc.cpp
> index 96e0a4c7e9..e977ef75bd 100644
> --- a/src/ipa/mali-c55/algorithms/agc.cpp
> +++ b/src/ipa/mali-c55/algorithms/agc.cpp
> @@ -122,8 +122,7 @@ Agc::Agc()
>  
>  int Agc::init(IPAContext &context, const ValueNode &tuningData)
>  {
> -       return agc_.init(tuningData, {
> -               .sensor = context.camHelper.get(),
> +       return agc_.init(tuningData, context.camHelper.get(), {
>                 .sensorInfo = context.sensorInfo,
>                 .sensorControls = context.sensorControls,
>                 .ctrlMap = context.ctrlMap,
> @@ -138,7 +137,6 @@ int Agc::configure(IPAContext &context,
>                 return ret;
>  
>         ret = agc_.configure(context.configuration.agc, context.activeState.agc, {
> -               .sensor = context.camHelper.get(),
>                 .sensorInfo = context.sensorInfo,
>                 .sensorControls = context.sensorControls,
>                 .ctrlMap = context.ctrlMap,
> diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp
> index bb0345ca87..79a7474cb2 100644
> --- a/src/ipa/rkisp1/algorithms/agc.cpp
> +++ b/src/ipa/rkisp1/algorithms/agc.cpp
> @@ -136,8 +136,7 @@ int Agc::init(IPAContext &context, const ValueNode &tuningData)
>  {
>         int ret;
>  
> -       ret = agc_.init(tuningData, {
> -               .sensor = context.camHelper.get(),
> +       ret = agc_.init(tuningData, context.camHelper.get(), {
>                 .sensorInfo = context.sensorInfo,
>                 .sensorControls = context.sensorControls,
>                 .ctrlMap = context.ctrlMap,
> @@ -163,7 +162,6 @@ int Agc::init(IPAContext &context, const ValueNode &tuningData)
>  int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo)
>  {
>         int ret = agc_.configure(context.configuration.agc, context.activeState.agc, {
> -               .sensor = context.camHelper.get(),
>                 .sensorInfo = context.sensorInfo,
>                 .sensorControls = context.sensorControls,
>                 .ctrlMap = context.ctrlMap,
> -- 
> 2.55.0
>