[libcamera-devel,RFC,RFC,DNI] utils: tuning: Draft a new tuning infrastructure

17394 diff mbox series

This patch introduces the interface of a new tuning infrastructure, with
samples of how I imagine different platforms would use it (aptly named
"libtuning" for now) to create their own tuning scripts.

The samples include (in reverse order of how they appear in the patch,
but in semantic order of how I want to show them):
- rkisp1, which for now only has ALSC, and comes with most of the
  descriptions in the form of comments
- raspberrypi's alsc_only, which is similar to rkisp1's, to show how
  different platforms could implement similar-but-slightly-different
  tuning scripts without much duplication
- raspberrypi's main one, which only has ALSC and a dummy AWB one for
  now

The two raspberrypi tuning scripts have a shared ALSC component, which
is in a separate file to show where platforms could place custom
components if libtuning doesn't support exactly what the platform's
tuning script wants.

Obviously none of this runs (hence the DNI) and none of it is
implemented yet, but I wanted to show how it would be used first and to
gather comments on it.

There are some questions nested in the descriptions comments as well.

Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
---
 utils/tuning/libtuning/__init__.py         |  0
 utils/tuning/libtuning/modules/__init__.py |  0
 utils/tuning/raspberrypi.py                | 37 ++++++++++++
 utils/tuning/raspberrypi/__init__.py       |  0
 utils/tuning/raspberrypi/alsc.py           | 16 ++++++
 utils/tuning/raspberrypi_alsc_only.py      | 16 ++++++
 utils/tuning/rkisp1.py                     | 67 ++++++++++++++++++++++
 7 files changed, 136 insertions(+)
 create mode 100644 utils/tuning/libtuning/__init__.py
 create mode 100644 utils/tuning/libtuning/modules/__init__.py
 create mode 100644 utils/tuning/raspberrypi.py
 create mode 100644 utils/tuning/raspberrypi/__init__.py
 create mode 100644 utils/tuning/raspberrypi/alsc.py
 create mode 100644 utils/tuning/raspberrypi_alsc_only.py
 create mode 100644 utils/tuning/rkisp1.py

Hi Paul,

On Mon, Sep 26, 2022 at 02:09:31PM +0900, Paul Elder wrote:
> This patch introduces the interface of a new tuning infrastructure, with
> samples of how I imagine different platforms would use it (aptly named
> "libtuning" for now) to create their own tuning scripts.
> 
> The samples include (in reverse order of how they appear in the patch,
> but in semantic order of how I want to show them):
> - rkisp1, which for now only has ALSC, and comes with most of the
>   descriptions in the form of comments
> - raspberrypi's alsc_only, which is similar to rkisp1's, to show how
>   different platforms could implement similar-but-slightly-different
>   tuning scripts without much duplication
> - raspberrypi's main one, which only has ALSC and a dummy AWB one for
>   now
> 
> The two raspberrypi tuning scripts have a shared ALSC component, which
> is in a separate file to show where platforms could place custom
> components if libtuning doesn't support exactly what the platform's
> tuning script wants.
> 
> Obviously none of this runs (hence the DNI) and none of it is
> implemented yet, but I wanted to show how it would be used first and to
> gather comments on it.
> 
> There are some questions nested in the descriptions comments as well.
> 
> Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
> ---
>  utils/tuning/libtuning/__init__.py         |  0
>  utils/tuning/libtuning/modules/__init__.py |  0
>  utils/tuning/raspberrypi.py                | 37 ++++++++++++
>  utils/tuning/raspberrypi/__init__.py       |  0
>  utils/tuning/raspberrypi/alsc.py           | 16 ++++++
>  utils/tuning/raspberrypi_alsc_only.py      | 16 ++++++
>  utils/tuning/rkisp1.py                     | 67 ++++++++++++++++++++++
>  7 files changed, 136 insertions(+)
>  create mode 100644 utils/tuning/libtuning/__init__.py
>  create mode 100644 utils/tuning/libtuning/modules/__init__.py
>  create mode 100644 utils/tuning/raspberrypi.py
>  create mode 100644 utils/tuning/raspberrypi/__init__.py
>  create mode 100644 utils/tuning/raspberrypi/alsc.py
>  create mode 100644 utils/tuning/raspberrypi_alsc_only.py
>  create mode 100644 utils/tuning/rkisp1.py
> 
> diff --git a/utils/tuning/libtuning/__init__.py b/utils/tuning/libtuning/__init__.py
> new file mode 100644
> index 00000000..e69de29b
> diff --git a/utils/tuning/libtuning/modules/__init__.py b/utils/tuning/libtuning/modules/__init__.py
> new file mode 100644
> index 00000000..e69de29b
> diff --git a/utils/tuning/raspberrypi.py b/utils/tuning/raspberrypi.py
> new file mode 100644
> index 00000000..322b9c36
> --- /dev/null
> +++ b/utils/tuning/raspberrypi.py
> @@ -0,0 +1,37 @@
> +import sys
> +
> +import libtuning as lt
> +from libtuning.modules import AWB
> +from libtuning.parsers import RaspberryPiParser
> +from libtuning.generators import RaspberryPiOutput
> +
> +from raspberrypi.alsc import ALSC as RaspberryPiALSC
> +
> +tuner = lt.Camera()

Nitpicking on naming, it feels weird to name a variable "tuner" when it
is of type "Camera".

> +
> +# These modules can also be custom modules. libtuning will come with utilities
> +# for handling stuff like images, so there shouldn't be too much boilerplate
> +# involved in creating custom modules, though I don't yet have a concrete
> +# vision on how custom implementations of modules would look.
> +tuner.add(RaspberryPiALSC)
> +
> +# Other tuning modules can be added like so.
> +# The order that the tuning modules will be executed is determined by the order
> +# that they're added.
> +# This is kind of an implementation detail, but the "context" is saved
> +# internally in lt.Camera, so modules that are added (and therefore executed)
> +# later can use the output of the previous modules. I'm thinking that a module
> +# that depends on values from another module has two modes of execution, for
> +# when those values are available and another for when they're not. Not quite
> +# sure concretely how to handle this yet.
> +tuner.add(AWB( # module parameters
> +               ))
> +
> +tuner.setInputType(RaspberryPiParser)
> +tuner.setOutputType(RaspberryPiOutput)
> +
> +# The order of the output doesn't necessarily have to be the same as the order
> +# of input, which is specified by the order of adding the modules above.
> +tuner.setOutputOrder(AWB, ALSC)
> +
> +tuner.run(sys.argv)
> diff --git a/utils/tuning/raspberrypi/__init__.py b/utils/tuning/raspberrypi/__init__.py
> new file mode 100644
> index 00000000..e69de29b
> diff --git a/utils/tuning/raspberrypi/alsc.py b/utils/tuning/raspberrypi/alsc.py
> new file mode 100644
> index 00000000..ff8a02fd
> --- /dev/null
> +++ b/utils/tuning/raspberrypi/alsc.py
> @@ -0,0 +1,16 @@
> +ALSC(do_color = lt.paramsIfUnset(True), \
> +               debug = [lt.debug.Plot], \
> +               luminance_strength = lt.params.IfUnSet(0.5) \
> +               sector_shape = (16, 12), \
> +               sector_x_gradient = lt.gradients.Linear, \
> +               sector_y_gradient = lt.gradients.Linear, \
> +               sector_x_remainder = lt.remainder.Append, \
> +               sector_y_remainder = lt.remainder.Append, \
> +               sector_average_function = lt.average_functions.Mean, \
> +               smoothing_function = lt.smoothing.MedianBlur, \
> +               smoothing_parameters = [3], \
> +               output_type = lt.type.Float, \
> +               output_color_channels = [lt.color.R, lt.color.G, lt.color.B], \
> +               output_luminance_channels = [lt.color.G] \
> +               output_range = [0, 3.999] \
> +               )
> diff --git a/utils/tuning/raspberrypi_alsc_only.py b/utils/tuning/raspberrypi_alsc_only.py
> new file mode 100644
> index 00000000..11a1fc23
> --- /dev/null
> +++ b/utils/tuning/raspberrypi_alsc_only.py
> @@ -0,0 +1,16 @@
> +import sys
> +
> +import libtuning as lt
> +from libtuning.modules import ALSC
> +from libtuning.parsers import RaspberryPiParser
> +from libtuning.generators import RaspberryPiOutput
> +
> +from raspberrypi.alsc import ALSC as RaspberryPiALSC
> +
> +tuner = lt.Camera()
> +tuner.add(RaspberryPiALSC)
> +tuner.setInputType(RaspberryPiParser)
> +tuner.setOutputType(RaspberryPiOutput)
> +tuner.setOutputOrder(ALSC)
> +
> +tuner.run(sys.argv)
> diff --git a/utils/tuning/rkisp1.py b/utils/tuning/rkisp1.py
> new file mode 100644
> index 00000000..8e26beba
> --- /dev/null
> +++ b/utils/tuning/rkisp1.py
> @@ -0,0 +1,67 @@
> +import sys
> +
> +import libtuning as lt
> +from libtuning.modules import ALSC
> +from libtuning.parsers import YamlParser
> +from libtuning.generators import YamlOutput
> +
> +tuner = lt.Camera()
> +tuner.add(ALSC(do_color = lt.paramsIfUnset(True), \
> +
> +               # This can support other debug options (I can't think of any rn
> +               # but for future-proofing)
> +               debug = [lt.debug.Plot], \
> +
> +               # The name of IfUnSet can obviously be changed, but really I
> +               # just want something that says "it must be specified in the
> +               # configuration, and get the value from there" or "if the value
> +               # is not in the configuration, use this"
> +               luminance_strength = lt.params.IfUnSet(0.5) \
> +
> +               sector_shape = (16, 16), \
> +
> +               # Other functions might include Circular, Hyperbolic, Gaussian,
> +               # Linear, Exponential, Logarithmic, etc
> +               # Of course, both need not be the same function
> +               # Some functions would need a sector_x_parameter (eg. sigma for Gaussian)
> +               # Alternatively: sector_x_sizes = [] ? I don't think it would work tho
> +               sector_x_gradient = lt.gradients.Parabolic, \
> +               sector_y_gradient = lt.gradients.Parabolic, \
> +
> +               # This is the function that will be used to average the pixels in each sector
> +               # This can also be a custom function.
> +               sector_average_function = lt.average_functions.Mean, \
> +
> +               # This is the function that will be used to smooth the color ratio values
> +               # This can also be a custom function.
> +               smoothing_function = lt.smoothing.MedianBlur, \
> +               smoothing_parameters = [3], \
> +
> +               # Are there any platforms that use integer values for their lsc table?
> +               output_type = lt.type.Float, \
> +
> +               # Required if and only if do_color can be or is True
> +               output_color_channels = [lt.color.R, lt.color.GR, lt.color.GB, lt.color.B], \
> +
> +               # Required if and only if do_color can be or is False
> +               output_luminance_channels = [lt.color.GR, lt.color.GB], \
> +
> +               # Automatically get the precision from this
> +               output_range = [0, 3.999] \
> +
> +               # Do we need a flag to enable/disable calculating sigmas? afaik
> +               # the rkisp1 IPA doesn't use it? But we could output it anyway
> +               # and algorithms can decide whether or not if they want to use
> +               # it. Oh well, no flag for now, we can always add it later if
> +               # there's a big demand, plus it's only two to three values and
> +               # not an entire table.
> +               ))
> +tuner.setInputType(YamlParser)
> +tuner.setOutputType(YamlOutput)
> +tuner.setOutputOrder(ALSC)

I may be missing something obvious, but isn't tuning supposed to use
images ? :-)

> +
> +# Maybe this should be wrapped in an if __main__ = '__main__' ? That way the

__name__ == '__main__'

> +# developer can control which tuner they want to be executed based on another
> +# layer of arguments? But I was thinking that this would handle *all* arguments
> +# based on the modules' and the input/output configurations.
> +tuner.run(sys.argv)

It looks like you have quite a few ideas on how to implement all this,
but based on this skeleton only, I have a hard time understanding them
:-) It would make sense to me to move forward a bit more and then
continue the discussion.