[libcamera-devel,v5,1/1] libcamera: controls: Add controls for HDR

19185 diff mbox series

We add an HdrMode control (to enable and disable HDR processing)
and an HdrChannel, which indicates what kind of HDR frame (short, long
or medium) has just arrived.

Currently the HdrMode supports the following values:

* Off - no HDR processing at all.
* MultiExposureUnmerged - frames at multiple different exposures are
  produced, but not merged together. They are returned "as is".
* MultiExposure - frames at multiple different exposures are merged
  to create HDR images.
* SingleExposure - multiple frames all at the same exposure are
  merged to create HDR images.
* Night - multiple frames will be combined to create "night mode"
  images.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
---
 src/libcamera/control_ids.yaml | 87 ++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

Hi Dave,

Thank you for the patch.

On Thu, Oct 26, 2023 at 12:11:31PM +0100, David Plowman via libcamera-devel wrote:
> We add an HdrMode control (to enable and disable HDR processing)
> and an HdrChannel, which indicates what kind of HDR frame (short, long
> or medium) has just arrived.
> 
> Currently the HdrMode supports the following values:
> 
> * Off - no HDR processing at all.
> * MultiExposureUnmerged - frames at multiple different exposures are
>   produced, but not merged together. They are returned "as is".
> * MultiExposure - frames at multiple different exposures are merged
>   to create HDR images.
> * SingleExposure - multiple frames all at the same exposure are
>   merged to create HDR images.
> * Night - multiple frames will be combined to create "night mode"
>   images.
> 
> Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
> ---
>  src/libcamera/control_ids.yaml | 87 ++++++++++++++++++++++++++++++++++
>  1 file changed, 87 insertions(+)
> 
> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> index f2e542f4..06a9128c 100644
> --- a/src/libcamera/control_ids.yaml
> +++ b/src/libcamera/control_ids.yaml
> @@ -774,6 +774,93 @@ controls:
>              Continuous AF is paused. No further state changes or lens movements
>              will occur until the AfPauseResume control is sent.
>  
> +  - HdrMode:
> +      type: int32_t
> +      description: |
> +        Control to set the mode to be used for High Dynamic Range (HDR)
> +        imaging. HDR techniques typically include multiple exposure, image
> +        fusion and tone mapping techniques to improve the dynamic range of the
> +        resulting images.
> +
> +        When using an HDR mode, images are tagged to indicate which HDR channel
> +        (long, medium or short) they come from.

Let's define the concept of channel.

	When using an HDR mode, images are captured with different sets	of AGC
	settings called HDR channels. Channels indicate in particular the type
	of exposure (short, medium or long) used to capture the raw image,
	before fusion. Each HDR image is tagged with the corresponding channel
	using the HdrChannel control.

> +
> +        \sa HdrChannel
> +
> +      enum:
> +        - name: HdrModeOff
> +          value: 0
> +          description: |
> +            HDR is disabled. Metadata for this frame will not include the
> +            HdrChannel control.
> +        - name: HdrModeMultiExposureUnmerged
> +          value: 1
> +          description: |
> +            Multiple exposures will be generated in an alternating fashion.
> +            However, they will not be merged together and will be returned to
> +            the application as they are. Each image will be tagged with the
> +            correct HDR channel, indicating what kind of exposure it is. The
> +            tag should be the same as in the HdrModeMultiExposure case.
> +
> +            The expectation is that an application using this mode would merge
> +            the frames to create HDR images for itself if it requires them.
> +        - name: HdrModeMultiExposure
> +          value: 2
> +          description: |
> +            Multiple exposures will be generated and merged to create HDR
> +            images. Each image will be tagged with the HDR channel (long, medium
> +            or short) that arrived and which caused this image to be output.
> +
> +            Systems that use two channels for HDR will return images tagged
> +            alternately as the short and long channel. Systems that use three
> +            channels for HDR will cycle through the short, medium and long
> +            channel before repeating.
> +        - name: HdrModeSingleExposure
> +          value: 3
> +          description: |
> +            Multiple frames all at a single exposure will be used to create HDR
> +            images. These images should be reported as all corresponding to the
> +            HDR short channel.
> +        - name: HdrModeNight
> +          value: 4
> +          description: |
> +            Multiple frames will be combined to produce "night mode" images. It
> +            is up to the implementation exactly which HDR channels it uses, and
> +            the images will all be tagged accordingly with the correct HDR
> +            channel information.

This is still very undefined, but that's fine, let's keep it as a
prototype and see where it leads.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

I'll wait for your opinion regarding the proposed change above before
merging this.

> +
> +  - HdrChannel:
> +      type: int32_t
> +      description: |
> +        This value is reported back to the application so that it can discover
> +        whether this capture corresponds to the short or long exposure image (or
> +        any other image used by the HDR procedure). An application can monitor
> +        the HDR channel to discover when the differently exposed images have
> +        arrived.
> +
> +        This metadata is only available when an HDR mode has been enabled.
> +
> +        \sa HdrMode
> +
> +      enum:
> +        - name: HdrChannelNone
> +          value: 0
> +          description: |
> +            This image does not correspond to any of the captures used to create
> +            an HDR image.
> +        - name: HdrChannelShort
> +          value: 1
> +          description: |
> +            This is a short exposure image.
> +        - name: HdrChannelMedium
> +          value: 2
> +          description: |
> +            This is a medium exposure image.
> +        - name: HdrChannelLong
> +          value: 3
> +          description: |
> +            This is a long exposure image.
> +
>    # ----------------------------------------------------------------------------
>    # Draft controls section
>

Hi Laurent

Thanks for the review.

On Thu, 2 Nov 2023 at 11:03, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Dave,
>
> Thank you for the patch.
>
> On Thu, Oct 26, 2023 at 12:11:31PM +0100, David Plowman via libcamera-devel wrote:
> > We add an HdrMode control (to enable and disable HDR processing)
> > and an HdrChannel, which indicates what kind of HDR frame (short, long
> > or medium) has just arrived.
> >
> > Currently the HdrMode supports the following values:
> >
> > * Off - no HDR processing at all.
> > * MultiExposureUnmerged - frames at multiple different exposures are
> >   produced, but not merged together. They are returned "as is".
> > * MultiExposure - frames at multiple different exposures are merged
> >   to create HDR images.
> > * SingleExposure - multiple frames all at the same exposure are
> >   merged to create HDR images.
> > * Night - multiple frames will be combined to create "night mode"
> >   images.
> >
> > Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> > Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
> > ---
> >  src/libcamera/control_ids.yaml | 87 ++++++++++++++++++++++++++++++++++
> >  1 file changed, 87 insertions(+)
> >
> > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > index f2e542f4..06a9128c 100644
> > --- a/src/libcamera/control_ids.yaml
> > +++ b/src/libcamera/control_ids.yaml
> > @@ -774,6 +774,93 @@ controls:
> >              Continuous AF is paused. No further state changes or lens movements
> >              will occur until the AfPauseResume control is sent.
> >
> > +  - HdrMode:
> > +      type: int32_t
> > +      description: |
> > +        Control to set the mode to be used for High Dynamic Range (HDR)
> > +        imaging. HDR techniques typically include multiple exposure, image
> > +        fusion and tone mapping techniques to improve the dynamic range of the
> > +        resulting images.
> > +
> > +        When using an HDR mode, images are tagged to indicate which HDR channel
> > +        (long, medium or short) they come from.
>
> Let's define the concept of channel.
>
>         When using an HDR mode, images are captured with different sets of AGC
>         settings called HDR channels. Channels indicate in particular the type
>         of exposure (short, medium or long) used to capture the raw image,
>         before fusion. Each HDR image is tagged with the corresponding channel
>         using the HdrChannel control.
>

I think that's fine, actually I think it's a worthwhile improvement!

Thanks!
David

> > +
> > +        \sa HdrChannel
> > +
> > +      enum:
> > +        - name: HdrModeOff
> > +          value: 0
> > +          description: |
> > +            HDR is disabled. Metadata for this frame will not include the
> > +            HdrChannel control.
> > +        - name: HdrModeMultiExposureUnmerged
> > +          value: 1
> > +          description: |
> > +            Multiple exposures will be generated in an alternating fashion.
> > +            However, they will not be merged together and will be returned to
> > +            the application as they are. Each image will be tagged with the
> > +            correct HDR channel, indicating what kind of exposure it is. The
> > +            tag should be the same as in the HdrModeMultiExposure case.
> > +
> > +            The expectation is that an application using this mode would merge
> > +            the frames to create HDR images for itself if it requires them.
> > +        - name: HdrModeMultiExposure
> > +          value: 2
> > +          description: |
> > +            Multiple exposures will be generated and merged to create HDR
> > +            images. Each image will be tagged with the HDR channel (long, medium
> > +            or short) that arrived and which caused this image to be output.
> > +
> > +            Systems that use two channels for HDR will return images tagged
> > +            alternately as the short and long channel. Systems that use three
> > +            channels for HDR will cycle through the short, medium and long
> > +            channel before repeating.
> > +        - name: HdrModeSingleExposure
> > +          value: 3
> > +          description: |
> > +            Multiple frames all at a single exposure will be used to create HDR
> > +            images. These images should be reported as all corresponding to the
> > +            HDR short channel.
> > +        - name: HdrModeNight
> > +          value: 4
> > +          description: |
> > +            Multiple frames will be combined to produce "night mode" images. It
> > +            is up to the implementation exactly which HDR channels it uses, and
> > +            the images will all be tagged accordingly with the correct HDR
> > +            channel information.
>
> This is still very undefined, but that's fine, let's keep it as a
> prototype and see where it leads.
>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> I'll wait for your opinion regarding the proposed change above before
> merging this.
>
> > +
> > +  - HdrChannel:
> > +      type: int32_t
> > +      description: |
> > +        This value is reported back to the application so that it can discover
> > +        whether this capture corresponds to the short or long exposure image (or
> > +        any other image used by the HDR procedure). An application can monitor
> > +        the HDR channel to discover when the differently exposed images have
> > +        arrived.
> > +
> > +        This metadata is only available when an HDR mode has been enabled.
> > +
> > +        \sa HdrMode
> > +
> > +      enum:
> > +        - name: HdrChannelNone
> > +          value: 0
> > +          description: |
> > +            This image does not correspond to any of the captures used to create
> > +            an HDR image.
> > +        - name: HdrChannelShort
> > +          value: 1
> > +          description: |
> > +            This is a short exposure image.
> > +        - name: HdrChannelMedium
> > +          value: 2
> > +          description: |
> > +            This is a medium exposure image.
> > +        - name: HdrChannelLong
> > +          value: 3
> > +          description: |
> > +            This is a long exposure image.
> > +
> >    # ----------------------------------------------------------------------------
> >    # Draft controls section
> >
>
> --
> Regards,
>
> Laurent Pinchart