[v2,00/35] Full dewarper support on imx8mp

24728 mbox series

Hi all,

the imx8mp has a very flexible dewarper that can be used to implement
cropping, digital zoom, pan, rotate, flip and actual lens dewarping. This
series implements all these features for libcamera.

Control wise, the functionality is split into multiple aspects.
ScalerCrop is now implemented using the vertex map, because that
provides pixel perfect control for arbitrary rectangles which is an
improvement over the current solution that only supports a limited set
of scaling factors. On top of that a few specific Dw100 specific
controls are added to implement the other functions. Lens dewarp is
based on calibration data in the tuning file and can be enabled/disabled
at runtime.

As prerequisite it requires an updated kernel driver that supports
programming the vertex map at runtime. The corresponding implementation
is well underway, but not yet upstream.

You can find the current version of the kernel driver here:
https://git.uk.ideasonboard.com/sklug/linux/src/branch/sklug/v6.16/topic/dw100-v2

Changes in v2:

In the discussions around v2 it got clear that the extended controls
need further discussions and can not be merged on mainline as is. I
moved them to the end of the series and marked them with DNI. The rest
of the series is valuable in itself as it brings pixel perfect ScalerCrop
and lens dewarping support without changing much on the controls side.

The vertexmap code was rewritten to use affine transforms and matrices
insted of manual transformation. This makes the code way cleaner and was
on my todo list anyways.

In the orientation handling a bug was fixed if the transformation
involved a transpose. This bug was visible by a camera configuration
that validated successfully, but the camera failed to configure.

Also a lot of documentation improvements were done and the series passes
CI successfully: https://gitlab.freedesktop.org/camera/libcamera/-/pipelines/1533035

For smaller fixes please see the changelogs on the individual patches.


Patch summary:

Patches 1-3 replace plain pointers to media devices
by shared pointers. The patches were already posted here:
https://patchwork.libcamera.org/project/libcamera/list/?series=5113 and
review should continue there.

Patches 4-9 contain more plumbing and add support for V4L2 requests in
libcamera.

Patches 10-33 contain the actual implementation and I guess it is best
to read the individual patches.

Patch 34-35 shall not be mainlined and is only there to keep
compatibility with prior versions of the kernel driver that didn't use
V4L2 requests and to show the extended controls that are possible with
the dw100.

Best regards,
Stefan

Kieran Bingham (3):
  libcamera: v4l2: Support fromEntityName with shared_ptr<MediaDevice>
  libcamera: pipeline: utilise shared MediaDevice pointers
  libcamera: v4l2: Remove fromEntityName(MediaDevice*)

Stefan Klug (29):
  libcamera: converter: Utilise shared MediaDevice pointers
  libcamera: Add support for V4L2 requests
  libcamera: converter: Add V4L2 request support
  libcamera: converter_v4l2_m2m: Add suport for V4L2 requests
  libcamera: converter_v4l2_m2m: Always set stride
  pipeline: rkisp1: Use V4L2 requests for the dewarper
  libcamera: rkisp1: Properly cancel buffers in dewarp case
  libcamera: converter_v4l2_m2m: Add debug logging for formats
  libcamera: rkisp1: Move useDewarper_ flag into RkISP1CameraData
  libcamera: rkisp1: Scale down in dewarper instead of resizer
  libcamera: rkisp1: Allow upscaling when the dewarper is present
  libcamera: converter: converter_v4l2_m2m: Add makeStream() function
  libcamera: converter_m2m: Make some parts protected
  libcamera: converter: Add dw100 vertex map class
  libcamera: converter_dw100: Use vertex map
  libcamera: rkisp1: Use the dw100 class instead of the generic v4l2
    converter
  libcamera: rkisp1: Use vertex map to implement ScalerCrop
  pipeline: rksip1: Move isRaw up in scope
  pipeline: rkisp1: Drop rawFormat variable
  pipeline: rkisp1: Enable the dewarper based on the tuning file
  libcamera: internal: camera_sensor: Add accessor for
    mountingOrientation_
  libcamera: Add transpose() function to size
  libcamera: rkisp1: Handle requested orientation using dewarper
  libcamera: dw100_vertexmap: Implement parametric dewarping
  pipeline: rkisp1: Load dewarp parameters from tuning file
  libcamera: Add and implement LensDewarpEnable control
  pipeline: rkisp1: Add warning to unclear format handling
  DNI libcamera: rkisp1: Implement dw100 specific features
  DNI pipeline: rkisp1: Workaround for customer kernels without requests

Umang Jain (2):
  libcamera: converter_v4l2_m2m: Add helper to apply controls
  libcamera: converter: Add dw100 converter class

Xavier Roumegue (1):
  include: linux: Update headers for Dw100 dewarper engine

 include/libcamera/geometry.h                  |   6 +
 include/libcamera/internal/camera_sensor.h    |   1 +
 include/libcamera/internal/converter.h        |  13 +-
 .../internal/converter/converter_dw100.h      |  43 ++
 .../converter/converter_dw100_vertexmap.h     |  92 +++
 .../internal/converter/converter_v4l2_m2m.h   |  29 +-
 .../libcamera/internal/converter/meson.build  |   2 +
 include/libcamera/internal/media_device.h     |   7 +
 include/libcamera/internal/meson.build        |   1 +
 include/libcamera/internal/pipeline_handler.h |   8 +-
 include/libcamera/internal/v4l2_device.h      |   5 +-
 include/libcamera/internal/v4l2_request.h     |  49 ++
 include/libcamera/internal/v4l2_subdevice.h   |   2 +-
 include/libcamera/internal/v4l2_videodevice.h |   5 +-
 include/linux/dw100.h                         |  14 +
 src/ipa/libipa/module.h                       |   4 +
 src/libcamera/control_ids_core.yaml           |   7 +
 src/libcamera/control_ids_draft.yaml          |  45 ++
 src/libcamera/converter.cpp                   |   8 +-
 src/libcamera/converter/converter_dw100.cpp   | 111 +++
 .../converter/converter_dw100_vertexmap.cpp   | 652 ++++++++++++++++++
 .../converter/converter_v4l2_m2m.cpp          | 202 +++++-
 src/libcamera/converter/meson.build           |   2 +
 src/libcamera/geometry.cpp                    |  10 +
 src/libcamera/media_device.cpp                |  47 ++
 src/libcamera/meson.build                     |   1 +
 src/libcamera/pipeline/imx8-isi/imx8-isi.cpp  |   2 +-
 src/libcamera/pipeline/ipu3/cio2.cpp          |   2 +-
 src/libcamera/pipeline/ipu3/cio2.h            |   2 +-
 src/libcamera/pipeline/ipu3/imgu.cpp          |   3 +-
 src/libcamera/pipeline/ipu3/imgu.h            |   4 +-
 src/libcamera/pipeline/ipu3/ipu3.cpp          |   4 +-
 src/libcamera/pipeline/mali-c55/mali-c55.cpp  |   2 +-
 src/libcamera/pipeline/rkisp1/rkisp1.cpp      | 564 +++++++++++----
 src/libcamera/pipeline/rkisp1/rkisp1_path.cpp |   2 +-
 src/libcamera/pipeline/rkisp1/rkisp1_path.h   |   2 +-
 .../pipeline/rpi/common/pipeline_base.cpp     |   6 +-
 .../pipeline/rpi/common/pipeline_base.h       |   9 +-
 src/libcamera/pipeline/rpi/pisp/pisp.cpp      |  10 +-
 src/libcamera/pipeline/rpi/vc4/vc4.cpp        |  13 +-
 src/libcamera/pipeline/simple/simple.cpp      |  15 +-
 src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |   6 +-
 src/libcamera/pipeline/vimc/vimc.cpp          |   6 +-
 src/libcamera/pipeline_handler.cpp            |  14 +-
 src/libcamera/sensor/camera_sensor.cpp        |   7 +
 src/libcamera/sensor/camera_sensor_legacy.cpp |   1 +
 src/libcamera/sensor/camera_sensor_raw.cpp    |   1 +
 src/libcamera/v4l2_device.cpp                 |  28 +-
 src/libcamera/v4l2_request.cpp                | 128 ++++
 src/libcamera/v4l2_subdevice.cpp              |   2 +-
 src/libcamera/v4l2_videodevice.cpp            |  12 +-
 test/delayed_controls.cpp                     |   2 +-
 test/libtest/buffer_source.cpp                |   2 +-
 53 files changed, 1992 insertions(+), 223 deletions(-)
 create mode 100644 include/libcamera/internal/converter/converter_dw100.h
 create mode 100644 include/libcamera/internal/converter/converter_dw100_vertexmap.h
 create mode 100644 include/libcamera/internal/v4l2_request.h
 create mode 100644 include/linux/dw100.h
 create mode 100644 src/libcamera/converter/converter_dw100.cpp
 create mode 100644 src/libcamera/converter/converter_dw100_vertexmap.cpp
 create mode 100644 src/libcamera/v4l2_request.cpp