[v1,00/33] Full dewarper support on imx8mp

24497 mbox series

Hi all,

the imx8mp has a very flexible dewarper that can be used to implement
cropping, digital zoom, pan, rotate, flip and actual lens dewarping. This
series implements all these features for libcamera.

Control wise, the functionality is split into multiple aspects.
ScalerCrop is now implemented using the vertex map, because that
provides pixel perfect control for arbitrary rectangles which is an
improvement over the current solution that only supports a limited set
of scaling factors. On top of that a few specific Dw100 specific
controls are added to implement the other functions. Lens dewarp is
based on calibration data in the tuning file and can be enabled/disabled
at runtime.

As prerequisite it requires an updated kernel driver that supports
programming the vertex map at runtime. The corresponding implementation
is well underway, but not yet upstream.

You can find the current version of the kernel driver here:
https://git.uk.ideasonboard.com/sklug/linux/src/branch/sklug/v6.16/topic/dw100-v2

Now to the series. Patches 2-4 replace plain pointers to media devices
by shared pointers. The patches were already posted here:
https://patchwork.libcamera.org/project/libcamera/list/?series=5113 and
review should continue there.

Patches 5-10 contain more plumbing and add support for V4L2 requests in
libcamera.

Patches 11-32 contain the actual implementation and I guess it is best
to read the individual patches.

Patch 33 shall not be mainlined and is only there to keep compatibility
with prior versions of the kernel driver that didn't use V4L2 requests.

As always there are things to improve but functionality wise it is well
tested. So I think it is a good time to get more eyes on the code.

Best regards,
Stefan



Kieran Bingham (4):
  libcamera: rkisp1: Only connect delayed controls at start/stop
  libcamera: v4l2: Support fromEntityName with shared_ptr<MediaDevice>
  libcamera: pipeline: utilise shared MediaDevice pointers
  libcamera: v4l2: Remove fromEntityName(MediaDevice*)

Stefan Klug (26):
  libcamera: converter: Utilise shared MediaDevice pointers
  libcamera: Add support for V4L2 requests
  libcamera: converter: Add V4L2 request support
  libcamera: converter_v4l2_m2m: Add suport for V4L2 requests
  libcamera: converter_v4l2_m2m: Always set stride
  pipeline: rkisp1: Use V4L2 requests for the dewarper
  libcamera: rkisp1: Properly cancel buffers in dewarp case
  libcamera: converter_v4l2_m2m: Add debug logging for formats
  libcamera: rkisp1: Move useDewarper_ flag into RkISP1CameraData
  libcamera: rkisp1: Scale down in dewarper instead of resizer
  libcamera: rkisp1: Allow upscaling when the dewarper is present
  libcamera: converter: converter_v4l2_m2m: Add makeStream() function
  libcamera: converter: Add dw100 vertex map class
  libcamera: converter_dw100: Use vertex map
  libcamera: rkisp1: Use the dw100 class instead of the generic v4l2
    converter
  libcamera: rkisp1: Implement dw100 specific features
  libcamera: rkisp1: Use vertex map to implement ScalerCrop
  libcamera: internal: camera_sensor: Add accessor for
    mountingOrientation_
  libcamera: Add transpose() function to size
  libcamera: rkisp1: Handle requested orientation using dewarper
  libcamera: dw100_vertexmap:  Implement parametric dewarping
  pipeline: rkisp1: Enable the dewarper based on the tuning file
  pipeline: rkisp1: Load dewarp parameters from tuning file
  libcamera: Add LensDewarpEnable control
  libcamera: rkisp1: Implement LensDewarpEnable control
  DNI pipeline: rkisp1: Workaround for customer kernels without requests

Umang Jain (2):
  libcamera: converter_v4l2_m2m: Add helper to apply controls
  libcamera: converter: Add dw100 converter class

Xavier Roumegue (1):
  include: linux: Update headers for Dw100 dewarper engine

 include/libcamera/geometry.h                  |   6 +
 include/libcamera/internal/camera_sensor.h    |   1 +
 include/libcamera/internal/converter.h        |  13 +-
 .../internal/converter/converter_dw100.h      |  45 ++
 .../converter/converter_dw100_vertexmap.h     |  97 ++++
 .../internal/converter/converter_v4l2_m2m.h   |  28 +-
 .../libcamera/internal/converter/meson.build  |   2 +
 include/libcamera/internal/media_device.h     |   7 +
 include/libcamera/internal/meson.build        |   1 +
 include/libcamera/internal/pipeline_handler.h |   8 +-
 include/libcamera/internal/v4l2_device.h      |   5 +-
 include/libcamera/internal/v4l2_request.h     |  49 ++
 include/libcamera/internal/v4l2_subdevice.h   |   2 +-
 include/libcamera/internal/v4l2_videodevice.h |   5 +-
 include/linux/dw100.h                         |  14 +
 src/ipa/libipa/module.h                       |   4 +
 src/libcamera/control_ids_core.yaml           |   6 +
 src/libcamera/control_ids_draft.yaml          |  39 +-
 src/libcamera/converter.cpp                   |   8 +-
 src/libcamera/converter/converter_dw100.cpp   |  99 ++++
 .../converter/converter_dw100_vertexmap.cpp   | 449 +++++++++++++++
 .../converter/converter_v4l2_m2m.cpp          |  80 ++-
 src/libcamera/converter/meson.build           |   2 +
 src/libcamera/geometry.cpp                    |  10 +
 src/libcamera/media_device.cpp                |  47 ++
 src/libcamera/meson.build                     |   1 +
 src/libcamera/pipeline/imx8-isi/imx8-isi.cpp  |   2 +-
 src/libcamera/pipeline/ipu3/cio2.cpp          |   2 +-
 src/libcamera/pipeline/ipu3/cio2.h            |   2 +-
 src/libcamera/pipeline/ipu3/imgu.cpp          |   3 +-
 src/libcamera/pipeline/ipu3/imgu.h            |   4 +-
 src/libcamera/pipeline/ipu3/ipu3.cpp          |   4 +-
 src/libcamera/pipeline/mali-c55/mali-c55.cpp  |   2 +-
 src/libcamera/pipeline/rkisp1/rkisp1.cpp      | 523 ++++++++++++++----
 src/libcamera/pipeline/rkisp1/rkisp1_path.cpp |   2 +-
 src/libcamera/pipeline/rkisp1/rkisp1_path.h   |   2 +-
 .../pipeline/rpi/common/pipeline_base.cpp     |   6 +-
 .../pipeline/rpi/common/pipeline_base.h       |   9 +-
 src/libcamera/pipeline/rpi/pisp/pisp.cpp      |  10 +-
 src/libcamera/pipeline/rpi/vc4/vc4.cpp        |  13 +-
 src/libcamera/pipeline/simple/simple.cpp      |  15 +-
 src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |   6 +-
 src/libcamera/pipeline/vimc/vimc.cpp          |   6 +-
 src/libcamera/pipeline_handler.cpp            |  14 +-
 src/libcamera/sensor/camera_sensor_legacy.cpp |   1 +
 src/libcamera/sensor/camera_sensor_raw.cpp    |   1 +
 src/libcamera/v4l2_device.cpp                 |  28 +-
 src/libcamera/v4l2_request.cpp                | 107 ++++
 src/libcamera/v4l2_subdevice.cpp              |   2 +-
 src/libcamera/v4l2_videodevice.cpp            |  12 +-
 test/delayed_controls.cpp                     |   2 +-
 test/libtest/buffer_source.cpp                |   2 +-
 52 files changed, 1606 insertions(+), 202 deletions(-)
 create mode 100644 include/libcamera/internal/converter/converter_dw100.h
 create mode 100644 include/libcamera/internal/converter/converter_dw100_vertexmap.h
 create mode 100644 include/libcamera/internal/v4l2_request.h
 create mode 100644 include/linux/dw100.h
 create mode 100644 src/libcamera/converter/converter_dw100.cpp
 create mode 100644 src/libcamera/converter/converter_dw100_vertexmap.cpp
 create mode 100644 src/libcamera/v4l2_request.cpp