From patchwork Mon Jan 20 00:24:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 2688 Return-Path: Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E076F60454 for ; Mon, 20 Jan 2020 01:24:42 +0100 (CET) Received: from pendragon.bb.dnainternet.fi (81-175-216-236.bb.dnainternet.fi [81.175.216.236]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 20857504 for ; Mon, 20 Jan 2020 01:24:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1579479882; bh=W/UQ/hA3M+jQVgiF4jfspF3OIahwe215bVt756Qadfg=; h=From:To:Subject:Date:From; b=d+9/rc1PqrjUtizIo+1ek9fohSw0z9Hg9iPcJm/EP17LO72lH1+udG5JQZ+L8Iozi lJlY3nVBwW48jjgHTT7LM1lU1ODDLztgqoKr/5/Dhg4ff/nlMwE+j8N3v/H0pEhFdO VZM6LVva/mCqWn9u5MIoM07WwvG3z6X1vX0vCdys= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Mon, 20 Jan 2020 02:24:18 +0200 Message-Id: <20200120002437.6633-1-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 00/19] Initial libcamera threading model X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jan 2020 00:24:43 -0000 Hello, This series defines and implements an initial version of the threading model. libcamera currently relies on the application providing an event loop. This cumbersome when the direct user of libcamera is not an application but a framework, as no event loop is available in that case. The Android camera HAL and the V4L2 adaptation layers had to create an internal thread to make an event loop available to libcamera, and handle synchronization between calls from the upper framework and the internal thread. A GStreamer element would require a similar implementation, resulting in lots of code duplication. Furthermore, relying on an application event loop may block libcamera from processing events if the application performs long blocking operations. This is not compatible with the real-time constraints of pipeline handlers and IPAs. For this reason, creating and managing threads internally to libcamera is a better option. This requires defining a threading model, as all of a sudden multiple contexts of execution will be available when only one context existed before. The first 7 patches are preparatory work in the form of small miscellaneous fixes or cleanups (01/19 and 05/19 to 07/19) and enhancements (making ConnectionTypeBlocking more useful in 02/19, and better log support for multithreading contexts in 03/19 and 04/19). Patch 08/19 then adds a test case to the signal test to catch an existing bug, and patch 09/19 fixes the bug while reworking the Signal implementation. The next two patches consist solely of documentation. Patch 10/19 defines the threading model and patch 11/19 documents it in core classes. Patch 12/19 then makes signal connection and disconnection thread-safe to support multi-threaded contexts. Patch 13/19 is the most disruptive change, as it creates an thread for the CameraManager. At the moment this is the only thread created by libcamera, and all pipeline handlers and camera are bound to it. This model is likely too simple and will need to be extended with more threads in the future to avoid low-priority blocking operations interfering with real-time camera constraints. The change is however a step in the right direction, as it brings consideration of multiple threads to all of libcamera. Patches 14/19 to 17/19 then document and implement the threading model in the Camera and PipelineHandler classes. As a side effect, along with 13/19, they move private data members of the CameraManager and Camera classes to private classes, helping with ABI breakage avoidance in future work. Note that patch 13/19 breaks operation of libcamera, and patch 16/19 fixes that. This is caused by the threading model implementation being split in two patches. I have decided to organize the changes this way to keep changes reviewable, but I'm open to alternatives. Patches 18/19 and 19/19 then simplify the V4L2 compatibility and Android camera HAL layers by removing the internal threads, as libcamera doesn't require an externally-provided event loop anymore. I have tested 18/19 with yavta, but 19/19 is currently only compile-tested. Laurent Pinchart (19): test: buffer_import: Propagate status code from buffer allocation libcamera: bound_method: Avoid deadlock with ConnectionTypeBlocking libcamera: thread: Add a method to return the ID of the current thread libcamera: log: Print the thread ID in the log libcamera: Replace ARRAY_SIZE with std::array libcamera: bound_method: Use std::index_sequence libcamera: Declare static local variables as const where applicable test: signal: Add additional disconnection tests for Object libcamera: signal: Make slots list private libcamera: Define the threading model libcamera: Document thread-safety attributes of core classes libcamera: signal: Make connection and disconnection thread-safe libcamera: camera_manager: Run the camera manager in a thread libcamera: camera: Move private data members to private implementation libcamera: camera: Centralize state checks in Private class libcamera: camera: Implement the threading model libcamera: pipeline_handler: Implement the threading model v4l2: Remove internal thread android: Remove internal thread Documentation/Doxyfile.in | 5 +- include/libcamera/bound_method.h | 38 +-- include/libcamera/camera.h | 28 +- include/libcamera/camera_manager.h | 13 +- include/libcamera/object.h | 4 +- include/libcamera/signal.h | 88 +++--- src/android/camera3_hal.cpp | 8 +- src/android/camera_device.cpp | 48 +-- src/android/camera_device.h | 17 +- src/android/camera_hal_manager.cpp | 78 ++--- src/android/camera_hal_manager.h | 16 +- src/android/camera_ops.cpp | 96 ++++++ src/android/camera_ops.h | 15 + src/android/camera_proxy.cpp | 180 ----------- src/android/camera_proxy.h | 42 --- src/android/meson.build | 2 +- src/libcamera/bound_method.cpp | 22 +- src/libcamera/camera.cpp | 364 +++++++++++++++-------- src/libcamera/camera_manager.cpp | 295 +++++++++++++----- src/libcamera/controls.cpp | 2 +- src/libcamera/device_enumerator.cpp | 2 + src/libcamera/event_notifier.cpp | 2 + src/libcamera/formats.cpp | 2 +- src/libcamera/framebuffer_allocator.cpp | 49 +-- src/libcamera/include/pipeline_handler.h | 4 +- src/libcamera/include/thread.h | 2 + src/libcamera/include/utils.h | 2 - src/libcamera/ipa_module.cpp | 8 +- src/libcamera/ipc_unixsocket.cpp | 2 + src/libcamera/log.cpp | 21 +- src/libcamera/object.cpp | 17 +- src/libcamera/pipeline_handler.cpp | 41 ++- src/libcamera/signal.cpp | 69 +++++ src/libcamera/thread.cpp | 106 +++++++ src/libcamera/timer.cpp | 20 +- src/libcamera/utils.cpp | 5 - src/libcamera/v4l2_videodevice.cpp | 9 +- src/v4l2/v4l2_camera.h | 2 +- src/v4l2/v4l2_camera_proxy.cpp | 43 +-- src/v4l2/v4l2_compat_manager.cpp | 48 +-- src/v4l2/v4l2_compat_manager.h | 13 +- test/camera/buffer_import.cpp | 18 +- test/ipc/unixsocket.cpp | 8 +- test/object-invoke.cpp | 20 ++ test/signal.cpp | 24 ++ 45 files changed, 1112 insertions(+), 786 deletions(-) create mode 100644 src/android/camera_ops.cpp create mode 100644 src/android/camera_ops.h delete mode 100644 src/android/camera_proxy.cpp delete mode 100644 src/android/camera_proxy.h