[libcamera-devel,RFC,v2,0/3] lc-compliance: Refactor using Googletest

This is a prototype of the refactoring of lc-compliance using Googletest as the framework. Patches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3 does the actual refactor. In v1 the tests were registered statically which simplified the work needed to add new tests, but it required the camera pointer to be a global variable, which wouldn't scale for usage with multiple cameras. For v2 the tests are registered dynamically so that a camera pointer can be passed to tests being instantiated. This requires manual registration of each parametrization of the tests, which I tried to make less cumbersome by defining the REGISTER_TESTS() macro (although for tests with different parameters, different macros would be needed...). It also requires each new test to declare a new class and be described in the TestBody() function of the class. Perhaps this could also be made easier with a macro? Also, since now a test is created for each of the cameras in the system, the -c flag was removed, as the camera to be tested can be selected by filtering only tests for that camera with --gtest_filter. Although we could re-add this flag later for convenience. The parameters for Gtest have yet to be wired up, but can currently be passed after double dashes, like lc-compliance -c '<camera_name>' -- --gtest_list_tests to list all current tests defined. The parameter --gtest_filter can be used to filter the individual tests or test suites to run. It uses string matching and globing for the test names so it's easy to run a whole test suite MyTestSuite with --gtest_list_tests='*MyTestSuite*'. This is a snippet of a successful run: [==========] Running 240 tests from 1 test suite. [----------] Global test environment set-up. [----------] 240 tests from SingleStream [ RUN ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 [20:42:43.656235633] [105482] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG Camera needs 4 requests, can't test only 1 [ SKIPPED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 (63 ms) [...] [ RUN ] SingleStream.Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.2-0bda:5511/Viewfinder/89 [20:51:53.678488142] [105482] INFO Camera camera.cpp:915 configuring streams: (0) 576x360-R8 [ OK ] SingleStream.Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.2-0bda:5511/Viewfinder/89 (6390 ms) [----------] 240 tests from SingleStream (556412 ms total) [----------] Global test environment tear-down [==========] 240 tests from 1 test suite ran. (556412 ms total) [ PASSED ] 192 tests. [ SKIPPED ] 48 tests, listed below: [ SKIPPED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 [ SKIPPED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 [...] And how a failure looks like: [ RUN ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 [20:55:13.218088360] [106518] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG ../src/lc-compliance/simple_capture.cpp:117: Failure Expected equality of these values: captureCount_ Which is: 5 captureLimit_ Which is: 6 [ FAILED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 (318 ms) The current complete list of tests is shown below for a single camera (each test will show once for each of the cameras, which makes the list very lengthy if multiple cameras are present): SingleStream. BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/89 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/89 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/89 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/89 Changes in v2: - Added patch 1 to make Camera::stop() idempotent as well from [1]. - Made test registration dynamic in patch 3 - Some other minor fixes [1] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html v1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html Nícolas F. R. A. Prado (3): libcamera: camera: Make stop() idempotent lc-compliance: Make SimpleCapture::stop() idempotent lc-compliance: Refactor using Googletest src/lc-compliance/main.cpp | 99 +++++++++--------- src/lc-compliance/meson.build | 3 + src/lc-compliance/simple_capture.cpp | 95 +++++++---------- src/lc-compliance/simple_capture.h | 8 +- src/lc-compliance/single_stream.cpp | 151 ++++++++++++++------------- src/lc-compliance/tests.h | 15 ++- src/libcamera/camera.cpp | 20 +++- 7 files changed, 199 insertions(+), 192 deletions(-)

Message ID	20210503193307.108607-1-nfraprado@collabora.com
Headers	show Return-Path: <libcamera-devel-bounces@lists.libcamera.org> sender: nfraprado) by bhuna.collabora.co.uk (Postfix) with ESMTPSA id 474E81F41E95; Mon, 3 May 2021 20:33:55 +0100 (BST) From: =?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= <nfraprado@collabora.com> To: libcamera-devel@lists.libcamera.org Date: Mon, 3 May 2021 16:33:04 -0300 Message-Id: <20210503193307.108607-1-nfraprado@collabora.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 0/3] lc-compliance: Refactor using Googletest Precedence: list Cc: kernel@collabora.com, =?utf-8?q?Andr=C3=A9_Almeida?= <andrealmeid@collabora.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" <libcamera-devel-bounces@lists.libcamera.org>
Series	lc-compliance: Refactor using Googletest
Related	show [libcamera-devel,RFC,v2,0/3] lc-compliance: Refactor using Googletest [libcamera-devel,RFC,v2,1/3] libcamera: camera: Make stop() idempotent [libcamera-devel,RFC,v2,2/3] lc-compliance: Make SimpleCapture::stop() idempotent [libcamera-devel,RFC,v2,3/3] lc-compliance: Refactor using Googletest

[libcamera-devel,RFC,v2,0/3] lc-compliance: Refactor using Googletest 12188 mbox series

Message

[libcamera-devel,RFC,v2,0/3] lc-compliance: Refactor using Googletest
mbox series