From patchwork Wed Jun 16 13:25:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= X-Patchwork-Id: 12610 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7ED57BD78E for ; Wed, 16 Jun 2021 13:26:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EC7C468941; Wed, 16 Jun 2021 15:26:41 +0200 (CEST) Received: from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B2DD46029A for ; Wed, 16 Jun 2021 15:26:40 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2804:14c:1a9:2434:2243:2b27:a2be:44ab]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: nfraprado) by bhuna.collabora.co.uk (Postfix) with ESMTPSA id 806071F4371A; Wed, 16 Jun 2021 14:26:38 +0100 (BST) From: =?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= To: libcamera-devel@lists.libcamera.org Date: Wed, 16 Jun 2021 10:25:30 -0300 Message-Id: <20210616132535.453411-1-nfraprado@collabora.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v8 0/5] lc-compliance: Refactor using Googletest X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kernel@collabora.com, =?utf-8?q?Andr=C3=A9_Almeida?= Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This is a refactoring of lc-compliance using Googletest as the framework. Patches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3 adds an Environment singleton to provide access to the camera from all tests. Patch 4 does the actual refactor. Patch 5 adds two new parameters, --list and --filter. lc-compliance is built around testing a single camera (and possibly optional secondary cameras in the future), which is passed through the -c flag. The list of tests can be queried with: lc-compliance --list By default all tests are run, but it's possible to select which tests to run by using: lc-compliance -c '' --filter '' where the test_filter is in the same format as Googletest's --gtest_filter. Quoting its help: --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS] Run only the tests whose name matches one of the positive patterns but none of the negative patterns. '?' matches any single character; '*' matches any substring; ':' separates two patterns. This is a snippet of a successful run: [==========] Running 120 tests from 1 test suite. [----------] Global test environment set-up. [----------] 120 tests from RolesAndRequests/FixedRequestsTest [ RUN ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1 [31:53:24.860985336] [271719] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG Camera needs 4 requests, can't test only 1 [ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1 (64 ms) [...] [ RUN ] RolesAndRequests/FixedRequestsTest.Unbalanced/Viewfinder_89 [31:56:18.607757616] [271719] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG [ OK ] RolesAndRequests/FixedRequestsTest.Unbalanced/Viewfinder_89 (3148 ms) [----------] 120 tests from RolesAndRequests/FixedRequestsTest (176896 ms total) [----------] Global test environment tear-down [==========] 120 tests from 1 test suite ran. (176896 ms total) [ PASSED ] 96 tests. [ SKIPPED ] 24 tests, listed below: [ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1 [ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_2 [ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_3 [ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/StillCapture_1 [...] And how a failure looks like: [ RUN ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_5 [31:59:59.528082413] [272962] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG ../src/lc-compliance/simple_capture.cpp:117: Failure Expected equality of these values: captureCount_ Which is: 5 captureLimit_ Which is: 6 [ FAILED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_5, where GetParam() = (0, 5) (317 ms) The current complete list of tests is shown below: RolesAndRequests/FixedRequestsTest. BalancedSingleCycle/Raw_1 # GetParam() = (0, 1) BalancedSingleCycle/Raw_2 # GetParam() = (0, 2) BalancedSingleCycle/Raw_3 # GetParam() = (0, 3) BalancedSingleCycle/Raw_5 # GetParam() = (0, 5) BalancedSingleCycle/Raw_8 # GetParam() = (0, 8) BalancedSingleCycle/Raw_13 # GetParam() = (0, 13) BalancedSingleCycle/Raw_21 # GetParam() = (0, 21) BalancedSingleCycle/Raw_34 # GetParam() = (0, 34) BalancedSingleCycle/Raw_55 # GetParam() = (0, 55) BalancedSingleCycle/Raw_89 # GetParam() = (0, 89) BalancedSingleCycle/StillCapture_1 # GetParam() = (1, 1) BalancedSingleCycle/StillCapture_2 # GetParam() = (1, 2) BalancedSingleCycle/StillCapture_3 # GetParam() = (1, 3) BalancedSingleCycle/StillCapture_5 # GetParam() = (1, 5) BalancedSingleCycle/StillCapture_8 # GetParam() = (1, 8) BalancedSingleCycle/StillCapture_13 # GetParam() = (1, 13) BalancedSingleCycle/StillCapture_21 # GetParam() = (1, 21) BalancedSingleCycle/StillCapture_34 # GetParam() = (1, 34) BalancedSingleCycle/StillCapture_55 # GetParam() = (1, 55) BalancedSingleCycle/StillCapture_89 # GetParam() = (1, 89) BalancedSingleCycle/VideoRecording_1 # GetParam() = (2, 1) BalancedSingleCycle/VideoRecording_2 # GetParam() = (2, 2) BalancedSingleCycle/VideoRecording_3 # GetParam() = (2, 3) BalancedSingleCycle/VideoRecording_5 # GetParam() = (2, 5) BalancedSingleCycle/VideoRecording_8 # GetParam() = (2, 8) BalancedSingleCycle/VideoRecording_13 # GetParam() = (2, 13) BalancedSingleCycle/VideoRecording_21 # GetParam() = (2, 21) BalancedSingleCycle/VideoRecording_34 # GetParam() = (2, 34) BalancedSingleCycle/VideoRecording_55 # GetParam() = (2, 55) BalancedSingleCycle/VideoRecording_89 # GetParam() = (2, 89) BalancedSingleCycle/Viewfinder_1 # GetParam() = (3, 1) BalancedSingleCycle/Viewfinder_2 # GetParam() = (3, 2) BalancedSingleCycle/Viewfinder_3 # GetParam() = (3, 3) BalancedSingleCycle/Viewfinder_5 # GetParam() = (3, 5) BalancedSingleCycle/Viewfinder_8 # GetParam() = (3, 8) BalancedSingleCycle/Viewfinder_13 # GetParam() = (3, 13) BalancedSingleCycle/Viewfinder_21 # GetParam() = (3, 21) BalancedSingleCycle/Viewfinder_34 # GetParam() = (3, 34) BalancedSingleCycle/Viewfinder_55 # GetParam() = (3, 55) BalancedSingleCycle/Viewfinder_89 # GetParam() = (3, 89) BalancedMultiCycle/Raw_1 # GetParam() = (0, 1) BalancedMultiCycle/Raw_2 # GetParam() = (0, 2) BalancedMultiCycle/Raw_3 # GetParam() = (0, 3) BalancedMultiCycle/Raw_5 # GetParam() = (0, 5) BalancedMultiCycle/Raw_8 # GetParam() = (0, 8) BalancedMultiCycle/Raw_13 # GetParam() = (0, 13) BalancedMultiCycle/Raw_21 # GetParam() = (0, 21) BalancedMultiCycle/Raw_34 # GetParam() = (0, 34) BalancedMultiCycle/Raw_55 # GetParam() = (0, 55) BalancedMultiCycle/Raw_89 # GetParam() = (0, 89) BalancedMultiCycle/StillCapture_1 # GetParam() = (1, 1) BalancedMultiCycle/StillCapture_2 # GetParam() = (1, 2) BalancedMultiCycle/StillCapture_3 # GetParam() = (1, 3) BalancedMultiCycle/StillCapture_5 # GetParam() = (1, 5) BalancedMultiCycle/StillCapture_8 # GetParam() = (1, 8) BalancedMultiCycle/StillCapture_13 # GetParam() = (1, 13) BalancedMultiCycle/StillCapture_21 # GetParam() = (1, 21) BalancedMultiCycle/StillCapture_34 # GetParam() = (1, 34) BalancedMultiCycle/StillCapture_55 # GetParam() = (1, 55) BalancedMultiCycle/StillCapture_89 # GetParam() = (1, 89) BalancedMultiCycle/VideoRecording_1 # GetParam() = (2, 1) BalancedMultiCycle/VideoRecording_2 # GetParam() = (2, 2) BalancedMultiCycle/VideoRecording_3 # GetParam() = (2, 3) BalancedMultiCycle/VideoRecording_5 # GetParam() = (2, 5) BalancedMultiCycle/VideoRecording_8 # GetParam() = (2, 8) BalancedMultiCycle/VideoRecording_13 # GetParam() = (2, 13) BalancedMultiCycle/VideoRecording_21 # GetParam() = (2, 21) BalancedMultiCycle/VideoRecording_34 # GetParam() = (2, 34) BalancedMultiCycle/VideoRecording_55 # GetParam() = (2, 55) BalancedMultiCycle/VideoRecording_89 # GetParam() = (2, 89) BalancedMultiCycle/Viewfinder_1 # GetParam() = (3, 1) BalancedMultiCycle/Viewfinder_2 # GetParam() = (3, 2) BalancedMultiCycle/Viewfinder_3 # GetParam() = (3, 3) BalancedMultiCycle/Viewfinder_5 # GetParam() = (3, 5) BalancedMultiCycle/Viewfinder_8 # GetParam() = (3, 8) BalancedMultiCycle/Viewfinder_13 # GetParam() = (3, 13) BalancedMultiCycle/Viewfinder_21 # GetParam() = (3, 21) BalancedMultiCycle/Viewfinder_34 # GetParam() = (3, 34) BalancedMultiCycle/Viewfinder_55 # GetParam() = (3, 55) BalancedMultiCycle/Viewfinder_89 # GetParam() = (3, 89) Unbalanced/Raw_1 # GetParam() = (0, 1) Unbalanced/Raw_2 # GetParam() = (0, 2) Unbalanced/Raw_3 # GetParam() = (0, 3) Unbalanced/Raw_5 # GetParam() = (0, 5) Unbalanced/Raw_8 # GetParam() = (0, 8) Unbalanced/Raw_13 # GetParam() = (0, 13) Unbalanced/Raw_21 # GetParam() = (0, 21) Unbalanced/Raw_34 # GetParam() = (0, 34) Unbalanced/Raw_55 # GetParam() = (0, 55) Unbalanced/Raw_89 # GetParam() = (0, 89) Unbalanced/StillCapture_1 # GetParam() = (1, 1) Unbalanced/StillCapture_2 # GetParam() = (1, 2) Unbalanced/StillCapture_3 # GetParam() = (1, 3) Unbalanced/StillCapture_5 # GetParam() = (1, 5) Unbalanced/StillCapture_8 # GetParam() = (1, 8) Unbalanced/StillCapture_13 # GetParam() = (1, 13) Unbalanced/StillCapture_21 # GetParam() = (1, 21) Unbalanced/StillCapture_34 # GetParam() = (1, 34) Unbalanced/StillCapture_55 # GetParam() = (1, 55) Unbalanced/StillCapture_89 # GetParam() = (1, 89) Unbalanced/VideoRecording_1 # GetParam() = (2, 1) Unbalanced/VideoRecording_2 # GetParam() = (2, 2) Unbalanced/VideoRecording_3 # GetParam() = (2, 3) Unbalanced/VideoRecording_5 # GetParam() = (2, 5) Unbalanced/VideoRecording_8 # GetParam() = (2, 8) Unbalanced/VideoRecording_13 # GetParam() = (2, 13) Unbalanced/VideoRecording_21 # GetParam() = (2, 21) Unbalanced/VideoRecording_34 # GetParam() = (2, 34) Unbalanced/VideoRecording_55 # GetParam() = (2, 55) Unbalanced/VideoRecording_89 # GetParam() = (2, 89) Unbalanced/Viewfinder_1 # GetParam() = (3, 1) Unbalanced/Viewfinder_2 # GetParam() = (3, 2) Unbalanced/Viewfinder_3 # GetParam() = (3, 3) Unbalanced/Viewfinder_5 # GetParam() = (3, 5) Unbalanced/Viewfinder_8 # GetParam() = (3, 8) Unbalanced/Viewfinder_13 # GetParam() = (3, 13) Unbalanced/Viewfinder_21 # GetParam() = (3, 21) Unbalanced/Viewfinder_34 # GetParam() = (3, 34) Unbalanced/Viewfinder_55 # GetParam() = (3, 55) Unbalanced/Viewfinder_89 # GetParam() = (3, 89) Changes in v8: - Thanks to Laurent: - Fixed lc-compliance's meson.build to disable when gtest is not installed - Fixed compiling errors in Clang - Thanks to Kieran: - Fixed coding style issues - Switched from malloc/free to new/delete in gtest parameter allocation v7: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021253.html Changes in v7: - Thanks to Jacopo: - Fixed style issue - Made CameraManager a unique_ptr and passed to Environment as raw pointer - Moved the check for buffers allocated to the beginning of SimpleCapture::stop() - Thanks to Niklas: - Removed intermediary filter string variable in Harness::initGtestParameters() v6: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021137.html Changes in v6: - Thanks to Niklas: - Made return of Environment::cameraId() be a const reference - Changed buildGtestParameters() into initGtestParameters() and removed the need for Harness::gtestArgc_ and Harness::gtestArgv_ v5: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020759.html Changes in v5: - Thanks to Laurent: - Fixed style issues - Stored CameraManager and cameraId instead of Camera in Environment, and acquire and release the camera for each test - Thanks to Niklas: - Moved buildGtestParameters() inside run() - Thanks to Laurent & Niklas: - Improved Environment singleton class instantiation and destruction v4: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020594.html Changes in v4: - Removed old lc-compliance results classes - Thanks to Niklas: - Added patch 3 with the environment singleton in its own files - Improved naming of tests - Thanks to Jacopo: - Added \todo in Camera::stop() v3: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020382.html Changes in v3: - Thanks to Niklas: - Went back to static test registration, and created a Singleton Environment class to store the camera global variable - Added patch 4 implementing list and filter parameters - Added a nameParameters() function to give meaningful names to the test parameters, removing the need to register each test suite for every role v2: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020056.html Changes in v2: - Added patch 1 to make Camera::stop() idempotent as well from [1]. - Made test registration dynamic in patch 3 - Some other minor fixes [1] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html v1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html Nícolas F. R. A. Prado (5): libcamera: camera: Make stop() idempotent lc-compliance: Make SimpleCapture::stop() idempotent lc-compliance: Add Environment singleton lc-compliance: Refactor using Googletest lc-compliance: Add list and filter parameters src/lc-compliance/environment.cpp | 20 ++++ src/lc-compliance/environment.h | 31 +++++ src/lc-compliance/main.cpp | 133 +++++++++++++++------ src/lc-compliance/meson.build | 6 +- src/lc-compliance/results.cpp | 75 ------------ src/lc-compliance/results.h | 47 -------- src/lc-compliance/simple_capture.cpp | 92 +++++--------- src/lc-compliance/simple_capture.h | 9 +- src/lc-compliance/single_stream.cpp | 172 ++++++++++++++++----------- src/lc-compliance/tests.h | 16 --- src/libcamera/camera.cpp | 20 +++- 11 files changed, 307 insertions(+), 314 deletions(-) create mode 100644 src/lc-compliance/environment.cpp create mode 100644 src/lc-compliance/environment.h delete mode 100644 src/lc-compliance/results.cpp delete mode 100644 src/lc-compliance/results.h delete mode 100644 src/lc-compliance/tests.h