From patchwork Mon May 3 19:33:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= X-Patchwork-Id: 12188 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 77695BDE78 for ; Mon, 3 May 2021 19:33:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BAEB168919; Mon, 3 May 2021 21:33:58 +0200 (CEST) Received: from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4F6CB688AE for ; Mon, 3 May 2021 21:33:57 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2804:14c:1a9:2978:995d:672b:100f:2fd9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: nfraprado) by bhuna.collabora.co.uk (Postfix) with ESMTPSA id 474E81F41E95; Mon, 3 May 2021 20:33:55 +0100 (BST) From: =?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= To: libcamera-devel@lists.libcamera.org Date: Mon, 3 May 2021 16:33:04 -0300 Message-Id: <20210503193307.108607-1-nfraprado@collabora.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 0/3] lc-compliance: Refactor using Googletest X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kernel@collabora.com, =?utf-8?q?Andr=C3=A9_Almeida?= Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This is a prototype of the refactoring of lc-compliance using Googletest as the framework. Patches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3 does the actual refactor. In v1 the tests were registered statically which simplified the work needed to add new tests, but it required the camera pointer to be a global variable, which wouldn't scale for usage with multiple cameras. For v2 the tests are registered dynamically so that a camera pointer can be passed to tests being instantiated. This requires manual registration of each parametrization of the tests, which I tried to make less cumbersome by defining the REGISTER_TESTS() macro (although for tests with different parameters, different macros would be needed...). It also requires each new test to declare a new class and be described in the TestBody() function of the class. Perhaps this could also be made easier with a macro? Also, since now a test is created for each of the cameras in the system, the -c flag was removed, as the camera to be tested can be selected by filtering only tests for that camera with --gtest_filter. Although we could re-add this flag later for convenience. The parameters for Gtest have yet to be wired up, but can currently be passed after double dashes, like lc-compliance -c '' -- --gtest_list_tests to list all current tests defined. The parameter --gtest_filter can be used to filter the individual tests or test suites to run. It uses string matching and globing for the test names so it's easy to run a whole test suite MyTestSuite with --gtest_list_tests='*MyTestSuite*'. This is a snippet of a successful run: [==========] Running 240 tests from 1 test suite. [----------] Global test environment set-up. [----------] 240 tests from SingleStream [ RUN ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 [20:42:43.656235633] [105482] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG Camera needs 4 requests, can't test only 1 [ SKIPPED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 (63 ms) [...] [ RUN ] SingleStream.Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.2-0bda:5511/Viewfinder/89 [20:51:53.678488142] [105482] INFO Camera camera.cpp:915 configuring streams: (0) 576x360-R8 [ OK ] SingleStream.Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.2-0bda:5511/Viewfinder/89 (6390 ms) [----------] 240 tests from SingleStream (556412 ms total) [----------] Global test environment tear-down [==========] 240 tests from 1 test suite ran. (556412 ms total) [ PASSED ] 192 tests. [ SKIPPED ] 48 tests, listed below: [ SKIPPED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 [ SKIPPED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 [...] And how a failure looks like: [ RUN ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 [20:55:13.218088360] [106518] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG ../src/lc-compliance/simple_capture.cpp:117: Failure Expected equality of these values: captureCount_ Which is: 5 captureLimit_ Which is: 6 [ FAILED ] SingleStream.BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 (318 ms) The current complete list of tests is shown below for a single camera (each test will show once for each of the cameras, which makes the list very lengthy if multiple cameras are present): SingleStream. BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/89 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/89 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/89 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/1 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/2 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/3 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/5 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/8 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/13 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/21 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/34 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/55 BalancedSingleCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/89 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/1 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/2 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/3 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/5 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/8 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/13 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/21 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/34 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/55 BalancedMultiCycle/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Raw/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Still/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Video/89 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/1 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/2 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/3 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/5 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/8 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/13 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/21 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/34 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/55 Unbalanced/\_SB_.PCI0.XHC_.RHUB.HS09-9:1.0-0bda:5511/Viewfinder/89 Changes in v2: - Added patch 1 to make Camera::stop() idempotent as well from [1]. - Made test registration dynamic in patch 3 - Some other minor fixes [1] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html v1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html Nícolas F. R. A. Prado (3): libcamera: camera: Make stop() idempotent lc-compliance: Make SimpleCapture::stop() idempotent lc-compliance: Refactor using Googletest src/lc-compliance/main.cpp | 99 +++++++++--------- src/lc-compliance/meson.build | 3 + src/lc-compliance/simple_capture.cpp | 95 +++++++---------- src/lc-compliance/simple_capture.h | 8 +- src/lc-compliance/single_stream.cpp | 151 ++++++++++++++------------- src/lc-compliance/tests.h | 15 ++- src/libcamera/camera.cpp | 20 +++- 7 files changed, 199 insertions(+), 192 deletions(-)