{"id":12554,"url":"https://patchwork.libcamera.org/api/covers/12554/?format=json","web_url":"https://patchwork.libcamera.org/cover/12554/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210610183732.174697-1-nfraprado@collabora.com>","date":"2021-06-10T18:37:27","name":"[libcamera-devel,v7,0/5] lc-compliance: Refactor using Googletest","submitter":{"id":84,"url":"https://patchwork.libcamera.org/api/people/84/?format=json","name":"Nícolas F. R. A. Prado","email":"nfraprado@collabora.com"},"mbox":"https://patchwork.libcamera.org/cover/12554/mbox/","series":[{"id":2119,"url":"https://patchwork.libcamera.org/api/series/2119/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2119","date":"2021-06-10T18:37:27","name":"lc-compliance: Refactor using Googletest","version":7,"mbox":"https://patchwork.libcamera.org/series/2119/mbox/"}],"comments":"https://patchwork.libcamera.org/api/covers/12554/comments/","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A9ECBC320B\n\tfor ;\n\tThu, 10 Jun 2021 18:38:30 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 373D168930;\n\tThu, 10 Jun 2021 20:38:30 +0200 (CEST)","from bhuna.collabora.co.uk (bhuna.collabora.co.uk\n\t[IPv6:2a00:1098:0:82:1000:25:2eeb:e3e3])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A5A426029B\n\tfor ;\n\tThu, 10 Jun 2021 20:38:29 +0200 (CEST)","from localhost.localdomain (unknown\n\t[IPv6:2804:14c:1a9:2978:fdf4:5e57:8aee:5d72])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id AF0241F441F0;\n\tThu, 10 Jun 2021 19:38:27 +0100 (BST)"],"From":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= ","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 10 Jun 2021 15:37:27 -0300","Message-Id":"<20210610183732.174697-1-nfraprado@collabora.com>","X-Mailer":"git-send-email 2.32.0","MIME-Version":"1.0","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v7 0/5] lc-compliance: Refactor using\n\tGoogletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"kernel@collabora.com, =?utf-8?q?Andr=C3=A9_Almeida?=\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"This is a refactoring of lc-compliance using Googletest as the framework.\n\nPatches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3\nadds an Environment singleton to provide access to the camera from all tests.\nPatch 4 does the actual refactor. Patch 5 adds two new parameters, --list and\n--filter.\n\nlc-compliance is built around testing a single camera (and possibly optional\nsecondary cameras in the future), which is passed through the -c flag. The list\nof tests can be queried with:\n\n\tlc-compliance --list\n\nBy default all tests are run, but it's possible to select which tests to run by\nusing:\n\n\tlc-compliance -c '' --filter ''\n\nwhere the test_filter is in the same format as Googletest's --gtest_filter.\nQuoting its help:\n\n --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]\n Run only the tests whose name matches one of the positive patterns but\n none of the negative patterns. '?' matches any single character; '*'\n matches any substring; ':' separates two patterns.\n\nThis is a snippet of a successful run:\n\n\t[==========] Running 120 tests from 1 test suite.\n\t[----------] Global test environment set-up.\n\t[----------] 120 tests from RolesAndRequests/FixedRequestsTest\n\t[ RUN ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1\n\t[31:53:24.860985336] [271719] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG\n\tCamera needs 4 requests, can't test only 1\n\t[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1 (64 ms)\n\n\t[...]\n\n\t[ RUN ] RolesAndRequests/FixedRequestsTest.Unbalanced/Viewfinder_89\n\t[31:56:18.607757616] [271719] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG\n\t[ OK ] RolesAndRequests/FixedRequestsTest.Unbalanced/Viewfinder_89 (3148 ms)\n\t[----------] 120 tests from RolesAndRequests/FixedRequestsTest (176896 ms total)\n\t\n\t[----------] Global test environment tear-down\n\t[==========] 120 tests from 1 test suite ran. (176896 ms total)\n\t[ PASSED ] 96 tests.\n\t[ SKIPPED ] 24 tests, listed below:\n\t[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1\n\t[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_2\n\t[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_3\n\t[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/StillCapture_1\n\n\t[...]\n\nAnd how a failure looks like:\n\n\t[ RUN ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_5\n\t[31:59:59.528082413] [272962] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG\n\t../src/lc-compliance/simple_capture.cpp:117: Failure\n\tExpected equality of these values:\n\t captureCount_\n\t Which is: 5\n\t captureLimit_\n\t Which is: 6\n\t[ FAILED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_5, where GetParam() = (0, 5) (317 ms)\n\nThe current complete list of tests is shown below:\n\n\tRolesAndRequests/FixedRequestsTest.\n\t BalancedSingleCycle/Raw_1 # GetParam() = (0, 1)\n\t BalancedSingleCycle/Raw_2 # GetParam() = (0, 2)\n\t BalancedSingleCycle/Raw_3 # GetParam() = (0, 3)\n\t BalancedSingleCycle/Raw_5 # GetParam() = (0, 5)\n\t BalancedSingleCycle/Raw_8 # GetParam() = (0, 8)\n\t BalancedSingleCycle/Raw_13 # GetParam() = (0, 13)\n\t BalancedSingleCycle/Raw_21 # GetParam() = (0, 21)\n\t BalancedSingleCycle/Raw_34 # GetParam() = (0, 34)\n\t BalancedSingleCycle/Raw_55 # GetParam() = (0, 55)\n\t BalancedSingleCycle/Raw_89 # GetParam() = (0, 89)\n\t BalancedSingleCycle/StillCapture_1 # GetParam() = (1, 1)\n\t BalancedSingleCycle/StillCapture_2 # GetParam() = (1, 2)\n\t BalancedSingleCycle/StillCapture_3 # GetParam() = (1, 3)\n\t BalancedSingleCycle/StillCapture_5 # GetParam() = (1, 5)\n\t BalancedSingleCycle/StillCapture_8 # GetParam() = (1, 8)\n\t BalancedSingleCycle/StillCapture_13 # GetParam() = (1, 13)\n\t BalancedSingleCycle/StillCapture_21 # GetParam() = (1, 21)\n\t BalancedSingleCycle/StillCapture_34 # GetParam() = (1, 34)\n\t BalancedSingleCycle/StillCapture_55 # GetParam() = (1, 55)\n\t BalancedSingleCycle/StillCapture_89 # GetParam() = (1, 89)\n\t BalancedSingleCycle/VideoRecording_1 # GetParam() = (2, 1)\n\t BalancedSingleCycle/VideoRecording_2 # GetParam() = (2, 2)\n\t BalancedSingleCycle/VideoRecording_3 # GetParam() = (2, 3)\n\t BalancedSingleCycle/VideoRecording_5 # GetParam() = (2, 5)\n\t BalancedSingleCycle/VideoRecording_8 # GetParam() = (2, 8)\n\t BalancedSingleCycle/VideoRecording_13 # GetParam() = (2, 13)\n\t BalancedSingleCycle/VideoRecording_21 # GetParam() = (2, 21)\n\t BalancedSingleCycle/VideoRecording_34 # GetParam() = (2, 34)\n\t BalancedSingleCycle/VideoRecording_55 # GetParam() = (2, 55)\n\t BalancedSingleCycle/VideoRecording_89 # GetParam() = (2, 89)\n\t BalancedSingleCycle/Viewfinder_1 # GetParam() = (3, 1)\n\t BalancedSingleCycle/Viewfinder_2 # GetParam() = (3, 2)\n\t BalancedSingleCycle/Viewfinder_3 # GetParam() = (3, 3)\n\t BalancedSingleCycle/Viewfinder_5 # GetParam() = (3, 5)\n\t BalancedSingleCycle/Viewfinder_8 # GetParam() = (3, 8)\n\t BalancedSingleCycle/Viewfinder_13 # GetParam() = (3, 13)\n\t BalancedSingleCycle/Viewfinder_21 # GetParam() = (3, 21)\n\t BalancedSingleCycle/Viewfinder_34 # GetParam() = (3, 34)\n\t BalancedSingleCycle/Viewfinder_55 # GetParam() = (3, 55)\n\t BalancedSingleCycle/Viewfinder_89 # GetParam() = (3, 89)\n\t BalancedMultiCycle/Raw_1 # GetParam() = (0, 1)\n\t BalancedMultiCycle/Raw_2 # GetParam() = (0, 2)\n\t BalancedMultiCycle/Raw_3 # GetParam() = (0, 3)\n\t BalancedMultiCycle/Raw_5 # GetParam() = (0, 5)\n\t BalancedMultiCycle/Raw_8 # GetParam() = (0, 8)\n\t BalancedMultiCycle/Raw_13 # GetParam() = (0, 13)\n\t BalancedMultiCycle/Raw_21 # GetParam() = (0, 21)\n\t BalancedMultiCycle/Raw_34 # GetParam() = (0, 34)\n\t BalancedMultiCycle/Raw_55 # GetParam() = (0, 55)\n\t BalancedMultiCycle/Raw_89 # GetParam() = (0, 89)\n\t BalancedMultiCycle/StillCapture_1 # GetParam() = (1, 1)\n\t BalancedMultiCycle/StillCapture_2 # GetParam() = (1, 2)\n\t BalancedMultiCycle/StillCapture_3 # GetParam() = (1, 3)\n\t BalancedMultiCycle/StillCapture_5 # GetParam() = (1, 5)\n\t BalancedMultiCycle/StillCapture_8 # GetParam() = (1, 8)\n\t BalancedMultiCycle/StillCapture_13 # GetParam() = (1, 13)\n\t BalancedMultiCycle/StillCapture_21 # GetParam() = (1, 21)\n\t BalancedMultiCycle/StillCapture_34 # GetParam() = (1, 34)\n\t BalancedMultiCycle/StillCapture_55 # GetParam() = (1, 55)\n\t BalancedMultiCycle/StillCapture_89 # GetParam() = (1, 89)\n\t BalancedMultiCycle/VideoRecording_1 # GetParam() = (2, 1)\n\t BalancedMultiCycle/VideoRecording_2 # GetParam() = (2, 2)\n\t BalancedMultiCycle/VideoRecording_3 # GetParam() = (2, 3)\n\t BalancedMultiCycle/VideoRecording_5 # GetParam() = (2, 5)\n\t BalancedMultiCycle/VideoRecording_8 # GetParam() = (2, 8)\n\t BalancedMultiCycle/VideoRecording_13 # GetParam() = (2, 13)\n\t BalancedMultiCycle/VideoRecording_21 # GetParam() = (2, 21)\n\t BalancedMultiCycle/VideoRecording_34 # GetParam() = (2, 34)\n\t BalancedMultiCycle/VideoRecording_55 # GetParam() = (2, 55)\n\t BalancedMultiCycle/VideoRecording_89 # GetParam() = (2, 89)\n\t BalancedMultiCycle/Viewfinder_1 # GetParam() = (3, 1)\n\t BalancedMultiCycle/Viewfinder_2 # GetParam() = (3, 2)\n\t BalancedMultiCycle/Viewfinder_3 # GetParam() = (3, 3)\n\t BalancedMultiCycle/Viewfinder_5 # GetParam() = (3, 5)\n\t BalancedMultiCycle/Viewfinder_8 # GetParam() = (3, 8)\n\t BalancedMultiCycle/Viewfinder_13 # GetParam() = (3, 13)\n\t BalancedMultiCycle/Viewfinder_21 # GetParam() = (3, 21)\n\t BalancedMultiCycle/Viewfinder_34 # GetParam() = (3, 34)\n\t BalancedMultiCycle/Viewfinder_55 # GetParam() = (3, 55)\n\t BalancedMultiCycle/Viewfinder_89 # GetParam() = (3, 89)\n\t Unbalanced/Raw_1 # GetParam() = (0, 1)\n\t Unbalanced/Raw_2 # GetParam() = (0, 2)\n\t Unbalanced/Raw_3 # GetParam() = (0, 3)\n\t Unbalanced/Raw_5 # GetParam() = (0, 5)\n\t Unbalanced/Raw_8 # GetParam() = (0, 8)\n\t Unbalanced/Raw_13 # GetParam() = (0, 13)\n\t Unbalanced/Raw_21 # GetParam() = (0, 21)\n\t Unbalanced/Raw_34 # GetParam() = (0, 34)\n\t Unbalanced/Raw_55 # GetParam() = (0, 55)\n\t Unbalanced/Raw_89 # GetParam() = (0, 89)\n\t Unbalanced/StillCapture_1 # GetParam() = (1, 1)\n\t Unbalanced/StillCapture_2 # GetParam() = (1, 2)\n\t Unbalanced/StillCapture_3 # GetParam() = (1, 3)\n\t Unbalanced/StillCapture_5 # GetParam() = (1, 5)\n\t Unbalanced/StillCapture_8 # GetParam() = (1, 8)\n\t Unbalanced/StillCapture_13 # GetParam() = (1, 13)\n\t Unbalanced/StillCapture_21 # GetParam() = (1, 21)\n\t Unbalanced/StillCapture_34 # GetParam() = (1, 34)\n\t Unbalanced/StillCapture_55 # GetParam() = (1, 55)\n\t Unbalanced/StillCapture_89 # GetParam() = (1, 89)\n\t Unbalanced/VideoRecording_1 # GetParam() = (2, 1)\n\t Unbalanced/VideoRecording_2 # GetParam() = (2, 2)\n\t Unbalanced/VideoRecording_3 # GetParam() = (2, 3)\n\t Unbalanced/VideoRecording_5 # GetParam() = (2, 5)\n\t Unbalanced/VideoRecording_8 # GetParam() = (2, 8)\n\t Unbalanced/VideoRecording_13 # GetParam() = (2, 13)\n\t Unbalanced/VideoRecording_21 # GetParam() = (2, 21)\n\t Unbalanced/VideoRecording_34 # GetParam() = (2, 34)\n\t Unbalanced/VideoRecording_55 # GetParam() = (2, 55)\n\t Unbalanced/VideoRecording_89 # GetParam() = (2, 89)\n\t Unbalanced/Viewfinder_1 # GetParam() = (3, 1)\n\t Unbalanced/Viewfinder_2 # GetParam() = (3, 2)\n\t Unbalanced/Viewfinder_3 # GetParam() = (3, 3)\n\t Unbalanced/Viewfinder_5 # GetParam() = (3, 5)\n\t Unbalanced/Viewfinder_8 # GetParam() = (3, 8)\n\t Unbalanced/Viewfinder_13 # GetParam() = (3, 13)\n\t Unbalanced/Viewfinder_21 # GetParam() = (3, 21)\n\t Unbalanced/Viewfinder_34 # GetParam() = (3, 34)\n\t Unbalanced/Viewfinder_55 # GetParam() = (3, 55)\n\t Unbalanced/Viewfinder_89 # GetParam() = (3, 89)\n\nChanges in v7:\n- Thanks to Jacopo:\n - Fixed style issue\n - Made CameraManager a unique_ptr and passed to Environment as raw pointer\n - Moved the check for buffers allocated to the beginning of\n SimpleCapture::stop()\n- Thanks to Niklas:\n - Removed intermediary filter string variable in Harness::initGtestParameters()\n\nv6: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021137.html\n\nChanges in v6:\n- Thanks to Niklas:\n - Made return of Environment::cameraId() be a const reference\n - Changed buildGtestParameters() into initGtestParameters() and removed the\n need for Harness::gtestArgc_ and Harness::gtestArgv_\n\nv5: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020759.html\n\nChanges in v5:\n- Thanks to Laurent:\n - Fixed style issues\n - Stored CameraManager and cameraId instead of Camera in Environment, and\n acquire and release the camera for each test\n- Thanks to Niklas:\n - Moved buildGtestParameters() inside run()\n- Thanks to Laurent & Niklas:\n - Improved Environment singleton class instantiation and destruction\n\nv4: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020594.html\n\nChanges in v4:\n- Removed old lc-compliance results classes\n- Thanks to Niklas:\n - Added patch 3 with the environment singleton in its own files\n - Improved naming of tests\n- Thanks to Jacopo:\n - Added \\todo in Camera::stop()\n\nv3: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020382.html\n\nChanges in v3:\n- Thanks to Niklas:\n - Went back to static test registration, and created a Singleton Environment\n class to store the camera global variable\n- Added patch 4 implementing list and filter parameters\n- Added a nameParameters() function to give meaningful names to the test\n parameters, removing the need to register each test suite for every role\n\nv2: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020056.html\n\nChanges in v2:\n- Added patch 1 to make Camera::stop() idempotent as well from [1].\n- Made test registration dynamic in patch 3\n- Some other minor fixes\n\n[1] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html\n\nv1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html\n\nNícolas F. R. A. Prado (5):\n libcamera: camera: Make stop() idempotent\n lc-compliance: Make SimpleCapture::stop() idempotent\n lc-compliance: Add Environment singleton\n lc-compliance: Refactor using Googletest\n lc-compliance: Add list and filter parameters\n\n src/lc-compliance/environment.cpp | 20 +++\n src/lc-compliance/environment.h | 31 +++++\n src/lc-compliance/main.cpp | 134 +++++++++++++++------\n src/lc-compliance/meson.build | 5 +-\n src/lc-compliance/results.cpp | 75 ------------\n src/lc-compliance/results.h | 47 --------\n src/lc-compliance/simple_capture.cpp | 92 +++++---------\n src/lc-compliance/simple_capture.h | 9 +-\n src/lc-compliance/single_stream.cpp | 174 ++++++++++++++++-----------\n src/lc-compliance/tests.h | 16 ---\n src/libcamera/camera.cpp | 20 ++-\n 11 files changed, 309 insertions(+), 314 deletions(-)\n create mode 100644 src/lc-compliance/environment.cpp\n create mode 100644 src/lc-compliance/environment.h\n delete mode 100644 src/lc-compliance/results.cpp\n delete mode 100644 src/lc-compliance/results.h\n delete mode 100644 src/lc-compliance/tests.h"}