Cover Letter Detail
Show a cover letter.
GET /api/covers/12785/?format=api
{ "id": 12785, "url": "https://patchwork.libcamera.org/api/covers/12785/?format=api", "web_url": "https://patchwork.libcamera.org/cover/12785/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20210702122114.724932-1-nfraprado@collabora.com>", "date": "2021-07-02T12:21:09", "name": "[libcamera-devel,v11,0/5] lc-compliance: Refactor using Googletest", "submitter": { "id": 84, "url": "https://patchwork.libcamera.org/api/people/84/?format=api", "name": "Nícolas F. R. A. Prado", "email": "nfraprado@collabora.com" }, "mbox": "https://patchwork.libcamera.org/cover/12785/mbox/", "series": [ { "id": 2201, "url": "https://patchwork.libcamera.org/api/series/2201/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=2201", "date": "2021-07-02T12:21:09", "name": "lc-compliance: Refactor using Googletest", "version": 11, "mbox": "https://patchwork.libcamera.org/series/2201/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/covers/12785/comments/", "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id AA45CC3222\n\tfor <parsemail@patchwork.libcamera.org>;\n\tFri, 2 Jul 2021 12:21:31 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3B2BD684F2;\n\tFri, 2 Jul 2021 14:21:31 +0200 (CEST)", "from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5B9DD684E8\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 2 Jul 2021 14:21:29 +0200 (CEST)", "from localhost.localdomain (unknown\n\t[IPv6:2804:14c:1a9:2434:bd70:78a1:919c:f01a])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id DCF221F449AB;\n\tFri, 2 Jul 2021 13:21:27 +0100 (BST)" ], "From": "=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= <nfraprado@collabora.com>", "To": "libcamera-devel@lists.libcamera.org", "Date": "Fri, 2 Jul 2021 09:21:09 -0300", "Message-Id": "<20210702122114.724932-1-nfraprado@collabora.com>", "X-Mailer": "git-send-email 2.32.0", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=UTF-8", "Content-Transfer-Encoding": "8bit", "Subject": "[libcamera-devel] [PATCH v11 0/5] lc-compliance: Refactor using\n\tGoogletest", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "Cc": "kernel@collabora.com, =?utf-8?q?Andr=C3=A9_Almeida?=\n\t<andrealmeid@collabora.com>", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "(We're now running lc-compliance on KernelCI, more details below)\n\nThis is a refactoring of lc-compliance using Googletest as the framework.\n\nPatches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3\nadds an Environment singleton to provide access to the camera from all tests.\nPatch 4 does the actual refactor. Patch 5 adds two new parameters, --list and\n--filter.\n\nlc-compliance on KernelCI\n-------------------------\n\nlc-compliance is already being run on KernelCI [1] for mt8173-elm-hana\n(Chromebook), which has an USB camera. All test runs of lc-compliance so far can\nbe seen on the Tests tab [2] by searching for \"lc-compliance\". I'm currently\nlooking into adding Rock Pi 4 boards to the lab so that the tests can run on\nthose too.\n\n[1] https://linux.kernelci.org/test/job/tip/branch/master/kernel/v5.13-rc6-239-g0b42677e2e5d/plan/lc-compliance/\n\n[2] https://linux.kernelci.org/test/\n\nI wrote a blog post on Collabora's blog [3] going over the process of adding\nlc-compliance to KernelCI, so check it out for more details on that. At the\nmoment lc-compliance is being compiled from my tree [4], but as soon as this\nseries get merged we can easily change KernelCI to compile from upstream.\n\n[3] https://www.collabora.com/news-and-blog/blog/2021/06/15/testing-cameras-with-lc-compliance-on-kernelci/\n\n[4] https://gitlab.collabora.com/nfraprado/libcamera/-/tree/lcc-gtest-v5\n\nlc-compliance Usage\n-------------------\n\nlc-compliance is built around testing a single camera (and possibly optional\nsecondary cameras in the future), which is passed through the -c flag. The list\nof tests can be queried with:\n\n\tlc-compliance --list\n\nBy default all tests are run, but it's possible to select which tests to run by\nusing:\n\n\tlc-compliance -c '<camera_name>' --filter '<test_filter>'\n\nwhere the test_filter is in the same format as Googletest's --gtest_filter.\nQuoting its help:\n\n --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]\n Run only the tests whose name matches one of the positive patterns but\n none of the negative patterns. '?' matches any single character; '*'\n matches any substring; ':' separates two patterns.\n\nThis is a snippet of a successful run:\n\n\t[==========] Running 120 tests from 1 test suite.\n\t[----------] Global test environment set-up.\n\t[----------] 120 tests from CaptureTests/SingleStream\n\t[ RUN ] CaptureTests/SingleStream.Capture/Raw_1\n\t[31:47:27.239018193] [240317] INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n\tCamera needs 4 requests, can't test only 1\n\t../src/lc-compliance/simple_capture.cpp:90: Skipped\n\t\n\t[ SKIPPED ] CaptureTests/SingleStream.Capture/Raw_1 (62 ms)\n\n\t[...]\n\n\t[ RUN ] CaptureTests/SingleStream.UnbalancedStop/Viewfinder_89\n\t[31:50:19.266015455] [240317] INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n\t[ OK ] CaptureTests/SingleStream.UnbalancedStop/Viewfinder_89 (3142 ms)\n\t[----------] 120 tests from CaptureTests/SingleStream (175170 ms total)\n\t\n\t[----------] Global test environment tear-down\n\t[==========] 120 tests from 1 test suite ran. (175170 ms total)\n\t[ PASSED ] 96 tests.\n\t[ SKIPPED ] 24 tests, listed below:\n\t[ SKIPPED ] CaptureTests/SingleStream.Capture/Raw_1\n\t[ SKIPPED ] CaptureTests/SingleStream.Capture/Raw_2\n\t[ SKIPPED ] CaptureTests/SingleStream.Capture/Raw_3\n\t[ SKIPPED ] CaptureTests/SingleStream.Capture/StillCapture_1\n\n\t[...]\n\nAnd how a failure looks like:\n\n\t[ RUN ] CaptureTests/SingleStream.Capture/Raw_5\n\t[31:55:30.651452784] [241401] INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n\t../src/lc-compliance/simple_capture.cpp:117: Failure\n\tExpected equality of these values:\n\t captureCount_\n\t Which is: 5\n\t captureLimit_\n\t Which is: 6\n\t[ FAILED ] CaptureTests/SingleStream.Capture/Raw_5, where GetParam() = (0, 5) (317 ms)\n\nThe current complete list of tests is shown below:\n\n\tCaptureTests/SingleStream.\n\t Capture/Raw_1 # GetParam() = (0, 1)\n\t Capture/Raw_2 # GetParam() = (0, 2)\n\t Capture/Raw_3 # GetParam() = (0, 3)\n\t Capture/Raw_5 # GetParam() = (0, 5)\n\t Capture/Raw_8 # GetParam() = (0, 8)\n\t Capture/Raw_13 # GetParam() = (0, 13)\n\t Capture/Raw_21 # GetParam() = (0, 21)\n\t Capture/Raw_34 # GetParam() = (0, 34)\n\t Capture/Raw_55 # GetParam() = (0, 55)\n\t Capture/Raw_89 # GetParam() = (0, 89)\n\t Capture/StillCapture_1 # GetParam() = (1, 1)\n\t Capture/StillCapture_2 # GetParam() = (1, 2)\n\t Capture/StillCapture_3 # GetParam() = (1, 3)\n\t Capture/StillCapture_5 # GetParam() = (1, 5)\n\t Capture/StillCapture_8 # GetParam() = (1, 8)\n\t Capture/StillCapture_13 # GetParam() = (1, 13)\n\t Capture/StillCapture_21 # GetParam() = (1, 21)\n\t Capture/StillCapture_34 # GetParam() = (1, 34)\n\t Capture/StillCapture_55 # GetParam() = (1, 55)\n\t Capture/StillCapture_89 # GetParam() = (1, 89)\n\t Capture/VideoRecording_1 # GetParam() = (2, 1)\n\t Capture/VideoRecording_2 # GetParam() = (2, 2)\n\t Capture/VideoRecording_3 # GetParam() = (2, 3)\n\t Capture/VideoRecording_5 # GetParam() = (2, 5)\n\t Capture/VideoRecording_8 # GetParam() = (2, 8)\n\t Capture/VideoRecording_13 # GetParam() = (2, 13)\n\t Capture/VideoRecording_21 # GetParam() = (2, 21)\n\t Capture/VideoRecording_34 # GetParam() = (2, 34)\n\t Capture/VideoRecording_55 # GetParam() = (2, 55)\n\t Capture/VideoRecording_89 # GetParam() = (2, 89)\n\t Capture/Viewfinder_1 # GetParam() = (3, 1)\n\t Capture/Viewfinder_2 # GetParam() = (3, 2)\n\t Capture/Viewfinder_3 # GetParam() = (3, 3)\n\t Capture/Viewfinder_5 # GetParam() = (3, 5)\n\t Capture/Viewfinder_8 # GetParam() = (3, 8)\n\t Capture/Viewfinder_13 # GetParam() = (3, 13)\n\t Capture/Viewfinder_21 # GetParam() = (3, 21)\n\t Capture/Viewfinder_34 # GetParam() = (3, 34)\n\t Capture/Viewfinder_55 # GetParam() = (3, 55)\n\t Capture/Viewfinder_89 # GetParam() = (3, 89)\n\t CaptureStartStop/Raw_1 # GetParam() = (0, 1)\n\t CaptureStartStop/Raw_2 # GetParam() = (0, 2)\n\t CaptureStartStop/Raw_3 # GetParam() = (0, 3)\n\t CaptureStartStop/Raw_5 # GetParam() = (0, 5)\n\t CaptureStartStop/Raw_8 # GetParam() = (0, 8)\n\t CaptureStartStop/Raw_13 # GetParam() = (0, 13)\n\t CaptureStartStop/Raw_21 # GetParam() = (0, 21)\n\t CaptureStartStop/Raw_34 # GetParam() = (0, 34)\n\t CaptureStartStop/Raw_55 # GetParam() = (0, 55)\n\t CaptureStartStop/Raw_89 # GetParam() = (0, 89)\n\t CaptureStartStop/StillCapture_1 # GetParam() = (1, 1)\n\t CaptureStartStop/StillCapture_2 # GetParam() = (1, 2)\n\t CaptureStartStop/StillCapture_3 # GetParam() = (1, 3)\n\t CaptureStartStop/StillCapture_5 # GetParam() = (1, 5)\n\t CaptureStartStop/StillCapture_8 # GetParam() = (1, 8)\n\t CaptureStartStop/StillCapture_13 # GetParam() = (1, 13)\n\t CaptureStartStop/StillCapture_21 # GetParam() = (1, 21)\n\t CaptureStartStop/StillCapture_34 # GetParam() = (1, 34)\n\t CaptureStartStop/StillCapture_55 # GetParam() = (1, 55)\n\t CaptureStartStop/StillCapture_89 # GetParam() = (1, 89)\n\t CaptureStartStop/VideoRecording_1 # GetParam() = (2, 1)\n\t CaptureStartStop/VideoRecording_2 # GetParam() = (2, 2)\n\t CaptureStartStop/VideoRecording_3 # GetParam() = (2, 3)\n\t CaptureStartStop/VideoRecording_5 # GetParam() = (2, 5)\n\t CaptureStartStop/VideoRecording_8 # GetParam() = (2, 8)\n\t CaptureStartStop/VideoRecording_13 # GetParam() = (2, 13)\n\t CaptureStartStop/VideoRecording_21 # GetParam() = (2, 21)\n\t CaptureStartStop/VideoRecording_34 # GetParam() = (2, 34)\n\t CaptureStartStop/VideoRecording_55 # GetParam() = (2, 55)\n\t CaptureStartStop/VideoRecording_89 # GetParam() = (2, 89)\n\t CaptureStartStop/Viewfinder_1 # GetParam() = (3, 1)\n\t CaptureStartStop/Viewfinder_2 # GetParam() = (3, 2)\n\t CaptureStartStop/Viewfinder_3 # GetParam() = (3, 3)\n\t CaptureStartStop/Viewfinder_5 # GetParam() = (3, 5)\n\t CaptureStartStop/Viewfinder_8 # GetParam() = (3, 8)\n\t CaptureStartStop/Viewfinder_13 # GetParam() = (3, 13)\n\t CaptureStartStop/Viewfinder_21 # GetParam() = (3, 21)\n\t CaptureStartStop/Viewfinder_34 # GetParam() = (3, 34)\n\t CaptureStartStop/Viewfinder_55 # GetParam() = (3, 55)\n\t CaptureStartStop/Viewfinder_89 # GetParam() = (3, 89)\n\t UnbalancedStop/Raw_1 # GetParam() = (0, 1)\n\t UnbalancedStop/Raw_2 # GetParam() = (0, 2)\n\t UnbalancedStop/Raw_3 # GetParam() = (0, 3)\n\t UnbalancedStop/Raw_5 # GetParam() = (0, 5)\n\t UnbalancedStop/Raw_8 # GetParam() = (0, 8)\n\t UnbalancedStop/Raw_13 # GetParam() = (0, 13)\n\t UnbalancedStop/Raw_21 # GetParam() = (0, 21)\n\t UnbalancedStop/Raw_34 # GetParam() = (0, 34)\n\t UnbalancedStop/Raw_55 # GetParam() = (0, 55)\n\t UnbalancedStop/Raw_89 # GetParam() = (0, 89)\n\t UnbalancedStop/StillCapture_1 # GetParam() = (1, 1)\n\t UnbalancedStop/StillCapture_2 # GetParam() = (1, 2)\n\t UnbalancedStop/StillCapture_3 # GetParam() = (1, 3)\n\t UnbalancedStop/StillCapture_5 # GetParam() = (1, 5)\n\t UnbalancedStop/StillCapture_8 # GetParam() = (1, 8)\n\t UnbalancedStop/StillCapture_13 # GetParam() = (1, 13)\n\t UnbalancedStop/StillCapture_21 # GetParam() = (1, 21)\n\t UnbalancedStop/StillCapture_34 # GetParam() = (1, 34)\n\t UnbalancedStop/StillCapture_55 # GetParam() = (1, 55)\n\t UnbalancedStop/StillCapture_89 # GetParam() = (1, 89)\n\t UnbalancedStop/VideoRecording_1 # GetParam() = (2, 1)\n\t UnbalancedStop/VideoRecording_2 # GetParam() = (2, 2)\n\t UnbalancedStop/VideoRecording_3 # GetParam() = (2, 3)\n\t UnbalancedStop/VideoRecording_5 # GetParam() = (2, 5)\n\t UnbalancedStop/VideoRecording_8 # GetParam() = (2, 8)\n\t UnbalancedStop/VideoRecording_13 # GetParam() = (2, 13)\n\t UnbalancedStop/VideoRecording_21 # GetParam() = (2, 21)\n\t UnbalancedStop/VideoRecording_34 # GetParam() = (2, 34)\n\t UnbalancedStop/VideoRecording_55 # GetParam() = (2, 55)\n\t UnbalancedStop/VideoRecording_89 # GetParam() = (2, 89)\n\t UnbalancedStop/Viewfinder_1 # GetParam() = (3, 1)\n\t UnbalancedStop/Viewfinder_2 # GetParam() = (3, 2)\n\t UnbalancedStop/Viewfinder_3 # GetParam() = (3, 3)\n\t UnbalancedStop/Viewfinder_5 # GetParam() = (3, 5)\n\t UnbalancedStop/Viewfinder_8 # GetParam() = (3, 8)\n\t UnbalancedStop/Viewfinder_13 # GetParam() = (3, 13)\n\t UnbalancedStop/Viewfinder_21 # GetParam() = (3, 21)\n\t UnbalancedStop/Viewfinder_34 # GetParam() = (3, 34)\n\t UnbalancedStop/Viewfinder_55 # GetParam() = (3, 55)\n\t UnbalancedStop/Viewfinder_89 # GetParam() = (3, 89)\n\nChanges in v11:\n- Thanks to Paul:\n - Updated statemachine test so that it always expects Camera::stop() to pass\n\nv10: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021891.html\n\nChanges in v10:\n- Thanks to Jacopo:\n - Fixed some code style issues\n - Stopped calling CameraManager::stop() in the path it didn't start()\n - Updated comment header\n - Changed ASSERT_FALSE() uses to ASSERT_EQ(...,0) where it made sense\n - Updated copyright headers\n - Improved \"Test single capture cycles\" test description\n- Thanks to Paul:\n - Added flag in meson.build to enable exceptions when compiling lc-compliance\n\nv9: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021753.html\n\nChanges in v9:\n- Thanks to Jacopo:\n - Removed the Harness class, substituting methods with static helper functions\n - Added EXPECT() for the number of buffers allocated in SimpleCapture::start()\n - Changed ASSERT_FALSE() to ASSERT_EQ(..., 0)\n - Changed tests naming\n - Renamed single_stream.cpp to capture_test.cpp\n - Fixed some style issues\n - Added to --help that further Googletest options can be passed as environment\n variables\n- Thanks to Paul:\n - Fixed compiling error on clang\n\nv8: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021398.html\n\nChanges in v8:\n- Thanks to Laurent:\n - Fixed lc-compliance's meson.build to disable when gtest is not installed\n - Fixed compiling errors in Clang\n- Thanks to Kieran:\n - Fixed coding style issues\n - Switched from malloc/free to new/delete in gtest parameter allocation\n\nv7: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021253.html\n\nChanges in v7:\n- Thanks to Jacopo:\n - Fixed style issue\n - Made CameraManager a unique_ptr and passed to Environment as raw pointer\n - Moved the check for buffers allocated to the beginning of\n SimpleCapture::stop()\n- Thanks to Niklas:\n - Removed intermediary filter string variable in Harness::initGtestParameters()\n\nv6: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021137.html\n\nChanges in v6:\n- Thanks to Niklas:\n - Made return of Environment::cameraId() be a const reference\n - Changed buildGtestParameters() into initGtestParameters() and removed the\n need for Harness::gtestArgc_ and Harness::gtestArgv_\n\nv5: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020759.html\n\nChanges in v5:\n- Thanks to Laurent:\n - Fixed style issues\n - Stored CameraManager and cameraId instead of Camera in Environment, and\n acquire and release the camera for each test\n- Thanks to Niklas:\n - Moved buildGtestParameters() inside run()\n- Thanks to Laurent & Niklas:\n - Improved Environment singleton class instantiation and destruction\n\nv4: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020594.html\n\nChanges in v4:\n- Removed old lc-compliance results classes\n- Thanks to Niklas:\n - Added patch 3 with the environment singleton in its own files\n - Improved naming of tests\n- Thanks to Jacopo:\n - Added \\todo in Camera::stop()\n\nv3: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020382.html\n\nChanges in v3:\n- Thanks to Niklas:\n - Went back to static test registration, and created a Singleton Environment\n class to store the camera global variable\n- Added patch 4 implementing list and filter parameters\n- Added a nameParameters() function to give meaningful names to the test\n parameters, removing the need to register each test suite for every role\n\nv2: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020056.html\n\nChanges in v2:\n- Added patch 1 to make Camera::stop() idempotent as well from [5].\n- Made test registration dynamic in patch 3\n- Some other minor fixes\n\n[5] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html\n\nv1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html\n\nNícolas F. R. A. Prado (5):\n libcamera: camera: Make stop() idempotent\n lc-compliance: Make SimpleCapture::stop() idempotent\n lc-compliance: Add Environment singleton\n lc-compliance: Refactor using Googletest\n lc-compliance: Add list and filter parameters\n\n src/lc-compliance/capture_test.cpp | 128 ++++++++++++++++++++\n src/lc-compliance/environment.cpp | 20 ++++\n src/lc-compliance/environment.h | 31 +++++\n src/lc-compliance/main.cpp | 171 +++++++++++++++++----------\n src/lc-compliance/meson.build | 9 +-\n src/lc-compliance/results.cpp | 75 ------------\n src/lc-compliance/results.h | 47 --------\n src/lc-compliance/simple_capture.cpp | 95 ++++++---------\n src/lc-compliance/simple_capture.h | 11 +-\n src/lc-compliance/single_stream.cpp | 97 ---------------\n src/lc-compliance/tests.h | 16 ---\n src/libcamera/camera.cpp | 20 +++-\n test/camera/statemachine.cpp | 15 +--\n 13 files changed, 358 insertions(+), 377 deletions(-)\n create mode 100644 src/lc-compliance/capture_test.cpp\n create mode 100644 src/lc-compliance/environment.cpp\n create mode 100644 src/lc-compliance/environment.h\n delete mode 100644 src/lc-compliance/results.cpp\n delete mode 100644 src/lc-compliance/results.h\n delete mode 100644 src/lc-compliance/single_stream.cpp\n delete mode 100644 src/lc-compliance/tests.h" }