[{"id":17999,"web_url":"https://patchwork.libcamera.org/comment/17999/","msgid":"<20210706104253.nhr2bxt4fchithfn@uno.localdomain>","date":"2021-07-06T10:42:53","subject":"Re: [libcamera-devel] [PATCH v11 0/5] lc-compliance: Refactor using\n\tGoogletest","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi Nicolas,\n\nOn Fri, Jul 02, 2021 at 09:21:09AM -0300, Nícolas F. R. A. Prado wrote:\n> (We're now running lc-compliance on KernelCI, more details below)\n>\n> This is a refactoring of lc-compliance using Googletest as the framework.\n>\n> Patches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3\n> adds an Environment singleton to provide access to the camera from all tests.\n> Patch 4 does the actual refactor. Patch 5 adds two new parameters, --list and\n> --filter.\n\nNow pushed to master, thank you!\n\n>\n> lc-compliance on KernelCI\n> -------------------------\n>\n> lc-compliance is already being run on KernelCI [1] for mt8173-elm-hana\n> (Chromebook), which has an USB camera. All test runs of lc-compliance so far can\n> be seen on the Tests tab [2] by searching for \"lc-compliance\". I'm currently\n> looking into adding Rock Pi 4 boards to the lab so that the tests can run on\n> those too.\n>\n> [1] https://linux.kernelci.org/test/job/tip/branch/master/kernel/v5.13-rc6-239-g0b42677e2e5d/plan/lc-compliance/\n>\n> [2] https://linux.kernelci.org/test/\n>\n> I wrote a blog post on Collabora's blog [3] going over the process of adding\n> lc-compliance to KernelCI, so check it out for more details on that. At the\n> moment lc-compliance is being compiled from my tree [4], but as soon as this\n> series get merged we can easily change KernelCI to compile from upstream.\n>\n> [3] https://www.collabora.com/news-and-blog/blog/2021/06/15/testing-cameras-with-lc-compliance-on-kernelci/\n>\n> [4] https://gitlab.collabora.com/nfraprado/libcamera/-/tree/lcc-gtest-v5\n>\n> lc-compliance Usage\n> -------------------\n>\n> lc-compliance is built around testing a single camera (and possibly optional\n> secondary cameras in the future), which is passed through the -c flag. The list\n> of tests can be queried with:\n>\n> \tlc-compliance --list\n>\n> By default all tests are run, but it's possible to select which tests to run by\n> using:\n>\n> \tlc-compliance -c '<camera_name>' --filter '<test_filter>'\n>\n> where the test_filter is in the same format as Googletest's --gtest_filter.\n> Quoting its help:\n>\n>   --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]\n>       Run only the tests whose name matches one of the positive patterns but\n>       none of the negative patterns. '?' matches any single character; '*'\n>       matches any substring; ':' separates two patterns.\n>\n> This is a snippet of a successful run:\n>\n> \t[==========] Running 120 tests from 1 test suite.\n> \t[----------] Global test environment set-up.\n> \t[----------] 120 tests from CaptureTests/SingleStream\n> \t[ RUN      ] CaptureTests/SingleStream.Capture/Raw_1\n> \t[31:47:27.239018193] [240317]  INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n> \tCamera needs 4 requests, can't test only 1\n> \t../src/lc-compliance/simple_capture.cpp:90: Skipped\n>\n> \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_1 (62 ms)\n>\n> \t[...]\n>\n> \t[ RUN      ] CaptureTests/SingleStream.UnbalancedStop/Viewfinder_89\n> \t[31:50:19.266015455] [240317]  INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n> \t[       OK ] CaptureTests/SingleStream.UnbalancedStop/Viewfinder_89 (3142 ms)\n> \t[----------] 120 tests from CaptureTests/SingleStream (175170 ms total)\n>\n> \t[----------] Global test environment tear-down\n> \t[==========] 120 tests from 1 test suite ran. (175170 ms total)\n> \t[  PASSED  ] 96 tests.\n> \t[  SKIPPED ] 24 tests, listed below:\n> \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_1\n> \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_2\n> \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_3\n> \t[  SKIPPED ] CaptureTests/SingleStream.Capture/StillCapture_1\n>\n> \t[...]\n>\n> And how a failure looks like:\n>\n> \t[ RUN      ] CaptureTests/SingleStream.Capture/Raw_5\n> \t[31:55:30.651452784] [241401]  INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n> \t../src/lc-compliance/simple_capture.cpp:117: Failure\n> \tExpected equality of these values:\n> \t  captureCount_\n> \t    Which is: 5\n> \t  captureLimit_\n> \t    Which is: 6\n> \t[  FAILED  ] CaptureTests/SingleStream.Capture/Raw_5, where GetParam() = (0, 5) (317 ms)\n>\n> The current complete list of tests is shown below:\n>\n> \tCaptureTests/SingleStream.\n> \t  Capture/Raw_1  # GetParam() = (0, 1)\n> \t  Capture/Raw_2  # GetParam() = (0, 2)\n> \t  Capture/Raw_3  # GetParam() = (0, 3)\n> \t  Capture/Raw_5  # GetParam() = (0, 5)\n> \t  Capture/Raw_8  # GetParam() = (0, 8)\n> \t  Capture/Raw_13  # GetParam() = (0, 13)\n> \t  Capture/Raw_21  # GetParam() = (0, 21)\n> \t  Capture/Raw_34  # GetParam() = (0, 34)\n> \t  Capture/Raw_55  # GetParam() = (0, 55)\n> \t  Capture/Raw_89  # GetParam() = (0, 89)\n> \t  Capture/StillCapture_1  # GetParam() = (1, 1)\n> \t  Capture/StillCapture_2  # GetParam() = (1, 2)\n> \t  Capture/StillCapture_3  # GetParam() = (1, 3)\n> \t  Capture/StillCapture_5  # GetParam() = (1, 5)\n> \t  Capture/StillCapture_8  # GetParam() = (1, 8)\n> \t  Capture/StillCapture_13  # GetParam() = (1, 13)\n> \t  Capture/StillCapture_21  # GetParam() = (1, 21)\n> \t  Capture/StillCapture_34  # GetParam() = (1, 34)\n> \t  Capture/StillCapture_55  # GetParam() = (1, 55)\n> \t  Capture/StillCapture_89  # GetParam() = (1, 89)\n> \t  Capture/VideoRecording_1  # GetParam() = (2, 1)\n> \t  Capture/VideoRecording_2  # GetParam() = (2, 2)\n> \t  Capture/VideoRecording_3  # GetParam() = (2, 3)\n> \t  Capture/VideoRecording_5  # GetParam() = (2, 5)\n> \t  Capture/VideoRecording_8  # GetParam() = (2, 8)\n> \t  Capture/VideoRecording_13  # GetParam() = (2, 13)\n> \t  Capture/VideoRecording_21  # GetParam() = (2, 21)\n> \t  Capture/VideoRecording_34  # GetParam() = (2, 34)\n> \t  Capture/VideoRecording_55  # GetParam() = (2, 55)\n> \t  Capture/VideoRecording_89  # GetParam() = (2, 89)\n> \t  Capture/Viewfinder_1  # GetParam() = (3, 1)\n> \t  Capture/Viewfinder_2  # GetParam() = (3, 2)\n> \t  Capture/Viewfinder_3  # GetParam() = (3, 3)\n> \t  Capture/Viewfinder_5  # GetParam() = (3, 5)\n> \t  Capture/Viewfinder_8  # GetParam() = (3, 8)\n> \t  Capture/Viewfinder_13  # GetParam() = (3, 13)\n> \t  Capture/Viewfinder_21  # GetParam() = (3, 21)\n> \t  Capture/Viewfinder_34  # GetParam() = (3, 34)\n> \t  Capture/Viewfinder_55  # GetParam() = (3, 55)\n> \t  Capture/Viewfinder_89  # GetParam() = (3, 89)\n> \t  CaptureStartStop/Raw_1  # GetParam() = (0, 1)\n> \t  CaptureStartStop/Raw_2  # GetParam() = (0, 2)\n> \t  CaptureStartStop/Raw_3  # GetParam() = (0, 3)\n> \t  CaptureStartStop/Raw_5  # GetParam() = (0, 5)\n> \t  CaptureStartStop/Raw_8  # GetParam() = (0, 8)\n> \t  CaptureStartStop/Raw_13  # GetParam() = (0, 13)\n> \t  CaptureStartStop/Raw_21  # GetParam() = (0, 21)\n> \t  CaptureStartStop/Raw_34  # GetParam() = (0, 34)\n> \t  CaptureStartStop/Raw_55  # GetParam() = (0, 55)\n> \t  CaptureStartStop/Raw_89  # GetParam() = (0, 89)\n> \t  CaptureStartStop/StillCapture_1  # GetParam() = (1, 1)\n> \t  CaptureStartStop/StillCapture_2  # GetParam() = (1, 2)\n> \t  CaptureStartStop/StillCapture_3  # GetParam() = (1, 3)\n> \t  CaptureStartStop/StillCapture_5  # GetParam() = (1, 5)\n> \t  CaptureStartStop/StillCapture_8  # GetParam() = (1, 8)\n> \t  CaptureStartStop/StillCapture_13  # GetParam() = (1, 13)\n> \t  CaptureStartStop/StillCapture_21  # GetParam() = (1, 21)\n> \t  CaptureStartStop/StillCapture_34  # GetParam() = (1, 34)\n> \t  CaptureStartStop/StillCapture_55  # GetParam() = (1, 55)\n> \t  CaptureStartStop/StillCapture_89  # GetParam() = (1, 89)\n> \t  CaptureStartStop/VideoRecording_1  # GetParam() = (2, 1)\n> \t  CaptureStartStop/VideoRecording_2  # GetParam() = (2, 2)\n> \t  CaptureStartStop/VideoRecording_3  # GetParam() = (2, 3)\n> \t  CaptureStartStop/VideoRecording_5  # GetParam() = (2, 5)\n> \t  CaptureStartStop/VideoRecording_8  # GetParam() = (2, 8)\n> \t  CaptureStartStop/VideoRecording_13  # GetParam() = (2, 13)\n> \t  CaptureStartStop/VideoRecording_21  # GetParam() = (2, 21)\n> \t  CaptureStartStop/VideoRecording_34  # GetParam() = (2, 34)\n> \t  CaptureStartStop/VideoRecording_55  # GetParam() = (2, 55)\n> \t  CaptureStartStop/VideoRecording_89  # GetParam() = (2, 89)\n> \t  CaptureStartStop/Viewfinder_1  # GetParam() = (3, 1)\n> \t  CaptureStartStop/Viewfinder_2  # GetParam() = (3, 2)\n> \t  CaptureStartStop/Viewfinder_3  # GetParam() = (3, 3)\n> \t  CaptureStartStop/Viewfinder_5  # GetParam() = (3, 5)\n> \t  CaptureStartStop/Viewfinder_8  # GetParam() = (3, 8)\n> \t  CaptureStartStop/Viewfinder_13  # GetParam() = (3, 13)\n> \t  CaptureStartStop/Viewfinder_21  # GetParam() = (3, 21)\n> \t  CaptureStartStop/Viewfinder_34  # GetParam() = (3, 34)\n> \t  CaptureStartStop/Viewfinder_55  # GetParam() = (3, 55)\n> \t  CaptureStartStop/Viewfinder_89  # GetParam() = (3, 89)\n> \t  UnbalancedStop/Raw_1  # GetParam() = (0, 1)\n> \t  UnbalancedStop/Raw_2  # GetParam() = (0, 2)\n> \t  UnbalancedStop/Raw_3  # GetParam() = (0, 3)\n> \t  UnbalancedStop/Raw_5  # GetParam() = (0, 5)\n> \t  UnbalancedStop/Raw_8  # GetParam() = (0, 8)\n> \t  UnbalancedStop/Raw_13  # GetParam() = (0, 13)\n> \t  UnbalancedStop/Raw_21  # GetParam() = (0, 21)\n> \t  UnbalancedStop/Raw_34  # GetParam() = (0, 34)\n> \t  UnbalancedStop/Raw_55  # GetParam() = (0, 55)\n> \t  UnbalancedStop/Raw_89  # GetParam() = (0, 89)\n> \t  UnbalancedStop/StillCapture_1  # GetParam() = (1, 1)\n> \t  UnbalancedStop/StillCapture_2  # GetParam() = (1, 2)\n> \t  UnbalancedStop/StillCapture_3  # GetParam() = (1, 3)\n> \t  UnbalancedStop/StillCapture_5  # GetParam() = (1, 5)\n> \t  UnbalancedStop/StillCapture_8  # GetParam() = (1, 8)\n> \t  UnbalancedStop/StillCapture_13  # GetParam() = (1, 13)\n> \t  UnbalancedStop/StillCapture_21  # GetParam() = (1, 21)\n> \t  UnbalancedStop/StillCapture_34  # GetParam() = (1, 34)\n> \t  UnbalancedStop/StillCapture_55  # GetParam() = (1, 55)\n> \t  UnbalancedStop/StillCapture_89  # GetParam() = (1, 89)\n> \t  UnbalancedStop/VideoRecording_1  # GetParam() = (2, 1)\n> \t  UnbalancedStop/VideoRecording_2  # GetParam() = (2, 2)\n> \t  UnbalancedStop/VideoRecording_3  # GetParam() = (2, 3)\n> \t  UnbalancedStop/VideoRecording_5  # GetParam() = (2, 5)\n> \t  UnbalancedStop/VideoRecording_8  # GetParam() = (2, 8)\n> \t  UnbalancedStop/VideoRecording_13  # GetParam() = (2, 13)\n> \t  UnbalancedStop/VideoRecording_21  # GetParam() = (2, 21)\n> \t  UnbalancedStop/VideoRecording_34  # GetParam() = (2, 34)\n> \t  UnbalancedStop/VideoRecording_55  # GetParam() = (2, 55)\n> \t  UnbalancedStop/VideoRecording_89  # GetParam() = (2, 89)\n> \t  UnbalancedStop/Viewfinder_1  # GetParam() = (3, 1)\n> \t  UnbalancedStop/Viewfinder_2  # GetParam() = (3, 2)\n> \t  UnbalancedStop/Viewfinder_3  # GetParam() = (3, 3)\n> \t  UnbalancedStop/Viewfinder_5  # GetParam() = (3, 5)\n> \t  UnbalancedStop/Viewfinder_8  # GetParam() = (3, 8)\n> \t  UnbalancedStop/Viewfinder_13  # GetParam() = (3, 13)\n> \t  UnbalancedStop/Viewfinder_21  # GetParam() = (3, 21)\n> \t  UnbalancedStop/Viewfinder_34  # GetParam() = (3, 34)\n> \t  UnbalancedStop/Viewfinder_55  # GetParam() = (3, 55)\n> \t  UnbalancedStop/Viewfinder_89  # GetParam() = (3, 89)\n>\n> Changes in v11:\n> - Thanks to Paul:\n>   - Updated statemachine test so that it always expects Camera::stop() to pass\n>\n> v10: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021891.html\n>\n> Changes in v10:\n> - Thanks to Jacopo:\n>   - Fixed some code style issues\n>   - Stopped calling CameraManager::stop() in the path it didn't start()\n>   - Updated comment header\n>   - Changed ASSERT_FALSE() uses to ASSERT_EQ(...,0) where it made sense\n>   - Updated copyright headers\n>   - Improved \"Test single capture cycles\" test description\n> - Thanks to Paul:\n>   - Added flag in meson.build to enable exceptions when compiling lc-compliance\n>\n> v9: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021753.html\n>\n> Changes in v9:\n> - Thanks to Jacopo:\n>   - Removed the Harness class, substituting methods with static helper functions\n>   - Added EXPECT() for the number of buffers allocated in SimpleCapture::start()\n>   - Changed ASSERT_FALSE() to ASSERT_EQ(..., 0)\n>   - Changed tests naming\n>   - Renamed single_stream.cpp to capture_test.cpp\n>   - Fixed some style issues\n>   - Added to --help that further Googletest options can be passed as environment\n>     variables\n> - Thanks to Paul:\n>   - Fixed compiling error on clang\n>\n> v8: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021398.html\n>\n> Changes in v8:\n> - Thanks to Laurent:\n>   - Fixed lc-compliance's meson.build to disable when gtest is not installed\n>   - Fixed compiling errors in Clang\n> - Thanks to Kieran:\n>   - Fixed coding style issues\n>   - Switched from malloc/free to new/delete in gtest parameter allocation\n>\n> v7: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021253.html\n>\n> Changes in v7:\n> - Thanks to Jacopo:\n>   - Fixed style issue\n>   - Made CameraManager a unique_ptr and passed to Environment as raw pointer\n>   - Moved the check for buffers allocated to the beginning of\n>     SimpleCapture::stop()\n> - Thanks to Niklas:\n>   - Removed intermediary filter string variable in Harness::initGtestParameters()\n>\n> v6: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021137.html\n>\n> Changes in v6:\n> - Thanks to Niklas:\n>   - Made return of Environment::cameraId() be a const reference\n>   - Changed buildGtestParameters() into initGtestParameters() and removed the\n>     need for Harness::gtestArgc_ and Harness::gtestArgv_\n>\n> v5: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020759.html\n>\n> Changes in v5:\n> - Thanks to Laurent:\n>   - Fixed style issues\n>   - Stored CameraManager and cameraId instead of Camera in Environment, and\n>     acquire and release the camera for each test\n> - Thanks to Niklas:\n>   - Moved buildGtestParameters() inside run()\n> - Thanks to Laurent & Niklas:\n>   - Improved Environment singleton class instantiation and destruction\n>\n> v4: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020594.html\n>\n> Changes in v4:\n> - Removed old lc-compliance results classes\n> - Thanks to Niklas:\n>   - Added patch 3 with the environment singleton in its own files\n>   - Improved naming of tests\n> - Thanks to Jacopo:\n>   - Added \\todo in Camera::stop()\n>\n> v3: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020382.html\n>\n> Changes in v3:\n> - Thanks to Niklas:\n>   - Went back to static test registration, and created a Singleton Environment\n>     class to store the camera global variable\n> - Added patch 4 implementing list and filter parameters\n> - Added a nameParameters() function to give meaningful names to the test\n>   parameters, removing the need to register each test suite for every role\n>\n> v2: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020056.html\n>\n> Changes in v2:\n> - Added patch 1 to make Camera::stop() idempotent as well from [5].\n> - Made test registration dynamic in patch 3\n> - Some other minor fixes\n>\n> [5] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html\n>\n> v1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html\n>\n> Nícolas F. R. A. Prado (5):\n>   libcamera: camera: Make stop() idempotent\n>   lc-compliance: Make SimpleCapture::stop() idempotent\n>   lc-compliance: Add Environment singleton\n>   lc-compliance: Refactor using Googletest\n>   lc-compliance: Add list and filter parameters\n>\n>  src/lc-compliance/capture_test.cpp   | 128 ++++++++++++++++++++\n>  src/lc-compliance/environment.cpp    |  20 ++++\n>  src/lc-compliance/environment.h      |  31 +++++\n>  src/lc-compliance/main.cpp           | 171 +++++++++++++++++----------\n>  src/lc-compliance/meson.build        |   9 +-\n>  src/lc-compliance/results.cpp        |  75 ------------\n>  src/lc-compliance/results.h          |  47 --------\n>  src/lc-compliance/simple_capture.cpp |  95 ++++++---------\n>  src/lc-compliance/simple_capture.h   |  11 +-\n>  src/lc-compliance/single_stream.cpp  |  97 ---------------\n>  src/lc-compliance/tests.h            |  16 ---\n>  src/libcamera/camera.cpp             |  20 +++-\n>  test/camera/statemachine.cpp         |  15 +--\n>  13 files changed, 358 insertions(+), 377 deletions(-)\n>  create mode 100644 src/lc-compliance/capture_test.cpp\n>  create mode 100644 src/lc-compliance/environment.cpp\n>  create mode 100644 src/lc-compliance/environment.h\n>  delete mode 100644 src/lc-compliance/results.cpp\n>  delete mode 100644 src/lc-compliance/results.h\n>  delete mode 100644 src/lc-compliance/single_stream.cpp\n>  delete mode 100644 src/lc-compliance/tests.h\n>\n> --\n> 2.32.0\n>","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id ADDC5BD794\n\tfor <parsemail@patchwork.libcamera.org>;\n\tTue,  6 Jul 2021 10:42:06 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 459EF68504;\n\tTue,  6 Jul 2021 12:42:06 +0200 (CEST)","from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net\n\t[217.70.183.199])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C045A60288\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tTue,  6 Jul 2021 12:42:05 +0200 (CEST)","(Authenticated sender: jacopo@jmondi.org)\n\tby relay9-d.mail.gandi.net (Postfix) with ESMTPSA id CAD34FF803;\n\tTue,  6 Jul 2021 10:42:04 +0000 (UTC)"],"Date":"Tue, 6 Jul 2021 12:42:53 +0200","From":"Jacopo Mondi <jacopo@jmondi.org>","To":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4=?= Prado <nfraprado@collabora.com>","Message-ID":"<20210706104253.nhr2bxt4fchithfn@uno.localdomain>","References":"<20210702122114.724932-1-nfraprado@collabora.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20210702122114.724932-1-nfraprado@collabora.com>","Subject":"Re: [libcamera-devel] [PATCH v11 0/5] lc-compliance: Refactor using\n\tGoogletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com, =?utf-8?q?A?=\n\t=?utf-8?b?bmRyw6k=?= Almeida <andrealmeid@collabora.com>","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"}},{"id":18000,"web_url":"https://patchwork.libcamera.org/comment/18000/","msgid":"<20210706121914.dy2syznr34mgtupw@notapiano>","date":"2021-07-06T12:19:14","subject":"Re: [libcamera-devel] [PATCH v11 0/5] lc-compliance: Refactor using\n\tGoogletest","submitter":{"id":84,"url":"https://patchwork.libcamera.org/api/people/84/","name":"Nícolas F. R. A. Prado","email":"nfraprado@collabora.com"},"content":"On Tue, Jul 06, 2021 at 12:42:53PM +0200, Jacopo Mondi wrote:\n> Hi Nicolas,\n> \n> On Fri, Jul 02, 2021 at 09:21:09AM -0300, Nícolas F. R. A. Prado wrote:\n> > (We're now running lc-compliance on KernelCI, more details below)\n> >\n> > This is a refactoring of lc-compliance using Googletest as the framework.\n> >\n> > Patches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3\n> > adds an Environment singleton to provide access to the camera from all tests.\n> > Patch 4 does the actual refactor. Patch 5 adds two new parameters, --list and\n> > --filter.\n> \n> Now pushed to master, thank you!\n\nAwesome! Thank you all for the reviews in this v11 journey :)\n\n> \n> >\n> > lc-compliance on KernelCI\n> > -------------------------\n> >\n> > lc-compliance is already being run on KernelCI [1] for mt8173-elm-hana\n> > (Chromebook), which has an USB camera. All test runs of lc-compliance so far can\n> > be seen on the Tests tab [2] by searching for \"lc-compliance\". I'm currently\n> > looking into adding Rock Pi 4 boards to the lab so that the tests can run on\n> > those too.\n> >\n> > [1] https://linux.kernelci.org/test/job/tip/branch/master/kernel/v5.13-rc6-239-g0b42677e2e5d/plan/lc-compliance/\n> >\n> > [2] https://linux.kernelci.org/test/\n> >\n> > I wrote a blog post on Collabora's blog [3] going over the process of adding\n> > lc-compliance to KernelCI, so check it out for more details on that. At the\n> > moment lc-compliance is being compiled from my tree [4], but as soon as this\n> > series get merged we can easily change KernelCI to compile from upstream.\n> >\n> > [3] https://www.collabora.com/news-and-blog/blog/2021/06/15/testing-cameras-with-lc-compliance-on-kernelci/\n> >\n> > [4] https://gitlab.collabora.com/nfraprado/libcamera/-/tree/lcc-gtest-v5\n> >\n> > lc-compliance Usage\n> > -------------------\n> >\n> > lc-compliance is built around testing a single camera (and possibly optional\n> > secondary cameras in the future), which is passed through the -c flag. The list\n> > of tests can be queried with:\n> >\n> > \tlc-compliance --list\n> >\n> > By default all tests are run, but it's possible to select which tests to run by\n> > using:\n> >\n> > \tlc-compliance -c '<camera_name>' --filter '<test_filter>'\n> >\n> > where the test_filter is in the same format as Googletest's --gtest_filter.\n> > Quoting its help:\n> >\n> >   --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]\n> >       Run only the tests whose name matches one of the positive patterns but\n> >       none of the negative patterns. '?' matches any single character; '*'\n> >       matches any substring; ':' separates two patterns.\n> >\n> > This is a snippet of a successful run:\n> >\n> > \t[==========] Running 120 tests from 1 test suite.\n> > \t[----------] Global test environment set-up.\n> > \t[----------] 120 tests from CaptureTests/SingleStream\n> > \t[ RUN      ] CaptureTests/SingleStream.Capture/Raw_1\n> > \t[31:47:27.239018193] [240317]  INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n> > \tCamera needs 4 requests, can't test only 1\n> > \t../src/lc-compliance/simple_capture.cpp:90: Skipped\n> >\n> > \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_1 (62 ms)\n> >\n> > \t[...]\n> >\n> > \t[ RUN      ] CaptureTests/SingleStream.UnbalancedStop/Viewfinder_89\n> > \t[31:50:19.266015455] [240317]  INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n> > \t[       OK ] CaptureTests/SingleStream.UnbalancedStop/Viewfinder_89 (3142 ms)\n> > \t[----------] 120 tests from CaptureTests/SingleStream (175170 ms total)\n> >\n> > \t[----------] Global test environment tear-down\n> > \t[==========] 120 tests from 1 test suite ran. (175170 ms total)\n> > \t[  PASSED  ] 96 tests.\n> > \t[  SKIPPED ] 24 tests, listed below:\n> > \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_1\n> > \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_2\n> > \t[  SKIPPED ] CaptureTests/SingleStream.Capture/Raw_3\n> > \t[  SKIPPED ] CaptureTests/SingleStream.Capture/StillCapture_1\n> >\n> > \t[...]\n> >\n> > And how a failure looks like:\n> >\n> > \t[ RUN      ] CaptureTests/SingleStream.Capture/Raw_5\n> > \t[31:55:30.651452784] [241401]  INFO Camera camera.cpp:912 configuring streams: (0) 1280x720-MJPEG\n> > \t../src/lc-compliance/simple_capture.cpp:117: Failure\n> > \tExpected equality of these values:\n> > \t  captureCount_\n> > \t    Which is: 5\n> > \t  captureLimit_\n> > \t    Which is: 6\n> > \t[  FAILED  ] CaptureTests/SingleStream.Capture/Raw_5, where GetParam() = (0, 5) (317 ms)\n> >\n> > The current complete list of tests is shown below:\n> >\n> > \tCaptureTests/SingleStream.\n> > \t  Capture/Raw_1  # GetParam() = (0, 1)\n> > \t  Capture/Raw_2  # GetParam() = (0, 2)\n> > \t  Capture/Raw_3  # GetParam() = (0, 3)\n> > \t  Capture/Raw_5  # GetParam() = (0, 5)\n> > \t  Capture/Raw_8  # GetParam() = (0, 8)\n> > \t  Capture/Raw_13  # GetParam() = (0, 13)\n> > \t  Capture/Raw_21  # GetParam() = (0, 21)\n> > \t  Capture/Raw_34  # GetParam() = (0, 34)\n> > \t  Capture/Raw_55  # GetParam() = (0, 55)\n> > \t  Capture/Raw_89  # GetParam() = (0, 89)\n> > \t  Capture/StillCapture_1  # GetParam() = (1, 1)\n> > \t  Capture/StillCapture_2  # GetParam() = (1, 2)\n> > \t  Capture/StillCapture_3  # GetParam() = (1, 3)\n> > \t  Capture/StillCapture_5  # GetParam() = (1, 5)\n> > \t  Capture/StillCapture_8  # GetParam() = (1, 8)\n> > \t  Capture/StillCapture_13  # GetParam() = (1, 13)\n> > \t  Capture/StillCapture_21  # GetParam() = (1, 21)\n> > \t  Capture/StillCapture_34  # GetParam() = (1, 34)\n> > \t  Capture/StillCapture_55  # GetParam() = (1, 55)\n> > \t  Capture/StillCapture_89  # GetParam() = (1, 89)\n> > \t  Capture/VideoRecording_1  # GetParam() = (2, 1)\n> > \t  Capture/VideoRecording_2  # GetParam() = (2, 2)\n> > \t  Capture/VideoRecording_3  # GetParam() = (2, 3)\n> > \t  Capture/VideoRecording_5  # GetParam() = (2, 5)\n> > \t  Capture/VideoRecording_8  # GetParam() = (2, 8)\n> > \t  Capture/VideoRecording_13  # GetParam() = (2, 13)\n> > \t  Capture/VideoRecording_21  # GetParam() = (2, 21)\n> > \t  Capture/VideoRecording_34  # GetParam() = (2, 34)\n> > \t  Capture/VideoRecording_55  # GetParam() = (2, 55)\n> > \t  Capture/VideoRecording_89  # GetParam() = (2, 89)\n> > \t  Capture/Viewfinder_1  # GetParam() = (3, 1)\n> > \t  Capture/Viewfinder_2  # GetParam() = (3, 2)\n> > \t  Capture/Viewfinder_3  # GetParam() = (3, 3)\n> > \t  Capture/Viewfinder_5  # GetParam() = (3, 5)\n> > \t  Capture/Viewfinder_8  # GetParam() = (3, 8)\n> > \t  Capture/Viewfinder_13  # GetParam() = (3, 13)\n> > \t  Capture/Viewfinder_21  # GetParam() = (3, 21)\n> > \t  Capture/Viewfinder_34  # GetParam() = (3, 34)\n> > \t  Capture/Viewfinder_55  # GetParam() = (3, 55)\n> > \t  Capture/Viewfinder_89  # GetParam() = (3, 89)\n> > \t  CaptureStartStop/Raw_1  # GetParam() = (0, 1)\n> > \t  CaptureStartStop/Raw_2  # GetParam() = (0, 2)\n> > \t  CaptureStartStop/Raw_3  # GetParam() = (0, 3)\n> > \t  CaptureStartStop/Raw_5  # GetParam() = (0, 5)\n> > \t  CaptureStartStop/Raw_8  # GetParam() = (0, 8)\n> > \t  CaptureStartStop/Raw_13  # GetParam() = (0, 13)\n> > \t  CaptureStartStop/Raw_21  # GetParam() = (0, 21)\n> > \t  CaptureStartStop/Raw_34  # GetParam() = (0, 34)\n> > \t  CaptureStartStop/Raw_55  # GetParam() = (0, 55)\n> > \t  CaptureStartStop/Raw_89  # GetParam() = (0, 89)\n> > \t  CaptureStartStop/StillCapture_1  # GetParam() = (1, 1)\n> > \t  CaptureStartStop/StillCapture_2  # GetParam() = (1, 2)\n> > \t  CaptureStartStop/StillCapture_3  # GetParam() = (1, 3)\n> > \t  CaptureStartStop/StillCapture_5  # GetParam() = (1, 5)\n> > \t  CaptureStartStop/StillCapture_8  # GetParam() = (1, 8)\n> > \t  CaptureStartStop/StillCapture_13  # GetParam() = (1, 13)\n> > \t  CaptureStartStop/StillCapture_21  # GetParam() = (1, 21)\n> > \t  CaptureStartStop/StillCapture_34  # GetParam() = (1, 34)\n> > \t  CaptureStartStop/StillCapture_55  # GetParam() = (1, 55)\n> > \t  CaptureStartStop/StillCapture_89  # GetParam() = (1, 89)\n> > \t  CaptureStartStop/VideoRecording_1  # GetParam() = (2, 1)\n> > \t  CaptureStartStop/VideoRecording_2  # GetParam() = (2, 2)\n> > \t  CaptureStartStop/VideoRecording_3  # GetParam() = (2, 3)\n> > \t  CaptureStartStop/VideoRecording_5  # GetParam() = (2, 5)\n> > \t  CaptureStartStop/VideoRecording_8  # GetParam() = (2, 8)\n> > \t  CaptureStartStop/VideoRecording_13  # GetParam() = (2, 13)\n> > \t  CaptureStartStop/VideoRecording_21  # GetParam() = (2, 21)\n> > \t  CaptureStartStop/VideoRecording_34  # GetParam() = (2, 34)\n> > \t  CaptureStartStop/VideoRecording_55  # GetParam() = (2, 55)\n> > \t  CaptureStartStop/VideoRecording_89  # GetParam() = (2, 89)\n> > \t  CaptureStartStop/Viewfinder_1  # GetParam() = (3, 1)\n> > \t  CaptureStartStop/Viewfinder_2  # GetParam() = (3, 2)\n> > \t  CaptureStartStop/Viewfinder_3  # GetParam() = (3, 3)\n> > \t  CaptureStartStop/Viewfinder_5  # GetParam() = (3, 5)\n> > \t  CaptureStartStop/Viewfinder_8  # GetParam() = (3, 8)\n> > \t  CaptureStartStop/Viewfinder_13  # GetParam() = (3, 13)\n> > \t  CaptureStartStop/Viewfinder_21  # GetParam() = (3, 21)\n> > \t  CaptureStartStop/Viewfinder_34  # GetParam() = (3, 34)\n> > \t  CaptureStartStop/Viewfinder_55  # GetParam() = (3, 55)\n> > \t  CaptureStartStop/Viewfinder_89  # GetParam() = (3, 89)\n> > \t  UnbalancedStop/Raw_1  # GetParam() = (0, 1)\n> > \t  UnbalancedStop/Raw_2  # GetParam() = (0, 2)\n> > \t  UnbalancedStop/Raw_3  # GetParam() = (0, 3)\n> > \t  UnbalancedStop/Raw_5  # GetParam() = (0, 5)\n> > \t  UnbalancedStop/Raw_8  # GetParam() = (0, 8)\n> > \t  UnbalancedStop/Raw_13  # GetParam() = (0, 13)\n> > \t  UnbalancedStop/Raw_21  # GetParam() = (0, 21)\n> > \t  UnbalancedStop/Raw_34  # GetParam() = (0, 34)\n> > \t  UnbalancedStop/Raw_55  # GetParam() = (0, 55)\n> > \t  UnbalancedStop/Raw_89  # GetParam() = (0, 89)\n> > \t  UnbalancedStop/StillCapture_1  # GetParam() = (1, 1)\n> > \t  UnbalancedStop/StillCapture_2  # GetParam() = (1, 2)\n> > \t  UnbalancedStop/StillCapture_3  # GetParam() = (1, 3)\n> > \t  UnbalancedStop/StillCapture_5  # GetParam() = (1, 5)\n> > \t  UnbalancedStop/StillCapture_8  # GetParam() = (1, 8)\n> > \t  UnbalancedStop/StillCapture_13  # GetParam() = (1, 13)\n> > \t  UnbalancedStop/StillCapture_21  # GetParam() = (1, 21)\n> > \t  UnbalancedStop/StillCapture_34  # GetParam() = (1, 34)\n> > \t  UnbalancedStop/StillCapture_55  # GetParam() = (1, 55)\n> > \t  UnbalancedStop/StillCapture_89  # GetParam() = (1, 89)\n> > \t  UnbalancedStop/VideoRecording_1  # GetParam() = (2, 1)\n> > \t  UnbalancedStop/VideoRecording_2  # GetParam() = (2, 2)\n> > \t  UnbalancedStop/VideoRecording_3  # GetParam() = (2, 3)\n> > \t  UnbalancedStop/VideoRecording_5  # GetParam() = (2, 5)\n> > \t  UnbalancedStop/VideoRecording_8  # GetParam() = (2, 8)\n> > \t  UnbalancedStop/VideoRecording_13  # GetParam() = (2, 13)\n> > \t  UnbalancedStop/VideoRecording_21  # GetParam() = (2, 21)\n> > \t  UnbalancedStop/VideoRecording_34  # GetParam() = (2, 34)\n> > \t  UnbalancedStop/VideoRecording_55  # GetParam() = (2, 55)\n> > \t  UnbalancedStop/VideoRecording_89  # GetParam() = (2, 89)\n> > \t  UnbalancedStop/Viewfinder_1  # GetParam() = (3, 1)\n> > \t  UnbalancedStop/Viewfinder_2  # GetParam() = (3, 2)\n> > \t  UnbalancedStop/Viewfinder_3  # GetParam() = (3, 3)\n> > \t  UnbalancedStop/Viewfinder_5  # GetParam() = (3, 5)\n> > \t  UnbalancedStop/Viewfinder_8  # GetParam() = (3, 8)\n> > \t  UnbalancedStop/Viewfinder_13  # GetParam() = (3, 13)\n> > \t  UnbalancedStop/Viewfinder_21  # GetParam() = (3, 21)\n> > \t  UnbalancedStop/Viewfinder_34  # GetParam() = (3, 34)\n> > \t  UnbalancedStop/Viewfinder_55  # GetParam() = (3, 55)\n> > \t  UnbalancedStop/Viewfinder_89  # GetParam() = (3, 89)\n> >\n> > Changes in v11:\n> > - Thanks to Paul:\n> >   - Updated statemachine test so that it always expects Camera::stop() to pass\n> >\n> > v10: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021891.html\n> >\n> > Changes in v10:\n> > - Thanks to Jacopo:\n> >   - Fixed some code style issues\n> >   - Stopped calling CameraManager::stop() in the path it didn't start()\n> >   - Updated comment header\n> >   - Changed ASSERT_FALSE() uses to ASSERT_EQ(...,0) where it made sense\n> >   - Updated copyright headers\n> >   - Improved \"Test single capture cycles\" test description\n> > - Thanks to Paul:\n> >   - Added flag in meson.build to enable exceptions when compiling lc-compliance\n> >\n> > v9: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021753.html\n> >\n> > Changes in v9:\n> > - Thanks to Jacopo:\n> >   - Removed the Harness class, substituting methods with static helper functions\n> >   - Added EXPECT() for the number of buffers allocated in SimpleCapture::start()\n> >   - Changed ASSERT_FALSE() to ASSERT_EQ(..., 0)\n> >   - Changed tests naming\n> >   - Renamed single_stream.cpp to capture_test.cpp\n> >   - Fixed some style issues\n> >   - Added to --help that further Googletest options can be passed as environment\n> >     variables\n> > - Thanks to Paul:\n> >   - Fixed compiling error on clang\n> >\n> > v8: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021398.html\n> >\n> > Changes in v8:\n> > - Thanks to Laurent:\n> >   - Fixed lc-compliance's meson.build to disable when gtest is not installed\n> >   - Fixed compiling errors in Clang\n> > - Thanks to Kieran:\n> >   - Fixed coding style issues\n> >   - Switched from malloc/free to new/delete in gtest parameter allocation\n> >\n> > v7: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021253.html\n> >\n> > Changes in v7:\n> > - Thanks to Jacopo:\n> >   - Fixed style issue\n> >   - Made CameraManager a unique_ptr and passed to Environment as raw pointer\n> >   - Moved the check for buffers allocated to the beginning of\n> >     SimpleCapture::stop()\n> > - Thanks to Niklas:\n> >   - Removed intermediary filter string variable in Harness::initGtestParameters()\n> >\n> > v6: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021137.html\n> >\n> > Changes in v6:\n> > - Thanks to Niklas:\n> >   - Made return of Environment::cameraId() be a const reference\n> >   - Changed buildGtestParameters() into initGtestParameters() and removed the\n> >     need for Harness::gtestArgc_ and Harness::gtestArgv_\n> >\n> > v5: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020759.html\n> >\n> > Changes in v5:\n> > - Thanks to Laurent:\n> >   - Fixed style issues\n> >   - Stored CameraManager and cameraId instead of Camera in Environment, and\n> >     acquire and release the camera for each test\n> > - Thanks to Niklas:\n> >   - Moved buildGtestParameters() inside run()\n> > - Thanks to Laurent & Niklas:\n> >   - Improved Environment singleton class instantiation and destruction\n> >\n> > v4: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020594.html\n> >\n> > Changes in v4:\n> > - Removed old lc-compliance results classes\n> > - Thanks to Niklas:\n> >   - Added patch 3 with the environment singleton in its own files\n> >   - Improved naming of tests\n> > - Thanks to Jacopo:\n> >   - Added \\todo in Camera::stop()\n> >\n> > v3: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020382.html\n> >\n> > Changes in v3:\n> > - Thanks to Niklas:\n> >   - Went back to static test registration, and created a Singleton Environment\n> >     class to store the camera global variable\n> > - Added patch 4 implementing list and filter parameters\n> > - Added a nameParameters() function to give meaningful names to the test\n> >   parameters, removing the need to register each test suite for every role\n> >\n> > v2: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020056.html\n> >\n> > Changes in v2:\n> > - Added patch 1 to make Camera::stop() idempotent as well from [5].\n> > - Made test registration dynamic in patch 3\n> > - Some other minor fixes\n> >\n> > [5] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html\n> >\n> > v1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html\n> >\n> > Nícolas F. R. A. Prado (5):\n> >   libcamera: camera: Make stop() idempotent\n> >   lc-compliance: Make SimpleCapture::stop() idempotent\n> >   lc-compliance: Add Environment singleton\n> >   lc-compliance: Refactor using Googletest\n> >   lc-compliance: Add list and filter parameters\n> >\n> >  src/lc-compliance/capture_test.cpp   | 128 ++++++++++++++++++++\n> >  src/lc-compliance/environment.cpp    |  20 ++++\n> >  src/lc-compliance/environment.h      |  31 +++++\n> >  src/lc-compliance/main.cpp           | 171 +++++++++++++++++----------\n> >  src/lc-compliance/meson.build        |   9 +-\n> >  src/lc-compliance/results.cpp        |  75 ------------\n> >  src/lc-compliance/results.h          |  47 --------\n> >  src/lc-compliance/simple_capture.cpp |  95 ++++++---------\n> >  src/lc-compliance/simple_capture.h   |  11 +-\n> >  src/lc-compliance/single_stream.cpp  |  97 ---------------\n> >  src/lc-compliance/tests.h            |  16 ---\n> >  src/libcamera/camera.cpp             |  20 +++-\n> >  test/camera/statemachine.cpp         |  15 +--\n> >  13 files changed, 358 insertions(+), 377 deletions(-)\n> >  create mode 100644 src/lc-compliance/capture_test.cpp\n> >  create mode 100644 src/lc-compliance/environment.cpp\n> >  create mode 100644 src/lc-compliance/environment.h\n> >  delete mode 100644 src/lc-compliance/results.cpp\n> >  delete mode 100644 src/lc-compliance/results.h\n> >  delete mode 100644 src/lc-compliance/single_stream.cpp\n> >  delete mode 100644 src/lc-compliance/tests.h\n> >\n> > --\n> > 2.32.0\n> >","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4A088BD794\n\tfor <parsemail@patchwork.libcamera.org>;\n\tTue,  6 Jul 2021 12:19:22 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B7692684EA;\n\tTue,  6 Jul 2021 14:19:21 +0200 (CEST)","from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C7CA060288\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tTue,  6 Jul 2021 14:19:20 +0200 (CEST)","from notapiano (unknown\n\t[IPv6:2804:14c:1a9:2434:6553:ad0c:9d2a:24db])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id 2A04A1F40E82;\n\tTue,  6 Jul 2021 13:19:18 +0100 (BST)"],"Date":"Tue, 6 Jul 2021 09:19:14 -0300","From":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4=?= Prado <nfraprado@collabora.com>","To":"Jacopo Mondi <jacopo@jmondi.org>","Message-ID":"<20210706121914.dy2syznr34mgtupw@notapiano>","References":"<20210702122114.724932-1-nfraprado@collabora.com>\n\t<20210706104253.nhr2bxt4fchithfn@uno.localdomain>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20210706104253.nhr2bxt4fchithfn@uno.localdomain>","Subject":"Re: [libcamera-devel] [PATCH v11 0/5] lc-compliance: Refactor using\n\tGoogletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com, =?utf-8?q?A?=\n\t=?utf-8?b?bmRyw6k=?= Almeida <andrealmeid@collabora.com>","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"}}]