[{"id":16797,"web_url":"https://patchwork.libcamera.org/comment/16797/","msgid":"","date":"2021-05-06T08:48:16","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Nícolas,\n\nThanks for your work!\n\nI really like how gtest helps make the actual tests to be easier to \nread.\n\nOn 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> Refactor lc-compliance using Googletest as the test framework.\n> \n> Signed-off-by: Nícolas F. R. A. Prado \n> ---\n> Changes in v2:\n> - Changed from static to dynamic test registration\n> - Removed -c flag\n> \n> There's still an issue with the refactoring that the shared_ptr of the tests\n> apparently aren't being deleted on time, which causes:\n> \n> \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> \tSegmentation fault (core dumped)\n> \n> I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> that didn't work. In fact, when just listing the tests, the constructor for the\n> tests isn't even called. So I think the issue has to do with the passing of the\n> camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> \n> \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> \n> Not sure how to solve this. Any tip would be welcome.\n\nI will leave this for now as I have a different worry about \nREGISTER_TESTS() design below.\n\n> \n> src/lc-compliance/main.cpp | 99 +++++++++---------\n> src/lc-compliance/meson.build | 3 +\n> src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> src/lc-compliance/simple_capture.h | 8 +-\n> src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> src/lc-compliance/tests.h | 15 ++-\n> 6 files changed, 175 insertions(+), 175 deletions(-)\n> \n> diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> index 54cee54aa978..add0d7729aec 100644\n> --- a/src/lc-compliance/main.cpp\n> +++ b/src/lc-compliance/main.cpp\n> @@ -9,6 +9,8 @@\n> #include \n> #include \n> \n> +#include \n> +\n> #include \n> \n> #include \"../cam/options.h\"\n> @@ -17,7 +19,6 @@\n> using namespace libcamera;\n> \n> enum {\n> -\tOptCamera = 'c',\n> \tOptHelp = 'h',\n> };\n> \n> @@ -28,14 +29,15 @@ public:\n> \t~Harness();\n> \n> \tint exec();\n> +\tint init();\n> +\tvoid registerTests();\n> \n> private:\n> -\tint init();\n> \tvoid listCameras();\n> \n> \tOptionsParser::Options options_;\n> \tstd::unique_ptr cm_;\n> -\tstd::shared_ptr camera_;\n> +\tstd::vector> cameras_;\n> };\n> \n> Harness::Harness(const OptionsParser::Options &options)\n> @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> \n> Harness::~Harness()\n> {\n> -\tif (camera_) {\n> -\t\tcamera_->release();\n> -\t\tcamera_.reset();\n> +\tfor (auto &c : cameras_) {\n> +\t\tc->release();\n> +\t\tc.reset();\n> \t}\n> \n> \tcm_->stop();\n> }\n> \n> -int Harness::exec()\n> -{\n> -\tint ret = init();\n> -\tif (ret)\n> -\t\treturn ret;\n> -\n> -\tstd::vector results;\n> -\n> -\tresults.push_back(testSingleStream(camera_));\n> -\n> -\tfor (const Results &result : results) {\n> -\t\tret = result.summary();\n> -\t\tif (ret)\n> -\t\t\treturn ret;\n> -\t}\n> -\n> -\treturn 0;\n> -}\n> -\n> int Harness::init()\n> {\n> \tint ret = cm_->start();\n> @@ -82,42 +65,26 @@ int Harness::init()\n> \t\treturn ret;\n> \t}\n> \n> -\tif (!options_.isSet(OptCamera)) {\n> -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> -\t\tlistCameras();\n> -\t\treturn -ENODEV;\n> -\t}\n> -\n> -\tconst std::string &cameraId = options_[OptCamera];\n> -\tcamera_ = cm_->get(cameraId);\n> -\tif (!camera_) {\n> -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> -\t\tlistCameras();\n> -\t\treturn -ENODEV;\n> +\tfor (auto cam : cm_->cameras()) {\n> +\t\tif (cam->acquire()) {\n> +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> +\t\t\treturn -EINVAL;\n> +\t\t}\n> +\t\tcameras_.push_back(cam);\n\nI don't like this, I think we need to retain the ability to select which \ncamera to test. And further down the line maybe allow more then one \ncamera to be selected if we want to test concurrent use of two specific \nones. But for the work in this series I think you can ignore multiple \ncameras.\n\nIf I understand the coverletter correctly the correct thing here the cli \narguments should be feed to gtest directly to be able to filter on what \ntests are run. Is there support in getst to also pass arguments to \ntests? Maybe we could remove the need to parse arguments ourself \ncompletely?\n\n> \t}\n> \n> -\tif (camera_->acquire()) {\n> -\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> -\t\treturn -EINVAL;\n> -\t}\n> -\n> -\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> -\n> \treturn 0;\n> }\n> \n> void Harness::listCameras()\n> {\n> -\tfor (const std::shared_ptr &cam : cm_->cameras())\n> -\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> +\tfor (const std::shared_ptr &c : cm_->cameras())\n> +\t\tstd::cout << \"- \" << c.get()->id() << std::endl;\n> }\n> \n> static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> {\n> \tOptionsParser parser;\n> -\tparser.addOption(OptCamera, OptionString,\n> -\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> -\t\t\t ArgumentRequired, \"camera\");\n> \tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> \t\t\t \"help\");\n> \n> @@ -133,6 +100,31 @@ static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> \treturn 0;\n> }\n> \n> +/*\n> + * Make asserts act like exceptions, otherwise they only fail (or skip) the\n> + * current function. From gtest documentation:\n> + * https://google.github.io/googletest/advanced.html#asserting-on-subroutines-with-an-exception\n> + */\n> +class ThrowListener : public testing::EmptyTestEventListener {\n> +\tvoid OnTestPartResult(const testing::TestPartResult& result) override {\n> +\t\tif (result.type() == testing::TestPartResult::kFatalFailure\n> +\t\t || result.type() == testing::TestPartResult::kSkip) {\n> +\t\t\tthrow testing::AssertionException(result);\n> +\t\t}\n> +\t}\n> +};\n> +\n> +\n> +void Harness::registerTests() {\n> +\tstd::map roles = {{Raw, \"Raw\"},\n> +\t\t\t\t\t\t {StillCapture, \"Still\"},\n> +\t\t\t\t\t\t {VideoRecording, \"Video\"},\n> +\t\t\t\t\t\t {Viewfinder, \"Viewfinder\"}};\n> +\tstd::vector requests = {1, 2, 3, 5, 8, 13, 21, 34, 55, 89};\n> +\n> +\tregisterSingleStreamTests(cameras_, roles, requests);\n> +}\n> +\n> int main(int argc, char **argv)\n> {\n> \tOptionsParser::Options options;\n> @@ -143,6 +135,13 @@ int main(int argc, char **argv)\n> \t\treturn EXIT_FAILURE;\n> \n> \tHarness harness(options);\n> +\tret = harness.init();\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tharness.registerTests();\n> \n> -\treturn harness.exec() ? EXIT_FAILURE : EXIT_SUCCESS;\n> +\t::testing::InitGoogleTest(&argc, argv);\n> +\ttesting::UnitTest::GetInstance()->listeners().Append(new ThrowListener);\n> +\treturn RUN_ALL_TESTS();\n> }\n> diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> index a2bfcceb1259..704bc18af3e1 100644\n> --- a/src/lc-compliance/meson.build\n> +++ b/src/lc-compliance/meson.build\n> @@ -18,10 +18,13 @@ lc_compliance_sources = files([\n> 'single_stream.cpp',\n> ])\n> \n> +gtest_dep = dependency('gtest')\n> +\n> lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> dependencies : [\n> libatomic,\n> libcamera_dep,\n> libevent,\n> + gtest_dep,\n> ],\n> install : true)\n> diff --git a/src/lc-compliance/simple_capture.cpp b/src/lc-compliance/simple_capture.cpp\n> index f90fe6d0f9aa..7731eb16f8c2 100644\n> --- a/src/lc-compliance/simple_capture.cpp\n> +++ b/src/lc-compliance/simple_capture.cpp\n> @@ -5,6 +5,8 @@\n> * simple_capture.cpp - Simple capture helper\n> */\n> \n> +#include \n> +\n> #include \"simple_capture.h\"\n> \n> using namespace libcamera;\n> @@ -20,38 +22,34 @@ SimpleCapture::~SimpleCapture()\n> \tstop();\n> }\n> \n> -Results::Result SimpleCapture::configure(StreamRole role)\n> +void SimpleCapture::configure(StreamRole role)\n> {\n> \tconfig_ = camera_->generateConfiguration({ role });\n> \n> -\tif (!config_)\n> -\t\treturn { Results::Skip, \"Role not supported by camera\" };\n> +\tif (!config_) {\n> +\t\tstd::cout << \"Role not supported by camera\" << std::endl;\n> +\t\tGTEST_SKIP();\n> +\t}\n> \n> \tif (config_->validate() != CameraConfiguration::Valid) {\n> \t\tconfig_.reset();\n> -\t\treturn { Results::Fail, \"Configuration not valid\" };\n> +\t\tFAIL() << \"Configuration not valid\";\n> \t}\n> \n> \tif (camera_->configure(config_.get())) {\n> \t\tconfig_.reset();\n> -\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> +\t\tFAIL() << \"Failed to configure camera\";\n> \t}\n> -\n> -\treturn { Results::Pass, \"Configure camera\" };\n> }\n> \n> -Results::Result SimpleCapture::start()\n> +void SimpleCapture::start()\n> {\n> \tStream *stream = config_->at(0).stream();\n> -\tif (allocator_->allocate(stream) < 0)\n> -\t\treturn { Results::Fail, \"Failed to allocate buffers\" };\n> +\tASSERT_GE(allocator_->allocate(stream), 0) << \"Failed to allocate buffers\";\n> \n> -\tif (camera_->start())\n> -\t\treturn { Results::Fail, \"Failed to start camera\" };\n> +\tASSERT_TRUE(!camera_->start()) << \"Failed to start camera\";\n> \n> \tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> -\n> -\treturn { Results::Pass, \"Started camera\" };\n> }\n> \n> void SimpleCapture::stop()\n> @@ -77,22 +75,19 @@ SimpleCaptureBalanced::SimpleCaptureBalanced(std::shared_ptr camera)\n> {\n> }\n> \n> -Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> +void SimpleCaptureBalanced::capture(unsigned int numRequests)\n> {\n> -\tResults::Result ret = start();\n> -\tif (ret.first != Results::Pass)\n> -\t\treturn ret;\n> +\tstart();\n> \n> \tStream *stream = config_->at(0).stream();\n> \tconst std::vector> &buffers = allocator_->buffers(stream);\n> \n> \t/* No point in testing less requests then the camera depth. */\n> \tif (buffers.size() > numRequests) {\n> -\t\t/* Cache buffers.size() before we destroy it in stop() */\n> -\t\tint buffers_size = buffers.size();\n> -\n> -\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers_size)\n> -\t\t\t+ \" requests, can't test only \" + std::to_string(numRequests) };\n> +\t\tstd::cout << \"Camera needs \" + std::to_string(buffers.size())\n> +\t\t\t+ \" requests, can't test only \"\n> +\t\t\t+ std::to_string(numRequests) << std::endl;\n> +\t\tGTEST_SKIP();\n> \t}\n> \n> \tqueueCount_ = 0;\n> @@ -103,14 +98,11 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> \tstd::vector> requests;\n> \tfor (const std::unique_ptr &buffer : buffers) {\n> \t\tstd::unique_ptr request = camera_->createRequest();\n> -\t\tif (!request)\n> -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> \n> -\t\tif (request->addBuffer(stream, buffer.get()))\n> -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> \n> -\t\tif (queueRequest(request.get()) < 0)\n> -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> +\t\tASSERT_GE(queueRequest(request.get()), 0) << \"Failed to queue request\";\n> \n> \t\trequests.push_back(std::move(request));\n> \t}\n> @@ -121,12 +113,7 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> \tstop();\n> \tdelete loop_;\n> \n> -\tif (captureCount_ != captureLimit_)\n> -\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) +\n> -\t\t\t\" request, wanted \" + std::to_string(captureLimit_) };\n> -\n> -\treturn { Results::Pass, \"Balanced capture of \" +\n> -\t\tstd::to_string(numRequests) + \" requests\" };\n> +\tASSERT_EQ(captureCount_, captureLimit_);\n> }\n> \n> int SimpleCaptureBalanced::queueRequest(Request *request)\n> @@ -158,11 +145,9 @@ SimpleCaptureUnbalanced::SimpleCaptureUnbalanced(std::shared_ptr camera)\n> {\n> }\n> \n> -Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> +void SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> {\n> -\tResults::Result ret = start();\n> -\tif (ret.first != Results::Pass)\n> -\t\treturn ret;\n> +\tstart();\n> \n> \tStream *stream = config_->at(0).stream();\n> \tconst std::vector> &buffers = allocator_->buffers(stream);\n> @@ -174,14 +159,11 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> \tstd::vector> requests;\n> \tfor (const std::unique_ptr &buffer : buffers) {\n> \t\tstd::unique_ptr request = camera_->createRequest();\n> -\t\tif (!request)\n> -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> \n> -\t\tif (request->addBuffer(stream, buffer.get()))\n> -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> \n> -\t\tif (camera_->queueRequest(request.get()) < 0)\n> -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> +\t\tASSERT_GE(camera_->queueRequest(request.get()), 0) << \"Failed to queue request\";\n> \n> \t\trequests.push_back(std::move(request));\n> \t}\n> @@ -192,7 +174,7 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> \tstop();\n> \tdelete loop_;\n> \n> -\treturn { status ? Results::Fail : Results::Pass, \"Unbalanced capture of \" + std::to_string(numRequests) + \" requests\" };\n> +\tASSERT_FALSE(status);\n> }\n> \n> void SimpleCaptureUnbalanced::requestComplete(Request *request)\n> diff --git a/src/lc-compliance/simple_capture.h b/src/lc-compliance/simple_capture.h\n> index d9de53fb63a3..0f8465083456 100644\n> --- a/src/lc-compliance/simple_capture.h\n> +++ b/src/lc-compliance/simple_capture.h\n> @@ -17,13 +17,13 @@\n> class SimpleCapture\n> {\n> public:\n> -\tResults::Result configure(libcamera::StreamRole role);\n> +\tvoid configure(libcamera::StreamRole role);\n> \n> protected:\n> \tSimpleCapture(std::shared_ptr camera);\n> \tvirtual ~SimpleCapture();\n> \n> -\tResults::Result start();\n> +\tvoid start();\n> \tvoid stop();\n> \n> \tvirtual void requestComplete(libcamera::Request *request) = 0;\n> @@ -40,7 +40,7 @@ class SimpleCaptureBalanced : public SimpleCapture\n> public:\n> \tSimpleCaptureBalanced(std::shared_ptr camera);\n> \n> -\tResults::Result capture(unsigned int numRequests);\n> +\tvoid capture(unsigned int numRequests);\n> \n> private:\n> \tint queueRequest(libcamera::Request *request);\n> @@ -56,7 +56,7 @@ class SimpleCaptureUnbalanced : public SimpleCapture\n> public:\n> \tSimpleCaptureUnbalanced(std::shared_ptr camera);\n> \n> -\tResults::Result capture(unsigned int numRequests);\n> +\tvoid capture(unsigned int numRequests);\n> \n> private:\n> \tvoid requestComplete(libcamera::Request *request) override;\n> diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> index 8318b42f42d6..eb6a6f305826 100644\n> --- a/src/lc-compliance/single_stream.cpp\n> +++ b/src/lc-compliance/single_stream.cpp\n> @@ -7,91 +7,94 @@\n> \n> #include \n> \n> +#include \n> +\n> #include \"simple_capture.h\"\n> #include \"tests.h\"\n> \n> using namespace libcamera;\n> \n> -Results::Result testRequestBalance(std::shared_ptr camera,\n> -\t\t\t\t StreamRole role, unsigned int startCycles,\n> -\t\t\t\t unsigned int numRequests)\n> -{\n> -\tSimpleCaptureBalanced capture(camera);\n> +class SingleStream : public testing::Test {\n> +public:\n> +\texplicit SingleStream(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> +\t\t: camera_(camera), role_(role), numRequests_(numRequests) {}\n> \n> -\tResults::Result ret = capture.configure(role);\n> -\tif (ret.first != Results::Pass)\n> -\t\treturn ret;\n> +protected:\n> +\tstd::shared_ptr camera_;\n> +\tStreamRole role_;\n> +\tunsigned int numRequests_;\n> +};\n> \n> -\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> -\t\tret = capture.capture(numRequests);\n> -\t\tif (ret.first != Results::Pass)\n> -\t\t\treturn ret;\n> -\t}\n> +/*\n> + * Test single capture cycles\n> + *\n> + * Makes sure the camera completes the exact number of requests queued. Example\n> + * failure is a camera that needs N+M requests queued to complete N requests to\n> + * the application.\n> + */\n> +class BalancedSingleCycle : public SingleStream {\n> +public:\n> +\texplicit BalancedSingleCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> +\t\t: SingleStream(camera, role, numRequests) {}\n> \n> -\treturn { Results::Pass, \"Balanced capture of \" +\n> -\t\tstd::to_string(numRequests) + \" requests with \" +\n> -\t\tstd::to_string(startCycles) + \" start cycles\" };\n> -}\n> +\tvoid TestBody() override {\n> +\t\tSimpleCaptureBalanced capture(camera_);\n> \n> -Results::Result testRequestUnbalance(std::shared_ptr camera,\n> -\t\t\t\t StreamRole role, unsigned int numRequests)\n> -{\n> -\tSimpleCaptureUnbalanced capture(camera);\n> +\t\tcapture.configure(role_);\n> \n> -\tResults::Result ret = capture.configure(role);\n> -\tif (ret.first != Results::Pass)\n> -\t\treturn ret;\n> +\t\tcapture.capture(numRequests_);\n> +\t};\n> +};\n> \n> -\treturn capture.capture(numRequests);\n> -}\n> +/*\n> + * Test multiple start/stop cycles\n> + *\n> + * Makes sure the camera supports multiple start/stop cycles. Example failure is\n> + * a camera that does not clean up correctly in its error path but is only\n> + * tested by single-capture applications.\n> + */\n> +class BalancedMultiCycle : public SingleStream {\n> +public:\n> +\texplicit BalancedMultiCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> +\t\t: SingleStream(camera, role, numRequests) {}\n> \n> -Results testSingleStream(std::shared_ptr camera)\n> -{\n> -\tstatic const std::vector> roles = {\n> -\t\t{ \"raw\", Raw },\n> -\t\t{ \"still\", StillCapture },\n> -\t\t{ \"video\", VideoRecording },\n> -\t\t{ \"viewfinder\", Viewfinder },\n> +\tvoid TestBody() override {\n> +\t\tunsigned int numRepeats = 3;\n> +\n> +\t\tSimpleCaptureBalanced capture(camera_);\n> +\n> +\t\tcapture.configure(role_);\n> +\n> +\t\tfor (unsigned int starts = 0; starts < numRepeats; starts++)\n> +\t\t\tcapture.capture(numRequests_);\n> \t};\n> -\tstatic const std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> -\n> -\tResults results(numRequests.size() * roles.size() * 3);\n> -\n> -\tfor (const auto &role : roles) {\n> -\t\tstd::cout << \"= Test role \" << role.first << std::endl;\n> -\t\t/*\n> -\t\t * Test single capture cycles\n> -\t\t *\n> -\t\t * Makes sure the camera completes the exact number of requests queued.\n> -\t\t * Example failure is a camera that needs N+M requests queued to\n> -\t\t * complete N requests to the application.\n> -\t\t */\n> -\t\tstd::cout << \"* Test single capture cycles\" << std::endl;\n> -\t\tfor (unsigned int num : numRequests)\n> -\t\t\tresults.add(testRequestBalance(camera, role.second, 1, num));\n> -\n> -\t\t/*\n> -\t\t * Test multiple start/stop cycles\n> -\t\t *\n> -\t\t * Makes sure the camera supports multiple start/stop cycles.\n> -\t\t * Example failure is a camera that does not clean up correctly in its\n> -\t\t * error path but is only tested by single-capture applications.\n> -\t\t */\n> -\t\tstd::cout << \"* Test multiple start/stop cycles\" << std::endl;\n> -\t\tfor (unsigned int num : numRequests)\n> -\t\t\tresults.add(testRequestBalance(camera, role.second, 3, num));\n> -\n> -\t\t/*\n> -\t\t * Test unbalanced stop\n> -\t\t *\n> -\t\t * Makes sure the camera supports a stop with requests queued.\n> -\t\t * Example failure is a camera that does not handle cancelation\n> -\t\t * of buffers coming back from the video device while stopping.\n> -\t\t */\n> -\t\tstd::cout << \"* Test unbalanced stop\" << std::endl;\n> -\t\tfor (unsigned int num : numRequests)\n> -\t\t\tresults.add(testRequestUnbalance(camera, role.second, num));\n> -\t}\n> -\n> -\treturn results;\n> +};\n> +\n> +/*\n> + * Test unbalanced stop\n> + *\n> + * Makes sure the camera supports a stop with requests queued. Example failure\n> + * is a camera that does not handle cancelation of buffers coming back from the\n> + * video device while stopping.\n> + */\n> +class Unbalanced : public SingleStream {\n> +public:\n> +\texplicit Unbalanced(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> +\t\t: SingleStream(camera, role, numRequests) {}\n> +\n> +\tvoid TestBody() override {\n> +\t\tSimpleCaptureUnbalanced capture(camera_);\n> +\n> +\t\tcapture.configure(role_);\n> +\n> +\t\tcapture.capture(numRequests_);\n> +\t};\n> +};\n> +\n> +void registerSingleStreamTests(std::vector> cameras,\n> +\t\tstd::map roles, std::vector requests)\n> +{\n> +\tREGISTER_TESTS(SingleStream, BalancedSingleCycle, cameras, roles, requests);\n> +\tREGISTER_TESTS(SingleStream, BalancedMultiCycle, cameras, roles, requests);\n> +\tREGISTER_TESTS(SingleStream, Unbalanced, cameras, roles, requests);\n> }\n> diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> index 396605214e4b..6d5a8b88c287 100644\n> --- a/src/lc-compliance/tests.h\n> +++ b/src/lc-compliance/tests.h\n> @@ -11,6 +11,19 @@\n> \n> #include \"results.h\"\n> \n> -Results testSingleStream(std::shared_ptr camera);\n> +void registerSingleStreamTests(std::vector> cameras,\n> +\t\tstd::map roles, std::vector requests);\n> +\n> +#define REGISTER_TESTS(testsuite, testcase, cameras, roles, requests) \\\n> +for (auto c : cameras) { \\\n> +\tfor (auto [r, rn] : roles) { \\\n> +\t\tfor (auto rq : requests) { \\\n> +\t\t\ttesting::RegisterTest( \\\n> +\t\t\t\t#testsuite, (std::string(#testcase) + \"/\" + c->id() + \"/\" + rn + \"/\" + std::to_string(rq)).c_str(), \\\n> +\t\t\t\tnullptr, nullptr, __FILE__, __LINE__, \\\n> +\t\t\t\t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> +\t\t} \\\n> +\t} \\\n> +}\n> \n> #endif /* __LC_COMPLIANCE_TESTS_H__ */\n> -- \n> 2.31.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2C716BDE7F\n\tfor ;\n\tThu, 6 May 2021 08:48:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6DC606891D;\n\tThu, 6 May 2021 10:48:19 +0200 (CEST)","from mail-lf1-x135.google.com (mail-lf1-x135.google.com\n\t[IPv6:2a00:1450:4864:20::135])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3F90968919\n\tfor ;\n\tThu, 6 May 2021 10:48:18 +0200 (CEST)","by mail-lf1-x135.google.com with SMTP id x19so6699805lfa.2\n\tfor ;\n\tThu, 06 May 2021 01:48:18 -0700 (PDT)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tu22sm654498lja.5.2021.05.06.01.48.16\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tThu, 06 May 2021 01:48:16 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"Y0Vno19G\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=ex/wtNEpLnu7x2Fn7IZKnbgRZIt0ryF2Moi9bRzbEVg=;\n\tb=Y0Vno19Gaq8kaiQJsGxXYBW/8mVPa6zTRMnRr11JfnjCr1t2rnEEqHWe3U5KHXFXXq\n\tF9wGkcqOAOaymSPEzpYov4No+0R9NNVIjNe8LEZUGhY3kx9ddwK80FTKsXPXpMrNQryy\n\tV6jBDoPMLYk4GGkf++0Lc49xTTDr3RtijTxolRcoUq82SD4X8jk207HRSW9PEnMuFCD1\n\trvaZYPYbbGvcwaNar/agpVKNQ66pa9LHV1sUgacTNSjRQ2VdSD0LcDrWsFZGIPm+IIC+\n\tTialj709o/6H8X8isJyYnplS5CMb4juE1xjI2SK7BucWXDUXxnIAjUslobFIZxeqpwcX\n\t6SPQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=ex/wtNEpLnu7x2Fn7IZKnbgRZIt0ryF2Moi9bRzbEVg=;\n\tb=gggiJbhgkTy93rPaPGEp43qUtg7nkpvbQJjY2tX26qFiZ9650HSTEHOBzqfQ9tm02+\n\tvTfAUVmBUwXe+5xK/ZOAUNerrVGbX9NFkL4pUIlrfjKokrilJoG5oX7f4t5kzp2UKHXn\n\tlu5nnp393QdwkLoaBR7QL6SVNtlQ/e6L6DulC891Yvggbj3uMZPJN1jjxH6a5t+9ltSR\n\tXwrN+hmGkVFKN/kVd9rpvX9gTY7R9V2NBfI7PNuURNIhDMha/Mp2jWpsXn3Tnb+HM4Hv\n\tL+uw15HZ2NXUY46JQ7XWgyfIHNLJwtiyZpzPemRk87aP0xSPx58EXo8nt+5XQ3nAB7W8\n\tPhNw==","X-Gm-Message-State":"AOAM530XODapdMZRPo5fBKtvVdz8uf2R0aQAofzv+50z+7X7vRXiiqTs\n\tDnmRDD2V0r8JLKyYEb2UApdekBXImdc5RA==","X-Google-Smtp-Source":"ABdhPJzm2Z2Lne8jD6e9Gg2ST/JpAGLnc8coN+KRmmRtX+eHEUAVIeMj1qENDGTl6N75KN9IMBHVQg==","X-Received":"by 2002:a05:6512:1287:: with SMTP id\n\tu7mr2227619lfs.447.1620290897454; \n\tThu, 06 May 2021 01:48:17 -0700 (PDT)","Date":"Thu, 6 May 2021 10:48:16 +0200","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"=?iso-8859-1?q?N=EDcolas_F=2E_R=2E_A=2E?= Prado ","Message-ID":"","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20210503193307.108607-4-nfraprado@collabora.com>","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com,\n\t=?iso-8859-1?q?Andr=E9?= Almeida ","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16815,"web_url":"https://patchwork.libcamera.org/comment/16815/","msgid":"","date":"2021-05-06T20:20:29","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":84,"url":"https://patchwork.libcamera.org/api/people/84/","name":"Nícolas F. R. A. Prado","email":"nfraprado@collabora.com"},"content":"Hi Niklas,\n\nthanks for the feedback.\n\nEm 2021-05-06 05:48, Niklas Söderlund escreveu:\n\n> Hi Nícolas,\n>\n> Thanks for your work!\n>\n> I really like how gtest helps make the actual tests to be easier to\n> read.\n>\n> On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > Refactor lc-compliance using Googletest as the test framework.\n> > \n> > Signed-off-by: Nícolas F. R. A. Prado \n> > ---\n> > Changes in v2:\n> > - Changed from static to dynamic test registration\n> > - Removed -c flag\n> > \n> > There's still an issue with the refactoring that the shared_ptr of the tests\n> > apparently aren't being deleted on time, which causes:\n> > \n> > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > \tSegmentation fault (core dumped)\n> > \n> > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > that didn't work. In fact, when just listing the tests, the constructor for the\n> > tests isn't even called. So I think the issue has to do with the passing of the\n> > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > \n> > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > \n> > Not sure how to solve this. Any tip would be welcome.\n>\n> I will leave this for now as I have a different worry about\n> REGISTER_TESTS() design below.\n>\n> > \n> > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > src/lc-compliance/meson.build | 3 +\n> > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > src/lc-compliance/simple_capture.h | 8 +-\n> > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > src/lc-compliance/tests.h | 15 ++-\n> > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > \n> > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > index 54cee54aa978..add0d7729aec 100644\n> > --- a/src/lc-compliance/main.cpp\n> > +++ b/src/lc-compliance/main.cpp\n> > @@ -9,6 +9,8 @@\n> > #include \n> > #include \n> > \n> > +#include \n> > +\n> > #include \n> > \n> > #include \"../cam/options.h\"\n> > @@ -17,7 +19,6 @@\n> > using namespace libcamera;\n> > \n> > enum {\n> > -\tOptCamera = 'c',\n> > \tOptHelp = 'h',\n> > };\n> > \n> > @@ -28,14 +29,15 @@ public:\n> > \t~Harness();\n> > \n> > \tint exec();\n> > +\tint init();\n> > +\tvoid registerTests();\n> > \n> > private:\n> > -\tint init();\n> > \tvoid listCameras();\n> > \n> > \tOptionsParser::Options options_;\n> > \tstd::unique_ptr cm_;\n> > -\tstd::shared_ptr camera_;\n> > +\tstd::vector> cameras_;\n> > };\n> > \n> > Harness::Harness(const OptionsParser::Options &options)\n> > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > \n> > Harness::~Harness()\n> > {\n> > -\tif (camera_) {\n> > -\t\tcamera_->release();\n> > -\t\tcamera_.reset();\n> > +\tfor (auto &c : cameras_) {\n> > +\t\tc->release();\n> > +\t\tc.reset();\n> > \t}\n> > \n> > \tcm_->stop();\n> > }\n> > \n> > -int Harness::exec()\n> > -{\n> > -\tint ret = init();\n> > -\tif (ret)\n> > -\t\treturn ret;\n> > -\n> > -\tstd::vector results;\n> > -\n> > -\tresults.push_back(testSingleStream(camera_));\n> > -\n> > -\tfor (const Results &result : results) {\n> > -\t\tret = result.summary();\n> > -\t\tif (ret)\n> > -\t\t\treturn ret;\n> > -\t}\n> > -\n> > -\treturn 0;\n> > -}\n> > -\n> > int Harness::init()\n> > {\n> > \tint ret = cm_->start();\n> > @@ -82,42 +65,26 @@ int Harness::init()\n> > \t\treturn ret;\n> > \t}\n> > \n> > -\tif (!options_.isSet(OptCamera)) {\n> > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > -\t\tlistCameras();\n> > -\t\treturn -ENODEV;\n> > -\t}\n> > -\n> > -\tconst std::string &cameraId = options_[OptCamera];\n> > -\tcamera_ = cm_->get(cameraId);\n> > -\tif (!camera_) {\n> > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > -\t\tlistCameras();\n> > -\t\treturn -ENODEV;\n> > +\tfor (auto cam : cm_->cameras()) {\n> > +\t\tif (cam->acquire()) {\n> > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > +\t\t\treturn -EINVAL;\n> > +\t\t}\n> > +\t\tcameras_.push_back(cam);\n>\n> I don't like this, I think we need to retain the ability to select which\n> camera to test. And further down the line maybe allow more then one\n> camera to be selected if we want to test concurrent use of two specific\n> ones. But for the work in this series I think you can ignore multiple\n> cameras.\n\nActually, that doesn't mean that multiple cameras will be tested, that's\ndetermined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\nonly test that camera. But I think I see what you mean, I shouldn't be acquiring\nall cameras, only the ones that I will actually test (and in this code there's a\nbig issue where if a single camera is already busy, even if it won't be tested,\nlc-compliance as a whole fails).\n\n>\n> If I understand the coverletter correctly the correct thing here the cli\n> arguments should be feed to gtest directly to be able to filter on what\n> tests are run. Is there support in getst to also pass arguments to\n> tests? Maybe we could remove the need to parse arguments ourself\n> completely?\n\nThere isn't any CLI argument in gtest to pass arguments to tests, but I think we\ncould solve both problems (this and the one above) like this:\n\n- In the main(), query the id of each camera from libcamera.\n- Register each of the tests dynamically in gtest passing as an argument the\n camera id, rather than a camera pointer (which is the current behavior).\n- To select which tests to run, and on which camera(s), the user just uses\n --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n the camera).\n- In the startup of each test the camera is acquired for that test, and when\n it's done it is released.\n\nImplications:\n- We no longer need to parse arguments and can drop that code, as both camera\n and test are specified directly with gtest's filter.\n- We only acquire the cameras that we'll actually test and only while they're\n being tested.\n- As a downside, a single camera will probably be acquired multiple times during\n a single run as lc-compliance moves from one test to the next, but I don't\n think this would be a big issue. (Maybe we could do this at the test suite\n level instead to greatly decrease the number of times of acquire(), but I'm\n not sure if it's possible, since we instantiate, by passing the camera id, at\n the test level, I'll have to test that).\n\nWhat do you think?\n\nThanks,\nNícolas\n\n>\n> > \t}\n> > \n> > -\tif (camera_->acquire()) {\n> > -\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > -\t\treturn -EINVAL;\n> > -\t}\n> > -\n> > -\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> > -\n> > \treturn 0;\n> > }\n> > \n> > void Harness::listCameras()\n> > {\n> > -\tfor (const std::shared_ptr &cam : cm_->cameras())\n> > -\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> > +\tfor (const std::shared_ptr &c : cm_->cameras())\n> > +\t\tstd::cout << \"- \" << c.get()->id() << std::endl;\n> > }\n> > \n> > static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > {\n> > \tOptionsParser parser;\n> > -\tparser.addOption(OptCamera, OptionString,\n> > -\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> > -\t\t\t ArgumentRequired, \"camera\");\n> > \tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> > \t\t\t \"help\");\n> > \n> > @@ -133,6 +100,31 @@ static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > \treturn 0;\n> > }\n> > \n> > +/*\n> > + * Make asserts act like exceptions, otherwise they only fail (or skip) the\n> > + * current function. From gtest documentation:\n> > + * https://google.github.io/googletest/advanced.html#asserting-on-subroutines-with-an-exception\n> > + */\n> > +class ThrowListener : public testing::EmptyTestEventListener {\n> > +\tvoid OnTestPartResult(const testing::TestPartResult& result) override {\n> > +\t\tif (result.type() == testing::TestPartResult::kFatalFailure\n> > +\t\t || result.type() == testing::TestPartResult::kSkip) {\n> > +\t\t\tthrow testing::AssertionException(result);\n> > +\t\t}\n> > +\t}\n> > +};\n> > +\n> > +\n> > +void Harness::registerTests() {\n> > +\tstd::map roles = {{Raw, \"Raw\"},\n> > +\t\t\t\t\t\t {StillCapture, \"Still\"},\n> > +\t\t\t\t\t\t {VideoRecording, \"Video\"},\n> > +\t\t\t\t\t\t {Viewfinder, \"Viewfinder\"}};\n> > +\tstd::vector requests = {1, 2, 3, 5, 8, 13, 21, 34, 55, 89};\n> > +\n> > +\tregisterSingleStreamTests(cameras_, roles, requests);\n> > +}\n> > +\n> > int main(int argc, char **argv)\n> > {\n> > \tOptionsParser::Options options;\n> > @@ -143,6 +135,13 @@ int main(int argc, char **argv)\n> > \t\treturn EXIT_FAILURE;\n> > \n> > \tHarness harness(options);\n> > +\tret = harness.init();\n> > +\tif (ret)\n> > +\t\treturn ret;\n> > +\n> > +\tharness.registerTests();\n> > \n> > -\treturn harness.exec() ? EXIT_FAILURE : EXIT_SUCCESS;\n> > +\t::testing::InitGoogleTest(&argc, argv);\n> > +\ttesting::UnitTest::GetInstance()->listeners().Append(new ThrowListener);\n> > +\treturn RUN_ALL_TESTS();\n> > }\n> > diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> > index a2bfcceb1259..704bc18af3e1 100644\n> > --- a/src/lc-compliance/meson.build\n> > +++ b/src/lc-compliance/meson.build\n> > @@ -18,10 +18,13 @@ lc_compliance_sources = files([\n> > 'single_stream.cpp',\n> > ])\n> > \n> > +gtest_dep = dependency('gtest')\n> > +\n> > lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> > dependencies : [\n> > libatomic,\n> > libcamera_dep,\n> > libevent,\n> > + gtest_dep,\n> > ],\n> > install : true)\n> > diff --git a/src/lc-compliance/simple_capture.cpp b/src/lc-compliance/simple_capture.cpp\n> > index f90fe6d0f9aa..7731eb16f8c2 100644\n> > --- a/src/lc-compliance/simple_capture.cpp\n> > +++ b/src/lc-compliance/simple_capture.cpp\n> > @@ -5,6 +5,8 @@\n> > * simple_capture.cpp - Simple capture helper\n> > */\n> > \n> > +#include \n> > +\n> > #include \"simple_capture.h\"\n> > \n> > using namespace libcamera;\n> > @@ -20,38 +22,34 @@ SimpleCapture::~SimpleCapture()\n> > \tstop();\n> > }\n> > \n> > -Results::Result SimpleCapture::configure(StreamRole role)\n> > +void SimpleCapture::configure(StreamRole role)\n> > {\n> > \tconfig_ = camera_->generateConfiguration({ role });\n> > \n> > -\tif (!config_)\n> > -\t\treturn { Results::Skip, \"Role not supported by camera\" };\n> > +\tif (!config_) {\n> > +\t\tstd::cout << \"Role not supported by camera\" << std::endl;\n> > +\t\tGTEST_SKIP();\n> > +\t}\n> > \n> > \tif (config_->validate() != CameraConfiguration::Valid) {\n> > \t\tconfig_.reset();\n> > -\t\treturn { Results::Fail, \"Configuration not valid\" };\n> > +\t\tFAIL() << \"Configuration not valid\";\n> > \t}\n> > \n> > \tif (camera_->configure(config_.get())) {\n> > \t\tconfig_.reset();\n> > -\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> > +\t\tFAIL() << \"Failed to configure camera\";\n> > \t}\n> > -\n> > -\treturn { Results::Pass, \"Configure camera\" };\n> > }\n> > \n> > -Results::Result SimpleCapture::start()\n> > +void SimpleCapture::start()\n> > {\n> > \tStream *stream = config_->at(0).stream();\n> > -\tif (allocator_->allocate(stream) < 0)\n> > -\t\treturn { Results::Fail, \"Failed to allocate buffers\" };\n> > +\tASSERT_GE(allocator_->allocate(stream), 0) << \"Failed to allocate buffers\";\n> > \n> > -\tif (camera_->start())\n> > -\t\treturn { Results::Fail, \"Failed to start camera\" };\n> > +\tASSERT_TRUE(!camera_->start()) << \"Failed to start camera\";\n> > \n> > \tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> > -\n> > -\treturn { Results::Pass, \"Started camera\" };\n> > }\n> > \n> > void SimpleCapture::stop()\n> > @@ -77,22 +75,19 @@ SimpleCaptureBalanced::SimpleCaptureBalanced(std::shared_ptr camera)\n> > {\n> > }\n> > \n> > -Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > +void SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > {\n> > -\tResults::Result ret = start();\n> > -\tif (ret.first != Results::Pass)\n> > -\t\treturn ret;\n> > +\tstart();\n> > \n> > \tStream *stream = config_->at(0).stream();\n> > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > \n> > \t/* No point in testing less requests then the camera depth. */\n> > \tif (buffers.size() > numRequests) {\n> > -\t\t/* Cache buffers.size() before we destroy it in stop() */\n> > -\t\tint buffers_size = buffers.size();\n> > -\n> > -\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers_size)\n> > -\t\t\t+ \" requests, can't test only \" + std::to_string(numRequests) };\n> > +\t\tstd::cout << \"Camera needs \" + std::to_string(buffers.size())\n> > +\t\t\t+ \" requests, can't test only \"\n> > +\t\t\t+ std::to_string(numRequests) << std::endl;\n> > +\t\tGTEST_SKIP();\n> > \t}\n> > \n> > \tqueueCount_ = 0;\n> > @@ -103,14 +98,11 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > \tstd::vector> requests;\n> > \tfor (const std::unique_ptr &buffer : buffers) {\n> > \t\tstd::unique_ptr request = camera_->createRequest();\n> > -\t\tif (!request)\n> > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > \n> > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > \n> > -\t\tif (queueRequest(request.get()) < 0)\n> > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > +\t\tASSERT_GE(queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > \n> > \t\trequests.push_back(std::move(request));\n> > \t}\n> > @@ -121,12 +113,7 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > \tstop();\n> > \tdelete loop_;\n> > \n> > -\tif (captureCount_ != captureLimit_)\n> > -\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) +\n> > -\t\t\t\" request, wanted \" + std::to_string(captureLimit_) };\n> > -\n> > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > -\t\tstd::to_string(numRequests) + \" requests\" };\n> > +\tASSERT_EQ(captureCount_, captureLimit_);\n> > }\n> > \n> > int SimpleCaptureBalanced::queueRequest(Request *request)\n> > @@ -158,11 +145,9 @@ SimpleCaptureUnbalanced::SimpleCaptureUnbalanced(std::shared_ptr camera)\n> > {\n> > }\n> > \n> > -Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > +void SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > {\n> > -\tResults::Result ret = start();\n> > -\tif (ret.first != Results::Pass)\n> > -\t\treturn ret;\n> > +\tstart();\n> > \n> > \tStream *stream = config_->at(0).stream();\n> > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > @@ -174,14 +159,11 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > \tstd::vector> requests;\n> > \tfor (const std::unique_ptr &buffer : buffers) {\n> > \t\tstd::unique_ptr request = camera_->createRequest();\n> > -\t\tif (!request)\n> > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > \n> > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > \n> > -\t\tif (camera_->queueRequest(request.get()) < 0)\n> > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > +\t\tASSERT_GE(camera_->queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > \n> > \t\trequests.push_back(std::move(request));\n> > \t}\n> > @@ -192,7 +174,7 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > \tstop();\n> > \tdelete loop_;\n> > \n> > -\treturn { status ? Results::Fail : Results::Pass, \"Unbalanced capture of \" + std::to_string(numRequests) + \" requests\" };\n> > +\tASSERT_FALSE(status);\n> > }\n> > \n> > void SimpleCaptureUnbalanced::requestComplete(Request *request)\n> > diff --git a/src/lc-compliance/simple_capture.h b/src/lc-compliance/simple_capture.h\n> > index d9de53fb63a3..0f8465083456 100644\n> > --- a/src/lc-compliance/simple_capture.h\n> > +++ b/src/lc-compliance/simple_capture.h\n> > @@ -17,13 +17,13 @@\n> > class SimpleCapture\n> > {\n> > public:\n> > -\tResults::Result configure(libcamera::StreamRole role);\n> > +\tvoid configure(libcamera::StreamRole role);\n> > \n> > protected:\n> > \tSimpleCapture(std::shared_ptr camera);\n> > \tvirtual ~SimpleCapture();\n> > \n> > -\tResults::Result start();\n> > +\tvoid start();\n> > \tvoid stop();\n> > \n> > \tvirtual void requestComplete(libcamera::Request *request) = 0;\n> > @@ -40,7 +40,7 @@ class SimpleCaptureBalanced : public SimpleCapture\n> > public:\n> > \tSimpleCaptureBalanced(std::shared_ptr camera);\n> > \n> > -\tResults::Result capture(unsigned int numRequests);\n> > +\tvoid capture(unsigned int numRequests);\n> > \n> > private:\n> > \tint queueRequest(libcamera::Request *request);\n> > @@ -56,7 +56,7 @@ class SimpleCaptureUnbalanced : public SimpleCapture\n> > public:\n> > \tSimpleCaptureUnbalanced(std::shared_ptr camera);\n> > \n> > -\tResults::Result capture(unsigned int numRequests);\n> > +\tvoid capture(unsigned int numRequests);\n> > \n> > private:\n> > \tvoid requestComplete(libcamera::Request *request) override;\n> > diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> > index 8318b42f42d6..eb6a6f305826 100644\n> > --- a/src/lc-compliance/single_stream.cpp\n> > +++ b/src/lc-compliance/single_stream.cpp\n> > @@ -7,91 +7,94 @@\n> > \n> > #include \n> > \n> > +#include \n> > +\n> > #include \"simple_capture.h\"\n> > #include \"tests.h\"\n> > \n> > using namespace libcamera;\n> > \n> > -Results::Result testRequestBalance(std::shared_ptr camera,\n> > -\t\t\t\t StreamRole role, unsigned int startCycles,\n> > -\t\t\t\t unsigned int numRequests)\n> > -{\n> > -\tSimpleCaptureBalanced capture(camera);\n> > +class SingleStream : public testing::Test {\n> > +public:\n> > +\texplicit SingleStream(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > +\t\t: camera_(camera), role_(role), numRequests_(numRequests) {}\n> > \n> > -\tResults::Result ret = capture.configure(role);\n> > -\tif (ret.first != Results::Pass)\n> > -\t\treturn ret;\n> > +protected:\n> > +\tstd::shared_ptr camera_;\n> > +\tStreamRole role_;\n> > +\tunsigned int numRequests_;\n> > +};\n> > \n> > -\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> > -\t\tret = capture.capture(numRequests);\n> > -\t\tif (ret.first != Results::Pass)\n> > -\t\t\treturn ret;\n> > -\t}\n> > +/*\n> > + * Test single capture cycles\n> > + *\n> > + * Makes sure the camera completes the exact number of requests queued. Example\n> > + * failure is a camera that needs N+M requests queued to complete N requests to\n> > + * the application.\n> > + */\n> > +class BalancedSingleCycle : public SingleStream {\n> > +public:\n> > +\texplicit BalancedSingleCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > +\t\t: SingleStream(camera, role, numRequests) {}\n> > \n> > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > -\t\tstd::to_string(numRequests) + \" requests with \" +\n> > -\t\tstd::to_string(startCycles) + \" start cycles\" };\n> > -}\n> > +\tvoid TestBody() override {\n> > +\t\tSimpleCaptureBalanced capture(camera_);\n> > \n> > -Results::Result testRequestUnbalance(std::shared_ptr camera,\n> > -\t\t\t\t StreamRole role, unsigned int numRequests)\n> > -{\n> > -\tSimpleCaptureUnbalanced capture(camera);\n> > +\t\tcapture.configure(role_);\n> > \n> > -\tResults::Result ret = capture.configure(role);\n> > -\tif (ret.first != Results::Pass)\n> > -\t\treturn ret;\n> > +\t\tcapture.capture(numRequests_);\n> > +\t};\n> > +};\n> > \n> > -\treturn capture.capture(numRequests);\n> > -}\n> > +/*\n> > + * Test multiple start/stop cycles\n> > + *\n> > + * Makes sure the camera supports multiple start/stop cycles. Example failure is\n> > + * a camera that does not clean up correctly in its error path but is only\n> > + * tested by single-capture applications.\n> > + */\n> > +class BalancedMultiCycle : public SingleStream {\n> > +public:\n> > +\texplicit BalancedMultiCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > +\t\t: SingleStream(camera, role, numRequests) {}\n> > \n> > -Results testSingleStream(std::shared_ptr camera)\n> > -{\n> > -\tstatic const std::vector> roles = {\n> > -\t\t{ \"raw\", Raw },\n> > -\t\t{ \"still\", StillCapture },\n> > -\t\t{ \"video\", VideoRecording },\n> > -\t\t{ \"viewfinder\", Viewfinder },\n> > +\tvoid TestBody() override {\n> > +\t\tunsigned int numRepeats = 3;\n> > +\n> > +\t\tSimpleCaptureBalanced capture(camera_);\n> > +\n> > +\t\tcapture.configure(role_);\n> > +\n> > +\t\tfor (unsigned int starts = 0; starts < numRepeats; starts++)\n> > +\t\t\tcapture.capture(numRequests_);\n> > \t};\n> > -\tstatic const std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> > -\n> > -\tResults results(numRequests.size() * roles.size() * 3);\n> > -\n> > -\tfor (const auto &role : roles) {\n> > -\t\tstd::cout << \"= Test role \" << role.first << std::endl;\n> > -\t\t/*\n> > -\t\t * Test single capture cycles\n> > -\t\t *\n> > -\t\t * Makes sure the camera completes the exact number of requests queued.\n> > -\t\t * Example failure is a camera that needs N+M requests queued to\n> > -\t\t * complete N requests to the application.\n> > -\t\t */\n> > -\t\tstd::cout << \"* Test single capture cycles\" << std::endl;\n> > -\t\tfor (unsigned int num : numRequests)\n> > -\t\t\tresults.add(testRequestBalance(camera, role.second, 1, num));\n> > -\n> > -\t\t/*\n> > -\t\t * Test multiple start/stop cycles\n> > -\t\t *\n> > -\t\t * Makes sure the camera supports multiple start/stop cycles.\n> > -\t\t * Example failure is a camera that does not clean up correctly in its\n> > -\t\t * error path but is only tested by single-capture applications.\n> > -\t\t */\n> > -\t\tstd::cout << \"* Test multiple start/stop cycles\" << std::endl;\n> > -\t\tfor (unsigned int num : numRequests)\n> > -\t\t\tresults.add(testRequestBalance(camera, role.second, 3, num));\n> > -\n> > -\t\t/*\n> > -\t\t * Test unbalanced stop\n> > -\t\t *\n> > -\t\t * Makes sure the camera supports a stop with requests queued.\n> > -\t\t * Example failure is a camera that does not handle cancelation\n> > -\t\t * of buffers coming back from the video device while stopping.\n> > -\t\t */\n> > -\t\tstd::cout << \"* Test unbalanced stop\" << std::endl;\n> > -\t\tfor (unsigned int num : numRequests)\n> > -\t\t\tresults.add(testRequestUnbalance(camera, role.second, num));\n> > -\t}\n> > -\n> > -\treturn results;\n> > +};\n> > +\n> > +/*\n> > + * Test unbalanced stop\n> > + *\n> > + * Makes sure the camera supports a stop with requests queued. Example failure\n> > + * is a camera that does not handle cancelation of buffers coming back from the\n> > + * video device while stopping.\n> > + */\n> > +class Unbalanced : public SingleStream {\n> > +public:\n> > +\texplicit Unbalanced(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > +\t\t: SingleStream(camera, role, numRequests) {}\n> > +\n> > +\tvoid TestBody() override {\n> > +\t\tSimpleCaptureUnbalanced capture(camera_);\n> > +\n> > +\t\tcapture.configure(role_);\n> > +\n> > +\t\tcapture.capture(numRequests_);\n> > +\t};\n> > +};\n> > +\n> > +void registerSingleStreamTests(std::vector> cameras,\n> > +\t\tstd::map roles, std::vector requests)\n> > +{\n> > +\tREGISTER_TESTS(SingleStream, BalancedSingleCycle, cameras, roles, requests);\n> > +\tREGISTER_TESTS(SingleStream, BalancedMultiCycle, cameras, roles, requests);\n> > +\tREGISTER_TESTS(SingleStream, Unbalanced, cameras, roles, requests);\n> > }\n> > diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> > index 396605214e4b..6d5a8b88c287 100644\n> > --- a/src/lc-compliance/tests.h\n> > +++ b/src/lc-compliance/tests.h\n> > @@ -11,6 +11,19 @@\n> > \n> > #include \"results.h\"\n> > \n> > -Results testSingleStream(std::shared_ptr camera);\n> > +void registerSingleStreamTests(std::vector> cameras,\n> > +\t\tstd::map roles, std::vector requests);\n> > +\n> > +#define REGISTER_TESTS(testsuite, testcase, cameras, roles, requests) \\\n> > +for (auto c : cameras) { \\\n> > +\tfor (auto [r, rn] : roles) { \\\n> > +\t\tfor (auto rq : requests) { \\\n> > +\t\t\ttesting::RegisterTest( \\\n> > +\t\t\t\t#testsuite, (std::string(#testcase) + \"/\" + c->id() + \"/\" + rn + \"/\" + std::to_string(rq)).c_str(), \\\n> > +\t\t\t\tnullptr, nullptr, __FILE__, __LINE__, \\\n> > +\t\t\t\t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > +\t\t} \\\n> > +\t} \\\n> > +}\n> > \n> > #endif /* __LC_COMPLIANCE_TESTS_H__ */\n> > -- \n> > 2.31.1\n> > \n>\n> --\n> Regards,\n> Niklas Söderlund\n>\n> --\n> To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id EB7CFBF829\n\tfor ;\n\tThu, 6 May 2021 20:21:07 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 46FB568918;\n\tThu, 6 May 2021 22:21:07 +0200 (CEST)","from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C2E7068901\n\tfor ;\n\tThu, 6 May 2021 22:21:05 +0200 (CEST)","from localhost (unknown\n\t[IPv6:2804:14c:1a9:2978:995d:672b:100f:2fd9])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id 8A4E11F43BF1;\n\tThu, 6 May 2021 21:21:03 +0100 (BST)"],"Mime-Version":"1.0","Date":"Thu, 06 May 2021 17:20:29 -0300","Message-Id":"","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","From":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= ","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com, =?utf-8?q?A?=\n\t=?utf-8?q?ndr=C3=A9_Almeida?= ","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16824,"web_url":"https://patchwork.libcamera.org/comment/16824/","msgid":"","date":"2021-05-07T09:11:53","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Nícolas,\n\nOn 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> Hi Niklas,\n> \n> thanks for the feedback.\n> \n> Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> \n> > Hi Nícolas,\n> >\n> > Thanks for your work!\n> >\n> > I really like how gtest helps make the actual tests to be easier to\n> > read.\n> >\n> > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > Refactor lc-compliance using Googletest as the test framework.\n> > > \n> > > Signed-off-by: Nícolas F. R. A. Prado \n> > > ---\n> > > Changes in v2:\n> > > - Changed from static to dynamic test registration\n> > > - Removed -c flag\n> > > \n> > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > apparently aren't being deleted on time, which causes:\n> > > \n> > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > \tSegmentation fault (core dumped)\n> > > \n> > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > \n> > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > \n> > > Not sure how to solve this. Any tip would be welcome.\n> >\n> > I will leave this for now as I have a different worry about\n> > REGISTER_TESTS() design below.\n> >\n> > > \n> > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > src/lc-compliance/meson.build | 3 +\n> > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > src/lc-compliance/simple_capture.h | 8 +-\n> > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > src/lc-compliance/tests.h | 15 ++-\n> > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > \n> > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > index 54cee54aa978..add0d7729aec 100644\n> > > --- a/src/lc-compliance/main.cpp\n> > > +++ b/src/lc-compliance/main.cpp\n> > > @@ -9,6 +9,8 @@\n> > > #include \n> > > #include \n> > > \n> > > +#include \n> > > +\n> > > #include \n> > > \n> > > #include \"../cam/options.h\"\n> > > @@ -17,7 +19,6 @@\n> > > using namespace libcamera;\n> > > \n> > > enum {\n> > > -\tOptCamera = 'c',\n> > > \tOptHelp = 'h',\n> > > };\n> > > \n> > > @@ -28,14 +29,15 @@ public:\n> > > \t~Harness();\n> > > \n> > > \tint exec();\n> > > +\tint init();\n> > > +\tvoid registerTests();\n> > > \n> > > private:\n> > > -\tint init();\n> > > \tvoid listCameras();\n> > > \n> > > \tOptionsParser::Options options_;\n> > > \tstd::unique_ptr cm_;\n> > > -\tstd::shared_ptr camera_;\n> > > +\tstd::vector> cameras_;\n> > > };\n> > > \n> > > Harness::Harness(const OptionsParser::Options &options)\n> > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > \n> > > Harness::~Harness()\n> > > {\n> > > -\tif (camera_) {\n> > > -\t\tcamera_->release();\n> > > -\t\tcamera_.reset();\n> > > +\tfor (auto &c : cameras_) {\n> > > +\t\tc->release();\n> > > +\t\tc.reset();\n> > > \t}\n> > > \n> > > \tcm_->stop();\n> > > }\n> > > \n> > > -int Harness::exec()\n> > > -{\n> > > -\tint ret = init();\n> > > -\tif (ret)\n> > > -\t\treturn ret;\n> > > -\n> > > -\tstd::vector results;\n> > > -\n> > > -\tresults.push_back(testSingleStream(camera_));\n> > > -\n> > > -\tfor (const Results &result : results) {\n> > > -\t\tret = result.summary();\n> > > -\t\tif (ret)\n> > > -\t\t\treturn ret;\n> > > -\t}\n> > > -\n> > > -\treturn 0;\n> > > -}\n> > > -\n> > > int Harness::init()\n> > > {\n> > > \tint ret = cm_->start();\n> > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > \t\treturn ret;\n> > > \t}\n> > > \n> > > -\tif (!options_.isSet(OptCamera)) {\n> > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > -\t\tlistCameras();\n> > > -\t\treturn -ENODEV;\n> > > -\t}\n> > > -\n> > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > -\tcamera_ = cm_->get(cameraId);\n> > > -\tif (!camera_) {\n> > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > -\t\tlistCameras();\n> > > -\t\treturn -ENODEV;\n> > > +\tfor (auto cam : cm_->cameras()) {\n> > > +\t\tif (cam->acquire()) {\n> > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > +\t\t\treturn -EINVAL;\n> > > +\t\t}\n> > > +\t\tcameras_.push_back(cam);\n> >\n> > I don't like this, I think we need to retain the ability to select which\n> > camera to test. And further down the line maybe allow more then one\n> > camera to be selected if we want to test concurrent use of two specific\n> > ones. But for the work in this series I think you can ignore multiple\n> > cameras.\n> \n> Actually, that doesn't mean that multiple cameras will be tested, that's\n> determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> all cameras, only the ones that I will actually test (and in this code there's a\n> big issue where if a single camera is already busy, even if it won't be tested,\n> lc-compliance as a whole fails).\n\nAhh I understand.\n\n> \n> >\n> > If I understand the coverletter correctly the correct thing here the cli\n> > arguments should be feed to gtest directly to be able to filter on what\n> > tests are run. Is there support in getst to also pass arguments to\n> > tests? Maybe we could remove the need to parse arguments ourself\n> > completely?\n> \n> There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> could solve both problems (this and the one above) like this:\n> \n> - In the main(), query the id of each camera from libcamera.\n> - Register each of the tests dynamically in gtest passing as an argument the\n> camera id, rather than a camera pointer (which is the current behavior).\n> - To select which tests to run, and on which camera(s), the user just uses\n> --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> the camera).\n> - In the startup of each test the camera is acquired for that test, and when\n> it's done it is released.\n> \n> Implications:\n> - We no longer need to parse arguments and can drop that code, as both camera\n> and test are specified directly with gtest's filter.\n> - We only acquire the cameras that we'll actually test and only while they're\n> being tested.\n> - As a downside, a single camera will probably be acquired multiple times during\n> a single run as lc-compliance moves from one test to the next, but I don't\n> think this would be a big issue. (Maybe we could do this at the test suite\n> level instead to greatly decrease the number of times of acquire(), but I'm\n> not sure if it's possible, since we instantiate, by passing the camera id, at\n> the test level, I'll have to test that).\n> \n> What do you think?\n\nI understand what you are trying to do but I see two issues.\n\n1. It depends on the implicit knowledge of users how to build the filter \n string and is rather unintuitive compared to having two distinct \n arguments, one to select camera and one to express the test filter.\n\n This could possibly be worked around by keeping the cam options code \n and building the gtest filter string internally by injecitng the \n selected camera name in the correct location, but the we have the \n next issue,\n\n2. I don't think it will be practically possible to express a test \n filter that selects a specific combination of two cameras.\n\nI do think we need to be able to parameterize test cases. If gtest does \nnot allow to do so using function arguments there are other ways of \ndoing so.\n\nI'm sure there are many solutions to this problem and maybe someone more \nfamiliar with gtest knows of a neat way to piggyback on its \ninfrastrcuture? Since I know very little about the gtest framework the \nfollowing is just an idea and I'm happy to yield.\n\nHow about using a singleton class that carries arguments constructed \nfrom cli arguments (camera id) before the gtest are executed. Each test \ncase could then consult this singleton if it needs to access parameters. \nI think you could take inspiration from the libcamera CameraManager \nclass.\n\nThis could (at lest for now) imply keeping the cam option parser. If so \nI think it would be nice if somewhen before we integrate this work that \nthe gtest filter string could be supplied as an argument using that \nparser. Of course if we can do it the other way around and add arguments \nto the gtest parser that would be even cooler as it would be less code \nfor us to maintain ;-)\n\n> \n> Thanks,\n> Nícolas\n> \n> >\n> > > \t}\n> > > \n> > > -\tif (camera_->acquire()) {\n> > > -\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > -\t\treturn -EINVAL;\n> > > -\t}\n> > > -\n> > > -\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> > > -\n> > > \treturn 0;\n> > > }\n> > > \n> > > void Harness::listCameras()\n> > > {\n> > > -\tfor (const std::shared_ptr &cam : cm_->cameras())\n> > > -\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> > > +\tfor (const std::shared_ptr &c : cm_->cameras())\n> > > +\t\tstd::cout << \"- \" << c.get()->id() << std::endl;\n> > > }\n> > > \n> > > static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > > {\n> > > \tOptionsParser parser;\n> > > -\tparser.addOption(OptCamera, OptionString,\n> > > -\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> > > -\t\t\t ArgumentRequired, \"camera\");\n> > > \tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> > > \t\t\t \"help\");\n> > > \n> > > @@ -133,6 +100,31 @@ static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > > \treturn 0;\n> > > }\n> > > \n> > > +/*\n> > > + * Make asserts act like exceptions, otherwise they only fail (or skip) the\n> > > + * current function. From gtest documentation:\n> > > + * https://google.github.io/googletest/advanced.html#asserting-on-subroutines-with-an-exception\n> > > + */\n> > > +class ThrowListener : public testing::EmptyTestEventListener {\n> > > +\tvoid OnTestPartResult(const testing::TestPartResult& result) override {\n> > > +\t\tif (result.type() == testing::TestPartResult::kFatalFailure\n> > > +\t\t || result.type() == testing::TestPartResult::kSkip) {\n> > > +\t\t\tthrow testing::AssertionException(result);\n> > > +\t\t}\n> > > +\t}\n> > > +};\n> > > +\n> > > +\n> > > +void Harness::registerTests() {\n> > > +\tstd::map roles = {{Raw, \"Raw\"},\n> > > +\t\t\t\t\t\t {StillCapture, \"Still\"},\n> > > +\t\t\t\t\t\t {VideoRecording, \"Video\"},\n> > > +\t\t\t\t\t\t {Viewfinder, \"Viewfinder\"}};\n> > > +\tstd::vector requests = {1, 2, 3, 5, 8, 13, 21, 34, 55, 89};\n> > > +\n> > > +\tregisterSingleStreamTests(cameras_, roles, requests);\n> > > +}\n> > > +\n> > > int main(int argc, char **argv)\n> > > {\n> > > \tOptionsParser::Options options;\n> > > @@ -143,6 +135,13 @@ int main(int argc, char **argv)\n> > > \t\treturn EXIT_FAILURE;\n> > > \n> > > \tHarness harness(options);\n> > > +\tret = harness.init();\n> > > +\tif (ret)\n> > > +\t\treturn ret;\n> > > +\n> > > +\tharness.registerTests();\n> > > \n> > > -\treturn harness.exec() ? EXIT_FAILURE : EXIT_SUCCESS;\n> > > +\t::testing::InitGoogleTest(&argc, argv);\n> > > +\ttesting::UnitTest::GetInstance()->listeners().Append(new ThrowListener);\n> > > +\treturn RUN_ALL_TESTS();\n> > > }\n> > > diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> > > index a2bfcceb1259..704bc18af3e1 100644\n> > > --- a/src/lc-compliance/meson.build\n> > > +++ b/src/lc-compliance/meson.build\n> > > @@ -18,10 +18,13 @@ lc_compliance_sources = files([\n> > > 'single_stream.cpp',\n> > > ])\n> > > \n> > > +gtest_dep = dependency('gtest')\n> > > +\n> > > lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> > > dependencies : [\n> > > libatomic,\n> > > libcamera_dep,\n> > > libevent,\n> > > + gtest_dep,\n> > > ],\n> > > install : true)\n> > > diff --git a/src/lc-compliance/simple_capture.cpp b/src/lc-compliance/simple_capture.cpp\n> > > index f90fe6d0f9aa..7731eb16f8c2 100644\n> > > --- a/src/lc-compliance/simple_capture.cpp\n> > > +++ b/src/lc-compliance/simple_capture.cpp\n> > > @@ -5,6 +5,8 @@\n> > > * simple_capture.cpp - Simple capture helper\n> > > */\n> > > \n> > > +#include \n> > > +\n> > > #include \"simple_capture.h\"\n> > > \n> > > using namespace libcamera;\n> > > @@ -20,38 +22,34 @@ SimpleCapture::~SimpleCapture()\n> > > \tstop();\n> > > }\n> > > \n> > > -Results::Result SimpleCapture::configure(StreamRole role)\n> > > +void SimpleCapture::configure(StreamRole role)\n> > > {\n> > > \tconfig_ = camera_->generateConfiguration({ role });\n> > > \n> > > -\tif (!config_)\n> > > -\t\treturn { Results::Skip, \"Role not supported by camera\" };\n> > > +\tif (!config_) {\n> > > +\t\tstd::cout << \"Role not supported by camera\" << std::endl;\n> > > +\t\tGTEST_SKIP();\n> > > +\t}\n> > > \n> > > \tif (config_->validate() != CameraConfiguration::Valid) {\n> > > \t\tconfig_.reset();\n> > > -\t\treturn { Results::Fail, \"Configuration not valid\" };\n> > > +\t\tFAIL() << \"Configuration not valid\";\n> > > \t}\n> > > \n> > > \tif (camera_->configure(config_.get())) {\n> > > \t\tconfig_.reset();\n> > > -\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> > > +\t\tFAIL() << \"Failed to configure camera\";\n> > > \t}\n> > > -\n> > > -\treturn { Results::Pass, \"Configure camera\" };\n> > > }\n> > > \n> > > -Results::Result SimpleCapture::start()\n> > > +void SimpleCapture::start()\n> > > {\n> > > \tStream *stream = config_->at(0).stream();\n> > > -\tif (allocator_->allocate(stream) < 0)\n> > > -\t\treturn { Results::Fail, \"Failed to allocate buffers\" };\n> > > +\tASSERT_GE(allocator_->allocate(stream), 0) << \"Failed to allocate buffers\";\n> > > \n> > > -\tif (camera_->start())\n> > > -\t\treturn { Results::Fail, \"Failed to start camera\" };\n> > > +\tASSERT_TRUE(!camera_->start()) << \"Failed to start camera\";\n> > > \n> > > \tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> > > -\n> > > -\treturn { Results::Pass, \"Started camera\" };\n> > > }\n> > > \n> > > void SimpleCapture::stop()\n> > > @@ -77,22 +75,19 @@ SimpleCaptureBalanced::SimpleCaptureBalanced(std::shared_ptr camera)\n> > > {\n> > > }\n> > > \n> > > -Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > +void SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > {\n> > > -\tResults::Result ret = start();\n> > > -\tif (ret.first != Results::Pass)\n> > > -\t\treturn ret;\n> > > +\tstart();\n> > > \n> > > \tStream *stream = config_->at(0).stream();\n> > > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > > \n> > > \t/* No point in testing less requests then the camera depth. */\n> > > \tif (buffers.size() > numRequests) {\n> > > -\t\t/* Cache buffers.size() before we destroy it in stop() */\n> > > -\t\tint buffers_size = buffers.size();\n> > > -\n> > > -\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers_size)\n> > > -\t\t\t+ \" requests, can't test only \" + std::to_string(numRequests) };\n> > > +\t\tstd::cout << \"Camera needs \" + std::to_string(buffers.size())\n> > > +\t\t\t+ \" requests, can't test only \"\n> > > +\t\t\t+ std::to_string(numRequests) << std::endl;\n> > > +\t\tGTEST_SKIP();\n> > > \t}\n> > > \n> > > \tqueueCount_ = 0;\n> > > @@ -103,14 +98,11 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > \tstd::vector> requests;\n> > > \tfor (const std::unique_ptr &buffer : buffers) {\n> > > \t\tstd::unique_ptr request = camera_->createRequest();\n> > > -\t\tif (!request)\n> > > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > > \n> > > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > > \n> > > -\t\tif (queueRequest(request.get()) < 0)\n> > > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > > +\t\tASSERT_GE(queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > > \n> > > \t\trequests.push_back(std::move(request));\n> > > \t}\n> > > @@ -121,12 +113,7 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > \tstop();\n> > > \tdelete loop_;\n> > > \n> > > -\tif (captureCount_ != captureLimit_)\n> > > -\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) +\n> > > -\t\t\t\" request, wanted \" + std::to_string(captureLimit_) };\n> > > -\n> > > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > > -\t\tstd::to_string(numRequests) + \" requests\" };\n> > > +\tASSERT_EQ(captureCount_, captureLimit_);\n> > > }\n> > > \n> > > int SimpleCaptureBalanced::queueRequest(Request *request)\n> > > @@ -158,11 +145,9 @@ SimpleCaptureUnbalanced::SimpleCaptureUnbalanced(std::shared_ptr camera)\n> > > {\n> > > }\n> > > \n> > > -Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > +void SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > {\n> > > -\tResults::Result ret = start();\n> > > -\tif (ret.first != Results::Pass)\n> > > -\t\treturn ret;\n> > > +\tstart();\n> > > \n> > > \tStream *stream = config_->at(0).stream();\n> > > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > > @@ -174,14 +159,11 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > \tstd::vector> requests;\n> > > \tfor (const std::unique_ptr &buffer : buffers) {\n> > > \t\tstd::unique_ptr request = camera_->createRequest();\n> > > -\t\tif (!request)\n> > > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > > \n> > > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > > \n> > > -\t\tif (camera_->queueRequest(request.get()) < 0)\n> > > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > > +\t\tASSERT_GE(camera_->queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > > \n> > > \t\trequests.push_back(std::move(request));\n> > > \t}\n> > > @@ -192,7 +174,7 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > \tstop();\n> > > \tdelete loop_;\n> > > \n> > > -\treturn { status ? Results::Fail : Results::Pass, \"Unbalanced capture of \" + std::to_string(numRequests) + \" requests\" };\n> > > +\tASSERT_FALSE(status);\n> > > }\n> > > \n> > > void SimpleCaptureUnbalanced::requestComplete(Request *request)\n> > > diff --git a/src/lc-compliance/simple_capture.h b/src/lc-compliance/simple_capture.h\n> > > index d9de53fb63a3..0f8465083456 100644\n> > > --- a/src/lc-compliance/simple_capture.h\n> > > +++ b/src/lc-compliance/simple_capture.h\n> > > @@ -17,13 +17,13 @@\n> > > class SimpleCapture\n> > > {\n> > > public:\n> > > -\tResults::Result configure(libcamera::StreamRole role);\n> > > +\tvoid configure(libcamera::StreamRole role);\n> > > \n> > > protected:\n> > > \tSimpleCapture(std::shared_ptr camera);\n> > > \tvirtual ~SimpleCapture();\n> > > \n> > > -\tResults::Result start();\n> > > +\tvoid start();\n> > > \tvoid stop();\n> > > \n> > > \tvirtual void requestComplete(libcamera::Request *request) = 0;\n> > > @@ -40,7 +40,7 @@ class SimpleCaptureBalanced : public SimpleCapture\n> > > public:\n> > > \tSimpleCaptureBalanced(std::shared_ptr camera);\n> > > \n> > > -\tResults::Result capture(unsigned int numRequests);\n> > > +\tvoid capture(unsigned int numRequests);\n> > > \n> > > private:\n> > > \tint queueRequest(libcamera::Request *request);\n> > > @@ -56,7 +56,7 @@ class SimpleCaptureUnbalanced : public SimpleCapture\n> > > public:\n> > > \tSimpleCaptureUnbalanced(std::shared_ptr camera);\n> > > \n> > > -\tResults::Result capture(unsigned int numRequests);\n> > > +\tvoid capture(unsigned int numRequests);\n> > > \n> > > private:\n> > > \tvoid requestComplete(libcamera::Request *request) override;\n> > > diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> > > index 8318b42f42d6..eb6a6f305826 100644\n> > > --- a/src/lc-compliance/single_stream.cpp\n> > > +++ b/src/lc-compliance/single_stream.cpp\n> > > @@ -7,91 +7,94 @@\n> > > \n> > > #include \n> > > \n> > > +#include \n> > > +\n> > > #include \"simple_capture.h\"\n> > > #include \"tests.h\"\n> > > \n> > > using namespace libcamera;\n> > > \n> > > -Results::Result testRequestBalance(std::shared_ptr camera,\n> > > -\t\t\t\t StreamRole role, unsigned int startCycles,\n> > > -\t\t\t\t unsigned int numRequests)\n> > > -{\n> > > -\tSimpleCaptureBalanced capture(camera);\n> > > +class SingleStream : public testing::Test {\n> > > +public:\n> > > +\texplicit SingleStream(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > +\t\t: camera_(camera), role_(role), numRequests_(numRequests) {}\n> > > \n> > > -\tResults::Result ret = capture.configure(role);\n> > > -\tif (ret.first != Results::Pass)\n> > > -\t\treturn ret;\n> > > +protected:\n> > > +\tstd::shared_ptr camera_;\n> > > +\tStreamRole role_;\n> > > +\tunsigned int numRequests_;\n> > > +};\n> > > \n> > > -\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> > > -\t\tret = capture.capture(numRequests);\n> > > -\t\tif (ret.first != Results::Pass)\n> > > -\t\t\treturn ret;\n> > > -\t}\n> > > +/*\n> > > + * Test single capture cycles\n> > > + *\n> > > + * Makes sure the camera completes the exact number of requests queued. Example\n> > > + * failure is a camera that needs N+M requests queued to complete N requests to\n> > > + * the application.\n> > > + */\n> > > +class BalancedSingleCycle : public SingleStream {\n> > > +public:\n> > > +\texplicit BalancedSingleCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > \n> > > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > > -\t\tstd::to_string(numRequests) + \" requests with \" +\n> > > -\t\tstd::to_string(startCycles) + \" start cycles\" };\n> > > -}\n> > > +\tvoid TestBody() override {\n> > > +\t\tSimpleCaptureBalanced capture(camera_);\n> > > \n> > > -Results::Result testRequestUnbalance(std::shared_ptr camera,\n> > > -\t\t\t\t StreamRole role, unsigned int numRequests)\n> > > -{\n> > > -\tSimpleCaptureUnbalanced capture(camera);\n> > > +\t\tcapture.configure(role_);\n> > > \n> > > -\tResults::Result ret = capture.configure(role);\n> > > -\tif (ret.first != Results::Pass)\n> > > -\t\treturn ret;\n> > > +\t\tcapture.capture(numRequests_);\n> > > +\t};\n> > > +};\n> > > \n> > > -\treturn capture.capture(numRequests);\n> > > -}\n> > > +/*\n> > > + * Test multiple start/stop cycles\n> > > + *\n> > > + * Makes sure the camera supports multiple start/stop cycles. Example failure is\n> > > + * a camera that does not clean up correctly in its error path but is only\n> > > + * tested by single-capture applications.\n> > > + */\n> > > +class BalancedMultiCycle : public SingleStream {\n> > > +public:\n> > > +\texplicit BalancedMultiCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > \n> > > -Results testSingleStream(std::shared_ptr camera)\n> > > -{\n> > > -\tstatic const std::vector> roles = {\n> > > -\t\t{ \"raw\", Raw },\n> > > -\t\t{ \"still\", StillCapture },\n> > > -\t\t{ \"video\", VideoRecording },\n> > > -\t\t{ \"viewfinder\", Viewfinder },\n> > > +\tvoid TestBody() override {\n> > > +\t\tunsigned int numRepeats = 3;\n> > > +\n> > > +\t\tSimpleCaptureBalanced capture(camera_);\n> > > +\n> > > +\t\tcapture.configure(role_);\n> > > +\n> > > +\t\tfor (unsigned int starts = 0; starts < numRepeats; starts++)\n> > > +\t\t\tcapture.capture(numRequests_);\n> > > \t};\n> > > -\tstatic const std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> > > -\n> > > -\tResults results(numRequests.size() * roles.size() * 3);\n> > > -\n> > > -\tfor (const auto &role : roles) {\n> > > -\t\tstd::cout << \"= Test role \" << role.first << std::endl;\n> > > -\t\t/*\n> > > -\t\t * Test single capture cycles\n> > > -\t\t *\n> > > -\t\t * Makes sure the camera completes the exact number of requests queued.\n> > > -\t\t * Example failure is a camera that needs N+M requests queued to\n> > > -\t\t * complete N requests to the application.\n> > > -\t\t */\n> > > -\t\tstd::cout << \"* Test single capture cycles\" << std::endl;\n> > > -\t\tfor (unsigned int num : numRequests)\n> > > -\t\t\tresults.add(testRequestBalance(camera, role.second, 1, num));\n> > > -\n> > > -\t\t/*\n> > > -\t\t * Test multiple start/stop cycles\n> > > -\t\t *\n> > > -\t\t * Makes sure the camera supports multiple start/stop cycles.\n> > > -\t\t * Example failure is a camera that does not clean up correctly in its\n> > > -\t\t * error path but is only tested by single-capture applications.\n> > > -\t\t */\n> > > -\t\tstd::cout << \"* Test multiple start/stop cycles\" << std::endl;\n> > > -\t\tfor (unsigned int num : numRequests)\n> > > -\t\t\tresults.add(testRequestBalance(camera, role.second, 3, num));\n> > > -\n> > > -\t\t/*\n> > > -\t\t * Test unbalanced stop\n> > > -\t\t *\n> > > -\t\t * Makes sure the camera supports a stop with requests queued.\n> > > -\t\t * Example failure is a camera that does not handle cancelation\n> > > -\t\t * of buffers coming back from the video device while stopping.\n> > > -\t\t */\n> > > -\t\tstd::cout << \"* Test unbalanced stop\" << std::endl;\n> > > -\t\tfor (unsigned int num : numRequests)\n> > > -\t\t\tresults.add(testRequestUnbalance(camera, role.second, num));\n> > > -\t}\n> > > -\n> > > -\treturn results;\n> > > +};\n> > > +\n> > > +/*\n> > > + * Test unbalanced stop\n> > > + *\n> > > + * Makes sure the camera supports a stop with requests queued. Example failure\n> > > + * is a camera that does not handle cancelation of buffers coming back from the\n> > > + * video device while stopping.\n> > > + */\n> > > +class Unbalanced : public SingleStream {\n> > > +public:\n> > > +\texplicit Unbalanced(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > +\n> > > +\tvoid TestBody() override {\n> > > +\t\tSimpleCaptureUnbalanced capture(camera_);\n> > > +\n> > > +\t\tcapture.configure(role_);\n> > > +\n> > > +\t\tcapture.capture(numRequests_);\n> > > +\t};\n> > > +};\n> > > +\n> > > +void registerSingleStreamTests(std::vector> cameras,\n> > > +\t\tstd::map roles, std::vector requests)\n> > > +{\n> > > +\tREGISTER_TESTS(SingleStream, BalancedSingleCycle, cameras, roles, requests);\n> > > +\tREGISTER_TESTS(SingleStream, BalancedMultiCycle, cameras, roles, requests);\n> > > +\tREGISTER_TESTS(SingleStream, Unbalanced, cameras, roles, requests);\n> > > }\n> > > diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> > > index 396605214e4b..6d5a8b88c287 100644\n> > > --- a/src/lc-compliance/tests.h\n> > > +++ b/src/lc-compliance/tests.h\n> > > @@ -11,6 +11,19 @@\n> > > \n> > > #include \"results.h\"\n> > > \n> > > -Results testSingleStream(std::shared_ptr camera);\n> > > +void registerSingleStreamTests(std::vector> cameras,\n> > > +\t\tstd::map roles, std::vector requests);\n> > > +\n> > > +#define REGISTER_TESTS(testsuite, testcase, cameras, roles, requests) \\\n> > > +for (auto c : cameras) { \\\n> > > +\tfor (auto [r, rn] : roles) { \\\n> > > +\t\tfor (auto rq : requests) { \\\n> > > +\t\t\ttesting::RegisterTest( \\\n> > > +\t\t\t\t#testsuite, (std::string(#testcase) + \"/\" + c->id() + \"/\" + rn + \"/\" + std::to_string(rq)).c_str(), \\\n> > > +\t\t\t\tnullptr, nullptr, __FILE__, __LINE__, \\\n> > > +\t\t\t\t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > +\t\t} \\\n> > > +\t} \\\n> > > +}\n> > > \n> > > #endif /* __LC_COMPLIANCE_TESTS_H__ */\n> > > -- \n> > > 2.31.1\n> > > \n> >\n> > --\n> > Regards,\n> > Niklas Söderlund\n> >\n> > --\n> > To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 514B0BF831\n\tfor ;\n\tFri, 7 May 2021 09:11:58 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B54EB6891B;\n\tFri, 7 May 2021 11:11:57 +0200 (CEST)","from mail-lj1-x233.google.com (mail-lj1-x233.google.com\n\t[IPv6:2a00:1450:4864:20::233])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2A7DE688E4\n\tfor ;\n\tFri, 7 May 2021 11:11:56 +0200 (CEST)","by mail-lj1-x233.google.com with SMTP id w15so10650682ljo.10\n\tfor ;\n\tFri, 07 May 2021 02:11:56 -0700 (PDT)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\ta26sm1293220lfl.127.2021.05.07.02.11.54\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 07 May 2021 02:11:54 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"KVpkOMG0\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=dNK8aSi0kHC3u0lYB83UnpAszrn0rXIcPyU6MQ29MWM=;\n\tb=KVpkOMG0v1LpJMCo26S/XJ59zF5xBLnLR7LT3udGmjfNWvo0GWdGnzrQMVy/XGBwa9\n\tv0GnmPcPqNPOgO8+deFnpyJr7U96ShMuMUmitvWWoAHZKZz4q0CoECPGWv3usDoTdWZL\n\tERUQxIOcqCYU4C1TMyyT18tqGpgtHxdd+Z52CHY5wNVv8WdDPIG90Wwak04IuENq3aEb\n\t8ZkYwjYvLOGT03BSMBqUwzWpmuI7Ajgex9zJYt7xnOKve6uhJ9dZtw8hWRbcQBSDtsWu\n\tpaLnwBqs1obsv2XkkEw3NmDWnnC8jpmE8NH/VlF9iWIFTCa0mjyVh1ydMplpu0AOQzqg\n\t6GGQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=dNK8aSi0kHC3u0lYB83UnpAszrn0rXIcPyU6MQ29MWM=;\n\tb=Rdg1IAUkF61U0aVqF6TFGiirfbUFs5lLbaBtqSpqcqkpONe4G+8aHmZCXPKkv/v3Ay\n\tb0wYrJE1Ci94lIthQQjAFJSBY6uVlSHW34SpD6TW41bXHek7JC4z6nmxB+CLuw24LU8n\n\tRYHkUcUhsQcHPLvTW4MxMlK9L6KCUqbzqM3ew8m4ScG9Ns4d5zW4Tg/Sr4lXceJNII+R\n\tYk3M0coWL49zvM3symv04L1+HoRa0kUEFf3WFGvheZoVZRy8bDrOco8jMMD9jkwqoT0e\n\tQSPx6MSNvTKr+B8w9uWZ3AWqZBUdxE2hu1XYy7akIDkGUnmXNOU4Yg3Wpqu79Az0gM0K\n\tsGkg==","X-Gm-Message-State":"AOAM531+ZxVEBCbLMFSiFDdOSq+QGjcToCCrGKXg3pIMn7jx1ZcnmpvR\n\t5bUkaVLkDL64iWq4ujl7F886bQ==","X-Google-Smtp-Source":"ABdhPJwL8eeioVU1Ziisz/kn6hBFl8zRAf2VeJDKt7gT1Y4w47ZVVNMXETntz1+2bbg+tK9CPpVyXA==","X-Received":"by 2002:a05:651c:31b:: with SMTP id\n\ta27mr6954219ljp.324.1620378715164; \n\tFri, 07 May 2021 02:11:55 -0700 (PDT)","Date":"Fri, 7 May 2021 11:11:53 +0200","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"=?iso-8859-1?q?N=EDcolas_F=2E_R=2E_A=2E?= Prado ","Message-ID":"","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com,\n\t=?iso-8859-1?q?Andr=E9?= Almeida ","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16833,"web_url":"https://patchwork.libcamera.org/comment/16833/","msgid":"","date":"2021-05-07T15:52:38","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":84,"url":"https://patchwork.libcamera.org/api/people/84/","name":"Nícolas F. R. A. Prado","email":"nfraprado@collabora.com"},"content":"Hi Niklas,\n\nEm 2021-05-07 06:11, Niklas Söderlund escreveu:\n\n> Hi Nícolas,\n>\n> On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > Hi Niklas,\n> > \n> > thanks for the feedback.\n> > \n> > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > \n> > > Hi Nícolas,\n> > >\n> > > Thanks for your work!\n> > >\n> > > I really like how gtest helps make the actual tests to be easier to\n> > > read.\n> > >\n> > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > \n> > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > ---\n> > > > Changes in v2:\n> > > > - Changed from static to dynamic test registration\n> > > > - Removed -c flag\n> > > > \n> > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > apparently aren't being deleted on time, which causes:\n> > > > \n> > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > \tSegmentation fault (core dumped)\n> > > > \n> > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > \n> > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > \n> > > > Not sure how to solve this. Any tip would be welcome.\n> > >\n> > > I will leave this for now as I have a different worry about\n> > > REGISTER_TESTS() design below.\n> > >\n> > > > \n> > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > src/lc-compliance/meson.build | 3 +\n> > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > src/lc-compliance/tests.h | 15 ++-\n> > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > \n> > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > index 54cee54aa978..add0d7729aec 100644\n> > > > --- a/src/lc-compliance/main.cpp\n> > > > +++ b/src/lc-compliance/main.cpp\n> > > > @@ -9,6 +9,8 @@\n> > > > #include \n> > > > #include \n> > > > \n> > > > +#include \n> > > > +\n> > > > #include \n> > > > \n> > > > #include \"../cam/options.h\"\n> > > > @@ -17,7 +19,6 @@\n> > > > using namespace libcamera;\n> > > > \n> > > > enum {\n> > > > -\tOptCamera = 'c',\n> > > > \tOptHelp = 'h',\n> > > > };\n> > > > \n> > > > @@ -28,14 +29,15 @@ public:\n> > > > \t~Harness();\n> > > > \n> > > > \tint exec();\n> > > > +\tint init();\n> > > > +\tvoid registerTests();\n> > > > \n> > > > private:\n> > > > -\tint init();\n> > > > \tvoid listCameras();\n> > > > \n> > > > \tOptionsParser::Options options_;\n> > > > \tstd::unique_ptr cm_;\n> > > > -\tstd::shared_ptr camera_;\n> > > > +\tstd::vector> cameras_;\n> > > > };\n> > > > \n> > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > \n> > > > Harness::~Harness()\n> > > > {\n> > > > -\tif (camera_) {\n> > > > -\t\tcamera_->release();\n> > > > -\t\tcamera_.reset();\n> > > > +\tfor (auto &c : cameras_) {\n> > > > +\t\tc->release();\n> > > > +\t\tc.reset();\n> > > > \t}\n> > > > \n> > > > \tcm_->stop();\n> > > > }\n> > > > \n> > > > -int Harness::exec()\n> > > > -{\n> > > > -\tint ret = init();\n> > > > -\tif (ret)\n> > > > -\t\treturn ret;\n> > > > -\n> > > > -\tstd::vector results;\n> > > > -\n> > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > -\n> > > > -\tfor (const Results &result : results) {\n> > > > -\t\tret = result.summary();\n> > > > -\t\tif (ret)\n> > > > -\t\t\treturn ret;\n> > > > -\t}\n> > > > -\n> > > > -\treturn 0;\n> > > > -}\n> > > > -\n> > > > int Harness::init()\n> > > > {\n> > > > \tint ret = cm_->start();\n> > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > \t\treturn ret;\n> > > > \t}\n> > > > \n> > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > -\t\tlistCameras();\n> > > > -\t\treturn -ENODEV;\n> > > > -\t}\n> > > > -\n> > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > -\tcamera_ = cm_->get(cameraId);\n> > > > -\tif (!camera_) {\n> > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > -\t\tlistCameras();\n> > > > -\t\treturn -ENODEV;\n> > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > +\t\tif (cam->acquire()) {\n> > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > +\t\t\treturn -EINVAL;\n> > > > +\t\t}\n> > > > +\t\tcameras_.push_back(cam);\n> > >\n> > > I don't like this, I think we need to retain the ability to select which\n> > > camera to test. And further down the line maybe allow more then one\n> > > camera to be selected if we want to test concurrent use of two specific\n> > > ones. But for the work in this series I think you can ignore multiple\n> > > cameras.\n> > \n> > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > all cameras, only the ones that I will actually test (and in this code there's a\n> > big issue where if a single camera is already busy, even if it won't be tested,\n> > lc-compliance as a whole fails).\n>\n> Ahh I understand.\n>\n> > \n> > >\n> > > If I understand the coverletter correctly the correct thing here the cli\n> > > arguments should be feed to gtest directly to be able to filter on what\n> > > tests are run. Is there support in getst to also pass arguments to\n> > > tests? Maybe we could remove the need to parse arguments ourself\n> > > completely?\n> > \n> > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > could solve both problems (this and the one above) like this:\n> > \n> > - In the main(), query the id of each camera from libcamera.\n> > - Register each of the tests dynamically in gtest passing as an argument the\n> > camera id, rather than a camera pointer (which is the current behavior).\n> > - To select which tests to run, and on which camera(s), the user just uses\n> > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > the camera).\n> > - In the startup of each test the camera is acquired for that test, and when\n> > it's done it is released.\n> > \n> > Implications:\n> > - We no longer need to parse arguments and can drop that code, as both camera\n> > and test are specified directly with gtest's filter.\n> > - We only acquire the cameras that we'll actually test and only while they're\n> > being tested.\n> > - As a downside, a single camera will probably be acquired multiple times during\n> > a single run as lc-compliance moves from one test to the next, but I don't\n> > think this would be a big issue. (Maybe we could do this at the test suite\n> > level instead to greatly decrease the number of times of acquire(), but I'm\n> > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > the test level, I'll have to test that).\n> > \n> > What do you think?\n>\n> I understand what you are trying to do but I see two issues.\n>\n> 1. It depends on the implicit knowledge of users how to build the filter\n> string and is rather unintuitive compared to having two distinct\n> arguments, one to select camera and one to express the test filter.\n>\n> This could possibly be worked around by keeping the cam options code\n> and building the gtest filter string internally by injecitng the\n> selected camera name in the correct location, but the we have the\n> next issue,\n>\n> 2. I don't think it will be practically possible to express a test\n> filter that selects a specific combination of two cameras.\n>\n> I do think we need to be able to parameterize test cases. If gtest does\n> not allow to do so using function arguments there are other ways of\n> doing so.\n\nI think I didn't express myself well. I meant to say that gtest's CLI doesn't\nhave a flag to allow passing parameters to tests, meaning that it would not be\npossible to parametrize the tests from the command line and also remove our\nown parser and rely on gtest's.\n\nBut gtest does allow test parametrization at runtime and I'm already relying on\nit since v2. Let me go a bit more over the way I'm doing this.\n\n- I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n\n\tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n\n- Since we may want to run the same test for all cameras, roles and numRequests,\n I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n gtest's testing::RegisterTest() with all possible combinations of parameters.\n\n- Based on the --gtest_filter CLI argument from gtest, it decides which tests\n are actually run\n\nSo the key point is that even though I parametrize the tests, I do it for all\ncombinations in order to later rely on gtest_filter to select which tests to\nrun.\n\nNow, I do agree with both of your points. And if we want to make the interface\nmore intuitive to use (which I also find important), there's no way around\nhaving our own parser.\n\nAnd if we're using our own camera flag, then it makes no sense to register tests\nfor all cameras, we can register tests only with the specified camera.\n\nI think here it would be helpful to think if the interface suits our problem.\nTo me we're looking at having:\n\n\tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n\nto only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\nthe camera flag is used to only register tests with that camera, and the filter\nis passed as --gtest_filter, possibly a bit adapted, to gtest.\n\nAnd if we imagine we have a test ConcurrentCameras, which receives two cameras\nas parameters, and we call\n\n\tlc-compliance --camera 'cameraId1,cameraId2'\n\nit would test both each of the cameras separately in all current tests which\naccept a single camera, like BalancedSingleCycle, as well as test both at the\nsame time in the ConcurrentCameras test (and we could make only the former or\nthe latter occur by using the filter parameter).\n\nI know we don't need to implement support for multiple cameras right now, I just\nwant to make sure it's possible and thinking about the interface and the\nscenarios help me picture how they'd work.\n\nThoughts?\n\nThanks,\nNícolas\n\n>\n> I'm sure there are many solutions to this problem and maybe someone more\n> familiar with gtest knows of a neat way to piggyback on its\n> infrastrcuture? Since I know very little about the gtest framework the\n> following is just an idea and I'm happy to yield.\n>\n> How about using a singleton class that carries arguments constructed\n> from cli arguments (camera id) before the gtest are executed. Each test\n> case could then consult this singleton if it needs to access parameters.\n> I think you could take inspiration from the libcamera CameraManager\n> class.\n>\n> This could (at lest for now) imply keeping the cam option parser. If so\n> I think it would be nice if somewhen before we integrate this work that\n> the gtest filter string could be supplied as an argument using that\n> parser. Of course if we can do it the other way around and add arguments\n> to the gtest parser that would be even cooler as it would be less code\n> for us to maintain ;-)\n>\n> > \n> > Thanks,\n> > Nícolas\n> > \n> > >\n> > > > \t}\n> > > > \n> > > > -\tif (camera_->acquire()) {\n> > > > -\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > -\t\treturn -EINVAL;\n> > > > -\t}\n> > > > -\n> > > > -\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> > > > -\n> > > > \treturn 0;\n> > > > }\n> > > > \n> > > > void Harness::listCameras()\n> > > > {\n> > > > -\tfor (const std::shared_ptr &cam : cm_->cameras())\n> > > > -\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> > > > +\tfor (const std::shared_ptr &c : cm_->cameras())\n> > > > +\t\tstd::cout << \"- \" << c.get()->id() << std::endl;\n> > > > }\n> > > > \n> > > > static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > > > {\n> > > > \tOptionsParser parser;\n> > > > -\tparser.addOption(OptCamera, OptionString,\n> > > > -\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> > > > -\t\t\t ArgumentRequired, \"camera\");\n> > > > \tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> > > > \t\t\t \"help\");\n> > > > \n> > > > @@ -133,6 +100,31 @@ static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > > > \treturn 0;\n> > > > }\n> > > > \n> > > > +/*\n> > > > + * Make asserts act like exceptions, otherwise they only fail (or skip) the\n> > > > + * current function. From gtest documentation:\n> > > > + * https://google.github.io/googletest/advanced.html#asserting-on-subroutines-with-an-exception\n> > > > + */\n> > > > +class ThrowListener : public testing::EmptyTestEventListener {\n> > > > +\tvoid OnTestPartResult(const testing::TestPartResult& result) override {\n> > > > +\t\tif (result.type() == testing::TestPartResult::kFatalFailure\n> > > > +\t\t || result.type() == testing::TestPartResult::kSkip) {\n> > > > +\t\t\tthrow testing::AssertionException(result);\n> > > > +\t\t}\n> > > > +\t}\n> > > > +};\n> > > > +\n> > > > +\n> > > > +void Harness::registerTests() {\n> > > > +\tstd::map roles = {{Raw, \"Raw\"},\n> > > > +\t\t\t\t\t\t {StillCapture, \"Still\"},\n> > > > +\t\t\t\t\t\t {VideoRecording, \"Video\"},\n> > > > +\t\t\t\t\t\t {Viewfinder, \"Viewfinder\"}};\n> > > > +\tstd::vector requests = {1, 2, 3, 5, 8, 13, 21, 34, 55, 89};\n> > > > +\n> > > > +\tregisterSingleStreamTests(cameras_, roles, requests);\n> > > > +}\n> > > > +\n> > > > int main(int argc, char **argv)\n> > > > {\n> > > > \tOptionsParser::Options options;\n> > > > @@ -143,6 +135,13 @@ int main(int argc, char **argv)\n> > > > \t\treturn EXIT_FAILURE;\n> > > > \n> > > > \tHarness harness(options);\n> > > > +\tret = harness.init();\n> > > > +\tif (ret)\n> > > > +\t\treturn ret;\n> > > > +\n> > > > +\tharness.registerTests();\n> > > > \n> > > > -\treturn harness.exec() ? EXIT_FAILURE : EXIT_SUCCESS;\n> > > > +\t::testing::InitGoogleTest(&argc, argv);\n> > > > +\ttesting::UnitTest::GetInstance()->listeners().Append(new ThrowListener);\n> > > > +\treturn RUN_ALL_TESTS();\n> > > > }\n> > > > diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> > > > index a2bfcceb1259..704bc18af3e1 100644\n> > > > --- a/src/lc-compliance/meson.build\n> > > > +++ b/src/lc-compliance/meson.build\n> > > > @@ -18,10 +18,13 @@ lc_compliance_sources = files([\n> > > > 'single_stream.cpp',\n> > > > ])\n> > > > \n> > > > +gtest_dep = dependency('gtest')\n> > > > +\n> > > > lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> > > > dependencies : [\n> > > > libatomic,\n> > > > libcamera_dep,\n> > > > libevent,\n> > > > + gtest_dep,\n> > > > ],\n> > > > install : true)\n> > > > diff --git a/src/lc-compliance/simple_capture.cpp b/src/lc-compliance/simple_capture.cpp\n> > > > index f90fe6d0f9aa..7731eb16f8c2 100644\n> > > > --- a/src/lc-compliance/simple_capture.cpp\n> > > > +++ b/src/lc-compliance/simple_capture.cpp\n> > > > @@ -5,6 +5,8 @@\n> > > > * simple_capture.cpp - Simple capture helper\n> > > > */\n> > > > \n> > > > +#include \n> > > > +\n> > > > #include \"simple_capture.h\"\n> > > > \n> > > > using namespace libcamera;\n> > > > @@ -20,38 +22,34 @@ SimpleCapture::~SimpleCapture()\n> > > > \tstop();\n> > > > }\n> > > > \n> > > > -Results::Result SimpleCapture::configure(StreamRole role)\n> > > > +void SimpleCapture::configure(StreamRole role)\n> > > > {\n> > > > \tconfig_ = camera_->generateConfiguration({ role });\n> > > > \n> > > > -\tif (!config_)\n> > > > -\t\treturn { Results::Skip, \"Role not supported by camera\" };\n> > > > +\tif (!config_) {\n> > > > +\t\tstd::cout << \"Role not supported by camera\" << std::endl;\n> > > > +\t\tGTEST_SKIP();\n> > > > +\t}\n> > > > \n> > > > \tif (config_->validate() != CameraConfiguration::Valid) {\n> > > > \t\tconfig_.reset();\n> > > > -\t\treturn { Results::Fail, \"Configuration not valid\" };\n> > > > +\t\tFAIL() << \"Configuration not valid\";\n> > > > \t}\n> > > > \n> > > > \tif (camera_->configure(config_.get())) {\n> > > > \t\tconfig_.reset();\n> > > > -\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> > > > +\t\tFAIL() << \"Failed to configure camera\";\n> > > > \t}\n> > > > -\n> > > > -\treturn { Results::Pass, \"Configure camera\" };\n> > > > }\n> > > > \n> > > > -Results::Result SimpleCapture::start()\n> > > > +void SimpleCapture::start()\n> > > > {\n> > > > \tStream *stream = config_->at(0).stream();\n> > > > -\tif (allocator_->allocate(stream) < 0)\n> > > > -\t\treturn { Results::Fail, \"Failed to allocate buffers\" };\n> > > > +\tASSERT_GE(allocator_->allocate(stream), 0) << \"Failed to allocate buffers\";\n> > > > \n> > > > -\tif (camera_->start())\n> > > > -\t\treturn { Results::Fail, \"Failed to start camera\" };\n> > > > +\tASSERT_TRUE(!camera_->start()) << \"Failed to start camera\";\n> > > > \n> > > > \tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> > > > -\n> > > > -\treturn { Results::Pass, \"Started camera\" };\n> > > > }\n> > > > \n> > > > void SimpleCapture::stop()\n> > > > @@ -77,22 +75,19 @@ SimpleCaptureBalanced::SimpleCaptureBalanced(std::shared_ptr camera)\n> > > > {\n> > > > }\n> > > > \n> > > > -Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > +void SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > {\n> > > > -\tResults::Result ret = start();\n> > > > -\tif (ret.first != Results::Pass)\n> > > > -\t\treturn ret;\n> > > > +\tstart();\n> > > > \n> > > > \tStream *stream = config_->at(0).stream();\n> > > > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > > > \n> > > > \t/* No point in testing less requests then the camera depth. */\n> > > > \tif (buffers.size() > numRequests) {\n> > > > -\t\t/* Cache buffers.size() before we destroy it in stop() */\n> > > > -\t\tint buffers_size = buffers.size();\n> > > > -\n> > > > -\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers_size)\n> > > > -\t\t\t+ \" requests, can't test only \" + std::to_string(numRequests) };\n> > > > +\t\tstd::cout << \"Camera needs \" + std::to_string(buffers.size())\n> > > > +\t\t\t+ \" requests, can't test only \"\n> > > > +\t\t\t+ std::to_string(numRequests) << std::endl;\n> > > > +\t\tGTEST_SKIP();\n> > > > \t}\n> > > > \n> > > > \tqueueCount_ = 0;\n> > > > @@ -103,14 +98,11 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > \tstd::vector> requests;\n> > > > \tfor (const std::unique_ptr &buffer : buffers) {\n> > > > \t\tstd::unique_ptr request = camera_->createRequest();\n> > > > -\t\tif (!request)\n> > > > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > > > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > > > \n> > > > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > > > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > > > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > > > \n> > > > -\t\tif (queueRequest(request.get()) < 0)\n> > > > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > > > +\t\tASSERT_GE(queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > > > \n> > > > \t\trequests.push_back(std::move(request));\n> > > > \t}\n> > > > @@ -121,12 +113,7 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > \tstop();\n> > > > \tdelete loop_;\n> > > > \n> > > > -\tif (captureCount_ != captureLimit_)\n> > > > -\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) +\n> > > > -\t\t\t\" request, wanted \" + std::to_string(captureLimit_) };\n> > > > -\n> > > > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > > > -\t\tstd::to_string(numRequests) + \" requests\" };\n> > > > +\tASSERT_EQ(captureCount_, captureLimit_);\n> > > > }\n> > > > \n> > > > int SimpleCaptureBalanced::queueRequest(Request *request)\n> > > > @@ -158,11 +145,9 @@ SimpleCaptureUnbalanced::SimpleCaptureUnbalanced(std::shared_ptr camera)\n> > > > {\n> > > > }\n> > > > \n> > > > -Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > +void SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > {\n> > > > -\tResults::Result ret = start();\n> > > > -\tif (ret.first != Results::Pass)\n> > > > -\t\treturn ret;\n> > > > +\tstart();\n> > > > \n> > > > \tStream *stream = config_->at(0).stream();\n> > > > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > > > @@ -174,14 +159,11 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > \tstd::vector> requests;\n> > > > \tfor (const std::unique_ptr &buffer : buffers) {\n> > > > \t\tstd::unique_ptr request = camera_->createRequest();\n> > > > -\t\tif (!request)\n> > > > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > > > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > > > \n> > > > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > > > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > > > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > > > \n> > > > -\t\tif (camera_->queueRequest(request.get()) < 0)\n> > > > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > > > +\t\tASSERT_GE(camera_->queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > > > \n> > > > \t\trequests.push_back(std::move(request));\n> > > > \t}\n> > > > @@ -192,7 +174,7 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > \tstop();\n> > > > \tdelete loop_;\n> > > > \n> > > > -\treturn { status ? Results::Fail : Results::Pass, \"Unbalanced capture of \" + std::to_string(numRequests) + \" requests\" };\n> > > > +\tASSERT_FALSE(status);\n> > > > }\n> > > > \n> > > > void SimpleCaptureUnbalanced::requestComplete(Request *request)\n> > > > diff --git a/src/lc-compliance/simple_capture.h b/src/lc-compliance/simple_capture.h\n> > > > index d9de53fb63a3..0f8465083456 100644\n> > > > --- a/src/lc-compliance/simple_capture.h\n> > > > +++ b/src/lc-compliance/simple_capture.h\n> > > > @@ -17,13 +17,13 @@\n> > > > class SimpleCapture\n> > > > {\n> > > > public:\n> > > > -\tResults::Result configure(libcamera::StreamRole role);\n> > > > +\tvoid configure(libcamera::StreamRole role);\n> > > > \n> > > > protected:\n> > > > \tSimpleCapture(std::shared_ptr camera);\n> > > > \tvirtual ~SimpleCapture();\n> > > > \n> > > > -\tResults::Result start();\n> > > > +\tvoid start();\n> > > > \tvoid stop();\n> > > > \n> > > > \tvirtual void requestComplete(libcamera::Request *request) = 0;\n> > > > @@ -40,7 +40,7 @@ class SimpleCaptureBalanced : public SimpleCapture\n> > > > public:\n> > > > \tSimpleCaptureBalanced(std::shared_ptr camera);\n> > > > \n> > > > -\tResults::Result capture(unsigned int numRequests);\n> > > > +\tvoid capture(unsigned int numRequests);\n> > > > \n> > > > private:\n> > > > \tint queueRequest(libcamera::Request *request);\n> > > > @@ -56,7 +56,7 @@ class SimpleCaptureUnbalanced : public SimpleCapture\n> > > > public:\n> > > > \tSimpleCaptureUnbalanced(std::shared_ptr camera);\n> > > > \n> > > > -\tResults::Result capture(unsigned int numRequests);\n> > > > +\tvoid capture(unsigned int numRequests);\n> > > > \n> > > > private:\n> > > > \tvoid requestComplete(libcamera::Request *request) override;\n> > > > diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> > > > index 8318b42f42d6..eb6a6f305826 100644\n> > > > --- a/src/lc-compliance/single_stream.cpp\n> > > > +++ b/src/lc-compliance/single_stream.cpp\n> > > > @@ -7,91 +7,94 @@\n> > > > \n> > > > #include \n> > > > \n> > > > +#include \n> > > > +\n> > > > #include \"simple_capture.h\"\n> > > > #include \"tests.h\"\n> > > > \n> > > > using namespace libcamera;\n> > > > \n> > > > -Results::Result testRequestBalance(std::shared_ptr camera,\n> > > > -\t\t\t\t StreamRole role, unsigned int startCycles,\n> > > > -\t\t\t\t unsigned int numRequests)\n> > > > -{\n> > > > -\tSimpleCaptureBalanced capture(camera);\n> > > > +class SingleStream : public testing::Test {\n> > > > +public:\n> > > > +\texplicit SingleStream(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > +\t\t: camera_(camera), role_(role), numRequests_(numRequests) {}\n> > > > \n> > > > -\tResults::Result ret = capture.configure(role);\n> > > > -\tif (ret.first != Results::Pass)\n> > > > -\t\treturn ret;\n> > > > +protected:\n> > > > +\tstd::shared_ptr camera_;\n> > > > +\tStreamRole role_;\n> > > > +\tunsigned int numRequests_;\n> > > > +};\n> > > > \n> > > > -\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> > > > -\t\tret = capture.capture(numRequests);\n> > > > -\t\tif (ret.first != Results::Pass)\n> > > > -\t\t\treturn ret;\n> > > > -\t}\n> > > > +/*\n> > > > + * Test single capture cycles\n> > > > + *\n> > > > + * Makes sure the camera completes the exact number of requests queued. Example\n> > > > + * failure is a camera that needs N+M requests queued to complete N requests to\n> > > > + * the application.\n> > > > + */\n> > > > +class BalancedSingleCycle : public SingleStream {\n> > > > +public:\n> > > > +\texplicit BalancedSingleCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > > \n> > > > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > > > -\t\tstd::to_string(numRequests) + \" requests with \" +\n> > > > -\t\tstd::to_string(startCycles) + \" start cycles\" };\n> > > > -}\n> > > > +\tvoid TestBody() override {\n> > > > +\t\tSimpleCaptureBalanced capture(camera_);\n> > > > \n> > > > -Results::Result testRequestUnbalance(std::shared_ptr camera,\n> > > > -\t\t\t\t StreamRole role, unsigned int numRequests)\n> > > > -{\n> > > > -\tSimpleCaptureUnbalanced capture(camera);\n> > > > +\t\tcapture.configure(role_);\n> > > > \n> > > > -\tResults::Result ret = capture.configure(role);\n> > > > -\tif (ret.first != Results::Pass)\n> > > > -\t\treturn ret;\n> > > > +\t\tcapture.capture(numRequests_);\n> > > > +\t};\n> > > > +};\n> > > > \n> > > > -\treturn capture.capture(numRequests);\n> > > > -}\n> > > > +/*\n> > > > + * Test multiple start/stop cycles\n> > > > + *\n> > > > + * Makes sure the camera supports multiple start/stop cycles. Example failure is\n> > > > + * a camera that does not clean up correctly in its error path but is only\n> > > > + * tested by single-capture applications.\n> > > > + */\n> > > > +class BalancedMultiCycle : public SingleStream {\n> > > > +public:\n> > > > +\texplicit BalancedMultiCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > > \n> > > > -Results testSingleStream(std::shared_ptr camera)\n> > > > -{\n> > > > -\tstatic const std::vector> roles = {\n> > > > -\t\t{ \"raw\", Raw },\n> > > > -\t\t{ \"still\", StillCapture },\n> > > > -\t\t{ \"video\", VideoRecording },\n> > > > -\t\t{ \"viewfinder\", Viewfinder },\n> > > > +\tvoid TestBody() override {\n> > > > +\t\tunsigned int numRepeats = 3;\n> > > > +\n> > > > +\t\tSimpleCaptureBalanced capture(camera_);\n> > > > +\n> > > > +\t\tcapture.configure(role_);\n> > > > +\n> > > > +\t\tfor (unsigned int starts = 0; starts < numRepeats; starts++)\n> > > > +\t\t\tcapture.capture(numRequests_);\n> > > > \t};\n> > > > -\tstatic const std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> > > > -\n> > > > -\tResults results(numRequests.size() * roles.size() * 3);\n> > > > -\n> > > > -\tfor (const auto &role : roles) {\n> > > > -\t\tstd::cout << \"= Test role \" << role.first << std::endl;\n> > > > -\t\t/*\n> > > > -\t\t * Test single capture cycles\n> > > > -\t\t *\n> > > > -\t\t * Makes sure the camera completes the exact number of requests queued.\n> > > > -\t\t * Example failure is a camera that needs N+M requests queued to\n> > > > -\t\t * complete N requests to the application.\n> > > > -\t\t */\n> > > > -\t\tstd::cout << \"* Test single capture cycles\" << std::endl;\n> > > > -\t\tfor (unsigned int num : numRequests)\n> > > > -\t\t\tresults.add(testRequestBalance(camera, role.second, 1, num));\n> > > > -\n> > > > -\t\t/*\n> > > > -\t\t * Test multiple start/stop cycles\n> > > > -\t\t *\n> > > > -\t\t * Makes sure the camera supports multiple start/stop cycles.\n> > > > -\t\t * Example failure is a camera that does not clean up correctly in its\n> > > > -\t\t * error path but is only tested by single-capture applications.\n> > > > -\t\t */\n> > > > -\t\tstd::cout << \"* Test multiple start/stop cycles\" << std::endl;\n> > > > -\t\tfor (unsigned int num : numRequests)\n> > > > -\t\t\tresults.add(testRequestBalance(camera, role.second, 3, num));\n> > > > -\n> > > > -\t\t/*\n> > > > -\t\t * Test unbalanced stop\n> > > > -\t\t *\n> > > > -\t\t * Makes sure the camera supports a stop with requests queued.\n> > > > -\t\t * Example failure is a camera that does not handle cancelation\n> > > > -\t\t * of buffers coming back from the video device while stopping.\n> > > > -\t\t */\n> > > > -\t\tstd::cout << \"* Test unbalanced stop\" << std::endl;\n> > > > -\t\tfor (unsigned int num : numRequests)\n> > > > -\t\t\tresults.add(testRequestUnbalance(camera, role.second, num));\n> > > > -\t}\n> > > > -\n> > > > -\treturn results;\n> > > > +};\n> > > > +\n> > > > +/*\n> > > > + * Test unbalanced stop\n> > > > + *\n> > > > + * Makes sure the camera supports a stop with requests queued. Example failure\n> > > > + * is a camera that does not handle cancelation of buffers coming back from the\n> > > > + * video device while stopping.\n> > > > + */\n> > > > +class Unbalanced : public SingleStream {\n> > > > +public:\n> > > > +\texplicit Unbalanced(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > > +\n> > > > +\tvoid TestBody() override {\n> > > > +\t\tSimpleCaptureUnbalanced capture(camera_);\n> > > > +\n> > > > +\t\tcapture.configure(role_);\n> > > > +\n> > > > +\t\tcapture.capture(numRequests_);\n> > > > +\t};\n> > > > +};\n> > > > +\n> > > > +void registerSingleStreamTests(std::vector> cameras,\n> > > > +\t\tstd::map roles, std::vector requests)\n> > > > +{\n> > > > +\tREGISTER_TESTS(SingleStream, BalancedSingleCycle, cameras, roles, requests);\n> > > > +\tREGISTER_TESTS(SingleStream, BalancedMultiCycle, cameras, roles, requests);\n> > > > +\tREGISTER_TESTS(SingleStream, Unbalanced, cameras, roles, requests);\n> > > > }\n> > > > diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> > > > index 396605214e4b..6d5a8b88c287 100644\n> > > > --- a/src/lc-compliance/tests.h\n> > > > +++ b/src/lc-compliance/tests.h\n> > > > @@ -11,6 +11,19 @@\n> > > > \n> > > > #include \"results.h\"\n> > > > \n> > > > -Results testSingleStream(std::shared_ptr camera);\n> > > > +void registerSingleStreamTests(std::vector> cameras,\n> > > > +\t\tstd::map roles, std::vector requests);\n> > > > +\n> > > > +#define REGISTER_TESTS(testsuite, testcase, cameras, roles, requests) \\\n> > > > +for (auto c : cameras) { \\\n> > > > +\tfor (auto [r, rn] : roles) { \\\n> > > > +\t\tfor (auto rq : requests) { \\\n> > > > +\t\t\ttesting::RegisterTest( \\\n> > > > +\t\t\t\t#testsuite, (std::string(#testcase) + \"/\" + c->id() + \"/\" + rn + \"/\" + std::to_string(rq)).c_str(), \\\n> > > > +\t\t\t\tnullptr, nullptr, __FILE__, __LINE__, \\\n> > > > +\t\t\t\t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > +\t\t} \\\n> > > > +\t} \\\n> > > > +}\n> > > > \n> > > > #endif /* __LC_COMPLIANCE_TESTS_H__ */\n> > > > -- \n> > > > 2.31.1\n> > > > \n> > >\n> > > --\n> > > Regards,\n> > > Niklas Söderlund\n> > >\n> > > --\n> > > To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.\n> > \n> > \n>\n> --\n> Regards,\n> Niklas Söderlund\n>\n> --\n> To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DC92BBF829\n\tfor ;\n\tFri, 7 May 2021 15:53:17 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4DB5468919;\n\tFri, 7 May 2021 17:53:17 +0200 (CEST)","from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0C48E602BE\n\tfor ;\n\tFri, 7 May 2021 17:53:16 +0200 (CEST)","from localhost (unknown\n\t[IPv6:2804:14c:1a9:2978:995d:672b:100f:2fd9])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id E5D6A1F41650;\n\tFri, 7 May 2021 16:53:13 +0100 (BST)"],"Mime-Version":"1.0","Date":"Fri, 07 May 2021 12:52:38 -0300","Message-Id":"","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","From":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= ","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com, =?utf-8?q?A?=\n\t=?utf-8?q?ndr=C3=A9_Almeida?= ","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16872,"web_url":"https://patchwork.libcamera.org/comment/16872/","msgid":"","date":"2021-05-10T21:04:13","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Nícolas,\n\nThanks for your work on this. I really think gtest is a step in the \nright direction!\n\nOn 2021-05-07 12:52:38 -0300, Nícolas F. R. A. Prado wrote:\n> Hi Niklas,\n> \n> Em 2021-05-07 06:11, Niklas Söderlund escreveu:\n> \n> > Hi Nícolas,\n> >\n> > On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > > Hi Niklas,\n> > > \n> > > thanks for the feedback.\n> > > \n> > > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > > \n> > > > Hi Nícolas,\n> > > >\n> > > > Thanks for your work!\n> > > >\n> > > > I really like how gtest helps make the actual tests to be easier to\n> > > > read.\n> > > >\n> > > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > > \n> > > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > > ---\n> > > > > Changes in v2:\n> > > > > - Changed from static to dynamic test registration\n> > > > > - Removed -c flag\n> > > > > \n> > > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > > apparently aren't being deleted on time, which causes:\n> > > > > \n> > > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > > \tSegmentation fault (core dumped)\n> > > > > \n> > > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > > \n> > > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > \n> > > > > Not sure how to solve this. Any tip would be welcome.\n> > > >\n> > > > I will leave this for now as I have a different worry about\n> > > > REGISTER_TESTS() design below.\n> > > >\n> > > > > \n> > > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > > src/lc-compliance/meson.build | 3 +\n> > > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > > src/lc-compliance/tests.h | 15 ++-\n> > > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > > \n> > > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > > index 54cee54aa978..add0d7729aec 100644\n> > > > > --- a/src/lc-compliance/main.cpp\n> > > > > +++ b/src/lc-compliance/main.cpp\n> > > > > @@ -9,6 +9,8 @@\n> > > > > #include \n> > > > > #include \n> > > > > \n> > > > > +#include \n> > > > > +\n> > > > > #include \n> > > > > \n> > > > > #include \"../cam/options.h\"\n> > > > > @@ -17,7 +19,6 @@\n> > > > > using namespace libcamera;\n> > > > > \n> > > > > enum {\n> > > > > -\tOptCamera = 'c',\n> > > > > \tOptHelp = 'h',\n> > > > > };\n> > > > > \n> > > > > @@ -28,14 +29,15 @@ public:\n> > > > > \t~Harness();\n> > > > > \n> > > > > \tint exec();\n> > > > > +\tint init();\n> > > > > +\tvoid registerTests();\n> > > > > \n> > > > > private:\n> > > > > -\tint init();\n> > > > > \tvoid listCameras();\n> > > > > \n> > > > > \tOptionsParser::Options options_;\n> > > > > \tstd::unique_ptr cm_;\n> > > > > -\tstd::shared_ptr camera_;\n> > > > > +\tstd::vector> cameras_;\n> > > > > };\n> > > > > \n> > > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > > \n> > > > > Harness::~Harness()\n> > > > > {\n> > > > > -\tif (camera_) {\n> > > > > -\t\tcamera_->release();\n> > > > > -\t\tcamera_.reset();\n> > > > > +\tfor (auto &c : cameras_) {\n> > > > > +\t\tc->release();\n> > > > > +\t\tc.reset();\n> > > > > \t}\n> > > > > \n> > > > > \tcm_->stop();\n> > > > > }\n> > > > > \n> > > > > -int Harness::exec()\n> > > > > -{\n> > > > > -\tint ret = init();\n> > > > > -\tif (ret)\n> > > > > -\t\treturn ret;\n> > > > > -\n> > > > > -\tstd::vector results;\n> > > > > -\n> > > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > > -\n> > > > > -\tfor (const Results &result : results) {\n> > > > > -\t\tret = result.summary();\n> > > > > -\t\tif (ret)\n> > > > > -\t\t\treturn ret;\n> > > > > -\t}\n> > > > > -\n> > > > > -\treturn 0;\n> > > > > -}\n> > > > > -\n> > > > > int Harness::init()\n> > > > > {\n> > > > > \tint ret = cm_->start();\n> > > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > > \t\treturn ret;\n> > > > > \t}\n> > > > > \n> > > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > > -\t\tlistCameras();\n> > > > > -\t\treturn -ENODEV;\n> > > > > -\t}\n> > > > > -\n> > > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > > -\tcamera_ = cm_->get(cameraId);\n> > > > > -\tif (!camera_) {\n> > > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > > -\t\tlistCameras();\n> > > > > -\t\treturn -ENODEV;\n> > > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > > +\t\tif (cam->acquire()) {\n> > > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > +\t\t\treturn -EINVAL;\n> > > > > +\t\t}\n> > > > > +\t\tcameras_.push_back(cam);\n> > > >\n> > > > I don't like this, I think we need to retain the ability to select which\n> > > > camera to test. And further down the line maybe allow more then one\n> > > > camera to be selected if we want to test concurrent use of two specific\n> > > > ones. But for the work in this series I think you can ignore multiple\n> > > > cameras.\n> > > \n> > > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > > all cameras, only the ones that I will actually test (and in this code there's a\n> > > big issue where if a single camera is already busy, even if it won't be tested,\n> > > lc-compliance as a whole fails).\n> >\n> > Ahh I understand.\n> >\n> > > \n> > > >\n> > > > If I understand the coverletter correctly the correct thing here the cli\n> > > > arguments should be feed to gtest directly to be able to filter on what\n> > > > tests are run. Is there support in getst to also pass arguments to\n> > > > tests? Maybe we could remove the need to parse arguments ourself\n> > > > completely?\n> > > \n> > > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > > could solve both problems (this and the one above) like this:\n> > > \n> > > - In the main(), query the id of each camera from libcamera.\n> > > - Register each of the tests dynamically in gtest passing as an argument the\n> > > camera id, rather than a camera pointer (which is the current behavior).\n> > > - To select which tests to run, and on which camera(s), the user just uses\n> > > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > > the camera).\n> > > - In the startup of each test the camera is acquired for that test, and when\n> > > it's done it is released.\n> > > \n> > > Implications:\n> > > - We no longer need to parse arguments and can drop that code, as both camera\n> > > and test are specified directly with gtest's filter.\n> > > - We only acquire the cameras that we'll actually test and only while they're\n> > > being tested.\n> > > - As a downside, a single camera will probably be acquired multiple times during\n> > > a single run as lc-compliance moves from one test to the next, but I don't\n> > > think this would be a big issue. (Maybe we could do this at the test suite\n> > > level instead to greatly decrease the number of times of acquire(), but I'm\n> > > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > > the test level, I'll have to test that).\n> > > \n> > > What do you think?\n> >\n> > I understand what you are trying to do but I see two issues.\n> >\n> > 1. It depends on the implicit knowledge of users how to build the filter\n> > string and is rather unintuitive compared to having two distinct\n> > arguments, one to select camera and one to express the test filter.\n> >\n> > This could possibly be worked around by keeping the cam options code\n> > and building the gtest filter string internally by injecitng the\n> > selected camera name in the correct location, but the we have the\n> > next issue,\n> >\n> > 2. I don't think it will be practically possible to express a test\n> > filter that selects a specific combination of two cameras.\n> >\n> > I do think we need to be able to parameterize test cases. If gtest does\n> > not allow to do so using function arguments there are other ways of\n> > doing so.\n> \n> I think I didn't express myself well. I meant to say that gtest's CLI doesn't\n> have a flag to allow passing parameters to tests, meaning that it would not be\n> possible to parametrize the tests from the command line and also remove our\n> own parser and rely on gtest's.\n\nCheck, I think we need to keep our own parser for now as we need to be \nable to parametrize the tests from the CLI.\n\n> But gtest does allow test parametrization at runtime and I'm already relying on\n> it since v2. Let me go a bit more over the way I'm doing this.\n> \n> - I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n> \n> \tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n> \n> - Since we may want to run the same test for all cameras, roles and numRequests,\n> I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n> gtest's testing::RegisterTest() with all possible combinations of parameters.\n> \n> - Based on the --gtest_filter CLI argument from gtest, it decides which tests\n> are actually run\n> \n> So the key point is that even though I parametrize the tests, I do it for all\n> combinations in order to later rely on gtest_filter to select which tests to\n> run.\n\nI see, that explains how the parameters get filled. I noticed it but did \nnot dig further, see below for an expansion of my thoughts now that I \nunderstand this detail :-)\n\n> \n> Now, I do agree with both of your points. And if we want to make the interface\n> more intuitive to use (which I also find important), there's no way around\n> having our own parser.\n> \n> And if we're using our own camera flag, then it makes no sense to register tests\n> for all cameras, we can register tests only with the specified camera.\n\nThis feels wrong. It feels like tests should be register without a \ncamera component. The camera (and possible other arguments) should come \nfrom either arguments [1] or a singleton parameter class [2].\n\n1. This do not seem to be possible with gtests as test parameters needs \n to be registerd.\n\n2. This would be similar to the global variable used in earlier versions \n but instead modeled as a class singleton like CameraManager.\n\nMaybe there are other options tho.\n\n> \n> I think here it would be helpful to think if the interface suits our problem.\n> To me we're looking at having:\n> \n> \tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n\nThis matches my view as well.\n\n> \n> to only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\n> the camera flag is used to only register tests with that camera, and the filter\n> is passed as --gtest_filter, possibly a bit adapted, to gtest.\n> \n> And if we imagine we have a test ConcurrentCameras, which receives two cameras\n> as parameters, and we call\n> \n> \tlc-compliance --camera 'cameraId1,cameraId2'\n\nThis also looks good. But for now lets focus on one camera, I think it \nwill give us enough headache :-)\n\n> \n> it would test both each of the cameras separately in all current tests which\n> accept a single camera, like BalancedSingleCycle, as well as test both at the\n> same time in the ConcurrentCameras test (and we could make only the former or\n> the latter occur by using the filter parameter).\n\nI have not thought enough about this to know if that is a good idea or \nnot. At the top of my head I still tend to think that lc-compliance \ntests a single 'primary' camera and any additional camera supplied are \nonly used for tests where two (or more) cameras are needed to test some \nspecific feature of the primary camera.\n\nThat is only the 'primary' camera is testes, the other camera(s) are \njust there to help/provoke the testing. But maybe I'm to narrow minded.\n\n> \n> I know we don't need to implement support for multiple cameras right now, I just\n> want to make sure it's possible and thinking about the interface and the\n> scenarios help me picture how they'd work.\n> \n> Thoughts?\n> \n> Thanks,\n> Nícolas\n> \n> >\n> > I'm sure there are many solutions to this problem and maybe someone more\n> > familiar with gtest knows of a neat way to piggyback on its\n> > infrastrcuture? Since I know very little about the gtest framework the\n> > following is just an idea and I'm happy to yield.\n> >\n> > How about using a singleton class that carries arguments constructed\n> > from cli arguments (camera id) before the gtest are executed. Each test\n> > case could then consult this singleton if it needs to access parameters.\n> > I think you could take inspiration from the libcamera CameraManager\n> > class.\n> >\n> > This could (at lest for now) imply keeping the cam option parser. If so\n> > I think it would be nice if somewhen before we integrate this work that\n> > the gtest filter string could be supplied as an argument using that\n> > parser. Of course if we can do it the other way around and add arguments\n> > to the gtest parser that would be even cooler as it would be less code\n> > for us to maintain ;-)\n> >\n> > > \n> > > Thanks,\n> > > Nícolas\n> > > \n> > > >\n> > > > > \t}\n> > > > > \n> > > > > -\tif (camera_->acquire()) {\n> > > > > -\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > -\t\treturn -EINVAL;\n> > > > > -\t}\n> > > > > -\n> > > > > -\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> > > > > -\n> > > > > \treturn 0;\n> > > > > }\n> > > > > \n> > > > > void Harness::listCameras()\n> > > > > {\n> > > > > -\tfor (const std::shared_ptr &cam : cm_->cameras())\n> > > > > -\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> > > > > +\tfor (const std::shared_ptr &c : cm_->cameras())\n> > > > > +\t\tstd::cout << \"- \" << c.get()->id() << std::endl;\n> > > > > }\n> > > > > \n> > > > > static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > > > > {\n> > > > > \tOptionsParser parser;\n> > > > > -\tparser.addOption(OptCamera, OptionString,\n> > > > > -\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> > > > > -\t\t\t ArgumentRequired, \"camera\");\n> > > > > \tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> > > > > \t\t\t \"help\");\n> > > > > \n> > > > > @@ -133,6 +100,31 @@ static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> > > > > \treturn 0;\n> > > > > }\n> > > > > \n> > > > > +/*\n> > > > > + * Make asserts act like exceptions, otherwise they only fail (or skip) the\n> > > > > + * current function. From gtest documentation:\n> > > > > + * https://google.github.io/googletest/advanced.html#asserting-on-subroutines-with-an-exception\n> > > > > + */\n> > > > > +class ThrowListener : public testing::EmptyTestEventListener {\n> > > > > +\tvoid OnTestPartResult(const testing::TestPartResult& result) override {\n> > > > > +\t\tif (result.type() == testing::TestPartResult::kFatalFailure\n> > > > > +\t\t || result.type() == testing::TestPartResult::kSkip) {\n> > > > > +\t\t\tthrow testing::AssertionException(result);\n> > > > > +\t\t}\n> > > > > +\t}\n> > > > > +};\n> > > > > +\n> > > > > +\n> > > > > +void Harness::registerTests() {\n> > > > > +\tstd::map roles = {{Raw, \"Raw\"},\n> > > > > +\t\t\t\t\t\t {StillCapture, \"Still\"},\n> > > > > +\t\t\t\t\t\t {VideoRecording, \"Video\"},\n> > > > > +\t\t\t\t\t\t {Viewfinder, \"Viewfinder\"}};\n> > > > > +\tstd::vector requests = {1, 2, 3, 5, 8, 13, 21, 34, 55, 89};\n> > > > > +\n> > > > > +\tregisterSingleStreamTests(cameras_, roles, requests);\n> > > > > +}\n> > > > > +\n> > > > > int main(int argc, char **argv)\n> > > > > {\n> > > > > \tOptionsParser::Options options;\n> > > > > @@ -143,6 +135,13 @@ int main(int argc, char **argv)\n> > > > > \t\treturn EXIT_FAILURE;\n> > > > > \n> > > > > \tHarness harness(options);\n> > > > > +\tret = harness.init();\n> > > > > +\tif (ret)\n> > > > > +\t\treturn ret;\n> > > > > +\n> > > > > +\tharness.registerTests();\n> > > > > \n> > > > > -\treturn harness.exec() ? EXIT_FAILURE : EXIT_SUCCESS;\n> > > > > +\t::testing::InitGoogleTest(&argc, argv);\n> > > > > +\ttesting::UnitTest::GetInstance()->listeners().Append(new ThrowListener);\n> > > > > +\treturn RUN_ALL_TESTS();\n> > > > > }\n> > > > > diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> > > > > index a2bfcceb1259..704bc18af3e1 100644\n> > > > > --- a/src/lc-compliance/meson.build\n> > > > > +++ b/src/lc-compliance/meson.build\n> > > > > @@ -18,10 +18,13 @@ lc_compliance_sources = files([\n> > > > > 'single_stream.cpp',\n> > > > > ])\n> > > > > \n> > > > > +gtest_dep = dependency('gtest')\n> > > > > +\n> > > > > lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> > > > > dependencies : [\n> > > > > libatomic,\n> > > > > libcamera_dep,\n> > > > > libevent,\n> > > > > + gtest_dep,\n> > > > > ],\n> > > > > install : true)\n> > > > > diff --git a/src/lc-compliance/simple_capture.cpp b/src/lc-compliance/simple_capture.cpp\n> > > > > index f90fe6d0f9aa..7731eb16f8c2 100644\n> > > > > --- a/src/lc-compliance/simple_capture.cpp\n> > > > > +++ b/src/lc-compliance/simple_capture.cpp\n> > > > > @@ -5,6 +5,8 @@\n> > > > > * simple_capture.cpp - Simple capture helper\n> > > > > */\n> > > > > \n> > > > > +#include \n> > > > > +\n> > > > > #include \"simple_capture.h\"\n> > > > > \n> > > > > using namespace libcamera;\n> > > > > @@ -20,38 +22,34 @@ SimpleCapture::~SimpleCapture()\n> > > > > \tstop();\n> > > > > }\n> > > > > \n> > > > > -Results::Result SimpleCapture::configure(StreamRole role)\n> > > > > +void SimpleCapture::configure(StreamRole role)\n> > > > > {\n> > > > > \tconfig_ = camera_->generateConfiguration({ role });\n> > > > > \n> > > > > -\tif (!config_)\n> > > > > -\t\treturn { Results::Skip, \"Role not supported by camera\" };\n> > > > > +\tif (!config_) {\n> > > > > +\t\tstd::cout << \"Role not supported by camera\" << std::endl;\n> > > > > +\t\tGTEST_SKIP();\n> > > > > +\t}\n> > > > > \n> > > > > \tif (config_->validate() != CameraConfiguration::Valid) {\n> > > > > \t\tconfig_.reset();\n> > > > > -\t\treturn { Results::Fail, \"Configuration not valid\" };\n> > > > > +\t\tFAIL() << \"Configuration not valid\";\n> > > > > \t}\n> > > > > \n> > > > > \tif (camera_->configure(config_.get())) {\n> > > > > \t\tconfig_.reset();\n> > > > > -\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> > > > > +\t\tFAIL() << \"Failed to configure camera\";\n> > > > > \t}\n> > > > > -\n> > > > > -\treturn { Results::Pass, \"Configure camera\" };\n> > > > > }\n> > > > > \n> > > > > -Results::Result SimpleCapture::start()\n> > > > > +void SimpleCapture::start()\n> > > > > {\n> > > > > \tStream *stream = config_->at(0).stream();\n> > > > > -\tif (allocator_->allocate(stream) < 0)\n> > > > > -\t\treturn { Results::Fail, \"Failed to allocate buffers\" };\n> > > > > +\tASSERT_GE(allocator_->allocate(stream), 0) << \"Failed to allocate buffers\";\n> > > > > \n> > > > > -\tif (camera_->start())\n> > > > > -\t\treturn { Results::Fail, \"Failed to start camera\" };\n> > > > > +\tASSERT_TRUE(!camera_->start()) << \"Failed to start camera\";\n> > > > > \n> > > > > \tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> > > > > -\n> > > > > -\treturn { Results::Pass, \"Started camera\" };\n> > > > > }\n> > > > > \n> > > > > void SimpleCapture::stop()\n> > > > > @@ -77,22 +75,19 @@ SimpleCaptureBalanced::SimpleCaptureBalanced(std::shared_ptr camera)\n> > > > > {\n> > > > > }\n> > > > > \n> > > > > -Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > > +void SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > > {\n> > > > > -\tResults::Result ret = start();\n> > > > > -\tif (ret.first != Results::Pass)\n> > > > > -\t\treturn ret;\n> > > > > +\tstart();\n> > > > > \n> > > > > \tStream *stream = config_->at(0).stream();\n> > > > > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > > > > \n> > > > > \t/* No point in testing less requests then the camera depth. */\n> > > > > \tif (buffers.size() > numRequests) {\n> > > > > -\t\t/* Cache buffers.size() before we destroy it in stop() */\n> > > > > -\t\tint buffers_size = buffers.size();\n> > > > > -\n> > > > > -\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers_size)\n> > > > > -\t\t\t+ \" requests, can't test only \" + std::to_string(numRequests) };\n> > > > > +\t\tstd::cout << \"Camera needs \" + std::to_string(buffers.size())\n> > > > > +\t\t\t+ \" requests, can't test only \"\n> > > > > +\t\t\t+ std::to_string(numRequests) << std::endl;\n> > > > > +\t\tGTEST_SKIP();\n> > > > > \t}\n> > > > > \n> > > > > \tqueueCount_ = 0;\n> > > > > @@ -103,14 +98,11 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > > \tstd::vector> requests;\n> > > > > \tfor (const std::unique_ptr &buffer : buffers) {\n> > > > > \t\tstd::unique_ptr request = camera_->createRequest();\n> > > > > -\t\tif (!request)\n> > > > > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > > > > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > > > > \n> > > > > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > > > > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > > > > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > > > > \n> > > > > -\t\tif (queueRequest(request.get()) < 0)\n> > > > > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > > > > +\t\tASSERT_GE(queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > > > > \n> > > > > \t\trequests.push_back(std::move(request));\n> > > > > \t}\n> > > > > @@ -121,12 +113,7 @@ Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> > > > > \tstop();\n> > > > > \tdelete loop_;\n> > > > > \n> > > > > -\tif (captureCount_ != captureLimit_)\n> > > > > -\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) +\n> > > > > -\t\t\t\" request, wanted \" + std::to_string(captureLimit_) };\n> > > > > -\n> > > > > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > > > > -\t\tstd::to_string(numRequests) + \" requests\" };\n> > > > > +\tASSERT_EQ(captureCount_, captureLimit_);\n> > > > > }\n> > > > > \n> > > > > int SimpleCaptureBalanced::queueRequest(Request *request)\n> > > > > @@ -158,11 +145,9 @@ SimpleCaptureUnbalanced::SimpleCaptureUnbalanced(std::shared_ptr camera)\n> > > > > {\n> > > > > }\n> > > > > \n> > > > > -Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > > +void SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > > {\n> > > > > -\tResults::Result ret = start();\n> > > > > -\tif (ret.first != Results::Pass)\n> > > > > -\t\treturn ret;\n> > > > > +\tstart();\n> > > > > \n> > > > > \tStream *stream = config_->at(0).stream();\n> > > > > \tconst std::vector> &buffers = allocator_->buffers(stream);\n> > > > > @@ -174,14 +159,11 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > > \tstd::vector> requests;\n> > > > > \tfor (const std::unique_ptr &buffer : buffers) {\n> > > > > \t\tstd::unique_ptr request = camera_->createRequest();\n> > > > > -\t\tif (!request)\n> > > > > -\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > > > > +\t\tASSERT_TRUE(request) << \"Can't create request\";\n> > > > > \n> > > > > -\t\tif (request->addBuffer(stream, buffer.get()))\n> > > > > -\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > > > > +\t\tASSERT_FALSE(request->addBuffer(stream, buffer.get())) << \"Can't set buffer for request\";\n> > > > > \n> > > > > -\t\tif (camera_->queueRequest(request.get()) < 0)\n> > > > > -\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > > > > +\t\tASSERT_GE(camera_->queueRequest(request.get()), 0) << \"Failed to queue request\";\n> > > > > \n> > > > > \t\trequests.push_back(std::move(request));\n> > > > > \t}\n> > > > > @@ -192,7 +174,7 @@ Results::Result SimpleCaptureUnbalanced::capture(unsigned int numRequests)\n> > > > > \tstop();\n> > > > > \tdelete loop_;\n> > > > > \n> > > > > -\treturn { status ? Results::Fail : Results::Pass, \"Unbalanced capture of \" + std::to_string(numRequests) + \" requests\" };\n> > > > > +\tASSERT_FALSE(status);\n> > > > > }\n> > > > > \n> > > > > void SimpleCaptureUnbalanced::requestComplete(Request *request)\n> > > > > diff --git a/src/lc-compliance/simple_capture.h b/src/lc-compliance/simple_capture.h\n> > > > > index d9de53fb63a3..0f8465083456 100644\n> > > > > --- a/src/lc-compliance/simple_capture.h\n> > > > > +++ b/src/lc-compliance/simple_capture.h\n> > > > > @@ -17,13 +17,13 @@\n> > > > > class SimpleCapture\n> > > > > {\n> > > > > public:\n> > > > > -\tResults::Result configure(libcamera::StreamRole role);\n> > > > > +\tvoid configure(libcamera::StreamRole role);\n> > > > > \n> > > > > protected:\n> > > > > \tSimpleCapture(std::shared_ptr camera);\n> > > > > \tvirtual ~SimpleCapture();\n> > > > > \n> > > > > -\tResults::Result start();\n> > > > > +\tvoid start();\n> > > > > \tvoid stop();\n> > > > > \n> > > > > \tvirtual void requestComplete(libcamera::Request *request) = 0;\n> > > > > @@ -40,7 +40,7 @@ class SimpleCaptureBalanced : public SimpleCapture\n> > > > > public:\n> > > > > \tSimpleCaptureBalanced(std::shared_ptr camera);\n> > > > > \n> > > > > -\tResults::Result capture(unsigned int numRequests);\n> > > > > +\tvoid capture(unsigned int numRequests);\n> > > > > \n> > > > > private:\n> > > > > \tint queueRequest(libcamera::Request *request);\n> > > > > @@ -56,7 +56,7 @@ class SimpleCaptureUnbalanced : public SimpleCapture\n> > > > > public:\n> > > > > \tSimpleCaptureUnbalanced(std::shared_ptr camera);\n> > > > > \n> > > > > -\tResults::Result capture(unsigned int numRequests);\n> > > > > +\tvoid capture(unsigned int numRequests);\n> > > > > \n> > > > > private:\n> > > > > \tvoid requestComplete(libcamera::Request *request) override;\n> > > > > diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> > > > > index 8318b42f42d6..eb6a6f305826 100644\n> > > > > --- a/src/lc-compliance/single_stream.cpp\n> > > > > +++ b/src/lc-compliance/single_stream.cpp\n> > > > > @@ -7,91 +7,94 @@\n> > > > > \n> > > > > #include \n> > > > > \n> > > > > +#include \n> > > > > +\n> > > > > #include \"simple_capture.h\"\n> > > > > #include \"tests.h\"\n> > > > > \n> > > > > using namespace libcamera;\n> > > > > \n> > > > > -Results::Result testRequestBalance(std::shared_ptr camera,\n> > > > > -\t\t\t\t StreamRole role, unsigned int startCycles,\n> > > > > -\t\t\t\t unsigned int numRequests)\n> > > > > -{\n> > > > > -\tSimpleCaptureBalanced capture(camera);\n> > > > > +class SingleStream : public testing::Test {\n> > > > > +public:\n> > > > > +\texplicit SingleStream(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > > +\t\t: camera_(camera), role_(role), numRequests_(numRequests) {}\n> > > > > \n> > > > > -\tResults::Result ret = capture.configure(role);\n> > > > > -\tif (ret.first != Results::Pass)\n> > > > > -\t\treturn ret;\n> > > > > +protected:\n> > > > > +\tstd::shared_ptr camera_;\n> > > > > +\tStreamRole role_;\n> > > > > +\tunsigned int numRequests_;\n> > > > > +};\n> > > > > \n> > > > > -\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> > > > > -\t\tret = capture.capture(numRequests);\n> > > > > -\t\tif (ret.first != Results::Pass)\n> > > > > -\t\t\treturn ret;\n> > > > > -\t}\n> > > > > +/*\n> > > > > + * Test single capture cycles\n> > > > > + *\n> > > > > + * Makes sure the camera completes the exact number of requests queued. Example\n> > > > > + * failure is a camera that needs N+M requests queued to complete N requests to\n> > > > > + * the application.\n> > > > > + */\n> > > > > +class BalancedSingleCycle : public SingleStream {\n> > > > > +public:\n> > > > > +\texplicit BalancedSingleCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > > > \n> > > > > -\treturn { Results::Pass, \"Balanced capture of \" +\n> > > > > -\t\tstd::to_string(numRequests) + \" requests with \" +\n> > > > > -\t\tstd::to_string(startCycles) + \" start cycles\" };\n> > > > > -}\n> > > > > +\tvoid TestBody() override {\n> > > > > +\t\tSimpleCaptureBalanced capture(camera_);\n> > > > > \n> > > > > -Results::Result testRequestUnbalance(std::shared_ptr camera,\n> > > > > -\t\t\t\t StreamRole role, unsigned int numRequests)\n> > > > > -{\n> > > > > -\tSimpleCaptureUnbalanced capture(camera);\n> > > > > +\t\tcapture.configure(role_);\n> > > > > \n> > > > > -\tResults::Result ret = capture.configure(role);\n> > > > > -\tif (ret.first != Results::Pass)\n> > > > > -\t\treturn ret;\n> > > > > +\t\tcapture.capture(numRequests_);\n> > > > > +\t};\n> > > > > +};\n> > > > > \n> > > > > -\treturn capture.capture(numRequests);\n> > > > > -}\n> > > > > +/*\n> > > > > + * Test multiple start/stop cycles\n> > > > > + *\n> > > > > + * Makes sure the camera supports multiple start/stop cycles. Example failure is\n> > > > > + * a camera that does not clean up correctly in its error path but is only\n> > > > > + * tested by single-capture applications.\n> > > > > + */\n> > > > > +class BalancedMultiCycle : public SingleStream {\n> > > > > +public:\n> > > > > +\texplicit BalancedMultiCycle(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > > > \n> > > > > -Results testSingleStream(std::shared_ptr camera)\n> > > > > -{\n> > > > > -\tstatic const std::vector> roles = {\n> > > > > -\t\t{ \"raw\", Raw },\n> > > > > -\t\t{ \"still\", StillCapture },\n> > > > > -\t\t{ \"video\", VideoRecording },\n> > > > > -\t\t{ \"viewfinder\", Viewfinder },\n> > > > > +\tvoid TestBody() override {\n> > > > > +\t\tunsigned int numRepeats = 3;\n> > > > > +\n> > > > > +\t\tSimpleCaptureBalanced capture(camera_);\n> > > > > +\n> > > > > +\t\tcapture.configure(role_);\n> > > > > +\n> > > > > +\t\tfor (unsigned int starts = 0; starts < numRepeats; starts++)\n> > > > > +\t\t\tcapture.capture(numRequests_);\n> > > > > \t};\n> > > > > -\tstatic const std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> > > > > -\n> > > > > -\tResults results(numRequests.size() * roles.size() * 3);\n> > > > > -\n> > > > > -\tfor (const auto &role : roles) {\n> > > > > -\t\tstd::cout << \"= Test role \" << role.first << std::endl;\n> > > > > -\t\t/*\n> > > > > -\t\t * Test single capture cycles\n> > > > > -\t\t *\n> > > > > -\t\t * Makes sure the camera completes the exact number of requests queued.\n> > > > > -\t\t * Example failure is a camera that needs N+M requests queued to\n> > > > > -\t\t * complete N requests to the application.\n> > > > > -\t\t */\n> > > > > -\t\tstd::cout << \"* Test single capture cycles\" << std::endl;\n> > > > > -\t\tfor (unsigned int num : numRequests)\n> > > > > -\t\t\tresults.add(testRequestBalance(camera, role.second, 1, num));\n> > > > > -\n> > > > > -\t\t/*\n> > > > > -\t\t * Test multiple start/stop cycles\n> > > > > -\t\t *\n> > > > > -\t\t * Makes sure the camera supports multiple start/stop cycles.\n> > > > > -\t\t * Example failure is a camera that does not clean up correctly in its\n> > > > > -\t\t * error path but is only tested by single-capture applications.\n> > > > > -\t\t */\n> > > > > -\t\tstd::cout << \"* Test multiple start/stop cycles\" << std::endl;\n> > > > > -\t\tfor (unsigned int num : numRequests)\n> > > > > -\t\t\tresults.add(testRequestBalance(camera, role.second, 3, num));\n> > > > > -\n> > > > > -\t\t/*\n> > > > > -\t\t * Test unbalanced stop\n> > > > > -\t\t *\n> > > > > -\t\t * Makes sure the camera supports a stop with requests queued.\n> > > > > -\t\t * Example failure is a camera that does not handle cancelation\n> > > > > -\t\t * of buffers coming back from the video device while stopping.\n> > > > > -\t\t */\n> > > > > -\t\tstd::cout << \"* Test unbalanced stop\" << std::endl;\n> > > > > -\t\tfor (unsigned int num : numRequests)\n> > > > > -\t\t\tresults.add(testRequestUnbalance(camera, role.second, num));\n> > > > > -\t}\n> > > > > -\n> > > > > -\treturn results;\n> > > > > +};\n> > > > > +\n> > > > > +/*\n> > > > > + * Test unbalanced stop\n> > > > > + *\n> > > > > + * Makes sure the camera supports a stop with requests queued. Example failure\n> > > > > + * is a camera that does not handle cancelation of buffers coming back from the\n> > > > > + * video device while stopping.\n> > > > > + */\n> > > > > +class Unbalanced : public SingleStream {\n> > > > > +public:\n> > > > > +\texplicit Unbalanced(std::shared_ptr camera, StreamRole role, unsigned int numRequests)\n> > > > > +\t\t: SingleStream(camera, role, numRequests) {}\n> > > > > +\n> > > > > +\tvoid TestBody() override {\n> > > > > +\t\tSimpleCaptureUnbalanced capture(camera_);\n> > > > > +\n> > > > > +\t\tcapture.configure(role_);\n> > > > > +\n> > > > > +\t\tcapture.capture(numRequests_);\n> > > > > +\t};\n> > > > > +};\n> > > > > +\n> > > > > +void registerSingleStreamTests(std::vector> cameras,\n> > > > > +\t\tstd::map roles, std::vector requests)\n> > > > > +{\n> > > > > +\tREGISTER_TESTS(SingleStream, BalancedSingleCycle, cameras, roles, requests);\n> > > > > +\tREGISTER_TESTS(SingleStream, BalancedMultiCycle, cameras, roles, requests);\n> > > > > +\tREGISTER_TESTS(SingleStream, Unbalanced, cameras, roles, requests);\n> > > > > }\n> > > > > diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> > > > > index 396605214e4b..6d5a8b88c287 100644\n> > > > > --- a/src/lc-compliance/tests.h\n> > > > > +++ b/src/lc-compliance/tests.h\n> > > > > @@ -11,6 +11,19 @@\n> > > > > \n> > > > > #include \"results.h\"\n> > > > > \n> > > > > -Results testSingleStream(std::shared_ptr camera);\n> > > > > +void registerSingleStreamTests(std::vector> cameras,\n> > > > > +\t\tstd::map roles, std::vector requests);\n> > > > > +\n> > > > > +#define REGISTER_TESTS(testsuite, testcase, cameras, roles, requests) \\\n> > > > > +for (auto c : cameras) { \\\n> > > > > +\tfor (auto [r, rn] : roles) { \\\n> > > > > +\t\tfor (auto rq : requests) { \\\n> > > > > +\t\t\ttesting::RegisterTest( \\\n> > > > > +\t\t\t\t#testsuite, (std::string(#testcase) + \"/\" + c->id() + \"/\" + rn + \"/\" + std::to_string(rq)).c_str(), \\\n> > > > > +\t\t\t\tnullptr, nullptr, __FILE__, __LINE__, \\\n> > > > > +\t\t\t\t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > +\t\t} \\\n> > > > > +\t} \\\n> > > > > +}\n> > > > > \n> > > > > #endif /* __LC_COMPLIANCE_TESTS_H__ */\n> > > > > -- \n> > > > > 2.31.1\n> > > > > \n> > > >\n> > > > --\n> > > > Regards,\n> > > > Niklas Söderlund\n> > > >\n> > > > --\n> > > > To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.\n> > > \n> > > \n> >\n> > --\n> > Regards,\n> > Niklas Söderlund\n> >\n> > --\n> > To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4D2FBBF839\n\tfor ;\n\tMon, 10 May 2021 21:04:18 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AFFD168911;\n\tMon, 10 May 2021 23:04:17 +0200 (CEST)","from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com\n\t[IPv6:2a00:1450:4864:20::22c])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CB3D2602BE\n\tfor ;\n\tMon, 10 May 2021 23:04:15 +0200 (CEST)","by mail-lj1-x22c.google.com with SMTP id v5so22505749ljg.12\n\tfor ;\n\tMon, 10 May 2021 14:04:15 -0700 (PDT)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tu10sm2383407lfq.243.2021.05.10.14.04.13\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tMon, 10 May 2021 14:04:14 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"FXWWewlO\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=2P2rGMTFHheWpebGfbQRVp67GJKRmGo0o/w37C/GCDM=;\n\tb=FXWWewlOptPcbovsPgf/nQVHr7WlMpbXFR4Zj04dv3IEwfUplfAT8UnrVJ5GPHNVO7\n\tZgssODvSsRsssdv+0kIECppgCoWAcGGUIwI07wFtyvvbZ8PZx5uEY2wh+jAQ4fplrpxQ\n\txEO3yjxxFo/E7fgeN9tFFkM0orxqLEuEYqyPcGA8FpORMuRE8cA7F9NBy1fNwjPbIjgp\n\tHFbXnRdjhKQo3bUxHcjMn+J5dI63aznsaTwAqAb4CEUZ9rt2lBnyCErmRJXSyuwQaynu\n\tW7H8iTPc7ZOzFYyB2QDqsf5PGy9vsNQgCS16ywOgyb8+WFQyvdkngclJDu0p2+EGHtY/\n\tMnFQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=2P2rGMTFHheWpebGfbQRVp67GJKRmGo0o/w37C/GCDM=;\n\tb=lCQ4Ti8fUSC5cDJ6PMLN+UtemTPdO5LnmDdApP+dq+KhBycG3lIN/JaGx4TCG2Cfal\n\tEdNyKgLCuTH/P71AdxcIGHEEYfqoo0IjFpRBEOjZzKV6QKlocL1XFcuTiVA4txas6sES\n\tEoUXAmtHrTv6Vp93MWFEoxngU1N2+E9jT7Uzr0ax6pVqLQxqDcF9PyLVc2UIiiOHEJtH\n\tobtqDwQJhzt6UpLfZaowXpPiTKpOUI3udGSJrKUFSatzpFfi3j7WO4GQBrp4sEAjrGVw\n\tTQUnY0V1kgfZldAqDmi0450PkrApEUvrzKA/BaFvTrMrCf09aUAqSUDk6cbCAw9nODFw\n\thmeg==","X-Gm-Message-State":"AOAM5332BRu+ESDfLntgD4GDAZoltUt6NGwAWSKVj1rwsGSb1S4h0TIi\n\tJd6JU12fQp4E1WnkB+7J93BN3A==","X-Google-Smtp-Source":"ABdhPJxXF7OAlbUjve2Y7sje1eECrHaKzMXWVsMengBGmFiuEI3PXp8q6X2z7mmcA195RxgFHAlZ9A==","X-Received":"by 2002:a2e:8903:: with SMTP id d3mr5642822lji.373.1620680654784;\n\tMon, 10 May 2021 14:04:14 -0700 (PDT)","Date":"Mon, 10 May 2021 23:04:13 +0200","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"=?iso-8859-1?q?N=EDcolas_F=2E_R=2E_A=2E?= Prado ","Message-ID":"","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com,\n\t=?iso-8859-1?q?Andr=E9?= Almeida ","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16904,"web_url":"https://patchwork.libcamera.org/comment/16904/","msgid":"","date":"2021-05-11T17:17:56","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":84,"url":"https://patchwork.libcamera.org/api/people/84/","name":"Nícolas F. R. A. Prado","email":"nfraprado@collabora.com"},"content":"Hi Niklas,\n\nEm 2021-05-10 18:04, Niklas Söderlund escreveu:\n\n> Hi Nícolas,\n>\n> Thanks for your work on this. I really think gtest is a step in the\n> right direction!\n>\n> On 2021-05-07 12:52:38 -0300, Nícolas F. R. A. Prado wrote:\n> > Hi Niklas,\n> > \n> > Em 2021-05-07 06:11, Niklas Söderlund escreveu:\n> > \n> > > Hi Nícolas,\n> > >\n> > > On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > > > Hi Niklas,\n> > > > \n> > > > thanks for the feedback.\n> > > > \n> > > > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > > > \n> > > > > Hi Nícolas,\n> > > > >\n> > > > > Thanks for your work!\n> > > > >\n> > > > > I really like how gtest helps make the actual tests to be easier to\n> > > > > read.\n> > > > >\n> > > > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > > > \n> > > > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > > > ---\n> > > > > > Changes in v2:\n> > > > > > - Changed from static to dynamic test registration\n> > > > > > - Removed -c flag\n> > > > > > \n> > > > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > > > apparently aren't being deleted on time, which causes:\n> > > > > > \n> > > > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > > > \tSegmentation fault (core dumped)\n> > > > > > \n> > > > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > > > \n> > > > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > > \n> > > > > > Not sure how to solve this. Any tip would be welcome.\n> > > > >\n> > > > > I will leave this for now as I have a different worry about\n> > > > > REGISTER_TESTS() design below.\n> > > > >\n> > > > > > \n> > > > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > > > src/lc-compliance/meson.build | 3 +\n> > > > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > > > src/lc-compliance/tests.h | 15 ++-\n> > > > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > > > \n> > > > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > > > index 54cee54aa978..add0d7729aec 100644\n> > > > > > --- a/src/lc-compliance/main.cpp\n> > > > > > +++ b/src/lc-compliance/main.cpp\n> > > > > > @@ -9,6 +9,8 @@\n> > > > > > #include \n> > > > > > #include \n> > > > > > \n> > > > > > +#include \n> > > > > > +\n> > > > > > #include \n> > > > > > \n> > > > > > #include \"../cam/options.h\"\n> > > > > > @@ -17,7 +19,6 @@\n> > > > > > using namespace libcamera;\n> > > > > > \n> > > > > > enum {\n> > > > > > -\tOptCamera = 'c',\n> > > > > > \tOptHelp = 'h',\n> > > > > > };\n> > > > > > \n> > > > > > @@ -28,14 +29,15 @@ public:\n> > > > > > \t~Harness();\n> > > > > > \n> > > > > > \tint exec();\n> > > > > > +\tint init();\n> > > > > > +\tvoid registerTests();\n> > > > > > \n> > > > > > private:\n> > > > > > -\tint init();\n> > > > > > \tvoid listCameras();\n> > > > > > \n> > > > > > \tOptionsParser::Options options_;\n> > > > > > \tstd::unique_ptr cm_;\n> > > > > > -\tstd::shared_ptr camera_;\n> > > > > > +\tstd::vector> cameras_;\n> > > > > > };\n> > > > > > \n> > > > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > > > \n> > > > > > Harness::~Harness()\n> > > > > > {\n> > > > > > -\tif (camera_) {\n> > > > > > -\t\tcamera_->release();\n> > > > > > -\t\tcamera_.reset();\n> > > > > > +\tfor (auto &c : cameras_) {\n> > > > > > +\t\tc->release();\n> > > > > > +\t\tc.reset();\n> > > > > > \t}\n> > > > > > \n> > > > > > \tcm_->stop();\n> > > > > > }\n> > > > > > \n> > > > > > -int Harness::exec()\n> > > > > > -{\n> > > > > > -\tint ret = init();\n> > > > > > -\tif (ret)\n> > > > > > -\t\treturn ret;\n> > > > > > -\n> > > > > > -\tstd::vector results;\n> > > > > > -\n> > > > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > > > -\n> > > > > > -\tfor (const Results &result : results) {\n> > > > > > -\t\tret = result.summary();\n> > > > > > -\t\tif (ret)\n> > > > > > -\t\t\treturn ret;\n> > > > > > -\t}\n> > > > > > -\n> > > > > > -\treturn 0;\n> > > > > > -}\n> > > > > > -\n> > > > > > int Harness::init()\n> > > > > > {\n> > > > > > \tint ret = cm_->start();\n> > > > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > > > \t\treturn ret;\n> > > > > > \t}\n> > > > > > \n> > > > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > > > -\t\tlistCameras();\n> > > > > > -\t\treturn -ENODEV;\n> > > > > > -\t}\n> > > > > > -\n> > > > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > > > -\tcamera_ = cm_->get(cameraId);\n> > > > > > -\tif (!camera_) {\n> > > > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > > > -\t\tlistCameras();\n> > > > > > -\t\treturn -ENODEV;\n> > > > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > > > +\t\tif (cam->acquire()) {\n> > > > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > > +\t\t\treturn -EINVAL;\n> > > > > > +\t\t}\n> > > > > > +\t\tcameras_.push_back(cam);\n> > > > >\n> > > > > I don't like this, I think we need to retain the ability to select which\n> > > > > camera to test. And further down the line maybe allow more then one\n> > > > > camera to be selected if we want to test concurrent use of two specific\n> > > > > ones. But for the work in this series I think you can ignore multiple\n> > > > > cameras.\n> > > > \n> > > > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > > > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > > > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > > > all cameras, only the ones that I will actually test (and in this code there's a\n> > > > big issue where if a single camera is already busy, even if it won't be tested,\n> > > > lc-compliance as a whole fails).\n> > >\n> > > Ahh I understand.\n> > >\n> > > > \n> > > > >\n> > > > > If I understand the coverletter correctly the correct thing here the cli\n> > > > > arguments should be feed to gtest directly to be able to filter on what\n> > > > > tests are run. Is there support in getst to also pass arguments to\n> > > > > tests? Maybe we could remove the need to parse arguments ourself\n> > > > > completely?\n> > > > \n> > > > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > > > could solve both problems (this and the one above) like this:\n> > > > \n> > > > - In the main(), query the id of each camera from libcamera.\n> > > > - Register each of the tests dynamically in gtest passing as an argument the\n> > > > camera id, rather than a camera pointer (which is the current behavior).\n> > > > - To select which tests to run, and on which camera(s), the user just uses\n> > > > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > > > the camera).\n> > > > - In the startup of each test the camera is acquired for that test, and when\n> > > > it's done it is released.\n> > > > \n> > > > Implications:\n> > > > - We no longer need to parse arguments and can drop that code, as both camera\n> > > > and test are specified directly with gtest's filter.\n> > > > - We only acquire the cameras that we'll actually test and only while they're\n> > > > being tested.\n> > > > - As a downside, a single camera will probably be acquired multiple times during\n> > > > a single run as lc-compliance moves from one test to the next, but I don't\n> > > > think this would be a big issue. (Maybe we could do this at the test suite\n> > > > level instead to greatly decrease the number of times of acquire(), but I'm\n> > > > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > > > the test level, I'll have to test that).\n> > > > \n> > > > What do you think?\n> > >\n> > > I understand what you are trying to do but I see two issues.\n> > >\n> > > 1. It depends on the implicit knowledge of users how to build the filter\n> > > string and is rather unintuitive compared to having two distinct\n> > > arguments, one to select camera and one to express the test filter.\n> > >\n> > > This could possibly be worked around by keeping the cam options code\n> > > and building the gtest filter string internally by injecitng the\n> > > selected camera name in the correct location, but the we have the\n> > > next issue,\n> > >\n> > > 2. I don't think it will be practically possible to express a test\n> > > filter that selects a specific combination of two cameras.\n> > >\n> > > I do think we need to be able to parameterize test cases. If gtest does\n> > > not allow to do so using function arguments there are other ways of\n> > > doing so.\n> > \n> > I think I didn't express myself well. I meant to say that gtest's CLI doesn't\n> > have a flag to allow passing parameters to tests, meaning that it would not be\n> > possible to parametrize the tests from the command line and also remove our\n> > own parser and rely on gtest's.\n>\n> Check, I think we need to keep our own parser for now as we need to be\n> able to parametrize the tests from the CLI.\n>\n> > But gtest does allow test parametrization at runtime and I'm already relying on\n> > it since v2. Let me go a bit more over the way I'm doing this.\n> > \n> > - I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n> > \n> > \tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n> > \n> > - Since we may want to run the same test for all cameras, roles and numRequests,\n> > I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n> > gtest's testing::RegisterTest() with all possible combinations of parameters.\n> > \n> > - Based on the --gtest_filter CLI argument from gtest, it decides which tests\n> > are actually run\n> > \n> > So the key point is that even though I parametrize the tests, I do it for all\n> > combinations in order to later rely on gtest_filter to select which tests to\n> > run.\n>\n> I see, that explains how the parameters get filled. I noticed it but did\n> not dig further, see below for an expansion of my thoughts now that I\n> understand this detail :-)\n>\n> > \n> > Now, I do agree with both of your points. And if we want to make the interface\n> > more intuitive to use (which I also find important), there's no way around\n> > having our own parser.\n> > \n> > And if we're using our own camera flag, then it makes no sense to register tests\n> > for all cameras, we can register tests only with the specified camera.\n>\n> This feels wrong. It feels like tests should be register without a\n> camera component. The camera (and possible other arguments) should come\n> from either arguments [1] or a singleton parameter class [2].\n>\n> 1. This do not seem to be possible with gtests as test parameters needs\n> to be registerd.\n>\n> 2. This would be similar to the global variable used in earlier versions\n> but instead modeled as a class singleton like CameraManager.\n>\n> Maybe there are other options tho.\n>\n> > \n> > I think here it would be helpful to think if the interface suits our problem.\n> > To me we're looking at having:\n> > \n> > \tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n>\n> This matches my view as well.\n>\n> > \n> > to only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\n> > the camera flag is used to only register tests with that camera, and the filter\n> > is passed as --gtest_filter, possibly a bit adapted, to gtest.\n> > \n> > And if we imagine we have a test ConcurrentCameras, which receives two cameras\n> > as parameters, and we call\n> > \n> > \tlc-compliance --camera 'cameraId1,cameraId2'\n>\n> This also looks good. But for now lets focus on one camera, I think it\n> will give us enough headache :-)\n>\n> > \n> > it would test both each of the cameras separately in all current tests which\n> > accept a single camera, like BalancedSingleCycle, as well as test both at the\n> > same time in the ConcurrentCameras test (and we could make only the former or\n> > the latter occur by using the filter parameter).\n>\n> I have not thought enough about this to know if that is a good idea or\n> not. At the top of my head I still tend to think that lc-compliance\n> tests a single 'primary' camera and any additional camera supplied are\n> only used for tests where two (or more) cameras are needed to test some\n> specific feature of the primary camera.\n>\n> That is only the 'primary' camera is testes, the other camera(s) are\n> just there to help/provoke the testing. But maybe I'm to narrow minded.\n\nOkay, so it now comes down to how lc-compliance will get used. Either:\n\n1. lc-compliance will be called to test a single camera, with optionally other\nsecondary cameras\n\n2. lc-compliance will be called to test multiple cameras at once (not\nnecessarily simultaneously, but in a single run), with the default case being\n\"Test all the cameras in this system\".\n\nGiven that the final objective of my internship is to have lc-compliance running\non real boards by KernelCI, to catch any kernel regressions that affect camera\nusage, I'm trying to think how it would work there. (I wonder if this discussion\nshould be happening together with KernelCI to decide on this...)\n\nSo I'm imagining a RockPi4 board with a camera sensor plugged in, running\nlc-compliance to catch any kernel regressions.\n\nThe limitation I see in 1 is that if a system has multiple cameras and want to\nensure that all keep working, it will need to either hardcode all camera ids and\nmanually call lc-compliance once for each one, or have a separate program that\nlists all cameras from libcamera (using 'cam -l' for example) and calls\nlc-compliance with each one. That's to say, running on all cameras is not as\neasy. And then there are multiple reports, one for each camera, instead of a\nsingle one for all cameras, which might be good or bad, not sure.\n\nThat said, since the RockPi4 (and also the Raspberry Pi 4) has a single CSI-2\nconnector, I believe it only supports a single camera anyway, so this shouldn't\nbe a problem for this board, but I'm not sure about others. Phones for example\nseem to have at least two independent cameras...\n\nRegarding the benefit of 1, I'm not sure. If we were to use a global variable\nlike in v1, it would make the code a lot simpler by making the test registration\nstatic. But that doesn't seem to be an option :). I'm not familiar with using a\nSingleton, so I'll take a look at CameraManager. But I assume it wouldn't be\naccessible statically, so the dynamic test registration complexities would have\nto stay.\n\nAlso, just to make sure we're on the same page, by using a Singleton, and again\nassuming it isn't stored in a global variable to be accessible from the tests\nstatically, even though we would eliminate the camera as a parameter for the\ntests, the reference to the singleton itself would need to be added as a\nparameter for the tests.\n\nThanks,\nNícolas\n\n>\n> > \n> > I know we don't need to implement support for multiple cameras right now, I just\n> > want to make sure it's possible and thinking about the interface and the\n> > scenarios help me picture how they'd work.\n> > \n> > Thoughts?\n> > \n> > Thanks,\n> > Nícolas\n> > \n> > >\n> > > I'm sure there are many solutions to this problem and maybe someone more\n> > > familiar with gtest knows of a neat way to piggyback on its\n> > > infrastrcuture? Since I know very little about the gtest framework the\n> > > following is just an idea and I'm happy to yield.\n> > >\n> > > How about using a singleton class that carries arguments constructed\n> > > from cli arguments (camera id) before the gtest are executed. Each test\n> > > case could then consult this singleton if it needs to access parameters.\n> > > I think you could take inspiration from the libcamera CameraManager\n> > > class.\n> > >\n> > > This could (at lest for now) imply keeping the cam option parser. If so\n> > > I think it would be nice if somewhen before we integrate this work that\n> > > the gtest filter string could be supplied as an argument using that\n> > > parser. Of course if we can do it the other way around and add arguments\n> > > to the gtest parser that would be even cooler as it would be less code\n> > > for us to maintain ;-)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8D676BF829\n\tfor ;\n\tTue, 11 May 2021 17:18:36 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EFB7B6891E;\n\tTue, 11 May 2021 19:18:35 +0200 (CEST)","from bhuna.collabora.co.uk (bhuna.collabora.co.uk [46.235.227.227])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 379FD61538\n\tfor ;\n\tTue, 11 May 2021 19:18:35 +0200 (CEST)","from localhost (unknown\n\t[IPv6:2804:14c:1a9:2978:985c:7892:ebcf:7a90])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id E86371F41C92;\n\tTue, 11 May 2021 18:18:32 +0100 (BST)"],"Mime-Version":"1.0","Date":"Tue, 11 May 2021 14:17:56 -0300","Message-Id":"","From":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= ","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t\n\t\n\t","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com, =?utf-8?q?A?=\n\t=?utf-8?q?ndr=C3=A9_Almeida?= ","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16906,"web_url":"https://patchwork.libcamera.org/comment/16906/","msgid":"","date":"2021-05-11T19:06:04","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":84,"url":"https://patchwork.libcamera.org/api/people/84/","name":"Nícolas F. R. A. Prado","email":"nfraprado@collabora.com"},"content":"Hello again,\n\nEm 2021-05-11 14:17, Nícolas F. R. A. Prado escreveu:\n\n> Hi Niklas,\n>\n> Em 2021-05-10 18:04, Niklas Söderlund escreveu:\n>\n> > Hi Nícolas,\n> >\n> > Thanks for your work on this. I really think gtest is a step in the\n> > right direction!\n> >\n> > On 2021-05-07 12:52:38 -0300, Nícolas F. R. A. Prado wrote:\n> > > Hi Niklas,\n> > > \n> > > Em 2021-05-07 06:11, Niklas Söderlund escreveu:\n> > > \n> > > > Hi Nícolas,\n> > > >\n> > > > On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > Hi Niklas,\n> > > > > \n> > > > > thanks for the feedback.\n> > > > > \n> > > > > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > > > > \n> > > > > > Hi Nícolas,\n> > > > > >\n> > > > > > Thanks for your work!\n> > > > > >\n> > > > > > I really like how gtest helps make the actual tests to be easier to\n> > > > > > read.\n> > > > > >\n> > > > > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > > > > \n> > > > > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > > > > ---\n> > > > > > > Changes in v2:\n> > > > > > > - Changed from static to dynamic test registration\n> > > > > > > - Removed -c flag\n> > > > > > > \n> > > > > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > > > > apparently aren't being deleted on time, which causes:\n> > > > > > > \n> > > > > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > > > > \tSegmentation fault (core dumped)\n> > > > > > > \n> > > > > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > > > > \n> > > > > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > > > \n> > > > > > > Not sure how to solve this. Any tip would be welcome.\n> > > > > >\n> > > > > > I will leave this for now as I have a different worry about\n> > > > > > REGISTER_TESTS() design below.\n> > > > > >\n> > > > > > > \n> > > > > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > > > > src/lc-compliance/meson.build | 3 +\n> > > > > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > > > > src/lc-compliance/tests.h | 15 ++-\n> > > > > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > > > > \n> > > > > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > > > > index 54cee54aa978..add0d7729aec 100644\n> > > > > > > --- a/src/lc-compliance/main.cpp\n> > > > > > > +++ b/src/lc-compliance/main.cpp\n> > > > > > > @@ -9,6 +9,8 @@\n> > > > > > > #include \n> > > > > > > #include \n> > > > > > > \n> > > > > > > +#include \n> > > > > > > +\n> > > > > > > #include \n> > > > > > > \n> > > > > > > #include \"../cam/options.h\"\n> > > > > > > @@ -17,7 +19,6 @@\n> > > > > > > using namespace libcamera;\n> > > > > > > \n> > > > > > > enum {\n> > > > > > > -\tOptCamera = 'c',\n> > > > > > > \tOptHelp = 'h',\n> > > > > > > };\n> > > > > > > \n> > > > > > > @@ -28,14 +29,15 @@ public:\n> > > > > > > \t~Harness();\n> > > > > > > \n> > > > > > > \tint exec();\n> > > > > > > +\tint init();\n> > > > > > > +\tvoid registerTests();\n> > > > > > > \n> > > > > > > private:\n> > > > > > > -\tint init();\n> > > > > > > \tvoid listCameras();\n> > > > > > > \n> > > > > > > \tOptionsParser::Options options_;\n> > > > > > > \tstd::unique_ptr cm_;\n> > > > > > > -\tstd::shared_ptr camera_;\n> > > > > > > +\tstd::vector> cameras_;\n> > > > > > > };\n> > > > > > > \n> > > > > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > \n> > > > > > > Harness::~Harness()\n> > > > > > > {\n> > > > > > > -\tif (camera_) {\n> > > > > > > -\t\tcamera_->release();\n> > > > > > > -\t\tcamera_.reset();\n> > > > > > > +\tfor (auto &c : cameras_) {\n> > > > > > > +\t\tc->release();\n> > > > > > > +\t\tc.reset();\n> > > > > > > \t}\n> > > > > > > \n> > > > > > > \tcm_->stop();\n> > > > > > > }\n> > > > > > > \n> > > > > > > -int Harness::exec()\n> > > > > > > -{\n> > > > > > > -\tint ret = init();\n> > > > > > > -\tif (ret)\n> > > > > > > -\t\treturn ret;\n> > > > > > > -\n> > > > > > > -\tstd::vector results;\n> > > > > > > -\n> > > > > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > > > > -\n> > > > > > > -\tfor (const Results &result : results) {\n> > > > > > > -\t\tret = result.summary();\n> > > > > > > -\t\tif (ret)\n> > > > > > > -\t\t\treturn ret;\n> > > > > > > -\t}\n> > > > > > > -\n> > > > > > > -\treturn 0;\n> > > > > > > -}\n> > > > > > > -\n> > > > > > > int Harness::init()\n> > > > > > > {\n> > > > > > > \tint ret = cm_->start();\n> > > > > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > > > > \t\treturn ret;\n> > > > > > > \t}\n> > > > > > > \n> > > > > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > > > > -\t\tlistCameras();\n> > > > > > > -\t\treturn -ENODEV;\n> > > > > > > -\t}\n> > > > > > > -\n> > > > > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > > > > -\tcamera_ = cm_->get(cameraId);\n> > > > > > > -\tif (!camera_) {\n> > > > > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > > > > -\t\tlistCameras();\n> > > > > > > -\t\treturn -ENODEV;\n> > > > > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > > > > +\t\tif (cam->acquire()) {\n> > > > > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > > > +\t\t\treturn -EINVAL;\n> > > > > > > +\t\t}\n> > > > > > > +\t\tcameras_.push_back(cam);\n> > > > > >\n> > > > > > I don't like this, I think we need to retain the ability to select which\n> > > > > > camera to test. And further down the line maybe allow more then one\n> > > > > > camera to be selected if we want to test concurrent use of two specific\n> > > > > > ones. But for the work in this series I think you can ignore multiple\n> > > > > > cameras.\n> > > > > \n> > > > > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > > > > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > > > > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > > > > all cameras, only the ones that I will actually test (and in this code there's a\n> > > > > big issue where if a single camera is already busy, even if it won't be tested,\n> > > > > lc-compliance as a whole fails).\n> > > >\n> > > > Ahh I understand.\n> > > >\n> > > > > \n> > > > > >\n> > > > > > If I understand the coverletter correctly the correct thing here the cli\n> > > > > > arguments should be feed to gtest directly to be able to filter on what\n> > > > > > tests are run. Is there support in getst to also pass arguments to\n> > > > > > tests? Maybe we could remove the need to parse arguments ourself\n> > > > > > completely?\n> > > > > \n> > > > > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > > > > could solve both problems (this and the one above) like this:\n> > > > > \n> > > > > - In the main(), query the id of each camera from libcamera.\n> > > > > - Register each of the tests dynamically in gtest passing as an argument the\n> > > > > camera id, rather than a camera pointer (which is the current behavior).\n> > > > > - To select which tests to run, and on which camera(s), the user just uses\n> > > > > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > > > > the camera).\n> > > > > - In the startup of each test the camera is acquired for that test, and when\n> > > > > it's done it is released.\n> > > > > \n> > > > > Implications:\n> > > > > - We no longer need to parse arguments and can drop that code, as both camera\n> > > > > and test are specified directly with gtest's filter.\n> > > > > - We only acquire the cameras that we'll actually test and only while they're\n> > > > > being tested.\n> > > > > - As a downside, a single camera will probably be acquired multiple times during\n> > > > > a single run as lc-compliance moves from one test to the next, but I don't\n> > > > > think this would be a big issue. (Maybe we could do this at the test suite\n> > > > > level instead to greatly decrease the number of times of acquire(), but I'm\n> > > > > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > > > > the test level, I'll have to test that).\n> > > > > \n> > > > > What do you think?\n> > > >\n> > > > I understand what you are trying to do but I see two issues.\n> > > >\n> > > > 1. It depends on the implicit knowledge of users how to build the filter\n> > > > string and is rather unintuitive compared to having two distinct\n> > > > arguments, one to select camera and one to express the test filter.\n> > > >\n> > > > This could possibly be worked around by keeping the cam options code\n> > > > and building the gtest filter string internally by injecitng the\n> > > > selected camera name in the correct location, but the we have the\n> > > > next issue,\n> > > >\n> > > > 2. I don't think it will be practically possible to express a test\n> > > > filter that selects a specific combination of two cameras.\n> > > >\n> > > > I do think we need to be able to parameterize test cases. If gtest does\n> > > > not allow to do so using function arguments there are other ways of\n> > > > doing so.\n> > > \n> > > I think I didn't express myself well. I meant to say that gtest's CLI doesn't\n> > > have a flag to allow passing parameters to tests, meaning that it would not be\n> > > possible to parametrize the tests from the command line and also remove our\n> > > own parser and rely on gtest's.\n> >\n> > Check, I think we need to keep our own parser for now as we need to be\n> > able to parametrize the tests from the CLI.\n> >\n> > > But gtest does allow test parametrization at runtime and I'm already relying on\n> > > it since v2. Let me go a bit more over the way I'm doing this.\n> > > \n> > > - I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n> > > \n> > > \tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n> > > \n> > > - Since we may want to run the same test for all cameras, roles and numRequests,\n> > > I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n> > > gtest's testing::RegisterTest() with all possible combinations of parameters.\n> > > \n> > > - Based on the --gtest_filter CLI argument from gtest, it decides which tests\n> > > are actually run\n> > > \n> > > So the key point is that even though I parametrize the tests, I do it for all\n> > > combinations in order to later rely on gtest_filter to select which tests to\n> > > run.\n> >\n> > I see, that explains how the parameters get filled. I noticed it but did\n> > not dig further, see below for an expansion of my thoughts now that I\n> > understand this detail :-)\n> >\n> > > \n> > > Now, I do agree with both of your points. And if we want to make the interface\n> > > more intuitive to use (which I also find important), there's no way around\n> > > having our own parser.\n> > > \n> > > And if we're using our own camera flag, then it makes no sense to register tests\n> > > for all cameras, we can register tests only with the specified camera.\n> >\n> > This feels wrong. It feels like tests should be register without a\n> > camera component. The camera (and possible other arguments) should come\n> > from either arguments [1] or a singleton parameter class [2].\n> >\n> > 1. This do not seem to be possible with gtests as test parameters needs\n> > to be registerd.\n> >\n> > 2. This would be similar to the global variable used in earlier versions\n> > but instead modeled as a class singleton like CameraManager.\n> >\n> > Maybe there are other options tho.\n> >\n> > > \n> > > I think here it would be helpful to think if the interface suits our problem.\n> > > To me we're looking at having:\n> > > \n> > > \tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n> >\n> > This matches my view as well.\n> >\n> > > \n> > > to only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\n> > > the camera flag is used to only register tests with that camera, and the filter\n> > > is passed as --gtest_filter, possibly a bit adapted, to gtest.\n> > > \n> > > And if we imagine we have a test ConcurrentCameras, which receives two cameras\n> > > as parameters, and we call\n> > > \n> > > \tlc-compliance --camera 'cameraId1,cameraId2'\n> >\n> > This also looks good. But for now lets focus on one camera, I think it\n> > will give us enough headache :-)\n> >\n> > > \n> > > it would test both each of the cameras separately in all current tests which\n> > > accept a single camera, like BalancedSingleCycle, as well as test both at the\n> > > same time in the ConcurrentCameras test (and we could make only the former or\n> > > the latter occur by using the filter parameter).\n> >\n> > I have not thought enough about this to know if that is a good idea or\n> > not. At the top of my head I still tend to think that lc-compliance\n> > tests a single 'primary' camera and any additional camera supplied are\n> > only used for tests where two (or more) cameras are needed to test some\n> > specific feature of the primary camera.\n> >\n> > That is only the 'primary' camera is testes, the other camera(s) are\n> > just there to help/provoke the testing. But maybe I'm to narrow minded.\n>\n> Okay, so it now comes down to how lc-compliance will get used. Either:\n>\n> 1. lc-compliance will be called to test a single camera, with optionally other\n> secondary cameras\n>\n> 2. lc-compliance will be called to test multiple cameras at once (not\n> necessarily simultaneously, but in a single run), with the default case being\n> \"Test all the cameras in this system\".\n>\n> Given that the final objective of my internship is to have lc-compliance running\n> on real boards by KernelCI, to catch any kernel regressions that affect camera\n> usage, I'm trying to think how it would work there. (I wonder if this discussion\n> should be happening together with KernelCI to decide on this...)\n>\n> So I'm imagining a RockPi4 board with a camera sensor plugged in, running\n> lc-compliance to catch any kernel regressions.\n>\n> The limitation I see in 1 is that if a system has multiple cameras and want to\n> ensure that all keep working, it will need to either hardcode all camera ids and\n> manually call lc-compliance once for each one, or have a separate program that\n> lists all cameras from libcamera (using 'cam -l' for example) and calls\n> lc-compliance with each one. That's to say, running on all cameras is not as\n> easy. And then there are multiple reports, one for each camera, instead of a\n> single one for all cameras, which might be good or bad, not sure.\n>\n> That said, since the RockPi4 (and also the Raspberry Pi 4) has a single CSI-2\n> connector, I believe it only supports a single camera anyway, so this shouldn't\n> be a problem for this board, but I'm not sure about others. Phones for example\n> seem to have at least two independent cameras...\n\nSo, I just read about the Singleton pattern on Wikipedia and checked out the\nCameraManager class (should've done it before replying :) ).\n\nFrom what I can tell, there's a big difference between the CameraManager class\nand a Singleton. The CameraManager just makes sure that there's a single\ninstance of it by saving a reference to it in an internal static variable. A\nSingleton apparently, in addition to doing that, makes its instatiation through\na public static function. This means that not only there's a single instance,\nit's also available from the global scope.\n\nSo I wanted to ask you to clarify which one you meant. If you meant that we\nshould have a class like CameraManager, which just makes sure it's the single\ninstance, my previous comments below are still valid.\n\nOn the other hand, if you meant that we should a have singleton, which is\navailable globally, then it is basically the same thing as having a global\nvariable, only a bit more organized, and my comments below are no longer valid.\nWe could indeed go back to static test registration, since the camera instance\ncould be accessed from any test as something like\n\n\tTestEnvironment::getInstance().camera()\n\nmaking the alternative 1 have the benefit of making the test registration and\ndefinition code simpler.\n\nThanks,\nNícolas\n\n>\n> Regarding the benefit of 1, I'm not sure. If we were to use a global variable\n> like in v1, it would make the code a lot simpler by making the test registration\n> static. But that doesn't seem to be an option :). I'm not familiar with using a\n> Singleton, so I'll take a look at CameraManager. But I assume it wouldn't be\n> accessible statically, so the dynamic test registration complexities would have\n> to stay.\n>\n> Also, just to make sure we're on the same page, by using a Singleton, and again\n> assuming it isn't stored in a global variable to be accessible from the tests\n> statically, even though we would eliminate the camera as a parameter for the\n> tests, the reference to the singleton itself would need to be added as a\n> parameter for the tests.\n>\n> Thanks,\n> Nícolas\n>\n> >\n> > > \n> > > I know we don't need to implement support for multiple cameras right now, I just\n> > > want to make sure it's possible and thinking about the interface and the\n> > > scenarios help me picture how they'd work.\n> > > \n> > > Thoughts?\n> > > \n> > > Thanks,\n> > > Nícolas\n> > > \n> > > >\n> > > > I'm sure there are many solutions to this problem and maybe someone more\n> > > > familiar with gtest knows of a neat way to piggyback on its\n> > > > infrastrcuture? Since I know very little about the gtest framework the\n> > > > following is just an idea and I'm happy to yield.\n> > > >\n> > > > How about using a singleton class that carries arguments constructed\n> > > > from cli arguments (camera id) before the gtest are executed. Each test\n> > > > case could then consult this singleton if it needs to access parameters.\n> > > > I think you could take inspiration from the libcamera CameraManager\n> > > > class.\n> > > >\n> > > > This could (at lest for now) imply keeping the cam option parser. If so\n> > > > I think it would be nice if somewhen before we integrate this work that\n> > > > the gtest filter string could be supplied as an argument using that\n> > > > parser. Of course if we can do it the other way around and add arguments\n> > > > to the gtest parser that would be even cooler as it would be less code\n> > > > for us to maintain ;-)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5370ABF839\n\tfor ;\n\tTue, 11 May 2021 19:06:45 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 904B26891F;\n\tTue, 11 May 2021 21:06:44 +0200 (CEST)","from bhuna.collabora.co.uk (bhuna.collabora.co.uk\n\t[IPv6:2a00:1098:0:82:1000:25:2eeb:e3e3])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AC3D16890C\n\tfor ;\n\tTue, 11 May 2021 21:06:43 +0200 (CEST)","from localhost (unknown\n\t[IPv6:2804:14c:1a9:2978:985c:7892:ebcf:7a90])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id 567381F4256E;\n\tTue, 11 May 2021 20:06:41 +0100 (BST)"],"Mime-Version":"1.0","Date":"Tue, 11 May 2021 16:06:04 -0300","Message-Id":"","From":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= ","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t\n\t\n\t\n\t","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com, =?utf-8?q?A?=\n\t=?utf-8?q?ndr=C3=A9_Almeida?= ","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16915,"web_url":"https://patchwork.libcamera.org/comment/16915/","msgid":"","date":"2021-05-12T08:49:53","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Nícolas,\n\nOn 2021-05-11 14:17:56 -0300, Nícolas F. R. A. Prado wrote:\n> Hi Niklas,\n> \n> Em 2021-05-10 18:04, Niklas Söderlund escreveu:\n> \n> > Hi Nícolas,\n> >\n> > Thanks for your work on this. I really think gtest is a step in the\n> > right direction!\n> >\n> > On 2021-05-07 12:52:38 -0300, Nícolas F. R. A. Prado wrote:\n> > > Hi Niklas,\n> > > \n> > > Em 2021-05-07 06:11, Niklas Söderlund escreveu:\n> > > \n> > > > Hi Nícolas,\n> > > >\n> > > > On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > Hi Niklas,\n> > > > > \n> > > > > thanks for the feedback.\n> > > > > \n> > > > > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > > > > \n> > > > > > Hi Nícolas,\n> > > > > >\n> > > > > > Thanks for your work!\n> > > > > >\n> > > > > > I really like how gtest helps make the actual tests to be easier to\n> > > > > > read.\n> > > > > >\n> > > > > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > > > > \n> > > > > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > > > > ---\n> > > > > > > Changes in v2:\n> > > > > > > - Changed from static to dynamic test registration\n> > > > > > > - Removed -c flag\n> > > > > > > \n> > > > > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > > > > apparently aren't being deleted on time, which causes:\n> > > > > > > \n> > > > > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > > > > \tSegmentation fault (core dumped)\n> > > > > > > \n> > > > > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > > > > \n> > > > > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > > > \n> > > > > > > Not sure how to solve this. Any tip would be welcome.\n> > > > > >\n> > > > > > I will leave this for now as I have a different worry about\n> > > > > > REGISTER_TESTS() design below.\n> > > > > >\n> > > > > > > \n> > > > > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > > > > src/lc-compliance/meson.build | 3 +\n> > > > > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > > > > src/lc-compliance/tests.h | 15 ++-\n> > > > > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > > > > \n> > > > > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > > > > index 54cee54aa978..add0d7729aec 100644\n> > > > > > > --- a/src/lc-compliance/main.cpp\n> > > > > > > +++ b/src/lc-compliance/main.cpp\n> > > > > > > @@ -9,6 +9,8 @@\n> > > > > > > #include \n> > > > > > > #include \n> > > > > > > \n> > > > > > > +#include \n> > > > > > > +\n> > > > > > > #include \n> > > > > > > \n> > > > > > > #include \"../cam/options.h\"\n> > > > > > > @@ -17,7 +19,6 @@\n> > > > > > > using namespace libcamera;\n> > > > > > > \n> > > > > > > enum {\n> > > > > > > -\tOptCamera = 'c',\n> > > > > > > \tOptHelp = 'h',\n> > > > > > > };\n> > > > > > > \n> > > > > > > @@ -28,14 +29,15 @@ public:\n> > > > > > > \t~Harness();\n> > > > > > > \n> > > > > > > \tint exec();\n> > > > > > > +\tint init();\n> > > > > > > +\tvoid registerTests();\n> > > > > > > \n> > > > > > > private:\n> > > > > > > -\tint init();\n> > > > > > > \tvoid listCameras();\n> > > > > > > \n> > > > > > > \tOptionsParser::Options options_;\n> > > > > > > \tstd::unique_ptr cm_;\n> > > > > > > -\tstd::shared_ptr camera_;\n> > > > > > > +\tstd::vector> cameras_;\n> > > > > > > };\n> > > > > > > \n> > > > > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > \n> > > > > > > Harness::~Harness()\n> > > > > > > {\n> > > > > > > -\tif (camera_) {\n> > > > > > > -\t\tcamera_->release();\n> > > > > > > -\t\tcamera_.reset();\n> > > > > > > +\tfor (auto &c : cameras_) {\n> > > > > > > +\t\tc->release();\n> > > > > > > +\t\tc.reset();\n> > > > > > > \t}\n> > > > > > > \n> > > > > > > \tcm_->stop();\n> > > > > > > }\n> > > > > > > \n> > > > > > > -int Harness::exec()\n> > > > > > > -{\n> > > > > > > -\tint ret = init();\n> > > > > > > -\tif (ret)\n> > > > > > > -\t\treturn ret;\n> > > > > > > -\n> > > > > > > -\tstd::vector results;\n> > > > > > > -\n> > > > > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > > > > -\n> > > > > > > -\tfor (const Results &result : results) {\n> > > > > > > -\t\tret = result.summary();\n> > > > > > > -\t\tif (ret)\n> > > > > > > -\t\t\treturn ret;\n> > > > > > > -\t}\n> > > > > > > -\n> > > > > > > -\treturn 0;\n> > > > > > > -}\n> > > > > > > -\n> > > > > > > int Harness::init()\n> > > > > > > {\n> > > > > > > \tint ret = cm_->start();\n> > > > > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > > > > \t\treturn ret;\n> > > > > > > \t}\n> > > > > > > \n> > > > > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > > > > -\t\tlistCameras();\n> > > > > > > -\t\treturn -ENODEV;\n> > > > > > > -\t}\n> > > > > > > -\n> > > > > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > > > > -\tcamera_ = cm_->get(cameraId);\n> > > > > > > -\tif (!camera_) {\n> > > > > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > > > > -\t\tlistCameras();\n> > > > > > > -\t\treturn -ENODEV;\n> > > > > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > > > > +\t\tif (cam->acquire()) {\n> > > > > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > > > +\t\t\treturn -EINVAL;\n> > > > > > > +\t\t}\n> > > > > > > +\t\tcameras_.push_back(cam);\n> > > > > >\n> > > > > > I don't like this, I think we need to retain the ability to select which\n> > > > > > camera to test. And further down the line maybe allow more then one\n> > > > > > camera to be selected if we want to test concurrent use of two specific\n> > > > > > ones. But for the work in this series I think you can ignore multiple\n> > > > > > cameras.\n> > > > > \n> > > > > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > > > > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > > > > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > > > > all cameras, only the ones that I will actually test (and in this code there's a\n> > > > > big issue where if a single camera is already busy, even if it won't be tested,\n> > > > > lc-compliance as a whole fails).\n> > > >\n> > > > Ahh I understand.\n> > > >\n> > > > > \n> > > > > >\n> > > > > > If I understand the coverletter correctly the correct thing here the cli\n> > > > > > arguments should be feed to gtest directly to be able to filter on what\n> > > > > > tests are run. Is there support in getst to also pass arguments to\n> > > > > > tests? Maybe we could remove the need to parse arguments ourself\n> > > > > > completely?\n> > > > > \n> > > > > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > > > > could solve both problems (this and the one above) like this:\n> > > > > \n> > > > > - In the main(), query the id of each camera from libcamera.\n> > > > > - Register each of the tests dynamically in gtest passing as an argument the\n> > > > > camera id, rather than a camera pointer (which is the current behavior).\n> > > > > - To select which tests to run, and on which camera(s), the user just uses\n> > > > > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > > > > the camera).\n> > > > > - In the startup of each test the camera is acquired for that test, and when\n> > > > > it's done it is released.\n> > > > > \n> > > > > Implications:\n> > > > > - We no longer need to parse arguments and can drop that code, as both camera\n> > > > > and test are specified directly with gtest's filter.\n> > > > > - We only acquire the cameras that we'll actually test and only while they're\n> > > > > being tested.\n> > > > > - As a downside, a single camera will probably be acquired multiple times during\n> > > > > a single run as lc-compliance moves from one test to the next, but I don't\n> > > > > think this would be a big issue. (Maybe we could do this at the test suite\n> > > > > level instead to greatly decrease the number of times of acquire(), but I'm\n> > > > > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > > > > the test level, I'll have to test that).\n> > > > > \n> > > > > What do you think?\n> > > >\n> > > > I understand what you are trying to do but I see two issues.\n> > > >\n> > > > 1. It depends on the implicit knowledge of users how to build the filter\n> > > > string and is rather unintuitive compared to having two distinct\n> > > > arguments, one to select camera and one to express the test filter.\n> > > >\n> > > > This could possibly be worked around by keeping the cam options code\n> > > > and building the gtest filter string internally by injecitng the\n> > > > selected camera name in the correct location, but the we have the\n> > > > next issue,\n> > > >\n> > > > 2. I don't think it will be practically possible to express a test\n> > > > filter that selects a specific combination of two cameras.\n> > > >\n> > > > I do think we need to be able to parameterize test cases. If gtest does\n> > > > not allow to do so using function arguments there are other ways of\n> > > > doing so.\n> > > \n> > > I think I didn't express myself well. I meant to say that gtest's CLI doesn't\n> > > have a flag to allow passing parameters to tests, meaning that it would not be\n> > > possible to parametrize the tests from the command line and also remove our\n> > > own parser and rely on gtest's.\n> >\n> > Check, I think we need to keep our own parser for now as we need to be\n> > able to parametrize the tests from the CLI.\n> >\n> > > But gtest does allow test parametrization at runtime and I'm already relying on\n> > > it since v2. Let me go a bit more over the way I'm doing this.\n> > > \n> > > - I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n> > > \n> > > \tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n> > > \n> > > - Since we may want to run the same test for all cameras, roles and numRequests,\n> > > I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n> > > gtest's testing::RegisterTest() with all possible combinations of parameters.\n> > > \n> > > - Based on the --gtest_filter CLI argument from gtest, it decides which tests\n> > > are actually run\n> > > \n> > > So the key point is that even though I parametrize the tests, I do it for all\n> > > combinations in order to later rely on gtest_filter to select which tests to\n> > > run.\n> >\n> > I see, that explains how the parameters get filled. I noticed it but did\n> > not dig further, see below for an expansion of my thoughts now that I\n> > understand this detail :-)\n> >\n> > > \n> > > Now, I do agree with both of your points. And if we want to make the interface\n> > > more intuitive to use (which I also find important), there's no way around\n> > > having our own parser.\n> > > \n> > > And if we're using our own camera flag, then it makes no sense to register tests\n> > > for all cameras, we can register tests only with the specified camera.\n> >\n> > This feels wrong. It feels like tests should be register without a\n> > camera component. The camera (and possible other arguments) should come\n> > from either arguments [1] or a singleton parameter class [2].\n> >\n> > 1. This do not seem to be possible with gtests as test parameters needs\n> > to be registerd.\n> >\n> > 2. This would be similar to the global variable used in earlier versions\n> > but instead modeled as a class singleton like CameraManager.\n> >\n> > Maybe there are other options tho.\n> >\n> > > \n> > > I think here it would be helpful to think if the interface suits our problem.\n> > > To me we're looking at having:\n> > > \n> > > \tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n> >\n> > This matches my view as well.\n> >\n> > > \n> > > to only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\n> > > the camera flag is used to only register tests with that camera, and the filter\n> > > is passed as --gtest_filter, possibly a bit adapted, to gtest.\n> > > \n> > > And if we imagine we have a test ConcurrentCameras, which receives two cameras\n> > > as parameters, and we call\n> > > \n> > > \tlc-compliance --camera 'cameraId1,cameraId2'\n> >\n> > This also looks good. But for now lets focus on one camera, I think it\n> > will give us enough headache :-)\n> >\n> > > \n> > > it would test both each of the cameras separately in all current tests which\n> > > accept a single camera, like BalancedSingleCycle, as well as test both at the\n> > > same time in the ConcurrentCameras test (and we could make only the former or\n> > > the latter occur by using the filter parameter).\n> >\n> > I have not thought enough about this to know if that is a good idea or\n> > not. At the top of my head I still tend to think that lc-compliance\n> > tests a single 'primary' camera and any additional camera supplied are\n> > only used for tests where two (or more) cameras are needed to test some\n> > specific feature of the primary camera.\n> >\n> > That is only the 'primary' camera is testes, the other camera(s) are\n> > just there to help/provoke the testing. But maybe I'm to narrow minded.\n> \n> Okay, so it now comes down to how lc-compliance will get used. Either:\n> \n> 1. lc-compliance will be called to test a single camera, with optionally other\n> secondary cameras\n> \n> 2. lc-compliance will be called to test multiple cameras at once (not\n> necessarily simultaneously, but in a single run), with the default case being\n> \"Test all the cameras in this system\".\n> \n> Given that the final objective of my internship is to have lc-compliance running\n> on real boards by KernelCI, to catch any kernel regressions that affect camera\n> usage, I'm trying to think how it would work there. (I wonder if this discussion\n> should be happening together with KernelCI to decide on this...)\n> \n> So I'm imagining a RockPi4 board with a camera sensor plugged in, running\n> lc-compliance to catch any kernel regressions.\n> \n> The limitation I see in 1 is that if a system has multiple cameras and want to\n> ensure that all keep working, it will need to either hardcode all camera ids and\n> manually call lc-compliance once for each one, or have a separate program that\n> lists all cameras from libcamera (using 'cam -l' for example) and calls\n> lc-compliance with each one. That's to say, running on all cameras is not as\n> easy. And then there are multiple reports, one for each camera, instead of a\n> single one for all cameras, which might be good or bad, not sure.\n> \n> That said, since the RockPi4 (and also the Raspberry Pi 4) has a single CSI-2\n> connector, I believe it only supports a single camera anyway, so this shouldn't\n> be a problem for this board, but I'm not sure about others. Phones for example\n> seem to have at least two independent cameras...\n\nI now understand your point of view better, thanks for brining up the \nKernelCI angle. I agree from a KernelCI perspective the goal is likely \nto test all cameras of the system to get the widest test base. But from \na (current) libcamera perspective the lc-complinance tool primary usage \nis to be able to test a particular camera (or pipeline) to prove changes \ndo not break things, kind of like v4l2-complinace is for V4L2.\n\nThat being said I think it's trivial for us to make lc-complinance \nsatisfy both use-cases. If the arg parsing is structured something like \nthis,\n\n if (cliargs.continas(\"--camera\")) {\n run_tests_for_camera(cliargs.value_for(\"--camera\");\n } else if (cliargs.continas(\"--run-all-cameras\")) {\n for (camera : camer_manager.list_all_cameras()) {\n run_tests_for_camera(camera)\n }\n }\n\nOr am I missing something?\n\n> \n> Regarding the benefit of 1, I'm not sure. If we were to use a global variable\n> like in v1, it would make the code a lot simpler by making the test registration\n> static. But that doesn't seem to be an option :). I'm not familiar with using a\n> Singleton, so I'll take a look at CameraManager. But I assume it wouldn't be\n> accessible statically, so the dynamic test registration complexities would have\n> to stay.\n> \n> Also, just to make sure we're on the same page, by using a Singleton, and again\n> assuming it isn't stored in a global variable to be accessible from the tests\n> statically, even though we would eliminate the camera as a parameter for the\n> tests, the reference to the singleton itself would need to be added as a\n> parameter for the tests.\n\nI will follow up on this in your second mail.\n\n> \n> Thanks,\n> Nícolas\n> \n> >\n> > > \n> > > I know we don't need to implement support for multiple cameras right now, I just\n> > > want to make sure it's possible and thinking about the interface and the\n> > > scenarios help me picture how they'd work.\n> > > \n> > > Thoughts?\n> > > \n> > > Thanks,\n> > > Nícolas\n> > > \n> > > >\n> > > > I'm sure there are many solutions to this problem and maybe someone more\n> > > > familiar with gtest knows of a neat way to piggyback on its\n> > > > infrastrcuture? Since I know very little about the gtest framework the\n> > > > following is just an idea and I'm happy to yield.\n> > > >\n> > > > How about using a singleton class that carries arguments constructed\n> > > > from cli arguments (camera id) before the gtest are executed. Each test\n> > > > case could then consult this singleton if it needs to access parameters.\n> > > > I think you could take inspiration from the libcamera CameraManager\n> > > > class.\n> > > >\n> > > > This could (at lest for now) imply keeping the cam option parser. If so\n> > > > I think it would be nice if somewhen before we integrate this work that\n> > > > the gtest filter string could be supplied as an argument using that\n> > > > parser. Of course if we can do it the other way around and add arguments\n> > > > to the gtest parser that would be even cooler as it would be less code\n> > > > for us to maintain ;-)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A27FEC31E4\n\tfor ;\n\tWed, 12 May 2021 08:49:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0C5C168920;\n\tWed, 12 May 2021 10:49:57 +0200 (CEST)","from mail-lj1-x232.google.com (mail-lj1-x232.google.com\n\t[IPv6:2a00:1450:4864:20::232])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 74236688E4\n\tfor ;\n\tWed, 12 May 2021 10:49:55 +0200 (CEST)","by mail-lj1-x232.google.com with SMTP id s25so3934820ljo.11\n\tfor ;\n\tWed, 12 May 2021 01:49:55 -0700 (PDT)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tq24sm2226320ljm.23.2021.05.12.01.49.53\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tWed, 12 May 2021 01:49:54 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"ulFq27jW\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=6X7V/LMYm2qZUz3mL3IwfjPIFwAIs9lyzyz1rnf9zvg=;\n\tb=ulFq27jWDaeFeKO1LhzpJn1ZsBXZNdZUyA3aZE2Hkhc77zM/flD7tygSB4mrFGqBnD\n\tCZFVbIhDZQqjRYGYFMgEA/6TSb1vBw+nU05v+pzb9oKTSp6bpoDtrTqdT3OoKl2a0XP1\n\tvKGQDKBej3wzsH/8FacCpEmLGkFqmyJFz1sUebQ9fALDFZDdDC/T/tFLQT3OafnmIwTX\n\tLQnW+LD2Z3RU5aYyfy0zjqiu4/eZ/k7bp4H37RuESFtoVmjv3iNJNgrpbdaaWY53wz2u\n\tVaxtFUcadRL2D5LGQpBWL5TRcjP1j9Henme79FSkplX0nZyxBKS25194MOlwYr8iiWbC\n\tptwQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=6X7V/LMYm2qZUz3mL3IwfjPIFwAIs9lyzyz1rnf9zvg=;\n\tb=EBC2ohOa3F0SIVfO+OgHmKoTm8rSiYCiM9bCxmc+/YxvpX5Ekj6Q8dN3RTnpJ8k1Uk\n\tC40U22INNKetocsU3WZ5hsWak5Z1U142tpM2cc5rYmEI/XcWyqCVRtiDtbxXMKZIAu2u\n\tM8gJN/Q0UTuzlRnIpZQ68xltchY2hK8JKQnnOVCcvhB+myVPKnnay6AjST6Zrn83rzfm\n\tI/M8ORNUttkmKLPejLeeWcIOsogjlkYawcJIdiXPZ2ys+T6FKaOBQMTgGHMmUkhNtZ47\n\tWRCMbsf4InlokTRBSgEnuvplrePXV3978A+2MLoQwtBTXhe48q5IyQwcTzH16g54reVr\n\tjaKw==","X-Gm-Message-State":"AOAM530f4C/6RbeEBi6SS1sVCdYhWfYoRVlcQg+fzPTGLIIpWw0a2sVp\n\t9ZYyJ6MaiDa5P0wP95K4PWpVIQ==","X-Google-Smtp-Source":"ABdhPJw76tdTN4wQMrVqmslzlgly/Rw5ptAiiX6XCg9P435WP+eu8u2HJNn+mNPOtbS8+Ti/WqeZcg==","X-Received":"by 2002:a2e:9656:: with SMTP id\n\tz22mr27076266ljh.168.1620809394632; \n\tWed, 12 May 2021 01:49:54 -0700 (PDT)","Date":"Wed, 12 May 2021 10:49:53 +0200","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"=?iso-8859-1?q?N=EDcolas_F=2E_R=2E_A=2E?= Prado ","Message-ID":"","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com,\n\t=?iso-8859-1?q?Andr=E9?= Almeida ","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16916,"web_url":"https://patchwork.libcamera.org/comment/16916/","msgid":"","date":"2021-05-12T08:56:36","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"On 2021-05-11 16:06:04 -0300, Nícolas F. R. A. Prado wrote:\n> Hello again,\n\nHello :-)\n\n> \n> Em 2021-05-11 14:17, Nícolas F. R. A. Prado escreveu:\n> \n> > Hi Niklas,\n> >\n> > Em 2021-05-10 18:04, Niklas Söderlund escreveu:\n> >\n> > > Hi Nícolas,\n> > >\n> > > Thanks for your work on this. I really think gtest is a step in the\n> > > right direction!\n> > >\n> > > On 2021-05-07 12:52:38 -0300, Nícolas F. R. A. Prado wrote:\n> > > > Hi Niklas,\n> > > > \n> > > > Em 2021-05-07 06:11, Niklas Söderlund escreveu:\n> > > > \n> > > > > Hi Nícolas,\n> > > > >\n> > > > > On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > Hi Niklas,\n> > > > > > \n> > > > > > thanks for the feedback.\n> > > > > > \n> > > > > > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > > > > > \n> > > > > > > Hi Nícolas,\n> > > > > > >\n> > > > > > > Thanks for your work!\n> > > > > > >\n> > > > > > > I really like how gtest helps make the actual tests to be easier to\n> > > > > > > read.\n> > > > > > >\n> > > > > > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > > > > > \n> > > > > > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > > > > > ---\n> > > > > > > > Changes in v2:\n> > > > > > > > - Changed from static to dynamic test registration\n> > > > > > > > - Removed -c flag\n> > > > > > > > \n> > > > > > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > > > > > apparently aren't being deleted on time, which causes:\n> > > > > > > > \n> > > > > > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > > > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > > > > > \tSegmentation fault (core dumped)\n> > > > > > > > \n> > > > > > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > > > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > > > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > > > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > > > > > \n> > > > > > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > > > > \n> > > > > > > > Not sure how to solve this. Any tip would be welcome.\n> > > > > > >\n> > > > > > > I will leave this for now as I have a different worry about\n> > > > > > > REGISTER_TESTS() design below.\n> > > > > > >\n> > > > > > > > \n> > > > > > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > > > > > src/lc-compliance/meson.build | 3 +\n> > > > > > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > > > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > > > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > > > > > src/lc-compliance/tests.h | 15 ++-\n> > > > > > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > > > > > \n> > > > > > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > > > > > index 54cee54aa978..add0d7729aec 100644\n> > > > > > > > --- a/src/lc-compliance/main.cpp\n> > > > > > > > +++ b/src/lc-compliance/main.cpp\n> > > > > > > > @@ -9,6 +9,8 @@\n> > > > > > > > #include \n> > > > > > > > #include \n> > > > > > > > \n> > > > > > > > +#include \n> > > > > > > > +\n> > > > > > > > #include \n> > > > > > > > \n> > > > > > > > #include \"../cam/options.h\"\n> > > > > > > > @@ -17,7 +19,6 @@\n> > > > > > > > using namespace libcamera;\n> > > > > > > > \n> > > > > > > > enum {\n> > > > > > > > -\tOptCamera = 'c',\n> > > > > > > > \tOptHelp = 'h',\n> > > > > > > > };\n> > > > > > > > \n> > > > > > > > @@ -28,14 +29,15 @@ public:\n> > > > > > > > \t~Harness();\n> > > > > > > > \n> > > > > > > > \tint exec();\n> > > > > > > > +\tint init();\n> > > > > > > > +\tvoid registerTests();\n> > > > > > > > \n> > > > > > > > private:\n> > > > > > > > -\tint init();\n> > > > > > > > \tvoid listCameras();\n> > > > > > > > \n> > > > > > > > \tOptionsParser::Options options_;\n> > > > > > > > \tstd::unique_ptr cm_;\n> > > > > > > > -\tstd::shared_ptr camera_;\n> > > > > > > > +\tstd::vector> cameras_;\n> > > > > > > > };\n> > > > > > > > \n> > > > > > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > > \n> > > > > > > > Harness::~Harness()\n> > > > > > > > {\n> > > > > > > > -\tif (camera_) {\n> > > > > > > > -\t\tcamera_->release();\n> > > > > > > > -\t\tcamera_.reset();\n> > > > > > > > +\tfor (auto &c : cameras_) {\n> > > > > > > > +\t\tc->release();\n> > > > > > > > +\t\tc.reset();\n> > > > > > > > \t}\n> > > > > > > > \n> > > > > > > > \tcm_->stop();\n> > > > > > > > }\n> > > > > > > > \n> > > > > > > > -int Harness::exec()\n> > > > > > > > -{\n> > > > > > > > -\tint ret = init();\n> > > > > > > > -\tif (ret)\n> > > > > > > > -\t\treturn ret;\n> > > > > > > > -\n> > > > > > > > -\tstd::vector results;\n> > > > > > > > -\n> > > > > > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > > > > > -\n> > > > > > > > -\tfor (const Results &result : results) {\n> > > > > > > > -\t\tret = result.summary();\n> > > > > > > > -\t\tif (ret)\n> > > > > > > > -\t\t\treturn ret;\n> > > > > > > > -\t}\n> > > > > > > > -\n> > > > > > > > -\treturn 0;\n> > > > > > > > -}\n> > > > > > > > -\n> > > > > > > > int Harness::init()\n> > > > > > > > {\n> > > > > > > > \tint ret = cm_->start();\n> > > > > > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > > > > > \t\treturn ret;\n> > > > > > > > \t}\n> > > > > > > > \n> > > > > > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > > > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > > > > > -\t\tlistCameras();\n> > > > > > > > -\t\treturn -ENODEV;\n> > > > > > > > -\t}\n> > > > > > > > -\n> > > > > > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > > > > > -\tcamera_ = cm_->get(cameraId);\n> > > > > > > > -\tif (!camera_) {\n> > > > > > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > > > > > -\t\tlistCameras();\n> > > > > > > > -\t\treturn -ENODEV;\n> > > > > > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > > > > > +\t\tif (cam->acquire()) {\n> > > > > > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > > > > +\t\t\treturn -EINVAL;\n> > > > > > > > +\t\t}\n> > > > > > > > +\t\tcameras_.push_back(cam);\n> > > > > > >\n> > > > > > > I don't like this, I think we need to retain the ability to select which\n> > > > > > > camera to test. And further down the line maybe allow more then one\n> > > > > > > camera to be selected if we want to test concurrent use of two specific\n> > > > > > > ones. But for the work in this series I think you can ignore multiple\n> > > > > > > cameras.\n> > > > > > \n> > > > > > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > > > > > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > > > > > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > > > > > all cameras, only the ones that I will actually test (and in this code there's a\n> > > > > > big issue where if a single camera is already busy, even if it won't be tested,\n> > > > > > lc-compliance as a whole fails).\n> > > > >\n> > > > > Ahh I understand.\n> > > > >\n> > > > > > \n> > > > > > >\n> > > > > > > If I understand the coverletter correctly the correct thing here the cli\n> > > > > > > arguments should be feed to gtest directly to be able to filter on what\n> > > > > > > tests are run. Is there support in getst to also pass arguments to\n> > > > > > > tests? Maybe we could remove the need to parse arguments ourself\n> > > > > > > completely?\n> > > > > > \n> > > > > > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > > > > > could solve both problems (this and the one above) like this:\n> > > > > > \n> > > > > > - In the main(), query the id of each camera from libcamera.\n> > > > > > - Register each of the tests dynamically in gtest passing as an argument the\n> > > > > > camera id, rather than a camera pointer (which is the current behavior).\n> > > > > > - To select which tests to run, and on which camera(s), the user just uses\n> > > > > > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > > > > > the camera).\n> > > > > > - In the startup of each test the camera is acquired for that test, and when\n> > > > > > it's done it is released.\n> > > > > > \n> > > > > > Implications:\n> > > > > > - We no longer need to parse arguments and can drop that code, as both camera\n> > > > > > and test are specified directly with gtest's filter.\n> > > > > > - We only acquire the cameras that we'll actually test and only while they're\n> > > > > > being tested.\n> > > > > > - As a downside, a single camera will probably be acquired multiple times during\n> > > > > > a single run as lc-compliance moves from one test to the next, but I don't\n> > > > > > think this would be a big issue. (Maybe we could do this at the test suite\n> > > > > > level instead to greatly decrease the number of times of acquire(), but I'm\n> > > > > > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > > > > > the test level, I'll have to test that).\n> > > > > > \n> > > > > > What do you think?\n> > > > >\n> > > > > I understand what you are trying to do but I see two issues.\n> > > > >\n> > > > > 1. It depends on the implicit knowledge of users how to build the filter\n> > > > > string and is rather unintuitive compared to having two distinct\n> > > > > arguments, one to select camera and one to express the test filter.\n> > > > >\n> > > > > This could possibly be worked around by keeping the cam options code\n> > > > > and building the gtest filter string internally by injecitng the\n> > > > > selected camera name in the correct location, but the we have the\n> > > > > next issue,\n> > > > >\n> > > > > 2. I don't think it will be practically possible to express a test\n> > > > > filter that selects a specific combination of two cameras.\n> > > > >\n> > > > > I do think we need to be able to parameterize test cases. If gtest does\n> > > > > not allow to do so using function arguments there are other ways of\n> > > > > doing so.\n> > > > \n> > > > I think I didn't express myself well. I meant to say that gtest's CLI doesn't\n> > > > have a flag to allow passing parameters to tests, meaning that it would not be\n> > > > possible to parametrize the tests from the command line and also remove our\n> > > > own parser and rely on gtest's.\n> > >\n> > > Check, I think we need to keep our own parser for now as we need to be\n> > > able to parametrize the tests from the CLI.\n> > >\n> > > > But gtest does allow test parametrization at runtime and I'm already relying on\n> > > > it since v2. Let me go a bit more over the way I'm doing this.\n> > > > \n> > > > - I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n> > > > \n> > > > \tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n> > > > \n> > > > - Since we may want to run the same test for all cameras, roles and numRequests,\n> > > > I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n> > > > gtest's testing::RegisterTest() with all possible combinations of parameters.\n> > > > \n> > > > - Based on the --gtest_filter CLI argument from gtest, it decides which tests\n> > > > are actually run\n> > > > \n> > > > So the key point is that even though I parametrize the tests, I do it for all\n> > > > combinations in order to later rely on gtest_filter to select which tests to\n> > > > run.\n> > >\n> > > I see, that explains how the parameters get filled. I noticed it but did\n> > > not dig further, see below for an expansion of my thoughts now that I\n> > > understand this detail :-)\n> > >\n> > > > \n> > > > Now, I do agree with both of your points. And if we want to make the interface\n> > > > more intuitive to use (which I also find important), there's no way around\n> > > > having our own parser.\n> > > > \n> > > > And if we're using our own camera flag, then it makes no sense to register tests\n> > > > for all cameras, we can register tests only with the specified camera.\n> > >\n> > > This feels wrong. It feels like tests should be register without a\n> > > camera component. The camera (and possible other arguments) should come\n> > > from either arguments [1] or a singleton parameter class [2].\n> > >\n> > > 1. This do not seem to be possible with gtests as test parameters needs\n> > > to be registerd.\n> > >\n> > > 2. This would be similar to the global variable used in earlier versions\n> > > but instead modeled as a class singleton like CameraManager.\n> > >\n> > > Maybe there are other options tho.\n> > >\n> > > > \n> > > > I think here it would be helpful to think if the interface suits our problem.\n> > > > To me we're looking at having:\n> > > > \n> > > > \tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n> > >\n> > > This matches my view as well.\n> > >\n> > > > \n> > > > to only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\n> > > > the camera flag is used to only register tests with that camera, and the filter\n> > > > is passed as --gtest_filter, possibly a bit adapted, to gtest.\n> > > > \n> > > > And if we imagine we have a test ConcurrentCameras, which receives two cameras\n> > > > as parameters, and we call\n> > > > \n> > > > \tlc-compliance --camera 'cameraId1,cameraId2'\n> > >\n> > > This also looks good. But for now lets focus on one camera, I think it\n> > > will give us enough headache :-)\n> > >\n> > > > \n> > > > it would test both each of the cameras separately in all current tests which\n> > > > accept a single camera, like BalancedSingleCycle, as well as test both at the\n> > > > same time in the ConcurrentCameras test (and we could make only the former or\n> > > > the latter occur by using the filter parameter).\n> > >\n> > > I have not thought enough about this to know if that is a good idea or\n> > > not. At the top of my head I still tend to think that lc-compliance\n> > > tests a single 'primary' camera and any additional camera supplied are\n> > > only used for tests where two (or more) cameras are needed to test some\n> > > specific feature of the primary camera.\n> > >\n> > > That is only the 'primary' camera is testes, the other camera(s) are\n> > > just there to help/provoke the testing. But maybe I'm to narrow minded.\n> >\n> > Okay, so it now comes down to how lc-compliance will get used. Either:\n> >\n> > 1. lc-compliance will be called to test a single camera, with optionally other\n> > secondary cameras\n> >\n> > 2. lc-compliance will be called to test multiple cameras at once (not\n> > necessarily simultaneously, but in a single run), with the default case being\n> > \"Test all the cameras in this system\".\n> >\n> > Given that the final objective of my internship is to have lc-compliance running\n> > on real boards by KernelCI, to catch any kernel regressions that affect camera\n> > usage, I'm trying to think how it would work there. (I wonder if this discussion\n> > should be happening together with KernelCI to decide on this...)\n> >\n> > So I'm imagining a RockPi4 board with a camera sensor plugged in, running\n> > lc-compliance to catch any kernel regressions.\n> >\n> > The limitation I see in 1 is that if a system has multiple cameras and want to\n> > ensure that all keep working, it will need to either hardcode all camera ids and\n> > manually call lc-compliance once for each one, or have a separate program that\n> > lists all cameras from libcamera (using 'cam -l' for example) and calls\n> > lc-compliance with each one. That's to say, running on all cameras is not as\n> > easy. And then there are multiple reports, one for each camera, instead of a\n> > single one for all cameras, which might be good or bad, not sure.\n> >\n> > That said, since the RockPi4 (and also the Raspberry Pi 4) has a single CSI-2\n> > connector, I believe it only supports a single camera anyway, so this shouldn't\n> > be a problem for this board, but I'm not sure about others. Phones for example\n> > seem to have at least two independent cameras...\n> \n> So, I just read about the Singleton pattern on Wikipedia and checked out the\n> CameraManager class (should've done it before replying :) ).\n> \n> From what I can tell, there's a big difference between the CameraManager class\n> and a Singleton. The CameraManager just makes sure that there's a single\n> instance of it by saving a reference to it in an internal static variable. A\n> Singleton apparently, in addition to doing that, makes its instatiation through\n> a public static function. This means that not only there's a single instance,\n> it's also available from the global scope.\n\nYou are correct, my bad. I thought CameraManager implemented the \nSingleton pattern.\n\n> \n> So I wanted to ask you to clarify which one you meant.\n\nAlways a good idea as I easily get lost ;-)\n\n> If you meant that we\n> should have a class like CameraManager, which just makes sure it's the single\n> instance, my previous comments below are still valid.\n> \n> On the other hand, if you meant that we should a have singleton, which is\n> available globally, then it is basically the same thing as having a global\n> variable, only a bit more organized, and my comments below are no longer valid.\n> We could indeed go back to static test registration, since the camera instance\n> could be accessed from any test as something like\n> \n> \tTestEnvironment::getInstance().camera()\n\nThis is what I meant, a more organized way to reach the global variable \nto allow for static test registration while still being able to \nparametrize the tests. Sorry for the confusion by bringing up \nCameraManager.\n\n> \n> making the alternative 1 have the benefit of making the test registration and\n> definition code simpler.\n> \n> Thanks,\n> Nícolas\n> \n> >\n> > Regarding the benefit of 1, I'm not sure. If we were to use a global variable\n> > like in v1, it would make the code a lot simpler by making the test registration\n> > static. But that doesn't seem to be an option :). I'm not familiar with using a\n> > Singleton, so I'll take a look at CameraManager. But I assume it wouldn't be\n> > accessible statically, so the dynamic test registration complexities would have\n> > to stay.\n> >\n> > Also, just to make sure we're on the same page, by using a Singleton, and again\n> > assuming it isn't stored in a global variable to be accessible from the tests\n> > statically, even though we would eliminate the camera as a parameter for the\n> > tests, the reference to the singleton itself would need to be added as a\n> > parameter for the tests.\n> >\n> > Thanks,\n> > Nícolas\n> >\n> > >\n> > > > \n> > > > I know we don't need to implement support for multiple cameras right now, I just\n> > > > want to make sure it's possible and thinking about the interface and the\n> > > > scenarios help me picture how they'd work.\n> > > > \n> > > > Thoughts?\n> > > > \n> > > > Thanks,\n> > > > Nícolas\n> > > > \n> > > > >\n> > > > > I'm sure there are many solutions to this problem and maybe someone more\n> > > > > familiar with gtest knows of a neat way to piggyback on its\n> > > > > infrastrcuture? Since I know very little about the gtest framework the\n> > > > > following is just an idea and I'm happy to yield.\n> > > > >\n> > > > > How about using a singleton class that carries arguments constructed\n> > > > > from cli arguments (camera id) before the gtest are executed. Each test\n> > > > > case could then consult this singleton if it needs to access parameters.\n> > > > > I think you could take inspiration from the libcamera CameraManager\n> > > > > class.\n> > > > >\n> > > > > This could (at lest for now) imply keeping the cam option parser. If so\n> > > > > I think it would be nice if somewhen before we integrate this work that\n> > > > > the gtest filter string could be supplied as an argument using that\n> > > > > parser. Of course if we can do it the other way around and add arguments\n> > > > > to the gtest parser that would be even cooler as it would be less code\n> > > > > for us to maintain ;-)","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4437EC31E3\n\tfor ;\n\tWed, 12 May 2021 08:56:41 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6BE226891A;\n\tWed, 12 May 2021 10:56:40 +0200 (CEST)","from mail-lf1-x129.google.com (mail-lf1-x129.google.com\n\t[IPv6:2a00:1450:4864:20::129])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 48AE6688E4\n\tfor ;\n\tWed, 12 May 2021 10:56:38 +0200 (CEST)","by mail-lf1-x129.google.com with SMTP id r5so15443353lfr.5\n\tfor ;\n\tWed, 12 May 2021 01:56:38 -0700 (PDT)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\ti30sm2863551lfc.94.2021.05.12.01.56.36\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tWed, 12 May 2021 01:56:36 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"Y3e4IpVw\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=aOUzimjgL9XZ6GYV0W3YyIbi3b9SdUtN2+QXeF5GC0g=;\n\tb=Y3e4IpVwZc8AVcVz9Qhy4jtRMRqyrDiijuxOJUfPt7vMzDQIbc9udhE9kDKngsjtka\n\tSZbkDEpXd39kU3Ed7ZYVF8sC0tJb0ze0PzJwJCZWAFV0veUQjwv9m2nKbj5JfzLbKzV7\n\trH2zegtFTXb951i5Cqvn48998DfyC7j3DnaMu7cqEzSheiKv8/XEIRzMXJ5VzP99fnrV\n\tYv+lkGUnaoDYJQtWHiwUzjUOYS1YCklax1L7jFngr9E6A52CBgeJ5xr+RW1klUje4Iah\n\tq84kz8tFFWGWb+MxmeX79HWzr4Cp+weF40J1Pe/VIosataX7Wg+PRZldgMqFU0ApoC5F\n\tLVIQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=aOUzimjgL9XZ6GYV0W3YyIbi3b9SdUtN2+QXeF5GC0g=;\n\tb=YlrQjC2GhsCYRA1IefzwyYV0SweUQC0n8KiwHAcBRaejUOwP0rA5ZQOvRbslywZ0Ue\n\t1QchXY5mUH1fQZD5ybGC/4iNwtVndtacKq7mPG/9+qweFuCZvdhaFNVvu8Jp1EdCbmBL\n\tcFiMTUPrtODy+5nALdHHB2Z8NUhHgvRRqsHPgORibM7MMRkoRLLIHelWMiWvptAw06t4\n\tUZ0xrujcwY+eztt8b7XVbwMQoz4j1W8ynZdu7ftHiCVvbZVBFfU/zemSbDhubxUrunpb\n\to4593GtaV4D8tPVqe5b1f2ZWYXSDf73XgHNByq7ossgZnJ+mYzBOqwbKL7OZZ+gW8/4m\n\tizvg==","X-Gm-Message-State":"AOAM532xBwvkDifTNkJleK0oP2EmKL5oCp2mQxzm5DxF2EkIe2hSwVm1\n\tk4awoNyFxOWNzPZcyFM1U1oGZA==","X-Google-Smtp-Source":"ABdhPJwc2PvGNWFIjnfMpZV4k4/YCPqS4exQGz4rqsuxUO2Bvr/vKNDHr2U8qr5cwHYoypze/NhQnw==","X-Received":"by 2002:a19:f112:: with SMTP id\n\tp18mr24927867lfh.441.1620809797253; \n\tWed, 12 May 2021 01:56:37 -0700 (PDT)","Date":"Wed, 12 May 2021 10:56:36 +0200","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"=?iso-8859-1?q?N=EDcolas_F=2E_R=2E_A=2E?= Prado ","Message-ID":"","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com,\n\t=?iso-8859-1?q?Andr=E9?= Almeida ","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16919,"web_url":"https://patchwork.libcamera.org/comment/16919/","msgid":"","date":"2021-05-12T13:43:35","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":84,"url":"https://patchwork.libcamera.org/api/people/84/","name":"Nícolas F. R. A. Prado","email":"nfraprado@collabora.com"},"content":"Hi Niklas,\n\nEm 2021-05-12 05:49, Niklas Söderlund escreveu:\n\n> Hi Nícolas,\n>\n> On 2021-05-11 14:17:56 -0300, Nícolas F. R. A. Prado wrote:\n> > Hi Niklas,\n> > \n> > Em 2021-05-10 18:04, Niklas Söderlund escreveu:\n> > \n> > > Hi Nícolas,\n> > >\n> > > Thanks for your work on this. I really think gtest is a step in the\n> > > right direction!\n> > >\n> > > On 2021-05-07 12:52:38 -0300, Nícolas F. R. A. Prado wrote:\n> > > > Hi Niklas,\n> > > > \n> > > > Em 2021-05-07 06:11, Niklas Söderlund escreveu:\n> > > > \n> > > > > Hi Nícolas,\n> > > > >\n> > > > > On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > Hi Niklas,\n> > > > > > \n> > > > > > thanks for the feedback.\n> > > > > > \n> > > > > > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > > > > > \n> > > > > > > Hi Nícolas,\n> > > > > > >\n> > > > > > > Thanks for your work!\n> > > > > > >\n> > > > > > > I really like how gtest helps make the actual tests to be easier to\n> > > > > > > read.\n> > > > > > >\n> > > > > > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > > > > > \n> > > > > > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > > > > > ---\n> > > > > > > > Changes in v2:\n> > > > > > > > - Changed from static to dynamic test registration\n> > > > > > > > - Removed -c flag\n> > > > > > > > \n> > > > > > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > > > > > apparently aren't being deleted on time, which causes:\n> > > > > > > > \n> > > > > > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > > > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > > > > > \tSegmentation fault (core dumped)\n> > > > > > > > \n> > > > > > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > > > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > > > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > > > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > > > > > \n> > > > > > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > > > > \n> > > > > > > > Not sure how to solve this. Any tip would be welcome.\n> > > > > > >\n> > > > > > > I will leave this for now as I have a different worry about\n> > > > > > > REGISTER_TESTS() design below.\n> > > > > > >\n> > > > > > > > \n> > > > > > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > > > > > src/lc-compliance/meson.build | 3 +\n> > > > > > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > > > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > > > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > > > > > src/lc-compliance/tests.h | 15 ++-\n> > > > > > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > > > > > \n> > > > > > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > > > > > index 54cee54aa978..add0d7729aec 100644\n> > > > > > > > --- a/src/lc-compliance/main.cpp\n> > > > > > > > +++ b/src/lc-compliance/main.cpp\n> > > > > > > > @@ -9,6 +9,8 @@\n> > > > > > > > #include \n> > > > > > > > #include \n> > > > > > > > \n> > > > > > > > +#include \n> > > > > > > > +\n> > > > > > > > #include \n> > > > > > > > \n> > > > > > > > #include \"../cam/options.h\"\n> > > > > > > > @@ -17,7 +19,6 @@\n> > > > > > > > using namespace libcamera;\n> > > > > > > > \n> > > > > > > > enum {\n> > > > > > > > -\tOptCamera = 'c',\n> > > > > > > > \tOptHelp = 'h',\n> > > > > > > > };\n> > > > > > > > \n> > > > > > > > @@ -28,14 +29,15 @@ public:\n> > > > > > > > \t~Harness();\n> > > > > > > > \n> > > > > > > > \tint exec();\n> > > > > > > > +\tint init();\n> > > > > > > > +\tvoid registerTests();\n> > > > > > > > \n> > > > > > > > private:\n> > > > > > > > -\tint init();\n> > > > > > > > \tvoid listCameras();\n> > > > > > > > \n> > > > > > > > \tOptionsParser::Options options_;\n> > > > > > > > \tstd::unique_ptr cm_;\n> > > > > > > > -\tstd::shared_ptr camera_;\n> > > > > > > > +\tstd::vector> cameras_;\n> > > > > > > > };\n> > > > > > > > \n> > > > > > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > > \n> > > > > > > > Harness::~Harness()\n> > > > > > > > {\n> > > > > > > > -\tif (camera_) {\n> > > > > > > > -\t\tcamera_->release();\n> > > > > > > > -\t\tcamera_.reset();\n> > > > > > > > +\tfor (auto &c : cameras_) {\n> > > > > > > > +\t\tc->release();\n> > > > > > > > +\t\tc.reset();\n> > > > > > > > \t}\n> > > > > > > > \n> > > > > > > > \tcm_->stop();\n> > > > > > > > }\n> > > > > > > > \n> > > > > > > > -int Harness::exec()\n> > > > > > > > -{\n> > > > > > > > -\tint ret = init();\n> > > > > > > > -\tif (ret)\n> > > > > > > > -\t\treturn ret;\n> > > > > > > > -\n> > > > > > > > -\tstd::vector results;\n> > > > > > > > -\n> > > > > > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > > > > > -\n> > > > > > > > -\tfor (const Results &result : results) {\n> > > > > > > > -\t\tret = result.summary();\n> > > > > > > > -\t\tif (ret)\n> > > > > > > > -\t\t\treturn ret;\n> > > > > > > > -\t}\n> > > > > > > > -\n> > > > > > > > -\treturn 0;\n> > > > > > > > -}\n> > > > > > > > -\n> > > > > > > > int Harness::init()\n> > > > > > > > {\n> > > > > > > > \tint ret = cm_->start();\n> > > > > > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > > > > > \t\treturn ret;\n> > > > > > > > \t}\n> > > > > > > > \n> > > > > > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > > > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > > > > > -\t\tlistCameras();\n> > > > > > > > -\t\treturn -ENODEV;\n> > > > > > > > -\t}\n> > > > > > > > -\n> > > > > > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > > > > > -\tcamera_ = cm_->get(cameraId);\n> > > > > > > > -\tif (!camera_) {\n> > > > > > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > > > > > -\t\tlistCameras();\n> > > > > > > > -\t\treturn -ENODEV;\n> > > > > > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > > > > > +\t\tif (cam->acquire()) {\n> > > > > > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > > > > +\t\t\treturn -EINVAL;\n> > > > > > > > +\t\t}\n> > > > > > > > +\t\tcameras_.push_back(cam);\n> > > > > > >\n> > > > > > > I don't like this, I think we need to retain the ability to select which\n> > > > > > > camera to test. And further down the line maybe allow more then one\n> > > > > > > camera to be selected if we want to test concurrent use of two specific\n> > > > > > > ones. But for the work in this series I think you can ignore multiple\n> > > > > > > cameras.\n> > > > > > \n> > > > > > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > > > > > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > > > > > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > > > > > all cameras, only the ones that I will actually test (and in this code there's a\n> > > > > > big issue where if a single camera is already busy, even if it won't be tested,\n> > > > > > lc-compliance as a whole fails).\n> > > > >\n> > > > > Ahh I understand.\n> > > > >\n> > > > > > \n> > > > > > >\n> > > > > > > If I understand the coverletter correctly the correct thing here the cli\n> > > > > > > arguments should be feed to gtest directly to be able to filter on what\n> > > > > > > tests are run. Is there support in getst to also pass arguments to\n> > > > > > > tests? Maybe we could remove the need to parse arguments ourself\n> > > > > > > completely?\n> > > > > > \n> > > > > > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > > > > > could solve both problems (this and the one above) like this:\n> > > > > > \n> > > > > > - In the main(), query the id of each camera from libcamera.\n> > > > > > - Register each of the tests dynamically in gtest passing as an argument the\n> > > > > > camera id, rather than a camera pointer (which is the current behavior).\n> > > > > > - To select which tests to run, and on which camera(s), the user just uses\n> > > > > > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > > > > > the camera).\n> > > > > > - In the startup of each test the camera is acquired for that test, and when\n> > > > > > it's done it is released.\n> > > > > > \n> > > > > > Implications:\n> > > > > > - We no longer need to parse arguments and can drop that code, as both camera\n> > > > > > and test are specified directly with gtest's filter.\n> > > > > > - We only acquire the cameras that we'll actually test and only while they're\n> > > > > > being tested.\n> > > > > > - As a downside, a single camera will probably be acquired multiple times during\n> > > > > > a single run as lc-compliance moves from one test to the next, but I don't\n> > > > > > think this would be a big issue. (Maybe we could do this at the test suite\n> > > > > > level instead to greatly decrease the number of times of acquire(), but I'm\n> > > > > > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > > > > > the test level, I'll have to test that).\n> > > > > > \n> > > > > > What do you think?\n> > > > >\n> > > > > I understand what you are trying to do but I see two issues.\n> > > > >\n> > > > > 1. It depends on the implicit knowledge of users how to build the filter\n> > > > > string and is rather unintuitive compared to having two distinct\n> > > > > arguments, one to select camera and one to express the test filter.\n> > > > >\n> > > > > This could possibly be worked around by keeping the cam options code\n> > > > > and building the gtest filter string internally by injecitng the\n> > > > > selected camera name in the correct location, but the we have the\n> > > > > next issue,\n> > > > >\n> > > > > 2. I don't think it will be practically possible to express a test\n> > > > > filter that selects a specific combination of two cameras.\n> > > > >\n> > > > > I do think we need to be able to parameterize test cases. If gtest does\n> > > > > not allow to do so using function arguments there are other ways of\n> > > > > doing so.\n> > > > \n> > > > I think I didn't express myself well. I meant to say that gtest's CLI doesn't\n> > > > have a flag to allow passing parameters to tests, meaning that it would not be\n> > > > possible to parametrize the tests from the command line and also remove our\n> > > > own parser and rely on gtest's.\n> > >\n> > > Check, I think we need to keep our own parser for now as we need to be\n> > > able to parametrize the tests from the CLI.\n> > >\n> > > > But gtest does allow test parametrization at runtime and I'm already relying on\n> > > > it since v2. Let me go a bit more over the way I'm doing this.\n> > > > \n> > > > - I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n> > > > \n> > > > \tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n> > > > \n> > > > - Since we may want to run the same test for all cameras, roles and numRequests,\n> > > > I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n> > > > gtest's testing::RegisterTest() with all possible combinations of parameters.\n> > > > \n> > > > - Based on the --gtest_filter CLI argument from gtest, it decides which tests\n> > > > are actually run\n> > > > \n> > > > So the key point is that even though I parametrize the tests, I do it for all\n> > > > combinations in order to later rely on gtest_filter to select which tests to\n> > > > run.\n> > >\n> > > I see, that explains how the parameters get filled. I noticed it but did\n> > > not dig further, see below for an expansion of my thoughts now that I\n> > > understand this detail :-)\n> > >\n> > > > \n> > > > Now, I do agree with both of your points. And if we want to make the interface\n> > > > more intuitive to use (which I also find important), there's no way around\n> > > > having our own parser.\n> > > > \n> > > > And if we're using our own camera flag, then it makes no sense to register tests\n> > > > for all cameras, we can register tests only with the specified camera.\n> > >\n> > > This feels wrong. It feels like tests should be register without a\n> > > camera component. The camera (and possible other arguments) should come\n> > > from either arguments [1] or a singleton parameter class [2].\n> > >\n> > > 1. This do not seem to be possible with gtests as test parameters needs\n> > > to be registerd.\n> > >\n> > > 2. This would be similar to the global variable used in earlier versions\n> > > but instead modeled as a class singleton like CameraManager.\n> > >\n> > > Maybe there are other options tho.\n> > >\n> > > > \n> > > > I think here it would be helpful to think if the interface suits our problem.\n> > > > To me we're looking at having:\n> > > > \n> > > > \tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n> > >\n> > > This matches my view as well.\n> > >\n> > > > \n> > > > to only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\n> > > > the camera flag is used to only register tests with that camera, and the filter\n> > > > is passed as --gtest_filter, possibly a bit adapted, to gtest.\n> > > > \n> > > > And if we imagine we have a test ConcurrentCameras, which receives two cameras\n> > > > as parameters, and we call\n> > > > \n> > > > \tlc-compliance --camera 'cameraId1,cameraId2'\n> > >\n> > > This also looks good. But for now lets focus on one camera, I think it\n> > > will give us enough headache :-)\n> > >\n> > > > \n> > > > it would test both each of the cameras separately in all current tests which\n> > > > accept a single camera, like BalancedSingleCycle, as well as test both at the\n> > > > same time in the ConcurrentCameras test (and we could make only the former or\n> > > > the latter occur by using the filter parameter).\n> > >\n> > > I have not thought enough about this to know if that is a good idea or\n> > > not. At the top of my head I still tend to think that lc-compliance\n> > > tests a single 'primary' camera and any additional camera supplied are\n> > > only used for tests where two (or more) cameras are needed to test some\n> > > specific feature of the primary camera.\n> > >\n> > > That is only the 'primary' camera is testes, the other camera(s) are\n> > > just there to help/provoke the testing. But maybe I'm to narrow minded.\n> > \n> > Okay, so it now comes down to how lc-compliance will get used. Either:\n> > \n> > 1. lc-compliance will be called to test a single camera, with optionally other\n> > secondary cameras\n> > \n> > 2. lc-compliance will be called to test multiple cameras at once (not\n> > necessarily simultaneously, but in a single run), with the default case being\n> > \"Test all the cameras in this system\".\n> > \n> > Given that the final objective of my internship is to have lc-compliance running\n> > on real boards by KernelCI, to catch any kernel regressions that affect camera\n> > usage, I'm trying to think how it would work there. (I wonder if this discussion\n> > should be happening together with KernelCI to decide on this...)\n> > \n> > So I'm imagining a RockPi4 board with a camera sensor plugged in, running\n> > lc-compliance to catch any kernel regressions.\n> > \n> > The limitation I see in 1 is that if a system has multiple cameras and want to\n> > ensure that all keep working, it will need to either hardcode all camera ids and\n> > manually call lc-compliance once for each one, or have a separate program that\n> > lists all cameras from libcamera (using 'cam -l' for example) and calls\n> > lc-compliance with each one. That's to say, running on all cameras is not as\n> > easy. And then there are multiple reports, one for each camera, instead of a\n> > single one for all cameras, which might be good or bad, not sure.\n> > \n> > That said, since the RockPi4 (and also the Raspberry Pi 4) has a single CSI-2\n> > connector, I believe it only supports a single camera anyway, so this shouldn't\n> > be a problem for this board, but I'm not sure about others. Phones for example\n> > seem to have at least two independent cameras...\n>\n> I now understand your point of view better, thanks for brining up the\n> KernelCI angle. I agree from a KernelCI perspective the goal is likely\n> to test all cameras of the system to get the widest test base. But from\n> a (current) libcamera perspective the lc-complinance tool primary usage\n> is to be able to test a particular camera (or pipeline) to prove changes\n> do not break things, kind of like v4l2-complinace is for V4L2.\n>\n> That being said I think it's trivial for us to make lc-complinance\n> satisfy both use-cases. If the arg parsing is structured something like\n> this,\n>\n> if (cliargs.continas(\"--camera\")) {\n> run_tests_for_camera(cliargs.value_for(\"--camera\");\n> } else if (cliargs.continas(\"--run-all-cameras\")) {\n> for (camera : camer_manager.list_all_cameras()) {\n> run_tests_for_camera(camera)\n> }\n> }\n>\n> Or am I missing something?\n\nUnfortunately I don't think that would work. All tests would run fine, but the\nissue is in the results reporting at the end. Instead of having a single report,\nwith the right number of tests failed, skipped or succeeded and details for each\none, there would be N consecutive reports, where N is the number of cameras,\neach one reporting about its own camera. From a user perspective that would be\nvery confusing, since you'd need to scroll through each of the reports to check\nthe result for each camera.\n\nAnother issue is that for KernelCI we need machine-parsable results, which we\ncan do with --gtest_output=json to write the result in the JSON format into a\nfile, but then I bet each camera's result would overwrite the previous one\n(haven't tried it yet, though).\n\n(Although thinking a bit more about it now, we could take advantage of the fact\nthat --gtest_output=(json|xml)[:DIRECTORY_PATH/|:FILE_PATH] allows to set the\nfile name to make a sure each camera result is written to a different file, but\nthis still doesn't solve the first case with the text output to the terminal for\nthe user)\n\nSo I don't see any way to have lc-compliance work with multiple cameras and have\na single complete results report without relying on dynamic registration of\ntests for each camera.\n\nIn any case, given your answer in the second email that we could indeed use the\nsingleton pattern and have the camera globally available, meaning we could go\nback to static registration of tests, I think it's worth it to have\nlc-compliance focused on a single camera (with optional secondary ones) and push\nthe burden of how to test all cameras to KernelCI (which I'll probably be the\none to solve anyway :P) in exchange for a lot easier test registration (and\ngetting rid of that REGISTER_TESTS macro I wrote that I hated from the start :)\n).\n\nSo assuming you're okay with that, I'll get working on it for v3.\n\nThanks,\nNícolas\n\n>\n> > \n> > Regarding the benefit of 1, I'm not sure. If we were to use a global variable\n> > like in v1, it would make the code a lot simpler by making the test registration\n> > static. But that doesn't seem to be an option :). I'm not familiar with using a\n> > Singleton, so I'll take a look at CameraManager. But I assume it wouldn't be\n> > accessible statically, so the dynamic test registration complexities would have\n> > to stay.\n> > \n> > Also, just to make sure we're on the same page, by using a Singleton, and again\n> > assuming it isn't stored in a global variable to be accessible from the tests\n> > statically, even though we would eliminate the camera as a parameter for the\n> > tests, the reference to the singleton itself would need to be added as a\n> > parameter for the tests.\n>\n> I will follow up on this in your second mail.\n>\n> > \n> > Thanks,\n> > Nícolas\n> > \n> > >\n> > > > \n> > > > I know we don't need to implement support for multiple cameras right now, I just\n> > > > want to make sure it's possible and thinking about the interface and the\n> > > > scenarios help me picture how they'd work.\n> > > > \n> > > > Thoughts?\n> > > > \n> > > > Thanks,\n> > > > Nícolas\n> > > > \n> > > > >\n> > > > > I'm sure there are many solutions to this problem and maybe someone more\n> > > > > familiar with gtest knows of a neat way to piggyback on its\n> > > > > infrastrcuture? Since I know very little about the gtest framework the\n> > > > > following is just an idea and I'm happy to yield.\n> > > > >\n> > > > > How about using a singleton class that carries arguments constructed\n> > > > > from cli arguments (camera id) before the gtest are executed. Each test\n> > > > > case could then consult this singleton if it needs to access parameters.\n> > > > > I think you could take inspiration from the libcamera CameraManager\n> > > > > class.\n> > > > >\n> > > > > This could (at lest for now) imply keeping the cam option parser. If so\n> > > > > I think it would be nice if somewhen before we integrate this work that\n> > > > > the gtest filter string could be supplied as an argument using that\n> > > > > parser. Of course if we can do it the other way around and add arguments\n> > > > > to the gtest parser that would be even cooler as it would be less code\n> > > > > for us to maintain ;-)\n>\n> --\n> Regards,\n> Niklas Söderlund\n>\n> --\n> To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E5F1EC31E6\n\tfor ;\n\tWed, 12 May 2021 13:44:19 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5375E68919;\n\tWed, 12 May 2021 15:44:16 +0200 (CEST)","from bhuna.collabora.co.uk (bhuna.collabora.co.uk\n\t[IPv6:2a00:1098:0:82:1000:25:2eeb:e3e3])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E9DDF602B7\n\tfor ;\n\tWed, 12 May 2021 15:44:14 +0200 (CEST)","from localhost (unknown\n\t[IPv6:2804:14c:1a9:2978:985c:7892:ebcf:7a90])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits))\n\t(No client certificate requested) (Authenticated sender: nfraprado)\n\tby bhuna.collabora.co.uk (Postfix) with ESMTPSA id 6E9811F42685;\n\tWed, 12 May 2021 14:44:12 +0100 (BST)"],"Mime-Version":"1.0","Date":"Wed, 12 May 2021 10:43:35 -0300","Message-Id":"","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","From":"=?utf-8?b?TsOtY29sYXMgRi4gUi4gQS4gUHJhZG8=?= ","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t\n\t\n\t\n\t\n\t","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com, =?utf-8?q?A?=\n\t=?utf-8?q?ndr=C3=A9_Almeida?= ","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16932,"web_url":"https://patchwork.libcamera.org/comment/16932/","msgid":"","date":"2021-05-13T08:59:14","subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hello Nícolas,\n\nOn 2021-05-12 10:43:35 -0300, Nícolas F. R. A. Prado wrote:\n> Hi Niklas,\n> \n> Em 2021-05-12 05:49, Niklas Söderlund escreveu:\n> \n> > Hi Nícolas,\n> >\n> > On 2021-05-11 14:17:56 -0300, Nícolas F. R. A. Prado wrote:\n> > > Hi Niklas,\n> > > \n> > > Em 2021-05-10 18:04, Niklas Söderlund escreveu:\n> > > \n> > > > Hi Nícolas,\n> > > >\n> > > > Thanks for your work on this. I really think gtest is a step in the\n> > > > right direction!\n> > > >\n> > > > On 2021-05-07 12:52:38 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > Hi Niklas,\n> > > > > \n> > > > > Em 2021-05-07 06:11, Niklas Söderlund escreveu:\n> > > > > \n> > > > > > Hi Nícolas,\n> > > > > >\n> > > > > > On 2021-05-06 17:20:29 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > > Hi Niklas,\n> > > > > > > \n> > > > > > > thanks for the feedback.\n> > > > > > > \n> > > > > > > Em 2021-05-06 05:48, Niklas Söderlund escreveu:\n> > > > > > > \n> > > > > > > > Hi Nícolas,\n> > > > > > > >\n> > > > > > > > Thanks for your work!\n> > > > > > > >\n> > > > > > > > I really like how gtest helps make the actual tests to be easier to\n> > > > > > > > read.\n> > > > > > > >\n> > > > > > > > On 2021-05-03 16:33:07 -0300, Nícolas F. R. A. Prado wrote:\n> > > > > > > > > Refactor lc-compliance using Googletest as the test framework.\n> > > > > > > > > \n> > > > > > > > > Signed-off-by: Nícolas F. R. A. Prado \n> > > > > > > > > ---\n> > > > > > > > > Changes in v2:\n> > > > > > > > > - Changed from static to dynamic test registration\n> > > > > > > > > - Removed -c flag\n> > > > > > > > > \n> > > > > > > > > There's still an issue with the refactoring that the shared_ptr of the tests\n> > > > > > > > > apparently aren't being deleted on time, which causes:\n> > > > > > > > > \n> > > > > > > > > \t[20:26:08.744507935] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media0 while still in use\n> > > > > > > > > \t[20:26:08.744548175] [103243] ERROR DeviceEnumerator device_enumerator.cpp:165 Removing media device /dev/media1 while still in use\n> > > > > > > > > \tSegmentation fault (core dumped)\n> > > > > > > > > \n> > > > > > > > > I tried explicitly resetting the shared_ptr on the destructor of the tests but\n> > > > > > > > > that didn't work. In fact, when just listing the tests, the constructor for the\n> > > > > > > > > tests isn't even called. So I think the issue has to do with the passing of the\n> > > > > > > > > camera shared pointer to the REGISTER_TESTS() macro through that functor (?):\n> > > > > > > > > \n> > > > > > > > > \t[=]() -> testsuite* { return new testcase(c, r, rq); }); \\\n> > > > > > > > > \n> > > > > > > > > Not sure how to solve this. Any tip would be welcome.\n> > > > > > > >\n> > > > > > > > I will leave this for now as I have a different worry about\n> > > > > > > > REGISTER_TESTS() design below.\n> > > > > > > >\n> > > > > > > > > \n> > > > > > > > > src/lc-compliance/main.cpp | 99 +++++++++---------\n> > > > > > > > > src/lc-compliance/meson.build | 3 +\n> > > > > > > > > src/lc-compliance/simple_capture.cpp | 74 +++++--------\n> > > > > > > > > src/lc-compliance/simple_capture.h | 8 +-\n> > > > > > > > > src/lc-compliance/single_stream.cpp | 151 ++++++++++++++-------------\n> > > > > > > > > src/lc-compliance/tests.h | 15 ++-\n> > > > > > > > > 6 files changed, 175 insertions(+), 175 deletions(-)\n> > > > > > > > > \n> > > > > > > > > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > > > > > > > > index 54cee54aa978..add0d7729aec 100644\n> > > > > > > > > --- a/src/lc-compliance/main.cpp\n> > > > > > > > > +++ b/src/lc-compliance/main.cpp\n> > > > > > > > > @@ -9,6 +9,8 @@\n> > > > > > > > > #include \n> > > > > > > > > #include \n> > > > > > > > > \n> > > > > > > > > +#include \n> > > > > > > > > +\n> > > > > > > > > #include \n> > > > > > > > > \n> > > > > > > > > #include \"../cam/options.h\"\n> > > > > > > > > @@ -17,7 +19,6 @@\n> > > > > > > > > using namespace libcamera;\n> > > > > > > > > \n> > > > > > > > > enum {\n> > > > > > > > > -\tOptCamera = 'c',\n> > > > > > > > > \tOptHelp = 'h',\n> > > > > > > > > };\n> > > > > > > > > \n> > > > > > > > > @@ -28,14 +29,15 @@ public:\n> > > > > > > > > \t~Harness();\n> > > > > > > > > \n> > > > > > > > > \tint exec();\n> > > > > > > > > +\tint init();\n> > > > > > > > > +\tvoid registerTests();\n> > > > > > > > > \n> > > > > > > > > private:\n> > > > > > > > > -\tint init();\n> > > > > > > > > \tvoid listCameras();\n> > > > > > > > > \n> > > > > > > > > \tOptionsParser::Options options_;\n> > > > > > > > > \tstd::unique_ptr cm_;\n> > > > > > > > > -\tstd::shared_ptr camera_;\n> > > > > > > > > +\tstd::vector> cameras_;\n> > > > > > > > > };\n> > > > > > > > > \n> > > > > > > > > Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > > > @@ -46,33 +48,14 @@ Harness::Harness(const OptionsParser::Options &options)\n> > > > > > > > > \n> > > > > > > > > Harness::~Harness()\n> > > > > > > > > {\n> > > > > > > > > -\tif (camera_) {\n> > > > > > > > > -\t\tcamera_->release();\n> > > > > > > > > -\t\tcamera_.reset();\n> > > > > > > > > +\tfor (auto &c : cameras_) {\n> > > > > > > > > +\t\tc->release();\n> > > > > > > > > +\t\tc.reset();\n> > > > > > > > > \t}\n> > > > > > > > > \n> > > > > > > > > \tcm_->stop();\n> > > > > > > > > }\n> > > > > > > > > \n> > > > > > > > > -int Harness::exec()\n> > > > > > > > > -{\n> > > > > > > > > -\tint ret = init();\n> > > > > > > > > -\tif (ret)\n> > > > > > > > > -\t\treturn ret;\n> > > > > > > > > -\n> > > > > > > > > -\tstd::vector results;\n> > > > > > > > > -\n> > > > > > > > > -\tresults.push_back(testSingleStream(camera_));\n> > > > > > > > > -\n> > > > > > > > > -\tfor (const Results &result : results) {\n> > > > > > > > > -\t\tret = result.summary();\n> > > > > > > > > -\t\tif (ret)\n> > > > > > > > > -\t\t\treturn ret;\n> > > > > > > > > -\t}\n> > > > > > > > > -\n> > > > > > > > > -\treturn 0;\n> > > > > > > > > -}\n> > > > > > > > > -\n> > > > > > > > > int Harness::init()\n> > > > > > > > > {\n> > > > > > > > > \tint ret = cm_->start();\n> > > > > > > > > @@ -82,42 +65,26 @@ int Harness::init()\n> > > > > > > > > \t\treturn ret;\n> > > > > > > > > \t}\n> > > > > > > > > \n> > > > > > > > > -\tif (!options_.isSet(OptCamera)) {\n> > > > > > > > > -\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > > > > > > > > -\t\tlistCameras();\n> > > > > > > > > -\t\treturn -ENODEV;\n> > > > > > > > > -\t}\n> > > > > > > > > -\n> > > > > > > > > -\tconst std::string &cameraId = options_[OptCamera];\n> > > > > > > > > -\tcamera_ = cm_->get(cameraId);\n> > > > > > > > > -\tif (!camera_) {\n> > > > > > > > > -\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > > > > > > > > -\t\tlistCameras();\n> > > > > > > > > -\t\treturn -ENODEV;\n> > > > > > > > > +\tfor (auto cam : cm_->cameras()) {\n> > > > > > > > > +\t\tif (cam->acquire()) {\n> > > > > > > > > +\t\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > > > > > > > > +\t\t\treturn -EINVAL;\n> > > > > > > > > +\t\t}\n> > > > > > > > > +\t\tcameras_.push_back(cam);\n> > > > > > > >\n> > > > > > > > I don't like this, I think we need to retain the ability to select which\n> > > > > > > > camera to test. And further down the line maybe allow more then one\n> > > > > > > > camera to be selected if we want to test concurrent use of two specific\n> > > > > > > > ones. But for the work in this series I think you can ignore multiple\n> > > > > > > > cameras.\n> > > > > > > \n> > > > > > > Actually, that doesn't mean that multiple cameras will be tested, that's\n> > > > > > > determined by the test filter, so doing --gtest_filter='.*idCamera1.*' would\n> > > > > > > only test that camera. But I think I see what you mean, I shouldn't be acquiring\n> > > > > > > all cameras, only the ones that I will actually test (and in this code there's a\n> > > > > > > big issue where if a single camera is already busy, even if it won't be tested,\n> > > > > > > lc-compliance as a whole fails).\n> > > > > >\n> > > > > > Ahh I understand.\n> > > > > >\n> > > > > > > \n> > > > > > > >\n> > > > > > > > If I understand the coverletter correctly the correct thing here the cli\n> > > > > > > > arguments should be feed to gtest directly to be able to filter on what\n> > > > > > > > tests are run. Is there support in getst to also pass arguments to\n> > > > > > > > tests? Maybe we could remove the need to parse arguments ourself\n> > > > > > > > completely?\n> > > > > > > \n> > > > > > > There isn't any CLI argument in gtest to pass arguments to tests, but I think we\n> > > > > > > could solve both problems (this and the one above) like this:\n> > > > > > > \n> > > > > > > - In the main(), query the id of each camera from libcamera.\n> > > > > > > - Register each of the tests dynamically in gtest passing as an argument the\n> > > > > > > camera id, rather than a camera pointer (which is the current behavior).\n> > > > > > > - To select which tests to run, and on which camera(s), the user just uses\n> > > > > > > --gtest_filter='testToRun.*idCamera' (or '.*idCamera.*' to run all tests for\n> > > > > > > the camera).\n> > > > > > > - In the startup of each test the camera is acquired for that test, and when\n> > > > > > > it's done it is released.\n> > > > > > > \n> > > > > > > Implications:\n> > > > > > > - We no longer need to parse arguments and can drop that code, as both camera\n> > > > > > > and test are specified directly with gtest's filter.\n> > > > > > > - We only acquire the cameras that we'll actually test and only while they're\n> > > > > > > being tested.\n> > > > > > > - As a downside, a single camera will probably be acquired multiple times during\n> > > > > > > a single run as lc-compliance moves from one test to the next, but I don't\n> > > > > > > think this would be a big issue. (Maybe we could do this at the test suite\n> > > > > > > level instead to greatly decrease the number of times of acquire(), but I'm\n> > > > > > > not sure if it's possible, since we instantiate, by passing the camera id, at\n> > > > > > > the test level, I'll have to test that).\n> > > > > > > \n> > > > > > > What do you think?\n> > > > > >\n> > > > > > I understand what you are trying to do but I see two issues.\n> > > > > >\n> > > > > > 1. It depends on the implicit knowledge of users how to build the filter\n> > > > > > string and is rather unintuitive compared to having two distinct\n> > > > > > arguments, one to select camera and one to express the test filter.\n> > > > > >\n> > > > > > This could possibly be worked around by keeping the cam options code\n> > > > > > and building the gtest filter string internally by injecitng the\n> > > > > > selected camera name in the correct location, but the we have the\n> > > > > > next issue,\n> > > > > >\n> > > > > > 2. I don't think it will be practically possible to express a test\n> > > > > > filter that selects a specific combination of two cameras.\n> > > > > >\n> > > > > > I do think we need to be able to parameterize test cases. If gtest does\n> > > > > > not allow to do so using function arguments there are other ways of\n> > > > > > doing so.\n> > > > > \n> > > > > I think I didn't express myself well. I meant to say that gtest's CLI doesn't\n> > > > > have a flag to allow passing parameters to tests, meaning that it would not be\n> > > > > possible to parametrize the tests from the command line and also remove our\n> > > > > own parser and rely on gtest's.\n> > > >\n> > > > Check, I think we need to keep our own parser for now as we need to be\n> > > > able to parametrize the tests from the CLI.\n> > > >\n> > > > > But gtest does allow test parametrization at runtime and I'm already relying on\n> > > > > it since v2. Let me go a bit more over the way I'm doing this.\n> > > > > \n> > > > > - I have a test defined, eg BalancedSingleCycle, which takes some parameters:\n> > > > > \n> > > > > \tstd::shared_ptr camera, StreamRole role, unsigned int numRequests\n> > > > > \n> > > > > - Since we may want to run the same test for all cameras, roles and numRequests,\n> > > > > I register it, at runtime, using my REGISTER_TESTS() macro, which calls\n> > > > > gtest's testing::RegisterTest() with all possible combinations of parameters.\n> > > > > \n> > > > > - Based on the --gtest_filter CLI argument from gtest, it decides which tests\n> > > > > are actually run\n> > > > > \n> > > > > So the key point is that even though I parametrize the tests, I do it for all\n> > > > > combinations in order to later rely on gtest_filter to select which tests to\n> > > > > run.\n> > > >\n> > > > I see, that explains how the parameters get filled. I noticed it but did\n> > > > not dig further, see below for an expansion of my thoughts now that I\n> > > > understand this detail :-)\n> > > >\n> > > > > \n> > > > > Now, I do agree with both of your points. And if we want to make the interface\n> > > > > more intuitive to use (which I also find important), there's no way around\n> > > > > having our own parser.\n> > > > > \n> > > > > And if we're using our own camera flag, then it makes no sense to register tests\n> > > > > for all cameras, we can register tests only with the specified camera.\n> > > >\n> > > > This feels wrong. It feels like tests should be register without a\n> > > > camera component. The camera (and possible other arguments) should come\n> > > > from either arguments [1] or a singleton parameter class [2].\n> > > >\n> > > > 1. This do not seem to be possible with gtests as test parameters needs\n> > > > to be registerd.\n> > > >\n> > > > 2. This would be similar to the global variable used in earlier versions\n> > > > but instead modeled as a class singleton like CameraManager.\n> > > >\n> > > > Maybe there are other options tho.\n> > > >\n> > > > > \n> > > > > I think here it would be helpful to think if the interface suits our problem.\n> > > > > To me we're looking at having:\n> > > > > \n> > > > > \tlc-compliance --camera 'cameraId' --filter 'BalancedSingleCycle'\n> > > >\n> > > > This matches my view as well.\n> > > >\n> > > > > \n> > > > > to only test camera 'cameraId' with a single test 'BalancedSingleCycle'. Here,\n> > > > > the camera flag is used to only register tests with that camera, and the filter\n> > > > > is passed as --gtest_filter, possibly a bit adapted, to gtest.\n> > > > > \n> > > > > And if we imagine we have a test ConcurrentCameras, which receives two cameras\n> > > > > as parameters, and we call\n> > > > > \n> > > > > \tlc-compliance --camera 'cameraId1,cameraId2'\n> > > >\n> > > > This also looks good. But for now lets focus on one camera, I think it\n> > > > will give us enough headache :-)\n> > > >\n> > > > > \n> > > > > it would test both each of the cameras separately in all current tests which\n> > > > > accept a single camera, like BalancedSingleCycle, as well as test both at the\n> > > > > same time in the ConcurrentCameras test (and we could make only the former or\n> > > > > the latter occur by using the filter parameter).\n> > > >\n> > > > I have not thought enough about this to know if that is a good idea or\n> > > > not. At the top of my head I still tend to think that lc-compliance\n> > > > tests a single 'primary' camera and any additional camera supplied are\n> > > > only used for tests where two (or more) cameras are needed to test some\n> > > > specific feature of the primary camera.\n> > > >\n> > > > That is only the 'primary' camera is testes, the other camera(s) are\n> > > > just there to help/provoke the testing. But maybe I'm to narrow minded.\n> > > \n> > > Okay, so it now comes down to how lc-compliance will get used. Either:\n> > > \n> > > 1. lc-compliance will be called to test a single camera, with optionally other\n> > > secondary cameras\n> > > \n> > > 2. lc-compliance will be called to test multiple cameras at once (not\n> > > necessarily simultaneously, but in a single run), with the default case being\n> > > \"Test all the cameras in this system\".\n> > > \n> > > Given that the final objective of my internship is to have lc-compliance running\n> > > on real boards by KernelCI, to catch any kernel regressions that affect camera\n> > > usage, I'm trying to think how it would work there. (I wonder if this discussion\n> > > should be happening together with KernelCI to decide on this...)\n> > > \n> > > So I'm imagining a RockPi4 board with a camera sensor plugged in, running\n> > > lc-compliance to catch any kernel regressions.\n> > > \n> > > The limitation I see in 1 is that if a system has multiple cameras and want to\n> > > ensure that all keep working, it will need to either hardcode all camera ids and\n> > > manually call lc-compliance once for each one, or have a separate program that\n> > > lists all cameras from libcamera (using 'cam -l' for example) and calls\n> > > lc-compliance with each one. That's to say, running on all cameras is not as\n> > > easy. And then there are multiple reports, one for each camera, instead of a\n> > > single one for all cameras, which might be good or bad, not sure.\n> > > \n> > > That said, since the RockPi4 (and also the Raspberry Pi 4) has a single CSI-2\n> > > connector, I believe it only supports a single camera anyway, so this shouldn't\n> > > be a problem for this board, but I'm not sure about others. Phones for example\n> > > seem to have at least two independent cameras...\n> >\n> > I now understand your point of view better, thanks for brining up the\n> > KernelCI angle. I agree from a KernelCI perspective the goal is likely\n> > to test all cameras of the system to get the widest test base. But from\n> > a (current) libcamera perspective the lc-complinance tool primary usage\n> > is to be able to test a particular camera (or pipeline) to prove changes\n> > do not break things, kind of like v4l2-complinace is for V4L2.\n> >\n> > That being said I think it's trivial for us to make lc-complinance\n> > satisfy both use-cases. If the arg parsing is structured something like\n> > this,\n> >\n> > if (cliargs.continas(\"--camera\")) {\n> > run_tests_for_camera(cliargs.value_for(\"--camera\");\n> > } else if (cliargs.continas(\"--run-all-cameras\")) {\n> > for (camera : camer_manager.list_all_cameras()) {\n> > run_tests_for_camera(camera)\n> > }\n> > }\n> >\n> > Or am I missing something?\n> \n> Unfortunately I don't think that would work. All tests would run fine, but the\n> issue is in the results reporting at the end. Instead of having a single report,\n> with the right number of tests failed, skipped or succeeded and details for each\n> one, there would be N consecutive reports, where N is the number of cameras,\n> each one reporting about its own camera. From a user perspective that would be\n> very confusing, since you'd need to scroll through each of the reports to check\n> the result for each camera.\n\nGood point, for some reason I thought we could aggregate each test run \nand create a lc-compliance report at the end. But as you point out maybe \nit's best to keep a one camera test view.\n\n> \n> Another issue is that for KernelCI we need machine-parsable results, which we\n> can do with --gtest_output=json to write the result in the JSON format into a\n> file, but then I bet each camera's result would overwrite the previous one\n> (haven't tried it yet, though).\n> \n> (Although thinking a bit more about it now, we could take advantage of the fact\n> that --gtest_output=(json|xml)[:DIRECTORY_PATH/|:FILE_PATH] allows to set the\n> file name to make a sure each camera result is written to a different file, but\n> this still doesn't solve the first case with the text output to the terminal for\n> the user)\n\nFor KernelCI this could be the way forward. A worst case maybe the \noutput files from multiple lc-compliance runs can be aggregated into a \nsingle report file and then feed back to KernelCI?\n\n> \n> So I don't see any way to have lc-compliance work with multiple cameras and have\n> a single complete results report without relying on dynamic registration of\n> tests for each camera.\n> \n> In any case, given your answer in the second email that we could indeed use the\n> singleton pattern and have the camera globally available, meaning we could go\n> back to static registration of tests, I think it's worth it to have\n> lc-compliance focused on a single camera (with optional secondary ones) and push\n> the burden of how to test all cameras to KernelCI (which I'll probably be the\n> one to solve anyway :P) in exchange for a lot easier test registration (and\n> getting rid of that REGISTER_TESTS macro I wrote that I hated from the start :)\n> ).\n> \n> So assuming you're okay with that, I'll get working on it for v3.\n\nI think this sounds like a good way forward! Thanks again for working on \nthis, I think switching to gtest really is a good step in the right \ndirection.\n\n> \n> Thanks,\n> Nícolas\n> \n> >\n> > > \n> > > Regarding the benefit of 1, I'm not sure. If we were to use a global variable\n> > > like in v1, it would make the code a lot simpler by making the test registration\n> > > static. But that doesn't seem to be an option :). I'm not familiar with using a\n> > > Singleton, so I'll take a look at CameraManager. But I assume it wouldn't be\n> > > accessible statically, so the dynamic test registration complexities would have\n> > > to stay.\n> > > \n> > > Also, just to make sure we're on the same page, by using a Singleton, and again\n> > > assuming it isn't stored in a global variable to be accessible from the tests\n> > > statically, even though we would eliminate the camera as a parameter for the\n> > > tests, the reference to the singleton itself would need to be added as a\n> > > parameter for the tests.\n> >\n> > I will follow up on this in your second mail.\n> >\n> > > \n> > > Thanks,\n> > > Nícolas\n> > > \n> > > >\n> > > > > \n> > > > > I know we don't need to implement support for multiple cameras right now, I just\n> > > > > want to make sure it's possible and thinking about the interface and the\n> > > > > scenarios help me picture how they'd work.\n> > > > > \n> > > > > Thoughts?\n> > > > > \n> > > > > Thanks,\n> > > > > Nícolas\n> > > > > \n> > > > > >\n> > > > > > I'm sure there are many solutions to this problem and maybe someone more\n> > > > > > familiar with gtest knows of a neat way to piggyback on its\n> > > > > > infrastrcuture? Since I know very little about the gtest framework the\n> > > > > > following is just an idea and I'm happy to yield.\n> > > > > >\n> > > > > > How about using a singleton class that carries arguments constructed\n> > > > > > from cli arguments (camera id) before the gtest are executed. Each test\n> > > > > > case could then consult this singleton if it needs to access parameters.\n> > > > > > I think you could take inspiration from the libcamera CameraManager\n> > > > > > class.\n> > > > > >\n> > > > > > This could (at lest for now) imply keeping the cam option parser. If so\n> > > > > > I think it would be nice if somewhen before we integrate this work that\n> > > > > > the gtest filter string could be supplied as an argument using that\n> > > > > > parser. Of course if we can do it the other way around and add arguments\n> > > > > > to the gtest parser that would be even cooler as it would be less code\n> > > > > > for us to maintain ;-)\n> >\n> > --\n> > Regards,\n> > Niklas Söderlund\n> >\n> > --\n> > To unsubscribe, send mail to kernel-unsubscribe@lists.collabora.co.uk.\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 25288C31EE\n\tfor ;\n\tThu, 13 May 2021 08:59:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 75A6968920;\n\tThu, 13 May 2021 10:59:19 +0200 (CEST)","from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com\n\t[IPv6:2a00:1450:4864:20::12f])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 980AB68919\n\tfor ;\n\tThu, 13 May 2021 10:59:17 +0200 (CEST)","by mail-lf1-x12f.google.com with SMTP id a2so16567745lfc.9\n\tfor ;\n\tThu, 13 May 2021 01:59:17 -0700 (PDT)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tu27sm226838lfm.239.2021.05.13.01.59.15\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tThu, 13 May 2021 01:59:15 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"rq4n44nL\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=O3L3ADbaV4tGDwJIAK61HyQj1uUi9IDhFDyjqWtY3ec=;\n\tb=rq4n44nLnmkGUMNgBteTT1KusCdMoAuzI58vAdSiDQkkX+vIZD0CBaTu6hWo2PnavJ\n\t3nwSH7ENOf46D5e9OJB+L0yXcLMndp57n36Z/kPT0WobsthY05PeTTAk0R85QX3rL/8v\n\t833OOIRH1jT50x85CkzHkUmauG2ELdNtQvX5KNLcwt1qbAJo9Lk5VYOjNGier6/M4v6n\n\tQiIMBw9vON5bCHHiqRN+Hj48UHtekxUJzva1pFGvhh0G+TwFHFZ2hbfu2mWN4kJPoiKY\n\tdzjTgxqULQBakPSCS5rasT1EAsD012VQOeTYyuMqZZv/W4Zv2nVDpiOoufphsG/uTxi2\n\tGq4g==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=O3L3ADbaV4tGDwJIAK61HyQj1uUi9IDhFDyjqWtY3ec=;\n\tb=MzUP8gaCdYUTQ4jRgYtCh6/JVPtkRmZ4re/iCrn1Th3O0ssWnbIY4KwyqwxKA6He5Z\n\tQ340hLcHLNv15LgbNO6bLKhmi/txna7+08MX8wI7cLk9LQUXaEIwyYlFnIk9ilH1yhuH\n\tWOlJuHRJWdwNgYSDk9mxyGvhIVo5MJuo759Mb5TvNQm0ssu+ELkas1fKqxSwmLxMn20C\n\tPFrArbP0p6ml91+RCkuwwraWsBXYspOSyfbXxSnC9rSnunr+e2JqZwr/QEgg6Blb+m2G\n\tyQI2Gn1kGvVBrGNo0xymwPINWdHEjv2JBpoKpW4e7xe/LadtCsU0Z8+KfcMmsDbShVhg\n\teOKw==","X-Gm-Message-State":"AOAM531VIe8OyjD+YwfNIX9hGPu8nwCXskiISjy2MIdezsAwHTe/Qlwk\n\tpFagljIf0n+IJ3VlAedQDUeIFw==","X-Google-Smtp-Source":"ABdhPJw+hQYC3kAZBDEbZuJWMAEhhVc+3OXKtIIVULSPuLRaEZ3ih6fTbxA5KsNYLI4BhRtUWZVYcA==","X-Received":"by 2002:ac2:5456:: with SMTP id\n\td22mr16801205lfn.383.1620896356340; \n\tThu, 13 May 2021 01:59:16 -0700 (PDT)","Date":"Thu, 13 May 2021 10:59:14 +0200","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"=?iso-8859-1?q?N=EDcolas_F=2E_R=2E_A=2E?= Prado ","Message-ID":"","References":"<20210503193307.108607-1-nfraprado@collabora.com>\n\t<20210503193307.108607-4-nfraprado@collabora.com>\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH v2 3/3] lc-compliance: Refactor\n\tusing Googletest","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org, kernel@collabora.com,\n\t=?iso-8859-1?q?Andr=E9?= Almeida ","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]