[{"id":16167,"web_url":"https://patchwork.libcamera.org/comment/16167/","msgid":"","date":"2021-04-11T12:00:14","subject":"Re: [libcamera-devel] [PATCH v5 1/2] lc-compliance: Add a libcamera\n\tcompliance tool","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Niklas,\n\nThanks for th\n\nOn 09/04/2021 17:00, Niklas Söderlund wrote:\n> Add a compliance tool to ease testing of cameras. In contrast to the\n> unit-tests under test/ that aims to test the internal components of\n> libcamera the compliance tool aims to test application use-cases and to\n> some extent the public API.\n> \n> This change adds the boilerplate code of a simple framework for the\n> creation of tests. The tests aim both to demonstrate the tool and to\n> catch real problems. The tests added are:\n> \n> - Test that if one queues exactly N requests to a camera exactly N\n> requests are eventually completed.\n> \n> - Test that a configured camera can be started and stopped multiple\n> times in an attempt to exercise cleanup code paths otherwise not\n> often tested with 'cam' for example.\n> \n> Signed-off-by: Niklas Söderlund \n> Acked-by: Kieran Bingham \n> Tested-by: Kieran Bingham \n> Reviewed-by: Laurent Pinchart \n\nBoth demo cases have help me a lot, and still do, and i can't wait to\nsee this tool merged !\nTested-by: Jean-Michel Hautbois \n\n> ---\n> * Changes since v1\n> - Improve language in commit message and comments.\n> - Test all roles as they may exercise different code paths in the\n> pipeline.\n> - Move SimpleCapture to its own .cpp/.h files.\n> \n> * Changes since v2\n> - Fold in a use-after-free bug fix from Kieran, thanks!\n> \n> * Changes since v3\n> - Update commit message.\n> - Moved command line option parsing to main() instead of in Harness.\n> - Return exit code 0 when display help due to -h.\n> - Rework to allow for a listCameras() function.\n> - Drop options empty check so available cameras are listed if no option\n> is given.\n> - Use EXIT_SUCCESS instead of 0.\n> - Style logging as \"PASS: foo\" instead of \"PASS - foo\".\n> - Include utility for std::pair.\n> - Add todo to not forget to check if result aggregator can be shared\n> with test/.\n> - Do not keep Result construction as one lines.\n> - Make const arrays static const.\n> - Remove unneeded camera exists check.\n> \n> * Changes since v4\n> - Fold libevent check back into src/lc-compliance/meson.build\n> - Add lc-compliance meson option.\n> - Make parseOptions() static.\n> - Simplify return codes from parseOptions().\n> - Fix indentation in meson.build.\n> ---\n> meson.build | 1 +\n> meson_options.txt | 5 +\n> src/lc-compliance/main.cpp | 148 ++++++++++++++++++++++++++\n> src/lc-compliance/meson.build | 27 +++++\n> src/lc-compliance/results.cpp | 75 +++++++++++++\n> src/lc-compliance/results.h | 47 +++++++++\n> src/lc-compliance/simple_capture.cpp | 151 +++++++++++++++++++++++++++\n> src/lc-compliance/simple_capture.h | 54 ++++++++++\n> src/lc-compliance/single_stream.cpp | 74 +++++++++++++\n> src/lc-compliance/tests.h | 16 +++\n> src/meson.build | 2 +\n> 11 files changed, 600 insertions(+)\n> create mode 100644 src/lc-compliance/main.cpp\n> create mode 100644 src/lc-compliance/meson.build\n> create mode 100644 src/lc-compliance/results.cpp\n> create mode 100644 src/lc-compliance/results.h\n> create mode 100644 src/lc-compliance/simple_capture.cpp\n> create mode 100644 src/lc-compliance/simple_capture.h\n> create mode 100644 src/lc-compliance/single_stream.cpp\n> create mode 100644 src/lc-compliance/tests.h\n> \n> diff --git a/meson.build b/meson.build\n> index 3a615ae37c8a0fb8..13d88301259d7077 100644\n> --- a/meson.build\n> +++ b/meson.build\n> @@ -173,5 +173,6 @@ summary({\n> 'V4L2 emulation support': v4l2_enabled,\n> 'cam application': cam_enabled,\n> 'qcam application': qcam_enabled,\n> + 'lc-compliance application': lc_compliance_enabled,\n> 'Unit tests': test_enabled,\n> }, section : 'Configuration')\n> diff --git a/meson_options.txt b/meson_options.txt\n> index 12de8fa161f8e64f..ee544513a98d1638 100644\n> --- a/meson_options.txt\n> +++ b/meson_options.txt\n> @@ -16,6 +16,11 @@ option('cam',\n> value : 'auto',\n> description : 'Compile the cam test application')\n> \n> +option('lc-compliance',\n> + type : 'feature',\n> + value : 'auto',\n> + description : 'Compile the lc-compliance test application')\n> +\n> option('documentation',\n> type : 'feature',\n> description : 'Generate the project documentation')\n> diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> new file mode 100644\n> index 0000000000000000..54cee54aa9788a88\n> --- /dev/null\n> +++ b/src/lc-compliance/main.cpp\n> @@ -0,0 +1,148 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * main.cpp - lc-compliance - The libcamera compliance tool\n> + */\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"../cam/options.h\"\n> +#include \"tests.h\"\n> +\n> +using namespace libcamera;\n> +\n> +enum {\n> +\tOptCamera = 'c',\n> +\tOptHelp = 'h',\n> +};\n> +\n> +class Harness\n> +{\n> +public:\n> +\tHarness(const OptionsParser::Options &options);\n> +\t~Harness();\n> +\n> +\tint exec();\n> +\n> +private:\n> +\tint init();\n> +\tvoid listCameras();\n> +\n> +\tOptionsParser::Options options_;\n> +\tstd::unique_ptr cm_;\n> +\tstd::shared_ptr camera_;\n> +};\n> +\n> +Harness::Harness(const OptionsParser::Options &options)\n> +\t: options_(options)\n> +{\n> +\tcm_ = std::make_unique();\n> +}\n> +\n> +Harness::~Harness()\n> +{\n> +\tif (camera_) {\n> +\t\tcamera_->release();\n> +\t\tcamera_.reset();\n> +\t}\n> +\n> +\tcm_->stop();\n> +}\n> +\n> +int Harness::exec()\n> +{\n> +\tint ret = init();\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tstd::vector results;\n> +\n> +\tresults.push_back(testSingleStream(camera_));\n> +\n> +\tfor (const Results &result : results) {\n> +\t\tret = result.summary();\n> +\t\tif (ret)\n> +\t\t\treturn ret;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +int Harness::init()\n> +{\n> +\tint ret = cm_->start();\n> +\tif (ret) {\n> +\t\tstd::cout << \"Failed to start camera manager: \"\n> +\t\t\t << strerror(-ret) << std::endl;\n> +\t\treturn ret;\n> +\t}\n> +\n> +\tif (!options_.isSet(OptCamera)) {\n> +\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> +\t\tlistCameras();\n> +\t\treturn -ENODEV;\n> +\t}\n> +\n> +\tconst std::string &cameraId = options_[OptCamera];\n> +\tcamera_ = cm_->get(cameraId);\n> +\tif (!camera_) {\n> +\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> +\t\tlistCameras();\n> +\t\treturn -ENODEV;\n> +\t}\n> +\n> +\tif (camera_->acquire()) {\n> +\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> +\t\treturn -EINVAL;\n> +\t}\n> +\n> +\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> +\n> +\treturn 0;\n> +}\n> +\n> +void Harness::listCameras()\n> +{\n> +\tfor (const std::shared_ptr &cam : cm_->cameras())\n> +\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> +}\n> +\n> +static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> +{\n> +\tOptionsParser parser;\n> +\tparser.addOption(OptCamera, OptionString,\n> +\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> +\t\t\t ArgumentRequired, \"camera\");\n> +\tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> +\t\t\t \"help\");\n> +\n> +\t*options = parser.parse(argc, argv);\n> +\tif (!options->valid())\n> +\t\treturn -EINVAL;\n> +\n> +\tif (options->isSet(OptHelp)) {\n> +\t\tparser.usage();\n> +\t\treturn -EINTR;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +int main(int argc, char **argv)\n> +{\n> +\tOptionsParser::Options options;\n> +\tint ret = parseOptions(argc, argv, &options);\n> +\tif (ret == -EINTR)\n> +\t\treturn EXIT_SUCCESS;\n> +\tif (ret < 0)\n> +\t\treturn EXIT_FAILURE;\n> +\n> +\tHarness harness(options);\n> +\n> +\treturn harness.exec() ? EXIT_FAILURE : EXIT_SUCCESS;\n> +}\n> diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> new file mode 100644\n> index 0000000000000000..a2bfcceb1259d12e\n> --- /dev/null\n> +++ b/src/lc-compliance/meson.build\n> @@ -0,0 +1,27 @@\n> +# SPDX-License-Identifier: CC0-1.0\n> +\n> +libevent = dependency('libevent_pthreads', required : get_option('lc-compliance'))\n> +\n> +if not libevent.found()\n> + lc_compliance_enabled = false\n> + subdir_done()\n> +endif\n> +\n> +lc_compliance_enabled = true\n> +\n> +lc_compliance_sources = files([\n> + '../cam/event_loop.cpp',\n> + '../cam/options.cpp',\n> + 'main.cpp',\n> + 'results.cpp',\n> + 'simple_capture.cpp',\n> + 'single_stream.cpp',\n> +])\n> +\n> +lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> + dependencies : [\n> + libatomic,\n> + libcamera_dep,\n> + libevent,\n> + ],\n> + install : true)\n> diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp\n> new file mode 100644\n> index 0000000000000000..f149f7859e286b8f\n> --- /dev/null\n> +++ b/src/lc-compliance/results.cpp\n> @@ -0,0 +1,75 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * results.cpp - Test result aggregator\n> + */\n> +\n> +#include \"results.h\"\n> +\n> +#include \n> +\n> +void Results::add(const Result &result)\n> +{\n> +\tif (result.first == Pass)\n> +\t\tpassed_++;\n> +\telse if (result.first == Fail)\n> +\t\tfailed_++;\n> +\telse if (result.first == Skip)\n> +\t\tskipped_++;\n> +\n> +\tprintResult(result);\n> +}\n> +\n> +void Results::add(Status status, const std::string &message)\n> +{\n> +\tadd({ status, message });\n> +}\n> +\n> +void Results::fail(const std::string &message)\n> +{\n> +\tadd(Fail, message);\n> +}\n> +\n> +void Results::pass(const std::string &message)\n> +{\n> +\tadd(Pass, message);\n> +}\n> +\n> +void Results::skip(const std::string &message)\n> +{\n> +\tadd(Skip, message);\n> +}\n> +\n> +int Results::summary() const\n> +{\n> +\tif (failed_ + passed_ + skipped_ != planned_) {\n> +\t\tstd::cout << \"Planned and executed number of tests differ \"\n> +\t\t\t << failed_ + passed_ + skipped_ << \" executed \"\n> +\t\t\t << planned_ << \" planned\" << std::endl;\n> +\n> +\t\treturn -EINVAL;\n> +\t}\n> +\n> +\tstd::cout << planned_ << \" tests executed, \"\n> +\t\t << passed_ << \" tests passed, \"\n> +\t\t << skipped_ << \" tests skipped and \"\n> +\t\t << failed_ << \" tests failed \" << std::endl;\n> +\n> +\treturn 0;\n> +}\n> +\n> +void Results::printResult(const Result &result)\n> +{\n> +\tstd::string prefix;\n> +\n> +\t/* \\todo Make parsable as TAP. */\n> +\tif (result.first == Pass)\n> +\t\tprefix = \"PASS\";\n> +\telse if (result.first == Fail)\n> +\t\tprefix = \"FAIL\";\n> +\telse if (result.first == Skip)\n> +\t\tprefix = \"SKIP\";\n> +\n> +\tstd::cout << \"- \" << prefix << \": \" << result.second << std::endl;\n> +}\n> diff --git a/src/lc-compliance/results.h b/src/lc-compliance/results.h\n> new file mode 100644\n> index 0000000000000000..2a3722b841a6410a\n> --- /dev/null\n> +++ b/src/lc-compliance/results.h\n> @@ -0,0 +1,47 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * results.h - Test result aggregator\n> + */\n> +#ifndef __LC_COMPLIANCE_RESULTS_H__\n> +#define __LC_COMPLIANCE_RESULTS_H__\n> +\n> +#include \n> +#include \n> +\n> +/* \\todo Check if result aggregator can be shared with self tests in test/ */\n> +class Results\n> +{\n> +public:\n> +\tenum Status {\n> +\t\tFail,\n> +\t\tPass,\n> +\t\tSkip,\n> +\t};\n> +\n> +\tusing Result = std::pair;\n> +\n> +\tResults(unsigned int planned)\n> +\t\t: planned_(planned), passed_(0), failed_(0), skipped_(0)\n> +\t{\n> +\t}\n> +\n> +\tvoid add(const Result &result);\n> +\tvoid add(Status status, const std::string &message);\n> +\tvoid fail(const std::string &message);\n> +\tvoid pass(const std::string &message);\n> +\tvoid skip(const std::string &message);\n> +\n> +\tint summary() const;\n> +\n> +private:\n> +\tvoid printResult(const Result &result);\n> +\n> +\tunsigned int planned_;\n> +\tunsigned int passed_;\n> +\tunsigned int failed_;\n> +\tunsigned int skipped_;\n> +};\n> +\n> +#endif /* __LC_COMPLIANCE_RESULTS_H__ */\n> diff --git a/src/lc-compliance/simple_capture.cpp b/src/lc-compliance/simple_capture.cpp\n> new file mode 100644\n> index 0000000000000000..389dc11fc60225c5\n> --- /dev/null\n> +++ b/src/lc-compliance/simple_capture.cpp\n> @@ -0,0 +1,151 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * simple_capture.cpp - Simple capture helper\n> + */\n> +\n> +#include \"simple_capture.h\"\n> +\n> +using namespace libcamera;\n> +\n> +SimpleCapture::SimpleCapture(std::shared_ptr camera)\n> +\t: camera_(camera), allocator_(std::make_unique(camera))\n> +{\n> +}\n> +\n> +SimpleCapture::~SimpleCapture()\n> +{\n> +}\n> +\n> +Results::Result SimpleCapture::configure(StreamRole role)\n> +{\n> +\tconfig_ = camera_->generateConfiguration({ role });\n> +\n> +\tif (config_->validate() != CameraConfiguration::Valid) {\n> +\t\tconfig_.reset();\n> +\t\treturn { Results::Fail, \"Configuration not valid\" };\n> +\t}\n> +\n> +\tif (camera_->configure(config_.get())) {\n> +\t\tconfig_.reset();\n> +\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> +\t}\n> +\n> +\treturn { Results::Pass, \"Configure camera\" };\n> +}\n> +\n> +Results::Result SimpleCapture::start()\n> +{\n> +\tStream *stream = config_->at(0).stream();\n> +\tif (allocator_->allocate(stream) < 0)\n> +\t\treturn { Results::Fail, \"Failed to allocate buffers\" };\n> +\n> +\tif (camera_->start())\n> +\t\treturn { Results::Fail, \"Failed to start camera\" };\n> +\n> +\tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> +\n> +\treturn { Results::Pass, \"Started camera\" };\n> +}\n> +\n> +Results::Result SimpleCapture::stop()\n> +{\n> +\tStream *stream = config_->at(0).stream();\n> +\n> +\tcamera_->stop();\n> +\n> +\tcamera_->requestCompleted.disconnect(this, &SimpleCapture::requestComplete);\n> +\n> +\tallocator_->free(stream);\n> +\n> +\treturn { Results::Pass, \"Stopped camera\" };\n> +}\n> +\n> +/* SimpleCaptureBalanced */\n> +\n> +SimpleCaptureBalanced::SimpleCaptureBalanced(std::shared_ptr camera)\n> +\t: SimpleCapture(camera)\n> +{\n> +}\n> +\n> +Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> +{\n> +\tResults::Result ret = start();\n> +\tif (ret.first != Results::Pass)\n> +\t\treturn ret;\n> +\n> +\tStream *stream = config_->at(0).stream();\n> +\tconst std::vector> &buffers = allocator_->buffers(stream);\n> +\n> +\t/* No point in testing less requests then the camera depth. */\n> +\tif (buffers.size() > numRequests) {\n> +\t\t/* Cache buffers.size() before we destroy it in stop() */\n> +\t\tint buffers_size = buffers.size();\n> +\t\tstop();\n> +\n> +\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers_size)\n> +\t\t\t+ \" requests, can't test only \" + std::to_string(numRequests) };\n> +\t}\n> +\n> +\tqueueCount_ = 0;\n> +\tcaptureCount_ = 0;\n> +\tcaptureLimit_ = numRequests;\n> +\n> +\t/* Queue the recommended number of reqeuests. */\n> +\tstd::vector> requests;\n> +\tfor (const std::unique_ptr &buffer : buffers) {\n> +\t\tstd::unique_ptr request = camera_->createRequest();\n> +\t\tif (!request) {\n> +\t\t\tstop();\n> +\t\t\treturn { Results::Fail, \"Can't create request\" };\n> +\t\t}\n> +\n> +\t\tif (request->addBuffer(stream, buffer.get())) {\n> +\t\t\tstop();\n> +\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> +\t\t}\n> +\n> +\t\tif (queueRequest(request.get()) < 0) {\n> +\t\t\tstop();\n> +\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> +\t\t}\n> +\n> +\t\trequests.push_back(std::move(request));\n> +\t}\n> +\n> +\t/* Run capture session. */\n> +\tloop_ = new EventLoop();\n> +\tloop_->exec();\n> +\tstop();\n> +\tdelete loop_;\n> +\n> +\tif (captureCount_ != captureLimit_)\n> +\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) +\n> +\t\t\t\" request, wanted \" + std::to_string(captureLimit_) };\n> +\n> +\treturn { Results::Pass, \"Balanced capture of \" +\n> +\t\tstd::to_string(numRequests) + \" requests\" };\n> +}\n> +\n> +int SimpleCaptureBalanced::queueRequest(Request *request)\n> +{\n> +\tqueueCount_++;\n> +\tif (queueCount_ > captureLimit_)\n> +\t\treturn 0;\n> +\n> +\treturn camera_->queueRequest(request);\n> +}\n> +\n> +void SimpleCaptureBalanced::requestComplete(Request *request)\n> +{\n> +\tcaptureCount_++;\n> +\tif (captureCount_ >= captureLimit_) {\n> +\t\tloop_->exit(0);\n> +\t\treturn;\n> +\t}\n> +\n> +\trequest->reuse(Request::ReuseBuffers);\n> +\tif (queueRequest(request))\n> +\t\tloop_->exit(-EINVAL);\n> +}\n> diff --git a/src/lc-compliance/simple_capture.h b/src/lc-compliance/simple_capture.h\n> new file mode 100644\n> index 0000000000000000..3a6afc538c623050\n> --- /dev/null\n> +++ b/src/lc-compliance/simple_capture.h\n> @@ -0,0 +1,54 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * simple_capture.h - Simple capture helper\n> + */\n> +#ifndef __LC_COMPLIANCE_SIMPLE_CAPTURE_H__\n> +#define __LC_COMPLIANCE_SIMPLE_CAPTURE_H__\n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"../cam/event_loop.h\"\n> +#include \"results.h\"\n> +\n> +class SimpleCapture\n> +{\n> +public:\n> +\tResults::Result configure(libcamera::StreamRole role);\n> +\n> +protected:\n> +\tSimpleCapture(std::shared_ptr camera);\n> +\tvirtual ~SimpleCapture();\n> +\n> +\tResults::Result start();\n> +\tResults::Result stop();\n> +\n> +\tvirtual void requestComplete(libcamera::Request *request) = 0;\n> +\n> +\tEventLoop *loop_;\n> +\n> +\tstd::shared_ptr camera_;\n> +\tstd::unique_ptr allocator_;\n> +\tstd::unique_ptr config_;\n> +};\n> +\n> +class SimpleCaptureBalanced : public SimpleCapture\n> +{\n> +public:\n> +\tSimpleCaptureBalanced(std::shared_ptr camera);\n> +\n> +\tResults::Result capture(unsigned int numRequests);\n> +\n> +private:\n> +\tint queueRequest(libcamera::Request *request);\n> +\tvoid requestComplete(libcamera::Request *request) override;\n> +\n> +\tunsigned int queueCount_;\n> +\tunsigned int captureCount_;\n> +\tunsigned int captureLimit_;\n> +};\n> +\n> +#endif /* __LC_COMPLIANCE_SIMPLE_CAPTURE_H__ */\n> diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> new file mode 100644\n> index 0000000000000000..e9ca1d58ecb959cd\n> --- /dev/null\n> +++ b/src/lc-compliance/single_stream.cpp\n> @@ -0,0 +1,74 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * single_stream.cpp - Test a single camera stream\n> + */\n> +\n> +#include \n> +\n> +#include \"simple_capture.h\"\n> +#include \"tests.h\"\n> +\n> +using namespace libcamera;\n> +\n> +Results::Result testRequestBalance(std::shared_ptr camera,\n> +\t\t\t\t StreamRole role, unsigned int startCycles,\n> +\t\t\t\t unsigned int numRequests)\n> +{\n> +\tSimpleCaptureBalanced capture(camera);\n> +\n> +\tResults::Result ret = capture.configure(role);\n> +\tif (ret.first != Results::Pass)\n> +\t\treturn ret;\n> +\n> +\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> +\t\tret = capture.capture(numRequests);\n> +\t\tif (ret.first != Results::Pass)\n> +\t\t\treturn ret;\n> +\t}\n> +\n> +\treturn { Results::Pass, \"Balanced capture of \" +\n> +\t\tstd::to_string(numRequests) + \" requests with \" +\n> +\t\tstd::to_string(startCycles) + \" start cycles\" };\n> +}\n> +\n> +Results testSingleStream(std::shared_ptr camera)\n> +{\n> +\tstatic const std::vector> roles = {\n> +\t\t{ \"raw\", Raw },\n> +\t\t{ \"still\", StillCapture },\n> +\t\t{ \"video\", VideoRecording },\n> +\t\t{ \"viewfinder\", Viewfinder },\n> +\t};\n> +\tstatic const std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> +\n> +\tResults results(numRequests.size() * roles.size() * 2);\n> +\n> +\tfor (const auto &role : roles) {\n> +\t\tstd::cout << \"= Test role \" << role.first << std::endl;\n> +\t\t/*\n> +\t\t * Test single capture cycles\n> +\t\t *\n> +\t\t * Makes sure the camera completes the exact number of requests queued.\n> +\t\t * Example failure is a camera that needs N+M requests queued to\n> +\t\t * complete N requests to the application.\n> +\t\t */\n> +\t\tstd::cout << \"* Test single capture cycles\" << std::endl;\n> +\t\tfor (unsigned int num : numRequests)\n> +\t\t\tresults.add(testRequestBalance(camera, role.second, 1, num));\n> +\n> +\t\t/*\n> +\t\t * Test multiple start/stop cycles\n> +\t\t *\n> +\t\t * Makes sure the camera supports multiple start/stop cycles.\n> +\t\t * Example failure is a camera that does not clean up correctly in its\n> +\t\t * error path but is only tested by single-capture applications.\n> +\t\t */\n> +\t\tstd::cout << \"* Test multiple start/stop cycles\" << std::endl;\n> +\t\tfor (unsigned int num : numRequests)\n> +\t\t\tresults.add(testRequestBalance(camera, role.second, 3, num));\n> +\t}\n> +\n> +\treturn results;\n> +}\n> diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> new file mode 100644\n> index 0000000000000000..396605214e4b8980\n> --- /dev/null\n> +++ b/src/lc-compliance/tests.h\n> @@ -0,0 +1,16 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * tests.h - Test modules\n> + */\n> +#ifndef __LC_COMPLIANCE_TESTS_H__\n> +#define __LC_COMPLIANCE_TESTS_H__\n> +\n> +#include \n> +\n> +#include \"results.h\"\n> +\n> +Results testSingleStream(std::shared_ptr camera);\n> +\n> +#endif /* __LC_COMPLIANCE_TESTS_H__ */\n> diff --git a/src/meson.build b/src/meson.build\n> index 8c1c8763063b7df8..29ad2d338dcbe1f2 100644\n> --- a/src/meson.build\n> +++ b/src/meson.build\n> @@ -21,6 +21,8 @@ subdir('android')\n> subdir('libcamera')\n> subdir('ipa')\n> \n> +subdir('lc-compliance')\n> +\n> subdir('cam')\n> subdir('qcam')\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 49392BD224\n\tfor ;\n\tSun, 11 Apr 2021 12:00:18 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 779AE687F7;\n\tSun, 11 Apr 2021 14:00:17 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9264F687F4\n\tfor ;\n\tSun, 11 Apr 2021 14:00:15 +0200 (CEST)","from [IPv6:2a01:e0a:169:7140:64b9:7d3a:bc41:c080] (unknown\n\t[IPv6:2a01:e0a:169:7140:64b9:7d3a:bc41:c080])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 23EE940A;\n\tSun, 11 Apr 2021 14:00:15 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"IN77JWe4\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1618142415;\n\tbh=NOA2L1GUhygALVeDeBv77sR1PQA07lAH1NMbOTWgSc0=;\n\th=Subject:To:References:From:Date:In-Reply-To:From;\n\tb=IN77JWe4O04PVpA0RFQaLW3t8A0EXYyHCpvaRMzneggZht6SExE5748Zv5mra3i3f\n\tmgxNaZcHFNxbsWOExg8gI+HXveET7jAbAiFqrUSeLfyc9iGp5o8jaL3YcK5XAKIvs+\n\tTj1jdamvW7ofZ33lNmWulSE4VQu1Cauesww5Sm8k=","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ,\n\tlibcamera-devel@lists.libcamera.org","References":"<20210409150037.1440033-1-niklas.soderlund@ragnatech.se>\n\t<20210409150037.1440033-2-niklas.soderlund@ragnatech.se>","From":"Jean-Michel Hautbois ","Message-ID":"","Date":"Sun, 11 Apr 2021 14:00:14 +0200","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101\n\tThunderbird/78.7.1","MIME-Version":"1.0","In-Reply-To":"<20210409150037.1440033-2-niklas.soderlund@ragnatech.se>","Content-Language":"en-US","Subject":"Re: [libcamera-devel] [PATCH v5 1/2] lc-compliance: Add a libcamera\n\tcompliance tool","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":16172,"web_url":"https://patchwork.libcamera.org/comment/16172/","msgid":"","date":"2021-04-12T07:55:05","subject":"Re: [libcamera-devel] [PATCH v5 1/2] lc-compliance: Add a libcamera\n\tcompliance tool","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Niklas,\n\nThank you for the patch.\n\nOn Fri, Apr 09, 2021 at 05:00:36PM +0200, Niklas Söderlund wrote:\n> Add a compliance tool to ease testing of cameras. In contrast to the\n> unit-tests under test/ that aims to test the internal components of\n> libcamera the compliance tool aims to test application use-cases and to\n> some extent the public API.\n> \n> This change adds the boilerplate code of a simple framework for the\n> creation of tests. The tests aim both to demonstrate the tool and to\n> catch real problems. The tests added are:\n> \n> - Test that if one queues exactly N requests to a camera exactly N\n> requests are eventually completed.\n> \n> - Test that a configured camera can be started and stopped multiple\n> times in an attempt to exercise cleanup code paths otherwise not\n> often tested with 'cam' for example.\n> \n> Signed-off-by: Niklas Söderlund \n> Acked-by: Kieran Bingham \n> Tested-by: Kieran Bingham \n> Reviewed-by: Laurent Pinchart \n> ---\n> * Changes since v1\n> - Improve language in commit message and comments.\n> - Test all roles as they may exercise different code paths in the\n> pipeline.\n> - Move SimpleCapture to its own .cpp/.h files.\n> \n> * Changes since v2\n> - Fold in a use-after-free bug fix from Kieran, thanks!\n> \n> * Changes since v3\n> - Update commit message.\n> - Moved command line option parsing to main() instead of in Harness.\n> - Return exit code 0 when display help due to -h.\n> - Rework to allow for a listCameras() function.\n> - Drop options empty check so available cameras are listed if no option\n> is given.\n> - Use EXIT_SUCCESS instead of 0.\n> - Style logging as \"PASS: foo\" instead of \"PASS - foo\".\n> - Include utility for std::pair.\n> - Add todo to not forget to check if result aggregator can be shared\n> with test/.\n> - Do not keep Result construction as one lines.\n> - Make const arrays static const.\n> - Remove unneeded camera exists check.\n> \n> * Changes since v4\n> - Fold libevent check back into src/lc-compliance/meson.build\n> - Add lc-compliance meson option.\n> - Make parseOptions() static.\n> - Simplify return codes from parseOptions().\n> - Fix indentation in meson.build.\n> ---\n> meson.build | 1 +\n> meson_options.txt | 5 +\n> src/lc-compliance/main.cpp | 148 ++++++++++++++++++++++++++\n> src/lc-compliance/meson.build | 27 +++++\n> src/lc-compliance/results.cpp | 75 +++++++++++++\n> src/lc-compliance/results.h | 47 +++++++++\n> src/lc-compliance/simple_capture.cpp | 151 +++++++++++++++++++++++++++\n> src/lc-compliance/simple_capture.h | 54 ++++++++++\n> src/lc-compliance/single_stream.cpp | 74 +++++++++++++\n> src/lc-compliance/tests.h | 16 +++\n> src/meson.build | 2 +\n> 11 files changed, 600 insertions(+)\n> create mode 100644 src/lc-compliance/main.cpp\n> create mode 100644 src/lc-compliance/meson.build\n> create mode 100644 src/lc-compliance/results.cpp\n> create mode 100644 src/lc-compliance/results.h\n> create mode 100644 src/lc-compliance/simple_capture.cpp\n> create mode 100644 src/lc-compliance/simple_capture.h\n> create mode 100644 src/lc-compliance/single_stream.cpp\n> create mode 100644 src/lc-compliance/tests.h\n> \n> diff --git a/meson.build b/meson.build\n> index 3a615ae37c8a0fb8..13d88301259d7077 100644\n> --- a/meson.build\n> +++ b/meson.build\n> @@ -173,5 +173,6 @@ summary({\n> 'V4L2 emulation support': v4l2_enabled,\n> 'cam application': cam_enabled,\n> 'qcam application': qcam_enabled,\n> + 'lc-compliance application': lc_compliance_enabled,\n> 'Unit tests': test_enabled,\n> }, section : 'Configuration')\n> diff --git a/meson_options.txt b/meson_options.txt\n> index 12de8fa161f8e64f..ee544513a98d1638 100644\n> --- a/meson_options.txt\n> +++ b/meson_options.txt\n> @@ -16,6 +16,11 @@ option('cam',\n> value : 'auto',\n> description : 'Compile the cam test application')\n> \n> +option('lc-compliance',\n> + type : 'feature',\n> + value : 'auto',\n> + description : 'Compile the lc-compliance test application')\n> +\n\nLet's keep the list of options alphabetically sorted.\n\n> option('documentation',\n> type : 'feature',\n> description : 'Generate the project documentation')\n> diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> new file mode 100644\n> index 0000000000000000..54cee54aa9788a88\n> --- /dev/null\n> +++ b/src/lc-compliance/main.cpp\n> @@ -0,0 +1,148 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * main.cpp - lc-compliance - The libcamera compliance tool\n> + */\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"../cam/options.h\"\n> +#include \"tests.h\"\n> +\n> +using namespace libcamera;\n> +\n> +enum {\n> +\tOptCamera = 'c',\n> +\tOptHelp = 'h',\n> +};\n> +\n> +class Harness\n> +{\n> +public:\n> +\tHarness(const OptionsParser::Options &options);\n> +\t~Harness();\n> +\n> +\tint exec();\n> +\n> +private:\n> +\tint init();\n> +\tvoid listCameras();\n> +\n> +\tOptionsParser::Options options_;\n> +\tstd::unique_ptr cm_;\n> +\tstd::shared_ptr camera_;\n> +};\n> +\n> +Harness::Harness(const OptionsParser::Options &options)\n> +\t: options_(options)\n> +{\n> +\tcm_ = std::make_unique();\n> +}\n> +\n> +Harness::~Harness()\n> +{\n> +\tif (camera_) {\n> +\t\tcamera_->release();\n> +\t\tcamera_.reset();\n> +\t}\n> +\n> +\tcm_->stop();\n> +}\n> +\n> +int Harness::exec()\n> +{\n> +\tint ret = init();\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tstd::vector results;\n> +\n> +\tresults.push_back(testSingleStream(camera_));\n> +\n> +\tfor (const Results &result : results) {\n> +\t\tret = result.summary();\n> +\t\tif (ret)\n> +\t\t\treturn ret;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +int Harness::init()\n> +{\n> +\tint ret = cm_->start();\n> +\tif (ret) {\n> +\t\tstd::cout << \"Failed to start camera manager: \"\n> +\t\t\t << strerror(-ret) << std::endl;\n> +\t\treturn ret;\n> +\t}\n> +\n> +\tif (!options_.isSet(OptCamera)) {\n> +\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> +\t\tlistCameras();\n> +\t\treturn -ENODEV;\n> +\t}\n> +\n> +\tconst std::string &cameraId = options_[OptCamera];\n> +\tcamera_ = cm_->get(cameraId);\n> +\tif (!camera_) {\n> +\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> +\t\tlistCameras();\n> +\t\treturn -ENODEV;\n> +\t}\n> +\n> +\tif (camera_->acquire()) {\n> +\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> +\t\treturn -EINVAL;\n> +\t}\n> +\n> +\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> +\n> +\treturn 0;\n> +}\n> +\n> +void Harness::listCameras()\n> +{\n> +\tfor (const std::shared_ptr &cam : cm_->cameras())\n> +\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> +}\n> +\n> +static int parseOptions(int argc, char **argv, OptionsParser::Options *options)\n> +{\n> +\tOptionsParser parser;\n> +\tparser.addOption(OptCamera, OptionString,\n> +\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> +\t\t\t ArgumentRequired, \"camera\");\n> +\tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> +\t\t\t \"help\");\n> +\n> +\t*options = parser.parse(argc, argv);\n> +\tif (!options->valid())\n> +\t\treturn -EINVAL;\n> +\n> +\tif (options->isSet(OptHelp)) {\n> +\t\tparser.usage();\n> +\t\treturn -EINTR;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +int main(int argc, char **argv)\n> +{\n> +\tOptionsParser::Options options;\n> +\tint ret = parseOptions(argc, argv, &options);\n> +\tif (ret == -EINTR)\n> +\t\treturn EXIT_SUCCESS;\n> +\tif (ret < 0)\n> +\t\treturn EXIT_FAILURE;\n> +\n> +\tHarness harness(options);\n> +\n> +\treturn harness.exec() ? EXIT_FAILURE : EXIT_SUCCESS;\n> +}\n> diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> new file mode 100644\n> index 0000000000000000..a2bfcceb1259d12e\n> --- /dev/null\n> +++ b/src/lc-compliance/meson.build\n> @@ -0,0 +1,27 @@\n> +# SPDX-License-Identifier: CC0-1.0\n> +\n> +libevent = dependency('libevent_pthreads', required : get_option('lc-compliance'))\n> +\n> +if not libevent.found()\n> + lc_compliance_enabled = false\n> + subdir_done()\n> +endif\n> +\n> +lc_compliance_enabled = true\n> +\n> +lc_compliance_sources = files([\n> + '../cam/event_loop.cpp',\n> + '../cam/options.cpp',\n> + 'main.cpp',\n> + 'results.cpp',\n> + 'simple_capture.cpp',\n> + 'single_stream.cpp',\n> +])\n> +\n> +lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> + dependencies : [\n> + libatomic,\n> + libcamera_dep,\n> + libevent,\n> + ],\n> + install : true)\n> diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp\n> new file mode 100644\n> index 0000000000000000..f149f7859e286b8f\n> --- /dev/null\n> +++ b/src/lc-compliance/results.cpp\n> @@ -0,0 +1,75 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * results.cpp - Test result aggregator\n> + */\n> +\n> +#include \"results.h\"\n> +\n> +#include \n> +\n> +void Results::add(const Result &result)\n> +{\n> +\tif (result.first == Pass)\n> +\t\tpassed_++;\n> +\telse if (result.first == Fail)\n> +\t\tfailed_++;\n> +\telse if (result.first == Skip)\n> +\t\tskipped_++;\n> +\n> +\tprintResult(result);\n> +}\n> +\n> +void Results::add(Status status, const std::string &message)\n> +{\n> +\tadd({ status, message });\n> +}\n> +\n> +void Results::fail(const std::string &message)\n> +{\n> +\tadd(Fail, message);\n> +}\n> +\n> +void Results::pass(const std::string &message)\n> +{\n> +\tadd(Pass, message);\n> +}\n> +\n> +void Results::skip(const std::string &message)\n> +{\n> +\tadd(Skip, message);\n> +}\n> +\n> +int Results::summary() const\n> +{\n> +\tif (failed_ + passed_ + skipped_ != planned_) {\n> +\t\tstd::cout << \"Planned and executed number of tests differ \"\n> +\t\t\t << failed_ + passed_ + skipped_ << \" executed \"\n> +\t\t\t << planned_ << \" planned\" << std::endl;\n> +\n> +\t\treturn -EINVAL;\n> +\t}\n> +\n> +\tstd::cout << planned_ << \" tests executed, \"\n> +\t\t << passed_ << \" tests passed, \"\n> +\t\t << skipped_ << \" tests skipped and \"\n> +\t\t << failed_ << \" tests failed \" << std::endl;\n> +\n> +\treturn 0;\n> +}\n> +\n> +void Results::printResult(const Result &result)\n> +{\n> +\tstd::string prefix;\n> +\n> +\t/* \\todo Make parsable as TAP. */\n> +\tif (result.first == Pass)\n> +\t\tprefix = \"PASS\";\n> +\telse if (result.first == Fail)\n> +\t\tprefix = \"FAIL\";\n> +\telse if (result.first == Skip)\n> +\t\tprefix = \"SKIP\";\n> +\n> +\tstd::cout << \"- \" << prefix << \": \" << result.second << std::endl;\n> +}\n> diff --git a/src/lc-compliance/results.h b/src/lc-compliance/results.h\n> new file mode 100644\n> index 0000000000000000..2a3722b841a6410a\n> --- /dev/null\n> +++ b/src/lc-compliance/results.h\n> @@ -0,0 +1,47 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * results.h - Test result aggregator\n> + */\n> +#ifndef __LC_COMPLIANCE_RESULTS_H__\n> +#define __LC_COMPLIANCE_RESULTS_H__\n> +\n> +#include \n> +#include \n> +\n> +/* \\todo Check if result aggregator can be shared with self tests in test/ */\n> +class Results\n> +{\n> +public:\n> +\tenum Status {\n> +\t\tFail,\n> +\t\tPass,\n> +\t\tSkip,\n> +\t};\n> +\n> +\tusing Result = std::pair;\n> +\n> +\tResults(unsigned int planned)\n> +\t\t: planned_(planned), passed_(0), failed_(0), skipped_(0)\n> +\t{\n> +\t}\n> +\n> +\tvoid add(const Result &result);\n> +\tvoid add(Status status, const std::string &message);\n> +\tvoid fail(const std::string &message);\n> +\tvoid pass(const std::string &message);\n> +\tvoid skip(const std::string &message);\n> +\n> +\tint summary() const;\n> +\n> +private:\n> +\tvoid printResult(const Result &result);\n> +\n> +\tunsigned int planned_;\n> +\tunsigned int passed_;\n> +\tunsigned int failed_;\n> +\tunsigned int skipped_;\n> +};\n> +\n> +#endif /* __LC_COMPLIANCE_RESULTS_H__ */\n> diff --git a/src/lc-compliance/simple_capture.cpp b/src/lc-compliance/simple_capture.cpp\n> new file mode 100644\n> index 0000000000000000..389dc11fc60225c5\n> --- /dev/null\n> +++ b/src/lc-compliance/simple_capture.cpp\n> @@ -0,0 +1,151 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * simple_capture.cpp - Simple capture helper\n> + */\n> +\n> +#include \"simple_capture.h\"\n> +\n> +using namespace libcamera;\n> +\n> +SimpleCapture::SimpleCapture(std::shared_ptr camera)\n> +\t: camera_(camera), allocator_(std::make_unique(camera))\n> +{\n> +}\n> +\n> +SimpleCapture::~SimpleCapture()\n> +{\n> +}\n> +\n> +Results::Result SimpleCapture::configure(StreamRole role)\n> +{\n> +\tconfig_ = camera_->generateConfiguration({ role });\n> +\n> +\tif (config_->validate() != CameraConfiguration::Valid) {\n> +\t\tconfig_.reset();\n> +\t\treturn { Results::Fail, \"Configuration not valid\" };\n> +\t}\n> +\n> +\tif (camera_->configure(config_.get())) {\n> +\t\tconfig_.reset();\n> +\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> +\t}\n> +\n> +\treturn { Results::Pass, \"Configure camera\" };\n> +}\n> +\n> +Results::Result SimpleCapture::start()\n> +{\n> +\tStream *stream = config_->at(0).stream();\n> +\tif (allocator_->allocate(stream) < 0)\n> +\t\treturn { Results::Fail, \"Failed to allocate buffers\" };\n> +\n> +\tif (camera_->start())\n> +\t\treturn { Results::Fail, \"Failed to start camera\" };\n> +\n> +\tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> +\n> +\treturn { Results::Pass, \"Started camera\" };\n> +}\n> +\n> +Results::Result SimpleCapture::stop()\n> +{\n> +\tStream *stream = config_->at(0).stream();\n> +\n> +\tcamera_->stop();\n> +\n> +\tcamera_->requestCompleted.disconnect(this, &SimpleCapture::requestComplete);\n> +\n> +\tallocator_->free(stream);\n> +\n> +\treturn { Results::Pass, \"Stopped camera\" };\n> +}\n> +\n> +/* SimpleCaptureBalanced */\n> +\n> +SimpleCaptureBalanced::SimpleCaptureBalanced(std::shared_ptr camera)\n> +\t: SimpleCapture(camera)\n> +{\n> +}\n> +\n> +Results::Result SimpleCaptureBalanced::capture(unsigned int numRequests)\n> +{\n> +\tResults::Result ret = start();\n> +\tif (ret.first != Results::Pass)\n> +\t\treturn ret;\n> +\n> +\tStream *stream = config_->at(0).stream();\n> +\tconst std::vector> &buffers = allocator_->buffers(stream);\n> +\n> +\t/* No point in testing less requests then the camera depth. */\n> +\tif (buffers.size() > numRequests) {\n> +\t\t/* Cache buffers.size() before we destroy it in stop() */\n> +\t\tint buffers_size = buffers.size();\n> +\t\tstop();\n> +\n> +\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers_size)\n> +\t\t\t+ \" requests, can't test only \" + std::to_string(numRequests) };\n> +\t}\n> +\n> +\tqueueCount_ = 0;\n> +\tcaptureCount_ = 0;\n> +\tcaptureLimit_ = numRequests;\n> +\n> +\t/* Queue the recommended number of reqeuests. */\n> +\tstd::vector> requests;\n> +\tfor (const std::unique_ptr &buffer : buffers) {\n> +\t\tstd::unique_ptr request = camera_->createRequest();\n> +\t\tif (!request) {\n> +\t\t\tstop();\n> +\t\t\treturn { Results::Fail, \"Can't create request\" };\n> +\t\t}\n> +\n> +\t\tif (request->addBuffer(stream, buffer.get())) {\n> +\t\t\tstop();\n> +\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> +\t\t}\n> +\n> +\t\tif (queueRequest(request.get()) < 0) {\n> +\t\t\tstop();\n> +\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> +\t\t}\n> +\n> +\t\trequests.push_back(std::move(request));\n> +\t}\n> +\n> +\t/* Run capture session. */\n> +\tloop_ = new EventLoop();\n> +\tloop_->exec();\n> +\tstop();\n> +\tdelete loop_;\n> +\n> +\tif (captureCount_ != captureLimit_)\n> +\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) +\n> +\t\t\t\" request, wanted \" + std::to_string(captureLimit_) };\n> +\n> +\treturn { Results::Pass, \"Balanced capture of \" +\n> +\t\tstd::to_string(numRequests) + \" requests\" };\n> +}\n> +\n> +int SimpleCaptureBalanced::queueRequest(Request *request)\n> +{\n> +\tqueueCount_++;\n> +\tif (queueCount_ > captureLimit_)\n> +\t\treturn 0;\n> +\n> +\treturn camera_->queueRequest(request);\n> +}\n> +\n> +void SimpleCaptureBalanced::requestComplete(Request *request)\n> +{\n> +\tcaptureCount_++;\n> +\tif (captureCount_ >= captureLimit_) {\n> +\t\tloop_->exit(0);\n> +\t\treturn;\n> +\t}\n> +\n> +\trequest->reuse(Request::ReuseBuffers);\n> +\tif (queueRequest(request))\n> +\t\tloop_->exit(-EINVAL);\n> +}\n> diff --git a/src/lc-compliance/simple_capture.h b/src/lc-compliance/simple_capture.h\n> new file mode 100644\n> index 0000000000000000..3a6afc538c623050\n> --- /dev/null\n> +++ b/src/lc-compliance/simple_capture.h\n> @@ -0,0 +1,54 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * simple_capture.h - Simple capture helper\n> + */\n> +#ifndef __LC_COMPLIANCE_SIMPLE_CAPTURE_H__\n> +#define __LC_COMPLIANCE_SIMPLE_CAPTURE_H__\n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \"../cam/event_loop.h\"\n> +#include \"results.h\"\n> +\n> +class SimpleCapture\n> +{\n> +public:\n> +\tResults::Result configure(libcamera::StreamRole role);\n> +\n> +protected:\n> +\tSimpleCapture(std::shared_ptr camera);\n> +\tvirtual ~SimpleCapture();\n> +\n> +\tResults::Result start();\n> +\tResults::Result stop();\n> +\n> +\tvirtual void requestComplete(libcamera::Request *request) = 0;\n> +\n> +\tEventLoop *loop_;\n> +\n> +\tstd::shared_ptr camera_;\n> +\tstd::unique_ptr allocator_;\n> +\tstd::unique_ptr config_;\n> +};\n> +\n> +class SimpleCaptureBalanced : public SimpleCapture\n> +{\n> +public:\n> +\tSimpleCaptureBalanced(std::shared_ptr camera);\n> +\n> +\tResults::Result capture(unsigned int numRequests);\n> +\n> +private:\n> +\tint queueRequest(libcamera::Request *request);\n> +\tvoid requestComplete(libcamera::Request *request) override;\n> +\n> +\tunsigned int queueCount_;\n> +\tunsigned int captureCount_;\n> +\tunsigned int captureLimit_;\n> +};\n> +\n> +#endif /* __LC_COMPLIANCE_SIMPLE_CAPTURE_H__ */\n> diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> new file mode 100644\n> index 0000000000000000..e9ca1d58ecb959cd\n> --- /dev/null\n> +++ b/src/lc-compliance/single_stream.cpp\n> @@ -0,0 +1,74 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * single_stream.cpp - Test a single camera stream\n> + */\n> +\n> +#include \n> +\n> +#include \"simple_capture.h\"\n> +#include \"tests.h\"\n> +\n> +using namespace libcamera;\n> +\n> +Results::Result testRequestBalance(std::shared_ptr camera,\n> +\t\t\t\t StreamRole role, unsigned int startCycles,\n> +\t\t\t\t unsigned int numRequests)\n> +{\n> +\tSimpleCaptureBalanced capture(camera);\n> +\n> +\tResults::Result ret = capture.configure(role);\n> +\tif (ret.first != Results::Pass)\n> +\t\treturn ret;\n> +\n> +\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> +\t\tret = capture.capture(numRequests);\n> +\t\tif (ret.first != Results::Pass)\n> +\t\t\treturn ret;\n> +\t}\n> +\n> +\treturn { Results::Pass, \"Balanced capture of \" +\n> +\t\tstd::to_string(numRequests) + \" requests with \" +\n> +\t\tstd::to_string(startCycles) + \" start cycles\" };\n> +}\n> +\n> +Results testSingleStream(std::shared_ptr camera)\n> +{\n> +\tstatic const std::vector> roles = {\n> +\t\t{ \"raw\", Raw },\n> +\t\t{ \"still\", StillCapture },\n> +\t\t{ \"video\", VideoRecording },\n> +\t\t{ \"viewfinder\", Viewfinder },\n> +\t};\n> +\tstatic const std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> +\n> +\tResults results(numRequests.size() * roles.size() * 2);\n> +\n> +\tfor (const auto &role : roles) {\n> +\t\tstd::cout << \"= Test role \" << role.first << std::endl;\n> +\t\t/*\n> +\t\t * Test single capture cycles\n> +\t\t *\n> +\t\t * Makes sure the camera completes the exact number of requests queued.\n> +\t\t * Example failure is a camera that needs N+M requests queued to\n> +\t\t * complete N requests to the application.\n> +\t\t */\n> +\t\tstd::cout << \"* Test single capture cycles\" << std::endl;\n> +\t\tfor (unsigned int num : numRequests)\n> +\t\t\tresults.add(testRequestBalance(camera, role.second, 1, num));\n> +\n> +\t\t/*\n> +\t\t * Test multiple start/stop cycles\n> +\t\t *\n> +\t\t * Makes sure the camera supports multiple start/stop cycles.\n> +\t\t * Example failure is a camera that does not clean up correctly in its\n> +\t\t * error path but is only tested by single-capture applications.\n> +\t\t */\n> +\t\tstd::cout << \"* Test multiple start/stop cycles\" << std::endl;\n> +\t\tfor (unsigned int num : numRequests)\n> +\t\t\tresults.add(testRequestBalance(camera, role.second, 3, num));\n> +\t}\n> +\n> +\treturn results;\n> +}\n> diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> new file mode 100644\n> index 0000000000000000..396605214e4b8980\n> --- /dev/null\n> +++ b/src/lc-compliance/tests.h\n> @@ -0,0 +1,16 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * tests.h - Test modules\n> + */\n> +#ifndef __LC_COMPLIANCE_TESTS_H__\n> +#define __LC_COMPLIANCE_TESTS_H__\n> +\n> +#include \n> +\n> +#include \"results.h\"\n> +\n> +Results testSingleStream(std::shared_ptr camera);\n> +\n> +#endif /* __LC_COMPLIANCE_TESTS_H__ */\n> diff --git a/src/meson.build b/src/meson.build\n> index 8c1c8763063b7df8..29ad2d338dcbe1f2 100644\n> --- a/src/meson.build\n> +++ b/src/meson.build\n> @@ -21,6 +21,8 @@ subdir('android')\n> subdir('libcamera')\n> subdir('ipa')\n> \n> +subdir('lc-compliance')\n> +\n> subdir('cam')\n> subdir('qcam')\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3A07CBD224\n\tfor ;\n\tMon, 12 Apr 2021 07:55:58 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A4B29687FB;\n\tMon, 12 Apr 2021 09:55:57 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3E77B687F5\n\tfor ;\n\tMon, 12 Apr 2021 09:55:56 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 9398E3F0;\n\tMon, 12 Apr 2021 09:55:55 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Q3FG8biG\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1618214155;\n\tbh=WeHIIBAdr9b2ufi8INu20S+4Vsqvr4b+WFtERpnivA8=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=Q3FG8biG2gtAKX4yslQgMEe0YgrKS3T7qDY+YRgAGgtFSMG2Xk0NNl9GQQXZCk4lS\n\tsmcfSXfOgS7EOy62/oPmFIfIjZKShYesTCSWBOzz/kTtDnZlZx9FVyuejv12IUPBMb\n\tX+vDPIVeKV9kwvmHLhT9b0+q3rbudDsIdF2A3X7I=","Date":"Mon, 12 Apr 2021 10:55:05 +0300","From":"Laurent Pinchart ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Message-ID":"","References":"<20210409150037.1440033-1-niklas.soderlund@ragnatech.se>\n\t<20210409150037.1440033-2-niklas.soderlund@ragnatech.se>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20210409150037.1440033-2-niklas.soderlund@ragnatech.se>","Subject":"Re: [libcamera-devel] [PATCH v5 1/2] lc-compliance: Add a libcamera\n\tcompliance tool","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]