[{"id":14855,"web_url":"https://patchwork.libcamera.org/comment/14855/","msgid":"","date":"2021-01-29T18:05:22","subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Niklas,\n\nThank you for your work, this looks quite interesting. However, I have a\nhigh level question, see below.\n\nOn Fri, 29 Jan 2021 at 16:53, Niklas Söderlund <\nniklas.soderlund@ragnatech.se> wrote:\n\n> Add a compliance tool to ease testing of cameras on target. In contrast\n> to the unit-tests under test/ that aims to test the internal components\n> of libcamera the compliance aims to test application use-cases and to\n> some extend the public API.\n>\n> This change adds the boilerplate code for a simple framework to write\n> tests in as well as two simple test. The tests aims both to demonstrate\n> the tool and to catch real problems. The tests added are:\n>\n> - Test that if one queues exactly N requests to a camera exactly N\n> requests are eventually completed.\n>\n> - Test that a configured camera can be started and stopped multiple\n> times in an attempt to exercise cleanup code paths otherwise not\n> often tested with 'cam' for example.\n>\n> Example pass run on Raspberry Pi:\n>\n> $ LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c\n> \"/base/soc/i2c0mux/i2c@1/imx219@10\"\n> [2:36:17.672447527] [11896] ERROR V4L2 v4l2_device.cpp:191 'imx219\n> 10-0010': Control 0x009a0922 not found\n> [2:36:17.673095289] [11896] ERROR V4L2 v4l2_subdevice.cpp:285 'imx219\n> 10-0010': Unable to get rectangle 2 on pad 0: Invalid argument\n> Using camera /base/soc/i2c0mux/i2c@1/imx219@10\n> Test single capture cycles\n> - SKIP - Camera needs 4 requests, can't test only 1\n> - SKIP - Camera needs 4 requests, can't test only 2\n> - SKIP - Camera needs 4 requests, can't test only 3\n>\n\nI'm a bit curious about the above 3 lines. From what I make out, it is\nsaying that the pipeline cannot run with < 4 on-the-go requests, is that\ncorrect? Does that determination come from the buffer count returned by\nPipelineHandler::GenerateConfiguration()? I *think* the Raspberry Pi stack\nshould be able to handle any number of on-the-go requests. I say think,\nbecause I do not normally run in that configuration, but I do know some of\nour users might have. The buffer count of 4 returned\nby PipelineHandlerRPi::generateConfiguration() is a somewhat educated guess\nfor a reasonable number of buffers. Is this something we need to update in\nour pipeline handler for this to work?\n\nRegards,\nNaush\n\n\n> - PASS - Balanced capture of 5 requests with 1 start cycles\n> - PASS - Balanced capture of 8 requests with 1 start cycles\n> - PASS - Balanced capture of 13 requests with 1 start cycles\n> - PASS - Balanced capture of 21 requests with 1 start cycles\n> - PASS - Balanced capture of 34 requests with 1 start cycles\n> - PASS - Balanced capture of 55 requests with 1 start cycles\n> - PASS - Balanced capture of 89 requests with 1 start cycles\n> Test multiple start/stop cycles\n> - SKIP - Camera needs 4 requests, can't test only 1\n> - SKIP - Camera needs 4 requests, can't test only 2\n> - SKIP - Camera needs 4 requests, can't test only 3\n> - PASS - Balanced capture of 5 requests with 3 start cycles\n> - PASS - Balanced capture of 8 requests with 3 start cycles\n> - PASS - Balanced capture of 13 requests with 3 start cycles\n> - PASS - Balanced capture of 21 requests with 3 start cycles\n> - PASS - Balanced capture of 34 requests with 3 start cycles\n> - PASS - Balanced capture of 55 requests with 3 start cycles\n> - PASS - Balanced capture of 89 requests with 3 start cycles\n> 20 tests executed, 14 tests passed, 6 tests skipped and 0 tests failed\n>\n> Example fail run on IPU3:\n>\n> - LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c \"\\_SB_.PCI0.I2C2.CAM0\"\n> - [0:31:36.992763102] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov13858\n> 8-0010': Control 0x009a0922 not found\n> - [0:31:36.992815436] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> 8-0010': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.992847049] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> 8-0010': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.992869029] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993152932] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993172833] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the\n> active area size\n> - [0:31:36.993208149] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993224502] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the\n> active area size\n> - [0:31:36.993354770] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov5670\n> 10-0036': Control 0x009a0922 not found\n> - [0:31:36.993377622] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> 10-0036': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993398093] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> 10-0036': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993415611] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993660192] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993678492] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the\n> active area size\n> - [0:31:36.993724648] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993741032] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the\n> active area size\n> - Using camera \\_SB_.PCI0.I2C2.CAM0\n> - Test single capture cycles\n> - - SKIP - Camera needs 4 requests, can't test only 1\n> - - SKIP - Camera needs 4 requests, can't test only 2\n> - - SKIP - Camera needs 4 requests, can't test only 3\n> - - PASS - Balanced capture of 5 requests with 1 start cycles\n> - - PASS - Balanced capture of 8 requests with 1 start cycles\n> - - PASS - Balanced capture of 13 requests with 1 start cycles\n> - - PASS - Balanced capture of 21 requests with 1 start cycles\n> - - PASS - Balanced capture of 34 requests with 1 start cycles\n> - [0:31:41.969783243] [6420] ERROR IPU3 cio2.cpp:264 CIO2 buffer underrun\n> application>\n>\n> Signed-off-by: Niklas Söderlund \n> ---\n> Hi,\n>\n> I have tested this on RkISP1, RPi and IPU3 on latest master [1]. I plan\n> to post separate patches which starts to address some of the problems\n> found while working. A fix for 'cam --captur=N' is coming shortly for\n> example.\n>\n> I still have to find a good fix for a races found in the\n> IPU3 pipeline between the CIO2 and IMGU (as showed above). What I don't\n> (yet) have a plan for is how to fix that request queueing errors are not\n> propagated in our application facing API leading the test application to\n> wait forever to a request the library knows have failed, ideas welcome.\n>\n> 1. 958c80a4f1c28301 (\"android: camera_device: Set AE precapture trigger\n> according to request\")\n> ---\n> src/lc-compliance/main.cpp | 139 ++++++++++++++++++\n> src/lc-compliance/meson.build | 24 ++++\n> src/lc-compliance/results.cpp | 75 ++++++++++\n> src/lc-compliance/results.h | 45 ++++++\n> src/lc-compliance/single_stream.cpp | 210 ++++++++++++++++++++++++++++\n> src/lc-compliance/tests.h | 16 +++\n> src/meson.build | 2 +\n> 7 files changed, 511 insertions(+)\n> create mode 100644 src/lc-compliance/main.cpp\n> create mode 100644 src/lc-compliance/meson.build\n> create mode 100644 src/lc-compliance/results.cpp\n> create mode 100644 src/lc-compliance/results.h\n> create mode 100644 src/lc-compliance/single_stream.cpp\n> create mode 100644 src/lc-compliance/tests.h\n>\n> diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> new file mode 100644\n> index 0000000000000000..e1cbce7eac3df2bc\n> --- /dev/null\n> +++ b/src/lc-compliance/main.cpp\n> @@ -0,0 +1,139 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * main.cpp - lc-compliance - The libcamera compliance tool\n> + */\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"../cam/options.h\"\n> +#include \"tests.h\"\n> +\n> +using namespace libcamera;\n> +\n> +class Harness\n> +{\n> +public:\n> + Harness();\n> + ~Harness();\n> +\n> + int exec(int argc, char **argv);\n> +\n> +private:\n> + enum {\n> + OptCamera = 'c',\n> + OptHelp = 'h',\n> + };\n> +\n> + int parseOptions(int argc, char **argv);\n> + int init(int argc, char **argv);\n> +\n> + OptionsParser::Options options_;\n> + std::unique_ptr cm_;\n> + std::shared_ptr camera_;\n> +};\n> +\n> +Harness::Harness()\n> +{\n> + cm_ = std::make_unique();\n> +}\n> +\n> +Harness::~Harness()\n> +{\n> + if (camera_) {\n> + camera_->release();\n> + camera_.reset();\n> + }\n> +\n> + cm_->stop();\n> +}\n> +\n> +int Harness::exec(int argc, char **argv)\n> +{\n> + int ret = init(argc, argv);\n> + if (ret)\n> + return ret;\n> +\n> + std::vector results;\n> +\n> + results.push_back(testSingleStream(camera_));\n> +\n> + for (const Results &result : results) {\n> + ret = result.summary();\n> + if (ret)\n> + return ret;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +int Harness::init(int argc, char **argv)\n> +{\n> + int ret = parseOptions(argc, argv);\n> + if (ret < 0)\n> + return ret;\n> +\n> + ret = cm_->start();\n> + if (ret) {\n> + std::cout << \"Failed to start camera manager: \"\n> + << strerror(-ret) << std::endl;\n> + return ret;\n> + }\n> +\n> + if (!options_.isSet(OptCamera)) {\n> + std::cout << \"No camera specified, available cameras:\" <<\n> std::endl;\n> + for (const std::shared_ptr &cam : cm_->cameras())\n> + std::cout << \"- \" << cam.get()->id() << std::endl;\n> + return -ENODEV;\n> + }\n> +\n> + const std::string &cameraId = options_[OptCamera];\n> + camera_ = cm_->get(cameraId);\n> + if (!camera_) {\n> + std::cout << \"Camera \" << cameraId << \" not found,\n> available cameras:\" << std::endl;\n> + for (const std::shared_ptr &cam : cm_->cameras())\n> + std::cout << \"- \" << cam.get()->id() << std::endl;\n> + return -ENODEV;\n> + }\n> +\n> + if (camera_->acquire()) {\n> + std::cout << \"Failed to acquire camera\" << std::endl;\n> + return -EINVAL;\n> + }\n> +\n> + std::cout << \"Using camera \" << cameraId << std::endl;\n> +\n> + return 0;\n> +}\n> +\n> +int Harness::parseOptions(int argc, char **argv)\n> +{\n> + OptionsParser parser;\n> + parser.addOption(OptCamera, OptionString,\n> + \"Specify which camera to operate on, by id\",\n> \"camera\",\n> + ArgumentRequired, \"camera\");\n> + parser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> + \"help\");\n> +\n> + options_ = parser.parse(argc, argv);\n> + if (!options_.valid())\n> + return -EINVAL;\n> +\n> + if (options_.empty() || options_.isSet(OptHelp)) {\n> + parser.usage();\n> + return options_.empty() ? -EINVAL : -EINTR;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +int main(int argc, char **argv)\n> +{\n> + Harness harness;\n> + return harness.exec(argc, argv) ? EXIT_FAILURE : 0;\n> +}\n> diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> new file mode 100644\n> index 0000000000000000..d700a307c71405b4\n> --- /dev/null\n> +++ b/src/lc-compliance/meson.build\n> @@ -0,0 +1,24 @@\n> +# SPDX-License-Identifier: CC0-1.0\n> +\n> +libevent = dependency('libevent_pthreads', required : false)\n> +\n> +if not libevent.found()\n> + warning('libevent_pthreads not found, \\'lc-compliance\\' application\n> will not be compiled')\n> + subdir_done()\n> +endif\n> +\n> +lc_compliance_sources = files([\n> + '../cam/event_loop.cpp',\n> + '../cam/options.cpp',\n> + 'main.cpp',\n> + 'results.cpp',\n> + 'single_stream.cpp',\n> +])\n> +\n> +lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> + dependencies : [\n> + libatomic,\n> + libcamera_dep,\n> + libevent,\n> + ],\n> + install : true)\n> diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp\n> new file mode 100644\n> index 0000000000000000..fb4242bf49a3268b\n> --- /dev/null\n> +++ b/src/lc-compliance/results.cpp\n> @@ -0,0 +1,75 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * results.cpp - Test result aggregator\n> + */\n> +\n> +#include \"results.h\"\n> +\n> +#include \n> +\n> +void Results::add(const Result &result)\n> +{\n> + if (result.first == Pass)\n> + passed_++;\n> + else if (result.first == Fail)\n> + failed_++;\n> + else if (result.first == Skip)\n> + skipped_++;\n> +\n> + printResult(result);\n> +}\n> +\n> +void Results::add(Status status, const std::string &message)\n> +{\n> + add({ status, message });\n> +}\n> +\n> +void Results::fail(const std::string &message)\n> +{\n> + add(Fail, message);\n> +}\n> +\n> +void Results::pass(const std::string &message)\n> +{\n> + add(Pass, message);\n> +}\n> +\n> +void Results::skip(const std::string &message)\n> +{\n> + add(Skip, message);\n> +}\n> +\n> +int Results::summary() const\n> +{\n> + if (failed_ + passed_ + skipped_ != planned_) {\n> + std::cout << \"Planned and executed numer of tests differ \"\n> + << failed_ + passed_ + skipped_ << \" executed \"\n> + << planned_ << \" planned\" << std::endl;\n> +\n> + return -EINVAL;\n> + }\n> +\n> + std::cout << planned_ << \" tests executed, \"\n> + << passed_ << \" tests passed, \"\n> + << skipped_ << \" tests skipped and \"\n> + << failed_ << \" tests failed \" << std::endl;\n> +\n> + return 0;\n> +}\n> +\n> +void Results::printResult(const Result &result)\n> +{\n> + std::string prefix;\n> +\n> + /* \\todo Make parsable as TAP. */\n> + if (result.first == Pass)\n> + prefix = \"PASS\";\n> + else if (result.first == Fail)\n> + prefix = \"FAIL\";\n> + else if (result.first == Skip)\n> + prefix = \"SKIP\";\n> +\n> + std::cout << \"- \" << prefix << \" - \" << result.second << std::endl;\n> +}\n> diff --git a/src/lc-compliance/results.h b/src/lc-compliance/results.h\n> new file mode 100644\n> index 0000000000000000..a02fd5ab46edd62c\n> --- /dev/null\n> +++ b/src/lc-compliance/results.h\n> @@ -0,0 +1,45 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * results.h - Test result aggregator\n> + */\n> +#ifndef __LC_COMPLIANCE_RESULTS_H__\n> +#define __LC_COMPLIANCE_RESULTS_H__\n> +\n> +#include \n> +\n> +class Results\n> +{\n> +public:\n> + enum Status {\n> + Fail,\n> + Pass,\n> + Skip,\n> + };\n> +\n> + using Result = std::pair;\n> +\n> + Results(unsigned int planned)\n> + : planned_(planned), passed_(0), failed_(0), skipped_(0)\n> + {\n> + }\n> +\n> + void add(const Result &result);\n> + void add(Status status, const std::string &message);\n> + void fail(const std::string &message);\n> + void pass(const std::string &message);\n> + void skip(const std::string &message);\n> +\n> + int summary() const;\n> +\n> +private:\n> + void printResult(const Result &result);\n> +\n> + unsigned int planned_;\n> + unsigned int passed_;\n> + unsigned int failed_;\n> + unsigned int skipped_;\n> +};\n> +\n> +#endif /* __LC_COMPLIANCE_RESULTS_H__ */\n> diff --git a/src/lc-compliance/single_stream.cpp\n> b/src/lc-compliance/single_stream.cpp\n> new file mode 100644\n> index 0000000000000000..d31b6f8d06487a85\n> --- /dev/null\n> +++ b/src/lc-compliance/single_stream.cpp\n> @@ -0,0 +1,210 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * single_stream.cpp - Test a single camera stream\n> + */\n> +\n> +#include \n> +\n> +#include \"../cam/event_loop.h\"\n> +#include \"tests.h\"\n> +\n> +using namespace libcamera;\n> +\n> +class SimpleCapture\n> +{\n> +public:\n> + SimpleCapture(std::shared_ptr camera)\n> + : camera_(camera),\n> allocator_(std::make_unique(camera))\n> + {\n> + }\n> +\n> + Results::Result configure(StreamRole role);\n> + Results::Result start();\n> + Results::Result capture(unsigned int numRequests);\n> + Results::Result stop();\n> +\n> +private:\n> + int queueRequest(Request *request);\n> + void requestComplete(Request *request);\n> +\n> + std::shared_ptr camera_;\n> + std::unique_ptr allocator_;\n> + std::unique_ptr config_;\n> +\n> + EventLoop *loop_;\n> + unsigned int queueCount_;\n> + unsigned int captureCount_;\n> + unsigned int captureLimit_;\n> +};\n> +\n> +Results::Result SimpleCapture::configure(StreamRole role)\n> +{\n> + config_ = camera_->generateConfiguration({ role });\n> +\n> + if (config_->validate() != CameraConfiguration::Valid) {\n> + config_.reset();\n> + return { Results::Fail, \"Configuration not valid\" };\n> + }\n> +\n> + if (camera_->configure(config_.get())) {\n> + config_.reset();\n> + return { Results::Fail, \"Failed to configure camera\" };\n> + }\n> +\n> + return { Results::Pass, \"Configure camera\" };\n> +}\n> +\n> +Results::Result SimpleCapture::start()\n> +{\n> + Stream *stream = config_->at(0).stream();\n> + if (allocator_->allocate(stream) < 0)\n> + return { Results::Fail, \"Failed to allocate buffers\n> camera\" };\n> +\n> + if (camera_->start())\n> + return { Results::Fail, \"Failed to start camera\" };\n> +\n> + camera_->requestCompleted.connect(this,\n> &SimpleCapture::requestComplete);\n> +\n> + return { Results::Pass, \"Started camera\" };\n> +}\n> +\n> +Results::Result SimpleCapture::capture(unsigned int numRequests)\n> +{\n> + Stream *stream = config_->at(0).stream();\n> + const std::vector> &buffers =\n> allocator_->buffers(stream);\n> +\n> + /* No point in testing less requests then the camera depth. */\n> + if (buffers.size() > numRequests)\n> + return { Results::Skip, \"Camera needs \" +\n> std::to_string(buffers.size()) + \" requests, can't test only \" +\n> std::to_string(numRequests) };\n> +\n> + queueCount_ = 0;\n> + captureCount_ = 0;\n> + captureLimit_ = numRequests;\n> +\n> + /* Queue the camera recommended number of reqeuests. */\n> + std::vector> requests;\n> + for (const std::unique_ptr &buffer :\n> allocator_->buffers(stream)) {\n> + std::unique_ptr request =\n> camera_->createRequest();\n> + if (!request)\n> + return { Results::Fail, \"Can't create request\" };\n> +\n> + if (request->addBuffer(stream, buffer.get()))\n> + return { Results::Fail, \"Can't set buffer for\n> request\" };\n> +\n> + if (queueRequest(request.get()) < 0)\n> + return { Results::Fail, \"Failed to queue request\"\n> };\n> +\n> + requests.push_back(std::move(request));\n> + }\n> +\n> + /* Run capture session. */\n> + loop_ = new EventLoop();\n> + loop_->exec();\n> + delete loop_;\n> +\n> + if (captureCount_ != captureLimit_)\n> + return { Results::Fail, \"Got \" +\n> std::to_string(captureCount_) + \" request, wanted \" +\n> std::to_string(captureLimit_) };\n> +\n> + return { Results::Pass, \"Balanced capture of \" +\n> std::to_string(numRequests) + \" requests\" };\n> +}\n> +\n> +Results::Result SimpleCapture::stop()\n> +{\n> + Stream *stream = config_->at(0).stream();\n> +\n> + camera_->stop();\n> +\n> + camera_->requestCompleted.disconnect(this,\n> &SimpleCapture::requestComplete);\n> +\n> + allocator_->free(stream);\n> +\n> + return { Results::Pass, \"Stopped camera\" };\n> +}\n> +\n> +int SimpleCapture::queueRequest(Request *request)\n> +{\n> + queueCount_++;\n> + if (queueCount_ > captureLimit_)\n> + return 0;\n> +\n> + return camera_->queueRequest(request);\n> +}\n> +\n> +void SimpleCapture::requestComplete(Request *request)\n> +{\n> + captureCount_++;\n> + if (captureCount_ >= captureLimit_) {\n> + loop_->exit(0);\n> + return;\n> + }\n> +\n> + request->reuse(Request::ReuseBuffers);\n> + if (queueRequest(request))\n> + loop_->exit(-EINVAL);\n> +}\n> +\n> +Results::Result testRequestBalance(std::shared_ptr camera,\n> + unsigned int startCycles,\n> + unsigned int numRequests)\n> +{\n> + SimpleCapture capture(camera);\n> + Results::Result ret;\n> +\n> + ret = capture.configure(Viewfinder);\n> + if (ret.first != Results::Pass)\n> + return ret;\n> +\n> + for (unsigned int starts = 0; starts < startCycles; starts++) {\n> + ret = capture.start();\n> + if (ret.first != Results::Pass)\n> + return ret;\n> +\n> + ret = capture.capture(numRequests);\n> + if (ret.first != Results::Pass) {\n> + capture.stop();\n> + return ret;\n> + }\n> +\n> + ret = capture.stop();\n> + if (ret.first != Results::Pass)\n> + return ret;\n> + }\n> +\n> + return { Results::Pass, \"Balanced capture of \" +\n> std::to_string(numRequests) + \" requests with \" +\n> std::to_string(startCycles) + \" start cycles\" };\n> +}\n> +\n> +Results testSingleStream(std::shared_ptr camera)\n> +{\n> + const std::vector numRequests = { 1, 2, 3, 5, 8, 13,\n> 21, 34, 55, 89 };\n> +\n> + Results results(numRequests.size() * 2);\n> +\n> + if (!camera)\n> + return results;\n> +\n> + /*\n> + * Test single capture cycles\n> + *\n> + * Makes sure the camera completes the exact number of requests\n> queued.\n> + * Example failure is a camera that needs N+M requests queued to\n> + * complete N requests to the application.\n> + */\n> + std::cout << \"Test single capture cycles\" << std::endl;\n> + for (unsigned int num : numRequests)\n> + results.add(testRequestBalance(camera, 1, num));\n> +\n> + /*\n> + * Test multiple start/stop cycles\n> + *\n> + * Makes sure the camera supports multiple start/stop cycles.\n> + * Example failure is a camera that does not clean up correctly in\n> its\n> + * error path but is only tested by single-capture applications.\n> + */\n> + std::cout << \"Test multiple start/stop cycles\" << std::endl;\n> + for (unsigned int num : numRequests)\n> + results.add(testRequestBalance(camera, 3, num));\n> +\n> + return results;\n> +}\n> diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> new file mode 100644\n> index 0000000000000000..396605214e4b8980\n> --- /dev/null\n> +++ b/src/lc-compliance/tests.h\n> @@ -0,0 +1,16 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * tests.h - Test modules\n> + */\n> +#ifndef __LC_COMPLIANCE_TESTS_H__\n> +#define __LC_COMPLIANCE_TESTS_H__\n> +\n> +#include \n> +\n> +#include \"results.h\"\n> +\n> +Results testSingleStream(std::shared_ptr camera);\n> +\n> +#endif /* __LC_COMPLIANCE_TESTS_H__ */\n> diff --git a/src/meson.build b/src/meson.build\n> index 4b75f05878bcb702..a8e1af7adf2ca9c8 100644\n> --- a/src/meson.build\n> +++ b/src/meson.build\n> @@ -18,6 +18,8 @@ subdir('android')\n> subdir('libcamera')\n> subdir('ipa')\n>\n> +subdir('lc-compliance')\n> +\n> subdir('cam')\n> subdir('qcam')\n>\n> --\n> 2.30.0\n>\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DDC4FC33BB\n\tfor ;\n\tFri, 29 Jan 2021 18:05:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4FC0F683BD;\n\tFri, 29 Jan 2021 19:05:42 +0100 (CET)","from mail-lj1-x235.google.com (mail-lj1-x235.google.com\n\t[IPv6:2a00:1450:4864:20::235])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 079BA683AB\n\tfor ;\n\tFri, 29 Jan 2021 19:05:39 +0100 (CET)","by mail-lj1-x235.google.com with SMTP id m22so4085653ljj.4\n\tfor ;\n\tFri, 29 Jan 2021 10:05:39 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"L0ItPNL8\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=XpOqgx26k2jjnvr0l/yNDU1275Icsn1fjDALRgWLW50=;\n\tb=L0ItPNL8mN26nVX41bB/J18SYrDs/+V9QknFHU/MzfbExCTS9Pq6gZFh5Tm+XXDpk7\n\t133Dk4pNzCtJnD3YGlbHdD+nSEzAhxetRrFyn0OhXm7hMVhfV7y+mJNGi1bRCqi0Yn0Q\n\txBarvHOe/cXvWGEgZ13TGUcldeiJ86QvdWtBD8/Wr73/p/eodpFIxaDjXRwphfCdVsEb\n\tQzt68CU8+d3j5LkbffZ80t9XR2m6MMSP6HV2iqZQT/rQhwA9FetgGosjWKpd/Lqa2yXZ\n\t/aTMZHMGp+mcDdkstcOU+A/+Bka7ufnHbm5FPRoB1WbNPVbPRjr6KZWTrBoPuCO0P4xz\n\t+aBQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=XpOqgx26k2jjnvr0l/yNDU1275Icsn1fjDALRgWLW50=;\n\tb=HXyvEEC+fQwB+xz3Dp1Xh5rUeMrEd8JfawcJCiD1I5MwS4KsH9WXDmcQVQDRD8wq6F\n\ttu8csM/zb8VA22uwgnDWk6HCLWTBdd/xVe9voV9rEIUjMVV5HJz7Y9GuerMMfDTaeRIu\n\tV+uSk3vvZoT4NjMK0LTGJsttTGH5HA0r/3IwAxudRadBLLWrRcqW2HDpAdcql+XENMmZ\n\t9+BGvHHabHTzFkQgZlKZpmvlgWZRlFhA+VMFst4sMZL+jXWqBEiqKMGVTLxsM1AeSTJU\n\t/V/zoqiyspsorn++/0mLBQUNhF1GolW21+1jNNdugShlk496rPDvm8NLU0zngskgAG6k\n\tMk/A==","X-Gm-Message-State":"AOAM531ePu9xVZXQtktQRn03qxvt97ytvYXsBxZWXQFYZyCpqsePzWrQ\n\t3q8u4cDgMUGPDCxEyiQvg7NOY6IuhvJOHCdsnozHcQ==","X-Google-Smtp-Source":"ABdhPJyFT+26qqnL6VpDm5s9yZsQ8MdQAC75lcSh/RuPOkTbUiOi0sBHuuNFkTPTLSQ+/r0huDSVEfErgkzn1xnnAZE=","X-Received":"by 2002:a2e:9992:: with SMTP id\n\tw18mr3104997lji.169.1611943538984; \n\tFri, 29 Jan 2021 10:05:38 -0800 (PST)","MIME-Version":"1.0","References":"<20210129165336.256739-1-niklas.soderlund@ragnatech.se>","In-Reply-To":"<20210129165336.256739-1-niklas.soderlund@ragnatech.se>","From":"Naushir Patuck ","Date":"Fri, 29 Jan 2021 18:05:22 +0000","Message-ID":"","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","Subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Content-Type":"multipart/mixed;\n\tboundary=\"===============6618546487669710010==\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":14856,"web_url":"https://patchwork.libcamera.org/comment/14856/","msgid":"<20210129184856.w475ktjhb6sy5p4m@basti-TUXEDO-Book-XA1510>","date":"2021-01-29T18:48:56","subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","submitter":{"id":78,"url":"https://patchwork.libcamera.org/api/people/78/","name":"Sebastian Fricke","email":"sebastian.fricke@posteo.net"},"content":"Hey Niklas,\n\nNice I was eagerly waiting for this patch, thank you for the work.\n\nHere is how it looks on my NanoPC-T4 (RkISP1)\nhttps://paste.debian.net/1183263/\n\nAnd below I have added some comments, I expect to play around with that\ntool in the near future :).\n\nOn 29.01.2021 17:53, Niklas Söderlund wrote:\n>Add a compliance tool to ease testing of cameras on target. In contrast\n\nCould you clarify this sentence a little more? Which target do you\nmean?\n\n>to the unit-tests under test/ that aims to test the internal components\n>of libcamera the compliance aims to test application use-cases and to\n\ns/compliance/compliance tool/\n\n>some extend the public API.\n>\n>This change adds the boilerplate code for a simple framework to write\n>tests in as well as two simple test. The tests aims both to demonstrate\n\nMaybe this sounds better?\ns/adds the boilerplate code for a simple framework to write tests in/\n adds the boilerplate code of a simple framework for the creation of\n tests/\n\ns/The tests aims/The tests aim/\n\n>the tool and to catch real problems. The tests added are:\n>\n> - Test that if one queues exactly N requests to a camera exactly N\n> requests are eventually completed.\n>\n> - Test that a configured camera can be started and stopped multiple\n> times in an attempt to exercise cleanup code paths otherwise not\n> often tested with 'cam' for example.\n>\n>Example pass run on Raspberry Pi:\n>\n> $ LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c \"/base/soc/i2c0mux/i2c@1/imx219@10\"\n> [2:36:17.672447527] [11896] ERROR V4L2 v4l2_device.cpp:191 'imx219 10-0010': Control 0x009a0922 not found\n> [2:36:17.673095289] [11896] ERROR V4L2 v4l2_subdevice.cpp:285 'imx219 10-0010': Unable to get rectangle 2 on pad 0: Invalid argument\n> Using camera /base/soc/i2c0mux/i2c@1/imx219@10\n> Test single capture cycles\n> - SKIP - Camera needs 4 requests, can't test only 1\n> - SKIP - Camera needs 4 requests, can't test only 2\n> - SKIP - Camera needs 4 requests, can't test only 3\n> - PASS - Balanced capture of 5 requests with 1 start cycles\n> - PASS - Balanced capture of 8 requests with 1 start cycles\n> - PASS - Balanced capture of 13 requests with 1 start cycles\n> - PASS - Balanced capture of 21 requests with 1 start cycles\n> - PASS - Balanced capture of 34 requests with 1 start cycles\n> - PASS - Balanced capture of 55 requests with 1 start cycles\n> - PASS - Balanced capture of 89 requests with 1 start cycles\n> Test multiple start/stop cycles\n> - SKIP - Camera needs 4 requests, can't test only 1\n> - SKIP - Camera needs 4 requests, can't test only 2\n> - SKIP - Camera needs 4 requests, can't test only 3\n> - PASS - Balanced capture of 5 requests with 3 start cycles\n> - PASS - Balanced capture of 8 requests with 3 start cycles\n> - PASS - Balanced capture of 13 requests with 3 start cycles\n> - PASS - Balanced capture of 21 requests with 3 start cycles\n> - PASS - Balanced capture of 34 requests with 3 start cycles\n> - PASS - Balanced capture of 55 requests with 3 start cycles\n> - PASS - Balanced capture of 89 requests with 3 start cycles\n> 20 tests executed, 14 tests passed, 6 tests skipped and 0 tests failed\n>\n>Example fail run on IPU3:\n>\n> - LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c \"\\_SB_.PCI0.I2C2.CAM0\"\n> - [0:31:36.992763102] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov13858 8-0010': Control 0x009a0922 not found\n> - [0:31:36.992815436] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.992847049] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.992869029] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993152932] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993172833] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the active area size\n> - [0:31:36.993208149] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993224502] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the active area size\n> - [0:31:36.993354770] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov5670 10-0036': Control 0x009a0922 not found\n> - [0:31:36.993377622] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993398093] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993415611] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993660192] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993678492] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the active area size\n> - [0:31:36.993724648] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> - [0:31:36.993741032] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the active area size\n> - Using camera \\_SB_.PCI0.I2C2.CAM0\n> - Test single capture cycles\n> - - SKIP - Camera needs 4 requests, can't test only 1\n> - - SKIP - Camera needs 4 requests, can't test only 2\n> - - SKIP - Camera needs 4 requests, can't test only 3\n> - - PASS - Balanced capture of 5 requests with 1 start cycles\n> - - PASS - Balanced capture of 8 requests with 1 start cycles\n> - - PASS - Balanced capture of 13 requests with 1 start cycles\n> - - PASS - Balanced capture of 21 requests with 1 start cycles\n> - - PASS - Balanced capture of 34 requests with 1 start cycles\n> - [0:31:41.969783243] [6420] ERROR IPU3 cio2.cpp:264 CIO2 buffer underrun\n> \n>\n>Signed-off-by: Niklas Söderlund \n>---\n>Hi,\n>\n>I have tested this on RkISP1, RPi and IPU3 on latest master [1]. I plan\n>to post separate patches which starts to address some of the problems\n>found while working. A fix for 'cam --captur=N' is coming shortly for\n>example.\n>\n>I still have to find a good fix for a races found in the\n>IPU3 pipeline between the CIO2 and IMGU (as showed above). What I don't\n>(yet) have a plan for is how to fix that request queueing errors are not\n>propagated in our application facing API leading the test application to\n>wait forever to a request the library knows have failed, ideas welcome.\n>\n>1. 958c80a4f1c28301 (\"android: camera_device: Set AE precapture trigger according to request\")\n>---\n> src/lc-compliance/main.cpp | 139 ++++++++++++++++++\n> src/lc-compliance/meson.build | 24 ++++\n> src/lc-compliance/results.cpp | 75 ++++++++++\n> src/lc-compliance/results.h | 45 ++++++\n> src/lc-compliance/single_stream.cpp | 210 ++++++++++++++++++++++++++++\n> src/lc-compliance/tests.h | 16 +++\n> src/meson.build | 2 +\n> 7 files changed, 511 insertions(+)\n> create mode 100644 src/lc-compliance/main.cpp\n> create mode 100644 src/lc-compliance/meson.build\n> create mode 100644 src/lc-compliance/results.cpp\n> create mode 100644 src/lc-compliance/results.h\n> create mode 100644 src/lc-compliance/single_stream.cpp\n> create mode 100644 src/lc-compliance/tests.h\n>\n>diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n>new file mode 100644\n>index 0000000000000000..e1cbce7eac3df2bc\n>--- /dev/null\n>+++ b/src/lc-compliance/main.cpp\n>@@ -0,0 +1,139 @@\n>+/* SPDX-License-Identifier: GPL-2.0-or-later */\n>+/*\n>+ * Copyright (C) 2020, Google Inc.\n>+ *\n>+ * main.cpp - lc-compliance - The libcamera compliance tool\n>+ */\n>+\n>+#include \n>+#include \n>+#include \n>+\n>+#include \n>+\n>+#include \"../cam/options.h\"\n>+#include \"tests.h\"\n>+\n>+using namespace libcamera;\n>+\n>+class Harness\n>+{\n>+public:\n>+\tHarness();\n>+\t~Harness();\n>+\n>+\tint exec(int argc, char **argv);\n>+\n>+private:\n>+\tenum {\n>+\t\tOptCamera = 'c',\n>+\t\tOptHelp = 'h',\n>+\t};\n>+\n>+\tint parseOptions(int argc, char **argv);\n>+\tint init(int argc, char **argv);\n>+\n>+\tOptionsParser::Options options_;\n>+\tstd::unique_ptr cm_;\n>+\tstd::shared_ptr camera_;\n>+};\n>+\n>+Harness::Harness()\n>+{\n>+\tcm_ = std::make_unique();\n>+}\n>+\n>+Harness::~Harness()\n>+{\n>+\tif (camera_) {\n>+\t\tcamera_->release();\n>+\t\tcamera_.reset();\n>+\t}\n>+\n>+\tcm_->stop();\n>+}\n>+\n>+int Harness::exec(int argc, char **argv)\n>+{\n>+\tint ret = init(argc, argv);\n>+\tif (ret)\n>+\t\treturn ret;\n>+\n>+\tstd::vector results;\n>+\n>+\tresults.push_back(testSingleStream(camera_));\n>+\n>+\tfor (const Results &result : results) {\n>+\t\tret = result.summary();\n>+\t\tif (ret)\n>+\t\t\treturn ret;\n>+\t}\n>+\n>+\treturn 0;\n>+}\n>+\n>+int Harness::init(int argc, char **argv)\n>+{\n>+\tint ret = parseOptions(argc, argv);\n>+\tif (ret < 0)\n>+\t\treturn ret;\n>+\n>+\tret = cm_->start();\n>+\tif (ret) {\n>+\t\tstd::cout << \"Failed to start camera manager: \"\n>+\t\t\t << strerror(-ret) << std::endl;\n>+\t\treturn ret;\n>+\t}\n>+\n>+\tif (!options_.isSet(OptCamera)) {\n>+\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n>+\t\tfor (const std::shared_ptr &cam : cm_->cameras())\n>+\t\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n>+\t\treturn -ENODEV;\n>+\t}\n>+\n>+\tconst std::string &cameraId = options_[OptCamera];\n>+\tcamera_ = cm_->get(cameraId);\n>+\tif (!camera_) {\n>+\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n>+\t\tfor (const std::shared_ptr &cam : cm_->cameras())\n>+\t\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n>+\t\treturn -ENODEV;\n>+\t}\n>+\n>+\tif (camera_->acquire()) {\n>+\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n>+\t\treturn -EINVAL;\n>+\t}\n>+\n>+\tstd::cout << \"Using camera \" << cameraId << std::endl;\n>+\n>+\treturn 0;\n>+}\n>+\n>+int Harness::parseOptions(int argc, char **argv)\n>+{\n>+\tOptionsParser parser;\n>+\tparser.addOption(OptCamera, OptionString,\n>+\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n>+\t\t\t ArgumentRequired, \"camera\");\n>+\tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n>+\t\t\t \"help\");\n>+\n>+\toptions_ = parser.parse(argc, argv);\n>+\tif (!options_.valid())\n>+\t\treturn -EINVAL;\n>+\n>+\tif (options_.empty() || options_.isSet(OptHelp)) {\n>+\t\tparser.usage();\n>+\t\treturn options_.empty() ? -EINVAL : -EINTR;\n>+\t}\n>+\n>+\treturn 0;\n>+}\n>+\n>+int main(int argc, char **argv)\n>+{\n>+\tHarness harness;\n>+\treturn harness.exec(argc, argv) ? EXIT_FAILURE : 0;\n>+}\n>diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n>new file mode 100644\n>index 0000000000000000..d700a307c71405b4\n>--- /dev/null\n>+++ b/src/lc-compliance/meson.build\n>@@ -0,0 +1,24 @@\n>+# SPDX-License-Identifier: CC0-1.0\n>+\n>+libevent = dependency('libevent_pthreads', required : false)\n>+\n>+if not libevent.found()\n>+ warning('libevent_pthreads not found, \\'lc-compliance\\' application will not be compiled')\n>+ subdir_done()\n>+endif\n>+\n>+lc_compliance_sources = files([\n>+ '../cam/event_loop.cpp',\n>+ '../cam/options.cpp',\n>+ 'main.cpp',\n>+ 'results.cpp',\n>+ 'single_stream.cpp',\n>+])\n>+\n>+lc_compliance = executable('lc-compliance', lc_compliance_sources,\n>+ dependencies : [\n>+ libatomic,\n>+ libcamera_dep,\n>+ libevent,\n>+ ],\n>+ install : true)\n>diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp\n>new file mode 100644\n>index 0000000000000000..fb4242bf49a3268b\n>--- /dev/null\n>+++ b/src/lc-compliance/results.cpp\n>@@ -0,0 +1,75 @@\n>+/* SPDX-License-Identifier: GPL-2.0-or-later */\n>+/*\n>+ * Copyright (C) 2020, Google Inc.\n>+ *\n>+ * results.cpp - Test result aggregator\n>+ */\n>+\n>+#include \"results.h\"\n>+\n>+#include \n>+\n>+void Results::add(const Result &result)\n>+{\n>+\tif (result.first == Pass)\n>+\t\tpassed_++;\n>+\telse if (result.first == Fail)\n>+\t\tfailed_++;\n>+\telse if (result.first == Skip)\n>+\t\tskipped_++;\n>+\n>+\tprintResult(result);\n>+}\n>+\n>+void Results::add(Status status, const std::string &message)\n>+{\n>+\tadd({ status, message });\n>+}\n>+\n>+void Results::fail(const std::string &message)\n>+{\n>+\tadd(Fail, message);\n>+}\n>+\n>+void Results::pass(const std::string &message)\n>+{\n>+\tadd(Pass, message);\n>+}\n>+\n>+void Results::skip(const std::string &message)\n>+{\n>+\tadd(Skip, message);\n>+}\n>+\n>+int Results::summary() const\n>+{\n>+\tif (failed_ + passed_ + skipped_ != planned_) {\n>+\t\tstd::cout << \"Planned and executed numer of tests differ \"\n\ns/numer/number/\n\n>+\t\t\t << failed_ + passed_ + skipped_ << \" executed \"\n>+\t\t\t << planned_ << \" planned\" << std::endl;\n>+\n>+\t\treturn -EINVAL;\n>+\t}\n>+\n>+\tstd::cout << planned_ << \" tests executed, \"\n>+\t\t << passed_ << \" tests passed, \"\n>+\t\t << skipped_ << \" tests skipped and \"\n>+\t\t << failed_ << \" tests failed \" << std::endl;\n>+\n>+\treturn 0;\n>+}\n>+\n>+void Results::printResult(const Result &result)\n>+{\n>+\tstd::string prefix;\n>+\n>+\t/* \\todo Make parsable as TAP. */\n\nWhat do you mean with TAP?\n\n>+\tif (result.first == Pass)\n>+\t\tprefix = \"PASS\";\n>+\telse if (result.first == Fail)\n>+\t\tprefix = \"FAIL\";\n>+\telse if (result.first == Skip)\n>+\t\tprefix = \"SKIP\";\n>+\n>+\tstd::cout << \"- \" << prefix << \" - \" << result.second << std::endl;\n>+}\n>diff --git a/src/lc-compliance/results.h b/src/lc-compliance/results.h\n>new file mode 100644\n>index 0000000000000000..a02fd5ab46edd62c\n>--- /dev/null\n>+++ b/src/lc-compliance/results.h\n>@@ -0,0 +1,45 @@\n>+/* SPDX-License-Identifier: GPL-2.0-or-later */\n>+/*\n>+ * Copyright (C) 2020, Google Inc.\n>+ *\n>+ * results.h - Test result aggregator\n>+ */\n>+#ifndef __LC_COMPLIANCE_RESULTS_H__\n>+#define __LC_COMPLIANCE_RESULTS_H__\n>+\n>+#include \n>+\n>+class Results\n>+{\n>+public:\n>+\tenum Status {\n>+\t\tFail,\n>+\t\tPass,\n>+\t\tSkip,\n>+\t};\n>+\n>+\tusing Result = std::pair;\n>+\n>+\tResults(unsigned int planned)\n>+\t\t: planned_(planned), passed_(0), failed_(0), skipped_(0)\n>+\t{\n>+\t}\n>+\n>+\tvoid add(const Result &result);\n>+\tvoid add(Status status, const std::string &message);\n>+\tvoid fail(const std::string &message);\n>+\tvoid pass(const std::string &message);\n>+\tvoid skip(const std::string &message);\n>+\n>+\tint summary() const;\n>+\n>+private:\n>+\tvoid printResult(const Result &result);\n>+\n>+\tunsigned int planned_;\n>+\tunsigned int passed_;\n>+\tunsigned int failed_;\n>+\tunsigned int skipped_;\n>+};\n>+\n>+#endif /* __LC_COMPLIANCE_RESULTS_H__ */\n>diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n>new file mode 100644\n>index 0000000000000000..d31b6f8d06487a85\n>--- /dev/null\n>+++ b/src/lc-compliance/single_stream.cpp\n>@@ -0,0 +1,210 @@\n>+/* SPDX-License-Identifier: GPL-2.0-or-later */\n>+/*\n>+ * Copyright (C) 2020, Google Inc.\n>+ *\n>+ * single_stream.cpp - Test a single camera stream\n>+ */\n>+\n>+#include \n>+\n>+#include \"../cam/event_loop.h\"\n>+#include \"tests.h\"\n>+\n>+using namespace libcamera;\n>+\n>+class SimpleCapture\n>+{\n>+public:\n>+\tSimpleCapture(std::shared_ptr camera)\n>+\t\t: camera_(camera), allocator_(std::make_unique(camera))\n>+\t{\n>+\t}\n>+\n>+\tResults::Result configure(StreamRole role);\n>+\tResults::Result start();\n>+\tResults::Result capture(unsigned int numRequests);\n>+\tResults::Result stop();\n>+\n>+private:\n>+\tint queueRequest(Request *request);\n>+\tvoid requestComplete(Request *request);\n>+\n>+\tstd::shared_ptr camera_;\n>+\tstd::unique_ptr allocator_;\n>+\tstd::unique_ptr config_;\n>+\n>+\tEventLoop *loop_;\n>+\tunsigned int queueCount_;\n>+\tunsigned int captureCount_;\n>+\tunsigned int captureLimit_;\n>+};\n>+\n>+Results::Result SimpleCapture::configure(StreamRole role)\n>+{\n>+\tconfig_ = camera_->generateConfiguration({ role });\n>+\n>+\tif (config_->validate() != CameraConfiguration::Valid) {\n>+\t\tconfig_.reset();\n>+\t\treturn { Results::Fail, \"Configuration not valid\" };\n>+\t}\n>+\n>+\tif (camera_->configure(config_.get())) {\n>+\t\tconfig_.reset();\n>+\t\treturn { Results::Fail, \"Failed to configure camera\" };\n>+\t}\n>+\n>+\treturn { Results::Pass, \"Configure camera\" };\n>+}\n>+\n>+Results::Result SimpleCapture::start()\n>+{\n>+\tStream *stream = config_->at(0).stream();\n>+\tif (allocator_->allocate(stream) < 0)\n>+\t\treturn { Results::Fail, \"Failed to allocate buffers camera\" };\n\ns/buffers camera/buffers/ ?\n\n>+\n>+\tif (camera_->start())\n>+\t\treturn { Results::Fail, \"Failed to start camera\" };\n>+\n>+\tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n>+\n>+\treturn { Results::Pass, \"Started camera\" };\n>+}\n>+\n>+Results::Result SimpleCapture::capture(unsigned int numRequests)\n>+{\n>+\tStream *stream = config_->at(0).stream();\n>+\tconst std::vector> &buffers = allocator_->buffers(stream);\n>+\n>+\t/* No point in testing less requests then the camera depth. */\n>+\tif (buffers.size() > numRequests)\n>+\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers.size()) + \" requests, can't test only \" + std::to_string(numRequests) };\n\nThis seems to be quite long\n\n>+\n>+\tqueueCount_ = 0;\n>+\tcaptureCount_ = 0;\n>+\tcaptureLimit_ = numRequests;\n>+\n>+\t/* Queue the camera recommended number of reqeuests. */\n\ns/camera recommended/recommended/ ?\n\n>+\tstd::vector> requests;\n>+\tfor (const std::unique_ptr &buffer : allocator_->buffers(stream)) {\n>+\t\tstd::unique_ptr request = camera_->createRequest();\n>+\t\tif (!request)\n>+\t\t\treturn { Results::Fail, \"Can't create request\" };\n>+\n>+\t\tif (request->addBuffer(stream, buffer.get()))\n>+\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n>+\n>+\t\tif (queueRequest(request.get()) < 0)\n>+\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n>+\n>+\t\trequests.push_back(std::move(request));\n>+\t}\n>+\n>+\t/* Run capture session. */\n>+\tloop_ = new EventLoop();\n>+\tloop_->exec();\n>+\tdelete loop_;\n>+\n>+\tif (captureCount_ != captureLimit_)\n>+\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) + \" request, wanted \" + std::to_string(captureLimit_) };\n\nThis is also pretty long\n\n>+\n>+\treturn { Results::Pass, \"Balanced capture of \" + std::to_string(numRequests) + \" requests\" };\n>+}\n>+\n>+Results::Result SimpleCapture::stop()\n>+{\n>+\tStream *stream = config_->at(0).stream();\n>+\n>+\tcamera_->stop();\n>+\n>+\tcamera_->requestCompleted.disconnect(this, &SimpleCapture::requestComplete);\n>+\n>+\tallocator_->free(stream);\n>+\n>+\treturn { Results::Pass, \"Stopped camera\" };\n>+}\n>+\n>+int SimpleCapture::queueRequest(Request *request)\n>+{\n>+\tqueueCount_++;\n>+\tif (queueCount_ > captureLimit_)\n>+\t\treturn 0;\n>+\n>+\treturn camera_->queueRequest(request);\n>+}\n>+\n>+void SimpleCapture::requestComplete(Request *request)\n>+{\n>+\tcaptureCount_++;\n>+\tif (captureCount_ >= captureLimit_) {\n>+\t\tloop_->exit(0);\n>+\t\treturn;\n>+\t}\n>+\n>+\trequest->reuse(Request::ReuseBuffers);\n>+\tif (queueRequest(request))\n>+\t\tloop_->exit(-EINVAL);\n>+}\n>+\n>+Results::Result testRequestBalance(std::shared_ptr camera,\n>+\t\t\t\t unsigned int startCycles,\n>+\t\t\t\t unsigned int numRequests)\n>+{\n>+\tSimpleCapture capture(camera);\n>+\tResults::Result ret;\n>+\n>+\tret = capture.configure(Viewfinder);\n>+\tif (ret.first != Results::Pass)\n>+\t\treturn ret;\n>+\n>+\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n>+\t\tret = capture.start();\n>+\t\tif (ret.first != Results::Pass)\n>+\t\t\treturn ret;\n>+\n>+\t\tret = capture.capture(numRequests);\n>+\t\tif (ret.first != Results::Pass) {\n>+\t\t\tcapture.stop();\n>+\t\t\treturn ret;\n>+\t\t}\n>+\n>+\t\tret = capture.stop();\n>+\t\tif (ret.first != Results::Pass)\n>+\t\t\treturn ret;\n>+\t}\n>+\n>+\treturn { Results::Pass, \"Balanced capture of \" + std::to_string(numRequests) + \" requests with \" + std::to_string(startCycles) + \" start cycles\" };\n>+}\n>+\n>+Results testSingleStream(std::shared_ptr camera)\n>+{\n>+\tconst std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n>+\n>+\tResults results(numRequests.size() * 2);\n>+\n>+\tif (!camera)\n>+\t\treturn results;\n>+\n>+\t/*\n>+\t * Test single capture cycles\n>+\t *\n>+\t * Makes sure the camera completes the exact number of requests queued.\n>+\t * Example failure is a camera that needs N+M requests queued to\n>+\t * complete N requests to the application.\n>+\t */\n>+\tstd::cout << \"Test single capture cycles\" << std::endl;\n>+\tfor (unsigned int num : numRequests)\n>+\t\tresults.add(testRequestBalance(camera, 1, num));\n>+\n>+\t/*\n>+\t * Test multiple start/stop cycles\n>+\t *\n>+\t * Makes sure the camera supports multiple start/stop cycles.\n>+\t * Example failure is a camera that does not clean up correctly in its\n>+\t * error path but is only tested by single-capture applications.\n>+\t */\n>+\tstd::cout << \"Test multiple start/stop cycles\" << std::endl;\n>+\tfor (unsigned int num : numRequests)\n>+\t\tresults.add(testRequestBalance(camera, 3, num));\n>+\n>+\treturn results;\n>+}\n>diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n>new file mode 100644\n>index 0000000000000000..396605214e4b8980\n>--- /dev/null\n>+++ b/src/lc-compliance/tests.h\n>@@ -0,0 +1,16 @@\n>+/* SPDX-License-Identifier: GPL-2.0-or-later */\n>+/*\n>+ * Copyright (C) 2020, Google Inc.\n>+ *\n>+ * tests.h - Test modules\n>+ */\n>+#ifndef __LC_COMPLIANCE_TESTS_H__\n>+#define __LC_COMPLIANCE_TESTS_H__\n>+\n>+#include \n>+\n>+#include \"results.h\"\n>+\n>+Results testSingleStream(std::shared_ptr camera);\n>+\n>+#endif /* __LC_COMPLIANCE_TESTS_H__ */\n>diff --git a/src/meson.build b/src/meson.build\n>index 4b75f05878bcb702..a8e1af7adf2ca9c8 100644\n>--- a/src/meson.build\n>+++ b/src/meson.build\n>@@ -18,6 +18,8 @@ subdir('android')\n> subdir('libcamera')\n> subdir('ipa')\n>\n>+subdir('lc-compliance')\n>+\n> subdir('cam')\n> subdir('qcam')\n>\n>-- \n>2.30.0\n>\n>_______________________________________________\n>libcamera-devel mailing list\n>libcamera-devel@lists.libcamera.org\n>https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BB30ABD808\n\tfor ;\n\tFri, 29 Jan 2021 18:49:00 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 17B59683BA;\n\tFri, 29 Jan 2021 19:49:00 +0100 (CET)","from mout01.posteo.de (mout01.posteo.de [185.67.36.65])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A84AE683B3\n\tfor ;\n\tFri, 29 Jan 2021 19:48:58 +0100 (CET)","from submission (posteo.de [89.146.220.130]) \n\tby mout01.posteo.de (Postfix) with ESMTPS id E1C4B160061\n\tfor ;\n\tFri, 29 Jan 2021 19:48:57 +0100 (CET)","from customer (localhost [127.0.0.1])\n\tby submission (posteo.de) with ESMTPSA id 4DS5w92xj2z9rxP;\n\tFri, 29 Jan 2021 19:48:57 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=posteo.net header.i=@posteo.net\n\theader.b=\"kIpSpt/i\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;\n\tt=1611946137; bh=EBMjH5ZjPhd9KMbwYXXuQACmtrZpnrTZ1MpCnLUGvA0=;\n\th=Date:From:To:Cc:Subject:From;\n\tb=kIpSpt/iTisVprCQBEqIYtu99aC8Yj6iY9ikHYBb+CGPxp4b+aCDoRETRF56Vjhjn\n\tcN1Am/6X1M0D2txsBVMq6H6NymIz9vnggIl++KIOafpxoJmxl74R5dUfL90J4dIQnq\n\t/tY40U405hEIVcXQ8YsoOagZpA1nA/kGaRYNwMgw5Y/Gyj+vUURWrfkwMo3zslwuNa\n\t3rrX2uqReXqRR0N/t5sax1CSAhommdGeYCnj/o4Y8BZzi0iSeG5d1QYqlALtTHqL/W\n\txdbWZsaTfkmyK2uOQjRIJINNHTPR3BYmo8XyQ2VHOfYZCQq/pebQ7t5ORBN3PCFHlP\n\tCB0T0LEnw5HfQ==","Date":"Fri, 29 Jan 2021 19:48:56 +0100","From":"Sebastian Fricke ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Message-ID":"<20210129184856.w475ktjhb6sy5p4m@basti-TUXEDO-Book-XA1510>","References":"<20210129165336.256739-1-niklas.soderlund@ragnatech.se>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20210129165336.256739-1-niklas.soderlund@ragnatech.se>","Subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Transfer-Encoding":"base64","Content-Type":"text/plain; charset=\"utf-8\"; Format=\"flowed\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":14857,"web_url":"https://patchwork.libcamera.org/comment/14857/","msgid":"","date":"2021-01-29T21:54:20","subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Sebastian,\n\nThanks for your feedback.\n\nOn 2021-01-29 19:48:56 +0100, Sebastian Fricke wrote:\n> Hey Niklas,\n> \n> Nice I was eagerly waiting for this patch, thank you for the work.\n> \n> Here is how it looks on my NanoPC-T4 (RkISP1)\n> https://paste.debian.net/1183263/\n> \n> And below I have added some comments, I expect to play around with that\n> tool in the near future :).\n\nI'm glad you like it.\n\n> \n> On 29.01.2021 17:53, Niklas Söderlund wrote:\n> > Add a compliance tool to ease testing of cameras on target. In contrast\n> \n> Could you clarify this sentence a little more? Which target do you\n> mean?\n\nThe develop machine is sometimes referred to as the 'host' and the \ndevice under test is referred to as the 'target'. I intended to hint at \nthat the unit-test under test/ where run on the host machine and the \nlc-compliance tool was intended to run on the target.\n\nBut reading it now this make little sens as lc-compliance is just as \nvalid to run on the host as you might test a device that is attached to \nit (USB camera). I will s/on target/.\n\n> \n> > to the unit-tests under test/ that aims to test the internal components\n> > of libcamera the compliance aims to test application use-cases and to\n> \n> s/compliance/compliance tool/\n> \n> > some extend the public API.\n> > \n> > This change adds the boilerplate code for a simple framework to write\n> > tests in as well as two simple test. The tests aims both to demonstrate\n> \n> Maybe this sounds better?\n> s/adds the boilerplate code for a simple framework to write tests in/\n> adds the boilerplate code of a simple framework for the creation of\n> tests/\n> \n> s/The tests aims/The tests aim/\n> \n> > the tool and to catch real problems. The tests added are:\n> > \n> > - Test that if one queues exactly N requests to a camera exactly N\n> > requests are eventually completed.\n> > \n> > - Test that a configured camera can be started and stopped multiple\n> > times in an attempt to exercise cleanup code paths otherwise not\n> > often tested with 'cam' for example.\n> > \n> > Example pass run on Raspberry Pi:\n> > \n> > $ LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c \"/base/soc/i2c0mux/i2c@1/imx219@10\"\n> > [2:36:17.672447527] [11896] ERROR V4L2 v4l2_device.cpp:191 'imx219 10-0010': Control 0x009a0922 not found\n> > [2:36:17.673095289] [11896] ERROR V4L2 v4l2_subdevice.cpp:285 'imx219 10-0010': Unable to get rectangle 2 on pad 0: Invalid argument\n> > Using camera /base/soc/i2c0mux/i2c@1/imx219@10\n> > Test single capture cycles\n> > - SKIP - Camera needs 4 requests, can't test only 1\n> > - SKIP - Camera needs 4 requests, can't test only 2\n> > - SKIP - Camera needs 4 requests, can't test only 3\n> > - PASS - Balanced capture of 5 requests with 1 start cycles\n> > - PASS - Balanced capture of 8 requests with 1 start cycles\n> > - PASS - Balanced capture of 13 requests with 1 start cycles\n> > - PASS - Balanced capture of 21 requests with 1 start cycles\n> > - PASS - Balanced capture of 34 requests with 1 start cycles\n> > - PASS - Balanced capture of 55 requests with 1 start cycles\n> > - PASS - Balanced capture of 89 requests with 1 start cycles\n> > Test multiple start/stop cycles\n> > - SKIP - Camera needs 4 requests, can't test only 1\n> > - SKIP - Camera needs 4 requests, can't test only 2\n> > - SKIP - Camera needs 4 requests, can't test only 3\n> > - PASS - Balanced capture of 5 requests with 3 start cycles\n> > - PASS - Balanced capture of 8 requests with 3 start cycles\n> > - PASS - Balanced capture of 13 requests with 3 start cycles\n> > - PASS - Balanced capture of 21 requests with 3 start cycles\n> > - PASS - Balanced capture of 34 requests with 3 start cycles\n> > - PASS - Balanced capture of 55 requests with 3 start cycles\n> > - PASS - Balanced capture of 89 requests with 3 start cycles\n> > 20 tests executed, 14 tests passed, 6 tests skipped and 0 tests failed\n> > \n> > Example fail run on IPU3:\n> > \n> > - LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c \"\\_SB_.PCI0.I2C2.CAM0\"\n> > - [0:31:36.992763102] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov13858 8-0010': Control 0x009a0922 not found\n> > - [0:31:36.992815436] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.992847049] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.992869029] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993152932] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993172833] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the active area size\n> > - [0:31:36.993208149] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993224502] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the active area size\n> > - [0:31:36.993354770] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov5670 10-0036': Control 0x009a0922 not found\n> > - [0:31:36.993377622] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993398093] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993415611] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993660192] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993678492] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the active area size\n> > - [0:31:36.993724648] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993741032] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the active area size\n> > - Using camera \\_SB_.PCI0.I2C2.CAM0\n> > - Test single capture cycles\n> > - - SKIP - Camera needs 4 requests, can't test only 1\n> > - - SKIP - Camera needs 4 requests, can't test only 2\n> > - - SKIP - Camera needs 4 requests, can't test only 3\n> > - - PASS - Balanced capture of 5 requests with 1 start cycles\n> > - - PASS - Balanced capture of 8 requests with 1 start cycles\n> > - - PASS - Balanced capture of 13 requests with 1 start cycles\n> > - - PASS - Balanced capture of 21 requests with 1 start cycles\n> > - - PASS - Balanced capture of 34 requests with 1 start cycles\n> > - [0:31:41.969783243] [6420] ERROR IPU3 cio2.cpp:264 CIO2 buffer underrun\n> > \n> > \n> > Signed-off-by: Niklas Söderlund \n> > ---\n> > Hi,\n> > \n> > I have tested this on RkISP1, RPi and IPU3 on latest master [1]. I plan\n> > to post separate patches which starts to address some of the problems\n> > found while working. A fix for 'cam --captur=N' is coming shortly for\n> > example.\n> > \n> > I still have to find a good fix for a races found in the\n> > IPU3 pipeline between the CIO2 and IMGU (as showed above). What I don't\n> > (yet) have a plan for is how to fix that request queueing errors are not\n> > propagated in our application facing API leading the test application to\n> > wait forever to a request the library knows have failed, ideas welcome.\n> > \n> > 1. 958c80a4f1c28301 (\"android: camera_device: Set AE precapture trigger according to request\")\n> > ---\n> > src/lc-compliance/main.cpp | 139 ++++++++++++++++++\n> > src/lc-compliance/meson.build | 24 ++++\n> > src/lc-compliance/results.cpp | 75 ++++++++++\n> > src/lc-compliance/results.h | 45 ++++++\n> > src/lc-compliance/single_stream.cpp | 210 ++++++++++++++++++++++++++++\n> > src/lc-compliance/tests.h | 16 +++\n> > src/meson.build | 2 +\n> > 7 files changed, 511 insertions(+)\n> > create mode 100644 src/lc-compliance/main.cpp\n> > create mode 100644 src/lc-compliance/meson.build\n> > create mode 100644 src/lc-compliance/results.cpp\n> > create mode 100644 src/lc-compliance/results.h\n> > create mode 100644 src/lc-compliance/single_stream.cpp\n> > create mode 100644 src/lc-compliance/tests.h\n> > \n> > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > new file mode 100644\n> > index 0000000000000000..e1cbce7eac3df2bc\n> > --- /dev/null\n> > +++ b/src/lc-compliance/main.cpp\n> > @@ -0,0 +1,139 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * main.cpp - lc-compliance - The libcamera compliance tool\n> > + */\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \"../cam/options.h\"\n> > +#include \"tests.h\"\n> > +\n> > +using namespace libcamera;\n> > +\n> > +class Harness\n> > +{\n> > +public:\n> > +\tHarness();\n> > +\t~Harness();\n> > +\n> > +\tint exec(int argc, char **argv);\n> > +\n> > +private:\n> > +\tenum {\n> > +\t\tOptCamera = 'c',\n> > +\t\tOptHelp = 'h',\n> > +\t};\n> > +\n> > +\tint parseOptions(int argc, char **argv);\n> > +\tint init(int argc, char **argv);\n> > +\n> > +\tOptionsParser::Options options_;\n> > +\tstd::unique_ptr cm_;\n> > +\tstd::shared_ptr camera_;\n> > +};\n> > +\n> > +Harness::Harness()\n> > +{\n> > +\tcm_ = std::make_unique();\n> > +}\n> > +\n> > +Harness::~Harness()\n> > +{\n> > +\tif (camera_) {\n> > +\t\tcamera_->release();\n> > +\t\tcamera_.reset();\n> > +\t}\n> > +\n> > +\tcm_->stop();\n> > +}\n> > +\n> > +int Harness::exec(int argc, char **argv)\n> > +{\n> > +\tint ret = init(argc, argv);\n> > +\tif (ret)\n> > +\t\treturn ret;\n> > +\n> > +\tstd::vector results;\n> > +\n> > +\tresults.push_back(testSingleStream(camera_));\n> > +\n> > +\tfor (const Results &result : results) {\n> > +\t\tret = result.summary();\n> > +\t\tif (ret)\n> > +\t\t\treturn ret;\n> > +\t}\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +int Harness::init(int argc, char **argv)\n> > +{\n> > +\tint ret = parseOptions(argc, argv);\n> > +\tif (ret < 0)\n> > +\t\treturn ret;\n> > +\n> > +\tret = cm_->start();\n> > +\tif (ret) {\n> > +\t\tstd::cout << \"Failed to start camera manager: \"\n> > +\t\t\t << strerror(-ret) << std::endl;\n> > +\t\treturn ret;\n> > +\t}\n> > +\n> > +\tif (!options_.isSet(OptCamera)) {\n> > +\t\tstd::cout << \"No camera specified, available cameras:\" << std::endl;\n> > +\t\tfor (const std::shared_ptr &cam : cm_->cameras())\n> > +\t\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> > +\t\treturn -ENODEV;\n> > +\t}\n> > +\n> > +\tconst std::string &cameraId = options_[OptCamera];\n> > +\tcamera_ = cm_->get(cameraId);\n> > +\tif (!camera_) {\n> > +\t\tstd::cout << \"Camera \" << cameraId << \" not found, available cameras:\" << std::endl;\n> > +\t\tfor (const std::shared_ptr &cam : cm_->cameras())\n> > +\t\t\tstd::cout << \"- \" << cam.get()->id() << std::endl;\n> > +\t\treturn -ENODEV;\n> > +\t}\n> > +\n> > +\tif (camera_->acquire()) {\n> > +\t\tstd::cout << \"Failed to acquire camera\" << std::endl;\n> > +\t\treturn -EINVAL;\n> > +\t}\n> > +\n> > +\tstd::cout << \"Using camera \" << cameraId << std::endl;\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +int Harness::parseOptions(int argc, char **argv)\n> > +{\n> > +\tOptionsParser parser;\n> > +\tparser.addOption(OptCamera, OptionString,\n> > +\t\t\t \"Specify which camera to operate on, by id\", \"camera\",\n> > +\t\t\t ArgumentRequired, \"camera\");\n> > +\tparser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> > +\t\t\t \"help\");\n> > +\n> > +\toptions_ = parser.parse(argc, argv);\n> > +\tif (!options_.valid())\n> > +\t\treturn -EINVAL;\n> > +\n> > +\tif (options_.empty() || options_.isSet(OptHelp)) {\n> > +\t\tparser.usage();\n> > +\t\treturn options_.empty() ? -EINVAL : -EINTR;\n> > +\t}\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +int main(int argc, char **argv)\n> > +{\n> > +\tHarness harness;\n> > +\treturn harness.exec(argc, argv) ? EXIT_FAILURE : 0;\n> > +}\n> > diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> > new file mode 100644\n> > index 0000000000000000..d700a307c71405b4\n> > --- /dev/null\n> > +++ b/src/lc-compliance/meson.build\n> > @@ -0,0 +1,24 @@\n> > +# SPDX-License-Identifier: CC0-1.0\n> > +\n> > +libevent = dependency('libevent_pthreads', required : false)\n> > +\n> > +if not libevent.found()\n> > + warning('libevent_pthreads not found, \\'lc-compliance\\' application will not be compiled')\n> > + subdir_done()\n> > +endif\n> > +\n> > +lc_compliance_sources = files([\n> > + '../cam/event_loop.cpp',\n> > + '../cam/options.cpp',\n> > + 'main.cpp',\n> > + 'results.cpp',\n> > + 'single_stream.cpp',\n> > +])\n> > +\n> > +lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> > + dependencies : [\n> > + libatomic,\n> > + libcamera_dep,\n> > + libevent,\n> > + ],\n> > + install : true)\n> > diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp\n> > new file mode 100644\n> > index 0000000000000000..fb4242bf49a3268b\n> > --- /dev/null\n> > +++ b/src/lc-compliance/results.cpp\n> > @@ -0,0 +1,75 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * results.cpp - Test result aggregator\n> > + */\n> > +\n> > +#include \"results.h\"\n> > +\n> > +#include \n> > +\n> > +void Results::add(const Result &result)\n> > +{\n> > +\tif (result.first == Pass)\n> > +\t\tpassed_++;\n> > +\telse if (result.first == Fail)\n> > +\t\tfailed_++;\n> > +\telse if (result.first == Skip)\n> > +\t\tskipped_++;\n> > +\n> > +\tprintResult(result);\n> > +}\n> > +\n> > +void Results::add(Status status, const std::string &message)\n> > +{\n> > +\tadd({ status, message });\n> > +}\n> > +\n> > +void Results::fail(const std::string &message)\n> > +{\n> > +\tadd(Fail, message);\n> > +}\n> > +\n> > +void Results::pass(const std::string &message)\n> > +{\n> > +\tadd(Pass, message);\n> > +}\n> > +\n> > +void Results::skip(const std::string &message)\n> > +{\n> > +\tadd(Skip, message);\n> > +}\n> > +\n> > +int Results::summary() const\n> > +{\n> > +\tif (failed_ + passed_ + skipped_ != planned_) {\n> > +\t\tstd::cout << \"Planned and executed numer of tests differ \"\n> \n> s/numer/number/\n\nGuess I miss Sweden :-)\n\n> \n> > +\t\t\t << failed_ + passed_ + skipped_ << \" executed \"\n> > +\t\t\t << planned_ << \" planned\" << std::endl;\n> > +\n> > +\t\treturn -EINVAL;\n> > +\t}\n> > +\n> > +\tstd::cout << planned_ << \" tests executed, \"\n> > +\t\t << passed_ << \" tests passed, \"\n> > +\t\t << skipped_ << \" tests skipped and \"\n> > +\t\t << failed_ << \" tests failed \" << std::endl;\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +void Results::printResult(const Result &result)\n> > +{\n> > +\tstd::string prefix;\n> > +\n> > +\t/* \\todo Make parsable as TAP. */\n> \n> What do you mean with TAP?\n\nTAP - Test Anything Protocol\nhttps://testanything.org/\n\nIt's a simple text protocol to convey test results. It has some \nadvantages for testing I find really useful.\n\n- Simple. No need to depend on any test library or such on the test \n point producer. It may be as simple as printf(). I have a rather large \n set of bash script that outputs TAP I use to test boards.\n\n- Robust. It allows application \"noise\" to be intertwined with test \n results. This is really useful if the only link you have between host \n and target is a serial console.\n\n- Portable. There are lots of tools that understands TAP so no need to \n reinvent the wheel which is grate if you want to use it for automatic \n testing for example.\n\n> \n> > +\tif (result.first == Pass)\n> > +\t\tprefix = \"PASS\";\n> > +\telse if (result.first == Fail)\n> > +\t\tprefix = \"FAIL\";\n> > +\telse if (result.first == Skip)\n> > +\t\tprefix = \"SKIP\";\n> > +\n> > +\tstd::cout << \"- \" << prefix << \" - \" << result.second << std::endl;\n> > +}\n> > diff --git a/src/lc-compliance/results.h b/src/lc-compliance/results.h\n> > new file mode 100644\n> > index 0000000000000000..a02fd5ab46edd62c\n> > --- /dev/null\n> > +++ b/src/lc-compliance/results.h\n> > @@ -0,0 +1,45 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * results.h - Test result aggregator\n> > + */\n> > +#ifndef __LC_COMPLIANCE_RESULTS_H__\n> > +#define __LC_COMPLIANCE_RESULTS_H__\n> > +\n> > +#include \n> > +\n> > +class Results\n> > +{\n> > +public:\n> > +\tenum Status {\n> > +\t\tFail,\n> > +\t\tPass,\n> > +\t\tSkip,\n> > +\t};\n> > +\n> > +\tusing Result = std::pair;\n> > +\n> > +\tResults(unsigned int planned)\n> > +\t\t: planned_(planned), passed_(0), failed_(0), skipped_(0)\n> > +\t{\n> > +\t}\n> > +\n> > +\tvoid add(const Result &result);\n> > +\tvoid add(Status status, const std::string &message);\n> > +\tvoid fail(const std::string &message);\n> > +\tvoid pass(const std::string &message);\n> > +\tvoid skip(const std::string &message);\n> > +\n> > +\tint summary() const;\n> > +\n> > +private:\n> > +\tvoid printResult(const Result &result);\n> > +\n> > +\tunsigned int planned_;\n> > +\tunsigned int passed_;\n> > +\tunsigned int failed_;\n> > +\tunsigned int skipped_;\n> > +};\n> > +\n> > +#endif /* __LC_COMPLIANCE_RESULTS_H__ */\n> > diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp\n> > new file mode 100644\n> > index 0000000000000000..d31b6f8d06487a85\n> > --- /dev/null\n> > +++ b/src/lc-compliance/single_stream.cpp\n> > @@ -0,0 +1,210 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * single_stream.cpp - Test a single camera stream\n> > + */\n> > +\n> > +#include \n> > +\n> > +#include \"../cam/event_loop.h\"\n> > +#include \"tests.h\"\n> > +\n> > +using namespace libcamera;\n> > +\n> > +class SimpleCapture\n> > +{\n> > +public:\n> > +\tSimpleCapture(std::shared_ptr camera)\n> > +\t\t: camera_(camera), allocator_(std::make_unique(camera))\n> > +\t{\n> > +\t}\n> > +\n> > +\tResults::Result configure(StreamRole role);\n> > +\tResults::Result start();\n> > +\tResults::Result capture(unsigned int numRequests);\n> > +\tResults::Result stop();\n> > +\n> > +private:\n> > +\tint queueRequest(Request *request);\n> > +\tvoid requestComplete(Request *request);\n> > +\n> > +\tstd::shared_ptr camera_;\n> > +\tstd::unique_ptr allocator_;\n> > +\tstd::unique_ptr config_;\n> > +\n> > +\tEventLoop *loop_;\n> > +\tunsigned int queueCount_;\n> > +\tunsigned int captureCount_;\n> > +\tunsigned int captureLimit_;\n> > +};\n> > +\n> > +Results::Result SimpleCapture::configure(StreamRole role)\n> > +{\n> > +\tconfig_ = camera_->generateConfiguration({ role });\n> > +\n> > +\tif (config_->validate() != CameraConfiguration::Valid) {\n> > +\t\tconfig_.reset();\n> > +\t\treturn { Results::Fail, \"Configuration not valid\" };\n> > +\t}\n> > +\n> > +\tif (camera_->configure(config_.get())) {\n> > +\t\tconfig_.reset();\n> > +\t\treturn { Results::Fail, \"Failed to configure camera\" };\n> > +\t}\n> > +\n> > +\treturn { Results::Pass, \"Configure camera\" };\n> > +}\n> > +\n> > +Results::Result SimpleCapture::start()\n> > +{\n> > +\tStream *stream = config_->at(0).stream();\n> > +\tif (allocator_->allocate(stream) < 0)\n> > +\t\treturn { Results::Fail, \"Failed to allocate buffers camera\" };\n> \n> s/buffers camera/buffers/ ?\n> \n> > +\n> > +\tif (camera_->start())\n> > +\t\treturn { Results::Fail, \"Failed to start camera\" };\n> > +\n> > +\tcamera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);\n> > +\n> > +\treturn { Results::Pass, \"Started camera\" };\n> > +}\n> > +\n> > +Results::Result SimpleCapture::capture(unsigned int numRequests)\n> > +{\n> > +\tStream *stream = config_->at(0).stream();\n> > +\tconst std::vector> &buffers = allocator_->buffers(stream);\n> > +\n> > +\t/* No point in testing less requests then the camera depth. */\n> > +\tif (buffers.size() > numRequests)\n> > +\t\treturn { Results::Skip, \"Camera needs \" + std::to_string(buffers.size()) + \" requests, can't test only \" + std::to_string(numRequests) };\n> \n> This seems to be quite long\n\nI agree, but I really like to have the strings on one line. Makes \ngrepping the code much easier. But I will bow to popular demand :-)\n\nI have taken all your language improvements onboard for next version, \nthanks!\n\n> \n> > +\n> > +\tqueueCount_ = 0;\n> > +\tcaptureCount_ = 0;\n> > +\tcaptureLimit_ = numRequests;\n> > +\n> > +\t/* Queue the camera recommended number of reqeuests. */\n> \n> s/camera recommended/recommended/ ?\n> \n> > +\tstd::vector> requests;\n> > +\tfor (const std::unique_ptr &buffer : allocator_->buffers(stream)) {\n> > +\t\tstd::unique_ptr request = camera_->createRequest();\n> > +\t\tif (!request)\n> > +\t\t\treturn { Results::Fail, \"Can't create request\" };\n> > +\n> > +\t\tif (request->addBuffer(stream, buffer.get()))\n> > +\t\t\treturn { Results::Fail, \"Can't set buffer for request\" };\n> > +\n> > +\t\tif (queueRequest(request.get()) < 0)\n> > +\t\t\treturn { Results::Fail, \"Failed to queue request\" };\n> > +\n> > +\t\trequests.push_back(std::move(request));\n> > +\t}\n> > +\n> > +\t/* Run capture session. */\n> > +\tloop_ = new EventLoop();\n> > +\tloop_->exec();\n> > +\tdelete loop_;\n> > +\n> > +\tif (captureCount_ != captureLimit_)\n> > +\t\treturn { Results::Fail, \"Got \" + std::to_string(captureCount_) + \" request, wanted \" + std::to_string(captureLimit_) };\n> \n> This is also pretty long\n> \n> > +\n> > +\treturn { Results::Pass, \"Balanced capture of \" + std::to_string(numRequests) + \" requests\" };\n> > +}\n> > +\n> > +Results::Result SimpleCapture::stop()\n> > +{\n> > +\tStream *stream = config_->at(0).stream();\n> > +\n> > +\tcamera_->stop();\n> > +\n> > +\tcamera_->requestCompleted.disconnect(this, &SimpleCapture::requestComplete);\n> > +\n> > +\tallocator_->free(stream);\n> > +\n> > +\treturn { Results::Pass, \"Stopped camera\" };\n> > +}\n> > +\n> > +int SimpleCapture::queueRequest(Request *request)\n> > +{\n> > +\tqueueCount_++;\n> > +\tif (queueCount_ > captureLimit_)\n> > +\t\treturn 0;\n> > +\n> > +\treturn camera_->queueRequest(request);\n> > +}\n> > +\n> > +void SimpleCapture::requestComplete(Request *request)\n> > +{\n> > +\tcaptureCount_++;\n> > +\tif (captureCount_ >= captureLimit_) {\n> > +\t\tloop_->exit(0);\n> > +\t\treturn;\n> > +\t}\n> > +\n> > +\trequest->reuse(Request::ReuseBuffers);\n> > +\tif (queueRequest(request))\n> > +\t\tloop_->exit(-EINVAL);\n> > +}\n> > +\n> > +Results::Result testRequestBalance(std::shared_ptr camera,\n> > +\t\t\t\t unsigned int startCycles,\n> > +\t\t\t\t unsigned int numRequests)\n> > +{\n> > +\tSimpleCapture capture(camera);\n> > +\tResults::Result ret;\n> > +\n> > +\tret = capture.configure(Viewfinder);\n> > +\tif (ret.first != Results::Pass)\n> > +\t\treturn ret;\n> > +\n> > +\tfor (unsigned int starts = 0; starts < startCycles; starts++) {\n> > +\t\tret = capture.start();\n> > +\t\tif (ret.first != Results::Pass)\n> > +\t\t\treturn ret;\n> > +\n> > +\t\tret = capture.capture(numRequests);\n> > +\t\tif (ret.first != Results::Pass) {\n> > +\t\t\tcapture.stop();\n> > +\t\t\treturn ret;\n> > +\t\t}\n> > +\n> > +\t\tret = capture.stop();\n> > +\t\tif (ret.first != Results::Pass)\n> > +\t\t\treturn ret;\n> > +\t}\n> > +\n> > +\treturn { Results::Pass, \"Balanced capture of \" + std::to_string(numRequests) + \" requests with \" + std::to_string(startCycles) + \" start cycles\" };\n> > +}\n> > +\n> > +Results testSingleStream(std::shared_ptr camera)\n> > +{\n> > +\tconst std::vector numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };\n> > +\n> > +\tResults results(numRequests.size() * 2);\n> > +\n> > +\tif (!camera)\n> > +\t\treturn results;\n> > +\n> > +\t/*\n> > +\t * Test single capture cycles\n> > +\t *\n> > +\t * Makes sure the camera completes the exact number of requests queued.\n> > +\t * Example failure is a camera that needs N+M requests queued to\n> > +\t * complete N requests to the application.\n> > +\t */\n> > +\tstd::cout << \"Test single capture cycles\" << std::endl;\n> > +\tfor (unsigned int num : numRequests)\n> > +\t\tresults.add(testRequestBalance(camera, 1, num));\n> > +\n> > +\t/*\n> > +\t * Test multiple start/stop cycles\n> > +\t *\n> > +\t * Makes sure the camera supports multiple start/stop cycles.\n> > +\t * Example failure is a camera that does not clean up correctly in its\n> > +\t * error path but is only tested by single-capture applications.\n> > +\t */\n> > +\tstd::cout << \"Test multiple start/stop cycles\" << std::endl;\n> > +\tfor (unsigned int num : numRequests)\n> > +\t\tresults.add(testRequestBalance(camera, 3, num));\n> > +\n> > +\treturn results;\n> > +}\n> > diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> > new file mode 100644\n> > index 0000000000000000..396605214e4b8980\n> > --- /dev/null\n> > +++ b/src/lc-compliance/tests.h\n> > @@ -0,0 +1,16 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * tests.h - Test modules\n> > + */\n> > +#ifndef __LC_COMPLIANCE_TESTS_H__\n> > +#define __LC_COMPLIANCE_TESTS_H__\n> > +\n> > +#include \n> > +\n> > +#include \"results.h\"\n> > +\n> > +Results testSingleStream(std::shared_ptr camera);\n> > +\n> > +#endif /* __LC_COMPLIANCE_TESTS_H__ */\n> > diff --git a/src/meson.build b/src/meson.build\n> > index 4b75f05878bcb702..a8e1af7adf2ca9c8 100644\n> > --- a/src/meson.build\n> > +++ b/src/meson.build\n> > @@ -18,6 +18,8 @@ subdir('android')\n> > subdir('libcamera')\n> > subdir('ipa')\n> > \n> > +subdir('lc-compliance')\n> > +\n> > subdir('cam')\n> > subdir('qcam')\n> > \n> > -- \n> > 2.30.0\n> > \n> > _______________________________________________\n> > libcamera-devel mailing list\n> > libcamera-devel@lists.libcamera.org\n> > https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C9A08C33BB\n\tfor ;\n\tFri, 29 Jan 2021 21:54:25 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1E5AF683BE;\n\tFri, 29 Jan 2021 22:54:25 +0100 (CET)","from mail-lf1-x12d.google.com (mail-lf1-x12d.google.com\n\t[IPv6:2a00:1450:4864:20::12d])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 31F1D683B3\n\tfor ;\n\tFri, 29 Jan 2021 22:54:23 +0100 (CET)","by mail-lf1-x12d.google.com with SMTP id f1so14557690lfu.3\n\tfor ;\n\tFri, 29 Jan 2021 13:54:23 -0800 (PST)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tb3sm2157398lfb.191.2021.01.29.13.54.21\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 29 Jan 2021 13:54:21 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"iimP69rp\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=pKWGpCyR9HbNaPGFbrYXGhIqgQu1wGo0e0i2k5iiUf8=;\n\tb=iimP69rpFhcq46eGr5O7Xrh3fSy7pcCaY2O+HT6zjLhNwNgovTTGcXZ1cSwr5gw+PX\n\twExLMjqvcQ/qfAHrrD4cDQBu9tnozlphfNqXsgFbxA2BlC66IqMx6KJJLQ+oxoG4xLMt\n\tc7tujl5oUdhoR+Mv0cWFZ4A8CLZdtkAzMjsR2y2rBdP7vq0fxprkhiTjgZSskM8pVEoG\n\txjB42JtJV02yWfnNni1AG64FKSejz7g+C8PB7SrphQtXG7P1gPoypm5Q0C6hevj+IcEz\n\t6tWFDNPb0fNhHOUPmVqKr2JszqAeKi4zmbC0RnJJGbUOs4c3Sh0Ax4RTPLlfgl01PWDK\n\tM9Ow==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=pKWGpCyR9HbNaPGFbrYXGhIqgQu1wGo0e0i2k5iiUf8=;\n\tb=gJTi+QojVyFcXCPuSLovKIqJtx9RAiPW17aUqM4/6jNXgtxJY4btyRF2XxFQtFKIdj\n\tJxUoJ0ATg7rRVyXuuPLN8iN1fjF1+GjOcggOniLoIOjQwku8E3m1OqtPAx2c88NQ7EOo\n\tk2jysLpLs9dTU8/02u0uLzSfK37Fumg36uJnF39R1pz7vFhAOX34XxqSGmCsok2u+NzK\n\tnYQhMw9ZHljDsrv5cY3W1a2ZEYIZu2uhkiLh3YEUZIX+q6P75BiGTJIUfwUk+1i7svHb\n\tsksWTlbj7fE/rezorxUwAN57qL96HD/JcxM+G/TMV6k8AYBd/9AjTb673G3W4O0NY0yZ\n\tl8Iw==","X-Gm-Message-State":"AOAM531m4EUPtxO55gtorw/PeVl/4ZX3MdnCQ1lTF182spUV9seWTpQ1\n\tBK3XWpWNcPJALjwOlXjlK9Z8qQ==","X-Google-Smtp-Source":"ABdhPJzZE+cfbyZMDSqtKlNSvBt/s9iHT46WsXhA1jT5nTfFbQgms6MCGR3A75O2eg4P2wNRQYsPkA==","X-Received":"by 2002:a05:6512:20da:: with SMTP id\n\tu26mr3295311lfr.266.1611957262260; \n\tFri, 29 Jan 2021 13:54:22 -0800 (PST)","Date":"Fri, 29 Jan 2021 22:54:20 +0100","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Sebastian Fricke ","Message-ID":"","References":"<20210129165336.256739-1-niklas.soderlund@ragnatech.se>\n\t<20210129184856.w475ktjhb6sy5p4m@basti-TUXEDO-Book-XA1510>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20210129184856.w475ktjhb6sy5p4m@basti-TUXEDO-Book-XA1510>","Subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":14858,"web_url":"https://patchwork.libcamera.org/comment/14858/","msgid":"","date":"2021-01-29T22:44:46","subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Naushir,\n\nThanks for your feedback.\n\nOn 2021-01-29 18:05:22 +0000, Naushir Patuck wrote:\n> Hi Niklas,\n> \n> Thank you for your work, this looks quite interesting. However, I have a\n> high level question, see below.\n> \n> On Fri, 29 Jan 2021 at 16:53, Niklas Söderlund <\n> niklas.soderlund@ragnatech.se> wrote:\n> \n> > Add a compliance tool to ease testing of cameras on target. In contrast\n> > to the unit-tests under test/ that aims to test the internal components\n> > of libcamera the compliance aims to test application use-cases and to\n> > some extend the public API.\n> >\n> > This change adds the boilerplate code for a simple framework to write\n> > tests in as well as two simple test. The tests aims both to demonstrate\n> > the tool and to catch real problems. The tests added are:\n> >\n> > - Test that if one queues exactly N requests to a camera exactly N\n> > requests are eventually completed.\n> >\n> > - Test that a configured camera can be started and stopped multiple\n> > times in an attempt to exercise cleanup code paths otherwise not\n> > often tested with 'cam' for example.\n> >\n> > Example pass run on Raspberry Pi:\n> >\n> > $ LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c\n> > \"/base/soc/i2c0mux/i2c@1/imx219@10\"\n> > [2:36:17.672447527] [11896] ERROR V4L2 v4l2_device.cpp:191 'imx219\n> > 10-0010': Control 0x009a0922 not found\n> > [2:36:17.673095289] [11896] ERROR V4L2 v4l2_subdevice.cpp:285 'imx219\n> > 10-0010': Unable to get rectangle 2 on pad 0: Invalid argument\n> > Using camera /base/soc/i2c0mux/i2c@1/imx219@10\n> > Test single capture cycles\n> > - SKIP - Camera needs 4 requests, can't test only 1\n> > - SKIP - Camera needs 4 requests, can't test only 2\n> > - SKIP - Camera needs 4 requests, can't test only 3\n> >\n> \n> I'm a bit curious about the above 3 lines. From what I make out, it is\n> saying that the pipeline cannot run with < 4 on-the-go requests, is that\n> correct?\n\nYes and no, or rather I don't know :-)\n\nThe StreamConfiguration::bufferCount was added ~ 1 year age as:\n\n 4a5febd7ddd798c7 (\"libcamera: streams: extend stream configuration with buffer count\")\n\n The camera needs to be configured with the number of buffers \n required to satisfy the applications use case. While the application \n can request any number of buffers, the pipeline must take the \n constraints of the Linux driver into consideration.\n\n +/**\n + * \\var StreamConfiguration::bufferCount\n + * \\brief Requested number of buffers to allocate for the stream\n + */\n\nIIRC the idea at the time was to allow for pipelines informing \napplications a minimum number of buffers needed for the kernel driver to \n\"work\". There are V4L2 drivers that does not start the streaming before \nN buffers were queued to the video device. I'm not sure if any still \nexists but I know they have, I authored one but have since mended my \nways.\n\nThere was also a need to configure the pipeline with an expected number \nof requests that could be in-flight by an application so the pipeline \ncould allocate internal resources/buffers. Specifically for inline ISPs, \nif only one buffer was used for the sensor to memory dmaengine the frame \nrate would be rather low even if the application queued multiple \nrequests.\n\nAs we progressed and added IPAs for real at least I started to think of \nthe bufferCount as the pipeline depth needed to keep the IPA running \nsmooth. That is the number of requests that needs to be queued to the \ncamera at any given point in time to not stall the 3A.\n\nI even started to think about/design how this could be done on a \nlibcamera core level to make pipelines life easier. Then we started the \ncamera model debate which I really like and how it could address more \nthen this particular ambiguity. As an example we also have \nStreamConfiguration::stride which is a read-only property from an \napplication point of view that probably also should be reworked so \napplications to some extend could ask for a stride.\n\n> Does that determination come from the buffer count returned by\n> PipelineHandler::GenerateConfiguration()?\n\nYes.\n\n> I *think* the Raspberry Pi stack\n> should be able to handle any number of on-the-go requests. I say think,\n> because I do not normally run in that configuration, but I do know some of\n> our users might have. The buffer count of 4 returned\n> by PipelineHandlerRPi::generateConfiguration() is a somewhat educated guess\n> for a reasonable number of buffers. Is this something we need to update in\n> our pipeline handler for this to work?\n\nI would wait for the dust to settle around the camera model. Maybe I was \ntoo draconic in my interpretation and skipping testing with buffers less \nthen the bufferCount. I have tested this for RPi with 1 request and it \nworks, but I'm unsure how this is done in other pipelines so I took the \nsafest route to start out.\n\n> \n> Regards,\n> Naush\n> \n> \n> > - PASS - Balanced capture of 5 requests with 1 start cycles\n> > - PASS - Balanced capture of 8 requests with 1 start cycles\n> > - PASS - Balanced capture of 13 requests with 1 start cycles\n> > - PASS - Balanced capture of 21 requests with 1 start cycles\n> > - PASS - Balanced capture of 34 requests with 1 start cycles\n> > - PASS - Balanced capture of 55 requests with 1 start cycles\n> > - PASS - Balanced capture of 89 requests with 1 start cycles\n> > Test multiple start/stop cycles\n> > - SKIP - Camera needs 4 requests, can't test only 1\n> > - SKIP - Camera needs 4 requests, can't test only 2\n> > - SKIP - Camera needs 4 requests, can't test only 3\n> > - PASS - Balanced capture of 5 requests with 3 start cycles\n> > - PASS - Balanced capture of 8 requests with 3 start cycles\n> > - PASS - Balanced capture of 13 requests with 3 start cycles\n> > - PASS - Balanced capture of 21 requests with 3 start cycles\n> > - PASS - Balanced capture of 34 requests with 3 start cycles\n> > - PASS - Balanced capture of 55 requests with 3 start cycles\n> > - PASS - Balanced capture of 89 requests with 3 start cycles\n> > 20 tests executed, 14 tests passed, 6 tests skipped and 0 tests failed\n> >\n> > Example fail run on IPU3:\n> >\n> > - LIBCAMERA_LOG_LEVELS=\"*:ERROR\" lc-compliance -c \"\\_SB_.PCI0.I2C2.CAM0\"\n> > - [0:31:36.992763102] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov13858\n> > 8-0010': Control 0x009a0922 not found\n> > - [0:31:36.992815436] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> > 8-0010': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.992847049] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> > 8-0010': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.992869029] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> > 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993152932] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> > 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993172833] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> > 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the\n> > active area size\n> > - [0:31:36.993208149] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858\n> > 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993224502] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> > 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the\n> > active area size\n> > - [0:31:36.993354770] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov5670\n> > 10-0036': Control 0x009a0922 not found\n> > - [0:31:36.993377622] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> > 10-0036': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993398093] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> > 10-0036': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993415611] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> > 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993660192] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> > 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993678492] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> > 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the\n> > active area size\n> > - [0:31:36.993724648] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670\n> > 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device\n> > - [0:31:36.993741032] [6420] ERROR CameraSensor camera_sensor.cpp:678\n> > 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the\n> > active area size\n> > - Using camera \\_SB_.PCI0.I2C2.CAM0\n> > - Test single capture cycles\n> > - - SKIP - Camera needs 4 requests, can't test only 1\n> > - - SKIP - Camera needs 4 requests, can't test only 2\n> > - - SKIP - Camera needs 4 requests, can't test only 3\n> > - - PASS - Balanced capture of 5 requests with 1 start cycles\n> > - - PASS - Balanced capture of 8 requests with 1 start cycles\n> > - - PASS - Balanced capture of 13 requests with 1 start cycles\n> > - - PASS - Balanced capture of 21 requests with 1 start cycles\n> > - - PASS - Balanced capture of 34 requests with 1 start cycles\n> > - [0:31:41.969783243] [6420] ERROR IPU3 cio2.cpp:264 CIO2 buffer underrun\n> > > application>\n> >\n> > Signed-off-by: Niklas Söderlund \n> > ---\n> > Hi,\n> >\n> > I have tested this on RkISP1, RPi and IPU3 on latest master [1]. I plan\n> > to post separate patches which starts to address some of the problems\n> > found while working. A fix for 'cam --captur=N' is coming shortly for\n> > example.\n> >\n> > I still have to find a good fix for a races found in the\n> > IPU3 pipeline between the CIO2 and IMGU (as showed above). What I don't\n> > (yet) have a plan for is how to fix that request queueing errors are not\n> > propagated in our application facing API leading the test application to\n> > wait forever to a request the library knows have failed, ideas welcome.\n> >\n> > 1. 958c80a4f1c28301 (\"android: camera_device: Set AE precapture trigger\n> > according to request\")\n> > ---\n> > src/lc-compliance/main.cpp | 139 ++++++++++++++++++\n> > src/lc-compliance/meson.build | 24 ++++\n> > src/lc-compliance/results.cpp | 75 ++++++++++\n> > src/lc-compliance/results.h | 45 ++++++\n> > src/lc-compliance/single_stream.cpp | 210 ++++++++++++++++++++++++++++\n> > src/lc-compliance/tests.h | 16 +++\n> > src/meson.build | 2 +\n> > 7 files changed, 511 insertions(+)\n> > create mode 100644 src/lc-compliance/main.cpp\n> > create mode 100644 src/lc-compliance/meson.build\n> > create mode 100644 src/lc-compliance/results.cpp\n> > create mode 100644 src/lc-compliance/results.h\n> > create mode 100644 src/lc-compliance/single_stream.cpp\n> > create mode 100644 src/lc-compliance/tests.h\n> >\n> > diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp\n> > new file mode 100644\n> > index 0000000000000000..e1cbce7eac3df2bc\n> > --- /dev/null\n> > +++ b/src/lc-compliance/main.cpp\n> > @@ -0,0 +1,139 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * main.cpp - lc-compliance - The libcamera compliance tool\n> > + */\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \"../cam/options.h\"\n> > +#include \"tests.h\"\n> > +\n> > +using namespace libcamera;\n> > +\n> > +class Harness\n> > +{\n> > +public:\n> > + Harness();\n> > + ~Harness();\n> > +\n> > + int exec(int argc, char **argv);\n> > +\n> > +private:\n> > + enum {\n> > + OptCamera = 'c',\n> > + OptHelp = 'h',\n> > + };\n> > +\n> > + int parseOptions(int argc, char **argv);\n> > + int init(int argc, char **argv);\n> > +\n> > + OptionsParser::Options options_;\n> > + std::unique_ptr cm_;\n> > + std::shared_ptr camera_;\n> > +};\n> > +\n> > +Harness::Harness()\n> > +{\n> > + cm_ = std::make_unique();\n> > +}\n> > +\n> > +Harness::~Harness()\n> > +{\n> > + if (camera_) {\n> > + camera_->release();\n> > + camera_.reset();\n> > + }\n> > +\n> > + cm_->stop();\n> > +}\n> > +\n> > +int Harness::exec(int argc, char **argv)\n> > +{\n> > + int ret = init(argc, argv);\n> > + if (ret)\n> > + return ret;\n> > +\n> > + std::vector results;\n> > +\n> > + results.push_back(testSingleStream(camera_));\n> > +\n> > + for (const Results &result : results) {\n> > + ret = result.summary();\n> > + if (ret)\n> > + return ret;\n> > + }\n> > +\n> > + return 0;\n> > +}\n> > +\n> > +int Harness::init(int argc, char **argv)\n> > +{\n> > + int ret = parseOptions(argc, argv);\n> > + if (ret < 0)\n> > + return ret;\n> > +\n> > + ret = cm_->start();\n> > + if (ret) {\n> > + std::cout << \"Failed to start camera manager: \"\n> > + << strerror(-ret) << std::endl;\n> > + return ret;\n> > + }\n> > +\n> > + if (!options_.isSet(OptCamera)) {\n> > + std::cout << \"No camera specified, available cameras:\" <<\n> > std::endl;\n> > + for (const std::shared_ptr &cam : cm_->cameras())\n> > + std::cout << \"- \" << cam.get()->id() << std::endl;\n> > + return -ENODEV;\n> > + }\n> > +\n> > + const std::string &cameraId = options_[OptCamera];\n> > + camera_ = cm_->get(cameraId);\n> > + if (!camera_) {\n> > + std::cout << \"Camera \" << cameraId << \" not found,\n> > available cameras:\" << std::endl;\n> > + for (const std::shared_ptr &cam : cm_->cameras())\n> > + std::cout << \"- \" << cam.get()->id() << std::endl;\n> > + return -ENODEV;\n> > + }\n> > +\n> > + if (camera_->acquire()) {\n> > + std::cout << \"Failed to acquire camera\" << std::endl;\n> > + return -EINVAL;\n> > + }\n> > +\n> > + std::cout << \"Using camera \" << cameraId << std::endl;\n> > +\n> > + return 0;\n> > +}\n> > +\n> > +int Harness::parseOptions(int argc, char **argv)\n> > +{\n> > + OptionsParser parser;\n> > + parser.addOption(OptCamera, OptionString,\n> > + \"Specify which camera to operate on, by id\",\n> > \"camera\",\n> > + ArgumentRequired, \"camera\");\n> > + parser.addOption(OptHelp, OptionNone, \"Display this help message\",\n> > + \"help\");\n> > +\n> > + options_ = parser.parse(argc, argv);\n> > + if (!options_.valid())\n> > + return -EINVAL;\n> > +\n> > + if (options_.empty() || options_.isSet(OptHelp)) {\n> > + parser.usage();\n> > + return options_.empty() ? -EINVAL : -EINTR;\n> > + }\n> > +\n> > + return 0;\n> > +}\n> > +\n> > +int main(int argc, char **argv)\n> > +{\n> > + Harness harness;\n> > + return harness.exec(argc, argv) ? EXIT_FAILURE : 0;\n> > +}\n> > diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build\n> > new file mode 100644\n> > index 0000000000000000..d700a307c71405b4\n> > --- /dev/null\n> > +++ b/src/lc-compliance/meson.build\n> > @@ -0,0 +1,24 @@\n> > +# SPDX-License-Identifier: CC0-1.0\n> > +\n> > +libevent = dependency('libevent_pthreads', required : false)\n> > +\n> > +if not libevent.found()\n> > + warning('libevent_pthreads not found, \\'lc-compliance\\' application\n> > will not be compiled')\n> > + subdir_done()\n> > +endif\n> > +\n> > +lc_compliance_sources = files([\n> > + '../cam/event_loop.cpp',\n> > + '../cam/options.cpp',\n> > + 'main.cpp',\n> > + 'results.cpp',\n> > + 'single_stream.cpp',\n> > +])\n> > +\n> > +lc_compliance = executable('lc-compliance', lc_compliance_sources,\n> > + dependencies : [\n> > + libatomic,\n> > + libcamera_dep,\n> > + libevent,\n> > + ],\n> > + install : true)\n> > diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp\n> > new file mode 100644\n> > index 0000000000000000..fb4242bf49a3268b\n> > --- /dev/null\n> > +++ b/src/lc-compliance/results.cpp\n> > @@ -0,0 +1,75 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * results.cpp - Test result aggregator\n> > + */\n> > +\n> > +#include \"results.h\"\n> > +\n> > +#include \n> > +\n> > +void Results::add(const Result &result)\n> > +{\n> > + if (result.first == Pass)\n> > + passed_++;\n> > + else if (result.first == Fail)\n> > + failed_++;\n> > + else if (result.first == Skip)\n> > + skipped_++;\n> > +\n> > + printResult(result);\n> > +}\n> > +\n> > +void Results::add(Status status, const std::string &message)\n> > +{\n> > + add({ status, message });\n> > +}\n> > +\n> > +void Results::fail(const std::string &message)\n> > +{\n> > + add(Fail, message);\n> > +}\n> > +\n> > +void Results::pass(const std::string &message)\n> > +{\n> > + add(Pass, message);\n> > +}\n> > +\n> > +void Results::skip(const std::string &message)\n> > +{\n> > + add(Skip, message);\n> > +}\n> > +\n> > +int Results::summary() const\n> > +{\n> > + if (failed_ + passed_ + skipped_ != planned_) {\n> > + std::cout << \"Planned and executed numer of tests differ \"\n> > + << failed_ + passed_ + skipped_ << \" executed \"\n> > + << planned_ << \" planned\" << std::endl;\n> > +\n> > + return -EINVAL;\n> > + }\n> > +\n> > + std::cout << planned_ << \" tests executed, \"\n> > + << passed_ << \" tests passed, \"\n> > + << skipped_ << \" tests skipped and \"\n> > + << failed_ << \" tests failed \" << std::endl;\n> > +\n> > + return 0;\n> > +}\n> > +\n> > +void Results::printResult(const Result &result)\n> > +{\n> > + std::string prefix;\n> > +\n> > + /* \\todo Make parsable as TAP. */\n> > + if (result.first == Pass)\n> > + prefix = \"PASS\";\n> > + else if (result.first == Fail)\n> > + prefix = \"FAIL\";\n> > + else if (result.first == Skip)\n> > + prefix = \"SKIP\";\n> > +\n> > + std::cout << \"- \" << prefix << \" - \" << result.second << std::endl;\n> > +}\n> > diff --git a/src/lc-compliance/results.h b/src/lc-compliance/results.h\n> > new file mode 100644\n> > index 0000000000000000..a02fd5ab46edd62c\n> > --- /dev/null\n> > +++ b/src/lc-compliance/results.h\n> > @@ -0,0 +1,45 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * results.h - Test result aggregator\n> > + */\n> > +#ifndef __LC_COMPLIANCE_RESULTS_H__\n> > +#define __LC_COMPLIANCE_RESULTS_H__\n> > +\n> > +#include \n> > +\n> > +class Results\n> > +{\n> > +public:\n> > + enum Status {\n> > + Fail,\n> > + Pass,\n> > + Skip,\n> > + };\n> > +\n> > + using Result = std::pair;\n> > +\n> > + Results(unsigned int planned)\n> > + : planned_(planned), passed_(0), failed_(0), skipped_(0)\n> > + {\n> > + }\n> > +\n> > + void add(const Result &result);\n> > + void add(Status status, const std::string &message);\n> > + void fail(const std::string &message);\n> > + void pass(const std::string &message);\n> > + void skip(const std::string &message);\n> > +\n> > + int summary() const;\n> > +\n> > +private:\n> > + void printResult(const Result &result);\n> > +\n> > + unsigned int planned_;\n> > + unsigned int passed_;\n> > + unsigned int failed_;\n> > + unsigned int skipped_;\n> > +};\n> > +\n> > +#endif /* __LC_COMPLIANCE_RESULTS_H__ */\n> > diff --git a/src/lc-compliance/single_stream.cpp\n> > b/src/lc-compliance/single_stream.cpp\n> > new file mode 100644\n> > index 0000000000000000..d31b6f8d06487a85\n> > --- /dev/null\n> > +++ b/src/lc-compliance/single_stream.cpp\n> > @@ -0,0 +1,210 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * single_stream.cpp - Test a single camera stream\n> > + */\n> > +\n> > +#include \n> > +\n> > +#include \"../cam/event_loop.h\"\n> > +#include \"tests.h\"\n> > +\n> > +using namespace libcamera;\n> > +\n> > +class SimpleCapture\n> > +{\n> > +public:\n> > + SimpleCapture(std::shared_ptr camera)\n> > + : camera_(camera),\n> > allocator_(std::make_unique(camera))\n> > + {\n> > + }\n> > +\n> > + Results::Result configure(StreamRole role);\n> > + Results::Result start();\n> > + Results::Result capture(unsigned int numRequests);\n> > + Results::Result stop();\n> > +\n> > +private:\n> > + int queueRequest(Request *request);\n> > + void requestComplete(Request *request);\n> > +\n> > + std::shared_ptr camera_;\n> > + std::unique_ptr allocator_;\n> > + std::unique_ptr config_;\n> > +\n> > + EventLoop *loop_;\n> > + unsigned int queueCount_;\n> > + unsigned int captureCount_;\n> > + unsigned int captureLimit_;\n> > +};\n> > +\n> > +Results::Result SimpleCapture::configure(StreamRole role)\n> > +{\n> > + config_ = camera_->generateConfiguration({ role });\n> > +\n> > + if (config_->validate() != CameraConfiguration::Valid) {\n> > + config_.reset();\n> > + return { Results::Fail, \"Configuration not valid\" };\n> > + }\n> > +\n> > + if (camera_->configure(config_.get())) {\n> > + config_.reset();\n> > + return { Results::Fail, \"Failed to configure camera\" };\n> > + }\n> > +\n> > + return { Results::Pass, \"Configure camera\" };\n> > +}\n> > +\n> > +Results::Result SimpleCapture::start()\n> > +{\n> > + Stream *stream = config_->at(0).stream();\n> > + if (allocator_->allocate(stream) < 0)\n> > + return { Results::Fail, \"Failed to allocate buffers\n> > camera\" };\n> > +\n> > + if (camera_->start())\n> > + return { Results::Fail, \"Failed to start camera\" };\n> > +\n> > + camera_->requestCompleted.connect(this,\n> > &SimpleCapture::requestComplete);\n> > +\n> > + return { Results::Pass, \"Started camera\" };\n> > +}\n> > +\n> > +Results::Result SimpleCapture::capture(unsigned int numRequests)\n> > +{\n> > + Stream *stream = config_->at(0).stream();\n> > + const std::vector> &buffers =\n> > allocator_->buffers(stream);\n> > +\n> > + /* No point in testing less requests then the camera depth. */\n> > + if (buffers.size() > numRequests)\n> > + return { Results::Skip, \"Camera needs \" +\n> > std::to_string(buffers.size()) + \" requests, can't test only \" +\n> > std::to_string(numRequests) };\n> > +\n> > + queueCount_ = 0;\n> > + captureCount_ = 0;\n> > + captureLimit_ = numRequests;\n> > +\n> > + /* Queue the camera recommended number of reqeuests. */\n> > + std::vector> requests;\n> > + for (const std::unique_ptr &buffer :\n> > allocator_->buffers(stream)) {\n> > + std::unique_ptr request =\n> > camera_->createRequest();\n> > + if (!request)\n> > + return { Results::Fail, \"Can't create request\" };\n> > +\n> > + if (request->addBuffer(stream, buffer.get()))\n> > + return { Results::Fail, \"Can't set buffer for\n> > request\" };\n> > +\n> > + if (queueRequest(request.get()) < 0)\n> > + return { Results::Fail, \"Failed to queue request\"\n> > };\n> > +\n> > + requests.push_back(std::move(request));\n> > + }\n> > +\n> > + /* Run capture session. */\n> > + loop_ = new EventLoop();\n> > + loop_->exec();\n> > + delete loop_;\n> > +\n> > + if (captureCount_ != captureLimit_)\n> > + return { Results::Fail, \"Got \" +\n> > std::to_string(captureCount_) + \" request, wanted \" +\n> > std::to_string(captureLimit_) };\n> > +\n> > + return { Results::Pass, \"Balanced capture of \" +\n> > std::to_string(numRequests) + \" requests\" };\n> > +}\n> > +\n> > +Results::Result SimpleCapture::stop()\n> > +{\n> > + Stream *stream = config_->at(0).stream();\n> > +\n> > + camera_->stop();\n> > +\n> > + camera_->requestCompleted.disconnect(this,\n> > &SimpleCapture::requestComplete);\n> > +\n> > + allocator_->free(stream);\n> > +\n> > + return { Results::Pass, \"Stopped camera\" };\n> > +}\n> > +\n> > +int SimpleCapture::queueRequest(Request *request)\n> > +{\n> > + queueCount_++;\n> > + if (queueCount_ > captureLimit_)\n> > + return 0;\n> > +\n> > + return camera_->queueRequest(request);\n> > +}\n> > +\n> > +void SimpleCapture::requestComplete(Request *request)\n> > +{\n> > + captureCount_++;\n> > + if (captureCount_ >= captureLimit_) {\n> > + loop_->exit(0);\n> > + return;\n> > + }\n> > +\n> > + request->reuse(Request::ReuseBuffers);\n> > + if (queueRequest(request))\n> > + loop_->exit(-EINVAL);\n> > +}\n> > +\n> > +Results::Result testRequestBalance(std::shared_ptr camera,\n> > + unsigned int startCycles,\n> > + unsigned int numRequests)\n> > +{\n> > + SimpleCapture capture(camera);\n> > + Results::Result ret;\n> > +\n> > + ret = capture.configure(Viewfinder);\n> > + if (ret.first != Results::Pass)\n> > + return ret;\n> > +\n> > + for (unsigned int starts = 0; starts < startCycles; starts++) {\n> > + ret = capture.start();\n> > + if (ret.first != Results::Pass)\n> > + return ret;\n> > +\n> > + ret = capture.capture(numRequests);\n> > + if (ret.first != Results::Pass) {\n> > + capture.stop();\n> > + return ret;\n> > + }\n> > +\n> > + ret = capture.stop();\n> > + if (ret.first != Results::Pass)\n> > + return ret;\n> > + }\n> > +\n> > + return { Results::Pass, \"Balanced capture of \" +\n> > std::to_string(numRequests) + \" requests with \" +\n> > std::to_string(startCycles) + \" start cycles\" };\n> > +}\n> > +\n> > +Results testSingleStream(std::shared_ptr camera)\n> > +{\n> > + const std::vector numRequests = { 1, 2, 3, 5, 8, 13,\n> > 21, 34, 55, 89 };\n> > +\n> > + Results results(numRequests.size() * 2);\n> > +\n> > + if (!camera)\n> > + return results;\n> > +\n> > + /*\n> > + * Test single capture cycles\n> > + *\n> > + * Makes sure the camera completes the exact number of requests\n> > queued.\n> > + * Example failure is a camera that needs N+M requests queued to\n> > + * complete N requests to the application.\n> > + */\n> > + std::cout << \"Test single capture cycles\" << std::endl;\n> > + for (unsigned int num : numRequests)\n> > + results.add(testRequestBalance(camera, 1, num));\n> > +\n> > + /*\n> > + * Test multiple start/stop cycles\n> > + *\n> > + * Makes sure the camera supports multiple start/stop cycles.\n> > + * Example failure is a camera that does not clean up correctly in\n> > its\n> > + * error path but is only tested by single-capture applications.\n> > + */\n> > + std::cout << \"Test multiple start/stop cycles\" << std::endl;\n> > + for (unsigned int num : numRequests)\n> > + results.add(testRequestBalance(camera, 3, num));\n> > +\n> > + return results;\n> > +}\n> > diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h\n> > new file mode 100644\n> > index 0000000000000000..396605214e4b8980\n> > --- /dev/null\n> > +++ b/src/lc-compliance/tests.h\n> > @@ -0,0 +1,16 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2020, Google Inc.\n> > + *\n> > + * tests.h - Test modules\n> > + */\n> > +#ifndef __LC_COMPLIANCE_TESTS_H__\n> > +#define __LC_COMPLIANCE_TESTS_H__\n> > +\n> > +#include \n> > +\n> > +#include \"results.h\"\n> > +\n> > +Results testSingleStream(std::shared_ptr camera);\n> > +\n> > +#endif /* __LC_COMPLIANCE_TESTS_H__ */\n> > diff --git a/src/meson.build b/src/meson.build\n> > index 4b75f05878bcb702..a8e1af7adf2ca9c8 100644\n> > --- a/src/meson.build\n> > +++ b/src/meson.build\n> > @@ -18,6 +18,8 @@ subdir('android')\n> > subdir('libcamera')\n> > subdir('ipa')\n> >\n> > +subdir('lc-compliance')\n> > +\n> > subdir('cam')\n> > subdir('qcam')\n> >\n> > --\n> > 2.30.0\n> >\n> > _______________________________________________\n> > libcamera-devel mailing list\n> > libcamera-devel@lists.libcamera.org\n> > https://lists.libcamera.org/listinfo/libcamera-devel\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 587D0BD808\n\tfor ;\n\tFri, 29 Jan 2021 22:44:51 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AA693683C0;\n\tFri, 29 Jan 2021 23:44:50 +0100 (CET)","from mail-lj1-x233.google.com (mail-lj1-x233.google.com\n\t[IPv6:2a00:1450:4864:20::233])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4DE45683B3\n\tfor ;\n\tFri, 29 Jan 2021 23:44:49 +0100 (CET)","by mail-lj1-x233.google.com with SMTP id b20so4452209ljo.1\n\tfor ;\n\tFri, 29 Jan 2021 14:44:49 -0800 (PST)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tm78sm2185059lfa.270.2021.01.29.14.44.47\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 29 Jan 2021 14:44:47 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"eZ4XtLBN\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=zdAcyVhL/+mMnatBLxAsMCRlDXOpmFnxiULOgSITYDw=;\n\tb=eZ4XtLBNDx6HUTCcayYgtsOXID59HDpIP+FnohhyvwbdZ8I6FTmmj1Tp23QIffgSz7\n\tko4nU/Nmndjp1QR25nIVhELT8W0v2tS0u0XUOflO9Psj97kxaBMZi5lDkpYdYhPXaZ3Q\n\tykhnJY32/B+qmgapEL6hxkGOv/qF0mf/5B05jNpvgOA2o8FqRFZ8dwIJ43CGwuMEx/TC\n\tIRQ2HVJ7LXddLtJ+v+bXWbDQgRjDGydKmhEYTIcXRFfvBFAbE+2yngEgPLAm2Ly7pG+2\n\tohSGWbrv2ywada0zM9Vt6ao1SUsWCPFKFKMndcsjyIdT5n6ZXFyE7udzVzE+tFV+aZWb\n\tTx+w==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=zdAcyVhL/+mMnatBLxAsMCRlDXOpmFnxiULOgSITYDw=;\n\tb=AcUb8Ud8ZxiAdRFpZ4f7VH8kdxe5FULMlkoS+HVPioWnx7RkW7X0xREUWTHxHwhv5z\n\tf+X3rdhJPWaNqReokBuDG2P/g/jqFcNi5tZYZsiq71J6MnH2zlCWAFG1jfrpC6FrZPro\n\tviBRPqAsmYF6oe43VJLTHimV4yIHI/3NB/T72W0T37aSIwVwu/aV1EKnD4qSd4K9hWr6\n\tS3hmHtm+lhDjoPpFCsRfgHYmMXgn3g5GF517V8+jHLAgW80x80g5PRIQf9K3Sv+nOiv7\n\t+3lPMqTs6sykZl3jA9Uy9vLzoyD2C50fzI/yooalZiEpk/zzvG6lIuLXuyLturUe/MHu\n\tqkHA==","X-Gm-Message-State":"AOAM533rni32xDaYFyRlgzxDdBv/lcXmI/ReWJ7/D1n/ixeznIS+96KU\n\tkDLvi8SF0D1bxuuVJOXitkIspn95vrxvVA==","X-Google-Smtp-Source":"ABdhPJzYgN5kGEjrzB/lmaxcHwnB/H7uG5QbA0Mcp+DsfJxC6IwP8FtzvRMBBGkshRO9W585P3bdEg==","X-Received":"by 2002:a05:651c:1107:: with SMTP id\n\td7mr3531594ljo.10.1611960288454; \n\tFri, 29 Jan 2021 14:44:48 -0800 (PST)","Date":"Fri, 29 Jan 2021 23:44:46 +0100","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Naushir Patuck ","Message-ID":"","References":"<20210129165336.256739-1-niklas.soderlund@ragnatech.se>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH] lc-compliance: Add a libcamera\n\tcompliance tool","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]