{"id":19743,"url":"https://patchwork.libcamera.org/api/patches/19743/?format=json","web_url":"https://patchwork.libcamera.org/patch/19743/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20240319120517.362082-4-stefan.klug@ideasonboard.com>","date":"2024-03-19T12:05:04","name":"[v3,03/16] libcamera: lc-compliance: Add initial set of per-frame-control tests","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"c864c7112cff6dac48f2bb4130c718772ea4b816","submitter":{"id":184,"url":"https://patchwork.libcamera.org/api/people/184/?format=json","name":"Stefan Klug","email":"stefan.klug@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/19743/mbox/","series":[{"id":4230,"url":"https://patchwork.libcamera.org/api/series/4230/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4230","date":"2024-03-19T12:05:01","name":"Preparation for per-frame-controls and initial tests","version":3,"mbox":"https://patchwork.libcamera.org/series/4230/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/19743/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/19743/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C9A62BD160\n\tfor ;\n\tTue, 19 Mar 2024 12:05:37 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EC06662CAB;\n\tTue, 19 Mar 2024 13:05:31 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 20A6962973\n\tfor ;\n\tTue, 19 Mar 2024 13:05:28 +0100 (CET)","from jasper.fritz.box (unknown\n\t[IPv6:2a00:6020:448c:6c00:1478:344b:8fcb:baf5])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 6F44B480;\n\tTue, 19 Mar 2024 13:05:01 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"k4Xvmih+\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1710849901;\n\tbh=YWcx60Xes0BHpk2TegmXEa7+lGhzQhQJfS3TcDAV8iI=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=k4Xvmih+w1jxLZBhr7oQwpNK1S3UfkuD236IJMWPnuBQrerhUebHDnpKfVMyCAjIK\n\t5ZBrfZXxDXu7qkrtMaXNw67tNY0qirT5gRv4f314Uq0m47A08rKSUhm0tDRQ6e6VPj\n\t/GJNhURw5PneCB/1ptk+ZSXGpmm+iNCEZva/X2Mg=","From":"Stefan Klug ","To":"libcamera-devel@lists.libcamera.org","Cc":"Stefan Klug ","Subject":"[PATCH v3 03/16] libcamera: lc-compliance: Add initial set of\n\tper-frame-control tests","Date":"Tue, 19 Mar 2024 13:05:04 +0100","Message-Id":"<20240319120517.362082-4-stefan.klug@ideasonboard.com>","X-Mailer":"git-send-email 2.40.1","In-Reply-To":"<20240319120517.362082-1-stefan.klug@ideasonboard.com>","References":"<20240319120517.362082-1-stefan.klug@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Add tests that check if controls (only exposure time and analogue gain at the\nmoment) get applied on the frame they were requested for. This is tested by\nlooking at the metadata and the mean brightness of the image center. At the\nmoment these tests fail. Fixes for the pipelines will be delivered in later\npatches (rkisp1 for now).\n\nTo run the pfc tests only:\nlc-compliance -c -f \"PerFrameControlTests.*\"\n\nNote that the current implementation is a bit picky on what the camera\nactually sees. If it is too dark (or too bright), the tests will fail.\nLooking at a white wall in a normally lit office usually works.\n\nThese tests are known to pass using a imx219 (RPi cam v2) with a imx8mp\n(debix-som) using the rkisp1 pipeline\n\nSigned-off-by: Stefan Klug \n---\n src/apps/lc-compliance/meson.build | 1 +\n .../lc-compliance/per_frame_controls_test.cpp | 398 ++++++++++++++++++\n 2 files changed, 399 insertions(+)\n create mode 100644 src/apps/lc-compliance/per_frame_controls_test.cpp","diff":"diff --git a/src/apps/lc-compliance/meson.build b/src/apps/lc-compliance/meson.build\nindex eb7b2d71..8f4eec55 100644\n--- a/src/apps/lc-compliance/meson.build\n+++ b/src/apps/lc-compliance/meson.build\n@@ -15,6 +15,7 @@ lc_compliance_sources = files([\n 'capture_test.cpp',\n 'environment.cpp',\n 'main.cpp',\n+ 'per_frame_controls_test.cpp',\n 'simple_capture.cpp',\n 'time_sheet.cpp',\n ])\ndiff --git a/src/apps/lc-compliance/per_frame_controls_test.cpp b/src/apps/lc-compliance/per_frame_controls_test.cpp\nnew file mode 100644\nindex 00000000..017e8d60\n--- /dev/null\n+++ b/src/apps/lc-compliance/per_frame_controls_test.cpp\n@@ -0,0 +1,398 @@\n+/* SPDX-License-Identifier: GPL-2.0-or-later */\n+/*\n+ * Copyright (C) 2024, Ideas on Board Oy\n+ *\n+ * per_frame_controls.cpp - Tests for per frame controls\n+ */\n+#include \n+\n+#include \"environment.h\"\n+#include \"simple_capture.h\"\n+#include \"time_sheet.h\"\n+\n+using namespace libcamera;\n+\n+class PerFrameControlTests : public testing::Test\n+{\n+protected:\n+\tvoid SetUp() override;\n+\tvoid TearDown() override;\n+\n+\tstd::shared_ptr camera_;\n+};\n+\n+/*\n+ * We use gtest's SetUp() and TearDown() instead of constructor and destructor\n+ * in order to be able to assert on them.\n+ */\n+void PerFrameControlTests::SetUp()\n+{\n+\tEnvironment *env = Environment::get();\n+\n+\tcamera_ = env->cm()->get(env->cameraId());\n+\n+\tASSERT_EQ(camera_->acquire(), 0);\n+}\n+\n+void PerFrameControlTests::TearDown()\n+{\n+\tif (!camera_)\n+\t\treturn;\n+\n+\tcamera_->release();\n+\tcamera_.reset();\n+}\n+\n+class PerFrameControlsCapture : public SimpleCapture\n+{\n+public:\n+\tPerFrameControlsCapture(std::shared_ptr camera);\n+\n+\tstd::shared_ptr\n+\tstartCaptureWithTimeSheet(unsigned int framesToCapture,\n+\t\t\t\t const libcamera::ControlList *controls = nullptr);\n+\n+\tvoid runCaptureSession();\n+\tint queueRequest(libcamera::Request *request);\n+\tvoid requestComplete(libcamera::Request *request) override;\n+\n+\tunsigned int queueCount_;\n+\tunsigned int captureCount_;\n+\tunsigned int captureLimit_;\n+\n+\tstd::weak_ptr timeSheet_;\n+};\n+\n+static const bool doImageTests = true;\n+\n+PerFrameControlsCapture::PerFrameControlsCapture(std::shared_ptr camera)\n+\t: SimpleCapture(camera)\n+{\n+}\n+\n+std::shared_ptr\n+PerFrameControlsCapture::startCaptureWithTimeSheet(unsigned int framesToCapture,\n+\t\t\t\t\t\t const ControlList *controls)\n+{\n+\tControlList ctrls(camera_->controls().idmap());\n+\n+\t/* Ensure defined default values */\n+\tctrls.set(controls::AeEnable, false);\n+\tctrls.set(controls::AeExposureMode, controls::ExposureCustom);\n+\tctrls.set(controls::ExposureTime, 10000);\n+\tctrls.set(controls::AnalogueGain, 1.0);\n+\n+\tif (controls)\n+\t\tctrls.merge(*controls, ControlList::MergePolicy::OverwriteExisting);\n+\n+\tstart(&ctrls);\n+\n+\tqueueCount_ = 0;\n+\tcaptureCount_ = 0;\n+\tcaptureLimit_ = framesToCapture;\n+\n+\tauto timeSheet = std::make_shared(captureLimit_,\n+\t\t\t\t\t\t camera_->controls().idmap());\n+\ttimeSheet_ = timeSheet;\n+\treturn timeSheet;\n+}\n+\n+int PerFrameControlsCapture::queueRequest(Request *request)\n+{\n+\tqueueCount_++;\n+\tif (queueCount_ > captureLimit_)\n+\t\treturn 0;\n+\n+\tauto ts = timeSheet_.lock();\n+\tif (ts)\n+\t\tts->prepareForQueue(request, queueCount_ - 1);\n+\n+\treturn camera_->queueRequest(request);\n+}\n+\n+void PerFrameControlsCapture::requestComplete(Request *request)\n+{\n+\tauto ts = timeSheet_.lock();\n+\tif (ts)\n+\t\tts->handleCompleteRequest(request);\n+\n+\tcaptureCount_++;\n+\tif (captureCount_ >= captureLimit_) {\n+\t\tloop_->exit(0);\n+\t\treturn;\n+\t}\n+\n+\trequest->reuse(Request::ReuseBuffers);\n+\tif (queueRequest(request))\n+\t\tloop_->exit(-EINVAL);\n+}\n+\n+void PerFrameControlsCapture::runCaptureSession()\n+{\n+\tStream *stream = config_->at(0).stream();\n+\tconst std::vector> &buffers = allocator_->buffers(stream);\n+\n+\t/* Queue the recommended number of requests. */\n+\tfor (const std::unique_ptr &buffer : buffers) {\n+\t\tstd::unique_ptr request = camera_->createRequest();\n+\t\trequest->addBuffer(stream, buffer.get());\n+\t\tqueueRequest(request.get());\n+\t\trequests_.push_back(std::move(request));\n+\t}\n+\n+\t/* Run capture session. */\n+\tloop_ = new EventLoop();\n+\tloop_->exec();\n+\tstop();\n+\tdelete loop_;\n+}\n+\n+TEST_F(PerFrameControlTests, testExposureGainChangeOnSameFrame)\n+{\n+\tPerFrameControlsCapture capture(camera_);\n+\tcapture.configure(StreamRole::VideoRecording);\n+\n+\tControlList startValues;\n+\tstartValues.set(controls::ExposureTime, 5000);\n+\tstartValues.set(controls::AnalogueGain, 1.0);\n+\n+\tauto timeSheet = capture.startCaptureWithTimeSheet(10, &startValues);\n+\tauto &ts = *timeSheet;\n+\n+\t/* wait a few frames to settle */\n+\tts[7].controls().set(controls::ExposureTime, 10000);\n+\tts[7].controls().set(controls::AnalogueGain, 4.0);\n+\n+\tcapture.runCaptureSession();\n+\n+\tASSERT_TRUE(ts[5].metadata().contains(controls::ExposureTime.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\tASSERT_TRUE(ts[5].metadata().contains(controls::AnalogueGain.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\n+\tEXPECT_NEAR(ts[3].metadata().get(controls::ExposureTime).value(), 5000, 20);\n+\tEXPECT_NEAR(ts[3].metadata().get(controls::AnalogueGain).value(), 1.0, 0.05);\n+\n+\t/* find the frame with the changes */\n+\tint exposureChangeIndex = 0;\n+\tfor (unsigned i = 3; i < ts.size(); i++) {\n+\t\tif (ts[i].metadata().get(controls::ExposureTime).value() > 7500) {\n+\t\t\texposureChangeIndex = i;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tint gainChangeIndex = 0;\n+\tfor (unsigned i = 3; i < ts.size(); i++) {\n+\t\tif (ts[i].metadata().get(controls::AnalogueGain).value() > 2.0) {\n+\t\t\tgainChangeIndex = i;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tEXPECT_NE(exposureChangeIndex, 0) << \"Exposure change not found in metadata\";\n+\tEXPECT_NE(gainChangeIndex, 0) << \"Gain change not found in metadata\";\n+\tEXPECT_EQ(exposureChangeIndex, gainChangeIndex)\n+\t\t<< \"Metadata contained gain and exposure changes on different frames\";\n+\n+\tif (doImageTests) {\n+\t\tint brightnessChangeIndex = 0;\n+\t\tfor (unsigned i = 3; i < ts.size(); i++) {\n+\t\t\tif (ts[i].brightnessChange() > 1.3) {\n+\t\t\t\tEXPECT_EQ(brightnessChangeIndex, 0)\n+\t\t\t\t\t<< \"Detected multiple frames with brightness increase\"\n+\t\t\t\t\t<< \" (Wrong control delays?)\";\n+\n+\t\t\t\tif (!brightnessChangeIndex)\n+\t\t\t\t\tbrightnessChangeIndex = i;\n+\t\t\t}\n+\t\t}\n+\n+\t\tEXPECT_EQ(exposureChangeIndex, brightnessChangeIndex)\n+\t\t\t<< \"Exposure change and measured brightness change were not on same\"\n+\t\t\t<< \" frame. (Wrong control delay?, Start frame event too late?)\";\n+\t\tEXPECT_EQ(exposureChangeIndex, gainChangeIndex)\n+\t\t\t<< \"Gain change and measured brightness change were not on same \"\n+\t\t\t<< \" frame. (Wrong control delay?, Start frame event too late?)\";\n+\t}\n+}\n+\n+TEST_F(PerFrameControlTests, testFramePreciseExposureChange)\n+{\n+\tPerFrameControlsCapture capture(camera_);\n+\tcapture.configure(StreamRole::VideoRecording);\n+\n+\tauto timeSheet = capture.startCaptureWithTimeSheet(10);\n+\tauto &ts = *timeSheet;\n+\n+\tts[3].controls().set(controls::ExposureTime, 5000);\n+\t/* wait a few frames to settle */\n+\tts[6].controls().set(controls::ExposureTime, 20000);\n+\n+\tcapture.runCaptureSession();\n+\n+\tASSERT_TRUE(ts[5].metadata().contains(controls::ExposureTime.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\n+\tEXPECT_NEAR(ts[5].metadata().get(controls::ExposureTime).value(), 5000, 20);\n+\tEXPECT_NEAR(ts[6].metadata().get(controls::ExposureTime).value(), 20000, 20);\n+\n+\tif (doImageTests) {\n+\t\t/* No increase just before setting exposure */\n+\t\tEXPECT_NEAR(ts[5].brightnessChange(), 1.0, 0.05)\n+\t\t\t<< \"Brightness changed too much before the expected time of change\"\n+\t\t\t<< \" (control delay too high?).\";\n+\t\t/*\n+\t\t * \\todo The change is brightness was a bit low\n+\t\t * (Exposure time increase by 4x resulted in a brightness increase of < 2).\n+\t\t * This should be investigated.\n+\t\t */\n+\t\tEXPECT_GT(ts[6].brightnessChange(), 1.3)\n+\t\t\t<< \"Brightness in frame \" << 6 << \" did not increase as expected\"\n+\t\t\t<< \" (reference: \" << ts[3].spotBrightness() << \" current: \"\n+\t\t\t<< ts[6].spotBrightness() << \" )\" << std::endl;\n+\n+\t\t/* No increase just after setting exposure */\n+\t\tEXPECT_NEAR(ts[7].brightnessChange(), 1.0, 0.05)\n+\t\t\t<< \"Brightness changed too much after the expected time of change\"\n+\t\t\t<< \" (control delay too low?).\";\n+\n+\t\t/* No increase just after setting exposure */\n+\t\tEXPECT_NEAR(ts[8].brightnessChange(), 1.0, 0.05)\n+\t\t\t<< \"Brightness changed too much 2 frames after the expected time\"\n+\t\t\t<< \" of change (control delay too low?).\";\n+\t}\n+}\n+\n+TEST_F(PerFrameControlTests, testFramePreciseGainChange)\n+{\n+\tPerFrameControlsCapture capture(camera_);\n+\tcapture.configure(StreamRole::VideoRecording);\n+\n+\tauto timeSheet = capture.startCaptureWithTimeSheet(10);\n+\tauto &ts = *timeSheet;\n+\n+\tts[3].controls().set(controls::AnalogueGain, 1.0);\n+\t/* wait a few frames to settle */\n+\tts[6].controls().set(controls::AnalogueGain, 4.0);\n+\n+\tcapture.runCaptureSession();\n+\n+\tASSERT_TRUE(ts[5].metadata().contains(controls::AnalogueGain.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\n+\tEXPECT_NEAR(ts[5].metadata().get(controls::AnalogueGain).value(), 1.0, 0.1);\n+\tEXPECT_NEAR(ts[6].metadata().get(controls::AnalogueGain).value(), 4.0, 0.1);\n+\n+\tif (doImageTests) {\n+\t\t/* No increase just before setting gain */\n+\t\tEXPECT_NEAR(ts[5].brightnessChange(), 1.0, 0.05)\n+\t\t\t<< \"Brightness changed too much before the expected time of change\"\n+\t\t\t<< \" (control delay too high?).\";\n+\t\t/*\n+\t\t * \\todo I see a brightness change of roughly half the expected one.\n+\t\t * This is not yet understood and needs investigation\n+\t\t */\n+\t\tEXPECT_GT(ts[6].brightnessChange(), 1.7)\n+\t\t\t<< \"Brightness in frame \" << 6 << \" did not increase as expected\"\n+\t\t\t<< \" (reference: \" << ts[5].spotBrightness()\n+\t\t\t<< \" current: \" << ts[6].spotBrightness() << \")\" << std::endl;\n+\n+\t\t/* No increase just after setting gain */\n+\t\tEXPECT_NEAR(ts[7].brightnessChange(), 1.0, 0.05)\n+\t\t\t<< \"Brightness changed too much after the expected time of change\"\n+\t\t\t<< \" (control delay too low?).\";\n+\n+\t\t/* No increase just after setting gain */\n+\t\tEXPECT_NEAR(ts[8].brightnessChange(), 1.0, 0.05)\n+\t\t\t<< \"Brightness changed too much after the expected time of change\"\n+\t\t\t<< \" (control delay too low?).\";\n+\t}\n+}\n+\n+TEST_F(PerFrameControlTests, testExposureGainFromFirstRequestGetsApplied)\n+{\n+\tPerFrameControlsCapture capture(camera_);\n+\tcapture.configure(StreamRole::VideoRecording);\n+\n+\tauto timeSheet = capture.startCaptureWithTimeSheet(5);\n+\tauto &ts = *timeSheet;\n+\n+\tts[0].controls().set(controls::ExposureTime, 10000);\n+\tts[0].controls().set(controls::AnalogueGain, 4.0);\n+\n+\tcapture.runCaptureSession();\n+\n+\tASSERT_TRUE(ts[4].metadata().contains(controls::ExposureTime.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\tASSERT_TRUE(ts[4].metadata().contains(controls::AnalogueGain.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\n+\t/* We expect it to be applied after 3 frames, the latest*/\n+\tEXPECT_NEAR(ts[4].metadata().get(controls::ExposureTime).value(), 10000, 20);\n+\tEXPECT_NEAR(ts[4].metadata().get(controls::AnalogueGain).value(), 4.0, 0.1);\n+}\n+\n+TEST_F(PerFrameControlTests, testExposureGainFromFirstAndSecondRequestGetsApplied)\n+{\n+\tPerFrameControlsCapture capture(camera_);\n+\tcapture.configure(StreamRole::VideoRecording);\n+\n+\tauto timeSheet = capture.startCaptureWithTimeSheet(5);\n+\tauto &ts = *timeSheet;\n+\n+\tts[0].controls().set(controls::ExposureTime, 8000);\n+\tts[0].controls().set(controls::AnalogueGain, 2.0);\n+\tts[1].controls().set(controls::ExposureTime, 10000);\n+\tts[1].controls().set(controls::AnalogueGain, 4.0);\n+\n+\tcapture.runCaptureSession();\n+\n+\tASSERT_TRUE(ts[4].metadata().contains(controls::ExposureTime.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\tASSERT_TRUE(ts[4].metadata().contains(controls::AnalogueGain.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\n+\t/* We expect it to be applied after 3 frames, the latest */\n+\tEXPECT_NEAR(ts[4].metadata().get(controls::ExposureTime).value(), 10000, 20);\n+\tEXPECT_NEAR(ts[4].metadata().get(controls::AnalogueGain).value(), 4.0, 0.1);\n+}\n+\n+TEST_F(PerFrameControlTests, testExposureGainIsAppliedOnFirstFrame)\n+{\n+\tPerFrameControlsCapture capture(camera_);\n+\tcapture.configure(StreamRole::VideoRecording);\n+\n+\tControlList startValues;\n+\tstartValues.set(controls::ExposureTime, 5000);\n+\tstartValues.set(controls::AnalogueGain, 1.0);\n+\n+\tauto ts1 = capture.startCaptureWithTimeSheet(3, &startValues);\n+\n+\tcapture.runCaptureSession();\n+\n+\tASSERT_TRUE((*ts1)[0].metadata().contains(controls::ExposureTime.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\tASSERT_TRUE((*ts1)[0].metadata().contains(controls::AnalogueGain.id()))\n+\t\t<< \"Required metadata entry is missing\";\n+\n+\tEXPECT_NEAR((*ts1)[0].metadata().get(controls::ExposureTime).value(), 5000, 20);\n+\tEXPECT_NEAR((*ts1)[0].metadata().get(controls::AnalogueGain).value(), 1.0, 0.02);\n+\n+\t/* Second capture with different values to ensure we don't hit default/old values */\n+\tstartValues.set(controls::ExposureTime, 15000);\n+\tstartValues.set(controls::AnalogueGain, 4.0);\n+\n+\tauto ts2 = capture.startCaptureWithTimeSheet(3, &startValues);\n+\n+\tcapture.runCaptureSession();\n+\n+\tEXPECT_NEAR((*ts2)[0].metadata().get(controls::ExposureTime).value(), 15000, 20);\n+\tEXPECT_NEAR((*ts2)[0].metadata().get(controls::AnalogueGain).value(), 4.0, 0.02);\n+\n+\tif (doImageTests) {\n+\t\t/* With 3x exposure and 4x gain we could expect a brightness increase of 2x */\n+\t\tdouble brightnessChange = ts2->get(1).spotBrightness() / ts1->get(1).spotBrightness();\n+\t\tEXPECT_GT(brightnessChange, 2.0);\n+\t}\n+}\n","prefixes":["v3","03/16"]}