{"id":11337,"url":"https://patchwork.libcamera.org/api/1.1/patches/11337/?format=json","web_url":"https://patchwork.libcamera.org/patch/11337/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210218170126.2060783-2-naush@raspberrypi.com>","date":"2021-02-18T17:01:23","name":"[libcamera-devel,v3,1/4] pipeline: ipa: raspberrypi: Pass exposure/gain values to IPA though controls","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"62ce97d72b787eab6470ab4c6f6a61d705421267","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/1.1/people/34/?format=json","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/11337/mbox/","series":[{"id":1708,"url":"https://patchwork.libcamera.org/api/1.1/series/1708/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1708","date":"2021-02-18T17:01:22","name":"Raspberry Pi: Embedded data usage","version":3,"mbox":"https://patchwork.libcamera.org/series/1708/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/11337/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/11337/checks/","tags":{},"headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D9790BD1F1\n\tfor <parsemail@patchwork.libcamera.org>;\n\tThu, 18 Feb 2021 17:01:34 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A703B689C1;\n\tThu, 18 Feb 2021 18:01:34 +0100 (CET)","from mail-wm1-x330.google.com (mail-wm1-x330.google.com\n\t[IPv6:2a00:1450:4864:20::330])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id BBFA968370\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tThu, 18 Feb 2021 18:01:32 +0100 (CET)","by mail-wm1-x330.google.com with SMTP id x4so4590396wmi.3\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tThu, 18 Feb 2021 09:01:32 -0800 (PST)","from naush-laptop.patuck.local ([88.97.76.4])\n\tby smtp.gmail.com with ESMTPSA id\n\tt30sm10504747wra.56.2021.02.18.09.01.30\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tThu, 18 Feb 2021 09:01:31 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"a0t8Kzyi\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=from:to:cc:subject:date:message-id:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=BGf6FqDf+QKZ9XQ/XwwRcdUDqGpiZdrGU8Nuikcqs10=;\n\tb=a0t8KzyiRc5ic+B+Tlt6ARj7VvH2Wy5P5yArm9/Pa/NUAL1GWM+wn93fFtS+69wTEr\n\tCx7SVy1ZG4ZgfbDTwPZdWNbPBBWpejuKtPnncNUyYIC86P4bekT5TaxCl/tuU/NIdvHd\n\tjzEJ3jEv5NcDVrMrEa1N3X0pzzpvBDU+ZffSiWxudgPGX3bfFzK7iCYp5etnfE8eKSTY\n\t20Bc3G9pdLSqY2j+mZSrMkRXaeHI8R4l+oIqQjyulD6gxEVmdUXTqiLIKg0piS5nRT0a\n\t5Dfc9iS4uQSp7v9NOsnJCWBzUXvYHokRuBLqXmeGfqkXOM0/QGJe7gXkbRzN366Cgh9l\n\tyY6g==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=BGf6FqDf+QKZ9XQ/XwwRcdUDqGpiZdrGU8Nuikcqs10=;\n\tb=gjpo2mQxU6QlCWkZp1CnuOP9aiR/hOSVQIX02euE5ncOB35A5aXT4baRhKqU86Oplt\n\tDt1VFjyg0CoJQR6kKh16BKEGOkEiYVzYHx0jqE29DROWbX8ISCB81fxF8qzSn9DsyDlF\n\tXB8sGIPTcmjhF+ZgitkK4oV47GgfOpEyTwRHVIfG0DevsFQkELicBX3wGmN/KrLOZVlK\n\tZj64oig4KioLvhEvYmSsLNZ0tVEuNpCBiSe2lrlt1n6pjD1gGeyrxeai7xcYGebDK1qO\n\trVpdXUI3wkqIxLWNodTh1xCjDcAuvz/8CjAvH8/SbWuuJPdetxCV0fxDkV+X5Hx276Ob\n\tYZxg==","X-Gm-Message-State":"AOAM532lAcBIr7nbpcrI3iazHuXYCXvW/x3VUOZrycxmkl5+dHE8Xu7c\n\t+3iuiVkxcwsk8X2fW7/Vi3tm5i8tZQsfewYY","X-Google-Smtp-Source":"ABdhPJzrfbokt6r7OxRgOhiXz4CXyCTaWAZ37tUy/Wn0yA1ZcU8er2W5CkE9MTYvoLc9rVsMo60fYw==","X-Received":"by 2002:a05:600c:4857:: with SMTP id\n\tj23mr4493482wmo.66.1613667691733; \n\tThu, 18 Feb 2021 09:01:31 -0800 (PST)","From":"Naushir Patuck <naush@raspberrypi.com>","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 18 Feb 2021 17:01:23 +0000","Message-Id":"<20210218170126.2060783-2-naush@raspberrypi.com>","X-Mailer":"git-send-email 2.25.1","In-Reply-To":"<20210218170126.2060783-1-naush@raspberrypi.com>","References":"<20210218170126.2060783-1-naush@raspberrypi.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v3 1/4] pipeline: ipa: raspberrypi: Pass\n\texposure/gain values to IPA though controls","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"},"content":"When running with sensors that had no embedded data, the pipeline handler\nwould fill a dummy embedded data buffer with gain/exposure values, and\npass this buffer to the IPA together with the bayer buffer. The IPA would\nextract these values for use in the controller algorithms.\n\nRework this logic entirely by having a new RPiCameraData::BayerFrame\nqueue to replace the existing bayer queue. In addition to storing the\nFrameBuffer pointer, this also stores all the controls tracked by\nDelayedControls for that frame in a ControlList. This includes include\nexposure and gain values. On signalling RPi::IPA_EVENT_SIGNAL_ISP_PREPARE\nIPA event, the pipeline handler now passes this ControlList from the\nRPiCameraData::BayerFrame queue.\n\nThe IPA now extracts the gain and exposure values from the ControlList\ninstead of using RPiController::MdParserRPi to parse the embedded data\nbuffer.\n\nSigned-off-by: Naushir Patuck <naush@raspberrypi.com>\n---\n include/libcamera/ipa/raspberrypi.mojom       |   2 +\n src/ipa/raspberrypi/raspberrypi.cpp           | 132 +++++++++++-------\n .../pipeline/raspberrypi/raspberrypi.cpp      |  58 ++++----\n 3 files changed, 108 insertions(+), 84 deletions(-)","diff":"diff --git a/include/libcamera/ipa/raspberrypi.mojom b/include/libcamera/ipa/raspberrypi.mojom\nindex 5a27b1e4fc2d..f733a2cd2a40 100644\n--- a/include/libcamera/ipa/raspberrypi.mojom\n+++ b/include/libcamera/ipa/raspberrypi.mojom\n@@ -29,6 +29,8 @@ struct SensorConfig {\n struct ISPConfig {\n \tuint32 embeddedBufferId;\n \tuint32 bayerBufferId;\n+\tbool embeddedBufferPresent;\n+\tControlList controls;\n };\n \n struct ConfigInput {\ndiff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp\nindex 1226ea514521..f9ab6417866e 100644\n--- a/src/ipa/raspberrypi/raspberrypi.cpp\n+++ b/src/ipa/raspberrypi/raspberrypi.cpp\n@@ -1,6 +1,6 @@\n /* SPDX-License-Identifier: BSD-2-Clause */\n /*\n- * Copyright (C) 2019-2020, Raspberry Pi (Trading) Ltd.\n+ * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd.\n  *\n  * rpi.cpp - Raspberry Pi Image Processing Algorithms\n  */\n@@ -101,9 +101,11 @@ private:\n \tbool validateIspControls();\n \tvoid queueRequest(const ControlList &controls);\n \tvoid returnEmbeddedBuffer(unsigned int bufferId);\n-\tvoid prepareISP(unsigned int bufferId);\n+\tvoid prepareISP(const ipa::RPi::ISPConfig &data);\n \tvoid reportMetadata();\n \tbool parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &deviceStatus);\n+\tvoid fillDeviceStatus(uint32_t exposureLines, uint32_t gainCode,\n+\t\t\t      struct DeviceStatus &deviceStatus);\n \tvoid processStats(unsigned int bufferId);\n \tvoid applyFrameDurations(double minFrameDuration, double maxFrameDuration);\n \tvoid applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls);\n@@ -447,7 +449,7 @@ void IPARPi::signalIspPrepare(const ipa::RPi::ISPConfig &data)\n \t * avoid running the control algos for a few frames in case\n \t * they are \"unreliable\".\n \t */\n-\tprepareISP(data.embeddedBufferId);\n+\tprepareISP(data);\n \tframeCount_++;\n \n \t/* Ready to push the input buffer into the ISP. */\n@@ -909,67 +911,84 @@ void IPARPi::returnEmbeddedBuffer(unsigned int bufferId)\n \tembeddedComplete.emit(bufferId & ipa::RPi::MaskID);\n }\n \n-void IPARPi::prepareISP(unsigned int bufferId)\n+void IPARPi::prepareISP(const ipa::RPi::ISPConfig &data)\n {\n \tstruct DeviceStatus deviceStatus = {};\n-\tbool success = parseEmbeddedData(bufferId, deviceStatus);\n+\tbool success = false;\n \n-\t/* Done with embedded data now, return to pipeline handler asap. */\n-\treturnEmbeddedBuffer(bufferId);\n+\tif (data.embeddedBufferPresent) {\n+\t\t/*\n+\t\t * Pipeline handler has supplied us with an embedded data buffer,\n+\t\t * so parse it and extract the exposure and gain.\n+\t\t */\n+\t\tsuccess = parseEmbeddedData(data.embeddedBufferId, deviceStatus);\n \n-\tif (success) {\n-\t\tControlList ctrls(ispCtrls_);\n+\t\t/* Done with embedded data now, return to pipeline handler asap. */\n+\t\treturnEmbeddedBuffer(data.embeddedBufferId);\n+\t}\n \n-\t\trpiMetadata_.Clear();\n-\t\trpiMetadata_.Set(\"device.status\", deviceStatus);\n-\t\tcontroller_.Prepare(&rpiMetadata_);\n+\tif (!success) {\n+\t\t/*\n+\t\t * Pipeline handler has not supplied an embedded data buffer,\n+\t\t * or embedded data buffer parsing has failed for some reason,\n+\t\t * so pull the exposure and gain values from the control list.\n+\t\t */\n+\t\tint32_t exposureLines = data.controls.get(V4L2_CID_EXPOSURE).get<int32_t>();\n+\t\tint32_t gainCode = data.controls.get(V4L2_CID_ANALOGUE_GAIN).get<int32_t>();\n+\t\tfillDeviceStatus(exposureLines, gainCode, deviceStatus);\n+\t}\n \n-\t\t/* Lock the metadata buffer to avoid constant locks/unlocks. */\n-\t\tstd::unique_lock<RPiController::Metadata> lock(rpiMetadata_);\n+\tControlList ctrls(ispCtrls_);\n \n-\t\tAwbStatus *awbStatus = rpiMetadata_.GetLocked<AwbStatus>(\"awb.status\");\n-\t\tif (awbStatus)\n-\t\t\tapplyAWB(awbStatus, ctrls);\n+\trpiMetadata_.Clear();\n+\trpiMetadata_.Set(\"device.status\", deviceStatus);\n+\tcontroller_.Prepare(&rpiMetadata_);\n \n-\t\tCcmStatus *ccmStatus = rpiMetadata_.GetLocked<CcmStatus>(\"ccm.status\");\n-\t\tif (ccmStatus)\n-\t\t\tapplyCCM(ccmStatus, ctrls);\n+\t/* Lock the metadata buffer to avoid constant locks/unlocks. */\n+\tstd::unique_lock<RPiController::Metadata> lock(rpiMetadata_);\n \n-\t\tAgcStatus *dgStatus = rpiMetadata_.GetLocked<AgcStatus>(\"agc.status\");\n-\t\tif (dgStatus)\n-\t\t\tapplyDG(dgStatus, ctrls);\n+\tAwbStatus *awbStatus = rpiMetadata_.GetLocked<AwbStatus>(\"awb.status\");\n+\tif (awbStatus)\n+\t\tapplyAWB(awbStatus, ctrls);\n \n-\t\tAlscStatus *lsStatus = rpiMetadata_.GetLocked<AlscStatus>(\"alsc.status\");\n-\t\tif (lsStatus)\n-\t\t\tapplyLS(lsStatus, ctrls);\n+\tCcmStatus *ccmStatus = rpiMetadata_.GetLocked<CcmStatus>(\"ccm.status\");\n+\tif (ccmStatus)\n+\t\tapplyCCM(ccmStatus, ctrls);\n \n-\t\tContrastStatus *contrastStatus = rpiMetadata_.GetLocked<ContrastStatus>(\"contrast.status\");\n-\t\tif (contrastStatus)\n-\t\t\tapplyGamma(contrastStatus, ctrls);\n+\tAgcStatus *dgStatus = rpiMetadata_.GetLocked<AgcStatus>(\"agc.status\");\n+\tif (dgStatus)\n+\t\tapplyDG(dgStatus, ctrls);\n \n-\t\tBlackLevelStatus *blackLevelStatus = rpiMetadata_.GetLocked<BlackLevelStatus>(\"black_level.status\");\n-\t\tif (blackLevelStatus)\n-\t\t\tapplyBlackLevel(blackLevelStatus, ctrls);\n+\tAlscStatus *lsStatus = rpiMetadata_.GetLocked<AlscStatus>(\"alsc.status\");\n+\tif (lsStatus)\n+\t\tapplyLS(lsStatus, ctrls);\n \n-\t\tGeqStatus *geqStatus = rpiMetadata_.GetLocked<GeqStatus>(\"geq.status\");\n-\t\tif (geqStatus)\n-\t\t\tapplyGEQ(geqStatus, ctrls);\n+\tContrastStatus *contrastStatus = rpiMetadata_.GetLocked<ContrastStatus>(\"contrast.status\");\n+\tif (contrastStatus)\n+\t\tapplyGamma(contrastStatus, ctrls);\n \n-\t\tDenoiseStatus *denoiseStatus = rpiMetadata_.GetLocked<DenoiseStatus>(\"denoise.status\");\n-\t\tif (denoiseStatus)\n-\t\t\tapplyDenoise(denoiseStatus, ctrls);\n+\tBlackLevelStatus *blackLevelStatus = rpiMetadata_.GetLocked<BlackLevelStatus>(\"black_level.status\");\n+\tif (blackLevelStatus)\n+\t\tapplyBlackLevel(blackLevelStatus, ctrls);\n \n-\t\tSharpenStatus *sharpenStatus = rpiMetadata_.GetLocked<SharpenStatus>(\"sharpen.status\");\n-\t\tif (sharpenStatus)\n-\t\t\tapplySharpen(sharpenStatus, ctrls);\n+\tGeqStatus *geqStatus = rpiMetadata_.GetLocked<GeqStatus>(\"geq.status\");\n+\tif (geqStatus)\n+\t\tapplyGEQ(geqStatus, ctrls);\n \n-\t\tDpcStatus *dpcStatus = rpiMetadata_.GetLocked<DpcStatus>(\"dpc.status\");\n-\t\tif (dpcStatus)\n-\t\t\tapplyDPC(dpcStatus, ctrls);\n+\tDenoiseStatus *denoiseStatus = rpiMetadata_.GetLocked<DenoiseStatus>(\"denoise.status\");\n+\tif (denoiseStatus)\n+\t\tapplyDenoise(denoiseStatus, ctrls);\n \n-\t\tif (!ctrls.empty())\n-\t\t\tsetIspControls.emit(ctrls);\n-\t}\n+\tSharpenStatus *sharpenStatus = rpiMetadata_.GetLocked<SharpenStatus>(\"sharpen.status\");\n+\tif (sharpenStatus)\n+\t\tapplySharpen(sharpenStatus, ctrls);\n+\n+\tDpcStatus *dpcStatus = rpiMetadata_.GetLocked<DpcStatus>(\"dpc.status\");\n+\tif (dpcStatus)\n+\t\tapplyDPC(dpcStatus, ctrls);\n+\n+\tif (!ctrls.empty())\n+\t\tsetIspControls.emit(ctrls);\n }\n \n bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &deviceStatus)\n@@ -985,6 +1004,7 @@ bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &devic\n \tRPiController::MdParser::Status status = helper_->Parser().Parse(mem.data());\n \tif (status != RPiController::MdParser::Status::OK) {\n \t\tLOG(IPARPI, Error) << \"Embedded Buffer parsing failed, error \" << status;\n+\t\treturn false;\n \t} else {\n \t\tuint32_t exposureLines, gainCode;\n \t\tif (helper_->Parser().GetExposureLines(exposureLines) != RPiController::MdParser::Status::OK) {\n@@ -992,21 +1012,29 @@ bool IPARPi::parseEmbeddedData(unsigned int bufferId, struct DeviceStatus &devic\n \t\t\treturn false;\n \t\t}\n \n-\t\tdeviceStatus.shutter_speed = helper_->Exposure(exposureLines);\n \t\tif (helper_->Parser().GetGainCode(gainCode) != RPiController::MdParser::Status::OK) {\n \t\t\tLOG(IPARPI, Error) << \"Gain failed\";\n \t\t\treturn false;\n \t\t}\n \n-\t\tdeviceStatus.analogue_gain = helper_->Gain(gainCode);\n-\t\tLOG(IPARPI, Debug) << \"Metadata - Exposure : \"\n-\t\t\t\t   << deviceStatus.shutter_speed << \" Gain : \"\n-\t\t\t\t   << deviceStatus.analogue_gain;\n+\t\tfillDeviceStatus(exposureLines, gainCode, deviceStatus);\n \t}\n \n \treturn true;\n }\n \n+void IPARPi::fillDeviceStatus(uint32_t exposureLines, uint32_t gainCode,\n+\t\t\t      struct DeviceStatus &deviceStatus)\n+{\n+\tdeviceStatus.shutter_speed = helper_->Exposure(exposureLines);\n+\tdeviceStatus.analogue_gain = helper_->Gain(gainCode);\n+\n+\tLOG(IPARPI, Debug) << \"Metadata - Exposure : \"\n+\t\t\t   << deviceStatus.shutter_speed\n+\t\t\t   << \" Gain : \"\n+\t\t\t   << deviceStatus.analogue_gain;\n+}\n+\n void IPARPi::processStats(unsigned int bufferId)\n {\n \tauto it = buffers_.find(bufferId);\ndiff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\nindex ba74ace183db..b43d86166c63 100644\n--- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n+++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n@@ -1,6 +1,6 @@\n /* SPDX-License-Identifier: LGPL-2.1-or-later */\n /*\n- * Copyright (C) 2019-2020, Raspberry Pi (Trading) Ltd.\n+ * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd.\n  *\n  * raspberrypi.cpp - Pipeline handler for Raspberry Pi devices\n  */\n@@ -197,7 +197,13 @@ public:\n \t */\n \tenum class State { Stopped, Idle, Busy, IpaComplete };\n \tState state_;\n-\tstd::queue<FrameBuffer *> bayerQueue_;\n+\n+\tstruct BayerFrame {\n+\t\tFrameBuffer *buffer;\n+\t\tControlList controls;\n+\t};\n+\n+\tstd::queue<BayerFrame> bayerQueue_;\n \tstd::queue<FrameBuffer *> embeddedQueue_;\n \tstd::deque<Request *> requestQueue_;\n \n@@ -222,7 +228,7 @@ public:\n private:\n \tvoid checkRequestCompleted();\n \tvoid tryRunPipeline();\n-\tbool findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *&embeddedBuffer);\n+\tbool findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&embeddedBuffer);\n \n \tunsigned int ispOutputCount_;\n };\n@@ -1383,29 +1389,14 @@ void RPiCameraData::unicamBufferDequeue(FrameBuffer *buffer)\n \t\t\t<< \", timestamp: \" << buffer->metadata().timestamp;\n \n \tif (stream == &unicam_[Unicam::Image]) {\n-\t\tbayerQueue_.push(buffer);\n-\t} else {\n-\t\tembeddedQueue_.push(buffer);\n-\n-\t\tControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence);\n-\n \t\t/*\n-\t\t * Sensor metadata is unavailable, so put the expected ctrl\n-\t\t * values (accounting for the staggered delays) into the empty\n-\t\t * metadata buffer.\n+\t\t * Lookup the sensor controls used for this frame sequence from\n+\t\t * StaggeredCtrl and queue them along with the frame buffer.\n \t\t */\n-\t\tif (!sensorMetadata_) {\n-\t\t\tunsigned int bufferId = unicam_[Unicam::Embedded].getBufferId(buffer);\n-\t\t\tauto it = mappedEmbeddedBuffers_.find(bufferId);\n-\t\t\tif (it != mappedEmbeddedBuffers_.end()) {\n-\t\t\t\tuint32_t *mem = reinterpret_cast<uint32_t *>(it->second.maps()[0].data());\n-\t\t\t\tmem[0] = ctrl.get(V4L2_CID_EXPOSURE).get<int32_t>();\n-\t\t\t\tmem[1] = ctrl.get(V4L2_CID_ANALOGUE_GAIN).get<int32_t>();\n-\t\t\t} else {\n-\t\t\t\tLOG(RPI, Warning) << \"Failed to find embedded buffer \"\n-\t\t\t\t\t\t  << bufferId;\n-\t\t\t}\n-\t\t}\n+\t\tControlList ctrl = delayedCtrls_->get(buffer->metadata().sequence);\n+\t\tbayerQueue_.push({ buffer, std::move(ctrl) });\n+\t} else {\n+\t\tembeddedQueue_.push(buffer);\n \t}\n \n \thandleState();\n@@ -1656,14 +1647,15 @@ void RPiCameraData::applyScalerCrop(const ControlList &controls)\n \n void RPiCameraData::tryRunPipeline()\n {\n-\tFrameBuffer *bayerBuffer, *embeddedBuffer;\n+\tFrameBuffer *embeddedBuffer;\n+\tBayerFrame bayerFrame;\n \n \t/* If any of our request or buffer queues are empty, we cannot proceed. */\n \tif (state_ != State::Idle || requestQueue_.empty() ||\n \t    bayerQueue_.empty() || embeddedQueue_.empty())\n \t\treturn;\n \n-\tif (!findMatchingBuffers(bayerBuffer, embeddedBuffer))\n+\tif (!findMatchingBuffers(bayerFrame, embeddedBuffer))\n \t\treturn;\n \n \t/* Take the first request from the queue and action the IPA. */\n@@ -1682,7 +1674,7 @@ void RPiCameraData::tryRunPipeline()\n \t/* Set our state to say the pipeline is active. */\n \tstate_ = State::Busy;\n \n-\tunsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerBuffer);\n+\tunsigned int bayerId = unicam_[Unicam::Image].getBufferId(bayerFrame.buffer);\n \tunsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer);\n \n \tLOG(RPI, Debug) << \"Signalling signalIspPrepare:\"\n@@ -1692,24 +1684,26 @@ void RPiCameraData::tryRunPipeline()\n \tipa::RPi::ISPConfig ispPrepare;\n \tispPrepare.embeddedBufferId = ipa::RPi::MaskEmbeddedData | embeddedId;\n \tispPrepare.bayerBufferId = ipa::RPi::MaskBayerData | bayerId;\n+\tispPrepare.embeddedBufferPresent = true;\n+\tispPrepare.controls = std::move(bayerFrame.controls);\n \tipa_->signalIspPrepare(ispPrepare);\n }\n \n-bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *&embeddedBuffer)\n+bool RPiCameraData::findMatchingBuffers(BayerFrame &bayerFrame, FrameBuffer *&embeddedBuffer)\n {\n \tunsigned int embeddedRequeueCount = 0, bayerRequeueCount = 0;\n \n \t/* Loop until we find a matching bayer and embedded data buffer. */\n \twhile (!bayerQueue_.empty()) {\n \t\t/* Start with the front of the bayer queue. */\n-\t\tbayerBuffer = bayerQueue_.front();\n+\t\tbayerFrame = bayerQueue_.front();\n \n \t\t/*\n \t\t * Find the embedded data buffer with a matching timestamp to pass to\n \t\t * the IPA. Any embedded buffers with a timestamp lower than the\n \t\t * current bayer buffer will be removed and re-queued to the driver.\n \t\t */\n-\t\tuint64_t ts = bayerBuffer->metadata().timestamp;\n+\t\tuint64_t ts = bayerFrame.buffer->metadata().timestamp;\n \t\tembeddedBuffer = nullptr;\n \t\twhile (!embeddedQueue_.empty()) {\n \t\t\tFrameBuffer *b = embeddedQueue_.front();\n@@ -1739,7 +1733,7 @@ bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *\n \t\t\t\t * the front of the queue. This buffer is now orphaned, so requeue\n \t\t\t\t * it back to the device.\n \t\t\t\t */\n-\t\t\t\tunicam_[Unicam::Image].queueBuffer(bayerQueue_.front());\n+\t\t\t\tunicam_[Unicam::Image].queueBuffer(bayerQueue_.front().buffer);\n \t\t\t\tbayerQueue_.pop();\n \t\t\t\tbayerRequeueCount++;\n \t\t\t\tLOG(RPI, Warning) << \"Dropping unmatched input frame in stream \"\n@@ -1757,7 +1751,7 @@ bool RPiCameraData::findMatchingBuffers(FrameBuffer *&bayerBuffer, FrameBuffer *\n \n \t\t\t\tLOG(RPI, Warning) << \"Flushing bayer stream!\";\n \t\t\t\twhile (!bayerQueue_.empty()) {\n-\t\t\t\t\tunicam_[Unicam::Image].queueBuffer(bayerQueue_.front());\n+\t\t\t\t\tunicam_[Unicam::Image].queueBuffer(bayerQueue_.front().buffer);\n \t\t\t\t\tbayerQueue_.pop();\n \t\t\t\t}\n \t\t\t\tflushedBuffers = true;\n","prefixes":["libcamera-devel","v3","1/4"]}