{"id":23060,"url":"https://patchwork.libcamera.org/api/1.1/patches/23060/?format=json","web_url":"https://patchwork.libcamera.org/patch/23060/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20250327185945.159481-3-mzamazal@redhat.com>","date":"2025-03-27T18:59:40","name":"[v6,2/6] ipa: simple: softisp: Extend to pass metadata","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"a5502c947d596f4700ac3b224129dd652eef19ff","submitter":{"id":177,"url":"https://patchwork.libcamera.org/api/1.1/people/177/?format=json","name":"Milan Zamazal","email":"mzamazal@redhat.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/23060/mbox/","series":[{"id":5087,"url":"https://patchwork.libcamera.org/api/1.1/series/5087/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=5087","date":"2025-03-27T18:59:38","name":"ipa: simple: Introduce metadata reporting","version":6,"mbox":"https://patchwork.libcamera.org/series/5087/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/23060/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/23060/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0666FC323E\n\tfor ;\n\tThu, 27 Mar 2025 19:00:06 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6897C6897F;\n\tThu, 27 Mar 2025 20:00:05 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2769A68971\n\tfor ;\n\tThu, 27 Mar 2025 20:00:03 +0100 (CET)","from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com\n\t(ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97])\n\tby relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n\tcipher=TLS_AES_256_GCM_SHA384) id us-mta-65-W1Zqvj7pMEC13tgILEcWGQ-1;\n\tThu, 27 Mar 2025 14:59:59 -0400","from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com\n\t(mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (2048 bits)\n\tserver-digest SHA256) (No client certificate requested)\n\tby mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTPS id D12FF180AB19; Thu, 27 Mar 2025 18:59:58 +0000 (UTC)","from mzamazal-thinkpadp1gen7.tpbc.com (unknown [10.45.224.17])\n\tby mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTP id 058D130001A1; Thu, 27 Mar 2025 18:59:56 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"AWyeI70V\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1743102002;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=FBL4j11uDGzu/i4OG1Qm8222bw75urCxPjWL1WsxBkk=;\n\tb=AWyeI70VDbvUbGvRfyTIDZVxlY3PZiUbBGrv/Rm+olJD3im8olU4xisdnZ+mNMUuvtq99O\n\tyjQ5FWsF1hZeD92gtvzUDsXFUv0BeUT2SjEfJ78FfutW5ALaEDwn07jiQj6B+lCafAzAZi\n\tVQiCPStq649PV2LxtgU6W3xcZac7oiw=","X-MC-Unique":"W1Zqvj7pMEC13tgILEcWGQ-1","X-Mimecast-MFC-AGG-ID":"W1Zqvj7pMEC13tgILEcWGQ_1743101998","From":"Milan Zamazal ","To":"libcamera-devel@lists.libcamera.org","Cc":"Kieran Bingham ,\n\tLaurent Pinchart ,\n\tMilan Zamazal ","Subject":"[PATCH v6 2/6] ipa: simple: softisp: Extend to pass metadata","Date":"Thu, 27 Mar 2025 19:59:40 +0100","Message-ID":"<20250327185945.159481-3-mzamazal@redhat.com>","In-Reply-To":"<20250327185945.159481-1-mzamazal@redhat.com>","References":"<20250327185945.159481-1-mzamazal@redhat.com>","MIME-Version":"1.0","X-Scanned-By":"MIMEDefang 3.4.1 on 10.30.177.4","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"LOSSDaqZhgYEwO9JbqArGLdTlsRkXbOKdPkh6_29wyU_1743101998","X-Mimecast-Originator":"redhat.com","Content-Transfer-Encoding":"8bit","content-type":"text/plain; charset=\"US-ASCII\"; x-default=true","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"From: Kieran Bingham \n\nExtend the Simple IPA IPC to support returning a metadata ControlList\nwhen the process call has completed.\n\nA new signal from the IPA is introduced to report the metadata,\nsimilarly to what the hardware pipelines do.\n\nMerge the metadata reported by the ISP into any completing request to\nprovide to the application. Completion of a request is delayed until\nthis is done; this doesn't apply to canceled requests.\n\nReviewed-by: Kieran Bingham \nSigned-off-by: Kieran Bingham \nSigned-off-by: Milan Zamazal \n---\n .../internal/software_isp/software_isp.h | 2 +\n include/libcamera/ipa/soft.mojom | 1 +\n src/ipa/simple/soft_simple.cpp | 7 +--\n src/libcamera/pipeline/simple/simple.cpp | 48 ++++++++++++++-----\n src/libcamera/software_isp/software_isp.cpp | 9 ++++\n 5 files changed, 49 insertions(+), 18 deletions(-)","diff":"diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h\nindex 0026cec2..1ee37dc7 100644\n--- a/include/libcamera/internal/software_isp/software_isp.h\n+++ b/include/libcamera/internal/software_isp/software_isp.h\n@@ -85,12 +85,14 @@ public:\n \tSignal inputBufferReady;\n \tSignal outputBufferReady;\n \tSignal ispStatsReady;\n+\tSignal metadataReady;\n \tSignal setSensorControls;\n \n private:\n \tvoid saveIspParams();\n \tvoid setSensorCtrls(const ControlList &sensorControls);\n \tvoid statsReady(uint32_t frame, uint32_t bufferId);\n+\tvoid saveMetadata(uint32_t frame, const ControlList &metadata);\n \tvoid inputReady(FrameBuffer *input);\n \tvoid outputReady(FrameBuffer *output);\n \ndiff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom\nindex ede74413..a8e6ecda 100644\n--- a/include/libcamera/ipa/soft.mojom\n+++ b/include/libcamera/ipa/soft.mojom\n@@ -33,4 +33,5 @@ interface IPASoftInterface {\n interface IPASoftEventInterface {\n \tsetSensorControls(libcamera.ControlList sensorControls);\n \tsetIspParams();\n+\tmetadataReady(uint32 frame, libcamera.ControlList metadata);\n };\ndiff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp\nindex a87c6cdd..4ef67c43 100644\n--- a/src/ipa/simple/soft_simple.cpp\n+++ b/src/ipa/simple/soft_simple.cpp\n@@ -299,15 +299,10 @@ void IPASoftSimple::processStats(const uint32_t frame,\n \tint32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get();\n \tframeContext.sensor.gain = camHelper_ ? camHelper_->gain(again) : again;\n \n-\t/*\n-\t * Software ISP currently does not produce any metadata. Use an empty\n-\t * ControlList for now.\n-\t *\n-\t * \\todo Implement proper metadata handling\n-\t */\n \tControlList metadata(controls::controls);\n \tfor (auto const &algo : algorithms())\n \t\talgo->process(context_, frame, frameContext, stats_, metadata);\n+\tmetadataReady.emit(frame, metadata);\n \n \t/* Sanity check */\n \tif (!sensorControls.contains(V4L2_CID_EXPOSURE) ||\ndiff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp\nindex 004a8d53..fd0ccdca 100644\n--- a/src/libcamera/pipeline/simple/simple.cpp\n+++ b/src/libcamera/pipeline/simple/simple.cpp\n@@ -182,19 +182,21 @@ LOG_DEFINE_CATEGORY(SimplePipeline)\n class SimplePipelineHandler;\n \n struct SimpleFrameInfo {\n-\tSimpleFrameInfo(uint32_t f, Request *r)\n-\t\t: frame(f), request(r)\n+\tSimpleFrameInfo(uint32_t f, Request *r, bool m)\n+\t\t: frame(f), request(r), metadataRequired(m), metadataProcessed(false)\n \t{\n \t}\n \n \tuint32_t frame;\n \tRequest *request;\n+\tbool metadataRequired;\n+\tbool metadataProcessed;\n };\n \n class SimpleFrames\n {\n public:\n-\tvoid create(Request *request);\n+\tvoid create(Request *request, bool metadataRequested);\n \tvoid destroy(uint32_t frame);\n \tvoid clear();\n \n@@ -204,10 +206,10 @@ private:\n \tstd::map frameInfo_;\n };\n \n-void SimpleFrames::create(Request *request)\n+void SimpleFrames::create(Request *request, bool metadataRequired)\n {\n \tconst uint32_t frame = request->sequence();\n-\tauto [it, inserted] = frameInfo_.try_emplace(frame, frame, request);\n+\tauto [it, inserted] = frameInfo_.try_emplace(frame, frame, request, metadataRequired);\n \tASSERT(inserted);\n }\n \n@@ -347,11 +349,12 @@ private:\n \tvoid tryPipeline(unsigned int code, const Size &size);\n \tstatic std::vector routedSourcePads(MediaPad *sink);\n \n-\tvoid completeRequest(Request *request);\n+\tvoid tryCompleteRequest(Request *request);\n \tvoid conversionInputDone(FrameBuffer *buffer);\n \tvoid conversionOutputDone(FrameBuffer *buffer);\n \n \tvoid ispStatsReady(uint32_t frame, uint32_t bufferId);\n+\tvoid metadataReady(uint32_t frame, const ControlList &metadata);\n \tvoid setSensorControls(const ControlList &sensorControls);\n };\n \n@@ -590,6 +593,7 @@ int SimpleCameraData::init()\n \t\t\tswIsp_->inputBufferReady.connect(this, &SimpleCameraData::conversionInputDone);\n \t\t\tswIsp_->outputBufferReady.connect(this, &SimpleCameraData::conversionOutputDone);\n \t\t\tswIsp_->ispStatsReady.connect(this, &SimpleCameraData::ispStatsReady);\n+\t\t\tswIsp_->metadataReady.connect(this, &SimpleCameraData::metadataReady);\n \t\t\tswIsp_->setSensorControls.connect(this, &SimpleCameraData::setSensorControls);\n \t\t}\n \t}\n@@ -835,7 +839,7 @@ void SimpleCameraData::imageBufferReady(FrameBuffer *buffer)\n \t\t\t/* No conversion, just complete the request. */\n \t\t\tRequest *request = buffer->request();\n \t\t\tpipe->completeBuffer(request, buffer);\n-\t\t\tcompleteRequest(request);\n+\t\t\ttryCompleteRequest(request);\n \t\t\treturn;\n \t\t}\n \n@@ -853,7 +857,10 @@ void SimpleCameraData::imageBufferReady(FrameBuffer *buffer)\n \t\tconst RequestOutputs &outputs = conversionQueue_.front();\n \t\tfor (auto &[stream, buf] : outputs.outputs)\n \t\t\tpipe->completeBuffer(outputs.request, buf);\n-\t\tcompleteRequest(outputs.request);\n+\t\tSimpleFrameInfo *info = frameInfo_.find(outputs.request->sequence());\n+\t\tif (info)\n+\t\t\tinfo->metadataRequired = false;\n+\t\ttryCompleteRequest(outputs.request);\n \t\tconversionQueue_.pop();\n \n \t\treturn;\n@@ -911,7 +918,7 @@ void SimpleCameraData::imageBufferReady(FrameBuffer *buffer)\n \n \t/* Otherwise simply complete the request. */\n \tpipe->completeBuffer(request, buffer);\n-\tcompleteRequest(request);\n+\ttryCompleteRequest(request);\n }\n \n void SimpleCameraData::clearIncompleteRequests()\n@@ -922,14 +929,20 @@ void SimpleCameraData::clearIncompleteRequests()\n \t}\n }\n \n-void SimpleCameraData::completeRequest(Request *request)\n+void SimpleCameraData::tryCompleteRequest(Request *request)\n {\n+\tif (request->hasPendingBuffers())\n+\t\treturn;\n+\n \tSimpleFrameInfo *info = frameInfo_.find(request->sequence());\n \tif (!info) {\n \t\t/* Something is really wrong, let's return. */\n \t\treturn;\n \t}\n \n+\tif (info->metadataRequired && !info->metadataProcessed)\n+\t\treturn;\n+\n \tframeInfo_.destroy(info->frame);\n \tpipe()->completeRequest(request);\n }\n@@ -947,7 +960,7 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer)\n \t/* Complete the buffer and the request. */\n \tRequest *request = buffer->request();\n \tif (pipe->completeBuffer(request, buffer))\n-\t\tcompleteRequest(request);\n+\t\ttryCompleteRequest(request);\n }\n \n void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId)\n@@ -956,6 +969,17 @@ void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId)\n \t\t\t delayedCtrls_->get(frame));\n }\n \n+void SimpleCameraData::metadataReady(uint32_t frame, const ControlList &metadata)\n+{\n+\tSimpleFrameInfo *info = frameInfo_.find(frame);\n+\tif (!info)\n+\t\treturn;\n+\n+\tinfo->request->metadata().merge(metadata);\n+\tinfo->metadataProcessed = true;\n+\ttryCompleteRequest(info->request);\n+}\n+\n void SimpleCameraData::setSensorControls(const ControlList &sensorControls)\n {\n \tdelayedCtrls_->push(sensorControls);\n@@ -1489,7 +1513,7 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request)\n \t\t}\n \t}\n \n-\tdata->frameInfo_.create(request);\n+\tdata->frameInfo_.create(request, !!data->swIsp_);\n \tif (data->useConversion_) {\n \t\tdata->conversionQueue_.push({ request, std::move(buffers) });\n \t\tif (data->swIsp_)\ndiff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp\nindex 4a74dcb6..6baa3fec 100644\n--- a/src/libcamera/software_isp/software_isp.cpp\n+++ b/src/libcamera/software_isp/software_isp.cpp\n@@ -55,6 +55,11 @@ LOG_DEFINE_CATEGORY(SoftwareIsp)\n * \\brief A signal emitted when the statistics for IPA are ready\n */\n \n+/**\n+ * \\var SoftwareIsp::metadataReady\n+ * \\brief A signal emitted when the metadata for IPA is ready\n+ */\n+\n /**\n * \\var SoftwareIsp::setSensorControls\n * \\brief A signal emitted when the values to write to the sensor controls are\n@@ -141,6 +146,10 @@ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor,\n \t}\n \n \tipa_->setIspParams.connect(this, &SoftwareIsp::saveIspParams);\n+\tipa_->metadataReady.connect(this,\n+\t\t\t\t [this](uint32_t frame, const ControlList &metadata) {\n+\t\t\t\t\t metadataReady.emit(frame, metadata);\n+\t\t\t\t });\n \tipa_->setSensorControls.connect(this, &SoftwareIsp::setSensorCtrls);\n \n \tdebayer_->moveToThread(&ispWorkerThread_);\n","prefixes":["v6","2/6"]}