Patch Detail
Show a patch.
GET /api/1.1/patches/22309/?format=api
{ "id": 22309, "url": "https://patchwork.libcamera.org/api/1.1/patches/22309/?format=api", "web_url": "https://patchwork.libcamera.org/patch/22309/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/1.1/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20241213094602.2083174-5-naush@raspberrypi.com>", "date": "2024-12-13T09:38:27", "name": "[4/6] controls: ipa: rpi: Add CNN controls", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": false, "hash": "c22c8de3cb75ccba0304c80ce88eb833ddf386d3", "submitter": { "id": 34, "url": "https://patchwork.libcamera.org/api/1.1/people/34/?format=api", "name": "Naushir Patuck", "email": "naush@raspberrypi.com" }, "delegate": null, "mbox": "https://patchwork.libcamera.org/patch/22309/mbox/", "series": [ { "id": 4881, "url": "https://patchwork.libcamera.org/api/1.1/series/4881/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=4881", "date": "2024-12-13T09:38:23", "name": "Raspberry Pi: Various changes", "version": 1, "mbox": "https://patchwork.libcamera.org/series/4881/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/patches/22309/comments/", "check": "pending", "checks": "https://patchwork.libcamera.org/api/patches/22309/checks/", "tags": {}, "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7DF0EC32F1\n\tfor <parsemail@patchwork.libcamera.org>;\n\tFri, 13 Dec 2024 09:46:19 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 608E367EF9;\n\tFri, 13 Dec 2024 10:46:17 +0100 (CET)", "from mail-wr1-x436.google.com (mail-wr1-x436.google.com\n\t[IPv6:2a00:1450:4864:20::436])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8A48567EEB\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 13 Dec 2024 10:46:11 +0100 (CET)", "by mail-wr1-x436.google.com with SMTP id\n\tffacd0b85a97d-385db79aafbso98994f8f.1\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 13 Dec 2024 01:46:11 -0800 (PST)", "from NAUSH-P-DELL.pitowers.org ([93.93.133.154])\n\tby smtp.gmail.com with ESMTPSA id\n\t5b1f17b1804b1-4362557c502sm43989105e9.11.2024.12.13.01.46.09\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 13 Dec 2024 01:46:10 -0800 (PST)" ], "Authentication-Results": "lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"EgHoTKgr\"; dkim-atps=neutral", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1734083170; x=1734687970;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=HSblzL9ARgj4fEX3/rzbo1k76l68wSLBOWWDbQdxmmA=;\n\tb=EgHoTKgrdvQ3b5HH3C3Qt5scVVaV6GrnQOPzkA+h+JLVYz4EgxK7ONzFaHtubr5tuk\n\tEHv/AYHWeJMrBpQLT1O6emjjMXfmH3FtwoXdT5LxOE74i2mkDY8IrWFuRgjQP+6RA+W+\n\tUiUq4kUF3MtTOl6U+FGTMKrXAhQDLIlEHXHWYFATH4VN98MDutKTn+sHny+HSGGx2M6Y\n\twjiWduFQAxEJhDMhtNV2U0rRqrHFbeSDd+gsjdacVGDLE0SjhiS0gyNg0BpctfsK1aCt\n\tJEvjdXBAxzv5dIx8HDA9wDscXYIM3a4yn1v5JOl6YcfOCwoGHIeiQ3ObUGorWRzzLsPo\n\tB1aQ==", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1734083170; x=1734687970;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=HSblzL9ARgj4fEX3/rzbo1k76l68wSLBOWWDbQdxmmA=;\n\tb=KFqtERW1lZOT2z4EwOUHHbtGIDngJEryAjAwCElfPVg3PNyqrfYBtDTpnIvyQUHePw\n\tEz2053fC43le9c6IbSXFvEZfuV3eyGLo87Uh9GtIrIwFyZYPifnRVJaX2oGOqISA0jYW\n\tpVpFrsbSnH0TtS3/8O2pzWXSx8jZ3D+mGxS00yqR20ZQol5gbmKVOwRHJCj60azL4gl/\n\tp8ntqtb6QMV8VVXGXN3vEWKr5rReSZAI7h5fdj+5QigtyLXt+VAs5YgGKpTswzmR2Pcv\n\tTuNBZfxILDyPpguxFS0JgMJ0cCTV3AjNebgOo9TjEvxygNizNy+bM3ZwfCh2J4RBYmGw\n\tdAyQ==", "X-Gm-Message-State": "AOJu0YwxMefbH6Xqciknyxb2hnyeEyyPmP3UshMLMzc1O8BM5kGqIdfX\n\tLsji40Scyv5XzksJgK1vkFpeOWPvaWZ88KMDhqY6nUph7uKMMEvsZH7tBI2P4AZ3Fpulwg7qZt1\n\tc", "X-Gm-Gg": "ASbGncvp/XmqBpmRYFeZ4euSg40GZ1Ymqyl6CVLpiAEn+63DlEDftSCrq15+4e308Sg\n\t9k8hQQ3P5OYGVDibVTv1ElHnuLYntHCqed0dEVWYL0fBystSBqvNxis8pGLqd9AbEhuZjAU//T8\n\tjIvC2VQDTRJKlwu7/VaxWTqNqSECgzXgu+CEd3bEsN9/s05cXiWICOl8LO+Siziw6QnQfEDzrlJ\n\tFXiZp2EzOri968J3JKueCS+ASRKSu7bA3ncOXO/twQ5qG5q4mabWjuhmUx0uARR0uyES14Vfj8n", "X-Google-Smtp-Source": "AGHT+IEVoP/wxnIvLYkndk/DMifMcLQEK/lFq3FufDJGDHUqJFZnSFf/5VAyJlple0uHRclWvPGaQg==", "X-Received": "by 2002:a5d:47a9:0:b0:385:fa20:6583 with SMTP id\n\tffacd0b85a97d-38880ac637dmr416660f8f.2.1734083170462; \n\tFri, 13 Dec 2024 01:46:10 -0800 (PST)", "From": "Naushir Patuck <naush@raspberrypi.com>", "To": "libcamera-devel@lists.libcamera.org", "Cc": "Naushir Patuck <naush@raspberrypi.com>", "Subject": "[PATCH 4/6] controls: ipa: rpi: Add CNN controls", "Date": "Fri, 13 Dec 2024 09:38:27 +0000", "Message-ID": "<20241213094602.2083174-5-naush@raspberrypi.com>", "X-Mailer": "git-send-email 2.43.0", "In-Reply-To": "<20241213094602.2083174-1-naush@raspberrypi.com>", "References": "<20241213094602.2083174-1-naush@raspberrypi.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "Add the follwing RPi vendor controls to handle Convolutional Neural\nNetwork processing:\n\nCnnOutputTensor\nCnnOutputTensorInfo\nCnnEnableInputTensor\nCnnInputTensor\nCnnInputTensorInfo\nCnnKpiInfo\n\nThese controls will be used to support the new Raspberry Pi AI Camera,\nusing an IMX500 sensor with on-board neural network processing.\n\nSigned-off-by: Naushir Patuck <naush@raspberrypi.com>\n---\n src/ipa/rpi/controller/controller.h | 33 +++++++++\n src/libcamera/control_ids_rpi.yaml | 108 ++++++++++++++++++++++++++++\n 2 files changed, 141 insertions(+)", "diff": "diff --git a/src/ipa/rpi/controller/controller.h b/src/ipa/rpi/controller/controller.h\nindex 64f93f414524..489188b44d9b 100644\n--- a/src/ipa/rpi/controller/controller.h\n+++ b/src/ipa/rpi/controller/controller.h\n@@ -25,6 +25,39 @@\n \n namespace RPiController {\n \n+/*\n+ * The following structures are used to export the CNN input/output tensor information\n+ * through the rpi::CnnOutputTensorInfo and rpi::CnnInputTensorInfo controls.\n+ * Applications must cast the span to these structures exactly.\n+ */\n+static constexpr unsigned int NetworkNameLen = 64;\n+static constexpr unsigned int MaxNumTensors = 16;\n+static constexpr unsigned int MaxNumDimensions = 16;\n+\n+struct OutputTensorInfo {\n+\tuint32_t tensorDataNum;\n+\tuint32_t numDimensions;\n+\tuint16_t size[MaxNumDimensions];\n+};\n+\n+struct CnnOutputTensorInfo {\n+\tchar networkName[NetworkNameLen];\n+\tuint32_t numTensors;\n+\tOutputTensorInfo info[MaxNumTensors];\n+};\n+\n+struct CnnInputTensorInfo {\n+\tchar networkName[NetworkNameLen];\n+\tuint32_t width;\n+\tuint32_t height;\n+\tuint32_t numChannels;\n+};\n+\n+struct CnnKpiInfo {\n+\tuint32_t dnnRuntime;\n+\tuint32_t dspRuntime;\n+};\n+\n class Algorithm;\n typedef std::unique_ptr<Algorithm> AlgorithmPtr;\n \ndiff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml\nindex 34bbdfc863c5..c0b5f63df525 100644\n--- a/src/libcamera/control_ids_rpi.yaml\n+++ b/src/libcamera/control_ids_rpi.yaml\n@@ -55,4 +55,112 @@ controls:\n official libcamera API support for per-stream controls in the future.\n \n \\sa ScalerCrop\n+\n+ - CnnOutputTensor:\n+ type: float\n+ size: [n]\n+ description: |\n+ This control returns a span of floating point values that represent the\n+ output tensors from a Convolutional Neural Network (CNN). The size and\n+ format of this array of values is entirely dependent on the neural\n+ network used, and further post-processing may need to be performed at\n+ the application level to generate the final desired output. This control\n+ is agnostic of the hardware or software used to generate the output\n+ tensors.\n+\n+ The structure of the span is described by the CnnOutputTensorInfo\n+ control.\n+\n+ \\sa CnnOutputTensorInfo\n+\n+ - CnnOutputTensorInfo:\n+ type: uint8_t\n+ size: [n]\n+ description: |\n+ This control returns the structure of the CnnOutputTensor. This structure\n+ takes the following form:\n+\n+ constexpr unsigned int NetworkNameLen = 64;\n+ constexpr unsigned int MaxNumTensors = 16;\n+ constexpr unsigned int MaxNumDimensions = 16;\n+\n+ struct CnnOutputTensorInfo {\n+ char networkName[NetworkNameLen];\n+ uint32_t numTensors;\n+ OutputTensorInfo info[MaxNumTensors];\n+ };\n+\n+ with\n+\n+ struct OutputTensorInfo {\n+ uint32_t tensorDataNum;\n+ uint32_t numDimensions;\n+ uint16_t size[MaxNumDimensions];\n+ };\n+\n+ networkName is the name of the CNN used,\n+ numTensors is the number of output tensors returned,\n+ tensorDataNum gives the number of elements in each output tensor,\n+ numDimensions gives the dimensionality of each output tensor,\n+ size gives the size of each dimension in each output tensor.\n+\n+ \\sa CnnOutputTensor\n+\n+ - CnnEnableInputTensor:\n+ type: bool\n+ description: |\n+ Boolean to control if the IPA returns the input tensor used by the CNN\n+ to generate the output tensors via the CnnInputTensor control. Because\n+ the input tensor may be relatively large, for efficiency reason avoid\n+ enabling input tensor output unless required for debugging purposes.\n+\n+ \\sa CnnInputTensor\n+\n+ - CnnInputTensor:\n+ type: uint8_t\n+ size: [n]\n+ description: |\n+ This control returns a span of uint8_t pixel values that represent the\n+ input tensor for a Convolutional Neural Network (CNN). The size and\n+ format of this array of values is entirely dependent on the neural\n+ network used, and further post-processing (e.g. pixel normalisations) may\n+ need to be performed at the application level to generate the final input\n+ image.\n+\n+ The structure of the span is described by the CnnInputTensorInfo\n+ control.\n+\n+ \\sa CnnInputTensorInfo\n+\n+ - CnnInputTensorInfo:\n+ type: uint8_t\n+ size: [n]\n+ description: |\n+ This control returns the structure of the CnnInputTensor. This structure\n+ takes the following form:\n+\n+ constexpr unsigned int NetworkNameLen = 64;\n+\n+ struct CnnInputTensorInfo {\n+ char networkName[NetworkNameLen];\n+ uint32_t width;\n+ uint32_t height;\n+ uint32_t numChannels;\n+ };\n+\n+ where\n+\n+ networkName is the name of the CNN used,\n+ width and height are the input tensor image width and height in pixels,\n+ numChannels is the number of channels in the input tensor image.\n+\n+ \\sa CnnInputTensor\n+\n+ - CnnKpiInfo:\n+ type: int32_t\n+ size: [2]\n+ description: |\n+ This control returns performance metrics for the CNN processing stage.\n+ Two values are returned in this span, the runtime of the CNN/DNN stage\n+ and the DSP stage in milliseconds.\n ...\n", "prefixes": [ "4/6" ] }