From patchwork Fri Dec 13 09:38:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 22309 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7DF0EC32F1 for ; Fri, 13 Dec 2024 09:46:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 608E367EF9; Fri, 13 Dec 2024 10:46:17 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="EgHoTKgr"; dkim-atps=neutral Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8A48567EEB for ; Fri, 13 Dec 2024 10:46:11 +0100 (CET) Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-385db79aafbso98994f8f.1 for ; Fri, 13 Dec 2024 01:46:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1734083170; x=1734687970; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HSblzL9ARgj4fEX3/rzbo1k76l68wSLBOWWDbQdxmmA=; b=EgHoTKgrdvQ3b5HH3C3Qt5scVVaV6GrnQOPzkA+h+JLVYz4EgxK7ONzFaHtubr5tuk EHv/AYHWeJMrBpQLT1O6emjjMXfmH3FtwoXdT5LxOE74i2mkDY8IrWFuRgjQP+6RA+W+ UiUq4kUF3MtTOl6U+FGTMKrXAhQDLIlEHXHWYFATH4VN98MDutKTn+sHny+HSGGx2M6Y wjiWduFQAxEJhDMhtNV2U0rRqrHFbeSDd+gsjdacVGDLE0SjhiS0gyNg0BpctfsK1aCt JEvjdXBAxzv5dIx8HDA9wDscXYIM3a4yn1v5JOl6YcfOCwoGHIeiQ3ObUGorWRzzLsPo B1aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734083170; x=1734687970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HSblzL9ARgj4fEX3/rzbo1k76l68wSLBOWWDbQdxmmA=; b=KFqtERW1lZOT2z4EwOUHHbtGIDngJEryAjAwCElfPVg3PNyqrfYBtDTpnIvyQUHePw Ez2053fC43le9c6IbSXFvEZfuV3eyGLo87Uh9GtIrIwFyZYPifnRVJaX2oGOqISA0jYW pVpFrsbSnH0TtS3/8O2pzWXSx8jZ3D+mGxS00yqR20ZQol5gbmKVOwRHJCj60azL4gl/ p8ntqtb6QMV8VVXGXN3vEWKr5rReSZAI7h5fdj+5QigtyLXt+VAs5YgGKpTswzmR2Pcv TuNBZfxILDyPpguxFS0JgMJ0cCTV3AjNebgOo9TjEvxygNizNy+bM3ZwfCh2J4RBYmGw dAyQ== X-Gm-Message-State: AOJu0YwxMefbH6Xqciknyxb2hnyeEyyPmP3UshMLMzc1O8BM5kGqIdfX Lsji40Scyv5XzksJgK1vkFpeOWPvaWZ88KMDhqY6nUph7uKMMEvsZH7tBI2P4AZ3Fpulwg7qZt1 c X-Gm-Gg: ASbGncvp/XmqBpmRYFeZ4euSg40GZ1Ymqyl6CVLpiAEn+63DlEDftSCrq15+4e308Sg 9k8hQQ3P5OYGVDibVTv1ElHnuLYntHCqed0dEVWYL0fBystSBqvNxis8pGLqd9AbEhuZjAU//T8 jIvC2VQDTRJKlwu7/VaxWTqNqSECgzXgu+CEd3bEsN9/s05cXiWICOl8LO+Siziw6QnQfEDzrlJ FXiZp2EzOri968J3JKueCS+ASRKSu7bA3ncOXO/twQ5qG5q4mabWjuhmUx0uARR0uyES14Vfj8n X-Google-Smtp-Source: AGHT+IEVoP/wxnIvLYkndk/DMifMcLQEK/lFq3FufDJGDHUqJFZnSFf/5VAyJlple0uHRclWvPGaQg== X-Received: by 2002:a5d:47a9:0:b0:385:fa20:6583 with SMTP id ffacd0b85a97d-38880ac637dmr416660f8f.2.1734083170462; Fri, 13 Dec 2024 01:46:10 -0800 (PST) Received: from NAUSH-P-DELL.pitowers.org ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4362557c502sm43989105e9.11.2024.12.13.01.46.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Dec 2024 01:46:10 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Cc: Naushir Patuck Subject: [PATCH 4/6] controls: ipa: rpi: Add CNN controls Date: Fri, 13 Dec 2024 09:38:27 +0000 Message-ID: <20241213094602.2083174-5-naush@raspberrypi.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241213094602.2083174-1-naush@raspberrypi.com> References: <20241213094602.2083174-1-naush@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add the follwing RPi vendor controls to handle Convolutional Neural Network processing: CnnOutputTensor CnnOutputTensorInfo CnnEnableInputTensor CnnInputTensor CnnInputTensorInfo CnnKpiInfo These controls will be used to support the new Raspberry Pi AI Camera, using an IMX500 sensor with on-board neural network processing. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman --- src/ipa/rpi/controller/controller.h | 33 +++++++++ src/libcamera/control_ids_rpi.yaml | 108 ++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) diff --git a/src/ipa/rpi/controller/controller.h b/src/ipa/rpi/controller/controller.h index 64f93f414524..489188b44d9b 100644 --- a/src/ipa/rpi/controller/controller.h +++ b/src/ipa/rpi/controller/controller.h @@ -25,6 +25,39 @@ namespace RPiController { +/* + * The following structures are used to export the CNN input/output tensor information + * through the rpi::CnnOutputTensorInfo and rpi::CnnInputTensorInfo controls. + * Applications must cast the span to these structures exactly. + */ +static constexpr unsigned int NetworkNameLen = 64; +static constexpr unsigned int MaxNumTensors = 16; +static constexpr unsigned int MaxNumDimensions = 16; + +struct OutputTensorInfo { + uint32_t tensorDataNum; + uint32_t numDimensions; + uint16_t size[MaxNumDimensions]; +}; + +struct CnnOutputTensorInfo { + char networkName[NetworkNameLen]; + uint32_t numTensors; + OutputTensorInfo info[MaxNumTensors]; +}; + +struct CnnInputTensorInfo { + char networkName[NetworkNameLen]; + uint32_t width; + uint32_t height; + uint32_t numChannels; +}; + +struct CnnKpiInfo { + uint32_t dnnRuntime; + uint32_t dspRuntime; +}; + class Algorithm; typedef std::unique_ptr AlgorithmPtr; diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml index 34bbdfc863c5..c0b5f63df525 100644 --- a/src/libcamera/control_ids_rpi.yaml +++ b/src/libcamera/control_ids_rpi.yaml @@ -55,4 +55,112 @@ controls: official libcamera API support for per-stream controls in the future. \sa ScalerCrop + + - CnnOutputTensor: + type: float + size: [n] + description: | + This control returns a span of floating point values that represent the + output tensors from a Convolutional Neural Network (CNN). The size and + format of this array of values is entirely dependent on the neural + network used, and further post-processing may need to be performed at + the application level to generate the final desired output. This control + is agnostic of the hardware or software used to generate the output + tensors. + + The structure of the span is described by the CnnOutputTensorInfo + control. + + \sa CnnOutputTensorInfo + + - CnnOutputTensorInfo: + type: uint8_t + size: [n] + description: | + This control returns the structure of the CnnOutputTensor. This structure + takes the following form: + + constexpr unsigned int NetworkNameLen = 64; + constexpr unsigned int MaxNumTensors = 16; + constexpr unsigned int MaxNumDimensions = 16; + + struct CnnOutputTensorInfo { + char networkName[NetworkNameLen]; + uint32_t numTensors; + OutputTensorInfo info[MaxNumTensors]; + }; + + with + + struct OutputTensorInfo { + uint32_t tensorDataNum; + uint32_t numDimensions; + uint16_t size[MaxNumDimensions]; + }; + + networkName is the name of the CNN used, + numTensors is the number of output tensors returned, + tensorDataNum gives the number of elements in each output tensor, + numDimensions gives the dimensionality of each output tensor, + size gives the size of each dimension in each output tensor. + + \sa CnnOutputTensor + + - CnnEnableInputTensor: + type: bool + description: | + Boolean to control if the IPA returns the input tensor used by the CNN + to generate the output tensors via the CnnInputTensor control. Because + the input tensor may be relatively large, for efficiency reason avoid + enabling input tensor output unless required for debugging purposes. + + \sa CnnInputTensor + + - CnnInputTensor: + type: uint8_t + size: [n] + description: | + This control returns a span of uint8_t pixel values that represent the + input tensor for a Convolutional Neural Network (CNN). The size and + format of this array of values is entirely dependent on the neural + network used, and further post-processing (e.g. pixel normalisations) may + need to be performed at the application level to generate the final input + image. + + The structure of the span is described by the CnnInputTensorInfo + control. + + \sa CnnInputTensorInfo + + - CnnInputTensorInfo: + type: uint8_t + size: [n] + description: | + This control returns the structure of the CnnInputTensor. This structure + takes the following form: + + constexpr unsigned int NetworkNameLen = 64; + + struct CnnInputTensorInfo { + char networkName[NetworkNameLen]; + uint32_t width; + uint32_t height; + uint32_t numChannels; + }; + + where + + networkName is the name of the CNN used, + width and height are the input tensor image width and height in pixels, + numChannels is the number of channels in the input tensor image. + + \sa CnnInputTensor + + - CnnKpiInfo: + type: int32_t + size: [2] + description: | + This control returns performance metrics for the CNN processing stage. + Two values are returned in this span, the runtime of the CNN/DNN stage + and the DSP stage in milliseconds. ...