[{"id":32717,"web_url":"https://patchwork.libcamera.org/comment/32717/","msgid":"","date":"2024-12-13T10:24:05","subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Naush\n\nThanks for the patch.\n\nOn Fri, 13 Dec 2024 at 09:46, Naushir Patuck wrote:\n>\n> Add the follwing RPi vendor controls to handle Convolutional Neural\n> Network processing:\n>\n> CnnOutputTensor\n> CnnOutputTensorInfo\n> CnnEnableInputTensor\n> CnnInputTensor\n> CnnInputTensorInfo\n> CnnKpiInfo\n>\n> These controls will be used to support the new Raspberry Pi AI Camera,\n> using an IMX500 sensor with on-board neural network processing.\n>\n> Signed-off-by: Naushir Patuck \n> ---\n> src/ipa/rpi/controller/controller.h | 33 +++++++++\n> src/libcamera/control_ids_rpi.yaml | 108 ++++++++++++++++++++++++++++\n> 2 files changed, 141 insertions(+)\n>\n> diff --git a/src/ipa/rpi/controller/controller.h b/src/ipa/rpi/controller/controller.h\n> index 64f93f414524..489188b44d9b 100644\n> --- a/src/ipa/rpi/controller/controller.h\n> +++ b/src/ipa/rpi/controller/controller.h\n> @@ -25,6 +25,39 @@\n>\n> namespace RPiController {\n>\n> +/*\n> + * The following structures are used to export the CNN input/output tensor information\n> + * through the rpi::CnnOutputTensorInfo and rpi::CnnInputTensorInfo controls.\n> + * Applications must cast the span to these structures exactly.\n> + */\n> +static constexpr unsigned int NetworkNameLen = 64;\n> +static constexpr unsigned int MaxNumTensors = 16;\n> +static constexpr unsigned int MaxNumDimensions = 16;\n\nDid I see elsewhere that we like to put \"k\" on the front of constants\nthese days? Not that I'm bothered either way...\n\n> +\n> +struct OutputTensorInfo {\n> + uint32_t tensorDataNum;\n> + uint32_t numDimensions;\n> + uint16_t size[MaxNumDimensions];\n> +};\n> +\n> +struct CnnOutputTensorInfo {\n> + char networkName[NetworkNameLen];\n> + uint32_t numTensors;\n> + OutputTensorInfo info[MaxNumTensors];\n> +};\n> +\n> +struct CnnInputTensorInfo {\n> + char networkName[NetworkNameLen];\n> + uint32_t width;\n> + uint32_t height;\n> + uint32_t numChannels;\n> +};\n> +\n> +struct CnnKpiInfo {\n> + uint32_t dnnRuntime;\n> + uint32_t dspRuntime;\n> +};\n> +\n\nI wondered momentarily whether these should be in a separate header\nfile, but honestly, there are so few I'm happy not to bother with it!\n\n> class Algorithm;\n> typedef std::unique_ptr AlgorithmPtr;\n>\n> diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml\n> index 34bbdfc863c5..c0b5f63df525 100644\n> --- a/src/libcamera/control_ids_rpi.yaml\n> +++ b/src/libcamera/control_ids_rpi.yaml\n> @@ -55,4 +55,112 @@ controls:\n> official libcamera API support for per-stream controls in the future.\n>\n> \\sa ScalerCrop\n> +\n> + - CnnOutputTensor:\n> + type: float\n> + size: [n]\n> + description: |\n> + This control returns a span of floating point values that represent the\n> + output tensors from a Convolutional Neural Network (CNN). The size and\n> + format of this array of values is entirely dependent on the neural\n> + network used, and further post-processing may need to be performed at\n> + the application level to generate the final desired output. This control\n> + is agnostic of the hardware or software used to generate the output\n> + tensors.\n> +\n> + The structure of the span is described by the CnnOutputTensorInfo\n> + control.\n> +\n> + \\sa CnnOutputTensorInfo\n> +\n> + - CnnOutputTensorInfo:\n> + type: uint8_t\n> + size: [n]\n> + description: |\n> + This control returns the structure of the CnnOutputTensor. This structure\n> + takes the following form:\n> +\n> + constexpr unsigned int NetworkNameLen = 64;\n> + constexpr unsigned int MaxNumTensors = 16;\n> + constexpr unsigned int MaxNumDimensions = 16;\n> +\n> + struct CnnOutputTensorInfo {\n> + char networkName[NetworkNameLen];\n> + uint32_t numTensors;\n> + OutputTensorInfo info[MaxNumTensors];\n> + };\n> +\n> + with\n> +\n> + struct OutputTensorInfo {\n> + uint32_t tensorDataNum;\n> + uint32_t numDimensions;\n> + uint16_t size[MaxNumDimensions];\n> + };\n> +\n> + networkName is the name of the CNN used,\n> + numTensors is the number of output tensors returned,\n> + tensorDataNum gives the number of elements in each output tensor,\n> + numDimensions gives the dimensionality of each output tensor,\n> + size gives the size of each dimension in each output tensor.\n> +\n> + \\sa CnnOutputTensor\n> +\n> + - CnnEnableInputTensor:\n> + type: bool\n> + description: |\n> + Boolean to control if the IPA returns the input tensor used by the CNN\n> + to generate the output tensors via the CnnInputTensor control. Because\n> + the input tensor may be relatively large, for efficiency reason avoid\n\ns/reason/reasons/\n\n> + enabling input tensor output unless required for debugging purposes.\n\nActually I found some of this just a bit tricky to parse and\nunderstand. Is \"via the CnnInputTensor control\" maybe superfluous?\nAlso \"input tensor output\" took me a moment. Maybe \"output of the\ninput tensor\" is easier?\n\n> +\n> + \\sa CnnInputTensor\n> +\n> + - CnnInputTensor:\n> + type: uint8_t\n> + size: [n]\n> + description: |\n> + This control returns a span of uint8_t pixel values that represent the\n> + input tensor for a Convolutional Neural Network (CNN). The size and\n> + format of this array of values is entirely dependent on the neural\n> + network used, and further post-processing (e.g. pixel normalisations) may\n> + need to be performed at the application level to generate the final input\n> + image.\n> +\n> + The structure of the span is described by the CnnInputTensorInfo\n> + control.\n> +\n> + \\sa CnnInputTensorInfo\n> +\n> + - CnnInputTensorInfo:\n> + type: uint8_t\n> + size: [n]\n> + description: |\n> + This control returns the structure of the CnnInputTensor. This structure\n> + takes the following form:\n> +\n> + constexpr unsigned int NetworkNameLen = 64;\n> +\n> + struct CnnInputTensorInfo {\n> + char networkName[NetworkNameLen];\n> + uint32_t width;\n> + uint32_t height;\n> + uint32_t numChannels;\n> + };\n> +\n> + where\n> +\n> + networkName is the name of the CNN used,\n> + width and height are the input tensor image width and height in pixels,\n> + numChannels is the number of channels in the input tensor image.\n> +\n> + \\sa CnnInputTensor\n> +\n> + - CnnKpiInfo:\n> + type: int32_t\n> + size: [2]\n> + description: |\n> + This control returns performance metrics for the CNN processing stage.\n> + Two values are returned in this span, the runtime of the CNN/DNN stage\n\nNo particular issue, just wondering why we sometimes have CNN and\nsometimes DNN. Are they the same, really (we define CNN, but did we\never define DNN)? Should we standardise?\n\nMinor edits aside:\n\nReviewed-by: David Plowman \n\nThanks!\nDavid\n\n> + and the DSP stage in milliseconds.\n> ...\n> --\n> 2.43.0\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 97434C32F0\n\tfor ;\n\tFri, 13 Dec 2024 10:24:19 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D444467EE9;\n\tFri, 13 Dec 2024 11:24:18 +0100 (CET)","from mail-qt1-x830.google.com (mail-qt1-x830.google.com\n\t[IPv6:2607:f8b0:4864:20::830])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7C418618AD\n\tfor ;\n\tFri, 13 Dec 2024 11:24:17 +0100 (CET)","by mail-qt1-x830.google.com with SMTP id\n\td75a77b69052e-467a0a6c9fcso19527751cf.3\n\tfor ;\n\tFri, 13 Dec 2024 02:24:17 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"JzIYvREQ\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1734085456; x=1734690256;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=LG3YQ79JuZUnASHkUaQcSFXOVQjWzdu0PX07pXlQcGY=;\n\tb=JzIYvREQB4H4N85b1ytm+g9Dx59E9PvDfd4pI7Il8mGCch6wRduV1yuPde6V9tXN5F\n\tfzrDsf14aHPgwPY0KzNBhdx5j9vIZqFg5zuQdXrUcnrdqoVw1tpB1/uwXZKAkr040TEV\n\tVbaJ8nop1dkrHNZ6mMyhC+5VfMA73wGwf3eEqk5qRw0sY45sBTOMk9+B2tgiXPhv2gHm\n\tQJl4JNG4E/WO57UIxO4xf+JsInJ3QY2AW60TS+91mIXoDJWgahPw+/gMUeJqJVQjIAB1\n\tt0qi/jkGbmLNabrzY5lw9vs9a08YHu1KeWP6r0Q7nuTuVcIscoasRpqoyDVwos1anDkN\n\tbpHg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1734085456; x=1734690256;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=LG3YQ79JuZUnASHkUaQcSFXOVQjWzdu0PX07pXlQcGY=;\n\tb=XdmuXwSi5XOgMYGlzuKcUwkAUxlZDNmbxVJtM5r2eP659p4fQAIeldJ9GURwMxFgO3\n\tU00XXLuzxGerRsH5dqHc14vFU7KnPJ60ZHHlQfBXidHRSqUjths4mRtXWGFICGbb/tAt\n\tDYKLkG10X4jH4bhwUQ5lM1meKu6hJWd8jw9dOD1cLs8rKhTzfzMZWQqSRmiUfKDA7AN9\n\tF1otU/NI3dZzbcJg6AFzC4Rcvj4WfrC6CFQkxjgInQb64Dzr3E+FfLVjHjv9VnJIEjGd\n\tkKC2Tdub4+zZ7uj3VCM5WQN4IiExc6F0XMQdWlxK9sG1gyjs26z8nl3HEeF7QXKzVNA0\n\trsig==","X-Gm-Message-State":"AOJu0Yx22qeochYcs5/2NbeQrPygslDHvVVTbwj4OL8GjbEY24Lzfzre\n\ts8TCWzCWNPLethTAJ9v6ivC9xP8iRhOcJtUFUoswJQR8WBBR8FAPcmft76W596hZAUPhG0/Mv/T\n\tLPDm3LQy/LqE64WVuk0eZ0i3IGjgQk30F/nSM572mK0wU7Lg5vMA=","X-Gm-Gg":"ASbGnctMuRqNoq9gG9AbKb8+JDtJZaLdxrsOAPkZ0m7dsdqMjyjKvxAA2hzaTedl1bF\n\tf9u0CwGBRd7MdLs9WaIWI7zssBlo/yOEI8je0Jg==","X-Google-Smtp-Source":"AGHT+IGIE4awnaZBiEe7+PUfEs6Ss7fqUJnETH/ohTdByThsmh2RwTMSIhjODrPDZuUVfx/QI+K5kET437bPzy15lHg=","X-Received":"by 2002:a05:622a:1e05:b0:467:8651:409a with SMTP id\n\td75a77b69052e-467a582466amr35462741cf.49.1734085456417;\n\tFri, 13 Dec 2024 02:24:16 -0800 (PST)","MIME-Version":"1.0","References":"<20241213094602.2083174-1-naush@raspberrypi.com>\n\t<20241213094602.2083174-5-naush@raspberrypi.com>","In-Reply-To":"<20241213094602.2083174-5-naush@raspberrypi.com>","From":"David Plowman ","Date":"Fri, 13 Dec 2024 10:24:05 +0000","Message-ID":"","Subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32721,"web_url":"https://patchwork.libcamera.org/comment/32721/","msgid":"","date":"2024-12-13T13:34:12","subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi David,\n\nThank you for the feedback on this + all the other patches. I've\nfixed up all the minors for v2, and also commented inline below.\n\nOn Fri, 13 Dec 2024 at 10:24, David Plowman\n wrote:\n>\n> Hi Naush\n>\n> Thanks for the patch.\n>\n> On Fri, 13 Dec 2024 at 09:46, Naushir Patuck wrote:\n> >\n> > Add the follwing RPi vendor controls to handle Convolutional Neural\n> > Network processing:\n> >\n> > CnnOutputTensor\n> > CnnOutputTensorInfo\n> > CnnEnableInputTensor\n> > CnnInputTensor\n> > CnnInputTensorInfo\n> > CnnKpiInfo\n> >\n> > These controls will be used to support the new Raspberry Pi AI Camera,\n> > using an IMX500 sensor with on-board neural network processing.\n> >\n> > Signed-off-by: Naushir Patuck \n> > ---\n> > src/ipa/rpi/controller/controller.h | 33 +++++++++\n> > src/libcamera/control_ids_rpi.yaml | 108 ++++++++++++++++++++++++++++\n> > 2 files changed, 141 insertions(+)\n> >\n> > diff --git a/src/ipa/rpi/controller/controller.h b/src/ipa/rpi/controller/controller.h\n> > index 64f93f414524..489188b44d9b 100644\n> > --- a/src/ipa/rpi/controller/controller.h\n> > +++ b/src/ipa/rpi/controller/controller.h\n> > @@ -25,6 +25,39 @@\n> >\n> > namespace RPiController {\n> >\n> > +/*\n> > + * The following structures are used to export the CNN input/output tensor information\n> > + * through the rpi::CnnOutputTensorInfo and rpi::CnnInputTensorInfo controls.\n> > + * Applications must cast the span to these structures exactly.\n> > + */\n> > +static constexpr unsigned int NetworkNameLen = 64;\n> > +static constexpr unsigned int MaxNumTensors = 16;\n> > +static constexpr unsigned int MaxNumDimensions = 16;\n>\n> Did I see elsewhere that we like to put \"k\" on the front of constants\n> these days? Not that I'm bothered either way...\n>\n> > +\n> > +struct OutputTensorInfo {\n> > + uint32_t tensorDataNum;\n> > + uint32_t numDimensions;\n> > + uint16_t size[MaxNumDimensions];\n> > +};\n> > +\n> > +struct CnnOutputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t numTensors;\n> > + OutputTensorInfo info[MaxNumTensors];\n> > +};\n> > +\n> > +struct CnnInputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t width;\n> > + uint32_t height;\n> > + uint32_t numChannels;\n> > +};\n> > +\n> > +struct CnnKpiInfo {\n> > + uint32_t dnnRuntime;\n> > + uint32_t dspRuntime;\n> > +};\n> > +\n>\n> I wondered momentarily whether these should be in a separate header\n> file, but honestly, there are so few I'm happy not to bother with it!\n\nI thought about this, but the definitions are small I left it in here.\n\n>\n> > class Algorithm;\n> > typedef std::unique_ptr AlgorithmPtr;\n> >\n> > diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml\n> > index 34bbdfc863c5..c0b5f63df525 100644\n> > --- a/src/libcamera/control_ids_rpi.yaml\n> > +++ b/src/libcamera/control_ids_rpi.yaml\n> > @@ -55,4 +55,112 @@ controls:\n> > official libcamera API support for per-stream controls in the future.\n> >\n> > \\sa ScalerCrop\n> > +\n> > + - CnnOutputTensor:\n> > + type: float\n> > + size: [n]\n> > + description: |\n> > + This control returns a span of floating point values that represent the\n> > + output tensors from a Convolutional Neural Network (CNN). The size and\n> > + format of this array of values is entirely dependent on the neural\n> > + network used, and further post-processing may need to be performed at\n> > + the application level to generate the final desired output. This control\n> > + is agnostic of the hardware or software used to generate the output\n> > + tensors.\n> > +\n> > + The structure of the span is described by the CnnOutputTensorInfo\n> > + control.\n> > +\n> > + \\sa CnnOutputTensorInfo\n> > +\n> > + - CnnOutputTensorInfo:\n> > + type: uint8_t\n> > + size: [n]\n> > + description: |\n> > + This control returns the structure of the CnnOutputTensor. This structure\n> > + takes the following form:\n> > +\n> > + constexpr unsigned int NetworkNameLen = 64;\n> > + constexpr unsigned int MaxNumTensors = 16;\n> > + constexpr unsigned int MaxNumDimensions = 16;\n> > +\n> > + struct CnnOutputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t numTensors;\n> > + OutputTensorInfo info[MaxNumTensors];\n> > + };\n> > +\n> > + with\n> > +\n> > + struct OutputTensorInfo {\n> > + uint32_t tensorDataNum;\n> > + uint32_t numDimensions;\n> > + uint16_t size[MaxNumDimensions];\n> > + };\n> > +\n> > + networkName is the name of the CNN used,\n> > + numTensors is the number of output tensors returned,\n> > + tensorDataNum gives the number of elements in each output tensor,\n> > + numDimensions gives the dimensionality of each output tensor,\n> > + size gives the size of each dimension in each output tensor.\n> > +\n> > + \\sa CnnOutputTensor\n> > +\n> > + - CnnEnableInputTensor:\n> > + type: bool\n> > + description: |\n> > + Boolean to control if the IPA returns the input tensor used by the CNN\n> > + to generate the output tensors via the CnnInputTensor control. Because\n> > + the input tensor may be relatively large, for efficiency reason avoid\n>\n> s/reason/reasons/\n>\n> > + enabling input tensor output unless required for debugging purposes.\n>\n> Actually I found some of this just a bit tricky to parse and\n> understand. Is \"via the CnnInputTensor control\" maybe superfluous?\n> Also \"input tensor output\" took me a moment. Maybe \"output of the\n> input tensor\" is easier?\n\nAgree, that did not read well. I've reworded it to:\n\n Boolean to control if the IPA returns (through metadata) the input\n tensor used by the CNN to generate the output tensors. Because the input\n tensor may be relatively large, for efficiency reasons avoid returning\n the input tensor unless required for debugging purposes.\n\n>\n> > +\n> > + \\sa CnnInputTensor\n> > +\n> > + - CnnInputTensor:\n> > + type: uint8_t\n> > + size: [n]\n> > + description: |\n> > + This control returns a span of uint8_t pixel values that represent the\n> > + input tensor for a Convolutional Neural Network (CNN). The size and\n> > + format of this array of values is entirely dependent on the neural\n> > + network used, and further post-processing (e.g. pixel normalisations) may\n> > + need to be performed at the application level to generate the final input\n> > + image.\n> > +\n> > + The structure of the span is described by the CnnInputTensorInfo\n> > + control.\n> > +\n> > + \\sa CnnInputTensorInfo\n> > +\n> > + - CnnInputTensorInfo:\n> > + type: uint8_t\n> > + size: [n]\n> > + description: |\n> > + This control returns the structure of the CnnInputTensor. This structure\n> > + takes the following form:\n> > +\n> > + constexpr unsigned int NetworkNameLen = 64;\n> > +\n> > + struct CnnInputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t width;\n> > + uint32_t height;\n> > + uint32_t numChannels;\n> > + };\n> > +\n> > + where\n> > +\n> > + networkName is the name of the CNN used,\n> > + width and height are the input tensor image width and height in pixels,\n> > + numChannels is the number of channels in the input tensor image.\n> > +\n> > + \\sa CnnInputTensor\n> > +\n> > + - CnnKpiInfo:\n> > + type: int32_t\n> > + size: [2]\n> > + description: |\n> > + This control returns performance metrics for the CNN processing stage.\n> > + Two values are returned in this span, the runtime of the CNN/DNN stage\n>\n> No particular issue, just wondering why we sometimes have CNN and\n> sometimes DNN. Are they the same, really (we define CNN, but did we\n> ever define DNN)? Should we standardise?\n\nI was following one particular vendor's terminology here - but for no\nreal reason. I've replaced it with CNN.\n\nRegards,\nNaush\n\n\n> Minor edits aside:\n>\n> Reviewed-by: David Plowman \n>\n> Thanks!\n> David\n>\n> > + and the DSP stage in milliseconds.\n> > ...\n> > --\n> > 2.43.0\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1C866C32F0\n\tfor ;\n\tFri, 13 Dec 2024 13:34:51 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 34B4667EFC;\n\tFri, 13 Dec 2024 14:34:50 +0100 (CET)","from mail-yb1-xb35.google.com (mail-yb1-xb35.google.com\n\t[IPv6:2607:f8b0:4864:20::b35])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C2FC06189C\n\tfor ;\n\tFri, 13 Dec 2024 14:34:47 +0100 (CET)","by mail-yb1-xb35.google.com with SMTP id\n\t3f1490d57ef6-e3a0d9aab47so224329276.1\n\tfor ;\n\tFri, 13 Dec 2024 05:34:47 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"tkxTqpwn\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1734096886; x=1734701686;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=1sQUwCCbh8Y97ZhP/QiTUHLBE1Sd52FDOfuvCr6DGX8=;\n\tb=tkxTqpwnMBGj6iI3G0uBbEVu8VkXADMzrZV8XeqtTKF87qim8eIbdMupf4KxK0Lo8v\n\tnHqsrMdHVzZGtgQSOKvepM2OGLXIOjTh2FvUMUcAG2XZw/GH9NWjqqYGcQ8mKpqR9ik0\n\tPdUQWCKmwY6VhfA/NUCJwskC2315Y+n9RmG3sTpIJOBkJBYXQDsGsGveHtYbSVKRhCVm\n\t/0zTjbCS+lBzlvNOSBPMgKmm7LJMtzi1dVl474uBs5+rR7cvq4SgI7IGL7G6lkXIlFU6\n\tIO+MLq/UTDyR1J6cKWG/lh6bB9GB7fkW3EcNIHi0R6rL2Zol4Xq5KnRnlCpN4Pj7UfEe\n\tVTIw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1734096886; x=1734701686;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=1sQUwCCbh8Y97ZhP/QiTUHLBE1Sd52FDOfuvCr6DGX8=;\n\tb=LdlaFN19YLtoeK5+Sw3iuHQIdt8BNENC9avMmMkfKyVCzyKsaIVtPBKSUFHEH1dNEw\n\tcz9fORJNH1gkJdrPTaZjUAxXR2dPZZu36ckS4JsQpabyHNhjpwTB/celHKU0apHFaDDX\n\tFnjSRsVhM9yImbtdo9U1UzAfe4POcXRok2MNFbDxJhjnzWuenXdc0G6drashpNEL0nBJ\n\tOQp0LOxyM3nJMBRGKq1NgSas9Tw8D/X42kPcFiKc+VdGxu7XQMT5kRZf9D9TOPwizi/U\n\t4PbG3quy3xghe1+mrEQjs51YI5PN6eKT0Jp1gS1tNcz2H8IpjbQNvtrrbcwEZ0Eu0Spu\n\tV2zQ==","X-Gm-Message-State":"AOJu0Yy2hhNcu9L7JTTIjQb67EijxNzV+FDusNFo0Xpjs/GDG6lR4PLQ\n\t9t6S22Twsa8Vo/WjZtRs3zOi21KdtL6zskafY6LG8fvWCeInYIvaUZXFgSGDYKrXh/axo9MagDM\n\tl/fK0anyonueh9rTKr1cF28Of2LMhj1Mr1IM3kA==","X-Gm-Gg":"ASbGncta/bTHbeoS871cIA7BRGDmgnPNrMd8iVT4fgmLrvF1PH4SEMDN70UCGmGN8uK\n\tdjEts1SxDE+Kmjmu6IbEuEBujgi5xsx3VjC4LSg/QRr/kCeT5d7jDqEpZEsCu2AREA2C7","X-Google-Smtp-Source":"AGHT+IGbRgMc42Bd81b/XbtgwdbqY2Ahw+CHOeYdEa215bCXyWf1LXhCXi+C/nC1kZYbS5bZjqR2EWqFmj2cY+jSj6w=","X-Received":"by 2002:a05:6902:988:b0:e39:8f3e:30a0 with SMTP id\n\t3f1490d57ef6-e434f83f2a4mr1022347276.6.1734096886534; Fri, 13 Dec 2024\n\t05:34:46 -0800 (PST)","MIME-Version":"1.0","References":"<20241213094602.2083174-1-naush@raspberrypi.com>\n\t<20241213094602.2083174-5-naush@raspberrypi.com>\n\t","In-Reply-To":"","From":"Naushir Patuck ","Date":"Fri, 13 Dec 2024 13:34:12 +0000","Message-ID":"","Subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","To":"David Plowman ","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32729,"web_url":"https://patchwork.libcamera.org/comment/32729/","msgid":"<20241215163753.GD9975@pendragon.ideasonboard.com>","date":"2024-12-15T16:37:53","subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Naush,\n\nThank you for the patch.\n\nOn Fri, Dec 13, 2024 at 09:38:27AM +0000, Naushir Patuck wrote:\n> Add the follwing RPi vendor controls to handle Convolutional Neural\n> Network processing:\n> \n> CnnOutputTensor\n> CnnOutputTensorInfo\n> CnnEnableInputTensor\n> CnnInputTensor\n> CnnInputTensorInfo\n> CnnKpiInfo\n> \n> These controls will be used to support the new Raspberry Pi AI Camera,\n> using an IMX500 sensor with on-board neural network processing.\n\nI think those controls should be reviewed in the context of the IMX500\nkernel driver. That would also help with the libcamera policy that\ndrivers need to be on their way to mainline. When do you plan to post it\nfor review on the linux-media mailing list ?\n\n> Signed-off-by: Naushir Patuck \n> ---\n> src/ipa/rpi/controller/controller.h | 33 +++++++++\n> src/libcamera/control_ids_rpi.yaml | 108 ++++++++++++++++++++++++++++\n> 2 files changed, 141 insertions(+)\n> \n> diff --git a/src/ipa/rpi/controller/controller.h b/src/ipa/rpi/controller/controller.h\n> index 64f93f414524..489188b44d9b 100644\n> --- a/src/ipa/rpi/controller/controller.h\n> +++ b/src/ipa/rpi/controller/controller.h\n> @@ -25,6 +25,39 @@\n> \n> namespace RPiController {\n> \n> +/*\n> + * The following structures are used to export the CNN input/output tensor information\n> + * through the rpi::CnnOutputTensorInfo and rpi::CnnInputTensorInfo controls.\n> + * Applications must cast the span to these structures exactly.\n> + */\n> +static constexpr unsigned int NetworkNameLen = 64;\n> +static constexpr unsigned int MaxNumTensors = 16;\n> +static constexpr unsigned int MaxNumDimensions = 16;\n> +\n> +struct OutputTensorInfo {\n> +\tuint32_t tensorDataNum;\n> +\tuint32_t numDimensions;\n> +\tuint16_t size[MaxNumDimensions];\n> +};\n> +\n> +struct CnnOutputTensorInfo {\n> +\tchar networkName[NetworkNameLen];\n> +\tuint32_t numTensors;\n> +\tOutputTensorInfo info[MaxNumTensors];\n> +};\n> +\n> +struct CnnInputTensorInfo {\n> +\tchar networkName[NetworkNameLen];\n> +\tuint32_t width;\n> +\tuint32_t height;\n> +\tuint32_t numChannels;\n> +};\n> +\n> +struct CnnKpiInfo {\n> +\tuint32_t dnnRuntime;\n> +\tuint32_t dspRuntime;\n> +};\n> +\n> class Algorithm;\n> typedef std::unique_ptr AlgorithmPtr;\n> \n> diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml\n> index 34bbdfc863c5..c0b5f63df525 100644\n> --- a/src/libcamera/control_ids_rpi.yaml\n> +++ b/src/libcamera/control_ids_rpi.yaml\n> @@ -55,4 +55,112 @@ controls:\n> official libcamera API support for per-stream controls in the future.\n> \n> \\sa ScalerCrop\n> +\n> + - CnnOutputTensor:\n> + type: float\n> + size: [n]\n> + description: |\n> + This control returns a span of floating point values that represent the\n> + output tensors from a Convolutional Neural Network (CNN). The size and\n> + format of this array of values is entirely dependent on the neural\n> + network used, and further post-processing may need to be performed at\n> + the application level to generate the final desired output. This control\n> + is agnostic of the hardware or software used to generate the output\n> + tensors.\n> +\n> + The structure of the span is described by the CnnOutputTensorInfo\n> + control.\n> +\n> + \\sa CnnOutputTensorInfo\n> +\n> + - CnnOutputTensorInfo:\n> + type: uint8_t\n> + size: [n]\n> + description: |\n> + This control returns the structure of the CnnOutputTensor. This structure\n> + takes the following form:\n> +\n> + constexpr unsigned int NetworkNameLen = 64;\n> + constexpr unsigned int MaxNumTensors = 16;\n> + constexpr unsigned int MaxNumDimensions = 16;\n> +\n> + struct CnnOutputTensorInfo {\n> + char networkName[NetworkNameLen];\n> + uint32_t numTensors;\n> + OutputTensorInfo info[MaxNumTensors];\n> + };\n> +\n> + with\n> +\n> + struct OutputTensorInfo {\n> + uint32_t tensorDataNum;\n> + uint32_t numDimensions;\n> + uint16_t size[MaxNumDimensions];\n> + };\n> +\n> + networkName is the name of the CNN used,\n> + numTensors is the number of output tensors returned,\n> + tensorDataNum gives the number of elements in each output tensor,\n> + numDimensions gives the dimensionality of each output tensor,\n> + size gives the size of each dimension in each output tensor.\n> +\n> + \\sa CnnOutputTensor\n> +\n> + - CnnEnableInputTensor:\n> + type: bool\n> + description: |\n> + Boolean to control if the IPA returns the input tensor used by the CNN\n> + to generate the output tensors via the CnnInputTensor control. Because\n> + the input tensor may be relatively large, for efficiency reason avoid\n> + enabling input tensor output unless required for debugging purposes.\n> +\n> + \\sa CnnInputTensor\n> +\n> + - CnnInputTensor:\n> + type: uint8_t\n> + size: [n]\n> + description: |\n> + This control returns a span of uint8_t pixel values that represent the\n> + input tensor for a Convolutional Neural Network (CNN). The size and\n> + format of this array of values is entirely dependent on the neural\n> + network used, and further post-processing (e.g. pixel normalisations) may\n> + need to be performed at the application level to generate the final input\n> + image.\n> +\n> + The structure of the span is described by the CnnInputTensorInfo\n> + control.\n> +\n> + \\sa CnnInputTensorInfo\n> +\n> + - CnnInputTensorInfo:\n> + type: uint8_t\n> + size: [n]\n> + description: |\n> + This control returns the structure of the CnnInputTensor. This structure\n> + takes the following form:\n> +\n> + constexpr unsigned int NetworkNameLen = 64;\n> +\n> + struct CnnInputTensorInfo {\n> + char networkName[NetworkNameLen];\n> + uint32_t width;\n> + uint32_t height;\n> + uint32_t numChannels;\n> + };\n> +\n> + where\n> +\n> + networkName is the name of the CNN used,\n> + width and height are the input tensor image width and height in pixels,\n> + numChannels is the number of channels in the input tensor image.\n> +\n> + \\sa CnnInputTensor\n> +\n> + - CnnKpiInfo:\n> + type: int32_t\n> + size: [2]\n> + description: |\n> + This control returns performance metrics for the CNN processing stage.\n> + Two values are returned in this span, the runtime of the CNN/DNN stage\n> + and the DSP stage in milliseconds.\n> ...","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 6C0EBC32F6\n\tfor ;\n\tSun, 15 Dec 2024 16:38:13 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4580E67F03;\n\tSun, 15 Dec 2024 17:38:12 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DBA6467EEE\n\tfor ;\n\tSun, 15 Dec 2024 17:38:09 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 946577E9;\n\tSun, 15 Dec 2024 17:37:33 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"IqduHSaD\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1734280653;\n\tbh=73PhTAH+GKAJ799PX8DYHh0OIatRcSbwBl24uCZeoVw=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=IqduHSaDEqHbGVYVOmKoab0Xy6xAdBg5i6VBvvm+s69JR9+Nz57eHZAjjAYbHvF4r\n\tzCKXMqQn58vmoFKzOfRZzb6zB/zlzX4hmy1LeMWPKn78K+hI+xWbWGGNE+BkvaLoix\n\tzEwjHM8QSSY0EdP98kgvF0MHXQrkmwyO75l44DmQ=","Date":"Sun, 15 Dec 2024 18:37:53 +0200","From":"Laurent Pinchart ","To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","Message-ID":"<20241215163753.GD9975@pendragon.ideasonboard.com>","References":"<20241213094602.2083174-1-naush@raspberrypi.com>\n\t<20241213094602.2083174-5-naush@raspberrypi.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20241213094602.2083174-5-naush@raspberrypi.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32755,"web_url":"https://patchwork.libcamera.org/comment/32755/","msgid":"","date":"2024-12-16T10:11:28","subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Laurent,\n\nOn Sun, 15 Dec 2024 at 16:38, Laurent Pinchart\n wrote:\n>\n> Hi Naush,\n>\n> Thank you for the patch.\n>\n> On Fri, Dec 13, 2024 at 09:38:27AM +0000, Naushir Patuck wrote:\n> > Add the follwing RPi vendor controls to handle Convolutional Neural\n> > Network processing:\n> >\n> > CnnOutputTensor\n> > CnnOutputTensorInfo\n> > CnnEnableInputTensor\n> > CnnInputTensor\n> > CnnInputTensorInfo\n> > CnnKpiInfo\n> >\n> > These controls will be used to support the new Raspberry Pi AI Camera,\n> > using an IMX500 sensor with on-board neural network processing.\n>\n> I think those controls should be reviewed in the context of the IMX500\n> kernel driver. That would also help with the libcamera policy that\n> drivers need to be on their way to mainline. When do you plan to post it\n> for review on the linux-media mailing list ?\n\nThe intention of these controls was to avoid tying them to the IMX500\nspecifically and be generic. Of course the only user of these\ncurrently would be the imx500, but there is no reliance on e.g. the\nIMX500 camera helper.\n\nWith regards to upstreaming, as soon as we have completed the streams\nAPI, I'll be posting the imx500, imx477 and imx708 drivers to\nlinux-media. However I have to be realistic with everyone, the IMX500\ndriver with neural network functionality has close to zero chance of\nbeing accepted upstream. We rely on closed firmware blobs to drive\nthe DSP, the models are also closed blobs that are made with closed\nsource (but freely available) software, and the output stream\nstructure cannot be documented as it is network dependent.\n\n So as not to waste everyone's time, I'll only be posting the imaging\npart of the imx500 driver for upstream. I can understand if this means\nyou don't want to merge this patch upstream. Let me know if you want\nthis patch removed, and we can get the other patches in this series\nmerged.\n\nRegards,\nNaush\n\n\n>\n> > Signed-off-by: Naushir Patuck \n> > ---\n> > src/ipa/rpi/controller/controller.h | 33 +++++++++\n> > src/libcamera/control_ids_rpi.yaml | 108 ++++++++++++++++++++++++++++\n> > 2 files changed, 141 insertions(+)\n> >\n> > diff --git a/src/ipa/rpi/controller/controller.h b/src/ipa/rpi/controller/controller.h\n> > index 64f93f414524..489188b44d9b 100644\n> > --- a/src/ipa/rpi/controller/controller.h\n> > +++ b/src/ipa/rpi/controller/controller.h\n> > @@ -25,6 +25,39 @@\n> >\n> > namespace RPiController {\n> >\n> > +/*\n> > + * The following structures are used to export the CNN input/output tensor information\n> > + * through the rpi::CnnOutputTensorInfo and rpi::CnnInputTensorInfo controls.\n> > + * Applications must cast the span to these structures exactly.\n> > + */\n> > +static constexpr unsigned int NetworkNameLen = 64;\n> > +static constexpr unsigned int MaxNumTensors = 16;\n> > +static constexpr unsigned int MaxNumDimensions = 16;\n> > +\n> > +struct OutputTensorInfo {\n> > + uint32_t tensorDataNum;\n> > + uint32_t numDimensions;\n> > + uint16_t size[MaxNumDimensions];\n> > +};\n> > +\n> > +struct CnnOutputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t numTensors;\n> > + OutputTensorInfo info[MaxNumTensors];\n> > +};\n> > +\n> > +struct CnnInputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t width;\n> > + uint32_t height;\n> > + uint32_t numChannels;\n> > +};\n> > +\n> > +struct CnnKpiInfo {\n> > + uint32_t dnnRuntime;\n> > + uint32_t dspRuntime;\n> > +};\n> > +\n> > class Algorithm;\n> > typedef std::unique_ptr AlgorithmPtr;\n> >\n> > diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml\n> > index 34bbdfc863c5..c0b5f63df525 100644\n> > --- a/src/libcamera/control_ids_rpi.yaml\n> > +++ b/src/libcamera/control_ids_rpi.yaml\n> > @@ -55,4 +55,112 @@ controls:\n> > official libcamera API support for per-stream controls in the future.\n> >\n> > \\sa ScalerCrop\n> > +\n> > + - CnnOutputTensor:\n> > + type: float\n> > + size: [n]\n> > + description: |\n> > + This control returns a span of floating point values that represent the\n> > + output tensors from a Convolutional Neural Network (CNN). The size and\n> > + format of this array of values is entirely dependent on the neural\n> > + network used, and further post-processing may need to be performed at\n> > + the application level to generate the final desired output. This control\n> > + is agnostic of the hardware or software used to generate the output\n> > + tensors.\n> > +\n> > + The structure of the span is described by the CnnOutputTensorInfo\n> > + control.\n> > +\n> > + \\sa CnnOutputTensorInfo\n> > +\n> > + - CnnOutputTensorInfo:\n> > + type: uint8_t\n> > + size: [n]\n> > + description: |\n> > + This control returns the structure of the CnnOutputTensor. This structure\n> > + takes the following form:\n> > +\n> > + constexpr unsigned int NetworkNameLen = 64;\n> > + constexpr unsigned int MaxNumTensors = 16;\n> > + constexpr unsigned int MaxNumDimensions = 16;\n> > +\n> > + struct CnnOutputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t numTensors;\n> > + OutputTensorInfo info[MaxNumTensors];\n> > + };\n> > +\n> > + with\n> > +\n> > + struct OutputTensorInfo {\n> > + uint32_t tensorDataNum;\n> > + uint32_t numDimensions;\n> > + uint16_t size[MaxNumDimensions];\n> > + };\n> > +\n> > + networkName is the name of the CNN used,\n> > + numTensors is the number of output tensors returned,\n> > + tensorDataNum gives the number of elements in each output tensor,\n> > + numDimensions gives the dimensionality of each output tensor,\n> > + size gives the size of each dimension in each output tensor.\n> > +\n> > + \\sa CnnOutputTensor\n> > +\n> > + - CnnEnableInputTensor:\n> > + type: bool\n> > + description: |\n> > + Boolean to control if the IPA returns the input tensor used by the CNN\n> > + to generate the output tensors via the CnnInputTensor control. Because\n> > + the input tensor may be relatively large, for efficiency reason avoid\n> > + enabling input tensor output unless required for debugging purposes.\n> > +\n> > + \\sa CnnInputTensor\n> > +\n> > + - CnnInputTensor:\n> > + type: uint8_t\n> > + size: [n]\n> > + description: |\n> > + This control returns a span of uint8_t pixel values that represent the\n> > + input tensor for a Convolutional Neural Network (CNN). The size and\n> > + format of this array of values is entirely dependent on the neural\n> > + network used, and further post-processing (e.g. pixel normalisations) may\n> > + need to be performed at the application level to generate the final input\n> > + image.\n> > +\n> > + The structure of the span is described by the CnnInputTensorInfo\n> > + control.\n> > +\n> > + \\sa CnnInputTensorInfo\n> > +\n> > + - CnnInputTensorInfo:\n> > + type: uint8_t\n> > + size: [n]\n> > + description: |\n> > + This control returns the structure of the CnnInputTensor. This structure\n> > + takes the following form:\n> > +\n> > + constexpr unsigned int NetworkNameLen = 64;\n> > +\n> > + struct CnnInputTensorInfo {\n> > + char networkName[NetworkNameLen];\n> > + uint32_t width;\n> > + uint32_t height;\n> > + uint32_t numChannels;\n> > + };\n> > +\n> > + where\n> > +\n> > + networkName is the name of the CNN used,\n> > + width and height are the input tensor image width and height in pixels,\n> > + numChannels is the number of channels in the input tensor image.\n> > +\n> > + \\sa CnnInputTensor\n> > +\n> > + - CnnKpiInfo:\n> > + type: int32_t\n> > + size: [2]\n> > + description: |\n> > + This control returns performance metrics for the CNN processing stage.\n> > + Two values are returned in this span, the runtime of the CNN/DNN stage\n> > + and the DSP stage in milliseconds.\n> > ...\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D510DC32F9\n\tfor ;\n\tMon, 16 Dec 2024 10:12:08 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id ED94367F4B;\n\tMon, 16 Dec 2024 11:12:07 +0100 (CET)","from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com\n\t[IPv6:2607:f8b0:4864:20::112d])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B488867F37\n\tfor ;\n\tMon, 16 Dec 2024 11:12:05 +0100 (CET)","by mail-yw1-x112d.google.com with SMTP id\n\t00721157ae682-6f01ae02e7fso3166727b3.3\n\tfor ;\n\tMon, 16 Dec 2024 02:12:05 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"bDPAUp34\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1734343924; x=1734948724;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=M8VzV6iIKiNWUp6mIGK6TyhgRJ3um2NVWKH8OLdWKjs=;\n\tb=bDPAUp34RFcK2rvCi0le4hNvFNDhPG2srVFyZvME3FdPndy34Gj0Z84nwsjZ8M3SLp\n\tD+g8xDnYvFuHRRLwA5j9A6PK9IiQ6cxc+aALEeXmae+UXXSK3UaA0J41N8N2GxLsRIQB\n\tycQk5YAUb5UtyMMuDSP2vypO1DIeec+wCBcRIcPydAN6TPrvxy5D9j4d/7tcBPxxb9nY\n\tGpi2PhvI2OQ1641cBgA21mrQFHb4W3CZLTKYU13ohzvnnC3Jnc3QF1w1F+k99opxrrHz\n\t8zvyD3LC2tfgfSCM2WaU6IwROPQ4NgOWzBc2P3bFowBlSI/uioMHfhA/roIxg1XaXPQz\n\tTpqg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1734343924; x=1734948724;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=M8VzV6iIKiNWUp6mIGK6TyhgRJ3um2NVWKH8OLdWKjs=;\n\tb=EeEMwS4wUaquo2GFXm1NL7OQTJF3cMo72y1oyi5LGAVnEoHuHoMJlJn8Oo7FIozs0Z\n\tz63f5Z17xS6TvjtCaC3LCy4RXRtS7Fqb7lrUPbeDF6kzAvUP942hzJLH4yScdY9IxHGx\n\t11TSuoWWQyN9mCjl/ehipNAVPt7Q/m8lFhferMsH9teS79xi554KhUni3Ua6ZzGch2rK\n\taP+N0dFbtxaU/VAT4P38e4q+pElu9oXf/CEai8E9shKPbtYpL7QyqY3YrVsEn2OBg1F2\n\trYVMAHFllZwKC2DjociXM5fhqrzK8+AIPKo2rCxvhHR/8fqWRxLLc1eNamojpib3G8LK\n\t5OKw==","X-Gm-Message-State":"AOJu0YxlSuT6OwWTL4iZntZgP/AXgKTJ5iXXhrwHanf8DV+2deHl9Ybu\n\tiKiWzCJtOUmODbL/wfwKcrOlLHyQj/++W/egaURGl291r4aRaV+HDSNyK2djU3Su0TdlBIqZffB\n\t9fKff1OreSjYVJfA7jRRjEhMxk1Ze6aDRavlKRw==","X-Gm-Gg":"ASbGncsQODwK004wtowWWPZFRY3ujHGAW0k/jkBNJq+B+Pj05u5Wgskuk1h8EK1+JjI\n\t6Ytags1aYIMDdlv3gY3oTfFqQMQ4K1YvTM3o1tPVFj4D5bYRxRDDAEII/29iXsXLA4BV4","X-Google-Smtp-Source":"AGHT+IFuDE6Chy9gRoVWHrmLxcnVCO10BOMSAMO4O/hvgUKE0LNxo8FrTGQl9lmRsy6xjRcfBGMGPhi1Uiup7yPss2g=","X-Received":"by 2002:a05:6902:2203:b0:e4c:66ab:ed41 with SMTP id\n\t3f1490d57ef6-e4c66abf1abmr1381137276.2.1734343924321; Mon, 16 Dec 2024\n\t02:12:04 -0800 (PST)","MIME-Version":"1.0","References":"<20241213094602.2083174-1-naush@raspberrypi.com>\n\t<20241213094602.2083174-5-naush@raspberrypi.com>\n\t<20241215163753.GD9975@pendragon.ideasonboard.com>","In-Reply-To":"<20241215163753.GD9975@pendragon.ideasonboard.com>","From":"Naushir Patuck ","Date":"Mon, 16 Dec 2024 10:11:28 +0000","Message-ID":"","Subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","To":"Laurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32856,"web_url":"https://patchwork.libcamera.org/comment/32856/","msgid":"<20241217230005.GM23470@pendragon.ideasonboard.com>","date":"2024-12-17T23:00:05","subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Naush,\n\nOn Mon, Dec 16, 2024 at 10:11:28AM +0000, Naushir Patuck wrote:\n> On Sun, 15 Dec 2024 at 16:38, Laurent Pinchart wrote:\n> > On Fri, Dec 13, 2024 at 09:38:27AM +0000, Naushir Patuck wrote:\n> > > Add the follwing RPi vendor controls to handle Convolutional Neural\n> > > Network processing:\n> > >\n> > > CnnOutputTensor\n> > > CnnOutputTensorInfo\n> > > CnnEnableInputTensor\n> > > CnnInputTensor\n> > > CnnInputTensorInfo\n> > > CnnKpiInfo\n> > >\n> > > These controls will be used to support the new Raspberry Pi AI Camera,\n> > > using an IMX500 sensor with on-board neural network processing.\n> >\n> > I think those controls should be reviewed in the context of the IMX500\n> > kernel driver. That would also help with the libcamera policy that\n> > drivers need to be on their way to mainline. When do you plan to post it\n> > for review on the linux-media mailing list ?\n> \n> The intention of these controls was to avoid tying them to the IMX500\n> specifically and be generic. Of course the only user of these\n> currently would be the imx500, but there is no reliance on e.g. the\n> IMX500 camera helper.\n> \n> With regards to upstreaming, as soon as we have completed the streams\n> API, I'll be posting the imx500, imx477 and imx708 drivers to\n> linux-media. However I have to be realistic with everyone, the IMX500\n> driver with neural network functionality has close to zero chance of\n> being accepted upstream. We rely on closed firmware blobs to drive\n> the DSP, the models are also closed blobs that are made with closed\n> source (but freely available) software, and the output stream\n> structure cannot be documented as it is network dependent.\n\nClose to zero is small, but I wouldn't entirely rule it out. Maybe not\nright now, but let's see how the situation will evolve.\n\n> So as not to waste everyone's time, I'll only be posting the imaging\n> part of the imx500 driver for upstream. I can understand if this means\n> you don't want to merge this patch upstream. Let me know if you want\n> this patch removed, and we can get the other patches in this series\n> merged.\n\nFor the time being that would be preferable. I'm sorry about that.\n\n> > > Signed-off-by: Naushir Patuck \n> > > ---\n> > > src/ipa/rpi/controller/controller.h | 33 +++++++++\n> > > src/libcamera/control_ids_rpi.yaml | 108 ++++++++++++++++++++++++++++\n> > > 2 files changed, 141 insertions(+)\n> > >\n> > > diff --git a/src/ipa/rpi/controller/controller.h b/src/ipa/rpi/controller/controller.h\n> > > index 64f93f414524..489188b44d9b 100644\n> > > --- a/src/ipa/rpi/controller/controller.h\n> > > +++ b/src/ipa/rpi/controller/controller.h\n> > > @@ -25,6 +25,39 @@\n> > >\n> > > namespace RPiController {\n> > >\n> > > +/*\n> > > + * The following structures are used to export the CNN input/output tensor information\n> > > + * through the rpi::CnnOutputTensorInfo and rpi::CnnInputTensorInfo controls.\n> > > + * Applications must cast the span to these structures exactly.\n> > > + */\n> > > +static constexpr unsigned int NetworkNameLen = 64;\n> > > +static constexpr unsigned int MaxNumTensors = 16;\n> > > +static constexpr unsigned int MaxNumDimensions = 16;\n> > > +\n> > > +struct OutputTensorInfo {\n> > > + uint32_t tensorDataNum;\n> > > + uint32_t numDimensions;\n> > > + uint16_t size[MaxNumDimensions];\n> > > +};\n> > > +\n> > > +struct CnnOutputTensorInfo {\n> > > + char networkName[NetworkNameLen];\n> > > + uint32_t numTensors;\n> > > + OutputTensorInfo info[MaxNumTensors];\n> > > +};\n> > > +\n> > > +struct CnnInputTensorInfo {\n> > > + char networkName[NetworkNameLen];\n> > > + uint32_t width;\n> > > + uint32_t height;\n> > > + uint32_t numChannels;\n> > > +};\n> > > +\n> > > +struct CnnKpiInfo {\n> > > + uint32_t dnnRuntime;\n> > > + uint32_t dspRuntime;\n> > > +};\n> > > +\n> > > class Algorithm;\n> > > typedef std::unique_ptr AlgorithmPtr;\n> > >\n> > > diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml\n> > > index 34bbdfc863c5..c0b5f63df525 100644\n> > > --- a/src/libcamera/control_ids_rpi.yaml\n> > > +++ b/src/libcamera/control_ids_rpi.yaml\n> > > @@ -55,4 +55,112 @@ controls:\n> > > official libcamera API support for per-stream controls in the future.\n> > >\n> > > \\sa ScalerCrop\n> > > +\n> > > + - CnnOutputTensor:\n> > > + type: float\n> > > + size: [n]\n> > > + description: |\n> > > + This control returns a span of floating point values that represent the\n> > > + output tensors from a Convolutional Neural Network (CNN). The size and\n> > > + format of this array of values is entirely dependent on the neural\n> > > + network used, and further post-processing may need to be performed at\n> > > + the application level to generate the final desired output. This control\n> > > + is agnostic of the hardware or software used to generate the output\n> > > + tensors.\n> > > +\n> > > + The structure of the span is described by the CnnOutputTensorInfo\n> > > + control.\n> > > +\n> > > + \\sa CnnOutputTensorInfo\n> > > +\n> > > + - CnnOutputTensorInfo:\n> > > + type: uint8_t\n> > > + size: [n]\n> > > + description: |\n> > > + This control returns the structure of the CnnOutputTensor. This structure\n> > > + takes the following form:\n> > > +\n> > > + constexpr unsigned int NetworkNameLen = 64;\n> > > + constexpr unsigned int MaxNumTensors = 16;\n> > > + constexpr unsigned int MaxNumDimensions = 16;\n> > > +\n> > > + struct CnnOutputTensorInfo {\n> > > + char networkName[NetworkNameLen];\n> > > + uint32_t numTensors;\n> > > + OutputTensorInfo info[MaxNumTensors];\n> > > + };\n> > > +\n> > > + with\n> > > +\n> > > + struct OutputTensorInfo {\n> > > + uint32_t tensorDataNum;\n> > > + uint32_t numDimensions;\n> > > + uint16_t size[MaxNumDimensions];\n> > > + };\n> > > +\n> > > + networkName is the name of the CNN used,\n> > > + numTensors is the number of output tensors returned,\n> > > + tensorDataNum gives the number of elements in each output tensor,\n> > > + numDimensions gives the dimensionality of each output tensor,\n> > > + size gives the size of each dimension in each output tensor.\n> > > +\n> > > + \\sa CnnOutputTensor\n> > > +\n> > > + - CnnEnableInputTensor:\n> > > + type: bool\n> > > + description: |\n> > > + Boolean to control if the IPA returns the input tensor used by the CNN\n> > > + to generate the output tensors via the CnnInputTensor control. Because\n> > > + the input tensor may be relatively large, for efficiency reason avoid\n> > > + enabling input tensor output unless required for debugging purposes.\n> > > +\n> > > + \\sa CnnInputTensor\n> > > +\n> > > + - CnnInputTensor:\n> > > + type: uint8_t\n> > > + size: [n]\n> > > + description: |\n> > > + This control returns a span of uint8_t pixel values that represent the\n> > > + input tensor for a Convolutional Neural Network (CNN). The size and\n> > > + format of this array of values is entirely dependent on the neural\n> > > + network used, and further post-processing (e.g. pixel normalisations) may\n> > > + need to be performed at the application level to generate the final input\n> > > + image.\n> > > +\n> > > + The structure of the span is described by the CnnInputTensorInfo\n> > > + control.\n> > > +\n> > > + \\sa CnnInputTensorInfo\n> > > +\n> > > + - CnnInputTensorInfo:\n> > > + type: uint8_t\n> > > + size: [n]\n> > > + description: |\n> > > + This control returns the structure of the CnnInputTensor. This structure\n> > > + takes the following form:\n> > > +\n> > > + constexpr unsigned int NetworkNameLen = 64;\n> > > +\n> > > + struct CnnInputTensorInfo {\n> > > + char networkName[NetworkNameLen];\n> > > + uint32_t width;\n> > > + uint32_t height;\n> > > + uint32_t numChannels;\n> > > + };\n> > > +\n> > > + where\n> > > +\n> > > + networkName is the name of the CNN used,\n> > > + width and height are the input tensor image width and height in pixels,\n> > > + numChannels is the number of channels in the input tensor image.\n> > > +\n> > > + \\sa CnnInputTensor\n> > > +\n> > > + - CnnKpiInfo:\n> > > + type: int32_t\n> > > + size: [2]\n> > > + description: |\n> > > + This control returns performance metrics for the CNN processing stage.\n> > > + Two values are returned in this span, the runtime of the CNN/DNN stage\n> > > + and the DSP stage in milliseconds.\n> > > ...","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C6B65BD1F1\n\tfor ;\n\tTue, 17 Dec 2024 23:00:10 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E4CA56805B;\n\tWed, 18 Dec 2024 00:00:09 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2FFC067FD3\n\tfor ;\n\tWed, 18 Dec 2024 00:00:08 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5DDBB75A;\n\tTue, 17 Dec 2024 23:59:30 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"fkxoYjUD\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1734476370;\n\tbh=Te3tqG5PlyDiBs+Gsx75oiOTCFaAhpOmhLqkXQUMEOQ=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=fkxoYjUDyJAVyAec7YIPeV0V3UuNDPYoYO+C8rbF184HFhywddrv1obtvG0R2KO26\n\tiHiNbtqZ+R+UmFOfGz2R2L+WzIVj1TKYu7qNh8WJft9XKY3CSOCkWBK1O5oB1rgA0b\n\tgWdEAzF28og9XzpaGgAmSUUaH0a56RR9saJYtJoM=","Date":"Wed, 18 Dec 2024 01:00:05 +0200","From":"Laurent Pinchart ","To":"Naushir Patuck ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH 4/6] controls: ipa: rpi: Add CNN controls","Message-ID":"<20241217230005.GM23470@pendragon.ideasonboard.com>","References":"<20241213094602.2083174-1-naush@raspberrypi.com>\n\t<20241213094602.2083174-5-naush@raspberrypi.com>\n\t<20241215163753.GD9975@pendragon.ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]