[{"id":19709,"web_url":"https://patchwork.libcamera.org/comment/19709/","msgid":"","date":"2021-09-16T13:17:07","subject":"Re: [libcamera-devel] [PATCH v2] Documentation: IPU3 IPA Design\n\tguide","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"content":"Hi Kieran,\n\nOn 16/09/2021 14:51, Kieran Bingham wrote:\n> The IPU3 IPA implements the basic 3A using the ImgU ISP.\n> \n> Provide an overview document to describe its operations, and provide a\n> block diagram to help visualise how the components are put together to\n> assist any new developers exploring the code.\n> \n> Signed-off-by: Kieran Bingham \n> Reviewed-by: Umang Jain \n> \n> ---\n> v2:\n> - /accelerator cluster/processing block/ (and refactor)\n> - /Pipeline/pipeline/\n> - /Camera Sensor/camera sensor/\n> - /CPU accessible/CPU-accessible/\n> - Remove updated control parameters from IPASessionConfiguration\n> - Expand pre-frame preparation to match post-frame with the event\n> descriptions.\n> - Add Sensor Controls brief\n> - Move to src/ipa/ipu3/\n> - Lower indentation of the block diagrams (keep under 80chars)\n> - reference mapBuffers() call for passing buffers in\n> - reference unmapBuffers() after stop()\n> \n> src/ipa/ipu3/ipu3-ipa-design-guide.rst | 155 +++++++++++++++++++++++++\n> 1 file changed, 155 insertions(+)\n> create mode 100644 src/ipa/ipu3/ipu3-ipa-design-guide.rst\n> \n> diff --git a/src/ipa/ipu3/ipu3-ipa-design-guide.rst b/src/ipa/ipu3/ipu3-ipa-design-guide.rst\n> new file mode 100644\n> index 000000000000..e5ee508d62e9\n> --- /dev/null\n> +++ b/src/ipa/ipu3/ipu3-ipa-design-guide.rst\n> @@ -0,0 +1,155 @@\n> +IPU3 IPA Architecture Design and Overview\n> +=========================================\n> +\n> +The IPU3 IPA is built as a modular and extensible framework with an\n> +upper layer to manage the interactions with the pipeline handler, and\n> +the image processing algorithms split to compartmentalise the processing\n> +required for each processing block, making use of the fixed-function\n> +accelerators provided by the ImgU ISP.\n\nI am not native, and it might very well explain it but... isn't that\nsentence very long ? I read it multiple times to have it all... :-/\n\n> +\n> +The core IPU3 class is responsible for initialisation and construction\n> +of the algorithm components, processing controls set by the requests\n> +from applications, and managing events from the pipeline handler.\n> +\n> +::\n> +\n> + ┌───────────────────────────────────────────┐\n> + │ IPU3 Pipeline Handler │\n> + │ ┌────────┐ ┌────────┐ ┌────────┐ │\n> + │ │ │ │ │ │ │ │\n> + │ │ Sensor ├───►│ CIO2 ├───►│ ImgU ├──►\n> + │ │ │ │ │ │ │ │\n> + │ └────────┘ └────────┘ └─▲────┬─┘ │ P: Parameter Buffer\n> + │ │P │ │ S: Statistics Buffer\n> + │ │ │S │\n> + └─┬───┬───┬──────┬────┬────┬────┬─┴────▼─┬──┘ 1: init()\n> + │ │ │ │ ▲ │ ▲ │ ▲ │ ▲ │ 2: configure()\n> + │1 │2 │3 │4│ │4│ │4│ │4│ │5 3: mapBuffers(), start()\n> + ▼ ▼ ▼ ▼ │ ▼ │ ▼ │ ▼ │ ▼ 4: processEvent()\n> + ┌──────────────────┴────┴────┴────┴─────────┐ 5: stop(), unmapBuffers()\n> + │ IPU3 IPA │\n> + │ ┌───────────────────────┐ │\n> + │ ┌───────────┐ │ Algorithms │ │\n> + │ │IPAContext │ │ ┌─────────┐ │ │\n> + │ │ ┌───────┐ │ │ │ ... │ │ │\n> + │ │ │ │ │ │ ┌─┴───────┐ │ │ │\n> + │ │ │ SC │ │ │ │ Tonemap ├─┘ │ │\n> + │ │ │ │ ◄───► ┌─┴───────┐ │ │ │\n> + │ │ ├───────┤ │ │ │ AWB ├─┘ │ │\n> + │ │ │ │ │ │ ┌─┴───────┐ │ │ │\n> + │ │ │ FC │ │ │ │ AGC ├─┘ │ │\n> + │ │ │ │ │ │ │ │ │ │\n> + │ │ └───────┘ │ │ └─────────┘ │ │\n> + │ └───────────┘ └───────────────────────┘ │\n> + └───────────────────────────────────────────┘\n> + SC: IPASessionConfiguration\n> + FC: IPAFrameContext(s)\n> +\n> +The IPA instance is constructed and initialised at the point a Camera is\n> +created by the IPU3 pipeline handler. The initialisation call provides\n> +details about which camera sensor is being used, and the controls that\n> +it has available, along with their default values and ranges.\n> +\n> +Buffers\n> +~~~~~~~\n> +\n> +The IPA will have Parameter and Statistics buffers shared with it from\n> +the IPU3 Pipeline handler. These buffers will be passed to the IPA using\n> +the ``mapBuffers()`` call before the ``start()`` operation occurs.\n> +\n> +The IPA will map the buffers into CPU-accessible memory, associated with\n> +a buffer ID, and further events for sending or receiving parameter and\n> +statistics buffers will reference the ID to avoid expensive memory\n> +mapping operations, or the passing of file handles during streaming.\n> +\n> +After the ``stop()`` operation occurs, these buffers will be unmapped\n> +when requested by the pipeline handler using the ``unmapBuffers()`` call\n> +and no further access to the buffers is permitted.\n> +\n> +Context\n> +~~~~~~~\n> +\n> +Algorithm calls will always have the ``IPAContext`` available to them.\n> +This context comprises of two parts:\n> +\n> +- IPA Session Configuration\n> +- IPA Frame Context\n> +\n> +The session configuration structure ``IPASessionConfiguration``\n> +represents constant parameters determined before streaming commenced\n> +during ``configure()``.\n> +\n> +The IPA Frame Context provides the storage for algorithms for a single\n> +frame operation.\n> +\n> +The ``IPAFrameContext`` structure may be extended to an array, list, or\n> +queue to store historical state for each frame, allowing algorithms to\n> +obtain and reference results of calculations which are deeply pipelined.\n> +This may only be done if an algorithm needs to know the context that was\n> +applied at the frame the statistics were produced for, rather than the\n> +previous or current frame.\n> +\n> +Presently there is a single ``IPAFrameContext`` without historical data,\n> +and the context is maintained and updated through successive processing\n> +operations.\n> +\n> +Operating\n> +~~~~~~~~~\n> +\n> +There are three main interactions with the algorithms for the IPU3 IPA\n> +to operate when running:\n> +\n> +- configure()\n> +- processEvent(``EventFillParams``)\n> +- processEvent(``EventStatReady``)\n> +\n> +The configuration phase allows the pipeline-handler to inform the IPA of\n> +the current stream configurations, which is then passed into each\n> +algorithm to provide an opportunity to identify and track state of the\n> +hardware, such as image size or ImgU pipeline configurations.\n> +\n> +Pre-frame preparation\n> +~~~~~~~~~~~~~~~~~~~~~\n> +\n> +When configured, the IPA is notified by the pipeline handler of the\n> +Camera ``start()`` event, after which incoming requests will be queued\n> +for processing, requiring a parameter buffer (``ipu3_uapi_params``) to\n> +be populated for the ImgU. This is given to the IPA through the\n> +``EventFillParams`` event, and then passed directly to each algorithm\n> +through the ``prepare()`` call allowing the ISP configuration to be\n> +updated for the needs of each component that the algorithm is\n> +responsible for.\n> +\n> +The algorithm should set the use flag (``ipu3_uapi_flags``) for any\n> +structure that it modifies, and it should take care to ensure that any\n> +structure set by a use flag is fully initialised to suitable values.\n> +\n> +The parameter buffer is returned to the pipeline handler through the\n> +``ActionParamFilled`` event, and from there queued to the ImgU along\n> +with a raw frame captured with the CIO2.\n> +\n> +Post-frame completion\n> +~~~~~~~~~~~~~~~~~~~~~\n> +\n> +When the capture of an image is completed, and successfully processed\n> +through the ImgU, the generated statistics buffer\n> +(``ipu3_uapi_stats_3a``) is given to the IPA through the\n> +``EventStatReady`` event. This provides the IPA with an opportunity to\n> +examine the results of the ISP and run the calculations required by each\n> +algorithm on the new data. The algorithms may require context from the\n> +operations of other algorithms, for example, the AWB might choose to use\n> +a scene brightness determined by the AGC. It is important that the\n> +algorithms are ordered to ensure that required results are determined\n> +before they are needed.\n> +\n> +The ordering of the algorithm processing is determined by their\n> +placement in the ``IPU3::algorithms_`` ordered list.\n> +\n> +Sensor Controls\n> +~~~~~~~~~~~~~~~\n> +\n> +The AutoExposure and AutoGain (AGC) algorithm differs slightly from the\n> +others as it requires operating directly on the sensor, as opposed to\n> +through the ImgU ISP. To support this, there is a dedicated action\n> +`ActionSetSensorControls` to allow the IPA to request controls to be set\n> +on the camera sensor through the pipeline handler.\n> \\ No newline at end of file\n\nWhere is this \"No newline\" coming from ?\n\nApart from that:\nReviewed-by: Jean-Michel Hautbois ","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C6FCABF01C\n\tfor ;\n\tThu, 16 Sep 2021 13:17:11 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 55B4169188;\n\tThu, 16 Sep 2021 15:17:11 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A32396916F\n\tfor ;\n\tThu, 16 Sep 2021 15:17:09 +0200 (CEST)","from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:83cf:f045:9f9f:a85e])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 39DF12A5;\n\tThu, 16 Sep 2021 15:17:09 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"LfQ/FZv+\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1631798229;\n\tbh=S7K1I1nYAFSAyRkcj38Xv+ImvQvUaQFxyUcwLWypZrk=;\n\th=Subject:To:Cc:References:From:Date:In-Reply-To:From;\n\tb=LfQ/FZv+4NVsbR7j4EKS9CLWwOINeUUsBhZDraYjWshJ6fHcx9kg+8lFTiyTgW7zh\n\tCQxtD4a44oWzPEoQn1nIQREmT4uZgG2Ga1x9u9czPFzQSA+rCbogsO8JxeCnBbY8o8\n\tK1HePNZwNDMK4jBYiMHC4IPYn6VKngsqzX+h3H6k=","To":"Kieran Bingham ,\n\tlibcamera devel ","References":"<20210916125125.1842826-1-kieran.bingham@ideasonboard.com>","From":"Jean-Michel Hautbois ","Message-ID":"","Date":"Thu, 16 Sep 2021 15:17:07 +0200","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101\n\tThunderbird/78.13.0","MIME-Version":"1.0","In-Reply-To":"<20210916125125.1842826-1-kieran.bingham@ideasonboard.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-US","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH v2] Documentation: IPU3 IPA Design\n\tguide","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]