Patch Detail
Show a patch.
GET /api/1.1/patches/14277/?format=api
{ "id": 14277, "url": "https://patchwork.libcamera.org/api/1.1/patches/14277/?format=api", "web_url": "https://patchwork.libcamera.org/patch/14277/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/1.1/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20211022151218.111966-12-jeanmichel.hautbois@ideasonboard.com>", "date": "2021-10-22T15:12:10", "name": "[libcamera-devel,v3,11/19] Documentation: IPU3 IPA Design guide", "commit_ref": null, "pull_url": null, "state": "changes-requested", "archived": false, "hash": "f231a2c02f4d5758b443dc2a9c2c4ecba45c07c1", "submitter": { "id": 75, "url": "https://patchwork.libcamera.org/api/1.1/people/75/?format=api", "name": "Jean-Michel Hautbois", "email": "jeanmichel.hautbois@ideasonboard.com" }, "delegate": null, "mbox": "https://patchwork.libcamera.org/patch/14277/mbox/", "series": [ { "id": 2651, "url": "https://patchwork.libcamera.org/api/1.1/series/2651/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=2651", "date": "2021-10-22T15:11:59", "name": "Document IPU3 IPA", "version": 3, "mbox": "https://patchwork.libcamera.org/series/2651/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/patches/14277/comments/", "check": "pending", "checks": "https://patchwork.libcamera.org/api/patches/14277/checks/", "tags": {}, "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 079D8BF415\n\tfor <parsemail@patchwork.libcamera.org>;\n\tFri, 22 Oct 2021 15:12:40 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B3F9168F70;\n\tFri, 22 Oct 2021 17:12:39 +0200 (CEST)", "from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DD08E68F65\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 22 Oct 2021 17:12:24 +0200 (CEST)", "from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:22cc:3af6:5ccb:8367])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 8675451D;\n\tFri, 22 Oct 2021 17:12:24 +0200 (CEST)" ], "Authentication-Results": "lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Pwx6mjG9\"; dkim-atps=neutral", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1634915544;\n\tbh=9fz9YCVeDDqgrNQUR8yRVkSaMmhDJxC5lqc4Fgj8ZvQ=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=Pwx6mjG9NhH1u8zbpG12PeYNUUoJecuK3Hr1hbYK6p5s44zOVzRvrpJThZ3DJBKog\n\tTafxwAKgAYv+TkJFuTW2ZNeJ2UdFwDZkjz+Ikd87lSc2isby1FE/h3Ws9i76TMOcr0\n\tveBDKk0P9UzP0wzMJKbaVBdDs8moTQZ79oeffniU=", "From": "Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>", "To": "libcamera-devel@lists.libcamera.org", "Date": "Fri, 22 Oct 2021 17:12:10 +0200", "Message-Id": "<20211022151218.111966-12-jeanmichel.hautbois@ideasonboard.com>", "X-Mailer": "git-send-email 2.32.0", "In-Reply-To": "<20211022151218.111966-1-jeanmichel.hautbois@ideasonboard.com>", "References": "<20211022151218.111966-1-jeanmichel.hautbois@ideasonboard.com>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=UTF-8", "Content-Transfer-Encoding": "8bit", "Subject": "[libcamera-devel] [PATCH v3 11/19] Documentation: IPU3 IPA Design\n\tguide", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "From: Kieran Bingham <kieran.bingham@ideasonboard.com>\n\nThe IPU3 IPA implements the basic 3A using the ImgU ISP.\n\nProvide an overview document to describe its operations, and provide a\nblock diagram to help visualise how the components are put together to\nassist any new developers exploring the code.\n\nSigned-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\nReviewed-by: Umang Jain <umang.jain@ideasonboard.com>\n\n---\n\nv2:\n - /accelerator cluster/processing block/ (and refactor)\n - /Pipeline/pipeline/\n - /Camera Sensor/camera sensor/\n - /CPU accessible/CPU-accessible/\n - Remove updated control parameters from IPASessionConfiguration\n - Expand pre-frame preparation to match post-frame with the event\n descriptions.\n - Add Sensor Controls brief\n - Move to src/ipa/ipu3/\n - Lower indentation of the block diagrams (keep under 80chars)\n - reference mapBuffers() call for passing buffers in\n - reference unmapBuffers() after stop()\n---\n src/ipa/ipu3/ipu3-ipa-design-guide.rst | 155 +++++++++++++++++++++++++\n 1 file changed, 155 insertions(+)\n create mode 100644 src/ipa/ipu3/ipu3-ipa-design-guide.rst", "diff": "diff --git a/src/ipa/ipu3/ipu3-ipa-design-guide.rst b/src/ipa/ipu3/ipu3-ipa-design-guide.rst\nnew file mode 100644\nindex 00000000..e5ee508d\n--- /dev/null\n+++ b/src/ipa/ipu3/ipu3-ipa-design-guide.rst\n@@ -0,0 +1,155 @@\n+IPU3 IPA Architecture Design and Overview\n+=========================================\n+\n+The IPU3 IPA is built as a modular and extensible framework with an\n+upper layer to manage the interactions with the pipeline handler, and\n+the image processing algorithms split to compartmentalise the processing\n+required for each processing block, making use of the fixed-function\n+accelerators provided by the ImgU ISP.\n+\n+The core IPU3 class is responsible for initialisation and construction\n+of the algorithm components, processing controls set by the requests\n+from applications, and managing events from the pipeline handler.\n+\n+::\n+\n+ ┌───────────────────────────────────────────┐\n+ │ IPU3 Pipeline Handler │\n+ │ ┌────────┐ ┌────────┐ ┌────────┐ │\n+ │ │ │ │ │ │ │ │\n+ │ │ Sensor ├───►│ CIO2 ├───►│ ImgU ├──►\n+ │ │ │ │ │ │ │ │\n+ │ └────────┘ └────────┘ └─▲────┬─┘ │ P: Parameter Buffer\n+ │ │P │ │ S: Statistics Buffer\n+ │ │ │S │\n+ └─┬───┬───┬──────┬────┬────┬────┬─┴────▼─┬──┘ 1: init()\n+ │ │ │ │ ▲ │ ▲ │ ▲ │ ▲ │ 2: configure()\n+ │1 │2 │3 │4│ │4│ │4│ │4│ │5 3: mapBuffers(), start()\n+ ▼ ▼ ▼ ▼ │ ▼ │ ▼ │ ▼ │ ▼ 4: processEvent()\n+ ┌──────────────────┴────┴────┴────┴─────────┐ 5: stop(), unmapBuffers()\n+ │ IPU3 IPA │\n+ │ ┌───────────────────────┐ │\n+ │ ┌───────────┐ │ Algorithms │ │\n+ │ │IPAContext │ │ ┌─────────┐ │ │\n+ │ │ ┌───────┐ │ │ │ ... │ │ │\n+ │ │ │ │ │ │ ┌─┴───────┐ │ │ │\n+ │ │ │ SC │ │ │ │ Tonemap ├─┘ │ │\n+ │ │ │ │ ◄───► ┌─┴───────┐ │ │ │\n+ │ │ ├───────┤ │ │ │ AWB ├─┘ │ │\n+ │ │ │ │ │ │ ┌─┴───────┐ │ │ │\n+ │ │ │ FC │ │ │ │ AGC ├─┘ │ │\n+ │ │ │ │ │ │ │ │ │ │\n+ │ │ └───────┘ │ │ └─────────┘ │ │\n+ │ └───────────┘ └───────────────────────┘ │\n+ └───────────────────────────────────────────┘\n+ SC: IPASessionConfiguration\n+ FC: IPAFrameContext(s)\n+\n+The IPA instance is constructed and initialised at the point a Camera is\n+created by the IPU3 pipeline handler. The initialisation call provides\n+details about which camera sensor is being used, and the controls that\n+it has available, along with their default values and ranges.\n+\n+Buffers\n+~~~~~~~\n+\n+The IPA will have Parameter and Statistics buffers shared with it from\n+the IPU3 Pipeline handler. These buffers will be passed to the IPA using\n+the ``mapBuffers()`` call before the ``start()`` operation occurs.\n+\n+The IPA will map the buffers into CPU-accessible memory, associated with\n+a buffer ID, and further events for sending or receiving parameter and\n+statistics buffers will reference the ID to avoid expensive memory\n+mapping operations, or the passing of file handles during streaming.\n+\n+After the ``stop()`` operation occurs, these buffers will be unmapped\n+when requested by the pipeline handler using the ``unmapBuffers()`` call\n+and no further access to the buffers is permitted.\n+\n+Context\n+~~~~~~~\n+\n+Algorithm calls will always have the ``IPAContext`` available to them.\n+This context comprises of two parts:\n+\n+- IPA Session Configuration\n+- IPA Frame Context\n+\n+The session configuration structure ``IPASessionConfiguration``\n+represents constant parameters determined before streaming commenced\n+during ``configure()``.\n+\n+The IPA Frame Context provides the storage for algorithms for a single\n+frame operation.\n+\n+The ``IPAFrameContext`` structure may be extended to an array, list, or\n+queue to store historical state for each frame, allowing algorithms to\n+obtain and reference results of calculations which are deeply pipelined.\n+This may only be done if an algorithm needs to know the context that was\n+applied at the frame the statistics were produced for, rather than the\n+previous or current frame.\n+\n+Presently there is a single ``IPAFrameContext`` without historical data,\n+and the context is maintained and updated through successive processing\n+operations.\n+\n+Operating\n+~~~~~~~~~\n+\n+There are three main interactions with the algorithms for the IPU3 IPA\n+to operate when running:\n+\n+- configure()\n+- processEvent(``EventFillParams``)\n+- processEvent(``EventStatReady``)\n+\n+The configuration phase allows the pipeline-handler to inform the IPA of\n+the current stream configurations, which is then passed into each\n+algorithm to provide an opportunity to identify and track state of the\n+hardware, such as image size or ImgU pipeline configurations.\n+\n+Pre-frame preparation\n+~~~~~~~~~~~~~~~~~~~~~\n+\n+When configured, the IPA is notified by the pipeline handler of the\n+Camera ``start()`` event, after which incoming requests will be queued\n+for processing, requiring a parameter buffer (``ipu3_uapi_params``) to\n+be populated for the ImgU. This is given to the IPA through the\n+``EventFillParams`` event, and then passed directly to each algorithm\n+through the ``prepare()`` call allowing the ISP configuration to be\n+updated for the needs of each component that the algorithm is\n+responsible for.\n+\n+The algorithm should set the use flag (``ipu3_uapi_flags``) for any\n+structure that it modifies, and it should take care to ensure that any\n+structure set by a use flag is fully initialised to suitable values.\n+\n+The parameter buffer is returned to the pipeline handler through the\n+``ActionParamFilled`` event, and from there queued to the ImgU along\n+with a raw frame captured with the CIO2.\n+\n+Post-frame completion\n+~~~~~~~~~~~~~~~~~~~~~\n+\n+When the capture of an image is completed, and successfully processed\n+through the ImgU, the generated statistics buffer\n+(``ipu3_uapi_stats_3a``) is given to the IPA through the\n+``EventStatReady`` event. This provides the IPA with an opportunity to\n+examine the results of the ISP and run the calculations required by each\n+algorithm on the new data. The algorithms may require context from the\n+operations of other algorithms, for example, the AWB might choose to use\n+a scene brightness determined by the AGC. It is important that the\n+algorithms are ordered to ensure that required results are determined\n+before they are needed.\n+\n+The ordering of the algorithm processing is determined by their\n+placement in the ``IPU3::algorithms_`` ordered list.\n+\n+Sensor Controls\n+~~~~~~~~~~~~~~~\n+\n+The AutoExposure and AutoGain (AGC) algorithm differs slightly from the\n+others as it requires operating directly on the sensor, as opposed to\n+through the ImgU ISP. To support this, there is a dedicated action\n+`ActionSetSensorControls` to allow the IPA to request controls to be set\n+on the camera sensor through the pipeline handler.\n\\ No newline at end of file\n", "prefixes": [ "libcamera-devel", "v3", "11/19" ] }