Cover Letter Detail
Show a cover letter.
GET /api/covers/22762/?format=api
{ "id": 22762, "url": "https://patchwork.libcamera.org/api/covers/22762/?format=api", "web_url": "https://patchwork.libcamera.org/cover/22762/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20250207101703.21149-1-mzamazal@redhat.com>", "date": "2025-02-07T10:16:52", "name": "[v7,00/10] Software ISP support for CCM", "submitter": { "id": 177, "url": "https://patchwork.libcamera.org/api/people/177/?format=api", "name": "Milan Zamazal", "email": "mzamazal@redhat.com" }, "mbox": "https://patchwork.libcamera.org/cover/22762/mbox/", "series": [ { "id": 4994, "url": "https://patchwork.libcamera.org/api/series/4994/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=4994", "date": "2025-02-07T10:16:52", "name": "Software ISP support for CCM", "version": 7, "mbox": "https://patchwork.libcamera.org/series/4994/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/covers/22762/comments/", "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C95CDC32EA\n\tfor <parsemail@patchwork.libcamera.org>;\n\tFri, 7 Feb 2025 10:17:17 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E7A8768600;\n\tFri, 7 Feb 2025 11:17:16 +0100 (CET)", "from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3DA84685BE\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tFri, 7 Feb 2025 11:17:15 +0100 (CET)", "from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com\n\t(ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97])\n\tby relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n\tcipher=TLS_AES_256_GCM_SHA384) id us-mta-376-pdAv49zmP3KR5r3r3Fn6AQ-1;\n\tFri, 07 Feb 2025 05:17:10 -0500", "from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com\n\t(mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com\n\t[10.30.177.15])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (2048 bits)\n\tserver-digest SHA256) (No client certificate requested)\n\tby mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTPS id 58B24180087B; Fri, 7 Feb 2025 10:17:09 +0000 (UTC)", "from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.226.123])\n\tby mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTP id DCB2719560AE; Fri, 7 Feb 2025 10:17:06 +0000 (UTC)" ], "Authentication-Results": "lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"amUrr+i+\"; dkim-atps=neutral", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1738923434;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding;\n\tbh=Rp9ZOcj615YcB3t+wFPPgCh9ygzt2/wx4aUFN9NoaV8=;\n\tb=amUrr+i+a4WwqIgFghwymNk3FtOhMbZfV5eGcCGyny2LEIAi/GuYkIVt+p7+hI1j9+gT+V\n\tcqiy5hbubJQeXfu44XsTHlC6qrzOngluJMtEs9oBDs2Luaf2YTVargGDELDEQTfRonGLOl\n\t5DW8gy9FuFUrgx64GYMujnSuBwkzoZw=", "X-MC-Unique": "pdAv49zmP3KR5r3r3Fn6AQ-1", "X-Mimecast-MFC-AGG-ID": "pdAv49zmP3KR5r3r3Fn6AQ", "From": "Milan Zamazal <mzamazal@redhat.com>", "To": "libcamera-devel@lists.libcamera.org", "Cc": "Milan Zamazal <mzamazal@redhat.com>,\n\tRobert Mader <robert.mader@collabora.com>,\n\tHans de Goede <hdegoede@redhat.com>,\n\tLaurent Pinchart <laurent.pinchart@ideasonboard.com>,\n\tKieran Bingham <kieran.bingham@ideasonboard.com>", "Subject": "[PATCH v7 00/10] Software ISP support for CCM", "Date": "Fri, 7 Feb 2025 11:16:52 +0100", "Message-ID": "<20250207101703.21149-1-mzamazal@redhat.com>", "MIME-Version": "1.0", "X-Scanned-By": "MIMEDefang 3.0 on 10.30.177.15", "X-Mimecast-Spam-Score": "0", "X-Mimecast-MFC-PROC-ID": "Lo_dfmmdKvPMwo7ZF1M0-fsF5VEcx1lZkIJhL8qcIq0_1738923429", "X-Mimecast-Originator": "redhat.com", "Content-Type": "text/plain; charset=UTF-8", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "This series implements application of color correction matrices in\nsoftware ISP.\n\nSupport for color temperature is added to auto white balance algorithm,\nthis is needed to obtain the right CCM. A new Ccm algorithm is added to\ndetermine the matrix. Both roughly follow what the hardware pipelines\ndo.\n\nLut algorithm is modified to incorporate the CCM and perform some\nprecomputations in the form of 4 lookup tables (one per each matrix\ncolumn and one for gamma).\n\nA tricky part is to support both CCM and the original (no-CCM)\ntransformations. CCM obviously adds a significant overhead; per-frame\ntime increases by ~45% on Debix Model A and ~85% on TI AM69 SK. This\nmeans CCM must be optional, which is determined by presence of Ccm\nalgorithm in the tuning file.\n\nThe performance critical debayering code must not include extra\nconditionals and must work efficiently for both the cases. The macros\nin debayering code are refactored and restructured for this. An added\nbonus is that this also removes some annoying code duplication. And\nsome more, partially already present, templating is used to select\nbetween CCM and non-CCM processing.\n\nPerformance is much influenced by the data structures used in\ndebayering. I tried to define something reasonable and performed only\nbasic performance testing. Suggestions for possible improvements are\nwelcome. There is already one from Hans about avoiding using std::clamp\nin debayering but I don’t have a working patch for this yet.\n\nWhen trying to use the following imx219 color correction matrix from rpi\nsubtree, I get quite noticeable color casts:\n\n - Ccm:\n ccms:\n - ct: 2860\n ccm: [ 2.12089, -0.52461, -0.59629,\n -0.85342, 2.80445, -0.95103,\n -0.26897, -1.14788, 2.41685 ]\n - ct: 2960\n ccm: [ 2.26962, -0.54174, -0.72789,\n -0.77008, 2.60271, -0.83262,\n -0.26036, -1.51254, 2.77289 ]\n - ct: 3603\n ccm: [ 2.18644, -0.66148, -0.52496,\n -0.77828, 2.69474, -0.91645,\n -0.25239, -0.83059, 2.08298 ]\n - ct: 4650\n ccm: [ 2.18174, -0.70887, -0.47287,\n -0.70196, 2.76426, -1.06231,\n -0.25157, -0.71978, 1.97135 ]\n - ct: 5858\n ccm: [ 2.32392, -0.88421, -0.43971,\n -0.63821, 2.58348, -0.94527,\n -0.28541, -0.54112, 1.82653 ]\n - ct: 7580\n ccm: [ 2.21175, -0.53242, -0.67933,\n -0.57875, 3.07922, -1.50047,\n -0.27709, -0.73338, 2.01048 ]\n\nOne of the possible explanations could be that the software ISP AWB\n(used to determine the color temperature) is no way accurate and\nproduces some (different) color casts too. But it can also be caused by\nsomething else.\n\nChanges in v7:\n- The changed lookup table types updated in docstrings.\n\nChanges in v6:\n- A transposed CCM tables access was used in debayering, fixed.\n Unfortunately this slows down the processing by almost 10% in my\n environment, apparently due to worse locality.\n- When red-blue swapping is applied in debayering, `r' and `b' items in\n the CCM lookup tables must be swapped too, fixed.\n- The multiplication is now CCM * RGB rather than RGB * CCM; this\n matches the representation of CCMs in the tuning files.\n- CCM and gain value type changed from double to float to be consistent\n with the rkisp1 pipeline and controls::ColourCorrectionMatrix.\n- Some code simplifications suggested by Laurent.\n- Comments added to lookup tables declarations.\n\nChanges in v5:\n- Local variable to store temperature removed; the context variable used\n directly instead.\n- Patch to use RGB<double> instead of a custom struct to store gains\n added.\n- Superfluous include of stdlib.h removed from ccm.cpp.\n- Error returned from Ccm::init when there is no CCM defined.\n- Ccm::kTemperatureThreshold moved to a constexpr global variable.\n- `const' added to const variables.\n- Separate color lookup tables for CCM / simple rather than unions.\n- ccmEnabled passed as a normal rather than template argument to\n setDebayerFunctions.\n- ccmEnabled is determined in init rather than in configure.\n\nChanges in v4:\n- utils::abs_diff used instead of the ugly typecasting.\n- A missing argument type added to Debayer::configure docstring.\n- CCM and related processing is disabled when tuning data is not\n available.\n- Unneeded initialization of context_.activeState.ccm.enabled to the\n default value removed.\n\nChanges in v3:\n- Rebased on master.\n- AWB gains applied.\n- ccmEnabled docstring moved to the right patch.\n- Small notes about std::variant and about std::clamp added to the\n commit message of the last patch.\n\nChanges in v2:\n- Fix of the gamma table index upper boundary.\n- Fix of red<->blue being temporarily swapped in an intermediate patch.\n- Fix of CCM transformation when swapRedBlueGains_ is true.\n- A related minor lookup data structures rearrangement, with an\n unintended but welcome effect of ~10% speedup with CCM in my\n environment. \n\nMilan Zamazal (10):\n libcamera: software_isp: Determine color temperature\n libcamera: software_isp: Use RGB type to represent gains\n libcamera: software_isp: Store color temperature to metadata\n libcamera: software_isp: lut: Remove maybe_unused on a used argument\n libcamera: software_isp: Use common code to store debayered pixels\n libcamera: software_isp: Use a macro to assign debayering methods\n libcamera: software_isp: Add CCM algorithm\n libcamera: software_isp: Add an example CCM to uncalibrated.yaml\n libcamera: software_isp: Track whether CCM is enabled\n libcamera: software_isp: Apply CCM in debayering\n\n .../internal/software_isp/debayer_params.h | 34 +++-\n .../internal/software_isp/software_isp.h | 1 +\n include/libcamera/ipa/soft.mojom | 2 +-\n src/ipa/simple/algorithms/awb.cpp | 29 +++-\n src/ipa/simple/algorithms/ccm.cpp | 76 +++++++++\n src/ipa/simple/algorithms/ccm.h | 43 +++++\n src/ipa/simple/algorithms/lut.cpp | 63 ++++---\n src/ipa/simple/algorithms/lut.h | 1 +\n src/ipa/simple/algorithms/meson.build | 1 +\n src/ipa/simple/data/uncalibrated.yaml | 9 +\n src/ipa/simple/ipa_context.h | 21 ++-\n src/ipa/simple/soft_simple.cpp | 8 +-\n src/libcamera/software_isp/debayer.cpp | 54 +++++-\n src/libcamera/software_isp/debayer.h | 3 +-\n src/libcamera/software_isp/debayer_cpu.cpp | 158 +++++++++++-------\n src/libcamera/software_isp/debayer_cpu.h | 39 +++--\n src/libcamera/software_isp/software_isp.cpp | 5 +-\n 17 files changed, 424 insertions(+), 123 deletions(-)\n create mode 100644 src/ipa/simple/algorithms/ccm.cpp\n create mode 100644 src/ipa/simple/algorithms/ccm.h" }