[{"id":32501,"web_url":"https://patchwork.libcamera.org/comment/32501/","msgid":"<1651334c-d1c0-4fe5-9fe9-2f7957c3d1d8@redhat.com>","date":"2024-12-03T17:19:41","subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","submitter":{"id":102,"url":"https://patchwork.libcamera.org/api/people/102/","name":"Hans de Goede","email":"hdegoede@redhat.com"},"content":"Hi Milan,\n\nOn 25-Nov-24 7:03 PM, Milan Zamazal wrote:\n> This series implements application of color correction matrices in\n> software ISP.\n> \n> Support for color temperature is added to auto white balance algorithm,\n> this is needed to obtain the right CCM. A new Ccm algorithm is added to\n> determine the matrix. Both roughly follow what the hardware pipelines\n> do.\n> \n> Lut algorithm is modified to incorporate the CCM and perform some\n> precomputations in the form of 10 lookup tables (one per each matrix\n> element and one for gamma).\n> \n> A tricky part is to support both CCM and the original (no-CCM)\n> transformations. CCM obviously adds a significant overhead; per-frame\n> time increases by ~45% on Debix Model A and ~85% on TI AM69 SK. This\n> means CCM must be optional, which is determined by presence of Ccm\n> algorithm in the tuning file.\n> \n> The performance critical debayering code must not include extra\n> conditionals and must work efficiently for both the cases. The macros\n> in debayering code are refactored and restructured for this. An added\n> bonus is that this also removes some annoying code duplication. And\n> some more, partially already present, templating is used to select\n> between CCM and non-CCM processing.\n> \n> Performance is much influenced by the data structures used in\n> debayering. I tried to define something reasonable and performed only\n> basic performance testing. Suggestions for possible improvements are\n> welcome.\n\nThank you for writing this. I have tested this on a ThinkPad X1 Yoga\nwith a 13th Gen Intel(R) CoreT i7-1370P.\n\nPinning qcam and all its threads to only the 6 performance cores /\n12 threads:\n\ntaskset 0xfff ./redhat-linux-build/src/apps/qcam/qcam\n\nI consistently get 6.3 ms / frame processing time without the CCM\nboth with and without your patches and 9.5 ms / frame with the CCM\nsnippet in uncalibrated.yaml activated. So it seems that we should\nbe able to activate the CCM with the software ISP in the IPU6 case.\n\nI did notice that the image becomes quite green with the default\n1 - 1 - 1 CCM, so it seems that currently using the CCM ignores\nthe color-gains calculated by the simple IPA.\n\nI'm pretty sure that real ISP-s have both color gains and a CCM,\nso I think that we should apply the color gains to the CCM\nwhen calculating the lookup tables.\n\nE.g. multiply CcmColumn.r by red-gain before using it to fill\nthe lookup-table, etc.\n\nAnd then it would be up to the IPA code to either do\nwhitebalance through the gains or directly in the ccm,\ne.g. when using a ccm to enhance saturation then awb\ncould still be done through the gains.\n\nRegards,\n\nHans\n\n\n\n> \n> See the commit messages for more details.\n> \n> Changes in v2:\n> - Fix of the gamma table index upper boundary.\n> - Fix of red<->blue being temporarily swapped in an intermediate patch.\n> - Fix of CCM transformation when swapRedBlueGains_ is true.\n> - A related minor lookup data structures rearrangement, with an\n> unintended but welcome effect of ~10% speedup with CCM in my\n> environment. \n> \n> Milan Zamazal (9):\n> libcamera: software_isp: Determine color temperature\n> libcamera: software_isp: Store color temperature to metadata\n> libcamera: software_isp: lut: Remove maybe_unused on a used argument\n> libcamera: software_isp: Use common code to store debayered pixels\n> libcamera: software_isp: Use a macro to assign debayering methods\n> libcamera: software_isp: Add CCM algorithm\n> libcamera: software_isp: Add an example CCM to uncalibrated.yaml\n> libcamera: software_isp: Track whether CCM is enabled\n> libcamera: software_isp: Apply CCM in debayering\n> \n> .../internal/software_isp/debayer_params.h | 20 ++-\n> include/libcamera/ipa/soft.mojom | 2 +-\n> src/ipa/simple/algorithms/awb.cpp | 18 ++-\n> src/ipa/simple/algorithms/ccm.cpp | 86 ++++++++++++\n> src/ipa/simple/algorithms/ccm.h | 45 ++++++\n> src/ipa/simple/algorithms/lut.cpp | 62 +++++---\n> src/ipa/simple/algorithms/lut.h | 1 +\n> src/ipa/simple/algorithms/meson.build | 1 +\n> src/ipa/simple/data/uncalibrated.yaml | 7 +\n> src/ipa/simple/ipa_context.h | 22 ++-\n> src/ipa/simple/soft_simple.cpp | 8 +-\n> src/libcamera/software_isp/debayer.cpp | 53 ++++++-\n> src/libcamera/software_isp/debayer.h | 3 +-\n> src/libcamera/software_isp/debayer_cpu.cpp | 132 ++++++++++--------\n> src/libcamera/software_isp/debayer_cpu.h | 31 ++--\n> src/libcamera/software_isp/software_isp.cpp | 11 +-\n> 16 files changed, 393 insertions(+), 109 deletions(-)\n> create mode 100644 src/ipa/simple/algorithms/ccm.cpp\n> create mode 100644 src/ipa/simple/algorithms/ccm.h\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2232FBDB1C\n\tfor ;\n\tTue, 3 Dec 2024 17:19:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4921766092;\n\tTue, 3 Dec 2024 18:19:49 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A2B7865FD6\n\tfor ;\n\tTue, 3 Dec 2024 18:19:46 +0100 (CET)","from mail-ej1-f72.google.com (mail-ej1-f72.google.com\n\t[209.85.218.72]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n\tus-mta-686-P_1cnYjfPWa7cykaVbtqcg-1; Tue, 03 Dec 2024 12:19:44 -0500","by mail-ej1-f72.google.com with SMTP id\n\ta640c23a62f3a-aa516e52376so408752466b.2\n\tfor ;\n\tTue, 03 Dec 2024 09:19:43 -0800 (PST)","from ?IPV6:2001:1c00:2a07:3a01:e7a9:b143:57e6:261b?\n\t(2001-1c00-2a07-3a01-e7a9-b143-57e6-261b.cable.dynamic.v6.ziggo.nl.\n\t[2001:1c00:2a07:3a01:e7a9:b143:57e6:261b])\n\tby smtp.gmail.com with ESMTPSA id\n\ta640c23a62f3a-aa5996ddbfbsm637947766b.51.2024.12.03.09.19.41\n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tTue, 03 Dec 2024 09:19:41 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"Lgg8Oaaz\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1733246385;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=XaZR9UP+utDHM+JTGNBHrUxryYJNRTd1Ft8zIeJnrno=;\n\tb=Lgg8Oaazc2Nq4rxFYsUq1CuP1XvtNqFLBHa763aruyQvhCzNi4DRb829eoe34oPICDSTtg\n\tGXRva4OnH3yY6DPpcp76Im0L5B5b+nVwnMGBhTQY6e7+6+e3wDfHN04rIXkbgJ8gLdGkB1\n\tdUTeT+D0D/T3QOBuUMV2/UwxBwMPNhs=","X-MC-Unique":"P_1cnYjfPWa7cykaVbtqcg-1","X-Mimecast-MFC-AGG-ID":"P_1cnYjfPWa7cykaVbtqcg","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1733246383; x=1733851183;\n\th=content-transfer-encoding:in-reply-to:from:content-language\n\t:references:cc:to:subject:user-agent:mime-version:date:message-id\n\t:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n\tbh=XaZR9UP+utDHM+JTGNBHrUxryYJNRTd1Ft8zIeJnrno=;\n\tb=tYgKQLfOEvnGbWHIausQdmMwcy8M8V8q1kuZigZKaAeT2V7oOIab1umM8i3woD4UrC\n\tYsqJOmP0qxUr2p16/Q4F4GZyvP0s9ApvrkGh+/gVYyCSkNuUNYgWDDJpbSorjrsi0q8T\n\tDKgNvWuL2i5Ftn4EJc9N2cfO22m/cWzyRsNffw8pPxjj/uUUaRN6l4tMMfHA+Z+85S8d\n\tl1EychVlhbC0vdc0N7OEGqbpDLaeKqpHtqT4xMA7WVutN+JoOvQAgO0mWDTroYVRpGBt\n\t0FcqQHN37055cQsVKBm0dsxiBWimUOhGpejU1RdooyezObdZH30xm2djFF3wfi38qI9/\n\tWs4g==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCWvX8Y5vzqMdld1l4AlG33L6J6IHyX3jOji2H6l67rO6X4PYN8vATIa8+wcNtcjIousB9h0TF7QGrxUAftrfVU=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YxdPuJlXQkP0qruSoTKHoBT8k23UauIkOcuGLvnneOLHiOiFA1U\n\tDWvlEatA4YZNlfzOHUbr1PWlvXqC/2Sj3dHtmF4NYaUayoixMgw6UMunHiZeUkFkxFfoRZO/aJm\n\tjz32rlPgt8eSS+zzI2FNXsPFf7zJ7JDZk1jShAq+JF3vosvL7PgtcqSMlsSBK5dXg4rKsEh8=","X-Gm-Gg":"ASbGncviUYY2lDtlj51SZ8iAJQGaTMMv5WFzKNKs9pNscgLHu0q96mpxK3LMcxlSb9g\n\tp8mt8m6kNfbnYfGzGviUHthiMQQq185VXzr7jro00p3eVlvCGRYWt2hVLO+umuoGqCqmEPaNzyf\n\tGWJ3h8XYWc/u83oO0nhFpvYTTLYQ76tAJ+Y/xwagcFs8xk6W3p6B/g+XV9HG0+HyTlcrZo6qQHG\n\tZEmITIhAXPOyd4eo6flh7RNrEXPMMBNSohnRbtZqfUmgzQN3wWVoJ8yqVAh3xWDfZ2A1N/AkvSQ\n\t0AKgWjZR6maBEGFaHi92/dGmlNceFrBUh2hBWhBvi5qZ/z6UWZQa8G/fIsWwk1lyYT9ZNWwLfOU\n\tWjzW1xJLJO4MGyaMY3vSh75M4ZEg=","X-Received":["by 2002:a17:906:4ca:b0:aa5:4b7f:e706 with SMTP id\n\ta640c23a62f3a-aa5f7d03e46mr232369966b.2.1733246382828; \n\tTue, 03 Dec 2024 09:19:42 -0800 (PST)","by 2002:a17:906:4ca:b0:aa5:4b7f:e706 with SMTP id\n\ta640c23a62f3a-aa5f7d03e46mr232368466b.2.1733246382427; \n\tTue, 03 Dec 2024 09:19:42 -0800 (PST)"],"X-Google-Smtp-Source":"AGHT+IEbjxhdNUHOT4EB4RwQToFJQ4FwlXxxDypXZCZLeMI1H2J0HWxtxF0BiOpMUD+TFMSqp63Q1Q==","Message-ID":"<1651334c-d1c0-4fe5-9fe9-2f7957c3d1d8@redhat.com>","Date":"Tue, 3 Dec 2024 18:19:41 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","To":"Milan Zamazal , libcamera-devel@lists.libcamera.org","Cc":"Robert Mader ","References":"<20241125180310.1254519-1-mzamazal@redhat.com>","From":"Hans de Goede ","In-Reply-To":"<20241125180310.1254519-1-mzamazal@redhat.com>","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"gwX6-LWjee7YE8TDQqjc_t5EwDAofJMSRx16KbSerqc_1733246383","X-Mimecast-Originator":"redhat.com","Content-Language":"en-US","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"7bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32502,"web_url":"https://patchwork.libcamera.org/comment/32502/","msgid":"<20241203172337.GA5196@pendragon.ideasonboard.com>","date":"2024-12-03T17:23:37","subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Tue, Dec 03, 2024 at 06:19:41PM +0100, Hans de Goede wrote:\n> Hi Milan,\n> \n> On 25-Nov-24 7:03 PM, Milan Zamazal wrote:\n> > This series implements application of color correction matrices in\n> > software ISP.\n> > \n> > Support for color temperature is added to auto white balance algorithm,\n> > this is needed to obtain the right CCM. A new Ccm algorithm is added to\n> > determine the matrix. Both roughly follow what the hardware pipelines\n> > do.\n> > \n> > Lut algorithm is modified to incorporate the CCM and perform some\n> > precomputations in the form of 10 lookup tables (one per each matrix\n> > element and one for gamma).\n> > \n> > A tricky part is to support both CCM and the original (no-CCM)\n> > transformations. CCM obviously adds a significant overhead; per-frame\n> > time increases by ~45% on Debix Model A and ~85% on TI AM69 SK. This\n> > means CCM must be optional, which is determined by presence of Ccm\n> > algorithm in the tuning file.\n> > \n> > The performance critical debayering code must not include extra\n> > conditionals and must work efficiently for both the cases. The macros\n> > in debayering code are refactored and restructured for this. An added\n> > bonus is that this also removes some annoying code duplication. And\n> > some more, partially already present, templating is used to select\n> > between CCM and non-CCM processing.\n> > \n> > Performance is much influenced by the data structures used in\n> > debayering. I tried to define something reasonable and performed only\n> > basic performance testing. Suggestions for possible improvements are\n> > welcome.\n> \n> Thank you for writing this. I have tested this on a ThinkPad X1 Yoga\n> with a 13th Gen Intel(R) CoreT i7-1370P.\n> \n> Pinning qcam and all its threads to only the 6 performance cores /\n> 12 threads:\n> \n> taskset 0xfff ./redhat-linux-build/src/apps/qcam/qcam\n> \n> I consistently get 6.3 ms / frame processing time without the CCM\n> both with and without your patches and 9.5 ms / frame with the CCM\n> snippet in uncalibrated.yaml activated. So it seems that we should\n> be able to activate the CCM with the software ISP in the IPU6 case.\n> \n> I did notice that the image becomes quite green with the default\n> 1 - 1 - 1 CCM, so it seems that currently using the CCM ignores\n> the color-gains calculated by the simple IPA.\n> \n> I'm pretty sure that real ISP-s have both color gains and a CCM,\n> so I think that we should apply the color gains to the CCM\n> when calculating the lookup tables.\n\nThey do, but they typically apply colour gains before CFA interpolation,\nnot after.\n\n> E.g. multiply CcmColumn.r by red-gain before using it to fill\n> the lookup-table, etc.\n> \n> And then it would be up to the IPA code to either do\n> whitebalance through the gains or directly in the ccm,\n> e.g. when using a ccm to enhance saturation then awb\n> could still be done through the gains.\n> \n> Regards,\n> \n> Hans\n> \n> > See the commit messages for more details.\n> > \n> > Changes in v2:\n> > - Fix of the gamma table index upper boundary.\n> > - Fix of red<->blue being temporarily swapped in an intermediate patch.\n> > - Fix of CCM transformation when swapRedBlueGains_ is true.\n> > - A related minor lookup data structures rearrangement, with an\n> > unintended but welcome effect of ~10% speedup with CCM in my\n> > environment. \n> > \n> > Milan Zamazal (9):\n> > libcamera: software_isp: Determine color temperature\n> > libcamera: software_isp: Store color temperature to metadata\n> > libcamera: software_isp: lut: Remove maybe_unused on a used argument\n> > libcamera: software_isp: Use common code to store debayered pixels\n> > libcamera: software_isp: Use a macro to assign debayering methods\n> > libcamera: software_isp: Add CCM algorithm\n> > libcamera: software_isp: Add an example CCM to uncalibrated.yaml\n> > libcamera: software_isp: Track whether CCM is enabled\n> > libcamera: software_isp: Apply CCM in debayering\n> > \n> > .../internal/software_isp/debayer_params.h | 20 ++-\n> > include/libcamera/ipa/soft.mojom | 2 +-\n> > src/ipa/simple/algorithms/awb.cpp | 18 ++-\n> > src/ipa/simple/algorithms/ccm.cpp | 86 ++++++++++++\n> > src/ipa/simple/algorithms/ccm.h | 45 ++++++\n> > src/ipa/simple/algorithms/lut.cpp | 62 +++++---\n> > src/ipa/simple/algorithms/lut.h | 1 +\n> > src/ipa/simple/algorithms/meson.build | 1 +\n> > src/ipa/simple/data/uncalibrated.yaml | 7 +\n> > src/ipa/simple/ipa_context.h | 22 ++-\n> > src/ipa/simple/soft_simple.cpp | 8 +-\n> > src/libcamera/software_isp/debayer.cpp | 53 ++++++-\n> > src/libcamera/software_isp/debayer.h | 3 +-\n> > src/libcamera/software_isp/debayer_cpu.cpp | 132 ++++++++++--------\n> > src/libcamera/software_isp/debayer_cpu.h | 31 ++--\n> > src/libcamera/software_isp/software_isp.cpp | 11 +-\n> > 16 files changed, 393 insertions(+), 109 deletions(-)\n> > create mode 100644 src/ipa/simple/algorithms/ccm.cpp\n> > create mode 100644 src/ipa/simple/algorithms/ccm.h","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 172E0BDC71\n\tfor ;\n\tTue, 3 Dec 2024 17:23:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4E0E56609D;\n\tTue, 3 Dec 2024 18:23:51 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D9F3F6608A\n\tfor ;\n\tTue, 3 Dec 2024 18:23:49 +0100 (CET)","from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi\n\t[81.175.209.231])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 2AA184CE;\n\tTue, 3 Dec 2024 18:23:22 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"RixtXi1r\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1733246602;\n\tbh=qR+oieplXnFzWGa/94EEPX3S3Pj8AkNMudZrwK7YA2w=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=RixtXi1rgo/crlweWSJpemiapRySkXlWNXGpjlUWsGyBzmHZ3ZyADWi1O/UQG4Lfa\n\tu5e/Z/AO3qyu6g+aEvuUbxRP4ewyKdOu3Jp9y0L9duFtMmcS4DjOkD7Cttk7WgwmvE\n\tUNjXYojdiQiuT4UEuK6du1gOJwPM97wJQY71+Bk0=","Date":"Tue, 3 Dec 2024 19:23:37 +0200","From":"Laurent Pinchart ","To":"Hans de Goede ","Cc":"Milan Zamazal , libcamera-devel@lists.libcamera.org,\n\tRobert Mader ","Subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","Message-ID":"<20241203172337.GA5196@pendragon.ideasonboard.com>","References":"<20241125180310.1254519-1-mzamazal@redhat.com>\n\t<1651334c-d1c0-4fe5-9fe9-2f7957c3d1d8@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<1651334c-d1c0-4fe5-9fe9-2f7957c3d1d8@redhat.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32505,"web_url":"https://patchwork.libcamera.org/comment/32505/","msgid":"<8734j3vldn.fsf@redhat.com>","date":"2024-12-04T09:38:12","subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","submitter":{"id":177,"url":"https://patchwork.libcamera.org/api/people/177/","name":"Milan Zamazal","email":"mzamazal@redhat.com"},"content":"Hi Hans,\n\nHans de Goede writes:\n\n> Hi Milan,\n>\n> On 25-Nov-24 7:03 PM, Milan Zamazal wrote:\n>> This series implements application of color correction matrices in\n>> software ISP.\n>> \n>> Support for color temperature is added to auto white balance algorithm,\n>> this is needed to obtain the right CCM. A new Ccm algorithm is added to\n>> determine the matrix. Both roughly follow what the hardware pipelines\n>> do.\n>> \n>> Lut algorithm is modified to incorporate the CCM and perform some\n>> precomputations in the form of 10 lookup tables (one per each matrix\n>> element and one for gamma).\n>> \n>> A tricky part is to support both CCM and the original (no-CCM)\n>> transformations. CCM obviously adds a significant overhead; per-frame\n>> time increases by ~45% on Debix Model A and ~85% on TI AM69 SK. This\n>> means CCM must be optional, which is determined by presence of Ccm\n>> algorithm in the tuning file.\n>> \n>> The performance critical debayering code must not include extra\n>> conditionals and must work efficiently for both the cases. The macros\n>> in debayering code are refactored and restructured for this. An added\n>> bonus is that this also removes some annoying code duplication. And\n>> some more, partially already present, templating is used to select\n>> between CCM and non-CCM processing.\n>> \n>> Performance is much influenced by the data structures used in\n>> debayering. I tried to define something reasonable and performed only\n>> basic performance testing. Suggestions for possible improvements are\n>> welcome.\n>\n> Thank you for writing this. I have tested this on a ThinkPad X1 Yoga\n> with a 13th Gen Intel(R) CoreT i7-1370P.\n>\n> Pinning qcam and all its threads to only the 6 performance cores /\n> 12 threads:\n>\n> taskset 0xfff ./redhat-linux-build/src/apps/qcam/qcam\n>\n> I consistently get 6.3 ms / frame processing time without the CCM\n> both with and without your patches and 9.5 ms / frame with the CCM\n> snippet in uncalibrated.yaml activated. So it seems that we should\n> be able to activate the CCM with the software ISP in the IPU6 case.\n\nCool, thank you for testing.\n\n> I did notice that the image becomes quite green with the default\n> 1 - 1 - 1 CCM, so it seems that currently using the CCM ignores\n> the color-gains calculated by the simple IPA.\n\nYes.\n\n> I'm pretty sure that real ISP-s have both color gains and a CCM,\n> so I think that we should apply the color gains to the CCM\n> when calculating the lookup tables.\n\nOK, since the CCM depends on temperature, I thought the gains might be\nalready included, but it indeed doesn't seem to be the case.\n\n> E.g. multiply CcmColumn.r by red-gain before using it to fill\n> the lookup-table, etc.\n\nThis would be easy to do.\n\n> And then it would be up to the IPA code to either do\n> whitebalance through the gains or directly in the ccm,\n> e.g. when using a ccm to enhance saturation then awb\n> could still be done through the gains.\n\nWhy should CCM + saturation need applying gains separately rather than\nin the CCM?\n\n> Regards,\n>\n> Hans\n>\n>\n>\n>> \n>> See the commit messages for more details.\n>> \n>> Changes in v2:\n>> - Fix of the gamma table index upper boundary.\n>> - Fix of red<->blue being temporarily swapped in an intermediate patch.\n>> - Fix of CCM transformation when swapRedBlueGains_ is true.\n>> - A related minor lookup data structures rearrangement, with an\n>> unintended but welcome effect of ~10% speedup with CCM in my\n>> environment. \n>> \n>> Milan Zamazal (9):\n>> libcamera: software_isp: Determine color temperature\n>> libcamera: software_isp: Store color temperature to metadata\n>> libcamera: software_isp: lut: Remove maybe_unused on a used argument\n>> libcamera: software_isp: Use common code to store debayered pixels\n>> libcamera: software_isp: Use a macro to assign debayering methods\n>> libcamera: software_isp: Add CCM algorithm\n>> libcamera: software_isp: Add an example CCM to uncalibrated.yaml\n>> libcamera: software_isp: Track whether CCM is enabled\n>> libcamera: software_isp: Apply CCM in debayering\n>> \n>> .../internal/software_isp/debayer_params.h | 20 ++-\n>> include/libcamera/ipa/soft.mojom | 2 +-\n>> src/ipa/simple/algorithms/awb.cpp | 18 ++-\n>> src/ipa/simple/algorithms/ccm.cpp | 86 ++++++++++++\n>> src/ipa/simple/algorithms/ccm.h | 45 ++++++\n>> src/ipa/simple/algorithms/lut.cpp | 62 +++++---\n>> src/ipa/simple/algorithms/lut.h | 1 +\n>> src/ipa/simple/algorithms/meson.build | 1 +\n>> src/ipa/simple/data/uncalibrated.yaml | 7 +\n>> src/ipa/simple/ipa_context.h | 22 ++-\n>> src/ipa/simple/soft_simple.cpp | 8 +-\n>> src/libcamera/software_isp/debayer.cpp | 53 ++++++-\n>> src/libcamera/software_isp/debayer.h | 3 +-\n>> src/libcamera/software_isp/debayer_cpu.cpp | 132 ++++++++++--------\n>> src/libcamera/software_isp/debayer_cpu.h | 31 ++--\n>> src/libcamera/software_isp/software_isp.cpp | 11 +-\n>> 16 files changed, 393 insertions(+), 109 deletions(-)\n>> create mode 100644 src/ipa/simple/algorithms/ccm.cpp\n>> create mode 100644 src/ipa/simple/algorithms/ccm.h\n>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 447C2BDB13\n\tfor ;\n\tWed, 4 Dec 2024 09:38:23 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2F3706609E;\n\tWed, 4 Dec 2024 10:38:22 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 31816618B5\n\tfor ;\n\tWed, 4 Dec 2024 10:38:18 +0100 (CET)","from mail-wr1-f70.google.com (mail-wr1-f70.google.com\n\t[209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n\tus-mta-298-wzC_tQPwOi-tk9b5kfP2lw-1; Wed, 04 Dec 2024 04:38:16 -0500","by mail-wr1-f70.google.com with SMTP id\n\tffacd0b85a97d-385e03f54d0so2217479f8f.3\n\tfor ;\n\tWed, 04 Dec 2024 01:38:16 -0800 (PST)","from nuthatch (ip-77-48-47-2.net.vodafone.cz. [77.48.47.2])\n\tby smtp.gmail.com with ESMTPSA id\n\tffacd0b85a97d-385ee4d932asm9498837f8f.26.2024.12.04.01.38.13\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tWed, 04 Dec 2024 01:38:13 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"dHTByAZY\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1733305097;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=E/y/esNM+fICuhx6O38HwQetOHolGN/oQSeAoxTyTRE=;\n\tb=dHTByAZYmdPVfV1Kfw3CDR8tf7hotcHdMUO8uBnza7lulTbVCpXRFRZilzbd9brWFAColR\n\tE4aGAL7+yZTZ8vsd+xcSW+IOxD4DeiFLdhAGZIsznuXCrUVW/UWwIHhUutVSkURyPSSDtL\n\tKY3dLrMqUXXB1e0NYUhCSkzvAarlQCA=","X-MC-Unique":"wzC_tQPwOi-tk9b5kfP2lw-1","X-Mimecast-MFC-AGG-ID":"wzC_tQPwOi-tk9b5kfP2lw","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1733305094; x=1733909894;\n\th=mime-version:user-agent:message-id:date:references:in-reply-to\n\t:subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=E/y/esNM+fICuhx6O38HwQetOHolGN/oQSeAoxTyTRE=;\n\tb=gB7g20yarOYgP8EHm7yVyr40Y33kAhS4hW1LTmhqBi12lldQWcGJ6HwWdV+SlQ9/MI\n\tdu8kQGqodNp43SUWzTeef3bw5Jd39hFQGeGu8/nr2s2H4wY2j8KPJA8C6dA42Am2oQYI\n\t1zBMPHQ9n0wVOqOMjK2xWD35/SDa02bNOpvvm//7bRVKaiA2gx0eMkVa+2UWKu2IZvqu\n\tcAclfAxKsk8LCHpV64/o+gR4xRfmpwHAxL2eVbESplSU363Xun9KnLdwJcLEmKuiX5Lv\n\tZoGFTvqh4tUftTcIrp2Ng3iIdaaO/EpOvEFHHkUezObhyV4JN6GIXOF00mdfcEicED+d\n\tlBuw==","X-Gm-Message-State":"AOJu0YyUwvhaCllmU64/3F0u4HQgwfuHZajPZGTJQEtvrMSjk6E0Xoid\n\tKVKbKQ9Kg59IEEEYIXIPlm6DkXsbMp+2GxwBv5Z5Tprf5QXwPeEDZW/2wl0kTuvBgUZUaowO25g\n\t6noDhrS1yK6ZgatSkGwnUO0ZBV2hbcZGL51C7qsRE4+g2jMpLm1Q6+sspJcnvfI676/oaocaODY\n\t3BQok=","X-Gm-Gg":"ASbGncuBwXb+wohB/DidqglIKgnfPAtBhysfD+Z3ZULc++znuRhdwr3w0wURYq5cEHC\n\tOD2IkbpVerutiwujCCUUUzinTkt9myQ+5lCCuNi/B3lK2HMH2U6EvF7PJNPiAqyPm/ZwjKIpQe+\n\t5Ovuy5Jqe3zabmh4eP1ituSMhfMQPWkzoyVyUYK+jBLUUWq0O1r5cU/Rb4Vizy7cSHjjYsagUAn\n\tAa23CztSGahuVvlaSufPW4+SgvxFUJ2ufgLFjnauSERgqlXl3ASmbGqHzghwI6UOKALu0A=","X-Received":["by 2002:a05:6000:186b:b0:385:ec89:2f07 with SMTP id\n\tffacd0b85a97d-38607ae4117mr2474432f8f.32.1733305094594; \n\tWed, 04 Dec 2024 01:38:14 -0800 (PST)","by 2002:a05:6000:186b:b0:385:ec89:2f07 with SMTP id\n\tffacd0b85a97d-38607ae4117mr2474413f8f.32.1733305094194; \n\tWed, 04 Dec 2024 01:38:14 -0800 (PST)"],"X-Google-Smtp-Source":"AGHT+IGCkSjvGA8W/J4F9M/9XnIP0OAzVcbY39aGfWRvcRq3Epf7EdCBUy6vH2CjVSCDnXqtok+3nQ==","From":"Milan Zamazal ","To":"Hans de Goede ","Cc":"libcamera-devel@lists.libcamera.org, Robert Mader\n\t","Subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","In-Reply-To":"<1651334c-d1c0-4fe5-9fe9-2f7957c3d1d8@redhat.com> (Hans de\n\tGoede's message of \"Tue, 3 Dec 2024 18:19:41 +0100\")","References":"<20241125180310.1254519-1-mzamazal@redhat.com>\n\t<1651334c-d1c0-4fe5-9fe9-2f7957c3d1d8@redhat.com>","Date":"Wed, 04 Dec 2024 10:38:12 +0100","Message-ID":"<8734j3vldn.fsf@redhat.com>","User-Agent":"Gnus/5.13 (Gnus v5.13)","MIME-Version":"1.0","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"tfzhE4ZlPzlipvnFKbBLqkUzZzMCr-FTGyEvKbxjopw_1733305095","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":32610,"web_url":"https://patchwork.libcamera.org/comment/32610/","msgid":"<173368419467.3479610.14371613209171858348@ping.linuxembedded.co.uk>","date":"2024-12-08T18:56:34","subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Milan,\n\nI'm trying to work through and test this series, but unfortunately I\nthink the master branch moved fast on this one and there are a couple of\nareas not compiling. It seems to be around the use of matrix which moved\nto libcamera/internal/matrix.h and then I also get issues with the\nactivestate, when I tried to work through those manually.\n\nCould you rebase this series when you get chance please?\n\n--\nKieran\n\nQuoting Milan Zamazal (2024-11-25 18:03:01)\n> This series implements application of color correction matrices in\n> software ISP.\n> \n> Support for color temperature is added to auto white balance algorithm,\n> this is needed to obtain the right CCM. A new Ccm algorithm is added to\n> determine the matrix. Both roughly follow what the hardware pipelines\n> do.\n> \n> Lut algorithm is modified to incorporate the CCM and perform some\n> precomputations in the form of 10 lookup tables (one per each matrix\n> element and one for gamma).\n> \n> A tricky part is to support both CCM and the original (no-CCM)\n> transformations. CCM obviously adds a significant overhead; per-frame\n> time increases by ~45% on Debix Model A and ~85% on TI AM69 SK. This\n> means CCM must be optional, which is determined by presence of Ccm\n> algorithm in the tuning file.\n> \n> The performance critical debayering code must not include extra\n> conditionals and must work efficiently for both the cases. The macros\n> in debayering code are refactored and restructured for this. An added\n> bonus is that this also removes some annoying code duplication. And\n> some more, partially already present, templating is used to select\n> between CCM and non-CCM processing.\n> \n> Performance is much influenced by the data structures used in\n> debayering. I tried to define something reasonable and performed only\n> basic performance testing. Suggestions for possible improvements are\n> welcome.\n> \n> See the commit messages for more details.\n> \n> Changes in v2:\n> - Fix of the gamma table index upper boundary.\n> - Fix of red<->blue being temporarily swapped in an intermediate patch.\n> - Fix of CCM transformation when swapRedBlueGains_ is true.\n> - A related minor lookup data structures rearrangement, with an\n> unintended but welcome effect of ~10% speedup with CCM in my\n> environment. \n> \n> Milan Zamazal (9):\n> libcamera: software_isp: Determine color temperature\n> libcamera: software_isp: Store color temperature to metadata\n> libcamera: software_isp: lut: Remove maybe_unused on a used argument\n> libcamera: software_isp: Use common code to store debayered pixels\n> libcamera: software_isp: Use a macro to assign debayering methods\n> libcamera: software_isp: Add CCM algorithm\n> libcamera: software_isp: Add an example CCM to uncalibrated.yaml\n> libcamera: software_isp: Track whether CCM is enabled\n> libcamera: software_isp: Apply CCM in debayering\n> \n> .../internal/software_isp/debayer_params.h | 20 ++-\n> include/libcamera/ipa/soft.mojom | 2 +-\n> src/ipa/simple/algorithms/awb.cpp | 18 ++-\n> src/ipa/simple/algorithms/ccm.cpp | 86 ++++++++++++\n> src/ipa/simple/algorithms/ccm.h | 45 ++++++\n> src/ipa/simple/algorithms/lut.cpp | 62 +++++---\n> src/ipa/simple/algorithms/lut.h | 1 +\n> src/ipa/simple/algorithms/meson.build | 1 +\n> src/ipa/simple/data/uncalibrated.yaml | 7 +\n> src/ipa/simple/ipa_context.h | 22 ++-\n> src/ipa/simple/soft_simple.cpp | 8 +-\n> src/libcamera/software_isp/debayer.cpp | 53 ++++++-\n> src/libcamera/software_isp/debayer.h | 3 +-\n> src/libcamera/software_isp/debayer_cpu.cpp | 132 ++++++++++--------\n> src/libcamera/software_isp/debayer_cpu.h | 31 ++--\n> src/libcamera/software_isp/software_isp.cpp | 11 +-\n> 16 files changed, 393 insertions(+), 109 deletions(-)\n> create mode 100644 src/ipa/simple/algorithms/ccm.cpp\n> create mode 100644 src/ipa/simple/algorithms/ccm.h\n> \n> -- \n> 2.44.2\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7941FC3259\n\tfor ;\n\tSun, 8 Dec 2024 18:56:41 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 617F367E3D;\n\tSun, 8 Dec 2024 19:56:40 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C60AD6612F\n\tfor ;\n\tSun, 8 Dec 2024 19:56:37 +0100 (CET)","from pendragon.ideasonboard.com\n\t(cpc89244-aztw30-2-0-cust6594.18-1.cable.virginm.net [86.31.185.195])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 8D0C96DF;\n\tSun, 8 Dec 2024 19:56:06 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Js/B+Byp\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1733684166;\n\tbh=ye6opgxQ9QuxCQ/7R+FssVZPVZWgGoC7kSvwFIOi0Xo=;\n\th=In-Reply-To:References:Subject:From:Cc:To:Date:From;\n\tb=Js/B+BypdqB09W2LO8uPUhFO/t3HWfiHmFROnjHw6qFzyoIQ9R5Fv7RSynL+bsiYb\n\tnQkcugNMy12hXp/eju8FLyZ3F5s2NLRceYnO20YR5mNQLR7a47VSSczh9K115DWDCn\n\toTAIWGCMzJyHAq0D2TgeoliLGaFCAAe4t2+PoT0E=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"<20241125180310.1254519-1-mzamazal@redhat.com>","References":"<20241125180310.1254519-1-mzamazal@redhat.com>","Subject":"Re: [PATCH v2 0/9] Software ISP support for CCM","From":"Kieran Bingham ","Cc":"Milan Zamazal ,\n\tRobert Mader ","To":"Milan Zamazal , libcamera-devel@lists.libcamera.org","Date":"Sun, 08 Dec 2024 18:56:34 +0000","Message-ID":"<173368419467.3479610.14371613209171858348@ping.linuxembedded.co.uk>","User-Agent":"alot/0.10","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]