[{"id":36286,"web_url":"https://patchwork.libcamera.org/comment/36286/","msgid":"<176057115572.635357.2931568201179646446@ping.linuxembedded.co.uk>","date":"2025-10-15T23:32:35","subject":"Re: [PATCH v3 39/39] libcamera: software_isp: Add a gpuisp todo list","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Quoting Bryan O'Donoghue (2025-10-15 02:22:51)\n> List the series of things to do in GPU ISP in perceived order of\n> difficulty.\n> \n> Signed-off-by: Bryan O'Donoghue \n\nIn whatever form this file takes:\n\nAcked-by: Kieran Bingham \n\n> ---\n> src/libcamera/software_isp/gpuisp-todo.txt | 83 ++++++++++++++++++++++\n> 1 file changed, 83 insertions(+)\n> create mode 100644 src/libcamera/software_isp/gpuisp-todo.txt\n> \n> diff --git a/src/libcamera/software_isp/gpuisp-todo.txt b/src/libcamera/software_isp/gpuisp-todo.txt\n> new file mode 100644\n> index 00000000..768dcc32\n> --- /dev/null\n> +++ b/src/libcamera/software_isp/gpuisp-todo.txt\n> @@ -0,0 +1,83 @@\n> +List the TODOs in perceived order of ease.\n> +\n> +Version 3:\n> +Use dma-buf handle to generate upload texture:\n> + - eglCreateImageKHR can be used to generate the upload texture i.e.\n> + to feed the bayer data into the GPU.\n> +\n> +Denoising:\n> + - As below still TBD\n> +\n> +Dead pixel correction:\n> + - As below still TBD\n> +\n> +Lense shading correction:\n> + - This is WIP but as yet still TBD\n> +\n> +Lense flare correction:\n> + - Not WIP still TBD\n> +\n> +processFrame() to run in its own thread:\n> + - processFrame() runs in the context of the Debayer::process()\n> + thread. Robert Mader suggested and it seems like a good\n> + suggestion too to run processFrame() in its own thread.\n> +\n> +Version 2:\n> +Make GPUISP default:\n> + - Right now the environment variable allows over-riding to swtich\n> + from CPU to GPU.\n> + - Once we support 24 BPP output on GPUISP we will have the same\n> + pixel format support as CPU and can set the default to GPU without\n> + regressing functionality\n> +\n> +glTexture1D:\n> + - Initial code was developed for < GLES 2.O but since we have fixed\n> + on GLES >= 2.0 this means we can use glTexture1D\n> + - Provided this is so amend the shaders to do val = texture(x, y, 0);\n> + not texture(x, y, 0.5) the 0.5 is because of using glTexture2D\n> +\n> +Denoising:\n> + - Run a denoise algorithm in the shaders\n> + - Supply a control to influence the noise-floor ?\n> +\n> +Dead pixel correction:\n> + - Add logic to correct dead pixels in the fragment shaders\n> +\n> +Version 1:\n> +24 bit output support:\n> + - Take the BPP we already capture and get a 24 bit GBM surface\n> + - Pass a compile-time parameter to the shaders to tell them to do\n> + gl_FragColor = rgb not gl_FragColor = rgba\n> + - Version 2:\n> + This is not possible.\n> + gl_FragColor expects vec4 not vec3 on the output.\n> + If you really want RGB888 run cpuisp.\n> +\n> +Surfaceless GBM:\n> + - We get a GBM surface and then have to swap buffers\n> + If we rework for surfaceless GBM and EGL then the swap buffer can\n> + be dropped.\n> + - Version 2:\n> + Complete GBM surface removed, memcpy() phase removed also\n> +\n> +dma-buf texture upload:\n> + - Currently we pass the input buffer to glCreateTexture2D.\n> + We should be able to make the upload of the input buffer go faster\n> + by using eglCreateImageKHR and enumerated the dma-buf contents.\n> + - Version 2:\n> + Complete sm8250 test platform shows 20x performance increase\n> + with CCM.\n> +\n> +Render-to-texture:\n> + - Right now we render to the GBM provided surface framebuffer\n> + and then memcpy from that buffer to the target output buffer.\n> + This necessitates flushing the cache on the target buffer in\n> + addition to the memcpy().\n> + - Render-to-texture where we generate the target framebuffer\n> + directly from a dma-buf handle will mitigate the memcpy() phase.\n> + - It should be the case then that the consumer of the output buffer\n> + i.e. the thing that's not libcamera is responsible to flush the cache\n> + if-and-only-if that user writes to the buffer.\n> + - We need to flush the cache on the buffer because we are memcpying() to it.\n> + - Version 2:\n> + Done in version 2\n> -- \n> 2.51.0\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BBD00C3259\n\tfor ;\n\tWed, 15 Oct 2025 23:32:40 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1F82360650;\n\tThu, 16 Oct 2025 01:32:40 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5362660643\n\tfor ;\n\tThu, 16 Oct 2025 01:32:38 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(cpc89244-aztw30-2-0-cust6594.18-1.cable.virginm.net [86.31.185.195])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id C84691306;\n\tThu, 16 Oct 2025 01:30:58 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"fQRe6XMH\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1760571058;\n\tbh=Om998I6Svt/GqDZ3HdflbTDq67/50MzrEMsG/y38piM=;\n\th=In-Reply-To:References:Subject:From:Cc:To:Date:From;\n\tb=fQRe6XMHwUGm2nYXR8pR4FSWMlsWP01GcsBn0M9wkskF3tF8XBpih5WvXtBF6dVE1\n\tWqu672rWczR57b3K0qKKugl+NmJGG286wRP60ypSW+C/um0d0vEB7gUHlpD5aBlxTQ\n\t6WxNGGHaXg6nOX3WHpn8ksKb0kJI2xftn31Dz8eE=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"<20251015012251.17508-40-bryan.odonoghue@linaro.org>","References":"<20251015012251.17508-1-bryan.odonoghue@linaro.org>\n\t<20251015012251.17508-40-bryan.odonoghue@linaro.org>","Subject":"Re: [PATCH v3 39/39] libcamera: software_isp: Add a gpuisp todo list","From":"Kieran Bingham ","Cc":"hdegoede@redhat.com, mzamazal@redhat.com, bryan.odonoghue@linaro.org,\n\tbod.linux@nxsw.ie","To":"Bryan O'Donoghue ,\n\tlibcamera-devel@lists.libcamera.org","Date":"Thu, 16 Oct 2025 00:32:35 +0100","Message-ID":"<176057115572.635357.2931568201179646446@ping.linuxembedded.co.uk>","User-Agent":"alot/0.9.1","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36323,"web_url":"https://patchwork.libcamera.org/comment/36323/","msgid":"<85ikgebyqq.fsf@mzamazal-thinkpadp1gen7.tpbc.csb>","date":"2025-10-16T16:25:01","subject":"Re: [PATCH v3 39/39] libcamera: software_isp: Add a gpuisp todo list","submitter":{"id":177,"url":"https://patchwork.libcamera.org/api/people/177/","name":"Milan Zamazal","email":"mzamazal@redhat.com"},"content":"Hi Bryan,\n\nBryan O'Donoghue writes:\n\n> List the series of things to do in GPU ISP in perceived order of\n> difficulty.\n>\n> Signed-off-by: Bryan O'Donoghue \n> ---\n> src/libcamera/software_isp/gpuisp-todo.txt | 83 ++++++++++++++++++++++\n> 1 file changed, 83 insertions(+)\n> create mode 100644 src/libcamera/software_isp/gpuisp-todo.txt\n>\n> diff --git a/src/libcamera/software_isp/gpuisp-todo.txt b/src/libcamera/software_isp/gpuisp-todo.txt\n> new file mode 100644\n> index 00000000..768dcc32\n> --- /dev/null\n> +++ b/src/libcamera/software_isp/gpuisp-todo.txt\n> @@ -0,0 +1,83 @@\n> +List the TODOs in perceived order of ease.\n> +\n> +Version 3:\n\nDo the versions here serve any useful purpose? I'd simply keep a plain\nlist of current items, ordered by difficulty as stated above. Whatever\ngets implemented should be dropped from here in the corresponding final\ncommit.\n\n> +Use dma-buf handle to generate upload texture:\n> +\t- eglCreateImageKHR can be used to generate the upload texture i.e.\n> +\t to feed the bayer data into the GPU.\n> +\n> +Denoising:\n> +\t- As below still TBD\n\nFYI I work on this and I have some working very early proof-of-concept.\nI managed to get rid of the artefacts demonstrated at the yesterday's\nsync by changing some value but it's a complete mystery to me why it\nhelps. So more fun before I can dive into the real thing.\n\n> +Dead pixel correction:\n> +\t- As below still TBD\n> +\n> +Lense shading correction:\n> +\t- This is WIP but as yet still TBD\n> +\n> +Lense flare correction:\n> +\t- Not WIP still TBD\n> +\n> +processFrame() to run in its own thread:\n> +\t- processFrame() runs in the context of the Debayer::process()\n> +\t thread. Robert Mader suggested and it seems like a good\n> +\t suggestion too to run processFrame() in its own thread.\n> +\n> +Version 2:\n> +Make GPUISP default:\n> +\t- Right now the environment variable allows over-riding to swtich\n\ns/swtich/switch/\n\n> +\t from CPU to GPU.\n> +\t- Once we support 24 BPP output on GPUISP we will have the same\n> +\t pixel format support as CPU and can set the default to GPU without\n> +\t regressing functionality\n\nTo keep or to drop, considering the related item below about this not\nbeing possible?\n\n> +glTexture1D:\n> +\t- Initial code was developed for < GLES 2.O but since we have fixed\n> +\t on GLES >= 2.0 this means we can use glTexture1D\n> +\t- Provided this is so amend the shaders to do val = texture(x, y, 0);\n> +\t not texture(x, y, 0.5) the 0.5 is because of using glTexture2D\n> +\n> +Denoising:\n> +\t- Run a denoise algorithm in the shaders\n> +\t- Supply a control to influence the noise-floor ?\n> +\n> +Dead pixel correction:\n> +\t- Add logic to correct dead pixels in the fragment shaders\n> +\n> +Version 1:\n> +24 bit output support:\n> +\t- Take the BPP we already capture and get a 24 bit GBM surface\n> +\t- Pass a compile-time parameter to the shaders to tell them to do\n> +\t gl_FragColor = rgb not gl_FragColor = rgba\n> +\t- Version 2:\n> +\t This is not possible.\n> +\t gl_FragColor expects vec4 not vec3 on the output.\n> +\t If you really want RGB888 run cpuisp.\n\nIf the following is done then let's drop it all from here?\n\n> +Surfaceless GBM:\n> +\t- We get a GBM surface and then have to swap buffers\n> +\t If we rework for surfaceless GBM and EGL then the swap buffer can\n> +\t be dropped.\n> +\t- Version 2:\n> +\t Complete GBM surface removed, memcpy() phase removed also\n> +dma-buf texture upload:\n> +\t- Currently we pass the input buffer to glCreateTexture2D.\n> +\t We should be able to make the upload of the input buffer go faster\n> +\t by using eglCreateImageKHR and enumerated the dma-buf contents.\n> +\t- Version 2:\n> +\t Complete sm8250 test platform shows 20x performance increase\n> +\t with CCM.\n> +\n> +Render-to-texture:\n> +\t- Right now we render to the GBM provided surface framebuffer\n> +\t and then memcpy from that buffer to the target output buffer.\n> +\t This necessitates flushing the cache on the target buffer in\n> +\t addition to the memcpy().\n> +\t- Render-to-texture where we generate the target framebuffer\n> +\t directly from a dma-buf handle will mitigate the memcpy() phase.\n> +\t- It should be the case then that the consumer of the output buffer\n> +\t i.e. the thing that's not libcamera is responsible to flush the cache\n> +\t if-and-only-if that user writes to the buffer.\n> +\t- We need to flush the cache on the buffer because we are memcpying() to it.\n> +\t- Version 2:\n> +\t Done in version 2","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 357E2BE080\n\tfor ;\n\tThu, 16 Oct 2025 16:25:12 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 544FE6068C;\n\tThu, 16 Oct 2025 18:25:11 +0200 (CEST)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 063BB605D7\n\tfor ;\n\tThu, 16 Oct 2025 18:25:09 +0200 (CEST)","from mail-ej1-f71.google.com (mail-ej1-f71.google.com\n\t[209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS\n\t(version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n\tus-mta-301-RPSKyPo6Oqy3hUx-hnSb2g-1; Thu, 16 Oct 2025 12:25:06 -0400","by mail-ej1-f71.google.com with SMTP id\n\ta640c23a62f3a-b5a8827e692so89445966b.2\n\tfor ;\n\tThu, 16 Oct 2025 09:25:04 -0700 (PDT)","from mzamazal-thinkpadp1gen7.tpbc.csb\n\t(ip-77-48-47-2.net.vodafone.cz. [77.48.47.2])\n\tby smtp.gmail.com with ESMTPSA id\n\ta640c23a62f3a-b5ccd7b202dsm564446866b.82.2025.10.16.09.25.01\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tThu, 16 Oct 2025 09:25:02 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"Tft+Gvi9\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1760631908;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=fNVtEXpQL0KIKvL8RrDlbDf6mlUaTWDgn+xSk+07ovo=;\n\tb=Tft+Gvi9yly7qx+HLplvi0dWIdSLIxnNY3ko3ahlhYjSAlT+mwKqreUtAO/hG/AZy4HBCP\n\tA+O5KdBw5hm754QD16yITuqmppHtJkuLUzdfjX3AnJyf96T/0ujAcG5A9mndi1NrCTOIqQ\n\tZeIrSE/RVwukWvFnBI692l0SO3mYVhc=","X-MC-Unique":"RPSKyPo6Oqy3hUx-hnSb2g-1","X-Mimecast-MFC-AGG-ID":"RPSKyPo6Oqy3hUx-hnSb2g_1760631903","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1760631903; x=1761236703;\n\th=mime-version:user-agent:message-id:date:references:in-reply-to\n\t:subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=fNVtEXpQL0KIKvL8RrDlbDf6mlUaTWDgn+xSk+07ovo=;\n\tb=U9OZSokxjkHUc465/lhsDtES5xB5eLuZm2HfIbgeDw4PhdoB1oc0Rsp8qkIsXr4dEJ\n\tHsWUWvZJSkgQSPKCDkehGxXMLF3XUISPNn2Z+s0gzM9qNolmBdGyWzmePUeZvUgC2lAU\n\tIdPxQR5yUU0vPwCVp8jQuq2HVmySGQf3xO+x0HYJ3IYkhy9HE+UpcV6mLepc3oCfnTQF\n\tUr9+n80133q326RaVOqGBxUP7pI7VYBe/GWRYwRqPR4NY2UwYXSSjF7d7IMA7/LVpgwn\n\tpP3bSxwUUd6P9TCV/WDNOFsu3LIMBDPVJ5uky+0f00iKDnRJ2Fn1gbONeBqYtxelc+tl\n\tAgbw==","X-Gm-Message-State":"AOJu0YyTfVKDMCrNNE1OPmKUBxSZvBz/pw3HjYpVce4Qnlm+TacZZlfC\n\tgHmbim47ukGTAbYjM7s4c1PvKaXEXUif9KSfXVhOdXxt5HdD54v8Lc9YceLSKhHFr6Pi+EquOhZ\n\tRbqo4b+1Jxn23Q6or/WogzWVfHReZ8QbuCARMk0FgdB6PIUhlcH+qpjZP5RUiR4mzzrHaSw4RHK\n\tjlhFsaqEw=","X-Gm-Gg":"ASbGncsWho/+NUqJWvMjE4PfOsvWUWpL12uWJc1zVQddXDmKPUeNabFvQv7Ttpnq7KW\n\t9i5hrUsen7L1dJj3kdLt91czoLGV2l/vldoTZlwLq2gndbvOdx5DoKMvvjL0+XNJtIoOjK+otK2\n\tfE0eMlTkIOKwDDk5geWTqJXJod6spHCN4oGd+rU4Zdzo1oeekeWQu4IcFN8vR/WiNk1AUwCqsGU\n\t+FE6Oyi5j3rRPc3TZiLXkDiCBCwXohvEi24rVnFUU4EPvfXP/PjPmQKZowwKj+qIKvaDrp2xLhl\n\tPSUWxN9RYttUInUiZZG+WjqHAw4y3n2jQ2TtfIkfDA3PEcVv0hEWmxxyQC6wsJebyrdRtTb/08p\n\tG5a9Jmrta/HPkEp4u9zRgkBXPpp8a/RidnoqrxJbfzznVivPxtKf6","X-Received":["by 2002:a17:906:6a0a:b0:b4a:ac1e:7866 with SMTP id\n\ta640c23a62f3a-b6473f42446mr62871466b.45.1760631903061; \n\tThu, 16 Oct 2025 09:25:03 -0700 (PDT)","by 2002:a17:906:6a0a:b0:b4a:ac1e:7866 with SMTP id\n\ta640c23a62f3a-b6473f42446mr62867066b.45.1760631902547; \n\tThu, 16 Oct 2025 09:25:02 -0700 (PDT)"],"X-Google-Smtp-Source":"AGHT+IGkqx353i1DcxnM4wgMCiLrX2DGvIAuniINV669UolOM4uq0Ij2oj6cS/8eR/nHSD8nDfRUxg==","From":"Milan Zamazal ","To":"Bryan O'Donoghue ","Cc":"libcamera-devel@lists.libcamera.org, hdegoede@redhat.com,\n\tbod.linux@nxsw.ie","Subject":"Re: [PATCH v3 39/39] libcamera: software_isp: Add a gpuisp todo list","In-Reply-To":"<20251015012251.17508-40-bryan.odonoghue@linaro.org> (Bryan\n\tO'Donoghue's message of \"Wed, 15 Oct 2025 02:22:51 +0100\")","References":"<20251015012251.17508-1-bryan.odonoghue@linaro.org>\n\t<20251015012251.17508-40-bryan.odonoghue@linaro.org>","Date":"Thu, 16 Oct 2025 18:25:01 +0200","Message-ID":"<85ikgebyqq.fsf@mzamazal-thinkpadp1gen7.tpbc.csb>","User-Agent":"Gnus/5.13 (Gnus v5.13)","MIME-Version":"1.0","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"c7USfPNEd6gzDp6qH0YfPQnsaybD8XAZzMtr1DyMY80_1760631903","X-Mimecast-Originator":"redhat.com","Content-Type":"text/plain","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]