From patchwork Wed Jun 11 01:32:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bryan O'Donoghue X-Patchwork-Id: 23506 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 90997BDE6B for ; Wed, 11 Jun 2025 01:32:58 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6EF2E68DBD; Wed, 11 Jun 2025 03:32:57 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="iwEyQk4i"; dkim-atps=neutral Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 43D566154C for ; Wed, 11 Jun 2025 03:32:55 +0200 (CEST) Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-4531e146a24so9700495e9.0 for ; Tue, 10 Jun 2025 18:32:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1749605574; x=1750210374; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=gJAbBkvyfXZuLPb+kwGJ2DuwOlkjpLcTwHpxY3RUYos=; b=iwEyQk4i5VFNfYGWOzZY+ZbPRGQIOd+1oeOKi9NbzZzQfEyBNGKh7fWIX+slFpmYCC Nq19NKL8QdsIjC7ajHtbQlp15Xhxa9EMveRwP0bHpK+8De0KJaZUlDTJCJo4V22bXRM2 La1hNBfTQP+9hZ9CVAtfj0emt9BnC5s69/2uzxrNypwbrCih4oVKGOFJ2ZUcz69lJYwQ d6WGANuqrZ5Pge/8YxqFKoouKj7POMTK28L2rT2x124/7yaOJxnArcs1LdqlrucjA6VR VLbvUF4OUhGPZp0bAXkGB4IjId2OySCnoSF7fbPlq09Vdz4h6o6F71E7BHD8Av7AyNTm JXNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749605574; x=1750210374; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gJAbBkvyfXZuLPb+kwGJ2DuwOlkjpLcTwHpxY3RUYos=; b=IZCaSESXOJo1oPnys+XJVdG9r2JYt9DGpwaI12QT4kKmxuuEZuAoTLZzYzs8hgTbAG 90g3Adx8uZc8ShDB7a1uB4//eZhhUfQYhDlXJ5XOpXPxyiwm+j7eLB+p5Vmp6lsllpVH 0/Vdaix76eoyvzOv1w0aOs5cacy+CJ5yCt2Hk9LcMfc3EvRm349hoLEPgq2oFWZ9yowc G7UcZ1RO/UCpTsl6Du8WKqg7GTwQ5TalW6jlHfoXmrsLBa6614rzWmyAf6NmoDasC2V4 CWWSIt99lRpnLDMsSZwsd3n8NsAUJ7yXbrTDYux93LbgFEk8Yzg3cmRS3zhbl4APA9Dp xfOA== X-Gm-Message-State: AOJu0YyO3OekGcLuCt1SYw1+mNrqXUnrISpletuh9brJeVALbCBcnSIL 5Bk0LkfoScrd6o335b/Oh3EPT+Lt1DyhEMVtpoemsqRL3dSNXz+479QHcdV6m0H0YJfBaveC31N rwOTujdQ= X-Gm-Gg: ASbGncsLQYMWpzT7M89rGj+4kEbVFsNjmrHhIr4ea+Hu0zMAx1nnbrn0LfRzb+ztY/V Wjzty8oecLoHP9p+BsuB0R9kk2dYczgSZkxRmulJZfeoWl3slp5DwacxKa6QU9BgE4NOsOarDMO b0ksYphzrxolmkRrrZuHoiXi3nHzUI9UOcObFhgeBnhfm2UmyPwRBpmqaL1vkt4j0tVJjnDAjXz vSLuOgOSnMosk2V9xkHQqJYDh2wqTOP4XFEO1NWk4QJX99Pxyru/szoCgfNFiPURbjs8hR0vCfr +hqls2BGoTBmt6e5W2wUvfDPi5hjTi6ZrNKhE7H6uKSKrN/12/e1kHaBuBYrMT7YxB7+LXybeOK Ro2rai8gGADpfKKdt3PONyHOwbcdbMImEyOCjL3tTEg== X-Google-Smtp-Source: AGHT+IGXUWRJbDFGXGpqICXN3qNOc3vD/Ocby5FRg5X+rxMO+yH8TxHE9JK5iz3IzKqNno0VNYC4nA== X-Received: by 2002:a05:600c:4f95:b0:43d:172:50b1 with SMTP id 5b1f17b1804b1-453248dbf28mr9494065e9.29.1749605574391; Tue, 10 Jun 2025 18:32:54 -0700 (PDT) Received: from inspiron14p-linux.ht.home (188-141-3-146.dynamic.upc.ie. [188.141.3.146]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4532514138asm5680625e9.3.2025.06.10.18.32.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Jun 2025 18:32:53 -0700 (PDT) From: Bryan O'Donoghue To: libcamera-devel@lists.libcamera.org Cc: Bryan O'Donoghue Subject: [PATCH 00/35] Add GLES 2.0 GPUISP to libcamera Date: Wed, 11 Jun 2025 02:32:10 +0100 Message-ID: <20250611013245.133785-1-bryan.odonoghue@linaro.org> X-Mailer: git-send-email 2.49.0 MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This series introduces a GLES 2.0 GPU ISP to libcamera. We have had extensive discussions, meetings and collaborative discussions about this topic over the last year or so. As an overview we want to start to move as much processing of software_isp into the GPU as possible. This is especially advantageous when we are talking about processing a framebuffer's worth of pixels as quickly as possible. The decision to use GLES 2.0 instead of say Vulcan stems from a desire to support as much in the way of older hardware as possible and the fact we already have upstream GLES 2.0 fragment shaders to do debayer. Generally the approach is - Move the fragment shaders out of qcam and into a common location - Update the existing SoftwareISP Debayer/DebayerCPU pair to facilitate addition of a new class DebayerEGL. - Introduce that class - Then do progressive change of the shaders and DebayerEGL class to make the modifications as transparent as possible in the git log. - Reuse as much of the SoftIPA data-structures and logic as possible. - Consume the data from SoftIPA in the Debayer Shaders so that CPUISP and GPUISP give similar - hopefully the same results but with GPUISP going faster. In order to get untiled and uncompressed pixel data out of the GPU framebuffer we need to tell the GPU how to store the data it is writing to that framebuffer. GPUs can store their framebuffer data in tiled or even compressed formats which is why the naive approach of running your fragment shader and then using glReadPixels(GL_RGBA); will be horrendously slow as glReadPixels must convert from the internal GPU format to the requested output format - an operation that for me takes ~ 10 milliseconds per frame. Instead we get the GPU to store its data as ARGB8888 swap buffers and memcpy() from the swapped buffer to our output frame. Right now this series supports 32 bit output formats only. The memcpy() also entails flushing the cache of the target buffer as per the terms of the dma-buf software contract. This leads us onto the main outstanding TODOs - 24 bit GBM buffer support leading - 24 bit output framebuffer support - Surfaceless GBM and eGL context with no swapbuffer - Render to texture If we render directly to a buffer provided to the GPU the output buffer we will not need to memcpy() to the output buffer nor will we need to invalidate the output buffer cache. - eglCreateImageKHR for the texture upload. This list is of the colour "make it go faster" not "make it work" which is why we are moving to start to submit a v1 for discussion in the full realisation it will have to go through several cycles of review giving us the opportunity to fix: - Doxygen is missing for new classes and methods - Some of the pipelines don't complete in gitlab - 24 bit output seems doable before merge - Render to texture perhaps even too For me on my Qualcomm hardware GPUISP works very well I get 30fps in qcam with about 75% CPU usage versus > 100% - cam goes faster which to me implies a good bit of time is being consumed in qcam itself. The series starts out with fixes and updates from Hans and finishes it out with shader modifications from Milan both of whom along with Kieran, Laurent and Maxime I'd like to thank for being some helpful and patient. Bryan O'Donoghue (27): libcamera: MappedFrameBuffer: Latch a pointer to the framebuffer libcamera: MappedFrameBuffer: Add MappedFrameBuffer::getPlaneFD() libcamera: software_isp: Move useful items from DebayerCpu to Debayer base class libcamera: software_isp: Move Bayer parans init from DebayerCpu to Debayer libcamera: software_isp: Move param select code to Debayer base class libcamera: software_isp: Move isStandardBayerOrder to base class libcamera: software_isp: Start the ISP thread in configure libcamera: software_isp: Move configure to worker thread libcamera: software_isp: debayer: Make the debayer_ object of type class Debayer not DebayerCpu libcamera: software_isp: debayer: Extend DebayerParams struct to hold a copy of per-frame CCM values libcamera: shaders: Move GL shader programs to src/libcamera/assets/shader utils: gen-shader-headers: Add a utility to generate headers from shaders meson: Automatically generate glsl_shaders.h from specified shader programs libcamera: software_isp: ccm: Populate CCM table to Debayer params structure libcamera: software_isp: lut: Make gain corrected CCM in lut.cpp available in debayer params libcamera: software_isp: gbm: Add in a GBM helper class for GPU surface access libcamera: software_isp: egl: Introduce an eGL base helper class libcamera: software_isp: debayer_egl: Add an eGL debayer class libcamera: software_isp: debayer_egl: Make DebayerEGL an environment option libcamera: shaders: Use highp not mediump for float precision libcamera: shaders: Extend debayer shaders to apply RGB gain values on output libcamera: software_isp: debayer_egl: Convert from identity CCM to CCM calculated by SoftIPA libcamera: software_isp: Switch on uncalibrated CCM to validate eGLDebayer libcamera: software_isp: Make isStandardBayerOrder static libcamera: software_isp: debayer_cpu: Make getInputConfig and getOutputConfig static libcamera: shaders: Extend bayer shaders to support swapping R and B on output libcamera: software_isp: Add a gpuisp todo list Hans de Goede (5): libcamera: swstats_cpu: Update statsProcessFn() / processLine0() documentation libcamera: swstats_cpu: Drop patternSize_ documentation libcamera: swstats_cpu: Move header to libcamera/internal/software_isp libcamera: software_isp: Move benchmark code to its own class libcamera: swstats_cpu: Add processFrame() method Milan Zamazal (3): libcamera: shaders: Fix neighbouring positions in 8-bit debayering libcamera: software_isp: GPU support for unpacked 10/12-bit formats libcamera: shaders: Rename bayer_8 to bayer_unpacked include/libcamera/internal/egl.h | 110 +++ include/libcamera/internal/gbm.h | 55 ++ .../libcamera/internal/mapped_framebuffer.h | 4 + include/libcamera/internal/meson.build | 11 + .../libcamera/internal/shaders}/RGB.frag | 2 +- .../internal/shaders}/YUV_2_planes.frag | 2 +- .../internal/shaders}/YUV_3_planes.frag | 2 +- .../internal/shaders}/YUV_packed.frag | 2 +- .../internal/shaders}/bayer_1x_packed.frag | 62 +- .../internal/shaders/bayer_unpacked.frag | 78 ++- .../internal/shaders/bayer_unpacked.vert | 8 +- .../libcamera/internal/shaders}/identity.vert | 0 .../libcamera/internal/shaders/meson.build | 10 + .../internal/software_isp/benchmark.h | 36 + .../internal/software_isp/debayer_params.h | 7 + .../internal/software_isp/meson.build | 2 + .../internal/software_isp/software_isp.h | 5 +- .../internal}/software_isp/swstats_cpu.h | 12 + src/apps/qcam/assets/shader/shaders.qrc | 16 +- src/apps/qcam/viewfinder_gl.cpp | 70 +- src/ipa/simple/algorithms/ccm.cpp | 4 +- src/ipa/simple/algorithms/lut.cpp | 1 + src/ipa/simple/data/uncalibrated.yaml | 12 +- src/libcamera/egl.cpp | 369 ++++++++++ src/libcamera/gbm.cpp | 137 ++++ src/libcamera/mapped_framebuffer.cpp | 7 + src/libcamera/meson.build | 34 + src/libcamera/software_isp/benchmark.cpp | 93 +++ src/libcamera/software_isp/debayer.cpp | 61 ++ src/libcamera/software_isp/debayer.h | 41 +- src/libcamera/software_isp/debayer_cpu.cpp | 81 +-- src/libcamera/software_isp/debayer_cpu.h | 44 +- src/libcamera/software_isp/debayer_egl.cpp | 632 ++++++++++++++++++ src/libcamera/software_isp/debayer_egl.h | 171 +++++ src/libcamera/software_isp/gpuisp-todo.txt | 42 ++ src/libcamera/software_isp/meson.build | 9 + src/libcamera/software_isp/software_isp.cpp | 37 +- src/libcamera/software_isp/swstats_cpu.cpp | 89 ++- utils/gen-shader-header.py | 38 ++ utils/gen-shader-headers.sh | 44 ++ utils/meson.build | 2 + 41 files changed, 2234 insertions(+), 208 deletions(-) create mode 100644 include/libcamera/internal/egl.h create mode 100644 include/libcamera/internal/gbm.h rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/RGB.frag (93%) rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/YUV_2_planes.frag (97%) rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/YUV_3_planes.frag (96%) rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/YUV_packed.frag (99%) rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/bayer_1x_packed.frag (76%) rename src/apps/qcam/assets/shader/bayer_8.frag => include/libcamera/internal/shaders/bayer_unpacked.frag (56%) rename src/apps/qcam/assets/shader/bayer_8.vert => include/libcamera/internal/shaders/bayer_unpacked.vert (85%) rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/identity.vert (100%) create mode 100644 include/libcamera/internal/shaders/meson.build create mode 100644 include/libcamera/internal/software_isp/benchmark.h rename {src/libcamera => include/libcamera/internal}/software_isp/swstats_cpu.h (86%) create mode 100644 src/libcamera/egl.cpp create mode 100644 src/libcamera/gbm.cpp create mode 100644 src/libcamera/software_isp/benchmark.cpp create mode 100644 src/libcamera/software_isp/debayer_egl.cpp create mode 100644 src/libcamera/software_isp/debayer_egl.h create mode 100644 src/libcamera/software_isp/gpuisp-todo.txt create mode 100755 utils/gen-shader-header.py create mode 100755 utils/gen-shader-headers.sh