{"id":23506,"url":"https://patchwork.libcamera.org/api/1.1/covers/23506/?format=json","web_url":"https://patchwork.libcamera.org/cover/23506/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20250611013245.133785-1-bryan.odonoghue@linaro.org>","date":"2025-06-11T01:32:10","name":"[00/35] Add GLES 2.0 GPUISP to libcamera","submitter":{"id":175,"url":"https://patchwork.libcamera.org/api/1.1/people/175/?format=json","name":"Bryan O'Donoghue","email":"bryan.odonoghue@linaro.org"},"mbox":"https://patchwork.libcamera.org/cover/23506/mbox/","series":[{"id":5212,"url":"https://patchwork.libcamera.org/api/1.1/series/5212/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=5212","date":"2025-06-11T01:32:10","name":"Add GLES 2.0 GPUISP to libcamera","version":1,"mbox":"https://patchwork.libcamera.org/series/5212/mbox/"}],"comments":"https://patchwork.libcamera.org/api/covers/23506/comments/","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 90997BDE6B\n\tfor ;\n\tWed, 11 Jun 2025 01:32:58 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6EF2E68DBD;\n\tWed, 11 Jun 2025 03:32:57 +0200 (CEST)","from mail-wm1-x334.google.com (mail-wm1-x334.google.com\n\t[IPv6:2a00:1450:4864:20::334])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 43D566154C\n\tfor ;\n\tWed, 11 Jun 2025 03:32:55 +0200 (CEST)","by mail-wm1-x334.google.com with SMTP id\n\t5b1f17b1804b1-4531e146a24so9700495e9.0\n\tfor ;\n\tTue, 10 Jun 2025 18:32:55 -0700 (PDT)","from inspiron14p-linux.ht.home (188-141-3-146.dynamic.upc.ie.\n\t[188.141.3.146]) by smtp.gmail.com with ESMTPSA id\n\t5b1f17b1804b1-4532514138asm5680625e9.3.2025.06.10.18.32.51\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tTue, 10 Jun 2025 18:32:53 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"iwEyQk4i\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=linaro.org; s=google; t=1749605574; x=1750210374;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:mime-version:message-id:date:subject:cc\n\t:to:from:from:to:cc:subject:date:message-id:reply-to;\n\tbh=gJAbBkvyfXZuLPb+kwGJ2DuwOlkjpLcTwHpxY3RUYos=;\n\tb=iwEyQk4i5VFNfYGWOzZY+ZbPRGQIOd+1oeOKi9NbzZzQfEyBNGKh7fWIX+slFpmYCC\n\tNq19NKL8QdsIjC7ajHtbQlp15Xhxa9EMveRwP0bHpK+8De0KJaZUlDTJCJo4V22bXRM2\n\tLa1hNBfTQP+9hZ9CVAtfj0emt9BnC5s69/2uzxrNypwbrCih4oVKGOFJ2ZUcz69lJYwQ\n\td6WGANuqrZ5Pge/8YxqFKoouKj7POMTK28L2rT2x124/7yaOJxnArcs1LdqlrucjA6VR\n\tVLbvUF4OUhGPZp0bAXkGB4IjId2OySCnoSF7fbPlq09Vdz4h6o6F71E7BHD8Av7AyNTm\n\tJXNA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1749605574; x=1750210374;\n\th=content-transfer-encoding:mime-version:message-id:date:subject:cc\n\t:to:from:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=gJAbBkvyfXZuLPb+kwGJ2DuwOlkjpLcTwHpxY3RUYos=;\n\tb=IZCaSESXOJo1oPnys+XJVdG9r2JYt9DGpwaI12QT4kKmxuuEZuAoTLZzYzs8hgTbAG\n\t90g3Adx8uZc8ShDB7a1uB4//eZhhUfQYhDlXJ5XOpXPxyiwm+j7eLB+p5Vmp6lsllpVH\n\t0/Vdaix76eoyvzOv1w0aOs5cacy+CJ5yCt2Hk9LcMfc3EvRm349hoLEPgq2oFWZ9yowc\n\tG7UcZ1RO/UCpTsl6Du8WKqg7GTwQ5TalW6jlHfoXmrsLBa6614rzWmyAf6NmoDasC2V4\n\tCWWSIt99lRpnLDMsSZwsd3n8NsAUJ7yXbrTDYux93LbgFEk8Yzg3cmRS3zhbl4APA9Dp\n\txfOA==","X-Gm-Message-State":"AOJu0YyO3OekGcLuCt1SYw1+mNrqXUnrISpletuh9brJeVALbCBcnSIL\n\t5Bk0LkfoScrd6o335b/Oh3EPT+Lt1DyhEMVtpoemsqRL3dSNXz+479QHcdV6m0H0YJfBaveC31N\n\trwOTujdQ=","X-Gm-Gg":"ASbGncsLQYMWpzT7M89rGj+4kEbVFsNjmrHhIr4ea+Hu0zMAx1nnbrn0LfRzb+ztY/V\n\tWjzty8oecLoHP9p+BsuB0R9kk2dYczgSZkxRmulJZfeoWl3slp5DwacxKa6QU9BgE4NOsOarDMO\n\tb0ksYphzrxolmkRrrZuHoiXi3nHzUI9UOcObFhgeBnhfm2UmyPwRBpmqaL1vkt4j0tVJjnDAjXz\n\tvSLuOgOSnMosk2V9xkHQqJYDh2wqTOP4XFEO1NWk4QJX99Pxyru/szoCgfNFiPURbjs8hR0vCfr\n\t+hqls2BGoTBmt6e5W2wUvfDPi5hjTi6ZrNKhE7H6uKSKrN/12/e1kHaBuBYrMT7YxB7+LXybeOK\n\tRo2rai8gGADpfKKdt3PONyHOwbcdbMImEyOCjL3tTEg==","X-Google-Smtp-Source":"AGHT+IGXUWRJbDFGXGpqICXN3qNOc3vD/Ocby5FRg5X+rxMO+yH8TxHE9JK5iz3IzKqNno0VNYC4nA==","X-Received":"by 2002:a05:600c:4f95:b0:43d:172:50b1 with SMTP id\n\t5b1f17b1804b1-453248dbf28mr9494065e9.29.1749605574391; \n\tTue, 10 Jun 2025 18:32:54 -0700 (PDT)","From":"Bryan O'Donoghue ","To":"libcamera-devel@lists.libcamera.org","Cc":"Bryan O'Donoghue ","Subject":"[PATCH 00/35] Add GLES 2.0 GPUISP to libcamera ","Date":"Wed, 11 Jun 2025 02:32:10 +0100","Message-ID":"<20250611013245.133785-1-bryan.odonoghue@linaro.org>","X-Mailer":"git-send-email 2.49.0","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"This series introduces a GLES 2.0 GPU ISP to libcamera.\n\nWe have had extensive discussions, meetings and collaborative discussions\nabout this topic over the last year or so.\n\nAs an overview we want to start to move as much processing of software_isp\ninto the GPU as possible. This is especially advantageous when we are\ntalking about processing a framebuffer's worth of pixels as quickly as\npossible.\n\nThe decision to use GLES 2.0 instead of say Vulcan stems from a desire to\nsupport as much in the way of older hardware as possible and the fact we\nalready have upstream GLES 2.0 fragment shaders to do debayer.\n\nGenerally the approach is\n\n- Move the fragment shaders out of qcam and into a common location\n- Update the existing SoftwareISP Debayer/DebayerCPU pair to facilitate\n addition of a new class DebayerEGL.\n- Introduce that class\n- Then do progressive change of the shaders and DebayerEGL class to make\n the modifications as transparent as possible in the git log.\n- Reuse as much of the SoftIPA data-structures and logic as possible.\n- Consume the data from SoftIPA in the Debayer Shaders so that CPUISP and\n GPUISP give similar - hopefully the same results but with GPUISP going\n faster.\n\nIn order to get untiled and uncompressed pixel data out of the GPU\nframebuffer we need to tell the GPU how to store the data it is writing to\nthat framebuffer. GPUs can store their framebuffer data in tiled or even\ncompressed formats which is why the naive approach of running your fragment\nshader and then using glReadPixels(GL_RGBA); will be horrendously slow as\nglReadPixels must convert from the internal GPU format to the requested\noutput format - an operation that for me takes ~ 10 milliseconds per frame.\n\nInstead we get the GPU to store its data as ARGB8888 swap buffers and\nmemcpy() from the swapped buffer to our output frame. Right now this series\nsupports 32 bit output formats only.\n\nThe memcpy() also entails flushing the cache of the target buffer as per\nthe terms of the dma-buf software contract.\n\nThis leads us onto the main outstanding TODOs\n\n- 24 bit GBM buffer support leading\n- 24 bit output framebuffer support\n- Surfaceless GBM and eGL context with no swapbuffer\n- Render to texture\n If we render directly to a buffer provided to the GPU the output\n buffer we will not need to memcpy() to the output buffer\n nor will we need to invalidate the output buffer cache.\n- eglCreateImageKHR for the texture upload.\n\nThis list is of the colour \"make it go faster\" not \"make it work\" which is\nwhy we are moving to start to submit a v1 for discussion in the full\nrealisation it will have to go through several cycles of review giving us\nthe opportunity to fix:\n\n- Doxygen is missing for new classes and methods\n- Some of the pipelines don't complete in gitlab\n- 24 bit output seems doable before merge\n- Render to texture perhaps even too\n\nFor me on my Qualcomm hardware GPUISP works very well I get 30fps in qcam\nwith about 75% CPU usage versus > 100% - cam goes faster which to me\nimplies a good bit of time is being consumed in qcam itself.\n\nThe series starts out with fixes and updates from Hans and finishes it out\nwith shader modifications from Milan both of whom along with Kieran,\nLaurent and Maxime I'd like to thank for being some helpful and patient.\n\nBryan O'Donoghue (27):\n libcamera: MappedFrameBuffer: Latch a pointer to the framebuffer\n libcamera: MappedFrameBuffer: Add MappedFrameBuffer::getPlaneFD()\n libcamera: software_isp: Move useful items from DebayerCpu to Debayer\n base class\n libcamera: software_isp: Move Bayer parans init from DebayerCpu to\n Debayer\n libcamera: software_isp: Move param select code to Debayer base class\n libcamera: software_isp: Move isStandardBayerOrder to base class\n libcamera: software_isp: Start the ISP thread in configure\n libcamera: software_isp: Move configure to worker thread\n libcamera: software_isp: debayer: Make the debayer_ object of type\n class Debayer not DebayerCpu\n libcamera: software_isp: debayer: Extend DebayerParams struct to hold\n a copy of per-frame CCM values\n libcamera: shaders: Move GL shader programs to\n src/libcamera/assets/shader\n utils: gen-shader-headers: Add a utility to generate headers from\n shaders\n meson: Automatically generate glsl_shaders.h from specified shader\n programs\n libcamera: software_isp: ccm: Populate CCM table to Debayer params\n structure\n libcamera: software_isp: lut: Make gain corrected CCM in lut.cpp\n available in debayer params\n libcamera: software_isp: gbm: Add in a GBM helper class for GPU\n surface access\n libcamera: software_isp: egl: Introduce an eGL base helper class\n libcamera: software_isp: debayer_egl: Add an eGL debayer class\n libcamera: software_isp: debayer_egl: Make DebayerEGL an environment\n option\n libcamera: shaders: Use highp not mediump for float precision\n libcamera: shaders: Extend debayer shaders to apply RGB gain values on\n output\n libcamera: software_isp: debayer_egl: Convert from identity CCM to CCM\n calculated by SoftIPA\n libcamera: software_isp: Switch on uncalibrated CCM to validate\n eGLDebayer\n libcamera: software_isp: Make isStandardBayerOrder static\n libcamera: software_isp: debayer_cpu: Make getInputConfig and\n getOutputConfig static\n libcamera: shaders: Extend bayer shaders to support swapping R and B\n on output\n libcamera: software_isp: Add a gpuisp todo list\n\nHans de Goede (5):\n libcamera: swstats_cpu: Update statsProcessFn() / processLine0()\n documentation\n libcamera: swstats_cpu: Drop patternSize_ documentation\n libcamera: swstats_cpu: Move header to libcamera/internal/software_isp\n libcamera: software_isp: Move benchmark code to its own class\n libcamera: swstats_cpu: Add processFrame() method\n\nMilan Zamazal (3):\n libcamera: shaders: Fix neighbouring positions in 8-bit debayering\n libcamera: software_isp: GPU support for unpacked 10/12-bit formats\n libcamera: shaders: Rename bayer_8 to bayer_unpacked\n\n include/libcamera/internal/egl.h | 110 +++\n include/libcamera/internal/gbm.h | 55 ++\n .../libcamera/internal/mapped_framebuffer.h | 4 +\n include/libcamera/internal/meson.build | 11 +\n .../libcamera/internal/shaders}/RGB.frag | 2 +-\n .../internal/shaders}/YUV_2_planes.frag | 2 +-\n .../internal/shaders}/YUV_3_planes.frag | 2 +-\n .../internal/shaders}/YUV_packed.frag | 2 +-\n .../internal/shaders}/bayer_1x_packed.frag | 62 +-\n .../internal/shaders/bayer_unpacked.frag | 78 ++-\n .../internal/shaders/bayer_unpacked.vert | 8 +-\n .../libcamera/internal/shaders}/identity.vert | 0\n .../libcamera/internal/shaders/meson.build | 10 +\n .../internal/software_isp/benchmark.h | 36 +\n .../internal/software_isp/debayer_params.h | 7 +\n .../internal/software_isp/meson.build | 2 +\n .../internal/software_isp/software_isp.h | 5 +-\n .../internal}/software_isp/swstats_cpu.h | 12 +\n src/apps/qcam/assets/shader/shaders.qrc | 16 +-\n src/apps/qcam/viewfinder_gl.cpp | 70 +-\n src/ipa/simple/algorithms/ccm.cpp | 4 +-\n src/ipa/simple/algorithms/lut.cpp | 1 +\n src/ipa/simple/data/uncalibrated.yaml | 12 +-\n src/libcamera/egl.cpp | 369 ++++++++++\n src/libcamera/gbm.cpp | 137 ++++\n src/libcamera/mapped_framebuffer.cpp | 7 +\n src/libcamera/meson.build | 34 +\n src/libcamera/software_isp/benchmark.cpp | 93 +++\n src/libcamera/software_isp/debayer.cpp | 61 ++\n src/libcamera/software_isp/debayer.h | 41 +-\n src/libcamera/software_isp/debayer_cpu.cpp | 81 +--\n src/libcamera/software_isp/debayer_cpu.h | 44 +-\n src/libcamera/software_isp/debayer_egl.cpp | 632 ++++++++++++++++++\n src/libcamera/software_isp/debayer_egl.h | 171 +++++\n src/libcamera/software_isp/gpuisp-todo.txt | 42 ++\n src/libcamera/software_isp/meson.build | 9 +\n src/libcamera/software_isp/software_isp.cpp | 37 +-\n src/libcamera/software_isp/swstats_cpu.cpp | 89 ++-\n utils/gen-shader-header.py | 38 ++\n utils/gen-shader-headers.sh | 44 ++\n utils/meson.build | 2 +\n 41 files changed, 2234 insertions(+), 208 deletions(-)\n create mode 100644 include/libcamera/internal/egl.h\n create mode 100644 include/libcamera/internal/gbm.h\n rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/RGB.frag (93%)\n rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/YUV_2_planes.frag (97%)\n rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/YUV_3_planes.frag (96%)\n rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/YUV_packed.frag (99%)\n rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/bayer_1x_packed.frag (76%)\n rename src/apps/qcam/assets/shader/bayer_8.frag => include/libcamera/internal/shaders/bayer_unpacked.frag (56%)\n rename src/apps/qcam/assets/shader/bayer_8.vert => include/libcamera/internal/shaders/bayer_unpacked.vert (85%)\n rename {src/apps/qcam/assets/shader => include/libcamera/internal/shaders}/identity.vert (100%)\n create mode 100644 include/libcamera/internal/shaders/meson.build\n create mode 100644 include/libcamera/internal/software_isp/benchmark.h\n rename {src/libcamera => include/libcamera/internal}/software_isp/swstats_cpu.h (86%)\n create mode 100644 src/libcamera/egl.cpp\n create mode 100644 src/libcamera/gbm.cpp\n create mode 100644 src/libcamera/software_isp/benchmark.cpp\n create mode 100644 src/libcamera/software_isp/debayer_egl.cpp\n create mode 100644 src/libcamera/software_isp/debayer_egl.h\n create mode 100644 src/libcamera/software_isp/gpuisp-todo.txt\n create mode 100755 utils/gen-shader-header.py\n create mode 100755 utils/gen-shader-headers.sh"}