From patchwork Wed Nov 18 09:57:16 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bryan O'Donoghue X-Patchwork-Id: 28971 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A6311BDE17 for ; Wed, 18 Nov 2026 09:58:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1A7016A8D9; Wed, 18 Nov 2026 10:58:07 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="xOP2L5kz"; dkim-atps=neutral Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 94BDB6A8C8 for ; Wed, 18 Nov 2026 10:57:58 +0100 (CET) Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-49465892f7bso535191f8f.1 for ; Wed, 18 Nov 2026 01:57:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1794995878; x=1795600678; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BTcx1UeBLlSMGaXkloThBotj5wVQaiibN+Q3tG4VB/8=; b=xOP2L5kzg9XlluICmJeRSRn+qfTlC8juyZ0Xwf/Fvk1zkgrH/hvcMfKUgqEpraEhiD fa5phwg268ZmBF7dQdTt0BzEr98OpJH79US+sWNyG1zNgePO4gG0/TvZsJkpF5vrG2wn ZtNzZr9Xz7w2Q85DzsQFhXbsg4jCuEvWuk1bEMWW+tZuHxhgulvOFvH5kRL4Spb2QH8h D0oID5oQF4p0sYpqRWqUeisEyLDv5B924Vd9nuCK4kYapqwa/Q86ABytFas2mMXQwpN4 Bg41L5p8wIVE7SZ/CJ8jjXj3OEaQWw7DjMz4BxA9DTQiTQlWenaQFl6845W5pctdj9e4 +Fwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794995878; x=1795600678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BTcx1UeBLlSMGaXkloThBotj5wVQaiibN+Q3tG4VB/8=; b=Q0b5peGSlRuaGCf7GrdBUIXJRsbYZ7zmet1dBYStnzBXso3PInCItOIfws4nCURq5g Gy9pycZI5hTvzSCrD5nNbafsuKdkVpigcDO2QuAA3aFj0+wV4JCz8ocPezt7m5/UowQ9 GOvnc7jFAlxYbM3sV87jxbwPQ3ISzBlCEGCX6aOtlHLRHBsTQ8MRZC7/YV1nIty88SHX qxeGvhI4TfKSZ27hPiwTOpgJ0Azzc3K3XCPqMKx4JGM421xrfj8STRuJil0Q3EUSOgHU ZqorFzR0fNb7Gcq2YJEXYqcYFBtePwB1k3LQT3yGhJa7016GzknJbOF2YMuWx7afPR7Q 4fVA== X-Gm-Message-State: APuU4Z+aNvlOdTasLEAycyPklpDJGAwX7DzVpEnCMIXpKK/OO2Btd/pJ L+tZJaJ/chLIjDTgJp+PFHDYs4wZft2ITvZ9ZXBZ3gSS4zQ+NsR5ecBBRva+E7gop2VyI2qLut+ Mx2WX/0I= X-Gm-Gg: AZFdTZdcrq9I5lnlOuCfGj/q0fq5SNPY6k+qmk2hfCfd/zFPAkrxCuUFTOCVI3fK8bo c3ySzlT2eirDINf1IWmeBXf01FkasLI7/6qkkh7K3nBRKuRY0cYhuBDMFQAfHOQbp+Bd6GTLi/N tny5+J1LyLOf543xYbtC/XLn3Gt3h8RVJbyf3/b5RCnOycyiuM9KslaSX9n+66DMxOJ29x7DOue fwSQaOZhqJLJzTZKT5yco5Tki3eQtOcTkOtnIGeP6YkyiLy6/6GComUjyrDHdDvP7QZftFKMuds i3Tbd+l8CUz+2yMeY7a4kh7e3fgZD/YBSClAejHlUiB536XGZuMJ4A6SH5S2dGPRpxI8o+a7LXG 5336cqxkqANKfExgRe1MiKP+R+9yPBh20SZshxl5bg5/4Dn/qXEUhZS3TxVMKxvH/jlRiec/wqZ WWwLLsF6jiwoEiI9jf0MkCxZCfmoF4fQ7fukZkz480p95Sj9aFCi07tzrem2EEVWexfIQn9B7NO i52jurRUIcQFSZf+yqz+GIQ6fIuNLwZfLfjIKJIWFF2O51Zby8wYql4DrddEMiY8bFwxb3ig7BQ yuZ8c3AOn4fdyqOFVAygn84SNA+vCwmazSSMKiAHweEqzw== X-Received: by 2002:a05:6000:41c9:b0:48d:ba3b:af36 with SMTP id ffacd0b85a97d-4a63b9d9524mr4088685f8f.37.1794995877792; Wed, 18 Nov 2026 01:57:57 -0800 (PST) Received: from inspiron14p-linux ([109.76.187.112]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4a655f9f4a7sm3940871f8f.34.2026.11.18.01.57.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Nov 2026 01:57:57 -0800 (PST) From: Bryan O'Donoghue To: libcamera-devel@lists.libcamera.org Cc: rstrode@redhat.com, robert.mader@collabora.com, barnabas.pocze@ideasonboard.com, Bryan O'Donoghue Subject: [RFC PATCH v2 09/26] libcamera: shaders: Split packed and unpacked demosiac up Date: Wed, 18 Nov 2026 10:57:16 +0100 Message-ID: <20261118095735.170587-10-bryan.odonoghue@linaro.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261118095735.170587-1-bryan.odonoghue@linaro.org> References: <20261118095735.170587-1-bryan.odonoghue@linaro.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Take the existing bayer_1x_packed.frag and split it up so that the texture2D() sampling happens in a first pass shader whose job it is to normalise the data-stream to GLR16F output. Thus far we have had two shaders for packed and unpacked MIPI streams which also differ in how they demosiac nearest-neighbour v bilinear. This is alot of maintenance we shouldn't really have to do and means that for reasons we don't seem to have chosen you get different demosiac quality if you have a packed or unpacked input stream. With a two-pass system though we can have the first shader pass produce a standard 16 bit float intermediary framebuffer which the subsequent pass can then demosiac. This lets us use the same code irrespective of the incoming MIPI stream which is obviously the right thing to do from a design/maintenance POV. Signed-off-by: Bryan O'Donoghue --- .../shaders/bayer_1x_packed_to_glr16f.frag | 93 +++++++++ .../shaders/bayer_glr16f_to_rgba.frag | 183 ++++++++++++++++++ .../shaders/bayer_unpacked_to_glr16f.frag | 46 +++++ src/libcamera/shaders/meson.build | 3 + 4 files changed, 325 insertions(+) create mode 100644 src/libcamera/shaders/bayer_1x_packed_to_glr16f.frag create mode 100644 src/libcamera/shaders/bayer_glr16f_to_rgba.frag create mode 100644 src/libcamera/shaders/bayer_unpacked_to_glr16f.frag diff --git a/src/libcamera/shaders/bayer_1x_packed_to_glr16f.frag b/src/libcamera/shaders/bayer_1x_packed_to_glr16f.frag new file mode 100644 index 000000000..57814dc49 --- /dev/null +++ b/src/libcamera/shaders/bayer_1x_packed_to_glr16f.frag @@ -0,0 +1,93 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Based on the code from http://jgt.akpeters.com/papers/McGuire08/ + * + * Copyright (c) 2008, Morgan McGuire. All rights reserved. + * + * Modified by Linaro Ltd for 10/12-bit packed raw Bayer format. + * Copyright (C) 2020, Linaro + * + * precursor_packed.frag - Precursor shader to decode MIPI packed raw Bayer + * data and output normalised single-channel float to an R16F FBO. + * Pairs with identity.vert. + */ + +#ifdef GL_ES +precision highp float; +#endif + +/* + * These constants are used to select the bytes containing the HS part of + * the pixel value: + * BPP - bytes per pixel, + * THRESHOLD_L = fract(BPP) * 0.5 + 0.02 + * THRESHOLD_H = 1.0 - fract(BPP) * 1.5 + 0.02 + * Let X is the x coordinate in the texture measured in bytes (so that the + * range is from 0 to (stride_-1)) aligned on the nearest pixel. + * E.g. for RAW10P: + * -------------+-------------------+-------------------+-- + * pixel No | 0 1 2 3 | 4 5 6 7 | ... + * -------------+-------------------+-------------------+-- + * byte offset | 0 1 2 3 4 | 5 6 7 8 9 | ... + * -------------+-------------------+-------------------+-- + * X | 0.0 1.25 2.5 3.75 | 5.0 6.25 7.5 8.75 | ... + * -------------+-------------------+-------------------+-- + * If fract(X) < THRESHOLD_L then the previous byte contains the LS + * bits of the pixel values and needs to be skipped. + * If fract(X) > THRESHOLD_H then the next byte contains the LS bits + * of the pixel values and needs to be skipped. + */ +#if defined(RAW10P) +#define BPP 1.25 +#define THRESHOLD_L 0.14 +#define THRESHOLD_H 0.64 +#elif defined(RAW12P) +#define BPP 1.5 +#define THRESHOLD_L 0.27 +#define THRESHOLD_H 0.27 +#else +#error Invalid raw format +#endif + +varying vec2 textureOut; + +/* the texture size in pixels */ +uniform vec2 tex_size; +uniform vec2 tex_step; + +uniform sampler2D tex_y; + +void main(void) +{ + /* + * center_bytes holds the coordinates of the MS byte of the pixel + * being sampled on the [0, stride-1/height-1] range. + * center_pixel holds the coordinates of the pixel being sampled + * on the [0, width/height-1] range. + */ + vec2 center_bytes; + vec2 center_pixel; + + /* + * The coordinates passed to the shader in textureOut may point + * to a place in between the pixels if the texture format doesn't + * match the image format. In particular, MIPI packed raw Bayer + * formats don't have a matching texture format. + * In this case align the coordinates to the left nearest pixel + * by hand. + */ + center_pixel = floor(textureOut * tex_size); + center_bytes.y = center_pixel.y; + + /* + * Add a small number (a few mantissa's LSBs) to avoid float + * representation issues. + */ + center_bytes.x = BPP * center_pixel.x + 0.02; + center_bytes.x = floor(center_bytes.x); + center_bytes *= tex_step; + + float C = texture2D(tex_y, center_bytes).r; + + gl_FragColor = vec4(C, 0.0, 0.0, 0.0); +} diff --git a/src/libcamera/shaders/bayer_glr16f_to_rgba.frag b/src/libcamera/shaders/bayer_glr16f_to_rgba.frag new file mode 100644 index 000000000..f3883a82b --- /dev/null +++ b/src/libcamera/shaders/bayer_glr16f_to_rgba.frag @@ -0,0 +1,183 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* +From http://jgt.akpeters.com/papers/McGuire08/ + +Efficient, High-Quality Bayer Demosaic Filtering on GPUs + +Morgan McGuire + +This paper appears in issue Volume 13, Number 4. +--------------------------------------------------------- +Copyright (c) 2008, Morgan McGuire. All rights reserved. + +Modified by Linaro Ltd to integrate it into libcamera. +Copyright (C) 2021-2026, Linaro +*/ + +//Pixel Shader +#ifdef GL_ES +precision highp float; +#endif + +/** Monochrome RGBA or GL_LUMINANCE Bayer encoded texture.*/ +uniform sampler2D tex_y; +varying vec4 center; +varying vec4 yCoord; +varying vec4 xCoord; +uniform mat3 ccm; +uniform float gamma; +uniform float contrastExp; + +float apply_contrast(float value) +{ + // Apply simple S-curve + if (value < 0.5) + return 0.5 * pow(value / 0.5, contrastExp); + else + return 1.0 - 0.5 * pow((1.0 - value) / 0.5, contrastExp); +} + +void main(void) { + vec3 rgb; + + /* Sample from R16F input texture */ + #define fetch(x, y) texture2D(tex_y, vec2(x, y)).r + + float C = fetch(center.x, center.y); // ( 0, 0) + const vec4 kC = vec4( 4.0, 6.0, 5.0, 5.0) / 8.0; + + // Determine which of four types of pixels we are on. + vec2 alternate = mod(floor(center.zw), 2.0); + + vec4 Dvec = vec4( + fetch(xCoord[1], yCoord[1]), // (-1,-1) + fetch(xCoord[1], yCoord[2]), // (-1, 1) + fetch(xCoord[2], yCoord[1]), // ( 1,-1) + fetch(xCoord[2], yCoord[2])); // ( 1, 1) + + vec4 PATTERN = (kC.xyz * C).xyzz; + + // Can also be a dot product with (1,1,1,1) on hardware where that is + // specially optimized. + // Equivalent to: D = Dvec[0] + Dvec[1] + Dvec[2] + Dvec[3]; + Dvec.xy += Dvec.zw; + Dvec.x += Dvec.y; + + vec4 value = vec4( + fetch(center.x, yCoord[0]), // ( 0,-2) + fetch(center.x, yCoord[1]), // ( 0,-1) + fetch(xCoord[0], center.y), // (-2, 0) + fetch(xCoord[1], center.y)); // (-1, 0) + + vec4 temp = vec4( + fetch(center.x, yCoord[3]), // ( 0, 2) + fetch(center.x, yCoord[2]), // ( 0, 1) + fetch(xCoord[3], center.y), // ( 2, 0) + fetch(xCoord[2], center.y)); // ( 1, 0) + + // Even the simplest compilers should be able to constant-fold these to + // avoid the division. + // Note that on scalar processors these constants force computation of some + // identical products twice. + const vec4 kA = vec4(-1.0, -1.5, 0.5, -1.0) / 8.0; + const vec4 kB = vec4( 2.0, 0.0, 0.0, 4.0) / 8.0; + const vec4 kD = vec4( 0.0, 2.0, -1.0, -1.0) / 8.0; + + // Conserve constant registers and take advantage of free swizzle on load + #define kE (kA.xywz) + #define kF (kB.xywz) + + value += temp; + + // There are five filter patterns (identity, cross, checker, + // theta, phi). Precompute the terms from all of them and then + // use swizzles to assign to color channels. + // + // Channel Matches + // x cross (e.g., EE G) + // y checker (e.g., EE B) + // z theta (e.g., EO R) + // w phi (e.g., EO R) + #define A (value[0]) + #define B (value[1]) + #define D (Dvec.x) + #define E (value[2]) + #define F (value[3]) + + // Avoid zero elements. On a scalar processor this saves two MADDs + // and it has no effect on a vector processor. + PATTERN.yzw += (kD.yz * D).xyy; + + PATTERN += (kA.xyz * A).xyzx + (kE.xyw * E).xyxz; + PATTERN.xw += kB.xw * B; + PATTERN.xz += kF.xz * F; + + rgb = (alternate.y == 0.0) ? + ((alternate.x == 0.0) ? + vec3(C, PATTERN.xy) : + vec3(PATTERN.z, C, PATTERN.w)) : + ((alternate.x == 0.0) ? + vec3(PATTERN.w, C, PATTERN.z) : + vec3(PATTERN.yx, C)); + + /* + * CCM is a 3x3 in the format + * + * +--------------+----------------+---------------+ + * | RedRedGain | RedGreenGain | RedBlueGain | + * +--------------+----------------+---------------+ + * | GreenRedGain | GreenGreenGain | GreenBlueGain | + * +--------------+----------------+---------------+ + * | BlueRedGain | BlueGreenGain | BlueBlueGain | + * +--------------+----------------+---------------+ + * + * Rout = RedRedGain * Rin + RedGreenGain * Gin + RedBlueGain * Bin + * Gout = GreenRedGain * Rin + GreenGreenGain * Gin + GreenBlueGain * Bin + * Bout = BlueRedGain * Rin + BlueGreenGain * Gin + BlueBlueGain * Bin + * + * We upload to the GPU without transposition glUniformMatrix3f(.., .., GL_FALSE, ccm); + * + * CPU + * float ccm [] = { + * RedRedGain, RedGreenGain, RedBlueGain, + * GreenRedGain, GreenGreenGain, GreenBlueGain, + * BlueRedGain, BlueGreenGain, BlueBlueGain, + * }; + * + * GPU + * ccm = { + * RedRedGain, GreenRedGain, BlueRedGain, + * RedGreenGain, GreenGreenGain, BlueGreenGain, + * RedBlueGain, GreenBlueGain, BlueBlueGain, + * } + * + * However the indexing for the mat data-type is column major hence + * ccm[0][0] = RedRedGain, ccm[0][1] = RedGreenGain, ccm[0][2] = RedBlueGain + * + */ + float rin, gin, bin; + rin = rgb.r; + gin = rgb.g; + bin = rgb.b; + + rgb.r = (rin * ccm[0][0]) + (gin * ccm[0][1]) + (bin * ccm[0][2]); + rgb.g = (rin * ccm[1][0]) + (gin * ccm[1][1]) + (bin * ccm[1][2]); + rgb.b = (rin * ccm[2][0]) + (gin * ccm[2][1]) + (bin * ccm[2][2]); + + /* + * Contrast + */ + rgb = clamp(rgb, 0.0, 1.0); + rgb.r = apply_contrast(rgb.r); + rgb.g = apply_contrast(rgb.g); + rgb.b = apply_contrast(rgb.b); + + /* Apply gamma after colour correction */ + rgb = pow(rgb, vec3(gamma)); + + #if defined (SWAP_BLUE) + gl_FragColor = vec4(rgb.bgr, 1.0); + #else + gl_FragColor = vec4(rgb, 1.0); + #endif +} diff --git a/src/libcamera/shaders/bayer_unpacked_to_glr16f.frag b/src/libcamera/shaders/bayer_unpacked_to_glr16f.frag new file mode 100644 index 000000000..00ed47ce8 --- /dev/null +++ b/src/libcamera/shaders/bayer_unpacked_to_glr16f.frag @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Based on the code from http://jgt.akpeters.com/papers/McGuire08/ + * + * Copyright (c) 2008, Morgan McGuire. All rights reserved. + * + * Modified by Linaro Ltd to integrate it into libcamera. + * Copyright (C) 2021, Linaro + * + * precursor_unpacked.frag - Precursor shader to decode unpacked raw Bayer + * data (R8, RG8 with split 10/12-bit values) and output normalised + * single-channel float to an R16F FBO. + * Pairs with identity.vert. + */ + +#ifdef GL_ES +precision highp float; +#endif + +varying vec2 textureOut; + +uniform sampler2D tex_y; + +void main(void) +{ + /* + * Reconstruct the pixel value from the texture format. + * + * For RAW10P the 10-bit value is split across two 8-bit channels: + * value = R / 4.0 + G * 64.0 + * For RAW12P the 12-bit value is split across two 8-bit channels: + * value = R / 16.0 + G * 16.0 + * Otherwise the value is a plain single-channel sample. + */ +#if defined(RAW10P) + vec4 p = texture2D(tex_y, textureOut); + float C = p.r / 4.0 + p.g * 64.0; +#elif defined(RAW12P) + vec4 p = texture2D(tex_y, textureOut); + float C = p.r / 16.0 + p.g * 16.0; +#else + float C = texture2D(tex_y, textureOut).r; +#endif + + gl_FragColor = vec4(C, 0.0, 0.0, 1.0); +} diff --git a/src/libcamera/shaders/meson.build b/src/libcamera/shaders/meson.build index c409ff9b0..8483e169e 100644 --- a/src/libcamera/shaders/meson.build +++ b/src/libcamera/shaders/meson.build @@ -3,6 +3,9 @@ # List of shader files to convert to header hex # for the purposes of inclusion in OpenGL debayering shader_files = files([ + 'bayer_1x_packed_to_glr16f.frag', + 'bayer_unpacked_to_glr16f.frag', + 'bayer_glr16f_to_rgba.frag', 'bayer_1x_packed.frag', 'bayer_unpacked.frag', 'bayer_unpacked.vert',