From patchwork Wed Aug 26 08:23:22 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28097 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 97890C333C for ; Wed, 26 Aug 2026 08:33:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7E3D268439; Wed, 26 Aug 2026 10:33:24 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="bQGPGZmo"; dkim-atps=neutral Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E3E4D6840C for ; Wed, 26 Aug 2026 10:23:32 +0200 (CEST) Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-c1c26d7e951so94770966b.0 for ; Wed, 26 Aug 2026 01:23:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732612; x=1788337412; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9M69w26wjssZbQ/ALWO958gZ0c+9M4q4+jNgBELWqso=; b=bQGPGZmoA7iEi1dXinz62knL+zfRDFlKeqT7ynOb+V9xlW8z133H5MKuYSeRa87z5N MYbG0aI96FBfge9g+ltGoqEu8fLEoRWAbS2d7GiOKTFaBgc7aQayaUyjI7LE9QPRqT4B ICs/gdmJtEgRxhl+2PUDAnSkgXRBK8ahLhw6Y+wuxTqGrXO9Nzzuuj6mkh5jCGAWqcGD DS3p4zodEwtgXsn0Ta9WcdM2WkdWbV6RvLNr5ft35ym9vpWPP8XFOrajIN2sNXAjUk/E nF0oeSn0e5LsApNWhVuRMLdPko7o5ko/jLKtKaSmgAaPv3g/h91FXh81PhLK0nSxOClr qSRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732612; x=1788337412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=9M69w26wjssZbQ/ALWO958gZ0c+9M4q4+jNgBELWqso=; b=fZneHDsd5e9iSGi12ZQ+RzIhdU407E10b5VajxX2dTOvu9wO3rYyrlpuYQvch74TNc ErUiWKZvHlpCxTETcThhz80WDThtKwVzd6Ofbgr27qnlcIvC8CERLmmDvxTlEUpJt+bY sgJ2dv+xJAYfMRxKYlY2myZm9A5w59uhQhV8HUOkr/hk6NELFA3R9WIGqG3jUv90waEZ cPoWmQf/xDyGB+1fC5r9Uft5UH7RyX8gYQL+KdTyGTbFVxrrCAJ8MLVNpU6PTEdcu5lD WSOFIHwLPWjlDkOAWpKSq/187ZtXzKjgjRsnqsDUhcdWCjvUNYswkqa1EMyjnco9ROQm fmqQ== X-Gm-Message-State: AFuF++lRkbTVWJguP6gklzNAds67HFCx04fbgJGDrElyNuElF1MF2MFK q21mosNaMY4/eSv2rVEEDKB8sKbbLDjtOwJ+H69XdI8f1ER6dqjHLEK1fAaToiVbqyA= X-Gm-Gg: AR+sD11dOEngXbJktWjx3OvkFDagB9IJ026a9SeEkhOLbhXPnOS5CGUICADvv3957a7 iW9qpS/PJiDOB7qMjXoAptE12EXoIpCGkKPxKlWXgr3Im/8Gi/qNbSQPVnKYVzANVdWFdbNKzN5 3kP5RjqNR3e5wELXdWefC7nYCegs3tyDbE6ZhSJYUo/zKdp+D7pC0cPF/uxFu/qy/fhZx/WVehT VBxZeLAicJdZsMfP4JM7z09Qhk7ZNgYP+/fs2gw3SiSPn917pa8oPRH+j2SJVBjF3j2dsizLNXz s7sllsg0yUj2iv27c5LzoAjGIi++XxU180KOwbl3JlQvKfkL+1Wsyy8R4cDhr40VKuTH1NmUXF0 evf+3RPmZ8QeRpxsH7sFsTDLJaKTHV+BfxmlLTXZLrt5n0XUEePMJJ6frz5w+ugXmrnclDYR4Wf ZB9bf4TqQ30qcVNcI9TfmZEcM7AZjD9CuHXlxRtYsLPsy33dcp7h59mbHI/ukI5snGbJW5FGMHt 5cAvgM+Cj38ytMcydP5JHt6SKIo41/1Cih7UMyM6rt8hsT6r7Kc7DQN0GrC1SYFb+aAqjXtaf+m loNqNbzOYPHi+tM= X-Received: by 2002:a17:906:fe4c:b0:c20:5213:56f1 with SMTP id a640c23a62f3a-c250c315049mr686467466b.13.1787732612303; Wed, 26 Aug 2026 01:23:32 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:31 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 1/7] libcamera: camera_sensor_properties: Add OV32C4 Date: Wed, 26 Aug 2026 10:23:22 +0200 Message-ID: <20260826082328.24176-2-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The OmniVision OV32C4 is a 32 megapixel RGBC sensor used as an under-display camera in laptops, behind an Intel IPU. Its kernel driver was submitted to linux-media. The unit cell size is the 0.702 um pixel pitch from the vendor product brief; it agrees with the 6528x4896 active array the same brief lists and with the crop rectangle the driver reports. No test pattern modes are listed because the driver does not implement V4L2_CID_TEST_PATTERN, and the sensor delays are left at the libcamera defaults because they have not been measured. Signed-off-by: Robert Bozik --- src/libcamera/sensor/camera_sensor_properties.cpp | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/libcamera/sensor/camera_sensor_properties.cpp b/src/libcamera/sensor/camera_sensor_properties.cpp index 819f07bb..9c3dd565 100644 --- a/src/libcamera/sensor/camera_sensor_properties.cpp +++ b/src/libcamera/sensor/camera_sensor_properties.cpp @@ -412,6 +412,11 @@ const CameraSensorProperties *CameraSensorProperties::get(const std::string &sen }, .sensorDelays = { }, } }, + { "ov32c4", { + .unitCellSize = { 702, 702 }, + .testPatternModes = {}, + .sensorDelays = { }, + } }, { "ov4689", { .unitCellSize = { 2000, 2000 }, .testPatternModes = { From patchwork Wed Aug 26 08:23:23 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28098 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1E877C333C for ; Wed, 26 Aug 2026 08:33:30 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A932E6843C; Wed, 26 Aug 2026 10:33:25 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="lxMrnVFd"; dkim-atps=neutral Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 641FB68406 for ; Wed, 26 Aug 2026 10:23:34 +0200 (CEST) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-c15d3cd51b2so89659266b.3 for ; Wed, 26 Aug 2026 01:23:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732614; x=1788337414; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YZkELr7iCSTgCCm9hYmaXDbC+1QVtiW84rLB7w3K9ZE=; b=lxMrnVFdOufVgYrAsHOdDNY4FrYmcup/S1ukanmVfngvl9roA/Vk9hA1zx+kt7z7XS acLIiC+lfpHrgdjyU/z3eT6tVY4YROeQuQzYmJH6EqbIT3TTYOYSQOhVS78KeuXPrjr+ nfto4RobXFFrVQalnQ4+0amAdsWd0TETwc2rVdDcGWMQW3QBccSQ7C21Evi8zC9Pr4Ok TRjmsWlWYogFuwyTjbHyTquGpu8GEpk9vck8Txs5tCfbW+QfjJc9FmAneLEqt7W6VuRE /mUbZaEAfqE0ekKaUPDVvTman59IlyugqGrRBbvjXxtYJ+p82RZFagsKtljlQmH+i7Np CyJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732614; x=1788337414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=YZkELr7iCSTgCCm9hYmaXDbC+1QVtiW84rLB7w3K9ZE=; b=XilmVYeH07WQ/cZVys5yBCWAt/luIFDw5VV1zduebbKv+US+LXG8HbFad8laaEfz6a LJue64fMyL+0ndQ408EoczcGur/DnpNYZdtq9La4jW44Y3a6ogd7SLWbbxaWpm8+S+aL ZftIM7rfkU8s+gRZP1upgmEQN35Qfs33w3lYNWT0PrA+Lxw3MDfw2oGvGC4qZ5UC6Mtc bo7d9xne3awGeFl+aXeWbQzyC9YFcH3ZH/uecNEYP2uoJPglasVByXhNsvFWZ3MZe1hA 05Tms+thmMmXtfwIKiW7n1pY9fmW/AIlTU/sF186JTei5noB90lE/QAUROaYOdYBMHyR MVOQ== X-Gm-Message-State: AFuF++lSks4B2cYMRVEIiZwEOxkOX0uZnPo68U+a+JVVzYfieWefL4ju 905/aRua1h9bC8QnDphuq7aIUM9RUKCMZj7L5n7+fuERbqHzEoNlio4w68VOWmFfFow= X-Gm-Gg: AR+sD10KpQl46o6SBEJKqU0/Y0+2gW66wmnf6JIX3Tphi50ErDgH2q5cY94TTsEoWtO SNAhTiMJ68LM09JrVZRX3w4drHXnrO8bnPW9m1esqtOBKIajopic4l45UoupvCmsk88sfDW/2bt C2dnCiCCra4juH1fyPAufs+XP579aROktggFsqKMkOvxEIHqu+1k+aHxX4AX3I80Wne4wtnahfh inPEi6Ta1qA3wnzHYGBc+12EDIUfMqDiXlMC5Hl11sCkfm5SIHVk4YeLClLpnvJoo8/TCq4/GKx 0m6bRzErlwtxj39Y7sobCxAkLQT6rrdqiHNafvDJStYd6zseorPYOXs0hTJ4lRERoVI1NjkhOPJ yo4D/YCOw15K3yHdRFmfQPcDjOmbt3X5WjRw/XCqUfMBoAxx7knEoEa8NWyl7OG/tGtzxFMD9Q2 x1l5D87GHCtkOSAPFLhe4TGMMUzw+vdpobmrUmZq8DZs+Jb/UK4niLcwOb5vBnbciOOnodUVG4M NAjZ6MXO2f4Lwhpj+iMtl/mIEpHjM5WfpH/AQop45NyoM67NBL84IZM6lnZeGLhP8Bs50KB08rj 06TVhZ+yKjiRXr0= X-Received: by 2002:a17:907:97d5:b0:c16:14d1:b529 with SMTP id a640c23a62f3a-c250c36ae0bmr570300266b.18.1787732613648; Wed, 26 Aug 2026 01:23:33 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:33 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 2/7] ipa: libipa: camera_sensor_helper: Add OV32C4 Date: Wed, 26 Aug 2026 10:23:23 +0200 Message-ID: <20260826082328.24176-3-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Both numbers were measured on the sensor rather than taken from the OmniVision helpers already present, because the obvious donor is wrong for this part. The analogue gain is linear in the register value, but unity is at 0x100, not at 0x80 as on ov13b10 and ov08x40 which use the same register: below 0x100 the output stops tracking the value written at all, and the chip itself powers up at 0x100. Using code/128 here would make the AE loop believe it has twice the gain it actually applied. The black level is 0x40 at 10 bits, measured with the sensor covered across the whole gain range, where it stays within one LSB. Signed-off-by: Robert Bozik --- src/ipa/libipa/camera_sensor_helper.cpp | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/ipa/libipa/camera_sensor_helper.cpp b/src/ipa/libipa/camera_sensor_helper.cpp index 82bf255d..6b746727 100644 --- a/src/ipa/libipa/camera_sensor_helper.cpp +++ b/src/ipa/libipa/camera_sensor_helper.cpp @@ -763,6 +763,23 @@ public: }; REGISTER_CAMERA_SENSOR_HELPER("ov2740", CameraSensorHelperOv2740) +class CameraSensorHelperOv32c4 : public CameraSensorHelper +{ +public: + CameraSensorHelperOv32c4() + { + /* + * Measured on the sensor: the black level is 0x40 at 10 bits, + * and the gain is linear in the register value with unity at + * 0x100, not at 0x80 as on other OmniVision sensors sharing + * the same gain register. + */ + blackLevel_ = 4096; + gain_ = AnalogueGainLinear{ 1, 0, 0, 256 }; + } +}; +REGISTER_CAMERA_SENSOR_HELPER("ov32c4", CameraSensorHelperOv32c4) + class CameraSensorHelperOv4689 : public CameraSensorHelper { public: From patchwork Wed Aug 26 08:23:24 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28099 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 55EE7C333D for ; Wed, 26 Aug 2026 08:33:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2B51468429; Wed, 26 Aug 2026 10:33:27 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Ezl875gr"; dkim-atps=neutral Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 113BC6841E for ; Wed, 26 Aug 2026 10:23:36 +0200 (CEST) Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-6a5e329c0f2so665533a12.3 for ; Wed, 26 Aug 2026 01:23:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732615; x=1788337415; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=DHG3k6WyhqdbeyVoS6R8003gMHNOFslse6PuucBXQko=; b=Ezl875grKyNOpqbU2sB0k07vdqlVboZE9SbcA7R5wQcLIMMgsyLskWAWTz4Olr3FcX YOmz2YwyCUMebibhUCCWkJBMER5aTMiQYDFdq2sDH90MppZ5C5uUEdEZU8dtNIC1DMms LqA9s3rLpQ3PjD0zdC/3GdQthLPPwjiEUVUNobbXdKEd+87tRHXjvimV1/VkJ8UguMvt vkudtT9hR3G9JU26HbiW2f0weXtBtrfEY4hq2tFiLRqlBVC95WeH89rzSpadavgRzfNL PPrh0dmEmNw3H4Ci/1UeRV75Fc7Ndv74BqyDsV14YnNGFH8HwWT5czPZTbWZFAbFxoam pGAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732615; x=1788337415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=DHG3k6WyhqdbeyVoS6R8003gMHNOFslse6PuucBXQko=; b=dgnxeSZdwoT1hp7Zv/GFOtWGRUXQ7Zo1JCVtHgQVmdtk6rl8we6CJ9Y0l0QEBPfMHN Dggvl5XIISAha1U0BvjSzaM6vlkUP92h6UoAe6ah+7JeiQSo7qd/4LcGV+/lKyyzb3oO inijO9rvhIhaV36+Bivlip5X0RVnuxQrl6gf9TCasn2Aa0ypcNYFkiC+N0RDEPdawGWT yAewGswAdO5DQ4RKIO4d05CPLeOex8/7mYSQQp8+FlTywqhs0hO6eObN2kvsrjSYqiRu RYg58Vyynbp5crm0jGFzW8EJNHkHf9SMIKAOcPfPC0+xW0AfvutaEndfRG060dt53q/H mP1g== X-Gm-Message-State: AFuF++mgGmsgQ5+/gPyZT+kJx1CH/gvKLRxwqNtmaTSHev0xREdL0/kM IO8AKbrRBQ0iHa1KguFuRQ5soRfalgjJhD5gPbz+F3jN4JysAeXZufyYpNobhdapURE= X-Gm-Gg: AR+sD11473kDWkDjvXeeSWvQ/5X7i5DxQTz4j4QKkchlOpIKxVU2fJ8stVMqsotcaHH 6L9xKgReZb6Rqk3AnN0LexdBWItK2XTgv/wLuE0hyTEjtsQSjYiRYJHcOU/+DL5Wsj9Srh/ZEOh 4JmS527mCyrHGfVaNACx8ab6zLDspzaFxXdpLlPS5hIoEinNzphEzaKlIl6NjMLQ9NFtBZUIFOG ibOXbXf7QVZ41X+rRMXsHUMOrPIcRDGgY1sM55zZhhKFCjortZWVwJfLaifESoq/Rq4SiqwYWaI 2ULi/YFpI2RceehA8L7hEakuu9Ct3CBOYtb+IGQbJoqTsP3Y2/HDWGicFJKdhNiJz9Dz7UOR9iT IFRsQaW6w0ELktCl63U9md/fXQff1Ry0VXr+svAZs0uEYuHn/auFZnlWvSt12vUX3CVnvkycnU+ +s6s9ERnZB//Jigh7PRCpE6Kw6zTVMUHqkB1pzOG2jDdKrSqLbigRK0TKSHnX4Nf2lEEdnB4GB2 jBu05WHYk2MEW7kdeXj4izbCFGDy/sULOs9POXFJDedQ2UGY6LNquZSMHZ6b3dS2oWToqYhNub/ cIyPFoZWJPGTp0MOss7T21YD0A== X-Received: by 2002:a17:907:9289:b0:c15:e118:9b99 with SMTP id a640c23a62f3a-c250c36b683mr668068066b.23.1787732615268; Wed, 26 Aug 2026 01:23:35 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:34 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 3/7] ipa: softisp: Support FrameDurationLimits through vertical blanking Date: Wed, 26 Aug 2026 10:23:24 +0200 Message-ID: <20260826082328.24176-4-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The software ISP AGC can only raise the exposure up to the limit the sensor reports for its current frame length. On a sensor that is at that limit and at maximum analogue gain in dim light, the only remaining way to get more signal is a longer frame, but neither the simple pipeline handler nor the IPA touched V4L2_CID_VBLANK, and FrameDurationLimits was not exposed at all. Expose FrameDurationLimits from the sensor timing information, default to the full range the sensor mode supports, and let the AGC lengthen the frame when the exposure needs it: the exposure may grow up to what the longest allowed frame permits, and the vertical blanking follows the exposure, so the frame rate only drops when a longer exposure is needed. Decreasing brightness reduces the gain first, then the exposure, and the frame shortens again with it. Applications restrict the range through FrameDurationLimits as usual. The maximum exposure for a given vertical blanking is derived from the limits the driver reports at configure time (frame length minus exposure maximum), which relies on the driver updating the exposure control range when the vertical blanking changes, as the OmniVision drivers do. Sensors without a V4L2_CID_VBLANK control keep the previous fixed-frame behaviour. The simple pipeline handler registers V4L2_CID_VBLANK with the delayed controls as a priority write, and applies it on its own first when controls are written directly, so that the exposure of the same frame is validated against the updated limits. The frame duration of each frame is reported in the FrameDuration metadata. Signed-off-by: Robert Bozik --- src/ipa/softisp/algorithms/agc.cpp | 181 ++++++++++++++++++++++- src/ipa/softisp/algorithms/agc.h | 8 + src/ipa/softisp/ipa_context.h | 21 +++ src/ipa/softisp/softisp.cpp | 46 +++++- src/libcamera/pipeline/simple/simple.cpp | 20 +++ 5 files changed, 268 insertions(+), 8 deletions(-) diff --git a/src/ipa/softisp/algorithms/agc.cpp b/src/ipa/softisp/algorithms/agc.cpp index 63b41544..ca403300 100644 --- a/src/ipa/softisp/algorithms/agc.cpp +++ b/src/ipa/softisp/algorithms/agc.cpp @@ -12,6 +12,9 @@ #include #include +#include + +#include #include "control_ids.h" @@ -19,6 +22,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPASoftIspExposure) +using namespace std::literals::chrono_literals; + namespace ipa::softisp::algorithms { /* @@ -65,15 +70,161 @@ Agc::Agc() { } +int Agc::init(IPAContext &context, [[maybe_unused]] const ValueNode &tuningData) +{ + /* + * Expose the frame duration limits the sensor can achieve in the + * current mode. Whether the IPA can actually change the frame duration + * is only known in configure(), when the sensor controls are available. + */ + const IPACameraSensorInfo &sensorInfo = context.sensorInfo; + if (!sensorInfo.pixelRate || !sensorInfo.minLineLength) { + LOG(IPASoftIspExposure, Warning) + << "Missing sensor timing information, " + << "FrameDurationLimits not exposed"; + return 0; + } + + utils::Duration lineDuration = sensorInfo.minLineLength * 1.0s / sensorInfo.pixelRate; + utils::Duration minDuration = lineDuration * sensorInfo.minFrameLength; + utils::Duration maxDuration = lineDuration * sensorInfo.maxFrameLength; + int64_t minFrameDuration = minDuration.get(); + int64_t maxFrameDuration = maxDuration.get(); + + context.ctrlMap[&controls::FrameDurationLimits] = + ControlInfo(minFrameDuration, maxFrameDuration, minFrameDuration); + + return 0; +} + +int Agc::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) +{ + auto &agc = context.activeState.agc; + const auto &cfg = context.configuration.agc; + + /* + * Default to the full range the sensor supports, applications restrict + * it through FrameDurationLimits. Without vblank control the frame + * duration is fixed at the sensor default. + */ + const auto it = context.ctrlMap.find(&controls::FrameDurationLimits); + if (it != context.ctrlMap.end() && cfg.vblankSupported) { + agc.minFrameDuration = std::chrono::microseconds(it->second.min().get()); + agc.maxFrameDuration = std::chrono::microseconds(it->second.max().get()); + } else { + agc.minFrameDuration = cfg.lineDuration * (cfg.frameHeight + cfg.vblankDef); + agc.maxFrameDuration = agc.minFrameDuration; + } + agc.vblank = cfg.vblankDef; + + return 0; +} + +void Agc::queueRequest(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, const ControlList &controls) +{ + auto &agc = context.activeState.agc; + + const auto &frameDurationLimits = controls.get(controls::FrameDurationLimits); + if (frameDurationLimits && context.configuration.agc.vblankSupported) { + const auto it = context.ctrlMap.find(&controls::FrameDurationLimits); + if (it != context.ctrlMap.end()) { + const ControlInfo &limits = it->second; + int64_t minFrameDuration = + std::clamp((*frameDurationLimits).front(), + limits.min().get(), + limits.max().get()); + int64_t maxFrameDuration = + std::clamp((*frameDurationLimits).back(), + limits.min().get(), + limits.max().get()); + if (maxFrameDuration < minFrameDuration) + maxFrameDuration = minFrameDuration; + + agc.minFrameDuration = std::chrono::microseconds(minFrameDuration); + agc.maxFrameDuration = std::chrono::microseconds(maxFrameDuration); + } + } + + frameContext.agc.minFrameDuration = agc.minFrameDuration; + frameContext.agc.maxFrameDuration = agc.maxFrameDuration; +} + +/* + * Translate the frame duration limits of the frame into a vblank range, + * clamped to what the sensor supports. + */ +void Agc::vblankRange(const IPAContext &context, const IPAFrameContext &frameContext, + int32_t &vblankLo, int32_t &vblankHi) const +{ + const auto &cfg = context.configuration.agc; + + if (!cfg.vblankSupported) { + vblankLo = vblankHi = cfg.vblankDef; + return; + } + + /* + * The limits are expressed in microseconds, which can't represent the + * line timing exactly. Round to the nearest line, so that a limit + * derived from a whole number of lines maps back to that number. + */ + const double minLines = std::round(frameContext.agc.minFrameDuration / cfg.lineDuration); + const double maxLines = std::round(frameContext.agc.maxFrameDuration / cfg.lineDuration); + const int64_t height = cfg.frameHeight; + + vblankLo = static_cast(std::clamp( + static_cast(minLines) - height, cfg.vblankMin, cfg.vblankMax)); + vblankHi = static_cast(std::clamp( + static_cast(maxLines) - height, cfg.vblankMin, cfg.vblankMax)); + if (vblankHi < vblankLo) + vblankHi = vblankLo; +} + +/* + * Maximum exposure the sensor accepts for a given vblank. The driver keeps + * exposureMargin lines between the exposure and the frame length. + */ +int32_t Agc::exposureMaxForVblank(const IPAContext &context, int32_t vblank) const +{ + const auto &cfg = context.configuration.agc; + + if (!cfg.vblankSupported) + return cfg.exposureMax; + + return std::max(cfg.exposureMin, + static_cast(cfg.frameHeight) + vblank - cfg.exposureMargin); +} + void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, double exposureMSV) { int32_t &exposure = frameContext.sensor.exposure; double &again = frameContext.sensor.gain; + int32_t &vblank = frameContext.sensor.vblank; + + int32_t vblankLo, vblankHi; + vblankRange(context, frameContext, vblankLo, vblankHi); + + /* + * The exposure may grow up to what the longest allowed frame permits; + * the vblank then follows the exposure, so the frame is only made + * longer when the exposure needs it. + */ + const int32_t exposureMax = exposureMaxForVblank(context, vblankHi); double error = kExposureOptimal - exposureMSV; - if (std::abs(error) <= kExposureSatisfactory) + if (std::abs(error) <= kExposureSatisfactory) { + /* Still honour changed frame duration limits. */ + exposure = std::clamp(exposure, context.configuration.agc.exposureMin, + exposureMax); + vblank = std::clamp(exposure + context.configuration.agc.exposureMargin - + static_cast(context.configuration.agc.frameHeight), + vblankLo, vblankHi); + context.activeState.agc.exposure = exposure; + context.activeState.agc.vblank = vblank; return; + } /* * Compute a proportional correction factor. The sign of the error @@ -85,8 +236,11 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou float factor = 1.0f + step; if (factor > 1.0f) { - /* Scene too dark: increase exposure first, then gain. */ - if (exposure < context.configuration.agc.exposureMax) { + /* + * Scene too dark: increase exposure first (lengthening the + * frame when the limits allow it), then gain. + */ + if (exposure < exposureMax) { int32_t next = static_cast(exposure * factor); exposure = std::max(next, exposure + 1); } else { @@ -111,17 +265,22 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou } exposure = std::clamp(exposure, context.configuration.agc.exposureMin, - context.configuration.agc.exposureMax); + exposureMax); again = std::clamp(again, context.configuration.agc.againMin, context.configuration.agc.againMax); + vblank = std::clamp(exposure + context.configuration.agc.exposureMargin - + static_cast(context.configuration.agc.frameHeight), + vblankLo, vblankHi); context.activeState.agc.exposure = exposure; context.activeState.agc.again = again; + context.activeState.agc.vblank = vblank; LOG(IPASoftIspExposure, Debug) << "exposureMSV " << exposureMSV << " error " << error << " factor " << factor - << " exp " << exposure << " again " << again; + << " exp " << exposure << " again " << again + << " vblank " << vblank << " (" << vblankLo << "-" << vblankHi << ")"; } void Agc::process(IPAContext &context, @@ -130,10 +289,16 @@ void Agc::process(IPAContext &context, const SwIspStats *stats, ControlList &metadata) { - utils::Duration exposureTime = - context.configuration.agc.lineDuration * frameContext.sensor.exposure; + const auto &cfg = context.configuration.agc; + utils::Duration exposureTime = cfg.lineDuration * frameContext.sensor.exposure; metadata.set(controls::ExposureTime, exposureTime.get()); metadata.set(controls::AnalogueGain, frameContext.sensor.gain); + if (cfg.vblankSupported) { + frameContext.agc.frameDuration = + cfg.lineDuration * (cfg.frameHeight + frameContext.sensor.vblank); + metadata.set(controls::FrameDuration, + frameContext.agc.frameDuration.get()); + } if (!context.activeState.agc.valid) { /* @@ -142,6 +307,7 @@ void Agc::process(IPAContext &context, */ context.activeState.agc.exposure = frameContext.sensor.exposure; context.activeState.agc.again = frameContext.sensor.gain; + context.activeState.agc.vblank = frameContext.sensor.vblank; context.activeState.agc.valid = true; } @@ -152,6 +318,7 @@ void Agc::process(IPAContext &context, */ frameContext.sensor.exposure = context.activeState.agc.exposure; frameContext.sensor.gain = context.activeState.agc.again; + frameContext.sensor.vblank = context.activeState.agc.vblank; return; } diff --git a/src/ipa/softisp/algorithms/agc.h b/src/ipa/softisp/algorithms/agc.h index 36944619..1aa77737 100644 --- a/src/ipa/softisp/algorithms/agc.h +++ b/src/ipa/softisp/algorithms/agc.h @@ -19,6 +19,11 @@ public: Agc(); ~Agc() = default; + int init(IPAContext &context, const ValueNode &tuningData) override; + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; + void queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const SwIspStats *stats, @@ -26,6 +31,9 @@ public: private: void updateExposure(IPAContext &context, IPAFrameContext &frameContext, double exposureMSV); + void vblankRange(const IPAContext &context, const IPAFrameContext &frameContext, + int32_t &vblankLo, int32_t &vblankHi) const; + int32_t exposureMaxForVblank(const IPAContext &context, int32_t vblank) const; }; } /* namespace ipa::softisp::algorithms */ diff --git a/src/ipa/softisp/ipa_context.h b/src/ipa/softisp/ipa_context.h index 2eed32f8..b5577768 100644 --- a/src/ipa/softisp/ipa_context.h +++ b/src/ipa/softisp/ipa_context.h @@ -31,6 +31,17 @@ struct IPASessionConfiguration { int32_t exposureMin, exposureMax; double againMin, againMax, again10, againMinStep; utils::Duration lineDuration; + /* + * Frame duration control through V4L2_CID_VBLANK. When the + * sensor doesn't expose the control, vblankSupported is false + * and the frame duration stays at whatever the sensor was + * configured with. + */ + bool vblankSupported; + int32_t vblankMin, vblankMax, vblankDef; + /* Lines the sensor keeps between max exposure and frame length */ + int32_t exposureMargin; + uint32_t frameHeight; } agc; struct { std::optional level; @@ -44,7 +55,10 @@ struct IPAActiveState { struct { int32_t exposure; double again; + int32_t vblank; bool valid; + utils::Duration minFrameDuration; + utils::Duration maxFrameDuration; } agc; struct { @@ -70,8 +84,15 @@ struct IPAFrameContext : public FrameContext { struct { int32_t exposure; double gain; + int32_t vblank; } sensor; + struct { + utils::Duration minFrameDuration; + utils::Duration maxFrameDuration; + utils::Duration frameDuration; + } agc; + float gamma; std::optional contrast; std::optional saturation; diff --git a/src/ipa/softisp/softisp.cpp b/src/ipa/softisp/softisp.cpp index aec04c52..612f9dfa 100644 --- a/src/ipa/softisp/softisp.cpp +++ b/src/ipa/softisp/softisp.cpp @@ -7,6 +7,7 @@ #include #include +#include #include #include @@ -224,6 +225,35 @@ int IPASoftIsp::configure(const IPAConfigInfo &configInfo) int32_t againMax = gainInfo.max().get(); int32_t againDef = gainInfo.def().get(); + /* + * Frame duration control. The sensor driver is expected to update the + * exposure limits when the vertical blanking changes (as e.g. the + * OmniVision drivers do), so the maximum exposure for a given vblank is + * derived here from the current limits: the difference between the + * frame length and the exposure maximum is the margin the driver keeps. + */ + auto &agc = context_.configuration.agc; + agc.frameHeight = context_.sensorInfo.outputSize.height; + agc.vblankSupported = false; + const auto vblankIt = sensorInfoMap_.find(V4L2_CID_VBLANK); + if (vblankIt != sensorInfoMap_.end()) { + const ControlInfo &vblankInfo = vblankIt->second; + agc.vblankMin = vblankInfo.min().get(); + agc.vblankMax = vblankInfo.max().get(); + agc.vblankDef = vblankInfo.def().get(); + agc.exposureMargin = static_cast(agc.frameHeight) + + agc.vblankDef - agc.exposureMax; + if (agc.exposureMargin >= 0 && agc.vblankMax > agc.vblankMin) { + agc.vblankSupported = true; + } else { + LOG(IPASoftIsp, Warning) + << "Unusable vblank limits " << agc.vblankMin + << "-" << agc.vblankMax << " (def " << agc.vblankDef + << "), exposure max " << agc.exposureMax + << ", frame duration control disabled"; + } + } + if (camHelper_) { context_.configuration.agc.againMin = camHelper_->gain(againMin); context_.configuration.agc.againMax = camHelper_->gain(againMax); @@ -260,7 +290,13 @@ int IPASoftIsp::configure(const IPAConfigInfo &configInfo) << context_.configuration.agc.exposureMax << ", gain " << context_.configuration.agc.againMin << "-" << context_.configuration.agc.againMax - << " (" << context_.configuration.agc.againMinStep << ")"; + << " (" << context_.configuration.agc.againMinStep << ")" + << (agc.vblankSupported + ? ", vblank " + std::to_string(agc.vblankMin) + "-" + + std::to_string(agc.vblankMax) + " (def " + + std::to_string(agc.vblankDef) + ", exposure margin " + + std::to_string(agc.exposureMargin) + " lines)" + : ", no vblank control"); return 0; } @@ -305,6 +341,12 @@ void IPASoftIsp::processStats(const uint32_t frame, sensorControls.get(V4L2_CID_EXPOSURE).get(); int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); frameContext.sensor.gain = camHelper_ ? camHelper_->gain(again) : again; + if (context_.configuration.agc.vblankSupported && + sensorControls.contains(V4L2_CID_VBLANK)) + frameContext.sensor.vblank = + sensorControls.get(V4L2_CID_VBLANK).get(); + else + frameContext.sensor.vblank = context_.configuration.agc.vblankDef; ControlList metadata(controls::controls); for (const auto &algo : algorithms()) @@ -324,6 +366,8 @@ void IPASoftIsp::processStats(const uint32_t frame, ctrls.set(V4L2_CID_EXPOSURE, frameContext.sensor.exposure); ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(camHelper_ ? camHelper_->gainCode(againNew) : againNew)); + if (context_.configuration.agc.vblankSupported) + ctrls.set(V4L2_CID_VBLANK, frameContext.sensor.vblank); setSensorControls.emit(ctrls); } diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index e20f08b6..0a0cfc55 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -570,6 +570,14 @@ SimpleCameraData::SimpleCameraData(SimplePipelineHandler *pipe, { V4L2_CID_ANALOGUE_GAIN, { delays.gainDelay, false } }, { V4L2_CID_EXPOSURE, { delays.exposureDelay, false } }, }; + /* + * The software ISP IPA controls the frame duration through vertical + * blanking when the sensor supports it. Write it with priority, so + * that the driver has updated the exposure limits before the exposure + * of the same frame is applied. + */ + if (sensor_->controls().count(V4L2_CID_VBLANK)) + params[V4L2_CID_VBLANK] = { delays.vblankDelay, true }; delayedCtrls_ = std::make_unique(sensor_->device(), params); LOG(SimplePipeline, Debug) @@ -1054,6 +1062,18 @@ void SimpleCameraData::setSensorControls(const ControlList &sensorControls) */ if (!frameStartEmitter_) { ControlList ctrls(sensorControls); + + /* + * Apply the vertical blanking on its own first, as the + * exposure limits depend on it and a single request with + * an exposure outside the current limits would be rejected. + */ + if (ctrls.contains(V4L2_CID_VBLANK)) { + ControlList vblank(sensor_->controls()); + vblank.set(V4L2_CID_VBLANK, ctrls.get(V4L2_CID_VBLANK)); + sensor_->setControls(&vblank); + } + sensor_->setControls(&ctrls); } } From patchwork Wed Aug 26 08:23:25 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28100 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3DE15C333C for ; Wed, 26 Aug 2026 08:33:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9B0E36843F; Wed, 26 Aug 2026 10:33:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="N+oQk21v"; dkim-atps=neutral Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 692E068406 for ; Wed, 26 Aug 2026 10:23:37 +0200 (CEST) Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-c1c52d920b8so77668566b.2 for ; Wed, 26 Aug 2026 01:23:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732617; x=1788337417; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=b0+DHPFnbUJqxMG7LCCRZqbuEFKsg5PZWPtq/mGAJ7U=; b=N+oQk21vEyN7GU1TFol+xxaOYYHNshuEETUOiBaQtZp1uKuPLpGphL71Ro0hK0o7XP D2WeCmMAp/bVLy20aHCwdKNjPB27NcDgsgRB76E/OQR2XbR5Prb1qniQUr3X1DkCgN18 0fzk5KwGm+hXpCICuwsIjR+DcUK/aen2G+s4oWJFu14qVPQVAcfc4cVhjM816jQmInpr kUYfpoS3zJgz+4gJMu6oBmC7eZ1qT/K7lpKnIqT8JnY277qHqYTucWDKNGMSgFFdN7NC n+NaYAmwCTIYNJ4SCD60zcgRfQBRW2JUet1eMnNB6bXoobrua/ms9LLPo/xhCqht5pdL bASA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732617; x=1788337417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=b0+DHPFnbUJqxMG7LCCRZqbuEFKsg5PZWPtq/mGAJ7U=; b=XbJuWVJUmo2vdKa54KquzCsNrp0RZWv+yUtP77JbN9dso7ASRKPb/kYy1VUaqOX7oF U4xkqpF7YvgZs5E/rmxv7z3dK98tM93HuEKrPA5GlA3dUKgIVgXUwopAHVNzmMrehD4w smfZsQTRj9k/UV/1QNORGPEnenlm+nNhz8sGGq5dx0YzUKH88KnoaqFtMbXzhKIosO8Y TKRT4rKSqan7N+FTrmRo4zdo5uEBV5NpsAXReDQHGrQQVUzG6YZFG4j8yLxyTaWzQU4d SK7sOO1sT72fYtAoyV6IDCZFUN1BJvai3YbjiV9unU6uxRSwHX6bpLUdrwVAqKbx+zTj 2wbQ== X-Gm-Message-State: AFuF++nw5iJjCWDtSC77A2rpICIgaD+GzQBmT7KGYcOuKBO2TlTUyPhg kFeEaoCQ12rwWD9fSQbLrgB35eRorgeLbSt48G8YK2mTcKvbguW5eaAjEeEtNpYMzog= X-Gm-Gg: AR+sD11nIAYdKVUrg+7yBeY7Ak/URoRuTQdA8Y3mMTIH/9ZQfjs32fR8+3VlHmxKbbc +1it8ZMMXxIhVLlHpHu4vtnBXSl+Ix85TMuQIZ88WJZqjCv9Nm8t3TIVHwPUtKJxVCfY/hei6V5 tstKljUeRTAzXFbWUavvud9dUCnU9fJbeSJ38tMoavXfaCt+ReoCmZBkTNANrsxC8iqrt3W1Cf9 RnoqPWCz+FQUaW+Uyhkmo2jNsTxylc2Dpwsfvg+qhEB1WlmT5fqwbbbVLavfKr1Ki7VKlg1II1F oQNz35mI4Z7Bj1ZgqHjy0xRpL56bOuAABgIIuIFRf9eT9bvzXGgf2O8hFd7pkuVYysn6dJethFN OHYg/Gyx/w4tHTA88UV8gwptuKTJ3DQQxslARpqRu/89AeYxb3hvg3OR5JpQbClO55b1D05xMJm JwYSK/AORCWpzgwLgijwFbk7ydxdBnhlM1OScj9mInc0kmcaKL5nOU+oqPb+yLqlc7TAD2Nr69S ZGfT6+VluQUEO7Zg9eOwkmGMQXuNPs5qNKiDviaOuTf9++uX672HCDDWlxAQHwVvDJA+o9SB6a2 kgc8wJsJX/2BqO0= X-Received: by 2002:a17:907:980e:b0:c25:2f12:1441 with SMTP id a640c23a62f3a-c252f121647mr11850166b.12.1787732616857; Wed, 26 Aug 2026 01:23:36 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:36 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 4/7] pipeline: simple: Pass the startup controls to the software ISP IPA Date: Wed, 26 Aug 2026 10:23:25 +0200 Message-ID: <20260826082328.24176-5-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The controls passed to Camera::start() were dropped by the simple pipeline handler, so applications that configure the camera at start, such as libcamerasrc with its framerate caps and the PipeWire libcamera source, had no effect on the software ISP: FrameDurationLimits, AWB mode or colour gains set that way were silently ignored. Extend the software ISP and its IPA start() with the control list, and have the IPA feed the controls through the algorithms' queueRequest() handlers with a throwaway frame context, so that they end up in the active state exactly as per-request controls do. Signed-off-by: Robert Bozik --- .../libcamera/internal/software_isp/software_isp.h | 2 +- include/libcamera/ipa/softisp.mojom | 2 +- src/ipa/softisp/softisp.cpp | 13 +++++++++++-- src/libcamera/pipeline/simple/simple.cpp | 4 ++-- src/libcamera/software_isp/software_isp.cpp | 5 +++-- 5 files changed, 18 insertions(+), 8 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index da893d24..9f3227d5 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -73,7 +73,7 @@ public: void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls); - int start(); + int start(const ControlList &controls); void stop(); void queueRequest(const uint32_t frame, const ControlList &controls); diff --git a/include/libcamera/ipa/softisp.mojom b/include/libcamera/ipa/softisp.mojom index a892d84f..d9c9842f 100644 --- a/include/libcamera/ipa/softisp.mojom +++ b/include/libcamera/ipa/softisp.mojom @@ -19,7 +19,7 @@ interface IPASoftIspInterface { libcamera.IPACameraSensorInfo sensorInfo, libcamera.ControlInfoMap sensorControls) => (int32 ret, libcamera.ControlInfoMap ipaControls, bool ccmEnabled); - start() => (int32 ret); + start(libcamera.ControlList controls) => (int32 ret); stop(); configure(IPAConfigInfo configInfo) => (int32 ret); diff --git a/src/ipa/softisp/softisp.cpp b/src/ipa/softisp/softisp.cpp index 612f9dfa..6d6b80d5 100644 --- a/src/ipa/softisp/softisp.cpp +++ b/src/ipa/softisp/softisp.cpp @@ -61,7 +61,7 @@ public: bool *ccmEnabled) override; int configure(const IPAConfigInfo &configInfo) override; - int start() override; + int start(const ControlList &controls) override; void stop() override; void queueRequest(const uint32_t frame, const ControlList &controls) override; @@ -301,8 +301,17 @@ int IPASoftIsp::configure(const IPAConfigInfo &configInfo) return 0; } -int IPASoftIsp::start() +int IPASoftIsp::start(const ControlList &controls) { + /* + * Apply the startup controls through the algorithms, as if they had + * been queued with a request. The frame context is a throwaway, the + * algorithms record what matters in the active state. + */ + IPAFrameContext frameContext{}; + for (const auto &algo : algorithms()) + algo->queueRequest(context_, 0, frameContext, controls); + return 0; } diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 0a0cfc55..4793a522 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1647,7 +1647,7 @@ int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, return data->video_->exportBuffers(count, buffers); } -int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlList *controls) +int SimplePipelineHandler::start(Camera *camera, const ControlList *controls) { SimpleCameraData *data = cameraData(camera); V4L2VideoDevice *video = data->video_; @@ -1705,7 +1705,7 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL if (data->converter_) ret = data->converter_->start(); else if (data->swIsp_) - ret = data->swIsp_->start(); + ret = data->swIsp_->start(controls ? *controls : ControlList()); else ret = 0; diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index ae86c20a..432cab24 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -374,11 +374,12 @@ int SoftwareIsp::queueBuffers(uint32_t frame, FrameBuffer *input, /** * \brief Starts the Software ISP streaming operation + * \param[in] controls The controls to apply before the first frame * \return 0 on success, any other value indicates an error */ -int SoftwareIsp::start() +int SoftwareIsp::start(const ControlList &controls) { - int ret = ipa_->start(); + int ret = ipa_->start(controls); if (ret) return ret; From patchwork Wed Aug 26 08:23:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28101 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4691DC333E for ; Wed, 26 Aug 2026 08:33:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EAE246842C; Wed, 26 Aug 2026 10:33:29 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="P6tJmsBv"; dkim-atps=neutral Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E079668425 for ; Wed, 26 Aug 2026 10:23:38 +0200 (CEST) Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-c1600d040e4so106221766b.1 for ; Wed, 26 Aug 2026 01:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732618; x=1788337418; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=sR7eQIRTodmLB3PIU5BlGuGEzYYEqBDYfXYra7p9Wek=; b=P6tJmsBvL1FQCj9ISQediregl71oRQePGrdcZOr15uIUdLcIxpDF5WuKiVhzZWi3WV cIJdOWTcWuGsgs3w2PHuZuIkKNxNfEDBavoVId4H5XZJzZn98ErYLwdnx7ytKlTblNrC axILdwDtH7L8E0vgFAPBlIaHn/zY85AjQ8gt6AbrWG+nosRhRBk6msaXtQd4mgwCS/Hx 3QNIF0froIFbVy743aFLkTTzcZ26l4cV9YpvfVjoT6pYy4xejUb0jHHBBolrl5JAtwNA tyTFy4XgwijbItga0nE7vhFXkfvb2SnHC4TCO6aGCfx/0YwBhbEZsM7eHklIpZqtxO6A 4c1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732618; x=1788337418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=sR7eQIRTodmLB3PIU5BlGuGEzYYEqBDYfXYra7p9Wek=; b=YgC9uPa752UZhmaFXhxeCpY8FBR7yrUueY0cp7mInLpoHWmVhGV0KKJOHgyUwzzdKS r3rrB7XjT2kwttLkDRNQm2sZT6BTv2UZZwZgQQjtji3RJ5Z1BG4zt2hB2WD1S17jBd0t rc4jE9cvr3WNgxC2Go55IBgg3StbSd+taDnk5JrTU961U/aJ2PGDOJJTLDqjMIXqW0US 18j2ZmX9w8neEx2ZzEqN0GHQvtCB60A8H4DI4VNuLvDFXCBYB6Alfhhz6t+nR4yGsMSE uXY6f8JtyEuXC6eY5KjN6Dl7FdWp3MVDorNEJsT0/pUGBQ2XBM1Ke/6lE+sJN2Z6J3IS laOg== X-Gm-Message-State: AFuF++n4M5z9gJotrTsqKdIexQjFNRRv9p2UtEY/XYkpsdKOVp7143SE TyUIWsdXoQ6lL21GgdCwsRzzs/WTY2Ua/2ejmMk7Af2ozTaPuEb0+hm82p3S9JrneEI= X-Gm-Gg: AR+sD10Y6Ku3513FnbGK3GnyAReidiu/q15Ndj1wqLBZkaP4EfInMnkh7CQM2oSXt+J tqUZrkgWxNiLq9s7rjcW1tF1DUoIUrXpa4um1OIGp/6ZMGM+0mPHXQyxw+vW/C1HkTn+pqN+MEQ yqDdYLpiGpVH/7b+MAn3mWbjpL5Jj3xcEdpNbgFoufudSCFHm1gkHayNrHzdraSxs7fMDcCvpcY uwn4ruepJuY7HWIXtXVq8LJDx0Envb4HRcx2wQW0rXonQCIRmo6DmKHoIIDhZzcaa0+B6Nt13yj HaZtXLq1j0RDUlKNV1zrihZJncqev2dbFHdPyE+AM/zNcUEjRrzUA5W19bp1WeZ9+vswzfcR98s bn9yHBvVxbb6WEVoZmX0KEE4NQGZsZUI7t+NkWBlflyOBKsmzje78ZUTA+efL6XdMT/FOoabGFl pAH+t0l4SQOxkToCHTzD4vHfT6TrXPIdBxxJhMijCged+3c4/TL6rc3BGitQWAqaTF9wXF+cNad oURUuh/ga07UN9Wc7YPCz/wurB7+fL7REEI8gQvdAUTh6i3UA7Vg3BMtz1I694kUDb6OxVSQ0fm 1PglK9mPxXXevYI= X-Received: by 2002:a17:906:6a1d:b0:c15:c323:36e5 with SMTP id a640c23a62f3a-c24e2d2b08amr1414392366b.10.1787732618137; Wed, 26 Aug 2026 01:23:38 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:37 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 5/7] ipa: softisp: Converge faster and add digital gain to the AGC Date: Wed, 26 Aug 2026 10:23:26 +0200 Message-ID: <20260826082328.24176-6-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AGC corrects the exposure by a small proportional step per statistics period, at most 15% and typically around 5%. Getting from a black start to a bright target then needs tens of statistics periods, and as each period spans several frames and the frames get long in low light, the convergence takes tens of seconds. Far from the target, jump by the ratio between the target and the measured mean sample value, bounded to a factor of two per step; the proportional correction takes over near the target so the convergence stays smooth. When the exposure and the analogue gain are both exhausted, apply a digital gain in the ISP by scaling the colour gains, after black level subtraction. It doesn't add information, so it is only used last and is bounded by the new maxDigitalGain tuning parameter (default 1.0, which disables it). The statistics are computed on the sensor data, so the histogram is scaled by the digital gain of the frame before the mean sample value is computed. The gain is reported in the DigitalGain metadata. Signed-off-by: Robert Bozik --- src/ipa/softisp/algorithms/agc.cpp | 94 ++++++++++++++++++++++++++---- src/ipa/softisp/algorithms/agc.h | 5 ++ src/ipa/softisp/ipa_context.h | 4 ++ 3 files changed, 91 insertions(+), 12 deletions(-) diff --git a/src/ipa/softisp/algorithms/agc.cpp b/src/ipa/softisp/algorithms/agc.cpp index ca403300..b6fc7203 100644 --- a/src/ipa/softisp/algorithms/agc.cpp +++ b/src/ipa/softisp/algorithms/agc.cpp @@ -66,12 +66,37 @@ static constexpr float kExpProportionalGain = 0.04; */ static constexpr float kExpMaxStep = 0.15; +/* + * Errors above this threshold are far from the target, and the small + * proportional steps would need tens of statistics periods to get there + * (a statistics period being several frames, and the frame possibly long + * in low light). For those, jump by the ratio between the target and the + * measured MSV instead, bounded by kExpMaxJump per step. The proportional + * correction takes over near the target, so the convergence stays smooth. + */ +static constexpr float kExpLargeError = 0.5; +static constexpr float kExpMaxJump = 2.0; + +/* + * Digital gain is applied by the ISP on top of the sensor exposure and + * analogue gain, only when those are exhausted. It doesn't add information, + * so the tuning file bounds it with maxDigitalGain (1.0 disables it). + */ +static constexpr double kDefaultMaxDigitalGain = 1.0; + Agc::Agc() { } -int Agc::init(IPAContext &context, [[maybe_unused]] const ValueNode &tuningData) +int Agc::init(IPAContext &context, const ValueNode &tuningData) { + maxDigitalGain_ = tuningData["maxDigitalGain"].get(kDefaultMaxDigitalGain); + if (maxDigitalGain_ < 1.0) { + LOG(IPASoftIspExposure, Warning) + << "maxDigitalGain " << maxDigitalGain_ << " below 1.0, ignored"; + maxDigitalGain_ = 1.0; + } + /* * Expose the frame duration limits the sensor can achieve in the * current mode. Whether the IPA can actually change the frame duration @@ -116,10 +141,24 @@ int Agc::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &co agc.maxFrameDuration = agc.minFrameDuration; } agc.vblank = cfg.vblankDef; + agc.dgain = 1.0; + context.configuration.agc.dgainMax = maxDigitalGain_; return 0; } +void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, DebayerParams *params) +{ + /* + * The digital gain scales the colour gains the ISP applies after black + * level subtraction. This runs after the AWB has set the gains, as the + * Agc algorithm is listed after Awb in the tuning file. + */ + frameContext.agc.digitalGain = context.activeState.agc.dgain; + params->gains *= frameContext.agc.digitalGain; +} + void Agc::queueRequest(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, const ControlList &controls) { @@ -201,6 +240,8 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou int32_t &exposure = frameContext.sensor.exposure; double &again = frameContext.sensor.gain; int32_t &vblank = frameContext.sensor.vblank; + double &dgain = frameContext.agc.digitalGain; + const double dgainMax = context.configuration.agc.dgainMax; int32_t vblankLo, vblankHi; vblankRange(context, frameContext, vblankLo, vblankHi); @@ -227,32 +268,47 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou } /* - * Compute a proportional correction factor. The sign of the error - * determines the direction: positive error means too dark (increase), - * negative means too bright (decrease). + * Compute the correction factor. The sign of the error determines the + * direction: positive error means too dark (increase), negative means + * too bright (decrease). Far from the target, jump by the measured + * ratio; near it, apply a small proportional step. */ - float step = std::clamp(static_cast(error) * kExpProportionalGain, - -kExpMaxStep, kExpMaxStep); - float factor = 1.0f + step; + float factor; + if (std::abs(error) > kExpLargeError) { + factor = std::clamp(static_cast(kExposureOptimal / std::max(exposureMSV, 0.1)), + 1.0f / kExpMaxJump, kExpMaxJump); + } else { + float step = std::clamp(static_cast(error) * kExpProportionalGain, + -kExpMaxStep, kExpMaxStep); + factor = 1.0f + step; + } if (factor > 1.0f) { /* * Scene too dark: increase exposure first (lengthening the - * frame when the limits allow it), then gain. + * frame when the limits allow it), then analogue gain, then + * digital gain. */ if (exposure < exposureMax) { int32_t next = static_cast(exposure * factor); exposure = std::max(next, exposure + 1); - } else { + } else if (again < context.configuration.agc.againMax) { double next = again * factor; if (next - again < context.configuration.agc.againMinStep) again += context.configuration.agc.againMinStep; else again = next; + } else { + dgain = std::min(dgain * factor, dgainMax); } } else { - /* Scene too bright: decrease gain first, then exposure. */ - if (again > context.configuration.agc.again10) { + /* + * Scene too bright: decrease digital gain first, then analogue + * gain, then exposure. + */ + if (dgain > 1.0) { + dgain = std::max(dgain * factor, 1.0); + } else if (again > context.configuration.agc.again10) { double next = again * factor; if (again - next < context.configuration.agc.againMinStep) again -= context.configuration.agc.againMinStep; @@ -272,14 +328,18 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou static_cast(context.configuration.agc.frameHeight), vblankLo, vblankHi); + dgain = std::clamp(dgain, 1.0, dgainMax); + context.activeState.agc.exposure = exposure; context.activeState.agc.again = again; context.activeState.agc.vblank = vblank; + context.activeState.agc.dgain = dgain; LOG(IPASoftIspExposure, Debug) << "exposureMSV " << exposureMSV << " error " << error << " factor " << factor << " exp " << exposure << " again " << again + << " dgain " << dgain << " vblank " << vblank << " (" << vblankLo << "-" << vblankHi << ")"; } @@ -293,6 +353,7 @@ void Agc::process(IPAContext &context, utils::Duration exposureTime = cfg.lineDuration * frameContext.sensor.exposure; metadata.set(controls::ExposureTime, exposureTime.get()); metadata.set(controls::AnalogueGain, frameContext.sensor.gain); + metadata.set(controls::DigitalGain, static_cast(frameContext.agc.digitalGain)); if (cfg.vblankSupported) { frameContext.agc.frameDuration = cfg.lineDuration * (cfg.frameHeight + frameContext.sensor.vblank); @@ -319,6 +380,7 @@ void Agc::process(IPAContext &context, frameContext.sensor.exposure = context.activeState.agc.exposure; frameContext.sensor.gain = context.activeState.agc.again; frameContext.sensor.vblank = context.activeState.agc.vblank; + frameContext.agc.digitalGain = context.activeState.agc.dgain; return; } @@ -344,8 +406,16 @@ void Agc::process(IPAContext &context, return; } + /* + * The statistics are computed on the sensor data, before the ISP + * applies the digital gain. Scale the histogram index by the digital + * gain of the frame so that the MSV reflects the output brightness. + */ + const double digitalGain = frameContext.agc.digitalGain; for (unsigned int i = 0; i < histogramSize; i++) { - unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + unsigned int scaled = std::min( + static_cast(i * digitalGain), histogramSize - 1); + unsigned int idx = (scaled - (scaled / yHistValsPerBinMod)) / yHistValsPerBin; exposureBins[idx] += histogram[blackLevelHistIdx + i]; } diff --git a/src/ipa/softisp/algorithms/agc.h b/src/ipa/softisp/algorithms/agc.h index 1aa77737..6e9ba728 100644 --- a/src/ipa/softisp/algorithms/agc.h +++ b/src/ipa/softisp/algorithms/agc.h @@ -24,6 +24,9 @@ public: void queueRequest(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ControlList &controls) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const SwIspStats *stats, @@ -34,6 +37,8 @@ private: void vblankRange(const IPAContext &context, const IPAFrameContext &frameContext, int32_t &vblankLo, int32_t &vblankHi) const; int32_t exposureMaxForVblank(const IPAContext &context, int32_t vblank) const; + + double maxDigitalGain_; }; } /* namespace ipa::softisp::algorithms */ diff --git a/src/ipa/softisp/ipa_context.h b/src/ipa/softisp/ipa_context.h index b5577768..de188f14 100644 --- a/src/ipa/softisp/ipa_context.h +++ b/src/ipa/softisp/ipa_context.h @@ -42,6 +42,8 @@ struct IPASessionConfiguration { /* Lines the sensor keeps between max exposure and frame length */ int32_t exposureMargin; uint32_t frameHeight; + /* Digital gain applied by the ISP after the sensor gain is exhausted */ + double dgainMax; } agc; struct { std::optional level; @@ -56,6 +58,7 @@ struct IPAActiveState { int32_t exposure; double again; int32_t vblank; + double dgain; bool valid; utils::Duration minFrameDuration; utils::Duration maxFrameDuration; @@ -91,6 +94,7 @@ struct IPAFrameContext : public FrameContext { utils::Duration minFrameDuration; utils::Duration maxFrameDuration; utils::Duration frameDuration; + double digitalGain; } agc; float gamma; From patchwork Wed Aug 26 08:23:27 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28102 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 28473C333D for ; Wed, 26 Aug 2026 08:33:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 042D368446; Wed, 26 Aug 2026 10:33:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="m03I33LF"; dkim-atps=neutral Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5E77068421 for ; Wed, 26 Aug 2026 10:23:40 +0200 (CEST) Received: by mail-ed1-x52f.google.com with SMTP id 4fb4d7f45d1cf-6a18e24ad25so884382a12.1 for ; Wed, 26 Aug 2026 01:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732620; x=1788337420; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WQwB7xsAKMXncRqqK9sNh6HsEyacyJk53ZfjeTdxXgo=; b=m03I33LFiLjSTXNZCOJB0lKyuZtdbvA6UnbzVY7XDpmx9EWBz+1jGcFD115qpi7JO2 4ZYyEkiFFYjgdxOiEbX39CL39jiZR05K5pfqCUzY+iqOLslaHXmvMQbTpTIne+IUYNBs YYF/jfGjEbTj21A7kWF8qkXs20lv0mGqsXHY+PLOAniHd5zn1s8rpBbT4j2Wa4q0JQQG m3hJTd3UBzrIgMqOOvI78DsX62Z/AbMPh4UQZiMywpbqa2ZLfzbOUklw6bDclEOOnG5p Xt2zUYMKHO1YUidoe7KbkFfXldFZtj/tYM/ChYFVjrnVZlfArI+Uq4lpZTJhUq7GWnVK 442Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732620; x=1788337420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=WQwB7xsAKMXncRqqK9sNh6HsEyacyJk53ZfjeTdxXgo=; b=ZeTN4yzyZgk19YFJqX+7RfjSVuDKlzItpxbmf4K3HLP9nQizNZLkFCoKrbByAya3QA xpQ39pxoLsXNPyHmR0J3Pl/jkOGNul1epAZKQPZlohA5ON1l3cfb9V+riPBia01Wgf7s dW5/+CgQ+JGWQcBgURl6yOwMl5BzkbII61AeUmVE+XWbj5wy+8rk1CbSrXMUGAqTXnwq KdDwE0H5uqo7R89K5SRkILFotV75QgfiyoPPS2G+dKJgwpiCL8sJt4L4+AINfmwJJJ0s tunW4C/89hdbgRYTyWhg5FKQKRdNh5ldM2giS+1wwaxOKbXDHL73BYPcpJSALA5om/Cn cnnA== X-Gm-Message-State: AFuF++l+W6PNJlZfmngJxrFeDicDXhRjIuo9eaFZD2MF32mQz9drpMVQ XfqzgXBFrXo7bnayhqkztz6wlxq9vHLY+2rb5zzQOAkfjNmswFj35id8gkS67ym/8FU= X-Gm-Gg: AR+sD13wJZo3iLe4RWs4B/DCjmVMs5tjhWngkMdB3DYbskvpnjkabbSuNnOD8ArpUO0 FdBl9Ct3jIjV8PoOuAoSQJMldbkcbjzPPtxVQb+yH7xXtm81d3V2idDqd/ZZr4Nms7b8Fu3e1jz PZmk4amaFjMVfjhZ4ll0TzIVfa8I54VM/e55nL/thWGc3D0FbEwHwGjUanWGHqlcetO2j9063C2 SWpW8C6PciYmkPwa45SHeTNDTY9IladE4THlkfhlKrey6XeOLUJTiior8+i14JXpd4m7wg2KbT8 FVjjzFheS8ggN31H59kFmjzBxV87T2p6DTv4WkUmG4IOn0SYJlH4bnM8RkalmHs78aMmmxfs/+V Ukk3JGwngfxJECc2aQDpgTfzO7sbYst5YY9HqC3+O0o+9Te+tv8KCFkCG8TT48hQzf2aRD/pZX6 jQUL36MnsDHB6wZPwIWlH3rF84ENHOlqyt+aDRxGHCLi51+2VB0F9kGZ+wFKQO2kyK3ZJwZ9Bpz kMkhnKgxySxANZje0VaBjiQVDep6Be651ZPagY+5ajRKC0IfK++q/qrZS6uhVkl+kNQpIuwrpN/ 3iwDeuvAh0TpBQ4= X-Received: by 2002:a17:906:ef04:b0:c20:af9d:454a with SMTP id a640c23a62f3a-c250c36aa50mr599124466b.16.1787732619590; Wed, 26 Aug 2026 01:23:39 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:38 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 6/7] ipa: softisp: Let the tuning file bound the default frame duration Date: Wed, 26 Aug 2026 10:23:27 +0200 Message-ID: <20260826082328.24176-7-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Applications that don't set FrameDurationLimits get the full range the sensor mode supports, so in low light the AGC lengthens the frame as far as the sensor allows. Not all applications that omit the control expect that: the PipeWire libcamera source, for instance, negotiates a frame rate with its clients but never passes it to libcamera, and a video call would then drop to a frame every second in a dim room. Add a maxFrameDuration tuning parameter (in microseconds) that bounds the maximum frame duration used when no FrameDurationLimits control is set. Explicit FrameDurationLimits keep allowing the full sensor range. Signed-off-by: Robert Bozik --- src/ipa/softisp/algorithms/agc.cpp | 24 ++++++++++++++++++++++++ src/ipa/softisp/algorithms/agc.h | 5 +++++ 2 files changed, 29 insertions(+) diff --git a/src/ipa/softisp/algorithms/agc.cpp b/src/ipa/softisp/algorithms/agc.cpp index b6fc7203..0037fa6e 100644 --- a/src/ipa/softisp/algorithms/agc.cpp +++ b/src/ipa/softisp/algorithms/agc.cpp @@ -84,6 +84,15 @@ static constexpr float kExpMaxJump = 2.0; */ static constexpr double kDefaultMaxDigitalGain = 1.0; +/* + * Applications that don't set FrameDurationLimits get the full range the + * sensor supports, which lets the AGC slow the frame rate down to whatever + * the sensor allows in low light. That is rarely what an application which + * didn't ask for it expects, so the tuning file can bound the default + * maximum frame duration with maxFrameDuration (in microseconds); explicit + * FrameDurationLimits still allow the full sensor range. + */ + Agc::Agc() { } @@ -91,6 +100,17 @@ Agc::Agc() int Agc::init(IPAContext &context, const ValueNode &tuningData) { maxDigitalGain_ = tuningData["maxDigitalGain"].get(kDefaultMaxDigitalGain); + + const auto tuningMaxFrameDuration = tuningData["maxFrameDuration"].get(); + if (tuningMaxFrameDuration) { + if (*tuningMaxFrameDuration == 0) { + LOG(IPASoftIspExposure, Warning) + << "maxFrameDuration must be positive, ignored"; + } else { + defaultMaxFrameDuration_ = + std::chrono::microseconds(*tuningMaxFrameDuration); + } + } if (maxDigitalGain_ < 1.0) { LOG(IPASoftIspExposure, Warning) << "maxDigitalGain " << maxDigitalGain_ << " below 1.0, ignored"; @@ -136,6 +156,10 @@ int Agc::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &co if (it != context.ctrlMap.end() && cfg.vblankSupported) { agc.minFrameDuration = std::chrono::microseconds(it->second.min().get()); agc.maxFrameDuration = std::chrono::microseconds(it->second.max().get()); + if (defaultMaxFrameDuration_) + agc.maxFrameDuration = std::clamp(*defaultMaxFrameDuration_, + agc.minFrameDuration, + agc.maxFrameDuration); } else { agc.minFrameDuration = cfg.lineDuration * (cfg.frameHeight + cfg.vblankDef); agc.maxFrameDuration = agc.minFrameDuration; diff --git a/src/ipa/softisp/algorithms/agc.h b/src/ipa/softisp/algorithms/agc.h index 6e9ba728..ad732b1a 100644 --- a/src/ipa/softisp/algorithms/agc.h +++ b/src/ipa/softisp/algorithms/agc.h @@ -7,6 +7,10 @@ #pragma once +#include + +#include + #include "algorithm.h" namespace libcamera { @@ -39,6 +43,7 @@ private: int32_t exposureMaxForVblank(const IPAContext &context, int32_t vblank) const; double maxDigitalGain_; + std::optional defaultMaxFrameDuration_; }; } /* namespace ipa::softisp::algorithms */ From patchwork Wed Aug 26 08:23:28 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28103 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0C00AC333F for ; Wed, 26 Aug 2026 08:33:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DB99868449; Wed, 26 Aug 2026 10:33:33 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AUCVT+xu"; dkim-atps=neutral Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 45C9768425 for ; Wed, 26 Aug 2026 10:23:42 +0200 (CEST) Received: by mail-ej1-x635.google.com with SMTP id a640c23a62f3a-c197e7e4e94so97532466b.2 for ; Wed, 26 Aug 2026 01:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732622; x=1788337422; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=L7EALqeayE9XM/VBr6eyx28jnmexVhT88+UrZ5gwy+c=; b=AUCVT+xuOiSe126G35fjb8P5xjmKNL5HWd0xqNVqzqh5hhZWhmkOzLLspQ7MY2K8X6 QTb/CWiW7aVVLBW7kU/yN5AG44GyBaYQIWQe1N+KEuUYZYcsgbWNJFWil/Q0gzcy/4Y5 Jnn+9jKmQH3pE3r76aacpd1tpgY61t9448ERdvRiVXjS+VWZdXjmCUepluD5IxUsEya7 178QqI1qEptQ3jTuYgUv0DrGROHGgbfy9RuEyS7oRagti7LRRu0F3Q45DMCkMR5UCoyV 8onwKF+nh4lfubTzovOoq4iUuJ606QCkjhRBpKMb8waQvIrapSTmAKHfWY5IhqhoK0Ez D5qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732622; x=1788337422; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=L7EALqeayE9XM/VBr6eyx28jnmexVhT88+UrZ5gwy+c=; b=BBwJ6aEBdlYcEZEFvNz4t02Vfuh6QwrNz9A37S+w38WQVw8FrUrn4s6KyCmqNSaarK WdOZe1QBhOzdOzJZlPNwNp8x+Lw9Bpu0EdzizqUsFmS95jZNtZy/OP1M3dSU2UC0i3tx EaiM+PLlUlw+P7ASGyMZXHnmu+kUcp08p6z1Gpjr9HMG/niCljsjCsGMroZOVMixTOib jRVUhbgwQS4ENPpAS4TTYTepgcqZLuMCoZfwwOmEjC+/QLxK3S1dFQ3tOc2rhE7NgW2H g9nplE/HHu5T0PfzErqWM6mqcWIgBgjr+ld9bICzFALuUkcCgYibIJHIqMTz9kvYsJQm UCoQ== X-Gm-Message-State: AFuF++mFA3FdmzigTUOglxp+/VVpwGmtiHppUC1yZGaU68zSUPHdrTPH J3s6tIrtNt8d1giHq4j3lWSVlb9Fxf5fwXZ5gRYdzCvrh9AY5Xpzuqryq0G2J9pgPUc= X-Gm-Gg: AR+sD12lq2da8zoZ3r7vrUVp9Q0CQTkMoBe/x6O9QcpqshiowhT/pKKPBgLSnFPdQVg b6grNkvLLDRU3MA+QZbJH45QmwQAr01KV+vmrIH5S/oeNLzgsvRgrUa6c/ZaY3DzbsY/r+cHEp0 LzSRfw9pjSjWnwTTX/TlhQyoxHpAdBS2tY50zRI7YkQkN8sB2r369jYYqsq41f/ui26urjpHect f57CbH54B0QM3rYWGtOsjnUGJZ2o73lavtaPHlZzjyW6u1ioGjrGPeOmSNs6kX4A/RV5Tk3rxhB iiQ7/ropyFA6++2FkABFrhJa+ArKisHufJQZbUgNwtbKev1xqhealllv6PmqVTumaSpekOnTKwe goBIkXJjXvnHK2dH1/eaAQ6Ae6XTLJORySHIdWS4Yww4KGThBTvAyiaeT8gMIY7TG/DbfJDNjOV 3Cy3iXBzL4FsAZvu6nO1I2PPN1Z+Jaa7B5SZKxLvDqX+lqlBqj+I5726oERVU7lOrDe2afJ7k/X cmk+qZupqU9HfToWKM6TZ1vIKggX4ksNpVcMt/gq1a9oNH0yC3VatTKxqxg4470C0U0WkwxbZv4 CdNwCyVp6vZbNG4= X-Received: by 2002:a17:907:ea8b:b0:c20:3402:bd35 with SMTP id a640c23a62f3a-c250c330f70mr573682966b.12.1787732621342; Wed, 26 Aug 2026 01:23:41 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:40 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 7/7] libcamera: software_isp: Add temporal noise reduction to the GPU debayer Date: Wed, 26 Aug 2026 10:23:28 +0200 Message-ID: <20260826082328.24176-8-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Under-display and other small sensors run at maximum gain in ordinary room light, and the software ISP then applies a digital gain on top. The resulting noise is not only visible as grain: the noise around the black level is symmetric, but once the black level is subtracted the negative half is clamped to zero, leaving a positive bias that lifts the blacks and desaturates the colours. Averaging frames before that subtraction removes both. Add a temporal noise reduction pass to the EGL debayer for the unpacked Bayer formats. The raw frame is blended with the previously filtered one into a history texture, and the debayer shader then samples the filtered frame instead of the raw input. Motion is detected on the mean of the 4x4 block around each pixel: where the current frame differs from the history by more than a few sigmas of the noise expected for that block, the current frame is used as is, so that moving edges do not ghost. The history is an RGBA8 texture, which every GLES 2.0 implementation can render to, laid out like the raw input texture (low byte, high byte) so that the debayer shader decodes either with the same code; the fraction of the filtered value is kept in the third channel for the precision of the recursive filter. The parameters come from a new Denoise IPA algorithm: alpha is the weight of the current frame (1.0 disables the filter); noiseSlope and noiseFloor describe the sensor noise model at unity analogue gain (variance in normalised raw units = noiseSlope * signal + noiseFloor), which the IPA scales for the analogue gain in use; motionSigma is the number of noise sigmas of the block mean above which a pixel is treated as moving. The CPU debayer ignores the parameters for now. Signed-off-by: Robert Bozik --- .../internal/software_isp/debayer_params.h | 14 ++ src/ipa/softisp/algorithms/denoise.cpp | 89 +++++++++ src/ipa/softisp/algorithms/denoise.h | 36 ++++ src/ipa/softisp/algorithms/meson.build | 1 + src/ipa/softisp/softisp.cpp | 1 + src/libcamera/shaders/meson.build | 1 + src/libcamera/shaders/temporal.frag | 131 +++++++++++++ src/libcamera/software_isp/debayer.cpp | 13 ++ src/libcamera/software_isp/debayer_egl.cpp | 172 ++++++++++++++++++ src/libcamera/software_isp/debayer_egl.h | 23 +++ 10 files changed, 481 insertions(+) create mode 100644 src/ipa/softisp/algorithms/denoise.cpp create mode 100644 src/ipa/softisp/algorithms/denoise.h create mode 100644 src/libcamera/shaders/temporal.frag diff --git a/include/libcamera/internal/software_isp/debayer_params.h b/include/libcamera/internal/software_isp/debayer_params.h index 1074720d..3ba794a3 100644 --- a/include/libcamera/internal/software_isp/debayer_params.h +++ b/include/libcamera/internal/software_isp/debayer_params.h @@ -25,6 +25,20 @@ struct DebayerParams { float gamma = 1.0; float contrastExp = 1.0; RGB gains = RGB({ 1.0, 1.0, 1.0 }); + /* + * Temporal noise reduction of the raw data. alpha is the weight of + * the current frame (1.0 disables the filter). Motion is detected + * where the frame differs from the history by more than motionSigma + * times the expected noise, whose variance in normalised raw units is + * noiseSlope * signal + noiseFloor (signal above black level, already + * scaled for the current analogue gain). + */ + struct { + float alpha = 1.0; + float noiseSlope = 0.0; + float noiseFloor = 0.0; + float motionSigma = 0.0; + } temporalDenoise; }; } /* namespace libcamera */ diff --git a/src/ipa/softisp/algorithms/denoise.cpp b/src/ipa/softisp/algorithms/denoise.cpp new file mode 100644 index 00000000..a13440f6 --- /dev/null +++ b/src/ipa/softisp/algorithms/denoise.cpp @@ -0,0 +1,89 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Robert Bozik + * + * Temporal noise reduction parameters + */ + +#include "denoise.h" + +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftIspDenoise) + +namespace ipa::softisp::algorithms { + +/* + * Temporal noise reduction blends each raw frame with the previously + * filtered one, before black level subtraction. The tuning file provides: + * + * - alpha: weight of the current frame, in ]0, 1]. Lower values average + * more frames (the noise standard deviation drops by roughly + * sqrt(alpha / (2 - alpha))) but react slower to changes; 1.0 disables + * the filter. + * - noiseSlope, noiseFloor: sensor noise model at unity analogue gain, + * in normalised raw units: variance = noiseSlope * gain * signal + + * noiseFloor * gain^2, with the signal above the black level. The + * slope is the shot noise, the floor the read noise. Both are measured + * from the difference of consecutive frames of a static scene. + * - motionSigma: difference between the current frame and the history, + * in sigmas of that noise, above which a pixel is considered to have + * moved and the current frame is used as is. + */ +static constexpr float kDefaultAlpha = 1.0f; +static constexpr float kDefaultMotionSigma = 3.0f; + +int Denoise::init([[maybe_unused]] IPAContext &context, const ValueNode &tuningData) +{ + alpha_ = tuningData["alpha"].get(kDefaultAlpha); + noiseSlope_ = tuningData["noiseSlope"].get(0.0); + noiseFloor_ = tuningData["noiseFloor"].get(0.0); + motionSigma_ = tuningData["motionSigma"].get(kDefaultMotionSigma); + + if (alpha_ <= 0.0f || alpha_ > 1.0f) { + LOG(IPASoftIspDenoise, Error) + << "alpha must be in ]0, 1], got " << alpha_; + return -EINVAL; + } + if (noiseSlope_ < 0.0f || noiseFloor_ < 0.0f || motionSigma_ <= 0.0f) { + LOG(IPASoftIspDenoise, Error) + << "noiseSlope and noiseFloor must not be negative, " + << "motionSigma must be positive"; + return -EINVAL; + } + if (alpha_ < 1.0f && noiseSlope_ == 0.0f && noiseFloor_ == 0.0f) { + LOG(IPASoftIspDenoise, Error) + << "A noise model (noiseSlope/noiseFloor) is required " + << "for temporal denoising"; + return -EINVAL; + } + + LOG(IPASoftIspDenoise, Info) + << "Temporal denoise alpha " << alpha_ + << " noise slope " << noiseSlope_ << " floor " << noiseFloor_ + << " motion " << motionSigma_ << " sigma"; + + return 0; +} + +void Denoise::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + DebayerParams *params) +{ + const double gain = std::max(context.activeState.agc.again, 1.0); + + params->temporalDenoise.alpha = alpha_; + params->temporalDenoise.noiseSlope = noiseSlope_ * gain; + params->temporalDenoise.noiseFloor = noiseFloor_ * gain * gain; + params->temporalDenoise.motionSigma = motionSigma_; +} + +REGISTER_IPA_ALGORITHM(Denoise, "Denoise") + +} /* namespace ipa::softisp::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/softisp/algorithms/denoise.h b/src/ipa/softisp/algorithms/denoise.h new file mode 100644 index 00000000..34c03fe4 --- /dev/null +++ b/src/ipa/softisp/algorithms/denoise.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Robert Bozik + * + * Temporal noise reduction parameters + */ + +#pragma once + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::softisp::algorithms { + +class Denoise : public Algorithm +{ +public: + Denoise() = default; + ~Denoise() = default; + + int init(IPAContext &context, const ValueNode &tuningData) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + +private: + float alpha_; + float noiseSlope_; + float noiseFloor_; + float motionSigma_; +}; + +} /* namespace ipa::softisp::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/softisp/algorithms/meson.build b/src/ipa/softisp/algorithms/meson.build index d240409e..ea87afa8 100644 --- a/src/ipa/softisp/algorithms/meson.build +++ b/src/ipa/softisp/algorithms/meson.build @@ -6,4 +6,5 @@ softisp_ipa_algorithms = files([ 'agc.cpp', 'blc.cpp', 'ccm.cpp', + 'denoise.cpp', ]) diff --git a/src/ipa/softisp/softisp.cpp b/src/ipa/softisp/softisp.cpp index 6d6b80d5..80155d19 100644 --- a/src/ipa/softisp/softisp.cpp +++ b/src/ipa/softisp/softisp.cpp @@ -164,6 +164,7 @@ int IPASoftIsp::init(const IPASettings &settings, params_->gamma = 1.0 / algorithms::kDefaultGamma; params_->contrastExp = 1.0; params_->gains = { { 1.0, 1.0, 1.0 } }; + params_->temporalDenoise = {}; /* combinedMatrix is reset for each frame. */ } diff --git a/src/libcamera/shaders/meson.build b/src/libcamera/shaders/meson.build index c409ff9b..147a2ac9 100644 --- a/src/libcamera/shaders/meson.build +++ b/src/libcamera/shaders/meson.build @@ -7,6 +7,7 @@ shader_files = files([ 'bayer_unpacked.frag', 'bayer_unpacked.vert', 'identity.vert', + 'temporal.frag', ]) # Generate header from shaders diff --git a/src/libcamera/shaders/temporal.frag b/src/libcamera/shaders/temporal.frag new file mode 100644 index 00000000..ec7611cb --- /dev/null +++ b/src/libcamera/shaders/temporal.frag @@ -0,0 +1,131 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Robert Bozik + * + * temporal.frag - Temporal noise reduction of raw Bayer data + * + * Blends the current raw frame with the previously filtered one, before + * black level subtraction, so that the noise is averaged before any + * clamping rectifies it. Where the frame changed by more than a few + * sigmas of the expected sensor noise the current frame is used as is, + * to avoid ghosting on motion. + * + * The filtered frame is kept in an RGBA8 texture, which every GLES 2.0 + * implementation can render to, laid out like the raw input so that the + * debayer shader can sample either. + */ + +#ifdef GL_ES +precision highp float; +#endif + +uniform sampler2D tex_y; /* Current raw frame */ +uniform sampler2D tex_hist; /* Previous filtered frame, packed */ +uniform float alpha; /* Weight of the current frame */ +uniform float noise_a; /* Noise variance per unit of signal */ +uniform float noise_b; /* Noise variance floor */ +uniform float motion_k; /* Motion threshold in noise sigmas */ +uniform float black; /* Black level, normalised */ +uniform float hist_valid; /* 0.0 on the first frame */ + +varying vec2 textureOut; + +/* + * The history texture is RGBA8 and holds the filtered value in the same + * layout as the raw input texture, so that the debayer shader samples + * either one with the same decoding: for the unpacked 10 and 12 bit + * formats the low byte in .r and the high byte in .g, for 8 bit formats + * the value in .r. The fraction of the value is kept in .b for the + * precision of the recursive filter; the debayer shader ignores it. + * + * The decoding mirrors bayer_unpacked.frag: (lo + 256 * hi) / 1020 for + * 10 bit, (lo + 256 * hi) / 4080 for 12 bit. + */ +#if defined(RAW10P) +#define RAW_SCALE 1020.0 +#elif defined(RAW12P) +#define RAW_SCALE 4080.0 +#endif + +#if defined(RAW_SCALE) +float fetch_cur(vec2 uv) +{ + vec4 p = texture2D(tex_y, uv); + return (p.r * 255.0 + p.g * 255.0 * 256.0) / RAW_SCALE; +} + +float fetch_hist(vec2 uv) +{ + vec4 p = texture2D(tex_hist, uv); + return (p.r * 255.0 + p.g * 255.0 * 256.0 + p.b) / RAW_SCALE; +} + +vec4 pack_hist(float v) +{ + float raw = clamp(v, 0.0, 1.0) * RAW_SCALE; + float ip = floor(raw); + float hi = floor(ip / 256.0); + float lo = ip - hi * 256.0; + return vec4(lo / 255.0, hi / 255.0, raw - ip, 1.0); +} +#else +float fetch_cur(vec2 uv) +{ + return texture2D(tex_y, uv).r; +} + +float fetch_hist(vec2 uv) +{ + vec4 p = texture2D(tex_hist, uv); + return p.r + p.b / 255.0; +} + +vec4 pack_hist(float v) +{ + float raw = clamp(v, 0.0, 1.0) * 255.0; + float ip = floor(raw); + return vec4(ip / 255.0, 0.0, raw - ip, 1.0); +} +#endif + +uniform vec2 tex_step; /* 1 / texture size, one texel */ + +void main(void) +{ + float cur = fetch_cur(textureOut); + float prev = fetch_hist(textureOut); + + /* + * Detect motion on the mean of the 4x4 block around the pixel (two + * Bayer quads each way) rather than on the pixel alone: the noise of + * the mean is a quarter of that of a pixel, so the threshold can sit + * close to the real noise and still catch subtle motion, such as the + * trailing edge of an object over a background of similar brightness. + */ + float curMean = 0.0; + float prevMean = 0.0; + for (int j = -1; j <= 2; j++) { + for (int i = -1; i <= 2; i++) { + vec2 uv = textureOut + vec2(float(i) * tex_step.x, float(j) * tex_step.y); + curMean += fetch_cur(uv); + prevMean += fetch_hist(uv); + } + } + curMean *= 1.0 / 16.0; + prevMean *= 1.0 / 16.0; + float diff = abs(curMean - prevMean); + + /* + * Expected noise of the block mean from the sensor noise model: the + * variance grows with the signal (shot noise), and the mean of 16 + * pixels has a sixteenth of the variance of one. + */ + float signal = max(prevMean - black, 0.0); + float sigma = sqrt((noise_a * signal + noise_b) / 16.0); + float threshold = motion_k * sigma; + + float w = mix(alpha, 1.0, smoothstep(threshold, 1.5 * threshold, diff)); + float v = hist_valid > 0.5 ? mix(prev, cur, w) : cur; + + gl_FragColor = pack_hist(v); +} diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index a4854e51..4a17515c 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -43,6 +43,19 @@ namespace libcamera { * \brief Contrast value to be used as an exponent */ +/** + * \var DebayerParams::temporalDenoise + * \brief Temporal noise reduction parameters + * + * The raw frame is blended with the filtered history before debayering. + * alpha is the weight of the current frame (1.0 disables the filter). + * Motion is detected where the frame differs from the history by more than + * motionSigma times the expected noise standard deviation, whose variance + * in normalised raw units is noiseSlope * signal + noiseFloor, the signal + * being measured above the black level and scaled for the current analogue + * gain. + */ + /** * \class Debayer * \brief Base debayering class diff --git a/src/libcamera/software_isp/debayer_egl.cpp b/src/libcamera/software_isp/debayer_egl.cpp index 97aa0379..913685b1 100644 --- a/src/libcamera/software_isp/debayer_egl.cpp +++ b/src/libcamera/software_isp/debayer_egl.cpp @@ -238,6 +238,23 @@ int DebayerEGL::initBayerShaders(PixelFormat inputFormat, PixelFormat outputForm break; }; + /* + * The temporal noise reduction pass writes the filtered raw frame to + * an RGBA8 texture laid out like the raw input, which the debayer + * shader then samples instead of the raw input with the same decoding. + * It is only implemented for the unpacked formats handled by the + * bayer_unpacked shader. + */ + temporalSupported_ = false; + if (fragmentShaderData.data() == bayer_unpacked_frag.data()) { + if (initTemporalShaders(shaderEnv) == 0) { + temporalSupported_ = true; + } else { + LOG(Debayer, Warning) + << "Temporal noise reduction unavailable"; + } + } + if (egl_.compileVertexShader(vertexShaderId_, vertexShaderData, shaderEnv)) { LOG(Debayer, Error) << "Compile vertex shader fail"; @@ -488,6 +505,144 @@ void DebayerEGL::setShaderVariableValues(eGLImage &eglImageIn, const DebayerPara return; } +int DebayerEGL::initTemporalShaders(const std::vector &shaderEnv) +{ + GLuint vertexShaderId = 0; + GLuint fragmentShaderId = 0; + + if (egl_.compileVertexShader(vertexShaderId, identity_vert, shaderEnv)) { + LOG(Debayer, Error) << "Compile temporal vertex shader fail"; + return -ENODEV; + } + utils::scope_exit vShaderGuard([&] { glDeleteShader(vertexShaderId); }); + + if (egl_.compileFragmentShader(fragmentShaderId, temporal_frag, shaderEnv)) { + LOG(Debayer, Error) << "Compile temporal fragment shader fail"; + return -ENODEV; + } + utils::scope_exit fShaderGuard([&] { glDeleteShader(fragmentShaderId); }); + + if (egl_.linkProgram(temporalProgramId_, vertexShaderId, fragmentShaderId)) { + LOG(Debayer, Error) << "Linking temporal program fail"; + return -ENODEV; + } + + temporalAttributeVertex_ = glGetAttribLocation(temporalProgramId_, "vertexIn"); + temporalAttributeTexture_ = glGetAttribLocation(temporalProgramId_, "textureIn"); + temporalUniformProjMatrix_ = glGetUniformLocation(temporalProgramId_, "proj_matrix"); + temporalUniformStrideFactor_ = glGetUniformLocation(temporalProgramId_, "stride_factor"); + temporalUniformDataIn_ = glGetUniformLocation(temporalProgramId_, "tex_y"); + temporalUniformHist_ = glGetUniformLocation(temporalProgramId_, "tex_hist"); + temporalUniformAlpha_ = glGetUniformLocation(temporalProgramId_, "alpha"); + temporalUniformNoiseA_ = glGetUniformLocation(temporalProgramId_, "noise_a"); + temporalUniformNoiseB_ = glGetUniformLocation(temporalProgramId_, "noise_b"); + temporalUniformMotionK_ = glGetUniformLocation(temporalProgramId_, "motion_k"); + temporalUniformBlack_ = glGetUniformLocation(temporalProgramId_, "black"); + temporalUniformHistValid_ = glGetUniformLocation(temporalProgramId_, "hist_valid"); + temporalUniformStep_ = glGetUniformLocation(temporalProgramId_, "tex_step"); + + /* + * Two history textures, the same size as the input texture so that + * the debayer shader can sample the filtered frame with the + * coordinates it computes for the raw input. + */ + const uint32_t histWidth = inputConfig_.stride / bytesPerPixel_; + for (unsigned int i = 0; i < 2; i++) { + temporalHistory_[i] = std::make_unique(GL_RGBA, histWidth, height_, + histWidth * 4, + GL_TEXTURE2 + i, 2 + i); + egl_.createOutputTexture2D(*temporalHistory_[i]); + } + + GLenum err = glGetError(); + if (err != GL_NO_ERROR) { + LOG(Debayer, Error) << "Temporal history textures error " << err; + return -ENODEV; + } + + return 0; +} + +/* + * Blend the raw input with the previous filtered frame into the other + * history texture, and return 0 with the debayer input redirected to it. + * With alpha at 1.0 the filter is disabled and the history reset, so that + * re-enabling it doesn't blend with a stale frame. + */ +int DebayerEGL::temporalPass(eGLImage &eglImageIn, const DebayerParams ¶ms) +{ + if (params.temporalDenoise.alpha >= 1.0f) { + temporalHistoryValid_ = false; + temporalActive_ = false; + return 0; + } + + eGLImage &prev = *temporalHistory_[temporalIndex_]; + eGLImage &next = *temporalHistory_[temporalIndex_ ^ 1]; + + glUseProgram(temporalProgramId_); + + static const GLfloat identityMatrix[] = { + 1, 0, 0, 0, + 0, 1, 0, 0, + 0, 0, 1, 0, + 0, 0, 0, 1 + }; + static const GLfloat vcoordinates[4][2] = { + { -1.0f, -1.0f }, + { -1.0f, +1.0f }, + { +1.0f, +1.0f }, + { +1.0f, -1.0f }, + }; + static const GLfloat tcoordinates[4][2] = { + { 0.0f, 0.0f }, + { 0.0f, 1.0f }, + { 1.0f, 1.0f }, + { 1.0f, 0.0f }, + }; + + glEnableVertexAttribArray(temporalAttributeVertex_); + glVertexAttribPointer(temporalAttributeVertex_, 2, GL_FLOAT, GL_TRUE, + 2 * sizeof(GLfloat), vcoordinates); + glEnableVertexAttribArray(temporalAttributeTexture_); + glVertexAttribPointer(temporalAttributeTexture_, 2, GL_FLOAT, GL_TRUE, + 2 * sizeof(GLfloat), tcoordinates); + + egl_.activateBindTexture(eglImageIn); + egl_.activateBindTexture(prev); + glUniform1i(temporalUniformDataIn_, eglImageIn.texture_unit_uniform_id_); + glUniform1i(temporalUniformHist_, prev.texture_unit_uniform_id_); + glUniformMatrix4fv(temporalUniformProjMatrix_, 1, GL_FALSE, identityMatrix); + glUniform1f(temporalUniformStrideFactor_, 1.0f); + glUniform1f(temporalUniformAlpha_, params.temporalDenoise.alpha); + glUniform1f(temporalUniformNoiseA_, params.temporalDenoise.noiseSlope); + glUniform1f(temporalUniformNoiseB_, params.temporalDenoise.noiseFloor); + glUniform1f(temporalUniformMotionK_, params.temporalDenoise.motionSigma); + glUniform1f(temporalUniformBlack_, static_cast(params.blackLevel.g())); + glUniform1f(temporalUniformHistValid_, temporalHistoryValid_ ? 1.0f : 0.0f); + glUniform2f(temporalUniformStep_, 1.0f / prev.width_, 1.0f / prev.height_); + + if (egl_.attachTextureToFBO(next)) + return -ENODEV; + + glViewport(0, 0, next.width_, next.height_); + glDrawArrays(GL_TRIANGLE_FAN, 0, DEBAYER_OPENGL_COORDS); + + GLenum err = glGetError(); + if (err != GL_NO_ERROR) { + LOG(eGL, Error) << "Temporal pass fail " << err; + return -ENODEV; + } + + temporalIndex_ ^= 1; + temporalHistoryValid_ = true; + temporalActive_ = true; + + glUseProgram(programId_); + + return 0; +} + eGLImage *DebayerEGL::getCachedInputFrameBuffer(FrameBuffer *input, std::optional *inMapped, std::optional *inDmaSyncer) { const SharedFD &fd = input->planes()[0].fd; @@ -585,8 +740,16 @@ int DebayerEGL::debayerGPU(FrameBuffer *input, FrameBuffer *output, const Debaye if (!eglImageOut) return -ENOMEM; + if (temporalSupported_ && temporalPass(*eglImageIn, params)) + return -ENODEV; + egl_.attachTextureToFBO(*eglImageOut); setShaderVariableValues(*eglImageIn, params); + if (temporalActive_) { + eGLImage &filtered = *temporalHistory_[temporalIndex_]; + egl_.activateBindTexture(filtered); + glUniform1i(textureUniformBayerDataIn_, filtered.texture_unit_uniform_id_); + } glViewport(0, 0, width_, height_); glClear(GL_COLOR_BUFFER_BIT); @@ -677,6 +840,15 @@ void DebayerEGL::stop() { eglImageOutCache_.clear(); eglImageInCache_.clear(); + temporalHistory_[0].reset(); + temporalHistory_[1].reset(); + temporalHistoryValid_ = false; + temporalActive_ = false; + + if (temporalProgramId_) { + glDeleteProgram(temporalProgramId_); + temporalProgramId_ = 0; + } if (programId_) glDeleteProgram(programId_); diff --git a/src/libcamera/software_isp/debayer_egl.h b/src/libcamera/software_isp/debayer_egl.h index 30e51a47..c273b0ee 100644 --- a/src/libcamera/software_isp/debayer_egl.h +++ b/src/libcamera/software_isp/debayer_egl.h @@ -64,6 +64,8 @@ public: private: static int getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config); int initBayerShaders(PixelFormat inputFormat, PixelFormat outputFormat); + int initTemporalShaders(const std::vector &shaderEnv); + int temporalPass(eGLImage &eglImageIn, const DebayerParams ¶ms); int getShaderVariableLocations(); void setShaderVariableValues(eGLImage &eGLImageIn, const DebayerParams ¶ms); int debayerGPU(FrameBuffer *input, FrameBuffer *output, const DebayerParams ¶ms, std::optional *mappedInputBuffer, std::optional *inputBufferDmaSyncer); @@ -76,6 +78,27 @@ private: GLuint fragmentShaderId_ = 0; GLuint programId_ = 0; + /* Temporal noise reduction pass */ + bool temporalSupported_ = false; + GLuint temporalProgramId_ = 0; + GLint temporalAttributeVertex_ = -1; + GLint temporalAttributeTexture_ = -1; + GLint temporalUniformProjMatrix_ = -1; + GLint temporalUniformStrideFactor_ = -1; + GLint temporalUniformDataIn_ = -1; + GLint temporalUniformHist_ = -1; + GLint temporalUniformAlpha_ = -1; + GLint temporalUniformNoiseA_ = -1; + GLint temporalUniformNoiseB_ = -1; + GLint temporalUniformMotionK_ = -1; + GLint temporalUniformBlack_ = -1; + GLint temporalUniformHistValid_ = -1; + GLint temporalUniformStep_ = -1; + std::unique_ptr temporalHistory_[2]; + unsigned int temporalIndex_ = 0; + bool temporalHistoryValid_ = false; + bool temporalActive_ = false; + /* Pointer to object representing input texture */ std::deque>> eglImageInCache_; std::deque>> eglImageOutCache_;