From patchwork Fri Feb 13 16:57:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 26145 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4A7C7C32EA for ; Fri, 13 Feb 2026 16:58:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 717D262201; Fri, 13 Feb 2026 17:58:11 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="sC/0dgme"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CD8BE621D2 for ; Fri, 13 Feb 2026 17:58:04 +0100 (CET) Received: from ping.linuxembedded.co.uk (cpc89244-aztw30-2-0-cust6594.18-1.cable.virginm.net [86.31.185.195]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B5EA5180F; Fri, 13 Feb 2026 17:57:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1771001835; bh=C5ujSuBcZfK72bFwDC2d74gukJ8/4Cq0fWaMVmqqLkY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=sC/0dgmeDhGMm8kDslql2yZkYWkNXXutpbdSLBpVscv480w46fhsIaKcZvh71tAeV NmfX45SG5+FbWvMyCGrk1i0AxoellZuPRjO2zRHZ6RDyIERxNzNmqIzNA96gV65C/n Y9kYwO7w2+0St3X01Qx5UN9HKZWWb9l/A/gtxk0E= From: Kieran Bingham Date: Fri, 13 Feb 2026 16:57:42 +0000 Subject: [PATCH v7 03/15] ipa: libipa: fixedpoint: Fix unsigned usage MIME-Version: 1.0 Message-Id: <20260213-kbingham-quantizers-v7-3-1626b9aaabf1@ideasonboard.com> References: <20260213-kbingham-quantizers-v7-0-1626b9aaabf1@ideasonboard.com> In-Reply-To: <20260213-kbingham-quantizers-v7-0-1626b9aaabf1@ideasonboard.com> To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Isaac Scott X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1771001883; l=2418; i=kieran.bingham@ideasonboard.com; s=20260207; h=from:subject:message-id; bh=C5ujSuBcZfK72bFwDC2d74gukJ8/4Cq0fWaMVmqqLkY=; b=ff+rXQ4cVwnOLIV8LFtTm0Qgjg1tG9O03K6vBJf/0shwwScmt3+fdjfzzP1SYfaBLknKc7VrX Oeupp2NAp/ZC3r7khiuTUjwEHc1o6einpXSzHIPjz4PO1lLDyy8xze6 X-Developer-Key: i=kieran.bingham@ideasonboard.com; a=ed25519; pk=FVXKN7YuwHc6UtbRUeTMAmranfsQomA+vnilfglWdaY= X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The fixedToFloatingPoint does not support unsigned Q types, and incorrectly sign-extends all values which have the top most bit set in the quantized values. Fix this by ensuring that only signed types perform sign extension, and simplify the calculation for unsigned types. Convert the storage of the test cases to signed types to correctly represent their intended purpose, to prevent test failures. Reviewed-by: Isaac Scott Signed-off-by: Kieran Bingham --- v6: - Use T{1} << F instead of 1 << F v7: - Fix checkstyle issues --- src/ipa/libipa/fixedpoint.h | 3 +++ test/ipa/libipa/fixedpoint.cpp | 6 +++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/src/ipa/libipa/fixedpoint.h b/src/ipa/libipa/fixedpoint.h index 709cf50f0fcd..aeb9bce3269b 100644 --- a/src/ipa/libipa/fixedpoint.h +++ b/src/ipa/libipa/fixedpoint.h @@ -49,6 +49,9 @@ constexpr R fixedToFloatingPoint(T number) static_assert(sizeof(int) >= sizeof(T)); static_assert(I + F <= sizeof(T) * 8); + if constexpr (std::is_unsigned_v) + return static_cast(number) / static_cast(T{ 1 } << F); + /* * Recreate the upper bits in case of a negative number by shifting the sign * bit from the fixed point to the first bit of the unsigned and then right shifting diff --git a/test/ipa/libipa/fixedpoint.cpp b/test/ipa/libipa/fixedpoint.cpp index 99eb662ddf4e..4b017e86a74f 100644 --- a/test/ipa/libipa/fixedpoint.cpp +++ b/test/ipa/libipa/fixedpoint.cpp @@ -68,7 +68,7 @@ protected: * The second 7.992 test is to test that unused bits don't * affect the result. */ - std::map testCases = { + std::map testCases = { { 7.992, 0x3ff }, { 0.2, 0x01a }, { -0.2, 0x7e6 }, @@ -81,14 +81,14 @@ protected: int ret; for (const auto &testCase : testCases) { - ret = testSingleFixedPoint<4, 7, uint16_t>(testCase.first, + ret = testSingleFixedPoint<4, 7, int16_t>(testCase.first, testCase.second); if (ret != TestPass) return ret; } /* Special case with a superfluous one in the unused bits */ - ret = testFixedToFloat<4, 7, uint16_t, double>(0xbff, 7.992); + ret = testFixedToFloat<4, 7, int16_t, double>(0xbff, 7.992); if (ret != TestPass) return ret;