From patchwork Mon Mar 24 17:07:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 23003 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2AF6CC3213 for ; Mon, 24 Mar 2025 17:08:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D266968959; Mon, 24 Mar 2025 18:08:20 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ehz8lwSq"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D394B68945 for ; Mon, 24 Mar 2025 18:08:18 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:854:5bc6:b2a7:e242]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 50E78A8F; Mon, 24 Mar 2025 18:06:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1742835992; bh=rjzPA6Us67ZjTb/h97K3bGEFgnhDfi+yLIZKMoqHl98=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ehz8lwSqBj+OUyB8X9cNv07AnpAQtNh7Te3saFBNbFkK27z0u4epv0CSlql/sRbL3 E+hAu54DBE44uXQLCAuyvIKIiWL67SI72xPPTRLwEZOIgs3DuaWa7z//y/OnQvSzAq xvbH2d8aIxN2yWvulEEk/yEW4SdAH8j09ALaxjOg= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH 2/5] libipa: histogram: Fix quantile() calculation for fractional results Date: Mon, 24 Mar 2025 18:07:37 +0100 Message-ID: <20250324170803.103296-3-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250324170803.103296-1-stefan.klug@ideasonboard.com> References: <20250324170803.103296-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The calculation of the frac variable is based solely on integers and therefore results in the fractional part being either 0 or 1. In the original code from RaspberryPi this is mitigated by casting the nominator to a double. This works for most cases, but fails when q is very small because of the quantization introduced by item being an integer. Fix both issues by doing the full calculation in double. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/libipa/histogram.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/ipa/libipa/histogram.cpp b/src/ipa/libipa/histogram.cpp index 10e44b54a0cf..c19a4cbbf3cd 100644 --- a/src/ipa/libipa/histogram.cpp +++ b/src/ipa/libipa/histogram.cpp @@ -130,7 +130,8 @@ double Histogram::quantile(double q, uint32_t first, uint32_t last) const if (cumulative_[first + 1] == cumulative_[first]) frac = 0; else - frac = (item - cumulative_[first]) / (cumulative_[first + 1] - cumulative_[first]); + frac = (q * total() - cumulative_[first]) / + (cumulative_[first + 1] - cumulative_[first]); return first + frac; }