From patchwork Thu Sep 29 19:53:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 17469 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 02D7DC327E for ; Thu, 29 Sep 2022 19:53:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 920F462397; Thu, 29 Sep 2022 21:53:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1664481204; bh=0Nx9nM4oNxTHghZxzwhc8DUbEgkHMrD7kCT65jlrVyk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=rh/DDtPkp+9uGMv0VMVm90xGKr1Uj/ZhRe+uyfq4KQMA1YQWraY3V1bCEbD1p587d LouLvxAcoD9rtbBeM681h+DK8fwPPCcEknvadYqRPvDM18O4Cg++YcKKcJCQLyFLtB /td8bGEo8vtKRoTqrWIBf4F1KEW4UKfvn1X58DtJmczDNL8hFV95UbM9n1PmM9iHbc FHdan23Hj2R/luYvTQOIPFXooqvB7xGWvqFimLlSCJFefqckZtmCSOlpwMiX970h0c rIXSFRYSOh7TXgle7RsWmwJcMQoobgjoafi9TSFDToCbUGao108tuwPBtbTwS/ZVaT mzhLuMliCWK2w== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 78CFB62384 for ; Thu, 29 Sep 2022 21:53:22 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="UP+r1Iqt"; dkim-atps=neutral Received: from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 054EE47C for ; Thu, 29 Sep 2022 21:53:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1664481202; bh=0Nx9nM4oNxTHghZxzwhc8DUbEgkHMrD7kCT65jlrVyk=; h=From:To:Subject:Date:In-Reply-To:References:From; b=UP+r1IqtLz8fhROYFPF7+GOu43OHmU51+oarVla+8EhFyASE6P3JULVlWUlKvU5/I uY+q5QNY+eIa4H/+WY7RlS2obPChpciW6sVgwpI+oAdDIRX8GeBJ8GdBwxi3tALs0m c/YKrm1duHcbQLIiNwywBpXERmpNTvEjYU11vwuk= To: libcamera-devel@lists.libcamera.org Date: Thu, 29 Sep 2022 22:53:17 +0300 Message-Id: <20220929195318.13577-2-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220929195318.13577-1-laurent.pinchart@ideasonboard.com> References: <20220929195318.13577-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/2] utils: rkisp1: gen-csc-table: Fix inverted CSC calculation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Laurent Pinchart via libcamera-devel From: Laurent Pinchart Reply-To: Laurent Pinchart Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Coefficient scaling in the inverted CSC table is wrong, as the luma-related coefficients are not in the first row, but in the first column. The simplest way to fix it is to remove the inversion handling from scale_coeff() and invert the matrix after scaling the coefficients. The multiplication by the fixed point precision must still happen after inversion, and must thus be moved out of scale_coeff(). Fixes: b5c4b6e0339a ("utils: rkisp1: gen-csc-table: Add support for inverting the CSC") Signed-off-by: Laurent Pinchart --- utils/rkisp1/gen-csc-table.py | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/utils/rkisp1/gen-csc-table.py b/utils/rkisp1/gen-csc-table.py index 83aea357966a..934ace377e17 100755 --- a/utils/rkisp1/gen-csc-table.py +++ b/utils/rkisp1/gen-csc-table.py @@ -64,9 +64,8 @@ class Quantization(enum.Enum): LIMITED = 1 -def scale_coeff(coeff, quantization, luma, precision, invert): - """Scale a coefficient to the output range dictated by the quantization and - the precision. +def scale_coeff(coeff, quantization, luma): + """Scale a coefficient to the output range dictated by the quantization. Parameters ---------- @@ -76,9 +75,6 @@ def scale_coeff(coeff, quantization, luma, precision, invert): The quantization, either FULL or LIMITED luma : bool True if the coefficient corresponds to a luma value, False otherwise - precision : int - The desired precision for the scaled coefficient as a number of - fractional bits """ # Assume the input range is 8 bits. The output range is set by the @@ -92,10 +88,7 @@ def scale_coeff(coeff, quantization, luma, precision, invert): else: out_range = 240 - 16 - if invert: - in_range, out_range = out_range, in_range - - return coeff * out_range / in_range * (1 << precision) + return coeff * out_range / in_range def round_array(values): @@ -172,20 +165,21 @@ def main(argv): encoding = encodings[args.encoding] quantization = Quantization[args.quantization.upper()] - if args.invert: - encoding = np.linalg.inv(encoding) - # Scale and round the encoding coefficients based on the precision and # quantization range. luma = True scaled_coeffs = [] for line in encoding: - line = [scale_coeff(coeff, quantization, luma, precision.fractional, args.invert) for coeff in line] + line = [scale_coeff(coeff, quantization, luma) for coeff in line] scaled_coeffs.append(line) luma = False + if args.invert: + scaled_coeffs = np.linalg.inv(scaled_coeffs) + rounded_coeffs = [] for line in scaled_coeffs: + line = [coeff * (1 << precision.fractional) for coeff in line] line = round_array(line) # Convert coefficients to the number of bits selected by the precision.