From patchwork Sat Oct 24 15:27:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Karnakov X-Patchwork-Id: 28598 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D86A2C324C for ; Sat, 24 Oct 2026 15:28:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7DAC768FC1; Sat, 24 Oct 2026 17:28:10 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="cf9OFpa5"; dkim-atps=neutral Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F107E68ECB for ; Sat, 24 Oct 2026 17:28:08 +0200 (CEST) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-4a2167bd087so10098095e9.3 for ; Sat, 24 Oct 2026 08:28:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1792855688; x=1793460488; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=w2CgL2MPoGMs6qH9iWRtQeUVO+0z/skAVqhcdkw3VSo=; b=cf9OFpa5y27zwrrTcvzFvDLNZdCEuPDYQnYW81g9L+z7ioXuj9/uUvjBtmbSYj1MpB 7wjm/j/hEQCkZtSAoWoosOMwxIqUr3fW2qMNgy+WlR62fQDSx2bEFv34NBXP8fN3kodr Hf2MxcGGmxMYKFO8q763ptzTrZgrmMn8zOIRFX/WpaKo7wyrVuMNLjKL9bmXx0t9vmzM wy1ESaVfrLOa3Jl/vUccZIg4wEp70kJuvoehEtL0OH7H4BBjly30d68nmt3eFJm0lo5I Q9xki4Lj5+TktBW63nnd1xNyzmiaPIgwrrsy5zIPk9tbbVu9dnM0nfxTsveP/IS0yy1a NVSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792855688; x=1793460488; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=w2CgL2MPoGMs6qH9iWRtQeUVO+0z/skAVqhcdkw3VSo=; b=sz4C/hwvAfgd5lymLV/cBkHXC/aBbafgZn4obMcAYQP5jNfg/B+fSUtd61+nhewK3S 83K1y2Z0oP7SrwU6anRoZ2B44UTVa0BkBOOrkg1pevDMoB/DtQrXB9XZk6y5wWXdJio1 dWoZ7VnjjimGVvt83O6SxyAAEj8uGqLtUBtS5/PCE1NiCMBiNnzbe4kxTFCfP//JovL6 Z7EAEzV1OsaqStvzIW3+0NYArkPptVtppb/Dj4Y3lq3oB9sieZPVvF/pONCsyqEUyhgG +pQm2nB8E7yeCgNX4Bc5TVZ+1RSTT9pY73I7QIKHHcguM4EmpRJJGZUxfXRzO7RYjR/P U1Qw== X-Gm-Message-State: AFq9FYJN43GE3jtuCMl6hiSSfpADeNNQc30WOide29wvSyNfzCtN0iWX Tb9EDQDkWUGTfOgqTyYHLIBWWZTLiNNMQgBvgZXACTusPSMWqawcZHecpjmAy3KGBWM= X-Gm-Gg: AYBFou181OTUjTN3KirIR+mO91h4XCLDFf4USURxYvSNAjbccEvqhp5nGoEvIW2zLGz P2WsaZ8MxypqiAL5xx+u6PGOLMztUY2pWDSdNzZXV/lyTYyulgAYK9lz7qbl1Wu6RkPxdW6iLKi 9BcPxstndIp8+3YAsv9nf73HShylj4aMDn6GfgUo8f5my1F/t2YPnFZGNEgvogzpdYvHG/FSu4r DGfhegvV049XrhnlRqddYZK/tfV44ELxAQbreaYeCcvo1k9Z1gpOKlAybfd7WaBLQmC4gE1rE7r tc77xVvbFDv/GkKiC2Vzy8A8df/oXiHQJsE5bXfEOLUdeXNuWDdiaMfmQ98n+unAkEdSKqlKjvO v+K5CQtSPNeyKSzntLrGCEgd/ezzeWXAFOMjR1C1Nn/0QCDXzobpZN3agAiNjLLKclvbtrY9YJe FiKeEnpXvRjy+6g7KKpw2CrkfPpysJ1iCbd+H/QYjwP3w8U4kmDZlMhzfbZKdRYgIIbGv6Qd7YT 7Z2WCWmU7/+Gyu1NND77rPnBkAyev+ORsOeH6/D5H1GrAY9vUIW1uhepi3XUuQB6ql4mgkggl1o QJF6HocMJ/itp6IhQR2Bko4h3HMl2yzQVmYI59WuYzYA1VlfgVOgphBaYhf8RlRgAwbonxWbhTD ukGozrAShmUZ6QArvLROM/cuscVMFEQ== X-Received: by 2002:a05:600c:a03:b0:4a2:267:25b3 with SMTP id 5b1f17b1804b1-4a35accbd93mr36233615e9.35.1792855688101; Sat, 24 Oct 2026 08:28:08 -0700 (PDT) Received: from arch.tail9f8374.ts.net (ip-176-199-103-181.um44.pools.vodafone-ip.de. [176.199.103.181]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a359f3e00asm115238085e9.0.2026.10.24.08.28.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Oct 2026 08:28:07 -0700 (PDT) From: Petr Karnakov To: libcamera-devel@lists.libcamera.org Cc: Jacopo Mondi , Petr Karnakov Subject: [PATCH v2] libcamera: pipeline: ipu3: Fix unsigned wrap in IF height search Date: Sat, 24 Oct 2026 17:27:43 +0200 Message-ID: <20261024152743.89334-1-pkarnakov@gmail.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261019172811.10179-1-pkarnakov@gmail.com> References: <20261019172811.10179-1-pkarnakov@gmail.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" calculatePipeConfig() tries IF heights down to in.height - kIFMaxCropHeight, and calculateBDSHeight() then crops up to another kIFMaxCropHeight lines from each of them. For an IF height below kIFMaxCropHeight, the minimum IF height in calculateBDSHeight() wraps around and becomes the lower bound of std::clamp() with lo > hi. With _GLIBCXX_ASSERTIONS this aborts, e.g. on a Surface Go 3 (ov5693, 1296x972 input) when 640x360 is requested. Don't try IF sizes below kIFMaxCropWidth x kIFMaxCropHeight. This also stops the search loops from wrapping, and running practically forever, for an input exactly kIFMaxCropHeight lines high. Closes: https://gitlab.freedesktop.org/camera/libcamera/-/work_items/366 Suggested-by: Jacopo Mondi Signed-off-by: Petr Karnakov --- Changes in v2: - Bound the IF search in calculatePipeConfig() instead of returning early from calculateBDSHeight() (Jacopo) src/libcamera/pipeline/ipu3/imgu.cpp | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/imgu.cpp b/src/libcamera/pipeline/ipu3/imgu.cpp index bb36c06e8..bf5c33da5 100644 --- a/src/libcamera/pipeline/ipu3/imgu.cpp +++ b/src/libcamera/pipeline/ipu3/imgu.cpp @@ -407,11 +407,18 @@ ImgUDevice::PipeConfig ImgUDevice::calculatePipeConfig(Pipe *pipe) float bdsSF = static_cast(in.width) / gdc.width; float sf = findScaleFactor(bdsSF, bdsScalingFactors, true); - /* Populate the configurations vector by scaling width and height. */ + /* + * Populate the configurations vector by scaling width and height. + * + * calculateBDSHeight() crops up to kIFMaxCropHeight lines from the IF + * height, so don't go below that, or its minimum IF height wraps. + */ unsigned int ifWidth = utils::alignUp(in.width, ImgUDevice::kIFAlignWidth); unsigned int ifHeight = utils::alignUp(in.height, ImgUDevice::kIFAlignHeight); - unsigned int minIfWidth = in.width - ImgUDevice::kIFMaxCropWidth; - unsigned int minIfHeight = in.height - ImgUDevice::kIFMaxCropHeight; + unsigned int minIfWidth = std::max(in.width - ImgUDevice::kIFMaxCropWidth, + ImgUDevice::kIFMaxCropWidth); + unsigned int minIfHeight = std::max(in.height - ImgUDevice::kIFMaxCropHeight, + ImgUDevice::kIFMaxCropHeight); while (ifWidth >= minIfWidth) { while (ifHeight >= minIfHeight) { Size iif{ ifWidth, ifHeight }; @@ -425,8 +432,6 @@ ImgUDevice::PipeConfig ImgUDevice::calculatePipeConfig(Pipe *pipe) /* Repeat search by scaling width first. */ ifWidth = utils::alignUp(in.width, ImgUDevice::kIFAlignWidth); ifHeight = utils::alignUp(in.height, ImgUDevice::kIFAlignHeight); - minIfWidth = in.width - ImgUDevice::kIFMaxCropWidth; - minIfHeight = in.height - ImgUDevice::kIFMaxCropHeight; while (ifHeight >= minIfHeight) { /* * \todo This procedure is probably broken: