From patchwork Fri Oct 24 14:15:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 24803 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6141BC3259 for ; Fri, 24 Oct 2025 14:40:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8B6CB60977; Fri, 24 Oct 2025 16:40:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="KHpi3tM2"; dkim-atps=neutral Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D25B460974 for ; Fri, 24 Oct 2025 16:40:56 +0200 (CEST) Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-3ee130237a8so1482677f8f.0 for ; Fri, 24 Oct 2025 07:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1761316856; x=1761921656; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=nDFe4dQU0l4w9wPRMvbWU+zyzN47eglyj+Xh9oe8AEM=; b=KHpi3tM2YxQWHDqNfAO2RkWql68MJ6naeTaOMtqX/B9oDmklC24VHGgJn7ezBwYlyc q+++KWzvRyhSIKv4R1IRnBk04UklxDfXWUg28BrQHO4+g7OQSh3KK7tnwK6E+VcivE8x m2R0TXSGkoxpLFZhaxONVUKOWcc5tL9t5J0OBKyMRkf9HmJ9WnMLhvrnC10jcJL9UuYk 5yA76UdPsVgfJy3j13CaTFsdpmCgFK/RRvWsI5fKB/XGhyCBsl1VcxaZfZVodJfOPPvS q8ptaOZAQDQox8nkKY+STHVps9HfjKN+mK9KBNxVdAbezrJy7KG3BVMAxFwe/xE4Va6g 9aeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761316856; x=1761921656; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nDFe4dQU0l4w9wPRMvbWU+zyzN47eglyj+Xh9oe8AEM=; b=vrly71iM2nVGfMB7wJmWVw/XbCwZOUPxPyOag7d7HriWlJMl+phBtcwezYHwFxISrI h4/jg9sVn8uglJW+TEgwpD+hrnr/tz1YU0l+HfAPYKzjuhk7WYBqNxPfSuYwlqi6U5Hg kgW0HHzuKlNQHQeR3/e4MpmrqIDRuabLiu0qdMKzRyvBMX2hCPJKxYxrAWy7nb0/XDKo 6xI3kwKNMPjJrvveSHhjkeZ5Z8dQUXHiW0SnTkZGtFUfyrI/SpvMnUq3DUlZQvD5qLDS G536GoQdBCxOJO5F9qvLuoCNLRPvZY/DjA+g0YomKKeF+qoG5DZNpuwdwNnyKe+jWhrY y3LQ== X-Gm-Message-State: AOJu0YzDb10TjVlClntoBV3QuXli86GqthX2eUkQVZQ3tye3202TRtOj HGiMMI/izUnnTMKU2VCGX/4PBfM6n5bitbEHnrBTEUrzKN1gaLFmufKCPs0UvYdeGRQqJ4hDLgp G5iHNk3o= X-Gm-Gg: ASbGncuWrYvUAy5zCQOw2gLTNzzlIPFCi9CVUyE98hsazDjx21yxVRc09ELOboFbV55 tHg2LppmJ26HWUpuV0RlPf1knpsEI+ZjYwi7CTULBl3S4RhPq1+X9TJXsKGaAfiwBUvgyhFETTk PXj8aguUP1+S0jQvP2hK3N22+QgbsvrYr/xvXVtBgC+XIc+RpIHJYq7HQ6UT2h/DCEhVZ2wRNV2 fDOhDMuHpBHEt43q5sQO3cfd3V4Wd6EERMZcks7VdCrMgH3wFZq+gvOcOzeYyJyXAXKNY90M2/f jjeUX2mJVnjbBjqHXMgYDSOUWSBi3lk7NSsVZGQnxYasdQLxkShUgazUuGIMh4R0UZdTGjOEM6T bUVgnsXv0UqAvGaMjAvyypvyIj+PrOkXkyGs2WwlnDeU78qmkwsPXxaym6lDcH3bfHASA8aH4g7 moKbYCD8RYnyRbd5EueShtIdXLSbYIU9HUpFykfmKOKipp/uTbvU8/C20J2T97cn2kPN6KzF/9g F6B/rUMXv7g8jNOWoLNsqDd9Q== X-Google-Smtp-Source: AGHT+IGKVd0noM1r1viNaSimgU0JZfUxFti9aRPiq+4YTvurwQm842ZHdWkQtz4Wu3JfaVOL9HWzZw== X-Received: by 2002:a5d:5f55:0:b0:427:5ed:296d with SMTP id ffacd0b85a97d-42990712615mr2208660f8f.28.1761316855907; Fri, 24 Oct 2025 07:40:55 -0700 (PDT) Received: from localhost.localdomain ([2a06:61c0:f337:0:8aba:2a85:b765:a23e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-429898ecabdsm9835993f8f.47.2025.10.24.07.40.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Oct 2025 07:40:54 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH 0/4] Raspberry Pi AWB using neural networks Date: Fri, 24 Oct 2025 15:15:59 +0100 Message-ID: <20251024144049.3311-1-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi everyone Here's our first go at doing AWB using neural networks. For both the PiSP and VC4 we take the image statistics and feed them to a neural network which tells us what it thinks the colour temperature should be. The first commit splits our existing Awb class into an Awb base class, and an AwbBayes derived class, so that the subsequent AwbNN class (2nd commit) can share code. Finally we update the tuning files for the new AWB and supply a TFLite model for each platform, though by default it remains off for now. Just switch the "disable" tag between the two AWB configurations to enable the new algorithm. We'll shortly be making public our image capture/annotation/training code which will include the full source for the model definitions. It will allow users to train their own, either from scratch, or incrementally using our datasets and models as a starting point. This is the work of our intern who was with us over the summer. Thanks! David Peter Bailey (4): ipa: rpi: controller: awb: Separate Bayesian Awb into AwbBayes ipa: rpi: controller: awb: Add Neural Network Awb ipa: rpi: pisp: vc4: Update tuning files for new awb and add model ipa: rpi: controller: Ignore algorithms starting with disable src/ipa/rpi/controller/controller.cpp | 6 + src/ipa/rpi/controller/meson.build | 10 + src/ipa/rpi/controller/rpi/awb.cpp | 409 ++--------------- src/ipa/rpi/controller/rpi/awb.h | 99 ++--- src/ipa/rpi/controller/rpi/awb_bayes.cpp | 444 +++++++++++++++++++ src/ipa/rpi/controller/rpi/awb_nn.cpp | 442 ++++++++++++++++++ src/ipa/rpi/pisp/data/awb_model.tflite | Bin 0 -> 47624 bytes src/ipa/rpi/pisp/data/imx219.json | 65 ++- src/ipa/rpi/pisp/data/imx296.json | 64 ++- src/ipa/rpi/pisp/data/imx296_16mm.json | 64 ++- src/ipa/rpi/pisp/data/imx296_6mm.json | 64 ++- src/ipa/rpi/pisp/data/imx477.json | 63 +++ src/ipa/rpi/pisp/data/imx477_16mm.json | 65 ++- src/ipa/rpi/pisp/data/imx477_6mm.json | 65 ++- src/ipa/rpi/pisp/data/imx477_scientific.json | 79 +++- src/ipa/rpi/pisp/data/imx500.json | 67 +++ src/ipa/rpi/pisp/data/imx708.json | 64 ++- src/ipa/rpi/pisp/data/imx708_wide.json | 62 +++ src/ipa/rpi/pisp/data/meson.build | 7 + src/ipa/rpi/pisp/data/ov5647.json | 63 +++ src/ipa/rpi/vc4/data/awb_model.tflite | Bin 0 -> 42976 bytes src/ipa/rpi/vc4/data/imx219.json | 64 +++ src/ipa/rpi/vc4/data/imx296.json | 64 +++ src/ipa/rpi/vc4/data/imx477.json | 69 +++ src/ipa/rpi/vc4/data/imx500.json | 67 +++ src/ipa/rpi/vc4/data/imx708.json | 72 +++ src/ipa/rpi/vc4/data/imx708_wide.json | 62 +++ src/ipa/rpi/vc4/data/meson.build | 8 + src/ipa/rpi/vc4/data/ov5647.json | 64 +++ 29 files changed, 2244 insertions(+), 428 deletions(-) create mode 100644 src/ipa/rpi/controller/rpi/awb_bayes.cpp create mode 100644 src/ipa/rpi/controller/rpi/awb_nn.cpp create mode 100644 src/ipa/rpi/pisp/data/awb_model.tflite create mode 100644 src/ipa/rpi/vc4/data/awb_model.tflite