From patchwork Tue Jun 16 06:41:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 26901 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DD4FCBF415 for ; Tue, 16 Jun 2026 06:42:05 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 424E6625BD; Tue, 16 Jun 2026 08:42:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="GHKn97d1"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2B30C623FC for ; Tue, 16 Jun 2026 08:41:51 +0200 (CEST) Received: from [127.0.1.1] (chfd-03-b2-v4wan-176392-cust229.vm15.cable.virginm.net [82.19.20.230]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id BE097166C; Tue, 16 Jun 2026 08:41:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1781592077; bh=IAQrKq63l2GMTBls3JxFXlAC9BHKn37LmkWDTjL3tmM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=GHKn97d1vlj0XHNDgzBG5mDObNIGJZL1Tag/POEAInijIJPpT0qFyZgyzuLWaYxwe ekUpd23oQmrB4WhDLa3CcRhapWB0Kh7zuNu34CrttyN4t1LtiGpf8bd4GhcDQ5T9S+ DjVo0k/7GYArBz+4l+VHWjd5sAIFfu9jPKdohf/s= From: Daniel Scally Date: Tue, 16 Jun 2026 07:41:44 +0100 Subject: [PATCH 10/10] ipa: ipu3: Add Lens Shading Correction algorithm MIME-Version: 1.0 Message-Id: <20260616-ipu3-libipa-rework-v1-10-d4448b54f1d8@ideasonboard.com> References: <20260616-ipu3-libipa-rework-v1-0-d4448b54f1d8@ideasonboard.com> In-Reply-To: <20260616-ipu3-libipa-rework-v1-0-d4448b54f1d8@ideasonboard.com> To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=14890; i=dan.scally@ideasonboard.com; h=from:subject:message-id; bh=IAQrKq63l2GMTBls3JxFXlAC9BHKn37LmkWDTjL3tmM=; b=owEBbQKS/ZANAwAKAchJV3psRXUyAcsmYgBqMPAq5Wqd7pFh3ch6GtXWD4sG8dC1pF828CBiI x28NYAMjMuJAjMEAAEKAB0WIQQqyuwyDnZdb+mxmm/ISVd6bEV1MgUCajDwKgAKCRDISVd6bEV1 MsUuEACnaJktsYMw4MzkCJ8+1ndl0S/ZWEVr8gO9XjiWSYvhu6joZ+DaLlX57eqD6WhMVb479xA f7te8XFKuDQSj50AFOTRMMtbIpgTAQRu/+O/MQOXK5GTAuDvz45iJT+G9cGNIHaGB2Q44tTIWFr nCpKSkY7ThK4UF9+1madRdQkCXnUKEUv/fR4QVMdpjafxiB8B45n2Bw8NHgppxPKaumDEAE1hkE 6g0/xOfzV3BBd7I1WKg0PYXFQaAcvJzOyajboAjHrwBFzH/OfgBUFIAxp8NcbzkWZ3xGlc8Z9lK RJjZKZbnZSVWGaCnBzDtxEXGg9jcjgt01tXnADJmkwPL6e5RU8AiDmNVEA1ksCaJFWNc4vLzpkU LZfMF/P2/cfvsblKOji8DaeZOsbcC9puDtzU2S6IqpI3aW0T6sdPhXUvm9lXJO2/njG9o1kxJ3n vd+kLWv/JTwTBY4S0H4Apg/mQkZqeQVuWiEnd+73eaySxI79NeGwuoJhtNYNLa9w5McpL9O6f5G R0fOW0whyw8VhqPnvBXBUM1qgWLi33UcjPHhlwgkws/6H02rEs6j+HRoWwhfiqLTXHGLbi0p1ZL v6Xy5ExR/Ay6PJPUpXs4BO+QnF8bfKjIp3NX1z9qT7RfWdn1QdhPCMWiLFz5b0R3W/witrXM2Aj D1V8NlJjLyb/Sxw== X-Developer-Key: i=dan.scally@ideasonboard.com; a=openpgp; fpr=EEC699ACA1B7CB5D31330C0BBD501C2A3546CCF6 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a lens shading correction algorithm for the IPU3 IPA, using the recently implemented libipa base algorithm class. Signed-off-by: Daniel Scally --- src/ipa/ipu3/algorithms/lsc.cpp | 290 ++++++++++++++++++++++++++++++++++++ src/ipa/ipu3/algorithms/lsc.h | 58 ++++++++ src/ipa/ipu3/algorithms/meson.build | 1 + src/ipa/ipu3/ipa_context.cpp | 10 ++ src/ipa/ipu3/ipa_context.h | 3 + 5 files changed, 362 insertions(+) diff --git a/src/ipa/ipu3/algorithms/lsc.cpp b/src/ipa/ipu3/algorithms/lsc.cpp new file mode 100644 index 0000000000000000000000000000000000000000..70f8e59b9a9238c44f96d4a2a31bd23a8377f662 --- /dev/null +++ b/src/ipa/ipu3/algorithms/lsc.cpp @@ -0,0 +1,290 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Ideas on Board, Oy + * + * IPU3 Lens Shading Correction algorithm + */ + +#include "lsc.h" + +/** + * \file lsc.h + */ + +namespace libcamera { + +namespace ipa::ipu3::algorithms { + +/** + * \class Lsc + * \brief IPU3 Lens Shading Correction algorithm + */ + +LOG_DEFINE_CATEGORY(IPU3Lsc) + +static constexpr unsigned int kMaxNumHCells = 73; +static constexpr unsigned int kMaxNumVCells = 56; +static constexpr int kColourTemperatureQuantization = 10; + +/** + * \copydoc libcamera::ipa::Algorithm::init + */ +int Lsc::init(IPAContext &context, const ValueNode &tuningData) +{ + /* + * The IPU3 lens shading block expects a table of data that isn't of a + * fixed size, but rather is configurable based on 4 parameters: + * + * block_width_log2: The log2 of the horizontal pixel count per cell + * block_height_log2: The log2 of the vertical pixel count per cell + * width: The number of horizontal cells + * height: The number of vertical cells + * + * The constructed grid should be capable of covering the image, but + * ideally won't extend past the edges of the image. Fixing either set + * of parameters for the algorithm as a whole is likely to result in + * suboptimal situations for some sensors, so let's determine them + * programmatically instead. + * + * What we want is the densest possible grid that ideally doesn't extend + * past the edges of the image at all. The maximum grid size is 73x56, + * which gives us the lower bounds on cell size. Unfortunately we can + * only specify sizes in powers of two, which can have the effect of + * making the grid much more coarse. For example for a 2592x1944 input + * image, 2592 / 73 = 35.5...which means we need to set blockWidthLog2 + * to 6 (I.E. 64) and have just 40.5 (or rather 41) cells horizontally. + */ + sensorWidth_ = context.sensorInfo.activeAreaSize.width; + sensorHeight_ = context.sensorInfo.activeAreaSize.height; + + unsigned int cellWidth = (sensorWidth_ + kMaxNumHCells - 1) / kMaxNumHCells; + unsigned int cellHeight = (sensorHeight_ + kMaxNumVCells - 1) / kMaxNumVCells; + + unsigned int minCellWidth = std::bit_ceil(cellWidth); + unsigned int minCellHeight = std::bit_ceil(cellHeight); + + numHCells_ = (sensorWidth_ + minCellWidth - 1) / minCellWidth; + numVCells_ = (sensorHeight_ + minCellHeight - 1) / minCellHeight; + + blockWidthLog2_ = std::bit_width(minCellWidth) - 1; + blockHeightLog2_ = std::bit_width(minCellHeight) - 1; + + LOG(IPU3Lsc, Debug) << "Calculated Grid configuration: " + << numHCells_ << "x" << numVCells_ << " cells of " + << minCellWidth << "x" << minCellHeight << " pixels"; + + /* + * We need to know if we're running the polynomial algorithm or not as + * things will behave slightly differently. + */ + polynomial_ = tuningData["polynomial"].get(false); + + return lscAlgo_.init(tuningData, context.ctrlMap, { + .keys = { "r", "gr", "gb", "b" }, + .numHCells = numHCells_, + .numVCells = numVCells_, + .sensorSize = context.sensorInfo.activeAreaSize + }); +} + +std::vector Lsc::calculatePositions(unsigned int dimension) +{ + std::vector positions(dimension); + for (double i = 0.0; i < dimension; i++) + positions[i] = i / (dimension - 1); + + return positions; +} + +/** + * \copydoc libcamera::ipa::Algorithm::configure + */ +int Lsc::configure(IPAContext &context, const IPAConfigInfo &configInfo) +{ + cropWidth_ = configInfo.sensorInfo.analogCrop.width; + cropHeight_ = configInfo.sensorInfo.analogCrop.height; + std::vector xPos = calculatePositions(numHCells_); + std::vector yPos = calculatePositions(numVCells_); + + return lscAlgo_.configure(context.activeState.lsc, + configInfo.sensorInfo.analogCrop, xPos, yPos); +} + +/** + * \copydoc libcamera::ipa::Algorithm::queueRequest + */ +void Lsc::queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) +{ + /* + * The base algorithm defines the LensShadingCorrectionEnable control + * with a default value of true, but actually the IPU3 driver defaults + * it to off. If this is the first frame, check for the control, but if + * there isn't one, force it on to fulfil the advertised default. + */ + if (frame == 0) { + const auto &lscEnable = controls.get(controls::LensShadingCorrectionEnable); + if (!lscEnable) { + frameContext.lsc.enabled = true; + frameContext.lsc.update = true; + } + } + + lscAlgo_.queueRequest(context.activeState.lsc, frameContext.lsc, controls); +} + +static unsigned int quantize(unsigned int value, unsigned int step) +{ + return std::lround(value / static_cast(step)) * step; +} + +/** + * \copydoc libcamera::ipa::Algorithm::prepare + */ +void Lsc::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, ipu3_uapi_params *params) +{ + uint32_t ct = frameContext.awb.temperatureK; + unsigned int quantizedCt = quantize(ct, kColourTemperatureQuantization); + + if (!frameContext.lsc.update) { + if (!frameContext.lsc.enabled) + return; + + /* + * Add a threshold so that oscillations around a quantization + * step don't lead to constant changes. + */ + if (utils::abs_diff(ct, lastAppliedCt_) < kColourTemperatureQuantization / 2) + return; + + if (quantizedCt == lastAppliedQuantizedCt_) + return; + } + + /* + * This flag tells the kernel driver that it should read the LSC params + * passed from userspace instead of using its cached copy. + */ + params->use.acc_shd = 1; + + /* + * Pass the enabled flag. If we're not enabled, we can then just bail + * out. + */ + ipu3_uapi_shd_config_static *config = ¶ms->acc_param.shd.shd; + config->general.shd_enable = frameContext.lsc.enabled; + + if (!frameContext.lsc.enabled) + return; + + config->grid.width = numHCells_; + config->grid.height = numVCells_; + config->grid.block_width_log2 = blockWidthLog2_; + config->grid.block_height_log2 = blockHeightLog2_; + config->grid.grid_height_per_slice = IPU3_UAPI_SHD_MAX_CELLS_PER_SET / numHCells_; + + /* + * The IPU3's documentation describes the x_start and y_start members + * as follows: + * + * "[X/Y] value of top left corner of sensor relative to ROI + * s13, [-4096, 0], default 0, only negative values." + * + * I interpret that as allowing us to configure the cropped rectangle + * relative to the full grid. That's useful if we're running the tabular + * algorithm, which would otherwise apply the full grid inappropriately. + * If we're running the polynomial one though the calculated grid is + * probably more appropriate than a coarse application of the full grid + * so let's tell the hardware not to bother correcting in that case. + */ + if (polynomial_) { + config->grid.x_start = 0; + config->grid.y_start = 0; + } else { + config->grid.x_start = (cropWidth_ - sensorWidth_) / 2; + config->grid.y_start = (cropHeight_ - sensorHeight_) / 2; + } + + /* No idea what this is, but the docs say it should be set as so */ + config->general.init_set_vrt_offst_ul = + config->grid.y_start >> (config->grid.block_height_log2 % + config->grid.grid_height_per_slice); + + /* + * Values in the LUT cease taking effect at 4096, and a value of 0.0 is + * "no correction" rather than black. The gain factor is described by + * the documentation like so: + * + * "Shift calculated anti shading value. Precision u2. 0x0 - gain factor + * [1, 5], means no shift interpolated value. 0x1 - gain factor [1, 9], + * means shift interpolated by 1. 0x2 - gain factor [1, 17], means shift + * interpolated by 2." + * + * The simplest interpretation for those pieces of information is I + * think that the LUT stores 12-bit Q numbers who's represented values + * depend on the gain_factor setting like so: + * + * 0: UQ<2, 10> representing values in range [0, 4) + * 1: UQ<3, 9> representing values in range [0, 8) + * 2: UQ<4, 8> representing values in range [0, 16) + * + * And that a base gain of 1.0 is added to those configured values. As a + * gain of more than 5.0 is fairly unlikely, let's fix gain_factor to 0 + * for now and revisit if needed. + */ + config->general.gain_factor = 0; + + /* + * Disable the black level settings here - we do that through another + * parameters block. + */ + config->black_level.bl_r = 0; + config->black_level.bl_gr = 0; + config->black_level.bl_gb = 0; + config->black_level.bl_b = 0; + + ipu3_uapi_shd_lut *lut = ¶ms->acc_param.shd.shd_lut; + + const auto &set = lscAlgo_.interpolateComponents(quantizedCt); + + unsigned int totalCells = numHCells_ * numVCells_; + unsigned int cellsPerSet = numHCells_ * config->grid.grid_height_per_slice; + unsigned int numSets = (numHCells_ + config->grid.grid_height_per_slice - 1) / + config->grid.grid_height_per_slice; + unsigned int i = 0; + + for (unsigned int s = 0; s < numSets; s++) { + for (unsigned int c = 0; c < cellsPerSet && i < totalCells; c++, i++) { + lut->sets[s].r_and_gr[c].r = set.at("r")[i]; + lut->sets[s].r_and_gr[c].gr = set.at("gr")[i]; + lut->sets[s].gb_and_b[c].gb = set.at("gb")[i]; + lut->sets[s].gb_and_b[c].b = set.at("b")[i]; + } + } + + lastAppliedCt_ = ct; + lastAppliedQuantizedCt_ = quantizedCt; + LOG(IPU3Lsc, Debug) + << "ct is " << ct << ", quantized to " + << quantizedCt; +} + +/** + * \copydoc libcamera::ipa::Algorithm::process + */ +void Lsc::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + [[maybe_unused]] const ipu3_uapi_stats_3a *stats, + ControlList &metadata) +{ + lscAlgo_.process(frameContext.lsc, metadata); +} + +REGISTER_IPA_ALGORITHM(Lsc, "Lsc") + +} /* ipa::ipu3::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/algorithms/lsc.h b/src/ipa/ipu3/algorithms/lsc.h new file mode 100644 index 0000000000000000000000000000000000000000..f86505029f005c9832c3efa213b86848222ddf62 --- /dev/null +++ b/src/ipa/ipu3/algorithms/lsc.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Ideas on Board, Oy + * + * IPU3 Lens Shading Correction algorithm + */ + +#pragma once + +#include "libipa/fixedpoint.h" +#include "libipa/lsc.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::ipu3::algorithms { + +class Lsc : public Algorithm +{ +public: + int init(IPAContext &context, const ValueNode &tuningData) override; + void queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) override; + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + ipu3_uapi_params *params) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ipu3_uapi_stats_3a *stats, + ControlList &metadata) override; + +private: + std::vector calculatePositions(unsigned int dimension); + + unsigned int numHCells_; + unsigned int numVCells_; + unsigned int blockWidthLog2_; + unsigned int blockHeightLog2_; + + unsigned int lastAppliedCt_; + unsigned int lastAppliedQuantizedCt_; + + unsigned int sensorWidth_; + unsigned int sensorHeight_; + unsigned int cropWidth_; + unsigned int cropHeight_; + + bool polynomial_; + + LscAlgorithm> lscAlgo_; +}; + +} /* ipa::ipu3::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build index 3dafd2fda9897942cf87d9640665c4fcff383859..70177f50415259e0c2bd207205c385dde07d6624 100644 --- a/src/ipa/ipu3/algorithms/meson.build +++ b/src/ipa/ipu3/algorithms/meson.build @@ -6,5 +6,6 @@ ipu3_ipa_algorithms = files([ 'awb.cpp', 'blc.cpp', 'ccm.cpp', + 'lsc.cpp', 'tone_mapping.cpp', ]) diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 7fcfd5e0e4ade92521cc2914dd07113235af8e45..071a7415b61f574d1b525d04d4434ac48fccb23e 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -127,6 +127,11 @@ namespace libcamera::ipa::ipu3 { * \brief Active gamma correction parameters for the IPA */ +/** + * \var IPAActiveState::lsc + * \brief Active lens shading correction parameters for the IPA + */ + /** * \var IPASessionConfiguration::sensor * \brief Sensor-specific configuration of the IPA @@ -191,4 +196,9 @@ namespace libcamera::ipa::ipu3 { * \brief Per-frame gamma correction parameters for the IPA */ +/** + * \var IPAFrameContext::lsc + * \brief Per-frame lens shading correction parameters for the IPA + */ + } /* namespace libcamera::ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index f157f223cbb3119f108d768b14fca514ac5661ca..b517639b1e5878138f653918cd2dc71b49d97264 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -21,6 +21,7 @@ #include #include #include +#include namespace libcamera { @@ -70,6 +71,7 @@ struct IPAActiveState { ipa::awb::ActiveState awb; ipa::ccm::ActiveState ccm; ipa::gamma::ActiveState gamma; + ipa::lsc::ActiveState lsc; }; struct IPAFrameContext : public FrameContext { @@ -81,6 +83,7 @@ struct IPAFrameContext : public FrameContext { ipa::awb::FrameContext awb; ipa::ccm::FrameContext ccm; ipa::gamma::FrameContext gamma; + ipa::lsc::FrameContext lsc; }; struct IPAContext {