From patchwork Fri Jun 26 13:05:59 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 27072 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D2041C3318 for ; Fri, 26 Jun 2026 13:06:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AD17965F19; Fri, 26 Jun 2026 15:06:23 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Wd749cbz"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 32D2565EE1 for ; Fri, 26 Jun 2026 15:06:08 +0200 (CEST) Received: from [127.0.1.1] (chfd-03-b2-v4wan-176392-cust229.vm15.cable.virginm.net [82.19.20.230]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3FD561E7; Fri, 26 Jun 2026 15:05:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1782479127; bh=FIBgslO/k1aow8xgwgmZBi8HwhIqX3xLhKNM4OUPFgY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Wd749cbzBOuQWBojgwuu2NovyKIT2IwMtS6FyRI6qrKDFeCDH6/V/US0K3sYVI2ib nLP4+GZJPA6dBPM8AzWcvjuBrsbC0n/ribQoHlBbB1SMZOkEQ+IVYT13P6Az4xpAl+ 0yMpuusagRFSmYXYBW2MO5KWl59eF9hv1mbz/AVc= From: Daniel Scally Date: Fri, 26 Jun 2026 14:05:59 +0100 Subject: [PATCH v2 12/12] ipa: ipu3: Add Lens Shading Correction algorithm MIME-Version: 1.0 Message-Id: <20260626-ipu3-libipa-rework-v2-12-41546e23de3e@ideasonboard.com> References: <20260626-ipu3-libipa-rework-v2-0-41546e23de3e@ideasonboard.com> In-Reply-To: <20260626-ipu3-libipa-rework-v2-0-41546e23de3e@ideasonboard.com> To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Jacopo Mondi X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=15230; i=dan.scally@ideasonboard.com; h=from:subject:message-id; bh=FIBgslO/k1aow8xgwgmZBi8HwhIqX3xLhKNM4OUPFgY=; b=owEBbQKS/ZANAwAKAchJV3psRXUyAcsmYgBqPnk760DlwdVPMyl90d5Px7YI6hV2AJOk55IJl 5YZrN56TeiJAjMEAAEKAB0WIQQqyuwyDnZdb+mxmm/ISVd6bEV1MgUCaj55OwAKCRDISVd6bEV1 MlDlEACmNEom8sWYOyDUfkTwV/UiMxEpIJ95xQUhNNRS8hX+tpr7BViRCf3ZlbucC3LMN5Rn4E+ Dccld6KAjn/oWuLOCnWyIiASm5u3OGFDxFZkF6dpFxd9zFbCyoAIG+WBZSUvgQ6OFsF1rJgkbfJ lcxFGeyKASnYDjB/YBVpYuiVsE+V275l5TH8cIBR5s8HjHnTmL+gQZBhaVT4o5uuYQqIi1a/vr/ 5G95vMHhF2y85ExOZQ12B3CLfN85FA5QkrkskCAvV91yo6osUZkx8AmmV3fM/Og8HcBdla+qnki XfaOCymIJqZOtvEuXkroCdpHMap6wQ5GJ7Irh9U0XPdjOhanIrvmJ/U94k7U56pPEtCbG1OzDEE 7B0NmdMtoVaDY7qs3C9Vhsm6vCOSlm0280XKCCkiZLFGBWaJE4c5gk1Rw73LRKdPNNZuLMjhtEA QAqXBSJ434yilHyh56eQ8pXogG/Z1ZnNq59Tj6L07q8BqQ9Ipg/yvPDNwnDiSasq8/uAq8FVv38 uWUmZK6q5wg3YnR0djY+0x7TDEd2VhNPsZVJ+4C8NDfGkTakC2S3jzz998SHMNqP5saueSfg0o2 UkZQko4i5CQ7lc458lWEnhoAvJLnZ9fYYCMF5bg9SfuXod1TXuv1u+qR37ZAPQtU9OkJ5ydKubU HJjkuwrtIqg6yig== X-Developer-Key: i=dan.scally@ideasonboard.com; a=openpgp; fpr=EEC699ACA1B7CB5D31330C0BBD501C2A3546CCF6 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a lens shading correction algorithm for the IPU3 IPA, using the recently implemented libipa base algorithm class. Reviewed-by: Jacopo Mondi Signed-off-by: Daniel Scally --- Changes in v2: - Fixed includes - Updated some documentation comments - Check tuning file for "type" key, not "polynomial" --- src/ipa/ipu3/algorithms/lsc.cpp | 296 ++++++++++++++++++++++++++++++++++++ src/ipa/ipu3/algorithms/lsc.h | 60 ++++++++ src/ipa/ipu3/algorithms/meson.build | 1 + src/ipa/ipu3/ipa_context.cpp | 10 ++ src/ipa/ipu3/ipa_context.h | 3 + 5 files changed, 370 insertions(+) diff --git a/src/ipa/ipu3/algorithms/lsc.cpp b/src/ipa/ipu3/algorithms/lsc.cpp new file mode 100644 index 0000000000000000000000000000000000000000..ab9775e6a64ed3d32e7ae3dd90f55f4128da40be --- /dev/null +++ b/src/ipa/ipu3/algorithms/lsc.cpp @@ -0,0 +1,296 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Ideas on Board, Oy + * + * IPU3 Lens Shading Correction algorithm + */ + +#include "lsc.h" + +#include +#include + +#include + +/** + * \file lsc.h + * \brief IPU3 Lens Shading Correction + */ + +namespace libcamera { + +namespace ipa::ipu3::algorithms { + +/** + * \class Lsc + * \brief IPU3 Lens Shading Correction algorithm + */ + +LOG_DEFINE_CATEGORY(IPU3Lsc) + +static constexpr unsigned int kMaxNumHCells = 73; +static constexpr unsigned int kMaxNumVCells = 56; +static constexpr int kColourTemperatureQuantization = 10; + +/** + * \copydoc libcamera::ipa::Algorithm::init + */ +int Lsc::init(IPAContext &context, const ValueNode &tuningData) +{ + /* + * The IPU3 lens shading block expects a table of data that isn't of a + * fixed size, but rather is configurable based on 4 parameters: + * + * block_width_log2: The log2 of the horizontal pixel count per cell + * block_height_log2: The log2 of the vertical pixel count per cell + * width: The number of horizontal cells + * height: The number of vertical cells + * + * The constructed grid should be capable of covering the image, but + * ideally won't extend past the edges of the image. Fixing either set + * of parameters for the algorithm as a whole is likely to result in + * suboptimal situations for some sensors, so let's determine them + * programmatically instead. + * + * What we want is the densest possible grid that ideally doesn't extend + * past the edges of the image at all. The maximum grid size is 73x56, + * which gives us the lower bounds on cell size. Unfortunately we can + * only specify sizes in powers of two, which can have the effect of + * making the grid much more coarse. For example for a 2592x1944 input + * image, 2592 / 73 = 35.5...which means we need to set blockWidthLog2 + * to 6 (I.E. 64) and have just 40.5 (or rather 41) cells horizontally. + */ + sensorWidth_ = context.sensorInfo.activeAreaSize.width; + sensorHeight_ = context.sensorInfo.activeAreaSize.height; + + unsigned int cellWidth = (sensorWidth_ + kMaxNumHCells - 1) / kMaxNumHCells; + unsigned int cellHeight = (sensorHeight_ + kMaxNumVCells - 1) / kMaxNumVCells; + + unsigned int minCellWidth = std::bit_ceil(cellWidth); + unsigned int minCellHeight = std::bit_ceil(cellHeight); + + numHCells_ = (sensorWidth_ + minCellWidth - 1) / minCellWidth; + numVCells_ = (sensorHeight_ + minCellHeight - 1) / minCellHeight; + + blockWidthLog2_ = std::bit_width(minCellWidth) - 1; + blockHeightLog2_ = std::bit_width(minCellHeight) - 1; + + LOG(IPU3Lsc, Debug) << "Calculated Grid configuration: " + << numHCells_ << "x" << numVCells_ << " cells of " + << minCellWidth << "x" << minCellHeight << " pixels"; + + /* + * We need to know if we're running the polynomial algorithm or not as + * things will behave slightly differently. + */ + polynomial_ = tuningData["type"].get(false); + + return lscAlgo_.init(tuningData, context.ctrlMap, { + .keys = { "r", "gr", "gb", "b" }, + .numHCells = numHCells_, + .numVCells = numVCells_, + .sensorSize = context.sensorInfo.activeAreaSize + }); +} + +std::vector Lsc::calculatePositions(unsigned int dimension) +{ + std::vector positions(dimension); + for (double i = 0.0; i < dimension; i++) + positions[i] = i / (dimension - 1); + + return positions; +} + +/** + * \copydoc libcamera::ipa::Algorithm::configure + */ +int Lsc::configure(IPAContext &context, const IPAConfigInfo &configInfo) +{ + cropWidth_ = configInfo.sensorInfo.analogCrop.width; + cropHeight_ = configInfo.sensorInfo.analogCrop.height; + std::vector xPos = calculatePositions(numHCells_); + std::vector yPos = calculatePositions(numVCells_); + + return lscAlgo_.configure(context.activeState.lsc, + configInfo.sensorInfo.analogCrop, xPos, yPos); +} + +/** + * \copydoc libcamera::ipa::Algorithm::queueRequest + */ +void Lsc::queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) +{ + /* + * The base algorithm defines the LensShadingCorrectionEnable control + * with a default value of true, but actually the IPU3 driver defaults + * it to off. If this is the first frame, check for the control, but if + * there isn't one, force it on to fulfil the advertised default. + */ + if (frame == 0) { + const auto &lscEnable = controls.get(controls::LensShadingCorrectionEnable); + if (!lscEnable) { + frameContext.lsc.enabled = true; + frameContext.lsc.update = true; + } + } + + lscAlgo_.queueRequest(context.activeState.lsc, frameContext.lsc, controls); +} + +static unsigned int quantize(unsigned int value, unsigned int step) +{ + return std::lround(value / static_cast(step)) * step; +} + +/** + * \copydoc libcamera::ipa::Algorithm::prepare + */ +void Lsc::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, ipu3_uapi_params *params) +{ + uint32_t ct = frameContext.awb.temperatureK; + unsigned int quantizedCt = quantize(ct, kColourTemperatureQuantization); + + if (!frameContext.lsc.update) { + if (!frameContext.lsc.enabled) + return; + + /* + * Add a threshold so that oscillations around a quantization + * step don't lead to constant changes. + */ + if (utils::abs_diff(ct, lastAppliedCt_) < kColourTemperatureQuantization / 2) + return; + + if (quantizedCt == lastAppliedQuantizedCt_) + return; + } + + /* + * This flag tells the kernel driver that it should read the LSC params + * passed from userspace instead of using its cached copy. + */ + params->use.acc_shd = 1; + + /* + * Pass the enabled flag. If we're not enabled, we can then just bail + * out. + */ + ipu3_uapi_shd_config_static *config = ¶ms->acc_param.shd.shd; + config->general.shd_enable = frameContext.lsc.enabled; + + if (!frameContext.lsc.enabled) + return; + + config->grid.width = numHCells_; + config->grid.height = numVCells_; + config->grid.block_width_log2 = blockWidthLog2_; + config->grid.block_height_log2 = blockHeightLog2_; + config->grid.grid_height_per_slice = IPU3_UAPI_SHD_MAX_CELLS_PER_SET / numHCells_; + + /* + * The IPU3's documentation describes the x_start and y_start members + * as follows: + * + * "[X/Y] value of top left corner of sensor relative to ROI + * s13, [-4096, 0], default 0, only negative values." + * + * I interpret that as allowing us to configure the cropped rectangle + * relative to the full grid. That's useful if we're running the tabular + * algorithm, which would otherwise apply the full grid inappropriately. + * If we're running the polynomial one though the calculated grid is + * probably more appropriate than a coarse application of the full grid + * so let's tell the hardware not to bother correcting in that case. + */ + if (polynomial_) { + config->grid.x_start = 0; + config->grid.y_start = 0; + } else { + config->grid.x_start = (cropWidth_ - sensorWidth_) / 2; + config->grid.y_start = (cropHeight_ - sensorHeight_) / 2; + } + + /* No idea what this is, but the docs say it should be set as so */ + config->general.init_set_vrt_offst_ul = + config->grid.y_start >> (config->grid.block_height_log2 % + config->grid.grid_height_per_slice); + + /* + * Values in the LUT cease taking effect at 4096, and a value of 0.0 is + * "no correction" rather than black. The gain factor is described by + * the documentation like so: + * + * "Shift calculated anti shading value. Precision u2. 0x0 - gain factor + * [1, 5], means no shift interpolated value. 0x1 - gain factor [1, 9], + * means shift interpolated by 1. 0x2 - gain factor [1, 17], means shift + * interpolated by 2." + * + * The simplest interpretation for those pieces of information is I + * think that the LUT stores 12-bit Q numbers who's represented values + * depend on the gain_factor setting like so: + * + * 0: UQ<2, 10> representing values in range [0, 4) + * 1: UQ<3, 9> representing values in range [0, 8) + * 2: UQ<4, 8> representing values in range [0, 16) + * + * And that a base gain of 1.0 is added to those configured values. As a + * gain of more than 5.0 is fairly unlikely, let's fix gain_factor to 0 + * for now and revisit if needed. + */ + config->general.gain_factor = 0; + + /* + * Disable the black level settings here - we do that through another + * parameters block. + */ + config->black_level.bl_r = 0; + config->black_level.bl_gr = 0; + config->black_level.bl_gb = 0; + config->black_level.bl_b = 0; + + ipu3_uapi_shd_lut *lut = ¶ms->acc_param.shd.shd_lut; + + const auto &set = lscAlgo_.interpolateComponents(quantizedCt); + + unsigned int totalCells = numHCells_ * numVCells_; + unsigned int cellsPerSet = numHCells_ * config->grid.grid_height_per_slice; + unsigned int numSets = (numHCells_ + config->grid.grid_height_per_slice - 1) / + config->grid.grid_height_per_slice; + unsigned int i = 0; + + for (unsigned int s = 0; s < numSets; s++) { + for (unsigned int c = 0; c < cellsPerSet && i < totalCells; c++, i++) { + lut->sets[s].r_and_gr[c].r = set.at("r")[i]; + lut->sets[s].r_and_gr[c].gr = set.at("gr")[i]; + lut->sets[s].gb_and_b[c].gb = set.at("gb")[i]; + lut->sets[s].gb_and_b[c].b = set.at("b")[i]; + } + } + + lastAppliedCt_ = ct; + lastAppliedQuantizedCt_ = quantizedCt; + LOG(IPU3Lsc, Debug) + << "ct is " << ct << ", quantized to " + << quantizedCt; +} + +/** + * \copydoc libcamera::ipa::Algorithm::process + */ +void Lsc::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + [[maybe_unused]] const ipu3_uapi_stats_3a *stats, + ControlList &metadata) +{ + lscAlgo_.process(frameContext.lsc, metadata); +} + +REGISTER_IPA_ALGORITHM(Lsc, "Lsc") + +} /* ipa::ipu3::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/algorithms/lsc.h b/src/ipa/ipu3/algorithms/lsc.h new file mode 100644 index 0000000000000000000000000000000000000000..050ce91ad115a93224efb161ae0a3536fe32cc7d --- /dev/null +++ b/src/ipa/ipu3/algorithms/lsc.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Ideas on Board, Oy + * + * IPU3 Lens Shading Correction algorithm + */ + +#pragma once + +#include + +#include "libipa/fixedpoint.h" +#include "libipa/lsc.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::ipu3::algorithms { + +class Lsc : public Algorithm +{ +public: + int init(IPAContext &context, const ValueNode &tuningData) override; + void queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) override; + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + ipu3_uapi_params *params) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ipu3_uapi_stats_3a *stats, + ControlList &metadata) override; + +private: + std::vector calculatePositions(unsigned int dimension); + + unsigned int numHCells_; + unsigned int numVCells_; + unsigned int blockWidthLog2_; + unsigned int blockHeightLog2_; + + unsigned int lastAppliedCt_; + unsigned int lastAppliedQuantizedCt_; + + unsigned int sensorWidth_; + unsigned int sensorHeight_; + unsigned int cropWidth_; + unsigned int cropHeight_; + + bool polynomial_; + + LscAlgorithm> lscAlgo_; +}; + +} /* ipa::ipu3::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build index 3dafd2fda9897942cf87d9640665c4fcff383859..70177f50415259e0c2bd207205c385dde07d6624 100644 --- a/src/ipa/ipu3/algorithms/meson.build +++ b/src/ipa/ipu3/algorithms/meson.build @@ -6,5 +6,6 @@ ipu3_ipa_algorithms = files([ 'awb.cpp', 'blc.cpp', 'ccm.cpp', + 'lsc.cpp', 'tone_mapping.cpp', ]) diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 9537802ceca5018118bdf4c71ef361c20fc44bfb..935264279a8a6346cb3e58ddb5577811b46163e8 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -127,6 +127,11 @@ namespace libcamera::ipa::ipu3 { * \brief Active gamma correction parameters for the IPA */ +/** + * \var IPAActiveState::lsc + * \brief Active lens shading correction parameters for the IPA + */ + /** * \var IPASessionConfiguration::sensor * \brief Sensor-specific configuration of the IPA @@ -186,4 +191,9 @@ namespace libcamera::ipa::ipu3 { * \brief Per-frame gamma correction parameters for the IPA */ +/** + * \var IPAFrameContext::lsc + * \brief Per-frame lens shading correction parameters for the IPA + */ + } /* namespace libcamera::ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index d650f2fe1ad8eab91b7128a47c9690e42b1595f1..d4bf2091a64fb1aad5c715a2d9d265ae3aecacb4 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -21,6 +21,7 @@ #include #include #include +#include namespace libcamera { @@ -68,6 +69,7 @@ struct IPAActiveState { ipa::awb::ActiveState awb; ipa::ccm::ActiveState ccm; ipa::gamma::ActiveState gamma; + ipa::lsc::ActiveState lsc; }; struct IPAFrameContext : public FrameContext { @@ -79,6 +81,7 @@ struct IPAFrameContext : public FrameContext { ipa::awb::FrameContext awb; ipa::ccm::FrameContext ccm; ipa::gamma::FrameContext gamma; + ipa::lsc::FrameContext lsc; }; struct IPAContext {