From patchwork Thu Sep 8 01:41:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 17309 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 62A79C3272 for ; Thu, 8 Sep 2022 01:42:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8913862093; Thu, 8 Sep 2022 03:42:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1662601339; bh=I13RnWDgq6Jmi4C51Te6s7Dnod9yFpwhgl3ZBNcL09k=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=qqmJnI6dL9DcvyDA1/9gy3O2OReYrngUW4fGDIfmjWB/13qA52Ppfs/+phNN+ctDA 7DuapL8Cf95OxLOkFoaKOQE7ePooz34NxXFrzLzShZNomf/gE6GDVS8EQtcAt+IJg3 5Lqp7vpKp94/NlapuS7M+zGaYJlciM3LG11sDawyc4yQfTpZNHWvEz00StKwmOvZFH ocK9WnXXMBNDe3ICRTD29b/4HQnZ7dE6532lChDa2seBJwWftMpF1alTp7sM2v9SiY 09fMnjAYMhTR8S7f/3eOUTOTGyR0D9PQyMo4rtLrozFd00Go7ohTlbXQFnuH+bUtik bmN/PdV0evPvA== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8CDD861FB8 for ; Thu, 8 Sep 2022 03:42:17 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="XjYOb5ld"; dkim-atps=neutral Received: from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id DE33F6CC for ; Thu, 8 Sep 2022 03:42:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1662601337; bh=I13RnWDgq6Jmi4C51Te6s7Dnod9yFpwhgl3ZBNcL09k=; h=From:To:Subject:Date:From; b=XjYOb5ldtbXbuZ2mU1KSXOQu6bvjreYjnyLCwLsvwxuIWFvf+rR3MgbG2utQTJJub Oos9ureLDcNENGAe3yR82Lzk03gET/wE2YeZOZ19PdhMBpOEMfsdF/f/7gWz6OFjS8 8wh3Fovd2gk/frguD3ydwJXIiHAPrDB8OBzf4Lb8= To: libcamera-devel@lists.libcamera.org Date: Thu, 8 Sep 2022 04:41:28 +0300 Message-Id: <20220908014200.28728-1-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 00/32] ipa: Frame context queue, IPU3 & RkISP consolidation, and RkISP1 improvements X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Laurent Pinchart via libcamera-devel From: Laurent Pinchart Reply-To: Laurent Pinchart Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hello, After lots of reshuffling of the "[PATCH v3 00/17] libcamera: Align IPU3 and RKISP1 interfaces" series, here's a v4 that has grown quite a bit on the RkISP1 side. I've taken over the previous version of the series as I thought it would be easier and faster to provide review as fixup commits. I ended up reshuffling modifications around, splitting and combining commits, not so much in an attempt to clean or fix the history, but out of a desire to make sure I correctly understood the changes. The result should hopefully be easy to follow. Patches 01/32 and 02/32 are small preliminary drive-by cleanups. The gist of the frame context queue introduction is in patches 03/32 to 08/32 that also update the IPA algorithm operations to pass the new objects around. There shouldn't be anything really new there, small changes are listed in individual changelogs. The next patches, from 09/32 to 11/32, port the IPU3 IPA module to the FCQueue class. They should be similar to the implementation in v3 that was spread within the patches that updated the IPA algorithm operations. I have dropped the other changes to the IPU3 IPA module, not because they were wrong, but because the series was growing big already. I will rebase them on top of this v4 and submit them separately (unless someone beats me to it). Patches 12/32 to 16/32 then performs the same for the RkISP1 IPA module. I have also dropped the lens-related patches from v3, as I wanted to focus on the frame context queue first. Those patches will also be rebased on top, they're not lost. Patch 17/32 then renames the IPAFrameContext structures to IPU3FrameContext and RkISP1FrameContext. This was part of v3, and I'm not too fond of it to be honest, as the other structures don't have the same naming scheme. I would prefer dropping this from the series and then discussing the naming scheme of IPA structures globally. The next 7 patches, from 18/32 to 24/32, port the RkISP1 algorithms to the frame context API. 18/32 starts by dropping the frameCount member variable of the active state as we can now use the frame number passed to the algorithm operations, and the next patches port the algorithms one by one, showcasing how data can be split between the active state and frame context. Different schemes may be possible, maybe with less duplication of member variables between the active state and frame context. I'm particularly interested in feedback on this. The last patch of this group, 24/32, documents the scheme used here. Finally, patches 25/32 to 32/32 improve the AWB implementation of the RkISP1 IPA module. Patch 25/32 in particular shows how the frame context allows fixing a defect of the current implementation. The rest are improvements that are detailed in the individual commit messages. We don't need to merge this series in one go. It is organized as three sets of patches (01/32 to 17/32 to introduce and use the frame context queue, 18/32 to 24/32 to showcase its usage in the RkISP1 IPA module, and 25/32 to 32/32 to improve the RkISP1 AWB). Each set is quite self-contained, but they are based on each other in that order. There is still work to do, in particular in the documentation, but I didn't want to spend more time on it before getting an approval of the general approach. Another point I'm not completely happy about is the naming of the FCQueue functions, and in particular the init() function (alloc() would already be better in my opinion, and I'd like to find an even better name) but that could also possibly be fixed on top. Jacopo Mondi (1): ipa: rkisp1: Remove unused class member Kieran Bingham (7): ipa: libipa: Provide a common base for frame contexts ipa: libipa: algorithm: prepare(): Pass frame and frame Context ipa: libipa: algorithm: process(): Pass frame number ipa: libipa: algorithm: queueRequest(): Pass frame context ipa: rkisp1: Rename frameContext to activeState ipa: rkisp1: Convert to use the FCQueue ipa: Rename IPAFrameContext structures Laurent Pinchart (22): ipa: ipu3: Fix style of Doxygen comment blocks ipa: ipu3: af: Pass context reference to afIsOutOfFocus() ipa: libipa: Pass a reference instead of pointer to Algorithm::process() ipa: ipu3: Use base FrameContext class ipa: ipu3: Use the FCQueue ipa: ipu3: Pass controls to algorithm's queueRequest() handler ipa: rkisp1: Sort documentation of the IPA context ipa: rkisp1: Use base FrameContext class ipa: rkisp1: Use frame number passed to Algorithm::prepare() ipa: rkisp1: agc: Store per-frame information in frame context ipa: rkisp1: awb: Store per-frame information in frame context ipa: rkisp1: cproc: Store per-frame information in frame context ipa: rkisp1: dpf: Store per-frame information in frame context ipa: rkisp1: filter: Store per-frame information in frame context ipa: rkisp1: Document the active state and frame context ipa: rkisp1: awb: Use frame context to fix gains calculations ipa: rkisp1: awb: Store color temperature as an integer ipa: rkisp1: awb: Log means, gains and temperature in debug message ipa: rkisp1: awb: Prevent RGB means from being negative ipa: rkisp1: awb: Clamp gains to prevent divisions by zero ipa: rkisp1: awb: Freeze AWB when means are too small ipa: rkisp1: awb: Remove bias from gain calculation Quentin Schulz (1): ipa: rkisp1: awb: Add support for RGB means Umang Jain (1): ipa: libipa: Introduce FrameContextQueue src/ipa/ipu3/algorithms/af.cpp | 34 +-- src/ipa/ipu3/algorithms/af.h | 9 +- src/ipa/ipu3/algorithms/agc.cpp | 10 +- src/ipa/ipu3/algorithms/agc.h | 5 +- src/ipa/ipu3/algorithms/awb.cpp | 8 +- src/ipa/ipu3/algorithms/awb.h | 7 +- src/ipa/ipu3/algorithms/blc.cpp | 10 +- src/ipa/ipu3/algorithms/blc.h | 4 +- src/ipa/ipu3/algorithms/tone_mapping.cpp | 18 +- src/ipa/ipu3/algorithms/tone_mapping.h | 6 +- src/ipa/ipu3/ipa_context.cpp | 45 +--- src/ipa/ipu3/ipa_context.h | 18 +- src/ipa/ipu3/ipu3.cpp | 39 +++- src/ipa/ipu3/module.h | 2 +- src/ipa/libipa/algorithm.cpp | 4 + src/ipa/libipa/algorithm.h | 7 +- src/ipa/libipa/fc_queue.cpp | 135 ++++++++++++ src/ipa/libipa/fc_queue.h | 118 ++++++++++ src/ipa/libipa/meson.build | 2 + src/ipa/rkisp1/algorithms/agc.cpp | 36 +-- src/ipa/rkisp1/algorithms/agc.h | 10 +- src/ipa/rkisp1/algorithms/awb.cpp | 269 ++++++++++++++++------- src/ipa/rkisp1/algorithms/awb.h | 12 +- src/ipa/rkisp1/algorithms/blc.cpp | 6 +- src/ipa/rkisp1/algorithms/blc.h | 4 +- src/ipa/rkisp1/algorithms/cproc.cpp | 52 +++-- src/ipa/rkisp1/algorithms/cproc.h | 5 +- src/ipa/rkisp1/algorithms/dpcc.cpp | 6 +- src/ipa/rkisp1/algorithms/dpcc.h | 4 +- src/ipa/rkisp1/algorithms/dpf.cpp | 25 ++- src/ipa/rkisp1/algorithms/dpf.h | 5 +- src/ipa/rkisp1/algorithms/filter.cpp | 32 +-- src/ipa/rkisp1/algorithms/filter.h | 5 +- src/ipa/rkisp1/algorithms/gsl.cpp | 6 +- src/ipa/rkisp1/algorithms/gsl.h | 4 +- src/ipa/rkisp1/algorithms/lsc.cpp | 5 +- src/ipa/rkisp1/algorithms/lsc.h | 4 +- src/ipa/rkisp1/ipa_context.cpp | 265 ++++++++++++++++------ src/ipa/rkisp1/ipa_context.h | 58 ++++- src/ipa/rkisp1/module.h | 2 +- src/ipa/rkisp1/rkisp1.cpp | 52 +++-- 41 files changed, 982 insertions(+), 366 deletions(-) create mode 100644 src/ipa/libipa/fc_queue.cpp create mode 100644 src/ipa/libipa/fc_queue.h