From patchwork Tue Dec 12 11:50:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 19305 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C48F1BD87C for ; Tue, 12 Dec 2023 11:51:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AA20C62B32; Tue, 12 Dec 2023 12:51:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1702381874; bh=LAXwjRZ5b+9xXTYr6AdLk2aQxO96bQdvdoo7bmAAeBE=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=zZ9ZReuNd2l9FG5OHd8RA/TgTj8S3oTLKPV3WuNiQVYcMBK1uKj5kdkmp9/4KOyHh HgEHRgpyvJ98hYL5Pn9fzbNaU89WgiGS+PZ0PEpjY9peomYSQA4+d04KiQaEOxMAkK HS9YxpLO9tSgkFlxfP17JE0LYe2fCxXF6q9sbsZleuxXfNOKlWb6zfchu4+Vw0ZEOJ WWJw8ziHJ/PTL8zPiMunzTX5ZKA/I7nmH+/MSWReEnuPVmKUWpjEhGfnwJ3CIXs5Mt ytMU8yjPjFBrDeP5PT2TX+XAHd2kXDqO50LaS3YuBRAXo3vn4ukMf5zZ9AOVBzU3wO O0JzUeCHwKv9w== Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 07E34629BD for ; Tue, 12 Dec 2023 12:51:13 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="M2SMZHCT"; dkim-atps=neutral Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-50c222a022dso5989943e87.1 for ; Tue, 12 Dec 2023 03:51:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1702381872; x=1702986672; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=qohYS+7OEys3DNnBs9zpd/ElDPXcE+JR4ISwmV7P3Og=; b=M2SMZHCTVikD0eOlAHy9Nro4iEPmU8PHBuMSy03g/4CAjvdf/Ljaxp1BsWe3ypzyC+ 7mNcTQA/sy9k8pAvZ/C1HBSj9tHjgTuhMW/AEmuFVYxJqTqAeV/Hw8djtOOQckfg/03Q Y/q5v7dK1+2ABNsjsvSiEptxIz7XV3EiJOtyiPtOvBznrztCsw5JYB4SgM40rmOySdc7 lT7BqikQOz6FNtuscRcivo53EpASIWfgVpv9UuqTpU1qYYAG2dMaD/9fIZGEY6M8HY0C k3O1NTMMGRNbQj36GHXF1qN899yYSsCegZYqYCQ4jyJhjVmKn0/O5vwoyTvEAN1QxgaY 0tqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702381872; x=1702986672; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qohYS+7OEys3DNnBs9zpd/ElDPXcE+JR4ISwmV7P3Og=; b=GNIQzi5JZBNCAMCsfWAxuA4txlZ5GIVw29Cd2eaQYUSvhieWGk5yoWfHxaLp9/kj7q grquriAwbrAms1cMUA1jcQ7iuN6IMqcIzbs4y3VH1icTzsV/FBh2AQWxsr9/5Yn7me8c 1eZsL5ljkotfQERRi9VtI8GZPfBNptIH77laIuO87E7V4sZbNVx1tYojdEHEluN6C8kA hvoW/28ztADDvLFzg2zlVUi/U3eaBj4CSnfaARjY5MYdO6ml0D7Y4EF9LCn5F/9WdcLl g/pLJQ0bx5Iup2ahDqGYsiEVAL1WNsJ4lwYFEfHbahBaxelqF+604LVGkCgWOaDiruvm Psug== X-Gm-Message-State: AOJu0Yzn8XyS1QUNNXMgYmkUrxUfHutLjYxvUSSxmZe4kQDA80L0U6ex ya7Mo1EMlQ9URW2tzJZN8FChUUK7OdtrSe5ptOg= X-Google-Smtp-Source: AGHT+IFjUlj6tBPrsOxAB6wAn1fsXhCk+cvM3KeNiT9f+EQdFIsi26zAiVHuQ0Asgl0UGRSyxsZ7Ug== X-Received: by 2002:a05:6512:401b:b0:50d:3b9c:f174 with SMTP id br27-20020a056512401b00b0050d3b9cf174mr2781424lfb.108.1702381872177; Tue, 12 Dec 2023 03:51:12 -0800 (PST) Received: from Lat-5310.. ([87.116.161.153]) by smtp.gmail.com with ESMTPSA id tx17-20020a1709078e9100b00a1b75e0e061sm6213188ejc.130.2023.12.12.03.51.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Dec 2023 03:51:11 -0800 (PST) To: libcamera-devel@lists.libcamera.org Date: Tue, 12 Dec 2023 14:50:39 +0300 Message-Id: <20231212115046.102726-1-andrey.konovalov@linaro.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 0/7] libcamera: introduce Software ISP and Software IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Andrey Konovalov via libcamera-devel From: Andrey Konovalov Reply-To: Andrey Konovalov Cc: mripard@redhat.com, g.martti@gmail.com, t.langendam@gmail.com, srinivas.kandagatla@linaro.org, pavel@ucw.cz, bryan.odonoghue@linaro.org, admin@dennisbonke.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Here is a draft implementation of Software ISP and Software IPA which provide debayering and implementation of image processing algorithms for systems without a hardware ISP, or in cases when there are no public drivers for the hardware ISP present in the system. The implementation of the Software ISP is a reference one. A naive AWB alorithm is implemented as part of a function which does debayering and statistics calculations - the algorithm part is to be moved to the IPA in the next version of the patch set. And for debayering itself there is already a more efficient implementation by Hans de Goede. This patch set is currently using the earlier debayering implementation as it is less lines of code and is OK for the initial discussion. Only RAW10P format from the sensor is currently supported, but other debayering functions can be easily added (which of them to call is decided based on the input format). The Software IPA has only auto exposure and AGC. For the AGC the analogue gain control of the camera sensor is used (if available). The algorithm is very much simplified, and is mostly included as a reference code. The 6th patch renames some variables in the simple pipeline handler for the Software ISP to use the same buffer handling code as the Converter currently does. This lets one to avoid adding extra code to the pipeline handler, but also makes the Software ISP and the Converter mutually exclusive. The Software ISP / IPA are intended to be used with the simple pipeline handler. The pipeline handler doesn't interact with the Software IPA directly - the Software IPA is hidden behind the Software ISP interface. To use the Software ISP the build must be configured with -Dpipelines=simple/linaro -Dipas=simple/linaro and the Converter must not be used (the latter is hardcoded in the simple pipeline handler). If the build is configured with just -Dpipelines=simple the Software ISP / IPA are not used, and the simple pipeline handler works in the same way as without this patchset. "linaro" in the -Dpipelines=simple/linaro -Dipas=simple/linaro is the name of the Software ISP / IPA implementation. -Dpipelines=simple/ -Dipas=simple/ should work too if another implementation were added. What implementation to use is the build time choice, only one implementation is included into the build. This patch set uses SharedMemObject class used by the RPi pipeline handler - the second patch in the series moves the header file to a common directory. This patch set has been tested on Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (not the standard RB5 camera mezzanine). git branch for the patch set v2: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03 v1 of the patch set: https://patchwork.libcamera.org/cover/19262/ Changes in v2 vs v1: - patches are restructured and reordered - the Software IPA is hidden behind the Software ISP interface. The pipeline handler doesn't interact with the Software IPA directly - instantiation of the Software ISP / IPA is configurable (at build time) - comment explaining the implementation limitations is added to SwIspLinaro::IspWorker::debayerRaw10P() Andrey Konovalov (7): libcamera: introduce SoftwareIsp class libcamera: internal: Move SharedMemObject class to a common directory libcamera: ipa: add Soft IPA common files libcamera: ipa: Soft IPA: add a Soft IPA implementation libcamera: software_isp: add SwIspLinaro implementation of SoftwareIsp libcamera: pipeline: simple: rename converterBuffers_ and related vars libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA include/libcamera/internal/meson.build | 3 + .../libcamera/internal}/shared_mem_object.h | 4 - include/libcamera/internal/software_isp.h | 106 ++++ .../internal/software_isp/meson.build | 6 + .../internal/software_isp/statistics-linaro.h | 17 + .../internal/software_isp/swisp_linaro.h | 117 ++++ include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 27 + meson_options.txt | 3 +- src/ipa/simple/common/meson.build | 17 + src/ipa/simple/common/soft_base.cpp | 66 ++ src/ipa/simple/common/soft_base.h | 47 ++ src/ipa/simple/linaro/data/meson.build | 9 + src/ipa/simple/linaro/data/soft.conf | 3 + src/ipa/simple/linaro/meson.build | 26 + src/ipa/simple/linaro/soft_linaro.cpp | 210 +++++++ src/ipa/simple/meson.build | 12 + src/libcamera/meson.build | 2 + src/libcamera/pipeline/simple/simple.cpp | 165 +++-- src/libcamera/software_isp.cpp | 62 ++ src/libcamera/software_isp/meson.build | 20 + src/libcamera/software_isp/swisp_linaro.cpp | 589 ++++++++++++++++++ 22 files changed, 1455 insertions(+), 57 deletions(-) rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (98%) create mode 100644 include/libcamera/internal/software_isp.h create mode 100644 include/libcamera/internal/software_isp/meson.build create mode 100644 include/libcamera/internal/software_isp/statistics-linaro.h create mode 100644 include/libcamera/internal/software_isp/swisp_linaro.h create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/common/meson.build create mode 100644 src/ipa/simple/common/soft_base.cpp create mode 100644 src/ipa/simple/common/soft_base.h create mode 100644 src/ipa/simple/linaro/data/meson.build create mode 100644 src/ipa/simple/linaro/data/soft.conf create mode 100644 src/ipa/simple/linaro/meson.build create mode 100644 src/ipa/simple/linaro/soft_linaro.cpp create mode 100644 src/ipa/simple/meson.build create mode 100644 src/libcamera/software_isp.cpp create mode 100644 src/libcamera/software_isp/meson.build create mode 100644 src/libcamera/software_isp/swisp_linaro.cpp Tested-by: Pavel Machek