From patchwork Fri Nov 6 15:13:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 28796 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0E587C335C for ; Fri, 6 Nov 2026 15:14:05 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A65746A7E1; Fri, 6 Nov 2026 16:14:04 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=qualcomm.com header.i=@qualcomm.com header.b="gKHcxEu5"; dkim=pass (2048-bit key; unprotected) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b="KXqBczv2"; dkim-atps=neutral Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 631F66A7E1 for ; Fri, 6 Nov 2026 16:14:00 +0100 (CET) Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 6A6F0Fft087780 for ; Fri, 6 Nov 2026 15:13:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h= cc:content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=qcppdkim1; bh=/PX1YKa+HYv ZBGow/hQN9LE9bVXsPOugmPY594g/Tdk=; b=gKHcxEu5XpOZd9xK1B933BRaABQ cnWEXz5MNrhIGfjmFVNyKe3stMAlfyZ3T4x64Cqwp8/tZzzmq5MQkP5zbEs2zFhU deti17VQBtxgRvXTjzuI5gnEkHpmbpzeU3xU76s3ptI4MCtvgYgwIYH3dawGdK95 8RGl9AZmdXY1h2LYIFObZed8B3SBtQdfx++ArCftVjKu8iH2YAiZhJgW5+sfWCvp NzVCg+EKum4S0bRpZ28dYSBIlzErLank0kLaCkeNfDAIqVi2lhrXd5bklLPKpWQ2 Hl0K74MNnoojw1sBy+UYiMUObkcg3yU5dy6UMWahECeWHT6Sy4oMAxIUH2g== Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4hs3pa3ak5-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Fri, 06 Nov 2026 15:13:58 +0000 (GMT) Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-93f2373e9b4so92953085a.1 for ; Fri, 06 Nov 2026 07:13:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oss.qualcomm.com; s=google; t=1793978038; x=1794582838; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/PX1YKa+HYvZBGow/hQN9LE9bVXsPOugmPY594g/Tdk=; b=KXqBczv2mSNyPYL9xufAJIE2nqp4NvfOkE1lotYBzO+8TNm1+Z1osiwpQCN6ShybUX egAAkYMueoGD42LFw7/9fFambRcO1eul/fvWVk+5lSIddnjrILq4qZMwUPpyjFTFY1+c sb7sp+4CRypFavxsJ4juu8AF9Bzu9RiNGwzuTd8WE2aQmtlAI/ec3jcvhVfHjbMnt8WH h+Hu52pW2FnqKrunwokzV/iqJ8ViO10o6+ykzw66ptqyF++EZO88WFzscdn5mb4+tBJV vW9R0Begn8z/KKCBJwVsfY+aW+nOnDQKccvzkleFgLipcGvgPcwFAiuOBC5yAeg3truG HZ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793978038; x=1794582838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/PX1YKa+HYvZBGow/hQN9LE9bVXsPOugmPY594g/Tdk=; b=wQ4cWk4YJkZb4RORiNEKaGaY2DkHo8KazMtY2gXpxdZ1IMf5R6nKlW7cWsPDnBAz8R 8keNvFXrdfurnyB7LAMX5ewacEOFjkVxoj1XPH+hYWZOaEgT20Itp2jmAdPCoC0CzC/m yK1eypHyIAzAqk8fY+3hH3bjW+KAP5ar4iP95IrtpkxRfr93Y6BR4D19ytcdW6cZJGt0 Y2EXtg14GcwMSG3ZECFviBlMVCmCOc4iPEVaO50s8qNnjx8M/S5DxvNPHD+xSX3p8IVI v46gxKDfcrtpFGtD6WKGeHWHLGWyff+0k/3cJ1DzKDXEkSykuHGiJdDMxRrc3Rc8YIXj qltA== X-Gm-Message-State: APuU4Z/rHF1neM/Ev9aLafnRUD+CfD73QV3p6knThpj7gxFa9nuZL8MW tyhNa/0g8gfHHAaDpu84lFR+jX/+cdC2ECtjTv3ThdbqQuzajMrf1ln40pkoTBJr99i6Mcs0h4p ud2aAhOv1dzISy2GNnkwwqAqitz9t3/oul6pZ1ZTzejvzbD7LvnkVfH1RfqeeAYfi67nxb0g7iF ueNuxxhKvx X-Gm-Gg: AYBFou07CZgZv8jC+4I3D/MxeOXHvmJbDM/9bDQBVH2bnCwZzqq6WCbvadY2CQB9rxg KKO60Bld7FmiWitAaxI1HyPJbLMsbW+Bf/fKb2DuJB35QXyu4OpMVpVluk1aKPSG6C/J9bzx0ZB uf0RqrsImb2aE8NgZrK/FbDWHB+FWQK2AFAKl4+I0ANC3IUs/WPQcYYk+1/22EvX7VN+SR1vNiQ evf7pI0J5pB+81/AWMoFyjRcbbQDTbkXXEA66PmBFg86P9Nt2l/UEBqtH3XwP7oMUAWSexTgKGJ /6a/lj11RcM7/DH18sUtpH1UyEg1EwWYUE4hAiM0SDAdJGl/j2E99pqGyq8XaheUcWO8aPGudUZ PQ7DtbXr3Z+SZoAGiVzX2uyCHt3t+5KMmECPea46+MW/LMGuG0VTVz2H6rcJDEZpE3I9oBcsKeW v+4LFthNGW4lUNJY1Z3FykvAYGZcFQ8HDOk7DJklyy6tT9X4vLMOrcNvR22lcywkWz X-Received: by 2002:a05:620a:4449:b0:945:c50d:569e with SMTP id af79cd13be357-945f34fb975mr383916785a.36.1793978038104; Fri, 06 Nov 2026 07:13:58 -0800 (PST) X-Received: by 2002:a05:620a:4449:b0:945:c50d:569e with SMTP id af79cd13be357-945f34fb975mr383910185a.36.1793978037359; Fri, 06 Nov 2026 07:13:57 -0800 (PST) Received: from shalem (2001-1c00-0c32-7800-5bfa-a036-83f0-f9ec.cable.dynamic.v6.ziggo.nl. [2001:1c00:c32:7800:5bfa:a036:83f0:f9ec]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c4d3569b856sm86042466b.46.2026.11.06.07.13.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2026 07:13:56 -0800 (PST) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Cc: Gjorgji Rosikopulos , Loic Poulain , Hans de Goede Subject: [PATCH v3 7/7] camss: Add Offline Processing Engine ISP support Date: Fri, 6 Nov 2026 16:13:41 +0100 Message-ID: <20261106151345.32491-8-johannes.goede@oss.qualcomm.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261106151345.32491-1-johannes.goede@oss.qualcomm.com> References: <20261106151345.32491-1-johannes.goede@oss.qualcomm.com> MIME-Version: 1.0 X-Proofpoint-GUID: Lq6L25evKqJb9Z-36IB8Id14Pmc9VURG X-Proofpoint-Spam-Info: AW1haW4tMjYxMTA2MDA1OSBTYWx0ZWRfX71YRJo6nGwt0 r3cdvBLvDqwGooc6mxdTjZ2GapKXDJHZyomYi7lHS8EsPfVH378FHlIN+8MBFyradxq0AretShJ 90F5jQD3PsKuuoWodUd/CXFh+7bVzdw= X-Authority-Analysis: v=2.4 cv=e7WT2qp/ c=1 sm=1 tr=0 ts=6aedeeb6 cx=c_pps a=HLyN3IcIa5EE8TELMZ618Q==:117 a=xqWC_Br6kY4A:10 a=zJRp6_cAMcgA:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=u7WPNUs3qKkmUXheDGA7:22 a=rJkE3RaqiGZ5pbrm-msn:22 a=VwQbUJbxAAAA:8 a=EUspDBNiAAAA:8 a=P511DZv0kOVs7XiUdvQA:9 a=bTQJ7kPSJx9SKPbeHEYW:22 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYxMTA2MDA1OSBTYWx0ZWRfX3oMFfwIANR2B A2kXcQWKPicTWWsgJKQVvQeBkYBrcg+ctsl/e93ugwInCdOxPm5xSKWjViaPAI7+/lL6oWDeNZN crpdWzqFSApl+91Oi+K8qiwpekdzJPdHau+FSaxFoMeMR452qRZjeX32yYggLWwZn+sEk5nRoGK yuYdQ7YjE12eN8OSu3v+079p/beXbHztXqVAt7VnB7RnIlVbJGe8iPdn4FOYxnNLk8EmYljjkP+ rHe9ZwXLlpAHVVFzVKTWraZ8+yygXrSGBpk3lnHUTELqovqcgOZtvzBoGy2ESFf4T99Tx4w3X/F RT2kLk4lJmajJZCHo+IAmjR2drnL4bN+KOz4+bio+oSgaIN7UNEwCdhaXqhWM0CKWP6ikoyqCBW d7EZU3J5zqAXdFA2uUZPjkVrrWcv7Tca0TWbXFFbULOUauE9sBNRLEpPgqalyI6XTXSfVpCycfM +nQfu8uHSAJicMcNlMg== X-Proofpoint-ORIG-GUID: Lq6L25evKqJb9Z-36IB8Id14Pmc9VURG X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293, Aquarius:18.0.1176, Hydra:6.1.138, FMLib:17.12.100.49 definitions=2026-11-06_05,2026-11-06_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 impostorscore=0 suspectscore=0 clxscore=1015 priorityscore=1501 malwarescore=0 adultscore=0 lowpriorityscore=0 bulkscore=0 phishscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2611040001 definitions=main-2611060059 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This adds support for the OPE through the v11 kernel driver series posted upstream recently (See Link tag). This is still in the proof of concept phase atm. With the biggest todo item being this using the softISP simple IPA atm, while it really should come with its own IPA. Link: https://lore.kernel.org/linux-media/20261023-camss-isp-ope-v11-0-0c2937195298@oss.qualcomm.com/ Signed-off-by: Hans de Goede --- src/libcamera/pipeline/camss/camss.cpp | 19 +- .../pipeline/camss/camss_isp_ope.cpp | 704 ++++++++++++++++++ src/libcamera/pipeline/camss/camss_isp_ope.h | 120 +++ src/libcamera/pipeline/camss/camss_params.h | 2 +- src/libcamera/pipeline/camss/meson.build | 3 + 5 files changed, 841 insertions(+), 7 deletions(-) create mode 100644 src/libcamera/pipeline/camss/camss_isp_ope.cpp create mode 100644 src/libcamera/pipeline/camss/camss_isp_ope.h diff --git a/src/libcamera/pipeline/camss/camss.cpp b/src/libcamera/pipeline/camss/camss.cpp index 20b64ea49..e34d1dbf7 100644 --- a/src/libcamera/pipeline/camss/camss.cpp +++ b/src/libcamera/pipeline/camss/camss.cpp @@ -38,6 +38,7 @@ #include "camss_csi.h" #include "camss_frames.h" #include "camss_isp.h" +#include "camss_isp_ope.h" #include "camss_isp_raw.h" #include "camss_isp_soft.h" @@ -525,12 +526,18 @@ bool PipelineHandlerCamss::match(DeviceEnumerator *enumerator) data->delayedCtrls_ = std::make_unique(sensor->device(), params); - data->processedIsp_ = std::make_unique(this, sensor, - &data->frameInfos_, - &data->controlInfo_); - if (!data->processedIsp_->isValid()) { - LOG(Camss, Error) << "Failed to create software ISP"; - continue; + data->processedIsp_ = CamssIspOpe::match(this, enumerator, sensor, + &data->frameInfos_, + &data->controlInfo_); + if (data->processedIsp_ == nullptr) { + data->processedIsp_ = + std::make_unique(this, sensor, + &data->frameInfos_, + &data->controlInfo_); + if (!data->processedIsp_->isValid()) { + LOG(Camss, Error) << "Failed to create software ISP"; + continue; + } } data->rawIsp_ = std::make_unique(this, sensor, diff --git a/src/libcamera/pipeline/camss/camss_isp_ope.cpp b/src/libcamera/pipeline/camss/camss_isp_ope.cpp new file mode 100644 index 000000000..d9dcb254a --- /dev/null +++ b/src/libcamera/pipeline/camss/camss_isp_ope.cpp @@ -0,0 +1,704 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Qualcomm CAMSS OPE ISP class + * + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + */ + +#include "camss_isp_ope.h" + +#include + +#include + +#include +#include +#include +#include +#include + +#include "libcamera/internal/camera_manager.h" +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/dma_buf_allocator.h" +#include "libcamera/internal/framebuffer.h" // FIXME drop once softIPA (ab)use is gone +#include "libcamera/internal/ipa_manager.h" +#include "libcamera/internal/media_device.h" +#include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swstats_cpu.h" +#include "libcamera/internal/v4l2_subdevice.h" +#include "libcamera/internal/v4l2_videodevice.h" + +#include "camss_frames.h" +#include "camss_params.h" +#include "camss_util.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(Camss) + +namespace { + +static const std::map inputFormatToMediaBus = { + { formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 }, + { formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 }, + { formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 }, + { formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 }, + { formats::SBGGR10_CSI2P, MEDIA_BUS_FMT_SBGGR10_1X10 }, + { formats::SGBRG10_CSI2P, MEDIA_BUS_FMT_SGBRG10_1X10 }, + { formats::SGRBG10_CSI2P, MEDIA_BUS_FMT_SGRBG10_1X10 }, + { formats::SRGGB10_CSI2P, MEDIA_BUS_FMT_SRGGB10_1X10 }, +}; + +static const std::map outputFormatToMediaBus = { + { formats::NV12, MEDIA_BUS_FMT_YUYV8_1_5X8 }, + { formats::NV21, MEDIA_BUS_FMT_YVYU8_1_5X8 }, + { formats::NV16, MEDIA_BUS_FMT_YUYV8_2X8 }, + { formats::NV61, MEDIA_BUS_FMT_YUYV8_2X8 }, + { formats::NV24, MEDIA_BUS_FMT_YUV8_1X24 }, + { formats::NV42, MEDIA_BUS_FMT_YUV8_1X24 }, + { formats::R8, MEDIA_BUS_FMT_Y8_1X8 }, +}; + +} /* namespace */ + +/** + * \class CamssIspOpe + * \brief CAMSS ISP class for the Offline Processing Engine (OPE) ISP + */ + +/** + * \brief Constructs CamssIspOpe object + * \param[in] pipe The pipeline handler in use + * \param[in] opeMediaDev The OPE MediaDevice + * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline + * \param[in] frameInfos Pointer to CamssFrames instance owned by the pipeline + * \param[out] ControlInfoMap to which to add ISP provided controls + */ +CamssIspOpe::CamssIspOpe(PipelineHandler *pipe, std::shared_ptr opeMediaDev, + const CameraSensor *sensor, CamssFrames *frameInfos, + ControlInfoMap *ispControls) + : opeMediaDev_(std::move(opeMediaDev)), sensor_(sensor), frameInfos_(frameInfos) +{ + sharedParams_ = SharedMemObject("softIsp_params"); + if (!sharedParams_) { + LOG(Camss, Error) << "Failed to create shared memory for parameters"; + return; + } + + const CameraManager &cm = *pipe->cameraManager(); + + stats_ = std::make_unique(cm); + if (!stats_->isValid()) { + LOG(Camss, Error) << "Failed to create SwStatsCpu object"; + return; + } + + stats_->statsReady.connect(this, + [&](uint32_t frame, uint32_t statsBufferId) { + statsReady.emit(frame, statsBufferId); + }); + + ipa_ = pipe->createIPA(0, 0); + if (!ipa_) { + LOG(Camss, Error) << "Creating IPA failed"; + return; + } + + /* + * The API tuning file is made from the sensor name. If the tuning file + * isn't found, fall back to the 'uncalibrated' file. + */ + std::string ipaTuningFile = + ipa_->configurationFile(sensor->model() + ".yaml", "uncalibrated.yaml"); + + IPACameraSensorInfo sensorInfo{}; + int ret = sensor->sensorInfo(&sensorInfo); + if (ret) { + LOG(Camss, Error) << "Camera sensor information not available"; + ipa_.reset(); + return; + } + + bool ccmEnabled; + bool lscEnabled; + ret = ipa_->init(IPASettings{ ipaTuningFile, sensor->model() }, + stats_->getStatsFD(), + sharedParams_.fd(), + sensorInfo, + sensor->controls(), + ispControls, + &ccmEnabled, + &lscEnabled); + if (ret) { + LOG(Camss, Error) << "IPA init failed"; + ipa_.reset(); + return; + } + + ipa_->paramsComputed.connect(this, &CamssIspOpe::paramsComputed); + ipa_->metadataReady.connect(this, + [&](uint32_t frame, const ControlList &metadata) { + metadataReady.emit(frame, metadata); + }); + ipa_->setSensorControls.connect(this, + [&](const ControlList &sensorControls) { + setSensorControls.emit(sensorControls); + }); +} + +CamssIspOpe::~CamssIspOpe() = default; + +bool CamssIspOpe::isValid() +{ + return !!ipa_; +} + +Size CamssIspOpe::getMargins([[maybe_unused]] PixelFormat inputFormat) +{ + /* TODO: Do we really not need any margins on OPE? */ + return Size(); +} + +int CamssIspOpe::init() +{ + MediaEntity *paramsEnt, *inputEnt, *procEnt, *dispEnt, *dispOutputEnt; + int ret; + + if (!isValid()) + return -ENOMEM; + + paramsEnt = opeMediaDev_->getEntityByName("ope_params"); + inputEnt = opeMediaDev_->getEntityByName("ope_input"); + procEnt = opeMediaDev_->getEntityByName("ope_proc"); + dispEnt = opeMediaDev_->getEntityByName("ope_disp"); + dispOutputEnt = opeMediaDev_->getEntityByName("ope_disp_output"); + + if (!paramsEnt || !inputEnt || !procEnt || !dispEnt || !dispOutputEnt) { + LOG(Camss, Error) << "Did not find expected entities"; + return -EINVAL; + } + + params_ = std::make_unique(paramsEnt); + ret = params_->open(); + if (ret) + return ret; + + params_->bufferReady.connect(this, &CamssIspOpe::parameterBufferReady); + + input_ = std::make_unique(inputEnt); + ret = input_->open(); + if (ret) + return ret; + + input_->bufferReady.connect(this, [&](FrameBuffer *f) { + inputBufferReady.emit(f); + }); + + proc_ = std::make_unique(procEnt); + ret = proc_->open(); + if (ret) + return ret; + + disp_ = std::make_unique(dispEnt); + ret = disp_->open(); + if (ret) + return ret; + + output_ = std::make_unique(dispOutputEnt); + ret = output_->open(); + if (ret) + return ret; + + output_->bufferReady.connect(this, [&](FrameBuffer *f) { + outputBufferReady.emit(f); + }); + + return 0; +} + +/* Note cfg is const / not modified when update_stride_and_size is false */ +int CamssIspOpe::trySetVideoCfg(V4L2VideoDevice *v4l2Dev, StreamConfiguration &cfg, + bool set, bool update_stride_and_size, + const char *msgPrefix) const +{ + int ret; + + V4L2DeviceFormat v4l2Format; + v4l2Format.fourcc = v4l2Dev->toV4L2PixelFormat(cfg.pixelFormat); + v4l2Format.size = cfg.size; + v4l2Format.planes[0].bpl = cfg.stride; + v4l2Format.planesCount = 1; + + if (set) + ret = v4l2Dev->setFormat(&v4l2Format); + else + ret = v4l2Dev->tryFormat(&v4l2Format); + + if (ret < 0) { + LOG(Camss, Error) << msgPrefix << " error: " << strerror(-ret); + return ret; + } + + if (update_stride_and_size) { + cfg.stride = v4l2Format.planes[0].bpl; + cfg.frameSize = v4l2Format.planes[0].size; + } + + if (!camssV4L2DeviceFormatMatchesStreamConfig(v4l2Format, cfg, msgPrefix)) + return -EINVAL; + + return 0; +} + +StreamConfiguration CamssIspOpe::generateConfiguration(const StreamConfiguration &raw) const +{ + if (!inputFormatToMediaBus.count(raw.pixelFormat)) { + LOG(Camss, Warning) << "Unsupported pixel format " << raw.pixelFormat; + return {}; + } + + if (trySetVideoCfg(input_.get(), const_cast(raw), + false, false, "OPE input try format")) + return {}; + + /* OPE always supports all output formats */ + std::vector sizesVector = { SizeRange(kMinOutputSize, raw.size, 2, 2) }; + std::map> formats; + for (auto it = outputFormatToMediaBus.begin(); it != outputFormatToMediaBus.end(); it++) + formats[it->first] = sizesVector; + + StreamConfiguration cfg{ StreamFormats{ formats } }; + cfg.size = raw.size; + cfg.pixelFormat = formats::NV12; /* Default to NV12 output */ + cfg.bufferCount = kBufferCount; + + return cfg; +} + +int CamssIspOpe::trySetSubdevFormat(V4L2Subdevice *subdev, + unsigned int pad, + const V4L2SubdeviceFormat &fmt, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const +{ + V4L2SubdeviceFormat subdevFormat = fmt; + + int ret = subdev->setFormat(pad, &subdevFormat, whence); + if (ret < 0) { + LOG(Camss, Error) << msgPrefix << " try/set format error: " << strerror(-ret); + return ret; + } + + if (subdevFormat.size != fmt.size || subdevFormat.code != fmt.code) { + LOG(Camss, Error) << msgPrefix << " mismatch asked " + << fmt.size << "/" << fmt.code << " got " + << subdevFormat.size << "/" << subdevFormat.code; + return -EINVAL; + } + + return 0; +} + +int CamssIspOpe::trySetSubdevSelection(V4L2Subdevice *subdev, + unsigned int pad, + unsigned int target, + const Rectangle &rect, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const +{ + struct Rectangle r = rect; + + int ret = subdev->setSelection(pad, target, &r, whence); + if (ret < 0) { + LOG(Camss, Error) << msgPrefix << " set selection error: " << strerror(-ret); + return ret; + } + + if (r != rect) { + LOG(Camss, Error) << msgPrefix << " mismatch asked " + << rect << " got " << r; + return -EINVAL; + } + + return 0; +} + +int CamssIspOpe::trySetPipelineConfig(const StreamConfiguration &inputCfg, + [[maybe_unused]] const StreamConfiguration &outputCfg, + V4L2Subdevice::Whence whence) const +{ + if (!inputFormatToMediaBus.count(inputCfg.pixelFormat)) { + LOG(Camss, Warning) << "Unsupported pixel format " << inputCfg.pixelFormat; + return -EINVAL; + } + + V4L2SubdeviceFormat subdevFormat; + subdevFormat.code = inputFormatToMediaBus.at(inputCfg.pixelFormat); + subdevFormat.size = inputCfg.size; + + int ret = trySetSubdevFormat(proc_.get(), PROC_PAD_INPUT, subdevFormat, + whence, "OPE proc input"); + if (ret < 0) + return ret; + + /* + * No input-crop, downscale instead to keep full field of view. + * \todo crop in x or y dimension to match requested output aspect, + * to keep square pixels after scaling. + */ + ret = trySetSubdevSelection(proc_.get(), PROC_PAD_INPUT, V4L2_SEL_TGT_CROP, + Rectangle(inputCfg.size), whence, "OPE input crop"); + if (ret < 0) + return ret; + + ret = trySetSubdevSelection(proc_.get(), PROC_PAD_DISP, V4L2_SEL_TGT_COMPOSE, + Rectangle(outputCfg.size), whence, "OPE output compose"); + if (ret < 0) + return ret; + + subdevFormat.code = outputFormatToMediaBus.at(outputCfg.pixelFormat); + subdevFormat.size = outputCfg.size; + + ret = trySetSubdevFormat(proc_.get(), PROC_PAD_DISP, subdevFormat, + whence, "OPE proc disp out"); + if (ret < 0) + return ret; + + ret = trySetSubdevFormat(disp_.get(), DISP_PAD_PROC, subdevFormat, + whence, "OPE disp proc in"); + if (ret < 0) + return ret; + + ret = trySetSubdevFormat(disp_.get(), DISP_PAD_OUTPUT, subdevFormat, + whence, "OPE disp out"); + if (ret < 0) + return ret; + + return 0; +} + +StreamConfiguration CamssIspOpe::validate(const StreamConfiguration &raw, const StreamConfiguration &req) const +{ + if (trySetVideoCfg(input_.get(), const_cast(raw), + false, false, "OPE input try format")) + return {}; + + StreamConfiguration cfg; + + for (auto it = outputFormatToMediaBus.begin(); it != outputFormatToMediaBus.end(); it++) { + if (it->first == req.pixelFormat) + cfg.pixelFormat = req.pixelFormat; + } + + if (!cfg.pixelFormat.isValid()) + cfg.pixelFormat = formats::NV12; /* Default to NV12 output */ + + if (SizeRange(kMinOutputSize, raw.size, 2, 2).contains(req.size)) + cfg.size = req.size; + else + cfg.size = raw.size; + + int ret = trySetPipelineConfig(raw, cfg, V4L2Subdevice::TryFormat); + if (ret < 0) + return {}; + + ret = trySetVideoCfg(output_.get(), cfg, false, true, "OPE output try format"); + if (ret < 0) + return {}; + + cfg.bufferCount = std::max(kBufferCount, req.bufferCount); + cfg.setStream(const_cast(&outStream_)); + + return cfg; +} + +int CamssIspOpe::configure(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg, + ControlInfoMap *ispControls) +{ + ipa::softisp::IPAConfigInfo configInfo; + configInfo.sensorControls = sensor_->controls(); + + int ret = ipa_->configure(configInfo, ispControls); + if (ret < 0) + return ret; + + ret = stats_->configure(inputCfg); + if (ret < 0) + return ret; + + /* Use 2/3 center of image to reduce CPU load */ + Rectangle statsWindow; + statsWindow.width = inputCfg.size.width * 2 / 3; + statsWindow.height = inputCfg.size.height * 2 / 3; + statsWindow.x = (inputCfg.size.width - statsWindow.width) / 2; + statsWindow.y = (inputCfg.size.height - statsWindow.height) / 2; + /* stats_->setWindow() takes care of necessary alignment itself */ + stats_->setWindow(statsWindow); + + ret = trySetVideoCfg(input_.get(), const_cast(inputCfg), + true, false, "OPE input set format"); + if (ret < 0) + return ret; + + ret = trySetPipelineConfig(inputCfg, outputCfg, V4L2Subdevice::ActiveFormat); + if (ret < 0) + return ret; + + ret = trySetVideoCfg(output_.get(), const_cast(outputCfg), + true, false, "OPE output set format"); + if (ret < 0) + return ret; + + inputBufferCount_ = inputCfg.bufferCount; + outputBufferCount_ = outputCfg.bufferCount; + + return 0; +} + +int CamssIspOpe::allocateBuffers(unsigned int bufferCount) +{ + int ret; + + ret = input_->importBuffers(inputBufferCount_); + if (ret < 0) + return ret; + + ret = output_->importBuffers(outputBufferCount_); + if (ret < 0) { + freeBuffers(); + return ret; + } + + ret = params_->allocateBuffers(bufferCount, ¶mBuffers_); + if (ret < 0) { + freeBuffers(); + return ret; + } + + /* Map buffers to the IPA. */ + unsigned int ipaBufferId = 1; + + /* Lambda function for future re-use with statsBuffers. */ + auto pushBuffers = [&](const std::vector> &buffers) { + for (const std::unique_ptr &buffer : buffers) { + std::span planes = buffer->planes(); + + buffer->setCookie(ipaBufferId++); + ipaBuffers_.emplace_back(buffer->cookie(), + std::vector{ planes.begin(), + planes.end() }); + } + }; + + pushBuffers(paramBuffers_); + + /* FIXME drop once using own camss IPA, which will map on IPA side */ + for (const IPABuffer &buffer : ipaBuffers_) { + const FrameBuffer fb(buffer.planes); + mappedIpaBuffers_.emplace(buffer.id, + MappedFrameBuffer(&fb, MappedFrameBuffer::MapFlag::ReadWrite)); + } + + frameInfos_->init(paramBuffers_, {}); + return 0; +} + +void CamssIspOpe::freeBuffers() +{ + frameInfos_->clear(); + + /* FIXME drop once using own camss IPA, which will map on IPA side */ + mappedIpaBuffers_.clear(); + + ipaBuffers_.clear(); + paramBuffers_.clear(); + params_->releaseBuffers(); + output_->releaseBuffers(); + input_->releaseBuffers(); +} + +int CamssIspOpe::exportOutputBuffers([[maybe_unused]] const Stream *stream, unsigned int count, + std::vector> *buffers) +{ + return output_->exportBuffers(count, buffers); +} + +void CamssIspOpe::queueBuffers(Request *request, FrameBuffer *inputBuffer) +{ + uint32_t frame = request->sequence(); + + DmaSyncer inputDmaSync(inputBuffer->planes()[0].fd, DmaSyncer::SyncType::Read); + + MappedFrameBuffer input(inputBuffer, MappedFrameBuffer::MapFlag::Read); + if (!input.isValid()) { + LOG(Camss, Error) << "mmap-ing buffer(s) failed"; + return; + } + + ipa_->queueRequest(frame, request->controls()); + stats_->processFrame(frame, 0, input); + ipa_->computeParams(frame); + /* paramsComputed() will queue all the buffers once the params are known. */ +} + +void CamssIspOpe::paramsComputed(uint32_t frame) +{ + CamssFrames::Info *info = frameInfos_->find(frame); + if (!info) + return; + + FrameBuffer *outputBuffer = nullptr; + for (const auto &[stream, buffer] : info->request->buffers()) { + if (stream == &outStream_) + outputBuffer = buffer; + } + if (!outputBuffer) { + LOG(Camss, Error) << "Missing output buffer"; + return; + } + + /* + * FIXME drop once using own camss IPA, which will do this on IPA side. + * IPA paramsComputed signal should also pass a bytesused value back. + */ + uint32_t bufferId = info->paramBuffer->cookie(); + ipa::camss::CamssParams params(mappedIpaBuffers_.at(bufferId).planes()[0]); + + auto awbGains = params.block(); + DebayerParams debayerParams = *sharedParams_; + /* Convert gains to 15uQ10 and store */ + awbGains->r_gain = std::clamp(debayerParams.gains.r(), 0.0, 31.0) * 1024; + awbGains->g_gain = std::clamp(debayerParams.gains.g(), 0.0, 31.0) * 1024; + awbGains->b_gain = std::clamp(debayerParams.gains.b(), 0.0, 31.0) * 1024; + /* Convert blacklevel to 0-65535 (u16) and store */ + awbGains->r_sub = debayerParams.blackLevel.r() * 65535; + awbGains->g_sub = debayerParams.blackLevel.g() * 65535; + awbGains->b_sub = debayerParams.blackLevel.b() * 65535; + /* Unused */ + awbGains->r_add = 0; + awbGains->g_add = 0; + awbGains->b_add = 0; + + /* + * a[] contains green input contributions to GBR output + * b[] contains blue input contributions to GBR output + * c[] contains red input contributions to GBR output + * k[] contains offset for GBR output + */ + auto colorCorrect = params.block(); + colorCorrect->qfactor = 1; /* Select 12sQ8 format for matrix values */ + colorCorrect->a[0] = std::clamp(debayerParams.combinedMatrix[1][1], -8.0f, 7.9f) * 256; + colorCorrect->a[1] = std::clamp(debayerParams.combinedMatrix[2][1], -8.0f, 7.9f) * 256; + colorCorrect->a[2] = std::clamp(debayerParams.combinedMatrix[0][1], -8.0f, 7.9f) * 256; + colorCorrect->b[0] = std::clamp(debayerParams.combinedMatrix[1][2], -8.0f, 7.9f) * 256; + colorCorrect->b[1] = std::clamp(debayerParams.combinedMatrix[2][2], -8.0f, 7.9f) * 256; + colorCorrect->b[2] = std::clamp(debayerParams.combinedMatrix[0][2], -8.0f, 7.9f) * 256; + colorCorrect->c[0] = std::clamp(debayerParams.combinedMatrix[1][0], -8.0f, 7.9f) * 256; + colorCorrect->c[1] = std::clamp(debayerParams.combinedMatrix[2][0], -8.0f, 7.9f) * 256; + colorCorrect->c[2] = std::clamp(debayerParams.combinedMatrix[0][0], -8.0f, 7.9f) * 256; + colorCorrect->k[0] = 0; + colorCorrect->k[1] = 0; + colorCorrect->k[2] = 0; + + info->paramBuffer->_d()->metadata().planes()[0].bytesused = params.bytesused(); + params_->queueBuffer(info->paramBuffer); + output_->queueBuffer(outputBuffer); + input_->queueBuffer(info->rawBuffer); +} + +void CamssIspOpe::parameterBufferReady(FrameBuffer *f) +{ + CamssFrames::Info *info = frameInfos_->find(f); + if (!info) + return; + + info->paramDequeued = true; + + paramBufferReady.emit(f); +} + +void CamssIspOpe::processStats(const uint32_t frame, const uint32_t statsBufferId, + const ControlList &sensorControls) +{ + ipa_->processStats(frame, statsBufferId, sensorControls); +} + +int CamssIspOpe::start() +{ + int ret = ipa_->start(); + if (ret) + return ret; + + ret = input_->streamOn(); + if (ret < 0) { + stop(); + return ret; + } + + ret = output_->streamOn(); + if (ret < 0) { + stop(); + return ret; + } + + ret = params_->streamOn(); + if (ret < 0) { + stop(); + return ret; + } + + return 0; +} + +void CamssIspOpe::stop() +{ + params_->streamOff(); + output_->streamOff(); + input_->streamOff(); + ipa_->stop(); +} + +/** + * \brief Match media devices and create OPE instance if found + * \param[in] pipe The pipeline handler in use + * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline + * \param[in] frameInfos Pointer to CamssFrames instance owned by the pipeline + * \param[out] ControlInfoMap to which to add ISP provided controls + */ +std::unique_ptr CamssIspOpe::match(PipelineHandler *pipe, + DeviceEnumerator *enumerator, + const CameraSensor *sensor, + CamssFrames *frameInfos, + ControlInfoMap *ispControls) +{ + std::shared_ptr opeMediaDev; + DeviceMatch opeDm("qcom-camss-ope"); + + opeDm.add("ope_params"); + opeDm.add("ope_input"); + opeDm.add("ope_proc"); + opeDm.add("ope_disp"); + opeDm.add("ope_disp_output"); + + opeMediaDev = pipe->acquireMediaDevice(enumerator, opeDm); + if (!opeMediaDev) { + LOG(Camss, Info) << "No OPE match for " << sensor->entity()->name(); + return nullptr; + } + + std::unique_ptr ope = + std::make_unique(pipe, std::move(opeMediaDev), sensor, + frameInfos, ispControls); + + if (ope->init() != 0) + return nullptr; + + LOG(Camss, Info) << "Using OPE for " << sensor->entity()->name(); + + return ope; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/camss/camss_isp_ope.h b/src/libcamera/pipeline/camss/camss_isp_ope.h new file mode 100644 index 000000000..b2134b00e --- /dev/null +++ b/src/libcamera/pipeline/camss/camss_isp_ope.h @@ -0,0 +1,120 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Qualcomm CAMSS OPE ISP class + * + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + */ + +#pragma once + +#include +#include + +#include +#include + +/* FIXME drop once using own camss IPA, which will map on IPA side */ +#include "libcamera/internal/mapped_framebuffer.h" +#include "libcamera/internal/shared_mem_object.h" +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/v4l2_subdevice.h" + +#include "camss_isp.h" + +namespace libcamera { + +class CameraSensor; +class CamssFrames; +class ControlInfoMap; +class DeviceEnumerator; +class MediaDevice; +class PipelineHandler; +class Stream; +class SwStatsCpu; +class V4L2VideoDevice; + +class CamssIspOpe : public CamssIsp +{ +public: + static std::unique_ptr match(PipelineHandler *pipe, + DeviceEnumerator *enumerator, + const CameraSensor *sensor, + CamssFrames *frameInfos, + ControlInfoMap *ispControls); + + bool isValid() override; + Size getMargins(PixelFormat inputFormat) override; + + CamssIspOpe(PipelineHandler *pipe, std::shared_ptr opeMediaDev, + const CameraSensor *sensor, CamssFrames *frameInfos, + ControlInfoMap *ispControls); + ~CamssIspOpe() override; + + StreamConfiguration generateConfiguration(const StreamConfiguration &raw) const override; + StreamConfiguration validate(const StreamConfiguration &raw, const StreamConfiguration &req) const override; + int configure(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg, + ControlInfoMap *ispControls) override; + + int allocateBuffers(unsigned int bufferCount) override; + void freeBuffers() override; + int exportOutputBuffers(const Stream *stream, unsigned int count, + std::vector> *buffers) override; + void queueBuffers(Request *request, FrameBuffer *input) override; + + void processStats(const uint32_t frame, const uint32_t statsBufferId, + const ControlList &sensorControls) override; + + void paramsComputed(uint32_t frame); + void parameterBufferReady(FrameBuffer *f); + + int start() override; + void stop() override; + +private: + static constexpr Size kMinOutputSize = Size(24, 16); + static constexpr unsigned int PROC_PAD_INPUT = 0; + static constexpr unsigned int PROC_PAD_PARAMS = 1; + static constexpr unsigned int PROC_PAD_DISP = 2; + static constexpr unsigned int DISP_PAD_PROC = 0; + static constexpr unsigned int DISP_PAD_OUTPUT = 1; + + int init(); + int trySetVideoCfg(V4L2VideoDevice *v4l2Dev, StreamConfiguration &cfg, + bool set, bool update_stride_and_size, + const char *msgPrefix) const; + int trySetSubdevFormat(V4L2Subdevice *subdev, + unsigned int pad, + const V4L2SubdeviceFormat &fmt, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const; + int trySetSubdevSelection(V4L2Subdevice *subdev, + unsigned int pad, + unsigned int target, + const Rectangle &rect, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const; + int trySetPipelineConfig(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg, + V4L2Subdevice::Whence whence) const; + + std::shared_ptr opeMediaDev_; + const CameraSensor *sensor_; + class CamssFrames *frameInfos_; + SharedMemObject sharedParams_; + std::unique_ptr stats_; + std::unique_ptr proc_; + std::unique_ptr disp_; + std::unique_ptr params_; + std::unique_ptr input_; + std::unique_ptr output_; + std::unique_ptr ipa_; + std::vector> paramBuffers_; + std::vector ipaBuffers_; + /* FIXME drop once using own camss IPA, which will map on IPA side */ + std::map mappedIpaBuffers_; + unsigned int inputBufferCount_; + unsigned int outputBufferCount_; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/camss/camss_params.h b/src/libcamera/pipeline/camss/camss_params.h index 51e6e639c..9244de7e0 100644 --- a/src/libcamera/pipeline/camss/camss_params.h +++ b/src/libcamera/pipeline/camss/camss_params.h @@ -22,7 +22,7 @@ namespace libcamera { namespace ipa::camss { -enum class CamssBlocks { +enum class CamssBlocks : uint16_t { AwbGains, ChromaEnh, ColorCorrect, diff --git a/src/libcamera/pipeline/camss/meson.build b/src/libcamera/pipeline/camss/meson.build index 518d85478..88a70546d 100644 --- a/src/libcamera/pipeline/camss/meson.build +++ b/src/libcamera/pipeline/camss/meson.build @@ -5,7 +5,10 @@ libcamera_internal_sources += files([ 'camss_csi.cpp', 'camss_frames.cpp', 'camss_isp.cpp', + 'camss_isp_ope.cpp', 'camss_isp_raw.cpp', 'camss_isp_soft.cpp', 'camss_util.cpp', + # FIXME hack + '../../../ipa/libipa/v4l2_params.cpp' ])