From patchwork Tue Jul 21 14:59:57 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 27431 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6036BBDE4C for ; Tue, 21 Jul 2026 15:00:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0CDD967E78; Tue, 21 Jul 2026 17:00:19 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=qualcomm.com header.i=@qualcomm.com header.b="aY4pvB1O"; dkim=pass (2048-bit key; unprotected) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b="h+HQHBUL"; dkim-atps=neutral Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 047E967E72 for ; Tue, 21 Jul 2026 17:00:16 +0200 (CEST) Received: from pps.filterd (m0279870.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66LDfwtK1912530 for ; Tue, 21 Jul 2026 15:00:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h= cc:content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=qcppdkim1; bh=iBY8Jj61JpQ 7zLskBS7JWNts/YX873WTeG1EgV3TanE=; b=aY4pvB1OHe/WYH3xrTFvFsCXcvC L7zXv1WrTiS6KuAnuCJ6MJY4YbJYhS7E3icrygRk+zPoHirEiC21mF8v/WNaag4T VijMT6eArkLtkZYUufCZM3KaAlTNFrBsKIbSMEutr1Ye84/qQPUiUSbhaN5fl5R8 gTpUMWQ+ipIHhEiJkDmaMNYPOPC4RKtzx8bfJQ325zQaP2rFMYzAOLIiwzccLgbq HRxxcAb4u5GofiEpS0qx0TDa88cTUA9fsVb2QpZR8W5Tc8awG9/8ARVueHTF73IL gCryPtJv7rvprh0PBr/HD3wUySVQtAFA+dJTr42DN2T18uwvB+1bryclLqw== Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4fj1duk0k7-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Tue, 21 Jul 2026 15:00:15 +0000 (GMT) Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-51c075c1e25so163191061cf.3 for ; Tue, 21 Jul 2026 08:00:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oss.qualcomm.com; s=google; t=1784646015; x=1785250815; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iBY8Jj61JpQ7zLskBS7JWNts/YX873WTeG1EgV3TanE=; b=h+HQHBUL0RqYyrF66XQD/XsoEd3kRl5u0MmUOMreHhS4LmCHv5Tj7mmn2B53u8o4AS /nREOwwz3VmWWpvniabM8N4kFj6sU4xGjJzlbRh9pR4H4ZAJjeeBvVGP653al9oYqwVE VBhmpCrd665U3o00xjjbN3uum2Qr0+jMmg0Dshwom2uHgIeErMYbRPtB9IfnrbalzROQ 1BMjs2gz4cLVStilnGWfw7OtHSyKMwbtvJnTuq17jRZ8TyAgzSO6Q8C8IzlIidvFmmne lbF8DtYLy1KCS8Di3osdvGcVrU4GQkHBzi88p4zHYMOaOpnq231DZpUvW9agwO9TP2PP 54TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784646015; x=1785250815; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iBY8Jj61JpQ7zLskBS7JWNts/YX873WTeG1EgV3TanE=; b=hxsURIYRkxPTwzpp1xnr7JAgZy5dA+A3pCEKZ4+YbMGrQesd3REABq2NhvDTXm6HiI bqHXdfAl0S/NdhrudZpnQzQJRTv7y96f0IYe4SvTwcLMw3u+MaGV3M+ZsWH42eMjm6dL a/fDj90+/pSXcmjtALvtQxFXcGItqJQ2Cg6rOcZYaNMQREGnvg/byaNFI1Y2xGWM1CcX dnvSXLck6Z/FO5BViKBWDMWTbqWpdwnXSiPTIS36hvItr90Nsgj2LP2tuiS3sjh4K38t 1wFMnfi6l/YBnECMMcoIZUxuNB6ZBEDzenbBPMbqVDTb2Nv4ybrAWuZVgJ/Wgg8VDsk8 O2uQ== X-Gm-Message-State: AOJu0YyI39sNo2f9cMgX0K6fyGV7tUpAC7pWUg9t7ffI8k7fOy3D7TOF FOJjLliYDnZe683SPBjEgIo12AwEim8Pu4Ap+/diYjwGiHW/+cnycBBRmrE45Am32DR5qPtE7lK EbL5PvsyhKTKwzgFY1yGSqmKsExrfesB+wAugEZwIuh++OIGbv/6Od7KmCv1vyb60Ctja0KiGiM Kr08PdIQMl X-Gm-Gg: AfdE7clE9WUchXpGkYXXaTFfRbGfv80SwEoSHnRe2vIeBvzFAL921GnUdRtc8Td3ynw 5ceXi0qifpRBWgiP47CU51jlzc1SMovNlc1OSsFZpFVjwug6Im8z/T3ODvhlNHwOaPB1Xne4sDo pLsiAAiKf+YRwMFd5sIMpUKxTsCFkD6AH9cXuHWS41v/lMsdDjxG3sf110/jF7LmDfeCiD7qWyW oXiK73XWkPd8qMaW7V6Hb1TGdt3nKXSYPF/a8676N8KWB1Ltz8ii350Iv1v1HYqjVlVCqG4cics fhRkPkhK/X0RC8osJT4UzBBSSMEQI1K7jH1lK5ilyILhCeNB116KISLfMZsJgC8GmIEooLiIrDA 4KvMklATSDgwcltkEUnEeFlAJbLR8v1oUKIVEV+s7cBBfntOFhOdlRBkGlBB2K6BefIQbIuS5wn RxgEwbJfR2uL1n09o= X-Received: by 2002:a05:622a:1ba2:b0:51c:6e:ef23 with SMTP id d75a77b69052e-5213c9e3db3mr177448231cf.32.1784646012314; Tue, 21 Jul 2026 08:00:12 -0700 (PDT) X-Received: by 2002:a05:622a:1ba2:b0:51c:6e:ef23 with SMTP id d75a77b69052e-5213c9e3db3mr177445831cf.32.1784646010484; Tue, 21 Jul 2026 08:00:10 -0700 (PDT) Received: from shalem (2001-1c00-0c32-7800-5bfa-a036-83f0-f9ec.cable.dynamic.v6.ziggo.nl. [2001:1c00:c32:7800:5bfa:a036:83f0:f9ec]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c1712a5f837sm615968066b.32.2026.07.21.08.00.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jul 2026 08:00:09 -0700 (PDT) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Cc: Loic Poulain , Hans de Goede Subject: [PATCH v2 7/7] camss: Add Offline Processing Engine ISP support Date: Tue, 21 Jul 2026 16:59:57 +0200 Message-ID: <20260721145958.68489-8-johannes.goede@oss.qualcomm.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260721145958.68489-1-johannes.goede@oss.qualcomm.com> References: <20260721145958.68489-1-johannes.goede@oss.qualcomm.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: ib5jDCn2Hczbq5p99WW5TufhQDw1XYnN X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzIxMDE1OCBTYWx0ZWRfX0BcZoi8n/r3f QSPfa5rw2rltU9N5luq+gHWqEyDnBGRZ1MdSbQ/t4W2o+AToLha+hQeRVNNdviwhX4bg4BcKfni gTKFzUCdH78vLE1LgwkN9794xkEBbH8qErpkM7iKWc+xJUJ8MOpDzdrWoC9Sy39bBskypR0zpGf mbbBZmP7qcYnGWjB1LiKYHBQffeNibDOFon3EksorkbyubISUEsKZls6RM5rYY6ZttRFHgWI7Vo z0eU0ETrWPl1w9dm5ee6AEWkF5cHeUqDMDrnGDD3vWBZenatKWYSRHvUhVs9mkv0mRKqAg1XXVu 6HZ5KyUz59P+0mhYJYkjyShaHjVNxI43CBO9LhuBDXzk5q2/BKBfO2n+BAzKgoztYKhTHm9trwp XxSGizbCGB+jkgTyPelX5ilzTA9SPuaTRc5Bz99jzuanXIQcAHi2aZhQHwVjDCCNGrItQ3wEqR+ VUux9xkR9+MMO5IEbCw== X-Authority-Analysis: v=2.4 cv=I9JVgtgg c=1 sm=1 tr=0 ts=6a5f897f cx=c_pps a=mPf7EqFMSY9/WdsSgAYMbA==:117 a=xqWC_Br6kY4A:10 a=RAioF0-LDSMA:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=u7WPNUs3qKkmUXheDGA7:22 a=gowsoOTTUOVcmtlkKump:22 a=VwQbUJbxAAAA:8 a=EUspDBNiAAAA:8 a=ISZBYy2OhyTthe2ftDAA:9 a=dawVfQjAaf238kedN5IG:22 X-Proofpoint-GUID: ib5jDCn2Hczbq5p99WW5TufhQDw1XYnN X-Proofpoint-Spam-Info: AW1haW4tMjYwNzIxMDE1OCBTYWx0ZWRfX3k1IkMrien8y zrpwEDxqMzLvTdbtp9nh7mcmiJ/e9xns2yOZRE5IGVWR5orH57f/mWykAgRwrnuyw/3Z4n0BkJ+ uZ8HNOaowCxFC8AlrottFwpVau5no/E= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293, Aquarius:18.0.1143, Hydra:6.1.134, FMLib:17.12.100.49 definitions=2026-07-21_03,2026-07-20_03,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 adultscore=0 spamscore=0 clxscore=1015 suspectscore=0 lowpriorityscore=0 phishscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607210158 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This adds support for the OPE through the v3 kernel driver series posted upstream recently (See Link tag). This is still in the proof of concept phase atm. With the biggest todo item being this using the softISP simple IPA atm, while it really should come with its own IPA. Link: https://lore.kernel.org/linux-media/20260508-camss-isp-ope-v3-0-bb1055274603@oss.qualcomm.com/ Signed-off-by: Hans de Goede --- src/libcamera/pipeline/camss/camss.cpp | 19 +- .../pipeline/camss/camss_isp_ope.cpp | 679 ++++++++++++++++++ src/libcamera/pipeline/camss/camss_isp_ope.h | 120 ++++ src/libcamera/pipeline/camss/camss_params.h | 2 +- src/libcamera/pipeline/camss/meson.build | 3 + 5 files changed, 816 insertions(+), 7 deletions(-) create mode 100644 src/libcamera/pipeline/camss/camss_isp_ope.cpp create mode 100644 src/libcamera/pipeline/camss/camss_isp_ope.h diff --git a/src/libcamera/pipeline/camss/camss.cpp b/src/libcamera/pipeline/camss/camss.cpp index 371d4f8a2..6b53bdacd 100644 --- a/src/libcamera/pipeline/camss/camss.cpp +++ b/src/libcamera/pipeline/camss/camss.cpp @@ -38,6 +38,7 @@ #include "camss_csi.h" #include "camss_frames.h" #include "camss_isp.h" +#include "camss_isp_ope.h" #include "camss_isp_soft.h" namespace libcamera { @@ -522,12 +523,18 @@ bool PipelineHandlerCamss::match(DeviceEnumerator *enumerator) data->delayedCtrls_ = std::make_unique(sensor->device(), params); - data->isp_ = std::make_unique(this, sensor, - &data->frameInfos_, - &data->controlInfo_); - if (!data->isp_->isValid()) { - LOG(Camss, Error) << "Failed to create software ISP"; - continue; + data->isp_ = CamssIspOpe::match(this, enumerator, sensor, + &data->frameInfos_, + &data->controlInfo_); + if (data->isp_ == nullptr) { + data->isp_ = + std::make_unique(this, sensor, + &data->frameInfos_, + &data->controlInfo_); + if (!data->isp_->isValid()) { + LOG(Camss, Error) << "Failed to create software ISP"; + continue; + } } data->isp_->inputBufferReady.connect(data->csi_.get(), diff --git a/src/libcamera/pipeline/camss/camss_isp_ope.cpp b/src/libcamera/pipeline/camss/camss_isp_ope.cpp new file mode 100644 index 000000000..454c72795 --- /dev/null +++ b/src/libcamera/pipeline/camss/camss_isp_ope.cpp @@ -0,0 +1,679 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Qualcomm CAMSS OPE ISP class + * + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + */ + +#include "camss_isp_ope.h" + +#include + +#include + +#include +#include +#include +#include +#include + +#include "libcamera/internal/camera_manager.h" +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/dma_buf_allocator.h" +#include "libcamera/internal/framebuffer.h" // FIXME drop once softIPA (ab)use is gone +#include "libcamera/internal/ipa_manager.h" +#include "libcamera/internal/media_device.h" +#include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swstats_cpu.h" +#include "libcamera/internal/v4l2_subdevice.h" +#include "libcamera/internal/v4l2_videodevice.h" + +#include "camss_frames.h" +#include "camss_params.h" +#include "camss_util.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(Camss) + +namespace { + +static const std::map inputFormatToMediaBus = { + { formats::SBGGR8, MEDIA_BUS_FMT_SBGGR8_1X8 }, + { formats::SGBRG8, MEDIA_BUS_FMT_SGBRG8_1X8 }, + { formats::SGRBG8, MEDIA_BUS_FMT_SGRBG8_1X8 }, + { formats::SRGGB8, MEDIA_BUS_FMT_SRGGB8_1X8 }, + { formats::SBGGR10_CSI2P, MEDIA_BUS_FMT_SBGGR10_1X10 }, + { formats::SGBRG10_CSI2P, MEDIA_BUS_FMT_SGBRG10_1X10 }, + { formats::SGRBG10_CSI2P, MEDIA_BUS_FMT_SGRBG10_1X10 }, + { formats::SRGGB10_CSI2P, MEDIA_BUS_FMT_SRGGB10_1X10 }, +}; + +static const std::map outputFormatToMediaBus = { + { formats::NV12, MEDIA_BUS_FMT_YUYV8_1_5X8 }, + { formats::NV21, MEDIA_BUS_FMT_YVYU8_1_5X8 }, + { formats::NV16, MEDIA_BUS_FMT_YUYV8_2X8 }, + { formats::NV61, MEDIA_BUS_FMT_YUYV8_2X8 }, + { formats::NV24, MEDIA_BUS_FMT_YUV8_1X24 }, + { formats::NV42, MEDIA_BUS_FMT_YUV8_1X24 }, + { formats::R8, MEDIA_BUS_FMT_Y8_1X8 }, +}; + +} /* namespace */ + +/** + * \class CamssIspOpe + * \brief CAMSS ISP class for the Offline Processing Engine (OPE) ISP + */ + +/** + * \brief Constructs CamssIspOpe object + * \param[in] pipe The pipeline handler in use + * \param[in] opeMediaDev The OPE MediaDevice + * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline + * \param[in] frameInfos Pointer to CamssFrames instance owned by the pipeline + * \param[out] ControlInfoMap to which to add ISP provided controls + */ +CamssIspOpe::CamssIspOpe(PipelineHandler *pipe, std::shared_ptr opeMediaDev, + const CameraSensor *sensor, CamssFrames *frameInfos, + ControlInfoMap *ispControls) + : opeMediaDev_(std::move(opeMediaDev)), sensor_(sensor), frameInfos_(frameInfos) +{ + sharedParams_ = SharedMemObject("softIsp_params"); + if (!sharedParams_) { + LOG(Camss, Error) << "Failed to create shared memory for parameters"; + return; + } + + const CameraManager &cm = *pipe->cameraManager(); + + stats_ = std::make_unique(cm); + if (!stats_->isValid()) { + LOG(Camss, Error) << "Failed to create SwStatsCpu object"; + return; + } + + stats_->statsReady.connect(this, + [&](uint32_t frame, uint32_t statsBufferId) { + statsReady.emit(frame, statsBufferId); + }); + + ipa_ = pipe->createIPA("simple", 0, 0); + if (!ipa_) { + LOG(Camss, Error) << "Creating IPA failed"; + return; + } + + /* + * The API tuning file is made from the sensor name. If the tuning file + * isn't found, fall back to the 'uncalibrated' file. + */ + std::string ipaTuningFile = + ipa_->configurationFile(sensor->model() + ".yaml", "uncalibrated.yaml"); + + IPACameraSensorInfo sensorInfo{}; + int ret = sensor->sensorInfo(&sensorInfo); + if (ret) { + LOG(Camss, Error) << "Camera sensor information not available"; + ipa_.reset(); + return; + } + + bool ccmEnabled; + ret = ipa_->init(IPASettings{ ipaTuningFile, sensor->model() }, + stats_->getStatsFD(), + sharedParams_.fd(), + sensorInfo, + sensor->controls(), + ispControls, + &ccmEnabled); + if (ret) { + LOG(Camss, Error) << "IPA init failed"; + ipa_.reset(); + return; + } + + ipa_->paramsComputed.connect(this, &CamssIspOpe::paramsComputed); + ipa_->metadataReady.connect(this, + [&](uint32_t frame, const ControlList &metadata) { + metadataReady.emit(frame, metadata); + }); + ipa_->setSensorControls.connect(this, + [&](const ControlList &sensorControls) { + setSensorControls.emit(sensorControls); + }); +} + +CamssIspOpe::~CamssIspOpe() = default; + +bool CamssIspOpe::isValid() +{ + return !!ipa_; +} + +Size CamssIspOpe::getMargins([[maybe_unused]] PixelFormat inputFormat) +{ + /* TODO: Do we really not need any margins on OPE? */ + return Size(); +} + +int CamssIspOpe::init() +{ + MediaEntity *paramsEnt, *inputEnt, *procEnt, *dispEnt, *dispOutputEnt; + int ret; + + if (!isValid()) + return -ENOMEM; + + paramsEnt = opeMediaDev_->getEntityByName("ope_params"); + inputEnt = opeMediaDev_->getEntityByName("ope_input"); + procEnt = opeMediaDev_->getEntityByName("ope_proc"); + dispEnt = opeMediaDev_->getEntityByName("ope_disp"); + dispOutputEnt = opeMediaDev_->getEntityByName("ope_disp_output"); + + if (!paramsEnt || !inputEnt || !procEnt || !dispEnt || !dispOutputEnt) { + LOG(Camss, Error) << "Did not find expected entities"; + return -EINVAL; + } + + params_ = std::make_unique(paramsEnt); + ret = params_->open(); + if (ret) + return ret; + + params_->bufferReady.connect(this, &CamssIspOpe::parameterBufferReady); + + input_ = std::make_unique(inputEnt); + ret = input_->open(); + if (ret) + return ret; + + input_->bufferReady.connect(this, [&](FrameBuffer *f) { + inputBufferReady.emit(f); + }); + + proc_ = std::make_unique(procEnt); + ret = proc_->open(); + if (ret) + return ret; + + disp_ = std::make_unique(dispEnt); + ret = disp_->open(); + if (ret) + return ret; + + output_ = std::make_unique(dispOutputEnt); + ret = output_->open(); + if (ret) + return ret; + + output_->bufferReady.connect(this, [&](FrameBuffer *f) { + outputBufferReady.emit(f); + }); + + return 0; +} + +/* Note cfg is const / not modified when update_stride_and_size is false */ +int CamssIspOpe::trySetVideoCfg(V4L2VideoDevice *v4l2Dev, StreamConfiguration &cfg, + bool set, bool update_stride_and_size, + const char *msgPrefix) const +{ + int ret; + + V4L2DeviceFormat v4l2Format; + v4l2Format.fourcc = v4l2Dev->toV4L2PixelFormat(cfg.pixelFormat); + v4l2Format.size = cfg.size; + v4l2Format.planes[0].bpl = cfg.stride; + v4l2Format.planesCount = 1; + + if (set) + ret = v4l2Dev->setFormat(&v4l2Format); + else + ret = v4l2Dev->tryFormat(&v4l2Format); + + if (ret < 0) { + LOG(Camss, Error) << msgPrefix << " error: " << strerror(-ret); + return ret; + } + + if (update_stride_and_size) { + cfg.stride = v4l2Format.planes[0].bpl; + cfg.frameSize = v4l2Format.planes[0].size; + } + + if (!camssV4L2DeviceFormatMatchesStreamConfig(v4l2Format, cfg, msgPrefix)) + return -EINVAL; + + return 0; +} + +StreamConfiguration CamssIspOpe::generateConfiguration(const StreamConfiguration &raw) const +{ + if (!inputFormatToMediaBus.count(raw.pixelFormat)) { + LOG(Camss, Warning) << "Unsupported pixel format " << raw.pixelFormat; + return {}; + } + + if (trySetVideoCfg(input_.get(), const_cast(raw), + false, false, "OPE input try format")) + return {}; + + /* OPE always supports all output formats */ + std::vector sizesVector = { SizeRange(kMinOutputSize, raw.size, 2, 2) }; + std::map> formats; + for (auto it = outputFormatToMediaBus.begin(); it != outputFormatToMediaBus.end(); it++) + formats[it->first] = sizesVector; + + StreamConfiguration cfg{ StreamFormats{ formats } }; + cfg.size = raw.size; + cfg.pixelFormat = formats::NV12; /* Default to NV12 output */ + cfg.bufferCount = kBufferCount; + + return cfg; +} + +int CamssIspOpe::trySetSubdevFormat(V4L2Subdevice *subdev, + unsigned int pad, + const V4L2SubdeviceFormat &fmt, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const +{ + V4L2SubdeviceFormat subdevFormat = fmt; + + int ret = subdev->setFormat(pad, &subdevFormat, whence); + if (ret < 0) { + LOG(Camss, Error) << msgPrefix << " try/set format error: " << strerror(-ret); + return ret; + } + + if (subdevFormat.size != fmt.size || subdevFormat.code != fmt.code) { + LOG(Camss, Error) << msgPrefix << " mismatch asked " + << fmt.size << "/" << fmt.code << " got " + << subdevFormat.size << "/" << subdevFormat.code; + return -EINVAL; + } + + return 0; +} + +int CamssIspOpe::trySetSubdevSelection(V4L2Subdevice *subdev, + unsigned int pad, + unsigned int target, + const Rectangle &rect, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const +{ + struct Rectangle r = rect; + + int ret = subdev->setSelection(pad, target, &r, whence); + if (ret < 0) { + LOG(Camss, Error) << msgPrefix << " set selection error: " << strerror(-ret); + return ret; + } + + if (r != rect) { + LOG(Camss, Error) << msgPrefix << " mismatch asked " + << rect << " got " << r; + return -EINVAL; + } + + return 0; +} + +int CamssIspOpe::trySetPipelineConfig(const StreamConfiguration &inputCfg, + [[maybe_unused]] const StreamConfiguration &outputCfg, + V4L2Subdevice::Whence whence) const +{ + if (!inputFormatToMediaBus.count(inputCfg.pixelFormat)) { + LOG(Camss, Warning) << "Unsupported pixel format " << inputCfg.pixelFormat; + return -EINVAL; + } + + V4L2SubdeviceFormat subdevFormat; + subdevFormat.code = inputFormatToMediaBus.at(inputCfg.pixelFormat); + subdevFormat.size = inputCfg.size; + + int ret = trySetSubdevFormat(proc_.get(), PROC_PAD_INPUT, subdevFormat, + whence, "OPE proc input"); + if (ret < 0) + return ret; + + /* + * No input-crop, downscale instead to keep full field of view. + * \todo crop in x or y dimension to match requested output aspect, + * to keep square pixels after scaling. + */ + ret = trySetSubdevSelection(proc_.get(), PROC_PAD_INPUT, V4L2_SEL_TGT_CROP, + Rectangle(inputCfg.size), whence, "OPE input crop"); + if (ret < 0) + return ret; + + ret = trySetSubdevSelection(proc_.get(), PROC_PAD_DISP, V4L2_SEL_TGT_COMPOSE, + Rectangle(outputCfg.size), whence, "OPE output compose"); + if (ret < 0) + return ret; + + subdevFormat.code = outputFormatToMediaBus.at(outputCfg.pixelFormat); + subdevFormat.size = outputCfg.size; + + ret = trySetSubdevFormat(proc_.get(), PROC_PAD_DISP, subdevFormat, + whence, "OPE proc disp out"); + if (ret < 0) + return ret; + + ret = trySetSubdevFormat(disp_.get(), DISP_PAD_PROC, subdevFormat, + whence, "OPE disp proc in"); + if (ret < 0) + return ret; + + ret = trySetSubdevFormat(disp_.get(), DISP_PAD_OUTPUT, subdevFormat, + whence, "OPE disp out"); + if (ret < 0) + return ret; + + return 0; +} + +StreamConfiguration CamssIspOpe::validate(const StreamConfiguration &raw, const StreamConfiguration &req) const +{ + if (trySetVideoCfg(input_.get(), const_cast(raw), + false, false, "OPE input try format")) + return {}; + + StreamConfiguration cfg; + + for (auto it = outputFormatToMediaBus.begin(); it != outputFormatToMediaBus.end(); it++) { + if (it->first == req.pixelFormat) + cfg.pixelFormat = req.pixelFormat; + } + + if (!cfg.pixelFormat.isValid()) + cfg.pixelFormat = formats::NV12; /* Default to NV12 output */ + + if (SizeRange(kMinOutputSize, raw.size, 2, 2).contains(req.size)) + cfg.size = req.size; + else + cfg.size = raw.size; + + int ret = trySetPipelineConfig(raw, cfg, V4L2Subdevice::TryFormat); + if (ret < 0) + return {}; + + ret = trySetVideoCfg(output_.get(), cfg, false, true, "OPE output try format"); + if (ret < 0) + return {}; + + cfg.bufferCount = std::max(kBufferCount, req.bufferCount); + cfg.setStream(const_cast(&outStream_)); + + return cfg; +} + +int CamssIspOpe::configure(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg) +{ + ipa::soft::IPAConfigInfo configInfo; + configInfo.sensorControls = sensor_->controls(); + + int ret = ipa_->configure(configInfo); + if (ret < 0) + return ret; + + ret = stats_->configure(inputCfg); + if (ret < 0) + return ret; + + /* Use 2/3 center of image to reduce CPU load */ + Rectangle statsWindow; + statsWindow.width = inputCfg.size.width * 2 / 3; + statsWindow.height = inputCfg.size.height * 2 / 3; + statsWindow.x = (inputCfg.size.width - statsWindow.width) / 2; + statsWindow.y = (inputCfg.size.height - statsWindow.height) / 2; + /* stats_->setWindow() takes care of necessary alignment itself */ + stats_->setWindow(statsWindow); + + ret = trySetVideoCfg(input_.get(), const_cast(inputCfg), + true, false, "OPE input set format"); + if (ret < 0) + return ret; + + ret = trySetPipelineConfig(inputCfg, outputCfg, V4L2Subdevice::ActiveFormat); + if (ret < 0) + return ret; + + ret = trySetVideoCfg(output_.get(), const_cast(outputCfg), + true, false, "OPE output set format"); + if (ret < 0) + return ret; + + inputBufferCount_ = inputCfg.bufferCount; + outputBufferCount_ = outputCfg.bufferCount; + + return 0; +} + +int CamssIspOpe::allocateBuffers(unsigned int bufferCount) +{ + int ret; + + ret = input_->importBuffers(inputBufferCount_); + if (ret < 0) + return ret; + + ret = output_->importBuffers(outputBufferCount_); + if (ret < 0) { + freeBuffers(); + return ret; + } + + ret = params_->allocateBuffers(bufferCount, ¶mBuffers_); + if (ret < 0) { + freeBuffers(); + return ret; + } + + /* Map buffers to the IPA. */ + unsigned int ipaBufferId = 1; + + /* Lambda function for future re-use with statsBuffers. */ + auto pushBuffers = [&](const std::vector> &buffers) { + for (const std::unique_ptr &buffer : buffers) { + Span planes = buffer->planes(); + + buffer->setCookie(ipaBufferId++); + ipaBuffers_.emplace_back(buffer->cookie(), + std::vector{ planes.begin(), + planes.end() }); + } + }; + + pushBuffers(paramBuffers_); + + /* FIXME drop once using own camss IPA, which will map on IPA side */ + for (const IPABuffer &buffer : ipaBuffers_) { + const FrameBuffer fb(buffer.planes); + mappedIpaBuffers_.emplace(buffer.id, + MappedFrameBuffer(&fb, MappedFrameBuffer::MapFlag::ReadWrite)); + } + + frameInfos_->init(paramBuffers_, {}, true); + return 0; +} + +void CamssIspOpe::freeBuffers() +{ + frameInfos_->clear(); + + /* FIXME drop once using own camss IPA, which will map on IPA side */ + mappedIpaBuffers_.clear(); + + ipaBuffers_.clear(); + paramBuffers_.clear(); + params_->releaseBuffers(); + output_->releaseBuffers(); + input_->releaseBuffers(); +} + +int CamssIspOpe::exportOutputBuffers([[maybe_unused]] const Stream *stream, unsigned int count, + std::vector> *buffers) +{ + return output_->exportBuffers(count, buffers); +} + +void CamssIspOpe::queueBuffers(Request *request, FrameBuffer *inputBuffer) +{ + uint32_t frame = request->sequence(); + + DmaSyncer inputDmaSync(inputBuffer->planes()[0].fd, DmaSyncer::SyncType::Read); + + MappedFrameBuffer input(inputBuffer, MappedFrameBuffer::MapFlag::Read); + if (!input.isValid()) { + LOG(Camss, Error) << "mmap-ing buffer(s) failed"; + return; + } + + ipa_->queueRequest(frame, request->controls()); + stats_->processFrame(frame, 0, input); + ipa_->computeParams(frame); + /* paramsComputed() will queue all the buffers once the params are known. */ +} + +void CamssIspOpe::paramsComputed(uint32_t frame) +{ + CamssFrames::Info *info = frameInfos_->find(frame); + if (!info) + return; + + FrameBuffer *outputBuffer = nullptr; + for (const auto &[stream, buffer] : info->request->buffers()) { + if (stream == &outStream_) + outputBuffer = buffer; + } + if (!outputBuffer) { + LOG(Camss, Error) << "Missing output buffer"; + return; + } + + /* + * FIXME drop once using own camss IPA, which will do this on IPA side. + * IPA paramsComputed signal should also pass a bytesused value back. + */ + uint32_t bufferId = info->paramBuffer->cookie(); + ipa::camss::CamssParams params(mappedIpaBuffers_.at(bufferId).planes()[0]); + auto awbGains = params.block(); + DebayerParams debayerParams = *sharedParams_; + /* Convert gains to 15uQ10 and store */ + awbGains->r_gain = std::clamp(debayerParams.gains.r(), 0.0f, 31.0f) * 1024; + awbGains->g_gain = std::clamp(debayerParams.gains.g(), 0.0f, 31.0f) * 1024; + awbGains->b_gain = std::clamp(debayerParams.gains.b(), 0.0f, 31.0f) * 1024; + /* Convert blacklevel to 0-65535 (u16) and store */ + awbGains->r_sub = debayerParams.blackLevel.r() * 65535; + awbGains->g_sub = debayerParams.blackLevel.g() * 65535; + awbGains->b_sub = debayerParams.blackLevel.b() * 65535; + /* Unused */ + awbGains->r_add = 0; + awbGains->g_add = 0; + awbGains->b_add = 0; + + info->paramBuffer->_d()->metadata().planes()[0].bytesused = params.bytesused(); + params_->queueBuffer(info->paramBuffer); + output_->queueBuffer(outputBuffer); + input_->queueBuffer(info->rawBuffer); +} + +void CamssIspOpe::parameterBufferReady(FrameBuffer *f) +{ + CamssFrames::Info *info = frameInfos_->find(f); + if (!info) + return; + + info->paramDequeued = true; + + paramBufferReady.emit(f); +} + +void CamssIspOpe::processStats(const uint32_t frame, const uint32_t statsBufferId, + const ControlList &sensorControls) +{ + ipa_->processStats(frame, statsBufferId, sensorControls); +} + +int CamssIspOpe::start() +{ + int ret = ipa_->start(); + if (ret) + return ret; + + ret = input_->streamOn(); + if (ret < 0) { + stop(); + return ret; + } + + ret = output_->streamOn(); + if (ret < 0) { + stop(); + return ret; + } + + ret = params_->streamOn(); + if (ret < 0) { + stop(); + return ret; + } + + return 0; +} + +void CamssIspOpe::stop() +{ + params_->streamOff(); + output_->streamOff(); + input_->streamOff(); + ipa_->stop(); +} + +/** + * \brief Match media devices and create OPE instance if found + * \param[in] pipe The pipeline handler in use + * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline + * \param[in] frameInfos Pointer to CamssFrames instance owned by the pipeline + * \param[out] ControlInfoMap to which to add ISP provided controls + */ +std::unique_ptr CamssIspOpe::match(PipelineHandler *pipe, + DeviceEnumerator *enumerator, + const CameraSensor *sensor, + CamssFrames *frameInfos, + ControlInfoMap *ispControls) +{ + std::shared_ptr opeMediaDev; + DeviceMatch opeDm("qcom-camss-ope"); + + opeDm.add("ope_params"); + opeDm.add("ope_input"); + opeDm.add("ope_proc"); + opeDm.add("ope_disp"); + opeDm.add("ope_disp_output"); + + opeMediaDev = pipe->acquireMediaDevice(enumerator, opeDm); + if (!opeMediaDev) { + LOG(Camss, Info) << "No OPE match for " << sensor->entity()->name(); + return nullptr; + } + + std::unique_ptr ope = + std::make_unique(pipe, std::move(opeMediaDev), sensor, + frameInfos, ispControls); + + if (ope->init() != 0) + return nullptr; + + LOG(Camss, Info) << "Using OPE for " << sensor->entity()->name(); + + return ope; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/camss/camss_isp_ope.h b/src/libcamera/pipeline/camss/camss_isp_ope.h new file mode 100644 index 000000000..8909f9bac --- /dev/null +++ b/src/libcamera/pipeline/camss/camss_isp_ope.h @@ -0,0 +1,120 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Qualcomm CAMSS OPE ISP class + * + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + */ + +#pragma once + +#include +#include + +#include +#include + +/* FIXME drop once using own camss IPA, which will map on IPA side */ +#include "libcamera/internal/mapped_framebuffer.h" +#include "libcamera/internal/shared_mem_object.h" +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/v4l2_subdevice.h" + +#include "camss_isp.h" + +namespace libcamera { + +class CameraSensor; +class CamssFrames; +class ControlInfoMap; +class DeviceEnumerator; +class MediaDevice; +class PipelineHandler; +class Stream; +class SwStatsCpu; +class V4L2VideoDevice; + +class CamssIspOpe : public CamssIsp +{ +public: + static std::unique_ptr match(PipelineHandler *pipe, + DeviceEnumerator *enumerator, + const CameraSensor *sensor, + CamssFrames *frameInfos, + ControlInfoMap *ispControls); + + bool isValid() override; + Size getMargins(PixelFormat inputFormat) override; + + CamssIspOpe(PipelineHandler *pipe, std::shared_ptr opeMediaDev, + const CameraSensor *sensor, CamssFrames *frameInfos, + ControlInfoMap *ispControls); + ~CamssIspOpe() override; + + StreamConfiguration generateConfiguration(const StreamConfiguration &raw) const override; + StreamConfiguration validate(const StreamConfiguration &raw, const StreamConfiguration &req) const override; + int configure(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg) override; + + int allocateBuffers(unsigned int bufferCount) override; + void freeBuffers() override; + int exportOutputBuffers(const Stream *stream, unsigned int count, + std::vector> *buffers) override; + void queueBuffers(Request *request, FrameBuffer *input) override; + + void processStats(const uint32_t frame, const uint32_t statsBufferId, + const ControlList &sensorControls) override; + + void paramsComputed(uint32_t frame); + void parameterBufferReady(FrameBuffer *f); + + int start() override; + void stop() override; + +private: + static constexpr Size kMinOutputSize = Size(24, 16); + static constexpr unsigned int PROC_PAD_INPUT = 0; + static constexpr unsigned int PROC_PAD_PARAMS = 1; + static constexpr unsigned int PROC_PAD_DISP = 2; + static constexpr unsigned int DISP_PAD_PROC = 0; + static constexpr unsigned int DISP_PAD_OUTPUT = 1; + + int init(); + int trySetVideoCfg(V4L2VideoDevice *v4l2Dev, StreamConfiguration &cfg, + bool set, bool update_stride_and_size, + const char *msgPrefix) const; + int trySetSubdevFormat(V4L2Subdevice *subdev, + unsigned int pad, + const V4L2SubdeviceFormat &fmt, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const; + int trySetSubdevSelection(V4L2Subdevice *subdev, + unsigned int pad, + unsigned int target, + const Rectangle &rect, + V4L2Subdevice::Whence whence, + const char *msgPrefix) const; + int trySetPipelineConfig(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg, + V4L2Subdevice::Whence whence) const; + + std::shared_ptr opeMediaDev_; + const CameraSensor *sensor_; + const Stream *rawStream_; + class CamssFrames *frameInfos_; + SharedMemObject sharedParams_; + std::unique_ptr stats_; + std::unique_ptr proc_; + std::unique_ptr disp_; + std::unique_ptr params_; + std::unique_ptr input_; + std::unique_ptr output_; + std::unique_ptr ipa_; + std::vector> paramBuffers_; + std::vector ipaBuffers_; + /* FIXME drop once using own camss IPA, which will map on IPA side */ + std::map mappedIpaBuffers_; + unsigned int inputBufferCount_; + unsigned int outputBufferCount_; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/camss/camss_params.h b/src/libcamera/pipeline/camss/camss_params.h index 6dec85fbc..c09d2c4c2 100644 --- a/src/libcamera/pipeline/camss/camss_params.h +++ b/src/libcamera/pipeline/camss/camss_params.h @@ -20,7 +20,7 @@ namespace libcamera { namespace ipa::camss { -enum class CamssBlocks { +enum class CamssBlocks : uint16_t { AwbGains, ChromaEnh, ColorCorrect, diff --git a/src/libcamera/pipeline/camss/meson.build b/src/libcamera/pipeline/camss/meson.build index 2f3931dd1..cffb53ef5 100644 --- a/src/libcamera/pipeline/camss/meson.build +++ b/src/libcamera/pipeline/camss/meson.build @@ -5,6 +5,9 @@ libcamera_internal_sources += files([ 'camss_csi.cpp', 'camss_frames.cpp', 'camss_isp.cpp', + 'camss_isp_ope.cpp', 'camss_isp_soft.cpp', 'camss_util.cpp', + # FIXME hack + '../../../ipa/libipa/v4l2_params.cpp' ])