From patchwork Tue May 23 17:04:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Bara X-Patchwork-Id: 18649 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C2184C31E9 for ; Tue, 23 May 2023 17:05:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0DF52626DE; Tue, 23 May 2023 19:05:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1684861501; bh=ESXPLeMaQ817J3EHsBdJ8AOWu2sUAj3LMUWOsrx4sLg=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=Ugj+sAkfway55ubNNOo2TmkGmP5UJPxpDUUzXDAcso4tZ+lfTWSbNZT48jtbIA8+t l3g6s5lg+j8pYIquSPn2dBOIux4jQOiEyNtW7GMrswYPHx6OVSCXy0IaKPamRmjiH6 idSJqw8ofmmpTHLLvbepYyqTXrnvmIKinAt+U93y6QYAgpXvlBnF+xnBrfyh3Fg1Aj Y+XB8niy2CCsaw4pKzFjIIOsbYoKe1GpHxK0rmJ3X+XxlJ6olWpOJ1E863zHxfoVcE 1xTSUDAFOFU/b9uZlTxPjIgbc/ZrcwpEBQDyruSG/u6MqIjybrk3bwS/6w970py/cT 9GZ5xmmBvT6FA== Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 16006626DD for ; Tue, 23 May 2023 19:04:59 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="lYVo5NYE"; dkim-atps=neutral Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-50bcb4a81ceso193459a12.2 for ; Tue, 23 May 2023 10:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684861498; x=1687453498; h=cc:to:message-id:content-transfer-encoding:mime-version:subject :date:from:from:to:cc:subject:date:message-id:reply-to; bh=9ytBgpWtWaJ7QUJRzjRkIsKVLtH4LxiyjpZMmMQ5Te8=; b=lYVo5NYEcIShOaMCairXwZNZ40QA+SRDc+ZuheY3JyznvZKCN7D+jEdO6C9cFc2+Ex DIF/UNiLOJ4cw6YCC8pl7Him5wbJ+3ycsy5ip3j3MGAX/bmLXSMrHerXdGXZYXD2sHxC 3B2R/Nm5KdO62YJjiKx8VTIygBgKv09uekBS2OC1i3yxRTlONrUxNk1D0vaxubuTfSEN cVZjUU+GJwnajlf9MLvOLaoEdNJEM65H5vNQKqmu+DTPU1/I6B6vTp0HCUZPbaEwmyEJ J4CPpGiN3DarFbjV3OSMllNUp7isJRZt45Rcd5H8yJQ6pwA8VOhKekzL1RhR3bU7O5eJ 3xEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684861498; x=1687453498; h=cc:to:message-id:content-transfer-encoding:mime-version:subject :date:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9ytBgpWtWaJ7QUJRzjRkIsKVLtH4LxiyjpZMmMQ5Te8=; b=WUJ2YLVFMzlcuRoR4GMtYzrlWa8CjnDeD+B8IJbmp8BkVdmgmUYXAYgEyJ4TrtZrSo R1B09jv9gzcaZGOa8hGHnM+ZwJi6HlBh/2PAHzfufyVkGi5riS+c0uDTwXDFqVllcLNM iBj8eDJLp5a5c7jyyMo7covp2HJ96nC+Uf7Fqa0IJsi6dxs9X23usp26VOHwVmDYpbR8 vFN3PjI6on4IsnXD8v0wXE9ZclD3hxv6JbMv05yZsk5HAxonFf2B4O5M5lGkTSEMSj57 6BSAib9RXhrq3vxrZ83WQOiLF+JrpcSwaTq7cIIV8ne1B+ZSgawJK65bxFiH5Y6qrUk2 nVhQ== X-Gm-Message-State: AC+VfDxNdMSKgmCZ3qAb5OGEnQPbGXCabHFAKJ7BGqzQTAwBDXjy7YJj chJNU8G7KYlpLr/ldf0c4PkZOwGZRv0= X-Google-Smtp-Source: ACHHUZ7Z1iorHkyL7b1wF3B6HHp223/TWXPW2afnkBzMrWrkKyXwPgTdtoMgq9vss1UGivTWOFMwjg== X-Received: by 2002:a17:907:16a9:b0:959:a9a1:589e with SMTP id hc41-20020a17090716a900b00959a9a1589emr14808716ejc.76.1684861497915; Tue, 23 May 2023 10:04:57 -0700 (PDT) Received: from [127.0.1.1] ([91.230.2.244]) by smtp.gmail.com with ESMTPSA id v21-20020a170906339500b0096efd44dbffsm4665136eja.105.2023.05.23.10.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 10:04:57 -0700 (PDT) Date: Tue, 23 May 2023 19:04:43 +0200 MIME-Version: 1.0 Message-Id: <20230523-rkisp1-vblank-v1-1-381c6f025ff6@skidata.com> X-B4-Tracking: v=1; b=H4sIACrybGQC/x2NQQqDQAwAvyI5N6C7rdheCz7Aa+khq1GD7SpJk YL49649DsMwGxirsMEt20B5FZM5JihOGbQjxYFRusTgcufzi/Ook9hS4BpeFCf01FWlK6srnwl SE8gYg1Jsx6N6k31YD7Eo9/L9jx7Q1Hd47vsPhIz8BX0AAAA= To: libcamera-devel@lists.libcamera.org X-Mailer: b4 0.12.2 Subject: [libcamera-devel] [PATCH RFC] rkisp1: adjust vblank to target framerate X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Benjamin Bara via libcamera-devel From: Benjamin Bara Reply-To: Benjamin Bara Cc: Benjamin Bara Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Benjamin Bara Set the vertical blanking of a sensor accordingly to a targeted framerate. E.g. gst_libcamera_clamp_and_set_frameduration() sets FrameDurationLimits to the initControls_, which are then passed to Camera::start() (and from there to PipelineHandler::start()). Example (imx327: 1080p@25; default/minimum hBlank: 280): vBlank = 0.04 * 148500000 / 2200 - 1080 = 1620 which results to: 16390.268661 (25.00 fps) cam0-stream0 seq: 000001 bytesused: 2073600/1036800 16390.308661 (25.00 fps) cam0-stream0 seq: 000002 bytesused: 2073600/1036800 When doing this on sensors where the allowed exposure range depends on vblank (e.g. on the imx327), the upper exposure limit is increased ( from 1123 to 2698 in the example above). As the sensor controls are "cached" in the IPA context, they must be updated. This is done by passing the updated values via the start() method. In general, this could be done independently of the IPA. Signed-off-by: Benjamin Bara --- Some checks are still missing in this version, but I wanted to check first if my approach fits and if this is something worth to look further into. I also saw that the getBlanking() method inside of the rpi IPA is doing something similar, but I guess this can be done independent of the sensor and IPA. Many thanks and best regards, Benjamin --- include/libcamera/ipa/rkisp1.mojom | 2 +- src/ipa/rkisp1/rkisp1.cpp | 18 ++++++++++++++-- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 37 +++++++++++++++++++++++++++++++- 3 files changed, 53 insertions(+), 4 deletions(-) --- base-commit: e8fccaea46b9e545282cd37d54b1acb168608a46 change-id: 20230523-rkisp1-vblank-3ad862689e4a Best regards, diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index 1009e970..0e0df9e8 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -19,7 +19,7 @@ interface IPARkISP1Interface { libcamera.IPACameraSensorInfo sensorInfo, libcamera.ControlInfoMap sensorControls) => (int32 ret, libcamera.ControlInfoMap ipaControls); - start() => (int32 ret); + start(libcamera.ControlInfoMap sensorControls) => (int32 ret); stop(); configure(IPAConfigInfo configInfo, diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 6544c925..2c092efa 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -53,7 +53,7 @@ public: const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls) override; - int start() override; + int start(const ControlInfoMap &sensorControls) override; void stop() override; int configure(const IPAConfigInfo &ipaConfig, @@ -197,8 +197,22 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, return 0; } -int IPARkISP1::start() +int IPARkISP1::start(const ControlInfoMap &sensorControls) { + /* + * it might be possible that VBLANK has been changed and that this has + * an impact on the exposure limits. therefore re-set them here. + */ + const auto itExp = sensorControls.find(V4L2_CID_EXPOSURE); + int32_t minExposure = itExp->second.min().get(); + int32_t maxExposure = itExp->second.max().get(); + context_.configuration.sensor.minShutterSpeed = + minExposure * context_.configuration.sensor.lineDuration; + context_.configuration.sensor.maxShutterSpeed = + maxExposure * context_.configuration.sensor.lineDuration; + LOG(IPARkISP1, Debug) + << "Exposure: [" << minExposure << ", " << maxExposure << "]"; + setControls(0); return 0; diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 8a30fe06..f9b3a3f7 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -914,12 +914,47 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL RkISP1CameraData *data = cameraData(camera); int ret; + /* + * \todo Move this to a IPA-indepent location where a camera sensor + * instance is available and the targeted frame duration is known. + * Additionally, the IPA's sensor controls must be set accordingly. + */ + auto frameDurations = controls->get(controls::FrameDurationLimits); + if (frameDurations && frameDurations->size() == 2) { + ControlList sensorControls = data->sensor_->controls(); + ControlList ctrls; + IPACameraSensorInfo sensorInfo; + if (data->sensor_->sensorInfo(&sensorInfo)) { + LOG(RkISP1, Error) << "couldn't fetch sensor info"; + } + + /* + * setup vertical blanking for target frame rate: + * frameWidth = width + hBlank + * frameHeight = height + vBlank + * frameDuration = frameWidth * frameHeight / pixelRate + * => + * vBlank = frameDuration [us] * pixelRate / frameWidth - height + */ + uint32_t frameWidth = sensorInfo.minLineLength; + uint32_t height = sensorInfo.outputSize.height; + uint64_t pixelRate = sensorInfo.pixelRate; + uint32_t maxFrameDuration = (*frameDurations)[1]; + int32_t vBlank = maxFrameDuration * pixelRate / (frameWidth * 1000000) - height; + LOG(RkISP1, Debug) << "Setting VBLANK to " << vBlank; + ctrls.set(V4L2_CID_VBLANK, vBlank); + data->sensor_->setControls(&ctrls); + data->sensor_->updateControlInfo(); + } else { + LOG(RkISP1, Debug) << "Skip setting VBLANK"; + } + /* Allocate buffers for internal pipeline usage. */ ret = allocateBuffers(camera); if (ret) return ret; - ret = data->ipa_->start(); + ret = data->ipa_->start(data->sensor_->controls()); if (ret) { freeBuffers(camera); LOG(RkISP1, Error)