From patchwork Mon Oct 3 08:39:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 17495 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B4E84C3287 for ; Mon, 3 Oct 2022 08:40:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4DD6D62CEA; Mon, 3 Oct 2022 10:40:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1664786409; bh=QtTA413PQEFNDkJeBHXFQjs3DSW2osPsuoHi+6m9fek=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=eis0EIOtqGUyYTDkGeuBGIc2nkGT7MI4tJnXO2x1NLuSLIybDkv9rO+Ah5Xd+2FN3 ipAV+pEHOqmAd2x+0VvKmDB9t1FXXEv9bfStvCyIcAlInnBWH1tOCxbFpyvoIAfpnR NGnXJOfUxg639HIC5a50aS+24xSIXPElsuATihL0jN8HeGmJPEKFbYBUrV5K84rpZr sjJ2gO7uferlQIPHZQwoBRL3vXo+8Efw22bwTCfPy7j8FWhtX545kt8DphwIXnaihd HQWibeI8xXUeaweUO1ls1wdYvgxZ1QA49zoEuPvklMaU2qXA20PtInj9p5MJcUVY0G KuwQiH8gxq+pw== Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D8D9762CD5 for ; Mon, 3 Oct 2022 10:40:03 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="oXQI6A6Y"; dkim-atps=neutral Received: by mail-wm1-x329.google.com with SMTP id e18so6525688wmq.3 for ; Mon, 03 Oct 2022 01:40:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=likkbiXx7bdJfq6g4Kcjz6CER4IRvJggljDa82NlX20=; b=oXQI6A6Ys0V4DA3WkXCwp5SJA76+tGv0rFzzxoS/UlS8JQnojjbRd71dWHd62zns38 L1ho4InA4P6NAvA6cZVcXlwhpszLXV4VSEl487POLGWPCXLWiRtRs/nRItuM9Gl1JD9Q /MDvn577r0ofwLF3+S4cbR00rD8R4yulJH/ZkkNLcTECsUmWw5iw22VSAoD0d05TUlUJ B+K7jVJ5DIRl1pvzCnu+AbwfePmOMf/W6Abs7+89JJT/7wSjaLeMnwH00T+KKeHzvn5J yCKX9P7mG6MPZHshcwCxAUS8W31xyvBDUDf01LkGI+FKKAJzhtosyhYzTWJtjoblHP5K GR/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=likkbiXx7bdJfq6g4Kcjz6CER4IRvJggljDa82NlX20=; b=cvljhLG4oamMiVYtCa8GXNPvcrVxF0eyGe7+tXxaD+OAmzLNxV5mNER7xJ+Zk05T0x tdqCD3aw/cpDu05S7e440JqKUd8d4VIvP9YuN8C8rNksvuL+LJYRUCTnKzujdVzcWijC G8YdO15tlNCUFjoKwkjIAAo+3F5fUq6o1roAc3EDSrhsY85bzwBh0RB//XHz/ahY2rNW Prdbvg38Ah1p+1w5NdALuXfOtL5xoh/SU96IojBREs8sLv+3KmquB+Ow1OEZY9cAog/G tjO8OAwSt+D7SICoRop81WiohZlDYEHN60vgYmZoO5n1LEbi2JjN9LpJWz11ecOpyl5a +ENg== X-Gm-Message-State: ACrzQf0K6Il2vDJXxyqw8hRpIROQLdUIZDsdJUeHTUNKuP0hDIrzC5HS BZ4iCQte8UqL8JIYMeX1SQNJphydcIzYqQ== X-Google-Smtp-Source: AMsMyM6fryKRsopOwp8JOqT1SYkvDXd/JCNwtWMQiCBNzsbjyzVNNr+qoryMJhad+pr1mpq8n0ln4g== X-Received: by 2002:a05:600c:42d4:b0:3b3:3de1:7564 with SMTP id j20-20020a05600c42d400b003b33de17564mr6265041wme.152.1664786403055; Mon, 03 Oct 2022 01:40:03 -0700 (PDT) Received: from naush-laptop.localdomain ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id i13-20020a5d55cd000000b0022ae59d472esm8922787wrw.112.2022.10.03.01.40.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Oct 2022 01:40:02 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Mon, 3 Oct 2022 09:39:34 +0100 Message-Id: <20221003083934.31629-9-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221003083934.31629-1-naush@raspberrypi.com> References: <20221003083934.31629-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 8/9] ipa: raspberrypi: Allow full line length control X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Naushir Patuck via libcamera-devel From: Naushir Patuck Reply-To: Naushir Patuck Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Rename CamHelper::getVBlanking to CamHelper::getBlanking, and update the calculations in that function to return both horizontal and vertical blanking values for a given exposure time and frame duration limits. The calculations are setup such that vertical blanking is extended to the maximum allowable value, and any remainder gets put into horizontal blanking. The calculated horizontal blanking value is now return to the pipeline handler to pass into DelayedControls to program into the sensor. Update the IPA to now specify the maximum frame duration from the maximum horizontal + vertical blanking values provided by the sensor mode. Additionally, the IPA now uses the frame specific horizontal blanking value (as returned by DelayedControls) in all instances. Signed-off-by: Naushir Patuck --- src/ipa/raspberrypi/cam_helper.cpp | 47 +++++++++++++++++------ src/ipa/raspberrypi/cam_helper.h | 7 ++-- src/ipa/raspberrypi/cam_helper_imx477.cpp | 24 +++++++----- src/ipa/raspberrypi/cam_helper_imx519.cpp | 24 +++++++----- src/ipa/raspberrypi/raspberrypi.cpp | 45 +++++++++++++--------- 5 files changed, 95 insertions(+), 52 deletions(-) diff --git a/src/ipa/raspberrypi/cam_helper.cpp b/src/ipa/raspberrypi/cam_helper.cpp index c255ab0cb53f..f5f034ece711 100644 --- a/src/ipa/raspberrypi/cam_helper.cpp +++ b/src/ipa/raspberrypi/cam_helper.cpp @@ -8,6 +8,7 @@ #include #include +#include #include #include @@ -74,33 +75,57 @@ Duration CamHelper::exposure(uint32_t exposureLines, const Duration lineLength) return exposureLines * lineLength; } -uint32_t CamHelper::getVBlanking(Duration &exposure, - Duration minFrameDuration, - Duration maxFrameDuration) const +std::pair CamHelper::getBlanking(Duration &exposure, + Duration minFrameDuration, + Duration maxFrameDuration) const { - uint32_t frameLengthMin, frameLengthMax, vblank; - uint32_t exposureLines = CamHelper::exposureLines(exposure, mode_.minLineLength); + uint32_t frameLengthMin, frameLengthMax, vblank, hblank; + Duration lineLength = mode_.minLineLength; assert(initialized_); /* * minFrameDuration and maxFrameDuration are clamped by the caller * based on the limits for the active sensor mode. + * + * frameLengthMax gets calculated on the smallest line length as we do + * not want to extend that unless absolutely necessary. */ frameLengthMin = minFrameDuration / mode_.minLineLength; frameLengthMax = maxFrameDuration / mode_.minLineLength; + /* + * Watch out for exposureLines overflowing a uint32_t when the exposure + * time is extremely (extremely!) long - as happens when the IPA calculates + * the maximum possible exposure time. + */ + uint32_t exposureLines = std::min(CamHelper::exposureLines(exposure, lineLength), + std::numeric_limits::max() - frameIntegrationDiff_); + uint32_t frameLengthLines = std::clamp(exposureLines + frameIntegrationDiff_, + frameLengthMin, frameLengthMax); + + /* + * If our frame length lines is above the maximum allowed, see if we can + * extend the line length to accommodate the requested frame length. + */ + if (frameLengthLines > mode_.maxFrameLength) { + Duration lineLengthAdjusted = lineLength * frameLengthLines / mode_.maxFrameLength; + lineLength = std::min(mode_.maxLineLength, lineLengthAdjusted); + frameLengthLines = mode_.maxFrameLength; + } + + hblank = lineLengthToHblank(lineLength); + vblank = frameLengthLines - mode_.height; + /* * Limit the exposure to the maximum frame duration requested, and * re-calculate if it has been clipped. */ - exposureLines = std::min(frameLengthMax - frameIntegrationDiff_, exposureLines); - exposure = CamHelper::exposure(exposureLines, mode_.minLineLength); + exposureLines = std::min(frameLengthLines - frameIntegrationDiff_, + CamHelper::exposureLines(exposure, lineLength)); + exposure = CamHelper::exposure(exposureLines, lineLength); - /* Limit the vblank to the range allowed by the frame length limits. */ - vblank = std::clamp(exposureLines + frameIntegrationDiff_, - frameLengthMin, frameLengthMax) - mode_.height; - return vblank; + return { vblank, hblank }; } Duration CamHelper::hblankToLineLength(uint32_t hblank) const diff --git a/src/ipa/raspberrypi/cam_helper.h b/src/ipa/raspberrypi/cam_helper.h index b5c0726ff00e..21ac101a0a0b 100644 --- a/src/ipa/raspberrypi/cam_helper.h +++ b/src/ipa/raspberrypi/cam_helper.h @@ -8,6 +8,7 @@ #include #include +#include #include #include @@ -82,9 +83,9 @@ public: const libcamera::utils::Duration lineLength) const; virtual libcamera::utils::Duration exposure(uint32_t exposureLines, const libcamera::utils::Duration lineLength) const; - virtual uint32_t getVBlanking(libcamera::utils::Duration &exposure, - libcamera::utils::Duration minFrameDuration, - libcamera::utils::Duration maxFrameDuration) const; + virtual std::pair getBlanking(libcamera::utils::Duration &exposure, + libcamera::utils::Duration minFrameDuration, + libcamera::utils::Duration maxFrameDuration) const; libcamera::utils::Duration hblankToLineLength(uint32_t hblank) const; uint32_t lineLengthToHblank(const libcamera::utils::Duration &duration) const; libcamera::utils::Duration lineLengthPckToDuration(uint32_t lineLengthPck) const; diff --git a/src/ipa/raspberrypi/cam_helper_imx477.cpp b/src/ipa/raspberrypi/cam_helper_imx477.cpp index 76a82cc51378..19a5e471c27e 100644 --- a/src/ipa/raspberrypi/cam_helper_imx477.cpp +++ b/src/ipa/raspberrypi/cam_helper_imx477.cpp @@ -46,8 +46,8 @@ public: uint32_t gainCode(double gain) const override; double gain(uint32_t gainCode) const override; void prepare(libcamera::Span buffer, Metadata &metadata) override; - uint32_t getVBlanking(Duration &exposure, Duration minFrameDuration, - Duration maxFrameDuration) const override; + std::pair getBlanking(Duration &exposure, Duration minFrameDuration, + Duration maxFrameDuration) const override; void getDelays(int &exposureDelay, int &gainDelay, int &vblankDelay, int &hblankDelay) const override; bool sensorEmbeddedDataPresent() const override; @@ -118,15 +118,19 @@ void CamHelperImx477::prepare(libcamera::Span buffer, Metadata &m } } -uint32_t CamHelperImx477::getVBlanking(Duration &exposure, - Duration minFrameDuration, - Duration maxFrameDuration) const +std::pair CamHelperImx477::getBlanking(Duration &exposure, + Duration minFrameDuration, + Duration maxFrameDuration) const { uint32_t frameLength, exposureLines; unsigned int shift = 0; - frameLength = mode_.height + CamHelper::getVBlanking(exposure, minFrameDuration, - maxFrameDuration); + auto [vblank, hblank] = CamHelper::getBlanking(exposure, minFrameDuration, + maxFrameDuration); + + frameLength = mode_.height + vblank; + Duration lineLength = hblankToLineLength(hblank); + /* * Check if the frame length calculated needs to be setup for long * exposure mode. This will require us to use a long exposure scale @@ -144,12 +148,12 @@ uint32_t CamHelperImx477::getVBlanking(Duration &exposure, if (shift) { /* Account for any rounding in the scaled frame length value. */ frameLength <<= shift; - exposureLines = CamHelperImx477::exposureLines(exposure, mode_.minLineLength); + exposureLines = CamHelperImx477::exposureLines(exposure, lineLength); exposureLines = std::min(exposureLines, frameLength - frameIntegrationDiff); - exposure = CamHelperImx477::exposure(exposureLines, mode_.minLineLength); + exposure = CamHelperImx477::exposure(exposureLines, lineLength); } - return frameLength - mode_.height; + return { frameLength - mode_.height, hblank }; } void CamHelperImx477::getDelays(int &exposureDelay, int &gainDelay, diff --git a/src/ipa/raspberrypi/cam_helper_imx519.cpp b/src/ipa/raspberrypi/cam_helper_imx519.cpp index 9dff1eeb899f..d2eb171912da 100644 --- a/src/ipa/raspberrypi/cam_helper_imx519.cpp +++ b/src/ipa/raspberrypi/cam_helper_imx519.cpp @@ -46,8 +46,8 @@ public: uint32_t gainCode(double gain) const override; double gain(uint32_t gainCode) const override; void prepare(libcamera::Span buffer, Metadata &metadata) override; - uint32_t getVBlanking(Duration &exposure, Duration minFrameDuration, - Duration maxFrameDuration) const override; + std::pair getBlanking(Duration &exposure, Duration minFrameDuration, + Duration maxFrameDuration) const override; void getDelays(int &exposureDelay, int &gainDelay, int &vblankDelay, int &hblankDelay) const override; bool sensorEmbeddedDataPresent() const override; @@ -118,15 +118,19 @@ void CamHelperImx519::prepare(libcamera::Span buffer, Metadata &m } } -uint32_t CamHelperImx519::getVBlanking(Duration &exposure, - Duration minFrameDuration, - Duration maxFrameDuration) const +std::pair CamHelperImx519::getBlanking(Duration &exposure, + Duration minFrameDuration, + Duration maxFrameDuration) const { uint32_t frameLength, exposureLines; unsigned int shift = 0; - frameLength = mode_.height + CamHelper::getVBlanking(exposure, minFrameDuration, - maxFrameDuration); + auto [vblank, hblank] = CamHelper::getBlanking(exposure, minFrameDuration, + maxFrameDuration); + + frameLength = mode_.height + vblank; + Duration lineLength = hblankToLineLength(hblank); + /* * Check if the frame length calculated needs to be setup for long * exposure mode. This will require us to use a long exposure scale @@ -144,12 +148,12 @@ uint32_t CamHelperImx519::getVBlanking(Duration &exposure, if (shift) { /* Account for any rounding in the scaled frame length value. */ frameLength <<= shift; - exposureLines = CamHelperImx519::exposureLines(exposure, mode_.minLineLength); + exposureLines = CamHelperImx519::exposureLines(exposure, lineLength); exposureLines = std::min(exposureLines, frameLength - frameIntegrationDiff); - exposure = CamHelperImx519::exposure(exposureLines, mode_.minLineLength); + exposure = CamHelperImx519::exposure(exposureLines, lineLength); } - return frameLength - mode_.height; + return { frameLength - mode_.height, hblank }; } void CamHelperImx519::getDelays(int &exposureDelay, int &gainDelay, diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 5d6b22ef6813..497a83939ae6 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -315,7 +315,7 @@ void IPARPi::start(const ControlList &controls, StartConfig *startConfig) } startConfig->dropFrameCount = dropFrameCount_; - const Duration maxSensorFrameDuration = mode_.maxFrameLength * mode_.minLineLength; + const Duration maxSensorFrameDuration = mode_.maxFrameLength * mode_.maxLineLength; startConfig->maxSensorFrameLengthMs = maxSensorFrameDuration.get(); firstStart_ = false; @@ -462,7 +462,7 @@ int IPARPi::configure(const IPACameraSensorInfo &sensorInfo, */ ControlInfoMap::Map ctrlMap = ipaControls; const Duration minSensorFrameDuration = mode_.minFrameLength * mode_.minLineLength; - const Duration maxSensorFrameDuration = mode_.maxFrameLength * mode_.minLineLength; + const Duration maxSensorFrameDuration = mode_.maxFrameLength * mode_.maxLineLength; ctrlMap[&controls::FrameDurationLimits] = ControlInfo(static_cast(minSensorFrameDuration.get()), static_cast(maxSensorFrameDuration.get())); @@ -475,7 +475,7 @@ int IPARPi::configure(const IPACameraSensorInfo &sensorInfo, * will limit the maximum control value based on the current VBLANK value. */ Duration maxShutter = Duration::max(); - helper_->getVBlanking(maxShutter, minSensorFrameDuration, maxSensorFrameDuration); + helper_->getBlanking(maxShutter, minSensorFrameDuration, maxSensorFrameDuration); const uint32_t exposureMin = sensorCtrls_.at(V4L2_CID_EXPOSURE).min().get(); ctrlMap[&controls::ExposureTime] = @@ -552,7 +552,7 @@ void IPARPi::reportMetadata() deviceStatus->shutterSpeed.get()); libcameraMetadata_.set(controls::AnalogueGain, deviceStatus->analogueGain); libcameraMetadata_.set(controls::FrameDuration, - helper_->exposure(deviceStatus->frameLength, mode_.minLineLength).get()); + helper_->exposure(deviceStatus->frameLength, deviceStatus->lineLength).get()); if (deviceStatus->sensorTemperature) libcameraMetadata_.set(controls::SensorTemperature, *deviceStatus->sensorTemperature); } @@ -1110,8 +1110,8 @@ void IPARPi::fillDeviceStatus(const ControlList &sensorControls) int32_t vblank = sensorControls.get(V4L2_CID_VBLANK).get(); int32_t hblank = sensorControls.get(V4L2_CID_HBLANK).get(); - deviceStatus.lineLength = (mode_.width + hblank) * (1.0s / mode_.pixelRate); - deviceStatus.shutterSpeed = helper_->exposure(exposureLines, mode_.minLineLength); + deviceStatus.lineLength = helper_->hblankToLineLength(hblank); + deviceStatus.shutterSpeed = helper_->exposure(exposureLines, deviceStatus.lineLength); deviceStatus.analogueGain = helper_->gain(gainCode); deviceStatus.frameLength = mode_.height + vblank; @@ -1157,7 +1157,7 @@ void IPARPi::applyAWB(const struct AwbStatus *awbStatus, ControlList &ctrls) void IPARPi::applyFrameDurations(Duration minFrameDuration, Duration maxFrameDuration) { const Duration minSensorFrameDuration = mode_.minFrameLength * mode_.minLineLength; - const Duration maxSensorFrameDuration = mode_.maxFrameLength * mode_.minLineLength; + const Duration maxSensorFrameDuration = mode_.maxFrameLength * mode_.maxLineLength; /* * This will only be applied once AGC recalculations occur. @@ -1178,11 +1178,11 @@ void IPARPi::applyFrameDurations(Duration minFrameDuration, Duration maxFrameDur /* * Calculate the maximum exposure time possible for the AGC to use. - * getVBlanking() will update maxShutter with the largest exposure + * getBlanking() will update maxShutter with the largest exposure * value possible. */ Duration maxShutter = Duration::max(); - helper_->getVBlanking(maxShutter, minFrameDuration_, maxFrameDuration_); + helper_->getBlanking(maxShutter, minFrameDuration_, maxFrameDuration_); RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); @@ -1200,10 +1200,11 @@ void IPARPi::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) */ gainCode = std::min(gainCode, maxSensorGainCode_); - /* getVBlanking might clip exposure time to the fps limits. */ + /* getBlanking might clip exposure time to the fps limits. */ Duration exposure = agcStatus->shutterTime; - int32_t vblanking = helper_->getVBlanking(exposure, minFrameDuration_, maxFrameDuration_); - int32_t exposureLines = helper_->exposureLines(exposure, mode_.minLineLength); + auto [vblank, hblank] = helper_->getBlanking(exposure, minFrameDuration_, maxFrameDuration_); + int32_t exposureLines = helper_->exposureLines(exposure, + helper_->hblankToLineLength(hblank)); LOG(IPARPI, Debug) << "Applying AGC Exposure: " << exposure << " (Shutter lines: " << exposureLines << ", AGC requested " @@ -1211,14 +1212,22 @@ void IPARPi::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) << agcStatus->analogueGain << " (Gain Code: " << gainCode << ")"; - /* - * Due to the behavior of V4L2, the current value of VBLANK could clip the - * exposure time without us knowing. The next time though this function should - * clip exposure correctly. - */ - ctrls.set(V4L2_CID_VBLANK, vblanking); + ctrls.set(V4L2_CID_VBLANK, static_cast(vblank)); ctrls.set(V4L2_CID_EXPOSURE, exposureLines); ctrls.set(V4L2_CID_ANALOGUE_GAIN, gainCode); + + /* + * At present, there is no way of knowing if a control is read-only. + * As a workaround, assume that if the minimum and maximum values of + * the V4L2_CID_HBLANK control are the same, it implies the control + * is read-only. This seems to be the case for all the cameras our IPA + * works with. + * + * \todo The control API ought to have a flag to specify if a control + * is read-only which could be used below. + */ + if (mode_.minLineLength != mode_.maxLineLength) + ctrls.set(V4L2_CID_HBLANK, static_cast(hblank)); } void IPARPi::applyDG(const struct AgcStatus *dgStatus, ControlList &ctrls)