From patchwork Thu Dec 10 13:34:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 10633 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 66CE6BD80A for ; Thu, 10 Dec 2020 13:34:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3752567F79; Thu, 10 Dec 2020 14:34:34 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="D6AYVjkS"; dkim-atps=neutral Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E687E60323 for ; Thu, 10 Dec 2020 14:34:32 +0100 (CET) Received: by mail-wm1-x342.google.com with SMTP id d3so4671367wmb.4 for ; Thu, 10 Dec 2020 05:34:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Zlt2o4yBWUIl/QoAfsK+KQt08ETtTX7ud6jiM7O9gVU=; b=D6AYVjkSTZfEJnFHsTpPmzGgzfHuLGRqnNjVFcDu6XeDoMHN6wKColNcecDMjWXK5M OXDIQ14mbMxOXdy6rlbGmu+O3owxjXfLSdw43XHHSfd+/gGGO7LSqQwE1gXSlCdJitKB 8dfc0Ib0nFfHcr15KhfvgYparj0g33C3j90GKJvecv6oMQ5Bwyup7+UWQbLfFi9loVRC s1W0qPdx57QUZyA/NylIuJPxRGRGC4k0l9tP+Z839TxYIo+8qOqRtu+yX4sPtxPI3pzQ rIZcNaw8Rft5POtFA2rz6ldg++WEfTthSpuHOBfKwT4YfblrLdQmn0GZELFdBm0lM5h4 ji5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Zlt2o4yBWUIl/QoAfsK+KQt08ETtTX7ud6jiM7O9gVU=; b=b8WeoAN/DFGh5PUOJAUa8m9fCZbpvYnIMRJsEGyvrlzrH7uk+2AFcyLfpl2/pMyI6k 56bP3dzWcIrVJMQfRiYbQSHvTjGOIsLDD/AZNH340R5X+S55SypfoHBoCFuJI6duXnUM gEUvo+R/r7m5iZiZqaVS6RKIXE6wlg6E/1gqhZSaTuLWcUPZdAIlKYRw8Q/6SrMekeP8 L0th1o9T4QztyPgtVYDiLDvfuRH5I6fk4XS1fslTBu+C/Ah8i6ayMQ0AhzgrfwrmUZ1O Z2kijHYLJqiyJa/AfEqiVywPWFPayjqbQYPJxgnYyPgLsf8agjb9DnI0rKlOZdbQ2P45 x7Sg== X-Gm-Message-State: AOAM532CCNNoD3hwsrjWs9u315bLfGzJMdi+ViRyeDA688hfIchCvWkI 0ypXm7Hyc7KpPlYSDOrAKDzSKW+Ioyi2cA== X-Google-Smtp-Source: ABdhPJxTJSO4Zfjb8I1YxKNtJ5Wuephzn7XEQBI2u6ThfqDqed+WQseDm5EafUcTUBaBWEUBcoUMcw== X-Received: by 2002:a1c:1c1:: with SMTP id 184mr8157707wmb.112.1607607272145; Thu, 10 Dec 2020 05:34:32 -0800 (PST) Received: from naushir-VirtualBox.pitowers.org ([2a00:1098:3142:14:a00:27ff:fe4d:f6a2]) by smtp.gmail.com with ESMTPSA id c129sm9395467wma.31.2020.12.10.05.34.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Dec 2020 05:34:31 -0800 (PST) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Date: Thu, 10 Dec 2020 13:34:25 +0000 Message-Id: <20201210133426.206679-2-naush@raspberrypi.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210133426.206679-1-naush@raspberrypi.com> References: <20201210133426.206679-1-naush@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 2/3] libcamera: raspberrypi: Add control of sensor vblanking X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add support for setting V4L2_CID_VBLANK appropriately when setting V4L2_CID_EXPOSURE. This will allow adaptive framerates during viewfinder use cases (e.g. when the exposure time goes above 33ms, we can reduce the framerate to lower than 30fps). The minimum and maximum frame durations are provided via libcamera controls, and will prioritise exposure time limits over any AGC request. V4L2_CID_VBLANK is controlled through the staggered writer, just like the exposure and gain controls. Signed-off-by: Naushir Patuck --- include/libcamera/ipa/raspberrypi.h | 1 + src/ipa/raspberrypi/cam_helper.cpp | 35 +++++++++++++++- src/ipa/raspberrypi/cam_helper.hpp | 15 ++++++- src/ipa/raspberrypi/cam_helper_imx219.cpp | 13 +++++- src/ipa/raspberrypi/cam_helper_imx477.cpp | 11 ++++- src/ipa/raspberrypi/cam_helper_ov5647.cpp | 11 ++++- src/ipa/raspberrypi/raspberrypi.cpp | 41 ++++++++++++++++--- .../pipeline/raspberrypi/raspberrypi.cpp | 3 +- 8 files changed, 117 insertions(+), 13 deletions(-) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index 01fe5abc..160ca681 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -65,6 +65,7 @@ static const ControlInfoMap Controls = { { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, { &controls::ColourCorrectionMatrix, ControlInfo(-16.0f, 16.0f) }, { &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) }, + { &controls::FrameDurations, ControlInfo(1.0e3f, 1.0e9f) }, }; } /* namespace RPi */ diff --git a/src/ipa/raspberrypi/cam_helper.cpp b/src/ipa/raspberrypi/cam_helper.cpp index c8ac3232..7fd3e7f6 100644 --- a/src/ipa/raspberrypi/cam_helper.cpp +++ b/src/ipa/raspberrypi/cam_helper.cpp @@ -34,8 +34,10 @@ CamHelper *CamHelper::Create(std::string const &cam_name) return nullptr; } -CamHelper::CamHelper(MdParser *parser) - : parser_(parser), initialized_(false) +CamHelper::CamHelper(MdParser *parser, unsigned int maxFrameLength, + unsigned int frameIntegrationDiff) + : parser_(parser), initialized_(false), maxFrameLength_(maxFrameLength), + frameIntegrationDiff_(frameIntegrationDiff) { } @@ -56,6 +58,35 @@ double CamHelper::Exposure(uint32_t exposure_lines) const return exposure_lines * mode_.line_length / 1000.0; } +uint32_t CamHelper::GetVBlanking(double &exposure, double minFrameDuration, + double maxFrameDuration) const +{ + uint32_t frameLengthMin, frameLengthMax, vblank; + uint32_t exposureLines = ExposureLines(exposure); + + assert(initialized_); + + /* + * Clip frame length by the frame duration range and the maximum allowable + * value in the sensor, given by maxFrameLength_. + */ + frameLengthMin = std::clamp(1e3 * minFrameDuration / mode_.line_length, + mode_.height, maxFrameLength_); + frameLengthMax = std::clamp(1e3 * maxFrameDuration / mode_.line_length, + mode_.height, maxFrameLength_); + /* + * Limit the exposure to the maximum frame duration requested, and + * re-calculate if it has been clipped. + */ + exposureLines = std::min(frameLengthMax - frameIntegrationDiff_, exposureLines); + exposure = Exposure(exposureLines); + + /* Limit the vblank to the range allowed by the frame length limits. */ + vblank = std::clamp(exposureLines + frameIntegrationDiff_, + frameLengthMin, frameLengthMax) - mode_.height; + return vblank; +} + void CamHelper::SetCameraMode(const CameraMode &mode) { mode_ = mode; diff --git a/src/ipa/raspberrypi/cam_helper.hpp b/src/ipa/raspberrypi/cam_helper.hpp index 044c2866..b1739a57 100644 --- a/src/ipa/raspberrypi/cam_helper.hpp +++ b/src/ipa/raspberrypi/cam_helper.hpp @@ -62,12 +62,15 @@ class CamHelper { public: static CamHelper *Create(std::string const &cam_name); - CamHelper(MdParser *parser); + CamHelper(MdParser *parser, unsigned int maxFrameLength, + unsigned int frameIntegrationDiff); virtual ~CamHelper(); void SetCameraMode(const CameraMode &mode); MdParser &Parser() const { return *parser_; } uint32_t ExposureLines(double exposure_us) const; double Exposure(uint32_t exposure_lines) const; // in us + virtual uint32_t GetVBlanking(double &exposure_us, double minFrameDuration, + double maxFrameDuration) const; virtual uint32_t GainCode(double gain) const = 0; virtual double Gain(uint32_t gain_code) const = 0; virtual void GetDelays(int &exposure_delay, int &gain_delay) const; @@ -76,10 +79,20 @@ public: virtual unsigned int HideFramesModeSwitch() const; virtual unsigned int MistrustFramesStartup() const; virtual unsigned int MistrustFramesModeSwitch() const; + protected: MdParser *parser_; CameraMode mode_; + +private: bool initialized_; + /* Largest possible frame length, in units of lines. */ + unsigned int maxFrameLength_; + /* + * Smallest difference between the frame length and integration time, + * in units of lines. + */ + unsigned int frameIntegrationDiff_; }; // This is for registering camera helpers with the system, so that the diff --git a/src/ipa/raspberrypi/cam_helper_imx219.cpp b/src/ipa/raspberrypi/cam_helper_imx219.cpp index db8ab879..8688ec09 100644 --- a/src/ipa/raspberrypi/cam_helper_imx219.cpp +++ b/src/ipa/raspberrypi/cam_helper_imx219.cpp @@ -49,13 +49,22 @@ public: double Gain(uint32_t gain_code) const override; unsigned int MistrustFramesModeSwitch() const override; bool SensorEmbeddedDataPresent() const override; + +private: + /* + * Smallest difference between the frame length and integration time, + * in units of lines. + */ + static constexpr int frameIntegrationDiff = 4; + /* Largest possible frame length, in units of lines. */ + static constexpr int maxFrameLength = 0xffff; }; CamHelperImx219::CamHelperImx219() #if ENABLE_EMBEDDED_DATA - : CamHelper(new MdParserImx219()) + : CamHelper(new MdParserImx219(), maxFrameLength, frameIntegrationDiff) #else - : CamHelper(new MdParserRPi()) + : CamHelper(new MdParserRPi(), maxFrameLength, frameIntegrationDiff) #endif { } diff --git a/src/ipa/raspberrypi/cam_helper_imx477.cpp b/src/ipa/raspberrypi/cam_helper_imx477.cpp index 0e896ac7..53961310 100644 --- a/src/ipa/raspberrypi/cam_helper_imx477.cpp +++ b/src/ipa/raspberrypi/cam_helper_imx477.cpp @@ -38,10 +38,19 @@ public: uint32_t GainCode(double gain) const override; double Gain(uint32_t gain_code) const override; bool SensorEmbeddedDataPresent() const override; + +private: + /* + * Smallest difference between the frame length and integration time, + * in units of lines. + */ + static constexpr int frameIntegrationDiff = 22; + /* Largest possible frame length, in units of lines. */ + static constexpr int maxFrameLength = 0xffdc; }; CamHelperImx477::CamHelperImx477() - : CamHelper(new MdParserImx477()) + : CamHelper(new MdParserImx477(), maxFrameLength, frameIntegrationDiff) { } diff --git a/src/ipa/raspberrypi/cam_helper_ov5647.cpp b/src/ipa/raspberrypi/cam_helper_ov5647.cpp index dc5d8275..7630c127 100644 --- a/src/ipa/raspberrypi/cam_helper_ov5647.cpp +++ b/src/ipa/raspberrypi/cam_helper_ov5647.cpp @@ -22,6 +22,15 @@ public: unsigned int HideFramesModeSwitch() const override; unsigned int MistrustFramesStartup() const override; unsigned int MistrustFramesModeSwitch() const override; + +private: + /* + * Smallest difference between the frame length and integration time, + * in units of lines. + */ + static constexpr int frameIntegrationDiff = 4; + /* Largest possible frame length, in units of lines. */ + static constexpr int maxFrameLength = 0xffff; }; /* @@ -30,7 +39,7 @@ public: */ CamHelperOv5647::CamHelperOv5647() - : CamHelper(new MdParserRPi()) + : CamHelper(new MdParserRPi(), maxFrameLength, frameIntegrationDiff) { } diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 60cfdc27..4e7c2b94 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -58,6 +58,8 @@ namespace libcamera { /* Configure the sensor with these values initially. */ constexpr double DefaultAnalogueGain = 1.0; constexpr unsigned int DefaultExposureTime = 20000; +constexpr double defaultMinFrameDuration = 1e6 / 30.0; +constexpr double defaultMaxFrameDuration = 1e6 / 0.01; LOG_DEFINE_CATEGORY(IPARPI) @@ -145,6 +147,9 @@ private: /* LS table allocation passed in from the pipeline handler. */ FileDescriptor lsTableHandle_; void *lsTable_; + + /* Frame duration (1/fps) given in microseconds. */ + double minFrameDuration_, maxFrameDuration_; }; int IPARPi::init(const IPASettings &settings) @@ -266,7 +271,8 @@ void IPARPi::configure(const CameraSensorInfo &sensorInfo, sensorMetadata = helper_->SensorEmbeddedDataPresent(); result->data.push_back(gainDelay); - result->data.push_back(exposureDelay); + result->data.push_back(exposureDelay); /* FOR EXPOSURE ctrl */ + result->data.push_back(exposureDelay); /* For VBLANK ctrl */ result->data.push_back(sensorMetadata); result->operation |= RPi::IPA_CONFIG_STAGGERED_WRITE; @@ -335,6 +341,8 @@ void IPARPi::configure(const CameraSensorInfo &sensorInfo, AgcStatus agcStatus; agcStatus.shutter_time = DefaultExposureTime; agcStatus.analogue_gain = DefaultAnalogueGain; + minFrameDuration_ = defaultMinFrameDuration; + maxFrameDuration_ = defaultMaxFrameDuration; applyAGC(&agcStatus, ctrls); result->controls.emplace_back(ctrls); @@ -712,6 +720,17 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + case controls::FRAME_DURATIONS: { + auto frameDurations = ctrl.second.get>(); + + /* This will be applied once AGC recalculations occur. */ + minFrameDuration_ = frameDurations[0]; + maxFrameDuration_ = frameDurations[1]; + libcameraMetadata_.set(controls::FrameDurations, + { frameDurations[0], frameDurations[1] }); + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() @@ -882,7 +901,12 @@ void IPARPi::applyAWB(const struct AwbStatus *awbStatus, ControlList &ctrls) void IPARPi::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) { int32_t gainCode = helper_->GainCode(agcStatus->analogue_gain); - int32_t exposureLines = helper_->ExposureLines(agcStatus->shutter_time); + + /* GetVBlanking might clip exposure time to the fps limits. */ + double exposure = agcStatus->shutter_time; + int32_t vblanking = helper_->GetVBlanking(exposure, minFrameDuration_, + maxFrameDuration_); + int32_t exposureLines = helper_->ExposureLines(exposure); if (unicamCtrls_.find(V4L2_CID_ANALOGUE_GAIN) == unicamCtrls_.end()) { LOG(IPARPI, Error) << "Can't find analogue gain control"; @@ -894,13 +918,20 @@ void IPARPi::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) return; } - LOG(IPARPI, Debug) << "Applying AGC Exposure: " << agcStatus->shutter_time - << " (Shutter lines: " << exposureLines << ") Gain: " + LOG(IPARPI, Debug) << "Applying AGC Exposure: " << exposure + << " (Shutter lines: " << exposureLines << ", AGC requested " + << agcStatus->shutter_time << ") Gain: " << agcStatus->analogue_gain << " (Gain Code: " << gainCode << ")"; - ctrls.set(V4L2_CID_ANALOGUE_GAIN, gainCode); + /* + * Due to the behavior of V4L2, the current value of VBLANK could clip the + * exposure time without us knowing. The next time though this function should + * clip exposure correctly. + */ + ctrls.set(V4L2_CID_VBLANK, vblanking); ctrls.set(V4L2_CID_EXPOSURE, exposureLines); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, gainCode); } void IPARPi::applyDG(const struct AgcStatus *dgStatus, ControlList &ctrls) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 2ec1f6e7..13349f31 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1221,7 +1221,8 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config) if (!staggeredCtrl_) { staggeredCtrl_.init(unicam_[Unicam::Image].dev(), { { V4L2_CID_ANALOGUE_GAIN, result.data[resultIdx++] }, - { V4L2_CID_EXPOSURE, result.data[resultIdx++] } }); + { V4L2_CID_EXPOSURE, result.data[resultIdx++] }, + { V4L2_CID_VBLANK, result.data[resultIdx++] } }); sensorMetadata_ = result.data[resultIdx++]; } }