From patchwork Mon Oct 19 12:51:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 10099 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4DFFCBDB1F for ; Mon, 19 Oct 2020 12:52:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F197360CE6; Mon, 19 Oct 2020 14:52:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="Ua6sUqmZ"; dkim-atps=neutral Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 728CD613AD for ; Mon, 19 Oct 2020 14:52:09 +0200 (CEST) Received: by mail-wr1-x42a.google.com with SMTP id j7so7197855wrt.9 for ; Mon, 19 Oct 2020 05:52:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Da3jfoEzqJ/NANYN3bEr5iKNEQ2R+Mn1bTNJdEXTVFk=; b=Ua6sUqmZxZ0EY/shdVdYAKjlwZo7YFiERPfEhkpRVL6T/iFOUxn51/vpNW0GULdYp+ w+JVk8xAYIVlnYjo+n4GIVjd2E9ALUeEG6OFBqPTAjsKUOGcwP9pzFt9Hb5sfZXc4eUz pdhr7iVkUQMaozVL4b3x2yB3589q62sjWhiwORAWi+/2WKDKxFCmoLgN+c1QPAzXPUA3 WWh6Kh513jW8K83AfHK0NpikK3r+0VnKmJm0XJYhesE/R+8eTWZ8BUSKQqoq/KgsdxDw opgISN1NPqU43PTEmrHI3Ad4mpgzHZ/qUsTheqHjoIty/AcBbjOOn9Z+IZB8ZD1gae3H tzgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Da3jfoEzqJ/NANYN3bEr5iKNEQ2R+Mn1bTNJdEXTVFk=; b=olp4htdYahCmyXFGqZ27FM1KIADtponD4mguQuD5Pcp6wa0sWKCZAKL54C1KBkkdWS ui2EuLDgiwqhrmYInPjXeRYU6C3LgXkfIaUFajcVQZicbLqLbep0l4ConQmwGrRkdLYb OuBpAnGuIzCUJF9C0YnfMCitxwug2pJVFx5J1UizTJxNonxOVUiXdika/M/EWPb9HHKs GZfiHKftsK12IBdKlIDDcMODZo+ZupzsjwPaqeN2J6YyeswajflRhCgVpE+XP99+he6J LgIupfQRfYvc/c8C6uJsi9MgZIDW7jZUdBypULS2pMEg/q1RdKFlM2QJaO6euNJ270DN m4zA== X-Gm-Message-State: AOAM53189c4vmqp2Tpb4DjKnTWRLnsPM/YJu5k5Dz/jQFebikqQI+Ffc eCGs+jZgrFi1Eug3W2+SDgkuX/1+IggJVQ== X-Google-Smtp-Source: ABdhPJzU4lXl9D53GGPW30F3rsVoVSnIb6R+CmUwkBDvKJZo6fBanDzOyQOtA3SL8MCpa9HlbUyENw== X-Received: by 2002:a5d:424e:: with SMTP id s14mr21392702wrr.149.1603111928827; Mon, 19 Oct 2020 05:52:08 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:1ce1:9965:4328:89c4]) by smtp.gmail.com with ESMTPSA id q6sm16634335wma.0.2020.10.19.05.52.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Oct 2020 05:52:08 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Mon, 19 Oct 2020 13:51:56 +0100 Message-Id: <20201019125156.26751-6-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201019125156.26751-1-david.plowman@raspberrypi.com> References: <20201019125156.26751-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 5/5] libcamera: pipeline: raspberrypi: Implementation of digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" During configure() we update the ScalerCropMaximum to the correct value for this camera mode (which we also note internally) and work out the minimum crop size allowed by the ISP. Whenever a new ScalerCrop request is received we check it's valid and apply it to the ISP V4L2 device. We also forward it to the IPA so that the IPA can return the values used in the image metadata. Signed-off-by: David Plowman Reviewed-by: Laurent Pinchart --- include/libcamera/ipa/raspberrypi.h | 1 + src/ipa/raspberrypi/raspberrypi.cpp | 7 ++ .../pipeline/raspberrypi/raspberrypi.cpp | 70 +++++++++++++++---- 3 files changed, 63 insertions(+), 15 deletions(-) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index b3041591..bb4e1a8c 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -60,6 +60,7 @@ static const ControlInfoMap Controls = { { &controls::Saturation, ControlInfo(0.0f, 32.0f) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, { &controls::ColourCorrectionMatrix, ControlInfo(-16.0f, 16.0f) }, + { &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) }, }; } /* namespace RPi */ diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 48a72dd2..e777f0b4 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -699,6 +699,13 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + case controls::SCALER_CROP: { + /* Just copy the information back. */ + Rectangle crop = ctrl.second.get(); + libcameraMetadata_.set(controls::ScalerCrop, crop); + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 26dbd257..43b860ae 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -193,6 +193,11 @@ public: bool flipsAlterBayerOrder_; BayerFormat::Order nativeBayerOrder_; + /* For handling digital zoom. */ + CameraSensorInfo sensorInfo_; + Rectangle lastIspCrop_; + Size ispMinSize_; + unsigned int dropFrameCount_; private: @@ -677,26 +682,30 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) return ret; } - /* Adjust aspect ratio by providing crops on the input image. */ - Rectangle crop{ 0, 0, sensorFormat.size }; - - int ar = maxSize.height * sensorFormat.size.width - maxSize.width * sensorFormat.size.height; - if (ar > 0) - crop.width = maxSize.width * sensorFormat.size.height / maxSize.height; - else if (ar < 0) - crop.height = maxSize.height * sensorFormat.size.width / maxSize.width; + /* Figure out the smallest selection the ISP will allow. */ + Rectangle testCrop(0, 0, 1, 1); + data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop); + data->ispMinSize_ = testCrop.size(); - crop.width &= ~1; - crop.height &= ~1; + /* Adjust aspect ratio by providing crops on the input image. */ + Rectangle crop = sensorFormat.size.boundedToAspectRatio(maxSize).centeredTo(sensorFormat.size.center()); + data->lastIspCrop_ = crop; - crop.x = (sensorFormat.size.width - crop.width) >> 1; - crop.y = (sensorFormat.size.height - crop.height) >> 1; data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop); ret = data->configureIPA(config); if (ret) LOG(RPI, Error) << "Failed to configure the IPA: " << ret; + /* + * Update the ScalerCropMaximum to the correct value for this camera mode. + * For us, it's the same as the "analogue crop". + * + * \todo Make this property the ScalerCrop maximum value when dynamic + * controls are available and set it at validate() time + */ + data->properties_.set(properties::ScalerCropMaximum, data->sensorInfo_.analogCrop); + return ret; } @@ -1147,8 +1156,8 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config) ipaConfig.data.push_back(static_cast(lsTable_.fd())); } - CameraSensorInfo sensorInfo = {}; - int ret = sensor_->sensorInfo(&sensorInfo); + /* We store the CameraSensorInfo for digital zoom calculations. */ + int ret = sensor_->sensorInfo(&sensorInfo_); if (ret) { LOG(RPI, Error) << "Failed to retrieve camera sensor info"; return ret; @@ -1157,7 +1166,7 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config) /* Ready the IPA - it must know about the sensor resolution. */ IPAOperationData result; - ipa_->configure(sensorInfo, streamConfig, entityControls, ipaConfig, + ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result); unsigned int resultIdx = 0; @@ -1588,6 +1597,37 @@ void RPiCameraData::tryRunPipeline() /* Take the first request from the queue and action the IPA. */ Request *request = requestQueue_.front(); + if (request->controls().contains(controls::ScalerCrop)) { + Rectangle crop = request->controls().get(controls::ScalerCrop); + + if (crop.width && crop.height) { + /* First scale the crop from sensor native to camera mode pixels. */ + crop.translateBy(-sensorInfo_.analogCrop.x, -sensorInfo_.analogCrop.y); + crop.rescaleTo(sensorInfo_.analogCrop.size(), sensorInfo_.outputSize); + + /* + * The crop that we set must be: + * 1. At least as big as ispMinSize_, once that's been + * enlarged to the same aspect ratio. + * 2. With the same mid-point, if possible. + * 3. But it can't go outside the sensor area. + */ + Size minSize = ispMinSize_.expandedToAspectRatio(crop.size()); + Size size = crop.size().expandedTo(minSize); + crop = size.centeredTo(crop.center()).clampedTo(Rectangle(sensorInfo_.outputSize)); + + if (crop != lastIspCrop_) + isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop); + lastIspCrop_ = crop; + } + + /* We must scale the actual crop used back into native pixels. */ + crop = lastIspCrop_.rescaledTo(sensorInfo_.outputSize, sensorInfo_.analogCrop.size()); + crop.translateBy(sensorInfo_.analogCrop.x, sensorInfo_.analogCrop.y); + + request->controls().set(controls::ScalerCrop, crop); + } + /* * Process all the user controls by the IPA. Once this is complete, we * queue the ISP output buffer listed in the request to start the HW