From patchwork Fri Oct 23 10:21:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 10218 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 29D58C3D3C for ; Fri, 23 Oct 2020 10:22:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D5EFF61795; Fri, 23 Oct 2020 12:22:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="c3haEIYM"; dkim-atps=neutral Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4961D619AB for ; Fri, 23 Oct 2020 12:22:08 +0200 (CEST) Received: by mail-wm1-x330.google.com with SMTP id k18so919988wmj.5 for ; Fri, 23 Oct 2020 03:22:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Mj39ccQTSMuGU/dKsqpOkarhfi/ZGEAhU1hwhonA6KU=; b=c3haEIYMNQiW31aRlTcyRs4m5ikyOx476rc3L3QcQ7ObV2JlQlZ3zB+XMoqFR2dS30 WfUNp17nc4mfvdNnos3JUgMCMNFtZh30YsOnq2vSVMzkNVVJTrxlMWinITa9Zr6GT3oz MOtGdvq+++44q7dbrcxSFFKvFqx55xShhGl0+v0xpp2Mi57LyKq5L6+ftNgxU5mFCYgy 0vJRSwWSnVW4Z5Dl4TLuZj06d9U8GecZpqVV3PAJCl+it5sp1sr9gVaQkmaqV37+3WUK Fa+ISIarN55kSK9cifmUER+Av/jR/c/jodz4cUGkJ3DQRef08YsPR9Pk8koWj9OuUQCH JygQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Mj39ccQTSMuGU/dKsqpOkarhfi/ZGEAhU1hwhonA6KU=; b=F0Bkmrt+iUYkwD1B7XfD3Sfi2E05pmmG7vRF+x16SqglkdrOXX/pvEvOvWh7GZ3gUA aHLk5T4GadGAp4F0QupCpK5tP8LQTdrsrTA+P33wxcssGVYTnBUw340vTTH2zRAzF3E+ Xb/+xXHsg/P+nloZE44Hf73Iy0N9qYkeTPMfGjaQwjkcOZduQ8piLaaI1GuRlpKSJXJQ MTVw/3YRSx354E1MXF5kKxxUUWxRxtf+hY8lKkO8Fd5qkynaBjR1kmLGTKT/tXXgDvJz JpYTLQ2F9hyevEr1P3sRVwO6lo8j/V4fQvnqE6dD2pEMYansBHnQFuMnYZ5yvB0ImQ0m nc7A== X-Gm-Message-State: AOAM533Lm2jIsXtcV9IyhoBciL/DGKisLFBb78ilq4brKUfB+PkgrZAE OjWjPJs18nUUzZ2cVW0dsEjmrqV0pDRVlA== X-Google-Smtp-Source: ABdhPJyUVOF7LAg8vAxYUYMmEkvuQ9v4ADc0JkuSzd0LY+LFYDaQBuGbLR+qSbOOSAEzCIxc0KzPuA== X-Received: by 2002:a1c:ab0b:: with SMTP id u11mr1610697wme.165.1603448527567; Fri, 23 Oct 2020 03:22:07 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id r128sm2477831wma.20.2020.10.23.03.22.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Oct 2020 03:22:06 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Fri, 23 Oct 2020 11:21:59 +0100 Message-Id: <20201023102159.26274-6-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201023102159.26274-1-david.plowman@raspberrypi.com> References: <20201023102159.26274-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v5 5/5] libcamera: pipeline: raspberrypi: Implementation of digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" During configure() we update the ScalerCropMaximum to the correct value for this camera mode and work out the minimum crop size allowed by the ISP. Whenever a new ScalerCrop request is received we check it's valid and apply it to the ISP V4L2 device. When the IPA returns its metadata to us we add the ScalerCrop information, rescaled to sensor native pixels. Signed-off-by: David Plowman Reviewed-by: Jacopo Mondi --- include/libcamera/ipa/raspberrypi.h | 1 + src/ipa/raspberrypi/raspberrypi.cpp | 5 ++ .../pipeline/raspberrypi/raspberrypi.cpp | 82 +++++++++++++++---- 3 files changed, 72 insertions(+), 16 deletions(-) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index b23baf2f..ff2faf86 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -62,6 +62,7 @@ static const ControlInfoMap Controls = { { &controls::Saturation, ControlInfo(0.0f, 32.0f) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, { &controls::ColourCorrectionMatrix, ControlInfo(-16.0f, 16.0f) }, + { &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) }, }; } /* namespace RPi */ diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 1c255aa2..f338ff8b 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -699,6 +699,11 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + case controls::SCALER_CROP: { + /* We do nothing with this, but should avoid the warning below. */ + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 763451a8..83e91576 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -193,6 +193,11 @@ public: bool flipsAlterBayerOrder_; BayerFormat::Order nativeBayerOrder_; + /* For handling digital zoom. */ + CameraSensorInfo sensorInfo_; + Rectangle lastIspCrop_; + Size ispMinSize_; + unsigned int dropFrameCount_; private: @@ -677,26 +682,31 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) return ret; } - /* Adjust aspect ratio by providing crops on the input image. */ - Rectangle crop{ 0, 0, sensorFormat.size }; - - int ar = maxSize.height * sensorFormat.size.width - maxSize.width * sensorFormat.size.height; - if (ar > 0) - crop.width = maxSize.width * sensorFormat.size.height / maxSize.height; - else if (ar < 0) - crop.height = maxSize.height * sensorFormat.size.width / maxSize.width; + /* Figure out the smallest selection the ISP will allow. */ + Rectangle testCrop(0, 0, 1, 1); + data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop); + data->ispMinSize_ = testCrop.size(); - crop.width &= ~1; - crop.height &= ~1; + /* Adjust aspect ratio by providing crops on the input image. */ + Size size = sensorFormat.size.boundedToAspectRatio(maxSize); + Rectangle crop = size.centeredTo(sensorFormat.size.center()); + data->lastIspCrop_ = crop; - crop.x = (sensorFormat.size.width - crop.width) >> 1; - crop.y = (sensorFormat.size.height - crop.height) >> 1; data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop); ret = data->configureIPA(config); if (ret) LOG(RPI, Error) << "Failed to configure the IPA: " << ret; + /* + * Update the ScalerCropMaximum to the correct value for this camera mode. + * For us, it's the same as the "analogue crop". + * + * \todo Make this property the ScalerCrop maximum value when dynamic + * controls are available and set it at validate() time + */ + data->properties_.set(properties::ScalerCropMaximum, data->sensorInfo_.analogCrop); + return ret; } @@ -1154,8 +1164,8 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config) ipaConfig.data.push_back(static_cast(lsTable_.fd())); } - CameraSensorInfo sensorInfo = {}; - int ret = sensor_->sensorInfo(&sensorInfo); + /* We store the CameraSensorInfo for digital zoom calculations. */ + int ret = sensor_->sensorInfo(&sensorInfo_); if (ret) { LOG(RPI, Error) << "Failed to retrieve camera sensor info"; return ret; @@ -1164,7 +1174,7 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config) /* Ready the IPA - it must know about the sensor resolution. */ IPAOperationData result; - ipa_->configure(sensorInfo, streamConfig, entityControls, ipaConfig, + ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result); unsigned int resultIdx = 0; @@ -1243,8 +1253,23 @@ void RPiCameraData::queueFrameAction([[maybe_unused]] unsigned int frame, FrameBuffer *buffer = isp_[Isp::Stats].getBuffers().at(bufferId); handleStreamBuffer(buffer, &isp_[Isp::Stats]); + /* Fill the Request metadata buffer with what the IPA has provided */ - requestQueue_.front()->metadata() = std::move(action.controls[0]); + Request *request = requestQueue_.front(); + request->metadata() = std::move(action.controls[0]); + + /* + * Also update the ScalerCrop in the metadata with what we actually + * used. But we must first rescale that from ISP (camera mode) pixels + * back into sensor native pixels. + * + * Sending this information on every frame may be helpful. + */ + Rectangle crop = lastIspCrop_.scaledBy(sensorInfo_.analogCrop.size(), + sensorInfo_.outputSize); + crop.translateBy(sensorInfo_.analogCrop.topLeft()); + request->metadata().set(controls::ScalerCrop, crop); + state_ = State::IpaComplete; break; } @@ -1595,6 +1620,31 @@ void RPiCameraData::tryRunPipeline() /* Take the first request from the queue and action the IPA. */ Request *request = requestQueue_.front(); + if (request->controls().contains(controls::ScalerCrop)) { + Rectangle crop = request->controls().get(controls::ScalerCrop); + + if (crop.width && crop.height) { + /* First scale the crop from sensor native to camera mode pixels. */ + crop.translateBy(-sensorInfo_.analogCrop.topLeft()); + crop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size()); + + /* + * The crop that we set must be: + * 1. At least as big as ispMinSize_, once that's been + * enlarged to the same aspect ratio. + * 2. With the same mid-point, if possible. + * 3. But it can't go outside the sensor area. + */ + Size minSize = ispMinSize_.expandedToAspectRatio(crop.size()); + Size size = crop.size().expandedTo(minSize); + crop = size.centeredTo(crop.center()).enclosedIn(Rectangle(sensorInfo_.outputSize)); + + if (crop != lastIspCrop_) + isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop); + lastIspCrop_ = crop; + } + } + /* * Process all the user controls by the IPA. Once this is complete, we * queue the ISP output buffer listed in the request to start the HW