From patchwork Thu Jul 2 10:53:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 8553 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 940B9BE905 for ; Thu, 2 Jul 2020 10:53:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 152AF60C5B; Thu, 2 Jul 2020 12:53:43 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="O4cuPLwe"; dkim-atps=neutral Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1FA8F603B3 for ; Thu, 2 Jul 2020 12:53:42 +0200 (CEST) Received: by mail-wm1-x32d.google.com with SMTP id 17so27370479wmo.1 for ; Thu, 02 Jul 2020 03:53:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sVayGIGwKG3GvXC6jRhNzM2V+z4slLKJl0H3Fq3690s=; b=O4cuPLwezUAhQLyEcdhIQlMvnVz39O4WrYUTDWFAJFzfwTsNtjQq9IZdvE7m/vKjzG LQqMsAqg27rxxdSTTkFVoH/N5f7pBGBPsnpzgszOTr0wG1Il9fICSuUbVKYZaJjzZngE HqP1hnUlRx/kpwGjJEYXRgxIBMXfFIKnCP8yVOIvgO6iTbIDph5/2UO4zqh1gzQDXVvd W2c3ayTIHswX+KiWkZtdTZurt7dbaIoAPv3cQPA7ltC0jipfvIca9hyRJnvvQIV+IbH8 Mg8MBS2hhjT/xVc/Uorv5Z1UzgtV33a8iJFE/jlUYR4j+3wpedM9wBAwBISjT7kW1HHg GwcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sVayGIGwKG3GvXC6jRhNzM2V+z4slLKJl0H3Fq3690s=; b=IdcpeCbYB4nAkPBGVBj1k+tTo7f57jmcxHxjcXuBT7Jhc5scACfxfexYS5JfzKoBPZ eJwK9m7Q2KlOFsrpFa90t6umMaRxH5bT2tisuutj1V9IarZ/7hhyGj09a74y5nB0usHL MsBmYgwfjlVTZ1u26j6f127KXJFoPRPGrueG2oKvNuxFuC98+whWVcICDzmaP1TC9ZGM FXAxo2ElrEZ8bP3ICLumzm/Rz4CcPjp2e9Q0mZEL91PFzeHeJKIVcEKhiSWgbdF0mzFI UVg36aykqWZ4aW6sXocpfiBerCeRdN860Hy4uCsbHT8iz6sszsHJE/S/pMbKHu5N1i7x eohw== X-Gm-Message-State: AOAM530kah4WYLoWrz/1Al8564SkL8fSp6a6Myz5Wf9PDVen4mEyybX4 TuCgbybFGWv8EmkhO5xwxVJl5gWEETE= X-Google-Smtp-Source: ABdhPJyMJJ0fOTfUQw1cpJtt1W5AdoKJ2/8LoqAL1PIh9XIxsX6AJKsgY/lOo4Lpie1mDTkq8Eoqtg== X-Received: by 2002:a1c:f003:: with SMTP id a3mr30549703wmb.119.1593687221529; Thu, 02 Jul 2020 03:53:41 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id p25sm9768444wmg.39.2020.07.02.03.53.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2020 03:53:41 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Thu, 2 Jul 2020 11:53:36 +0100 Message-Id: <20200702105337.31161-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200702105337.31161-1-david.plowman@raspberrypi.com> References: <20200702105337.31161-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/2] libcamera: Implement digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" These changes add a Digital Zoom control, tacking a rectangle as its argument, indicating the region of the sensor output that will be zoomed up to the final output size. Additionally, we need have a method returning the "sensorCrop" which gives the dimensions of the sensor output within which we can pan and zoom. --- include/libcamera/camera.h | 2 ++ include/libcamera/internal/pipeline_handler.h | 4 +++ src/libcamera/camera.cpp | 26 +++++++++++++++++++ src/libcamera/control_ids.yaml | 10 +++++++ 4 files changed, 42 insertions(+) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 9c0e58f..d57b606 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -92,6 +92,8 @@ public: std::unique_ptr generateConfiguration(const StreamRoles &roles); int configure(CameraConfiguration *config); + Size const &getSensorCrop() const; + Request *createRequest(uint64_t cookie = 0); int queueRequest(Request *request); diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index 22e629a..37e069a 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -89,6 +89,8 @@ public: const char *name() const { return name_; } + Size const &getSensorCrop() const { return sensorCrop_; } + protected: void registerCamera(std::shared_ptr camera, std::unique_ptr data); @@ -100,6 +102,8 @@ protected: CameraManager *manager_; + Size sensorCrop_; + private: void mediaDeviceDisconnected(MediaDevice *media); virtual void disconnect(); diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 69a1b44..6614c93 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -793,6 +793,32 @@ int Camera::configure(CameraConfiguration *config) return 0; } +/** + * \brief Return the size of the sensor image being used to form the output + * + * This method returns the size, in pixels, of the raw image read from the + * sensor and which is used to form the output image(s). Note that these + * values take account of any cropping performed on the sensor output so + * as to produce the correct aspect ratio. It would normally be necessary + * to retrieve these values in order to calculate correct parameters for + * digital zoom. + * + * Example: a sensor mode may produce a 1920x1440 output image. But if an + * application has requested a 16:9 image, the values returned here would + * be 1920x1080 - the largest portion of the sensor output that provides + * the correct aspect ratio. + * + * \context This function is \threadsafe. It will only return valid + * (non-zero) values when the camera has been configured. + * + * \return The dimensions of the sensor image in use. + */ + +Size const &Camera::getSensorCrop() const +{ + return p_->pipe_->getSensorCrop(); +} + /** * \brief Create a request object for the camera * \param[in] cookie Opaque cookie for application use diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 8c3e4c7..ac72e2a 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -251,4 +251,14 @@ controls: higher than anyone could reasonably want. Negative values are not allowed. Note also that sharpening is not applied to raw streams. + + - DigitalZoom: + type: Rectangle + description: | + Sets the portion of the full sensor image, in pixels, that will be + used for digital zoom. That is, this part of the sensor output will + be scaled up to make the full size output image (and everything else + discarded). To obtain the "full sensor image" that is available, the + method Camera::getOutputCrop() should be called once the camera is + configured. An application may pan and zoom within this rectangle. ... From patchwork Thu Jul 2 10:53:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 8554 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BEE69BE905 for ; Thu, 2 Jul 2020 10:53:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 88FA160C55; Thu, 2 Jul 2020 12:53:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="DBh2mNrc"; dkim-atps=neutral Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7D4AF60C58 for ; Thu, 2 Jul 2020 12:53:43 +0200 (CEST) Received: by mail-wr1-x443.google.com with SMTP id h5so27594730wrc.7 for ; Thu, 02 Jul 2020 03:53:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gkLgitQJ/2bvHDpVFnNbn5whUaq3pqiy9fByQxOHyHI=; b=DBh2mNrc9goErsH8FPd6zRbhI/wGQGAfjfR1x3ZPASaZirpon1OJ2PgdknqSRPY2a1 QkUyzq5K0R4iy9XAfTEJ39lnPAWYFQi7wMkHpsmX2+Xp2JfeTAWhRoVGU6eWQJo8JJOB HWr1aMM1yc0EZz81ZGYl5s2iNDOylibMeSnn3r7Vl1Ww+pHC5qcOVEXUX0KsVA2uMOhh THh5kfoXkeE20sof3gvEth0RkHX0W9/O2pV6e+Kra97rjjAhh1zZ5h/sKjfbPj3F6QnW GNdJumD4tjliJeR5vqVvl5SOX6i67UnRzhU0h0nllG4FsUhfJ8PtKKQrzB8GLACSmws6 Qezw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gkLgitQJ/2bvHDpVFnNbn5whUaq3pqiy9fByQxOHyHI=; b=igB4MScdSPBykrgqX6g0X1+jAZYguITcBty4JqIOG2vONZLofhT/SBKBkvZc/DrfIm vJ9HS+69m1gCKUo3o0G8DrkwXvguNJhlhiLZdTSZiyCqWcSPrMe8M85oLPhWJ8uu1+FU 2zux8b+CyMtY6jApig+x18q1T1HGYjS9tfCAJUmxzF3Yvmd8/KuQyBjA+F8ahib2F9Qo fL/hBwaiSm891ig+wXdTBlWZk5BN7OetVcm7CR/tdH8xngMSmKaOrVzP009yVzOdaruF 5H9yYMJbQv0qsjsxhuyGM4a1bwgJRiisM0e51SYEPUyPJJvF9oJ3lAYRgTY7yIgylfaA Yl1w== X-Gm-Message-State: AOAM530ma5Pfw1sGaSFT7P9sUQ0pZNLzUlL5QvyDAoIRv+LtTbvPRFHg LsbCIRnLwW65icr5nACIkcyNT39Ci84= X-Google-Smtp-Source: ABdhPJzB0S8MwvFc44jm57Zbn4Itz4bWBvPecTjpSyAJJe6YqfPxu3+GfHWtfRwqjF8gF/TLZ0UVFw== X-Received: by 2002:adf:f10a:: with SMTP id r10mr649815wro.406.1593687222845; Thu, 02 Jul 2020 03:53:42 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id p25sm9768444wmg.39.2020.07.02.03.53.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2020 03:53:42 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Thu, 2 Jul 2020 11:53:37 +0100 Message-Id: <20200702105337.31161-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200702105337.31161-1-david.plowman@raspberrypi.com> References: <20200702105337.31161-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/2] libcamera: raspberrypi: Implement digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" These changes implement digital zoom for the Raspberry Pi. We detect the digital zoom control and update the V4L2 "selection" before starting the ISP. Libcamera metadata is always filled in by the IPAs, so we intercept it when it comes back to the pipeline handler and update the digital zoom region with the rectangle that we actually used. --- include/libcamera/ipa/raspberrypi.h | 1 + src/ipa/raspberrypi/raspberrypi.cpp | 10 ++++ .../pipeline/raspberrypi/raspberrypi.cpp | 59 +++++++++++++++++++ 3 files changed, 70 insertions(+) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index a18ce9a..e66402e 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -52,6 +52,7 @@ static const ControlInfoMap RPiControls = { { &controls::Contrast, ControlInfo(0.0f, 32.0f) }, { &controls::Saturation, ControlInfo(0.0f, 32.0f) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, + { &controls::DigitalZoom, ControlInfo(Rectangle{ 0, 0, 0, 0 }, Rectangle{ 65535, 65535, 65535, 65535 }, Rectangle{ 0, 0, 0, 0 }) }, }; } /* namespace libcamera */ diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index bc89ab5..b4e42c1 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -645,6 +645,16 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + case controls::DIGITAL_ZOOM: { + /* + * We send the zoom info back in the metadata where the pipeline + * handler will update it to the values actually used. + */ + Rectangle crop = ctrl.second.get(); + libcameraMetadata_.set(controls::DigitalZoom, crop); + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 9d887b7..25cbe13 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -37,6 +37,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(RPI) +static constexpr int ISP_MIN_SIZE = 64; + using V4L2PixFmtMap = std::map>; namespace { @@ -322,6 +324,13 @@ public: void handleStreamBuffer(FrameBuffer *buffer, const RPiStream *stream); void handleState(); + void initSensorCrop(Rectangle const &crop) + { + /* The initial zoom region is the whole of the sensorCrop_. */ + sensorCrop_ = crop; + zoomRect_ = Rectangle{ 0, 0, crop.width, crop.height }; + } + CameraSensor *sensor_; /* Array of Unicam and ISP device streams and associated buffers/streams. */ RPiDevice unicam_; @@ -358,6 +367,14 @@ private: bool dropFrame_; int ispOutputCount_; + /* + * sensorCrop_ is the largest region of the full sensor output that matches + * the desired aspect ratio, and therefore represents the area within + * which we can pan and zoom. zoomRect_ is the portion from within the + * sensorCrop_ that pan/zoom is currently using. + */ + Rectangle sensorCrop_; + Rectangle zoomRect_; }; class RPiCameraConfiguration : public CameraConfiguration @@ -763,6 +780,14 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) crop.x = (sensorFormat.size.width - crop.width) >> 1; crop.y = (sensorFormat.size.height - crop.height) >> 1; + + if (crop.width < ISP_MIN_SIZE || crop.height < ISP_MIN_SIZE) { + LOG(RPI, Error) << "Crop from sensor " << crop.width << "x" << crop.height << " is too small"; + return -1; + } + sensorCrop_ = Size(crop.width, crop.height); + data->initSensorCrop(crop); + data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop); ret = configureIPA(camera); @@ -1226,6 +1251,13 @@ void RPiCameraData::queueFrameAction(unsigned int frame, const IPAOperationData handleStreamBuffer(buffer, &isp_[Isp::Stats]); /* Fill the Request metadata buffer with what the IPA has provided */ requestQueue_.front()->metadata() = std::move(action.controls[0]); + ControlList &metadata = requestQueue_.front()->metadata(); + /* + * The IPA sends the zoom metadata back to us, but we're the ones who + * actually know what we used, so must fill it in correctly. + */ + if (metadata.contains(controls::DigitalZoom)) + metadata.set(controls::DigitalZoom, zoomRect_); state_ = State::IpaComplete; break; } @@ -1548,6 +1580,33 @@ void RPiCameraData::tryRunPipeline() */ Request *request = requestQueue_.front(); + if (request->controls().contains(controls::DigitalZoom)) { + Rectangle rect = request->controls().get(controls::DigitalZoom); + /* + * If a new digital zoom value was given, check that it lies within the + * available sensorCrop_, coercing it if necessary. + */ + if (rect.width && rect.height) { + zoomRect_ = rect; + if (zoomRect_.width < ISP_MIN_SIZE) + zoomRect_.width = ISP_MIN_SIZE; + else if (zoomRect_.width > sensorCrop_.width) + zoomRect_.width = sensorCrop_.width; + if (zoomRect_.height < ISP_MIN_SIZE) + zoomRect_.height = ISP_MIN_SIZE; + else if (zoomRect_.height > sensorCrop_.height) + zoomRect_.height = sensorCrop_.height; + if (zoomRect_.x + zoomRect_.width > sensorCrop_.width) + zoomRect_.x = sensorCrop_.width - zoomRect_.width; + if (zoomRect_.y + zoomRect_.height > sensorCrop_.height) + zoomRect_.y = sensorCrop_.height - zoomRect_.height; + } + Rectangle sensorRect = zoomRect_; + sensorRect.x += sensorCrop_.x; + sensorRect.y += sensorCrop_.y; + isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &sensorRect); + } + /* * Process all the user controls by the IPA. Once this is complete, we * queue the ISP output buffer listed in the request to start the HW