From patchwork Thu Jul 9 09:15:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 8713 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2096FBD790 for ; Thu, 9 Jul 2020 09:16:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DD72C611AD; Thu, 9 Jul 2020 11:16:08 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="EkCT+/If"; dkim-atps=neutral Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4CF0561186 for ; Thu, 9 Jul 2020 11:16:07 +0200 (CEST) Received: by mail-wm1-x342.google.com with SMTP id l2so1066837wmf.0 for ; Thu, 09 Jul 2020 02:16:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JxYFmoILtnLrUDRRdBNhPp9I1szEqxBDjAjdoZD8Yc8=; b=EkCT+/If0EbTWmiIExd9SY0UDzEE6WDOMyCJYjROBuXSmvhWV0sFKN7zz4U9q/hY+M 5XXF6DyQtGjWkvUY9s11ekg2tua9m3kAde1HckeRkA+o0/KsA5q3yv2eYBXmNAgxGoaZ Qnia4sq+XbYToDv5DeDLc4DQCNJRPmkog5DBsiqflg2QyRIcYomVLekXal5AiWptKWnb eU0qbxBo4Gb9u4ScQRUnOTrUF/nkmYWQjdrBApSlCtIDtKAG3y9WB1BfFjhCWBJotJy3 R8MuTSTfKaF5PNfg511hlUfdnlvrNjz66mBtAQhPcPw4na8pRmraKhTNk/GtAjf9SXeG 1wMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JxYFmoILtnLrUDRRdBNhPp9I1szEqxBDjAjdoZD8Yc8=; b=baJvQs45fR65oMrfVoDNA+7aQURWnn0C+joiaJNtSDuQwvm9MgoWn5Xh6+Z6R4VYED aBZxHiF3D0TGef61fW1Yq/PAq7TQc+ETyFIPzERbCl6Tm6KcjHhdsm87grREP1D1yG1b eXZko+CdX9JgY15ZjPw5bO27llIaBts7kNPELr7WbDLwQZTKEx+0DAnwE5YPyvWGlCKU N6gVGOiFGwYG08CeI6e3Ojk9w8xYqhDH+UETr9B8KFUTGOWIL18YMu+E8AmxOVKtluLL zbiAN6U6X3zxhBgBgw2BckSvWyi26qTS+5HYXIQ+xUouuhU5IQCN57UMQm3cFt5gp1ld ur9Q== X-Gm-Message-State: AOAM530RURtJpAhmb3jZd9KUUcgPOTJwWQJgLhuRFhBdZPbTI+oE5qWY QR7G/JdLatVAwydMtxVY6xH4v2PagFU= X-Google-Smtp-Source: ABdhPJxYVCJylPo/gu6YJ9S53LEOlhm+Ap35Gn7mWejRAZo0obEfhiGukTI/OoAIpOFZ6pXq56zUBA== X-Received: by 2002:a1c:7706:: with SMTP id t6mr12538569wmi.3.1594286166652; Thu, 09 Jul 2020 02:16:06 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id h84sm4135765wme.22.2020.07.09.02.16.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jul 2020 02:16:06 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Thu, 9 Jul 2020 10:15:54 +0100 Message-Id: <20200709091555.1617-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200709091555.1617-1-david.plowman@raspberrypi.com> References: <20200709091555.1617-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/2] libcamera: Implement digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" These changes add a Digital Zoom control, tacking a rectangle as its argument, indicating the region of the sensor output that will be zoomed up to the final output size. Additionally, we need have a method returning the "sensorCrop" which gives the dimensions of the sensor output within which we can pan and zoom. --- include/libcamera/camera.h | 2 ++ include/libcamera/internal/pipeline_handler.h | 4 +++ src/libcamera/camera.cpp | 26 +++++++++++++++++++ src/libcamera/control_ids.yaml | 10 +++++++ 4 files changed, 42 insertions(+) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 4d1a4a9..dd07f7a 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -92,6 +92,8 @@ public: std::unique_ptr generateConfiguration(const StreamRoles &roles = {}); int configure(CameraConfiguration *config); + Size const &getSensorCrop() const; + Request *createRequest(uint64_t cookie = 0); int queueRequest(Request *request); diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index 22e629a..37e069a 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -89,6 +89,8 @@ public: const char *name() const { return name_; } + Size const &getSensorCrop() const { return sensorCrop_; } + protected: void registerCamera(std::shared_ptr camera, std::unique_ptr data); @@ -100,6 +102,8 @@ protected: CameraManager *manager_; + Size sensorCrop_; + private: void mediaDeviceDisconnected(MediaDevice *media); virtual void disconnect(); diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 69a1b44..6614c93 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -793,6 +793,32 @@ int Camera::configure(CameraConfiguration *config) return 0; } +/** + * \brief Return the size of the sensor image being used to form the output + * + * This method returns the size, in pixels, of the raw image read from the + * sensor and which is used to form the output image(s). Note that these + * values take account of any cropping performed on the sensor output so + * as to produce the correct aspect ratio. It would normally be necessary + * to retrieve these values in order to calculate correct parameters for + * digital zoom. + * + * Example: a sensor mode may produce a 1920x1440 output image. But if an + * application has requested a 16:9 image, the values returned here would + * be 1920x1080 - the largest portion of the sensor output that provides + * the correct aspect ratio. + * + * \context This function is \threadsafe. It will only return valid + * (non-zero) values when the camera has been configured. + * + * \return The dimensions of the sensor image in use. + */ + +Size const &Camera::getSensorCrop() const +{ + return p_->pipe_->getSensorCrop(); +} + /** * \brief Create a request object for the camera * \param[in] cookie Opaque cookie for application use diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 988b501..5a099d5 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -262,4 +262,14 @@ controls: In this respect, it is not necessarily aimed at providing a way to implement a focus algorithm by the application, rather an indication of how in-focus a frame is. + + - DigitalZoom: + type: Rectangle + description: | + Sets the portion of the full sensor image, in pixels, that will be + used for digital zoom. That is, this part of the sensor output will + be scaled up to make the full size output image (and everything else + discarded). To obtain the "full sensor image" that is available, the + method Camera::getOutputCrop() should be called once the camera is + configured. An application may pan and zoom within this rectangle. ... From patchwork Thu Jul 9 09:15:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 8714 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 85ADEBD790 for ; Thu, 9 Jul 2020 09:16:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 508F961190; Thu, 9 Jul 2020 11:16:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="PsfAul/6"; dkim-atps=neutral Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C382561190 for ; Thu, 9 Jul 2020 11:16:08 +0200 (CEST) Received: by mail-wm1-x330.google.com with SMTP id l2so1066905wmf.0 for ; Thu, 09 Jul 2020 02:16:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UC8cy68txEeNI8Pq3ScFqRcZuj6BWeZjqq2AYqY2pqE=; b=PsfAul/6dHW2zpwis5a4R9cxo3VBizbKwmjaCOwQWbA/HKetUg/4tOrVvWTOW86FNd LyGqNLcYSTLOPhXKfUFtRWxuq42OAp02OXrzEOhMY3oBGWNkXMcONajuVNaqKamjusiv vAbflUv2jEAh0X8RAd5qPyQjej2G20ByizhzVEdOVZYOGMKsOmvBC5gCL0jzW7hH4+Pp 4RN7R5phW8/c0O2qwvi06BC7Nl8hHLqRrPvHCm2lgYwO+FrRsahETG5emRycd7E36eEE IkA4CCfaD3h8UdWna7y9joNXqMdIdH+VdJCw2+fiqJgUuW+yS9xdNtQRqmq7MMk1B9TH pG2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UC8cy68txEeNI8Pq3ScFqRcZuj6BWeZjqq2AYqY2pqE=; b=YSTclfHb6jYHHZXejh7c5xXBvmFkCE1UWDbBzVN5RxVnxk1u01vob7ZsSrWsycTsMb 1VMqTCPWKjWvsN4JLxkUNTymIvYaq+MQhqw9gdAMqC4gGhRLMzHTH6wvAL0sqRUer3mB Shn3MOSI9PXNpu1ex0xYTxfoQJ18mWd4ZWz/QHicm3VrxDlxBTVXzEHZCZbdE0MkAwjb LC78KmpAqgKkQTtAZqV0R0P2E8bKYceV8HfyvucCYw92EtVZT7Q01DmjJ5+ju/5hXH61 ief4nUtyW5PVAXwoXv9ZKzJb28H0DLGwU8d1PE7QdIhKA3fYhkMsj0Mgqx90uK2XTOCC tnaQ== X-Gm-Message-State: AOAM5306ZWVwaa4TWY3jZ38W+ASGAItl9x5DyLQIeRm0Cr5bSm89FUUy RRuRFlu2tqJv65WrjbkvxRxcmDY2tVM= X-Google-Smtp-Source: ABdhPJwWb5u9GU+keBeN0Mc8kEZ0XhfgSoEVfSDIokifudtWsKIZ2NYdDYwmRa9InB5E3TsObh0aXg== X-Received: by 2002:a1c:238d:: with SMTP id j135mr12857809wmj.71.1594286168036; Thu, 09 Jul 2020 02:16:08 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id h84sm4135765wme.22.2020.07.09.02.16.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jul 2020 02:16:07 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Thu, 9 Jul 2020 10:15:55 +0100 Message-Id: <20200709091555.1617-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200709091555.1617-1-david.plowman@raspberrypi.com> References: <20200709091555.1617-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/2] libcamera: raspberrypi: Implement digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" These changes implement digital zoom for the Raspberry Pi. We detect the digital zoom control and update the V4L2 "selection" before starting the ISP. We also update the value in the control that we send to the IPA, so that it has the correct value. --- include/libcamera/ipa/raspberrypi.h | 1 + src/ipa/raspberrypi/raspberrypi.cpp | 10 ++++ .../pipeline/raspberrypi/raspberrypi.cpp | 56 ++++++++++++++++++- 3 files changed, 66 insertions(+), 1 deletion(-) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index a18ce9a..e66402e 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -52,6 +52,7 @@ static const ControlInfoMap RPiControls = { { &controls::Contrast, ControlInfo(0.0f, 32.0f) }, { &controls::Saturation, ControlInfo(0.0f, 32.0f) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, + { &controls::DigitalZoom, ControlInfo(Rectangle{ 0, 0, 0, 0 }, Rectangle{ 65535, 65535, 65535, 65535 }, Rectangle{ 0, 0, 0, 0 }) }, }; } /* namespace libcamera */ diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index b1f2786..3c68078 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -658,6 +658,16 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + case controls::DIGITAL_ZOOM: { + /* + * We send the zoom info back in the metadata where the pipeline + * handler will update it to the values actually used. + */ + Rectangle crop = ctrl.second.get(); + libcameraMetadata_.set(controls::DigitalZoom, crop); + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index f4966f8..55db11d 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -287,7 +287,8 @@ class RPiCameraData : public CameraData public: RPiCameraData(PipelineHandler *pipe) : CameraData(pipe), sensor_(nullptr), lsTable_(nullptr), - state_(State::Stopped), dropFrame_(false), ispOutputCount_(0) + state_(State::Stopped), dropFrame_(false), ispOutputCount_(0), + ispMinSize_(0) { } @@ -322,6 +323,14 @@ public: void handleStreamBuffer(FrameBuffer *buffer, const RPiStream *stream); void handleState(); + void initSensorCrop(Rectangle const &crop, unsigned int ispMinSize) + { + /* The initial zoom region is the whole of the sensorCrop_. */ + sensorCrop_ = crop; + zoomRect_ = Rectangle{ 0, 0, crop.width, crop.height }; + ispMinSize_ = ispMinSize; + } + CameraSensor *sensor_; /* Array of Unicam and ISP device streams and associated buffers/streams. */ RPiDevice unicam_; @@ -358,6 +367,15 @@ private: bool dropFrame_; int ispOutputCount_; + /* + * sensorCrop_ is the largest region of the full sensor output that matches + * the desired aspect ratio, and therefore represents the area within + * which we can pan and zoom. zoomRect_ is the portion from within the + * sensorCrop_ that pan/zoom is currently using. + */ + Rectangle sensorCrop_; + Rectangle zoomRect_; + unsigned int ispMinSize_; }; class RPiCameraConfiguration : public CameraConfiguration @@ -744,6 +762,10 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) return ret; } + Rectangle testCrop = { 0, 0, 1, 1 }; + data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop); + const unsigned int ispMinSize = testCrop.width; + /* Adjust aspect ratio by providing crops on the input image. */ Rectangle crop = { .x = 0, @@ -763,8 +785,12 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) crop.x = (sensorFormat.size.width - crop.width) >> 1; crop.y = (sensorFormat.size.height - crop.height) >> 1; + data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop); + sensorCrop_ = Size(crop.width, crop.height); + data->initSensorCrop(crop, ispMinSize); + ret = configureIPA(camera); if (ret) LOG(RPI, Error) << "Failed to configure the IPA: " << ret; @@ -1553,6 +1579,34 @@ void RPiCameraData::tryRunPipeline() */ Request *request = requestQueue_.front(); + if (request->controls().contains(controls::DigitalZoom)) { + Rectangle rect = request->controls().get(controls::DigitalZoom); + /* + * If a new digital zoom value was given, check that it lies within the + * available sensorCrop_, coercing it if necessary. + */ + if (rect.width && rect.height) { + zoomRect_ = rect; + if (zoomRect_.width < ispMinSize_) + zoomRect_.width = ispMinSize_; + else if (zoomRect_.width > sensorCrop_.width) + zoomRect_.width = sensorCrop_.width; + if (zoomRect_.height < ispMinSize_) + zoomRect_.height = ispMinSize_; + else if (zoomRect_.height > sensorCrop_.height) + zoomRect_.height = sensorCrop_.height; + if (zoomRect_.x + zoomRect_.width > sensorCrop_.width) + zoomRect_.x = sensorCrop_.width - zoomRect_.width; + if (zoomRect_.y + zoomRect_.height > sensorCrop_.height) + zoomRect_.y = sensorCrop_.height - zoomRect_.height; + } + Rectangle sensorRect = zoomRect_; + sensorRect.x += sensorCrop_.x; + sensorRect.y += sensorCrop_.y; + isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &sensorRect); + request->controls().set(controls::DigitalZoom, zoomRect_); + } + /* * Process all the user controls by the IPA. Once this is complete, we * queue the ISP output buffer listed in the request to start the HW