From patchwork Thu Jul 23 08:43:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 8936 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CC1B0BD86F for ; Thu, 23 Jul 2020 08:43:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9B6016112A; Thu, 23 Jul 2020 10:43:46 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="QH68CWdb"; dkim-atps=neutral Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3373B60939 for ; Thu, 23 Jul 2020 10:43:44 +0200 (CEST) Received: by mail-wm1-x344.google.com with SMTP id o8so4197075wmh.4 for ; Thu, 23 Jul 2020 01:43:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fucOP9zskuOzWC8DYYa7syEbPMpQ+PXRcZIhZYYEiFY=; b=QH68CWdb37u8QXTAW4pIh2VJA+nrPJToR/P3GtDPBmpssEZ6eVuUySYbsxHqkZAVSZ yg+So5vi79Mk4tTunxREyWq1MUvlUaPATR91ZG55k0lMgdR7xrkLLSYXm8cyOirPk56N Jr/12PkUR0BLMQTc7rdvsyU2OWRDymseEGoR/sXaH/vlro4/3GIs9PLHh6vRY4IppvC0 b7xVDAOSWPwOwyE7BRt1Ja7UmdbgDiSY/TbCPew4Vb8z0KQI/uiKbI/XOLlc9Y08BOxv 796frSh3LbgQh3BM/NOd3HXTWmQx5J14lF+e1DuXnHeGw9bTLuS1c9+IEkUIrcY91Z7o VraA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fucOP9zskuOzWC8DYYa7syEbPMpQ+PXRcZIhZYYEiFY=; b=rYbhYujfzFgqxPj3St2OM2R+bH96hrd4kLl/fX/7FCRgFngPot0sEmJhfAECB4Q3po celF48rmGNiIQcPDIT6sY+eKfytdSpQu8/BHph+d0ZOWxyRCZlUAxVD9u5agS17jrLb2 6zz2GG/2HFRvGJmi5m7qPvn5fafZD3rOvnEp+xdNuU4EVfCcmjpKJJAoXNlP/lSn2qSS EjyocbjcP7K5sFALgtUnMjDjlIcgE1dbVmlSRv4VjVkkKUCCVhtIfOqe6hjHESNWbFrm PkjbBfYCDp7wPE60UuCD1WfdCAij1KcIxXdqsuMJABiBfcZZcVXmANzMk8cx+KGZijow /NPw== X-Gm-Message-State: AOAM530CYFoAhjNnz1M8uUdHxOWqro3IdipMHXgTUeMntPdfgZPlA+z/ DmiN+jxKTRT2lCqSHiRKYPS9Ou/o3ES27g== X-Google-Smtp-Source: ABdhPJzXMcT/2zSW/WfITTmf1YN3+kEGmfFlFpwwEFZK/1Ug1YoXFl9Xwj3O0zWEU4Q5HSrUJxFaow== X-Received: by 2002:a1c:ed01:: with SMTP id l1mr3240685wmh.33.1595493823543; Thu, 23 Jul 2020 01:43:43 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id t2sm2809450wmb.25.2020.07.23.01.43.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2020 01:43:43 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Thu, 23 Jul 2020 09:43:37 +0100 Message-Id: <20200723084338.13711-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200723084338.13711-1-david.plowman@raspberrypi.com> References: <20200723084338.13711-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 1/2] libcamera: Infrastructure for digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" These changes add a Digital Zoom control, taking a rectangle as its argument, indicating the region of the sensor output that the pipeline will "zoom up" to the final output size. Additionally, we need to have a method returning the "pipelineCrop" which gives the dimensions of the sensor output, taken by the pipeline, and within which we can subsequently pan and zoom. Signed-off-by: David Plowman --- include/libcamera/camera.h | 2 ++ include/libcamera/internal/pipeline_handler.h | 4 +++ src/libcamera/camera.cpp | 27 +++++++++++++++++++ src/libcamera/control_ids.yaml | 10 +++++++ 4 files changed, 43 insertions(+) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 4d1a4a9..6819b8e 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -92,6 +92,8 @@ public: std::unique_ptr generateConfiguration(const StreamRoles &roles = {}); int configure(CameraConfiguration *config); + Size const &getPipelineCrop() const; + Request *createRequest(uint64_t cookie = 0); int queueRequest(Request *request); diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index 22e629a..5bfe890 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -89,6 +89,8 @@ public: const char *name() const { return name_; } + Size const &getPipelineCrop() const { return pipelineCrop_; } + protected: void registerCamera(std::shared_ptr camera, std::unique_ptr data); @@ -100,6 +102,8 @@ protected: CameraManager *manager_; + Size pipelineCrop_; + private: void mediaDeviceDisconnected(MediaDevice *media); virtual void disconnect(); diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 69a1b44..f8b8ec6 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -793,6 +793,33 @@ int Camera::configure(CameraConfiguration *config) return 0; } +/** + * \brief Return the size of the sensor image being used by the pipeline + * to create the output. + * + * This method returns the size, in pixels, of the raw image read from the + * sensor and which is used by the pipeline to form the output image(s) + * (rescaling if necessary). Note that these values take account of any + * cropping performed on the sensor output so as to produce the correct + * aspect ratio. It would normally be necessary to retrieve these values + * in order to calculate correct parameters for digital zoom. + * + * Example: a sensor mode may produce a 1920x1440 output image. But if an + * application has requested a 16:9 image, the values returned here might + * be 1920x1080 - the largest portion of the sensor output that provides + * the correct aspect ratio. + * + * \context This function is \threadsafe. It will only return valid + * (non-zero) values when the camera has been configured. + * + * \return The dimensions of the sensor image used by the pipeline. + */ + +Size const &Camera::getPipelineCrop() const +{ + return p_->pipe_->getPipelineCrop(); +} + /** * \brief Create a request object for the camera * \param[in] cookie Opaque cookie for application use diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 988b501..5a099d5 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -262,4 +262,14 @@ controls: In this respect, it is not necessarily aimed at providing a way to implement a focus algorithm by the application, rather an indication of how in-focus a frame is. + + - DigitalZoom: + type: Rectangle + description: | + Sets the portion of the full sensor image, in pixels, that will be + used for digital zoom. That is, this part of the sensor output will + be scaled up to make the full size output image (and everything else + discarded). To obtain the "full sensor image" that is available, the + method Camera::getOutputCrop() should be called once the camera is + configured. An application may pan and zoom within this rectangle. ... From patchwork Thu Jul 23 08:43:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 8937 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EE1C2BD86F for ; Thu, 23 Jul 2020 08:43:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1AC2A61146; Thu, 23 Jul 2020 10:43:47 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="edfQQA/7"; dkim-atps=neutral Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9664B60536 for ; Thu, 23 Jul 2020 10:43:45 +0200 (CEST) Received: by mail-wm1-x32d.google.com with SMTP id y24so231931wma.1 for ; Thu, 23 Jul 2020 01:43:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ueNwn2zsSD1A5AY+qG+4p8wVmR8jVQtrsuCUmF/L9V8=; b=edfQQA/73RiTwe5PWCuWIMCQkGKaW9UT63yBjbM+3/D00LUGfOvG2YbOvgAkiC2TYF 7AN9FDaC6mu9D/lmkDqzwDJLUEBRNmw/DtHNHN5bJmafd1LDIPpXz+YAkDkRWG4Cpw3h vkRcJb1IVoOnAiDbd4CHftyFoLLs45fXK17Dh8hvwoYNDNWz++mHXU4+PG7EFM1FsU51 ifAF3/4qAhUCbysYml5+b38d8HhxokOgfYT5C7krTDDXEyvJqNzizpRMKWraVRsaiZdn T2njPxoTs1KrlWweehxIX2TrFRR9XIBI548ThCcgW/NBqGNNd42TWmtydceU+6sPIvKZ sR6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ueNwn2zsSD1A5AY+qG+4p8wVmR8jVQtrsuCUmF/L9V8=; b=KcQYtyCsO1pJpC1veKvc4kzd9sEEqEVG6gGCwAus0njO7PWYHRhsJcQX5GVzobQU4D SzYfpjaKTVpqEyWKrS6g3OnrsiBKKbqHI40HKuw456/ClXxxBEEqPh++hC0zreSv9Hyk rkrpyDY65Fv8GF+iG4rokYmP/ITNAfiP75qkuwV6jOnpqxQSze5JG/WK4+RICA5LxJ0p OWjML8Cli8nhuRhdp6+RPAYdySU276H9EN6iOf1zNsm3WLfczcOkG8JWW/Zrpyc52f+x ug2+WGcgZOI7/qng5RPdn8TOeljrksx9+0z6MuFmZxVU6L8d4sMw5e3QSHc6ult+UmBo pd6w== X-Gm-Message-State: AOAM5331YuHxVFmaFJ/OPEQu8NM4TPgjq+rVnCahrhfJ+zzZT2cxL6E3 wAs78DOdhMiuoO4UzsOMDydLvzuHo0CMSA== X-Google-Smtp-Source: ABdhPJziqnoyNRdpwQWIPoaV/29q1EI9Cag3VDbFs5q+gsz8oh1cXjqN8buxWgHt2nmtmu3h1QWwRg== X-Received: by 2002:a1c:18e:: with SMTP id 136mr3093026wmb.93.1595493824921; Thu, 23 Jul 2020 01:43:44 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id t2sm2809450wmb.25.2020.07.23.01.43.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2020 01:43:44 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Thu, 23 Jul 2020 09:43:38 +0100 Message-Id: <20200723084338.13711-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200723084338.13711-1-david.plowman@raspberrypi.com> References: <20200723084338.13711-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 2/2] libcamera: raspberrypi: Implement digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" These changes implement digital zoom for the Raspberry Pi. We detect the digital zoom control and update the V4L2 "selection" before starting the ISP. We also update the value in the control that we send to the IPA, so that it has the correct value. Signed-off-by: David Plowman --- include/libcamera/ipa/raspberrypi.h | 1 + src/ipa/raspberrypi/raspberrypi.cpp | 10 +++ .../pipeline/raspberrypi/raspberrypi.cpp | 72 +++++++++++++++++++ 3 files changed, 83 insertions(+) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index a493776..f42e13e 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -56,6 +56,7 @@ static const ControlInfoMap RPiControls = { { &controls::Contrast, ControlInfo(0.0f, 32.0f) }, { &controls::Saturation, ControlInfo(0.0f, 32.0f) }, { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, + { &controls::DigitalZoom, ControlInfo(Rectangle(0, 0, 0, 0), Rectangle(65535, 65535, 65535, 65535), Rectangle(0, 0, 0, 0)) }, }; } /* namespace libcamera */ diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 7bd0488..f759d4d 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -679,6 +679,16 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + case controls::DIGITAL_ZOOM: { + /* + * We send the zoom info back in the metadata where the pipeline + * handler will update it to the values actually used. + */ + Rectangle crop = ctrl.second.get(); + libcameraMetadata_.set(controls::DigitalZoom, crop); + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index bf1c771..6bc595f 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -308,6 +308,14 @@ public: void handleStreamBuffer(FrameBuffer *buffer, const RPiStream *stream); void handleState(); + void initPipelineCrop(Rectangle const &crop, Size const &ispMinSize) + { + /* The initial zoom region is the whole of the pipelineCrop_. */ + pipelineCrop_ = crop; + zoomRect_ = Rectangle(0, 0, crop.width, crop.height); + ispMinSize_ = ispMinSize; + } + CameraSensor *sensor_; /* Array of Unicam and ISP device streams and associated buffers/streams. */ RPiDevice unicam_; @@ -344,6 +352,15 @@ private: bool dropFrame_; int ispOutputCount_; + /* + * pipelineCrop_ is the largest region of the full sensor output that matches + * the desired aspect ratio, and therefore represents the area within + * which we can pan and zoom. zoomRect_ is the portion from within the + * sensorCrop_ that pan/zoom is currently using. + */ + Rectangle pipelineCrop_; + Rectangle zoomRect_; + Size ispMinSize_; }; class RPiCameraConfiguration : public CameraConfiguration @@ -736,6 +753,10 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) return ret; } + Rectangle testCrop(0, 0, 1, 1); + data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop); + unsigned int minSize = std::max(testCrop.width, testCrop.height); + /* Adjust aspect ratio by providing crops on the input image. */ Rectangle crop{ 0, 0, sensorFormat.size }; @@ -750,8 +771,17 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) crop.x = (sensorFormat.size.width - crop.width) >> 1; crop.y = (sensorFormat.size.height - crop.height) >> 1; + data->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop); + pipelineCrop_ = Size(crop.width, crop.height); + Size ispMinSize(minSize, minSize); + if (crop.width > crop.height) + ispMinSize.width = minSize * crop.width / crop.height; + else if (crop.width < crop.height) + ispMinSize.height = minSize * crop.height / crop.width; + data->initPipelineCrop(crop, ispMinSize); + ret = data->configureIPA(); if (ret) LOG(RPI, Error) << "Failed to configure the IPA: " << ret; @@ -1488,6 +1518,30 @@ void RPiCameraData::checkRequestCompleted() } } +static void fixup_dimension(int &start, unsigned int &size, + unsigned int minSize, unsigned int maxSize) +{ + /* + * If the size being requested it too small we must increase it, but + * we try to keep the mid point the same. Though if this would mean + * extending beyond the available bounds, then we have to line ourselves + * up flush. + */ + int mid = start + size / 2; + size = std::min(std::max(size, minSize), maxSize); + start = mid - size / 2; + if (start < 0) + start = 0; + else if (start + size > maxSize) + start = maxSize - size; +} + +static void fixup_rectangle(Rectangle ®ion, Size const &minSize, Size const &maxSize) +{ + fixup_dimension(region.x, region.width, minSize.width, maxSize.width); + fixup_dimension(region.y, region.height, minSize.height, maxSize.height); +} + void RPiCameraData::tryRunPipeline() { FrameBuffer *bayerBuffer, *embeddedBuffer; @@ -1541,6 +1595,24 @@ void RPiCameraData::tryRunPipeline() */ Request *request = requestQueue_.front(); + if (request->controls().contains(controls::DigitalZoom)) { + Rectangle rect = request->controls().get(controls::DigitalZoom); + /* + * If a new digital zoom value was given, check that it lies within the + * available pipelineCrop_, and that it's not smaller than permitted. + */ + if (rect.width && rect.height) { + zoomRect_ = rect; + fixup_rectangle(zoomRect_, ispMinSize_, + Size(pipelineCrop_.width, pipelineCrop_.height)); + } + Rectangle sensorRect = zoomRect_; + sensorRect.x += pipelineCrop_.x; + sensorRect.y += pipelineCrop_.y; + isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &sensorRect); + request->controls().set(controls::DigitalZoom, zoomRect_); + } + /* * Process all the user controls by the IPA. Once this is complete, we * queue the ISP output buffer listed in the request to start the HW