[{"id":12996,"web_url":"https://patchwork.libcamera.org/comment/12996/","msgid":"<228707fc-3cc8-1434-34cc-4c4f1dec5e33@ideasonboard.com>","date":"2020-10-05T20:54:34","subject":"Re: [libcamera-devel] [PATCH v3 5/6] libcamera: pipeline:\n\traspberrypi: Implementation of digital zoom","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi David,\n\nOn 29/09/2020 17:39, David Plowman wrote:\n> During configure() we update the SensorOutputSize to the correct value\n> for this camera mode (which we also note internally) and work out the\n> minimum crop size allowed by the ISP.\n> \n> Whenever a new IspCrop request is received we check it's valid and\n> apply it to the ISP V4L2 device. We also forward it to the IPA so that\n> the IPA can return the values used in the image metadata.\n> \n> Signed-off-by: David Plowman \n> Reviewed-by: Jacopo Mondi \n> ---\n> include/libcamera/ipa/raspberrypi.h | 1 +\n> src/ipa/raspberrypi/raspberrypi.cpp | 7 +++\n> .../pipeline/raspberrypi/raspberrypi.cpp | 47 +++++++++++++++++++\n> 3 files changed, 55 insertions(+)\n> \n> diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h\n> index b3041591..9ef5f6fc 100644\n> --- a/include/libcamera/ipa/raspberrypi.h\n> +++ b/include/libcamera/ipa/raspberrypi.h\n> @@ -60,6 +60,7 @@ static const ControlInfoMap Controls = {\n> \t{ &controls::Saturation, ControlInfo(0.0f, 32.0f) },\n> \t{ &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) },\n> \t{ &controls::ColourCorrectionMatrix, ControlInfo(-16.0f, 16.0f) },\n> +\t{ &controls::IspCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) },\n> };\n> \n> } /* namespace RPi */\n> diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp\n> index b0c7d1c1..dc73c20b 100644\n> --- a/src/ipa/raspberrypi/raspberrypi.cpp\n> +++ b/src/ipa/raspberrypi/raspberrypi.cpp\n> @@ -699,6 +699,13 @@ void IPARPi::queueRequest(const ControlList &controls)\n> \t\t\tbreak;\n> \t\t}\n> \n> +\t\tcase controls::ISP_CROP: {\n> +\t\t\t/* Just copy the information back. */\n> +\t\t\tRectangle crop = ctrl.second.get();\n> +\t\t\tlibcameraMetadata_.set(controls::IspCrop, crop);\n> +\t\t\tbreak;\n> +\t\t}\n> +\n> \t\tdefault:\n> \t\t\tLOG(IPARPI, Warning)\n> \t\t\t\t<< \"Ctrl \" << controls::controls.at(ctrl.first)->name()\n> diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> index d2bee150..4a6d0bf4 100644\n> --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp\n> @@ -193,6 +193,11 @@ public:\n> \tbool flipsAlterBayerOrder_;\n> \tBayerFormat::Order nativeBayerOrder_;\n> \n> +\t/* For handling digital zoom. */\n> +\tSize ispMinSize_;\n> +\tSize sensorOutputSize_;\n> +\tRectangle lastIspCrop_;\n> +\n> \tunsigned int dropFrameCount_;\n> \n> private:\n> @@ -587,6 +592,14 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n> \t */\n> \tret = data->isp_[Isp::Input].dev()->setFormat(&sensorFormat);\n> \n> +\t/*\n> +\t * Update the SensorOutputSize to the correct value for this camera mode.\n> +\t *\n> +\t * \\todo Move this property to CameraConfiguration when that\n> +\t * feature will be made available and set it at validate() time\n> +\t */\n> +\tdata->properties_.set(properties::SensorOutputSize, sensorFormat.size);\n> +\n> \t/*\n> \t * See which streams are requested, and route the user\n> \t * StreamConfiguration appropriately.\n> @@ -677,6 +690,16 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n> \t\treturn ret;\n> \t}\n> \n> +\t/*\n> +\t * Figure out the smallest selection the ISP will allow. We also store\n> +\t * the output image size, in pixels, from the sensor. These will be\n> +\t * used for digital zoom.\n> +\t */\n> +\tRectangle testCrop(0, 0, 1, 1);\n> +\tdata->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &testCrop);\n> +\tdata->ispMinSize_ = testCrop.size();\n\nI might have called testCrop minCrop, but I don't think that matters\nmuch. testCrop is ok too.\n\nPart of me is itching to say this should happen at camera creation time\nor such, as I don't think it would change on any configure parameters?\n\nBut arguing against myself - that would then involve changing settings\non the device which we should avoid, and as this is used on the\nper-frame controls, I think this is actually probably OK here. It's not\na big overhead for configure()\n\n\n> +\tdata->sensorOutputSize_ = sensorFormat.size;\n> +\n> \t/* Adjust aspect ratio by providing crops on the input image. */\n> \tRectangle crop{ 0, 0, sensorFormat.size };\n> \n> @@ -693,6 +716,8 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)\n> \tcrop.y = (sensorFormat.size.height - crop.height) >> 1;\n> \tdata->isp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop);\n> \n> +\tdata->lastIspCrop_ = crop;\n> +\n> \tret = data->configureIPA(config);\n> \tif (ret)\n> \t\tLOG(RPI, Error) << \"Failed to configure the IPA: \" << ret;\n> @@ -1588,6 +1613,28 @@ void RPiCameraData::tryRunPipeline()\n> \t/* Take the first request from the queue and action the IPA. */\n> \tRequest *request = requestQueue_.front();\n> \n> +\tif (request->controls().contains(controls::IspCrop)) {\n> +\t\tRectangle crop = request->controls().get(controls::IspCrop);\n> +\t\tif (crop.width && crop.height) {\n> +\t\t\t/*\n> +\t\t\t * The crop that we set must be:\n> +\t\t\t * 1. At least as big as ispMinSize_, once that's been\n> +\t\t\t * enlarged to the same aspect ratio.\n> +\t\t\t * 2. With the same mid-point, if possible.\n> +\t\t\t * 3. But it can't go outside the sensor area.\n> +\t\t\t */\n> +\t\t\tSize minSize = ispMinSize_.alignedUpToAspectRatio(crop.size());\n\nI like how the helpers make this expressive, over the underlying\ncalculations.\n\n> +\t\t\tSize size = crop.size().expandedTo(minSize);\n> +\t\t\tcrop = size.centredTo(crop).boundedTo(Rectangle(sensorOutputSize_));\n> +\n> +\t\t\tif (crop != lastIspCrop_)\n> +\t\t\t\tisp_[Isp::Input].dev()->setSelection(V4L2_SEL_TGT_CROP, &crop);\n> +\t\t\tlastIspCrop_ = crop;\n> +\t\t}\n> +\n> +\t\trequest->controls().set(controls::IspCrop, lastIspCrop_);\n> +\t}\n> +\n\nReviewed-by: Kieran Bingham \n\n> \t/*\n> \t * Process all the user controls by the IPA. Once this is complete, we\n> \t * queue the ISP output buffer listed in the request to start the HW\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9F728C3B5C\n\tfor ;\n\tMon, 5 Oct 2020 20:54:39 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2C2AC63C46;\n\tMon, 5 Oct 2020 22:54:39 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 231C563BED\n\tfor ;\n\tMon, 5 Oct 2020 22:54:38 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 81E353B;\n\tMon, 5 Oct 2020 22:54:37 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"pTw6HsxX\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1601931277;\n\tbh=2IDm9SzaXyFb+lKJNzKLSeuQ6PljZ/bbzw5jpgGep5M=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=pTw6HsxXm8NT87rbO0Saajz5aURgN35OUG4e+2ZFsjCcMiZMeQrCEWAJQv/dfDFae\n\tBmgWS8+y64cFEc8DSdSfDZ2y3j32Rke3QXyQh3wU9TfUg+is2PiRpTmIVhDAq5kaCL\n\tuhq3Fhuo3PGbCer7ZQWzxsN5rdWBg6LQ4f2iDjfk=","To":"David Plowman ,\n\tlibcamera-devel@lists.libcamera.org","References":"<20200929164000.15429-1-david.plowman@raspberrypi.com>\n\t<20200929164000.15429-6-david.plowman@raspberrypi.com>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<228707fc-3cc8-1434-34cc-4c4f1dec5e33@ideasonboard.com>","Date":"Mon, 5 Oct 2020 21:54:34 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"<20200929164000.15429-6-david.plowman@raspberrypi.com>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [PATCH v3 5/6] libcamera: pipeline:\n\traspberrypi: Implementation of digital zoom","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]