[{"id":12603,"web_url":"https://patchwork.libcamera.org/comment/12603/","msgid":"","date":"2020-09-21T06:24:45","subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi again everyone\n\nI was wondering if anyone else had any thoughts on this. In spite of\nthe long email which is perhaps a bit off-putting I don't think\nthere's actually very much complicated stuff going on. Anyway, I'm\nkeen to try and stop this discussion from getting marooned on the\nbacklog for too long if we can!\n\nThanks again\nDavid\n\nOn Mon, 7 Sep 2020 at 17:44, David Plowman\n wrote:\n>\n> Hi everyone\n>\n> I wanted to return to the question of digital zoom. Rather than simply\n> writing a long email I've included an implementation that shows where\n> I am. It's changed in the light of exchanges with Jacopo and others,\n> and I would say it's \"for discussion\" at this point. Principally:\n>\n> 1. I'd like users to have access to CameraSensorInfo::outputSize, and\n> I'm using Camera::properties() to get it. Unfortunately it changes\n> with every camera mode, so will we have to do this differently?\n>\n> 2. The terminology is a bit more Android-like. Because, why not?\n>\n> 3. I'm no longer enforcing any particular method of cropping. But to\n> make life easier for applications I've added some geometry helpers. To\n> understand the rationale behind these I've included some example code\n> further down that I think an application might use.\n>\n> This work is currently made up of 4 commits:\n>\n> 1. The SensorOutputSize property.\n>\n> 2. The SensorCrop control.\n>\n> 3. The geometry helper functions.\n>\n> 4. Implementation in the Raspberry Pi pipeline handler.\n>\n> Example Application Code\n>\n> Assume we have:\n>\n> camera - pointer to the Camera object\n> streamCfg - reference to the StreamConfiguration of the output image\n> request - pointer to a Request object\n> zoomFactor - floating point zoom factor, >= 1\n> dx - amount in pixels to pan left (-ve) or right (+ve)\n> dy - amount in pixels to pan up (-ve) or down (+ve)\n>\n> You'll want to understand the helper methods:\n>\n> Size Size::aspectRatioDown(const Size &ar) const\n> Returns a letter/pillarboxed version of this size.\n>\n> Size operator/(const Size &size, float f)\n> Downscale a Size object by f.\n>\n> Rectangle Size::centre(const Rectangle ®ion, int offsetX = 0, int offsetY = 0) const\n> Returns a Rectangle of this size centred at the same point as region,\n> optionally with offsets.\n>\n> Rectangle Rectangle::clamp(const Rectangle &boundary) const\n> Translates the rectangle so that no part of it lies outside boundary.\n>\n> 1. Basic zooming into the centre\n>\n> Size area = camera->properties().get(properties::SensorOutputSize);\n> Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;\n> Rectangle crop = size.centre(Rectangle(area));\n> request->controls.set(controls::SensorCrop, crop);\n>\n> (I'm hoping this first one is easy enough to understand!)\n>\n> 2. Zooming anywhere in the sensor output (even into parts of the image that\n> were not intially visible)\n>\n> Size area = camera->properties().get(properties::SensorOutputSize);\n> Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;\n> Rectangle crop = size.centre(Rectangle(area), dx, dy).clamp(Rectangle(area));\n> request->controls.set(controls::SensorCrop, crop);\n>\n> (As before, but we have offsets dx and dy to enable panning, so we\n> must also clamp the result to be sure it's within the sensor area.)\n>\n> 3. Zooming as before, but not allowing you to pan into anything that\n> wasn't initially visible\n>\n> Size area = camera->properties().get(properties::SensorOutputSize);\n> Rectangle maxCrop = area.aspectRatioDown(streamCfg.size).centre(area);\n> Size size = maxCrop.size() / zoomFactor;\n> Rectangle crop = size.centre(maxCrop, dx, dy).clamp(maxCrop);\n> request->controls.set(controls::SensorCrop, crop);\n>\n> (The subtle difference here is that we clamp the result against maxCrop,\n> not the full sensor area.)\n>\n> 4. Android\n>\n> This time we're given the \"crop\" (a Rectangle) that Android has requested. I\n> think Android only requires us to sort out the aspect ratio.\n>\n> Size size = crop.size().aspectRatioDown(streamCfg.size);\n> request->controls.set(controls::SensorCrop, size.centre(crop));\n>\n>\n> That's as far as I've got. Thoughts and comments welcome!\n>\n> Best regards\n> David\n>\n> David Plowman (4):\n> libcamera: Add SensorOutputSize property\n> libcamera: Add SensorCrop control\n> libcamera: Add geometry helper functions\n> libcamera: pipeline: raspberrypi: Implementation of digital zoom\n>\n> include/libcamera/geometry.h | 18 +++\n> include/libcamera/ipa/raspberrypi.h | 1 +\n> src/ipa/raspberrypi/raspberrypi.cpp | 7 +\n> src/libcamera/camera_sensor.cpp | 3 +\n> src/libcamera/control_ids.yaml | 9 ++\n> src/libcamera/geometry.cpp | 129 ++++++++++++++++++\n> .../pipeline/raspberrypi/raspberrypi.cpp | 41 ++++++\n> src/libcamera/property_ids.yaml | 6 +\n> 8 files changed, 214 insertions(+)\n>\n> --\n> 2.20.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C05D4C3B5B\n\tfor ;\n\tMon, 21 Sep 2020 06:24:57 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 59B4862FBF;\n\tMon, 21 Sep 2020 08:24:57 +0200 (CEST)","from mail-oi1-x241.google.com (mail-oi1-x241.google.com\n\t[IPv6:2607:f8b0:4864:20::241])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1DAE160363\n\tfor ;\n\tMon, 21 Sep 2020 08:24:56 +0200 (CEST)","by mail-oi1-x241.google.com with SMTP id x14so15727162oic.9\n\tfor ;\n\tSun, 20 Sep 2020 23:24:56 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"HnDkqeVa\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to;\n\tbh=J2H6jz6SVRYUsGj4sG0bIBea53TzpU9B3EtQHh5haFM=;\n\tb=HnDkqeVatgFTEYKloNbNXLGTY+s5m6vU4aksBadM748t3urvi8DmZELYU4IGrxnFDt\n\tELty33vTxSBEWpmLLRQuF1nrv3kCvOcIKn+d/KQJ7U7yh+Z0Q7S8HhtLJuTMN7QTAcFc\n\tD4eQ55CPW5lPqXSy+oCKnqZSgB5oemHQ2DGQ73e5AUQ6J4y6VluUcfYsrvg3LNXvajLM\n\tHYUkS2OZj0sook+NRdx3FFpaAva16/2pDzMj3LmJ7Adnuk88WlhAwIdgDYVTCQft7gh7\n\t/qyl0ZKimanEoYZhEf0id1opHrME/X2qyzn/JWbpzkWGJ6pIyZxRCd1sB/KWCnpIooFP\n\tnEWA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to;\n\tbh=J2H6jz6SVRYUsGj4sG0bIBea53TzpU9B3EtQHh5haFM=;\n\tb=iagZbLrzj8CFF+DBVBZ3A3EeEPR4ukG2lJslZJQ5nw4Serd3aQZvtHft5NZOOLzFls\n\tczzsZbb0a+GlGviw4NgSgMXMQe4OGhCeJ9C+b9HpePU6OpNPpekJlz1IJ/nX/nUEjF1s\n\tyaXrvWWIGZE41QUm5GXFBIsZFtGG/epa5IIbYdPUZQcmUjIcM21jtr2FUuk8GSEQFWpS\n\tOjD6GW/U0pbgT2euN4BDF+f0lWzBpbSzAlJrmu384ahvpgseymim2UxXuEZXBcYV/raQ\n\t3cT1EPQslmfEiszOzHtXICGIsyOlEPhbLLyiLHuHWSwUBYMJs/KyaDkbWQDxqz0YGv0S\n\tRWsQ==","X-Gm-Message-State":"AOAM530sgqWMu2zs61d2Z8JNBy9JqxkaFRDjoMTCzYBFKXUKtzua2GrR\n\tejfZLqDAYp0TZilc868UbISxCklRqH0Cn7mc7ekM6uYgISA=","X-Google-Smtp-Source":"ABdhPJySz8EPpsEBaAElkoip+robiq6DfijLte1Vm+i7eHvnKLisWImSuA5QgLSPxUaz0XqX0EwR+loM/U2v4b3zDPU=","X-Received":"by 2002:aca:ef89:: with SMTP id\n\tn131mr15771327oih.22.1600669494735; \n\tSun, 20 Sep 2020 23:24:54 -0700 (PDT)","MIME-Version":"1.0","References":"<20200907164450.13082-1-david.plowman@raspberrypi.com>","In-Reply-To":"<20200907164450.13082-1-david.plowman@raspberrypi.com>","From":"David Plowman ","Date":"Mon, 21 Sep 2020 07:24:45 +0100","Message-ID":"","To":"libcamera-devel@lists.libcamera.org","Subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":12618,"web_url":"https://patchwork.libcamera.org/comment/12618/","msgid":"<20200921104138.2bcsw6mbhhgr2bvq@uno.localdomain>","date":"2020-09-21T10:41:38","subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi David,\n\nOn Mon, Sep 21, 2020 at 07:24:45AM +0100, David Plowman wrote:\n> Hi again everyone\n>\n> I was wondering if anyone else had any thoughts on this. In spite of\n> the long email which is perhaps a bit off-putting I don't think\n> there's actually very much complicated stuff going on. Anyway, I'm\n> keen to try and stop this discussion from getting marooned on the\n> backlog for too long if we can!\n\nSorry sorry for the late reply. We also left quite some discussion\npending on the previous version, let's try to catch up here.\n\n>\n> Thanks again\n> David\n>\n> On Mon, 7 Sep 2020 at 17:44, David Plowman\n> wrote:\n> >\n> > Hi everyone\n> >\n> > I wanted to return to the question of digital zoom. Rather than simply\n> > writing a long email I've included an implementation that shows where\n> > I am. It's changed in the light of exchanges with Jacopo and others,\n> > and I would say it's \"for discussion\" at this point. Principally:\n> >\n> > 1. I'd like users to have access to CameraSensorInfo::outputSize, and\n> > I'm using Camera::properties() to get it. Unfortunately it changes\n> > with every camera mode, so will we have to do this differently?\n> >\n\nI'll reply to this in 1/4. I think it's not optimal, but we should not\nblock this feature any longer. Although, I'm afraid that application\nmight be forced to change the way they retrieve the SensorCrop (from\nCamera to CameraConfiguration). How fast do yuo expect this will be\nadopted? Do you plan to build anything on top ?\n\n> > 2. The terminology is a bit more Android-like. Because, why not?\n> >\n\n:)\n\n> > 3. I'm no longer enforcing any particular method of cropping. But to\n> > make life easier for applications I've added some geometry helpers. To\n> > understand the rationale behind these I've included some example code\n> > further down that I think an application might use.\n\nNice!\n\n> >\n> > This work is currently made up of 4 commits:\n> >\n> > 1. The SensorOutputSize property.\n> >\n> > 2. The SensorCrop control.\n> >\n> > 3. The geometry helper functions.\n> >\n> > 4. Implementation in the Raspberry Pi pipeline handler.\n> >\n\nRepluies in individual patches\n\n> > Example Application Code\n> >\n> > Assume we have:\n> >\n> > camera - pointer to the Camera object\n> > streamCfg - reference to the StreamConfiguration of the output image\n> > request - pointer to a Request object\n> > zoomFactor - floating point zoom factor, >= 1\n> > dx - amount in pixels to pan left (-ve) or right (+ve)\n> > dy - amount in pixels to pan up (-ve) or down (+ve)\n> >\n> > You'll want to understand the helper methods:\n> >\n> > Size Size::aspectRatioDown(const Size &ar) const\n> > Returns a letter/pillarboxed version of this size.\n> >\n> > Size operator/(const Size &size, float f)\n> > Downscale a Size object by f.\n> >\n> > Rectangle Size::centre(const Rectangle ®ion, int offsetX = 0, int offsetY = 0) const\n> > Returns a Rectangle of this size centred at the same point as region,\n> > optionally with offsets.\n> >\n> > Rectangle Rectangle::clamp(const Rectangle &boundary) const\n> > Translates the rectangle so that no part of it lies outside boundary.\n> >\n> > 1. Basic zooming into the centre\n> >\n> > Size area = camera->properties().get(properties::SensorOutputSize);\n> > Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;\n> > Rectangle crop = size.centre(Rectangle(area));\n> > request->controls.set(controls::SensorCrop, crop);\n> >\n> > (I'm hoping this first one is easy enough to understand!)\n> >\n> > 2. Zooming anywhere in the sensor output (even into parts of the image that\n> > were not intially visible)\n> >\n> > Size area = camera->properties().get(properties::SensorOutputSize);\n> > Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;\n> > Rectangle crop = size.centre(Rectangle(area), dx, dy).clamp(Rectangle(area));\n> > request->controls.set(controls::SensorCrop, crop);\n> >\n> > (As before, but we have offsets dx and dy to enable panning, so we\n> > must also clamp the result to be sure it's within the sensor area.)\n> >\n> > 3. Zooming as before, but not allowing you to pan into anything that\n> > wasn't initially visible\n> >\n> > Size area = camera->properties().get(properties::SensorOutputSize);\n> > Rectangle maxCrop = area.aspectRatioDown(streamCfg.size).centre(area);\n> > Size size = maxCrop.size() / zoomFactor;\n> > Rectangle crop = size.centre(maxCrop, dx, dy).clamp(maxCrop);\n> > request->controls.set(controls::SensorCrop, crop);\n> >\n> > (The subtle difference here is that we clamp the result against maxCrop,\n> > not the full sensor area.)\n\nThanks! very informative!\n\n> >\n> > 4. Android\n> >\n> > This time we're given the \"crop\" (a Rectangle) that Android has requested. I\n> > think Android only requires us to sort out the aspect ratio.\n> >\n> > Size size = crop.size().aspectRatioDown(streamCfg.size);\n> > request->controls.set(controls::SensorCrop, size.centre(crop));\n\nWe'll here have to deal with the fact Android specifies its sensor\ncrop with the full size sensor pixel array as reference, not the actual\nsensor output frame. This shall be handled in the HAL though.\n\n> >\n> >\n> > That's as far as I've got. Thoughts and comments welcome!\n> >\n> > Best regards\n> > David\n> >\n> > David Plowman (4):\n> > libcamera: Add SensorOutputSize property\n> > libcamera: Add SensorCrop control\n> > libcamera: Add geometry helper functions\n> > libcamera: pipeline: raspberrypi: Implementation of digital zoom\n> >\n> > include/libcamera/geometry.h | 18 +++\n> > include/libcamera/ipa/raspberrypi.h | 1 +\n> > src/ipa/raspberrypi/raspberrypi.cpp | 7 +\n> > src/libcamera/camera_sensor.cpp | 3 +\n> > src/libcamera/control_ids.yaml | 9 ++\n> > src/libcamera/geometry.cpp | 129 ++++++++++++++++++\n> > .../pipeline/raspberrypi/raspberrypi.cpp | 41 ++++++\n> > src/libcamera/property_ids.yaml | 6 +\n> > 8 files changed, 214 insertions(+)\n> >\n> > --\n> > 2.20.1\n> >\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8A4B9C3B5B\n\tfor ;\n\tMon, 21 Sep 2020 10:37:48 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E7AEA62FBF;\n\tMon, 21 Sep 2020 12:37:47 +0200 (CEST)","from relay2-d.mail.gandi.net (relay2-d.mail.gandi.net\n\t[217.70.183.194])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7714860366\n\tfor ;\n\tMon, 21 Sep 2020 12:37:46 +0200 (CEST)","from uno.localdomain (93-34-118-233.ip49.fastwebnet.it\n\t[93.34.118.233]) (Authenticated sender: jacopo@jmondi.org)\n\tby relay2-d.mail.gandi.net (Postfix) with ESMTPSA id 025AE4000D;\n\tMon, 21 Sep 2020 10:37:45 +0000 (UTC)"],"X-Originating-IP":"93.34.118.233","Date":"Mon, 21 Sep 2020 12:41:38 +0200","From":"Jacopo Mondi ","To":"David Plowman ","Message-ID":"<20200921104138.2bcsw6mbhhgr2bvq@uno.localdomain>","References":"<20200907164450.13082-1-david.plowman@raspberrypi.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":12626,"web_url":"https://patchwork.libcamera.org/comment/12626/","msgid":"<20200921130845.navin5h6othpwtkz@uno.localdomain>","date":"2020-09-21T13:08:45","subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi David,\n just noticed that....\n\nOn Mon, Sep 21, 2020 at 12:41:38PM +0200, Jacopo Mondi wrote:\n> Hi David,\n>\n> On Mon, Sep 21, 2020 at 07:24:45AM +0100, David Plowman wrote:\n> > Hi again everyone\n> >\n> > I was wondering if anyone else had any thoughts on this. In spite of\n> > the long email which is perhaps a bit off-putting I don't think\n> > there's actually very much complicated stuff going on. Anyway, I'm\n> > keen to try and stop this discussion from getting marooned on the\n> > backlog for too long if we can!\n>\n> Sorry sorry for the late reply. We also left quite some discussion\n> pending on the previous version, let's try to catch up here.\n>\n> >\n> > Thanks again\n> > David\n> >\n> > On Mon, 7 Sep 2020 at 17:44, David Plowman\n> > wrote:\n> > >\n> > > Hi everyone\n> > >\n> > > I wanted to return to the question of digital zoom. Rather than simply\n> > > writing a long email I've included an implementation that shows where\n> > > I am. It's changed in the light of exchanges with Jacopo and others,\n> > > and I would say it's \"for discussion\" at this point. Principally:\n\nAll patches miss your Signed-off-by, but reading this, it might be\nvery well intentional.\n\nThanks\n j","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D2564C3B5B\n\tfor ;\n\tMon, 21 Sep 2020 13:04:54 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5CB3E62FD3;\n\tMon, 21 Sep 2020 15:04:54 +0200 (CEST)","from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net\n\t[217.70.183.198])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4A72D62B90\n\tfor ;\n\tMon, 21 Sep 2020 15:04:53 +0200 (CEST)","from uno.localdomain (93-34-118-233.ip49.fastwebnet.it\n\t[93.34.118.233]) (Authenticated sender: jacopo@jmondi.org)\n\tby relay6-d.mail.gandi.net (Postfix) with ESMTPSA id D119FC000D;\n\tMon, 21 Sep 2020 13:04:52 +0000 (UTC)"],"X-Originating-IP":"93.34.118.233","Date":"Mon, 21 Sep 2020 15:08:45 +0200","From":"Jacopo Mondi ","To":"David Plowman ","Message-ID":"<20200921130845.navin5h6othpwtkz@uno.localdomain>","References":"<20200907164450.13082-1-david.plowman@raspberrypi.com>\n\t\n\t<20200921104138.2bcsw6mbhhgr2bvq@uno.localdomain>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20200921104138.2bcsw6mbhhgr2bvq@uno.localdomain>","Subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":12628,"web_url":"https://patchwork.libcamera.org/comment/12628/","msgid":"","date":"2020-09-21T13:50:32","subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi Jacopo\n\nThanks for the comments!\n\nOn Mon, 21 Sep 2020 at 11:37, Jacopo Mondi wrote:\n>\n> Hi David,\n>\n> On Mon, Sep 21, 2020 at 07:24:45AM +0100, David Plowman wrote:\n> > Hi again everyone\n> >\n> > I was wondering if anyone else had any thoughts on this. In spite of\n> > the long email which is perhaps a bit off-putting I don't think\n> > there's actually very much complicated stuff going on. Anyway, I'm\n> > keen to try and stop this discussion from getting marooned on the\n> > backlog for too long if we can!\n>\n> Sorry sorry for the late reply. We also left quite some discussion\n> pending on the previous version, let's try to catch up here.\n>\n> >\n> > Thanks again\n> > David\n> >\n> > On Mon, 7 Sep 2020 at 17:44, David Plowman\n> > wrote:\n> > >\n> > > Hi everyone\n> > >\n> > > I wanted to return to the question of digital zoom. Rather than simply\n> > > writing a long email I've included an implementation that shows where\n> > > I am. It's changed in the light of exchanges with Jacopo and others,\n> > > and I would say it's \"for discussion\" at this point. Principally:\n> > >\n> > > 1. I'd like users to have access to CameraSensorInfo::outputSize, and\n> > > I'm using Camera::properties() to get it. Unfortunately it changes\n> > > with every camera mode, so will we have to do this differently?\n> > >\n>\n> I'll reply to this in 1/4. I think it's not optimal, but we should not\n> block this feature any longer. Although, I'm afraid that application\n> might be forced to change the way they retrieve the SensorCrop (from\n> Camera to CameraConfiguration). How fast do yuo expect this will be\n> adopted? Do you plan to build anything on top ?\n\nAs you know, we're building libcamera versions of our existing\nraspicam apps, and digital zoom is a feature those applications have.\nBut beyond that, I don't see it spreading very far just yet. In the\nmeantime, anyway, I'm not keen to make those new apps public while\nthey're still missing some key features, though I'd be happy to share\nthem privately if anyone from the libcamera team was curious.\n\nI was also wondering about adding digital zoom to qcam as a stream\noption at startup, otherwise it's a bit hard to try without our as-yet\nunavailable libcamera apps. Would that be helpful?\n\nThanks again for all the feedback. I'll get on and produce a proper\nset of patches in the next day or two (properly signed too!)\n\nBest regards\nDavid\n\n>\n> > > 2. The terminology is a bit more Android-like. Because, why not?\n> > >\n>\n> :)\n>\n> > > 3. I'm no longer enforcing any particular method of cropping. But to\n> > > make life easier for applications I've added some geometry helpers. To\n> > > understand the rationale behind these I've included some example code\n> > > further down that I think an application might use.\n>\n> Nice!\n>\n> > >\n> > > This work is currently made up of 4 commits:\n> > >\n> > > 1. The SensorOutputSize property.\n> > >\n> > > 2. The SensorCrop control.\n> > >\n> > > 3. The geometry helper functions.\n> > >\n> > > 4. Implementation in the Raspberry Pi pipeline handler.\n> > >\n>\n> Repluies in individual patches\n>\n> > > Example Application Code\n> > >\n> > > Assume we have:\n> > >\n> > > camera - pointer to the Camera object\n> > > streamCfg - reference to the StreamConfiguration of the output image\n> > > request - pointer to a Request object\n> > > zoomFactor - floating point zoom factor, >= 1\n> > > dx - amount in pixels to pan left (-ve) or right (+ve)\n> > > dy - amount in pixels to pan up (-ve) or down (+ve)\n> > >\n> > > You'll want to understand the helper methods:\n> > >\n> > > Size Size::aspectRatioDown(const Size &ar) const\n> > > Returns a letter/pillarboxed version of this size.\n> > >\n> > > Size operator/(const Size &size, float f)\n> > > Downscale a Size object by f.\n> > >\n> > > Rectangle Size::centre(const Rectangle ®ion, int offsetX = 0, int offsetY = 0) const\n> > > Returns a Rectangle of this size centred at the same point as region,\n> > > optionally with offsets.\n> > >\n> > > Rectangle Rectangle::clamp(const Rectangle &boundary) const\n> > > Translates the rectangle so that no part of it lies outside boundary.\n> > >\n> > > 1. Basic zooming into the centre\n> > >\n> > > Size area = camera->properties().get(properties::SensorOutputSize);\n> > > Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;\n> > > Rectangle crop = size.centre(Rectangle(area));\n> > > request->controls.set(controls::SensorCrop, crop);\n> > >\n> > > (I'm hoping this first one is easy enough to understand!)\n> > >\n> > > 2. Zooming anywhere in the sensor output (even into parts of the image that\n> > > were not intially visible)\n> > >\n> > > Size area = camera->properties().get(properties::SensorOutputSize);\n> > > Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;\n> > > Rectangle crop = size.centre(Rectangle(area), dx, dy).clamp(Rectangle(area));\n> > > request->controls.set(controls::SensorCrop, crop);\n> > >\n> > > (As before, but we have offsets dx and dy to enable panning, so we\n> > > must also clamp the result to be sure it's within the sensor area.)\n> > >\n> > > 3. Zooming as before, but not allowing you to pan into anything that\n> > > wasn't initially visible\n> > >\n> > > Size area = camera->properties().get(properties::SensorOutputSize);\n> > > Rectangle maxCrop = area.aspectRatioDown(streamCfg.size).centre(area);\n> > > Size size = maxCrop.size() / zoomFactor;\n> > > Rectangle crop = size.centre(maxCrop, dx, dy).clamp(maxCrop);\n> > > request->controls.set(controls::SensorCrop, crop);\n> > >\n> > > (The subtle difference here is that we clamp the result against maxCrop,\n> > > not the full sensor area.)\n>\n> Thanks! very informative!\n>\n> > >\n> > > 4. Android\n> > >\n> > > This time we're given the \"crop\" (a Rectangle) that Android has requested. I\n> > > think Android only requires us to sort out the aspect ratio.\n> > >\n> > > Size size = crop.size().aspectRatioDown(streamCfg.size);\n> > > request->controls.set(controls::SensorCrop, size.centre(crop));\n>\n> We'll here have to deal with the fact Android specifies its sensor\n> crop with the full size sensor pixel array as reference, not the actual\n> sensor output frame. This shall be handled in the HAL though.\n>\n> > >\n> > >\n> > > That's as far as I've got. Thoughts and comments welcome!\n> > >\n> > > Best regards\n> > > David\n> > >\n> > > David Plowman (4):\n> > > libcamera: Add SensorOutputSize property\n> > > libcamera: Add SensorCrop control\n> > > libcamera: Add geometry helper functions\n> > > libcamera: pipeline: raspberrypi: Implementation of digital zoom\n> > >\n> > > include/libcamera/geometry.h | 18 +++\n> > > include/libcamera/ipa/raspberrypi.h | 1 +\n> > > src/ipa/raspberrypi/raspberrypi.cpp | 7 +\n> > > src/libcamera/camera_sensor.cpp | 3 +\n> > > src/libcamera/control_ids.yaml | 9 ++\n> > > src/libcamera/geometry.cpp | 129 ++++++++++++++++++\n> > > .../pipeline/raspberrypi/raspberrypi.cpp | 41 ++++++\n> > > src/libcamera/property_ids.yaml | 6 +\n> > > 8 files changed, 214 insertions(+)\n> > >\n> > > --\n> > > 2.20.1\n> > >\n> > _______________________________________________\n> > libcamera-devel mailing list\n> > libcamera-devel@lists.libcamera.org\n> > https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A2570C3B5B\n\tfor ;\n\tMon, 21 Sep 2020 13:50:47 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3192762FD5;\n\tMon, 21 Sep 2020 15:50:47 +0200 (CEST)","from mail-oi1-x229.google.com (mail-oi1-x229.google.com\n\t[IPv6:2607:f8b0:4864:20::229])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9324662B90\n\tfor ;\n\tMon, 21 Sep 2020 15:50:45 +0200 (CEST)","by mail-oi1-x229.google.com with SMTP id u126so16974649oif.13\n\tfor ;\n\tMon, 21 Sep 2020 06:50:45 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"GCNtYTjn\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=Z2zSLiPlTdrdNWw9frHVDjndwh9JFl1XRgZOorlcKjY=;\n\tb=GCNtYTjnrv7e6sX++PruNk30i+0IbHe1itLQNbuimQeaF0YUg2BFlg5GzDpTSNRfQX\n\tw9ZvfpXFTccF8B8DrOpW4ZQW2YbosRryqBzlQQ1Woh/Ku/evJfVF1tOn4IJ5uAyNG9Gq\n\tbw8jNqW95viF3N2getAN5R/k1vRA3rOHJSbO6gJpiNkEG1fvtg9z2eFPKqLIYnjCGCCy\n\txKvu5J500ssp8KYzl+BI9EySaAGNtb7+0TCiQ3BW+CA/6IaRApTDNi44K3Uelg2qRPJT\n\twrm8WTJReeQxxBtj6v/MCLP2NBN6Jp9h/o6wLuNLzjRA48tXxJNJjl2G7TefcqV47mOi\n\tYGDA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=Z2zSLiPlTdrdNWw9frHVDjndwh9JFl1XRgZOorlcKjY=;\n\tb=ZSv7b4NFBCgKjd7Q9nH/izuS8qmUwgYOLZcAoMhYhNxCQrddSvVrMJZBSEvvhsf8K6\n\tXUw5pqcN24C1TZ00zxpFDUyYH0H5Ayu+ugfSVYim9JbOBtfSmOb6Ydc+H/Zpi7vAf72L\n\trX7Z2iPh/+PNHa3Pk9OXVuniRnJ2xxz97XiKJok57FETEZxqETqhQ0+EypQxWA2OjEUg\n\tWu17cFgx8rF6Yb2dkeowUa6eE/lVjpiUPJpyYPCzKpSVt8flsMlP2HATxcr0GugGkyEB\n\tDUh3I/zXSmB4y6mQUgiGUQKugN7ODRoYFCu08pSmEsIys9jC+8JauySe471BkidUNfhQ\n\tG8PQ==","X-Gm-Message-State":"AOAM531eLgJeew8UV7aPjOR9BfRD1JHFj9IoMmNaVHVJkxq1QfeJg1T2\n\t6v4+hNtLVc/ocSxcqeSj476LHta0Wkofv1hor8JpPVlu1AI=","X-Google-Smtp-Source":"ABdhPJwupVRAge7TDypNBj9lZfqSSCIuEYlz8uUzn9ge2Xda4AFJ/XFK9LgzT466Mnrtewt2LZPnzV+F4z0v+3qHa4Y=","X-Received":"by 2002:aca:5546:: with SMTP id\n\tj67mr18450377oib.107.1600696244159; \n\tMon, 21 Sep 2020 06:50:44 -0700 (PDT)","MIME-Version":"1.0","References":"<20200907164450.13082-1-david.plowman@raspberrypi.com>\n\t\n\t<20200921104138.2bcsw6mbhhgr2bvq@uno.localdomain>","In-Reply-To":"<20200921104138.2bcsw6mbhhgr2bvq@uno.localdomain>","From":"David Plowman ","Date":"Mon, 21 Sep 2020 14:50:32 +0100","Message-ID":"","To":"Jacopo Mondi ","Subject":"Re: [libcamera-devel] [RFC PATCH 0/4] Digital zoom","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]