From patchwork Mon Sep 7 16:44:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 9511 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 09341BDB1C for ; Mon, 7 Sep 2020 16:44:58 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 827D662BDF; Mon, 7 Sep 2020 18:44:57 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="YJq/WXqh"; dkim-atps=neutral Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 52D6260372 for ; Mon, 7 Sep 2020 18:44:56 +0200 (CEST) Received: by mail-wm1-x32b.google.com with SMTP id y15so2139173wmi.0 for ; Mon, 07 Sep 2020 09:44:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Yj6SmQyP5xG17/iMFzbXgzO8Vd4w1SmfSV3LfK1CnTc=; b=YJq/WXqhClk52m1QnoBB9qKHaswIIC5rTYdBKgz1m7cLNM4omeZPgU4gHix0SGMYOF hw6Gq34qrqGfKq8ATMyS70Iedr2yYOZ9T7vwPCwMvfP3VI3GTqKWT2wvQNwTHJVKTTHl 7h//f7pEIVWnfbnPxeDMN+mPK8EmmgxyFMKjUrpg7hLybylyjkgUz/SkitFJNZGbdgLd oVhv1l7IJ7QO90LIvuxDOpb5Y5tk9CIvHzuywxlMhbL5WYId+pqK7cGdX4aPIVSAWbKl uuggjfya33MfCgHUzumJiw18X/F+SYGpf9zEsQE63vqFV651hpYsNM/P8/zht6r4YmX/ vWHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Yj6SmQyP5xG17/iMFzbXgzO8Vd4w1SmfSV3LfK1CnTc=; b=DvyDWGXy+YZMw5I7Z+VGfNVaXxSbdGDvVdhq9RPqbRcROI3REu0FyeOAyGjj6Urqmc yGb9R6MfUMqjO+ZNjbs8Lg2u/MSG90OsGxFne/JIfLXnW9kDlbhElPIp/O232bEv03Lo EJcdQ10h/GRUAtHZPoEtKmvxeqEaGASzgrgJ0dl7LqLzVk+wSuT3T6FXqr1hZwKvxsmJ CY6OUEu+Uhbz30MLuNJuqpW2n5nL10gK80BckgX6oYfMUCWa9vQ+so7jlRCw4VuE72aa 1eq6xd2mCVTyW+1kmlZ8dy8Qr0D2mW/nZfHlZGIix92/CJ7e16q5vlKXSpDbahBVDVbH tiRw== X-Gm-Message-State: AOAM53246kZ7VYaiW2a2lUW36p/FWjv3C5IixgzQuQsiXvpsruU+I38J 4kSGbY7EzFHMdZZ2wibL1XQDa88cancRqw== X-Google-Smtp-Source: ABdhPJzCZHgVfxvUgY3MGEGKHgrQs58rcmjJ0Li3qUlyy6BqO/ybD3taSxP46l0gsWf3jVWsPI/+uw== X-Received: by 2002:a1c:32c3:: with SMTP id y186mr185766wmy.15.1599497095488; Mon, 07 Sep 2020 09:44:55 -0700 (PDT) Received: from pi4-davidp.lan (plowpeople3.plus.com. [80.229.223.72]) by smtp.gmail.com with ESMTPSA id v3sm27707033wmh.6.2020.09.07.09.44.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2020 09:44:55 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Mon, 7 Sep 2020 17:44:46 +0100 Message-Id: <20200907164450.13082-1-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 0/4] Digital zoom X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi everyone I wanted to return to the question of digital zoom. Rather than simply writing a long email I've included an implementation that shows where I am. It's changed in the light of exchanges with Jacopo and others, and I would say it's "for discussion" at this point. Principally: 1. I'd like users to have access to CameraSensorInfo::outputSize, and I'm using Camera::properties() to get it. Unfortunately it changes with every camera mode, so will we have to do this differently? 2. The terminology is a bit more Android-like. Because, why not? 3. I'm no longer enforcing any particular method of cropping. But to make life easier for applications I've added some geometry helpers. To understand the rationale behind these I've included some example code further down that I think an application might use. This work is currently made up of 4 commits: 1. The SensorOutputSize property. 2. The SensorCrop control. 3. The geometry helper functions. 4. Implementation in the Raspberry Pi pipeline handler. Example Application Code Assume we have: camera - pointer to the Camera object streamCfg - reference to the StreamConfiguration of the output image request - pointer to a Request object zoomFactor - floating point zoom factor, >= 1 dx - amount in pixels to pan left (-ve) or right (+ve) dy - amount in pixels to pan up (-ve) or down (+ve) You'll want to understand the helper methods: Size Size::aspectRatioDown(const Size &ar) const Returns a letter/pillarboxed version of this size. Size operator/(const Size &size, float f) Downscale a Size object by f. Rectangle Size::centre(const Rectangle ®ion, int offsetX = 0, int offsetY = 0) const Returns a Rectangle of this size centred at the same point as region, optionally with offsets. Rectangle Rectangle::clamp(const Rectangle &boundary) const Translates the rectangle so that no part of it lies outside boundary. 1. Basic zooming into the centre Size area = camera->properties().get(properties::SensorOutputSize); Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor; Rectangle crop = size.centre(Rectangle(area)); request->controls.set(controls::SensorCrop, crop); (I'm hoping this first one is easy enough to understand!) 2. Zooming anywhere in the sensor output (even into parts of the image that were not intially visible) Size area = camera->properties().get(properties::SensorOutputSize); Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor; Rectangle crop = size.centre(Rectangle(area), dx, dy).clamp(Rectangle(area)); request->controls.set(controls::SensorCrop, crop); (As before, but we have offsets dx and dy to enable panning, so we must also clamp the result to be sure it's within the sensor area.) 3. Zooming as before, but not allowing you to pan into anything that wasn't initially visible Size area = camera->properties().get(properties::SensorOutputSize); Rectangle maxCrop = area.aspectRatioDown(streamCfg.size).centre(area); Size size = maxCrop.size() / zoomFactor; Rectangle crop = size.centre(maxCrop, dx, dy).clamp(maxCrop); request->controls.set(controls::SensorCrop, crop); (The subtle difference here is that we clamp the result against maxCrop, not the full sensor area.) 4. Android This time we're given the "crop" (a Rectangle) that Android has requested. I think Android only requires us to sort out the aspect ratio. Size size = crop.size().aspectRatioDown(streamCfg.size); request->controls.set(controls::SensorCrop, size.centre(crop)); That's as far as I've got. Thoughts and comments welcome! Best regards David David Plowman (4): libcamera: Add SensorOutputSize property libcamera: Add SensorCrop control libcamera: Add geometry helper functions libcamera: pipeline: raspberrypi: Implementation of digital zoom include/libcamera/geometry.h | 18 +++ include/libcamera/ipa/raspberrypi.h | 1 + src/ipa/raspberrypi/raspberrypi.cpp | 7 + src/libcamera/camera_sensor.cpp | 3 + src/libcamera/control_ids.yaml | 9 ++ src/libcamera/geometry.cpp | 129 ++++++++++++++++++ .../pipeline/raspberrypi/raspberrypi.cpp | 41 ++++++ src/libcamera/property_ids.yaml | 6 + 8 files changed, 214 insertions(+)