Cover Letter Detail
Show a cover letter.
GET /api/1.1/covers/9244/?format=api
{ "id": 9244, "url": "https://patchwork.libcamera.org/api/1.1/covers/9244/?format=api", "web_url": "https://patchwork.libcamera.org/cover/9244/", "project": { "id": 1, "url": "https://patchwork.libcamera.org/api/1.1/projects/1/?format=api", "name": "libcamera", "link_name": "libcamera", "list_id": "libcamera_core", "list_email": "libcamera-devel@lists.libcamera.org", "web_url": "", "scm_url": "", "webscm_url": "" }, "msgid": "<20200806061706.1025788-1-hiroh@chromium.org>", "date": "2020-08-06T06:17:05", "name": "[libcamera-devel,0/1] Proposal of mapping between camera configurations and requested configurations", "submitter": { "id": 63, "url": "https://patchwork.libcamera.org/api/1.1/people/63/?format=api", "name": "Hirokazu Honda", "email": "hiroh@chromium.org" }, "mbox": "https://patchwork.libcamera.org/cover/9244/mbox/", "series": [ { "id": 1202, "url": "https://patchwork.libcamera.org/api/1.1/series/1202/?format=api", "web_url": "https://patchwork.libcamera.org/project/libcamera/list/?series=1202", "date": "2020-08-06T06:17:05", "name": "Proposal of mapping between camera configurations and requested configurations", "version": 1, "mbox": "https://patchwork.libcamera.org/series/1202/mbox/" } ], "comments": "https://patchwork.libcamera.org/api/covers/9244/comments/", "headers": { "Return-Path": "<libcamera-devel-bounces@lists.libcamera.org>", "X-Original-To": "parsemail@patchwork.libcamera.org", "Delivered-To": "parsemail@patchwork.libcamera.org", "Received": [ "from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5363ABD86F\n\tfor <parsemail@patchwork.libcamera.org>;\n\tThu, 6 Aug 2020 06:17:16 +0000 (UTC)", "from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C07E260730;\n\tThu, 6 Aug 2020 08:17:15 +0200 (CEST)", "from mail-pf1-x435.google.com (mail-pf1-x435.google.com\n\t[IPv6:2607:f8b0:4864:20::435])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 09AE16038D\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tThu, 6 Aug 2020 08:17:15 +0200 (CEST)", "by mail-pf1-x435.google.com with SMTP id y206so14064682pfb.10\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed, 05 Aug 2020 23:17:14 -0700 (PDT)", "from hiroh.tok.corp.google.com\n\t([2401:fa00:8f:2:de4a:3eff:fe7d:f78f])\n\tby smtp.gmail.com with ESMTPSA id\n\t21sm6388267pfa.4.2020.08.05.23.17.11\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tWed, 05 Aug 2020 23:17:12 -0700 (PDT)" ], "Authentication-Results": "lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"bZ7JlOO6\"; dkim-atps=neutral", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; h=from:to:cc:subject:date:message-id:mime-version\n\t:content-transfer-encoding;\n\tbh=5/xjfMfQqiXvIMqi/+QRN0LjQ4kbejnYRIWicBVDUbs=;\n\tb=bZ7JlOO6pXZ6KcjYzQTUGA/C52zxj2y9gKxnNztEGXs5jAvh6UAjJ/3/M166fX9OIV\n\th5kidNM4M/aFlwI6OxRmxiMSnYJCnPI/woSr6oFK4WRlFFuA4z7YqtAc4V5hSe1MqAr0\n\tiY/5BDog/JOi7oA6WAehLdaviYX/YGGXyLcAA=", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version\n\t:content-transfer-encoding;\n\tbh=5/xjfMfQqiXvIMqi/+QRN0LjQ4kbejnYRIWicBVDUbs=;\n\tb=JTTTon1S7jXPKSDhf1gSRUYTJXAFck2UWUAQYb2yFMVMKtkuynKHaykGxc7wj0kZrU\n\tojZMnxgnmFgZCoALIm0oEBu8HdiTUjIUhYqX5Z6fqsxexEWpkBTArTnTj9KUgp+HyLto\n\tIMgza1XueDJTgcsPsIYVYh29iGdHFAStJNxeLZSXSrcY2HGM+CDJbNUBjhU8Li2yaWFD\n\tS154j1z0DhZAIvbI61fpgdqSA45ytCVgZCasRaWGn/HejOLmcy4nJpVa+I/EZv/mzfI6\n\tXsidxcqf/mK6ktgr8Fgyr52yT79t+SmyPf2V+cRgFHQE+bJWKL0xXcBYxfrE8H6AJEkH\n\tS9Hg==", "X-Gm-Message-State": "AOAM532lvQ8ShQ9uvQpFE39SKcdNu5DiPuqsUXY/yAB6vStno0SBe+Wc\n\t+/Xoh8KTvSpfc3ofMkvkERx7halLBVM=", "X-Google-Smtp-Source": "ABdhPJyBlbovMgxIBFEZhdYWytvTxNOgTOSr9BcEharHP54tg0pTJxWVW4qo5lVYOf2O2XjHKuX4ew==", "X-Received": "by 2002:a62:1c8b:: with SMTP id\n\tc133mr6969621pfc.134.1596694633088; \n\tWed, 05 Aug 2020 23:17:13 -0700 (PDT)", "From": "Hirokazu Honda <hiroh@chromium.org>", "To": "libcamera-devel@lists.libcamera.org", "Date": "Thu, 6 Aug 2020 15:17:05 +0900", "Message-Id": "<20200806061706.1025788-1-hiroh@chromium.org>", "X-Mailer": "git-send-email 2.28.0.236.gb10cc79966-goog", "MIME-Version": "1.0", "Subject": "[libcamera-devel] [PATCH 0/1] Proposal of mapping between camera\n\tconfigurations and requested configurations", "X-BeenThere": "libcamera-devel@lists.libcamera.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<libcamera-devel.lists.libcamera.org>", "List-Unsubscribe": "<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>", "List-Archive": "<https://lists.libcamera.org/pipermail/libcamera-devel/>", "List-Post": "<mailto:libcamera-devel@lists.libcamera.org>", "List-Help": "<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>", "List-Subscribe": "<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>", "Cc": "hanlinchen@chromium.org", "Content-Type": "text/plain; charset=\"utf-8\"", "Content-Transfer-Encoding": "base64", "Errors-To": "libcamera-devel-bounces@lists.libcamera.org", "Sender": "\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>" }, "content": "This is a proposal about how to map camera configurations and requested\nconfigurations in Android Camera HAL adaptation layer. Please also see the\nsample code in the following patch.\n\n# Software Stream Processing in libcamera\n\n_hiroh@chromium.org / Draft: 2020-08-06_\n\n\n# Objective\n\nPerform frame processing in libcamera to achieve requested stream configurations that are not supported natively by the camera hardware, but required by the Android Camera HAL interface.\n\n\n# Background\n\n\n### Libcamera\n\nIn addition to its native API, libcamera[^1] provides a number of camera APIs, for example, V4L2 Webcam API and Android Camera HAL3. The platform specific implementations are wrapped in libcamera core and a caller of libcamera doesn’t have to take care the platform.\n\n\n### Android Camera HAL\n\nChrome OS camera stack uses Android Camera HAL[^2] interface. Libcamera provides Android Camera HAL with an adaptation layer[^3] between libcamera core part and Android HAL, which is called Android HAL adaptation layer in this document.\n\nTo present a uniform set of capabilities to the API users, Android Camera HAL API[^4] allows caller to request stream configurations that are beyond the device capabilities. For example, while a camera device is able to produce a single stream, a HAL caller requests three possibly different resolution streams (PRIV, YUV, JPEG). However, libcamera core implementation produces camera-capable streams. Therefore, we have to create three streams from the single stream produced by libcamera.\n\nRequests beyond the device capability is supported only in Android HAL at this moment. I describe the design in this document that the stream processing is performed in Android HAL adaptation layer.\n\n\n# Overview\n\n\n## Current implementation\n\nThe requested stream configuration is given by _camera3_device_t->ops->configure_streams()_ in Android Camera HAL. This delegates CameraDevice::configureStreams()[^5] in libcamera. The current implementation attempts all the given configurations and succeeds if and only if the camera device can produces them without any adjustments.\n\n\n### libcamera::CameraConfiguration\n\n It is CameraConfiguration[^6] that judges whether adjustments are required, or even requested configurations are infeasible.\n\nThe procedure of configuration is that CameraDevice\n\n\n\n1. Adds every configuration by CameraConfiguration::addConfiguration().\n2. Assorts the added configurations by CameraConfiguration::validate().\n\nCameraConfiguration, especially for validate(), is implemented per pipeline. For instance, the CameraConfiguration implementation for IPU3 is IPU3CameraConfiguration[^7].\n\nvalidate() returns one of the below,\n\n\n\n* Valid\n * A camera can produce streams with requested configurations.\n* Adjusted\n * A camera cannot produce streams with requested configurations as-is, but can produce streams with different pixel formats or resolutions.\n* Invalid\n * A camera cannot produce streams with either requested configurations or different pixel formats and resolutions. For instance, this is returned when the larger resolution is requested than the maximum supported one?\n\nWhat we need to resolve is, when Adjusted is returned, to map adjusted camera streams to requested camera streams and required processing.\n\n\n## Stream processing\n\nThe processing to be thought of are followings.\n\n\n\n* Down-scaling\n * We don’t perform up-scaling because it affects stream qualities\n * Down-scaling is allowed for the same ratio to avoid producing distorted frames. For instance, scaling from 1280x720 (16:9) to 480x360 (4:3) is not allowed.\n* Cropping\n * Cropping is executed only to change the frame ratio. Thus it must be done after down-scaling if required. For example, to convert 1280x720 to 480x360, first down-scale to 640x360 and then crop to 480x360.\n* Format conversion\n * Pixel format conversion\n * JPEG encoding\n\n\n# Proposal\n\nBasically we only need to consider a mapping algorithm after validate(). However, to obtain less processing and better stream qualities, we should reorder given configurations within validate().\n\nI described how to map after validate() first, and next how to reorder within validate().\n\n\n## How to map after validate()\n\nFor each requested stream, we try to find a best-fit camera stream as follows.\n\n\n\n1. Filter out smaller resolutions than the requested one.\n2. Prioritize smaller resolutions to reduce number of processed pixels\n3. If there is same ratio and same format as requested ones, select it\n4. Otherwise, select one of the same ratio’s ones. If there is no same ratio one, select one of the same format’s ones.\n5. If there is neither same ratio’s nor format’s one, select any convertible stream.\n\nThe required processings are\n\n\n* No-op [Same resolution and same format]\n* Scale [Same ratio and same format, but different resolution]\n* Pixel format conversion [Same resolution but different format]\n* Scale and Pixel format conversion [Same ratio but different format and resolution]\n* Scle and Crop [Same format, but different ratio]\n* Scale, Crop and Pixel format conversion [Different ratio and format]\n\n\n## How to sort within validate()\n\nSince up-scaling is not allowed in the proposal mapping, it is important to configure larger resolutions. Otherwise, configureStream() fails. Besides, producing as many as different ratios would be better so that we don’t have to get rid of captured content by cropping.\n\nThis can be done in Android HAL adaptation layer, however I would do this within validate() because the above sorting strategy is general and each configuration class should have more information to prioritize the requested configurations.\n\n[^1]:\n https://libcamera.org/index.html\n\n[^2]:\n https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h\n\n[^3]:\n https://git.linuxtv.org/libcamera.git/tree/src/android\n\n[^4]:\n https://developer.android.com/reference/android/hardware/camera2/CameraDevice#regular-capture\n\n[^5]:\n https://git.linuxtv.org/libcamera.git/tree/src/android/camera_device.cpp#n934\n\n[^6]:\n https://git.linuxtv.org/libcamera.git/tree/include/libcamera/camera.h#n28\n\n[^7]:\n https://git.linuxtv.org/libcamera.git/tree/src/libcamera/pipeline/ipu3/ipu3.cpp#n62\n\n--\n2.28.0.236.gb10cc79966-goog" }