From patchwork Thu Aug 6 06:17:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hirokazu Honda X-Patchwork-Id: 9244 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5363ABD86F for ; Thu, 6 Aug 2020 06:17:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C07E260730; Thu, 6 Aug 2020 08:17:15 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="bZ7JlOO6"; dkim-atps=neutral Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 09AE16038D for ; Thu, 6 Aug 2020 08:17:15 +0200 (CEST) Received: by mail-pf1-x435.google.com with SMTP id y206so14064682pfb.10 for ; Wed, 05 Aug 2020 23:17:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=5/xjfMfQqiXvIMqi/+QRN0LjQ4kbejnYRIWicBVDUbs=; b=bZ7JlOO6pXZ6KcjYzQTUGA/C52zxj2y9gKxnNztEGXs5jAvh6UAjJ/3/M166fX9OIV h5kidNM4M/aFlwI6OxRmxiMSnYJCnPI/woSr6oFK4WRlFFuA4z7YqtAc4V5hSe1MqAr0 iY/5BDog/JOi7oA6WAehLdaviYX/YGGXyLcAA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=5/xjfMfQqiXvIMqi/+QRN0LjQ4kbejnYRIWicBVDUbs=; b=JTTTon1S7jXPKSDhf1gSRUYTJXAFck2UWUAQYb2yFMVMKtkuynKHaykGxc7wj0kZrU ojZMnxgnmFgZCoALIm0oEBu8HdiTUjIUhYqX5Z6fqsxexEWpkBTArTnTj9KUgp+HyLto IMgza1XueDJTgcsPsIYVYh29iGdHFAStJNxeLZSXSrcY2HGM+CDJbNUBjhU8Li2yaWFD S154j1z0DhZAIvbI61fpgdqSA45ytCVgZCasRaWGn/HejOLmcy4nJpVa+I/EZv/mzfI6 Xsidxcqf/mK6ktgr8Fgyr52yT79t+SmyPf2V+cRgFHQE+bJWKL0xXcBYxfrE8H6AJEkH S9Hg== X-Gm-Message-State: AOAM532lvQ8ShQ9uvQpFE39SKcdNu5DiPuqsUXY/yAB6vStno0SBe+Wc +/Xoh8KTvSpfc3ofMkvkERx7halLBVM= X-Google-Smtp-Source: ABdhPJyBlbovMgxIBFEZhdYWytvTxNOgTOSr9BcEharHP54tg0pTJxWVW4qo5lVYOf2O2XjHKuX4ew== X-Received: by 2002:a62:1c8b:: with SMTP id c133mr6969621pfc.134.1596694633088; Wed, 05 Aug 2020 23:17:13 -0700 (PDT) Received: from hiroh.tok.corp.google.com ([2401:fa00:8f:2:de4a:3eff:fe7d:f78f]) by smtp.gmail.com with ESMTPSA id 21sm6388267pfa.4.2020.08.05.23.17.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Aug 2020 23:17:12 -0700 (PDT) From: Hirokazu Honda To: libcamera-devel@lists.libcamera.org Date: Thu, 6 Aug 2020 15:17:05 +0900 Message-Id: <20200806061706.1025788-1-hiroh@chromium.org> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 0/1] Proposal of mapping between camera configurations and requested configurations X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: hanlinchen@chromium.org Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This is a proposal about how to map camera configurations and requested configurations in Android Camera HAL adaptation layer. Please also see the sample code in the following patch. # Software Stream Processing in libcamera _hiroh@chromium.org / Draft: 2020-08-06_ # Objective Perform frame processing in libcamera to achieve requested stream configurations that are not supported natively by the camera hardware, but required by the Android Camera HAL interface. # Background ### Libcamera In addition to its native API, libcamera[^1] provides a number of camera APIs, for example, V4L2 Webcam API and Android Camera HAL3. The platform specific implementations are wrapped in libcamera core and a caller of libcamera doesn’t have to take care the platform. ### Android Camera HAL Chrome OS camera stack uses Android Camera HAL[^2] interface. Libcamera provides Android Camera HAL with an adaptation layer[^3] between libcamera core part and Android HAL, which is called Android HAL adaptation layer in this document. To present a uniform set of capabilities to the API users, Android Camera HAL API[^4] allows caller to request stream configurations that are beyond the device capabilities. For example, while a camera device is able to produce a single stream, a HAL caller requests three possibly different resolution streams (PRIV, YUV, JPEG). However, libcamera core implementation produces camera-capable streams. Therefore, we have to create three streams from the single stream produced by libcamera. Requests beyond the device capability is supported only in Android HAL at this moment. I describe the design in this document that the stream processing is performed in Android HAL adaptation layer. # Overview ## Current implementation The requested stream configuration is given by _camera3_device_t->ops->configure_streams()_ in Android Camera HAL. This delegates CameraDevice::configureStreams()[^5] in libcamera. The current implementation attempts all the given configurations and succeeds if and only if the camera device can produces them without any adjustments. ### libcamera::CameraConfiguration It is CameraConfiguration[^6] that judges whether adjustments are required, or even requested configurations are infeasible. The procedure of configuration is that CameraDevice 1. Adds every configuration by CameraConfiguration::addConfiguration(). 2. Assorts the added configurations by CameraConfiguration::validate(). CameraConfiguration, especially for validate(), is implemented per pipeline. For instance, the CameraConfiguration implementation for IPU3 is IPU3CameraConfiguration[^7]. validate() returns one of the below, * Valid * A camera can produce streams with requested configurations. * Adjusted * A camera cannot produce streams with requested configurations as-is, but can produce streams with different pixel formats or resolutions. * Invalid * A camera cannot produce streams with either requested configurations or different pixel formats and resolutions. For instance, this is returned when the larger resolution is requested than the maximum supported one? What we need to resolve is, when Adjusted is returned, to map adjusted camera streams to requested camera streams and required processing. ## Stream processing The processing to be thought of are followings. * Down-scaling * We don’t perform up-scaling because it affects stream qualities * Down-scaling is allowed for the same ratio to avoid producing distorted frames. For instance, scaling from 1280x720 (16:9) to 480x360 (4:3) is not allowed. * Cropping * Cropping is executed only to change the frame ratio. Thus it must be done after down-scaling if required. For example, to convert 1280x720 to 480x360, first down-scale to 640x360 and then crop to 480x360. * Format conversion * Pixel format conversion * JPEG encoding # Proposal Basically we only need to consider a mapping algorithm after validate(). However, to obtain less processing and better stream qualities, we should reorder given configurations within validate(). I described how to map after validate() first, and next how to reorder within validate(). ## How to map after validate() For each requested stream, we try to find a best-fit camera stream as follows. 1. Filter out smaller resolutions than the requested one. 2. Prioritize smaller resolutions to reduce number of processed pixels 3. If there is same ratio and same format as requested ones, select it 4. Otherwise, select one of the same ratio’s ones. If there is no same ratio one, select one of the same format’s ones. 5. If there is neither same ratio’s nor format’s one, select any convertible stream. The required processings are * No-op [Same resolution and same format] * Scale [Same ratio and same format, but different resolution] * Pixel format conversion [Same resolution but different format] * Scale and Pixel format conversion [Same ratio but different format and resolution] * Scle and Crop [Same format, but different ratio] * Scale, Crop and Pixel format conversion [Different ratio and format] ## How to sort within validate() Since up-scaling is not allowed in the proposal mapping, it is important to configure larger resolutions. Otherwise, configureStream() fails. Besides, producing as many as different ratios would be better so that we don’t have to get rid of captured content by cropping. This can be done in Android HAL adaptation layer, however I would do this within validate() because the above sorting strategy is general and each configuration class should have more information to prioritize the requested configurations. [^1]: https://libcamera.org/index.html [^2]: https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h [^3]: https://git.linuxtv.org/libcamera.git/tree/src/android [^4]: https://developer.android.com/reference/android/hardware/camera2/CameraDevice#regular-capture [^5]: https://git.linuxtv.org/libcamera.git/tree/src/android/camera_device.cpp#n934 [^6]: https://git.linuxtv.org/libcamera.git/tree/include/libcamera/camera.h#n28 [^7]: https://git.linuxtv.org/libcamera.git/tree/src/libcamera/pipeline/ipu3/ipu3.cpp#n62 --- 2.28.0.236.gb10cc79966-goog