From patchwork Fri Aug 12 09:08:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17086 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 70140C3272 for ; Fri, 12 Aug 2022 09:08:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E146763327; Fri, 12 Aug 2022 11:08:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295326; bh=29yAFgEUXgO/HQGbpev1YLTzXJFLwiDR8/fWXyc8Vwo=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=1ZvKG30TjpaWscM5nwLWznFBMM00uVJVWC+dWeNQlr2xYrdaQH4P/OsjWpJeqisTE snovQkatw6TACOb5eZcmPK/+4kfjAwaMUOPirZkMQTijsnDbGA2SAfb6ls34+s5X1i qaCzop0SraT3JNATgQwCO/u6D1Dlr7lskbquswH+fkitYpwcbf+NJbJ1Qj7//RfwRc iADYi4uqjgwi4+PDJm+0nZAySSaRq//dqIiTJVbBIEdmZVlI/x6u5RsHEkfM8UAN+T mdvt4Wy0uq0J2AHdxIHO3wXxTDjOjyCdeRIBhKrWTInTS8rlPU922lTqVwgKbxlwEQ NousRt0CpLgjw== Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5F48663326 for ; Fri, 12 Aug 2022 11:08:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="biwKdXhb"; dkim-atps=neutral Received: by mail-pg1-x536.google.com with SMTP id q16so323653pgq.6 for ; Fri, 12 Aug 2022 02:08:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc; bh=bNaNPCkZcVjd2jXt2ZOgjr0tUGshs+CpLx3V6pvqums=; b=biwKdXhbn9YI8quZH8mIhjQRsobOnzSVcx65n6S5Vvmzq9R6IpMxJ598EDODy9HmEb 0rbh62E6yL7hIydFnwwLBReCq3KU47R4oMY5czyZgwj+SJbVIYRmnCWwUPtsan5tmsmi zydbUvSMOhPh3s+4DK4wKeV/u0Vti3t1pySdY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc; bh=bNaNPCkZcVjd2jXt2ZOgjr0tUGshs+CpLx3V6pvqums=; b=3QYD4dHKuzyDaWT+r0bWvAhvZL3S7kdFzU1OnGX9PeV/sTiwcEkSwa6U4wGWjfdplh dJ4RzCqp7p97AlElpVNtY3GZ8XLHkcH9D1DNwOEmr8XqEQLpEy/CGxnWUeUPpJLfe1Wh XuDRMJk6jNuYzKU9ufFOFZbdriZZkppmBR7ptobZ2dSqqTVPU8jpOn3gTZypAetztUR1 ZuYEwgmzZbY+c67jeP76ufHnkfiTVUSJMDXIo7ELT4Dji9CaK/19xEDjF0RODiUbZAnP BjCNjUkiNfNdHcXt6VVZr520yHwEFyLDMcbczt+io5+6daCKv2BQGQ8Ybu/vFgrp39z3 +xZg== X-Gm-Message-State: ACgBeo0OV6WK7pDcDCgIRj/1A2C4hUCRJQ8D4GbubYCTXIU8l7BWIXz7 uS0YNlFJ3JOuoR44UvTDzGJ5aZ2M9KgVKg== X-Google-Smtp-Source: AA6agR7iqi8+HPn8kW3V5iRBBz7BJnoPzV1Xu3iRNnNFiSjDMQgjnZsHDT9b1BS5+tv/BM3RQ1UbeA== X-Received: by 2002:a63:318a:0:b0:41c:b82a:dc6d with SMTP id x132-20020a63318a000000b0041cb82adc6dmr2417373pgx.319.1660295323465; Fri, 12 Aug 2022 02:08:43 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id kb7-20020a17090ae7c700b001f56a5e5d2fsm15839434pjb.2.2022.08.12.02.08.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:43 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:31 +0800 Message-Id: <20220812090838.1784703-1-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 0/7] Implement Android Partial Result Featrue X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi, The series is to implement Android partial result feature, which allows pipeline handler returns early buffers and metadata back to application before request completed. The feature is as a preparation of implementing the following description of Android camera3 spec: * The order of returning metadata and buffers for a single result does not * matter, but buffers for a given stream must be returned in FIFO order. So * the buffer for request 5 for stream A must always be returned before the * buffer for request 6 for stream A. This also applies to the result * metadata; the metadata for request 5 must be returned before the metadata * for request 6. * * However, different streams are independent of each other, so it is * acceptable and expected that the buffer for request 5 for stream A may be * returned after the buffer for request 6 for stream B is. And it is * acceptable that the result metadata for request 6 for stream B is * returned before the buffer for request 5 for stream A is. The spec is critical for ZSL implementation, because the still capture can be return in an order independent to pending preview requests. This will enhance still capture performance by: 1. Use cached raw frame for still capture, skip pending sensor exposure. 2. Skip pending preview requests and return still capture concurently. 2. HAL can pick 3A stable raw frame, so the application doesn't need to wait for the process of stablizing 3A (by AEPrecapture and AFTrigger). Because Android relies on metadata to process buffer properly, bufferCompleted signal is not sufficient to report partial result to framework. The patch series allows returning metadata earlier for pipeline handler and refactor Android adaptor to return partial result as soon as possible. V2: 1. Add patch to for flush() waiting for all requests done before return. 2. Change the final result returns in its o submission order. 3. Reorganize the reuturning order of the results to meet Android's requirement that the buffers of the same stream returns in FIFO. Han-Lin Chen (7): libcamera: Camera: Add RequestCompletionMode to configure the completion order libcamera: Camera: Add signals for completion of metadata and partial result libcamera: ipu3: returning partial results android: Move StreamBuffer out of Camera3RequestDescriptor android: Add JpegExifMetadata to store tags setting into Exif android: Implement partial result feature based on partialResultCompleted signal android: function flush() waits for all requests completed before return include/libcamera/camera.h | 10 + include/libcamera/internal/camera.h | 4 + include/libcamera/internal/pipeline_handler.h | 4 + include/libcamera/request.h | 40 + src/android/camera_capabilities.cpp | 2 +- src/android/camera_capabilities.h | 2 + src/android/camera_device.cpp | 884 ++++++++++++------ src/android/camera_device.h | 45 +- src/android/camera_request.cpp | 90 +- src/android/camera_request.h | 104 ++- src/android/camera_stream.cpp | 18 +- src/android/camera_stream.h | 10 +- src/android/jpeg/post_processor_jpeg.cpp | 17 +- src/android/jpeg/post_processor_jpeg.h | 2 +- src/android/post_processor.h | 4 +- src/android/yuv/post_processor_yuv.cpp | 2 +- src/android/yuv/post_processor_yuv.h | 2 +- src/ipa/ipu3/ipu3.cpp | 2 - src/libcamera/camera.cpp | 78 +- src/libcamera/pipeline/ipu3/ipu3.cpp | 65 +- src/libcamera/pipeline_handler.cpp | 141 ++- src/libcamera/request.cpp | 170 +++- 22 files changed, 1278 insertions(+), 418 deletions(-)