From patchwork Wed Apr 7 16:06:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 11861 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 85F27BD1F6 for ; Wed, 7 Apr 2021 16:06:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 39389687D3; Wed, 7 Apr 2021 18:06:16 +0200 (CEST) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 194D1687A1 for ; Wed, 7 Apr 2021 18:06:14 +0200 (CEST) X-Originating-IP: 93.34.118.233 Received: from uno.lan (93-34-118-233.ip49.fastwebnet.it [93.34.118.233]) (Authenticated sender: jacopo@jmondi.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 825D6E0004; Wed, 7 Apr 2021 16:06:13 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Wed, 7 Apr 2021 18:06:42 +0200 Message-Id: <20210407160644.58326-2-jacopo@jmondi.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210407160644.58326-1-jacopo@jmondi.org> References: <20210407160644.58326-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/3] libcamera: controls: Destage 'SensorTimestamp' X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Destage the 'SensorTimestamp' control, which is used by pipeline handlers to report the time when the first active line of the sensor's pixel array is exposed. Signed-off-by: Jacopo Mondi Reviewed-by: Hirokazu Honda --- src/libcamera/control_ids.yaml | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index b4771f9def89..f025819aedfd 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -360,6 +360,20 @@ controls: size: [2] + - SensorTimestamp: + type: int64_t + description: | + The time when the first row of the image sensor active array is exposed, + The timestamp represents a monotonically increasing counter since the + system boot time expressed in nanoseconds. + + Pipeline handlers set this control in a completed Request metadata to + allow application to compute the sensor's frame period duration by + comparing consecutive capture results. + + \todo Define how the sensor timestamp has to be used in the reprocessing + use case. + # ---------------------------------------------------------------------------- # Draft controls section @@ -547,13 +561,6 @@ controls: value: 3 description: The AWB algorithm is locked. - - SensorTimestamp: - type: int64_t - draft: true - description: | - Control to report the start of exposure of the first row of the captured - image. Currently identical to ANDROID_SENSOR_TIMESTAMP. - - SensorRollingShutterSkew: type: int64_t draft: true From patchwork Wed Apr 7 16:06:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 11862 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B1097BD1F6 for ; Wed, 7 Apr 2021 16:06:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 616D9687E3; Wed, 7 Apr 2021 18:06:17 +0200 (CEST) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EB7F2687D3 for ; Wed, 7 Apr 2021 18:06:14 +0200 (CEST) X-Originating-IP: 93.34.118.233 Received: from uno.lan (93-34-118-233.ip49.fastwebnet.it [93.34.118.233]) (Authenticated sender: jacopo@jmondi.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 40350E000E; Wed, 7 Apr 2021 16:06:14 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Wed, 7 Apr 2021 18:06:43 +0200 Message-Id: <20210407160644.58326-3-jacopo@jmondi.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210407160644.58326-1-jacopo@jmondi.org> References: <20210407160644.58326-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/3] libcamera: ipu3: Report sensor timestamp X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Report the sensor's timestamp in the Request metadata by using the CIO2 buffer timestamp as a good-enough approximation. The buffer's timestamp is recorded at DMA-transfer time, and it does not theoretically matches the 'start of exposure' definition, but when used to compare two consecutive frames it gives an acceptable estimation of the sensor frame period duration. Signed-off-by: Jacopo Mondi Reviewed-by: Hirokazu Honda Reviewed-by: Laurent Pinchart --- src/libcamera/pipeline/ipu3/ipu3.cpp | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 519cad4f8148..d52e24f847a2 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -1251,6 +1251,15 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer) Request *request = info->request; + /* + * Record the sensor's timestamp in the request metadata. + * + * \todo The sensor timestamp should be better estimated by connecting + * to the V4L2Device::frameStart signal. + */ + request->metadata().set(controls::SensorTimestamp, + buffer->metadata().timestamp); + /* If the buffer is cancelled force a complete of the whole request. */ if (buffer->metadata().status == FrameMetadata::FrameCancelled) { for (auto it : request->buffers()) From patchwork Wed Apr 7 16:06:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 11863 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 30C9DBD1F6 for ; Wed, 7 Apr 2021 16:06:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AC847687E7; Wed, 7 Apr 2021 18:06:17 +0200 (CEST) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 925FE687A1 for ; Wed, 7 Apr 2021 18:06:15 +0200 (CEST) X-Originating-IP: 93.34.118.233 Received: from uno.lan (93-34-118-233.ip49.fastwebnet.it [93.34.118.233]) (Authenticated sender: jacopo@jmondi.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 05B63E0008; Wed, 7 Apr 2021 16:06:14 +0000 (UTC) From: Jacopo Mondi To: libcamera-devel@lists.libcamera.org Date: Wed, 7 Apr 2021 18:06:44 +0200 Message-Id: <20210407160644.58326-4-jacopo@jmondi.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210407160644.58326-1-jacopo@jmondi.org> References: <20210407160644.58326-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/3] android: camera_device: Use controls::SensorTimestamp X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Use the controls::SensorTimestamp value to populate ANDROID_SENSOR_TIMESTAMP result metadata, if the Camera provides it. Use the same control to notify the shutter even to the camera framework otherwise fall-back to the timestamp of the first completed buffer if it is not available. Signed-off-by: Jacopo Mondi Reviewed-by: Hirokazu Honda --- src/android/camera_device.cpp | 33 ++++++++++++++++++++------------- src/android/camera_device.h | 3 +-- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 89044efa7ebe..749fe5c3dedc 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -2019,7 +2019,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques void CameraDevice::requestComplete(Request *request) { - const Request::BufferMap &buffers = request->buffers(); camera3_buffer_status status = CAMERA3_BUFFER_STATUS_OK; std::unique_ptr resultMetadata; Camera3RequestDescriptor *descriptor = @@ -2034,14 +2033,7 @@ void CameraDevice::requestComplete(Request *request) LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " << descriptor->buffers_.size() << " streams"; - /* - * \todo The timestamp used for the metadata is currently always taken - * from the first buffer (which may be the first stream) in the Request. - * It might be appropriate to return a 'correct' (as determined by - * pipeline handlers) timestamp in the Request itself. - */ - uint64_t timestamp = buffers.begin()->second->metadata().timestamp; - resultMetadata = getResultMetadata(*descriptor, timestamp); + resultMetadata = getResultMetadata(*descriptor); /* Handle any JPEG compression. */ for (camera3_stream_buffer_t &buffer : descriptor->buffers_) { @@ -2086,6 +2078,19 @@ void CameraDevice::requestComplete(Request *request) captureResult.output_buffers = descriptor->buffers_.data(); if (status == CAMERA3_BUFFER_STATUS_OK) { + int64_t timestamp; + + if (request->metadata().contains(controls::SensorTimestamp)) { + timestamp = request->metadata().get(controls::SensorTimestamp); + } else { + /* + * Use the timestamp from the first buffer (which may be + * the first stream) in the Request if the pipeline does + * not report the sensor timestamp. + */ + const Request::BufferMap &buffers = request->buffers(); + timestamp = buffers.begin()->second->metadata().timestamp; + } notifyShutter(descriptor->frameNumber_, timestamp); captureResult.partial_result = 1; @@ -2147,8 +2152,7 @@ void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream) * Produce a set of fixed result metadata. */ std::unique_ptr -CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor, - int64_t timestamp) const +CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) const { const ControlList &metadata = descriptor.request_->metadata(); const CameraMetadata &settings = descriptor.settings_; @@ -2274,8 +2278,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor, resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, &value32, 1); - resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, ×tamp, 1); - value = ANDROID_STATISTICS_FACE_DETECT_MODE_OFF; resultMetadata->addEntry(ANDROID_STATISTICS_FACE_DETECT_MODE, &value, 1); @@ -2314,6 +2316,11 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor, &exposure, 1); } + if (metadata.contains(controls::SensorTimestamp)) { + int64_t timestamp = metadata.get(controls::SensorTimestamp); + resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, ×tamp, 1); + } + if (metadata.contains(controls::ScalerCrop)) { Rectangle crop = metadata.get(controls::ScalerCrop); int32_t cropRect[] = { diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 11bdfec8d587..73e5009ac274 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -102,8 +102,7 @@ private: libcamera::PixelFormat toPixelFormat(int format) const; int processControls(Camera3RequestDescriptor *descriptor); std::unique_ptr getResultMetadata( - const Camera3RequestDescriptor &descriptor, - int64_t timestamp) const; + const Camera3RequestDescriptor &descriptor) const; unsigned int id_; camera3_device_t camera3Device_;