From patchwork Thu Aug 11 15:02:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17077 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CC501BE173 for ; Thu, 11 Aug 2022 15:02:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 765666332F; Thu, 11 Aug 2022 17:02:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660230156; bh=4PmH1XUl/7NgnSb0To7bqDlLGeP4zFBEEZ82sxD102o=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=giM2Wzk3LrpGI0EghUiFIINmK3tHyu+DocLKA0NsxpyYnh9sh6X0iI040hK6+il5B QpfKzkEl7FXHt/1AmEzHDU1g5abbOp+d+nuqzIyNmpMszP5VmQaV/d1EexLE3whWAX BRHF45ARAbMcT6jZw2lQdwLzqXdEkhS4eDjiQPwWU8ZNPxa0W8/PFuhqGF3RPCT6rR g66Vp1h5Rd3+Is0bc5AVL51eU1rvp9qTXeY75CGnxf8nPvsQxznVqUAbLRlVPtFAWx xzFCAFtlMqy3Ig374ugWZwyYvrmx0+K1x1UtDXEXeR2DPo5McJANUpWuLFvEfQbtHl StRHMa1MzhtsQ== Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [IPv6:2001:4b98:dc4:8::228]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B180563325 for ; Thu, 11 Aug 2022 17:02:34 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id 3E3D01BF20A; Thu, 11 Aug 2022 15:02:32 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Aug 2022 17:02:14 +0200 Message-Id: <20220811150219.62066-2-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220811150219.62066-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/6] controls: Introduce AEGC-related controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls to model the AEGC algorithm block. The three controls allow applications to select the exposure time and analogue gain computation calculation mode (auto vs manual) independently from one another, while the AeState control reports the global state for the AEGC algorithm. The new controls are meant to replace the existing AeEnable and AeLocked controls, which are momentarily kept not to break compilation of platforms making use of them. Bug: https://bugs.libcamera.org/show_bug.cgi?id=42 Bug: https://bugs.libcamera.org/show_bug.cgi?id=43 Bug: https://bugs.libcamera.org/show_bug.cgi?id=47 Signed-off-by: Paul Elder Signed-off-by: Jacopo Mondi --- src/libcamera/control_ids.yaml | 273 +++++++++++++++++++++++++++------ 1 file changed, 223 insertions(+), 50 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index c93f362463dd..694659af4b7d 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -25,6 +25,75 @@ controls: \sa AeEnable + - AeState: + type: int32_t + description: | + Control to report the AEGC algorithm state. + + The AEGC algorithm computes the exposure time and the analogue gain + values to be applied to the image sensor. + + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and + AnalogueGainMode controls, which allow applications to decide how + the exposure time and gain are computed, in Auto or Manual mode, + independently from one another. + + The AeState control reports the AEGC algorithm state through a single + value and describes it as a single computation block which computes + both the exposure time and the analogue gain values. + + When both the exposure time and analogue gain values are configured to + be in Manual mode, the AEGC algorithm is quiescent and does not actively + compute any value and the AeState control will report AeStateIdle. + + When at least the exposure time or analogue gain are configured to be + computed by the AEGC algorithm, the AeState control will report if the + algorithm has converged to stable values for all of the controls set + to be computed in Auto mode. + + \sa AnalogueGainMode + \sa ExposureTimeMode + + enum: + - name: AeStateIdle + value: 0 + description: | + The AEGC algorithm is inactive. + + This state is returned when both AnalogueGainMode and + ExposureTimeMode are set to Manual and the algorithm is not + actively computing any value. + + When ExposureTimeMode or AnalogueGainMode are set to Auto mode, the + AEGC algorithm might spontaneously initiate a new scan, in which + case the AeState control is moved to AeStateSearching. + - name: AeStateSearching + value: 1 + description: | + The AEGC algorithm is actively computing new values, for either the + exposure time or the analogue gain, but has not converged to a + stable result yet. + + This state is returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to auto and the algorithm hasn't + converged yet. + + The AEGC algorithm converges once stable values are computed for + any of the controls set to be computed in Auto mode. + + Once the algorithm converges the state is moved to AeStateConverged. + - name: AeStateConverged + value: 2 + description: | + The AEGC algorithm has converged. + + This state is returned if at least one of AnalogueGainMode + or ExposureTimeMode is set to Auto, and the AEGC algorithm has + converged to stable value. + + The AEGC algorithm might spontaneously re-initiate an AE scan, in + which case the state is moved to AeStateSearching. + # AeMeteringMode needs further attention: # - Auto-generate max enum value. # - Better handling of custom types. @@ -93,6 +162,13 @@ controls: how the desired total exposure is divided between the shutter time and the sensor's analogue gain. The exposure modes are platform specific, and not all exposure modes may be supported. + + When one of AnalogueGainMode or ExposureTimeMode is set to Manual, + the fixed values will override any choices made by AeExposureMode. + + \sa AnalogueGainMode + \sa ExposureTimeMode + enum: - name: ExposureNormal value: 0 @@ -111,13 +187,15 @@ controls: type: float description: | Specify an Exposure Value (EV) parameter. The EV parameter will only be - applied if the AE algorithm is currently enabled. + applied if the AE algorithm is currently enabled, that is, at least one + of AnalogueGainMode and ExposureTimeMode are in Auto mode. By convention EV adjusts the exposure as log2. For example EV = [-2, -1, 0.5, 0, 0.5, 1, 2] results in an exposure adjustment of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x]. - \sa AeEnable + \sa AnalogueGainMode + \sa ExposureTimeMode - ExposureTime: type: int32_t @@ -125,17 +203,95 @@ controls: Exposure time (shutter speed) for the frame applied in the sensor device. This value is specified in micro-seconds. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if ExposureTimeMode is Manual. If + this control is set when ExposureTimeMode is Auto, the value will be + ignored and will not be retained. + + When reported in metadata, this control indicates what exposure time + was used for the current request, regardless of ExposureTimeMode. + ExposureTimeMode will indicate the source of the exposure time value, + whether it came from the AE algorithm or not. + + \sa AnalogueGain + \sa ExposureTimeMode + + - ExposureTimeMode: + type: int32_t + description: | + Controls the source of the exposure time that is applied to the image + sensor. When set to Auto, the AE algorithm computes the exposure time + and configures the image sensor accordingly. When set to Manual, + the value of the ExposureTime control is used. + + When transitioning from Auto to Manual mode and no ExposureTime control + is provided by the application, the last value computed by the AE + algorithm when the mode was Auto will be used. If the ExposureTimeMode + was never set to Auto (either because the camera started in Manual mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + If ExposureTimeModeManual is supported, the ExposureTime control must + also be supported. + + The set of ExposureTimeMode modes that are supported by the camera must + have an intersection with the supported set of AnalogueGainMode modes. + + Flickerless exposure mode transitions + + Applications that transition from ExposureTimeModeAuto to the direct + control of the exposure time should aim to do so by selecting an + ExposureTime value as close as possible to the last value computed by + the auto exposure algorithm in order to avoid any visible flickering. + + To select the correct value to use as ExposureTime value, applications + should accommodate the natural delay in applying controls caused by the + capture pipeline frame depth. - \sa AnalogueGain AeEnable + When switching to manual exposure mode, applications should not + immediately specify an ExposureTime value in the same request where + ExposureTimeMode is set to Manual. They should instead wait for the + first Request where ExposureTimeMode is reported as + ExposureTimeModeManual in the Request metadata, and use the reported + ExposureTime to populate the control value in the next Request to be + queued to the Camera. - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + The implementation of the auto-exposure algorithm should equally try to + minimize flickering and when transitioning from manual exposure mode to + auto exposure use the last value provided by the application as starting + point. + + 1. Start with ExposureTimeMode set to Auto + + 2. Set ExposureTimeMode to Manual + + 3. Wait for the first completed request that has ExposureTimeMode + set to Manual + + 4. Copy the value reported in ExposureTime into a new request, and + submit it + + 5. Proceed to run manual exposure time + + \sa ExposureTime + enum: + - name: ExposureTimeModeAuto + value: 0 + description: | + The exposure time will be calculated automatically and set by the + AE algorithm. + + If ExposureTime is set while this mode is active, it will be + ignored, and it will also not be retained. + - name: ExposureTimeModeManual + value: 1 + description: | + The exposure time will not be updated by the AE algorithm. + + When transitioning from Auto to Manual mode, the last computed + exposure value is used until a new value is specified through the + ExposureTime control. If an ExposureTime value is specified in the + same request where the ExposureTimeMode is changed from Auto to + Manual, the provided ExposureTime is applied immediately. - AnalogueGain: type: float @@ -144,17 +300,64 @@ controls: The value of the control specifies the gain multiplier applied to all colour channels. This value cannot be lower than 1.0. - Setting this value means that it is now fixed and the AE algorithm may - not change it. Setting it back to zero returns it to the control of the - AE algorithm. + This control will only take effect if AnalogueGainMode is Manual. If + this control is set when AnalogueGainMode is Auto, the value will be + ignored and will not be retained. - \sa ExposureTime AeEnable + When reported in metadata, this control indicates what analogue gain + was used for the current request, regardless of AnalogueGainMode. + AnalogueGainMode will indicate the source of the analogue gain value, + whether it came from the AEGC algorithm or not. + + \sa ExposureTime + \sa AnalogueGainMode - \todo Document the interactions between AeEnable and setting a fixed - value for this control. Consider interactions with other AE features, - such as aperture and aperture/shutter priority mode, and decide if - control of which features should be automatically adjusted shouldn't - better be handled through a separate AE mode control. + - AnalogueGainMode: + type: int32_t + description: | + Controls the source of the analogue gain that is applied to the image + sensor. When set to Auto, the AEGC algorithm computes the analogue gain + and configures the image sensor accordingly. When set to Manual, + the value of the AnalogueGain control is used. + + When transitioning from Auto to Manual mode and no AnalogueGain control + is provided by the application, the last value computed by the AEGC + algorithm when the mode was Auto will be used. If the AnalogueGainMode + was never set to Auto (either because the camera started in Manual mode, + or Auto is not supported by the camera), the camera should use a + best-effort default value. + + If AnalogueGainModeManual is supported, the AnalogueGain control must + also be supported. + + The set of AnalogueGainMode modes that are supported by the camera must + have an intersection with the supported set of ExposureTimeMode modes. + + The same procedure described for performing flickerless transitions in + the ExposureTimeMode control documentation should be applied to + analogue gain. + + \sa ExposureTimeMode + \sa AnalogueGain + enum: + - name: AnalogueGainModeAuto + value: 0 + description: | + The analogue gain will be calculated automatically and set by the + AEGC algorithm. + + If AnalogueGain is set while this mode is active, it will be + ignored, and it will also not be retained. + - name: AnalogueGainModeManual + value: 1 + description: | + The analogue gain will not be updated by the AEGC algorithm. + + When transitioning from Auto to Manual mode, the last computed + gain value is used until a new value is specified through the + AnalogueGain control. If an AnalogueGain value is specified in the + same request where the AnalogueGainMode is changed from Auto to + Manual, the provided AnalogueGain is applied immediately. - Brightness: type: float @@ -767,36 +970,6 @@ controls: High quality aberration correction which might reduce the frame rate. - - AeState: - type: int32_t - draft: true - description: | - Control to report the current AE algorithm state. Currently identical to - ANDROID_CONTROL_AE_STATE. - - Current state of the AE algorithm. - enum: - - name: AeStateInactive - value: 0 - description: The AE algorithm is inactive. - - name: AeStateSearching - value: 1 - description: The AE algorithm has not converged yet. - - name: AeStateConverged - value: 2 - description: The AE algorithm has converged. - - name: AeStateLocked - value: 3 - description: The AE algorithm is locked. - - name: AeStateFlashRequired - value: 4 - description: The AE algorithm would need a flash for good results - - name: AeStatePrecapture - value: 5 - description: | - The AE algorithm has started a pre-capture metering session. - \sa AePrecaptureTrigger - - AwbState: type: int32_t draft: true From patchwork Thu Aug 11 15:02:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17078 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 26D7CBE173 for ; Thu, 11 Aug 2022 15:02:38 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D7B0263332; Thu, 11 Aug 2022 17:02:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660230157; bh=KzJ0/tq9QCpf6bmGfhk/V4Tbdp+Mebx49XIlXRkGLbQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=EyqiP4+9qy0+NRw6GQPMTsC/5VA3XFB9Qbz6ZTTvpki2/5w8YUVezd1Fzz59qvh8H pao5zfVDqLGBzkCADqL/BJ/+G+Gy+K5I74/G3ZngLLoje71TpcaepBfvif/f82CRAV IYPbyg2fAf2G2dhSPMQZE9bhDgotgzEZSYELKN7T7kAF1Jl9CjGcSx35BrNehLzcWn vQNyJ28IlDn2WKo8VuNUg9opGEXq1YUAx+rfeSZR51MJ22eWQF3ZbzLU/j3SzFmvBU 4VRozg2xqoUBQlum8tfJpnPGWzImfaNkWz+OnllH+XjTZwLmcqsnjvXMf8Ngu0nE5q ScOZT3ZinkgWA== Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [IPv6:2001:4b98:dc4:8::228]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4602C63327 for ; Thu, 11 Aug 2022 17:02:36 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id EE6151BF205; Thu, 11 Aug 2022 15:02:34 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Aug 2022 17:02:15 +0200 Message-Id: <20220811150219.62066-3-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220811150219.62066-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/6] libcamera: uvcvideo: Register ExposureTimeMode control X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Port the UVC pipeline handler to use the new ExposureTimeMode control when processing Camera controls in place of the AeEnable control. The V4L2_CID_EXPOSURE_AUTO control allows 4 possible values, which map to ExposureTimeModeAuto and ExposureTimeModeManual. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder --- src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 54 ++++++++++++++++++-- 1 file changed, 49 insertions(+), 5 deletions(-) diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp index fbe02cdcd520..08a37da0d90d 100644 --- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp +++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp @@ -266,7 +266,7 @@ int PipelineHandlerUVC::processControl(ControlList *controls, unsigned int id, cid = V4L2_CID_CONTRAST; else if (id == controls::Saturation) cid = V4L2_CID_SATURATION; - else if (id == controls::AeEnable) + else if (id == controls::ExposureTimeMode) cid = V4L2_CID_EXPOSURE_AUTO; else if (id == controls::ExposureTime) cid = V4L2_CID_EXPOSURE_ABSOLUTE; @@ -580,7 +580,7 @@ void UVCCameraData::addControl(uint32_t cid, const ControlInfo &v4l2Info, id = &controls::Saturation; break; case V4L2_CID_EXPOSURE_AUTO: - id = &controls::AeEnable; + id = &controls::ExposureTimeMode; break; case V4L2_CID_EXPOSURE_ABSOLUTE: id = &controls::ExposureTime; @@ -593,6 +593,7 @@ void UVCCameraData::addControl(uint32_t cid, const ControlInfo &v4l2Info, } /* Map the control info. */ + const std::vector &v4l2Values = v4l2Info.values(); int32_t min = v4l2Info.min().get(); int32_t max = v4l2Info.max().get(); int32_t def = v4l2Info.def().get(); @@ -630,10 +631,53 @@ void UVCCameraData::addControl(uint32_t cid, const ControlInfo &v4l2Info, }; break; - case V4L2_CID_EXPOSURE_AUTO: - info = ControlInfo{ false, true, true }; + case V4L2_CID_EXPOSURE_AUTO: { + /* + * From the V4L2_CID_EXPOSURE_AUTO documentation: + * + * ------------------------------------------------------------ + * V4L2_EXPOSURE_AUTO: + * Automatic exposure time, automatic iris aperture. + * + * V4L2_EXPOSURE_MANUAL: + * Manual exposure time, manual iris. + * + * V4L2_EXPOSURE_SHUTTER_PRIORITY: + * Manual exposure time, auto iris. + * + * V4L2_EXPOSURE_APERTURE_PRIORITY: + * Auto exposure time, manual iris. + *------------------------------------------------------------- + * + * ExposureTimeModeAuto = { V4L2_EXPOSURE_AUTO, + * V4L2_EXPOSURE_APERTURE_PRIORITY } + * + * + * ExposureTimeModeManual = { V4L2_EXPOSURE_MANUAL, + * V4L2_EXPOSURE_SHUTTER_PRIORITY } + */ + std::array values{}; + + auto it = std::find_if(v4l2Values.begin(), v4l2Values.end(), + [&](const ControlValue &val) { + return (val.get() == V4L2_EXPOSURE_APERTURE_PRIORITY || + val.get() == V4L2_EXPOSURE_AUTO) ? true : false; + }); + if (it != v4l2Values.end()) + values.back() = static_cast(controls::ExposureTimeModeAuto); + + it = std::find_if(v4l2Values.begin(), v4l2Values.end(), + [&](const ControlValue &val) { + return (val.get() == V4L2_EXPOSURE_SHUTTER_PRIORITY || + val.get() == V4L2_EXPOSURE_MANUAL) ? true : false; + }); + if (it != v4l2Values.end()) + values.back() = static_cast(controls::ExposureTimeModeManual); + + int32_t *data = values.data(); + info = ControlInfo{Span(data, 2), values[0]}; break; - + } case V4L2_CID_EXPOSURE_ABSOLUTE: /* * ExposureTime is in units of 1 µs, and UVC expects From patchwork Thu Aug 11 15:02:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17079 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 82493BE173 for ; Thu, 11 Aug 2022 15:02:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4337563335; Thu, 11 Aug 2022 17:02:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660230159; bh=YOf6JRjouAQ7XrFNBxTgBRYmqQiIF4DOQVqpRAdzCe4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=GD8EAaH3VQ6cnlcNiAwmBFI5lxnlze6TLnJUwhjupmwny3mq5XBnDAFUBrfwwXE+k A0/z97CyvmPX6g9VG08jBKqZtHxvxLGHfY75ifB3WY3mI1xLylJ1gT+wpX+RjRMJ+H CFUiRJyNuv9O1R7BC9eMVTNEyREkamNpehHyzrpJQncqmY/aalV10fzeZ+QinGQEtJ iqdPzYU/kF6tAxFtY7EWr+vIs8VPpO5pTWZdymkxG1JpQ0lWeCGH6AYYg3+0BDFh0y UC0xS2ig1EoXtAXwLTWeEB8mt2n2ZqSbyewoc0FgyjpEzGFxW9cZX7W/oDT7PiHltx iiqYOI7QMtvBg== Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [IPv6:2001:4b98:dc4:8::228]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 723566332D for ; Thu, 11 Aug 2022 17:02:37 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id 88AD41BF205; Thu, 11 Aug 2022 15:02:36 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Aug 2022 17:02:16 +0200 Message-Id: <20220811150219.62066-4-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220811150219.62066-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 3/6] test: ipa_data_serialization: Use ExposureTimeMode X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Replace the deprecated AeEnable control with ExposureTimeMode in ipa_data_serialization test. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder --- test/serialization/ipa_data_serializer_test.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/serialization/ipa_data_serializer_test.cpp b/test/serialization/ipa_data_serializer_test.cpp index d2050a868b38..f9b3e727127c 100644 --- a/test/serialization/ipa_data_serializer_test.cpp +++ b/test/serialization/ipa_data_serializer_test.cpp @@ -33,7 +33,7 @@ using namespace std; using namespace libcamera; static const ControlInfoMap Controls = ControlInfoMap({ - { &controls::AeEnable, ControlInfo(false, true) }, + { &controls::ExposureTimeMode, ControlInfo(false, true) }, { &controls::ExposureTime, ControlInfo(0, 999999) }, { &controls::AnalogueGain, ControlInfo(1.0f, 32.0f) }, { &controls::ColourGains, ControlInfo(0.0f, 32.0f) }, From patchwork Thu Aug 11 15:02:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17080 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D97C5BE173 for ; Thu, 11 Aug 2022 15:02:40 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9B3C363338; Thu, 11 Aug 2022 17:02:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660230160; bh=YBSL0m/ZatXDKFzO1smEbnnzup40c/qz/H/qUpQcA3s=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=jM2MgrCzDnOiciKCOggrMHFSRTMIvOQeSsOW6UFW6SzdP5Kwawe5z+fn/FIPqGJzT jHWdJubcWgvuE+w6tEzQNWcqR1gWIvSjIwLQUitK3UFTYc9Cr0B5CL1yZSthHSfPhN +2Svzw6bqg2aL4vU0kH87fSioOusXdpprt33pesgYQRx393QIWC1rHIqeeHrEmkDrk YMQyzTjSeTjVX+dyTs2Ia0KTmAVEwbspmB6z8z4h0DnG53ZKEAYnWZ4goWQ9vzS7yY DIYCU2p4Ohu2928zLF2svYf543E2VgbRaokHryUiAAA41DeNFb5HYccRsfKddX8ViF JMPSBjnKFoTmQ== Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [IPv6:2001:4b98:dc4:8::228]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AFA1E63330 for ; Thu, 11 Aug 2022 17:02:38 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id A7ED01BF20A; Thu, 11 Aug 2022 15:02:37 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Aug 2022 17:02:17 +0200 Message-Id: <20220811150219.62066-5-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220811150219.62066-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 4/6] ipa: raspberry: Port to the new AEGC controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The newly introduced controls to driver the AEGC algorithm allow to control the computation of the exposure time and analogue gain separately. The RPi AEGC implementation already computes the shutter and gain values separately but does not expose separate functions to control them. Augment the AgcAlgorithm interface to allow to pause/resume the shutter and gain automatic computations separately and plumb them to the newly introduced controls. Add safety checks to ignore ExposureTime and AnalogueGain values if the algorithms are not paused, and report the correct AeState value by checking if both algorithms have been paused or if they have converged. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder --- .../raspberrypi/controller/agc_algorithm.h | 6 ++ src/ipa/raspberrypi/controller/rpi/agc.cpp | 24 +++++- src/ipa/raspberrypi/controller/rpi/agc.h | 8 +- src/ipa/raspberrypi/raspberrypi.cpp | 76 ++++++++++++++++--- 4 files changed, 99 insertions(+), 15 deletions(-) diff --git a/src/ipa/raspberrypi/controller/agc_algorithm.h b/src/ipa/raspberrypi/controller/agc_algorithm.h index 3a91444c3a61..bf9c501db553 100644 --- a/src/ipa/raspberrypi/controller/agc_algorithm.h +++ b/src/ipa/raspberrypi/controller/agc_algorithm.h @@ -17,6 +17,12 @@ class AgcAlgorithm : public Algorithm public: AgcAlgorithm(Controller *controller) : Algorithm(controller) {} /* An AGC algorithm must provide the following: */ + virtual void pauseShutter(); + virtual void resumeShutter(); + virtual bool isShutterPaused() const; + virtual void pauseGain(); + virtual void resumeGain(); + virtual bool isGainPaused() const; virtual unsigned int getConvergenceFrames() const = 0; virtual void setEv(double ev) = 0; virtual void setFlickerPeriod(libcamera::utils::Duration flickerPeriod) = 0; diff --git a/src/ipa/raspberrypi/controller/rpi/agc.cpp b/src/ipa/raspberrypi/controller/rpi/agc.cpp index bd54a639d637..3d04724349f7 100644 --- a/src/ipa/raspberrypi/controller/rpi/agc.cpp +++ b/src/ipa/raspberrypi/controller/rpi/agc.cpp @@ -275,18 +275,36 @@ bool Agc::isPaused() const return false; } -void Agc::pause() +void Agc::pauseShutter() { fixedShutter_ = status_.shutterTime; - fixedAnalogueGain_ = status_.analogueGain; } -void Agc::resume() +void Agc::resumeShutter() { fixedShutter_ = 0s; +} + +bool Agc::isShutterPaused() const +{ + return fixedShutter_ != 0s; +} + +void Agc::pauseGain() +{ + fixedAnalogueGain_ = status_.analogueGain; +} + +void Agc::resumeGain() +{ fixedAnalogueGain_ = 0; } +bool Agc::isGainPaused() const +{ + return fixedAnalogueGain_ != 0; +} + unsigned int Agc::getConvergenceFrames() const { /* diff --git a/src/ipa/raspberrypi/controller/rpi/agc.h b/src/ipa/raspberrypi/controller/rpi/agc.h index 6d6b0e5ad857..cfb57f41848b 100644 --- a/src/ipa/raspberrypi/controller/rpi/agc.h +++ b/src/ipa/raspberrypi/controller/rpi/agc.h @@ -77,8 +77,12 @@ public: int read(const libcamera::YamlObject ¶ms) override; /* AGC handles "pausing" for itself. */ bool isPaused() const override; - void pause() override; - void resume() override; + void pauseShutter() override; + void resumeShutter() override; + bool isShutterPaused() const override; + void pauseGain() override; + void resumeGain() override; + bool isGainPaused() const override; unsigned int getConvergenceFrames() const override; void setEv(double ev) override; void setFlickerPeriod(libcamera::utils::Duration flickerPeriod) override; diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 69c73f8c780a..c041aac008eb 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -74,8 +74,13 @@ constexpr Duration controllerMinFrameDuration = 1.0s / 30.0; /* List of controls handled by the Raspberry Pi IPA */ static const ControlInfoMap::Map ipaControls{ - { &controls::AeEnable, ControlInfo(false, true) }, + { &controls::ExposureTimeMode, + ControlInfo(static_cast(controls::ExposureTimeModeAuto), + static_cast(controls::ExposureTimeModeManual)) }, { &controls::ExposureTime, ControlInfo(0, 66666) }, + { &controls::AnalogueGainMode, + ControlInfo(static_cast(controls::AnalogueGainModeAuto), + static_cast(controls::AnalogueGainModeManual)) }, { &controls::AnalogueGain, ControlInfo(1.0f, 16.0f) }, { &controls::AeMeteringMode, ControlInfo(controls::AeMeteringModeValues) }, { &controls::AeConstraintMode, ControlInfo(controls::AeConstraintModeValues) }, @@ -556,9 +561,18 @@ void IPARPi::reportMetadata() } AgcStatus *agcStatus = rpiMetadata_.getLocked("agc.status"); - if (agcStatus) { - libcameraMetadata_.set(controls::AeLocked, agcStatus->locked); + if (agcStatus) libcameraMetadata_.set(controls::DigitalGain, agcStatus->digitalGain); + + RPiController::AgcAlgorithm *agc = dynamic_cast( + controller_.getAlgorithm("agc")); + if (agc) { + if (agc->isShutterPaused() && agc->isGainPaused()) + libcameraMetadata_.set(controls::AeState, controls::AeStateIdle); + else if (agcStatus) + libcameraMetadata_.set(controls::AeState, + agcStatus->locked ? controls::AeStateConverged + : controls::AeStateSearching); } LuxStatus *luxStatus = rpiMetadata_.getLocked("lux.status"); @@ -703,20 +717,22 @@ void IPARPi::queueRequest(const ControlList &controls) << " = " << ctrl.second.toString(); switch (ctrl.first) { - case controls::AE_ENABLE: { - RPiController::Algorithm *agc = controller_.getAlgorithm("agc"); + case controls::EXPOSURE_TIME_MODE: { + RPiController::AgcAlgorithm *agc = dynamic_cast( + controller_.getAlgorithm("agc")); if (!agc) { LOG(IPARPI, Warning) - << "Could not set AE_ENABLE - no AGC algorithm"; + << "Could not set EXPOSURE_TIME_MODE - no AGC algorithm"; break; } - if (ctrl.second.get() == false) - agc->pause(); + if (ctrl.second.get() == controls::ExposureTimeModeManual) + agc->pauseShutter(); else - agc->resume(); + agc->resumeShutter(); - libcameraMetadata_.set(controls::AeEnable, ctrl.second.get()); + libcameraMetadata_.set(controls::ExposureTimeMode, + ctrl.second.get()); break; } @@ -729,6 +745,13 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + /* + * Ignore manual exposure time when the auto exposure + * algorithm is running. + */ + if (!agc->isShutterPaused()) + break; + /* The control provides units of microseconds. */ agc->setFixedShutter(ctrl.second.get() * 1.0us); @@ -736,6 +759,25 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + case controls::ANALOGUE_GAIN_MODE: { + RPiController::AgcAlgorithm *agc = dynamic_cast( + controller_.getAlgorithm("agc")); + if (!agc) { + LOG(IPARPI, Warning) + << "Could not set ANALOGUE_GAIN_MODE - no AGC algorithm"; + break; + } + + if (ctrl.second.get() == controls::AnalogueGainModeManual) + agc->pauseGain(); + else + agc->resumeGain(); + + libcameraMetadata_.set(controls::AnalogueGainMode, + ctrl.second.get()); + break; + } + case controls::ANALOGUE_GAIN: { RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); @@ -745,6 +787,13 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + /* + * Ignore manual analogue gain value when the auto gain + * algorithm is running. + */ + if (!agc->isGainPaused()) + break; + agc->setFixedAnalogueGain(ctrl.second.get()); libcameraMetadata_.set(controls::AnalogueGain, @@ -801,6 +850,13 @@ void IPARPi::queueRequest(const ControlList &controls) break; } + /* + * Ignore AE_EXPOSURE_MODE if the shutter or the gain + * are in auto mode. + */ + if (!agc->isShutterPaused() || !agc->isGainPaused()) + break; + int32_t idx = ctrl.second.get(); if (ExposureModeTable.count(idx)) { agc->setExposureMode(ExposureModeTable.at(idx)); From patchwork Thu Aug 11 15:02:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17081 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3EDD2BE173 for ; Thu, 11 Aug 2022 15:02:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F11086333C; Thu, 11 Aug 2022 17:02:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660230163; bh=hOZPkh2xuR9o9pecJHmkjnSBInfMf4EmLkB0U2PWgc4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=NKIrdhG4NOGelB5O+1IxKt5pYlrHah546zXfvcQRU3m2AjPuvtF7br3n9CXW3rja7 LJBOKEAHR7tuBeAVFLbMtOtHVGRv6stH+RRnH/+ktu14kWQf4YfIPsgLA5D70L40jZ GhWqvRAewaY1SshgyiKPuCwrFbeheFfSJy9dnO7fqx57+M85S0M/qlh5yok33fWy/G tHoK1H2TMoj1xwtdWcaW/e9+9AjjO8HXr7G10I96fGoZQ6Zrh+Z+p4xpvf9BncITjQ RNJ55XYcPZ1yt+hl0EONXPc58gpcTuUQ2FwJ1bkaGtcIk3e9tBiZ49LNfHdspdDpej mVdf4YqgEoMGA== Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D91CB63330 for ; Thu, 11 Aug 2022 17:02:39 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id E70151BF205; Thu, 11 Aug 2022 15:02:38 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Aug 2022 17:02:18 +0200 Message-Id: <20220811150219.62066-6-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220811150219.62066-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 5/6] ipa: rkisp1: Remove AE-related controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The RkISP1 IPA registers the AeEnabled control as available but does not handle it yet and returns the AeLocked control in the prepareMetadata() function with a value currently hard-coded to 0. Remove the AEGC-related controls as they currently have no purpose and both AeLocked and AeEnable will be removed in the next patch. A proper handling of the AEGC algorithm will have to be implemented using the newly introduced AEGC controls. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder --- src/ipa/rkisp1/rkisp1.cpp | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 17d42d38eb45..dac3a43f803d 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -64,7 +64,6 @@ protected: private: void setControls(unsigned int frame); - void prepareMetadata(unsigned int frame, unsigned int aeState); std::map buffers_; std::map mappedBuffers_; @@ -91,7 +90,6 @@ namespace { /* List of controls handled by the RkISP1 IPA */ const ControlInfoMap::Map rkisp1Controls{ - { &controls::AeEnable, ControlInfo(false, true) }, { &controls::Brightness, ControlInfo(-1.0f, 0.993f) }, { &controls::Contrast, ControlInfo(0.0f, 1.993f) }, { &controls::Saturation, ControlInfo(0.0f, 1.993f) }, @@ -321,14 +319,15 @@ void IPARkISP1::processStatsBuffer(const uint32_t frame, const uint32_t bufferId context_.frameContext.sensor.gain = camHelper_->gain(sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); - unsigned int aeState = 0; - for (auto const &algo : algorithms()) algo->process(context_, nullptr, stats); setControls(frame); - prepareMetadata(frame, aeState); + /* \todo: prepare metadata. */ + + ControlList ctrls(controls::controls); + metadataReady.emit(frame, ctrls); } void IPARkISP1::setControls(unsigned int frame) @@ -343,16 +342,6 @@ void IPARkISP1::setControls(unsigned int frame) setSensorControls.emit(frame, ctrls); } -void IPARkISP1::prepareMetadata(unsigned int frame, unsigned int aeState) -{ - ControlList ctrls(controls::controls); - - if (aeState) - ctrls.set(controls::AeLocked, aeState == 2); - - metadataReady.emit(frame, ctrls); -} - } /* namespace ipa::rkisp1 */ /* From patchwork Thu Aug 11 15:02:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17082 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AA017C3275 for ; Thu, 11 Aug 2022 15:02:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 751136333A; Thu, 11 Aug 2022 17:02:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660230163; bh=afaZN8vbiqmLB0nz7no8tWe9gCm+D/lzo+Zw05i3oTk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=GThj4kbVJR6QqhnZ4vgFnVeR4DW37tKAcHEG6z3rHcnTYLkXuw2miDT8evvm4pvqL a53u0Pk+LUiglE+ApKiqgNioMkmvxjoaxbRjjiUb56Dr1NN+fxe5IhyAowgvto9YrT PqDUQa1cT8AXlrGBLsz31nIYN5HriD963BJH22YjeYvcFjRMioaX3XXQ5tILP7TZ5R +2O7px5p9EL4RvAjnr6v6UmCeCxpE1QvpDsOqqh/HoryglD7vC0dlRJbjFw4hvMjb+ I2ofKG5FK6rWHtCz6H+KBRL6i1KW4ieoxTxuTSKn3HYrD6uGhRv/PKyRIsbUoYgxJI r2ia6tlz5+Vkg== Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 472AC63330 for ; Thu, 11 Aug 2022 17:02:41 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id 263FE1BF20A; Thu, 11 Aug 2022 15:02:39 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Thu, 11 Aug 2022 17:02:19 +0200 Message-Id: <20220811150219.62066-7-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220811150219.62066-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 6/6] controls: Remove AeEnable and AeLocked X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the codebase has been ported to use the new AEGC controls remove the definition of AeEnable and AeLocked. Signed-off-by: Jacopo Mondi Reviewed-by: Paul Elder --- src/libcamera/control_ids.yaml | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml index 694659af4b7d..f7f8b2e525d8 100644 --- a/src/libcamera/control_ids.yaml +++ b/src/libcamera/control_ids.yaml @@ -7,24 +7,6 @@ # Unless otherwise stated, all controls are bi-directional, i.e. they can be # set through Request::controls() and returned out through Request::metadata(). controls: - - AeEnable: - type: bool - description: | - Enable or disable the AE. - - \sa ExposureTime AnalogueGain - - - AeLocked: - type: bool - description: | - Report the lock status of a running AE algorithm. - - If the AE algorithm is locked the value shall be set to true, if it's - converging it shall be set to false. If the AE algorithm is not - running the control shall not be present in the metadata control list. - - \sa AeEnable - - AeState: type: int32_t description: | From patchwork Fri Sep 23 08:52:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17389 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E3A53C0DA4 for ; Fri, 23 Sep 2022 08:52:40 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4B2E46222C; Fri, 23 Sep 2022 10:52:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1663923160; bh=+Hr2MDQTRcmktC4am1RnCysmVamhKenRq/kVjVwA7ZE=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=I/eJ9mfQQavVKSYH7bf01UQFLctNC56DvhtvociHKG99OHmUtmBQGBdPWrFqzF4Bh GeSgTKIev4gXbyihzQu1fsdop6XVz4G/Ge9vVIvT4ahsYKJN6+OA2Vd7zzKzH30cFO oD4a25z3Wxm7jOp0uqNqX58c+/YLqDg/q0gR9BBR2AOlMt7Yjn2XbhW2wCn1YCKY+6 25qxS796A3miFCrCltHLTBlu1BFLVswxGR8kyMZjh5bOEXey5jDOBWLoH3qStV6hX+ +sCTBfYc3pHwwNjtKbumMNMcR8b9tjjaI8NKbN2so1LQZomOvAnM30tO5V3H7onkoL +1M+OmUVRYaUA== Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [IPv6:2001:4b98:dc4:8::232]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CCD52621BC for ; Fri, 23 Sep 2022 10:52:38 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id 90C07200005; Fri, 23 Sep 2022 08:52:37 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Fri, 23 Sep 2022 10:52:21 +0200 Message-Id: <20220923085222.7759-1-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220811150219.62066-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 7/6] Documentation: design: ae: Document the design for AE controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Document the design and rationale for the AE-related controls. Signed-off-by: Paul Elder Signed-off-by: Jacopo Mondi --- v1->v2: - s/Inactive/Idle - s/Disabled/Manual --- Documentation/design/ae.rst | 260 ++++++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 Documentation/design/ae.rst -- 2.37.3 diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst new file mode 100644 index 000000000000..ecef301ba3df --- /dev/null +++ b/Documentation/design/ae.rst @@ -0,0 +1,260 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +Design of Exposure and Gain controls +==================================== + +This document explains the design and rationale of the controls related to +exposure and gain. This includes the all-encompassing auto-exposure (AE), the +manual exposure control, and the manual gain control. + +Description of the problem +-------------------------- + +Sub controls +^^^^^^^^^^^^ + +There are more than one control that make up exposure: exposure, gain, and +aperture (though for now we will not consider aperture). We already had +individual controls for setting the values of manual exposure and manual gain, +but for switching between auto mode and manual mode we only had a high-level +boolean AeEnable control that would set *both* exposure and gain to auto mode +or manual mode; we had no way to set one to auto and the other to manual. + +So, we need to introduce two new controls to act as "levers" to indicate +individually for exposure and gain if the value would come from AEGC or if it +would come from the manual control value. + +Aperture priority +^^^^^^^^^^^^^^^^^ + +We eventually will support aperture, and so whatever our solution is for having +only some controls on auto and the others on manual needs to be extensible. + +Flickering when going from auto to manual +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a manual exposure or gain value is requested by the application, it costs +a few frames worth of time for them to take effect. This means that during a +transition from auto to manual, there would be flickering in the control values +and the transition won't be smooth. + +Take for instance the following flow, where we start on auto exposure (which +for the purposes of the example increments by 1 each frame) and we want to +switch seamlessly to manual exposure, which involves copying the exposure value +computed by the auto exposure algorithm: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A 2 2 2 2 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 2 2 + +As we can see, after frame N+2 completes, we copy the exposure value that was +used for frame N+2 (which was computed by AE algorithm), and queue that value +into request N+3 with manual mode on. However, as it takes two frames for the +exposure to be set, the exposure still changes since it is set by AE, and we +get a flicker in the exposure during the switch from auto to manual. + +A solution is to *not submit* any exposure value when manual mode is enabled, +and wait until the manual mode as been "applied" before copying the exposure +value: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + + Mode requested: Auto Auto Auto Manual Manual Manual Manual + Exp requested: N/A N/A N/A None None None 5 + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8 + + Mode used: Auto Auto Auto Auto Auto Manual Manual + Exp used: 0 1 2 3 4 5 5 + +In practice, this works. However, libcamera has a policy where once a control +is submitted, its value is saved and does not need to be resubmitted (even +though this isn't implemented yet). So if the manual exposure value was set +while auto mode was on, in theory the value would be saved, so when manual mode +is enabled, the exposure value that was previously set would immediately be +used. Clearly this solution isn't correct, but it can serve as the basis for a +proper solution, with some more rigorous rules. + +Existing solutions +------------------ + +Raspberry Pi +^^^^^^^^^^^^ + +The raspberry pi IPA got around the lack of individual AeEnable controls for +exposure and gain by using magic values. When AeEnable was false, if one of the +manual control values was set to 0 then the value computed by AEGC would be +used for just that control. This solution isn't desirable, as it prevents +that magic value from being used as a valid value. + +To get around the flickering issue, when AeEnable was false, the raspberry pi +AEGC would simply stop updating the values to be set. As mentioned above, since +the value retention mechanism hasn't actually been implemented yet, this +worked. But, it's not a proper solution. + +Android +^^^^^^^ + +The Android HAL specification requires that exposure and gain (sensitivity) +must both be manual or both be auto. It cannot be that one is manual while the +other is auto, so they simply don't support sub controls. + +For the flickering issue, the Android HAL has an AeLock control. To transition +from auto to manual, the application would keep AE on auto, and turn on the +lock. Once the lock has propagated through, then the value can be copied from +the result into the request and the lock disabled and the mode set to manual. + +The problem with this solution is, besides the extra complexity, that it is +ambiguous what happens if there is a state transition from manual to locked +(even though it's a state transition that doesn't make sense). If locked is +defined to "use the last automatically computed values" then it could use the +values from the last time it AE was set to auto, or it would be undefined if AE +was never auto (eg. it started out as manual), or if AE is implemented to run +in the background it could just use the current values that are computed. If +locked is defined to "use the last value that was set" there would be less +ambiguity. Still, it's better if we can make it impossible to execute this +nonsensical state transition, and if we can reduce the complexity of having +this extra control or extra setting on a lever. + +Summary of goals +---------------- + + - We need a lock of some sort, to instruct the AEGC to not update output + results + + - We need manual modes, to override the values computed by the AEGC + + - We need to support seamless transitions from auto to manual, and do so + without flickering + + - We need custom minimum values for the manual controls; that is, no magic + values for enabling/disabling auto + + - All of these need to be done with AE sub-controls (exposure time, analogue + gain) and be extensible to aperture in the future + +Our solution +------------ + +A diagram of our solution: + +:: + + +----------------------------+-------------+------------------+-----------------+ + | INPUT | ALGORITHM | RESULT | OUTPUT | + +----------------------------+-------------+------------------+-----------------+ + + ExposureTimeMode ExposureTimeMode + ---------------------+----------------------------------------+-----------------> + 0: Auto | | + 1: Manual | V + | |\ + | | \ + | /----------------------------------> | 1| ExposureTime + | | +-------------+ exposure time | | --------------> + \--)--> | | --------------> | 0| + ExposureTime | | | | / + ------------------------+--> | | |/ + | | AeState + | AEGC | -----------------------------------> + AnalogueGain | | + ------------------------+--> | | |\ + | | | | \ + /--)--> | | --------------> | 0| AnalogueGain + | | +-------------+ analogue gain | | --------------> + | \----------------------------------> | 1| + | | / + | |/ + | ^ + AnalogueGainMode | | AnalogueGainMode + ---------------------+----------------------------------------+-----------------> + 0: Auto + 1: Manual + + +The diagram is divided in four sections horizontally: + + - Input: The values received from the request controls + + - Algorithm: The algorithm itself + + - Result: The values calculated by the algorithm + + - Output: The values that sent in result metadata and applied to the device + +The four input controls are divided between manual values (ExposureTime and +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The +former are the manual values, the latter control how they're applied. The two +modes are independent from each other, and each can take one of two values: + + - Auto (0): The AGC computes the value normally. The AGC result is applied + to the output. The manual value is ignored *and is not retained*. + + - Manual (1): The AGC uses the manual value internally. The corresponding + manual control from the request is applied to the output. The AGC result + is ignored. + +The AeState control reports the state of the unified AEGC block. If both +ExposureTimeMode and AnalogueGainMode are set to disabled then it will report +Idle. If at least one of the two is set to auto, then AeState will report +if the AEGC has Converged or not (Searching). This control replaces the old +AeLocked control, as it was insufficient for reporting the AE state. + +There is a caveat to the disabled mode: the manual control value is not +retained if it is set during auto mode. This means that if the disabled mode is +entered without also setting the manual value, then it will enter a state +similar to "locked", where the last automatically computed value while the mode +was auto will be used. Once the manual value is set, then that will be used and +retained as usual. + +This simulates an auto -> locked -> manual or auto -> manual state transition, +and makes it impossible to do the nonsensical manual -> locked state +transition. + +We specifically do not have a "master AE control" like the old AeEnable. This +is because we have the individual mode controls, and if we had a master AE +control it would be a "control that sets other controls", which could easily +get out of control. + +With this solution, the earlier example would become: + +:: + + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10| + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ + Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab + Exp requested: N/A None None None None 10 None 10 10 + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10 + + Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab + Exp used: 2 3 4 5 5 5 5 10 10 + +This example is extended by a few frames to exhibit the simulated "locked" +state. At frame N+5 the application has confirmed that the auto mode has been +disabled, but does not provide a manual value until request N+7. Thus, the +value that is used in requests N+5 and N+6 (where the mode is disabled), comes +from the last value that was used when the mode was auto, which comes from +frame N+4. + +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for +the exposure to be applied. N+8 does not supply a manual value, but the last +supplied value is retained, so a manual value of 10 is still used and set in +frame N+10. + +Although this behavior is the same as what we had with waiting for the manual +mode to propagate (in the section "Description of the problem"), this time it +is correct as we have defined specifically that if a manual value was specified +while the mode was auto, it will not be retained. From patchwork Fri Sep 23 08:52:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacopo Mondi X-Patchwork-Id: 17390 X-Patchwork-Delegate: jacopo@jmondi.org Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EC41CC327E for ; Fri, 23 Sep 2022 08:52:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BB512621BC; Fri, 23 Sep 2022 10:52:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1663923160; bh=wWKBEGgh/ilXdEvxaLJPHBUL0oV5TdmHRJ3CCSuJzKQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=VofWZZsy8UsBjzm6d6kkQT1Nc6dyd4Kn+xdKlurrxLL5NtUarkKTs8Z87Huj89GZx L2SKVoNg56SFecLKhjJ47L919bDgoMXsxlydtqLnXh9Pu8jgO6/PdPZL+seF+GgdjK zwhQVCCj3wmtVkXTKa4ZGWemV6ABaJuUBUqryopDoJ8zfjBnXhaI5cmcssdLCydYog r3xsAfy0wxhNCkFW9ZWzsv5EqET0IG74sU53Ry7xuHx5TYWqAgccDD48OwMjNmkd5T L4t+4s+/zcYDEO88nXjHMTf+PSfjGRclPntricUpyrmO3y7PFWzbqI7dccsTPe2fpF noNdZBgt2hblw== Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [IPv6:2001:4b98:dc4:8::232]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ABC0662229 for ; Fri, 23 Sep 2022 10:52:39 +0200 (CEST) Received: (Authenticated sender: jacopo@jmondi.org) by mail.gandi.net (Postfix) with ESMTPSA id E509E200003; Fri, 23 Sep 2022 08:52:38 +0000 (UTC) To: libcamera-devel@lists.libcamera.org Date: Fri, 23 Sep 2022 10:52:22 +0200 Message-Id: <20220923085222.7759-2-jacopo@jmondi.org> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220923085222.7759-1-jacopo@jmondi.org> References: <20220811150219.62066-1-jacopo@jmondi.org> <20220923085222.7759-1-jacopo@jmondi.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 8/6] libcamera: control_ids: Add high-level documentation for AE X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Jacopo Mondi via libcamera-devel From: Jacopo Mondi Reply-To: Jacopo Mondi Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add high-level documentation for AE, describing the states and the transitions and how one would use the AE controls. Signed-off-by: Paul Elder Signed-off-by: Jacopo Mondi --- v1->v2: - s/Inactive/Idle - s/Disabled/Manual -- src/libcamera/control_ids.cpp.in | 82 ++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) -- 2.37.3 diff --git a/src/libcamera/control_ids.cpp.in b/src/libcamera/control_ids.cpp.in index 5fb1c2c30558..4b3504f2c62c 100644 --- a/src/libcamera/control_ids.cpp.in +++ b/src/libcamera/control_ids.cpp.in @@ -10,6 +10,88 @@ #include #include +/** + * \page AE Auto-exposure Support + * + * libcamera offers the following controls related to exposure and gain: + * - AnalogueGain + * - AnalogueGainMode + * - ExposureTime + * - ExposureTimeMode + * - AeState + * + * Auto-exposure and auto-gain can be enabled and disabled separately using the + * ExposureTimeMode and AnalogueGainMode controls respectively. There is no + * overarching AeEnable control. + * + * For each of exposure and gain, we can model it with three states: auto, + * locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only + * have two values, as the locked state is simulated. + * + * /---------------------------------\ + * | | + * V | + * +--------+ +--------+ + * | | ---------------------> | | + * | Auto | +--------+ | Manual | + * | | ----> | Locked | ----> | | + * +--------+ +--------+ +--------+ + * ^ | + * | | + * \----------------/ + * + * Notice from the state diagram that locked to manual is a one-way state + * change, as the reverse direction is nonsensical (see the design document for + * more details on this topic). + * + * The exposure/gain is in the Auto state when + * ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value + * that is computed by the AE algorithm is applied to the image sensor. Any + * value that is supplied in the ExposureTime/AnalogueGain control is ignored + * and is not retained. + * + * If ExposureTimeMode/AnalogueGainMode is set to Manual, it can put us in + * either the Locked or Manual state. The difference is in if + * ExposureTime/AnalogueGain has been supplied. If it has not yet been + * supplied, then we are in the Locked state. If it has been supplied, then we + * are in the Manual state. + * + * In both the Locked state and the Manual state the exposure/gain value does + * not come from the AE algorithm. In the Locked state the value comes from the + * last value computed by the AE algorithm while the state was Auto, or if the + * state was never Auto (e.g. we started in Locked, or the camera doesn't + * support Auto), then the value should be a best-effort default value. In the + * Manual state the value comes from the value supplied in the + * ExposureTime/AnalogueGain control. + * + * To transition from the Locked state to the Manual state, a value needs to be + * submitted in ExposureTime/AnalogueGain. Once the state has transitioned to + * Manual, then this value will be retained, and doesn't need to be resubmitted + * if it doesn't change. + * + * To transition to the Auto state, simply set + * ExposureTimeMode/AnalogueGainMode to Auto. + * + * + * The AeState metadata reports the state of the AE algorithm. As AE cannot + * compute exposure and gain separately, the state of the AE component is + * unified. There are three states: Idle, Searching, and Converged. + * + * The state shall be Idle if both ExposureTimeMode and AnalogueGainMode + * are set to Manual. If the camera only supports one of the two controls, + * then the state shall be Idle if that one control is set to Manual. If + * the camera does not support Manual for at least one of the two controls, + * then the state will never be Idle, as AE will always be running. + * + * The state shall be Searching if at least one of exposure or gain calculated + * by the AE algorithm is used (that is, at least one of the two modes is Auto), + * *and* the value(s) have not converged yet. + * + * The state shall be Converged if at least one of exposure or gain calculated + * by the AE algorithm is used (that is, at least one of the two modes is Auto), + * *and* the value(s) have converged. + */ + /** * \file control_ids.h * \brief Camera control identifiers