[{"id":36511,"web_url":"https://patchwork.libcamera.org/comment/36511/","msgid":"","date":"2025-10-28T09:53:15","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":239,"url":"https://patchwork.libcamera.org/api/people/239/","name":"Hans de Goede","email":"hansg@kernel.org"},"content":"Hi Jacopo,\n\nOn 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> By definition, the exposure time that can be programmed on a sensor is\n> limited by the frame duration and whenever the frame duration is\n> updated, the exposure time limits should be updated as well.\n> \n> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> libipa, where the exposure limits are computed at configure() time and\n> never updated.\n> \n> This has two implications:\n> 1) The AGC algorithms will always operate with an exposure time bound to\n> the frame duration programmed at startup time\n> 2) The Camera::controls() limits are not updated to reflect the new\n> constraints\n> \n> This series addresses issue 1) by changing the AgcMeanLuminance class\n> and the associated helpers to regulate the exposure time base on the\n> currently programmed max frame duration using a sensor-specific margin\n> that is now reported by the CameraHelper class.\n\nI have a question about the above paragraph. The sensor-specific margin\nis already known by the kernel's sensor driver and properly written\nsensor drivers will update the exposure range reported by the query-ctrl\nioctl after changing the vblank control.\n\nSo I think it would be better to rely on the kernel for this info\nand refresh the cached exposure-control limits when changing the vblank\ncontrol rather then duplicating this info in the CameraHelper class ?\n\nOne could even calculate the exposure-margin by calculating the configured\nframe-time (in a unit of lines) and then substracting the max-exposure\nreported by the sensor from the frame-time-in-lines.\n\n> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> to the maximum allowed frame duration, initializing the algorithm\n> with the sensor's maximum capabilities results in a very low frame rate,\n> unless the user specifies a FrameDurationLimits range in which the\n> algorithm should operate. To avoid slowing the frame rate down too much\n> compute a \"default\" frame rate at algorithm configuration time, were\n> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> of preference.\n\nI must admit I'm not familiar with the AgcMeanLuminance algorithm.\nDoes this algorithm also set / used to set the vblank control to\nthe maximum allowed frame duration? Or does it dynamically adjust\nvblank when it needs more exposure \"range\" ?\n\nI've been thinking about adding support to dynamically increase\nvblank when more exposure time is needed in the softISP AGC code,\nbut it would probably be better to first convert that over to\nthe shared AGC helpers. I know that Kieran is working on this.\n\nAt least some UVC cameras have an auto-exposure priority (or is mode?)\nv4l2 control which can be set to \"aperture priority\" to make them\nautomatically increase frameduration if necessary because of low\nlight conditions. I think we want something similar for libcamera's\nAGC.\n\nRegards,\n\nHans\n\n\n\n\n\n\n> \n> The last patch is not for inclusion as Mali support needs to be\n> finalized upstream first, but shows how is it possible to handle\n> frame duration limits and exposure with the new model (and I needed that\n> patch for testing on Mali anyway).\n> \n> Tested with camshark on Mali-C55 and with a test scripts that sets a\n> longer frame duration (66 msecs) and by validating that the exposure\n> time is increased to match the new limits on RkISP1.\n> \n> RkISP1Agc agc.cpp:644 Divided up exposure time, analogue gain, quantization gain and digital gain are 33251.27us, 10.6667, 1.00018 and 6.49639\n> 61.545331 (47.58 fps) cam0-stream0 seq: 000050 bytesused: 2073600/1036800\n> kISP1Agc agc.cpp:644 Divided up exposure time, analogue gain, quantization gain and digital gain are 66578.16us, 10.6667, 1.00018 and 3.24451\n> 61.566241 (47.82 fps) cam0-stream0 seq: 000051 bytesused: 2073600/1036800\n> \n> Signed-off-by: Jacopo Mondi \n> ---\n> Changes in v2:\n> - Rework of v1 that moves the exposure limits computation to the\n> AgcMeanLuminance exposure helpers\n> \n> ---\n> Jacopo Mondi (9):\n> ipa: camera_sensor_helper: Introduce exposureMargin()\n> ipa: ipu3: Move CameraHelper to context\n> ipa: libipa: agc: Make Agc::configure() set limits\n> ipa: libipa: agc: Configure with frame duration\n> ipa: libipa: agc: Initialize exposure with frame duration\n> ipa: libipa: agc: Set exposure limits with frame duration\n> ipa: libipa: agc: Initialize a sensible frame duration\n> ipa: libipa: agc: Calculate frame duration in helper\n> [DNI] ipa: mali: Handle FrameDurationLimits\n> \n> Kieran Bingham (1):\n> ipa: mali-c55: Move CameraHelper to context\n> \n> src/ipa/ipu3/algorithms/agc.cpp | 32 +++++++---\n> src/ipa/ipu3/ipa_context.cpp | 3 +\n> src/ipa/ipu3/ipa_context.h | 3 +\n> src/ipa/ipu3/ipu3.cpp | 22 +++----\n> src/ipa/libipa/agc_mean_luminance.cpp | 73 ++++++++++++++++++---\n> src/ipa/libipa/agc_mean_luminance.h | 18 +++++-\n> src/ipa/libipa/camera_sensor_helper.cpp | 72 +++++++++++++++++++++\n> src/ipa/libipa/camera_sensor_helper.h | 2 +\n> src/ipa/libipa/exposure_mode_helper.cpp | 109 +++++++++++++++++++++++++++-----\n> src/ipa/libipa/exposure_mode_helper.h | 16 ++++-\n> src/ipa/mali-c55/algorithms/agc.cpp | 76 +++++++++++++++++++---\n> src/ipa/mali-c55/ipa_context.cpp | 6 ++\n> src/ipa/mali-c55/ipa_context.h | 12 ++++\n> src/ipa/mali-c55/mali-c55.cpp | 44 ++++++-------\n> src/ipa/rkisp1/algorithms/agc.cpp | 42 ++++++------\n> 15 files changed, 430 insertions(+), 100 deletions(-)\n> ---\n> base-commit: 36f9cdcdb4a3c69e10b7da8a3de8354421cdcb56\n> change-id: 20251017-exposure-limits-5c48252fe548\n> \n> Best regards,","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 59610BE080\n\tfor ;\n\tTue, 28 Oct 2025 09:53:22 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 66C8E607A3;\n\tTue, 28 Oct 2025 10:53:21 +0100 (CET)","from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 12A8A603ED\n\tfor ;\n\tTue, 28 Oct 2025 10:53:20 +0100 (CET)","from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58])\n\tby tor.source.kernel.org (Postfix) with ESMTP id A90176022B;\n\tTue, 28 Oct 2025 09:53:18 +0000 (UTC)","by smtp.kernel.org (Postfix) with ESMTPSA id 47632C4CEE7;\n\tTue, 28 Oct 2025 09:53:17 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=kernel.org header.i=@kernel.org\n\theader.b=\"LlHZp4Qs\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;\n\ts=k20201202; t=1761645198;\n\tbh=ciukWR0mYu6+GaZwsTGbz2YqRZpDKdnyZGcS/wFuQhY=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=LlHZp4QsvMdXCrMdAzcCHbTSrQaRiowtILw8+qni9IFYwKqq9sj47PQPNfKuy+/vC\n\tj6JIaWOvOez4oLlBPkhOfXeVPZcYyyM6mSxLRQjR5ZmQJTh6fzOcN9NeoVaemvpKLA\n\t4sjw4AxGtLbN704zDinA0s6DPiz6/L81JvKm17PaYVpK2WBG8zrKaMqxa8A1Irtl37\n\t+hjDyFzFv0MrqcBpMwiXKp2V/SdGrMG8tSIe3gVhcH9chDY4iQHtpfjpi/AL/zpWp6\n\txvS4fPQSR2GaU9GTiz7pKkDbcslIOSPnkafOK/MhFvCNuM/+tIJR82yrFfBncGTME+\n\t8EiTV2LStjlEA==","Message-ID":"","Date":"Tue, 28 Oct 2025 10:53:15 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Jacopo Mondi , =?utf-8?q?Niklas_S=C3=B6?=\n\t=?utf-8?q?derlund?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org","Cc":"Kieran Bingham ","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>","From":"Hans de Goede ","Content-Language":"en-US, nl","In-Reply-To":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"7bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36514,"web_url":"https://patchwork.libcamera.org/comment/36514/","msgid":"<96a07c26-ddf6-48b1-83d5-d81d4909a392@collabora.com>","date":"2025-10-28T11:26:07","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":140,"url":"https://patchwork.libcamera.org/api/people/140/","name":"Robert Mader","email":"robert.mader@collabora.com"},"content":"Just gave it a go on my PinePhone Pro (rkisp1/rk3399) and can confirm it \nstill fixes low framerates for me. So for the common and rkisp1 bits:\n\nTested-by: Robert Mader\n\nOn 10/28/25 10:31, Jacopo Mondi wrote:\n> By definition, the exposure time that can be programmed on a sensor is\n> limited by the frame duration and whenever the frame duration is\n> updated, the exposure time limits should be updated as well.\n>\n> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> libipa, where the exposure limits are computed at configure() time and\n> never updated.\n>\n> This has two implications:\n> 1) The AGC algorithms will always operate with an exposure time bound to\n> the frame duration programmed at startup time\n> 2) The Camera::controls() limits are not updated to reflect the new\n> constraints\n>\n> This series addresses issue 1) by changing the AgcMeanLuminance class\n> and the associated helpers to regulate the exposure time base on the\n> currently programmed max frame duration using a sensor-specific margin\n> that is now reported by the CameraHelper class.\n>\n> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> to the maximum allowed frame duration, initializing the algorithm\n> with the sensor's maximum capabilities results in a very low frame rate,\n> unless the user specifies a FrameDurationLimits range in which the\n> algorithm should operate. To avoid slowing the frame rate down too much\n> compute a \"default\" frame rate at algorithm configuration time, were\n> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> of preference.\n>\n> The last patch is not for inclusion as Mali support needs to be\n> finalized upstream first, but shows how is it possible to handle\n> frame duration limits and exposure with the new model (and I needed that\n> patch for testing on Mali anyway).\n>\n> Tested with camshark on Mali-C55 and with a test scripts that sets a\n> longer frame duration (66 msecs) and by validating that the exposure\n> time is increased to match the new limits on RkISP1.\n>\n> RkISP1Agc agc.cpp:644 Divided up exposure time, analogue gain, quantization gain and digital gain are 33251.27us, 10.6667, 1.00018 and 6.49639\n> 61.545331 (47.58 fps) cam0-stream0 seq: 000050 bytesused: 2073600/1036800\n> kISP1Agc agc.cpp:644 Divided up exposure time, analogue gain, quantization gain and digital gain are 66578.16us, 10.6667, 1.00018 and 3.24451\n> 61.566241 (47.82 fps) cam0-stream0 seq: 000051 bytesused: 2073600/1036800\n>\n> Signed-off-by: Jacopo Mondi\n> ---\n> Changes in v2:\n> - Rework of v1 that moves the exposure limits computation to the\n> AgcMeanLuminance exposure helpers\n>\n> ---\n> Jacopo Mondi (9):\n> ipa: camera_sensor_helper: Introduce exposureMargin()\n> ipa: ipu3: Move CameraHelper to context\n> ipa: libipa: agc: Make Agc::configure() set limits\n> ipa: libipa: agc: Configure with frame duration\n> ipa: libipa: agc: Initialize exposure with frame duration\n> ipa: libipa: agc: Set exposure limits with frame duration\n> ipa: libipa: agc: Initialize a sensible frame duration\n> ipa: libipa: agc: Calculate frame duration in helper\n> [DNI] ipa: mali: Handle FrameDurationLimits\n>\n> Kieran Bingham (1):\n> ipa: mali-c55: Move CameraHelper to context\n>\n> src/ipa/ipu3/algorithms/agc.cpp | 32 +++++++---\n> src/ipa/ipu3/ipa_context.cpp | 3 +\n> src/ipa/ipu3/ipa_context.h | 3 +\n> src/ipa/ipu3/ipu3.cpp | 22 +++----\n> src/ipa/libipa/agc_mean_luminance.cpp | 73 ++++++++++++++++++---\n> src/ipa/libipa/agc_mean_luminance.h | 18 +++++-\n> src/ipa/libipa/camera_sensor_helper.cpp | 72 +++++++++++++++++++++\n> src/ipa/libipa/camera_sensor_helper.h | 2 +\n> src/ipa/libipa/exposure_mode_helper.cpp | 109 +++++++++++++++++++++++++++-----\n> src/ipa/libipa/exposure_mode_helper.h | 16 ++++-\n> src/ipa/mali-c55/algorithms/agc.cpp | 76 +++++++++++++++++++---\n> src/ipa/mali-c55/ipa_context.cpp | 6 ++\n> src/ipa/mali-c55/ipa_context.h | 12 ++++\n> src/ipa/mali-c55/mali-c55.cpp | 44 ++++++-------\n> src/ipa/rkisp1/algorithms/agc.cpp | 42 ++++++------\n> 15 files changed, 430 insertions(+), 100 deletions(-)\n> ---\n> base-commit: 36f9cdcdb4a3c69e10b7da8a3de8354421cdcb56\n> change-id: 20251017-exposure-limits-5c48252fe548\n>\n> Best regards,","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E0357C3259\n\tfor ;\n\tTue, 28 Oct 2025 11:26:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D3B9C607A7;\n\tTue, 28 Oct 2025 12:26:19 +0100 (CET)","from sender4-pp-f112.zoho.com (sender4-pp-f112.zoho.com\n\t[136.143.188.112])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C6680606DE\n\tfor ;\n\tTue, 28 Oct 2025 12:26:17 +0100 (CET)","by mx.zohomail.com with SMTPS id 1761650769717460.13321503495456; \n\tTue, 28 Oct 2025 04:26:09 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=collabora.com\n\theader.i=robert.mader@collabora.com header.b=\"H3pNCjj2\"; \n\tdkim-atps=neutral","ARC-Seal":"i=1; a=rsa-sha256; t=1761650773; cv=none; \n\td=zohomail.com; s=zohoarc; \n\tb=cB2r1gcDPJQr/j0FVCvIu2FKi3WkXC51PlgRAX/EoHkOzzLiU2AvAmwcyLvXBmcXFBPaC/FvcdmnxogxgmwxnJ3BKy2cNqw5SooFZPtYi80jyGWoi8WaKObCi13XzAVs5c2VIUWPoDmirjDj+M5elwOdxeDzVVztgqW8ATWF8Mk=","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; \n\ts=zohoarc; t=1761650773;\n\th=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To;\n\tbh=Ostzdw71lFFbs+FPasPtMcKwpxO0zVxbmQz9dgUU3AM=; \n\tb=VGWfyAloifz369/6MvsEgtBiKDOCJH6HyG7P+pfqwWiZJSr88D5mbNI9ZbGgW70+Jv8vBFl5kZPJMFPz0MCb7SN39WncKSBdCp3IgwZRQR4XCLndZEuDqui6vd+6VjqQXkV4y0g8CG8z8CdpN4iGG/Kp6HzKgKKbmqvgYIzU+Ds=","ARC-Authentication-Results":"i=1; mx.zohomail.com;\n\tdkim=pass header.i=collabora.com;\n\tspf=pass smtp.mailfrom=robert.mader@collabora.com;\n\tdmarc=pass header.from=","DKIM-Signature":"v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1761650773;\n\ts=zohomail; d=collabora.com; i=robert.mader@collabora.com;\n\th=Content-Type:Message-ID:Date:Date:MIME-Version:Subject:Subject:To:To:Cc:Cc:References:From:From:In-Reply-To:Message-Id:Reply-To;\n\tbh=Ostzdw71lFFbs+FPasPtMcKwpxO0zVxbmQz9dgUU3AM=;\n\tb=H3pNCjj2TzOwiPVsgnDqJtx5m/wAT6oFI/Wmjt7IgEpX4y1YgR/1SAXflSGdtOjH\n\twl5ffrqVJrfFfvwd4DtvHgiuQPYpng/sgqzQ78aOapcXEEoLpMmTPhtu6KsxqBZGwsW\n\tf2Z5+yKl4lsPbKQ5/6ORRZIavFa1Ng46mgZftCw4=","Content-Type":"multipart/alternative;\n\tboundary=\"------------sK5gj6I0Cx6S8Gp0Zbdje8wo\"","Message-ID":"<96a07c26-ddf6-48b1-83d5-d81d4909a392@collabora.com>","Date":"Tue, 28 Oct 2025 12:26:07 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Jacopo Mondi , =?utf-8?q?Niklas_S=C3=B6?=\n\t=?utf-8?q?derlund?= ,\n\tlibcamera-devel@lists.libcamera.org","Cc":"Kieran Bingham ","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>","Content-Language":"en-US, de-DE","From":"Robert Mader ","In-Reply-To":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36515,"web_url":"https://patchwork.libcamera.org/comment/36515/","msgid":"","date":"2025-10-28T11:42:07","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Hans\n\nOn Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> Hi Jacopo,\n>\n> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> > By definition, the exposure time that can be programmed on a sensor is\n> > limited by the frame duration and whenever the frame duration is\n> > updated, the exposure time limits should be updated as well.\n> >\n> > This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> > libipa, where the exposure limits are computed at configure() time and\n> > never updated.\n> >\n> > This has two implications:\n> > 1) The AGC algorithms will always operate with an exposure time bound to\n> > the frame duration programmed at startup time\n> > 2) The Camera::controls() limits are not updated to reflect the new\n> > constraints\n> >\n> > This series addresses issue 1) by changing the AgcMeanLuminance class\n> > and the associated helpers to regulate the exposure time base on the\n> > currently programmed max frame duration using a sensor-specific margin\n> > that is now reported by the CameraHelper class.\n>\n> I have a question about the above paragraph. The sensor-specific margin\n> is already known by the kernel's sensor driver and properly written\n> sensor drivers will update the exposure range reported by the query-ctrl\n> ioctl after changing the vblank control.\n>\n> So I think it would be better to rely on the kernel for this info\n> and refresh the cached exposure-control limits when changing the vblank\n> control rather then duplicating this info in the CameraHelper class ?\n\nWe discussed this with Niklas in v1\nhttps://patchwork.libcamera.org/patch/24695/#36372\n\nMy main concern (one more ioctl per frame apart) is that relying on\nthe kernel to update the control limits will introduce a delay of a\nvariable number of frames that correponds to the pipeline depth plus\nthe control delays.\n\nWhenever processing statistics for frame X and a new vblank value is\ncalculated to accommodate the newly computed exposure value (within\nthe FrameDurationLimits) we push the new value to DelayedControls\nwhich caches it in a queue. Whenever a new frame is started the\nqueue of controls is popped and controls get applied to the device. They\nwill take effect a few frames later (while the kernel interface\nupdates the control limits immediately if I'm not mistaken).\n\nThe IPA operates \"in the future\" and as once it has consumed request X\n(which will be realized on the device at frame X + depth) it shall\ncalculate new values from frame X+1. This means that once the IPA\nstate is updated to new limits, all calculations from that point on\nshould use the new state, and we can't rely on the device statee for this.\n\nWe currently do not properly track Request id/frame sequences, and we\nkind of ignore the pipeline depth as we keep consuming requests as\nsoon as they arrive, but it's not unresonable to assume that the\noverall delay, once we properly track the depth and compute parameters\nearly enough, can be around 6 or 7 frames.\n\n>\n> One could even calculate the exposure-margin by calculating the configured\n> frame-time (in a unit of lines) and then substracting the max-exposure\n> reported by the sensor from the frame-time-in-lines.\n\nStefan suggested that too.\n\nWhen you configure() you take the sensor's exposure time and the frame\nlength and the diff between the two is the margin. However there is no\nindication anywhere that the sensor driver has to initialize controls\nin this way and considering how slightly different sensor drivers are\nI wouldn't be on them being consistent.\n\n>\n> > As the AgcMeanLuminance algorithm tries to push the exposure time up\n> > to the maximum allowed frame duration, initializing the algorithm\n> > with the sensor's maximum capabilities results in a very low frame rate,\n> > unless the user specifies a FrameDurationLimits range in which the\n> > algorithm should operate. To avoid slowing the frame rate down too much\n> > compute a \"default\" frame rate at algorithm configuration time, were\n> > the canonical 30, 15 and 10 FPS values are tested and selected in order\n> > of preference.\n>\n> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> Does this algorithm also set / used to set the vblank control to\n> the maximum allowed frame duration? Or does it dynamically adjust\n> vblank when it needs more exposure \"range\" ?\n\nIt didn't, as it didn't calculate the frame duration.\n\nThe only IPA that use AgcMeanLuminance and handles VBLANK is the\nRkISP1 and as you can see in 09/10 the way it does that is to take\nthe exposure time calculated by AgcMeanLuminance::calculateNewEV()\nand use it as the desired frame duration. This is wrong as it breaks\nthe assumption we have a safe margin. It only doesn't break as the\ndriver will clamp exposure once VBLANK is applied, but I presume this\nalso causes a mis-alignment between what the algorithm think the\nexposure time is and what is actually applied to the device (and I\npresume this is what Stefan has reported causing oscillations when I\nnaively proposed to ignore the margin and let the driver adjust\nexposure).\n\nThis is the reason for 09/10: the AgcMeanLuminance now calculates the\nduration as well using the (exposure time it has computed + margin) and\nsends it back to the IPA that transforms it in a proper vblank value\nto be eventually set on the device.\n\n>\n> I've been thinking about adding support to dynamically increase\n> vblank when more exposure time is needed in the softISP AGC code,\n> but it would probably be better to first convert that over to\n> the shared AGC helpers. I know that Kieran is working on this.\n\nYes, I think that's a desirable goal.\n\nMy whole effort in this series has been around moving as much as\npossible of the logic to the shared algorithm so that all IPA using it\nwill get a consistent behaviour from \"free\".\n\nAlso patch 10/10 does what you want to do on Simple for Mali, so that\ncould serve as a reference.\n\n>\n> At least some UVC cameras have an auto-exposure priority (or is mode?)\n> v4l2 control which can be set to \"aperture priority\" to make them\n> automatically increase frameduration if necessary because of low\n> light conditions. I think we want something similar for libcamera's\n> AGC.\n\nYes, that's why we re-worked the AGC-related controls to have a\nseparate ExposureTimeMode and AnalogueGainMode compared to the single\n\"big button\" AeEnable.\n\nTo implement priorities application can fix one of the two and let the\nalgorithm adjust the other.\n\nThe exposure time will anyway always be clamped to the user selected\nFrameDurationLimits[] so that has to be properly adjusted too.\n\nThanks\n j\n>\n> Regards,\n>\n> Hans\n>\n>\n>\n>\n>\n>\n> >\n> > The last patch is not for inclusion as Mali support needs to be\n> > finalized upstream first, but shows how is it possible to handle\n> > frame duration limits and exposure with the new model (and I needed that\n> > patch for testing on Mali anyway).\n> >\n> > Tested with camshark on Mali-C55 and with a test scripts that sets a\n> > longer frame duration (66 msecs) and by validating that the exposure\n> > time is increased to match the new limits on RkISP1.\n> >\n> > RkISP1Agc agc.cpp:644 Divided up exposure time, analogue gain, quantization gain and digital gain are 33251.27us, 10.6667, 1.00018 and 6.49639\n> > 61.545331 (47.58 fps) cam0-stream0 seq: 000050 bytesused: 2073600/1036800\n> > kISP1Agc agc.cpp:644 Divided up exposure time, analogue gain, quantization gain and digital gain are 66578.16us, 10.6667, 1.00018 and 3.24451\n> > 61.566241 (47.82 fps) cam0-stream0 seq: 000051 bytesused: 2073600/1036800\n> >\n> > Signed-off-by: Jacopo Mondi \n> > ---\n> > Changes in v2:\n> > - Rework of v1 that moves the exposure limits computation to the\n> > AgcMeanLuminance exposure helpers\n> >\n> > ---\n> > Jacopo Mondi (9):\n> > ipa: camera_sensor_helper: Introduce exposureMargin()\n> > ipa: ipu3: Move CameraHelper to context\n> > ipa: libipa: agc: Make Agc::configure() set limits\n> > ipa: libipa: agc: Configure with frame duration\n> > ipa: libipa: agc: Initialize exposure with frame duration\n> > ipa: libipa: agc: Set exposure limits with frame duration\n> > ipa: libipa: agc: Initialize a sensible frame duration\n> > ipa: libipa: agc: Calculate frame duration in helper\n> > [DNI] ipa: mali: Handle FrameDurationLimits\n> >\n> > Kieran Bingham (1):\n> > ipa: mali-c55: Move CameraHelper to context\n> >\n> > src/ipa/ipu3/algorithms/agc.cpp | 32 +++++++---\n> > src/ipa/ipu3/ipa_context.cpp | 3 +\n> > src/ipa/ipu3/ipa_context.h | 3 +\n> > src/ipa/ipu3/ipu3.cpp | 22 +++----\n> > src/ipa/libipa/agc_mean_luminance.cpp | 73 ++++++++++++++++++---\n> > src/ipa/libipa/agc_mean_luminance.h | 18 +++++-\n> > src/ipa/libipa/camera_sensor_helper.cpp | 72 +++++++++++++++++++++\n> > src/ipa/libipa/camera_sensor_helper.h | 2 +\n> > src/ipa/libipa/exposure_mode_helper.cpp | 109 +++++++++++++++++++++++++++-----\n> > src/ipa/libipa/exposure_mode_helper.h | 16 ++++-\n> > src/ipa/mali-c55/algorithms/agc.cpp | 76 +++++++++++++++++++---\n> > src/ipa/mali-c55/ipa_context.cpp | 6 ++\n> > src/ipa/mali-c55/ipa_context.h | 12 ++++\n> > src/ipa/mali-c55/mali-c55.cpp | 44 ++++++-------\n> > src/ipa/rkisp1/algorithms/agc.cpp | 42 ++++++------\n> > 15 files changed, 430 insertions(+), 100 deletions(-)\n> > ---\n> > base-commit: 36f9cdcdb4a3c69e10b7da8a3de8354421cdcb56\n> > change-id: 20251017-exposure-limits-5c48252fe548\n> >\n> > Best regards,\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 69BE1BE080\n\tfor ;\n\tTue, 28 Oct 2025 11:42:13 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8A286607D3;\n\tTue, 28 Oct 2025 12:42:12 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 13791606DE\n\tfor ;\n\tTue, 28 Oct 2025 12:42:11 +0100 (CET)","from ideasonboard.com (mob-5-90-58-13.net.vodafone.it [5.90.58.13])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 4D86B11D5;\n\tTue, 28 Oct 2025 12:40:22 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"OFO2T4F+\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1761651622;\n\tbh=nvg9R66Zu87gM57ykcipA3mqS8TuHooEacJGQ+DUAgU=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=OFO2T4F+g5QiTps0HuMr6hR3FsP+EYh2BJmW8xKvelmiFBBkAYaQhJ0VoX80TUBog\n\tRr/u1SWDiDcg69gT7D0IvkANn3qjrUzPES0O9ndA0xzTByDRiiEmEQODsIKdo3jzgN\n\tmt/teW2BH7ctwR2N+L5PIAXsyNwJSlXwy9yF63Pw=","Date":"Tue, 28 Oct 2025 12:42:07 +0100","From":"Jacopo Mondi ","To":"Hans de Goede ","Cc":"Jacopo Mondi , Niklas =?utf-8?b?U8O2?=\n\t=?utf-8?q?derlund?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36516,"web_url":"https://patchwork.libcamera.org/comment/36516/","msgid":"","date":"2025-10-28T14:48:58","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":239,"url":"https://patchwork.libcamera.org/api/people/239/","name":"Hans de Goede","email":"hansg@kernel.org"},"content":"Hi Jacopo,\n\nOn 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> Hi Hans\n> \n> On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n>> Hi Jacopo,\n>>\n>> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n>>> By definition, the exposure time that can be programmed on a sensor is\n>>> limited by the frame duration and whenever the frame duration is\n>>> updated, the exposure time limits should be updated as well.\n>>>\n>>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n>>> libipa, where the exposure limits are computed at configure() time and\n>>> never updated.\n>>>\n>>> This has two implications:\n>>> 1) The AGC algorithms will always operate with an exposure time bound to\n>>> the frame duration programmed at startup time\n>>> 2) The Camera::controls() limits are not updated to reflect the new\n>>> constraints\n>>>\n>>> This series addresses issue 1) by changing the AgcMeanLuminance class\n>>> and the associated helpers to regulate the exposure time base on the\n>>> currently programmed max frame duration using a sensor-specific margin\n>>> that is now reported by the CameraHelper class.\n>>\n>> I have a question about the above paragraph. The sensor-specific margin\n>> is already known by the kernel's sensor driver and properly written\n>> sensor drivers will update the exposure range reported by the query-ctrl\n>> ioctl after changing the vblank control.\n>>\n>> So I think it would be better to rely on the kernel for this info\n>> and refresh the cached exposure-control limits when changing the vblank\n>> control rather then duplicating this info in the CameraHelper class ?\n> \n> We discussed this with Niklas in v1\n> https://patchwork.libcamera.org/patch/24695/#36372\n> \n> My main concern (one more ioctl per frame apart) is that relying on\n> the kernel to update the control limits will introduce a delay of a\n> variable number of frames that correponds to the pipeline depth plus\n> the control delays.\n> \n> Whenever processing statistics for frame X and a new vblank value is\n> calculated to accommodate the newly computed exposure value (within\n> the FrameDurationLimits) we push the new value to DelayedControls\n> which caches it in a queue. Whenever a new frame is started the\n> queue of controls is popped and controls get applied to the device. They\n> will take effect a few frames later (while the kernel interface\n> updates the control limits immediately if I'm not mistaken).\n\nRight I forgot about the delayed-controls stuff, not being able\nto get the new exposure limits from the kernel then makes sense.\n\nQuestion: is there code somewhere to guarantee that when delayed\ncontrols changes vblank it applies the vblank before the exposure?\n\nIIRC libcamera uses set-ext-ctrls to set them all at once, but\nmost (all?) sensor drivers do not implement this, so the kernel\njust applies them 1-by-1 internally. And if vblank gets increased\nonly after exposure then for one frame exposure will still get\nclamped based on the old vblank.\n\nThis problem actually sounds like a more likely cause for Stefan's\noscillation issue you mention below.\n\n>> One could even calculate the exposure-margin by calculating the configured\n>> frame-time (in a unit of lines) and then substracting the max-exposure\n>> reported by the sensor from the frame-time-in-lines.\n> \n> Stefan suggested that too.\n> \n> When you configure() you take the sensor's exposure time and the frame\n> length and the diff between the two is the margin. However there is no\n> indication anywhere that the sensor driver has to initialize controls\n> in this way and considering how slightly different sensor drivers are\n> I wouldn't be on them being consistent.\n\nHmm, I would say that we need to fix the sensor drivers then. But since\nwe need the camerahelper for the analog-gain bits I guess that having\nto add the exposure vs framelength margin is not a big problem.\n\n>>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n>>> to the maximum allowed frame duration, initializing the algorithm\n>>> with the sensor's maximum capabilities results in a very low frame rate,\n>>> unless the user specifies a FrameDurationLimits range in which the\n>>> algorithm should operate. To avoid slowing the frame rate down too much\n>>> compute a \"default\" frame rate at algorithm configuration time, were\n>>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n>>> of preference.\n>>\n>> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n>> Does this algorithm also set / used to set the vblank control to\n>> the maximum allowed frame duration? Or does it dynamically adjust\n>> vblank when it needs more exposure \"range\" ?\n> \n> It didn't, as it didn't calculate the frame duration.\n> \n> The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> RkISP1 and as you can see in 09/10 the way it does that is to take\n> the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> and use it as the desired frame duration. This is wrong as it breaks\n> the assumption we have a safe margin. It only doesn't break as the\n> driver will clamp exposure once VBLANK is applied, but I presume this\n> also causes a mis-alignment between what the algorithm think the\n> exposure time is and what is actually applied to the device (and I\n> presume this is what Stefan has reported causing oscillations when I\n> naively proposed to ignore the margin and let the driver adjust\n> exposure).\n\nSee above, applying exposure before an increased vblank will lead\nto a much bigger clamp which is a more likely cause of oscillation.\n\n> This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> duration as well using the (exposure time it has computed + margin) and\n> sends it back to the IPA that transforms it in a proper vblank value\n> to be eventually set on the device.\n> \n>>\n>> I've been thinking about adding support to dynamically increase\n>> vblank when more exposure time is needed in the softISP AGC code,\n>> but it would probably be better to first convert that over to\n>> the shared AGC helpers. I know that Kieran is working on this.\n> \n> Yes, I think that's a desirable goal.\n> \n> My whole effort in this series has been around moving as much as\n> possible of the logic to the shared algorithm so that all IPA using it\n> will get a consistent behaviour from \"free\".\n\nAck.\n\nThank you for taking the time to answer all my questions.\n\nRegards,\n\nHans","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0D9F6BE080\n\tfor ;\n\tTue, 28 Oct 2025 14:49:06 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2A9CE607AE;\n\tTue, 28 Oct 2025 15:49:05 +0100 (CET)","from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B840D606DE\n\tfor ;\n\tTue, 28 Oct 2025 15:49:03 +0100 (CET)","from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58])\n\tby sea.source.kernel.org (Postfix) with ESMTP id 2E3CE45F05;\n\tTue, 28 Oct 2025 14:49:02 +0000 (UTC)","by smtp.kernel.org (Postfix) with ESMTPSA id D9B1EC4CEE7;\n\tTue, 28 Oct 2025 14:49:00 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=kernel.org header.i=@kernel.org\n\theader.b=\"Ul+tVRoK\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;\n\ts=k20201202; t=1761662942;\n\tbh=Q+8aZ/80Gg4AonOXSyoWfIVRIw5njP0/8JRIXZYervA=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=Ul+tVRoK0pXmnsQz6Fy5+KWRMQKL3JBjNTSavz6SsM0ksnLD9r7IxdJVa+5qkC5e0\n\tg0LORDRj16aT82vXzAZ1HsGRpdGOYKFjbmBzu/Akbz3q8VE5zJqk3lip1W/4uMeT6d\n\t4bxv8MA1H4gxJypKl9SvTyMPnbkZLmCrfFXbwEatRrYQDimR6MeSERNvC0QF0z03RM\n\teoIw1rFcogXdE/lMIQ1yu1cgLcsMH91n8HrAGmnaYOS8yGERGacpKF/zPUVMfmW1KD\n\tWeXsFrvdZtP9rstQHDlj8kb5YWRaaxok6lNdqSzcva8XoZs5fbaQYNbjQh//gPc5+V\n\t2sjdlUJCi4OJA==","Message-ID":"","Date":"Tue, 28 Oct 2025 15:48:58 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Jacopo Mondi ","Cc":"=?utf-8?q?Niklas_S=C3=B6derlund?= ,\n\tRobert Mader , \n\tlibcamera-devel@lists.libcamera.org, Kieran Bingham\n\t","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t","From":"Hans de Goede ","Content-Language":"en-US, nl","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"7bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36517,"web_url":"https://patchwork.libcamera.org/comment/36517/","msgid":"","date":"2025-10-28T14:58:15","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Hans,\n\nOn Tue, 28 Oct 2025 at 14:49, Hans de Goede wrote:\n>\n> Hi Jacopo,\n>\n> On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > Hi Hans\n> >\n> > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> >> Hi Jacopo,\n> >>\n> >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> >>> By definition, the exposure time that can be programmed on a sensor is\n> >>> limited by the frame duration and whenever the frame duration is\n> >>> updated, the exposure time limits should be updated as well.\n> >>>\n> >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> >>> libipa, where the exposure limits are computed at configure() time and\n> >>> never updated.\n> >>>\n> >>> This has two implications:\n> >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> >>> the frame duration programmed at startup time\n> >>> 2) The Camera::controls() limits are not updated to reflect the new\n> >>> constraints\n> >>>\n> >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> >>> and the associated helpers to regulate the exposure time base on the\n> >>> currently programmed max frame duration using a sensor-specific margin\n> >>> that is now reported by the CameraHelper class.\n> >>\n> >> I have a question about the above paragraph. The sensor-specific margin\n> >> is already known by the kernel's sensor driver and properly written\n> >> sensor drivers will update the exposure range reported by the query-ctrl\n> >> ioctl after changing the vblank control.\n> >>\n> >> So I think it would be better to rely on the kernel for this info\n> >> and refresh the cached exposure-control limits when changing the vblank\n> >> control rather then duplicating this info in the CameraHelper class ?\n> >\n> > We discussed this with Niklas in v1\n> > https://patchwork.libcamera.org/patch/24695/#36372\n> >\n> > My main concern (one more ioctl per frame apart) is that relying on\n> > the kernel to update the control limits will introduce a delay of a\n> > variable number of frames that correponds to the pipeline depth plus\n> > the control delays.\n> >\n> > Whenever processing statistics for frame X and a new vblank value is\n> > calculated to accommodate the newly computed exposure value (within\n> > the FrameDurationLimits) we push the new value to DelayedControls\n> > which caches it in a queue. Whenever a new frame is started the\n> > queue of controls is popped and controls get applied to the device. They\n> > will take effect a few frames later (while the kernel interface\n> > updates the control limits immediately if I'm not mistaken).\n>\n> Right I forgot about the delayed-controls stuff, not being able\n> to get the new exposure limits from the kernel then makes sense.\n>\n> Question: is there code somewhere to guarantee that when delayed\n> controls changes vblank it applies the vblank before the exposure?\n>\n> IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> most (all?) sensor drivers do not implement this, so the kernel\n> just applies them 1-by-1 internally. And if vblank gets increased\n> only after exposure then for one frame exposure will still get\n> clamped based on the old vblank.\n>\n> This problem actually sounds like a more likely cause for Stefan's\n> oscillation issue you mention below.\n\nWhen constructing the DelayedControls object, we mark the VBLANK\ncontrol as \"priority write\". This ensures that VBLANK is written\nahead of any (and particularly EXPOSURE) controls for this very\nreason.\n\nRegards,\nNaush\n\n\n>\n> >> One could even calculate the exposure-margin by calculating the configured\n> >> frame-time (in a unit of lines) and then substracting the max-exposure\n> >> reported by the sensor from the frame-time-in-lines.\n> >\n> > Stefan suggested that too.\n> >\n> > When you configure() you take the sensor's exposure time and the frame\n> > length and the diff between the two is the margin. However there is no\n> > indication anywhere that the sensor driver has to initialize controls\n> > in this way and considering how slightly different sensor drivers are\n> > I wouldn't be on them being consistent.\n>\n> Hmm, I would say that we need to fix the sensor drivers then. But since\n> we need the camerahelper for the analog-gain bits I guess that having\n> to add the exposure vs framelength margin is not a big problem.\n>\n> >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> >>> to the maximum allowed frame duration, initializing the algorithm\n> >>> with the sensor's maximum capabilities results in a very low frame rate,\n> >>> unless the user specifies a FrameDurationLimits range in which the\n> >>> algorithm should operate. To avoid slowing the frame rate down too much\n> >>> compute a \"default\" frame rate at algorithm configuration time, were\n> >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> >>> of preference.\n> >>\n> >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> >> Does this algorithm also set / used to set the vblank control to\n> >> the maximum allowed frame duration? Or does it dynamically adjust\n> >> vblank when it needs more exposure \"range\" ?\n> >\n> > It didn't, as it didn't calculate the frame duration.\n> >\n> > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > and use it as the desired frame duration. This is wrong as it breaks\n> > the assumption we have a safe margin. It only doesn't break as the\n> > driver will clamp exposure once VBLANK is applied, but I presume this\n> > also causes a mis-alignment between what the algorithm think the\n> > exposure time is and what is actually applied to the device (and I\n> > presume this is what Stefan has reported causing oscillations when I\n> > naively proposed to ignore the margin and let the driver adjust\n> > exposure).\n>\n> See above, applying exposure before an increased vblank will lead\n> to a much bigger clamp which is a more likely cause of oscillation.\n>\n> > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > duration as well using the (exposure time it has computed + margin) and\n> > sends it back to the IPA that transforms it in a proper vblank value\n> > to be eventually set on the device.\n> >\n> >>\n> >> I've been thinking about adding support to dynamically increase\n> >> vblank when more exposure time is needed in the softISP AGC code,\n> >> but it would probably be better to first convert that over to\n> >> the shared AGC helpers. I know that Kieran is working on this.\n> >\n> > Yes, I think that's a desirable goal.\n> >\n> > My whole effort in this series has been around moving as much as\n> > possible of the logic to the shared algorithm so that all IPA using it\n> > will get a consistent behaviour from \"free\".\n>\n> Ack.\n>\n> Thank you for taking the time to answer all my questions.\n>\n> Regards,\n>\n> Hans\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 57710C3259\n\tfor ;\n\tTue, 28 Oct 2025 14:58:56 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 78A43607BC;\n\tTue, 28 Oct 2025 15:58:55 +0100 (CET)","from mail-vs1-xe2e.google.com (mail-vs1-xe2e.google.com\n\t[IPv6:2607:f8b0:4864:20::e2e])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C8EE0606DE\n\tfor ;\n\tTue, 28 Oct 2025 15:58:53 +0100 (CET)","by mail-vs1-xe2e.google.com with SMTP id\n\tada2fe7eead31-5db2281bb3aso409568137.3\n\tfor ;\n\tTue, 28 Oct 2025 07:58:53 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"Tj8+YcjL\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1761663532; x=1762268332;\n\tdarn=lists.libcamera.org; \n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:from:to:cc:subject:date:message-id:reply-to;\n\tbh=QvkKtYxP6xe+TY6flF3L773TTlNVfB5oo1CSBC0pBTE=;\n\tb=Tj8+YcjLWhF6MxjvrnaI+DFJ4oWmT7qONpiycMFxBerK/bx/oQRJV2lptVRDX/ZefR\n\tgavezqsHZiUavUZEZtmHB0VMWyrvDUfCPAw6kNXwrMZkQqFjp4uDQS15GRvnwJy++kU7\n\tzE2rT51+cgCdCvTfAEHycH5oNEUFUWn3mgC7Zpw5PQYtzCwcQCccYCmZBnqoP8wL5mzC\n\tOmgRoW9r/HacVrgV/3uE2PvgSfPqHgtQViR17IMoJuqJbku41IVauQHqCRQbeJhJ62cy\n\taOiBv1DxcvBaN0j1uAQXHoqcLt4bfTvICSpr7zVTicb3GrV14xZj74olrvwpsCBdewot\n\tKfKA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1761663532; x=1762268332;\n\th=cc:to:subject:message-id:date:from:in-reply-to:references\n\t:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id\n\t:reply-to;\n\tbh=QvkKtYxP6xe+TY6flF3L773TTlNVfB5oo1CSBC0pBTE=;\n\tb=d0ZRsmm3dFwK/qOsfRNp2rdOHtfIuO3ltW7TPv5Mayb/8MplZiGpCjhkvZ+IOCsDul\n\tWNiCocifUI4Hey5UjKN4Z7jSRoAs0AdP5eo3bcOt9uA4x8gpG/hktDuwBnmJ+7ELPx4s\n\teum1iZrmwcoMVGZN5d6dhWpSYq0XDfM6PfnLjwpVi2KnyQo4uNnndFZessOAPHmZEDK6\n\tETJDvAHI5yInbVfjtgker7J8NxRd6QmwPiGhX5K8IkHwDAkv8s6asrLKvlQBXv4lMcBP\n\tZVMi7MbHU0Ny8cEN2b/JzgDlCzwxcDt/Bdxf/P3orV9MRmBxP2UrQU3z/UKtwUHgooBs\n\t4X9w==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCUO2fSPI2hfzORHjoyY2vPs3YHRa5AhmCbLpnqBaLq0nQIvokuKO0VT9i0LgTdWBXdjIBFDToTx1YyU/v0CQ+k=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YwzrLfvUdAE3fVDSzNZi+w5YWM/MpI91CxKC5k/IRwODAMTFgkm\n\tjdn+ZjWe81VjPiNeM1bnApPnEMbQU33TnQ+ADX3Wd3yehXkCtixFVypD6fQOQjYil/uDOqRqZCF\n\te3vp0FxkT0zTRsg/izYNtaufpt+nN+OGTez2vZSUv6w==","X-Gm-Gg":"ASbGnctYp3/DqGzEVsdUbXmzZWkkt/6gJiO1TuS6narxxZswReQ+tO3U6XAettzBTSO\n\tY9e9/QTQZfoZKXkCsTMu8AO+vn7KYiOj8a3jM+yQg5ZJjxTZhvsr/kY759Vrb0y9L8Rnb5+wszI\n\t5bl7FstVIXiNL56FGptmzxX1wC31/XsrDC+47UJAFVBzgp045wsaSRv1WL5wE/IT0s9SKe2jmqM\n\tWZlMl4xrgI4i6mem55pniQeYxtgUXv2nBPb5cccqWxH9EOws56hbUGiS4Sr","X-Google-Smtp-Source":"AGHT+IEwNFtTvB0Rg2IehweQPZdjLHjRHwacy+7uvJ1bLPmzcylyXIazE14FJ0r4o93JtOgZklba8zswByGb6iV/B/M=","X-Received":"by 2002:a05:6102:6a95:b0:5d6:1ae:1a1d with SMTP id\n\tada2fe7eead31-5db7cc2d2e9mr668930137.8.1761663532352; Tue, 28 Oct 2025\n\t07:58:52 -0700 (PDT)","MIME-Version":"1.0","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t","In-Reply-To":"","From":"Naushir Patuck ","Date":"Tue, 28 Oct 2025 14:58:15 +0000","X-Gm-Features":"AWmQ_bk-8k2d9GD6iOMoXWxXKB8rR0bIWFK5J8rTzJ4qellMgmkZNPFtAeteuC8","Message-ID":"","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Hans de Goede ","Cc":"Jacopo Mondi , =?utf-8?q?Niklas_S?=\n\t=?utf-8?b?w7ZkZXJsdW5k?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org, \n\tKieran Bingham ","Content-Type":"text/plain; charset=\"UTF-8\"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36520,"web_url":"https://patchwork.libcamera.org/comment/36520/","msgid":"","date":"2025-10-28T15:27:42","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Hans\n\nOn Tue, Oct 28, 2025 at 03:48:58PM +0100, Hans de Goede wrote:\n> Hi Jacopo,\n>\n> On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > Hi Hans\n> >\n> > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> >> Hi Jacopo,\n> >>\n> >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> >>> By definition, the exposure time that can be programmed on a sensor is\n> >>> limited by the frame duration and whenever the frame duration is\n> >>> updated, the exposure time limits should be updated as well.\n> >>>\n> >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> >>> libipa, where the exposure limits are computed at configure() time and\n> >>> never updated.\n> >>>\n> >>> This has two implications:\n> >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> >>> the frame duration programmed at startup time\n> >>> 2) The Camera::controls() limits are not updated to reflect the new\n> >>> constraints\n> >>>\n> >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> >>> and the associated helpers to regulate the exposure time base on the\n> >>> currently programmed max frame duration using a sensor-specific margin\n> >>> that is now reported by the CameraHelper class.\n> >>\n> >> I have a question about the above paragraph. The sensor-specific margin\n> >> is already known by the kernel's sensor driver and properly written\n> >> sensor drivers will update the exposure range reported by the query-ctrl\n> >> ioctl after changing the vblank control.\n> >>\n> >> So I think it would be better to rely on the kernel for this info\n> >> and refresh the cached exposure-control limits when changing the vblank\n> >> control rather then duplicating this info in the CameraHelper class ?\n> >\n> > We discussed this with Niklas in v1\n> > https://patchwork.libcamera.org/patch/24695/#36372\n> >\n> > My main concern (one more ioctl per frame apart) is that relying on\n> > the kernel to update the control limits will introduce a delay of a\n> > variable number of frames that correponds to the pipeline depth plus\n> > the control delays.\n> >\n> > Whenever processing statistics for frame X and a new vblank value is\n> > calculated to accommodate the newly computed exposure value (within\n> > the FrameDurationLimits) we push the new value to DelayedControls\n> > which caches it in a queue. Whenever a new frame is started the\n> > queue of controls is popped and controls get applied to the device. They\n> > will take effect a few frames later (while the kernel interface\n> > updates the control limits immediately if I'm not mistaken).\n>\n> Right I forgot about the delayed-controls stuff, not being able\n> to get the new exposure limits from the kernel then makes sense.\n>\n> Question: is there code somewhere to guarantee that when delayed\n> controls changes vblank it applies the vblank before the exposure?\n\nYes, DelayedControls has a \"priority\" parameters to make sure VBLANK\nis updated before EXPOSURE\nhttps://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/delayed_controls.cpp#n50\nhttps://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/delayed_controls.cpp#n232\n\n>\n> IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> most (all?) sensor drivers do not implement this, so the kernel\n> just applies them 1-by-1 internally. And if vblank gets increased\n> only after exposure then for one frame exposure will still get\n> clamped based on the old vblank.\n>\n> This problem actually sounds like a more likely cause for Stefan's\n> oscillation issue you mention below.\n\nYou're right, thankfully RPi handled this when they first implemented\nDelayedControls\n\n>\n> >> One could even calculate the exposure-margin by calculating the configured\n> >> frame-time (in a unit of lines) and then substracting the max-exposure\n> >> reported by the sensor from the frame-time-in-lines.\n> >\n> > Stefan suggested that too.\n> >\n> > When you configure() you take the sensor's exposure time and the frame\n> > length and the diff between the two is the margin. However there is no\n> > indication anywhere that the sensor driver has to initialize controls\n> > in this way and considering how slightly different sensor drivers are\n> > I wouldn't be on them being consistent.\n>\n> Hmm, I would say that we need to fix the sensor drivers then. But since\n> we need the camerahelper for the analog-gain bits I guess that having\n> to add the exposure vs framelength margin is not a big problem.\n>\n> >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> >>> to the maximum allowed frame duration, initializing the algorithm\n> >>> with the sensor's maximum capabilities results in a very low frame rate,\n> >>> unless the user specifies a FrameDurationLimits range in which the\n> >>> algorithm should operate. To avoid slowing the frame rate down too much\n> >>> compute a \"default\" frame rate at algorithm configuration time, were\n> >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> >>> of preference.\n> >>\n> >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> >> Does this algorithm also set / used to set the vblank control to\n> >> the maximum allowed frame duration? Or does it dynamically adjust\n> >> vblank when it needs more exposure \"range\" ?\n> >\n> > It didn't, as it didn't calculate the frame duration.\n> >\n> > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > and use it as the desired frame duration. This is wrong as it breaks\n> > the assumption we have a safe margin. It only doesn't break as the\n> > driver will clamp exposure once VBLANK is applied, but I presume this\n> > also causes a mis-alignment between what the algorithm think the\n> > exposure time is and what is actually applied to the device (and I\n> > presume this is what Stefan has reported causing oscillations when I\n> > naively proposed to ignore the margin and let the driver adjust\n> > exposure).\n>\n> See above, applying exposure before an increased vblank will lead\n> to a much bigger clamp which is a more likely cause of oscillation.\n>\n> > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > duration as well using the (exposure time it has computed + margin) and\n> > sends it back to the IPA that transforms it in a proper vblank value\n> > to be eventually set on the device.\n> >\n> >>\n> >> I've been thinking about adding support to dynamically increase\n> >> vblank when more exposure time is needed in the softISP AGC code,\n> >> but it would probably be better to first convert that over to\n> >> the shared AGC helpers. I know that Kieran is working on this.\n> >\n> > Yes, I think that's a desirable goal.\n> >\n> > My whole effort in this series has been around moving as much as\n> > possible of the logic to the shared algorithm so that all IPA using it\n> > will get a consistent behaviour from \"free\".\n>\n> Ack.\n>\n> Thank you for taking the time to answer all my questions.\n>\n\nNo worries at all, this is very helpful as some of my understanding of\nthe issue has to be validated, and that's exactly what reviews are for ;)\n\nThanks\n j\n\n> Regards,\n>\n> Hans\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0E497BE080\n\tfor ;\n\tTue, 28 Oct 2025 15:27:49 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4F5BB607E0;\n\tTue, 28 Oct 2025 16:27:48 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8143E606DE\n\tfor ;\n\tTue, 28 Oct 2025 16:27:46 +0100 (CET)","from ideasonboard.com (mob-5-90-58-13.net.vodafone.it [5.90.58.13])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id AC1C716CD;\n\tTue, 28 Oct 2025 16:25:57 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"ZZnRZ5SA\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1761665157;\n\tbh=qlklLeC4j/YFKbZ1rtJtszzG1fpnwlKEzpe1VYqfzYI=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=ZZnRZ5SAIlwBb3m+FyhPt7WMCXBZF7OeEC9LaHuN+F3gZ4vGczFcP056vqUZEYzF1\n\tI0CexIhy7etggmt5x7mxJVXPzqW9x532fYl17JYwvXGPnKXxNn2AwHUYeqw3/b/b3u\n\tFU2jJyVOJ+IKvUpJH81+Xznwvq6XLONzfYcjgvFg=","Date":"Tue, 28 Oct 2025 16:27:42 +0100","From":"Jacopo Mondi ","To":"Hans de Goede ","Cc":"Jacopo Mondi , Niklas =?utf-8?b?U8O2?=\n\t=?utf-8?q?derlund?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36521,"web_url":"https://patchwork.libcamera.org/comment/36521/","msgid":"<20251028153842.GC365372@ragnatech.se>","date":"2025-10-28T15:38:42","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hello,\n\nOn 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> Hi Jacopo,\n> \n> On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > Hi Hans\n> > \n> > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> >> Hi Jacopo,\n> >>\n> >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> >>> By definition, the exposure time that can be programmed on a sensor is\n> >>> limited by the frame duration and whenever the frame duration is\n> >>> updated, the exposure time limits should be updated as well.\n> >>>\n> >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> >>> libipa, where the exposure limits are computed at configure() time and\n> >>> never updated.\n> >>>\n> >>> This has two implications:\n> >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> >>> the frame duration programmed at startup time\n> >>> 2) The Camera::controls() limits are not updated to reflect the new\n> >>> constraints\n> >>>\n> >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> >>> and the associated helpers to regulate the exposure time base on the\n> >>> currently programmed max frame duration using a sensor-specific margin\n> >>> that is now reported by the CameraHelper class.\n> >>\n> >> I have a question about the above paragraph. The sensor-specific margin\n> >> is already known by the kernel's sensor driver and properly written\n> >> sensor drivers will update the exposure range reported by the query-ctrl\n> >> ioctl after changing the vblank control.\n> >>\n> >> So I think it would be better to rely on the kernel for this info\n> >> and refresh the cached exposure-control limits when changing the vblank\n> >> control rather then duplicating this info in the CameraHelper class ?\n> > \n> > We discussed this with Niklas in v1\n> > https://patchwork.libcamera.org/patch/24695/#36372\n> > \n> > My main concern (one more ioctl per frame apart) is that relying on\n> > the kernel to update the control limits will introduce a delay of a\n> > variable number of frames that correponds to the pipeline depth plus\n> > the control delays.\n> > \n> > Whenever processing statistics for frame X and a new vblank value is\n> > calculated to accommodate the newly computed exposure value (within\n> > the FrameDurationLimits) we push the new value to DelayedControls\n> > which caches it in a queue. Whenever a new frame is started the\n> > queue of controls is popped and controls get applied to the device. They\n> > will take effect a few frames later (while the kernel interface\n> > updates the control limits immediately if I'm not mistaken).\n> \n> Right I forgot about the delayed-controls stuff, not being able\n> to get the new exposure limits from the kernel then makes sense.\n\nI too understand why it's easier to not consult the kernel for the new \nlimits. I have no real strong feeling that the correct solution is to \nhave this limit calculation done in the kernel or user-space. What I do \nfeel somewhat strongly about is if we do the limit calculation in \nuser-space we should never consult the kernel for the limits.\n\nThis just mixes two ways of doing the same thing and is very confusing \nwhen reading the code. So if doing this in user-space is the way forward \nI think we should drop the reading of min and max of V4L2_CID_EXPOSURE \nin IPARkISP1::configure() and instead use this method there too.\n\nI also feel, a little less strongly as it's a big work item, that it's \nreally wrong to duplicate the calculation both in kernel and user-space.\n\nAs this IMHO exposes a weakness in the V4L2 API we should think about \nhow we want the V4L2_CID_EXPOSURE control to report min/max values. Do \nwe really want it to report its min/max values based on the current \nV4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL \nfor each frame.\n\nCan we simplify the control to report the min and max values the sensor \ncan do provided the VBLANK setting allows it, and just reject any value \nat set time that do not conform to the current VBLANK setting?\n\nIf so we can stop doing stuff in sensor drivers such as in IMX219 to \nallow for this control dependency in each driver.\n\n static int imx219_set_ctrl(struct v4l2_ctrl *ctrl)\n {\n ...\n\n if (ctrl->id == V4L2_CID_VBLANK) {\n \tint exposure_max, exposure_def;\n\n \t/* Update max exposure while meeting expected vblanking */\n \texposure_max = format->height + ctrl->val - 4;\n \texposure_def = (exposure_max < IMX219_EXPOSURE_DEFAULT) ?\n \t\texposure_max : IMX219_EXPOSURE_DEFAULT;\n \t__v4l2_ctrl_modify_range(imx219->exposure,\n \t\t\t\t imx219->exposure->minimum,\n \t\t\t\t exposure_max, imx219->exposure->step,\n \t\t\t\t exposure_def);\n }\n\n ...\n }\n\n>\n> Question: is there code somewhere to guarantee that when delayed\n> controls changes vblank it applies the vblank before the exposure?\n> \n> IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> most (all?) sensor drivers do not implement this, so the kernel\n> just applies them 1-by-1 internally. And if vblank gets increased\n> only after exposure then for one frame exposure will still get\n> clamped based on the old vblank.\n> \n> This problem actually sounds like a more likely cause for Stefan's\n> oscillation issue you mention below.\n> \n> >> One could even calculate the exposure-margin by calculating the configured\n> >> frame-time (in a unit of lines) and then substracting the max-exposure\n> >> reported by the sensor from the frame-time-in-lines.\n> > \n> > Stefan suggested that too.\n> > \n> > When you configure() you take the sensor's exposure time and the frame\n> > length and the diff between the two is the margin. However there is no\n> > indication anywhere that the sensor driver has to initialize controls\n> > in this way and considering how slightly different sensor drivers are\n> > I wouldn't be on them being consistent.\n> \n> Hmm, I would say that we need to fix the sensor drivers then. But since\n> we need the camerahelper for the analog-gain bits I guess that having\n> to add the exposure vs framelength margin is not a big problem.\n> \n> >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> >>> to the maximum allowed frame duration, initializing the algorithm\n> >>> with the sensor's maximum capabilities results in a very low frame rate,\n> >>> unless the user specifies a FrameDurationLimits range in which the\n> >>> algorithm should operate. To avoid slowing the frame rate down too much\n> >>> compute a \"default\" frame rate at algorithm configuration time, were\n> >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> >>> of preference.\n> >>\n> >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> >> Does this algorithm also set / used to set the vblank control to\n> >> the maximum allowed frame duration? Or does it dynamically adjust\n> >> vblank when it needs more exposure \"range\" ?\n> > \n> > It didn't, as it didn't calculate the frame duration.\n> > \n> > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > and use it as the desired frame duration. This is wrong as it breaks\n> > the assumption we have a safe margin. It only doesn't break as the\n> > driver will clamp exposure once VBLANK is applied, but I presume this\n> > also causes a mis-alignment between what the algorithm think the\n> > exposure time is and what is actually applied to the device (and I\n> > presume this is what Stefan has reported causing oscillations when I\n> > naively proposed to ignore the margin and let the driver adjust\n> > exposure).\n> \n> See above, applying exposure before an increased vblank will lead\n> to a much bigger clamp which is a more likely cause of oscillation.\n> \n> > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > duration as well using the (exposure time it has computed + margin) and\n> > sends it back to the IPA that transforms it in a proper vblank value\n> > to be eventually set on the device.\n> > \n> >>\n> >> I've been thinking about adding support to dynamically increase\n> >> vblank when more exposure time is needed in the softISP AGC code,\n> >> but it would probably be better to first convert that over to\n> >> the shared AGC helpers. I know that Kieran is working on this.\n> > \n> > Yes, I think that's a desirable goal.\n> > \n> > My whole effort in this series has been around moving as much as\n> > possible of the logic to the shared algorithm so that all IPA using it\n> > will get a consistent behaviour from \"free\".\n> \n> Ack.\n> \n> Thank you for taking the time to answer all my questions.\n> \n> Regards,\n> \n> Hans\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 6EC9AC3259\n\tfor ;\n\tTue, 28 Oct 2025 15:38:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B2347607D3;\n\tTue, 28 Oct 2025 16:38:48 +0100 (CET)","from fout-a6-smtp.messagingengine.com\n\t(fout-a6-smtp.messagingengine.com [103.168.172.149])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CE38B606DE\n\tfor ;\n\tTue, 28 Oct 2025 16:38:46 +0100 (CET)","from phl-compute-05.internal (phl-compute-05.internal\n\t[10.202.2.45])\n\tby mailfout.phl.internal (Postfix) with ESMTP id 01E76EC0569;\n\tTue, 28 Oct 2025 11:38:46 -0400 (EDT)","from phl-mailfrontend-01 ([10.202.2.162])\n\tby phl-compute-05.internal (MEProxy); Tue, 28 Oct 2025 11:38:46 -0400","by mail.messagingengine.com (Postfix) with ESMTPA; Tue,\n\t28 Oct 2025 11:38:44 -0400 (EDT)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=ragnatech.se header.i=@ragnatech.se\n\theader.b=\"avW3dUlc\"; dkim=pass (2048-bit key;\n\tunprotected) header.d=messagingengine.com\n\theader.i=@messagingengine.com header.b=\"Q4Bv3FFg\"; \n\tdkim-atps=neutral","DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=ragnatech.se; h=\n\tcc:cc:content-transfer-encoding:content-type:content-type:date\n\t:date:from:from:in-reply-to:in-reply-to:message-id:mime-version\n\t:references:reply-to:subject:subject:to:to; s=fm2; t=1761665925;\n\tx=1761752325; bh=QSjT2xNn4/1/I0hZDqXrgrMY1x7IQfRcblvqVByW//U=; b=\n\tavW3dUlcDrN8BUSs1QD+HFXLoBBtAhSKKhcNscL4zigPi9pryzRozKKW7wZnIKAQ\n\tJtGUWOXI2VwucrSt+8pDkusvoioPpAMaY0nya5Ht/ff4FMx8flmv2RluLVkOapw0\n\tmRL2/jkM+2Y2wLCwKfXv9TZYi/bG/qrxOLtu0FbqeyZJTugE7AuFVif/y0QOcx4I\n\trSm0BQTMto1bd9aMqwcLvJkpwf2scRuzPVIlR6dZlulX3mVFVDP1+UNgq23DFUAi\n\tZIIbzq75yR7FbV0wF34EDQAPTThOMqOfja4V8eRNvPGlf2iHg8eb4k8hL6iHTs46\n\tiMFwYA7cISHA52kfvT9PJg==","v=1; a=rsa-sha256; c=relaxed/relaxed; d=\n\tmessagingengine.com; h=cc:cc:content-transfer-encoding\n\t:content-type:content-type:date:date:feedback-id:feedback-id\n\t:from:from:in-reply-to:in-reply-to:message-id:mime-version\n\t:references:reply-to:subject:subject:to:to:x-me-proxy\n\t:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1761665925; x=\n\t1761752325; bh=QSjT2xNn4/1/I0hZDqXrgrMY1x7IQfRcblvqVByW//U=; b=Q\n\t4Bv3FFgTI4JpnTLXf8KqfP4G/MB/U2WOw/tQCC2Ikk/bx8hR/D1yHbTXyJMX+57v\n\t0EXJk0rsqnfYcjqw1o2x9oAYGLZ2YoWKx1bWiv9sOG9Ttv1FNI/j5zHAPqGQCbMW\n\tgxjN0Tt6k1ZkKr2sqA2/M1pUztnvdHE2htT272O6PCDYhIM7iB/K+0qE8aHTt6mv\n\tIyD8azvW4NNT99RAZr8w638mmG5OOD6CbvTEZ6MrC7eXTWfv1DtZaFHmu9IPE14f\n\t73Vqgf9K4OM3r2RyKVg66EzJT75CkXSSrhjffKtoSLvB8Np0LwDoyULUOSNwz1LD\n\taVicqvMJYUN6fNPgjezuQ=="],"X-ME-Sender":"\n\t","X-ME-Received":"","X-ME-Proxy-Cause":"gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdduieduvdefucetufdoteggodetrf\n\tdotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu\n\trghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf\n\tgurhepfffhvfevuffkfhggtggugfgjsehtkeertddttdejnecuhfhrohhmpefpihhklhgr\n\tshcuufpnuggvrhhluhhnugcuoehnihhklhgrshdrshhouggvrhhluhhnugesrhgrghhnrg\n\thtvggthhdrshgvqeenucggtffrrghtthgvrhhnpeehheekteejhfdttedthffgvdegieeu\n\tjeeiheetveetteeftdehledtffffgeejheenucffohhmrghinheplhhisggtrghmvghrrg\n\tdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm\n\tpehnihhklhgrshdrshhouggvrhhluhhnugesrhgrghhnrghtvggthhdrshgvpdhnsggprh\n\tgtphhtthhopeehpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehhrghnshhgsehk\n\tvghrnhgvlhdrohhrghdprhgtphhtthhopehjrggtohhpohdrmhhonhguihesihguvggrsh\n\thonhgsohgrrhgurdgtohhmpdhrtghpthhtoheprhhosggvrhhtrdhmrgguvghrsegtohhl\n\tlhgrsghorhgrrdgtohhmpdhrtghpthhtoheplhhisggtrghmvghrrgdquggvvhgvlheslh\n\thishhtshdrlhhisggtrghmvghrrgdrohhrghdprhgtphhtthhopehkihgvrhgrnhdrsghi\n\tnhhghhgrmhesihguvggrshhonhgsohgrrhgurdgtohhm","X-ME-Proxy":"\n\t\n\t\n\t\n\t","Feedback-ID":"i80c9496c:Fastmail","Date":"Tue, 28 Oct 2025 16:38:42 +0100","From":"Niklas =?utf-8?q?S=C3=B6derlund?= ","To":"Hans de Goede ","Cc":"Jacopo Mondi ,\n\tRobert Mader ,\n\tlibcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"<20251028153842.GC365372@ragnatech.se>","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36607,"web_url":"https://patchwork.libcamera.org/comment/36607/","msgid":"<54xd52swj5ftm6sbv6habzmjzojbqle6u4kvt442mdvz3ay74u@c4tkaqvly5pf>","date":"2025-11-02T13:36:36","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Niklas,\n\nOn Tue, Oct 28, 2025 at 04:38:42PM +0100, Niklas Söderlund wrote:\n> Hello,\n>\n> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> > Hi Jacopo,\n> >\n> > On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > > Hi Hans\n> > >\n> > > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> > >> Hi Jacopo,\n> > >>\n> > >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> > >>> By definition, the exposure time that can be programmed on a sensor is\n> > >>> limited by the frame duration and whenever the frame duration is\n> > >>> updated, the exposure time limits should be updated as well.\n> > >>>\n> > >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> > >>> libipa, where the exposure limits are computed at configure() time and\n> > >>> never updated.\n> > >>>\n> > >>> This has two implications:\n> > >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> > >>> the frame duration programmed at startup time\n> > >>> 2) The Camera::controls() limits are not updated to reflect the new\n> > >>> constraints\n> > >>>\n> > >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> > >>> and the associated helpers to regulate the exposure time base on the\n> > >>> currently programmed max frame duration using a sensor-specific margin\n> > >>> that is now reported by the CameraHelper class.\n> > >>\n> > >> I have a question about the above paragraph. The sensor-specific margin\n> > >> is already known by the kernel's sensor driver and properly written\n> > >> sensor drivers will update the exposure range reported by the query-ctrl\n> > >> ioctl after changing the vblank control.\n> > >>\n> > >> So I think it would be better to rely on the kernel for this info\n> > >> and refresh the cached exposure-control limits when changing the vblank\n> > >> control rather then duplicating this info in the CameraHelper class ?\n> > >\n> > > We discussed this with Niklas in v1\n> > > https://patchwork.libcamera.org/patch/24695/#36372\n> > >\n> > > My main concern (one more ioctl per frame apart) is that relying on\n> > > the kernel to update the control limits will introduce a delay of a\n> > > variable number of frames that correponds to the pipeline depth plus\n> > > the control delays.\n> > >\n> > > Whenever processing statistics for frame X and a new vblank value is\n> > > calculated to accommodate the newly computed exposure value (within\n> > > the FrameDurationLimits) we push the new value to DelayedControls\n> > > which caches it in a queue. Whenever a new frame is started the\n> > > queue of controls is popped and controls get applied to the device. They\n> > > will take effect a few frames later (while the kernel interface\n> > > updates the control limits immediately if I'm not mistaken).\n> >\n> > Right I forgot about the delayed-controls stuff, not being able\n> > to get the new exposure limits from the kernel then makes sense.\n>\n> I too understand why it's easier to not consult the kernel for the new\n> limits. I have no real strong feeling that the correct solution is to\n> have this limit calculation done in the kernel or user-space. What I do\n> feel somewhat strongly about is if we do the limit calculation in\n> user-space we should never consult the kernel for the limits.\n\nCurrently the EXPOSURE control limits from the kernel interface are\nonly queried when initializing the IPA. To be honest this doesn't\nbother me that much, but you already know that :)\n\nBut I agree that if we move the max exposure computation to be based\non the frame duration, we could even ignore the EXPOSURE limits\ncompletely, provided we get a way to know the minimum exposure value.\nI checked a few drivers and this doesn't always correspond to the\nexposure margin which I already added to the CameraSensorHelper.\n\nI could add a CameraSensorHelper::exposureMin(), to be honest this\nseems quite some churn when we could easily get the min exposure from\nthe v4l2-control. Do you have alternative proposals ?\n\nI think I could improve the here proposed setLimit() interface, as\nmaxExposure is only used to signal when a fixed exposure should be\nused.\n\nI could pass the min exposure to Agc::configure() and only update the\nframe duration in setLimit() with an optional \"fixed exposure\"\nparameter in case a fixed manual exposure is desired...\n\n>\n> This just mixes two ways of doing the same thing and is very confusing\n> when reading the code. So if doing this in user-space is the way forward\n> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> in IPARkISP1::configure() and instead use this method there too.\n>\n> I also feel, a little less strongly as it's a big work item, that it's\n> really wrong to duplicate the calculation both in kernel and user-space.\n\nThat I don't agree for the reasons explained already. This is not only\nabout the kernel interface, it's the IPA internal calculation that\nneed to know what the current exposure limit (in real-time) to\nregulate itself and know how to split the desired exposure between\nshutter time and gains. It could even be considered an internal IPA\nparameter..\n\n>\n> As this IMHO exposes a weakness in the V4L2 API we should think about\n> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> we really want it to report its min/max values based on the current\n> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> for each frame.\n>\n> Can we simplify the control to report the min and max values the sensor\n> can do provided the VBLANK setting allows it, and just reject any value\n> at set time that do not conform to the current VBLANK setting?\n\nWon't the v4l2 control framework would need a min and a max to adjust the\nvalue passed to the driver ?\n\n>\n> If so we can stop doing stuff in sensor drivers such as in IMX219 to\n> allow for this control dependency in each driver.\n>\n> static int imx219_set_ctrl(struct v4l2_ctrl *ctrl)\n> {\n> ...\n>\n> if (ctrl->id == V4L2_CID_VBLANK) {\n> \tint exposure_max, exposure_def;\n>\n> \t/* Update max exposure while meeting expected vblanking */\n> \texposure_max = format->height + ctrl->val - 4;\n> \texposure_def = (exposure_max < IMX219_EXPOSURE_DEFAULT) ?\n> \t\texposure_max : IMX219_EXPOSURE_DEFAULT;\n> \t__v4l2_ctrl_modify_range(imx219->exposure,\n> \t\t\t\t imx219->exposure->minimum,\n> \t\t\t\t exposure_max, imx219->exposure->step,\n> \t\t\t\t exposure_def);\n> }\n>\n> ...\n> }\n>\n> >\n> > Question: is there code somewhere to guarantee that when delayed\n> > controls changes vblank it applies the vblank before the exposure?\n> >\n> > IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> > most (all?) sensor drivers do not implement this, so the kernel\n> > just applies them 1-by-1 internally. And if vblank gets increased\n> > only after exposure then for one frame exposure will still get\n> > clamped based on the old vblank.\n> >\n> > This problem actually sounds like a more likely cause for Stefan's\n> > oscillation issue you mention below.\n> >\n> > >> One could even calculate the exposure-margin by calculating the configured\n> > >> frame-time (in a unit of lines) and then substracting the max-exposure\n> > >> reported by the sensor from the frame-time-in-lines.\n> > >\n> > > Stefan suggested that too.\n> > >\n> > > When you configure() you take the sensor's exposure time and the frame\n> > > length and the diff between the two is the margin. However there is no\n> > > indication anywhere that the sensor driver has to initialize controls\n> > > in this way and considering how slightly different sensor drivers are\n> > > I wouldn't be on them being consistent.\n> >\n> > Hmm, I would say that we need to fix the sensor drivers then. But since\n> > we need the camerahelper for the analog-gain bits I guess that having\n> > to add the exposure vs framelength margin is not a big problem.\n> >\n> > >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> > >>> to the maximum allowed frame duration, initializing the algorithm\n> > >>> with the sensor's maximum capabilities results in a very low frame rate,\n> > >>> unless the user specifies a FrameDurationLimits range in which the\n> > >>> algorithm should operate. To avoid slowing the frame rate down too much\n> > >>> compute a \"default\" frame rate at algorithm configuration time, were\n> > >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> > >>> of preference.\n> > >>\n> > >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> > >> Does this algorithm also set / used to set the vblank control to\n> > >> the maximum allowed frame duration? Or does it dynamically adjust\n> > >> vblank when it needs more exposure \"range\" ?\n> > >\n> > > It didn't, as it didn't calculate the frame duration.\n> > >\n> > > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > > and use it as the desired frame duration. This is wrong as it breaks\n> > > the assumption we have a safe margin. It only doesn't break as the\n> > > driver will clamp exposure once VBLANK is applied, but I presume this\n> > > also causes a mis-alignment between what the algorithm think the\n> > > exposure time is and what is actually applied to the device (and I\n> > > presume this is what Stefan has reported causing oscillations when I\n> > > naively proposed to ignore the margin and let the driver adjust\n> > > exposure).\n> >\n> > See above, applying exposure before an increased vblank will lead\n> > to a much bigger clamp which is a more likely cause of oscillation.\n> >\n> > > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > > duration as well using the (exposure time it has computed + margin) and\n> > > sends it back to the IPA that transforms it in a proper vblank value\n> > > to be eventually set on the device.\n> > >\n> > >>\n> > >> I've been thinking about adding support to dynamically increase\n> > >> vblank when more exposure time is needed in the softISP AGC code,\n> > >> but it would probably be better to first convert that over to\n> > >> the shared AGC helpers. I know that Kieran is working on this.\n> > >\n> > > Yes, I think that's a desirable goal.\n> > >\n> > > My whole effort in this series has been around moving as much as\n> > > possible of the logic to the shared algorithm so that all IPA using it\n> > > will get a consistent behaviour from \"free\".\n> >\n> > Ack.\n> >\n> > Thank you for taking the time to answer all my questions.\n> >\n> > Regards,\n> >\n> > Hans\n> >\n> >\n>\n> --\n> Kind Regards,\n> Niklas Söderlund","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3034DBDE4C\n\tfor ;\n\tSun, 2 Nov 2025 13:36:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8680B6096B;\n\tSun, 2 Nov 2025 14:36:42 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 23945606E6\n\tfor ;\n\tSun, 2 Nov 2025 14:36:40 +0100 (CET)","from ideasonboard.com (mob-5-90-50-232.net.vodafone.it\n\t[5.90.50.232])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 7DF89A98;\n\tSun, 2 Nov 2025 14:34:47 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"OK6A1Lfe\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1762090487;\n\tbh=704htFMHQDsJZuwzY01kUth6thegc0uQs5Ahi6Zpdg8=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=OK6A1LfeIwP3mjEgm6CpjrAtz652ozhCqOxjxehJ1Gw2O+ovdtwhNPqWt7J7fJueL\n\tRxMoXrtQkqvsCnEH1AIlo+ijzPSDw0mRNszealjAGz29fYk5Vt41RXgIaKmQvWt7dF\n\t/DMx+8kg4FLjs7Dx7LQq4tWXpKA2sAoO36YuS+Qs=","Date":"Sun, 2 Nov 2025 14:36:36 +0100","From":"Jacopo Mondi ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Cc":"Hans de Goede , \n\tJacopo Mondi ,\n\tRobert Mader , \n\tlibcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"<54xd52swj5ftm6sbv6habzmjzojbqle6u4kvt442mdvz3ay74u@c4tkaqvly5pf>","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20251028153842.GC365372@ragnatech.se>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36620,"web_url":"https://patchwork.libcamera.org/comment/36620/","msgid":"<20251102160419.GK365372@ragnatech.se>","date":"2025-11-02T16:04:19","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Jacopo,\n\nOn 2025-11-02 14:36:36 +0100, Jacopo Mondi wrote:\n> Hi Niklas,\n> \n> On Tue, Oct 28, 2025 at 04:38:42PM +0100, Niklas Söderlund wrote:\n> > Hello,\n> >\n> > On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> > > Hi Jacopo,\n> > >\n> > > On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > > > Hi Hans\n> > > >\n> > > > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> > > >> Hi Jacopo,\n> > > >>\n> > > >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> > > >>> By definition, the exposure time that can be programmed on a sensor is\n> > > >>> limited by the frame duration and whenever the frame duration is\n> > > >>> updated, the exposure time limits should be updated as well.\n> > > >>>\n> > > >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> > > >>> libipa, where the exposure limits are computed at configure() time and\n> > > >>> never updated.\n> > > >>>\n> > > >>> This has two implications:\n> > > >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> > > >>> the frame duration programmed at startup time\n> > > >>> 2) The Camera::controls() limits are not updated to reflect the new\n> > > >>> constraints\n> > > >>>\n> > > >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> > > >>> and the associated helpers to regulate the exposure time base on the\n> > > >>> currently programmed max frame duration using a sensor-specific margin\n> > > >>> that is now reported by the CameraHelper class.\n> > > >>\n> > > >> I have a question about the above paragraph. The sensor-specific margin\n> > > >> is already known by the kernel's sensor driver and properly written\n> > > >> sensor drivers will update the exposure range reported by the query-ctrl\n> > > >> ioctl after changing the vblank control.\n> > > >>\n> > > >> So I think it would be better to rely on the kernel for this info\n> > > >> and refresh the cached exposure-control limits when changing the vblank\n> > > >> control rather then duplicating this info in the CameraHelper class ?\n> > > >\n> > > > We discussed this with Niklas in v1\n> > > > https://patchwork.libcamera.org/patch/24695/#36372\n> > > >\n> > > > My main concern (one more ioctl per frame apart) is that relying on\n> > > > the kernel to update the control limits will introduce a delay of a\n> > > > variable number of frames that correponds to the pipeline depth plus\n> > > > the control delays.\n> > > >\n> > > > Whenever processing statistics for frame X and a new vblank value is\n> > > > calculated to accommodate the newly computed exposure value (within\n> > > > the FrameDurationLimits) we push the new value to DelayedControls\n> > > > which caches it in a queue. Whenever a new frame is started the\n> > > > queue of controls is popped and controls get applied to the device. They\n> > > > will take effect a few frames later (while the kernel interface\n> > > > updates the control limits immediately if I'm not mistaken).\n> > >\n> > > Right I forgot about the delayed-controls stuff, not being able\n> > > to get the new exposure limits from the kernel then makes sense.\n> >\n> > I too understand why it's easier to not consult the kernel for the new\n> > limits. I have no real strong feeling that the correct solution is to\n> > have this limit calculation done in the kernel or user-space. What I do\n> > feel somewhat strongly about is if we do the limit calculation in\n> > user-space we should never consult the kernel for the limits.\n> \n> Currently the EXPOSURE control limits from the kernel interface are\n> only queried when initializing the IPA. To be honest this doesn't\n> bother me that much, but you already know that :)\n> \n> But I agree that if we move the max exposure computation to be based\n> on the frame duration, we could even ignore the EXPOSURE limits\n> completely, provided we get a way to know the minimum exposure value.\n> I checked a few drivers and this doesn't always correspond to the\n> exposure margin which I already added to the CameraSensorHelper.\n> \n> I could add a CameraSensorHelper::exposureMin(), to be honest this\n> seems quite some churn when we could easily get the min exposure from\n> the v4l2-control. Do you have alternative proposals ?\n\nNo, and I think that highlights my concern.\n\nIf we can't be sure we can to 100% calculate the exposure limits in \nuser-space for all sensors, with their special cases etc, based on the \nVBLANK setting and need to check the kernel sometimes. Then I think it's \ndangerous to do so. What if the next issue us that the min exposure time \nalso depends on VBLANK for another sensor for example?\n\nBut if we can be sure we can do the limit calculation in user-space I \nthink we should. But then we should do so all the time to prepare for \ndeprecating the V4L2_CID_EXPOSURE control.\n\nI will not rage on about this anymore, or I will try not too ;-) When I \nsee the same thing done twice in two different locations, the results \nnot correlated, and even worse IMHO sometimes done using method A and \nsometimes using method B my head says danger, danger, this will come \nback and bite me!\n\nThe current code do not do the right thing either, and is currently \nbiting me. I agree too something that will potentially bite me in the \nfuture and gives better images now is better then fighting V4L2 API for \nyears before this can be fixed properly.\n\n> \n> I think I could improve the here proposed setLimit() interface, as\n> maxExposure is only used to signal when a fixed exposure should be\n> used.\n> \n> I could pass the min exposure to Agc::configure() and only update the\n> frame duration in setLimit() with an optional \"fixed exposure\"\n> parameter in case a fixed manual exposure is desired...\n> \n> >\n> > This just mixes two ways of doing the same thing and is very confusing\n> > when reading the code. So if doing this in user-space is the way forward\n> > I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> > in IPARkISP1::configure() and instead use this method there too.\n> >\n> > I also feel, a little less strongly as it's a big work item, that it's\n> > really wrong to duplicate the calculation both in kernel and user-space.\n> \n> That I don't agree for the reasons explained already. This is not only\n> about the kernel interface, it's the IPA internal calculation that\n> need to know what the current exposure limit (in real-time) to\n> regulate itself and know how to split the desired exposure between\n> shutter time and gains. It could even be considered an internal IPA\n> parameter..\n> \n> >\n> > As this IMHO exposes a weakness in the V4L2 API we should think about\n> > how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> > we really want it to report its min/max values based on the current\n> > V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> > for each frame.\n> >\n> > Can we simplify the control to report the min and max values the sensor\n> > can do provided the VBLANK setting allows it, and just reject any value\n> > at set time that do not conform to the current VBLANK setting?\n> \n> Won't the v4l2 control framework would need a min and a max to adjust the\n> value passed to the driver ?\n> \n> >\n> > If so we can stop doing stuff in sensor drivers such as in IMX219 to\n> > allow for this control dependency in each driver.\n> >\n> > static int imx219_set_ctrl(struct v4l2_ctrl *ctrl)\n> > {\n> > ...\n> >\n> > if (ctrl->id == V4L2_CID_VBLANK) {\n> > \tint exposure_max, exposure_def;\n> >\n> > \t/* Update max exposure while meeting expected vblanking */\n> > \texposure_max = format->height + ctrl->val - 4;\n> > \texposure_def = (exposure_max < IMX219_EXPOSURE_DEFAULT) ?\n> > \t\texposure_max : IMX219_EXPOSURE_DEFAULT;\n> > \t__v4l2_ctrl_modify_range(imx219->exposure,\n> > \t\t\t\t imx219->exposure->minimum,\n> > \t\t\t\t exposure_max, imx219->exposure->step,\n> > \t\t\t\t exposure_def);\n> > }\n> >\n> > ...\n> > }\n> >\n> > >\n> > > Question: is there code somewhere to guarantee that when delayed\n> > > controls changes vblank it applies the vblank before the exposure?\n> > >\n> > > IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> > > most (all?) sensor drivers do not implement this, so the kernel\n> > > just applies them 1-by-1 internally. And if vblank gets increased\n> > > only after exposure then for one frame exposure will still get\n> > > clamped based on the old vblank.\n> > >\n> > > This problem actually sounds like a more likely cause for Stefan's\n> > > oscillation issue you mention below.\n> > >\n> > > >> One could even calculate the exposure-margin by calculating the configured\n> > > >> frame-time (in a unit of lines) and then substracting the max-exposure\n> > > >> reported by the sensor from the frame-time-in-lines.\n> > > >\n> > > > Stefan suggested that too.\n> > > >\n> > > > When you configure() you take the sensor's exposure time and the frame\n> > > > length and the diff between the two is the margin. However there is no\n> > > > indication anywhere that the sensor driver has to initialize controls\n> > > > in this way and considering how slightly different sensor drivers are\n> > > > I wouldn't be on them being consistent.\n> > >\n> > > Hmm, I would say that we need to fix the sensor drivers then. But since\n> > > we need the camerahelper for the analog-gain bits I guess that having\n> > > to add the exposure vs framelength margin is not a big problem.\n> > >\n> > > >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> > > >>> to the maximum allowed frame duration, initializing the algorithm\n> > > >>> with the sensor's maximum capabilities results in a very low frame rate,\n> > > >>> unless the user specifies a FrameDurationLimits range in which the\n> > > >>> algorithm should operate. To avoid slowing the frame rate down too much\n> > > >>> compute a \"default\" frame rate at algorithm configuration time, were\n> > > >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> > > >>> of preference.\n> > > >>\n> > > >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> > > >> Does this algorithm also set / used to set the vblank control to\n> > > >> the maximum allowed frame duration? Or does it dynamically adjust\n> > > >> vblank when it needs more exposure \"range\" ?\n> > > >\n> > > > It didn't, as it didn't calculate the frame duration.\n> > > >\n> > > > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > > > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > > > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > > > and use it as the desired frame duration. This is wrong as it breaks\n> > > > the assumption we have a safe margin. It only doesn't break as the\n> > > > driver will clamp exposure once VBLANK is applied, but I presume this\n> > > > also causes a mis-alignment between what the algorithm think the\n> > > > exposure time is and what is actually applied to the device (and I\n> > > > presume this is what Stefan has reported causing oscillations when I\n> > > > naively proposed to ignore the margin and let the driver adjust\n> > > > exposure).\n> > >\n> > > See above, applying exposure before an increased vblank will lead\n> > > to a much bigger clamp which is a more likely cause of oscillation.\n> > >\n> > > > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > > > duration as well using the (exposure time it has computed + margin) and\n> > > > sends it back to the IPA that transforms it in a proper vblank value\n> > > > to be eventually set on the device.\n> > > >\n> > > >>\n> > > >> I've been thinking about adding support to dynamically increase\n> > > >> vblank when more exposure time is needed in the softISP AGC code,\n> > > >> but it would probably be better to first convert that over to\n> > > >> the shared AGC helpers. I know that Kieran is working on this.\n> > > >\n> > > > Yes, I think that's a desirable goal.\n> > > >\n> > > > My whole effort in this series has been around moving as much as\n> > > > possible of the logic to the shared algorithm so that all IPA using it\n> > > > will get a consistent behaviour from \"free\".\n> > >\n> > > Ack.\n> > >\n> > > Thank you for taking the time to answer all my questions.\n> > >\n> > > Regards,\n> > >\n> > > Hans\n> > >\n> > >\n> >\n> > --\n> > Kind Regards,\n> > Niklas Söderlund","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3C404C3241\n\tfor ;\n\tSun, 2 Nov 2025 16:04:26 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 694606096B;\n\tSun, 2 Nov 2025 17:04:25 +0100 (CET)","from fout-a8-smtp.messagingengine.com\n\t(fout-a8-smtp.messagingengine.com [103.168.172.151])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D074B606E6\n\tfor ;\n\tSun, 2 Nov 2025 17:04:22 +0100 (CET)","from phl-compute-02.internal (phl-compute-02.internal\n\t[10.202.2.42])\n\tby mailfout.phl.internal (Postfix) with ESMTP id DABAEEC02CF;\n\tSun, 2 Nov 2025 11:04:21 -0500 (EST)","from phl-mailfrontend-01 ([10.202.2.162])\n\tby phl-compute-02.internal (MEProxy); Sun, 02 Nov 2025 11:04:21 -0500","by mail.messagingengine.com (Postfix) with ESMTPA; Sun,\n\t2 Nov 2025 11:04:20 -0500 (EST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=ragnatech.se header.i=@ragnatech.se\n\theader.b=\"V2zXL38b\"; dkim=pass (2048-bit key;\n\tunprotected) header.d=messagingengine.com\n\theader.i=@messagingengine.com header.b=\"totjJUJu\"; \n\tdkim-atps=neutral","DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=ragnatech.se; h=\n\tcc:cc:content-transfer-encoding:content-type:content-type:date\n\t:date:from:from:in-reply-to:in-reply-to:message-id:mime-version\n\t:references:reply-to:subject:subject:to:to; s=fm2; t=1762099461;\n\tx=1762185861; bh=hyBP0veN4ABsTy56P7ZBDbCtsqfE9TInmLaGdWxrnpc=; b=\n\tV2zXL38bpx/xYAM8EOt3293lrYSTbxY9n19lpfLK8bw3bKoI19v2zfwQR4qoiydQ\n\tc0mTDZxOZUTN4x3zZ4MoZRRJ6OIN42VMa4J28PfxdA8U9EO47g7Fz/khOdX0DcP7\n\tHbZavb7mmr+b+9v3GMDpEKjkAPLas0TL8VyYiCeX6KGLJ4pJZ/Sjo2z4PB3RMtLy\n\tzY7QW3PwPVAAt+6HuwuQm++Cwf42KvyUG3r3kLrcV/3XHv1ekSpYFeS6wQ+vEP6D\n\tqOhAab98w4hKQvf85+/7k6r+D/JFtiPoLa+Ptr5ykmRaHc2MYyC9ryoRqVVxp+g5\n\tf+7L/GAu9+9qmZFne+ZAiw==","v=1; a=rsa-sha256; c=relaxed/relaxed; d=\n\tmessagingengine.com; h=cc:cc:content-transfer-encoding\n\t:content-type:content-type:date:date:feedback-id:feedback-id\n\t:from:from:in-reply-to:in-reply-to:message-id:mime-version\n\t:references:reply-to:subject:subject:to:to:x-me-proxy\n\t:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1762099461; x=\n\t1762185861; bh=hyBP0veN4ABsTy56P7ZBDbCtsqfE9TInmLaGdWxrnpc=; b=t\n\totjJUJuPGSa0f/hy124hR6/nJaSokUd91ya8+kJUNCwnF15xgjxiIg0mW6pVly+A\n\tMktodeqB03b/+YK2ku/OVUw2x9SSGQde5IaxHIUmfuDymK0IDZUVUMBif2539cHM\n\tG8Sz0/Voq3oeLxnYFRSMuie0Z2IsD3SpAVziYjD/nXsuCkgi+O/r4ZXhc8e48iAg\n\thlDqR71ScvrARbefHcJY+AlH4T1XSt2x7bCO8hQCB4ih+GtM6MPZ0VS2X6tLo9Sn\n\tS7zV/bTcBhx4dvPAP8Z4r1AcJNcwes+CLBtTQPs4XhXSDX14/wZgQISZUwc4mTSq\n\tDFMd0x54gG9ARXL2QbfWg=="],"X-ME-Sender":"\n\t","X-ME-Received":"","X-ME-Proxy-Cause":"gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggddujeehieekucetufdoteggodetrf\n\tdotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu\n\trghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf\n\tgurhepfffhvfevuffkfhggtggugfgjsehtkeertddttdejnecuhfhrohhmpefpihhklhgr\n\tshcuufpnuggvrhhluhhnugcuoehnihhklhgrshdrshhouggvrhhluhhnugesrhgrghhnrg\n\thtvggthhdrshgvqeenucggtffrrghtthgvrhhnpeehheekteejhfdttedthffgvdegieeu\n\tjeeiheetveetteeftdehledtffffgeejheenucffohhmrghinheplhhisggtrghmvghrrg\n\tdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm\n\tpehnihhklhgrshdrshhouggvrhhluhhnugesrhgrghhnrghtvggthhdrshgvpdhnsggprh\n\tgtphhtthhopeehpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehjrggtohhpohdr\n\tmhhonhguihesihguvggrshhonhgsohgrrhgurdgtohhmpdhrtghpthhtohephhgrnhhsgh\n\teskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheprhhosggvrhhtrdhmrgguvghrsegtohhl\n\tlhgrsghorhgrrdgtohhmpdhrtghpthhtoheplhhisggtrghmvghrrgdquggvvhgvlheslh\n\thishhtshdrlhhisggtrghmvghrrgdrohhrghdprhgtphhtthhopehkihgvrhgrnhdrsghi\n\tnhhghhgrmhesihguvggrshhonhgsohgrrhgurdgtohhm","X-ME-Proxy":"\n\t\n\t\n\t\n\t","Feedback-ID":"i80c9496c:Fastmail","Date":"Sun, 2 Nov 2025 17:04:19 +0100","From":"Niklas =?utf-8?q?S=C3=B6derlund?= ","To":"Jacopo Mondi ","Cc":"Hans de Goede ,\n\tRobert Mader ,\n\tlibcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"<20251102160419.GK365372@ragnatech.se>","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<54xd52swj5ftm6sbv6habzmjzojbqle6u4kvt442mdvz3ay74u@c4tkaqvly5pf>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<54xd52swj5ftm6sbv6habzmjzojbqle6u4kvt442mdvz3ay74u@c4tkaqvly5pf>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36622,"web_url":"https://patchwork.libcamera.org/comment/36622/","msgid":"<20251102164223.GL365372@ragnatech.se>","date":"2025-11-02T16:42:23","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hello (again) Jacopo,\n\nOn 2025-11-02 17:04:22 +0100, Niklas Söderlund wrote:\n> Hi Jacopo,\n> \n> On 2025-11-02 14:36:36 +0100, Jacopo Mondi wrote:\n> > Hi Niklas,\n> > \n> > On Tue, Oct 28, 2025 at 04:38:42PM +0100, Niklas Söderlund wrote:\n> > > Hello,\n> > >\n> > > On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> > > > Hi Jacopo,\n> > > >\n> > > > On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > > > > Hi Hans\n> > > > >\n> > > > > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> > > > >> Hi Jacopo,\n> > > > >>\n> > > > >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> > > > >>> By definition, the exposure time that can be programmed on a sensor is\n> > > > >>> limited by the frame duration and whenever the frame duration is\n> > > > >>> updated, the exposure time limits should be updated as well.\n> > > > >>>\n> > > > >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> > > > >>> libipa, where the exposure limits are computed at configure() time and\n> > > > >>> never updated.\n> > > > >>>\n> > > > >>> This has two implications:\n> > > > >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> > > > >>> the frame duration programmed at startup time\n> > > > >>> 2) The Camera::controls() limits are not updated to reflect the new\n> > > > >>> constraints\n> > > > >>>\n> > > > >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> > > > >>> and the associated helpers to regulate the exposure time base on the\n> > > > >>> currently programmed max frame duration using a sensor-specific margin\n> > > > >>> that is now reported by the CameraHelper class.\n> > > > >>\n> > > > >> I have a question about the above paragraph. The sensor-specific margin\n> > > > >> is already known by the kernel's sensor driver and properly written\n> > > > >> sensor drivers will update the exposure range reported by the query-ctrl\n> > > > >> ioctl after changing the vblank control.\n> > > > >>\n> > > > >> So I think it would be better to rely on the kernel for this info\n> > > > >> and refresh the cached exposure-control limits when changing the vblank\n> > > > >> control rather then duplicating this info in the CameraHelper class ?\n> > > > >\n> > > > > We discussed this with Niklas in v1\n> > > > > https://patchwork.libcamera.org/patch/24695/#36372\n> > > > >\n> > > > > My main concern (one more ioctl per frame apart) is that relying on\n> > > > > the kernel to update the control limits will introduce a delay of a\n> > > > > variable number of frames that correponds to the pipeline depth plus\n> > > > > the control delays.\n> > > > >\n> > > > > Whenever processing statistics for frame X and a new vblank value is\n> > > > > calculated to accommodate the newly computed exposure value (within\n> > > > > the FrameDurationLimits) we push the new value to DelayedControls\n> > > > > which caches it in a queue. Whenever a new frame is started the\n> > > > > queue of controls is popped and controls get applied to the device. They\n> > > > > will take effect a few frames later (while the kernel interface\n> > > > > updates the control limits immediately if I'm not mistaken).\n> > > >\n> > > > Right I forgot about the delayed-controls stuff, not being able\n> > > > to get the new exposure limits from the kernel then makes sense.\n> > >\n> > > I too understand why it's easier to not consult the kernel for the new\n> > > limits. I have no real strong feeling that the correct solution is to\n> > > have this limit calculation done in the kernel or user-space. What I do\n> > > feel somewhat strongly about is if we do the limit calculation in\n> > > user-space we should never consult the kernel for the limits.\n> > \n> > Currently the EXPOSURE control limits from the kernel interface are\n> > only queried when initializing the IPA. To be honest this doesn't\n> > bother me that much, but you already know that :)\n> > \n> > But I agree that if we move the max exposure computation to be based\n> > on the frame duration, we could even ignore the EXPOSURE limits\n> > completely, provided we get a way to know the minimum exposure value.\n> > I checked a few drivers and this doesn't always correspond to the\n> > exposure margin which I already added to the CameraSensorHelper.\n> > \n> > I could add a CameraSensorHelper::exposureMin(), to be honest this\n> > seems quite some churn when we could easily get the min exposure from\n> > the v4l2-control. Do you have alternative proposals ?\n> \n> No, and I think that highlights my concern.\n> \n> If we can't be sure we can to 100% calculate the exposure limits in \n> user-space for all sensors, with their special cases etc, based on the \n> VBLANK setting and need to check the kernel sometimes. Then I think it's \n> dangerous to do so. What if the next issue us that the min exposure time \n> also depends on VBLANK for another sensor for example?\n> \n> But if we can be sure we can do the limit calculation in user-space I \n> think we should. But then we should do so all the time to prepare for \n> deprecating the V4L2_CID_EXPOSURE control.\n> \n> I will not rage on about this anymore, or I will try not too ;-) When I \n> see the same thing done twice in two different locations, the results \n> not correlated, and even worse IMHO sometimes done using method A and \n> sometimes using method B my head says danger, danger, this will come \n> back and bite me!\n> \n> The current code do not do the right thing either, and is currently \n> biting me. I agree too something that will potentially bite me in the \n> future and gives better images now is better then fighting V4L2 API for \n> years before this can be fixed properly.\n\nI said I would try ;-)\n\nOne (half ugly) idea if we do want to use the kernel to calculate the \nvalues would be to at IPA init time in the pipeline handler do a \"range \nfinding run\". The pipeline would run over the sensors VBLANK setting \nrange recording the min and max EXPOSURE values, build a lookup table it \nthen gives to the IPA.\n\nThe IPA then uses this table internally to map a VBLANK value to a \nEXPOSURE min/max limit. This could likely be done in a sensor helper \nclass.\n\n> \n> > \n> > I think I could improve the here proposed setLimit() interface, as\n> > maxExposure is only used to signal when a fixed exposure should be\n> > used.\n> > \n> > I could pass the min exposure to Agc::configure() and only update the\n> > frame duration in setLimit() with an optional \"fixed exposure\"\n> > parameter in case a fixed manual exposure is desired...\n> > \n> > >\n> > > This just mixes two ways of doing the same thing and is very confusing\n> > > when reading the code. So if doing this in user-space is the way forward\n> > > I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> > > in IPARkISP1::configure() and instead use this method there too.\n> > >\n> > > I also feel, a little less strongly as it's a big work item, that it's\n> > > really wrong to duplicate the calculation both in kernel and user-space.\n> > \n> > That I don't agree for the reasons explained already. This is not only\n> > about the kernel interface, it's the IPA internal calculation that\n> > need to know what the current exposure limit (in real-time) to\n> > regulate itself and know how to split the desired exposure between\n> > shutter time and gains. It could even be considered an internal IPA\n> > parameter..\n> > \n> > >\n> > > As this IMHO exposes a weakness in the V4L2 API we should think about\n> > > how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> > > we really want it to report its min/max values based on the current\n> > > V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> > > for each frame.\n> > >\n> > > Can we simplify the control to report the min and max values the sensor\n> > > can do provided the VBLANK setting allows it, and just reject any value\n> > > at set time that do not conform to the current VBLANK setting?\n> > \n> > Won't the v4l2 control framework would need a min and a max to adjust the\n> > value passed to the driver ?\n> > \n> > >\n> > > If so we can stop doing stuff in sensor drivers such as in IMX219 to\n> > > allow for this control dependency in each driver.\n> > >\n> > > static int imx219_set_ctrl(struct v4l2_ctrl *ctrl)\n> > > {\n> > > ...\n> > >\n> > > if (ctrl->id == V4L2_CID_VBLANK) {\n> > > \tint exposure_max, exposure_def;\n> > >\n> > > \t/* Update max exposure while meeting expected vblanking */\n> > > \texposure_max = format->height + ctrl->val - 4;\n> > > \texposure_def = (exposure_max < IMX219_EXPOSURE_DEFAULT) ?\n> > > \t\texposure_max : IMX219_EXPOSURE_DEFAULT;\n> > > \t__v4l2_ctrl_modify_range(imx219->exposure,\n> > > \t\t\t\t imx219->exposure->minimum,\n> > > \t\t\t\t exposure_max, imx219->exposure->step,\n> > > \t\t\t\t exposure_def);\n> > > }\n> > >\n> > > ...\n> > > }\n> > >\n> > > >\n> > > > Question: is there code somewhere to guarantee that when delayed\n> > > > controls changes vblank it applies the vblank before the exposure?\n> > > >\n> > > > IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> > > > most (all?) sensor drivers do not implement this, so the kernel\n> > > > just applies them 1-by-1 internally. And if vblank gets increased\n> > > > only after exposure then for one frame exposure will still get\n> > > > clamped based on the old vblank.\n> > > >\n> > > > This problem actually sounds like a more likely cause for Stefan's\n> > > > oscillation issue you mention below.\n> > > >\n> > > > >> One could even calculate the exposure-margin by calculating the configured\n> > > > >> frame-time (in a unit of lines) and then substracting the max-exposure\n> > > > >> reported by the sensor from the frame-time-in-lines.\n> > > > >\n> > > > > Stefan suggested that too.\n> > > > >\n> > > > > When you configure() you take the sensor's exposure time and the frame\n> > > > > length and the diff between the two is the margin. However there is no\n> > > > > indication anywhere that the sensor driver has to initialize controls\n> > > > > in this way and considering how slightly different sensor drivers are\n> > > > > I wouldn't be on them being consistent.\n> > > >\n> > > > Hmm, I would say that we need to fix the sensor drivers then. But since\n> > > > we need the camerahelper for the analog-gain bits I guess that having\n> > > > to add the exposure vs framelength margin is not a big problem.\n> > > >\n> > > > >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> > > > >>> to the maximum allowed frame duration, initializing the algorithm\n> > > > >>> with the sensor's maximum capabilities results in a very low frame rate,\n> > > > >>> unless the user specifies a FrameDurationLimits range in which the\n> > > > >>> algorithm should operate. To avoid slowing the frame rate down too much\n> > > > >>> compute a \"default\" frame rate at algorithm configuration time, were\n> > > > >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> > > > >>> of preference.\n> > > > >>\n> > > > >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> > > > >> Does this algorithm also set / used to set the vblank control to\n> > > > >> the maximum allowed frame duration? Or does it dynamically adjust\n> > > > >> vblank when it needs more exposure \"range\" ?\n> > > > >\n> > > > > It didn't, as it didn't calculate the frame duration.\n> > > > >\n> > > > > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > > > > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > > > > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > > > > and use it as the desired frame duration. This is wrong as it breaks\n> > > > > the assumption we have a safe margin. It only doesn't break as the\n> > > > > driver will clamp exposure once VBLANK is applied, but I presume this\n> > > > > also causes a mis-alignment between what the algorithm think the\n> > > > > exposure time is and what is actually applied to the device (and I\n> > > > > presume this is what Stefan has reported causing oscillations when I\n> > > > > naively proposed to ignore the margin and let the driver adjust\n> > > > > exposure).\n> > > >\n> > > > See above, applying exposure before an increased vblank will lead\n> > > > to a much bigger clamp which is a more likely cause of oscillation.\n> > > >\n> > > > > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > > > > duration as well using the (exposure time it has computed + margin) and\n> > > > > sends it back to the IPA that transforms it in a proper vblank value\n> > > > > to be eventually set on the device.\n> > > > >\n> > > > >>\n> > > > >> I've been thinking about adding support to dynamically increase\n> > > > >> vblank when more exposure time is needed in the softISP AGC code,\n> > > > >> but it would probably be better to first convert that over to\n> > > > >> the shared AGC helpers. I know that Kieran is working on this.\n> > > > >\n> > > > > Yes, I think that's a desirable goal.\n> > > > >\n> > > > > My whole effort in this series has been around moving as much as\n> > > > > possible of the logic to the shared algorithm so that all IPA using it\n> > > > > will get a consistent behaviour from \"free\".\n> > > >\n> > > > Ack.\n> > > >\n> > > > Thank you for taking the time to answer all my questions.\n> > > >\n> > > > Regards,\n> > > >\n> > > > Hans\n> > > >\n> > > >\n> > >\n> > > --\n> > > Kind Regards,\n> > > Niklas Söderlund\n> \n> -- \n> Kind Regards,\n> Niklas Söderlund","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0E70CC3241\n\tfor ;\n\tSun, 2 Nov 2025 16:42:30 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 68FAE6096B;\n\tSun, 2 Nov 2025 17:42:29 +0100 (CET)","from fhigh-a3-smtp.messagingengine.com\n\t(fhigh-a3-smtp.messagingengine.com [103.168.172.154])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0CF86606E6\n\tfor ;\n\tSun, 2 Nov 2025 17:42:27 +0100 (CET)","from phl-compute-04.internal (phl-compute-04.internal\n\t[10.202.2.44])\n\tby mailfhigh.phl.internal (Postfix) with ESMTP id 1773614000EC;\n\tSun, 2 Nov 2025 11:42:26 -0500 (EST)","from phl-mailfrontend-02 ([10.202.2.163])\n\tby phl-compute-04.internal (MEProxy); Sun, 02 Nov 2025 11:42:26 -0500","by mail.messagingengine.com (Postfix) with ESMTPA; Sun,\n\t2 Nov 2025 11:42:25 -0500 (EST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=ragnatech.se header.i=@ragnatech.se\n\theader.b=\"KCb/BJoa\"; dkim=pass (2048-bit key;\n\tunprotected) header.d=messagingengine.com\n\theader.i=@messagingengine.com header.b=\"gFB7meZ2\"; \n\tdkim-atps=neutral","DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=ragnatech.se; h=\n\tcc:cc:content-transfer-encoding:content-type:content-type:date\n\t:date:from:from:in-reply-to:in-reply-to:message-id:mime-version\n\t:references:reply-to:subject:subject:to:to; s=fm2; t=1762101746;\n\tx=1762188146; bh=tNe3D0L+W2E/oAejCEIfVuX6UJiJhDs3MxBkLW5+TNM=; b=\n\tKCb/BJoaZu1nk1X+j09HMpMMxZVl5lfUxLIvEFnRJyzTZep+dU4WZQVTYOO5EgSm\n\tSMxS7fbSPNm60AftYIv8/TXjWe5oXRlqeoPJzFK0UysGfXrTFo1f2MDBlSLa9Kgr\n\tgFk/uXcPn8kEcy5rc0URvnbF/GDJMXz7DSfmcEJtst1qLs33BBejv0yNBoovys57\n\t6rHPA1+wti/8oYu8KZrfb0AQZp+d5dU99syCipHrcVuMOAS2WqjmMEOmJ8Rk5sHT\n\tH4OyrbwkFKD/XMk0m1bkMPgU+CF4icx/kckO0LL1ezwK5SsH5o60NI6wNebyYf9B\n\t0ee64RLMPCLaZPiqq+vTzQ==","v=1; a=rsa-sha256; c=relaxed/relaxed; d=\n\tmessagingengine.com; h=cc:cc:content-transfer-encoding\n\t:content-type:content-type:date:date:feedback-id:feedback-id\n\t:from:from:in-reply-to:in-reply-to:message-id:mime-version\n\t:references:reply-to:subject:subject:to:to:x-me-proxy\n\t:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1762101746; x=\n\t1762188146; bh=tNe3D0L+W2E/oAejCEIfVuX6UJiJhDs3MxBkLW5+TNM=; b=g\n\tFB7meZ25h+ZUGRdqU0RutYbGwySStd4caal2IJvAhz8THrrdzSuamLpch5HVlmEU\n\tCdDeoR8EIMwXJTzK6uEzxbO5WoXvHWDZWeE+6JEhCdKhjTIyQqH9Uh+kMQOBmDJt\n\t5wxSEBujCCDkboOHixcbB8poEDeTl2j9a0hVx64nIG8wV4uTWZ1y0LAPQ0d9qeuE\n\tFT8qyEqSUw9UWQOVqAFSSjzQluD+M+Y7w5pK5a8l/ngiDWzIl7fF1vYG665EY/BI\n\tcxwHBM4rv50vdpAZicI/gbb+g/XM53dZXL57D9OZg5VeL75f20rpkfBLyk3KkRSL\n\tK8Xqw4AqlvhWgeyHN46gw=="],"X-ME-Sender":"\n\t","X-ME-Received":"","X-ME-Proxy-Cause":"gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggddujeehjeeiucetufdoteggodetrf\n\tdotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu\n\trghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf\n\tgurhepfffhvfevuffkfhggtggugfgjsehtkeertddttdejnecuhfhrohhmpefpihhklhgr\n\tshcuufpnuggvrhhluhhnugcuoehnihhklhgrshdrshhouggvrhhluhhnugesrhgrghhnrg\n\thtvggthhdrshgvqeenucggtffrrghtthgvrhhnpeehheekteejhfdttedthffgvdegieeu\n\tjeeiheetveetteeftdehledtffffgeejheenucffohhmrghinheplhhisggtrghmvghrrg\n\tdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhm\n\tpehnihhklhgrshdrshhouggvrhhluhhnugesrhgrghhnrghtvggthhdrshgvpdhnsggprh\n\tgtphhtthhopeehpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehjrggtohhpohdr\n\tmhhonhguihesihguvggrshhonhgsohgrrhgurdgtohhmpdhrtghpthhtohephhgrnhhsgh\n\teskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheprhhosggvrhhtrdhmrgguvghrsegtohhl\n\tlhgrsghorhgrrdgtohhmpdhrtghpthhtoheplhhisggtrghmvghrrgdquggvvhgvlheslh\n\thishhtshdrlhhisggtrghmvghrrgdrohhrghdprhgtphhtthhopehkihgvrhgrnhdrsghi\n\tnhhghhgrmhesihguvggrshhonhgsohgrrhgurdgtohhm","X-ME-Proxy":"\n\t\n\t\n\t\n\t","Feedback-ID":"i80c9496c:Fastmail","Date":"Sun, 2 Nov 2025 17:42:23 +0100","From":"Niklas =?utf-8?q?S=C3=B6derlund?= ","To":"Jacopo Mondi ","Cc":"Hans de Goede ,\n\tRobert Mader ,\n\tlibcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"<20251102164223.GL365372@ragnatech.se>","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<54xd52swj5ftm6sbv6habzmjzojbqle6u4kvt442mdvz3ay74u@c4tkaqvly5pf>\n\t<20251102160419.GK365372@ragnatech.se>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20251102160419.GK365372@ragnatech.se>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36623,"web_url":"https://patchwork.libcamera.org/comment/36623/","msgid":"<20251102173157.GD27255@pendragon.ideasonboard.com>","date":"2025-11-02T17:31:57","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Tue, Oct 28, 2025 at 02:58:15PM +0000, Naushir Patuck wrote:\n> On Tue, 28 Oct 2025 at 14:49, Hans de Goede wrote:\n> > On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> > >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> > >>> By definition, the exposure time that can be programmed on a sensor is\n> > >>> limited by the frame duration and whenever the frame duration is\n> > >>> updated, the exposure time limits should be updated as well.\n> > >>>\n> > >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> > >>> libipa, where the exposure limits are computed at configure() time and\n> > >>> never updated.\n> > >>>\n> > >>> This has two implications:\n> > >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> > >>> the frame duration programmed at startup time\n> > >>> 2) The Camera::controls() limits are not updated to reflect the new\n> > >>> constraints\n> > >>>\n> > >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> > >>> and the associated helpers to regulate the exposure time base on the\n> > >>> currently programmed max frame duration using a sensor-specific margin\n> > >>> that is now reported by the CameraHelper class.\n> > >>\n> > >> I have a question about the above paragraph. The sensor-specific margin\n> > >> is already known by the kernel's sensor driver and properly written\n> > >> sensor drivers will update the exposure range reported by the query-ctrl\n> > >> ioctl after changing the vblank control.\n> > >>\n> > >> So I think it would be better to rely on the kernel for this info\n> > >> and refresh the cached exposure-control limits when changing the vblank\n> > >> control rather then duplicating this info in the CameraHelper class ?\n> > >\n> > > We discussed this with Niklas in v1\n> > > https://patchwork.libcamera.org/patch/24695/#36372\n> > >\n> > > My main concern (one more ioctl per frame apart) is that relying on\n> > > the kernel to update the control limits will introduce a delay of a\n> > > variable number of frames that correponds to the pipeline depth plus\n> > > the control delays.\n> > >\n> > > Whenever processing statistics for frame X and a new vblank value is\n> > > calculated to accommodate the newly computed exposure value (within\n> > > the FrameDurationLimits) we push the new value to DelayedControls\n> > > which caches it in a queue. Whenever a new frame is started the\n> > > queue of controls is popped and controls get applied to the device. They\n> > > will take effect a few frames later (while the kernel interface\n> > > updates the control limits immediately if I'm not mistaken).\n> >\n> > Right I forgot about the delayed-controls stuff, not being able\n> > to get the new exposure limits from the kernel then makes sense.\n> >\n> > Question: is there code somewhere to guarantee that when delayed\n> > controls changes vblank it applies the vblank before the exposure?\n> >\n> > IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> > most (all?) sensor drivers do not implement this, so the kernel\n> > just applies them 1-by-1 internally. And if vblank gets increased\n> > only after exposure then for one frame exposure will still get\n> > clamped based on the old vblank.\n> >\n> > This problem actually sounds like a more likely cause for Stefan's\n> > oscillation issue you mention below.\n> \n> When constructing the DelayedControls object, we mark the VBLANK\n> control as \"priority write\".\n\nJacopo, have you noticed we don't do so in the rkisp1 pipeline handler ?\nIt seems to be a bug.\n\n> This ensures that VBLANK is written\n> ahead of any (and particularly EXPOSURE) controls for this very\n> reason.\n\nIt's a workaround for a control framework limitation. The framework\nfirst validates all control values against current boundaries, and then\nproceeds to changing controls one by one. Fixing that in the kernel is\nlikely quite difficult. We could do with a better API for controls in\nV4L2 (both userspace API and in-kernel API), but it would be a\nsubstantial amount of work.\n\n> > >> One could even calculate the exposure-margin by calculating the configured\n> > >> frame-time (in a unit of lines) and then substracting the max-exposure\n> > >> reported by the sensor from the frame-time-in-lines.\n> > >\n> > > Stefan suggested that too.\n> > >\n> > > When you configure() you take the sensor's exposure time and the frame\n> > > length and the diff between the two is the margin. However there is no\n> > > indication anywhere that the sensor driver has to initialize controls\n> > > in this way and considering how slightly different sensor drivers are\n> > > I wouldn't be on them being consistent.\n> >\n> > Hmm, I would say that we need to fix the sensor drivers then. But since\n> > we need the camerahelper for the analog-gain bits I guess that having\n> > to add the exposure vs framelength margin is not a big problem.\n> >\n> > >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> > >>> to the maximum allowed frame duration, initializing the algorithm\n> > >>> with the sensor's maximum capabilities results in a very low frame rate,\n> > >>> unless the user specifies a FrameDurationLimits range in which the\n> > >>> algorithm should operate. To avoid slowing the frame rate down too much\n> > >>> compute a \"default\" frame rate at algorithm configuration time, were\n> > >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> > >>> of preference.\n> > >>\n> > >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> > >> Does this algorithm also set / used to set the vblank control to\n> > >> the maximum allowed frame duration? Or does it dynamically adjust\n> > >> vblank when it needs more exposure \"range\" ?\n> > >\n> > > It didn't, as it didn't calculate the frame duration.\n> > >\n> > > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > > and use it as the desired frame duration. This is wrong as it breaks\n> > > the assumption we have a safe margin. It only doesn't break as the\n> > > driver will clamp exposure once VBLANK is applied, but I presume this\n> > > also causes a mis-alignment between what the algorithm think the\n> > > exposure time is and what is actually applied to the device (and I\n> > > presume this is what Stefan has reported causing oscillations when I\n> > > naively proposed to ignore the margin and let the driver adjust\n> > > exposure).\n> >\n> > See above, applying exposure before an increased vblank will lead\n> > to a much bigger clamp which is a more likely cause of oscillation.\n> >\n> > > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > > duration as well using the (exposure time it has computed + margin) and\n> > > sends it back to the IPA that transforms it in a proper vblank value\n> > > to be eventually set on the device.\n> > >\n> > >>\n> > >> I've been thinking about adding support to dynamically increase\n> > >> vblank when more exposure time is needed in the softISP AGC code,\n> > >> but it would probably be better to first convert that over to\n> > >> the shared AGC helpers. I know that Kieran is working on this.\n> > >\n> > > Yes, I think that's a desirable goal.\n> > >\n> > > My whole effort in this series has been around moving as much as\n> > > possible of the logic to the shared algorithm so that all IPA using it\n> > > will get a consistent behaviour from \"free\".\n> >\n> > Ack.\n> >\n> > Thank you for taking the time to answer all my questions.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8E63EBDE4C\n\tfor ;\n\tSun, 2 Nov 2025 17:32:13 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 92EFB60A8B;\n\tSun, 2 Nov 2025 18:32:12 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 750F7606E6\n\tfor ;\n\tSun, 2 Nov 2025 18:32:11 +0100 (CET)","from pendragon.ideasonboard.com (82-203-160-149.bb.dnainternet.fi\n\t[82.203.160.149])\n\tby perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id CE8048D4;\n\tSun, 2 Nov 2025 18:30:18 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Jq1CA8Gx\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1762104619;\n\tbh=q48DSxGFsbQQE/kQYRWiF6E3kizmAq+yoWgtw9AXOgM=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=Jq1CA8Gx26vU9vlU3kVe/1G6rx2RNHaVRV1E6Vp5YYeW/3pXorVeSUw7b5PEPaXXN\n\tcWp9WfVklFyaNLlgRTcZVjpOB3swbS0Xa64g0RsY4WD5ntOLTySCAgXEpKyd051xuv\n\tgBt/5tpv/zZU0T2ul8Jb50msxkdgluDiAmWPPeDs=","Date":"Sun, 2 Nov 2025 19:31:57 +0200","From":"Laurent Pinchart ","To":"Naushir Patuck ","Cc":"Hans de Goede , Jacopo Mondi\n\t, Niklas =?utf-8?q?S=C3=B6derlund?=\n\t, Robert Mader\n\t, libcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"<20251102173157.GD27255@pendragon.ideasonboard.com>","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36624,"web_url":"https://patchwork.libcamera.org/comment/36624/","msgid":"","date":"2025-11-02T18:05:05","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Laurent\n\nOn Sun, Nov 02, 2025 at 07:31:57PM +0200, Laurent Pinchart wrote:\n> On Tue, Oct 28, 2025 at 02:58:15PM +0000, Naushir Patuck wrote:\n> > On Tue, 28 Oct 2025 at 14:49, Hans de Goede wrote:\n> > > On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > > > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> > > >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> > > >>> By definition, the exposure time that can be programmed on a sensor is\n> > > >>> limited by the frame duration and whenever the frame duration is\n> > > >>> updated, the exposure time limits should be updated as well.\n> > > >>>\n> > > >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> > > >>> libipa, where the exposure limits are computed at configure() time and\n> > > >>> never updated.\n> > > >>>\n> > > >>> This has two implications:\n> > > >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> > > >>> the frame duration programmed at startup time\n> > > >>> 2) The Camera::controls() limits are not updated to reflect the new\n> > > >>> constraints\n> > > >>>\n> > > >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> > > >>> and the associated helpers to regulate the exposure time base on the\n> > > >>> currently programmed max frame duration using a sensor-specific margin\n> > > >>> that is now reported by the CameraHelper class.\n> > > >>\n> > > >> I have a question about the above paragraph. The sensor-specific margin\n> > > >> is already known by the kernel's sensor driver and properly written\n> > > >> sensor drivers will update the exposure range reported by the query-ctrl\n> > > >> ioctl after changing the vblank control.\n> > > >>\n> > > >> So I think it would be better to rely on the kernel for this info\n> > > >> and refresh the cached exposure-control limits when changing the vblank\n> > > >> control rather then duplicating this info in the CameraHelper class ?\n> > > >\n> > > > We discussed this with Niklas in v1\n> > > > https://patchwork.libcamera.org/patch/24695/#36372\n> > > >\n> > > > My main concern (one more ioctl per frame apart) is that relying on\n> > > > the kernel to update the control limits will introduce a delay of a\n> > > > variable number of frames that correponds to the pipeline depth plus\n> > > > the control delays.\n> > > >\n> > > > Whenever processing statistics for frame X and a new vblank value is\n> > > > calculated to accommodate the newly computed exposure value (within\n> > > > the FrameDurationLimits) we push the new value to DelayedControls\n> > > > which caches it in a queue. Whenever a new frame is started the\n> > > > queue of controls is popped and controls get applied to the device. They\n> > > > will take effect a few frames later (while the kernel interface\n> > > > updates the control limits immediately if I'm not mistaken).\n> > >\n> > > Right I forgot about the delayed-controls stuff, not being able\n> > > to get the new exposure limits from the kernel then makes sense.\n> > >\n> > > Question: is there code somewhere to guarantee that when delayed\n> > > controls changes vblank it applies the vblank before the exposure?\n> > >\n> > > IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> > > most (all?) sensor drivers do not implement this, so the kernel\n> > > just applies them 1-by-1 internally. And if vblank gets increased\n> > > only after exposure then for one frame exposure will still get\n> > > clamped based on the old vblank.\n> > >\n> > > This problem actually sounds like a more likely cause for Stefan's\n> > > oscillation issue you mention below.\n> >\n> > When constructing the DelayedControls object, we mark the VBLANK\n> > control as \"priority write\".\n>\n> Jacopo, have you noticed we don't do so in the rkisp1 pipeline handler ?\n> It seems to be a bug.\n>\n\nNot at all!\n\nI can quickly send a patch, but I wonder how this went unnoticed till\nnow\n\n> > This ensures that VBLANK is written\n> > ahead of any (and particularly EXPOSURE) controls for this very\n> > reason.\n>\n> It's a workaround for a control framework limitation. The framework\n> first validates all control values against current boundaries, and then\n> proceeds to changing controls one by one. Fixing that in the kernel is\n> likely quite difficult. We could do with a better API for controls in\n> V4L2 (both userspace API and in-kernel API), but it would be a\n> substantial amount of work.\n>\n> > > >> One could even calculate the exposure-margin by calculating the configured\n> > > >> frame-time (in a unit of lines) and then substracting the max-exposure\n> > > >> reported by the sensor from the frame-time-in-lines.\n> > > >\n> > > > Stefan suggested that too.\n> > > >\n> > > > When you configure() you take the sensor's exposure time and the frame\n> > > > length and the diff between the two is the margin. However there is no\n> > > > indication anywhere that the sensor driver has to initialize controls\n> > > > in this way and considering how slightly different sensor drivers are\n> > > > I wouldn't be on them being consistent.\n> > >\n> > > Hmm, I would say that we need to fix the sensor drivers then. But since\n> > > we need the camerahelper for the analog-gain bits I guess that having\n> > > to add the exposure vs framelength margin is not a big problem.\n> > >\n> > > >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> > > >>> to the maximum allowed frame duration, initializing the algorithm\n> > > >>> with the sensor's maximum capabilities results in a very low frame rate,\n> > > >>> unless the user specifies a FrameDurationLimits range in which the\n> > > >>> algorithm should operate. To avoid slowing the frame rate down too much\n> > > >>> compute a \"default\" frame rate at algorithm configuration time, were\n> > > >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> > > >>> of preference.\n> > > >>\n> > > >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> > > >> Does this algorithm also set / used to set the vblank control to\n> > > >> the maximum allowed frame duration? Or does it dynamically adjust\n> > > >> vblank when it needs more exposure \"range\" ?\n> > > >\n> > > > It didn't, as it didn't calculate the frame duration.\n> > > >\n> > > > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > > > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > > > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > > > and use it as the desired frame duration. This is wrong as it breaks\n> > > > the assumption we have a safe margin. It only doesn't break as the\n> > > > driver will clamp exposure once VBLANK is applied, but I presume this\n> > > > also causes a mis-alignment between what the algorithm think the\n> > > > exposure time is and what is actually applied to the device (and I\n> > > > presume this is what Stefan has reported causing oscillations when I\n> > > > naively proposed to ignore the margin and let the driver adjust\n> > > > exposure).\n> > >\n> > > See above, applying exposure before an increased vblank will lead\n> > > to a much bigger clamp which is a more likely cause of oscillation.\n> > >\n> > > > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > > > duration as well using the (exposure time it has computed + margin) and\n> > > > sends it back to the IPA that transforms it in a proper vblank value\n> > > > to be eventually set on the device.\n> > > >\n> > > >>\n> > > >> I've been thinking about adding support to dynamically increase\n> > > >> vblank when more exposure time is needed in the softISP AGC code,\n> > > >> but it would probably be better to first convert that over to\n> > > >> the shared AGC helpers. I know that Kieran is working on this.\n> > > >\n> > > > Yes, I think that's a desirable goal.\n> > > >\n> > > > My whole effort in this series has been around moving as much as\n> > > > possible of the logic to the shared algorithm so that all IPA using it\n> > > > will get a consistent behaviour from \"free\".\n> > >\n> > > Ack.\n> > >\n> > > Thank you for taking the time to answer all my questions.\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B979BC3241\n\tfor ;\n\tSun, 2 Nov 2025 18:05:11 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AFB1360A80;\n\tSun, 2 Nov 2025 19:05:10 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E06E5606E6\n\tfor ;\n\tSun, 2 Nov 2025 19:05:08 +0100 (CET)","from ideasonboard.com (93-61-96-190.ip145.fastwebnet.it\n\t[93.61.96.190])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 2945C1781;\n\tSun, 2 Nov 2025 19:03:16 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"aRuL/4yB\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1762106596;\n\tbh=qavj+sU706DgfL+GHUpxwV82vThSKDlZ0k5hEZRzHsE=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=aRuL/4yBIbShgukQkzEDsDltWlrsjVR2hjqVMqJMDS6GYHSwNWSQhtTqN4LCQ0Y+z\n\tnc3TyYO+7kzD4NU42juHs3C5lYMUrqsAIm1Jo07tHi+ZTFB6R5SaEieB/jdIBWduhN\n\tylpsgWDYJWoXCxJjLRKY3l2QyH38Sn0+a/sUx+8Q=","Date":"Sun, 2 Nov 2025 19:05:05 +0100","From":"Jacopo Mondi ","To":"Laurent Pinchart ","Cc":"Naushir Patuck , \n\tHans de Goede ,\n\tJacopo Mondi , Niklas =?utf-8?b?U8O2?=\n\t=?utf-8?q?derlund?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t\n\t<20251102173157.GD27255@pendragon.ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20251102173157.GD27255@pendragon.ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36625,"web_url":"https://patchwork.libcamera.org/comment/36625/","msgid":"<20251102192246.GE27255@pendragon.ideasonboard.com>","date":"2025-11-02T19:22:46","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"On Sun, Nov 02, 2025 at 05:42:23PM +0100, Niklas Söderlund wrote:\n> On 2025-11-02 17:04:22 +0100, Niklas Söderlund wrote:\n> > On 2025-11-02 14:36:36 +0100, Jacopo Mondi wrote:\n> > > On Tue, Oct 28, 2025 at 04:38:42PM +0100, Niklas Söderlund wrote:\n> > > > On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> > > > > On 28-Oct-25 12:42 PM, Jacopo Mondi wrote:\n> > > > > > On Tue, Oct 28, 2025 at 10:53:15AM +0100, Hans de Goede wrote:\n> > > > > >> On 28-Oct-25 10:31 AM, Jacopo Mondi wrote:\n> > > > > >>> By definition, the exposure time that can be programmed on a sensor is\n> > > > > >>> limited by the frame duration and whenever the frame duration is\n> > > > > >>> updated, the exposure time limits should be updated as well.\n> > > > > >>>\n> > > > > >>> This is not the case for IPAs using the AgcMeanLuminance algorithm from\n> > > > > >>> libipa, where the exposure limits are computed at configure() time and\n> > > > > >>> never updated.\n> > > > > >>>\n> > > > > >>> This has two implications:\n> > > > > >>> 1) The AGC algorithms will always operate with an exposure time bound to\n> > > > > >>> the frame duration programmed at startup time\n> > > > > >>> 2) The Camera::controls() limits are not updated to reflect the new\n> > > > > >>> constraints\n> > > > > >>>\n> > > > > >>> This series addresses issue 1) by changing the AgcMeanLuminance class\n> > > > > >>> and the associated helpers to regulate the exposure time base on the\n> > > > > >>> currently programmed max frame duration using a sensor-specific margin\n> > > > > >>> that is now reported by the CameraHelper class.\n> > > > > >>\n> > > > > >> I have a question about the above paragraph. The sensor-specific margin\n> > > > > >> is already known by the kernel's sensor driver and properly written\n> > > > > >> sensor drivers will update the exposure range reported by the query-ctrl\n> > > > > >> ioctl after changing the vblank control.\n> > > > > >>\n> > > > > >> So I think it would be better to rely on the kernel for this info\n> > > > > >> and refresh the cached exposure-control limits when changing the vblank\n> > > > > >> control rather then duplicating this info in the CameraHelper class ?\n> > > > > >\n> > > > > > We discussed this with Niklas in v1\n> > > > > > https://patchwork.libcamera.org/patch/24695/#36372\n> > > > > >\n> > > > > > My main concern (one more ioctl per frame apart) is that relying on\n> > > > > > the kernel to update the control limits will introduce a delay of a\n> > > > > > variable number of frames that correponds to the pipeline depth plus\n> > > > > > the control delays.\n> > > > > >\n> > > > > > Whenever processing statistics for frame X and a new vblank value is\n> > > > > > calculated to accommodate the newly computed exposure value (within\n> > > > > > the FrameDurationLimits) we push the new value to DelayedControls\n> > > > > > which caches it in a queue. Whenever a new frame is started the\n> > > > > > queue of controls is popped and controls get applied to the device. They\n> > > > > > will take effect a few frames later (while the kernel interface\n> > > > > > updates the control limits immediately if I'm not mistaken).\n> > > > >\n> > > > > Right I forgot about the delayed-controls stuff, not being able\n> > > > > to get the new exposure limits from the kernel then makes sense.\n> > > >\n> > > > I too understand why it's easier to not consult the kernel for the new\n> > > > limits. I have no real strong feeling that the correct solution is to\n> > > > have this limit calculation done in the kernel or user-space. What I do\n> > > > feel somewhat strongly about is if we do the limit calculation in\n> > > > user-space we should never consult the kernel for the limits.\n> > > \n> > > Currently the EXPOSURE control limits from the kernel interface are\n> > > only queried when initializing the IPA. To be honest this doesn't\n> > > bother me that much, but you already know that :)\n> > > \n> > > But I agree that if we move the max exposure computation to be based\n> > > on the frame duration, we could even ignore the EXPOSURE limits\n> > > completely, provided we get a way to know the minimum exposure value.\n> > > I checked a few drivers and this doesn't always correspond to the\n> > > exposure margin which I already added to the CameraSensorHelper.\n> > > \n> > > I could add a CameraSensorHelper::exposureMin(), to be honest this\n> > > seems quite some churn when we could easily get the min exposure from\n> > > the v4l2-control. Do you have alternative proposals ?\n> > \n> > No, and I think that highlights my concern.\n> > \n> > If we can't be sure we can to 100% calculate the exposure limits in \n> > user-space for all sensors, with their special cases etc, based on the \n> > VBLANK setting and need to check the kernel sometimes. Then I think it's \n> > dangerous to do so. What if the next issue us that the min exposure time \n> > also depends on VBLANK for another sensor for example?\n> > \n> > But if we can be sure we can do the limit calculation in user-space I \n> > think we should. But then we should do so all the time to prepare for \n> > deprecating the V4L2_CID_EXPOSURE control.\n\nWe still need a control to set the exposure time, don't we ?\n\n> > I will not rage on about this anymore, or I will try not too ;-) When I \n> > see the same thing done twice in two different locations, the results \n> > not correlated, and even worse IMHO sometimes done using method A and \n> > sometimes using method B my head says danger, danger, this will come \n> > back and bite me!\n> > \n> > The current code do not do the right thing either, and is currently \n> > biting me. I agree too something that will potentially bite me in the \n> > future and gives better images now is better then fighting V4L2 API for \n> > years before this can be fixed properly.\n> \n> I said I would try ;-)\n> \n> One (half ugly) idea if we do want to use the kernel to calculate the \n> values would be to at IPA init time in the pipeline handler do a \"range \n> finding run\". The pipeline would run over the sensors VBLANK setting \n> range recording the min and max EXPOSURE values, build a lookup table it \n> then gives to the IPA.\n> \n> The IPA then uses this table internally to map a VBLANK value to a \n> EXPOSURE min/max limit. This could likely be done in a sensor helper \n> class.\n\nI agree with part of your arguments above. Duplicating the same code in\nmultiple places is usually a sign that something goes wrong. That could\nbe the case here too. Taking a step back, it's important to understand\nwhy we perform the same calculations in kernel drivers and in libcamera:\n\n- In the kernel, the boundary calculations are used to ensure invalid\n register values will be adjusted. This is more for safety than\n correctness, even if most sensors will probably recover easily from\n invalid exposure time values.\n\n- In libcamera, the calculations are meant for correctness of\n algorithms. We need to perform exposure time calculations every frame,\n and going through one (or more) round-trip to the kernel will impact\n performance.\n\nThere's a case to be made for reporting from kernel drivers parameters\nthat libcamera needs. We're doing so today in a way that splits data\nbetween the kernel and userspace, and uses all kinds of heuristics. This\nis the result of a UAPI that doesn't really match the needs of\nuserspace. I'm not blaming the developers of the V4L2 control API, it's\nold and was designed for different types of devices and different\npurposes than what we're dealing with now, but it's nonetheless a bad\nmatch.\n\nI believe we can improve the UAPI to better suit our needs. That will be\nlots of work, and we can't wait for that to be complete before fixing\nthe issue that this patch series addresses. Not to mention that we'll\nneed a very motivated champion to make this happen :-)\n\n> > > I think I could improve the here proposed setLimit() interface, as\n> > > maxExposure is only used to signal when a fixed exposure should be\n> > > used.\n> > > \n> > > I could pass the min exposure to Agc::configure() and only update the\n> > > frame duration in setLimit() with an optional \"fixed exposure\"\n> > > parameter in case a fixed manual exposure is desired...\n> > > \n> > > >\n> > > > This just mixes two ways of doing the same thing and is very confusing\n> > > > when reading the code. So if doing this in user-space is the way forward\n> > > > I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> > > > in IPARkISP1::configure() and instead use this method there too.\n> > > >\n> > > > I also feel, a little less strongly as it's a big work item, that it's\n> > > > really wrong to duplicate the calculation both in kernel and user-space.\n> > > \n> > > That I don't agree for the reasons explained already. This is not only\n> > > about the kernel interface, it's the IPA internal calculation that\n> > > need to know what the current exposure limit (in real-time) to\n> > > regulate itself and know how to split the desired exposure between\n> > > shutter time and gains. It could even be considered an internal IPA\n> > > parameter..\n> > > \n> > > >\n> > > > As this IMHO exposes a weakness in the V4L2 API we should think about\n> > > > how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> > > > we really want it to report its min/max values based on the current\n> > > > V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> > > > for each frame.\n> > > >\n> > > > Can we simplify the control to report the min and max values the sensor\n> > > > can do provided the VBLANK setting allows it, and just reject any value\n> > > > at set time that do not conform to the current VBLANK setting?\n> > > \n> > > Won't the v4l2 control framework would need a min and a max to adjust the\n> > > value passed to the driver ?\n> > > \n> > > >\n> > > > If so we can stop doing stuff in sensor drivers such as in IMX219 to\n> > > > allow for this control dependency in each driver.\n> > > >\n> > > > static int imx219_set_ctrl(struct v4l2_ctrl *ctrl)\n> > > > {\n> > > > ...\n> > > >\n> > > > if (ctrl->id == V4L2_CID_VBLANK) {\n> > > > \tint exposure_max, exposure_def;\n> > > >\n> > > > \t/* Update max exposure while meeting expected vblanking */\n> > > > \texposure_max = format->height + ctrl->val - 4;\n> > > > \texposure_def = (exposure_max < IMX219_EXPOSURE_DEFAULT) ?\n> > > > \t\texposure_max : IMX219_EXPOSURE_DEFAULT;\n> > > > \t__v4l2_ctrl_modify_range(imx219->exposure,\n> > > > \t\t\t\t imx219->exposure->minimum,\n> > > > \t\t\t\t exposure_max, imx219->exposure->step,\n> > > > \t\t\t\t exposure_def);\n> > > > }\n> > > >\n> > > > ...\n> > > > }\n> > > >\n> > > > >\n> > > > > Question: is there code somewhere to guarantee that when delayed\n> > > > > controls changes vblank it applies the vblank before the exposure?\n> > > > >\n> > > > > IIRC libcamera uses set-ext-ctrls to set them all at once, but\n> > > > > most (all?) sensor drivers do not implement this, so the kernel\n> > > > > just applies them 1-by-1 internally. And if vblank gets increased\n> > > > > only after exposure then for one frame exposure will still get\n> > > > > clamped based on the old vblank.\n> > > > >\n> > > > > This problem actually sounds like a more likely cause for Stefan's\n> > > > > oscillation issue you mention below.\n> > > > >\n> > > > > >> One could even calculate the exposure-margin by calculating the configured\n> > > > > >> frame-time (in a unit of lines) and then substracting the max-exposure\n> > > > > >> reported by the sensor from the frame-time-in-lines.\n> > > > > >\n> > > > > > Stefan suggested that too.\n> > > > > >\n> > > > > > When you configure() you take the sensor's exposure time and the frame\n> > > > > > length and the diff between the two is the margin. However there is no\n> > > > > > indication anywhere that the sensor driver has to initialize controls\n> > > > > > in this way and considering how slightly different sensor drivers are\n> > > > > > I wouldn't be on them being consistent.\n> > > > >\n> > > > > Hmm, I would say that we need to fix the sensor drivers then. But since\n> > > > > we need the camerahelper for the analog-gain bits I guess that having\n> > > > > to add the exposure vs framelength margin is not a big problem.\n> > > > >\n> > > > > >>> As the AgcMeanLuminance algorithm tries to push the exposure time up\n> > > > > >>> to the maximum allowed frame duration, initializing the algorithm\n> > > > > >>> with the sensor's maximum capabilities results in a very low frame rate,\n> > > > > >>> unless the user specifies a FrameDurationLimits range in which the\n> > > > > >>> algorithm should operate. To avoid slowing the frame rate down too much\n> > > > > >>> compute a \"default\" frame rate at algorithm configuration time, were\n> > > > > >>> the canonical 30, 15 and 10 FPS values are tested and selected in order\n> > > > > >>> of preference.\n> > > > > >>\n> > > > > >> I must admit I'm not familiar with the AgcMeanLuminance algorithm.\n> > > > > >> Does this algorithm also set / used to set the vblank control to\n> > > > > >> the maximum allowed frame duration? Or does it dynamically adjust\n> > > > > >> vblank when it needs more exposure \"range\" ?\n> > > > > >\n> > > > > > It didn't, as it didn't calculate the frame duration.\n> > > > > >\n> > > > > > The only IPA that use AgcMeanLuminance and handles VBLANK is the\n> > > > > > RkISP1 and as you can see in 09/10 the way it does that is to take\n> > > > > > the exposure time calculated by AgcMeanLuminance::calculateNewEV()\n> > > > > > and use it as the desired frame duration. This is wrong as it breaks\n> > > > > > the assumption we have a safe margin. It only doesn't break as the\n> > > > > > driver will clamp exposure once VBLANK is applied, but I presume this\n> > > > > > also causes a mis-alignment between what the algorithm think the\n> > > > > > exposure time is and what is actually applied to the device (and I\n> > > > > > presume this is what Stefan has reported causing oscillations when I\n> > > > > > naively proposed to ignore the margin and let the driver adjust\n> > > > > > exposure).\n> > > > >\n> > > > > See above, applying exposure before an increased vblank will lead\n> > > > > to a much bigger clamp which is a more likely cause of oscillation.\n> > > > >\n> > > > > > This is the reason for 09/10: the AgcMeanLuminance now calculates the\n> > > > > > duration as well using the (exposure time it has computed + margin) and\n> > > > > > sends it back to the IPA that transforms it in a proper vblank value\n> > > > > > to be eventually set on the device.\n> > > > > >\n> > > > > >>\n> > > > > >> I've been thinking about adding support to dynamically increase\n> > > > > >> vblank when more exposure time is needed in the softISP AGC code,\n> > > > > >> but it would probably be better to first convert that over to\n> > > > > >> the shared AGC helpers. I know that Kieran is working on this.\n> > > > > >\n> > > > > > Yes, I think that's a desirable goal.\n> > > > > >\n> > > > > > My whole effort in this series has been around moving as much as\n> > > > > > possible of the logic to the shared algorithm so that all IPA using it\n> > > > > > will get a consistent behaviour from \"free\".\n> > > > >\n> > > > > Ack.\n> > > > >\n> > > > > Thank you for taking the time to answer all my questions.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9628BC3241\n\tfor ;\n\tSun, 2 Nov 2025 19:23:02 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E3FCA6096B;\n\tSun, 2 Nov 2025 20:23:01 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A9C9D606E6\n\tfor ;\n\tSun, 2 Nov 2025 20:23:00 +0100 (CET)","from pendragon.ideasonboard.com (82-203-160-149.bb.dnainternet.fi\n\t[82.203.160.149])\n\tby perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 017058D4;\n\tSun, 2 Nov 2025 20:21:07 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"t90WZEPK\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1762111268;\n\tbh=bFv+Ljcc5MMnC2NFFDjPu+5dqT3tylGkEGUIGo2TV4k=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=t90WZEPKpGoQgRw3ZXg1Rsr1MBGNA7U+yEZV/Utcp5ohqy9cPoU8V4YRCe7gozAst\n\t8eP0iIDSQ3eFGdiTPhTZibMAsUk+JEGrCvnlfY9Opcz+/6RpYw5UTDs3l4VG/0/Xxp\n\tmx60itstaxNDHj9aoeQXAuQbwIJjVL6d8DMoDiR8=","Date":"Sun, 2 Nov 2025 21:22:46 +0200","From":"Laurent Pinchart ","To":"Niklas =?utf-8?q?S=C3=B6derlund?= ","Cc":"Jacopo Mondi ,\n\tHans de Goede ,\n\tRobert Mader ,\n\tlibcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"<20251102192246.GE27255@pendragon.ideasonboard.com>","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<54xd52swj5ftm6sbv6habzmjzojbqle6u4kvt442mdvz3ay74u@c4tkaqvly5pf>\n\t<20251102160419.GK365372@ragnatech.se>\n\t<20251102164223.GL365372@ragnatech.se>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20251102164223.GL365372@ragnatech.se>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36634,"web_url":"https://patchwork.libcamera.org/comment/36634/","msgid":"<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>","date":"2025-11-03T09:01:15","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":239,"url":"https://patchwork.libcamera.org/api/people/239/","name":"Hans de Goede","email":"hansg@kernel.org"},"content":"Hi,\n\nOn 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n\n...\n\n>> Right I forgot about the delayed-controls stuff, not being able\n>> to get the new exposure limits from the kernel then makes sense.\n> \n> I too understand why it's easier to not consult the kernel for the new \n> limits. I have no real strong feeling that the correct solution is to \n> have this limit calculation done in the kernel or user-space. What I do \n> feel somewhat strongly about is if we do the limit calculation in \n> user-space we should never consult the kernel for the limits.\n> \n> This just mixes two ways of doing the same thing and is very confusing \n> when reading the code. So if doing this in user-space is the way forward \n> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE \n> in IPARkISP1::configure() and instead use this method there too.\n> \n> I also feel, a little less strongly as it's a big work item, that it's \n> really wrong to duplicate the calculation both in kernel and user-space.\n> \n> As this IMHO exposes a weakness in the V4L2 API we should think about \n> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do \n> we really want it to report its min/max values based on the current \n> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL \n> for each frame.\n> \n> Can we simplify the control to report the min and max values the sensor \n> can do provided the VBLANK setting allows it, and just reject any value \n> at set time that do not conform to the current VBLANK setting?\n\nOne complicating factor here is that some sensors will do vblank stretching\nwhen you ask for a higher exposure. IOW some sensors behave as if a bigger\nvblank is set when setting exposure > (frame-length + margin) to still\ngive you the desired exposure.\n\nATM I think that for all new sensor drivers we try to avoid this auto\nvblank increase by clamping the exposure range.\n\nIf we don't clamp the exposure range, then we will get inconsistent\nbehavior between sensors.\n\nWe could only clamp the value when written and keep the advertised\nrange the same though, but that feels wrong. Arguably allowing vblank\nstretch would be nice to do, then normally the IPA will keep\nthe exposure within the frame-length - exposure range, but it could\nthen do higher exposures without needed to reprogram vblank, but\nonly one some sensors ...\n\nRegards,\n\nHans","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E5E62BDE4C\n\tfor ;\n\tMon, 3 Nov 2025 09:01:23 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C89E9609D8;\n\tMon, 3 Nov 2025 10:01:22 +0100 (CET)","from tor.source.kernel.org (tor.source.kernel.org\n\t[IPv6:2600:3c04:e001:324:0:1991:8:25])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4B4D960805\n\tfor ;\n\tMon, 3 Nov 2025 10:01:20 +0100 (CET)","from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58])\n\tby tor.source.kernel.org (Postfix) with ESMTP id 7399460008;\n\tMon, 3 Nov 2025 09:01:18 +0000 (UTC)","by smtp.kernel.org (Postfix) with ESMTPSA id 09FCCC4CEE7;\n\tMon, 3 Nov 2025 09:01:16 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=kernel.org header.i=@kernel.org\n\theader.b=\"XTog/OhN\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;\n\ts=k20201202; t=1762160478;\n\tbh=7PvXHQLHB5sM0U3Ewl7MVGTl8MU7bhuLKLIDG5VnkI0=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=XTog/OhNL8Lfz1t1SaQgzFF8Y5ZG3gzYw+6oIZBD3+guOWmFDa7ejaOgIYA4u7czw\n\tSaopM558lSXyOa2geXXRKEr8iMRGjAiQR9+SnWmxPG443/iRyV/TfVtzJqkGwU29A7\n\tSJ4fS01as/UmrgRP7CF1cZzNBQzL+DE0BTZgN90Woh22/PRpsBCcAYZEsBIySd0NZr\n\tFOKiAt7oFJUoC34rY1X08+b25o8janj9//l1UuviE6nNtmnd8HjviGTfzJKeCCfZu8\n\t6YIJr8+IqJk03YchNO7oDIrQxXqj4iUkkhVh2Yy0fFe5N0fRVEEyEND08PCLdH1L6t\n\tRuilahKsgNifQ==","Message-ID":"<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>","Date":"Mon, 3 Nov 2025 10:01:15 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","Cc":"Jacopo Mondi ,\n\tRobert Mader ,\n\tlibcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>","From":"Hans de Goede ","Content-Language":"en-US, nl","In-Reply-To":"<20251028153842.GC365372@ragnatech.se>","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36681,"web_url":"https://patchwork.libcamera.org/comment/36681/","msgid":"<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>","date":"2025-11-04T15:53:04","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Hans\n\nOn Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n> Hi,\n>\n> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n> > On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n>\n> ...\n>\n> >> Right I forgot about the delayed-controls stuff, not being able\n> >> to get the new exposure limits from the kernel then makes sense.\n> >\n> > I too understand why it's easier to not consult the kernel for the new\n> > limits. I have no real strong feeling that the correct solution is to\n> > have this limit calculation done in the kernel or user-space. What I do\n> > feel somewhat strongly about is if we do the limit calculation in\n> > user-space we should never consult the kernel for the limits.\n> >\n> > This just mixes two ways of doing the same thing and is very confusing\n> > when reading the code. So if doing this in user-space is the way forward\n> > I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> > in IPARkISP1::configure() and instead use this method there too.\n> >\n> > I also feel, a little less strongly as it's a big work item, that it's\n> > really wrong to duplicate the calculation both in kernel and user-space.\n> >\n> > As this IMHO exposes a weakness in the V4L2 API we should think about\n> > how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> > we really want it to report its min/max values based on the current\n> > V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> > for each frame.\n> >\n> > Can we simplify the control to report the min and max values the sensor\n> > can do provided the VBLANK setting allows it, and just reject any value\n> > at set time that do not conform to the current VBLANK setting?\n>\n> One complicating factor here is that some sensors will do vblank stretching\n> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n> vblank is set when setting exposure > (frame-length + margin) to still\n> give you the desired exposure.\n\nInteresting. Do you have any pointer to mainline drivers that do that ?\n\n>\n> ATM I think that for all new sensor drivers we try to avoid this auto\n> vblank increase by clamping the exposure range.\n>\n> If we don't clamp the exposure range, then we will get inconsistent\n> behavior between sensors.\n>\n> We could only clamp the value when written and keep the advertised\n> range the same though, but that feels wrong. Arguably allowing vblank\n> stretch would be nice to do, then normally the IPA will keep\n> the exposure within the frame-length - exposure range, but it could\n> then do higher exposures without needed to reprogram vblank, but\n> only one some sensors ...\n>\n> Regards,\n>\n> Hans\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DE95CBDE4C\n\tfor ;\n\tTue, 4 Nov 2025 15:53:08 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 33FC560A80;\n\tTue, 4 Nov 2025 16:53:08 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 42D8C6069A\n\tfor ;\n\tTue, 4 Nov 2025 16:53:07 +0100 (CET)","from ideasonboard.com (93-61-96-190.ip145.fastwebnet.it\n\t[93.61.96.190])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 782381771;\n\tTue, 4 Nov 2025 16:51:13 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"rKAxT+VS\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1762271473;\n\tbh=TDffe8NVmONtx8VidnqXXaZZPn5acNV0sFaOKDaVDKA=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=rKAxT+VSet0klaTlrb74o8nvonRBIbUhyXI8Vvmk1ZO1kkqVBb5AS7NAFiKpCAkre\n\tZTX0Elhczdt2ZLqtsFpycq6yYsKfN/SXosVM3Ii3JA+bkOpmqJM+nKUUzGr+Secfmf\n\t6FXo0/sn2+A7W4TniFJG8BPtRRkoF4RR/bYPA3Zs=","Date":"Tue, 4 Nov 2025 16:53:04 +0100","From":"Jacopo Mondi ","To":"Hans de Goede ","Cc":"Niklas =?utf-8?q?S=C3=B6derlund?= ,\n\tJacopo Mondi , Robert Mader\n\t, libcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36687,"web_url":"https://patchwork.libcamera.org/comment/36687/","msgid":"<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>","date":"2025-11-04T17:23:23","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":239,"url":"https://patchwork.libcamera.org/api/people/239/","name":"Hans de Goede","email":"hansg@kernel.org"},"content":"Hi Jacopo,\n\nOn 4-Nov-25 16:53, Jacopo Mondi wrote:\n> Hi Hans\n> \n> On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n>> Hi,\n>>\n>> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n>>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n>>\n>> ...\n>>\n>>>> Right I forgot about the delayed-controls stuff, not being able\n>>>> to get the new exposure limits from the kernel then makes sense.\n>>>\n>>> I too understand why it's easier to not consult the kernel for the new\n>>> limits. I have no real strong feeling that the correct solution is to\n>>> have this limit calculation done in the kernel or user-space. What I do\n>>> feel somewhat strongly about is if we do the limit calculation in\n>>> user-space we should never consult the kernel for the limits.\n>>>\n>>> This just mixes two ways of doing the same thing and is very confusing\n>>> when reading the code. So if doing this in user-space is the way forward\n>>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n>>> in IPARkISP1::configure() and instead use this method there too.\n>>>\n>>> I also feel, a little less strongly as it's a big work item, that it's\n>>> really wrong to duplicate the calculation both in kernel and user-space.\n>>>\n>>> As this IMHO exposes a weakness in the V4L2 API we should think about\n>>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n>>> we really want it to report its min/max values based on the current\n>>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n>>> for each frame.\n>>>\n>>> Can we simplify the control to report the min and max values the sensor\n>>> can do provided the VBLANK setting allows it, and just reject any value\n>>> at set time that do not conform to the current VBLANK setting?\n>>\n>> One complicating factor here is that some sensors will do vblank stretching\n>> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n>> vblank is set when setting exposure > (frame-length + margin) to still\n>> give you the desired exposure.\n> \n> Interesting. Do you have any pointer to mainline drivers that do that ?\n\nI'm not talking about the driver stretching the vblank I'm talking\nabout the hw (the sensor) stretching the vblank. I think that in newer\ndrivers we stop this from happening by clamping the exposure.\n\nI'm 100% certain that some hw does this as I've seen this happen.\nI can do some digging if you want some specific sensor model examples\nwhere the hw is doing this.\n\nRegards,\n\nHans","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 4027ABDE4C\n\tfor ;\n\tTue, 4 Nov 2025 17:23:31 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 960B760856;\n\tTue, 4 Nov 2025 18:23:30 +0100 (CET)","from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4879A6069A\n\tfor ;\n\tTue, 4 Nov 2025 18:23:28 +0100 (CET)","from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58])\n\tby tor.source.kernel.org (Postfix) with ESMTP id DDC16601F5;\n\tTue, 4 Nov 2025 17:23:26 +0000 (UTC)","by smtp.kernel.org (Postfix) with ESMTPSA id 5C5B0C4CEF8;\n\tTue, 4 Nov 2025 17:23:25 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=kernel.org header.i=@kernel.org\n\theader.b=\"INR4QIsN\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;\n\ts=k20201202; t=1762277006;\n\tbh=C28ht2VSD9oUl7XqNg+JAqp0HkQ/CnBHP2WPkrTU0vo=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=INR4QIsNyn6PL8SiF92soyhCSPhkd/Qz2H+8Uu6AjgSih9Cuk+uzzNFxsyvPwevId\n\tOvGel6+vcXMoFpXNrq16/EmwOifo9aw3YT1PoFWdk1lMMwMl81ZMy/qsQMIjLng7wB\n\t9fYvB3HicZ3m2BEtB7GqrdB1HnA7Xa89TuSH4b6SNVcVcfwy5ECTL4X9ST7shyNvSR\n\tUBdxuL3zuHN/NaETzW3QQNa0BYYNjSaRNEu/uX/+gRSrOFtJHbnaMfrENTav3/NWty\n\tO/w3qaQEXTTXsxuGOyIYieVpIFPtOPmEhqjdIF7nqDXtJ3DfpIqdMhHp3pVZwQexdI\n\tiun2Llfcd2sXQ==","Message-ID":"<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>","Date":"Tue, 4 Nov 2025 18:23:23 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Jacopo Mondi ","Cc":"=?utf-8?q?Niklas_S=C3=B6derlund?= ,\n\tRobert Mader , \n\tlibcamera-devel@lists.libcamera.org, Kieran Bingham\n\t","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>","From":"Hans de Goede ","Content-Language":"en-US, nl","In-Reply-To":"<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36704,"web_url":"https://patchwork.libcamera.org/comment/36704/","msgid":"","date":"2025-11-05T10:50:43","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi,\n\nOn Tue, 4 Nov 2025 at 17:23, Hans de Goede wrote:\n>\n> Hi Jacopo,\n>\n> On 4-Nov-25 16:53, Jacopo Mondi wrote:\n> > Hi Hans\n> >\n> > On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n> >> Hi,\n> >>\n> >> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n> >>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> >>\n> >> ...\n> >>\n> >>>> Right I forgot about the delayed-controls stuff, not being able\n> >>>> to get the new exposure limits from the kernel then makes sense.\n> >>>\n> >>> I too understand why it's easier to not consult the kernel for the new\n> >>> limits. I have no real strong feeling that the correct solution is to\n> >>> have this limit calculation done in the kernel or user-space. What I do\n> >>> feel somewhat strongly about is if we do the limit calculation in\n> >>> user-space we should never consult the kernel for the limits.\n> >>>\n> >>> This just mixes two ways of doing the same thing and is very confusing\n> >>> when reading the code. So if doing this in user-space is the way forward\n> >>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> >>> in IPARkISP1::configure() and instead use this method there too.\n> >>>\n> >>> I also feel, a little less strongly as it's a big work item, that it's\n> >>> really wrong to duplicate the calculation both in kernel and user-space.\n> >>>\n> >>> As this IMHO exposes a weakness in the V4L2 API we should think about\n> >>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> >>> we really want it to report its min/max values based on the current\n> >>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> >>> for each frame.\n> >>>\n> >>> Can we simplify the control to report the min and max values the sensor\n> >>> can do provided the VBLANK setting allows it, and just reject any value\n> >>> at set time that do not conform to the current VBLANK setting?\n> >>\n> >> One complicating factor here is that some sensors will do vblank stretching\n> >> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n> >> vblank is set when setting exposure > (frame-length + margin) to still\n> >> give you the desired exposure.\n> >\n> > Interesting. Do you have any pointer to mainline drivers that do that ?\n>\n> I'm not talking about the driver stretching the vblank I'm talking\n> about the hw (the sensor) stretching the vblank. I think that in newer\n> drivers we stop this from happening by clamping the exposure.\n>\n> I'm 100% certain that some hw does this as I've seen this happen.\n> I can do some digging if you want some specific sensor model examples\n> where the hw is doing this.\n\nWe have worked with sensors (e.g. IMX477 and IMX708) that support\nauto-increasing the frame length / VBLANK if the exposure time\nstretches past the current frame length. However, we have turned off\nthis feature in the sensor and handle it manually like we do for all\nother sensors.\n\nNaush\n\n\n>\n> Regards,\n>\n> Hans\n>\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7C851C3241\n\tfor ;\n\tWed, 5 Nov 2025 10:51:25 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A489E608CF;\n\tWed, 5 Nov 2025 11:51:24 +0100 (CET)","from mail-vk1-xa2b.google.com (mail-vk1-xa2b.google.com\n\t[IPv6:2607:f8b0:4864:20::a2b])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 54653608CF\n\tfor ;\n\tWed, 5 Nov 2025 11:51:22 +0100 (CET)","by mail-vk1-xa2b.google.com with SMTP id\n\t71dfb90a1353d-5598dff5ba7so10080e0c.1\n\tfor ;\n\tWed, 05 Nov 2025 02:51:22 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"QqpE8uwv\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1762339881; x=1762944681;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=umj6YsJPQb+35sjfl87JvLWYxclQACTjQpE2/QLj/FA=;\n\tb=QqpE8uwv0Qfky7WeBufEMN54LNDuVq5vAH++j74BtVNRZIV3P+5uZsJ+3g5pMd7kXd\n\tgQKNyJO9pMXs8Ny1IjqT22je/eieFRxGeAd+/NcD4DvKNiPxqcwO8U2qy6/qXCPXb2hB\n\twDMOJjqWThP1nftmsnN5joScWkZEL2yB668nPg5TaJY33IMof/uhuTPeWiXO9/gxGkz2\n\tu60GEanxD2AQ9IPgWAjEL2N/CuTyqeFulHpPs81KU4jzT4nusnLrjC8YLgDzAn0xz5gX\n\tMaBwvi9AwoKkVUIF3XtpzUDYXopWHwyFuJZ8ZWhuR2yiBRBSs2fTILeyMfwDWjP1AroI\n\tG2Kg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1762339881; x=1762944681;\n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=umj6YsJPQb+35sjfl87JvLWYxclQACTjQpE2/QLj/FA=;\n\tb=mKKbXpIs1E0r0oRw7JEmQhPVHZgCpezgVyRLOKXbOp8jg0JMWh94cTZQhekToBYLrR\n\tFQo66yywsvAhIOydVpmWgIzT6pbTi//YdzY3nNsJapxIvkOnFilchDsKTp+ztcfQ+VpN\n\tqJHAvG1bo4UjPejVFi/8gQmDkOetASmiFO2BzbyzrTdTMS3qVn1Y3CFEkcH8iPOEHp9/\n\tjZssmLd66Yti3+QmuO9f/krckPNMcL/FupVsDUdRFTuyB1/7MTTA7em4mvOcbILFcEpE\n\t70GBCdkIZXkWkMFyUPwzoqSOtoyYeu8b0n0DQaU9rdGCb5/LBXDXX9w6Rieh0PI1ZBmD\n\tUUWg==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCWdp/2CcRiL7OGuilHXvr0qdkYuEo37e1GVbvQV73MQt9du2GeBne6iaKAwKRXTLGYTfU3gaMa/A0FIT7rFuWE=@lists.libcamera.org","X-Gm-Message-State":"AOJu0Yw9rQ0ZPjYQjZinXu3JG0rwuhR4jyHMw2aNOoIry8rlhFbOm2Tb\n\tV0zOyG8WCpmX1k0IqxmFQ39bEJU79VxUsjSHQ7qtoKACfFE4OeHhjiOHhIm8+OB1mgDvsxFi5ys\n\t5Qhcp3TF3iSnPreBtQXLwl3hbS0nKsf5ICoUqJLTLE6NaoK5ZVxWD","X-Gm-Gg":"ASbGncualPdCBH4ezrRL+JA8mAyncJutFNdAZMWnFO3nhRIFuN16h0pOYdEFLJkg4Cb\n\tA6FixBLaBA3/E8NLrS9EouUys9LHGBBG5P861cuE3Hb1BeJA05FzgJvsp0yJqBX9VTdauLrkdkX\n\tVBmQp8SaPFl+W04OUXBi9P+cb9BcdmNQ7Hu1M+IaFa+xKaKRW0K4+FJKYFZkQ0lKP/mHESuAPXs\n\tGTxlkPI+BX13AA5U+/DxU1kmghmtP07tUYScVRr1e1I6XdgLS92ybcPnlOfWM/jLTlmizOZAflm\n\tvcF8NVmPMPg0HSIUZmlKQrZzESw=","X-Google-Smtp-Source":"AGHT+IGyR7vO2URJhRFL8ZGQsEVhSNQpSRpeDn9JCx76PKHRWRBUwGXreQNkCiYBajjzXuDVLoEwCWoHWyofp3Q98cE=","X-Received":"by 2002:a05:6102:4412:b0:5db:d2b7:952a with SMTP id\n\tada2fe7eead31-5dd891078dbmr395845137.2.1762339880724; Wed, 05 Nov 2025\n\t02:51:20 -0800 (PST)","MIME-Version":"1.0","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>\n\t<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>","In-Reply-To":"<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>","From":"Naushir Patuck ","Date":"Wed, 5 Nov 2025 10:50:43 +0000","X-Gm-Features":"AWmQ_bmhg7umjVdqdhOeRfNwOAaXnQKaexbN8eNEpnl6ePnEVwkSuC-G03uFqfo","Message-ID":"","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Hans de Goede ","Cc":"Jacopo Mondi , =?utf-8?q?Niklas_S?=\n\t=?utf-8?b?w7ZkZXJsdW5k?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org, \n\tKieran Bingham ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36723,"web_url":"https://patchwork.libcamera.org/comment/36723/","msgid":"","date":"2025-11-06T08:29:05","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Hans\n\nOn Tue, Nov 04, 2025 at 06:23:23PM +0100, Hans de Goede wrote:\n> Hi Jacopo,\n>\n> On 4-Nov-25 16:53, Jacopo Mondi wrote:\n> > Hi Hans\n> >\n> > On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n> >> Hi,\n> >>\n> >> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n> >>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> >>\n> >> ...\n> >>\n> >>>> Right I forgot about the delayed-controls stuff, not being able\n> >>>> to get the new exposure limits from the kernel then makes sense.\n> >>>\n> >>> I too understand why it's easier to not consult the kernel for the new\n> >>> limits. I have no real strong feeling that the correct solution is to\n> >>> have this limit calculation done in the kernel or user-space. What I do\n> >>> feel somewhat strongly about is if we do the limit calculation in\n> >>> user-space we should never consult the kernel for the limits.\n> >>>\n> >>> This just mixes two ways of doing the same thing and is very confusing\n> >>> when reading the code. So if doing this in user-space is the way forward\n> >>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> >>> in IPARkISP1::configure() and instead use this method there too.\n> >>>\n> >>> I also feel, a little less strongly as it's a big work item, that it's\n> >>> really wrong to duplicate the calculation both in kernel and user-space.\n> >>>\n> >>> As this IMHO exposes a weakness in the V4L2 API we should think about\n> >>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> >>> we really want it to report its min/max values based on the current\n> >>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> >>> for each frame.\n> >>>\n> >>> Can we simplify the control to report the min and max values the sensor\n> >>> can do provided the VBLANK setting allows it, and just reject any value\n> >>> at set time that do not conform to the current VBLANK setting?\n> >>\n> >> One complicating factor here is that some sensors will do vblank stretching\n> >> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n> >> vblank is set when setting exposure > (frame-length + margin) to still\n> >> give you the desired exposure.\n> >\n> > Interesting. Do you have any pointer to mainline drivers that do that ?\n>\n> I'm not talking about the driver stretching the vblank I'm talking\n> about the hw (the sensor) stretching the vblank. I think that in newer\n> drivers we stop this from happening by clamping the exposure.\n>\n> I'm 100% certain that some hw does this as I've seen this happen.\n> I can do some digging if you want some specific sensor model examples\n> where the hw is doing this.\n\nI think the fact some sensor supporting this feature not an issue if\nwe operate with algorithms designed to always clamp the shutter time\nto the frame duration. Hence I would be more interested in knowing\nwhich usage model you would consider best for the whole\nlibcamera/libipa handling of exposure/frame duration.\n\nCurrently, the design is to have the maximum achievable shutter time\nto be always constrained by the frame duration. Users should take\ncare of explicitly slowing down the camera frame rate in order to\nachieve a larger shutter time.\n\nI think it is justified by the face that the desired exposure is\nrealized by the combined effect of the shutter time and gains, and\nhence limiting the shutter time won't imply not being able to reach\nthe desired exposure levels.\n\nAlternative approaches where the exposure time drives the frame\nduration could be implemented with helpers, similar to the already\ndiscussed \"aperture priority\" vs \"shutter priority\" modes.\n\nWhat we have to make sure is the controls we define to driver the AGC\nallows us to create such helpers easily.\n\nDo you think we're on the same page here ?\n\nThanks\n j\n\n>\n> Regards,\n>\n> Hans\n>\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2AF7DC3241\n\tfor ;\n\tThu, 6 Nov 2025 08:29:11 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 361D860A80;\n\tThu, 6 Nov 2025 09:29:10 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B795D6096B\n\tfor ;\n\tThu, 6 Nov 2025 09:29:08 +0100 (CET)","from ideasonboard.com (93-61-96-190.ip145.fastwebnet.it\n\t[93.61.96.190])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 9FE84593;\n\tThu, 6 Nov 2025 09:27:13 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"HXla2PL5\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1762417633;\n\tbh=XUyP4HvLoFryj4tKAc56mLs7sqVsAM9XXBXpemUZoR8=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=HXla2PL5sh6PHCektGpLj3x4K1MQWXagGvrevT2wCJoHrjJWGG8IB5oU+p+L1XXiM\n\tfKzdlOC0v509k8yO20bG9t73mCxcQ+mHIyZTxk2+PQXe1hgg0nIgL3l5GN+I6VNomM\n\tfIq08sCk/J/kVAn/B7GZsZuov8+vkdgR1OqQ7/6I=","Date":"Thu, 6 Nov 2025 09:29:05 +0100","From":"Jacopo Mondi ","To":"Hans de Goede ","Cc":"Jacopo Mondi , Niklas =?utf-8?b?U8O2?=\n\t=?utf-8?q?derlund?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","Message-ID":"","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>\n\t<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36754,"web_url":"https://patchwork.libcamera.org/comment/36754/","msgid":"<05eed1a8-8de8-49e3-9d65-44aa86d35d45@oss.qualcomm.com>","date":"2025-11-07T16:41:45","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":242,"url":"https://patchwork.libcamera.org/api/people/242/","name":"Hans de Goede","email":"johannes.goede@oss.qualcomm.com"},"content":"Hi Jacopo,\n\nOn 6-Nov-25 9:29 AM, Jacopo Mondi wrote:\n> Hi Hans\n> \n> On Tue, Nov 04, 2025 at 06:23:23PM +0100, Hans de Goede wrote:\n>> Hi Jacopo,\n>>\n>> On 4-Nov-25 16:53, Jacopo Mondi wrote:\n>>> Hi Hans\n>>>\n>>> On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n>>>> Hi,\n>>>>\n>>>> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n>>>>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n>>>>\n>>>> ...\n>>>>\n>>>>>> Right I forgot about the delayed-controls stuff, not being able\n>>>>>> to get the new exposure limits from the kernel then makes sense.\n>>>>>\n>>>>> I too understand why it's easier to not consult the kernel for the new\n>>>>> limits. I have no real strong feeling that the correct solution is to\n>>>>> have this limit calculation done in the kernel or user-space. What I do\n>>>>> feel somewhat strongly about is if we do the limit calculation in\n>>>>> user-space we should never consult the kernel for the limits.\n>>>>>\n>>>>> This just mixes two ways of doing the same thing and is very confusing\n>>>>> when reading the code. So if doing this in user-space is the way forward\n>>>>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n>>>>> in IPARkISP1::configure() and instead use this method there too.\n>>>>>\n>>>>> I also feel, a little less strongly as it's a big work item, that it's\n>>>>> really wrong to duplicate the calculation both in kernel and user-space.\n>>>>>\n>>>>> As this IMHO exposes a weakness in the V4L2 API we should think about\n>>>>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n>>>>> we really want it to report its min/max values based on the current\n>>>>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n>>>>> for each frame.\n>>>>>\n>>>>> Can we simplify the control to report the min and max values the sensor\n>>>>> can do provided the VBLANK setting allows it, and just reject any value\n>>>>> at set time that do not conform to the current VBLANK setting?\n>>>>\n>>>> One complicating factor here is that some sensors will do vblank stretching\n>>>> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n>>>> vblank is set when setting exposure > (frame-length + margin) to still\n>>>> give you the desired exposure.\n>>>\n>>> Interesting. Do you have any pointer to mainline drivers that do that ?\n>>\n>> I'm not talking about the driver stretching the vblank I'm talking\n>> about the hw (the sensor) stretching the vblank. I think that in newer\n>> drivers we stop this from happening by clamping the exposure.\n>>\n>> I'm 100% certain that some hw does this as I've seen this happen.\n>> I can do some digging if you want some specific sensor model examples\n>> where the hw is doing this.\n> \n> I think the fact some sensor supporting this feature not an issue if\n> we operate with algorithms designed to always clamp the shutter time\n> to the frame duration. Hence I would be more interested in knowing\n> which usage model you would consider best for the whole\n> libcamera/libipa handling of exposure/frame duration.\n> \n> Currently, the design is to have the maximum achievable shutter time\n> to be always constrained by the frame duration. Users should take\n> care of explicitly slowing down the camera frame rate in order to\n> achieve a larger shutter time.\n> \n> I think it is justified by the face that the desired exposure is\n> realized by the combined effect of the shutter time and gains, and\n> hence limiting the shutter time won't imply not being able to reach\n> the desired exposure levels.\n\nRight, except that in low light you may need to boost the gain\nto a level where you get significant noise and at some point\nthe gain is mostly just amplifying noise (discussion continued\nbelow).\n\n> Alternative approaches where the exposure time drives the frame\n> duration could be implemented with helpers, similar to the already\n> discussed \"aperture priority\" vs \"shutter priority\" modes.\n\nRight, I do believe that we really need these modes, at least\nfor laptops. Unlike the RPI cameras which have pretty nice large\nlenses. The laptop designs are optimized for small bezels and\nthus small lenses which means not a whole lot of light, which\nreally starts to bite at low light conditions.\n\nAnd I've seen both Intel's proprietary IPU6 stack as well as\nmany UVC webcams lower the framerate to compensate (somewhat).\n\nI'm fine with defaulting to never allowing the need for\nhigher exposure to lower the framerate, but as you say I do\nbelieve we need a control to allow the AGC lowering the framerate\nto get larger exposure times. And I think those controls should\nlikely even default to on devices with small lenses, but first\nwe need to have / implement such an exposure-mode control to begin\nwith :)\n\n> What we have to make sure is the controls we define to driver the AGC\n> allows us to create such helpers easily.\n> \n> Do you think we're on the same page here ?\n\nYes I believe that we are on the same page here.\n\nRegards,\n\nHans","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 64272BDE4C\n\tfor ;\n\tFri, 7 Nov 2025 16:41:52 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 91FDA60A81;\n\tFri, 7 Nov 2025 17:41:51 +0100 (CET)","from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com\n\t[205.220.168.131])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 79D7B609D8\n\tfor ;\n\tFri, 7 Nov 2025 17:41:50 +0100 (CET)","from pps.filterd (m0279866.ppops.net [127.0.0.1])\n\tby mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id\n\t5A7DPcVH3130550 for ;\n\tFri, 7 Nov 2025 16:41:48 GMT","from mail-qt1-f197.google.com (mail-qt1-f197.google.com\n\t[209.85.160.197])\n\tby mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4a9hn08hde-1\n\t(version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT)\n\tfor ;\n\tFri, 07 Nov 2025 16:41:48 +0000 (GMT)","by mail-qt1-f197.google.com with SMTP id\n\td75a77b69052e-4ed6466f2baso27198161cf.0\n\tfor ;\n\tFri, 07 Nov 2025 08:41:48 -0800 (PST)","from ?IPV6:2001:1c00:c32:7800:5bfa:a036:83f0:f9ec?\n\t(2001-1c00-0c32-7800-5bfa-a036-83f0-f9ec.cable.dynamic.v6.ziggo.nl.\n\t[2001:1c00:c32:7800:5bfa:a036:83f0:f9ec])\n\tby smtp.gmail.com with ESMTPSA id\n\ta640c23a62f3a-b72bf97d3c0sm303276066b.36.2025.11.07.08.41.45\n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tFri, 07 Nov 2025 08:41:46 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=qualcomm.com header.i=@qualcomm.com\n\theader.b=\"VqEAAoaf\"; dkim=pass (2048-bit key;\n\tunprotected) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com\n\theader.b=\"ca6yoeGc\"; dkim-atps=neutral","DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h=\n\tcc:content-transfer-encoding:content-type:date:from:in-reply-to\n\t:message-id:mime-version:references:subject:to; s=qcppdkim1; bh=\n\tR40mDjTY39vsqGDjLWUw+wNhTdUd0Jf7qhDUCZBEbsk=; b=VqEAAoaffDZfyTPE\n\tw+ijeRvDXKtlDsQd/H5z3K17LAIi6l4bTcz3ufGGBkwj3x5AdCxSC9GbIw9rT8KG\n\tuOqtbHiHQoiThypK34hSS4d+r+fpga21nbEXnPefYrDFCej2dTkSwlDH44JmQCDU\n\tajjBbpDBAL4HQaIYJvPPcpapUL2JF/cDfmp+6EpTxlSMFAnHo4b6fHHRiyri8Htm\n\t2n9tf+Ojcxa3mJRCVaN0lLueDHqLWNYssWnGadTqaQrSwA/fGFJi9BfV9Pz12hua\n\tfTt8QTbafkHwpY72LBc3nNTdwIYDbrCFQSvoreOM1vxRrBDP1oye09/WigZBosTu\n\trYKQ+w==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=oss.qualcomm.com; s=google; t=1762533707; x=1763138507;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:in-reply-to:content-language:references\n\t:cc:to:subject:from:user-agent:mime-version:date:message-id:from:to\n\t:cc:subject:date:message-id:reply-to;\n\tbh=R40mDjTY39vsqGDjLWUw+wNhTdUd0Jf7qhDUCZBEbsk=;\n\tb=ca6yoeGcKK7CVqlmFxYqEmHzpz8feK5yner/HRu56jXhrKyonADQaY9/V8yM52uNQp\n\tSGQ6+miXmjv4cKlWaZm80gQvfTrZ5v+OzJ6dkkYnv+CiJpMLCvsK0TasN0gHkEUPC0Kw\n\thC3cQkA8iQiJgkXI93oPHywXPuKbzg1lsGcTBgoaOZ38NWgOuIOerQ95McdVqEf3pBJz\n\tkDnm2hFm825vFKTEdg4/2PISGmVCzFqpNEj/P4a5W4EVUOHVOzHCaeYW2waJtu24FeKc\n\t0B7OESxbRsHIEZna8DMqsfM5kvnTYc18XUxzVAR8P3l1TqF7osRzP7CSMKpjucYKZmHH\n\tPpgg=="],"X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1762533707; x=1763138507;\n\th=content-transfer-encoding:in-reply-to:content-language:references\n\t:cc:to:subject:from:user-agent:mime-version:date:message-id:x-gm-gg\n\t:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n\tbh=R40mDjTY39vsqGDjLWUw+wNhTdUd0Jf7qhDUCZBEbsk=;\n\tb=WcbB5XgUYHkUR23ik62L4JBiYMNZPoueEqHyX8hpddEZ2eenvSBlphSQ5E+Iv3AwsQ\n\tDBf5aT5ratMzdQlkfQmdVin8AKsVqBhBx6TxUk+NzsqrxMxkdXqVTMVsMI0RZZYMrW7D\n\tWksyJLpDfTsqe+Qi9W7gXbjhXK8CBLcexhfCr45QR3GLEdJ7d8fIqfOJykoI3kTbsPOh\n\tXFRU80iXD3T9qCf2iG9HNzrFU/c9MfxpcIFl41JA/bg3MQSWIQHb8Mq3Av4BVUCUKjtH\n\tzyVVtApbmVZUlNYBVUICz/q7BlvxZrr6ypLazT+GKefjbQPxSkE0Qo6eRFJbgIQhWzfm\n\t71Mw==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCWfYAt1i9PViFj+RkGSsGvs5xhDBrDrZJLsWQuTGIS7zlTGfI5z21zZVZimyXwYulskkkWt2LLUzFPel3nyOfE=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YyJZkxCZ84LWAGopppr6aAjj31q0r4Kzt6WF8NS7cB/dGXn9Xkq\n\twNLiW5ndeq6L6GdGTGyhaDcHt0OQFLD+HsjZ5X6pAB7D+irE/hZOYchdSsm3Ol51qOVLulR9VTC\n\tEP7n3GMyrJJcU1A388vQIkpgCcuJgjXRuvGWzk0C3FYLCxGgtr0nZtKYFv/TPy1fRDI3j3Xikt4\n\tOCU8aW/YNr0RU=","X-Gm-Gg":"ASbGncsXxVj1CDI+bRUQgIS8BwgrN7cGJpoU60VjS3koCceKvz7oVYUW4n9iY0XP5zP\n\t0zbiGroZgMMOWnAtIg/Ro3xDoWiPpQar3Kc70Yd7+5IUy9AWlxrcnGWJAZ2x3Kuy4XXS6fV1PPD\n\tieA7lrqmw5b5jhk6vv1Ktvb1Fy0owIAIf8h1h+KAyHBy/3MKhcw0f+DzCsiP3ADmSMkgoqWnKG/\n\tMg/619lhdBVkBFWkl4khM4+xz4uH2pE70wvgPakwl1+0u4NVdGozlci1VqiIZZ8lT7IWuUvE8RV\n\typdhSnZyyhngid4Qbd6r8R9tm2naM4l4v2U7naG7SndZ4Wm74MOYXwA0/s1ah5WTcW50j9dxGRx\n\tB8nL7FNvDrOuX1IW1wQpdz0VrJ8s4lnsGkFI24F+2iGwH/KqRIL1hjYhGXqXh8n56OSflD3Y+lU\n\tuWtQNaz9Ec8HaWi9gscRo4sDJ66onlswoLZYd4TMAN+IEWF6Iqqok=","X-Received":["by 2002:a05:622a:48e:b0:4ec:3cfb:5589 with SMTP id\n\td75a77b69052e-4ed949a7a34mr48707571cf.34.1762533707384; \n\tFri, 07 Nov 2025 08:41:47 -0800 (PST)","by 2002:a05:622a:48e:b0:4ec:3cfb:5589 with SMTP id\n\td75a77b69052e-4ed949a7a34mr48707091cf.34.1762533706759; \n\tFri, 07 Nov 2025 08:41:46 -0800 (PST)"],"X-Google-Smtp-Source":"AGHT+IElH7LRoo7mOI+vS5GLfK6sQrJmqGxrfag2PVKbmiWUcihJYgfT2tvF29kgh5BxZNfZXwG4Zw==","Message-ID":"<05eed1a8-8de8-49e3-9d65-44aa86d35d45@oss.qualcomm.com>","Date":"Fri, 7 Nov 2025 17:41:45 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","From":"Hans de Goede ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Jacopo Mondi ","Cc":"=?utf-8?q?Niklas_S=C3=B6derlund?= ,\n\tRobert Mader , \n\tlibcamera-devel@lists.libcamera.org, Kieran Bingham\n\t","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>\n\t<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>\n\t","Content-Language":"en-US, nl","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","X-Proofpoint-Spam-Details-Enc":"AW1haW4tMjUxMTA3MDEzOCBTYWx0ZWRfX+KQ4cFA2CQOu\n\tAesoYiXNEbc0t75TbtZSCAyMzRH3fs/5YfkHf6jypoXZN0mp2hhrg5q2BRISHIxiaEmmp5f7o1e\n\tcpln7nGXLSvr7VSL1J/sqgG4S8KiOJRLB/qGKlKmxbEgi/4ruBRqucqBffEN/9c78inFb91b3va\n\tlmC10ManEkWu0Oce4mokfZ5bzqLwiGxqWx4AAPgKPjzZiyzDjwgk90cvdy02V/foT5eF27Eik1N\n\tvBuEXoslPFVSwdM24fZYL51Rip/gkiN+nqV+DWQTJWsgTVQJM+eibZNsqDmn1TEIHL4iE9fgeBj\n\tCZwgoCW7MGUv8LfMqTxbqbCtLi4RHSC2TSjR4t822G+Aayi9UaRBu4j+JUAPWSuU3uDlMp7pZ76\n\tAG93FfRfU0NWOFwr8SBcqeQGbi/2vg==","X-Proofpoint-GUID":"iZVIoYZ9l6AOlSwrOzyS5W9IreNajdsf","X-Authority-Analysis":"v=2.4 cv=MsJfKmae c=1 sm=1 tr=0 ts=690e214c cx=c_pps\n\ta=EVbN6Ke/fEF3bsl7X48z0g==:117 a=xqWC_Br6kY4A:10 a=IkcTkHD0fZMA:10\n\ta=6UeiqGixMTsA:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22\n\ta=CTmLhlSaaBTrNX_BkgwA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10\n\ta=a_PwQJl-kcHnX1M80qC6:22 a=HhbK4dLum7pmb74im6QT:22","X-Proofpoint-ORIG-GUID":"iZVIoYZ9l6AOlSwrOzyS5W9IreNajdsf","X-Proofpoint-Virus-Version":"vendor=baseguard\n\tengine=ICAP:2.0.293, Aquarius:18.0.1121, Hydra:6.1.9,\n\tFMLib:17.12.100.49\n\tdefinitions=2025-11-07_04,2025-11-06_01,2025-10-01_01","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\timpostorscore=0 suspectscore=0 bulkscore=0 adultscore=0\n\tmalwarescore=0\n\tspamscore=0 priorityscore=1501 clxscore=1015 phishscore=0\n\tlowpriorityscore=0\n\tclassifier=typeunknown authscore=0 authtc= authcc= route=outbound\n\tadjust=0\n\treason=mlx scancount=1 engine=8.22.0-2510240001\n\tdefinitions=main-2511070138","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36756,"web_url":"https://patchwork.libcamera.org/comment/36756/","msgid":"","date":"2025-11-10T09:05:36","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Hans,\n\nOn Fri, 7 Nov 2025 at 16:41, Hans de Goede\n wrote:\n>\n> Hi Jacopo,\n>\n> On 6-Nov-25 9:29 AM, Jacopo Mondi wrote:\n> > Hi Hans\n> >\n> > On Tue, Nov 04, 2025 at 06:23:23PM +0100, Hans de Goede wrote:\n> >> Hi Jacopo,\n> >>\n> >> On 4-Nov-25 16:53, Jacopo Mondi wrote:\n> >>> Hi Hans\n> >>>\n> >>> On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n> >>>> Hi,\n> >>>>\n> >>>> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n> >>>>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> >>>>\n> >>>> ...\n> >>>>\n> >>>>>> Right I forgot about the delayed-controls stuff, not being able\n> >>>>>> to get the new exposure limits from the kernel then makes sense.\n> >>>>>\n> >>>>> I too understand why it's easier to not consult the kernel for the new\n> >>>>> limits. I have no real strong feeling that the correct solution is to\n> >>>>> have this limit calculation done in the kernel or user-space. What I do\n> >>>>> feel somewhat strongly about is if we do the limit calculation in\n> >>>>> user-space we should never consult the kernel for the limits.\n> >>>>>\n> >>>>> This just mixes two ways of doing the same thing and is very confusing\n> >>>>> when reading the code. So if doing this in user-space is the way forward\n> >>>>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> >>>>> in IPARkISP1::configure() and instead use this method there too.\n> >>>>>\n> >>>>> I also feel, a little less strongly as it's a big work item, that it's\n> >>>>> really wrong to duplicate the calculation both in kernel and user-space.\n> >>>>>\n> >>>>> As this IMHO exposes a weakness in the V4L2 API we should think about\n> >>>>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> >>>>> we really want it to report its min/max values based on the current\n> >>>>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> >>>>> for each frame.\n> >>>>>\n> >>>>> Can we simplify the control to report the min and max values the sensor\n> >>>>> can do provided the VBLANK setting allows it, and just reject any value\n> >>>>> at set time that do not conform to the current VBLANK setting?\n> >>>>\n> >>>> One complicating factor here is that some sensors will do vblank stretching\n> >>>> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n> >>>> vblank is set when setting exposure > (frame-length + margin) to still\n> >>>> give you the desired exposure.\n> >>>\n> >>> Interesting. Do you have any pointer to mainline drivers that do that ?\n> >>\n> >> I'm not talking about the driver stretching the vblank I'm talking\n> >> about the hw (the sensor) stretching the vblank. I think that in newer\n> >> drivers we stop this from happening by clamping the exposure.\n> >>\n> >> I'm 100% certain that some hw does this as I've seen this happen.\n> >> I can do some digging if you want some specific sensor model examples\n> >> where the hw is doing this.\n> >\n> > I think the fact some sensor supporting this feature not an issue if\n> > we operate with algorithms designed to always clamp the shutter time\n> > to the frame duration. Hence I would be more interested in knowing\n> > which usage model you would consider best for the whole\n> > libcamera/libipa handling of exposure/frame duration.\n> >\n> > Currently, the design is to have the maximum achievable shutter time\n> > to be always constrained by the frame duration. Users should take\n> > care of explicitly slowing down the camera frame rate in order to\n> > achieve a larger shutter time.\n> >\n> > I think it is justified by the face that the desired exposure is\n> > realized by the combined effect of the shutter time and gains, and\n> > hence limiting the shutter time won't imply not being able to reach\n> > the desired exposure levels.\n>\n> Right, except that in low light you may need to boost the gain\n> to a level where you get significant noise and at some point\n> the gain is mostly just amplifying noise (discussion continued\n> below).\n>\n> > Alternative approaches where the exposure time drives the frame\n> > duration could be implemented with helpers, similar to the already\n> > discussed \"aperture priority\" vs \"shutter priority\" modes.\n>\n> Right, I do believe that we really need these modes, at least\n> for laptops. Unlike the RPI cameras which have pretty nice large\n> lenses. The laptop designs are optimized for small bezels and\n> thus small lenses which means not a whole lot of light, which\n> really starts to bite at low light conditions.\n>\n> And I've seen both Intel's proprietary IPU6 stack as well as\n> many UVC webcams lower the framerate to compensate (somewhat).\n>\n> I'm fine with defaulting to never allowing the need for\n> higher exposure to lower the framerate, but as you say I do\n> believe we need a control to allow the AGC lowering the framerate\n> to get larger exposure times. And I think those controls should\n> likely even default to on devices with small lenses, but first\n> we need to have / implement such an exposure-mode control to begin\n> with :)\n\nI believe the current FrameDurationLimits will allow you to achieve\neverything you need here. From the control definition:\n\n - FrameDurationLimits:\n type: int64_t\n direction: inout\n description: |\n The minimum and maximum (in that order) frame duration, expressed in\n microseconds.\n\n When provided by applications, the control specifies the sensor frame\n duration interval the pipeline has to use. This limits the largest\n exposure time the sensor can use. For example, if a maximum frame\n duration of 33ms is requested (corresponding to 30 frames per second),\n the sensor will not be able to raise the exposure time above 33ms.\n A fixed frame duration is achieved by setting the minimum and maximum\n values to be the same. Setting both values to 0 reverts to using the\n camera defaults.\n\n The maximum frame duration provides the absolute limit to the exposure\n time computed by the AE algorithm and it overrides any exposure mode\n setting specified with controls::AeExposureMode. Similarly, when a\n manual exposure time is set through controls::ExposureTime, it also\n gets clipped to the limits set by this control. When reported in\n metadata, the control expresses the minimum and maximum frame durations\n used after being clipped to the sensor provided frame duration limits.\n\nSo this control takes a min and max value to use by the IPA. If you\nspecify, say, min = (1/30)s and max = (1/15)s, this should limit the\nAGC to increase the shutter time upto 66ms if the scene conditions\nrequire. This is the mechanism RPi uses to allow AEC to increase the\nexposure without ramping up the analogue gain, if required.\n\nRegards,\nNaush\n\n>\n> > What we have to make sure is the controls we define to driver the AGC\n> > allows us to create such helpers easily.\n> >\n> > Do you think we're on the same page here ?\n>\n> Yes I believe that we are on the same page here.\n>\n> Regards,\n>\n> Hans\n>\n>\n>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 79857C3263\n\tfor ;\n\tMon, 10 Nov 2025 09:06:17 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 33FCC60A8B;\n\tMon, 10 Nov 2025 10:06:16 +0100 (CET)","from mail-vk1-xa2f.google.com (mail-vk1-xa2f.google.com\n\t[IPv6:2607:f8b0:4864:20::a2f])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 793C46069A\n\tfor ;\n\tMon, 10 Nov 2025 10:06:13 +0100 (CET)","by mail-vk1-xa2f.google.com with SMTP id\n\t71dfb90a1353d-559671d0bb6so89157e0c.0\n\tfor ;\n\tMon, 10 Nov 2025 01:06:13 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"jcCKKXsV\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1762765572; x=1763370372;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=Srf5SRx5vJAYNwjJspp+ZFDsbaUEI0FiTCqtkSy6LAs=;\n\tb=jcCKKXsVmbmYH/jLepc1nhnpts6mxpTi04ezNOmMf6Kp/JDS0j7xeHcAhWwCAm/V4F\n\tkl2v+Wdsf3YbXhPxacSILlgyTmkxK1pkrdovRtFk8a7xKX8+pIRk0IPp45Ks3Gg/ZLvB\n\tPt/o9uRa3Y3+tw13KXY8blRUnSLumGiB8SyB2aUzgzOqPAs4xwZa61XmKKA/gNNn7Iu2\n\tRlDpbZwArZNrkU621Fc7zepJl27LxHVynDoL0pjbMKkV4HKLDE5RrFx0tBx53WRdc7E9\n\t4ZMuQBC53SswmzCjnILlteiDx40mtLI2/7xaZBlzza9hNR3CX0JvTUAXqcLxVEd6wOCe\n\tmSdQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1762765572; x=1763370372;\n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from\n\t:to:cc:subject:date:message-id:reply-to;\n\tbh=Srf5SRx5vJAYNwjJspp+ZFDsbaUEI0FiTCqtkSy6LAs=;\n\tb=gwJjHoRI+pzdrvSk2csjhpwAmAt5pwoaqJI9vppCBW4ZFd6Nri41tbPlYut0xIvuFU\n\tR7w5QqFmag1BWNqaV3sSOPfaniNfUH4nyiE1LmgR1rl91SGFQ4dz1LYRZH10oaG+qLHV\n\tSwceIQpOJnTidVe5dxLSoEbePdxpgNfQEcx4uKiN/18lMCD51ATJO3B3NWlsvg2HKZiq\n\ty0+s399KRbABixNxrlkgMUmK7Q+H04jwbOvPasE0uC1WpJW4Y+cDLXk7EYfSxVvuhmje\n\tBJZuMZyP5EqRJKaCFMCISN1NNr6btGL+Rp4Kvpt9/XScmmJlC8W/z3FPxUMJfOJV5jxh\n\trQjA==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCXFrS9zEu/v9R2SgZ3upHH4SybPlq43W2hAyMjugJcWCrJHPH+f2e3j5bqaEaOD6JSabNDGBQcHHse8kYzO/oE=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YyKuEWa6QGNx5c9D/KEY7Xil5euSNlVzeokGMKJ7DRUoB2imWzW\n\t1rScRnudHITl70AH4gPlWsWn+fLuRqJDCwcGG/F942ZrXHx+D5Fy3Klsau5FOqcVt3quKwRhQIK\n\t0RMr/MM3iaxCED5V48Pa30yAyRN6YRlxQzPSKnfrMUg==","X-Gm-Gg":"ASbGncv3oIHIVkX26gfv6HwK6mDI7GyrILYMQPU2hiW9oU0FQQ8bKGkba6ed2CcGX1C\n\tb1dn4j3l3eyiDqJwjW8qfOnhLuAYOPeCOr6ILAbYfSfoGloIpKT4H1SaL8uEg+i6l+bKCDX9/m/\n\tOxzoQzsor33LB8w4dHPKd573Duja2Y9wdx4aGUaXTFdsxspZN5EwPAXbcA71ahBupzn9KLMHT1b\n\t/wv/vCnAgoYKnlXfASZKCNnWU3ZTbbfWGh0k7697tODwXRqU02y2u6tA+YfH03rPUNkcy1xXkYf\n\tQ4hvUHSFZcxD94s9","X-Google-Smtp-Source":"AGHT+IHzZQswXn/GHosoWcW8reTvQSjuR492knb1XM5NGAmQi14zF/t6R6Mg5vRwgUxIRpDuaoid3fBISTUhygYzwi4=","X-Received":"by 2002:a05:6122:105a:b0:559:860b:7b1c with SMTP id\n\t71dfb90a1353d-559b329a2c6mr729042e0c.2.1762765572039; Mon, 10 Nov 2025\n\t01:06:12 -0800 (PST)","MIME-Version":"1.0","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>\n\t<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>\n\t\n\t<05eed1a8-8de8-49e3-9d65-44aa86d35d45@oss.qualcomm.com>","In-Reply-To":"<05eed1a8-8de8-49e3-9d65-44aa86d35d45@oss.qualcomm.com>","From":"Naushir Patuck ","Date":"Mon, 10 Nov 2025 09:05:36 +0000","X-Gm-Features":"AWmQ_bn_zKMLE-GbezA5Sq2nLj0ajtQzvkC5TsV7iLnR9eXAJXC14x-QsONV93w","Message-ID":"","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Hans de Goede ","Cc":"Jacopo Mondi , =?utf-8?q?Niklas_S?=\n\t=?utf-8?b?w7ZkZXJsdW5k?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org, \n\tKieran Bingham ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36757,"web_url":"https://patchwork.libcamera.org/comment/36757/","msgid":"<23bc1f82-b0f2-4c57-9596-f02b13d06534@oss.qualcomm.com>","date":"2025-11-10T09:40:38","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":242,"url":"https://patchwork.libcamera.org/api/people/242/","name":"Hans de Goede","email":"johannes.goede@oss.qualcomm.com"},"content":"Hi Naush,\n\nOn 10-Nov-25 10:05, Naushir Patuck wrote:\n> Hi Hans,\n> \n> On Fri, 7 Nov 2025 at 16:41, Hans de Goede\n> wrote:\n>>\n>> Hi Jacopo,\n>>\n>> On 6-Nov-25 9:29 AM, Jacopo Mondi wrote:\n>>> Hi Hans\n>>>\n>>> On Tue, Nov 04, 2025 at 06:23:23PM +0100, Hans de Goede wrote:\n>>>> Hi Jacopo,\n>>>>\n>>>> On 4-Nov-25 16:53, Jacopo Mondi wrote:\n>>>>> Hi Hans\n>>>>>\n>>>>> On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n>>>>>> Hi,\n>>>>>>\n>>>>>> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n>>>>>>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n>>>>>>\n>>>>>> ...\n>>>>>>\n>>>>>>>> Right I forgot about the delayed-controls stuff, not being able\n>>>>>>>> to get the new exposure limits from the kernel then makes sense.\n>>>>>>>\n>>>>>>> I too understand why it's easier to not consult the kernel for the new\n>>>>>>> limits. I have no real strong feeling that the correct solution is to\n>>>>>>> have this limit calculation done in the kernel or user-space. What I do\n>>>>>>> feel somewhat strongly about is if we do the limit calculation in\n>>>>>>> user-space we should never consult the kernel for the limits.\n>>>>>>>\n>>>>>>> This just mixes two ways of doing the same thing and is very confusing\n>>>>>>> when reading the code. So if doing this in user-space is the way forward\n>>>>>>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n>>>>>>> in IPARkISP1::configure() and instead use this method there too.\n>>>>>>>\n>>>>>>> I also feel, a little less strongly as it's a big work item, that it's\n>>>>>>> really wrong to duplicate the calculation both in kernel and user-space.\n>>>>>>>\n>>>>>>> As this IMHO exposes a weakness in the V4L2 API we should think about\n>>>>>>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n>>>>>>> we really want it to report its min/max values based on the current\n>>>>>>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n>>>>>>> for each frame.\n>>>>>>>\n>>>>>>> Can we simplify the control to report the min and max values the sensor\n>>>>>>> can do provided the VBLANK setting allows it, and just reject any value\n>>>>>>> at set time that do not conform to the current VBLANK setting?\n>>>>>>\n>>>>>> One complicating factor here is that some sensors will do vblank stretching\n>>>>>> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n>>>>>> vblank is set when setting exposure > (frame-length + margin) to still\n>>>>>> give you the desired exposure.\n>>>>>\n>>>>> Interesting. Do you have any pointer to mainline drivers that do that ?\n>>>>\n>>>> I'm not talking about the driver stretching the vblank I'm talking\n>>>> about the hw (the sensor) stretching the vblank. I think that in newer\n>>>> drivers we stop this from happening by clamping the exposure.\n>>>>\n>>>> I'm 100% certain that some hw does this as I've seen this happen.\n>>>> I can do some digging if you want some specific sensor model examples\n>>>> where the hw is doing this.\n>>>\n>>> I think the fact some sensor supporting this feature not an issue if\n>>> we operate with algorithms designed to always clamp the shutter time\n>>> to the frame duration. Hence I would be more interested in knowing\n>>> which usage model you would consider best for the whole\n>>> libcamera/libipa handling of exposure/frame duration.\n>>>\n>>> Currently, the design is to have the maximum achievable shutter time\n>>> to be always constrained by the frame duration. Users should take\n>>> care of explicitly slowing down the camera frame rate in order to\n>>> achieve a larger shutter time.\n>>>\n>>> I think it is justified by the face that the desired exposure is\n>>> realized by the combined effect of the shutter time and gains, and\n>>> hence limiting the shutter time won't imply not being able to reach\n>>> the desired exposure levels.\n>>\n>> Right, except that in low light you may need to boost the gain\n>> to a level where you get significant noise and at some point\n>> the gain is mostly just amplifying noise (discussion continued\n>> below).\n>>\n>>> Alternative approaches where the exposure time drives the frame\n>>> duration could be implemented with helpers, similar to the already\n>>> discussed \"aperture priority\" vs \"shutter priority\" modes.\n>>\n>> Right, I do believe that we really need these modes, at least\n>> for laptops. Unlike the RPI cameras which have pretty nice large\n>> lenses. The laptop designs are optimized for small bezels and\n>> thus small lenses which means not a whole lot of light, which\n>> really starts to bite at low light conditions.\n>>\n>> And I've seen both Intel's proprietary IPU6 stack as well as\n>> many UVC webcams lower the framerate to compensate (somewhat).\n>>\n>> I'm fine with defaulting to never allowing the need for\n>> higher exposure to lower the framerate, but as you say I do\n>> believe we need a control to allow the AGC lowering the framerate\n>> to get larger exposure times. And I think those controls should\n>> likely even default to on devices with small lenses, but first\n>> we need to have / implement such an exposure-mode control to begin\n>> with :)\n> \n> I believe the current FrameDurationLimits will allow you to achieve\n> everything you need here. From the control definition:\n> \n> - FrameDurationLimits:\n> type: int64_t\n> direction: inout\n> description: |\n> The minimum and maximum (in that order) frame duration, expressed in\n> microseconds.\n> \n> When provided by applications, the control specifies the sensor frame\n> duration interval the pipeline has to use. This limits the largest\n> exposure time the sensor can use. For example, if a maximum frame\n> duration of 33ms is requested (corresponding to 30 frames per second),\n> the sensor will not be able to raise the exposure time above 33ms.\n> A fixed frame duration is achieved by setting the minimum and maximum\n> values to be the same. Setting both values to 0 reverts to using the\n> camera defaults.\n> \n> The maximum frame duration provides the absolute limit to the exposure\n> time computed by the AE algorithm and it overrides any exposure mode\n> setting specified with controls::AeExposureMode. Similarly, when a\n> manual exposure time is set through controls::ExposureTime, it also\n> gets clipped to the limits set by this control. When reported in\n> metadata, the control expresses the minimum and maximum frame durations\n> used after being clipped to the sensor provided frame duration limits.\n> \n> So this control takes a min and max value to use by the IPA. If you\n> specify, say, min = (1/30)s and max = (1/15)s, this should limit the\n> AGC to increase the shutter time upto 66ms if the scene conditions\n> require. This is the mechanism RPi uses to allow AEC to increase the\n> exposure without ramping up the analogue gain, if required.\n\nRight, this works if you are in full control of the applications and thus\nof the framerate range the application requests.\n\nBut when using e.g. browsers for video conferencing they will just ask\nfor a fixed framerate. I'm not even sure if the getUserMedia web APIs\nused by webrtc in-browser apps support a framerate range. So we may want to\nhave another mechanism to allow to allow lowering the framerate in low\nlight conditions, which on UVC cams at least is typically done through\nan aperture priority exposure mode.\n\nBut I see from the description of the current frameDuration control that\nthis would go against that description. So maybe we need to have pipewire\nalways request a fps range of (desired-fps / 2) - desired-fps. Even then\nit would still be nice to have a control to tell the AGC if it should\noptimize for getting as much fps as possible (only lowering the fps\nif it needs a gain of say above 4.0 otherwise) or if it should try to\nkeep the gain as low as possible and immediately start lowering fps\nif necessary.\n\nRegards,\n\nHans","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A5313C3241\n\tfor ;\n\tMon, 10 Nov 2025 09:40:45 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B4144609E0;\n\tMon, 10 Nov 2025 10:40:44 +0100 (CET)","from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com\n\t[205.220.180.131])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B1CA86069A\n\tfor ;\n\tMon, 10 Nov 2025 10:40:42 +0100 (CET)","from pps.filterd (m0279870.ppops.net [127.0.0.1])\n\tby mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id\n\t5AA9CgAQ3146244 for ;\n\tMon, 10 Nov 2025 09:40:41 GMT","from mail-qt1-f198.google.com (mail-qt1-f198.google.com\n\t[209.85.160.198])\n\tby mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4abd7hr3ht-1\n\t(version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT)\n\tfor ;\n\tMon, 10 Nov 2025 09:40:40 +0000 (GMT)","by mail-qt1-f198.google.com with SMTP id\n\td75a77b69052e-4edb31eed49so27522761cf.2\n\tfor ;\n\tMon, 10 Nov 2025 01:40:40 -0800 (PST)","from [10.40.98.122] ([78.108.130.194])\n\tby smtp.gmail.com with ESMTPSA id\n\ta640c23a62f3a-b72bdbcea81sm1034315266b.13.2025.11.10.01.40.38\n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tMon, 10 Nov 2025 01:40:39 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=qualcomm.com header.i=@qualcomm.com\n\theader.b=\"WjUBjTex\"; dkim=pass (2048-bit key;\n\tunprotected) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com\n\theader.b=\"KMKMiMyy\"; dkim-atps=neutral","DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h=\n\tcc:content-transfer-encoding:content-type:date:from:in-reply-to\n\t:message-id:mime-version:references:subject:to; s=qcppdkim1; bh=\n\t9N13Wib4VuJmKBXYICXQfrV0QSJpTef4jGiPPRQIxxE=; b=WjUBjTexjaxFGDfO\n\tqokl/+12OCEPrTyUYis7M6yd1l1fa5fKzRnnGSz5cin3+gG2JlObRCvzPIJ6smz4\n\tuA2d6Rqr+lPOd/mn73tKIPXA1VF+SxoKH1vqZYbMPLOY0kjUQ0rBovG4af5EhgtV\n\tgQHj9Hc1mCQTZqWeOc6Lc2Lis1aHtWrdnRK2gzsjNlnQxK3YBFZu7WUtgs6KTVRG\n\ti5QF1xxCUrQsTtxFBjqQlu0zo2hxVBc1wrlykmV8Scz+6+9PRg2/PDBLda7d1tW5\n\tEGN+tCs6EW4yW+WGErzcTjTyONVC1/DbYqHQexJf8nHVcwR0frcYlkn1G+HywUVt\n\trxNNkA==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=oss.qualcomm.com; s=google; t=1762767640; x=1763372440;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:in-reply-to:content-language:references\n\t:cc:to:subject:from:user-agent:mime-version:date:message-id:from:to\n\t:cc:subject:date:message-id:reply-to;\n\tbh=9N13Wib4VuJmKBXYICXQfrV0QSJpTef4jGiPPRQIxxE=;\n\tb=KMKMiMyyOG9pjCadyG2P6kxLPsO2DEVtwQvIq8Jaq9rfJi+tsaRO+TWJE3GbxjJdF+\n\tQhQm6nuF4BUpEO3pwV2CoxVF6eMbqk8V50HacMg/PFLPMSDAGwR4zeN3/XFAZHe6wQu4\n\te3CoAp1S6POQxc0gmhhUXRBk2CnSBVqQ9OnKVosdyl0LA/uaixhxQYzHGKMHKaB7Bfg/\n\tOwAj8oqDgzePGkVtr17jO3oZnxEeWrA++eHv+tDYBIirnpDZYJ0O9OH3uDeFgD/85ysl\n\t8oA9JCnUfzlMxD2pb6FrMhyV/CDyObM5l+EnXNriRrXdlqeabS1RPUOiErnFPAJVD2it\n\tRYYQ=="],"X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1762767640; x=1763372440;\n\th=content-transfer-encoding:in-reply-to:content-language:references\n\t:cc:to:subject:from:user-agent:mime-version:date:message-id:x-gm-gg\n\t:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n\tbh=9N13Wib4VuJmKBXYICXQfrV0QSJpTef4jGiPPRQIxxE=;\n\tb=wiYD8rjA7oBdih5DdGZ8pAMbO/JcXof5ZuLFFhpQJy10hid7nI1Nk5l054OIBx/glQ\n\tL1+2MKBDk8AFSjoVNLUenAS6Q74QS2o3n6LIwM2lQQT8QbFnkA5SQbX6HuTY/vIp2w8Z\n\t2/cpamHDn445nPYyEvNvtw7cS1MD6F7YXIQrYCIpAcJG4QtpX2u4zZpUfvFQs8BkcnB0\n\tujR5x7DoFzTi70GS3QDbTu2kKjuRLlYZD+NgZAD4RaV0Rehg97paQB5zbIZ5xN9wZqQC\n\tGjCcd6YCUzFvZnbn0trIZ7oQRtDor2sj+DlAYxO/8lFZL8CjL32x84g9x91B8oOX3ozp\n\tI8hQ==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCUc5SzHWO+uqZIixvMREgh2CnmNFT/6mTPLNlWON86Zu4YtmAjgN/yTVp9ZV0oQMBhPgFRwojUU0F55yh/oy6I=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YzjC76orXyMdAaFIxYWxAEICpg/v/kzTs7FLmsoy4crXLiWrO6W\n\tMK0/qk71LfWWQ8cVkQwINIPkG5BpE+oRi0DS82xEAleidOS4H9lvYQX233zpxBTgq/vCrckYzZ1\n\tc9dM52u/uPq72McO8WUFES7AyIB32owjOl2PvBe3wb8VyVDCOeuC31TO5t0+RIv5qr7iJwayGpj\n\tcw7lFqSLxK","X-Gm-Gg":"ASbGncsGLtP636ob5Z8LLAGphetumtE8jc4HQkrgJdJbuIHhWTRarDrjvQ8oVGBbVl1\n\t9Pe9lMC6QV7CWoFqhCPWp3BDAD93OU4DKrJyF8+EDsyl2cPUl10+mVr1MpzWY/1rsqWFopk0olJ\n\t47Rip1oL6w/TtqmmD6Ox4ij2ZmNObaHE+YfPoOndijI+DIBk29cQfc32/SMjweY1BS9KRu3twJX\n\tPSH4Gi+wnVHy59mD4BSr5bRXzP9PvQIuQVkLuMMGzPR/1lhQT7eCoFmL0lqNLpFIHy/tW1XFIJ6\n\t3pFlGtM9qf6gGrG5mPjFNKoVQngRU74cNcW6CdFL8AfiL65maYnFh+XdA79nxEufdQ+XiX0v7tt\n\t0ABmuT/Nwqye9RpB1yBI=","X-Received":["by 2002:a05:622a:1916:b0:4e8:aa11:5869 with SMTP id\n\td75a77b69052e-4eda4fe17demr99213751cf.70.1762767640220; \n\tMon, 10 Nov 2025 01:40:40 -0800 (PST)","by 2002:a05:622a:1916:b0:4e8:aa11:5869 with SMTP id\n\td75a77b69052e-4eda4fe17demr99213531cf.70.1762767639702; \n\tMon, 10 Nov 2025 01:40:39 -0800 (PST)"],"X-Google-Smtp-Source":"AGHT+IHnFuXKx8x+oLJKxQvtucOU2XHH5i5uCLXtyPe9/SaxKepWbqiK2p1ijI5VfQlLMhuvzw7j2g==","Message-ID":"<23bc1f82-b0f2-4c57-9596-f02b13d06534@oss.qualcomm.com>","Date":"Mon, 10 Nov 2025 10:40:38 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","From":"Hans de Goede ","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Naushir Patuck ","Cc":"Jacopo Mondi , =?utf-8?q?Niklas_S=C3=B6?=\n\t=?utf-8?q?derlund?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org,\n\tKieran Bingham ","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>\n\t<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>\n\t\n\t<05eed1a8-8de8-49e3-9d65-44aa86d35d45@oss.qualcomm.com>\n\t","Content-Language":"en-US, nl","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","X-Proofpoint-GUID":"gS9MFg9BD0gIs4cb1yXRNZepQmh9UsBC","X-Proofpoint-Spam-Details-Enc":"AW1haW4tMjUxMTEwMDA4NCBTYWx0ZWRfX5wIXiwC9pYIJ\n\tu/B5WddTuxtejCnOQXqyaL6ykYg+ekqvOdLiuBRtaeBoYy3Hk2fHjpngFrik2eASl1NF25KuHfD\n\tiKPtpTC/GDasFhK96kL69VEKe16xDit0n1lHO1rpI6iC2MfT4FJvC/q46aJxZSxKRjNTSlTC/ys\n\truHO8zmqCXCVXfI1ICh+qQ0ymKU7HjGeVkC3LJN7neQuBDx6trKtaDskMM1hRMh431kedQXcwap\n\twpEoTIVEkagxOu4mpDurIicSS9Jp5CJGTBvGbrD8UCEK2eYu7kjLRvFGDTWPGle5471GkDkQE1E\n\tvjWPQ8SvQJRN9zdAfPhctDntCxUxYR4euwO/awtl4HF7nZfP8ViYvJlST8xE2XfvYXpi62pSB2b\n\tf0fHqOSjDmco+R9QT1eIJi3EKn+UBw==","X-Proofpoint-ORIG-GUID":"gS9MFg9BD0gIs4cb1yXRNZepQmh9UsBC","X-Authority-Analysis":"v=2.4 cv=Yt4ChoYX c=1 sm=1 tr=0 ts=6911b319 cx=c_pps\n\ta=mPf7EqFMSY9/WdsSgAYMbA==:117 a=rrvG0T/C2D967D07Ol03YQ==:17\n\ta=IkcTkHD0fZMA:10 a=6UeiqGixMTsA:10 a=s4-Qcg_JpJYA:10\n\ta=VkNPw1HP01LnGYTKEx00:22 a=EUspDBNiAAAA:8 a=B_cds2-wcZVfNlSnbCgA:9\n\ta=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=dawVfQjAaf238kedN5IG:22\n\ta=HhbK4dLum7pmb74im6QT:22","X-Proofpoint-Virus-Version":"vendor=baseguard\n\tengine=ICAP:2.0.293, Aquarius:18.0.1121, Hydra:6.1.9,\n\tFMLib:17.12.100.49\n\tdefinitions=2025-11-10_04,2025-11-06_01,2025-10-01_01","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tbulkscore=0 adultscore=0 spamscore=0 phishscore=0 malwarescore=0\n\timpostorscore=0 clxscore=1015 suspectscore=0 lowpriorityscore=0\n\tpriorityscore=1501 classifier=typeunknown authscore=0 authtc= authcc=\n\troute=outbound adjust=0 reason=mlx scancount=1\n\tengine=8.22.0-2510240001\n\tdefinitions=main-2511100084","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36759,"web_url":"https://patchwork.libcamera.org/comment/36759/","msgid":"","date":"2025-11-10T09:57:57","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":34,"url":"https://patchwork.libcamera.org/api/people/34/","name":"Naushir Patuck","email":"naush@raspberrypi.com"},"content":"Hi Hans,\n\nOn Mon, 10 Nov 2025 at 09:40, Hans de Goede\n wrote:\n>\n> Hi Naush,\n>\n> On 10-Nov-25 10:05, Naushir Patuck wrote:\n> > Hi Hans,\n> >\n> > On Fri, 7 Nov 2025 at 16:41, Hans de Goede\n> > wrote:\n> >>\n> >> Hi Jacopo,\n> >>\n> >> On 6-Nov-25 9:29 AM, Jacopo Mondi wrote:\n> >>> Hi Hans\n> >>>\n> >>> On Tue, Nov 04, 2025 at 06:23:23PM +0100, Hans de Goede wrote:\n> >>>> Hi Jacopo,\n> >>>>\n> >>>> On 4-Nov-25 16:53, Jacopo Mondi wrote:\n> >>>>> Hi Hans\n> >>>>>\n> >>>>> On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n> >>>>>> Hi,\n> >>>>>>\n> >>>>>> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n> >>>>>>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> >>>>>>\n> >>>>>> ...\n> >>>>>>\n> >>>>>>>> Right I forgot about the delayed-controls stuff, not being able\n> >>>>>>>> to get the new exposure limits from the kernel then makes sense.\n> >>>>>>>\n> >>>>>>> I too understand why it's easier to not consult the kernel for the new\n> >>>>>>> limits. I have no real strong feeling that the correct solution is to\n> >>>>>>> have this limit calculation done in the kernel or user-space. What I do\n> >>>>>>> feel somewhat strongly about is if we do the limit calculation in\n> >>>>>>> user-space we should never consult the kernel for the limits.\n> >>>>>>>\n> >>>>>>> This just mixes two ways of doing the same thing and is very confusing\n> >>>>>>> when reading the code. So if doing this in user-space is the way forward\n> >>>>>>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> >>>>>>> in IPARkISP1::configure() and instead use this method there too.\n> >>>>>>>\n> >>>>>>> I also feel, a little less strongly as it's a big work item, that it's\n> >>>>>>> really wrong to duplicate the calculation both in kernel and user-space.\n> >>>>>>>\n> >>>>>>> As this IMHO exposes a weakness in the V4L2 API we should think about\n> >>>>>>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> >>>>>>> we really want it to report its min/max values based on the current\n> >>>>>>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> >>>>>>> for each frame.\n> >>>>>>>\n> >>>>>>> Can we simplify the control to report the min and max values the sensor\n> >>>>>>> can do provided the VBLANK setting allows it, and just reject any value\n> >>>>>>> at set time that do not conform to the current VBLANK setting?\n> >>>>>>\n> >>>>>> One complicating factor here is that some sensors will do vblank stretching\n> >>>>>> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n> >>>>>> vblank is set when setting exposure > (frame-length + margin) to still\n> >>>>>> give you the desired exposure.\n> >>>>>\n> >>>>> Interesting. Do you have any pointer to mainline drivers that do that ?\n> >>>>\n> >>>> I'm not talking about the driver stretching the vblank I'm talking\n> >>>> about the hw (the sensor) stretching the vblank. I think that in newer\n> >>>> drivers we stop this from happening by clamping the exposure.\n> >>>>\n> >>>> I'm 100% certain that some hw does this as I've seen this happen.\n> >>>> I can do some digging if you want some specific sensor model examples\n> >>>> where the hw is doing this.\n> >>>\n> >>> I think the fact some sensor supporting this feature not an issue if\n> >>> we operate with algorithms designed to always clamp the shutter time\n> >>> to the frame duration. Hence I would be more interested in knowing\n> >>> which usage model you would consider best for the whole\n> >>> libcamera/libipa handling of exposure/frame duration.\n> >>>\n> >>> Currently, the design is to have the maximum achievable shutter time\n> >>> to be always constrained by the frame duration. Users should take\n> >>> care of explicitly slowing down the camera frame rate in order to\n> >>> achieve a larger shutter time.\n> >>>\n> >>> I think it is justified by the face that the desired exposure is\n> >>> realized by the combined effect of the shutter time and gains, and\n> >>> hence limiting the shutter time won't imply not being able to reach\n> >>> the desired exposure levels.\n> >>\n> >> Right, except that in low light you may need to boost the gain\n> >> to a level where you get significant noise and at some point\n> >> the gain is mostly just amplifying noise (discussion continued\n> >> below).\n> >>\n> >>> Alternative approaches where the exposure time drives the frame\n> >>> duration could be implemented with helpers, similar to the already\n> >>> discussed \"aperture priority\" vs \"shutter priority\" modes.\n> >>\n> >> Right, I do believe that we really need these modes, at least\n> >> for laptops. Unlike the RPI cameras which have pretty nice large\n> >> lenses. The laptop designs are optimized for small bezels and\n> >> thus small lenses which means not a whole lot of light, which\n> >> really starts to bite at low light conditions.\n> >>\n> >> And I've seen both Intel's proprietary IPU6 stack as well as\n> >> many UVC webcams lower the framerate to compensate (somewhat).\n> >>\n> >> I'm fine with defaulting to never allowing the need for\n> >> higher exposure to lower the framerate, but as you say I do\n> >> believe we need a control to allow the AGC lowering the framerate\n> >> to get larger exposure times. And I think those controls should\n> >> likely even default to on devices with small lenses, but first\n> >> we need to have / implement such an exposure-mode control to begin\n> >> with :)\n> >\n> > I believe the current FrameDurationLimits will allow you to achieve\n> > everything you need here. From the control definition:\n> >\n> > - FrameDurationLimits:\n> > type: int64_t\n> > direction: inout\n> > description: |\n> > The minimum and maximum (in that order) frame duration, expressed in\n> > microseconds.\n> >\n> > When provided by applications, the control specifies the sensor frame\n> > duration interval the pipeline has to use. This limits the largest\n> > exposure time the sensor can use. For example, if a maximum frame\n> > duration of 33ms is requested (corresponding to 30 frames per second),\n> > the sensor will not be able to raise the exposure time above 33ms.\n> > A fixed frame duration is achieved by setting the minimum and maximum\n> > values to be the same. Setting both values to 0 reverts to using the\n> > camera defaults.\n> >\n> > The maximum frame duration provides the absolute limit to the exposure\n> > time computed by the AE algorithm and it overrides any exposure mode\n> > setting specified with controls::AeExposureMode. Similarly, when a\n> > manual exposure time is set through controls::ExposureTime, it also\n> > gets clipped to the limits set by this control. When reported in\n> > metadata, the control expresses the minimum and maximum frame durations\n> > used after being clipped to the sensor provided frame duration limits.\n> >\n> > So this control takes a min and max value to use by the IPA. If you\n> > specify, say, min = (1/30)s and max = (1/15)s, this should limit the\n> > AGC to increase the shutter time upto 66ms if the scene conditions\n> > require. This is the mechanism RPi uses to allow AEC to increase the\n> > exposure without ramping up the analogue gain, if required.\n>\n> Right, this works if you are in full control of the applications and thus\n> of the framerate range the application requests.\n>\n> But when using e.g. browsers for video conferencing they will just ask\n> for a fixed framerate. I'm not even sure if the getUserMedia web APIs\n> used by webrtc in-browser apps support a framerate range. So we may want to\n> have another mechanism to allow to allow lowering the framerate in low\n> light conditions, which on UVC cams at least is typically done through\n> an aperture priority exposure mode.\n\nIf the application/pipewire has set a fixed framerate to use, I don't\nthink the IPA should do anything other than run at this framerate,\nregardless of lighting conditions. But perhaps this is not what you\nmean here?\n\n>\n> But I see from the description of the current frameDuration control that\n> this would go against that description. So maybe we need to have pipewire\n> always request a fps range of (desired-fps / 2) - desired-fps. Even then\n> it would still be nice to have a control to tell the AGC if it should\n> optimize for getting as much fps as possible (only lowering the fps\n> if it needs a gain of say above 4.0 otherwise) or if it should try to\n> keep the gain as low as possible and immediately start lowering fps\n> if necessary.\n\nThis makes sense, and how we achieve this is through the \"exposure\nprofiles\" tuning parameters. It defines how the AGC algorithm splits\nthe required exposure into shutter and gain provided to the sensor. I\nbelieve libipa does have exposure profile handling as well.\n\nRegards,\nNaush\n\n\n>\n> Regards,\n>\n> Hans\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C81B4C3241\n\tfor ;\n\tMon, 10 Nov 2025 09:58:37 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id CE8BB60A8B;\n\tMon, 10 Nov 2025 10:58:36 +0100 (CET)","from mail-ua1-x934.google.com (mail-ua1-x934.google.com\n\t[IPv6:2607:f8b0:4864:20::934])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CF1FC6069A\n\tfor ;\n\tMon, 10 Nov 2025 10:58:34 +0100 (CET)","by mail-ua1-x934.google.com with SMTP id\n\ta1e0cc1a2514c-937232a3b0cso46878241.3\n\tfor ;\n\tMon, 10 Nov 2025 01:58:34 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"R7Mz4YpY\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1762768713; x=1763373513;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=tUPW5uvrlcrLZFaiMJ9KT/XlYWJn8KxJ070Mw3JAwOk=;\n\tb=R7Mz4YpYth4YOEi4l3EY6hiCuj2q1341+hsvAOH8Xw8T1PEqpZD4+9hcSNxs5lsSlz\n\tPH3vsQIJBS+nUOzxUU4ksIwCmk0PKUNePmK6MqxQ8Z54gwCUSsHmiOusxez7LXIKfgJA\n\tgFw/mGQnf8ljarQq0+Tap0zr+ZgCv7oeYQaW1GH3VAliCIpoZGFUU/x+/Rc+5HQ0E74u\n\tGAOQj/H4Kf6TXne/+qO+8N7tdVCBinS3vY5xtpDv2ZIT7sD8Vexkt5qLBYjN4RiBfRjH\n\tjZQf2okQZSekzOCZQodXwKZbVCyU6aoMSdVM4RI+2scJ5vKM7dRUdEIUxAdF8EzOMIZ5\n\t4SJw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1762768713; x=1763373513;\n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from\n\t:to:cc:subject:date:message-id:reply-to;\n\tbh=tUPW5uvrlcrLZFaiMJ9KT/XlYWJn8KxJ070Mw3JAwOk=;\n\tb=wJA8MaOiTddAu02hB0ZHfc7db1wKZAlGCvdCcecPtjsW9lqRw3R5q2fGhv53mFiPZ0\n\tUAOWbIAaZUO5l9WAYLBywdDvfPgFkCkTyOjqRULSqhWRAzdobl+Adt2Mv/BZyIo1FdI5\n\trMQFOszXzlmpaBxYvjr3QOu2JjdNcnDO1weFOQI9k92cPjw/9Rboz0LsabFlY03U8pon\n\tk6m79JM4K/U29t6BFQisXUskaV4QP+izuK7DlyJ0ZQqEBwNRe68O32oUhxg9fABgxh7x\n\tDSSg/wVElTmzewKZvMEt7NtGmC+0e3gWRG309ZORUqjKXAMPUepFBfo+z7JVsBcsFUSn\n\tC5rA==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCXGIoLt2VCHCXgzgkBLMftS6p5+K/c65SfxAkMnmKAhRZgrXHvU0iVBc9r/H+ZXkS0DhaD4Y9vj0+wJLbPMr5g=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YxoMBjfH8mOaEDlEGNsn+XZQF+c+kICiMB2Fh9Lgp2XIxTUYQok\n\tOLwm5VGuVjnQ4ZgbPcm44wHp53JFNy1KvR7+f83s49UDU3B0OkWcDIIpFWawcMBdOcP6asJbgUK\n\t8RLcZ7zdZhTSFrfWBOf4gNWs08j1wm/nNXYD9o8zBKg==","X-Gm-Gg":"ASbGncto3M2tA9hONYUtFJbD687tLgE7SOgbwZT2p1FtoUOSKCGHNzVGBOWKg8jQBnA\n\tqa9nLBcF2jT2cg1la4MnqBdXZRQEHvxvMbXkfYmySa7wtVWVqkO++9YvCJOTJJfenxxqBp4i29P\n\tA6Y6rBwizQeZfDemHTe+ekod1FGniv/54QkWJxPm0816MVqe3K3we3QOQmk4jlGBaEj9Z0PcVQF\n\tBVDYN7Q0cr/XNHeMAdYafzw4Nu+5rLk102cjTo64AHrYc1FRmEPdEj0jku7YKqs0mKbd0faYbJu\n\tZlKGhE76NuKj0ra9","X-Google-Smtp-Source":"AGHT+IHPCcXNQ+EXDgRSbpVnnLWqpoh47IeBaVL5+WUm4/Z5ZIrp85Km2oSbywIqMjs9sZAh5Y4ucNKh/EIqCZVUd3Q=","X-Received":"by 2002:a05:6102:54a3:b0:5ca:6fe8:8b0f with SMTP id\n\tada2fe7eead31-5ddc4662cf6mr908004137.2.1762768713405; Mon, 10 Nov 2025\n\t01:58:33 -0800 (PST)","MIME-Version":"1.0","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>\n\t<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>\n\t\n\t<05eed1a8-8de8-49e3-9d65-44aa86d35d45@oss.qualcomm.com>\n\t\n\t<23bc1f82-b0f2-4c57-9596-f02b13d06534@oss.qualcomm.com>","In-Reply-To":"<23bc1f82-b0f2-4c57-9596-f02b13d06534@oss.qualcomm.com>","From":"Naushir Patuck ","Date":"Mon, 10 Nov 2025 09:57:57 +0000","X-Gm-Features":"AWmQ_blZ8_wENTPhkwa5kHu2BAjHdF3Q6AgUdEj0Yrl1E-XQOmBKoSLIJHusNvI","Message-ID":"","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Hans de Goede ","Cc":"Jacopo Mondi , =?utf-8?q?Niklas_S?=\n\t=?utf-8?b?w7ZkZXJsdW5k?= , Robert Mader\n\t, libcamera-devel@lists.libcamera.org, \n\tKieran Bingham ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36761,"web_url":"https://patchwork.libcamera.org/comment/36761/","msgid":"","date":"2025-11-10T10:20:28","subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/people/42/","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"content":"Hi everyone\n\nOn Mon, 10 Nov 2025 at 09:58, Naushir Patuck wrote:\n>\n> Hi Hans,\n>\n> On Mon, 10 Nov 2025 at 09:40, Hans de Goede\n> wrote:\n> >\n> > Hi Naush,\n> >\n> > On 10-Nov-25 10:05, Naushir Patuck wrote:\n> > > Hi Hans,\n> > >\n> > > On Fri, 7 Nov 2025 at 16:41, Hans de Goede\n> > > wrote:\n> > >>\n> > >> Hi Jacopo,\n> > >>\n> > >> On 6-Nov-25 9:29 AM, Jacopo Mondi wrote:\n> > >>> Hi Hans\n> > >>>\n> > >>> On Tue, Nov 04, 2025 at 06:23:23PM +0100, Hans de Goede wrote:\n> > >>>> Hi Jacopo,\n> > >>>>\n> > >>>> On 4-Nov-25 16:53, Jacopo Mondi wrote:\n> > >>>>> Hi Hans\n> > >>>>>\n> > >>>>> On Mon, Nov 03, 2025 at 10:01:15AM +0100, Hans de Goede wrote:\n> > >>>>>> Hi,\n> > >>>>>>\n> > >>>>>> On 28-Oct-25 4:38 PM, Niklas Söderlund wrote:\n> > >>>>>>> On 2025-10-28 15:48:58 +0100, Hans de Goede wrote:\n> > >>>>>>\n> > >>>>>> ...\n> > >>>>>>\n> > >>>>>>>> Right I forgot about the delayed-controls stuff, not being able\n> > >>>>>>>> to get the new exposure limits from the kernel then makes sense.\n> > >>>>>>>\n> > >>>>>>> I too understand why it's easier to not consult the kernel for the new\n> > >>>>>>> limits. I have no real strong feeling that the correct solution is to\n> > >>>>>>> have this limit calculation done in the kernel or user-space. What I do\n> > >>>>>>> feel somewhat strongly about is if we do the limit calculation in\n> > >>>>>>> user-space we should never consult the kernel for the limits.\n> > >>>>>>>\n> > >>>>>>> This just mixes two ways of doing the same thing and is very confusing\n> > >>>>>>> when reading the code. So if doing this in user-space is the way forward\n> > >>>>>>> I think we should drop the reading of min and max of V4L2_CID_EXPOSURE\n> > >>>>>>> in IPARkISP1::configure() and instead use this method there too.\n> > >>>>>>>\n> > >>>>>>> I also feel, a little less strongly as it's a big work item, that it's\n> > >>>>>>> really wrong to duplicate the calculation both in kernel and user-space.\n> > >>>>>>>\n> > >>>>>>> As this IMHO exposes a weakness in the V4L2 API we should think about\n> > >>>>>>> how we want the V4L2_CID_EXPOSURE control to report min/max values. Do\n> > >>>>>>> we really want it to report its min/max values based on the current\n> > >>>>>>> V4L2_CID_VBLANK setting as it's too expensive to query it with an IOCTL\n> > >>>>>>> for each frame.\n> > >>>>>>>\n> > >>>>>>> Can we simplify the control to report the min and max values the sensor\n> > >>>>>>> can do provided the VBLANK setting allows it, and just reject any value\n> > >>>>>>> at set time that do not conform to the current VBLANK setting?\n> > >>>>>>\n> > >>>>>> One complicating factor here is that some sensors will do vblank stretching\n> > >>>>>> when you ask for a higher exposure. IOW some sensors behave as if a bigger\n> > >>>>>> vblank is set when setting exposure > (frame-length + margin) to still\n> > >>>>>> give you the desired exposure.\n> > >>>>>\n> > >>>>> Interesting. Do you have any pointer to mainline drivers that do that ?\n> > >>>>\n> > >>>> I'm not talking about the driver stretching the vblank I'm talking\n> > >>>> about the hw (the sensor) stretching the vblank. I think that in newer\n> > >>>> drivers we stop this from happening by clamping the exposure.\n> > >>>>\n> > >>>> I'm 100% certain that some hw does this as I've seen this happen.\n> > >>>> I can do some digging if you want some specific sensor model examples\n> > >>>> where the hw is doing this.\n> > >>>\n> > >>> I think the fact some sensor supporting this feature not an issue if\n> > >>> we operate with algorithms designed to always clamp the shutter time\n> > >>> to the frame duration. Hence I would be more interested in knowing\n> > >>> which usage model you would consider best for the whole\n> > >>> libcamera/libipa handling of exposure/frame duration.\n> > >>>\n> > >>> Currently, the design is to have the maximum achievable shutter time\n> > >>> to be always constrained by the frame duration. Users should take\n> > >>> care of explicitly slowing down the camera frame rate in order to\n> > >>> achieve a larger shutter time.\n> > >>>\n> > >>> I think it is justified by the face that the desired exposure is\n> > >>> realized by the combined effect of the shutter time and gains, and\n> > >>> hence limiting the shutter time won't imply not being able to reach\n> > >>> the desired exposure levels.\n> > >>\n> > >> Right, except that in low light you may need to boost the gain\n> > >> to a level where you get significant noise and at some point\n> > >> the gain is mostly just amplifying noise (discussion continued\n> > >> below).\n> > >>\n> > >>> Alternative approaches where the exposure time drives the frame\n> > >>> duration could be implemented with helpers, similar to the already\n> > >>> discussed \"aperture priority\" vs \"shutter priority\" modes.\n> > >>\n> > >> Right, I do believe that we really need these modes, at least\n> > >> for laptops. Unlike the RPI cameras which have pretty nice large\n> > >> lenses. The laptop designs are optimized for small bezels and\n> > >> thus small lenses which means not a whole lot of light, which\n> > >> really starts to bite at low light conditions.\n> > >>\n> > >> And I've seen both Intel's proprietary IPU6 stack as well as\n> > >> many UVC webcams lower the framerate to compensate (somewhat).\n> > >>\n> > >> I'm fine with defaulting to never allowing the need for\n> > >> higher exposure to lower the framerate, but as you say I do\n> > >> believe we need a control to allow the AGC lowering the framerate\n> > >> to get larger exposure times. And I think those controls should\n> > >> likely even default to on devices with small lenses, but first\n> > >> we need to have / implement such an exposure-mode control to begin\n> > >> with :)\n> > >\n> > > I believe the current FrameDurationLimits will allow you to achieve\n> > > everything you need here. From the control definition:\n> > >\n> > > - FrameDurationLimits:\n> > > type: int64_t\n> > > direction: inout\n> > > description: |\n> > > The minimum and maximum (in that order) frame duration, expressed in\n> > > microseconds.\n> > >\n> > > When provided by applications, the control specifies the sensor frame\n> > > duration interval the pipeline has to use. This limits the largest\n> > > exposure time the sensor can use. For example, if a maximum frame\n> > > duration of 33ms is requested (corresponding to 30 frames per second),\n> > > the sensor will not be able to raise the exposure time above 33ms.\n> > > A fixed frame duration is achieved by setting the minimum and maximum\n> > > values to be the same. Setting both values to 0 reverts to using the\n> > > camera defaults.\n> > >\n> > > The maximum frame duration provides the absolute limit to the exposure\n> > > time computed by the AE algorithm and it overrides any exposure mode\n> > > setting specified with controls::AeExposureMode. Similarly, when a\n> > > manual exposure time is set through controls::ExposureTime, it also\n> > > gets clipped to the limits set by this control. When reported in\n> > > metadata, the control expresses the minimum and maximum frame durations\n> > > used after being clipped to the sensor provided frame duration limits.\n> > >\n> > > So this control takes a min and max value to use by the IPA. If you\n> > > specify, say, min = (1/30)s and max = (1/15)s, this should limit the\n> > > AGC to increase the shutter time upto 66ms if the scene conditions\n> > > require. This is the mechanism RPi uses to allow AEC to increase the\n> > > exposure without ramping up the analogue gain, if required.\n> >\n> > Right, this works if you are in full control of the applications and thus\n> > of the framerate range the application requests.\n> >\n> > But when using e.g. browsers for video conferencing they will just ask\n> > for a fixed framerate. I'm not even sure if the getUserMedia web APIs\n> > used by webrtc in-browser apps support a framerate range. So we may want to\n> > have another mechanism to allow to allow lowering the framerate in low\n> > light conditions, which on UVC cams at least is typically done through\n> > an aperture priority exposure mode.\n>\n> If the application/pipewire has set a fixed framerate to use, I don't\n> think the IPA should do anything other than run at this framerate,\n> regardless of lighting conditions. But perhaps this is not what you\n> mean here?\n>\n> >\n> > But I see from the description of the current frameDuration control that\n> > this would go against that description. So maybe we need to have pipewire\n> > always request a fps range of (desired-fps / 2) - desired-fps. Even then\n> > it would still be nice to have a control to tell the AGC if it should\n> > optimize for getting as much fps as possible (only lowering the fps\n> > if it needs a gain of say above 4.0 otherwise) or if it should try to\n> > keep the gain as low as possible and immediately start lowering fps\n> > if necessary.\n>\n> This makes sense, and how we achieve this is through the \"exposure\n> profiles\" tuning parameters. It defines how the AGC algorithm splits\n> the required exposure into shutter and gain provided to the sensor. I\n> believe libipa does have exposure profile handling as well.\n\nPersonally, I've always been of the opinion that there's a bigger issue here.\n\nFor example, a very common use case might be video conferencing. But\non a Raspberry Pi, when using particular sensors, we might have strong\nrecommendations on how the camera system should be configured. It\nmight include \"use this AGC mode\", \"activate HDR\", \"use a higher\ndefault EV\" and so on. How is Pipewire to know about this? Either it\nhas some quite sophisticated tables of parameters and settings, which\nfeels a bit undesirable to me. Or it can simply tell libcamera a\n\"high-level\" use case, such as \"I am doing a video conference\", and\nthen libcamera (or our pipeline handler underneath) deals with it.\n\nWhat do people think?\n\nDavid\n\n>\n> Regards,\n> Naush\n>\n>\n> >\n> > Regards,\n> >\n> > Hans\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 54095C3263\n\tfor ;\n\tMon, 10 Nov 2025 10:20:44 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 029A660A80;\n\tMon, 10 Nov 2025 11:20:44 +0100 (CET)","from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com\n\t[IPv6:2607:f8b0:4864:20::f32])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6CDF06069A\n\tfor ;\n\tMon, 10 Nov 2025 11:20:42 +0100 (CET)","by mail-qv1-xf32.google.com with SMTP id\n\t6a1803df08f44-88058103dcfso20263266d6.3\n\tfor ;\n\tMon, 10 Nov 2025 02:20:42 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"Xjwy1iHP\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1762770041; x=1763374841;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=pWSTUeiZqzgepll8J1EXWSBys+kDrCab8fiFKEHxe7w=;\n\tb=Xjwy1iHPsb07VZeJmkGWtOB89mPfr8KAt26ieXzILMgRuS/kZQIprgeZTKocf9q1Aa\n\tmgkvc8PXsLBca86U99eugg2xibZAzJ63F7ky39cDZh/2eaUpwZWy4tU7midVUc6sTn8s\n\t3OhTUi6KqzEVItSya+apAqNohynbxBvDNrRJ3tifWLLlyA1nrDw/BXRqrfJDhplnmFjq\n\tSVTfKLqbftP4+oMtXe/8xymOqdDJwjqhesgvH+GwlrcDLJ/JWW8ohrXL91cNkPrRzSTO\n\tAUiR5na8l1EC/uYUGWD3dmWiKiSZAZQEUdBNEuY2sgbexYb6ULRFs33f1gF9eCLmcR2O\n\tmlPg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1762770041; x=1763374841;\n\th=content-transfer-encoding:cc:to:subject:message-id:date:from\n\t:in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from\n\t:to:cc:subject:date:message-id:reply-to;\n\tbh=pWSTUeiZqzgepll8J1EXWSBys+kDrCab8fiFKEHxe7w=;\n\tb=a1yn1wxw7SHZiU5NRE7WIuQxDJucBMAtUg9WxDmtc6ukNqUA6/olFPpMo3GWgXRWgi\n\tx8qnRzv2z4ZL51a4PYfKs7w3rbrsvrLBxPqrPOFV8eHqB8glsG2ZcQ/CpwEV+6sb2rtl\n\teTDC0vVmb4IbKs3/F6Iip8VTEzAIkC9CjH+MwVChsuVfZ94tNeu2lURaF2SxwAiKOZ7f\n\t1+wXjZnzZdHr8759wNwTuAhd+1LxvZ6scDYkwf9aNagMsKbl8VS5TcD0fW7icELuIzdV\n\tW3Igl9iVronz/cSW+T/wpQOHnCTquznyYw6iA+l39q6voRuH20UpknErWFJGYdJ74AdD\n\tVhjQ==","X-Forwarded-Encrypted":"i=1;\n\tAJvYcCXocBcAwPq6EobTK8eGkokxFcuEfuRsrr6YlOddGobKVEy13QCYD9JVSTRTgUsGtZ9hWmHjRzKH9jbZSPY4WcY=@lists.libcamera.org","X-Gm-Message-State":"AOJu0YxOXFtBQBXuu89vji0PpduSNb5ryfjb6UblxIcdDQ+8TNk7+f8C\n\tssCP1+b+LXFvI4eM9wmm7Q7E4FzkwDaHDVLcfAmo+WW4zUxDSv2Wt7+X7UL1KNGXWoWtmBkowJF\n\tAkbDyBvlGHD1h2VszUl7XvfAAgskVhp0bLGurRt901g==","X-Gm-Gg":"ASbGncsg2pTO2w04kGlGlNco31wl9C2qjN8Vjes4aaQYga+LIiVb5MtSijIFUFPuT09\n\t9YJU2BET23iq0jEPbPthk9QZwZ8WSnb1cHkYzy3+rxhFos1taXpqJLd7zkkEtOt6PU9WnzJnLdY\n\tZWl8YXpiRJWicrjy7Rx+3RYAnjnMHQzesGPwMpm+M4M1pSwjI29/YAYsJmYHET7Mq/U2XV11HCx\n\tREWf8h/sXvDtI1PRy2zN0Ki0cSsM0fxso638nY6FSkwuVz3fsKMAQ1rkicr5wzLhnojdlyc/9RS\n\tVrQMIYS2xnLocUHZQ2nGX7qJNpajvh5Y5JKRqzrS5Tr1H2arbZIqAvWSd/GemdVGzCzpFXCTlkJ\n\tT","X-Google-Smtp-Source":"AGHT+IFQKzJVyD51rj5bqrNJDGMTkUa1Ugy8gSAZjnADSq448aUKgWXb9yXXNPsam6A5FynLUH7r5IPHI8hzQ1xZeRU=","X-Received":"by 2002:a05:6214:29c5:b0:880:88fa:d75a with SMTP id\n\t6a1803df08f44-88238715327mr106118006d6.67.1762770041148;\n\tMon, 10 Nov 2025 02:20:41 -0800 (PST)","MIME-Version":"1.0","References":"<20251028-exposure-limits-v2-0-a8b5a318323e@ideasonboard.com>\n\t\n\t\n\t\n\t<20251028153842.GC365372@ragnatech.se>\n\t<9d2a9135-3b9c-45f1-9131-fc8ac54cb56b@kernel.org>\n\t<3kpzvwo3hhvfsljdvvobriwlq57u4ivkssomhmcwsqzjzmb473@ydsd2ms2wzzd>\n\t<93ce834e-470a-4815-aab4-0aa2f4d59bef@kernel.org>\n\t\n\t<05eed1a8-8de8-49e3-9d65-44aa86d35d45@oss.qualcomm.com>\n\t\n\t<23bc1f82-b0f2-4c57-9596-f02b13d06534@oss.qualcomm.com>\n\t","In-Reply-To":"","From":"David Plowman ","Date":"Mon, 10 Nov 2025 10:20:28 +0000","X-Gm-Features":"AWmQ_bksyFt8PV4bXuB8FP6VZf4LaZN1XzEUY2gJzoarq6FucKX2BeLVn5eLc6s","Message-ID":"","Subject":"Re: [PATCH v2 00/10] libipa: agc: Calculate exposure limits","To":"Naushir Patuck ","Cc":"Hans de Goede , Jacopo Mondi\n\t, =?utf-8?q?Niklas_S=C3=B6derlund?=\n\t, Robert Mader\n\t, libcamera-devel@lists.libcamera.org, \n\tKieran Bingham ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]