| Message ID | 20251024085130.995967-1-stefan.klug@ideasonboard.com |
|---|---|
| Headers | show |
| Series |
|
| Related | show |
Hi Stefan Yes indeed, always excited when folks want to talk about PFC! On Fri, 24 Oct 2025 at 09:51, Stefan Klug <stefan.klug@ideasonboard.com> wrote: > > Hi all, > > This series accumulated for a while but I think it is time to get it > into the public. > > From a very highlevel view, this series implements per frame control on > the rkisp1. It does not yet touch the possible PFC mechanics like > fast-tracked controls. It was only tested on the imx8mp together with > the dewarper so there might still be issues on other rkisp1 platforms. In many ways there are probably too many implementation specifics about non-Raspberry Pi code for me really to appreciate what's going on, nonetheless some degree of high level understanding might still be helpful. Also, by "fast-tracked controls" I assume you're referring to Barnabas's proposal? As I remarked at the time, I liked that so long as it involves a queue, and I don't really have too many concerns about their interaction. > > The series depends on [PATCH v2 00/35] Full dewarper support on imx8mp > and won't apply otherwise. A complete branch is available here: > https://git.uk.ideasonboard.com/sklug/libcamera/src/branch/sklug/imx8mp-dewarp-and-regulation-v1 > > There are a lot of smaller cleanup patches in there, but to get the > overall picture I'll try to summarize my thoughts behind that: > > To do correct per-frame-control we need to synchronize multiple things: I guess this is the obvious question - what do we mean by "correct per-frame-control"? Can you perhaps give a brief description of what that means here, and perhaps how it compares with Raspberry Pi's version (as shown in Nice)? > - The sensor needs to be fed with parameters according to the specific > delays > - The ISP parameters need to be in sync also. This is especially > important if for example digital gain in the ISP is used to mitigate > quantization in the sensor gain and therefore need to perfectly match > the sensor timing. > > The current IPA model has basically two loops: > > The sensor controls loop: > - Pipeline::statsBufferReady() > - IPA::processStats() > - IPA::setSensorControls.emit() > - Pipeline::DelayedControls.push() > - IPA::metadataReady.emit() > - Pipeline::tryCompleteRequest() > > The ISP params loop: > - Pipeline::queueRequestDevice() > - IPA::queueRequest() > - IPA::computeParams() > - Pipeline::paramsComputed() > - Pipeline::queueParams() > > This means the sensor controls and params are only indirectly coupled. > To add to the complexity, in the sensor controls loop, the frame context > for the frame that produced the stats is used as basis for the call to > setSensorControls(). That is the point where I often fail to follow the > chain of events. > > The reworked model now changes the mechanics: In multiple discussions it > got clear that even though theoretically the ISP params have a 0-1 frame > delay and are therefore 1 frame faster than typical sensor delays it is > more important to ensure that sensor data and ISP params are in sync > than to optimiza for that one frame. We can still have a look at that > later. Stating that sensor and params should be in sync also means that > we can produce them at the same time. So the pipeline was reworked with > the following premises in mind: I certainly agree with keeping sensor and ISP parameters in sync. Not doing so would make them very difficult for applications to use, and I wouldn't want to inflict that on them! > > - Pipeline and IPA now internally handle sensor sequence numbers instead > of request numbers. > > The stats loop only updates active state and fills metadata: > - ISP calculates stats > - IPA::processStats() > - IPA::metadataReady.emit() > - Pipeline::tryCompleteRequest() > > The sensor loop triggers calculation of params + sensor controls: > - Pipeline::startFrame() > - Pipeline::DelayedControls.applyControls(n+delay) > - IPA::computeParams(n+delay+1) > - IPA::setSensorControls.emit() > - Pipeline::DelayedControls.push() > - Pipeline::paramsComputed() > - Pipeline::queueParams() > > The request loop: > - Pipeline::queueRequestDevice() > - IPA::queueRequest() > > The main change is that sensor controls and ISP params are computed in > response to computeParams(). This has the nice effect, that we just need > to ensure that we call computeParams() early enough to send it out to > the sensor as that is usually the longest delay. > > A request underrun is easy to handle as the params/stats machinery just > continues to run with scratch buffers in the kernel. The only change to > do there is to repurpose the IPA::queueRequest to a > IPA::initializeFrameContext and call that on demand when params for a > given frame are needed. > > Additionally to this conceptual change, the stats, image and params > buffers were decoupled. Due to the asyncronous nature of V4L2 and the > usage of scratch buffers in the kernel it is impossible to ensure that a > grouping of (stats, image and params) stays in sync. Especially under > high load or with bad connectors, this breaks. Decoupling them makes a > few things easier but also brings a bit of complexity. This rework and > most of the loop structure change is unfortunately in one big commit > (patch 27/35) as I didn't see a way to split that into easy to digest > parts. > > For the (re)synchronization to work properly, I needed to add > synchronization helpers. In the current code we have a minimal resync > facility that adds 1 if it detects a sequence that is bigger than the > expected one. The problem here is the queue in the kernel. If a offset > is detected when dequing a buffer, we can compensate for that offset > when queuing new buffers. But in case there are n buffers queued we will > most likely observe that error another n times until the compensation > applies. By then we have overcompensated by n and started a beautiful > oscillation. To handle that we need to track the observed error and the > applied compensation over time. > > Another reason for this series was to get started faster. Therefor > controls passed to Camera::start() are now handled properly. In that > context a initial params buffer is also filled. This ensures that the > stats on frame 0 are calculated based on the correct settings and > prevents large oscillations at the beginning of the stream. > > Open Todos: > - The semantics of delayed controls was modified to not delay, but more > to a record and query. Maybe we should rename it? > - The changed delayed controls breaks the unit tests and most likely > other pipelines > - Lot's of debug logs are generated internally. That might be a bit of > an overhead but is super useful to debug the pipeline. So I'd somehow > like to keep them in, but be able to compile time disable them. > - As always, a bit more cleanup on the patches and commit messages... > > I'm excited to hear your opinions. Me too! Best regards David > > Best regards, > Stefan > > Jacopo Mondi (1): > libcamera: v4l2_videodevice: Do not hide frame drops > > Stefan Klug (34): > libcamera: delayed_controls: Add push() function that accepts a > sequence number > libcamera: delayed_controls: Handle missed pushes > libcamera: delayed_controls: Increase log level for dummy pushes > libcamera: delayed_controls: Queue noop when needed, not before > libcamera: delayed_controls: Add maxDelay() function > pipeline: rkisp1: Include frame number when pushing to delayed > controls > libipa: fc_queue: Rename template argument to FC > libipa: fc_queue: Add trailing underscore to private members of > FrameContext > ipa: rkisp1: Refactor setControls() > pipeline: rkisp1: Add a frameStart function to handle > DelayedControls::applyControls > ipa: rkisp1: Move setSensorControls signal to computeParams > pipeline: rkisp1: Fix controls in raw mode > ipa: rkisp1: Add initializeFrameContext() function > pipeline: rkisp1: Apply initial controls > ipa: rkisp1: Set frameContext.agc in queueRequest for auto mode also > ipa: rkisp1: agc: Process frame duration at the right time > ipa: rkisp1: agc: Fix vblank, when computeParams prepare is not called > libcamera: delayed_controls: Change semantics of sequence numbers > libipa: algorithm: Update docs > libcamera: delayed_controls: Ignore double pushes for the same frame > number > ipa: rkisp1: Allow processStats to be called without stats buffer > ipa: rkisp1: Lazy initialise frame context > ipa: rkisp1: Ensure controls don't get lost > pipeline: rkisp1: Add SequenceSyncHelper class > ipa: rkisp1: awb: Ignore empty AWB statistics > pipeline: rkisp1: Decouple image, stats and param buffers > pipline: rkisp1: Reinstantiate maxQueuedRequestsDevice limit > ipa: libipa: Reduce log level when obtaining an uninitialized frame > context > pipeline: rkisp1: Correctly handle params buffer for frame 0 > WIP ipa: rkisp1: Post quantization gain as digital gain in metadata > WIP: rkisp1: agc: Add digital gain > libipa: agc_mean_luminance: Make startup frames and regulations speed > configurable > ipa: rkisp1: Increase regulation speed > [WIP] Add imx335 delays > > include/libcamera/internal/delayed_controls.h | 3 + > include/libcamera/internal/v4l2_videodevice.h | 1 - > include/libcamera/ipa/rkisp1.mojom | 10 +- > src/ipa/libipa/agc_mean_luminance.cpp | 25 +- > src/ipa/libipa/agc_mean_luminance.h | 4 + > src/ipa/libipa/algorithm.cpp | 28 +- > src/ipa/libipa/fc_queue.cpp | 8 +- > src/ipa/libipa/fc_queue.h | 75 +- > src/ipa/rkisp1/algorithms/agc.cpp | 56 +- > src/ipa/rkisp1/algorithms/agc.h | 3 +- > src/ipa/rkisp1/algorithms/awb.cpp | 7 + > src/ipa/rkisp1/ipa_context.h | 4 + > src/ipa/rkisp1/rkisp1.cpp | 105 ++- > src/libcamera/delayed_controls.cpp | 83 +- > src/libcamera/pipeline/rkisp1/rkisp1.cpp | 788 ++++++++++++------ > .../pipeline/rkisp1/sequence_sync_helper.h | 69 ++ > .../sensor/camera_sensor_properties.cpp | 7 +- > src/libcamera/v4l2_videodevice.cpp | 15 - > test/meson.build | 2 +- > 19 files changed, 903 insertions(+), 390 deletions(-) > create mode 100644 src/libcamera/pipeline/rkisp1/sequence_sync_helper.h > > -- > 2.48.1 > >
Hi David, Thanks for your message. Quoting David Plowman (2025-10-24 13:00:30) > Hi Stefan > > Yes indeed, always excited when folks want to talk about PFC! > > On Fri, 24 Oct 2025 at 09:51, Stefan Klug <stefan.klug@ideasonboard.com> wrote: > > > > Hi all, > > > > This series accumulated for a while but I think it is time to get it > > into the public. > > > > From a very highlevel view, this series implements per frame control on > > the rkisp1. It does not yet touch the possible PFC mechanics like > > fast-tracked controls. It was only tested on the imx8mp together with > > the dewarper so there might still be issues on other rkisp1 platforms. > > In many ways there are probably too many implementation specifics > about non-Raspberry Pi code for me really to appreciate what's going > on, nonetheless some degree of high level understanding might still be > helpful. > > Also, by "fast-tracked controls" I assume you're referring to > Barnabas's proposal? As I remarked at the time, I liked that so long > as it involves a queue, and I don't really have too many concerns > about their interaction. > > > > > The series depends on [PATCH v2 00/35] Full dewarper support on imx8mp > > and won't apply otherwise. A complete branch is available here: > > https://git.uk.ideasonboard.com/sklug/libcamera/src/branch/sklug/imx8mp-dewarp-and-regulation-v1 > > > > There are a lot of smaller cleanup patches in there, but to get the > > overall picture I'll try to summarize my thoughts behind that: > > > > To do correct per-frame-control we need to synchronize multiple things: > > I guess this is the obvious question - what do we mean by "correct > per-frame-control"? Can you perhaps give a brief description of what > that means here, and perhaps how it compares with Raspberry Pi's > version (as shown in Nice)? Maybe the word PFC is already a bit burned. The internal working title was merely "regulation rework". My main motivation was to have a deterministic machinery that (assuming everything is in flow and there are enough requests queued in) finishes a request with exactly the settings that were requested. This was not the case on rkisp1 and made any discussion about what we could do (like for example the RPi model with queuing less buffers but still having deterministic control) a nightmare to even think about. This series only modifies the inner workings. On top of that we can implement things like the above mentioned proposal or sequenceId that you proposed. But I didn't have a basis where I could guarantee that the ISP params really are in sync with the sensor params and stats which lead to oscillations all over the place and especially at start. So with this series hopefully merged sometime, I expect there to be mental space to look at the open points in the PFC space. One big piece of that puzzle is in my opinion the long wanted separation of buffers from requests. But that is for another day. So tldr, I don't want to push a new concept for PFC that goes against the RPi version. My plan is to build the ground so we can actually work on PFC. So PFC in this series is more "The pipeline handler knows for every frame what's in there :-)" I hope that answers your question. Best regards, Stefan > > > - The sensor needs to be fed with parameters according to the specific > > delays > > - The ISP parameters need to be in sync also. This is especially > > important if for example digital gain in the ISP is used to mitigate > > quantization in the sensor gain and therefore need to perfectly match > > the sensor timing. > > > > The current IPA model has basically two loops: > > > > The sensor controls loop: > > - Pipeline::statsBufferReady() > > - IPA::processStats() > > - IPA::setSensorControls.emit() > > - Pipeline::DelayedControls.push() > > - IPA::metadataReady.emit() > > - Pipeline::tryCompleteRequest() > > > > The ISP params loop: > > - Pipeline::queueRequestDevice() > > - IPA::queueRequest() > > - IPA::computeParams() > > - Pipeline::paramsComputed() > > - Pipeline::queueParams() > > > > This means the sensor controls and params are only indirectly coupled. > > To add to the complexity, in the sensor controls loop, the frame context > > for the frame that produced the stats is used as basis for the call to > > setSensorControls(). That is the point where I often fail to follow the > > chain of events. > > > > The reworked model now changes the mechanics: In multiple discussions it > > got clear that even though theoretically the ISP params have a 0-1 frame > > delay and are therefore 1 frame faster than typical sensor delays it is > > more important to ensure that sensor data and ISP params are in sync > > than to optimiza for that one frame. We can still have a look at that > > later. Stating that sensor and params should be in sync also means that > > we can produce them at the same time. So the pipeline was reworked with > > the following premises in mind: > > I certainly agree with keeping sensor and ISP parameters in sync. Not > doing so would make them very difficult for applications to use, and I > wouldn't want to inflict that on them! > > > > > - Pipeline and IPA now internally handle sensor sequence numbers instead > > of request numbers. > > > > The stats loop only updates active state and fills metadata: > > - ISP calculates stats > > - IPA::processStats() > > - IPA::metadataReady.emit() > > - Pipeline::tryCompleteRequest() > > > > The sensor loop triggers calculation of params + sensor controls: > > - Pipeline::startFrame() > > - Pipeline::DelayedControls.applyControls(n+delay) > > - IPA::computeParams(n+delay+1) > > - IPA::setSensorControls.emit() > > - Pipeline::DelayedControls.push() > > - Pipeline::paramsComputed() > > - Pipeline::queueParams() > > > > The request loop: > > - Pipeline::queueRequestDevice() > > - IPA::queueRequest() > > > > The main change is that sensor controls and ISP params are computed in > > response to computeParams(). This has the nice effect, that we just need > > to ensure that we call computeParams() early enough to send it out to > > the sensor as that is usually the longest delay. > > > > A request underrun is easy to handle as the params/stats machinery just > > continues to run with scratch buffers in the kernel. The only change to > > do there is to repurpose the IPA::queueRequest to a > > IPA::initializeFrameContext and call that on demand when params for a > > given frame are needed. > > > > Additionally to this conceptual change, the stats, image and params > > buffers were decoupled. Due to the asyncronous nature of V4L2 and the > > usage of scratch buffers in the kernel it is impossible to ensure that a > > grouping of (stats, image and params) stays in sync. Especially under > > high load or with bad connectors, this breaks. Decoupling them makes a > > few things easier but also brings a bit of complexity. This rework and > > most of the loop structure change is unfortunately in one big commit > > (patch 27/35) as I didn't see a way to split that into easy to digest > > parts. > > > > For the (re)synchronization to work properly, I needed to add > > synchronization helpers. In the current code we have a minimal resync > > facility that adds 1 if it detects a sequence that is bigger than the > > expected one. The problem here is the queue in the kernel. If a offset > > is detected when dequing a buffer, we can compensate for that offset > > when queuing new buffers. But in case there are n buffers queued we will > > most likely observe that error another n times until the compensation > > applies. By then we have overcompensated by n and started a beautiful > > oscillation. To handle that we need to track the observed error and the > > applied compensation over time. > > > > Another reason for this series was to get started faster. Therefor > > controls passed to Camera::start() are now handled properly. In that > > context a initial params buffer is also filled. This ensures that the > > stats on frame 0 are calculated based on the correct settings and > > prevents large oscillations at the beginning of the stream. > > > > Open Todos: > > - The semantics of delayed controls was modified to not delay, but more > > to a record and query. Maybe we should rename it? > > - The changed delayed controls breaks the unit tests and most likely > > other pipelines > > - Lot's of debug logs are generated internally. That might be a bit of > > an overhead but is super useful to debug the pipeline. So I'd somehow > > like to keep them in, but be able to compile time disable them. > > - As always, a bit more cleanup on the patches and commit messages... > > > > I'm excited to hear your opinions. > > Me too! > > Best regards > David > > > > > Best regards, > > Stefan > > > > Jacopo Mondi (1): > > libcamera: v4l2_videodevice: Do not hide frame drops > > > > Stefan Klug (34): > > libcamera: delayed_controls: Add push() function that accepts a > > sequence number > > libcamera: delayed_controls: Handle missed pushes > > libcamera: delayed_controls: Increase log level for dummy pushes > > libcamera: delayed_controls: Queue noop when needed, not before > > libcamera: delayed_controls: Add maxDelay() function > > pipeline: rkisp1: Include frame number when pushing to delayed > > controls > > libipa: fc_queue: Rename template argument to FC > > libipa: fc_queue: Add trailing underscore to private members of > > FrameContext > > ipa: rkisp1: Refactor setControls() > > pipeline: rkisp1: Add a frameStart function to handle > > DelayedControls::applyControls > > ipa: rkisp1: Move setSensorControls signal to computeParams > > pipeline: rkisp1: Fix controls in raw mode > > ipa: rkisp1: Add initializeFrameContext() function > > pipeline: rkisp1: Apply initial controls > > ipa: rkisp1: Set frameContext.agc in queueRequest for auto mode also > > ipa: rkisp1: agc: Process frame duration at the right time > > ipa: rkisp1: agc: Fix vblank, when computeParams prepare is not called > > libcamera: delayed_controls: Change semantics of sequence numbers > > libipa: algorithm: Update docs > > libcamera: delayed_controls: Ignore double pushes for the same frame > > number > > ipa: rkisp1: Allow processStats to be called without stats buffer > > ipa: rkisp1: Lazy initialise frame context > > ipa: rkisp1: Ensure controls don't get lost > > pipeline: rkisp1: Add SequenceSyncHelper class > > ipa: rkisp1: awb: Ignore empty AWB statistics > > pipeline: rkisp1: Decouple image, stats and param buffers > > pipline: rkisp1: Reinstantiate maxQueuedRequestsDevice limit > > ipa: libipa: Reduce log level when obtaining an uninitialized frame > > context > > pipeline: rkisp1: Correctly handle params buffer for frame 0 > > WIP ipa: rkisp1: Post quantization gain as digital gain in metadata > > WIP: rkisp1: agc: Add digital gain > > libipa: agc_mean_luminance: Make startup frames and regulations speed > > configurable > > ipa: rkisp1: Increase regulation speed > > [WIP] Add imx335 delays > > > > include/libcamera/internal/delayed_controls.h | 3 + > > include/libcamera/internal/v4l2_videodevice.h | 1 - > > include/libcamera/ipa/rkisp1.mojom | 10 +- > > src/ipa/libipa/agc_mean_luminance.cpp | 25 +- > > src/ipa/libipa/agc_mean_luminance.h | 4 + > > src/ipa/libipa/algorithm.cpp | 28 +- > > src/ipa/libipa/fc_queue.cpp | 8 +- > > src/ipa/libipa/fc_queue.h | 75 +- > > src/ipa/rkisp1/algorithms/agc.cpp | 56 +- > > src/ipa/rkisp1/algorithms/agc.h | 3 +- > > src/ipa/rkisp1/algorithms/awb.cpp | 7 + > > src/ipa/rkisp1/ipa_context.h | 4 + > > src/ipa/rkisp1/rkisp1.cpp | 105 ++- > > src/libcamera/delayed_controls.cpp | 83 +- > > src/libcamera/pipeline/rkisp1/rkisp1.cpp | 788 ++++++++++++------ > > .../pipeline/rkisp1/sequence_sync_helper.h | 69 ++ > > .../sensor/camera_sensor_properties.cpp | 7 +- > > src/libcamera/v4l2_videodevice.cpp | 15 - > > test/meson.build | 2 +- > > 19 files changed, 903 insertions(+), 390 deletions(-) > > create mode 100644 src/libcamera/pipeline/rkisp1/sequence_sync_helper.h > > > > -- > > 2.48.1 > > > >
Hi all, This series accumulated for a while but I think it is time to get it into the public. From a very highlevel view, this series implements per frame control on the rkisp1. It does not yet touch the possible PFC mechanics like fast-tracked controls. It was only tested on the imx8mp together with the dewarper so there might still be issues on other rkisp1 platforms. The series depends on [PATCH v2 00/35] Full dewarper support on imx8mp and won't apply otherwise. A complete branch is available here: https://git.uk.ideasonboard.com/sklug/libcamera/src/branch/sklug/imx8mp-dewarp-and-regulation-v1 There are a lot of smaller cleanup patches in there, but to get the overall picture I'll try to summarize my thoughts behind that: To do correct per-frame-control we need to synchronize multiple things: - The sensor needs to be fed with parameters according to the specific delays - The ISP parameters need to be in sync also. This is especially important if for example digital gain in the ISP is used to mitigate quantization in the sensor gain and therefore need to perfectly match the sensor timing. The current IPA model has basically two loops: The sensor controls loop: - Pipeline::statsBufferReady() - IPA::processStats() - IPA::setSensorControls.emit() - Pipeline::DelayedControls.push() - IPA::metadataReady.emit() - Pipeline::tryCompleteRequest() The ISP params loop: - Pipeline::queueRequestDevice() - IPA::queueRequest() - IPA::computeParams() - Pipeline::paramsComputed() - Pipeline::queueParams() This means the sensor controls and params are only indirectly coupled. To add to the complexity, in the sensor controls loop, the frame context for the frame that produced the stats is used as basis for the call to setSensorControls(). That is the point where I often fail to follow the chain of events. The reworked model now changes the mechanics: In multiple discussions it got clear that even though theoretically the ISP params have a 0-1 frame delay and are therefore 1 frame faster than typical sensor delays it is more important to ensure that sensor data and ISP params are in sync than to optimiza for that one frame. We can still have a look at that later. Stating that sensor and params should be in sync also means that we can produce them at the same time. So the pipeline was reworked with the following premises in mind: - Pipeline and IPA now internally handle sensor sequence numbers instead of request numbers. The stats loop only updates active state and fills metadata: - ISP calculates stats - IPA::processStats() - IPA::metadataReady.emit() - Pipeline::tryCompleteRequest() The sensor loop triggers calculation of params + sensor controls: - Pipeline::startFrame() - Pipeline::DelayedControls.applyControls(n+delay) - IPA::computeParams(n+delay+1) - IPA::setSensorControls.emit() - Pipeline::DelayedControls.push() - Pipeline::paramsComputed() - Pipeline::queueParams() The request loop: - Pipeline::queueRequestDevice() - IPA::queueRequest() The main change is that sensor controls and ISP params are computed in response to computeParams(). This has the nice effect, that we just need to ensure that we call computeParams() early enough to send it out to the sensor as that is usually the longest delay. A request underrun is easy to handle as the params/stats machinery just continues to run with scratch buffers in the kernel. The only change to do there is to repurpose the IPA::queueRequest to a IPA::initializeFrameContext and call that on demand when params for a given frame are needed. Additionally to this conceptual change, the stats, image and params buffers were decoupled. Due to the asyncronous nature of V4L2 and the usage of scratch buffers in the kernel it is impossible to ensure that a grouping of (stats, image and params) stays in sync. Especially under high load or with bad connectors, this breaks. Decoupling them makes a few things easier but also brings a bit of complexity. This rework and most of the loop structure change is unfortunately in one big commit (patch 27/35) as I didn't see a way to split that into easy to digest parts. For the (re)synchronization to work properly, I needed to add synchronization helpers. In the current code we have a minimal resync facility that adds 1 if it detects a sequence that is bigger than the expected one. The problem here is the queue in the kernel. If a offset is detected when dequing a buffer, we can compensate for that offset when queuing new buffers. But in case there are n buffers queued we will most likely observe that error another n times until the compensation applies. By then we have overcompensated by n and started a beautiful oscillation. To handle that we need to track the observed error and the applied compensation over time. Another reason for this series was to get started faster. Therefor controls passed to Camera::start() are now handled properly. In that context a initial params buffer is also filled. This ensures that the stats on frame 0 are calculated based on the correct settings and prevents large oscillations at the beginning of the stream. Open Todos: - The semantics of delayed controls was modified to not delay, but more to a record and query. Maybe we should rename it? - The changed delayed controls breaks the unit tests and most likely other pipelines - Lot's of debug logs are generated internally. That might be a bit of an overhead but is super useful to debug the pipeline. So I'd somehow like to keep them in, but be able to compile time disable them. - As always, a bit more cleanup on the patches and commit messages... I'm excited to hear your opinions. Best regards, Stefan Jacopo Mondi (1): libcamera: v4l2_videodevice: Do not hide frame drops Stefan Klug (34): libcamera: delayed_controls: Add push() function that accepts a sequence number libcamera: delayed_controls: Handle missed pushes libcamera: delayed_controls: Increase log level for dummy pushes libcamera: delayed_controls: Queue noop when needed, not before libcamera: delayed_controls: Add maxDelay() function pipeline: rkisp1: Include frame number when pushing to delayed controls libipa: fc_queue: Rename template argument to FC libipa: fc_queue: Add trailing underscore to private members of FrameContext ipa: rkisp1: Refactor setControls() pipeline: rkisp1: Add a frameStart function to handle DelayedControls::applyControls ipa: rkisp1: Move setSensorControls signal to computeParams pipeline: rkisp1: Fix controls in raw mode ipa: rkisp1: Add initializeFrameContext() function pipeline: rkisp1: Apply initial controls ipa: rkisp1: Set frameContext.agc in queueRequest for auto mode also ipa: rkisp1: agc: Process frame duration at the right time ipa: rkisp1: agc: Fix vblank, when computeParams prepare is not called libcamera: delayed_controls: Change semantics of sequence numbers libipa: algorithm: Update docs libcamera: delayed_controls: Ignore double pushes for the same frame number ipa: rkisp1: Allow processStats to be called without stats buffer ipa: rkisp1: Lazy initialise frame context ipa: rkisp1: Ensure controls don't get lost pipeline: rkisp1: Add SequenceSyncHelper class ipa: rkisp1: awb: Ignore empty AWB statistics pipeline: rkisp1: Decouple image, stats and param buffers pipline: rkisp1: Reinstantiate maxQueuedRequestsDevice limit ipa: libipa: Reduce log level when obtaining an uninitialized frame context pipeline: rkisp1: Correctly handle params buffer for frame 0 WIP ipa: rkisp1: Post quantization gain as digital gain in metadata WIP: rkisp1: agc: Add digital gain libipa: agc_mean_luminance: Make startup frames and regulations speed configurable ipa: rkisp1: Increase regulation speed [WIP] Add imx335 delays include/libcamera/internal/delayed_controls.h | 3 + include/libcamera/internal/v4l2_videodevice.h | 1 - include/libcamera/ipa/rkisp1.mojom | 10 +- src/ipa/libipa/agc_mean_luminance.cpp | 25 +- src/ipa/libipa/agc_mean_luminance.h | 4 + src/ipa/libipa/algorithm.cpp | 28 +- src/ipa/libipa/fc_queue.cpp | 8 +- src/ipa/libipa/fc_queue.h | 75 +- src/ipa/rkisp1/algorithms/agc.cpp | 56 +- src/ipa/rkisp1/algorithms/agc.h | 3 +- src/ipa/rkisp1/algorithms/awb.cpp | 7 + src/ipa/rkisp1/ipa_context.h | 4 + src/ipa/rkisp1/rkisp1.cpp | 105 ++- src/libcamera/delayed_controls.cpp | 83 +- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 788 ++++++++++++------ .../pipeline/rkisp1/sequence_sync_helper.h | 69 ++ .../sensor/camera_sensor_properties.cpp | 7 +- src/libcamera/v4l2_videodevice.cpp | 15 - test/meson.build | 2 +- 19 files changed, 903 insertions(+), 390 deletions(-) create mode 100644 src/libcamera/pipeline/rkisp1/sequence_sync_helper.h