| Message ID | 20260312160009.18654-2-david.plowman@raspberrypi.com |
|---|---|
| State | New |
| Headers | show |
| Series |
|
| Related | show |
Hi everyone I'd like to give this topic another prod. I talked about it a bit at Nice, and revisiting this patch seems like a way to start some discussion, though the precise implementation here is more for illustration. One of the principal motivations is for things like burst captures, where you don't want to get held up at the back of the request queue. Does this seem like a reasonable thing to do, or are there better alternatives? Thanks David On Thu, 12 Mar 2026 at 16:00, David Plowman <david.plowman@raspberrypi.com> wrote: > > Add `Camera::queueControls()` whose purpose is to apply controls as > soon as possible, without going through `Request::controls()`. > > A new virtual function `PipelineHandler::queueControlsDevice()` is > provided for pipeline handler to implement fast-tracked application of > controls. If the pipeline handler does not implement that > functionality, or it fails, then a fallback mechanism is used. The > controls will be saved for later, and they will be merged into the > control list of the next available request sent to the pipeline > handler (`Camera::Private::waitingRequests_`). > > This patch is derived directly from Barnabas's previous verion that > implemented the same idea but with a single ControlList, rather than > allowing multiple ControlLists to be queued up for consecutive > frames. > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com> > --- > include/libcamera/camera.h | 1 + > include/libcamera/internal/camera.h | 1 + > include/libcamera/internal/pipeline_handler.h | 7 ++ > src/libcamera/camera.cpp | 61 +++++++++++++++ > src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ > 5 files changed, 146 insertions(+) > > diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h > index b24a2974..93a484e4 100644 > --- a/include/libcamera/camera.h > +++ b/include/libcamera/camera.h > @@ -147,6 +147,7 @@ public: > > std::unique_ptr<Request> createRequest(uint64_t cookie = 0); > int queueRequest(Request *request); > + int queueControls(ControlList &&controls); > > int start(const ControlList *controls = nullptr); > int stop(); > diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h > index 8a2e9ed5..17dda925 100644 > --- a/include/libcamera/internal/camera.h > +++ b/include/libcamera/internal/camera.h > @@ -38,6 +38,7 @@ public: > > std::list<Request *> queuedRequests_; > std::queue<Request *> waitingRequests_; > + std::queue<ControlList> queuedControls_; > ControlInfoMap controlInfo_; > ControlList properties_; > > diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h > index b4f97477..c25213de 100644 > --- a/include/libcamera/internal/pipeline_handler.h > +++ b/include/libcamera/internal/pipeline_handler.h > @@ -57,6 +57,7 @@ public: > > void registerRequest(Request *request); > void queueRequest(Request *request); > + int queueControls(Camera *camera, ControlList controls); > > bool completeBuffer(Request *request, FrameBuffer *buffer); > void completeRequest(Request *request); > @@ -76,6 +77,12 @@ protected: > unsigned int useCount() const { return useCount_; } > > virtual int queueRequestDevice(Camera *camera, Request *request) = 0; > + > + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) > + { > + return -EOPNOTSUPP; > + } > + > virtual void stopDevice(Camera *camera) = 0; > > virtual bool acquireDevice(Camera *camera); > diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp > index f724a1be..f0244707 100644 > --- a/src/libcamera/camera.cpp > +++ b/src/libcamera/camera.cpp > @@ -637,6 +637,16 @@ Camera::Private::~Private() > * queued requests was reached. > */ > > +/** > + * \var Camera::Private::queuedControls_ > + * \brief The queue of pending control lists > + * > + * This queue maintains a list of all the control lists that need to be sent > + * to the pipeline handler with subsequent requests. The top item in the queue > + * will always be sent with the next request going to > + * PipelineHandler::queueRequestDevice(). > + */ > + > /** > * \var Camera::Private::controlInfo_ > * \brief The set of controls supported by the camera > @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) > return 0; > } > > +/** > + * \brief Queue controls to be applied as soon as possible > + * \param[in] controls The list of controls to queue > + * > + * This function tries to ensure that the controls in \a controls are applied > + * to the camera as soon as possible. If there are still pending controls waiting > + * to be applied (because of previous calls to Camera::queueControls), then > + * these controls will be applied as soon as possible on a frame after those. > + * > + * The exact guarantees are camera dependent, but it is guaranteed that the > + * controls will be applied no later than with the next \ref Request"request" > + * that the application \ref Camera::queueRequest() "queues" (after any requests > + * have been *used up" for sending previously queued controls). > + * > + * \context This function is \threadsafe. It may only be called when the camera > + * is in the Running state as defined in \ref camera_operation. > + * > + * \return 0 on success or a negative error code otherwise > + * \retval -ENODEV The camera has been disconnected from the system > + * \retval -EACCES The camera is not running > + */ > +int Camera::queueControls(ControlList &&controls) > +{ > + Private *const d = _d(); > + > + /* > + * Like requests, controls can't be queued if the camera is not running. > + * Controls can be applied immediately when the camera starts using the > + * Camera::Start method. > + */ > + > + int ret = d->isAccessAllowed(Private::CameraRunning); > + if (ret < 0) > + return ret; > + > + /* > + * We want to be able to queue empty control lists, as this gives a way of > + * forcing another frame with the same controls as last time, before queueing > + * another control list that might change them again. > + */ > + > + patchControlList(controls); > + > + /* > + * \todo Or `ConnectionTypeBlocking` to get the return value? > + */ > + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); > + > + return 0; > +} > + > /** > * \brief Start capture from camera > * \param[in] controls Controls to be applied before starting the Camera > diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp > index 5c469e5b..1a87b28c 100644 > --- a/src/libcamera/pipeline_handler.cpp > +++ b/src/libcamera/pipeline_handler.cpp > @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) > ASSERT(data->queuedRequests_.empty()); > ASSERT(data->waitingRequests_.empty()); > > + /* > + * Clear out any unapplied controls. If an application wants to be > + * sure controls have been applied, it should wait before stopping. > + */ > + data->queuedControls_ = {}; > + > data->requestSequence_ = 0; > } > > @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) > request->_d()->prepare(300ms); > } > > +/** > + * \brief Queue controls to apply as soon as possible > + * \param[in] camera The camera > + * \param[in] controls The controls to apply > + * > + * This function tries to queue \a controls immediately to the device by > + * calling queueControlsDevice(). If that fails, then a fallback mechanism > + * is used to ensure that \a controls will be merged into the control list > + * of the next available request submitted to the pipeline handler. > + * > + * \context This function is called from the CameraManager thread. > + */ > +int PipelineHandler::queueControls(Camera *camera, ControlList controls) > +{ > + Camera::Private *data = camera->_d(); > + int ret = queueControlsDevice(camera, controls); > + > + /* > + * Don't worry about later request's controls overriding the ones > + * sent here - the application needs to deal with that. > + */ > + > + if (ret == -EOPNOTSUPP) { > + /* > + * Fall back to adding the controls to the next request that enters the > + * pipeline handler. See PipelineHandler::doQueueRequest(). > + */ > + data->queuedControls_.push(std::move(controls)); > + > + /* Counts as "success". */ > + ret = 0; > + > + } else if (ret < 0) { > + /* > + * The pipeline handler is claiming to support queueControlsDevice, > + * but it has failed. This is an error. > + */ > + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; > + } > + > + return ret; > +} > + > /** > * \brief Queue one requests to the device > */ > @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) > return; > } > > + if (!data->queuedControls_.empty()) { > + /* > + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is > + * needed to ensure that if `request` is newer than pendingControls_, > + * then its controls take precedence. > + */ > + request->controls().merge(data->queuedControls_.front(), > + ControlList::MergePolicy::KeepExisting); > + } > + > int ret = queueRequestDevice(camera, request); > if (ret) > cancelRequest(request); > + else if (!data->queuedControls_.empty()) > + data->queuedControls_.pop(); > } > > /** > @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) > * \return 0 on success or a negative error code otherwise > */ > > +/** > + * \fn PipelineHandler::queueControlsDevice() > + * \brief Queue controls to be applied as soon as possible > + * \param[in] camera The camera > + * \param[in] controls The controls to apply > + * > + * This function queues \a controls to \a camera so that they can be > + * applied as soon as possible > + * > + * \context This function is called from the CameraManager thread. > + * > + * \return 0 on success or a negative error code otherwise > + * \return -EOPNOTSUPP if fast-tracking controls is not supported > + */ > + > /** > * \brief Complete a buffer for a request > * \param[in] request The request the buffer belongs to > -- > 2.47.3 >
Hi 2026. 07. 23. 18:17 keltezéssel, David Plowman írta: > Hi everyone > > I'd like to give this topic another prod. I talked about it a bit at > Nice, and revisiting this patch seems like a way to start some > discussion, though the precise implementation here is more for > illustration. > > One of the principal motivations is for things like burst captures, > where you don't want to get held up at the back of the request queue. > > Does this seem like a reasonable thing to do, or are there better alternatives? I started typing out a reply in a different thread of yours, but it got too long, so I though it would be more effective to write a quick reply here. I believe there is largely agreement that some kind non-request-submission-tied control setting mechanism is desirable. So I think it would be useful to examine the motivating use cases in a bit more detail. Sorry if this was already done somewhere, I couldn't recall. So the way I understand it, there is a user that is cycling through a set of requests, keeping most of them queued. This works fine, however, sometimes, maybe due to some external event, it wants to capture a number of frames with specific settings. The reason why applying the controls in the subsequent requests is undesirable is because of the delay that the already queued requests introduce. The proposed solution is to have two parallel queues: one for requests, and one for controls. Furthermore, the two are consumed in lockstep. And there is an assumption that the control is queue is mostly empty. This way when the user submits a control list, it can effectively "skip" the requests in the parallel request queue, and arrive near the front of the control queue. And thus it will be "applied to an earlier request", reducing latency. Is this a faithful description? If so, I have two questions: * what if the user wants to enable/disable various streams for the "burst capture"? do you have any thoughts about how that API would work? * is it guaranteed that every control list will be applied to the results of the corresponding request? if not, how would that work with the user enabling/disabling various streams (during the "burst capture")? Thanks, Barnabás Pőcze > > Thanks > > David > > On Thu, 12 Mar 2026 at 16:00, David Plowman > <david.plowman@raspberrypi.com> wrote: >> >> Add `Camera::queueControls()` whose purpose is to apply controls as >> soon as possible, without going through `Request::controls()`. >> >> A new virtual function `PipelineHandler::queueControlsDevice()` is >> provided for pipeline handler to implement fast-tracked application of >> controls. If the pipeline handler does not implement that >> functionality, or it fails, then a fallback mechanism is used. The >> controls will be saved for later, and they will be merged into the >> control list of the next available request sent to the pipeline >> handler (`Camera::Private::waitingRequests_`). >> >> This patch is derived directly from Barnabas's previous verion that >> implemented the same idea but with a single ControlList, rather than >> allowing multiple ControlLists to be queued up for consecutive >> frames. >> >> Signed-off-by: David Plowman <david.plowman@raspberrypi.com> >> --- >> include/libcamera/camera.h | 1 + >> include/libcamera/internal/camera.h | 1 + >> include/libcamera/internal/pipeline_handler.h | 7 ++ >> src/libcamera/camera.cpp | 61 +++++++++++++++ >> src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ >> 5 files changed, 146 insertions(+) >> >> diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h >> index b24a2974..93a484e4 100644 >> --- a/include/libcamera/camera.h >> +++ b/include/libcamera/camera.h >> @@ -147,6 +147,7 @@ public: >> >> std::unique_ptr<Request> createRequest(uint64_t cookie = 0); >> int queueRequest(Request *request); >> + int queueControls(ControlList &&controls); >> >> int start(const ControlList *controls = nullptr); >> int stop(); >> diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h >> index 8a2e9ed5..17dda925 100644 >> --- a/include/libcamera/internal/camera.h >> +++ b/include/libcamera/internal/camera.h >> @@ -38,6 +38,7 @@ public: >> >> std::list<Request *> queuedRequests_; >> std::queue<Request *> waitingRequests_; >> + std::queue<ControlList> queuedControls_; >> ControlInfoMap controlInfo_; >> ControlList properties_; >> >> diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h >> index b4f97477..c25213de 100644 >> --- a/include/libcamera/internal/pipeline_handler.h >> +++ b/include/libcamera/internal/pipeline_handler.h >> @@ -57,6 +57,7 @@ public: >> >> void registerRequest(Request *request); >> void queueRequest(Request *request); >> + int queueControls(Camera *camera, ControlList controls); >> >> bool completeBuffer(Request *request, FrameBuffer *buffer); >> void completeRequest(Request *request); >> @@ -76,6 +77,12 @@ protected: >> unsigned int useCount() const { return useCount_; } >> >> virtual int queueRequestDevice(Camera *camera, Request *request) = 0; >> + >> + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) >> + { >> + return -EOPNOTSUPP; >> + } >> + >> virtual void stopDevice(Camera *camera) = 0; >> >> virtual bool acquireDevice(Camera *camera); >> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp >> index f724a1be..f0244707 100644 >> --- a/src/libcamera/camera.cpp >> +++ b/src/libcamera/camera.cpp >> @@ -637,6 +637,16 @@ Camera::Private::~Private() >> * queued requests was reached. >> */ >> >> +/** >> + * \var Camera::Private::queuedControls_ >> + * \brief The queue of pending control lists >> + * >> + * This queue maintains a list of all the control lists that need to be sent >> + * to the pipeline handler with subsequent requests. The top item in the queue >> + * will always be sent with the next request going to >> + * PipelineHandler::queueRequestDevice(). >> + */ >> + >> /** >> * \var Camera::Private::controlInfo_ >> * \brief The set of controls supported by the camera >> @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) >> return 0; >> } >> >> +/** >> + * \brief Queue controls to be applied as soon as possible >> + * \param[in] controls The list of controls to queue >> + * >> + * This function tries to ensure that the controls in \a controls are applied >> + * to the camera as soon as possible. If there are still pending controls waiting >> + * to be applied (because of previous calls to Camera::queueControls), then >> + * these controls will be applied as soon as possible on a frame after those. >> + * >> + * The exact guarantees are camera dependent, but it is guaranteed that the >> + * controls will be applied no later than with the next \ref Request"request" >> + * that the application \ref Camera::queueRequest() "queues" (after any requests >> + * have been *used up" for sending previously queued controls). >> + * >> + * \context This function is \threadsafe. It may only be called when the camera >> + * is in the Running state as defined in \ref camera_operation. >> + * >> + * \return 0 on success or a negative error code otherwise >> + * \retval -ENODEV The camera has been disconnected from the system >> + * \retval -EACCES The camera is not running >> + */ >> +int Camera::queueControls(ControlList &&controls) >> +{ >> + Private *const d = _d(); >> + >> + /* >> + * Like requests, controls can't be queued if the camera is not running. >> + * Controls can be applied immediately when the camera starts using the >> + * Camera::Start method. >> + */ >> + >> + int ret = d->isAccessAllowed(Private::CameraRunning); >> + if (ret < 0) >> + return ret; >> + >> + /* >> + * We want to be able to queue empty control lists, as this gives a way of >> + * forcing another frame with the same controls as last time, before queueing >> + * another control list that might change them again. >> + */ >> + >> + patchControlList(controls); >> + >> + /* >> + * \todo Or `ConnectionTypeBlocking` to get the return value? >> + */ >> + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); >> + >> + return 0; >> +} >> + >> /** >> * \brief Start capture from camera >> * \param[in] controls Controls to be applied before starting the Camera >> diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp >> index 5c469e5b..1a87b28c 100644 >> --- a/src/libcamera/pipeline_handler.cpp >> +++ b/src/libcamera/pipeline_handler.cpp >> @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) >> ASSERT(data->queuedRequests_.empty()); >> ASSERT(data->waitingRequests_.empty()); >> >> + /* >> + * Clear out any unapplied controls. If an application wants to be >> + * sure controls have been applied, it should wait before stopping. >> + */ >> + data->queuedControls_ = {}; >> + >> data->requestSequence_ = 0; >> } >> >> @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) >> request->_d()->prepare(300ms); >> } >> >> +/** >> + * \brief Queue controls to apply as soon as possible >> + * \param[in] camera The camera >> + * \param[in] controls The controls to apply >> + * >> + * This function tries to queue \a controls immediately to the device by >> + * calling queueControlsDevice(). If that fails, then a fallback mechanism >> + * is used to ensure that \a controls will be merged into the control list >> + * of the next available request submitted to the pipeline handler. >> + * >> + * \context This function is called from the CameraManager thread. >> + */ >> +int PipelineHandler::queueControls(Camera *camera, ControlList controls) >> +{ >> + Camera::Private *data = camera->_d(); >> + int ret = queueControlsDevice(camera, controls); >> + >> + /* >> + * Don't worry about later request's controls overriding the ones >> + * sent here - the application needs to deal with that. >> + */ >> + >> + if (ret == -EOPNOTSUPP) { >> + /* >> + * Fall back to adding the controls to the next request that enters the >> + * pipeline handler. See PipelineHandler::doQueueRequest(). >> + */ >> + data->queuedControls_.push(std::move(controls)); >> + >> + /* Counts as "success". */ >> + ret = 0; >> + >> + } else if (ret < 0) { >> + /* >> + * The pipeline handler is claiming to support queueControlsDevice, >> + * but it has failed. This is an error. >> + */ >> + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; >> + } >> + >> + return ret; >> +} >> + >> /** >> * \brief Queue one requests to the device >> */ >> @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) >> return; >> } >> >> + if (!data->queuedControls_.empty()) { >> + /* >> + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is >> + * needed to ensure that if `request` is newer than pendingControls_, >> + * then its controls take precedence. >> + */ >> + request->controls().merge(data->queuedControls_.front(), >> + ControlList::MergePolicy::KeepExisting); >> + } >> + >> int ret = queueRequestDevice(camera, request); >> if (ret) >> cancelRequest(request); >> + else if (!data->queuedControls_.empty()) >> + data->queuedControls_.pop(); >> } >> >> /** >> @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) >> * \return 0 on success or a negative error code otherwise >> */ >> >> +/** >> + * \fn PipelineHandler::queueControlsDevice() >> + * \brief Queue controls to be applied as soon as possible >> + * \param[in] camera The camera >> + * \param[in] controls The controls to apply >> + * >> + * This function queues \a controls to \a camera so that they can be >> + * applied as soon as possible >> + * >> + * \context This function is called from the CameraManager thread. >> + * >> + * \return 0 on success or a negative error code otherwise >> + * \return -EOPNOTSUPP if fast-tracking controls is not supported >> + */ >> + >> /** >> * \brief Complete a buffer for a request >> * \param[in] request The request the buffer belongs to >> -- >> 2.47.3 >>
Hi Barnabas Thanks for the message. I think you were right about everything, but I can certainly expand a bit here and there. On Fri, 7 Aug 2026 at 18:43, Barnabás Pőcze <barnabas.pocze@ideasonboard.com> wrote: > > Hi > > 2026. 07. 23. 18:17 keltezéssel, David Plowman írta: > > Hi everyone > > > > I'd like to give this topic another prod. I talked about it a bit at > > Nice, and revisiting this patch seems like a way to start some > > discussion, though the precise implementation here is more for > > illustration. > > > > One of the principal motivations is for things like burst captures, > > where you don't want to get held up at the back of the request queue. > > > > Does this seem like a reasonable thing to do, or are there better alternatives? > > I started typing out a reply in a different thread of yours, but it got too long, > so I though it would be more effective to write a quick reply here. > > I believe there is largely agreement that some kind non-request-submission-tied control > setting mechanism is desirable. > > So I think it would be useful to examine the motivating use cases in a bit more detail. > Sorry if this was already done somewhere, I couldn't recall. > > So the way I understand it, there is a user that is cycling through a set of requests, > keeping most of them queued. This works fine, however, sometimes, maybe due to some > external event, it wants to capture a number of frames with specific settings. The > reason why applying the controls in the subsequent requests is undesirable is because > of the delay that the already queued requests introduce. Yes, agree. Burst captures are the most obvious example, perhaps. But pretty much any user event (such as changing a white balance mode) is better if it happens quickly. > > The proposed solution is to have two parallel queues: one for requests, and one for controls. > Furthermore, the two are consumed in lockstep. And there is an assumption that the control > is queue is mostly empty. This way when the user submits a control list, it can effectively > "skip" the requests in the parallel request queue, and arrive near the front of the control > queue. And thus it will be "applied to an earlier request", reducing latency. Yes, the point being that it will be applied as soon as possible. As a side note, I also quite like the API you get where to make as many control lists as you need, and you can "fire and forget" them. You don't have to manage the queue of control lists yourself by feeding the top one into a request every time one completes and you recycle it. (OK, it's not difficult, but why make an API that's deliberately annoying?) > > Is this a faithful description? If so, I have two questions: Yes, think so! > > * what if the user wants to enable/disable various streams for the "burst capture"? > do you have any thoughts about how that API would work? Indeed, this is why I raised the question the other week about being able to enable streams using a control. The kind of use case is that you're capturing only low resolution images (e.g. for preview), but then want to do a burst capture with different exposures where you also want a high resolution buffer. The obvious way to synchronise this with a queue of control lists is to turn it into a control as well. We don't have any dedicated way to handle per-stream controls, so I think we're probably left with having either an array of flags (one per stream) or a separate control per stream. But I'm definitely open to offers there! I don't see any particular problem with enabling streams in both the request and a control list, as I expect applications would be using one method or the other. The pipeline handler could simply "or" them together, or complain if they disagree, I don't think it matters very much. Though if the expectation is that control lists in a request are applied with that request, then the control is presumably equivalent to the field in the request itself? > * is it guaranteed that every control list will be applied to the results of the > corresponding request? if not, how would that work with the user enabling/disabling > various streams (during the "burst capture")? Yes, though we haven't really said what the "corresponding request" means. In my use cases, I intend to take the top control list and apply it as soon as possible. It will be applied not to the request that was at the head of the queue at that moment, but a few frames later (because it takes a few frames to apply a control list). I also expect control lists, once in the control list queue, to have a sequence number, and for completed requests to report exactly which control list (according to its sequence number) they have applied. Warning: digression follows... In the "Android" case (by which I mean the scheme where a control list in a request is applied for that request), you will find yourself looking ahead at the control lists further up the request queue, deciding what needs to be applied with the top request. It's just my opinion, but even if you don't explicitly have a queue of control lists, you do still have a kind of "virtual" control list queue here. I find the whole thing easier to think about like this. The "Android"/ "Raspberry Pi" distinction ("Raspberry Pi" being the behaviour I described previously) just becomes a question of when you consume control lists from the control list queue: "Raspberry Pi" - just consume the top control list every time. "Android" - the control list queue entries need to record their target request number, then you just consume control lists as far as topRequest->sequenceNumber + pipelineDepth. Getting all the controls to synchronise correctly is, in my experience anyway, quite tricky for pipeline handlers, so I do sometimes wonder whether there's anything libcamera can do to help. But that's perhaps a question for another time! David > > > Thanks, > Barnabás Pőcze > > > > > Thanks > > > > David > > > > On Thu, 12 Mar 2026 at 16:00, David Plowman > > <david.plowman@raspberrypi.com> wrote: > >> > >> Add `Camera::queueControls()` whose purpose is to apply controls as > >> soon as possible, without going through `Request::controls()`. > >> > >> A new virtual function `PipelineHandler::queueControlsDevice()` is > >> provided for pipeline handler to implement fast-tracked application of > >> controls. If the pipeline handler does not implement that > >> functionality, or it fails, then a fallback mechanism is used. The > >> controls will be saved for later, and they will be merged into the > >> control list of the next available request sent to the pipeline > >> handler (`Camera::Private::waitingRequests_`). > >> > >> This patch is derived directly from Barnabas's previous verion that > >> implemented the same idea but with a single ControlList, rather than > >> allowing multiple ControlLists to be queued up for consecutive > >> frames. > >> > >> Signed-off-by: David Plowman <david.plowman@raspberrypi.com> > >> --- > >> include/libcamera/camera.h | 1 + > >> include/libcamera/internal/camera.h | 1 + > >> include/libcamera/internal/pipeline_handler.h | 7 ++ > >> src/libcamera/camera.cpp | 61 +++++++++++++++ > >> src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ > >> 5 files changed, 146 insertions(+) > >> > >> diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h > >> index b24a2974..93a484e4 100644 > >> --- a/include/libcamera/camera.h > >> +++ b/include/libcamera/camera.h > >> @@ -147,6 +147,7 @@ public: > >> > >> std::unique_ptr<Request> createRequest(uint64_t cookie = 0); > >> int queueRequest(Request *request); > >> + int queueControls(ControlList &&controls); > >> > >> int start(const ControlList *controls = nullptr); > >> int stop(); > >> diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h > >> index 8a2e9ed5..17dda925 100644 > >> --- a/include/libcamera/internal/camera.h > >> +++ b/include/libcamera/internal/camera.h > >> @@ -38,6 +38,7 @@ public: > >> > >> std::list<Request *> queuedRequests_; > >> std::queue<Request *> waitingRequests_; > >> + std::queue<ControlList> queuedControls_; > >> ControlInfoMap controlInfo_; > >> ControlList properties_; > >> > >> diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h > >> index b4f97477..c25213de 100644 > >> --- a/include/libcamera/internal/pipeline_handler.h > >> +++ b/include/libcamera/internal/pipeline_handler.h > >> @@ -57,6 +57,7 @@ public: > >> > >> void registerRequest(Request *request); > >> void queueRequest(Request *request); > >> + int queueControls(Camera *camera, ControlList controls); > >> > >> bool completeBuffer(Request *request, FrameBuffer *buffer); > >> void completeRequest(Request *request); > >> @@ -76,6 +77,12 @@ protected: > >> unsigned int useCount() const { return useCount_; } > >> > >> virtual int queueRequestDevice(Camera *camera, Request *request) = 0; > >> + > >> + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) > >> + { > >> + return -EOPNOTSUPP; > >> + } > >> + > >> virtual void stopDevice(Camera *camera) = 0; > >> > >> virtual bool acquireDevice(Camera *camera); > >> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp > >> index f724a1be..f0244707 100644 > >> --- a/src/libcamera/camera.cpp > >> +++ b/src/libcamera/camera.cpp > >> @@ -637,6 +637,16 @@ Camera::Private::~Private() > >> * queued requests was reached. > >> */ > >> > >> +/** > >> + * \var Camera::Private::queuedControls_ > >> + * \brief The queue of pending control lists > >> + * > >> + * This queue maintains a list of all the control lists that need to be sent > >> + * to the pipeline handler with subsequent requests. The top item in the queue > >> + * will always be sent with the next request going to > >> + * PipelineHandler::queueRequestDevice(). > >> + */ > >> + > >> /** > >> * \var Camera::Private::controlInfo_ > >> * \brief The set of controls supported by the camera > >> @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) > >> return 0; > >> } > >> > >> +/** > >> + * \brief Queue controls to be applied as soon as possible > >> + * \param[in] controls The list of controls to queue > >> + * > >> + * This function tries to ensure that the controls in \a controls are applied > >> + * to the camera as soon as possible. If there are still pending controls waiting > >> + * to be applied (because of previous calls to Camera::queueControls), then > >> + * these controls will be applied as soon as possible on a frame after those. > >> + * > >> + * The exact guarantees are camera dependent, but it is guaranteed that the > >> + * controls will be applied no later than with the next \ref Request"request" > >> + * that the application \ref Camera::queueRequest() "queues" (after any requests > >> + * have been *used up" for sending previously queued controls). > >> + * > >> + * \context This function is \threadsafe. It may only be called when the camera > >> + * is in the Running state as defined in \ref camera_operation. > >> + * > >> + * \return 0 on success or a negative error code otherwise > >> + * \retval -ENODEV The camera has been disconnected from the system > >> + * \retval -EACCES The camera is not running > >> + */ > >> +int Camera::queueControls(ControlList &&controls) > >> +{ > >> + Private *const d = _d(); > >> + > >> + /* > >> + * Like requests, controls can't be queued if the camera is not running. > >> + * Controls can be applied immediately when the camera starts using the > >> + * Camera::Start method. > >> + */ > >> + > >> + int ret = d->isAccessAllowed(Private::CameraRunning); > >> + if (ret < 0) > >> + return ret; > >> + > >> + /* > >> + * We want to be able to queue empty control lists, as this gives a way of > >> + * forcing another frame with the same controls as last time, before queueing > >> + * another control list that might change them again. > >> + */ > >> + > >> + patchControlList(controls); > >> + > >> + /* > >> + * \todo Or `ConnectionTypeBlocking` to get the return value? > >> + */ > >> + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); > >> + > >> + return 0; > >> +} > >> + > >> /** > >> * \brief Start capture from camera > >> * \param[in] controls Controls to be applied before starting the Camera > >> diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp > >> index 5c469e5b..1a87b28c 100644 > >> --- a/src/libcamera/pipeline_handler.cpp > >> +++ b/src/libcamera/pipeline_handler.cpp > >> @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) > >> ASSERT(data->queuedRequests_.empty()); > >> ASSERT(data->waitingRequests_.empty()); > >> > >> + /* > >> + * Clear out any unapplied controls. If an application wants to be > >> + * sure controls have been applied, it should wait before stopping. > >> + */ > >> + data->queuedControls_ = {}; > >> + > >> data->requestSequence_ = 0; > >> } > >> > >> @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) > >> request->_d()->prepare(300ms); > >> } > >> > >> +/** > >> + * \brief Queue controls to apply as soon as possible > >> + * \param[in] camera The camera > >> + * \param[in] controls The controls to apply > >> + * > >> + * This function tries to queue \a controls immediately to the device by > >> + * calling queueControlsDevice(). If that fails, then a fallback mechanism > >> + * is used to ensure that \a controls will be merged into the control list > >> + * of the next available request submitted to the pipeline handler. > >> + * > >> + * \context This function is called from the CameraManager thread. > >> + */ > >> +int PipelineHandler::queueControls(Camera *camera, ControlList controls) > >> +{ > >> + Camera::Private *data = camera->_d(); > >> + int ret = queueControlsDevice(camera, controls); > >> + > >> + /* > >> + * Don't worry about later request's controls overriding the ones > >> + * sent here - the application needs to deal with that. > >> + */ > >> + > >> + if (ret == -EOPNOTSUPP) { > >> + /* > >> + * Fall back to adding the controls to the next request that enters the > >> + * pipeline handler. See PipelineHandler::doQueueRequest(). > >> + */ > >> + data->queuedControls_.push(std::move(controls)); > >> + > >> + /* Counts as "success". */ > >> + ret = 0; > >> + > >> + } else if (ret < 0) { > >> + /* > >> + * The pipeline handler is claiming to support queueControlsDevice, > >> + * but it has failed. This is an error. > >> + */ > >> + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; > >> + } > >> + > >> + return ret; > >> +} > >> + > >> /** > >> * \brief Queue one requests to the device > >> */ > >> @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) > >> return; > >> } > >> > >> + if (!data->queuedControls_.empty()) { > >> + /* > >> + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is > >> + * needed to ensure that if `request` is newer than pendingControls_, > >> + * then its controls take precedence. > >> + */ > >> + request->controls().merge(data->queuedControls_.front(), > >> + ControlList::MergePolicy::KeepExisting); > >> + } > >> + > >> int ret = queueRequestDevice(camera, request); > >> if (ret) > >> cancelRequest(request); > >> + else if (!data->queuedControls_.empty()) > >> + data->queuedControls_.pop(); > >> } > >> > >> /** > >> @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) > >> * \return 0 on success or a negative error code otherwise > >> */ > >> > >> +/** > >> + * \fn PipelineHandler::queueControlsDevice() > >> + * \brief Queue controls to be applied as soon as possible > >> + * \param[in] camera The camera > >> + * \param[in] controls The controls to apply > >> + * > >> + * This function queues \a controls to \a camera so that they can be > >> + * applied as soon as possible > >> + * > >> + * \context This function is called from the CameraManager thread. > >> + * > >> + * \return 0 on success or a negative error code otherwise > >> + * \return -EOPNOTSUPP if fast-tracking controls is not supported > >> + */ > >> + > >> /** > >> * \brief Complete a buffer for a request > >> * \param[in] request The request the buffer belongs to > >> -- > >> 2.47.3 > >> >
2026. 08. 10. 13:10 keltezéssel, David Plowman írta: > Hi Barnabas > > Thanks for the message. I think you were right about everything, but I > can certainly expand a bit here and there. > > On Fri, 7 Aug 2026 at 18:43, Barnabás Pőcze > <barnabas.pocze@ideasonboard.com> wrote: >> >> Hi >> >> 2026. 07. 23. 18:17 keltezéssel, David Plowman írta: >>> Hi everyone >>> >>> I'd like to give this topic another prod. I talked about it a bit at >>> Nice, and revisiting this patch seems like a way to start some >>> discussion, though the precise implementation here is more for >>> illustration. >>> >>> One of the principal motivations is for things like burst captures, >>> where you don't want to get held up at the back of the request queue. >>> >>> Does this seem like a reasonable thing to do, or are there better alternatives? >> >> I started typing out a reply in a different thread of yours, but it got too long, >> so I though it would be more effective to write a quick reply here. >> >> I believe there is largely agreement that some kind non-request-submission-tied control >> setting mechanism is desirable. >> >> So I think it would be useful to examine the motivating use cases in a bit more detail. >> Sorry if this was already done somewhere, I couldn't recall. >> >> So the way I understand it, there is a user that is cycling through a set of requests, >> keeping most of them queued. This works fine, however, sometimes, maybe due to some >> external event, it wants to capture a number of frames with specific settings. The >> reason why applying the controls in the subsequent requests is undesirable is because >> of the delay that the already queued requests introduce. > > Yes, agree. Burst captures are the most obvious example, perhaps. But > pretty much any user event (such as changing a white balance mode) is > better if it happens quickly. > >> >> The proposed solution is to have two parallel queues: one for requests, and one for controls. >> Furthermore, the two are consumed in lockstep. And there is an assumption that the control >> is queue is mostly empty. This way when the user submits a control list, it can effectively >> "skip" the requests in the parallel request queue, and arrive near the front of the control >> queue. And thus it will be "applied to an earlier request", reducing latency. > > Yes, the point being that it will be applied as soon as possible. > > As a side note, I also quite like the API you get where to make as > many control lists as you need, and you can "fire and forget" them. > You don't have to manage the queue of control lists yourself by > feeding the top one into a request every time one completes and you > recycle it. (OK, it's not difficult, but why make an API that's > deliberately annoying?) > >> >> Is this a faithful description? If so, I have two questions: > > Yes, think so! > >> >> * what if the user wants to enable/disable various streams for the "burst capture"? >> do you have any thoughts about how that API would work? > > Indeed, this is why I raised the question the other week about being > able to enable streams using a control. > > The kind of use case is that you're capturing only low resolution > images (e.g. for preview), but then want to do a burst capture with > different exposures where you also want a high resolution buffer. > > The obvious way to synchronise this with a queue of control lists is > to turn it into a control as well. > > We don't have any dedicated way to handle per-stream controls, so I > think we're probably left with having either an array of flags (one > per stream) or a separate control per stream. But I'm definitely open > to offers there! Sidenote, I believe per-stream controls are planned, or at least they exist on the "would be good to have" list. > > I don't see any particular problem with enabling streams in both the > request and a control list, as I expect applications would be using > one method or the other. The pipeline handler could simply "or" them > together, or complain if they disagree, I don't think it matters very > much. Though if the expectation is that control lists in a request are > applied with that request, then the control is presumably equivalent > to the field in the request itself? I fear that by putting the stream-enable flags into a control list, it is effectively turned into a request. So arguably now there are two request queues. And what happens, in effect, is that two requests are merged at from the two request queues, and the result is what is actually applied. But the only reason the two queues are needed is because the "first" request queue is filled up by the "normal" requests (to keep the camera running), and the "second" one is then needed to, in effect, modify already submitted requests. So now I'm wondering, couldn't these use cases be address by keeping the "first" request queue mostly empty? For example, one random idea, one could have a so called "idle" request, which is repeating and is processed when there are no other requests. A user can then queue an idle request for their normal operation (e.g. preview), and in the burst capture use case, the new requests would be applied right away since the request queue is essentially empty. > >> * is it guaranteed that every control list will be applied to the results of the >> corresponding request? if not, how would that work with the user enabling/disabling >> various streams (during the "burst capture")? > > Yes, though we haven't really said what the "corresponding request" means. > > In my use cases, I intend to take the top control list and apply it as > soon as possible. It will be applied not to the request that was at > the head of the queue at that moment, but a few frames later (because > it takes a few frames to apply a control list). I also expect control > lists, once in the control list queue, to have a sequence number, and > for completed requests to report exactly which control list (according > to its sequence number) they have applied. And what would you expect to happen if the user, after start, queues 10 control lists with some controls + stream B, and then 10 requests with stream A? Based on the above, it would not be guaranteed that the 10 requests complete with the "matching" controls in effect with both stream A and stream B, correct? I think my main point here is the interaction of streams and controls. For example, the current request api effectively requires android-style per-frame control because if the user enables streams A and B and sets some controls, they will - I would argue - expect the controls to apply to the result, which contains data from stream A and B. And the way I understand what rpi implements today, is that each request has the `rpi::ControlListSequence` metadata, which denotes the latest request, the control list of which has been applied (and is in effect on the current result). And I quite like that approach for per-frame control, and I have only recently come to the conclusion that it is somewhat incompatible with the current public libcamera api wrt. stream handling. Because essentially one would have to delay the "stream enablement" with the controls to accurately capture what the user wants. And I'm kind of seeing a similar issue if the stream-enable flags become part of control lists, the pipeline handler will have to go in and override/delay the user choice, etc. Any thoughts on this? Maybe I'm missing something? > > Warning: digression follows... > > In the "Android" case (by which I mean the scheme where a control list > in a request is applied for that request), you will find yourself > looking ahead at the control lists further up the request queue, > deciding what needs to be applied with the top request. > > It's just my opinion, but even if you don't explicitly have a queue of > control lists, you do still have a kind of "virtual" control list > queue here. I find the whole thing easier to think about like this. > > The "Android"/ "Raspberry Pi" distinction ("Raspberry Pi" being the > behaviour I described previously) just becomes a question of when you > consume control lists from the control list queue: > > "Raspberry Pi" - just consume the top control list every time. > > "Android" - the control list queue entries need to record their target > request number, then you just consume control lists as far as > topRequest->sequenceNumber + pipelineDepth. > > Getting all the controls to synchronise correctly is, in my experience > anyway, quite tricky for pipeline handlers, so I do sometimes wonder > whether there's anything libcamera can do to help. But that's perhaps > a question for another time! > > David > >> >> >> Thanks, >> Barnabás Pőcze >> >>> >>> Thanks >>> >>> David >>> >>> On Thu, 12 Mar 2026 at 16:00, David Plowman >>> <david.plowman@raspberrypi.com> wrote: >>>> >>>> Add `Camera::queueControls()` whose purpose is to apply controls as >>>> soon as possible, without going through `Request::controls()`. >>>> >>>> A new virtual function `PipelineHandler::queueControlsDevice()` is >>>> provided for pipeline handler to implement fast-tracked application of >>>> controls. If the pipeline handler does not implement that >>>> functionality, or it fails, then a fallback mechanism is used. The >>>> controls will be saved for later, and they will be merged into the >>>> control list of the next available request sent to the pipeline >>>> handler (`Camera::Private::waitingRequests_`). >>>> >>>> This patch is derived directly from Barnabas's previous verion that >>>> implemented the same idea but with a single ControlList, rather than >>>> allowing multiple ControlLists to be queued up for consecutive >>>> frames. >>>> >>>> Signed-off-by: David Plowman <david.plowman@raspberrypi.com> >>>> --- >>>> include/libcamera/camera.h | 1 + >>>> include/libcamera/internal/camera.h | 1 + >>>> include/libcamera/internal/pipeline_handler.h | 7 ++ >>>> src/libcamera/camera.cpp | 61 +++++++++++++++ >>>> src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ >>>> 5 files changed, 146 insertions(+) >>>> >>>> diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h >>>> index b24a2974..93a484e4 100644 >>>> --- a/include/libcamera/camera.h >>>> +++ b/include/libcamera/camera.h >>>> @@ -147,6 +147,7 @@ public: >>>> >>>> std::unique_ptr<Request> createRequest(uint64_t cookie = 0); >>>> int queueRequest(Request *request); >>>> + int queueControls(ControlList &&controls); >>>> >>>> int start(const ControlList *controls = nullptr); >>>> int stop(); >>>> diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h >>>> index 8a2e9ed5..17dda925 100644 >>>> --- a/include/libcamera/internal/camera.h >>>> +++ b/include/libcamera/internal/camera.h >>>> @@ -38,6 +38,7 @@ public: >>>> >>>> std::list<Request *> queuedRequests_; >>>> std::queue<Request *> waitingRequests_; >>>> + std::queue<ControlList> queuedControls_; >>>> ControlInfoMap controlInfo_; >>>> ControlList properties_; >>>> >>>> diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h >>>> index b4f97477..c25213de 100644 >>>> --- a/include/libcamera/internal/pipeline_handler.h >>>> +++ b/include/libcamera/internal/pipeline_handler.h >>>> @@ -57,6 +57,7 @@ public: >>>> >>>> void registerRequest(Request *request); >>>> void queueRequest(Request *request); >>>> + int queueControls(Camera *camera, ControlList controls); >>>> >>>> bool completeBuffer(Request *request, FrameBuffer *buffer); >>>> void completeRequest(Request *request); >>>> @@ -76,6 +77,12 @@ protected: >>>> unsigned int useCount() const { return useCount_; } >>>> >>>> virtual int queueRequestDevice(Camera *camera, Request *request) = 0; >>>> + >>>> + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) >>>> + { >>>> + return -EOPNOTSUPP; >>>> + } >>>> + >>>> virtual void stopDevice(Camera *camera) = 0; >>>> >>>> virtual bool acquireDevice(Camera *camera); >>>> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp >>>> index f724a1be..f0244707 100644 >>>> --- a/src/libcamera/camera.cpp >>>> +++ b/src/libcamera/camera.cpp >>>> @@ -637,6 +637,16 @@ Camera::Private::~Private() >>>> * queued requests was reached. >>>> */ >>>> >>>> +/** >>>> + * \var Camera::Private::queuedControls_ >>>> + * \brief The queue of pending control lists >>>> + * >>>> + * This queue maintains a list of all the control lists that need to be sent >>>> + * to the pipeline handler with subsequent requests. The top item in the queue >>>> + * will always be sent with the next request going to >>>> + * PipelineHandler::queueRequestDevice(). >>>> + */ >>>> + >>>> /** >>>> * \var Camera::Private::controlInfo_ >>>> * \brief The set of controls supported by the camera >>>> @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) >>>> return 0; >>>> } >>>> >>>> +/** >>>> + * \brief Queue controls to be applied as soon as possible >>>> + * \param[in] controls The list of controls to queue >>>> + * >>>> + * This function tries to ensure that the controls in \a controls are applied >>>> + * to the camera as soon as possible. If there are still pending controls waiting >>>> + * to be applied (because of previous calls to Camera::queueControls), then >>>> + * these controls will be applied as soon as possible on a frame after those. >>>> + * >>>> + * The exact guarantees are camera dependent, but it is guaranteed that the >>>> + * controls will be applied no later than with the next \ref Request"request" >>>> + * that the application \ref Camera::queueRequest() "queues" (after any requests >>>> + * have been *used up" for sending previously queued controls). >>>> + * >>>> + * \context This function is \threadsafe. It may only be called when the camera >>>> + * is in the Running state as defined in \ref camera_operation. >>>> + * >>>> + * \return 0 on success or a negative error code otherwise >>>> + * \retval -ENODEV The camera has been disconnected from the system >>>> + * \retval -EACCES The camera is not running >>>> + */ >>>> +int Camera::queueControls(ControlList &&controls) >>>> +{ >>>> + Private *const d = _d(); >>>> + >>>> + /* >>>> + * Like requests, controls can't be queued if the camera is not running. >>>> + * Controls can be applied immediately when the camera starts using the >>>> + * Camera::Start method. >>>> + */ >>>> + >>>> + int ret = d->isAccessAllowed(Private::CameraRunning); >>>> + if (ret < 0) >>>> + return ret; >>>> + >>>> + /* >>>> + * We want to be able to queue empty control lists, as this gives a way of >>>> + * forcing another frame with the same controls as last time, before queueing >>>> + * another control list that might change them again. >>>> + */ >>>> + >>>> + patchControlList(controls); >>>> + >>>> + /* >>>> + * \todo Or `ConnectionTypeBlocking` to get the return value? >>>> + */ >>>> + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); >>>> + >>>> + return 0; >>>> +} >>>> + >>>> /** >>>> * \brief Start capture from camera >>>> * \param[in] controls Controls to be applied before starting the Camera >>>> diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp >>>> index 5c469e5b..1a87b28c 100644 >>>> --- a/src/libcamera/pipeline_handler.cpp >>>> +++ b/src/libcamera/pipeline_handler.cpp >>>> @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) >>>> ASSERT(data->queuedRequests_.empty()); >>>> ASSERT(data->waitingRequests_.empty()); >>>> >>>> + /* >>>> + * Clear out any unapplied controls. If an application wants to be >>>> + * sure controls have been applied, it should wait before stopping. >>>> + */ >>>> + data->queuedControls_ = {}; >>>> + >>>> data->requestSequence_ = 0; >>>> } >>>> >>>> @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) >>>> request->_d()->prepare(300ms); >>>> } >>>> >>>> +/** >>>> + * \brief Queue controls to apply as soon as possible >>>> + * \param[in] camera The camera >>>> + * \param[in] controls The controls to apply >>>> + * >>>> + * This function tries to queue \a controls immediately to the device by >>>> + * calling queueControlsDevice(). If that fails, then a fallback mechanism >>>> + * is used to ensure that \a controls will be merged into the control list >>>> + * of the next available request submitted to the pipeline handler. >>>> + * >>>> + * \context This function is called from the CameraManager thread. >>>> + */ >>>> +int PipelineHandler::queueControls(Camera *camera, ControlList controls) >>>> +{ >>>> + Camera::Private *data = camera->_d(); >>>> + int ret = queueControlsDevice(camera, controls); >>>> + >>>> + /* >>>> + * Don't worry about later request's controls overriding the ones >>>> + * sent here - the application needs to deal with that. >>>> + */ >>>> + >>>> + if (ret == -EOPNOTSUPP) { >>>> + /* >>>> + * Fall back to adding the controls to the next request that enters the >>>> + * pipeline handler. See PipelineHandler::doQueueRequest(). >>>> + */ >>>> + data->queuedControls_.push(std::move(controls)); >>>> + >>>> + /* Counts as "success". */ >>>> + ret = 0; >>>> + >>>> + } else if (ret < 0) { >>>> + /* >>>> + * The pipeline handler is claiming to support queueControlsDevice, >>>> + * but it has failed. This is an error. >>>> + */ >>>> + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; >>>> + } >>>> + >>>> + return ret; >>>> +} >>>> + >>>> /** >>>> * \brief Queue one requests to the device >>>> */ >>>> @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) >>>> return; >>>> } >>>> >>>> + if (!data->queuedControls_.empty()) { >>>> + /* >>>> + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is >>>> + * needed to ensure that if `request` is newer than pendingControls_, >>>> + * then its controls take precedence. >>>> + */ >>>> + request->controls().merge(data->queuedControls_.front(), >>>> + ControlList::MergePolicy::KeepExisting); >>>> + } >>>> + >>>> int ret = queueRequestDevice(camera, request); >>>> if (ret) >>>> cancelRequest(request); >>>> + else if (!data->queuedControls_.empty()) >>>> + data->queuedControls_.pop(); >>>> } >>>> >>>> /** >>>> @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) >>>> * \return 0 on success or a negative error code otherwise >>>> */ >>>> >>>> +/** >>>> + * \fn PipelineHandler::queueControlsDevice() >>>> + * \brief Queue controls to be applied as soon as possible >>>> + * \param[in] camera The camera >>>> + * \param[in] controls The controls to apply >>>> + * >>>> + * This function queues \a controls to \a camera so that they can be >>>> + * applied as soon as possible >>>> + * >>>> + * \context This function is called from the CameraManager thread. >>>> + * >>>> + * \return 0 on success or a negative error code otherwise >>>> + * \return -EOPNOTSUPP if fast-tracking controls is not supported >>>> + */ >>>> + >>>> /** >>>> * \brief Complete a buffer for a request >>>> * \param[in] request The request the buffer belongs to >>>> -- >>>> 2.47.3 >>>> >>
Hi again On Mon, 10 Aug 2026 at 14:07, Barnabás Pőcze <barnabas.pocze@ideasonboard.com> wrote: > > 2026. 08. 10. 13:10 keltezéssel, David Plowman írta: > > Hi Barnabas > > > > Thanks for the message. I think you were right about everything, but I > > can certainly expand a bit here and there. > > > > On Fri, 7 Aug 2026 at 18:43, Barnabás Pőcze > > <barnabas.pocze@ideasonboard.com> wrote: > >> > >> Hi > >> > >> 2026. 07. 23. 18:17 keltezéssel, David Plowman írta: > >>> Hi everyone > >>> > >>> I'd like to give this topic another prod. I talked about it a bit at > >>> Nice, and revisiting this patch seems like a way to start some > >>> discussion, though the precise implementation here is more for > >>> illustration. > >>> > >>> One of the principal motivations is for things like burst captures, > >>> where you don't want to get held up at the back of the request queue. > >>> > >>> Does this seem like a reasonable thing to do, or are there better alternatives? > >> > >> I started typing out a reply in a different thread of yours, but it got too long, > >> so I though it would be more effective to write a quick reply here. > >> > >> I believe there is largely agreement that some kind non-request-submission-tied control > >> setting mechanism is desirable. > >> > >> So I think it would be useful to examine the motivating use cases in a bit more detail. > >> Sorry if this was already done somewhere, I couldn't recall. > >> > >> So the way I understand it, there is a user that is cycling through a set of requests, > >> keeping most of them queued. This works fine, however, sometimes, maybe due to some > >> external event, it wants to capture a number of frames with specific settings. The > >> reason why applying the controls in the subsequent requests is undesirable is because > >> of the delay that the already queued requests introduce. > > > > Yes, agree. Burst captures are the most obvious example, perhaps. But > > pretty much any user event (such as changing a white balance mode) is > > better if it happens quickly. > > > >> > >> The proposed solution is to have two parallel queues: one for requests, and one for controls. > >> Furthermore, the two are consumed in lockstep. And there is an assumption that the control > >> is queue is mostly empty. This way when the user submits a control list, it can effectively > >> "skip" the requests in the parallel request queue, and arrive near the front of the control > >> queue. And thus it will be "applied to an earlier request", reducing latency. > > > > Yes, the point being that it will be applied as soon as possible. > > > > As a side note, I also quite like the API you get where to make as > > many control lists as you need, and you can "fire and forget" them. > > You don't have to manage the queue of control lists yourself by > > feeding the top one into a request every time one completes and you > > recycle it. (OK, it's not difficult, but why make an API that's > > deliberately annoying?) > > > >> > >> Is this a faithful description? If so, I have two questions: > > > > Yes, think so! > > > >> > >> * what if the user wants to enable/disable various streams for the "burst capture"? > >> do you have any thoughts about how that API would work? > > > > Indeed, this is why I raised the question the other week about being > > able to enable streams using a control. > > > > The kind of use case is that you're capturing only low resolution > > images (e.g. for preview), but then want to do a burst capture with > > different exposures where you also want a high resolution buffer. > > > > The obvious way to synchronise this with a queue of control lists is > > to turn it into a control as well. > > > > We don't have any dedicated way to handle per-stream controls, so I > > think we're probably left with having either an array of flags (one > > per stream) or a separate control per stream. But I'm definitely open > > to offers there! > > Sidenote, I believe per-stream controls are planned, or at least they exist > on the "would be good to have" list. It has been mentioned from time to time. It would clearly be good, and help us to get rid of the fairly horrible "ScalerCrops" (plural) control that we have. I also see a possible future one day where we can have different colour spaces on our output branches, so it could apply there too. > > > > > > I don't see any particular problem with enabling streams in both the > > request and a control list, as I expect applications would be using > > one method or the other. The pipeline handler could simply "or" them > > together, or complain if they disagree, I don't think it matters very > > much. Though if the expectation is that control lists in a request are > > applied with that request, then the control is presumably equivalent > > to the field in the request itself? > > I fear that by putting the stream-enable flags into a control list, it is > effectively turned into a request. So arguably now there are two request queues. > And what happens, in effect, is that two requests are merged at from the two request > queues, and the result is what is actually applied. > > But the only reason the two queues are needed is because the "first" request > queue is filled up by the "normal" requests (to keep the camera running), and > the "second" one is then needed to, in effect, modify already submitted requests. > > So now I'm wondering, couldn't these use cases be address by keeping the "first" > request queue mostly empty? For example, one random idea, one could have a so > called "idle" request, which is repeating and is processed when there are no > other requests. A user can then queue an idle request for their normal operation > (e.g. preview), and in the burst capture use case, the new requests would be applied > right away since the request queue is essentially empty. This certainly reminds me a bit of Android's "repeating requests", which I believe saved applications from permanently spamming the request queue. I have a lot of sympathy with that approach actually, but I think Android has moved on and doesn't do this any more. They have their reasons, I'm sure, possibly even to do with making their model of PFC work. I agree that having a "normal" request queue and a "priority" queue achieves a similar thing. In my case I'd want to apply those controls immediately, so they'd actually end up applying to a different request than the one they were queued with (even one of the "normal" ones). To be fair, I don't particularly mind that, though it might seem a bit strange? As I also mentioned, I'm not super-keen on the fact that these priority requests have to be created, and their lifetimes managed, whereas control lists just look after themselves. > > > > > >> * is it guaranteed that every control list will be applied to the results of the > >> corresponding request? if not, how would that work with the user enabling/disabling > >> various streams (during the "burst capture")? > > > > Yes, though we haven't really said what the "corresponding request" means. > > > > In my use cases, I intend to take the top control list and apply it as > > soon as possible. It will be applied not to the request that was at > > the head of the queue at that moment, but a few frames later (because > > it takes a few frames to apply a control list). I also expect control > > lists, once in the control list queue, to have a sequence number, and > > for completed requests to report exactly which control list (according > > to its sequence number) they have applied. > > And what would you expect to happen if the user, after start, queues 10 control > lists with some controls + stream B, and then 10 requests with stream A? Based > on the above, it would not be guaranteed that the 10 requests complete with > the "matching" controls in effect with both stream A and stream B, correct? Correct. I suppose if you queue them all immediately when you start streaming, they might happen to coincide in the first 10 frames (after pipelineDepth frames, where nothing happened), but there's no guarantee. In a free-running system, with many requests already in the queue, then the frames with stream B enabled will typically come out much earlier. > > I think my main point here is the interaction of streams and controls. For example, > the current request api effectively requires android-style per-frame control because > if the user enables streams A and B and sets some controls, they will - I would argue - > expect the controls to apply to the result, which contains data from stream A and B. Yes, I agree. If there are flags in the request to enable streams, then I think I would expect them to be applied for that request. (Though I expect it's clear I don't really want to use these flags.) > > And the way I understand what rpi implements today, is that each request has the > `rpi::ControlListSequence` metadata, which denotes the latest request, the control list > of which has been applied (and is in effect on the current result). Correct. > > And I quite like that approach for per-frame control, and I have only recently come > to the conclusion that it is somewhat incompatible with the current public libcamera > api wrt. stream handling. Because essentially one would have to delay the > "stream enablement" with the controls to accurately capture what the user wants. > > And I'm kind of seeing a similar issue if the stream-enable flags become part of > control lists, the pipeline handler will have to go in and override/delay the > user choice, etc. Right, but this is covered by having a control for it. The control gets delayed along with all the other controls for pipelineDepth frames (or whatever), and then it would be applied. > > Any thoughts on this? Maybe I'm missing something? I agree with your earlier observation that things can seem a bit incompatible. Just my opinion, but I always think the Android style of API is trying to make the camera system seem like it's kind of "synchronous". Which seems like a good thing, only the difficulty is, cameras aren't! David > > > > > Warning: digression follows... > > > > In the "Android" case (by which I mean the scheme where a control list > > in a request is applied for that request), you will find yourself > > looking ahead at the control lists further up the request queue, > > deciding what needs to be applied with the top request. > > > > It's just my opinion, but even if you don't explicitly have a queue of > > control lists, you do still have a kind of "virtual" control list > > queue here. I find the whole thing easier to think about like this. > > > > The "Android"/ "Raspberry Pi" distinction ("Raspberry Pi" being the > > behaviour I described previously) just becomes a question of when you > > consume control lists from the control list queue: > > > > "Raspberry Pi" - just consume the top control list every time. > > > > "Android" - the control list queue entries need to record their target > > request number, then you just consume control lists as far as > > topRequest->sequenceNumber + pipelineDepth. > > > > Getting all the controls to synchronise correctly is, in my experience > > anyway, quite tricky for pipeline handlers, so I do sometimes wonder > > whether there's anything libcamera can do to help. But that's perhaps > > a question for another time! > > > > David > > > >> > >> > >> Thanks, > >> Barnabás Pőcze > >> > >>> > >>> Thanks > >>> > >>> David > >>> > >>> On Thu, 12 Mar 2026 at 16:00, David Plowman > >>> <david.plowman@raspberrypi.com> wrote: > >>>> > >>>> Add `Camera::queueControls()` whose purpose is to apply controls as > >>>> soon as possible, without going through `Request::controls()`. > >>>> > >>>> A new virtual function `PipelineHandler::queueControlsDevice()` is > >>>> provided for pipeline handler to implement fast-tracked application of > >>>> controls. If the pipeline handler does not implement that > >>>> functionality, or it fails, then a fallback mechanism is used. The > >>>> controls will be saved for later, and they will be merged into the > >>>> control list of the next available request sent to the pipeline > >>>> handler (`Camera::Private::waitingRequests_`). > >>>> > >>>> This patch is derived directly from Barnabas's previous verion that > >>>> implemented the same idea but with a single ControlList, rather than > >>>> allowing multiple ControlLists to be queued up for consecutive > >>>> frames. > >>>> > >>>> Signed-off-by: David Plowman <david.plowman@raspberrypi.com> > >>>> --- > >>>> include/libcamera/camera.h | 1 + > >>>> include/libcamera/internal/camera.h | 1 + > >>>> include/libcamera/internal/pipeline_handler.h | 7 ++ > >>>> src/libcamera/camera.cpp | 61 +++++++++++++++ > >>>> src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ > >>>> 5 files changed, 146 insertions(+) > >>>> > >>>> diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h > >>>> index b24a2974..93a484e4 100644 > >>>> --- a/include/libcamera/camera.h > >>>> +++ b/include/libcamera/camera.h > >>>> @@ -147,6 +147,7 @@ public: > >>>> > >>>> std::unique_ptr<Request> createRequest(uint64_t cookie = 0); > >>>> int queueRequest(Request *request); > >>>> + int queueControls(ControlList &&controls); > >>>> > >>>> int start(const ControlList *controls = nullptr); > >>>> int stop(); > >>>> diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h > >>>> index 8a2e9ed5..17dda925 100644 > >>>> --- a/include/libcamera/internal/camera.h > >>>> +++ b/include/libcamera/internal/camera.h > >>>> @@ -38,6 +38,7 @@ public: > >>>> > >>>> std::list<Request *> queuedRequests_; > >>>> std::queue<Request *> waitingRequests_; > >>>> + std::queue<ControlList> queuedControls_; > >>>> ControlInfoMap controlInfo_; > >>>> ControlList properties_; > >>>> > >>>> diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h > >>>> index b4f97477..c25213de 100644 > >>>> --- a/include/libcamera/internal/pipeline_handler.h > >>>> +++ b/include/libcamera/internal/pipeline_handler.h > >>>> @@ -57,6 +57,7 @@ public: > >>>> > >>>> void registerRequest(Request *request); > >>>> void queueRequest(Request *request); > >>>> + int queueControls(Camera *camera, ControlList controls); > >>>> > >>>> bool completeBuffer(Request *request, FrameBuffer *buffer); > >>>> void completeRequest(Request *request); > >>>> @@ -76,6 +77,12 @@ protected: > >>>> unsigned int useCount() const { return useCount_; } > >>>> > >>>> virtual int queueRequestDevice(Camera *camera, Request *request) = 0; > >>>> + > >>>> + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) > >>>> + { > >>>> + return -EOPNOTSUPP; > >>>> + } > >>>> + > >>>> virtual void stopDevice(Camera *camera) = 0; > >>>> > >>>> virtual bool acquireDevice(Camera *camera); > >>>> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp > >>>> index f724a1be..f0244707 100644 > >>>> --- a/src/libcamera/camera.cpp > >>>> +++ b/src/libcamera/camera.cpp > >>>> @@ -637,6 +637,16 @@ Camera::Private::~Private() > >>>> * queued requests was reached. > >>>> */ > >>>> > >>>> +/** > >>>> + * \var Camera::Private::queuedControls_ > >>>> + * \brief The queue of pending control lists > >>>> + * > >>>> + * This queue maintains a list of all the control lists that need to be sent > >>>> + * to the pipeline handler with subsequent requests. The top item in the queue > >>>> + * will always be sent with the next request going to > >>>> + * PipelineHandler::queueRequestDevice(). > >>>> + */ > >>>> + > >>>> /** > >>>> * \var Camera::Private::controlInfo_ > >>>> * \brief The set of controls supported by the camera > >>>> @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) > >>>> return 0; > >>>> } > >>>> > >>>> +/** > >>>> + * \brief Queue controls to be applied as soon as possible > >>>> + * \param[in] controls The list of controls to queue > >>>> + * > >>>> + * This function tries to ensure that the controls in \a controls are applied > >>>> + * to the camera as soon as possible. If there are still pending controls waiting > >>>> + * to be applied (because of previous calls to Camera::queueControls), then > >>>> + * these controls will be applied as soon as possible on a frame after those. > >>>> + * > >>>> + * The exact guarantees are camera dependent, but it is guaranteed that the > >>>> + * controls will be applied no later than with the next \ref Request"request" > >>>> + * that the application \ref Camera::queueRequest() "queues" (after any requests > >>>> + * have been *used up" for sending previously queued controls). > >>>> + * > >>>> + * \context This function is \threadsafe. It may only be called when the camera > >>>> + * is in the Running state as defined in \ref camera_operation. > >>>> + * > >>>> + * \return 0 on success or a negative error code otherwise > >>>> + * \retval -ENODEV The camera has been disconnected from the system > >>>> + * \retval -EACCES The camera is not running > >>>> + */ > >>>> +int Camera::queueControls(ControlList &&controls) > >>>> +{ > >>>> + Private *const d = _d(); > >>>> + > >>>> + /* > >>>> + * Like requests, controls can't be queued if the camera is not running. > >>>> + * Controls can be applied immediately when the camera starts using the > >>>> + * Camera::Start method. > >>>> + */ > >>>> + > >>>> + int ret = d->isAccessAllowed(Private::CameraRunning); > >>>> + if (ret < 0) > >>>> + return ret; > >>>> + > >>>> + /* > >>>> + * We want to be able to queue empty control lists, as this gives a way of > >>>> + * forcing another frame with the same controls as last time, before queueing > >>>> + * another control list that might change them again. > >>>> + */ > >>>> + > >>>> + patchControlList(controls); > >>>> + > >>>> + /* > >>>> + * \todo Or `ConnectionTypeBlocking` to get the return value? > >>>> + */ > >>>> + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); > >>>> + > >>>> + return 0; > >>>> +} > >>>> + > >>>> /** > >>>> * \brief Start capture from camera > >>>> * \param[in] controls Controls to be applied before starting the Camera > >>>> diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp > >>>> index 5c469e5b..1a87b28c 100644 > >>>> --- a/src/libcamera/pipeline_handler.cpp > >>>> +++ b/src/libcamera/pipeline_handler.cpp > >>>> @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) > >>>> ASSERT(data->queuedRequests_.empty()); > >>>> ASSERT(data->waitingRequests_.empty()); > >>>> > >>>> + /* > >>>> + * Clear out any unapplied controls. If an application wants to be > >>>> + * sure controls have been applied, it should wait before stopping. > >>>> + */ > >>>> + data->queuedControls_ = {}; > >>>> + > >>>> data->requestSequence_ = 0; > >>>> } > >>>> > >>>> @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) > >>>> request->_d()->prepare(300ms); > >>>> } > >>>> > >>>> +/** > >>>> + * \brief Queue controls to apply as soon as possible > >>>> + * \param[in] camera The camera > >>>> + * \param[in] controls The controls to apply > >>>> + * > >>>> + * This function tries to queue \a controls immediately to the device by > >>>> + * calling queueControlsDevice(). If that fails, then a fallback mechanism > >>>> + * is used to ensure that \a controls will be merged into the control list > >>>> + * of the next available request submitted to the pipeline handler. > >>>> + * > >>>> + * \context This function is called from the CameraManager thread. > >>>> + */ > >>>> +int PipelineHandler::queueControls(Camera *camera, ControlList controls) > >>>> +{ > >>>> + Camera::Private *data = camera->_d(); > >>>> + int ret = queueControlsDevice(camera, controls); > >>>> + > >>>> + /* > >>>> + * Don't worry about later request's controls overriding the ones > >>>> + * sent here - the application needs to deal with that. > >>>> + */ > >>>> + > >>>> + if (ret == -EOPNOTSUPP) { > >>>> + /* > >>>> + * Fall back to adding the controls to the next request that enters the > >>>> + * pipeline handler. See PipelineHandler::doQueueRequest(). > >>>> + */ > >>>> + data->queuedControls_.push(std::move(controls)); > >>>> + > >>>> + /* Counts as "success". */ > >>>> + ret = 0; > >>>> + > >>>> + } else if (ret < 0) { > >>>> + /* > >>>> + * The pipeline handler is claiming to support queueControlsDevice, > >>>> + * but it has failed. This is an error. > >>>> + */ > >>>> + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; > >>>> + } > >>>> + > >>>> + return ret; > >>>> +} > >>>> + > >>>> /** > >>>> * \brief Queue one requests to the device > >>>> */ > >>>> @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) > >>>> return; > >>>> } > >>>> > >>>> + if (!data->queuedControls_.empty()) { > >>>> + /* > >>>> + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is > >>>> + * needed to ensure that if `request` is newer than pendingControls_, > >>>> + * then its controls take precedence. > >>>> + */ > >>>> + request->controls().merge(data->queuedControls_.front(), > >>>> + ControlList::MergePolicy::KeepExisting); > >>>> + } > >>>> + > >>>> int ret = queueRequestDevice(camera, request); > >>>> if (ret) > >>>> cancelRequest(request); > >>>> + else if (!data->queuedControls_.empty()) > >>>> + data->queuedControls_.pop(); > >>>> } > >>>> > >>>> /** > >>>> @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) > >>>> * \return 0 on success or a negative error code otherwise > >>>> */ > >>>> > >>>> +/** > >>>> + * \fn PipelineHandler::queueControlsDevice() > >>>> + * \brief Queue controls to be applied as soon as possible > >>>> + * \param[in] camera The camera > >>>> + * \param[in] controls The controls to apply > >>>> + * > >>>> + * This function queues \a controls to \a camera so that they can be > >>>> + * applied as soon as possible > >>>> + * > >>>> + * \context This function is called from the CameraManager thread. > >>>> + * > >>>> + * \return 0 on success or a negative error code otherwise > >>>> + * \return -EOPNOTSUPP if fast-tracking controls is not supported > >>>> + */ > >>>> + > >>>> /** > >>>> * \brief Complete a buffer for a request > >>>> * \param[in] request The request the buffer belongs to > >>>> -- > >>>> 2.47.3 > >>>> > >> >
2026. 08. 10. 17:13 keltezéssel, David Plowman írta: > Hi again > > On Mon, 10 Aug 2026 at 14:07, Barnabás Pőcze > <barnabas.pocze@ideasonboard.com> wrote: >> >> 2026. 08. 10. 13:10 keltezéssel, David Plowman írta: >>> Hi Barnabas >>> >>> Thanks for the message. I think you were right about everything, but I >>> can certainly expand a bit here and there. >>> >>> On Fri, 7 Aug 2026 at 18:43, Barnabás Pőcze >>> <barnabas.pocze@ideasonboard.com> wrote: >>>> >>>> Hi >>>> >>>> 2026. 07. 23. 18:17 keltezéssel, David Plowman írta: >>>>> Hi everyone >>>>> >>>>> I'd like to give this topic another prod. I talked about it a bit at >>>>> Nice, and revisiting this patch seems like a way to start some >>>>> discussion, though the precise implementation here is more for >>>>> illustration. >>>>> >>>>> One of the principal motivations is for things like burst captures, >>>>> where you don't want to get held up at the back of the request queue. >>>>> >>>>> Does this seem like a reasonable thing to do, or are there better alternatives? >>>> >>>> I started typing out a reply in a different thread of yours, but it got too long, >>>> so I though it would be more effective to write a quick reply here. >>>> >>>> I believe there is largely agreement that some kind non-request-submission-tied control >>>> setting mechanism is desirable. >>>> >>>> So I think it would be useful to examine the motivating use cases in a bit more detail. >>>> Sorry if this was already done somewhere, I couldn't recall. >>>> >>>> So the way I understand it, there is a user that is cycling through a set of requests, >>>> keeping most of them queued. This works fine, however, sometimes, maybe due to some >>>> external event, it wants to capture a number of frames with specific settings. The >>>> reason why applying the controls in the subsequent requests is undesirable is because >>>> of the delay that the already queued requests introduce. >>> >>> Yes, agree. Burst captures are the most obvious example, perhaps. But >>> pretty much any user event (such as changing a white balance mode) is >>> better if it happens quickly. >>> >>>> >>>> The proposed solution is to have two parallel queues: one for requests, and one for controls. >>>> Furthermore, the two are consumed in lockstep. And there is an assumption that the control >>>> is queue is mostly empty. This way when the user submits a control list, it can effectively >>>> "skip" the requests in the parallel request queue, and arrive near the front of the control >>>> queue. And thus it will be "applied to an earlier request", reducing latency. >>> >>> Yes, the point being that it will be applied as soon as possible. >>> >>> As a side note, I also quite like the API you get where to make as >>> many control lists as you need, and you can "fire and forget" them. >>> You don't have to manage the queue of control lists yourself by >>> feeding the top one into a request every time one completes and you >>> recycle it. (OK, it's not difficult, but why make an API that's >>> deliberately annoying?) >>> >>>> >>>> Is this a faithful description? If so, I have two questions: >>> >>> Yes, think so! >>> >>>> >>>> * what if the user wants to enable/disable various streams for the "burst capture"? >>>> do you have any thoughts about how that API would work? >>> >>> Indeed, this is why I raised the question the other week about being >>> able to enable streams using a control. >>> >>> The kind of use case is that you're capturing only low resolution >>> images (e.g. for preview), but then want to do a burst capture with >>> different exposures where you also want a high resolution buffer. >>> >>> The obvious way to synchronise this with a queue of control lists is >>> to turn it into a control as well. >>> >>> We don't have any dedicated way to handle per-stream controls, so I >>> think we're probably left with having either an array of flags (one >>> per stream) or a separate control per stream. But I'm definitely open >>> to offers there! >> >> Sidenote, I believe per-stream controls are planned, or at least they exist >> on the "would be good to have" list. > > It has been mentioned from time to time. It would clearly be good, and > help us to get rid of the fairly horrible "ScalerCrops" (plural) > control that we have. I also see a possible future one day where we > can have different colour spaces on our output branches, so it could > apply there too. > >> >> >>> >>> I don't see any particular problem with enabling streams in both the >>> request and a control list, as I expect applications would be using >>> one method or the other. The pipeline handler could simply "or" them >>> together, or complain if they disagree, I don't think it matters very >>> much. Though if the expectation is that control lists in a request are >>> applied with that request, then the control is presumably equivalent >>> to the field in the request itself? >> >> I fear that by putting the stream-enable flags into a control list, it is >> effectively turned into a request. So arguably now there are two request queues. >> And what happens, in effect, is that two requests are merged at from the two request >> queues, and the result is what is actually applied. >> >> But the only reason the two queues are needed is because the "first" request >> queue is filled up by the "normal" requests (to keep the camera running), and >> the "second" one is then needed to, in effect, modify already submitted requests. >> >> So now I'm wondering, couldn't these use cases be address by keeping the "first" >> request queue mostly empty? For example, one random idea, one could have a so >> called "idle" request, which is repeating and is processed when there are no >> other requests. A user can then queue an idle request for their normal operation >> (e.g. preview), and in the burst capture use case, the new requests would be applied >> right away since the request queue is essentially empty. > > This certainly reminds me a bit of Android's "repeating requests", > which I believe saved applications from permanently spamming the > request queue. I have a lot of sympathy with that approach actually, > but I think Android has moved on and doesn't do this any more. They > have their reasons, I'm sure, possibly even to do with making their > model of PFC work. > > I agree that having a "normal" request queue and a "priority" queue > achieves a similar thing. In my case I'd want to apply those controls > immediately, so they'd actually end up applying to a different request > than the one they were queued with (even one of the "normal" ones). To > be fair, I don't particularly mind that, though it might seem a bit > strange? > > As I also mentioned, I'm not super-keen on the fact that these > priority requests have to be created, and their lifetimes managed, > whereas control lists just look after themselves. > That's a fair point, but I don't see it as a big issue given that it is already the status quo. And I think there are opportunities for simplification. >> >> >>> >>>> * is it guaranteed that every control list will be applied to the results of the >>>> corresponding request? if not, how would that work with the user enabling/disabling >>>> various streams (during the "burst capture")? >>> >>> Yes, though we haven't really said what the "corresponding request" means. >>> >>> In my use cases, I intend to take the top control list and apply it as >>> soon as possible. It will be applied not to the request that was at >>> the head of the queue at that moment, but a few frames later (because >>> it takes a few frames to apply a control list). I also expect control >>> lists, once in the control list queue, to have a sequence number, and >>> for completed requests to report exactly which control list (according >>> to its sequence number) they have applied. >> >> And what would you expect to happen if the user, after start, queues 10 control >> lists with some controls + stream B, and then 10 requests with stream A? Based >> on the above, it would not be guaranteed that the 10 requests complete with >> the "matching" controls in effect with both stream A and stream B, correct? > > Correct. I suppose if you queue them all immediately when you start > streaming, they might happen to coincide in the first 10 frames (after > pipelineDepth frames, where nothing happened), but there's no > guarantee. In a free-running system, with many requests already in the > queue, then the frames with stream B enabled will typically come out > much earlier. > >> >> I think my main point here is the interaction of streams and controls. For example, >> the current request api effectively requires android-style per-frame control because >> if the user enables streams A and B and sets some controls, they will - I would argue - >> expect the controls to apply to the result, which contains data from stream A and B. > > Yes, I agree. If there are flags in the request to enable streams, > then I think I would expect them to be applied for that request. > (Though I expect it's clear I don't really want to use these flags.) Right, but there seems to a deeper conflict between what you want and the current libcamera public api than I have originally imagined. So I wanted to confirm some details. > >> >> And the way I understand what rpi implements today, is that each request has the >> `rpi::ControlListSequence` metadata, which denotes the latest request, the control list >> of which has been applied (and is in effect on the current result). > > Correct. > >> >> And I quite like that approach for per-frame control, and I have only recently come >> to the conclusion that it is somewhat incompatible with the current public libcamera >> api wrt. stream handling. Because essentially one would have to delay the >> "stream enablement" with the controls to accurately capture what the user wants. >> >> And I'm kind of seeing a similar issue if the stream-enable flags become part of >> control lists, the pipeline handler will have to go in and override/delay the >> user choice, etc. > > Right, but this is covered by having a control for it. The control > gets delayed along with all the other controls for pipelineDepth > frames (or whatever), and then it would be applied. So in your ideal world the streams would be exclusively selected via a control, and added to the request during completion when it is finalized which streams will actually be captured from? > >> >> Any thoughts on this? Maybe I'm missing something? > > I agree with your earlier observation that things can seem a bit > incompatible. Just my opinion, but I always think the Android style of > API is trying to make the camera system seem like it's kind of > "synchronous". Which seems like a good thing, only the difficulty is, > cameras aren't! > > David > >> >>> >>> Warning: digression follows... >>> >>> In the "Android" case (by which I mean the scheme where a control list >>> in a request is applied for that request), you will find yourself >>> looking ahead at the control lists further up the request queue, >>> deciding what needs to be applied with the top request. >>> >>> It's just my opinion, but even if you don't explicitly have a queue of >>> control lists, you do still have a kind of "virtual" control list >>> queue here. I find the whole thing easier to think about like this. >>> >>> The "Android"/ "Raspberry Pi" distinction ("Raspberry Pi" being the >>> behaviour I described previously) just becomes a question of when you >>> consume control lists from the control list queue: >>> >>> "Raspberry Pi" - just consume the top control list every time. >>> >>> "Android" - the control list queue entries need to record their target >>> request number, then you just consume control lists as far as >>> topRequest->sequenceNumber + pipelineDepth. >>> >>> Getting all the controls to synchronise correctly is, in my experience >>> anyway, quite tricky for pipeline handlers, so I do sometimes wonder >>> whether there's anything libcamera can do to help. But that's perhaps >>> a question for another time! >>> >>> David >>> >>>> >>>> >>>> Thanks, >>>> Barnabás Pőcze >>>> >>>>> >>>>> Thanks >>>>> >>>>> David >>>>> >>>>> On Thu, 12 Mar 2026 at 16:00, David Plowman >>>>> <david.plowman@raspberrypi.com> wrote: >>>>>> >>>>>> Add `Camera::queueControls()` whose purpose is to apply controls as >>>>>> soon as possible, without going through `Request::controls()`. >>>>>> >>>>>> A new virtual function `PipelineHandler::queueControlsDevice()` is >>>>>> provided for pipeline handler to implement fast-tracked application of >>>>>> controls. If the pipeline handler does not implement that >>>>>> functionality, or it fails, then a fallback mechanism is used. The >>>>>> controls will be saved for later, and they will be merged into the >>>>>> control list of the next available request sent to the pipeline >>>>>> handler (`Camera::Private::waitingRequests_`). >>>>>> >>>>>> This patch is derived directly from Barnabas's previous verion that >>>>>> implemented the same idea but with a single ControlList, rather than >>>>>> allowing multiple ControlLists to be queued up for consecutive >>>>>> frames. >>>>>> >>>>>> Signed-off-by: David Plowman <david.plowman@raspberrypi.com> >>>>>> --- >>>>>> include/libcamera/camera.h | 1 + >>>>>> include/libcamera/internal/camera.h | 1 + >>>>>> include/libcamera/internal/pipeline_handler.h | 7 ++ >>>>>> src/libcamera/camera.cpp | 61 +++++++++++++++ >>>>>> src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ >>>>>> 5 files changed, 146 insertions(+) >>>>>> >>>>>> diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h >>>>>> index b24a2974..93a484e4 100644 >>>>>> --- a/include/libcamera/camera.h >>>>>> +++ b/include/libcamera/camera.h >>>>>> @@ -147,6 +147,7 @@ public: >>>>>> >>>>>> std::unique_ptr<Request> createRequest(uint64_t cookie = 0); >>>>>> int queueRequest(Request *request); >>>>>> + int queueControls(ControlList &&controls); >>>>>> >>>>>> int start(const ControlList *controls = nullptr); >>>>>> int stop(); >>>>>> diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h >>>>>> index 8a2e9ed5..17dda925 100644 >>>>>> --- a/include/libcamera/internal/camera.h >>>>>> +++ b/include/libcamera/internal/camera.h >>>>>> @@ -38,6 +38,7 @@ public: >>>>>> >>>>>> std::list<Request *> queuedRequests_; >>>>>> std::queue<Request *> waitingRequests_; >>>>>> + std::queue<ControlList> queuedControls_; >>>>>> ControlInfoMap controlInfo_; >>>>>> ControlList properties_; >>>>>> >>>>>> diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h >>>>>> index b4f97477..c25213de 100644 >>>>>> --- a/include/libcamera/internal/pipeline_handler.h >>>>>> +++ b/include/libcamera/internal/pipeline_handler.h >>>>>> @@ -57,6 +57,7 @@ public: >>>>>> >>>>>> void registerRequest(Request *request); >>>>>> void queueRequest(Request *request); >>>>>> + int queueControls(Camera *camera, ControlList controls); >>>>>> >>>>>> bool completeBuffer(Request *request, FrameBuffer *buffer); >>>>>> void completeRequest(Request *request); >>>>>> @@ -76,6 +77,12 @@ protected: >>>>>> unsigned int useCount() const { return useCount_; } >>>>>> >>>>>> virtual int queueRequestDevice(Camera *camera, Request *request) = 0; >>>>>> + >>>>>> + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) >>>>>> + { >>>>>> + return -EOPNOTSUPP; >>>>>> + } >>>>>> + >>>>>> virtual void stopDevice(Camera *camera) = 0; >>>>>> >>>>>> virtual bool acquireDevice(Camera *camera); >>>>>> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp >>>>>> index f724a1be..f0244707 100644 >>>>>> --- a/src/libcamera/camera.cpp >>>>>> +++ b/src/libcamera/camera.cpp >>>>>> @@ -637,6 +637,16 @@ Camera::Private::~Private() >>>>>> * queued requests was reached. >>>>>> */ >>>>>> >>>>>> +/** >>>>>> + * \var Camera::Private::queuedControls_ >>>>>> + * \brief The queue of pending control lists >>>>>> + * >>>>>> + * This queue maintains a list of all the control lists that need to be sent >>>>>> + * to the pipeline handler with subsequent requests. The top item in the queue >>>>>> + * will always be sent with the next request going to >>>>>> + * PipelineHandler::queueRequestDevice(). >>>>>> + */ >>>>>> + >>>>>> /** >>>>>> * \var Camera::Private::controlInfo_ >>>>>> * \brief The set of controls supported by the camera >>>>>> @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) >>>>>> return 0; >>>>>> } >>>>>> >>>>>> +/** >>>>>> + * \brief Queue controls to be applied as soon as possible >>>>>> + * \param[in] controls The list of controls to queue >>>>>> + * >>>>>> + * This function tries to ensure that the controls in \a controls are applied >>>>>> + * to the camera as soon as possible. If there are still pending controls waiting >>>>>> + * to be applied (because of previous calls to Camera::queueControls), then >>>>>> + * these controls will be applied as soon as possible on a frame after those. >>>>>> + * >>>>>> + * The exact guarantees are camera dependent, but it is guaranteed that the >>>>>> + * controls will be applied no later than with the next \ref Request"request" >>>>>> + * that the application \ref Camera::queueRequest() "queues" (after any requests >>>>>> + * have been *used up" for sending previously queued controls). >>>>>> + * >>>>>> + * \context This function is \threadsafe. It may only be called when the camera >>>>>> + * is in the Running state as defined in \ref camera_operation. >>>>>> + * >>>>>> + * \return 0 on success or a negative error code otherwise >>>>>> + * \retval -ENODEV The camera has been disconnected from the system >>>>>> + * \retval -EACCES The camera is not running >>>>>> + */ >>>>>> +int Camera::queueControls(ControlList &&controls) >>>>>> +{ >>>>>> + Private *const d = _d(); >>>>>> + >>>>>> + /* >>>>>> + * Like requests, controls can't be queued if the camera is not running. >>>>>> + * Controls can be applied immediately when the camera starts using the >>>>>> + * Camera::Start method. >>>>>> + */ >>>>>> + >>>>>> + int ret = d->isAccessAllowed(Private::CameraRunning); >>>>>> + if (ret < 0) >>>>>> + return ret; >>>>>> + >>>>>> + /* >>>>>> + * We want to be able to queue empty control lists, as this gives a way of >>>>>> + * forcing another frame with the same controls as last time, before queueing >>>>>> + * another control list that might change them again. >>>>>> + */ >>>>>> + >>>>>> + patchControlList(controls); >>>>>> + >>>>>> + /* >>>>>> + * \todo Or `ConnectionTypeBlocking` to get the return value? >>>>>> + */ >>>>>> + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); >>>>>> + >>>>>> + return 0; >>>>>> +} >>>>>> + >>>>>> /** >>>>>> * \brief Start capture from camera >>>>>> * \param[in] controls Controls to be applied before starting the Camera >>>>>> diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp >>>>>> index 5c469e5b..1a87b28c 100644 >>>>>> --- a/src/libcamera/pipeline_handler.cpp >>>>>> +++ b/src/libcamera/pipeline_handler.cpp >>>>>> @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) >>>>>> ASSERT(data->queuedRequests_.empty()); >>>>>> ASSERT(data->waitingRequests_.empty()); >>>>>> >>>>>> + /* >>>>>> + * Clear out any unapplied controls. If an application wants to be >>>>>> + * sure controls have been applied, it should wait before stopping. >>>>>> + */ >>>>>> + data->queuedControls_ = {}; >>>>>> + >>>>>> data->requestSequence_ = 0; >>>>>> } >>>>>> >>>>>> @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) >>>>>> request->_d()->prepare(300ms); >>>>>> } >>>>>> >>>>>> +/** >>>>>> + * \brief Queue controls to apply as soon as possible >>>>>> + * \param[in] camera The camera >>>>>> + * \param[in] controls The controls to apply >>>>>> + * >>>>>> + * This function tries to queue \a controls immediately to the device by >>>>>> + * calling queueControlsDevice(). If that fails, then a fallback mechanism >>>>>> + * is used to ensure that \a controls will be merged into the control list >>>>>> + * of the next available request submitted to the pipeline handler. >>>>>> + * >>>>>> + * \context This function is called from the CameraManager thread. >>>>>> + */ >>>>>> +int PipelineHandler::queueControls(Camera *camera, ControlList controls) >>>>>> +{ >>>>>> + Camera::Private *data = camera->_d(); >>>>>> + int ret = queueControlsDevice(camera, controls); >>>>>> + >>>>>> + /* >>>>>> + * Don't worry about later request's controls overriding the ones >>>>>> + * sent here - the application needs to deal with that. >>>>>> + */ >>>>>> + >>>>>> + if (ret == -EOPNOTSUPP) { >>>>>> + /* >>>>>> + * Fall back to adding the controls to the next request that enters the >>>>>> + * pipeline handler. See PipelineHandler::doQueueRequest(). >>>>>> + */ >>>>>> + data->queuedControls_.push(std::move(controls)); >>>>>> + >>>>>> + /* Counts as "success". */ >>>>>> + ret = 0; >>>>>> + >>>>>> + } else if (ret < 0) { >>>>>> + /* >>>>>> + * The pipeline handler is claiming to support queueControlsDevice, >>>>>> + * but it has failed. This is an error. >>>>>> + */ >>>>>> + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; >>>>>> + } >>>>>> + >>>>>> + return ret; >>>>>> +} >>>>>> + >>>>>> /** >>>>>> * \brief Queue one requests to the device >>>>>> */ >>>>>> @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) >>>>>> return; >>>>>> } >>>>>> >>>>>> + if (!data->queuedControls_.empty()) { >>>>>> + /* >>>>>> + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is >>>>>> + * needed to ensure that if `request` is newer than pendingControls_, >>>>>> + * then its controls take precedence. >>>>>> + */ >>>>>> + request->controls().merge(data->queuedControls_.front(), >>>>>> + ControlList::MergePolicy::KeepExisting); >>>>>> + } >>>>>> + >>>>>> int ret = queueRequestDevice(camera, request); >>>>>> if (ret) >>>>>> cancelRequest(request); >>>>>> + else if (!data->queuedControls_.empty()) >>>>>> + data->queuedControls_.pop(); >>>>>> } >>>>>> >>>>>> /** >>>>>> @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) >>>>>> * \return 0 on success or a negative error code otherwise >>>>>> */ >>>>>> >>>>>> +/** >>>>>> + * \fn PipelineHandler::queueControlsDevice() >>>>>> + * \brief Queue controls to be applied as soon as possible >>>>>> + * \param[in] camera The camera >>>>>> + * \param[in] controls The controls to apply >>>>>> + * >>>>>> + * This function queues \a controls to \a camera so that they can be >>>>>> + * applied as soon as possible >>>>>> + * >>>>>> + * \context This function is called from the CameraManager thread. >>>>>> + * >>>>>> + * \return 0 on success or a negative error code otherwise >>>>>> + * \return -EOPNOTSUPP if fast-tracking controls is not supported >>>>>> + */ >>>>>> + >>>>>> /** >>>>>> * \brief Complete a buffer for a request >>>>>> * \param[in] request The request the buffer belongs to >>>>>> -- >>>>>> 2.47.3 >>>>>> >>>> >>
diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index b24a2974..93a484e4 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -147,6 +147,7 @@ public: std::unique_ptr<Request> createRequest(uint64_t cookie = 0); int queueRequest(Request *request); + int queueControls(ControlList &&controls); int start(const ControlList *controls = nullptr); int stop(); diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h index 8a2e9ed5..17dda925 100644 --- a/include/libcamera/internal/camera.h +++ b/include/libcamera/internal/camera.h @@ -38,6 +38,7 @@ public: std::list<Request *> queuedRequests_; std::queue<Request *> waitingRequests_; + std::queue<ControlList> queuedControls_; ControlInfoMap controlInfo_; ControlList properties_; diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index b4f97477..c25213de 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -57,6 +57,7 @@ public: void registerRequest(Request *request); void queueRequest(Request *request); + int queueControls(Camera *camera, ControlList controls); bool completeBuffer(Request *request, FrameBuffer *buffer); void completeRequest(Request *request); @@ -76,6 +77,12 @@ protected: unsigned int useCount() const { return useCount_; } virtual int queueRequestDevice(Camera *camera, Request *request) = 0; + + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) + { + return -EOPNOTSUPP; + } + virtual void stopDevice(Camera *camera) = 0; virtual bool acquireDevice(Camera *camera); diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index f724a1be..f0244707 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -637,6 +637,16 @@ Camera::Private::~Private() * queued requests was reached. */ +/** + * \var Camera::Private::queuedControls_ + * \brief The queue of pending control lists + * + * This queue maintains a list of all the control lists that need to be sent + * to the pipeline handler with subsequent requests. The top item in the queue + * will always be sent with the next request going to + * PipelineHandler::queueRequestDevice(). + */ + /** * \var Camera::Private::controlInfo_ * \brief The set of controls supported by the camera @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) return 0; } +/** + * \brief Queue controls to be applied as soon as possible + * \param[in] controls The list of controls to queue + * + * This function tries to ensure that the controls in \a controls are applied + * to the camera as soon as possible. If there are still pending controls waiting + * to be applied (because of previous calls to Camera::queueControls), then + * these controls will be applied as soon as possible on a frame after those. + * + * The exact guarantees are camera dependent, but it is guaranteed that the + * controls will be applied no later than with the next \ref Request"request" + * that the application \ref Camera::queueRequest() "queues" (after any requests + * have been *used up" for sending previously queued controls). + * + * \context This function is \threadsafe. It may only be called when the camera + * is in the Running state as defined in \ref camera_operation. + * + * \return 0 on success or a negative error code otherwise + * \retval -ENODEV The camera has been disconnected from the system + * \retval -EACCES The camera is not running + */ +int Camera::queueControls(ControlList &&controls) +{ + Private *const d = _d(); + + /* + * Like requests, controls can't be queued if the camera is not running. + * Controls can be applied immediately when the camera starts using the + * Camera::Start method. + */ + + int ret = d->isAccessAllowed(Private::CameraRunning); + if (ret < 0) + return ret; + + /* + * We want to be able to queue empty control lists, as this gives a way of + * forcing another frame with the same controls as last time, before queueing + * another control list that might change them again. + */ + + patchControlList(controls); + + /* + * \todo Or `ConnectionTypeBlocking` to get the return value? + */ + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); + + return 0; +} + /** * \brief Start capture from camera * \param[in] controls Controls to be applied before starting the Camera diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 5c469e5b..1a87b28c 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) ASSERT(data->queuedRequests_.empty()); ASSERT(data->waitingRequests_.empty()); + /* + * Clear out any unapplied controls. If an application wants to be + * sure controls have been applied, it should wait before stopping. + */ + data->queuedControls_ = {}; + data->requestSequence_ = 0; } @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) request->_d()->prepare(300ms); } +/** + * \brief Queue controls to apply as soon as possible + * \param[in] camera The camera + * \param[in] controls The controls to apply + * + * This function tries to queue \a controls immediately to the device by + * calling queueControlsDevice(). If that fails, then a fallback mechanism + * is used to ensure that \a controls will be merged into the control list + * of the next available request submitted to the pipeline handler. + * + * \context This function is called from the CameraManager thread. + */ +int PipelineHandler::queueControls(Camera *camera, ControlList controls) +{ + Camera::Private *data = camera->_d(); + int ret = queueControlsDevice(camera, controls); + + /* + * Don't worry about later request's controls overriding the ones + * sent here - the application needs to deal with that. + */ + + if (ret == -EOPNOTSUPP) { + /* + * Fall back to adding the controls to the next request that enters the + * pipeline handler. See PipelineHandler::doQueueRequest(). + */ + data->queuedControls_.push(std::move(controls)); + + /* Counts as "success". */ + ret = 0; + + } else if (ret < 0) { + /* + * The pipeline handler is claiming to support queueControlsDevice, + * but it has failed. This is an error. + */ + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; + } + + return ret; +} + /** * \brief Queue one requests to the device */ @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) return; } + if (!data->queuedControls_.empty()) { + /* + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is + * needed to ensure that if `request` is newer than pendingControls_, + * then its controls take precedence. + */ + request->controls().merge(data->queuedControls_.front(), + ControlList::MergePolicy::KeepExisting); + } + int ret = queueRequestDevice(camera, request); if (ret) cancelRequest(request); + else if (!data->queuedControls_.empty()) + data->queuedControls_.pop(); } /** @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) * \return 0 on success or a negative error code otherwise */ +/** + * \fn PipelineHandler::queueControlsDevice() + * \brief Queue controls to be applied as soon as possible + * \param[in] camera The camera + * \param[in] controls The controls to apply + * + * This function queues \a controls to \a camera so that they can be + * applied as soon as possible + * + * \context This function is called from the CameraManager thread. + * + * \return 0 on success or a negative error code otherwise + * \return -EOPNOTSUPP if fast-tracking controls is not supported + */ + /** * \brief Complete a buffer for a request * \param[in] request The request the buffer belongs to
Add `Camera::queueControls()` whose purpose is to apply controls as soon as possible, without going through `Request::controls()`. A new virtual function `PipelineHandler::queueControlsDevice()` is provided for pipeline handler to implement fast-tracked application of controls. If the pipeline handler does not implement that functionality, or it fails, then a fallback mechanism is used. The controls will be saved for later, and they will be merged into the control list of the next available request sent to the pipeline handler (`Camera::Private::waitingRequests_`). This patch is derived directly from Barnabas's previous verion that implemented the same idea but with a single ControlList, rather than allowing multiple ControlLists to be queued up for consecutive frames. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> --- include/libcamera/camera.h | 1 + include/libcamera/internal/camera.h | 1 + include/libcamera/internal/pipeline_handler.h | 7 ++ src/libcamera/camera.cpp | 61 +++++++++++++++ src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ 5 files changed, 146 insertions(+)